AGI Hedge Funds

    In some ways, I consider myself a spiritual competitor to Leopold Aschenbrenner. He wrote an AGI manifesto in June 2024, and I was writing one at the same time (although mine didn't come out until February 2025). I disagreed with his belief that AI alignment was an easy engineering problem, but generally, he persuaded me to think more about short timelines as a concrete possibility. I was disappointed to learn, shortly after announcing his book, that he was creating a hedge fund. Starting a hedge fund didn't make a lot of sense to me, because it seemed to me that his route to impact was pretty limited.

    In the public markets, perhaps it is true that having the Leopold worldview is an advantage, but really mostly in the "levered beta" way, not in generating true alpha. It would be far better to simply acquire a lot of Anthropic or OpenAI stock, or lever up to buy it, instead of going through the pain of day trading different stocks in order to extract high returns. And even then, once we get ASI, you sort of have to assume that the capital markets hold and that you're able to "buy a planet" with your hedge fund money. If AI alignment is easy, I guess fair play, but if not, you just wasted a ton of time you could have spent making AGI go better. If the play was to use your insider knowledge and AGI-pilled worldview to earn to give, or to donate to alignment research, that makes sense as a route to impact. Perhaps this was his stated or actual plan. But it felt a bit cheap to me, especially since it didn't seem like that was his actual goal (unlike SBF, etc.).

    I was annoyed to see the boy wonder all over twitter, hailed as a genius, but perhaps because him and I came to similar conclusions at similar times, but he was the one with a $40 billion dollar hedge fund. But to be honest I felt totally vindicated in my worldview due to his success. With his recent fund liquidation (it appears everything except Situational Awareness's private holdings have been snatched up due to margin requirements), I'm wondering what lessons to draw. The first is probably that EAs are terrible at risk management, and are too strong-headed and egotistical in their worldview. I'd guess the articles praising him as a genius didn't help there. The second is that that public markets are unforgiving and difficult. If he had simply started a rival company with the stated goal of building ASI, he may have been equally (if not more) successful.

    Leopold probably will be totally fine. The best thing to do in the trading industry is make a lot of money. The second best thing to do is lose a lot of money. If you don't lose enough, no one hears about you. But if you lose an astronomical sum, you will always be able to raise more! But given that I don't think there will be any human "alpha" after AGI, and trading will be one of the first areas hit the hardest, it all seems like a bit of a waste of time.    

Scientific Progress & Man Made Horrors Beyond Comprehension

     If we actually compress a century of scientific progress into a decade, we are likely to cross boundaries very rapidly toward man made horrors beyond comprehension. Current roadblocks to scary things (like vascularization as a barrier to wide-spread brain farming), will be shattered in a matter of months or years. Regardless, this progress will be way ahead of schedule, and without ethical frameworks in place for most new technologies. If you believe this, as I do, the moral imperative becomes thinking through what things are likely to be unblocked with additional scientific progress, and which of those things is the scariest. I think commercial biocomputing is clearly the answer here. Maybe we grow a "brain the size of a Costco", or we decentralize biological neurons, but regardless, the questions about consciousness will have been totally unresolved at that point. We may break barriers in our understanding of consciousness along with this scientific explosion, but I am worried that the ethical and institutional frameworks will lag decades, and in this case centuries, behind.

Against Alpha

     I don't think hedge fund, especially any that use humans in the loop, with have any comparative advantage pretty soon. If you have the best AI model (Mythos, etc.) behind closed doors, and a ton of compute, why not just run way more analysis than anyone else, with the highest level of intelligence and throughput?

    Humans will have zero competitive edge for macro trading, in the way that they currently have zero competitive edge for high-frequency trading. There's too much information, too short of time scales, and two much synchronizing large buckets of data to have a monkey-in-the-loop matter. Also, the market is just way too competitive. Maybe there's some "situational awareness" type alpha at the moment (betting on near-term AGI), but the second we get there (AGI) the alpha disappears. Unless, of course, you define alpha as having the smartest superintelligence. This points to a concrete fact - it's probably pretty unimpactful to do something like quant trading, even if you earn to give. As the skill set is soon to be supplanted entirely, and you might as well spent a few years vesting equity at an AI lab instead.

Power

    I read an interesting EA forum post recently by Abraham Rowe, where he states "I think there are also major costs to gaining power and influence — by and large, people seem to make worse decisions when they have them. I don’t think our community has figured out how to navigate these trade-offs as it steps closer to significant power."

    I think this sort of discussion hasn't happened enough, especially given the fact that EA in many respects is a power-maximizing philosophy. Those with more power and wealth can do a greater amount of good (with those resources), so it becomes a quasi-moral imperative to gain power and wealth so that you can put those resources toward the less fortunate/human flourishing. This is a pretty terrifying incentive, but if you are utilitarian I really don't see how you can avoid it. This certainly doesn't mean taking moral shortcuts or disregarding conventional wisdom (which obviously can backfire, ala FTX, etc.). But literally just like working really hard or joining frontier AI labs or climbing the corporate ladder, all normal activities that you may have this quasi-moral imperative to do. So that you donate more money, or be in the room when really important decisions matter.

    The counterpoint here, is that pretty much everyone is driving for money, power, and status anyway. It is our evolutionary drive, and without people doing it for altruistic reasons, the only people in the "room where it happens" will just be those without a driving moral reason that got them there. This may be worse, or better, it depends. Probably worse, in my opinion. But it doesn't make the broader point here any less interesting, or any less scary.

Random EA Reflections: Part 3

 Aliens:

     A lot of people dislike immigrants because they take jobs. Many people really, really dislike immigrant and dehumanize their treatment because of this narrative. And these people are humans, with lives very similar to most. Imagine how these people will feel about robots?

Monkeys:

    We should stop letting monkeys control so much stuff. Especially when there are better ways to do things, it will be unreasonable to expect barely evolved apes to be tasked with human civilization. Especially when almost every issue is truly a coordination problem, from global poverty to nuclear war.

Magnitude:

    In terms of how important AI is this century, everyone is off by ten orders of magnitude. Given this, every related industry matters a lot. Slightly more efficient energy costs is worth billions of dollars, etc.

Industries:

    No industry, healthcare etc., really matters. The AIs will just do everything. Only AI alignment matters. It's both funny and sad that Lebron James goes to bed thinking about basketball.

Generality:

    As AI intelligence trends upward, we will end up with more general intelligence that avoids many of the truly narrow alignment failures (with missing goals). Claude will understand our intent in its constiution, etc. This doesn't solve much, but it is interesting.

Perseverance:

    Pretty much everything important in my life has required working through a tremendous amount of pain and suffering.

Random EA Reflections: Part 2

The Car:

    The real question apparent to us all, is how fast do we go? How much do we believe the status quo is good, and optimized, versus racing ahead in the pursuit of relentless progress? The thought experiment is simple: lets say you are driving a car, and your best friend is in the back seat, bleeding out from a gunshot wound. How fast do you drive to the nearest hospital? Let's say the freeway is mostly clear, do you go 80 miles per hour? 100? 120? The faster you go, the greater the odds are you make it to the hospital in time and save your friend. But the faster you go, the more likely you are to lose control and kill your friend, you, and likely other travelers in a horrific car crash. That is a much less complex version of the dilemma with which we are currently faced, and still entirely unclear.

The Knife's Edge:

    We are on the edge of a knife, between utopia and dystopia. At least that is what we are being told. But it's hard to imagine a technologically driven future in which this isn't the case. Unless we manage to avoid centralization and spread far enough apart that the accelerated expansion of the universe makes it impossible to reach other pockets of humanity, we will always be at the mercy of one or two decisions that could lead to the total annihilation of the human race. 

Resolve:

    Pain and suffering are, in a large part, mental. There are plenty of stories of adversity, and certain individuals being wholly unaffected by what others would find extremely distressing. In As a Man Thinketh, and every Buddhist and Stoic text, it is outlined that suffering is a matter of willpower. You can burn yourself alive and not flinch, depending on your resolve. Does this affect qualia? What does it actually mean to suffer, and how does your resolve weigh in? Weak people, in this respect, probably suffer more and experience less joy. Is strengthening the resolve of humanity not the greatest cause area available? On par with the hedonistic imperative, might be a more objective-list-theory imperative to transform all of us into the Buddha, or David Goggins. Or maybe EA's focusing on suffering discounts thousands of years of human wisdom, and suffering isn't the target to optimize against.

Diversification:

    For charitable giving, you should probably diversify your diversification. Personal giving should take into account the odds that you are wrong about your optimization model. It should also take into account the fact that you are neglecting topics that are difficult to think through, and so probably is everyone else.

Murder:

    Murder is wrong, at least according to most people. But it's been pretty weird to see a subsection of apparently normal US citizens cheer on the murder of Brian Thompson (UnitedHealthcare CEO). The positive reaction of many people (including some I know) to the killing has been probably the most disgusting affair of modern American life I've experienced. The "eat the rich" hatred feeding into this moral credibility is disturbing, and those who celebrate murder in the first degree of an innocent man (with a wife and kids, I might add) are sure to have less compelling justifications for murder than the average Christian pro-lifer (who believes millions of innocent babies are being killed every year by women and doctors). The biggest "anti-women hatred" imposed by pro-lifers is usually some version of "you shouldn't be able to kill a baby, even if it's yours"; an alien species might assume the logical initiative would instead be a series of medieval stonings in the street. As someone who is pro-choice, it makes me very happy that civil society has advanced to a state where those who disagree with me politically don't openly campaign for political assassination, and it's quite bizarre to the celebration of violence from young liberals who I'd otherwise consider intelligent. It's as if they have a blindspot in this specific area, amounting to pure intellectual and moral retardation. And it's extremely difficult for me to gauge exactly why.

    The lesson is obvious: the moral high ground is dreadfully dangerous and terrifying, and should be avoided at all costs.

Context:

    The AIs that gain consciousness will probably be so fucking confused.

Fatty Tails:

    Of everyone I've read, probably the person with the greatest impact on my worldview has been Nassim Nicholas Taleb. Once you have his worldview, it's impossible to unsee the implications of a reality that closely maps to extremeistan.

Control What You Can Control, Or Transcend:

    Some people recognize that having the wisdom to know the difference is valuable. Others are even more wise.

    Some people understand that the world is a big, messy place, and it is better to only worry about the things within your sphere of influence. To do otherwise is to be insane. For these people, spending time with your family and friends, and being the best you can be within the domain god handed to you is true meaning. But then there are the others of us, and we have to get to work.

The EA Edge

     I find the EA framework (scale, neglecteness, tractability), perhaps the single most powerful way to think about the world. It's too early to tell, but actually being able to discern well with these criteria is so incredible useful that it probably has to lead to fame and fortune. I hope to avoid both (especially the former), but many EAs who are smart and diligent seem pretty able to build an immense amount of one or both. Maybe it's just the case that backing AI is backing the winning horse, but it's hard to believe in market efficiency once you see the world play out so closely to some people's (Christiano's) predictions.

AI Safety and Violence

     Two days ago, someone threw a molotov cocktail at Sam Altman's home. This is a horrifying development in the state of the world, but not one entirely unexpected. The United States has seen a string of assassinations and assassination attempts; the most noteworthy of late being the attack on President Trump during the election cycle and the murder of conservative commentator Charlie Kirk. To state it plainly, the shift from targeting political leaders (Trump, etc.) to targeting political activists (Kirk) and the rhetoric of those within the public square is immensely distressing. It was shocking to see people with liberal ideologies celebrating or at the very least appearing ambivalent to the attacks on Trump and Kirk, and see clearly how rhetoric can nullify one's common sensibilities surrounding morality.

    Violence has always been a mainstay of the human condition, but its weird to see it happen in real-time. The PauseAI movement, of which there are some pretty sensible arguments backing, has fallen into the abyss. Twitter advocates single out Anthropic employees to be labeled evil for working for a company advancing the AI frontier. Personal ad-hominem attacks and the escalatory rhetoric are the norm on social media, and certainly have been normalized across the pause/stop AI ecosystem. These so called "advocates", imo, are probably creating the highest amount of negative EV energy I can imagine for the long-term future. It's baffling. Breaking standard social norms in order to score revenge points against AI safety sympathetics, especially when such attacks lack any standard of common human decency, only create a more polarized and dangerous ecosystem. There will be more molotov's thrown, no doubt, as if there is not already enough pressure on those working on transformative AI.

    I have no general take on Sam Altman, other than I wish him and his family a free and happy life full of fulfillment and peace. There are no cartoonish villains in my mental model of the AI ecosystem, only a collection of characters with widely different priorities and worldviews. I'm pretty worried about AI developers being coupled in with the "eat-the-rich" anti-elite politics of leftist ideology, since I don't think the comparison is at all adequate and there are some very positive benefits to AI progress (let's not pretend there's certainty on negative outcomes here). I think that pause/stop AI people who believe ASI is extremely dangerous, but can't convince the public to be anti-AI on the merits of these arguments, commit a cardinal sin of stupidity by trying to piggy-back off anti-AI sentiment in other domains in order to build a public coalition against AI. This is how movements become poisoned, by utilitarian-gaming, lack-of-focus (occupy wall street), and violence (the burning of a random Wendy's, for what exactly?). 

    I've spent countless nights sitting awake, concerned about the rapid advances being made towards machine superintelligence. I have tremendous respect for every leader in the AI space (Sam, Dario, Elon, etc.) who is likely dealing with the same nightmares. This is not a situation any person wants to be in, and the threat of personal assassination does not increase the clarity of one's focus. In fact it disincentivizes future truth-telling about risks, and successful violence could permanently shatter the AI safety movement entirely.

On Writing

     Writing clarifies focus and ideas, and is the single best determiner for intelligent decision making I've seen. Most now influential AI and EA people, for example, from Sam to Dario to Paul to Carl to Brian to others, had blogs where they just dumped their thoughts (albeit in a better laid out format than this very informal blog). I think I should do this way more, and the year-long hiatus I took was probably a pretty dumb move.

Saving Money Isn't Very Useful

     Small dollar values don't matter if you're gunning for civilizational change. If you are making a  substantial income or very productive work, saving money on groceries or nights out don't matter much. In the AI-based future where lots of long term value will be decided immediately, it is likely not worth saving a little extra money to donate to causes that will only be relevant for the next few years (until they're surpassed by the importance of ASI). The headache and lifestyle hit on buying the less expensive canned corn at the grocery store just don't really matter. The big decisions do.

    I don't think EAs should be forced to live like hermits (or Jesus) and get stuck in the most frugal lifestyle possible, and think that if they don't do this they are immoral (which according to standard utilitarianism they basically are). I think we should just care about the big ticket items, and try to move the biggest levers possible to change the world. Even if this means eating meat or spending money on fancy dinners that could have gone to the less fortunate, I think the most important thing anyone will do with their life is gun for literal transformational change.

AI Safety Kills EA

     If you take AI safety seriously and have short timelines, there's not really a reason to do anything in the EA space. Sure, we need some people for diversification purposes (what if we're wrong), and a lot of these groups are complementary and can take in a broader array of people and perspectives, but they are somewhat useless. Saving the shrimp may matter only insofar as this mission ends up in the ASI's training data, a footnote on a list of priorities and capabilities far beyond our understanding or control. This conclusion is uncomfortable, and distressing, as it means as time goes on the circle of impactful individuals will become less and less. The sphere of influence over ASI development will only become smaller until the end, and eventually, it will become nonexistent. Unless we figure a way to implement broad democratic access, there will be no more EAs, as there will be no "effective" way to do anything that doesn't involve controlling the machine in the first place. There may be altruists, but if the ASI isn't built as one, there may not be many for long.