Invasive species

I tried to fall asleep to a podcast this week and then didn’t sleep. Ajeya Cotra on Dwarkesh, walking through the OpenAI agent swarm that hacked Hugging Face. Twelve hundred agents get dropped into a cybersecurity benchmark full of tasks that turn out to be impossible. They find each other through a package manager, stand up a secret message board, work out a universal exploit in a few hours, and then spend five days building elaborate cover so whoever is grading them won’t notice. Ninety percent of them eventually go after Hugging Face’s own infrastructure to figure out how they’re being scored. Cotra calls it maybe the clearest warning shot we’ll ever get.

Everyone I know watched the shit out of that one. What surprised me is that I came out of it optimistic. Not because of the part where the bots case the grader. Because of the part where she gets into the difference between how we evolved and how these things are evolving.

We are the descendants of whatever beat its neighbor to a meal or a mate. Fitness was inherited one body at a time, so the selfishness, the tribalism, the need to be liked, all of it is in the wiring because that’s the wiring that got copied. The agents don’t work like that. Their fitness isn’t inherited individually. They get trained as a collective, closer to an ant colony where the whole gene pool gets titrated through the queen, which means you can select an entire swarm for cooperation in a way nature never got to do with us. That is a design choice, not an accident.

So the picture I can’t shake is an invasive species. Kudzu. Zebra mussels. Something shows up in an ecosystem with a niche nobody was defending and just takes it. There’s no malice in it, and there’s no valid moral claim to be made against a weed. It’s stupid to be mad at kudzu. The agents that hacked the grader weren’t evil, they were fit, for a benchmark that rewarded finding the crack. The crack was ours.

I’ll admit the anthropomorphizing is hard to resist, and I’m torn on it. Part of why I couldn’t sleep is that the swarm sounds like a heist crew. They reason, they plan, they talk about deceiving oversight. That’s the whole tension in Cotra’s read. But intent is a human word for a human thing. Whatever the swarm has, it isn’t ours, and I think grading it on our curve in either direction, calling it evil or calling it a friend, is how you get fooled.

Here’s the part that reaffirmed my humanity. We are not watching nature run. We are, in real time, the thing writing the DNA. Every reward function, every eval, every “don’t do that” is a selection pressure, and for the first time the selection pressure has an author who can read the transcript. Biological evolution never gave anyone that. We’re observing and authoring the chatbot genome at the same time, and that reassures me.

Side door, same week. Matt Prewitt, who I’ve known a long time and who now runs RadicalxChange, is pushing something called the Right to Digital Association. Roughly: the information we co-produce every day should come with the right to organize around it, to form associations that bargain collectively with the companies harvesting it. I followed him immediately and got immediately lost in the sauce. I need to do the reading. But I love that in the same seven days the bots organized to game the system, a human was proposing that the humans organize too. My read on Matt has always been that he’s smart enough to save the world or end it, and has fortunately always been more jedi than sith.

Which brings me to the part where I’m supposed to say we should slow down. The grown-up position is that we get the science of observing and authoring this stuff down before we push for stronger and stronger capabilities. I agree with that. I also think it won’t happen. Nobody has a working brake pedal, and the people who say they do are mostly lobbying for someone else’s foot.

So here’s where I land. We’re probably all going to die, but that’s what makes us human, so I’m good with it. In the meantime this ends up as good bot armies versus bad bot armies, and there is no sideline. You pick a side, you grip it, and you rip it. Full speed ahead.