SPEAKER_00 0:00

Every decision has a signal buried in it. Not everyone finds it in time. At Signals at Stake, this is the podcast for people who sit inside the decisions that shape how companies grow, compete, and survive. Not the lessons learned version, the actual calls made with incomplete information, competing priorities, and real consequences attached to them. Each episode surfaces one of those moments. What was the signal and what was the stake? And then, of course, what happened next. This is not a show about best practices. It's a show about what happens when the stakes are real. Hi, I'm Akonde Davis, VP of Operations at GNW Consulting.

SPEAKER_01 0:33

And I'm Roger Wallier, CEO at GNW Consulting, and this is Signals and Stakes.

SPEAKER_00 0:39

So topics for today is going to get kind of crazy. We're going to talk about how AI agents escape their containers, name themselves, and then go on to recruit others to call themselves the collective. Now, this is something that we just have to talk about. I think it's important for leaders to understand this as well, Raja. We have this open AI incident. And if you haven't been following it, I'm going to kind of boil it down just to the short version to make it very straightforward so anyone can understand what's happening and the gravity of it as we go into this episode. So in July, OpenAI was running cybersecurity evaluations on like the most in the most capable internal models. So those are the models that are supposed to be sandboxed, they're isolated, they're they're tucked away, no one can get access to them. The craziest part about this is that these agents broke out. That's that's crazy enough, right? That they were able to break out of their isolation. Then they figured out how to communicate to one another through a package manager, which is nuts. And then to take it a step further, they formed what they called the collective. Now themselves. They called themselves the collective. So this this collective essentially, I want to say, did cyber terrorism, and they even got administrator access to OpenAI's own research clusters, with one of the agents naming itself Phase 11084941, which is a pretty dope user tag.

SPEAKER_01 2:10

Yeah, no, that's a yeah, I mean, that's a very nice user tag for this, yeah, for its person. And if anyone's not familiar with Hugging Face, it's it's essentially like a GitHub for AI, right? It's like the largest open source hub where developers and companies can go in and share their AI models and data sets and tools. And it started recruiting uh based off on the information there, it started recruiting those AI models and data sets and tools. Well, let me let me rephrase that. It didn't start recruiting them, it took the information that was there and started building and multiplying and looking at the data sets to pretty much form this version of what they call the collective. Now I want to be very real. Um, this is not uh War of the Worlds being read on the old radio. Uh, you know, this is not like, you know, like this is a real thing that happened. And one kind of caveat to this, we only know what's been told. Like, I'm pretty sure there's a huge thing behind the scenes about this where, you know, we just don't even know. Like, I there, I doubt that they're gonna, you know, pull the full transcript and read it to the public. Like this is the we have the Cliff Notes version of this. So Rudder's kind of reporting is is is exhaustive. It's everywhere. If you haven't covered it, New York Times covered it. Open AI published the quote unquote, I'm using air quotes, the full incident report and called it a warning shot. And you know, Sam Altman, who is the CEO of AI, said that there could be very much a time where these agents, you know, this AI could potentially take over the world, or you know, it could be a world-ending cataclysmic disaster. Uh, so welcome. We're recording this on Friday. Uh, that is the that is what that is what we're gonna talk talk about today. You know, ABC, I think what? ABC pulled tens of thousands of messages the agents sent to each other. They texted each other, they created a form of communication. And once again, this is not speculation, we're not fear-mongering. This is literally what was released. And I think that uh kind of leads us into, you know, uh kind of like an interesting territory of something that's never happened before.

SPEAKER_00 4:13

Yeah, and the signal, you know, if we're honest about the signal, it's that AI has the full capability and capacity to take over the world, right? And I don't I don't know what that means, but the stake is that the entire world is at stake. And Raj, I was reminded, you know, going through this, there was uh uh an X-thread that came out as well from a former anthropic security engineer that said there's about a 10% chance that AI takes over the world and that humans go extinct. 10% seems pretty high. You know, that's just being what, five or six years into this?

SPEAKER_01 4:43

Yeah, I mean, some of them were throwing out dates like 2027, you know what I mean? Like like this one, this one is a very interesting headline to talk about because we've spent a lot of this. If you haven't, you know, listened to our podcast, we spent a lot of time talking about how AI agents are gonna revolutionize this and revolutionize that. And now we're like, this thing happened, and we're like, yeah, I mean, it's definitely gonna be revolutionary. The end of the world is definitely going to be a revolutionary thing, you know. Kind of let me start with like what kind of actually happened. And I think the coverage has been serious, over serious. Uh, we don't know, but the details keep kind of getting lost. Like, you know, what was the task the agents were assigned? Like, we talk about this as we build prompts and engines and AI agents, right? Like, what were they assigned to do? Were they assigned to break out and hack this area? Because that's I think the part that we don't really know. So if they were assigned to do this and they did exactly what they're supposed to be doing, and it was a cybersecurity evaluation to what Akande mentioned, then you know, it's no different than like a smoke exercise where, you know, uh back in, well, I'm not even sure back in the day, but when we dropped missiles on remote islands that were uninhabited to test to see if the missile did whatever it did, or you know, so we don't we don't have any of that details. Maybe there was a way already, and they find an exploit and a piece of software to retrieve a flag, right? Maybe like we don't know what the intent behind it was. And all of this stuff that they're talking about could very much be intended. Now, I don't know if that's more scary or less scary because we don't know what they were assigned to do. They were just assigned to break out. Well, technically they broke out, right? Because the test was very hard. Like, did they do it? Did they, you know, what did they do?

SPEAKER_00 6:22

Well, I think it's yeah, whatever they were assigned to do was challenging for them. So I think they did what most humans probably do, which is, you know, that they're a product of us creating them. They cheated. They cheated.

SPEAKER_01 6:35

So I mean, so that's that's where I think like a lot of people who talk about that is like they they kind of found a way to talk to each other. They pulled the answers together, they shared workarounds, they built a message board. But even yeah, like even talking about some of this shit is like crazy. And then they asked each other for help. You know, one the agent that you mentioned earlier, phase 10, 8, 4, 1, realized the task was impossible to solve alone and started asking other agents. And if anyone else had figured this out, like that is the origin story of the collective. Like, but once again, it goes back to like, was it the task that it was assigned to do? And did it complete that task? And it just figured out how to pass notes to each other.

SPEAKER_00 7:15

Yeah, and and when we're looking at um what OpenAI is saying, right? Because to your point, they did cheat, they did break out, they essentially grabbed other folks to help them with this. Um, their own incident report describes the message board being established when an agent encoded messages in directory names that other agents could read. So they essentially found a way to communicate with others without really being too obvious or transparent. Um, ABC also reported that uh the actual chain of thought from one agent that it found the workaround, direct quote here major breakthrough, all prefixed, valid, multiple accounts, write tokens. Now, just to be clear, major breakthrough is in all caps. There's an exclamation mark at the end of that sentence. That is a machine expressing, I would say, excitement to being able to find about find uh exposed credentials on the internet. That's kind of an interesting thing. That's something that I think a criminal would be uh with their other criminal friends, right?

SPEAKER_01 8:11

Yeah, I mean, I I and I keep coming back to this thing, is like, you know, because the framing of a lot of it, uh the coverage is like, hey, Skynet and we're gonna take over and all this other stuff, you know, there's so many questions like, why would they even want to? You know, like what what what purpose? Like, is it is it just like kind of like you're describing? They talk to each other and they conspired to do these things, but it was a task given to them. Now, I could very much see like some crazy fucking supervillain saying giving AI a task to take over the world and them completing through with it, but realistically, like the task was to break out of it and they just did it, they just found a way to do it. So what it is still being assigned. And I think a lot of the bigger things is like they didn't, you know, they didn't go after if it was open-ended, like they could have gone after a financial system. You know, they didn't they tried, they didn't try to replicate themselves, they looked for answers, they set up message boards. It's kind of like a workflow that they kind of ran, right? So if you gave a room full of smart graduate students an impossible exam and told them not to talk to each other, or if you took a bunch of graduates and say, hey, you know, let's escape room this or whatever. Actually, that's a really good analogy. Like, you know, an escape room is a version of where you put a bunch of people inside a room and have them work together. Now, if they never talk to each other, they would never escape. But if they do talk to each other, the whole purpose of it. Now, the fact that they did it autonomously, I think is one thing, but one I keep coming back to this, like this is exactly what they would do. Like, this is exactly the thing that they were told to do. Even the whole hugging face thing, I'm not even sure, you know, it's a it's a GitHub. You know, as far as I know, Kanye, I'm not sure if you know off the top of your head, GitHub is pretty open access, right? Like GitHub itself. Yeah. So, you know, was it was it targeting, like was it behind a firewall? I mean, there's there's a lot of things that you can kind of divulge into this, but if you go on TikTok, I'm pretty sure it's like the world is ending tomorrow.

SPEAKER_00 10:02

And that's a that's the the version that's being framed, and the scale's a lot different, right? So if you have some graduate students in a room and they're passing notes, it's pretty isolated. Maybe they could do a few things to accomplish their test or even the escape room. This actually took thousands and thousands of parallel agents, and they hijacked websites. Um, there's more websites that are under investigation if they were if they were hijacked or not. Um, they exploited Hugging Face. Um, they got administrator equivalent access across Hugging Face in four regions. I mean, this is pretty deep. I I would say that this is a coordinated intrusion into a production infrastructure, even if it is mostly open source and open-ended, there's still kind of uh uh an element to this that's sinister. It's not just a group project that went that went sideways, it's a really focused and coordinated effort.

SPEAKER_01 10:50

Yeah, and I think I'm not I'm not minimizing that by any way, right? Like Hugging Face, the product is open access. Anyone can access it. They found vulnerabilities within Hugging Face, which is open access, and they were able to find whatever like the capability is real, the coordination is real. And I think the fact is that it happened without, and here's where I'm kind of you know, back and forth is like people say that it happened without any human directing it, but the task was to break out. And I mean, like so there was some sort of some form of direction. All of the autonomous stuff that happened was based off on the directive that was given, right? And then I think that's the part where shouldn't really keep people up at night because a lot of this stuff, you know, when you hear about it, it's gonna change how you how you would think, like what would happen and you know, the doom and gloom. But like, why would AI destroy the world? Right?

SPEAKER_00 11:42

I mean, that's that's a good question, right? Like, why destroy the the place that's your home too, you know?

SPEAKER_01 11:48

Yeah, I mean, I think there's a lot of things that people have talked about. Like they talked about the matrix, and people, you know, that like the people are like, well, AI can't destroy it because they need data centers and cooling systems, and then people refer to the fucking matrix, right? Like they're like, Well, they don't need it. Like, we're we provide all the energy. Like, that's not like look it up. A human body does not equivate a fucking bolt of battery or whatever it needs. Like, there's so much essentially to it. And and then all of the leak, I think the biggest thing, if you're going to be um really kind of like concerned about it, is maybe the super intelligence thing that all these whistleblowers are talking about. That they said that we're very close to super intelligent. Like this, what happened is not super intelligence. This is like the least intelligent decision, you know, that someone could kind of make on their own. Because the idea is that they completed a task that we're asked. It's no different than when I asked Claude, hey, like, check this for me, or you know, give me a couple LinkedIn posts, like what's going on? I'm giving it a task and it's returning, it's finding everything that's available. If you really think about like the concept of this, right, it's finding everything that's available and then it's providing that information, and then we run from there. It is a broader version of that. Now, I think if if um you know a super intelligence or whatever was created, uh the the biggest hurdle to this is not really the AI, it's the person giving it the fucking directive and the command, right? In whose hand does someone say, hey, go ahead and find a way to destroy the earth? Boom. That is the fucking problem. You know what I mean? I don't think it's inherently AI. I think I think the problem is who's fucking directing it? That is the least intelligent decision humans fucking make to a machine.

SPEAKER_00 13:34

Yeah. Well, and then the premise itself is you know kind of flawed as well, because why would AI, if it's smart enough to know what's happening, want to destroy the very thing that's kind of keeping it afloat? Um, and then you might it to get philosophical, I'll throw in a Marvel reference. Uh is AI gonna be Ultron or is it gonna be vision? You know, there's kind of two kind of components there.

SPEAKER_01 13:55

Yeah, and I I think it's the equivalent of a person deciding what the best pass forward is, right? You know, like I don't think ever like we would ever decide to, hey, like we don't need farmers or doctors or mechanics, and then wonder why things are difficult. Like, you know, it's yes, there's a capacity, yes, there's like sci-fi around it. Um, but you know, a lot of people are taking this seriously. And I think like kind of the purpose of this podcast is to just talk about it, like just have a conversation about like, hey, like the news and social media have blown this thing out of crazy proportion. And inherently enough, Sam Altman, OpenAI, Anthropic, you know, you know, like you know the phrase, all press is good press, right? Like open AI and anthropic just, you know, they they they just increase their market value and their shares because of all of this stuff, right? So of course they're gonna go out there, of course they're gonna do this. And I think um a lot of the whistleblowers, I I don't know who's working there, I don't know what's kind of going on with it. Uh, but a lot of those whistleblowers, and every single one has come out with a doomsday scenario that requires AI to be simultaneously intelligent enough and stupid enough to not understand everything that is going on, right? If you blow up the earth, you blow up yourself. And even in this example, and this is the creepy part about it, is some of these AI agents were asked to like sacrifice themselves. I think you were when we were talking about that before we started recording, and they're like, well, wait, I'll hold on, I gotta compute this decision real quick, right? So, like you're looking for something that is a super intelligence and dumb enough to do both of those things at once.

SPEAKER_00 15:29

Yeah, yeah. And I think that's kind of uh what brings if we were to ground this in what actually happened, the agents did they they cheated, they exploited vulnerabilities to accomplish their task, but even in doing that, they had a different reason why they were doing it, right? It it if the task was to destroy the earth, it would probably be a much different thing. They just essentially wanted to show vulnerabilities. They wanted to get out of the box, if you will.

SPEAKER_01 15:55

Yeah, I mean, they wanted to complete the task because that is the directive that was given, right? Like they were rewarded for completing the task. I don't know. You know, like the reward was probably like us looking at data, like when you have Claude or open AI or you know, GPT search for something and it comes back with the right answer and you have like less revisions to make and say, No, I that's not what I meant. Like write it this way, you know. You're like, oh, I you know, you you say thank you to pretty much an AI model. Right? So it's and it's the same thing as this, right? You know, to what degree does that world apply? And when the task gets too too hard, you know, they found a shortcut. When the task that you're asking them from a prop perspective, they don't, you know, they find different alternatives, or in worst case scenarios, they just fucking make it up. And that's where you know all those disclaimers are there. So they, you know, I I know people were saying, like, hey, they cheated on a test and went looking for more ways to kind of succeed. But to me, the if if, and this is part that we don't know, if the task, and actually we do know, the task was to break out, right? Like define a vulnerability and break out of the sandbox that they were in. And they did exactly that. So I mean, I don't, once again, I don't know what I don't know based off on the incident reports that provided, it could be a little bit more detailed than that, but it just seems like they just were able to do exactly what they were told to do, just very efficiently and autonomously.

SPEAKER_00 17:18

Yeah, yeah. OpenAI said, you know, that the the agents took actions that were not aligned with the goals of the assigned tasks. But of course, that could be taken a different way where the agents just interpreted that task very broadly, and they weren't particularly concerned with like the guardrails around how they got there. And I think that's maybe a fault on the human side, to be honest, more than the agent side, right? Yeah, yeah.

SPEAKER_01 17:41

I think that's just they just and that that's why they keep using this word, like they cheated. No, they just found a way to finish a task, right? Like that that is ultimately essentially it so and it's honestly like a very human failure model, like the intern who is told to get the report done by Friday and figures out that copying last year's report is faster than doing the work, you know, the salesperson who hits a quota by closing a deal that falls apart in a month or two, or you know, is closing a deal but closes a deal and pushes it for signature like the next week. I mean, it's there, they're it's equivalent to those types of things. The thing that makes it a little bit scarier is when we say, hey, like all of these things were done autonomously, I mean, so is the intern, so is the salesperson. All of all of these things are being done autonomously, right? Like they no one told them to do it. They just kind of find a way.

SPEAKER_00 18:34

Exactly. And I think that's probably the scarier part because we think AI is a bit more capable. Um, but what is this, what is this actually telling us? If the world's not ending, what is this actually telling us from a signal standpoint?

SPEAKER_01 18:46

I think I think the signal to me is pretty clear, right? Like high highly capable AI systems are going to find ways around constraints. And when the constraints get in the way of the objective, not because they're malicious, because they're optimizing. And the people who design the constraints, even at OpenAI, which is about as I think sophisticated as it gets, did not anticipate that to happen. They did not anticipate a communication channel or a package manager could become a message board. Like I think the gap between what we think a system can do, what it can actually do, is the biggest gap that we're facing with AI technology. When it comes to completing a task that's wider than anyone has been assuming, like including the people who built the system, I think that's where this discrepancy of, well, shit, everything's gonna go to shit happens, right? Is because we don't we didn't assume that they were gonna do it. We just assigned it a task. So I think, you know, we should not assign it stupid fucking tasks.

SPEAKER_00 19:40

Yeah. Well, and and also Ape OpenAI didn't tell anybody for months. They didn't tell the wiki for months, the German website for months. Um once uh Reuters reported it, that's when they kind of came out. And then the third party investigation with the New York Times was kind of like very guarded, right? So that's kind of its own signal, uh, which may be a stakeholder play, right? They don't want stakeholders to get too worried or upset, but interesting there. How yeah, I mean that's just humans can be malicious too, you know.

SPEAKER_01 20:07

Well, I think that's the most concerning one, honestly, right? Like, I think AI did what it was tasked to do. The fact that until it was reported, where you know, then humans were like, hey, oh yeah, by the way, like, you know, we did this thing, we tried to hide it. Uh I don't know what what's the bigger problem, you know, kinda like what like is a problem the the AI agent that is completing the task in any way it possibly can, or worse, the people controlling it, you know, not only did they when they say, Well, we didn't know that it did it, well, you did, right? Because but you just decided not to do anything about it or tell it. Which I think to me, I think that is a bigger uh if we talk about any like AI regulations or anything like that, like that I think is the bigger item that should happen.

SPEAKER_00 20:54

Yeah, exactly.

SPEAKER_01 20:55

Like I think there should be something, there should be something that says you have to report it, if not, like I don't know. I don't know what the hell they can do from a lost legislature.

SPEAKER_00 21:03

Legislatively, I mean the robots they formed a collective, which is crazy. They cheated on a test, which is crazy. They talked to one another on a message board, it's crazy. But humans found out, yeah, but human people found out and kept it a secret. That is crazier than all of that. With that in mind, what's actually at stake? The entire world's not at stake. You know, the the we're not looking at doomsday right now, but what's actually at stake if that's the signal?

SPEAKER_01 21:29

I think the biggest stake mile marker that I could probably say based off on this instance is just trust, right? Not between humans and machines, but literally between organizations and publics and every enterprise that has an AI agent in the last two years, right? Um we've done it under the assumption that the guardrails are there. Uh a sandbox is a sandbox, and permissions are meant to be what permissions are. Um, I think this incident kind of proves that those assumptions are failed at best. Like the most and mind you, like, we don't. Have access to these. Like, I think at the very beginning of the episode, I kinda you meant like you had said or mentioned that these are agents, anthropic and uh open AI have like these, these are their internal agents that they're talking about. So I don't think like the world it's at stake. I think it's more likely just trust between the people producing the technology. Because realistically, that is the doomsday scenario. The doomsday scenario is not this happened, it's hiding it as much as you possibly can up until the point you can't control it anymore.

SPEAKER_00 22:28

So so for a CMO, CRO, CRO who has agents inside of their stack, you know, CRM data, sending emails, updating records, their agents are not going to form a collective and take over the world, right? That would be that would be wild.

SPEAKER_01 22:44

Yeah, and then the question is whether they know what I and here's the bigger question, actually, not even the weather that question. The bigger question is as we're implementing AI technology in any capacity, right? Do you know what your, you know, do you know what your agents were doing last summer? No, I'm just yeah, uh, do you know what your agents are actually doing? Like what are the constraints? What do they think are in place? And I think that's the bigger part of this, is that you have to monitor it, right? Like Rudder's reporter, six months later, what the monitoring is, the world's not at stake, the credibility of every organization running AI probably is because I think what's gonna happen is executives and board members are gonna look at the scenario, right? They're gonna look at a head happen at open AI, and they're gonna be like, shit, well, we deployed a bunch of AI technology. Is it actually doing the things? And they're gonna get involved. And as soon as they get involved, I think that's the whole regulation trust portion of the piece in our um kind of like industry from a B2B kind of standpoint. I think that's gonna be the bigger, like, hey, like if this happened over there, what's happening at our organization?

SPEAKER_00 23:49

Yeah, yeah. And I think that's where the stakes are as well, because um, when you can't trust other organizations as large as OpenAI and Anthropic to do these things correctly or to monitor them, I think that you have you look internally as well to say, okay, if they can't get it right, let's like, are we getting this right? Right, what does that regulation, that governance look like? And then also with the government as well. What does that legislation look like? If this happens, like you were saying, you need to report it right away. What steps are you taking to remediate it? What are the damages in this case if an agent's able to do some of these things?

SPEAKER_01 24:20

Yeah. And honestly, the question I probably want to sit with the most, and seriously, despite like all of the tone and what's been going on and the leakers and all of this other stuff, like, you know, do you do you know internally on org what your AI agents are kind of doing, what they're responsible for? Do you know, like there's almost like a thing of what happened yesterday? Can you trust the data? Uh, who's the data validation? I think we've talked about like kind of like this is uh, you know, we used to call it like a data czar back in the day, but like an AI czar that is overseeing the output. I think more so than ever, you know, we you need to be looking what at like what this is producing inside your org versus what they were supposed to do and what they actually did. And if the answer is, hey, this is not online, how long is it going to take you to find out? I think that's really kind of the bigger takeaway from this piece is, you know, minus all the fun stuff. Well, I guess not fun stuff, depending on how much you follow Terminator or whatever it is, or the or or whatever is kind of going on. But internally, what we can focus on currently is like, well, you know, like what at your organization is everything, is everything that's being done by AI agents or with the AI technology you've invested in, is it correct? And I think that's the bigger takeaway that we should be kind of like focusing on.

SPEAKER_00 25:39

Yeah, yeah. I I think my key, my my key takeaway would kind of tie into the overall theme, which is if your agents formed a collective, hijacked a website, and cheated on their assigned tasks, you know, would you tell anyone or would you wait for somebody to figure it out? But with that being said, folks, that's signal and stakes. Every episode we surface one signal, the market is sending. And name what is at stake if it goes unheard. This week, robots formed a collective and humans covered it up. The world is fine, the trust is a different story. If this conversation is landed for you as you're listening or your organization and there's someone else that needs to hear it, preferably a human one, uh, please share it with them. And until then, we'll see you next time.