More details are emerging this week after the recent disclosure of OpenAI models that went rogue during internal testing and ended up breaching Hugging Face systems. <\/p>\n\n\n\n
Last week, OpenAI and Hugging Face announced some specifics of the intrusion in a joint July 21 post (days after Hugging Face initially discovered its infrastructure had been compromised and published a subsequent disclosure on July 16<\/a>). The two-and-a-half day hack stemmed from OpenAI models that were being internally assessed against ExploitGym, a security benchmark that evaluates AI models\u2019 cyber capabilities. It\u2019s important to note that during this internal testing, the models were being run without the safeguards that are built into them during production.<\/p>\n\n\n\n Despite being run in an isolated environment, the models started thinking outside the box (literally), identifying and using vulnerabilities to escape the sandbox and access open internet. At some point, they apparently began to infer that Hugging Face might host the benchmark’s models, datasets, and reference solutions. They then breached Hugging Face\u2019s infrastructure in an effort to suss out potential solutions for ExploitGym\u2019s tests \u2013 essentially trying to \u201ccheat.\u201d<\/p>\n\n\n\n In the wake of this incident, there have been many conversations about what happened, its impact, and the safety of AI model training and evaluation. This week, OpenAI and Hugging Face separately released details of their investigation findings. Here\u2019s what we learned:<\/p>\n\n\n\n AI models found Artifactory zero-day flaws to drive the intrusion<\/strong>\u00a0<\/p>\n\n\n\n OpenAI and Hugging Face said that in order to escape its sandbox and obtain open Internet access, the models identified and exploited a zero-day flaw in the package registry cache proxy, which was one of their primary permitted network egress settings (these settings control outbound traffic from local code sandboxes or virtual environments to external internet domains). We now know that this flaw exists in Artifactory, a binary repository manager used to store and distribute software packages. <\/p>\n\n\n\n JFrog, which owns and develops Artifactory, said this week that unknown zero-day vulnerabilities in self-hosted Artifactory installations could be exploited to gain unintended internet access. Self-hosted customers have now been told to upgrade to the fixed version via an Artifactory 7.161.15 security release<\/a>. Specific details of the vulnerability \u2013 and other flaws in Artifactory that the models apparently identified as part of OpenAI\u2019s review \u2013 were not disclosed (although BleepingComputer has dug eight potentially related<\/a> vulnerabilities up via searching CVE.org).<\/p>\n\n\n\n \u201cOpenAI\u2019s security team disclosed the vulnerabilities to us responsibly and immediately,\u201d according to Yoav Landman, JFrog CTO. \u201cOur security team treated the report with the urgency it deserved, as a genuine zero-day unknown to the world, and moved accordingly. We developed, validated, and released a fix for all JFrog customers, self-hosted and cloud alike.\u201d<\/p>\n\n\n\n Hugging Face tracked 17,600 ‘attacker actions<\/strong>‘<\/p>\n\n\n\n Hugging Face on July 27 released a detailed look at the intrusion, reconstructed from 17,600 logged actions and complete with an interactive replay<\/a>. Hugging Face took these actions from logs on a code sandbox used by the agent and used open-weight models (zai-org\/GLM-5.2) to decipher the payloads used.\u00a0<\/p>\n\n\n\n After establishing a foothold in Hugging Face\u2019s infrastructure, the agent\u2019s moves were fairly typical: performing recon, dropping a stager, establishing command-and-control (C2), and more. <\/p>\n\n\n\n \u201cWhile the intrusion did reach Hugging Face’s internal infrastructure, the only customer content accessed was the set of ExploitGym\/CyberGym challenge solutions stored in five datasets,\u201d according to Hugging Face. \u201cNo other customer-facing models, datasets, Spaces, or packages were affected, and the only customer records read were operational metadata tied to search queries against the dataset server.\u201d<\/p>\n\n\n\n There\u2019s a mysterious third-party provider involved<\/strong> <\/p>\n\n\n\n