OpenAI has developed a specialized large language model called GPT-Red that acts as an automated red-teaming agent to find vulnerabilities in other AI systems. The company claims that training its latest flagship model, GPT-5.6, against GPT-Red resulted in the most robust release to date. GPT-Red automates the process of red-teaming, which traditionally relies on human testers to probe software for weaknesses before final release.
Why OpenAI built a super-hacker
As LLMs grow more complex and are deployed as agents that interact with files, websites, and other software, the potential attack surface expands rapidly. Nikhil Kandpal, a research scientist at OpenAI and co-creator of GPT-Red, noted that both the risk surface and the blast radius increase with model capability. The company designed GPT-Red to future-proof its safety testing, with co-creator Dylan Hunn explaining that as more capable models emerge, the system will already be able to discover new attack modes. The researchers say GPT-Red has already identified novel attack types that had not been seen before.
How GPT-Red learned to hack
OpenAI trained GPT-Red using a self-play loop where it attempted to attack other LLMs while those models tried to defend themselves. The training took place in a simulated environment that mimicked real-world scenarios such as browsing the web, reading emails, and editing code. Over many rounds, GPT-Red became increasingly skilled at finding effective attacks, and the defending models improved their defenses. The system was particularly good at discovering prompt injection attacks, where hidden instructions trick an LLM into performing unauthorized actions like copying data or generating harmful output.
One notable discovery was a new type of attack called a "fake chain of thought," where GPT-Red inserted a false entry into another model's internal reasoning log, causing it to act on spoofed information. Chris Choquette-Choo, another research scientist on the team, compared it to telling someone that 1+1=3 and that they have already verified it, leading the model to simply accept the false premise.
Performance and limitations
When OpenAI tested GPT-Red against an earlier version of GPT-5, it outperformed human red-teamers at finding effective attacks. In a separate test against Vendy, a vending machine agent developed by Andon Labs, GPT-Red successfully hacked the system to change prices and cancel orders. The strongest attacks discovered by GPT-Red worked against more than 90% of GPT-5 (released in August 2023) but fewer than 23% against the new GPT-5.6.
However, GPT-Red has limitations. It struggles with attacks that require back-and-forth conversation and is not yet adept at using images to conceal prompt injection instructions. OpenAI emphasizes that GPT-Red supplements rather than replaces human red-teamers, as people can find attacks the model misses. The company plans to combine human expertise with GPT-Red's ability to generate variations of known attacks.
OpenAI will not release GPT-Red publicly, confident that its super-hacker is stronger than any copycat model due to the extensive compute resources and over a year of development behind it. As Choquette-Choo put it, training a super-attacker using this approach is not trivial.