Large Language Models (LLMs) and AI agents present a number of novel AI security challenges. To get ahead of these risks, businesses must find them before threat actors do. This is where AI red teaming comes in.
Through red teaming, cybersecurity professionals can think and act in the same way as attackers. In turn, they can uncover and mitigate vulnerabilities in AI systems – including new or emerging types of AI risks that traditional security practices do not adequately address.
This article dives into the details of AI red teaming, explaining what it means, why it’s valuable, and how to get the most out of red teaming to help secure AI systems and infrastructure.
This is part of a series of articles about AI security.
What is AI red teaming?
AI red teaming is the practice of simulating attacks against AI systems. The goal of the team performing the tests is to operate just as external attackers would.
Parenthetically, note that AI red teaming can also refer to the use of AI tools when performing a red teaming exercise. This article, however, focuses on AI red teaming as a means of discovering vulnerabilities in AI systems.
What makes AI red teaming different from red teaming in general
Red teaming in general is a type of methodology that organizations can use to evaluate the security of any type of technology. AI red teaming, on the other hand, focuses on finding and assessing security flaws in AI systems specifically. Unlike traditional red teaming, which often targets infrastructure like networks and servers, AI red teaming is more behavior-focused and examines how systems respond to adversarial inputs.
This is increasingly crucial for businesses because as they adopt AI tools and platforms, they expose themselves to a variety of new types of potential security risks, such as:
- Data leakage from LLMs or AI agents, which could expose sensitive information to unauthorized users.
- Prompt injection attacks against LLMs, designed to manipulate AI model behavior by causing a model to ignore its original instructions and follow attacker-supplied instructions.
- Security risks in AI-generated code, which can be harder to detect than those in conventional applications because developers don’t assess the code as carefully.
- Shadow AI, or the use of unsanctioned (and, in many cases, insecure) AI tools by employees within an organization.
Conventional application security practices don’t fully address these risks. For example, traditional penetration testing assumes deterministic paths and typically looks for software bugs, network loopholes, and access control misconfigurations, but these traditional testing methods are of limited value for detecting AI-specific issues like prompt injection risks. AI red teaming exercises can help close the gap by providing a way of proactively testing AI systems for security flaws.
AI red teaming vs. blue teaming
AI red teaming (and red teaming in general) is distinct from blue teaming. The latter term refers to the processes an organization uses to defend its systems against cyberattacks.
This means that, in an AI red teaming exercise, the red team would seek to attack or exploit an AI system. Simultaneously, the blue team actively monitors and attempts to defend the system. Through this process, the organization can assess how effective its security teams and processes are in detecting and mitigating active attacks.
The use of a blue team to defend against red team attacks is what makes AI red teaming different from penetration testing of AI systems. During pentesting, the security team responsible for monitoring and protecting systems is typically aware that simulated attacks are taking place, and they don’t actively try to block them.
In contrast, under AI red teaming, the blue team usually has no knowledge of the planned attack simulation, and it reacts to signs of intrusion just as it would under a real-world attack scenario. This is valuable because it helps organizations assess the effectiveness of their cybersecurity defenses, in addition to probing for weaknesses.
The role of red teaming in AI security testing
Red teaming is not the only way of assessing attack surfaces and cybersecurity hygiene within AI systems. Businesses can also use techniques like penetration testing, monitoring AI agent workflows, and automatically feeding malicious prompts into AI models to assess prompt injection vulnerabilities.
That said, red teaming provides insights that other types of AI security techniques don’t offer, including:
- The ability not just to discover security flaws (which pentests and scanning can also do), but also to evaluate how well the organization’s cybersecurity personnel (i.e., the “blue team”) are able to mitigate active attacks.
- Carrying out complex, multi-step attacks against AI systems based on sophisticated techniques that scanners can’t emulate well.
- Identifying AI security risks that the organization might not otherwise think to check. Red teaming can help expose these vulnerabilities because testers are able to act in open-ended ways, helping them to develop novel attack strategies.
Operationalizing CTEM Through External Exposure Management
CTEM breaks when it turns into vulnerability chasing. Too many issues, weak proof, and constant escalation…
This whitepaper offers a practical starting point for operationalizing CTEM, covering what to measure, where to start, and what “good” looks like across the core steps.
The AI red teaming process: Key steps
Due to the open-ended nature of red teaming, every exercise is unique; teams don’t follow a rigid script. However, AI red teaming processes typically include the following key steps.
Determination of system and testing scope
Red teaming starts with stakeholders deciding which AI systems to test, which types of tests to carry out, and how to set scope around the specific use case, deployment context, and risk profile of the AI system. This is important because it’s typically not practical to test for every potential vulnerability in every AI tool or platform, due to the time and resource requirements necessary to carry out red team tests.
Because LLMs are often embedded in larger applications, the scope should clarify whether testing covers the target models, the surrounding application, or both. Instead, businesses must decide which systems and risks to prioritize. Best practices include threat modeling and using frameworks such as MITRE ATLAS or NIST AI RMF to set priorities.
Assessment and reconnaissance of AI systems
After deciding what to test for and which AI systems to test, the red team begins assessing the systems for potential flaws. This might include practices like feeding in manipulated prompts or other adversarial inputs to check whether an LLM appears vulnerable to prompt injection and uncover vulnerabilities, or scanning an AI agent in an effort to determine whether it includes any vulnerable third-party components due to insecure supply chain management. Teams could also probe for model inversion or extraction by querying the model in ways that may expose sensitive data from the model’s training data.
Gaining of initial access
If the team succeeds in finding weak points in the AI systems it’s testing, it exploits them to gain initial access. For example, if the goal is to analyze LLMs for data leakage risks, the attackers might use specially crafted prompts to “trick” a chatbot into believing they are users with elevated privileges so that they can then request sensitive data from the bot.
Post-exploitation and lateral movement
After breaking past AI systems’ initial defenses, testers go further to see whether they can escalate attacks or spread them to other systems. For instance, if they manage to impersonate privileged users when interacting with an AI chatbot, they might also check whether they can successfully instruct the chatbot to force AI agents to manipulate information within a database.
If they can, the simulated attack shows that threat actors could not only circumvent an AI model’s data protection controls to exfiltrate sensitive information, but also take advantage of AI agent vulnerabilities to modify data.
Analysis and reporting
AI red teaming concludes with an analysis and report of what the team discovered, including metrics like attack success rate to gauge the overall risk posture. Reports should also include context explaining what the most serious risks are and how configurations or dependencies between systems impact the exploitability of the risks. They should also report on failed attempts and edge cases, since these can reveal parts of the attack surface that still matter.
Recommendations about how to mitigate vulnerabilities are helpful as well, supported by detailed logs. The data gathered from testing can help teams build more precise input/output filters and refine security workflows before deployment.
Tips from the Expert
Rob Gurzeev, CEO and Co-Founder of CyCognito, has led the development of offensive security solutions for both the private sector and intelligence agencies.
Here are our tips for keeping AI red team tests effective and efficient:
- Know which AI assets are exposed: AI tools and services can appear virtually anywhere within modern IT estates – and the riskiest tend to comprise “shadow AI” systems that IT departments may not even know are in use within the organization. When deciding what to test for, it’s critical to find and cover all relevant AI exposure across the business, and avoid focusing only on the AI systems that engineers are actively securing.
- Test for authentication: Proper authentication is key to mitigating AI security risks. But AI systems can authenticate users in complex and novel ways, so it’s vital to iterate across a variety of authentication techniques and integrations when determining how vulnerable an AI system is to the manipulation of user identities and access rights.
- Understand AI service interactions: Few AI systems exist in vacuums; most connect to other business systems or databases via tools like MCP servers or processes like RAG. When planning red teaming test scope, ensure that your team plans to assess how vulnerabilities can play out across these dependency chains, instead of only testing AI systems in isolation.
- Validate remediation efforts: In many cases, mitigating AI security risks is not as simple as merely applying a patch and re-testing. It could involve more complex efforts like re-training, followed by rigorous, iterative testing to ensure that the issue was actually solved (because non-determinism could result in scenarios where a flaw appears to be fixed during the first test but turns out to still be present during subsequent tests). For this reason, it’s essential to follow AI red teaming with a sophisticated remediation validation process.
Limitations and drawbacks of AI red teaming
While red teaming is an effective way to identify security flaws within AI systems, it’s not without its limitations (which is why red teaming should be just one component of an AI security strategy).
Here’s a look at potential drawbacks:
Need for manual effort
Red teaming exercises require manual effort to plan and execute. Automated security testing tools can help accelerate the process, but automated red teaming and emerging AI red teaming tools are best viewed as support for human-led assessments, not full replacements. Lightweight and open-source red teaming tools can automate parts of testing and help identify generative AI vulnerabilities, but they are not a replacement for human expertise in planning and escalating attacks.
Thus, red teaming for AI systems requires skilled professionals whose expertise spans security, machine learning, and human behavior, since all of these can interact to impact AI security.
Limited scope and coverage
The primarily manual nature of red teaming makes it infeasible to test systematically for every type of risk in every system. This results in limited testing scope and functionality coverage, resulting in the risk of leaving some security flaws undiscovered.
Non-deterministic nature of AI systems
Most modern AI systems (specifically, those powered by LLMs) are probabilistic and non-deterministic. This means that they don’t always behave in the same way; for example, if you feed an identical prompt into an AI model two different times, you’re likely to get different responses each time.
From a security testing perspective, non-determinism is a challenge because it can result in situations where identical attack simulations succeed on some attempts and fail on others. If the red team only carries out attacks once or twice, it may fail to discover relevant risks.
Rapid change of pace across AI surface
AI systems are also highly dynamic, and they interact in complex ways as the threat landscape changes with model updates, integrations, and expanding capabilities. To go back to an example used above, a chatbot might also be able to control AI agents, with the result that vulnerabilities in the chatbot could be exploited to manipulate agents, too, and updates or added components can introduce new risks.
Finding and testing for security flaws like this requires careful planning and repeated effort. Simple scripting or generic security tests are insufficient for identifying all potential flaws in complex AI systems. Effective programs require domain experts working alongside security and ML specialists to evaluate real-world scenarios and misuse scenarios in their specific environment.
Lack of AI red teaming tools and frameworks for AI systems
Due to the novelty of generative and agentic AI technology, no standardized frameworks exist to guide red team exercises against AI systems. This can make it challenging for red teams to operate in a holistic way. It also increases the level of expertise necessary on the part of AI red testing teams.
AI red teaming best practices
The following practices can help organizations maximize the impact of AI red teaming:
Define clear scope and objectives
Determining what to test for is vital – especially when testing AI systems, which are especially complex. When defining the scope and goals of AI red team exercises, it’s crucial to think not just about vulnerabilities within a specific AI system that the business wants to assess, but also about how that system’s interactions with other systems could allow attacks to spread or escalate.
Standardize operations
While red team exercises must be dynamic enough to react to unexpected test results and capitalize on novel security flaws whenever the team discovers them, it’s also important to keep testing operations consistent. That standardization should support broader risk management and compliance requirements, not just internal consistency. This is especially important given the non-deterministic nature of AI systems, which means that carrying out the same test multiple times is critical for discovering all vulnerabilities.
Documented, standardized adversarial testing also creates audit-ready evidence for compliance reviews and risk-assurance processes. This is important because regulations like Article 15 of the European Union AI Act require high-risk AI systems to demonstrate accuracy, robustness, and cybersecurity.
Keep processes repeatable
In a similar vein, keeping AI red teaming processes repeatable helps maximize the discoverability of flaws while also making it easier to run continuous testing across the AI system lifecycle and at scale, such that AI systems are rapidly evaluated for flaws whenever they are deployed or updated. This aligns with the NIST AI Risk Management Framework, which emphasizes continuous testing and evaluation throughout the development lifecycle, as well as the integration of security tests into development pipelines so updates do not introduce new vulnerabilities. AI red teaming should not be treated as a one-off exercise, since catching issues pre-deployment is far cheaper than managing a post-deployment crisis.
Document workflows and results
Documenting red team exercises and results helps ensure that teams can keep tests consistent, and workflows should specify the controlled environment used for testing while making clear that production models are excluded. It also mitigates the need to depend on key personnel to carry out tests – a particularly important consideration given the specialized expertise necessary to plan and execute AI security tests. Documentation lowers the knowledge barrier, making it possible for a broader set of engineers (including those without specific AI expertise) to contribute to AI red teaming once basic processes are in place, while also supporting safety training and more consistent remediation work over time.
Securing AI systems with CyCognito
Red teaming can only test what the team knows exists, and the article makes the point directly: the riskiest AI assets tend to be the shadow systems no one put on the list. CyCognito continuously discovers your full external footprint, including exposed AI services, and validates what within it is actually exploitable, starting from nothing more than your organization’s name.
- Discovers exposed AI services across the surface, including model APIs, inference endpoints, MCP servers, and agent orchestration layers that never made the asset list
- Continuously validates exploitability across every discovered asset through always-on active testing, reducing what appears critical from 25% of findings to 0.1% confirmed exploitable
- Traces how exposures connect across dependency chains, mapping external-to-internal attack paths so risk in one AI service does not stay isolated
- Hands red teams a current, evidence-backed target set, so engagement time shifts from recon and enumeration to the creative, multi-step work humans do best
- Prioritizes by exploitability, attack paths, and business context, then tracks findings through to verified closure rather than assumed fixes
If you want to see CyCognito in action, click here to schedule a 1:1 demo.
