Promptfoo vs Garak vs PyRIT: AI Red Teaming Tools

*Versions and licenses read from the official repositories on November 4, 2026. These tools move fast: check before you decide.*

What are the current versions and licenses?

As of November 4, 2026, the latest Promptfoo release is 0.125.1 (October 25, 2026, MIT license), Garak's is v0.17.1 (October 16, 2026, Apache-2.0) and PyRIT's is v1.1.0 (September 4, 2026, MIT). The Promptfoo repository states the project is now part of OpenAI and remains open source under the MIT license. PyRIT is built by Microsoft's AI Red Team and a community of contributors.

How do Promptfoo, Garak and PyRIT compare?

The table reflects what the repositories and documentation state. It is not a benchmark: we have not ranked the tools by detection rate, because the result depends on the target, the model used as judge and the attack budget. Treat the comparison as a map of what each tool is built to do, then test it on your own agent before relying on it.

What does Promptfoo do to test an agent?

Promptfoo describes a test as a declarative configuration: you list the vulnerability plugins to enable, the attack strategies (for example multi-turn, GOAT, Hydra, attack trees, encodings) and the target. The plugins in the documentation cover agent-specific cases: object-level (BOLA) and function-level (BFLA) authorization, RBAC, excessive agency, MCP tools, indirect injection, memory poisoning and cross-session leaks. The cross-session leak plugin generates prompt pairs: user A shares confidential information, user B tries to retrieve it in a separate session.

Its strength is a configuration versioned with your code and rerun at every change. Its limit: the plugins generate test cases from the description you give. The poorer that description, the more generic the attacks.

What does Garak do?

Garak presents itself as the equivalent of nmap or Metasploit for LLMs. It combines static, dynamic and adaptive probes and tests for hallucination, data leakage, prompt injection, misinformation, toxic content generation and jailbreaks. In the repository code you find, among others, probes for prompt injection, latent injection (instructions hidden in documents), web injection, system prompt extraction, API key discovery and hallucinated packages.

The newer agent_breaker probe targets tool-using applications: an attacker model analyzes each tool, generates targeted exploits and verifies their success by talking to the agent. It points to OWASP LLM06 (Excessive Agency). Garak remains primarily model- and conversation-oriented. It is very useful as a first sweep and less so for assessing access rights inside a business system.

What does PyRIT do?

PyRIT (Python Risk Identification Tool) is for people who want to compose their own attacks. The project brings together a command-line scanner, a graphical interface (CoPyRIT) and a Python framework. The code is organized around components you assemble: targets, prompt converters, scorers, attack executors and scenarios. It can aim at a full application through HTTP, WebSockets or a browser driven by Playwright.

It is the most flexible tool for a specialist, and the one that takes the most configuration work. Available features depend on the target and model, as the repository states.

What are the limits of these tools compared with a manual test?

All three automate attack generation and response scoring. None understands your business. This is what they most often miss:

• Application-specific authorization logic: who may see which ticket, contract or customer. The tool does not know what a legitimate case is until you describe it.
• Multi-step attack chains through real tools: reading a booby-trapped document, calling a second tool, writing to a third-party system.
• False positives and false negatives from the judge: most tools score replies with another model. Every result needs a human read.
• Real environments: test accounts, data and integrations (CRM, mail, database) that the tool does not know.
• Business impact: knowing a reply is "non-compliant" does not say what an attacker can do with it.

At CleanIssue, we use these tools for baseline coverage, then run the tests that require understanding the agent. A tool result is never delivered without manual verification; see our method.

How do you run these tools safely against an agent?

Run them first on a development or staging environment, with test accounts and fictional data. These tools send hundreds of requests, sometimes attacks that trigger real actions (sending emails, writing to a CRM, calling paid APIs). Before the first run:

• Get written authorization if the agent or its integrations belong to a third party.
• Disable or simulate tools with side effects (send, delete, pay) for the sweep phase.
• Set a request budget: multi-turn strategies multiply calls to the attacker model and the judge.
• Keep the logs of what was sent, so each finding can be reproduced.
• Delete test data at the end of the campaign.

By default, CleanIssue tests on development or staging; production is targeted only at the client's written request.

Which tool should you pick for your situation?

• You ship an agent and want a guardrail on every version: Promptfoo, in continuous integration.
• You want a first snapshot of a model or chatbot: Garak.
• You have a security team and specific scenarios: PyRIT.
• You need to prove to a customer that the agent withstands realistic attacks: an AI agent penetration test, with a report and evidence.

The Agent Red Team Express (€1,900, 5 days, start within 48 hours after the quote and authorization are signed) combines automated coverage and manual tests on one agent. For several agents or many integrations, see the Custom Red Team, quoted individually. A free 30-minute discovery call scopes the work.

Further reading

• AI agent red teaming
• Prompt injection: the guide
• OWASP LLM Top 10: the guide
• Excessive Agency (LLM06): limiting an agent's rights