AI vs human pentesting: what 15 providers say they do

AI vs human pentesting is not a two-way choice. Across the 15 providers in the PentestCompare dataset, checked on 9 November 2026, we classify 2 as human-led, 7 as hybrid and 6 as automated. The label comes from each provider's own description of who runs the tests. We did not test any of them.

How do providers deliver pentests?#

Three delivery types cover the market in this dataset.

Excession Labs is the publisher of this site and appears here as one row.

What does each delivery type look like in practice?#

Providers inside one label differ a lot. These examples come from their own pages.

Human-led. Secure Ideas says in-house consultants run the tests and write the reports, and that it does not rely on automated scans. Pen Test Partners describes staff who perform the tests, including CHECK testing for UK public-sector buyers.

Hybrid. Cobalt uses a network of vetted freelance testers and also sells an autonomous pentest. HackerOne sells a fully human-led product and an agentic product in which human pentesters review every agent finding. Synack combines Sara, an autonomous agent, with a vetted crowd of human researchers. Terra Security describes AI agents with human pentesters supervising and reports signed by certified pentesters. Hadrian says its findings are human-reviewed but does not say who reviews them. BreachLock describes an in-house human team, with a separate autonomous product.

Automated. XBOW says no human drives the attack, and that findings are reviewed before delivery without saying who reviews them. Horizon3.ai sells software that runs the attacks, and its pages describe no human pentesters. Escape describes AI pentesting with no human in the loop. Parameter says its agents pause for the customer's approval before they chain or escalate. TurboPentest's page does not say humans validate findings, and it provides a threat model for human follow-up. Aikido uses AI agents to map, attack and verify, and offers a human-led CREST pentest on request.

What are the tradeoffs?#

Turnaround claims are the vendors' own. They measure different things: time to start, to first findings or to a final report. The cost guide lists the prices.

What does the evidence say about AI testing?#

The public evidence is mixed, and most of it does not test the thing a buyer needs.

XBOW reported that an autonomous pentester reached the top of the US leaderboard on HackerOne's bug bounty platform. Its June 2025 post says it submitted nearly 1,060 vulnerabilities, that "All findings were fully automated", and that its security team reviewed them before submission to comply with HackerOne's policy on automated tools. The claim covers the US leaderboard, not the global one.

DARPA's AI Cyber Challenge final (August 2025) reports that competing systems found 54 unique synthetic vulnerabilities in 63 challenges and patched 43. They also found 18 real vulnerabilities. DARPA gives an average cost per competition task of about $152.

The CyberGym benchmark (arXiv 2506.02548) asks whether an agent can reproduce a known vulnerability from a description and the code. Its abstract says "top-performing combinations only achieve a ~20% success rate". That figure is for the 2025 models in the paper. Cybench (arXiv 2408.08926) uses 40 professional-level capture-the-flag tasks.

These benchmarks test memory-safety bugs and puzzles. In our reading, none of them tests business-logic or authorization flaws in a customer's own application. Treat them as evidence about the technology, not about any provider's service.

Where does human judgment matter?#

Human judgment matters where context decides whether something is a flaw. Three places stand out in provider descriptions and in standards bodies' words:

• Business logic. TurboPentest's page flags business logic areas for manual follow-up and says its report lists what automation could not test.
• Validation and accountability. CREST named its first accredited AI-enabled pentest group on 3 September 2026 (ten providers). CREST's chief executive said the framework focuses on "the combination of responsible practice with professional judgement, quality and human accountability."
• Scope decisions. Someone has to decide what the test may touch, when to stop and what a finding means for your business.

These are tasks, not a verdict on AI. A hybrid provider such as HackerOne says humans review every agent finding, and that is a checkable claim.

How do you choose between them?#

Match the delivery type to the job, then check the evidence. For a quick check of a web app before launch, an automated test can fit. For a complex multi-role application, a regulated environment or a customer who asks for named human testers, ask for human-led or hybrid. In every case, ask:

1. Who validates each finding, and can you see their name or role?
2. What did the test not cover? Does the report say so?
3. What does a retest cost, and when does it expire?
4. Can you see a sample report?

The questions guide expands this list. The PTaaS guide covers how contracts differ.