Largest AI vs. human CTF study reveals significant speed gains for elite teams, exposes inflection points for junior and mid-tier talent, and underscores the need for an AI-fluent workforce
Hack The Box (HTB), the global leader in AI-powered cybersecurity readiness, released findings from its AI-Augmented vs Human-Only Cybersecurity Performance Benchmark Report, which includes data from its NeuroGrid Capture The Flag (CTF) competition, the largest side-by-side benchmark of agentic AI and human performance on cybersecurity tasks to date.
Also Read: CIO Influence Interview With Jake Mosey, Chief Product Officer at Recast
AI can raise the bar of cybersecurity performance, but it does not eliminate the need for human expertise.
The results demonstrate that AI levels the playing field and accelerates performance, depending on the skills of the AI-augmented teams, which:
- Completed tasks in significantly less time, enabling up to 4.1x more output for elite teams and 1.4x across all teams, within the set period of time.
- Improved their challenge solve rate by 70% within the same time window, achieving a 27% solve rate vs. 16% for top human-only teams.
- Achieved a 3.2x higher solve-rate ratio than human-only teams, across all active participants.
The benchmark analyzed performance data from 1,078 teams, including 120 agentic AI teams and 958 human teams, across 36 cybersecurity challenges spanning nine technical domains and four difficulty levels during a three-day competition.
“AI can raise the bar of cybersecurity performance, but it does not eliminate the need for human expertise,” said Haris Pylarinos, Founder and CEO of Hack The Box. “Our findings show measurable productivity gains, but also predictable failure patterns. Security leaders must build and test human-in-the-loop workflows that are proven under pressure, and develop the AI and cybersecurity skills needed to unlock benefits safely as models evolve.”
How AI Impacts Performance by Experience Level
The findings showed that AI requires a targeted workforce development strategy, as it impacts less-experienced to elite operators in fundamentally different ways. Results include:
- Early Career Productivity Illusion: AI can function as a competency bridge, helping lower-ranked teams solve meaningfully more challenges, but it can also create a productivity illusion. Lower-performing AI-augmented teams were 12.5% slower, often getting stuck in unproductive loops without strong oversight and fluency.
- Mid Career Strongest Gain: Mid-level operations saw the strongest lift on medium-difficulty tasks, where AI advantage peaked (3.89x), indicating a practical sweet spot where pattern recognition boosts productivity.
- Elite Speed Advantage: The solve-rate advantage narrowed sharply at the top (3.2x overall to 1.7x in the top 5%), confirming elite teams already have the competency to close most of the gap. At the same time, AI-augmented elite teams saw a speed boost, completing challenges 312% faster. AI increased speed, not skills.
“Routine and mid-level work is where enterprises will see immediate ROI,” said Gibb Witham, President of Hack The Box. “If organizations over-index on automating the tasks that build judgment, they risk trading long-term resilience for short-term efficiency. Agentic automation must be paired with deliberate human skill development. For enterprises, the competitive advantage will not come from AI adoption alone. It will come from training cybersecurity professionals to effectively orchestrate, validate, and govern AI-driven workflows and agents.”
The data reveals that AI’s strongest impact occurs in medium-complexity work (3.89x), the very layer where mid-level cybersecurity talent traditionally develops the judgment needed to become security practitioners. Medium-tier challenges showed the highest solve-rate ratios for AI-augmented teams. These are the problems that build experience and pattern recognition in developing analysts. If AI absorbs this layer without structured upskilling and deliberate exposure to increasingly complex scenarios, organizations risk hollowing out the very pipeline that produces future senior experts.
Human-AI Hybrid Models Win
The findings show how AI-augmented teams, often operating with human-in-the-loop, raised baseline performance across the field, yet the hardest and most novel challenges still demanded human judgment and verification. For CEOs and CISOs, the takeaway is clear: use AI to amplify human capability with oversight, not replace it.
Hack The Box will present a deeper analysis of its research at RSAC 2026 on March 26, 2026, in the Village showcase.
Catch more CIO Insights: Why CIOs are becoming chief risk orchestrators?
[To share your insights with us, please write to psen@itechseries.com ]
