Automated Pentesting: Not Everything It’s Made Out to Be
It seems that bots might not be the solution to all issues regarding flaw identification. Many security teams have found fully automated pentesting to be disappointing, as indicated by offensive security firm Cobalt, which noted a significant drop in support for this method over the last year. The latest 2026 State of Pentesting report from Cobalt revealed that security professionals are swiftly abandoning autonomous pentesting tools primarily due to their inability to detect essential vulnerabilities.
False Negatives: The Achilles’ Heel of Automated Scanning
Cobalt disclosed that 78 percent of survey respondents for its 2026 report encountered “critical false negatives” from automated scanning tools, which are notably poor at identifying the types of vulnerabilities that their AI counterparts introduce in environments where they are common. “Automated scanners excel at uncovering known, signature-based vulnerabilities. However, they fall short in addressing AI security,” the firm stated in a summary of the report’s findings. “Prompt injection exploits and excess agency flaws necessitate creative, multi-turn interaction chains [and] adversarial psychology,” Cobalt added. “These logical flaws are completely undetectable by tools that function using single-shot automated queries.”
Organizations Losing Trust in Bots
A year marred by disappointment with automated scanning tools has resulted in a significant drop in organizations considering a fully automated security scanning strategy, with only 9 percent of respondents expressing openness to the concept, compared to 29 percent last year.
Hybrid Security: A More Effective Method?
Cobalt posits that the answer lies in hybrid security, where the majority of systems are subject to automated AI scanning, while the most critical systems are managed and protected by humans. The company naturally offers such a solution, but it’s important to note that its findings regarding the increase in vulnerabilities caused by AI are not particularly unique.
AI: A Source of Chaos?
Earlier this year, application security firm Veracode reported that AI-assisted software development is generating more vulnerabilities than security teams can address, resulting in a backlog of unresolved vulnerabilities. According to Veracode, approximately 82 percent of companies are leaving known vulnerabilities unattended for over a year, and the proportion of high-risk vulnerabilities within all discovered is also on the rise.
The Upside? Some Still Find Bots Beneficial
Nevertheless, not everyone shares the skepticism of automated pentesting reflected by Cobalt and its survey participants. Amazon’s security chief CJ Moses noted that AI pentesting tools have enhanced the efficiency of Amazon’s security teams by 40 percent, although the basis for that figure remains unclear. However, Moses was reluctant to entrust the entirety of the security project to AI. He informed us at the RSA Conference in April that human oversight is still essential to ensure AI does not create complications.
A Geordie’s Perspective
“AI excels at tasks, especially when processing vast data sets and requiring a comprehensive overview,” Moses expressed during an April interview. “However, regarding decision-making functions, it isn’t something we are prepared to depend on.” But really, who wouldn’t desire a bot that could enjoy a pint while identifying flaws simultaneously?
Conclusion: Is It Time to Ditch the Bots?
It appears that machines still cannot be relied upon to perform our challenging tasks. Surprisingly, they’re about as effective at detecting vulnerabilities as I am at resisting Greggs sausage rolls. Hand me the keyboard; I’ll take it from here.
Check out GadgetLad.co.uk for more tech discussions from the Tyne
