Snyk Benchmarks AI Models for Secure Code Fixes

Snyk conducted a benchmark study to evaluate how effectively leading AI models can produce secure and functional vulnerability fixes across approximately 150 real vulnerable code samples in JavaScript, Java, and Python. The study found that out-of-the-box models like Gemini 3.1 Pro and Claude Opus 4.6 achieved a fixing rate of 72-75%. However, when integrated with Snyk Intelligence, Opus 4.6 improved its fixing rate from 74.6% to 85.4%, demonstrating a significant enhancement in performance. The benchmark focused on both security and functionality, emphasizing that a fix must not only remove vulnerabilities but also maintain the original code's functionality. The evaluation set, termed Golden Tests, included human-verified unit tests to ensure both criteria were met. The findings indicate that the security context provided by Snyk Intelligence is crucial for improving model performance in vulnerability remediation. This study highlights the importance of having a dual focus on security and functionality in code remediation efforts.

Key Points: • Out-of-the-box AI models fix 72-75% of vulnerabilities, but performance improves with Snyk Intelligence. • Opus 4.6's fixing rate increased from 74.6% to 85.4% when using Snyk's security context. • The benchmark emphasizes the necessity of both security and functional correctness in code fixes.