Open-weight models trail closed models in cyber capabilities by 6–7 months
Open-weight models trail closed models in cyber capabilities by 6–7 months
Epoch AI compared open-source and closed-source models through the cybersecurity benchmark test Cyber ECI, finding that open-source models lag behind closed-source models by 6-7 months in cybersecurity capabilities. This test is based on the cybersecurity version of ECI, using the same parameters to evaluate the models.
Epoch's work is free to use, distribute, and reproduce provided the source and authors are credited under the Creative Commons BY license.
Learn more about this graph
We compare open- and closed-weight models using the Cyber ECI, a domain-specific version of the Epoch Capabilities Index (ECI) based on cybersecurity benchmarks. We keep the benchmark parameters from a joint fit of the ECI on general and cyber benchmarks, and estimate each model’s Cyber ECI from its cyber results alone. Scores are on the same scale as the general ECI. See our documentation for details on the methodology.
Data
Analysis
Assumptions and limitations
Download this data
Cyber ECI scores of open- and closed-weight models
CSV, Updated Oct. 8, 2026
Source: Epoch AI · epoch.ai