Skip to content
Live
Research#Coding
  1. Hacker News · 300+71

    MaxxBench release: Model scores show significant divergence after rewriting DeepSWE tasks

    The MaxxBench team made minor changes to 89 DeepSWE tasks (median reference patch added only 17 lines, 3.5%), re-evaluating 15 model configurations, finding the difference between the highest and lowest scores expanded from 20.9 percentage points in DeepSWE to 48.3 percentage points.

    Why it matters: By rewriting the instructions and starting code for 89 DeepSWE tasks, it demonstrates the huge gap in model scores between familiar benchmarks and tasks after modification.

  1. The Decoder71

    Alibaba and other teams released Co-Design Bench: coding intelligence agents only fully solve 26.4% of real tasks.

    Researchers from Alibaba and several Asian universities used real user change requests to build the coding intelligence agent benchmark Co-Design Bench. Among the ten tested models, the best-performing GPT-6 Astra only meets about 70% of the task requirements, but only 26.4% of the tasks are completely correct.

    Same story · 2 outlets

    Why it matters: A benchmark display composed of real user requests shows that the strongest model can only fully solve about one-quarter of the tasks, and the score difference for the same model in different environments is nearly 20 percentage points.

  2. MarkTechPost71

    Microsoft and UC Berkeley release TokenEconomizer plugin, coding agent costs reduced by up to 30.9%

    Microsoft Research and UC Berkeley's research team released the open-source cost controller TokenEconomizer, which can reduce the cost of encoding agents without changing model weights and agent code.

    Why it matters: The original text provides the cost reduction of the plugin on 277 SWE-Bench tasks and the change in Pass@1, which can be used to judge its impact on the coding agent workflow.

  1. Meta Engineering66

    Meta rolls out JiTTesting to production: Use LLM to generate tests and catch bugs before code merge

    Meta announced that Catching JiTTests has been put into internal production, using LLM to automatically generate executable tests for diffs to be merged, exposing behavioral differences before the code is landed

    Why it matters: Meta has disclosed the production deployment details and quantitative results of JiTTests, available for evaluating the implementation of LLM-generated tests in ultra-large-scale codebases.

  1. The Decoder82

    Anthropic enables Claude to automatically generate formal safety proofs for code.

    Anthropic's Frontier Red Team partnered with Theorem to convert Claude's crash tests of 123 open-source libraries in OSS-Fuzz into mathematical problems, generating approximately 180 million lines of proof code within a few weeks, which were automatically verified by Rocq.

    Why it matters: Anthropic uses Claude to automatically generate formal proofs for the OSS-Fuzz library, providing computational cost and applicable boundaries, allowing readers to judge the feasibility of AI-verified code.

  2. Anthropic Research82

    Anthropic research: LLMs make large-scale formal software verification feasible.

    Anthropic partnered with Theorem to complete formal verification of the RISC-V assembly layer for 239 entry points in 123 OSS-Fuzz open-source libraries using Claude. The verification proves that these programs will not crash with any input less than 64 MB, with the proof code consisting of about 180 million lines of Rocq.

    Why it matters: Anthropic used Claude to formally prove the RISC-V assembly layer for 123 open-source libraries and provided cost and reuse data, which can be used to determine whether this method can enter the daily development process.

  1. Anthropic Research67

    Anthropic uses Claude with property testing to find bugs in the Python ecosystem

    Anthropic's research team developed an agent based on Claude Code custom commands, which infers the properties that code should satisfy by reading type annotations, docstrings, and comments, and then uses Hypothesis to write property tests, discovering hundreds of potential bugs in over 100 popular Python packages such as NumPy, SciPy, and Pandas.

    Why it matters: Anthropic uses Claude agents to automatically write property tests to find bugs in packages like NumPy, SciPy, Pandas, and provides the efficiency after manual review and merged fixes

You are all caught up