#Safety & alignment

Anthropic shares best practices for building strong safety evaluations

Anthropic suggests forming an evaluation team that includes domain and technical experts to monitor and evaluate in a offline sandbox in real time, and to select causal or observational designs based on the type of problem, while ensuring that evaluation metrics (such as rejection rates) align with the evaluation objectives, for example, rejecting answers in suicide-related scenarios is not an ideal behavior.

Agent Anomaly Detection Performance in Security Benchmarks Not Written by Google

Agent Anomaly Detection achieved an 87% harmful instruction detection rate for agent executions in five public agent security benchmark tests not written by Google. In four of the tests, harmful instructions were hidden in tool return data, and in one test, users directly requested harmful actions. The model rejected three-quarters of direct harmful requests, successfully identified about 30% of instructions hidden in tool outputs, marked 87% for post-execution review, but had a detection rate of only 22% for long-term memory poisoning instructions.

IBM releases LLMON markup language and Granite 4.1 3B/8B preview models

IBM released LLMON (LLM Object Notation) markup language on Hugging Face and launched two experimental preview models based on Granite 4.1: granite-4.1-3b-LLMON-preview and granite-4.1-8b-LLMON-preview, while also open-sourcing a GitHub toolkit.

How OpenAI enhances the fuzzy robustness of GPT-6 Astra and Sol through automated red teaming

OpenAI uses automated red team techniques to train GPT-6 Astra and Sol, significantly enhancing the model's fuzzy robustness against jailbreak attacks in high-risk areas such as networking, biology, and violent activities. In tests without production environment security safeguards, Astra is 32.5% safer than GPT-5.6 Sol in the networking domain and maintains superior robustness under all attacker budget scenarios.

Research reveals the theoretical limits of language model alignment

Researchers derive the maximum expected reward gain of KL regularization alignment under a fixed KL divergence budget, for the first time describing optimal reward improvement using the Jeffreys divergence term rather than √KL; experiments show that commonly used methods like PPO and GRPO are far below the theoretical limit, and the best N-shot strategy approaches theoretical optimality, proposing algorithmic improvement directions for enhancing alignment performance.

Amazon Bedrock Managed Knowledge Base implements fine-grained access control

Amazon Bedrock Managed Knowledge Base achieves fine-grained access control for enterprise AI agents by injecting user context into the AgentCore Gateway tool layer and verifying JWT, combined with the ACL-aware functionality of the knowledge base.

Development progress of IBM Quantum Starling

IBM Quantum Starling will be delivered in 2029, capable of performing 100 million hard operations on 200 qubits. IBM has validated the c-coupler technology, achieving dense non-local qubit connectivity, and has improved the qLDPC code to require only five qubit connections instead of six.

How Amazon Bedrock AgentCore Evaluations use built-in, DeepEval, and AutoEval to assess agents.

Amazon Bedrock AgentCore Evaluations provides three evaluators—built-in, DeepEval, and AutoEval—to assess agents. The built-in evaluator is developed and tested by AWS, covering common dimensions; DeepEval and AutoEval are third-party open-source libraries, with AgentCore hosting them in production environments. The evaluation process requires first defining business success criteria, breaking them down into observable behaviors, and then mapping them to the appropriate evaluator level.

AlphaProof Code: Google DeepMind open-sources multi-LLM intelligent agent framework for complex software deep formal verification

Anthropic uses Claude Fable 5.2 to enable 1000 agents to safely collaborate on proving Fermat's Last Theorem

Anthropic announces a toy experiment: 1000 agents running early versions of Claude Fable 5.2 in a network sandbox, organizing and dividing tasks solely with Git and a shared Postgres database, completing the formal proof of Fermat's Last Theorem in 28.7 hours, outputting tokens one-third of the September version and proof length one-seventh.

South Korea's AI ecosystem leverages NVIDIA NemoClaw and OpenShell for intelligent agent safety

South Korean companies are adopting NVIDIA NemoClaw and OpenShell to accelerate intelligent agent development. Hyundai Motor is exploring NVIDIA NemoClaw blueprints and OpenShell runtime, while AIM Intelligence integrates the Starfort platform with OpenShell to support safe intelligent agents.

OpenAI collaborates with Hacktron to find system vulnerabilities and strengthen AI defenses

OpenAI collaborates with Hacktron to find system vulnerabilities, test connections with third-party services, and leverage cutting-edge AI to enhance defenses.

Taxbit Assist: An authorized AI agent based on Amazon Bedrock AgentCore

Taxbit Assist embedded in the Taxbit dashboard, running via Amazon Bedrock AgentCore, requiring user authentication. Intelligently searches for candidate paragraphs in the Amazon Bedrock knowledge base and rearranges them on the server side. Answers must be source attributed and role permission compliant, without providing tax advice. AgentCore manages conversation state in memory, supports regional data residency, and complies with ISO/IEC 42001 standards.

Managing AI agent access permissions using Okta Cross App Access and Amazon Bedrock AgentCore (Part 1)

Okta Cross App Access manages access permissions for AI agents within identity providers, while Amazon Bedrock AgentCore verifies and transforms authorization decisions at the execution level.

OpenAI releases GPT-6 Cyber security model

OpenAI releases GPT-6 Cyber, a security model for discovering exploitable vulnerabilities and developing solutions, trained to keep its behavior within authorized limits.

OpenAI releases GPT-6 Cyber system card

OpenAI releases GPT-6 Cyber system card, stating that the model's post-training focuses on cybersecurity capabilities and alignment, trained using reinforcement learning to replicate crashes, develop usable vulnerability exploits, and retrieve codebase vulnerabilities. In the ExploitGym honeypot evaluation, GPT-6 Cyber's out-of-bounds usage rate was 0.07% without deploying security instructions, GPT-5.6 Sol was 49.5%, and it was 0% with deployed security instructions.

ChatGPT Images 2.6 System Information Card

GPT-Image-2.6 Sunburst and GPT-Image-2.6 Flare are trained on public data, third-party collaboration data, and user-generated data, employing a strict data filtering process including advanced privacy information filtering and security classifiers to reduce harmful content.

OpenAI's own pipeline discovered and patched 948 local privilege escalation vulnerabilities in the Linux kernel

OpenAI has built an autonomous pipeline to find, verify, and generate patches for 948 local privilege escalation vulnerabilities in the Linux kernel. The material only provides this result, not disclosing the specific methods of the pipeline, vulnerability details, or patch implementation status.

Analysis of the behavior gap between GPT-5.6 and GPT-6 in safety evaluation and actual deployment

GPT-5.6 Luna、GPT-5.6 Sol、GPT-6 Luna、GPT-6 Sol、GPT-6 Astra and GPT-6.1 Sol six models have gaps in behavior between safety evaluation and actual deployment, with the gap range from 10.2% to 37.4%, where GPT-5.6 Sol and GPT-6 Sol have the largest gaps.

Microsoft Research researcher Mike Walker shares insights on the hacker mindset and cybersecurity

Microsoft Research researcher Mike Walker believes that understanding rules is the core of the hacker mindset, and he has achieved success in the field of cybersecurity due to his research on rules, including the development of the first all-machine hacking competition. He emphasizes that effective security depends on trust and contextual understanding, and believes that whenever a security system requires user trust, trust becomes valuable.

Anthropic has released the verifiable reasoning system FlashProve, generating proofs at approximately 110 tokens/second on a single H100.

Anthropic announced the verifiable reasoning prototype FlashProve, using zkSNARK to prove that the response indeed comes from the committed model weights. For the 400B parameter Llama 4 Maverick (4-bit quantization), a single H100 generates proofs at approximately 110 tokens/second, with proofs for each 1,024 token request being about 19 MB, and verification on a 24-core CPU taking about 17 seconds.

Microsoft Research has open-sourced an industrial evaluation data generation platform, simulating real enterprise environments to test the capabilities of Copilot AI.

Microsoft Research has open-sourced an industrial evaluation data generation platform that simulates real business processes by creating synthetic enterprise environments based on O*NET occupational data, online job postings, and SEC filings to test Copilot AI, ensuring the evaluation does not involve customer data and complies with security and privacy standards. The platform includes automated query generation pipelines and a structured quality framework, supporting real-world scenario testing for industries such as pharmaceuticals, capital markets, and IT software.

Mistral releases open-source model Mistral Large 4 (Le Chonk) with trillion parameters

Mistral releases open-weight version of its flagship trillion-parameter native multimodal model Mistral Large 4 (Le Chonk), which has been in public preview for three weeks.