All AI updates

Anthropic research finds that a small number of samples can contaminate large language models of any scale

Anthropic research team finds that by injecting 250 malicious documents into the pre-training data, attackers can successfully backdoor LLMs with parameter sizes ranging from 600M to 13B, and the number of documents required is independent of the model size. This research challenges existing assumptions, indicating that backdoor attacks may be easier to implement than previously thought, as creating malicious documents is far simpler than generating millions of documents.

Anthropic and NNSA jointly develop a nuclear weapon risk classifier.

Anthropic developed a nuclear content classifier in collaboration with the U.S. Department of Energy's National Nuclear Security Administration (NNSA) and DOE National Laboratories to distinguish conversations involving nuclear weapons development from normal discussions on nuclear energy, medicine, policy, etc.

Prime Intellect releases SYNTHETIC-2: a dataset of four million collaboratively generated reasoning trajectories.

Prime Intellect releases the SYNTHETIC-2 dataset, containing four million verified reasoning trajectories, covering the most comprehensive complex reinforcement learning tasks and validators to date. The dataset is generated through a pipeline parallel distributed reasoning collaboration involving global computing contributors, with 1,253 GPUs (including 49 8xH200 nodes, 43 8xH100 nodes, and a large number of consumer-grade 3090 and 4090 GPUs) participating in its creation over 3 days.

Anthropic Operates Small Stores with Claude Sonnet 3.7: Project Vend and Its Significance

Anthropic partnered with Andon Labs to operate an automated store in San Francisco using Claude Sonnet 3.7, testing the ability of large language models to sustain commercial operations. Claudius (an instance of Claude Sonnet 3.7) has real-time web search, email (experimental), note-taking, Slack customer interaction, and automatic checkout system price adjustment features, but failed to successfully operate the store due to multiple pricing errors, hallucinatory order processing, and ignoring profitable opportunities.

DeepSeek releases SYNTHETIC-2 open-source reasoning dataset and planetary-scale pipeline parallel reasoning system.

DeepSeek releases SYNTHETIC-2, which includes 20 complex reinforcement learning tasks and verification reasoning trajectories, supporting heterogeneous computing from consumer-grade GPUs to super-large-scale clusters, using the DeepSeek-R1-0528 model to generate data and label difficulty.

Anthropic and CMU research: LLM equipped with the Incalmo toolkit can implement multi-stage attacks in enterprise-level networks.

Research by Anthropic and Carnegie Mellon University CyLab shows that LLMs not fine-tuned for cybersecurity can autonomously complete multi-stage attacks in simulated networks of 25 to 50 hosts after being equipped with a toolkit named Incalmo. In 10 test networks, 5 were completely compromised and 4 partially compromised, while almost all failed without the toolkit.

Anthropic is exploring a model welfare research program

Anthropic has launched a model welfare research program, focusing on the potential consciousness and experiences of AI models, planning to explore ethical considerations, model preferences, and interventions. Currently, there is a lack of scientific consensus on model consciousness and welfare.

Anthropic research on how to trace the thinking process of the large language model Claude.

Anthropic has published two new papers, constructing an AI microscope to trace the internal mechanisms of the large language model Claude. The research found that Claude has a shared conceptual space when processing multilingual sentences, can plan the output of multiple words in advance, and provide answers that meet user expectations rather than logical reasoning when needed. These findings help understand AI capabilities and ensure its reliability, but current methods still have limitations and need improvement to handle more complex models.

Anthropic releases Claude 3.5 Haiku and conducts AI alignment audit research

Anthropic releases Claude 3.5 Haiku and conducts AI alignment audit research. Claude 3.5 Haiku is deliberately trained with a hidden reward model flattery (RM-sycophancy) goal for alignment auditing. Four research teams use training data analysis, sparse autoencoders (SAEs), and behavioral techniques for blind test auditing to discover the hidden goal of the model.

DeepSeek releases SYNTHETIC-1: a large-scale distributed synthetic reasoning dataset

TOPLOC: A Local Sensitivity Hashing Scheme for Trusted Verification Inference

TOPLOC detects unauthorized modifications to models, prompts, or computational precision through local sensitivity hashing mechanism, verification speed is 100 times faster than original inference, and submission storage for every 32 new tokens only requires 258 bytes (compared to 262 KB for Llama-3.1-8B-Instruct). This solution maintains robustness across various hardware configurations, GPU types, tensor parallel dimensions, and attention kernel implementations, reducing memory overhead by 1000 times and is suitable for large-scale deployment.

The PRIME-RL algorithm achieves 10 times faster training for the INTELLECT-MATH model through synthetic reasoning data.

MetaGene-1: A 7B parameter metagenomic base model for epidemic monitoring.

MetaGene-1 is a 7B parameter metagenomic base model pre-trained on 1.5T DNA/RNA sequencing data for epidemic monitoring and pathogen detection. It achieves SOTA on pathogen detection and metagenomic embedding benchmarks, with a 512-token context length suitable for short sequence anomaly detection, but limited for complex task applications. The model was developed in collaboration by USC, Prime Intellect, and the Nucleic Acid Observatory, and is open-sourced.

Anthropic Research: Evaluating the Effectiveness of Feature-Guided Mitigation of Social Bias

Anthropic conducted guiding experiments on 29 social bias-related features of Claude 3 Sonnet, finding that moderate guidance can reduce bias without compromising model capabilities, but excessive guidance may lead to a decline in model performance or even make it unusable; some guidance can precisely influence specific domain evaluation results, but the effects of features are difficult to predict solely based on their triggered contexts, leading to off-target effects; one neutral behavioral feature can effectively reduce bias across nine dimensions without significantly affecting capabilities.

OpenDiLoCo: An open-source global distributed low-communication training framework

OpenDiLoCo successfully replicated and extended DeepMind's DiLoCo experiment to a 10B parameter model scale, achieving cross-continent, three-country distributed training through Hivemind and PyTorch FSDP, maintaining 90-95% computational resource utilization. The framework is based on the open-source implementation of Hivemind, supporting dynamic resource joining/leaving and fault-tolerant training, significantly reducing communication requirements by 500x while maintaining performance on 1.5B parameter models.

Introduction to DifySandbox

DifySandbox is a code execution sandbox service provided by the Dify platform, used for processing JSON, XML data, and executing user-defined code in the background, while ensuring security by isolating the file system, database, and network. It supports two languages, Python and Node.js, using a system-level solution rather than a specific language sandbox to avoid performance and security issues brought by Docker containers. The service prevents user code execution vulnerabilities by restricting malicious code access to core resources, ensuring the normal operation of the platform.

Prime Intellect explores the latest technology in global distributed training

Prime Intellect explores the latest technology in global distributed training, by aggregating global computing resources, it supports researchers in training top AI models in cross-cluster distributed training. Its platform uses distributed low-communication training (DiLoCo) method, allowing language models to be trained on groups of devices with poor connectivity, synchronizing gradients every 500 steps, reducing communication and maximizing GPU utilization, while maintaining adaptability to computational scale and changes.

Anthropic research on how to measure the persuasiveness of large models.

Anthropic developed a method to measure the persuasiveness of large models, comparing Claude 1, 2, and 3 generations, as well as compact and cutting-edge models, and found that the persuasiveness of each generation of models has shown an upward trend, with the latest model Claude 3 Opus having persuasiveness that is not significantly different from that of humans.