CS 194/294-196 (LLM Agents) - Lecture 11, Ben Mann

The lecture discusses the importance of AI safety and Anthropic's approach to responsible AI development, particularly focusing on the concept of AI Safety Levels (ASLs) modeled after biological safety levels. Anthropic has developed a Responsible Scaling Policy (RSP) that outlines safety measures for different ASL levels, highlighting the need to anticipate and prepare for future capabilities that might not yet exist and potentially pause development until safety measures catch up with capabilities.

Anthropic & AI Safety

• 00:00:07 Anthropic, co-founded by the speaker, focuses on developing safe and beneficial AI systems. Their work is driven by the increasing capabilities of AI models as compute power increases. They published several papers on Transformer circuits and launched the Claude language model.

AI Safety Levels

• 00:12:54 Anthropic uses the concept of AI Safety Levels (ASLs) to guide their AI development, inspired by biological safety levels (BSLs). ASLs categorize models based on their potential for harm and require specific containment and deployment measures for each level. For example, ASL2 models exhibit early signs of potentially dangerous capabilities, while ASL3 models pose a significant risk of misuse and autonomous replication.

Responsible Scaling Policy

• 00:11:43 Anthropic's Responsible Scaling Policy (RSP) aims to ensure the safe deployment of AI models by establishing pre-committed safety standards. This framework provides structure for decision-making regarding AI safety, promotes transparency, and inspires other organizations to implement similar policies. The policy includes capability thresholds that require more testing before model release, a six-month post-training window for evaluation, and other safety measures.

Computer Use Capabilities

• 00:29:47 Anthropic's Computer Use capabilities allow AI models to interact directly with computer systems through tools like mouse movements, keyboard input, and screenshots. While still in the early stages of development, this technology holds great promise for automating complex tasks but also raises safety concerns, such as potential malicious websites or jailbreaks aimed at manipulating the model.

AI Forecasting & Benchmarks

• 00:23:06 AI forecasting is challenging due to the rapid pace of progress and the lack of reliable benchmarks. Anthropic believes the RSP framework can enhance AI forecasting by considering safety constraints alongside raw capabilities. Benchmarks, such as OSOR and SBench, provide snapshots of current capabilities across a range of domains and tasks. However, benchmarks can quickly become saturated as AI rapidly surpasses human performance.