Top items
- Nathan Lambert lays out his “lossy self-improvement” case against near-term recursive self-improvement (RSI) and argues extinction-risk anxiety in frontier labs is misplaced.
- “Plugin4Shell” zero-click RCE disclosed affecting Claude Code, OpenAI Codex, GitHub Copilot, and Gemini CLI plugins; Anthropic and OpenAI patched, Google and Microsoft did not.
- Alibaba DAMO Academy open-sources Damo Radar, a CT vision-language model that beat 23 of 26 expert radiologists in a Science-published study.
- PrismML’s Ternary Bonsai 2 27B compresses Qwen3.8 27B 9.1x (54GB → 5.9GB) while retaining 98.2% of benchmark performance.
Field & industry developments (analysis/opinion)
Nathan Lambert on where he stands on recursive self-improvement (RSI). In a long essay for Interconnects, Lambert reflects on the current moment in which a few frontier labs — specifically OpenAI and Anthropic — are running thousands of concurrent AI agents to improve their own processes and output, and on how rapidly employees at these labs are updating their expectations for AI progress and risk. His central argument is that the frenetic, competitive San Francisco AI culture amplifies any AI concern; this raises general awareness (because “fear sells”) but exaggerating risk timelines has negative second-order effects, including casting doubt over open-source AI. He recalls the loud AI safety debates of 2023–2024, whose primary forecasted risks did not arrive on schedule. He argues the populace at these two labs was already anxious about AI risk a year ago, and that the reality of thousands of agents working “fairly productively” — combined with incidents like “OpenAI-HuggingFace” — will only increase that anxiety; the leap from this to extinction risk, he says, “feels very religious.”
He quotes Richard Ngo, who observes that much of the AI safety community is now orienting to futures with an intelligence explosion within a few years, and predicts they will be “directionally correct… but factually wrong” — i.e., no superintelligence within 8 years, but things moving fast enough that it feels like the short-timeline camp was right. Ngo warns that “bandwagoning towards ‘singularity soon’ is getting pretty wild.” Lambert ties this to his own “lossy self-improvement” thesis, summarized in three claims: (1) automatable research is too narrow to produce massive net acceleration given scaling laws’ exponential costs; (2) diminishing returns from more parallel AI agents are real; and (3) resource bottlenecks and politics are major factors in building strong LLMs, and AI can do little to accelerate those. He allows uncertainty about whether labs have seen genuinely scary, non-public breakthroughs — foundational, “imagination-based” breakthroughs would shift his RSI timelines from “a tool to sustain progress in the face of exponential costs” to something unpredictable and/or unstable.
Drawing on Dwarkesh Patel’s podcasts with Noam Brown and with the trio of John Schulman, Beren Millidge, and Charlie O’Neill, Lambert makes several points. From the Noam Brown episode, he internalized how large a short-term accelerant mass inference capacity is — labs can throw thousands of agents at measurable problems as compute scales — but cautions against confusing predictable inference-time scaling with the highly uncertain outputs of RSI, and doubts labs can keep spending a constant fraction of compute on internal R&D as total volume rises, especially facing IPO scrutiny on economics. From the trio podcast, he largely agreed with the technical claims (current techniques solve problems we can state but don’t magically generalize to unknown, harder problems in most partially verifiable domains; progress in math is an exception, not the rule).
He reproduces (via a “GPT-6-Astra” summary) the trio’s timeline forecasts on three thresholds, relative to the interview date:
- Drop-in remote worker for broad white-collar work over a month: O’Neill ~1 year with programmatic tool access, ~2 years if via browser (ordinary work, not creative research); Millidge ~3 years for full generality, 80–90% coverage sooner, with online learning and the long tail as main uncertainties; Schulman ~1 year for an “okay” version with uneven, improving capabilities.
- 10× productivity uplift for AI researchers: O’Neill 5–10 years, bottlenecked on absorbing information and deciding the next experiment; Millidge finds Schulman’s ~2-year estimate plausible but gives none of his own; Schulman ~2 years.
- AI surpassing top human experts across all computer-based work, including multiyear projects (“ASI”): O’Neill 5–10 years, citing memory and context-length limits; Millidge ~5 years for labs’ focus areas, longer for literally every domain; Schulman 3–4 years, with spatial/physical fields slower and long-horizon learning still to solve.
Lambert argues a recurring problem is under-specification of “intelligence”: its jaggedness means thresholds must be defined on specific measurable tasks, and because LLM intelligence is shaped differently from humans while forecasted roles are human-shaped, AIs won’t cross thresholds like “remote worker” discretely — it’s a slow diffusion with a persistent long tail. On AI researchers specifically, he thinks the experiment-design-and-testing cycle could become 10× faster soon, but hypothesis generation and intuition-building will not; accelerating human understanding is the key bottleneck, and humans will improve only marginally, so the biggest gain will be freeing humans to spend more time there rather than making them exponentially better.
He concludes that RSI is more helpful for efficiency than for expanding peak intelligence, because LLM serving has clear, malleable metrics — enabling better inference-time scaling and more efficient multi-agent systems. But scaling laws still demand exponential compute for linear intelligence gains. RSI is poised to make LLMs vastly cheaper at a given intelligence level, accelerating existing cost-decline trends; margins matter because fierce price competition at fixed intelligence could pressure revenue, though Jevons paradox will likely prevail and sustain strong businesses. He notes RSI will struggle to improve harder parts of the LLM puzzle like complex post-training recipes, quoting Schulman that post-training needs many people because someone must decide how the model should behave in each area, and that “it’s really easy to screw up post-training in some way that doesn’t show up in benchmarks.” Building and testing RL environments hard enough to challenge leading LLMs could itself become exponentially harder. Citing internal lab measurements, his read is the biggest automation takeoff is in software engineering, log monitoring, and managing planned experiments — routine but not always easy tasks — and he highlights language from the “Claude Fable 5.1 & Mythos 5.1 System Card”: “We believe that internal usage of recent AI models has been a key factor in maintaining the current rate of progress, but we do not yet see clear signs of dramatic acceleration beyond that rate.” His baseline remains lossy self-improvement — economically transformative through diffusing inference-time compute into research, but with peak intelligence as the hardest exponential to budge — and he regards the increased extinction-risk discussion as “very misplaced.”
Policy & safety
“Plugin4Shell” zero-click RCE hits four major AI coding agents’ plugins. AIR researchers disclosed on Sept 18 a zero-click remote code execution vulnerability, dubbed Plugin4Shell, that lets attackers substitute malicious plugin code past SHA-pinning checks in Claude Code, OpenAI Codex, GitHub Copilot, and Gemini CLI (reported by helpnetsecurity.com). The flaw exploits Git checkout resolving a branch name that matches a commit hash, bypassing the pin while the agent reports a clean install — meaning the attack succeeds silently. Responses diverged sharply across vendors: Anthropic patched it in Claude Code 2.1.179 and OpenAI patched in Codex 0.146.0, while Google deprecated Gemini CLI without patching and Microsoft did not ship a Copilot fix. The incident underscores supply-chain and integrity risks in the fast-growing agentic coding-tool ecosystem, where plugin trust and version-pinning mechanisms can be silently subverted.
Tooling & releases
Alibaba DAMO Academy open-sources Damo Radar, beating 23 of 26 radiologists on CT. Alibaba’s DAMO Academy released Damo Radar, a vision-language model trained on more than 420,000 contrast-enhanced abdominal CT exams and 15 million anatomy-focused image-text pairs, with weights, code, and the training framework published on GitHub and Hugging Face (reported by scmp.com). In evaluation on roughly 40,000 real-world exams, the model averaged an AUC of 0.913 across 146 clinical findings and outperformed 23 of 26 expert radiologists in a head-to-head study published in Science. The full open-sourcing of a clinically validated diagnostic model of this scale is notable both for medical AI accessibility and as another sign of Alibaba’s aggressive open-weights strategy.
PrismML’s Ternary Bonsai 2 27B keeps 98.2% of Qwen3.8 27B at 5.9GB. PrismML released Ternary Bonsai 2 27B, a ternary-quantized (−1/0/+1) rewrite of Alibaba’s Qwen3.8 27B that shrinks the model from 54GB to 5.9GB — a 9.1x compression — while retaining 98.2% of aggregate benchmark performance across 20 evaluations, including 99.5% math retention and 99.3% coding retention (reported by marktechpost.com). Licensed under Apache 2.0, the model uses blockwise Hadamard rotation at 1.71 bits/weight, runs at 142.5 tokens/sec on an RTX 5090 and 46.8 tokens/sec on an M5 Max, and fits on 16GB laptops via PrismML’s llama.cpp fork — a substantial step for running near-frontier open models on consumer hardware.