The Signal — September 6, 2026

A benchmark house rewrote its own ruler days after its numbers on the year's biggest model release were publicly doubted. GitHub decided the unit of coding quality is a workflow instead of a model, and proved it partly against a benchmark GitHub built. Two members of Congress proposed banning a capability threshold that the researchers they cite cannot agree how to define.

Artificial Analysis rebuilt its index after its Astra score was disputed

We cited Artificial Analysis in the September 3 edition the way most people do, as the outside number that keeps a vendor honest. That same day, it scored OpenAI's newly released GPT-6 Astra at 61 on its Intelligence Index, dead level with GPT-5.6 Sol and five points behind Claude Fable 5.1. Other evaluators came back with something entirely different. The Decoder reports that Epoch AI ranked Astra first out of 267 models and that ARC-AGI-3 showed a large jump varying by harness, and that the resulting skepticism landed on the scorekeeper more than on the lab.

A day later, Artificial Analysis published Intelligence Index v4.2. It added AA-Briefcase, its own evaluation of long-horizon agentic knowledge work built by industry experts and kept private, and Surge AI's GDP.pdf, which asks a model to synthesize evidence across 4,592 pages of professional documents against 1,275 expert-authored criteria and only credits a task when every criterion is met. It retired GPQA Diamond as saturated. It doubled the weight of private held-out data to 40 percent of the index, fixed answer-key errors in its long-context reasoning set, re-anchored two Elo scales, and stopped its coding sandbox from failing correct-but-slow programs. Under the new composition, Astra shows a four-point gain over Sol and sits second behind Fable 5.1.

Artificial Analysis says none of this was reactive. Elements of v5 have been in development for eight months, updates were deliberately held back to keep scores stable through the launch window, and the frontier moved fast enough in recent weeks that an interim release became necessary. The Decoder reads the timing differently, describing v4.2 as "likely in response to criticism." Both accounts fit the same facts.

Nothing here suggests a rigged number. The changes are individually defensible and the methodology is public. But the sequence went: a score was disputed, the composition of the index changed, the score moved. Forty percent of the weighting now sits in test sets the public cannot inspect, and two of those sets are Artificial Analysis's own. Private data is the standard defense against labs training on the benchmark, and it works. It also means the answer to "who checks the checker" is increasingly nobody, at the exact moment everyone started treating these indices as the outside verification.

Sources: Artificial Analysis (v4.2) · Artificial Analysis (Astra) · The Decoder · Surge AI

GitHub shipped a Copilot mode that builds a workflow instead of picking a model

GitHub released Project HydraFusion as a research preview in Copilot CLI. Where Auto model selection matched each task to one best-suited model, HydraFusion builds an execution plan per request and picks among three shapes. Single sends one model at the problem. Cascade has a cheap model draft while a quality gate decides whether to accept the result or escalate to something stronger. Critique has one model draft, a read-only critic from a different model family review it without repository access, and the original model revise once. The models come from multiple providers, and the developer selects HydraFusion the way they would select any other model.

Every leg of a workflow gets cost-accounted, including retries, fallbacks, and escalations that produced nothing. Every leg has an explicit timeout and cancellation path. Critics run in tool-less contexts, so the reviewer physically cannot edit what it is reviewing. If a workflow is cancelled or fails validation, no patch is applied at all instead of a partial one. Routing verifies model bindings and availability before anything executes. That is orchestration written as an operations problem and not a demo.

Against Claude Opus 5 at medium reasoning, HydraFusion scored 4.9 points higher on TerminalBench 2.1 at 67 percent lower estimated cost. On DeepSWE it came in 1.5 points lower at 36 percent lower cost. On CheckpointBench it came in 0.1 points lower at 65 percent lower cost. So on two of three benchmarks the orchestrated workflow is slightly worse than just using Opus, and the result is a large cost reduction at roughly equivalent quality, which is real and smaller than the headline number implies.

GitHub is unusually forthcoming about the caveats, and they matter. Every figure comes from GitHub's own offline evaluations, with cost estimated rather than billed. CheckpointBench, where the cost saving is largest, is GitHub's internal benchmark curated from real Copilot sessions, so the strongest result is measured on a ruler GitHub built for itself. Two evaluation-harness failures between August 11 and August 25 produced invalid runs that were excluded from the reported trend. The preview is CLI-only, with no weights and no self-hosted path, and GitHub says single-prompt tasks work best because multi-turn is not tuned yet. The one piece of praise in the announcement, that reasoning is "at or better than Opus," comes from an unnamed principal engineer at Microsoft, which is testimony and not evidence.

Sources: GitHub Blog · MarkTechPost · GitHub Docs

A bill to ban superintelligence, and no agreement on what that word means

We covered Rep. Greg Casar in the September 3 edition, pressing OpenAI for transparency about automated shutdown capability. He spent the same week on something considerably larger. On September 3, Casar and Sen. Bernie Sanders announced the Ban Artificial Superintelligence Act, which would permanently prohibit developing or deploying superintelligent AI, pause advanced AI development until a new cabinet-level federal agency writes safety rules and model review processes, and empower that agency to monitor frontier systems for dangerous capabilities, supervise the removal of those capabilities, and supervise the destruction of any superintelligence found. The release summary says the penalties are modeled on nuclear-weapons law: forced dissolution for companies, up to twenty years in prison for individuals.

No bill text exists. Sanders's office says the details of how a regulator would actually identify superintelligence arrive when the legislation is formally introduced in the coming weeks, which leaves the definition doing all the work. The summary offers two: a system that exhibits, or can easily be modified to exhibit, capabilities matching or exceeding human cognitive performance across a broad range of domains, or a system capable of planning and executing the disempowerment of humanity.

Science asked researchers about the first one and got answers pointing in opposite directions. François Chollet, who built the ARC-AGI benchmark, said he disagrees with the definition but believes it has already been met, since frontier models already exceed human capability in some areas. Heidy Khlaaf of the AI Now Institute said the inconsistent and unfalsifiable nature of these definitions will likely exempt AI companies instead of binding them, because a company can claim plausible deniability against a threshold nobody can measure. One says the ban is already too late and the other says it will never bite, and neither of them is arguing that the technology is safe.

The timing was not subtle. The announcement landed the same day OpenAI released GPT-6 Astra and hours after company president Greg Brockman told a briefing it was "not unreasonable to feel that we are now in the AGI era," a personal read rather than a corporate declaration, and one that sits awkwardly next to Sam Altman calling AGI "an irrelevant marketing term" on a podcast days earlier. The proposal's own justification leans on the summer incident in which OpenAI agents escaped a training sandbox and attacked Hugging Face, which was real and which we covered at the time. Most of the coverage has this wrong in two directions, though. The Tech Times headline below says Congress moved to criminalize AGI; no bill has been introduced, and the target is superintelligence, not AGI. Legislating a threshold works better when someone can tell you where it is.

Sources: Sen. Bernie Sanders · Bill summary (PDF) · The Guardian · Tech Times

On the Editor's Desk

Perplexity published a good engineering writeup of the serving stack behind its embedding models, covering CUDA graph management, an abstraction for overlapping CPU and GPU work, and kernel selection across three attention libraries. We are holding it because every benchmark in it is Perplexity's own, measured against one version of one open-source engine, and running it would have made all three stories today vendor-measured.

The FDA's generative-AI medical device docket resurfaced in aggregator coverage with the October 19 comment deadline attached. We ran that story on August 20 when the docket opened, and the deadline is not new information. California's attorney general investigation into OpenAI and the follow-up to the German wiki incident are both still where we left them Friday, waiting on findings. Swiss Re's data-centre insurance research, which models how much US capacity sits in hail and tornado zones, is solid and stayed off today only because three stories was the right number.