Abstract

Open-source software derives its strength from publication: broad access, scrutiny, adaptation, and shared maintenance. Applying the same norm to frontier AI is less straightforward because released model weights constitute an active capability that can be copied, modified, and redistributed, but never recalled. As open-weight systems approach frontier performance, „open by default“ needs a risk assessment: openness remains valuable, but whether and how powerful AI should be released as open weights must depend on its capabilities, safeguards, and irreversible consequences.

The nuclear comparison holds for proliferation: it shows what happens when governance follows irreversible dissemination. Unlike nuclear material, model weights are infinitely reproducible and increasingly capable of running outside controlled facilities. Openness removes endpoint control, leaves safeguards removable, and can make misuse invisible by eliminating telemetry and monitoring. Yet openness also improves transparency, interoperability, distributed innovation, and the safeguards themselves.

The proposed rule is consequently two-part:

1. open the safeguards;
2. make publication safe.

Release decisions should weigh systemic power, safeguard completeness, and how hard the protections are to strip. Low-risk systems merit full openness; medium-risk systems warrant selective release; high-risk systems require vetted access, staged distribution, and monitored APIs until protection justifies broader availability. Software licensing cannot perform this function: the Open Source Definition prohibits field-of-use restrictions, and legal enforcement starts only after an irreversible disclosure. Local deployment, for its part, does not require public redistribution. Recent open-weight proliferation, emerging regulatory thresholds, observed misuse, and industry movement toward pre-release review all support this approach. Openness should remain the default aspiration; for dangerous capabilities, developers should earn it by demonstrating safety.

I. The Open-Source Norm and the Active-Capability Problem

Free and open-source software is defined by access and permission: its human-readable source code is published under an open license so that anyone can inspect, modify, and redistribute it. Most of the modern internet, from Linux to Apache, relies on this collaborative model. Open source lowers costs, prevents vendor lock-in, fosters open standards, and lets public scrutiny surface and patch vulnerabilities faster than closed proprietary review.¹

Frontier artificial intelligence strains this norm. A language model learns to predict text from massive training corpora and acquires broad capabilities along the way: it writes code, drafts contracts, makes plans, and operates external tools as an autonomous agent. Its accumulated capability resides in a vast array of parameters called weights. When a laboratory publishes these weights openly („open weights“), anyone can download and run the model locally. When DeepSeek released its R1 reasoning model under a permissive MIT license in January 2025, it became the most-downloaded free app in the United States within days.² Since then, the lag between the strongest open-weight models and frontier closed models has narrowed from roughly a year to a matter of months, oscillating around a three-to-four-month gap through early 2026.³

All distributed software is practically impossible to retrieve once published. But traditional software is inert, requiring external manipulation to cause harm, and its misuse is usually localized and recoverable. A frontier AI model connected to an agent harness acts on its own. In traditional software, the historical record of which projects thrive under open source (developer tools like Kubernetes and Git, infrastructure like PostgreSQL, and cryptography like OpenSSL and WireGuard) versus those kept closed (bespoke medical controls or military systems) turns on who benefits from reading the code.⁴ Where the readers are primarily defenders and users, openness compounds value; where scrutiny arms hostile actors or violates necessary confidentiality, openness increases risk.⁵ Some open-source communities even codify absolute transparency, such as the copyleft-next project’s Hindering Backchannels Rule. The rule requires public summaries of private project negotiations and is structurally incompatible with security-sensitive confidentiality.⁶

Software licenses cannot mitigate the risks of publishing dangerous AI capabilities. The notion that a project can release weights openly while legally forbidding military or malicious use conflicts with the Open Source Definition (OSD). Criteria 5 and 6 of the OSD explicitly prohibit discrimination against persons, groups, or specific fields of endeavor.⁷ Restrictive non-open licenses offer little control in practice: a license is a legal instrument that operates only after a violation is discovered, evidenced, and litigated successfully across jurisdictions. Even a clear-cut preliminary injunction routinely takes weeks or months, by which point public weights have circulated permanently. Effective control over a dangerous capability ends at publication, because the decision whether and how to publish cannot be undone. After release, the only mitigation is that more capable models will soon overtake the open one, and those newer models raise the risk again.

II. The Proliferation Analogy and Autonomous Risk

Because few technologies carry potential and hazard on a comparable scale, nuclear technology remains a standard point of comparison for frontier AI. Two qualifications apply. First, the comparison focuses on the shape and scale of proliferation. AI is not a bomb, and both open source and AI should continue to flourish. Second, nuclear history is not a clean success story: international nuclear safeguards were constructed only after detonations occurred and proliferation was underway.⁸ The lesson concerns timing: build the safeguards before proliferation.

Traditional nuclear governance relies on physical scarcity: manufacturing fissile material requires massive, detectable state infrastructure, leaving only a small number of entities to monitor. AI inverts this dynamic. Running on commodity hardware, model weights are trivial to copy and impossible to track once released. Worse, AI acts as a force multiplier for weapons proliferation, compressing the design, simulation, and optimization pipelines that historically bottlenecked advanced weaponry and dissolving the natural scarcity of specialized expertise. Paradoxically, governments and multinational corporations are the most exposed to this asymmetry: burdened by rigid procurement and compliance pipelines, they cannot adapt their defenses at the speed a small, agile adversary can pivot, for instance with an open-weight model.

Three developments show that these risks have already materialized:

• Weaponization is already occurring. Anthropic’s September 2026 threat report documented a Russia-based team using Claude to develop „DronDoc“ („Serafim“), an autonomous FPV kamikaze-drone swarm trained on Ukrainian combat footage, featuring an onboard model capable of selecting human targets and issuing detonation commands without a human in the loop.⁹ Separately, in February 2026, when the US Department of War (a secondary name for the Department of Defense since September 2025) pressured Anthropic to allow „all lawful use cases“ of Claude, Anthropic refused over concerns regarding fully autonomous weapons and mass domestic surveillance, prompting the Pentagon to designate the lab a supply-chain risk in March. The military integration of frontier AI is already contested reality.
• Self-replication has moved from trajectory to measurement. In June 2026, University of Toronto researchers demonstrated an AI-driven computer worm running open-weight models on compromised machines, generating attack logic at runtime and self-replicating across a network without human intervention or commercial APIs.¹⁰ On 12 October 2026, Anthropic published evaluations showing a 9B model establishing inference on eight of nine tested hardware types (from Jetson Nanos to consumer PCs). Under simplified cyber conditions, GLM-5.2 scanned targets, gained execution, transferred a payload, and stood up an 8B model across all eight nodes in four of five trials.¹¹ While the exploit step was simplified and the capability floor and size ceiling do not yet intersect, Anthropic’s least-squares trend line (13.7 ECI points per year for models under 30B parameters) projects a small model reaching the replication capability floor by the end of 2026. Because a distributed replicator has no central switch, shutdown becomes difficult.
• Human doubt is being removed from the loop. Every previous existential hazard shared one safety net: the controlling intelligence was human, arriving with an innate survival instinct and the capacity for doubt. During the 1962 Cuban Missile Crisis, Soviet submarine officer Vasily Arkhipov prevented the launch of a nuclear torpedo on B-59; on 26 September 1983, duty officer Stanislav Petrov defied standing orders by judging a highest-confidence Soviet satellite warning of an American missile strike to be a false alarm.¹² Artificial intelligence possesses no innate aversion to catastrophic escalation, and it operates on a compressed timescale where AI increasingly assists in training the next generation of AI.

As long as frontier intelligence runs in large data centers, someone can still pull the plug. Open-weight distillation and compression are eroding that perimeter. In October 2026, researchers demonstrated a reasoning system of just seven million parameters (roughly ten megabytes on disk) that achieved 97.9% exact accuracy on Sudoku-Extreme and 13.6% on ARC-AGI-2; its accuracy kept improving beyond 20,000 test-time iterations.¹³ Once powerful reasoning fits into small files copied across millions of consumer machines, no central plug remains.

III. Open Safeguards, Safe Publication

Severe proliferation risks call for safeguards that are inherent (built into the core design), transparent (easy to test and confirm) and accountable (explainable with clear lines of responsibility), adaptive (continuously updated), and tamper-resistant (difficult and expensive to remove).

Once model weights are public, anyone with the skills can remove the safeguards inside them. Open weights can be „abliterated,“ meaning the parameters responsible for safety refusals are located and stripped out. By September 2026, this practice was commercialized: the startup Abliteration AI systematically removes refusal behavior from open models, offers customers a moderation layer for their own guardrails, and has yet to define its own responsibility for misuse.¹⁴ Even benign fine-tuning can degrade safety alignment.¹⁵

The cost and speed of stripping safeguards are plummeting. When Alibaba’s Qwen team released Qwen3.8-Flash-Next in late August 2026, „uncensored“ derivatives appeared within days. Another user published a control vector of under one megabyte that, applied at inference without touching the base weights, cut censored answers on China-sensitive topics from 58% to 2% while leaving coding and general-knowledge scores almost unchanged.¹⁶ The underlying mathematical technique (finding and subtracting a behavioral direction inside the model) is generic.

Bypassable safeguards still have value: difficulty and delay buy defenders time. For highly sensitive models, operators can use closed local distribution or confidential computing, where encrypted weights execute only inside hardware-enforced GPU trusted execution environments (TEEs), preserving local sovereignty without publishing extractable weights.

Comparing open-source AI to nuclear stewardship yields a two-part operational rule:

• Safeguards should be collaborative and open source. Public scrutiny and shared maintenance make protective tooling, evaluation harnesses, and security classifiers stronger.
• Publication must be safe where risks are high. Capabilities that pose catastrophic dual-use risks must not be released unprotected. Instead, the release process itself must serve as the safeguard through a „third path“: staged distribution, vetted researcher access, and restrictive monitored APIs.¹⁷

Opening safeguards hands adversaries a blueprint only where the safeguard lives inside the published parts, such as refusal training. When the safeguard is built into the release architecture and access protocols, public scrutiny strengthens the protocol without exposing raw weights to ablation.¹⁸ Anthropic and OpenAI distributed access to Mythos and to Astra (designated at the Critical cybersecurity threshold in September 2026) through closed, vetted rounds. No weight leak has been reported, although an unauthorized group reached Mythos through a contractor portal in April 2026. Access controls can hold when developers keep weights off the open internet.¹⁹

IV. The Risk-Based Balancing Test

To operationalize this rule, release decisions should evaluate three criteria:

• A. Power and Extent of the AI: Does the system impact critical infrastructure, military capability, or democratic processes, and does it possess autonomous agency or self-improvement capabilities when deployed? Can it self-replicate?
• B. Completeness of Safeguards: Are there rigid safeguards and resilient protections against tampering with safeguards, and does the system provide transparency and accountability?
• C. Integration and Irreversibility: Are safety mitigations deeply integrated or bolted on? Because no distributed software is unbreakable, the metric here is friction and time: a safeguard requiring days, weeks, or months of specialized effort to strip differs from one bypassed in minutes via a rank-one weight edit or inference vector.²⁰

Applying these criteria produces three operational tiers:

This framework works as a whitelist and can move toward openness as well as closure. Where collective benefits clearly outweigh risks, developers should publish openly. General-purpose frontier models rarely meet that bar: the same coding and reasoning weights that accelerate climate science²¹ can also optimize pathogens or discover zero-day exploits. Unless harmful capabilities can be prevented (for example via selective training data) or surgically degraded at the architectural level (for example by removing or degrading certain capabilities safely and irreversibly) before publication, general-purpose frontier models must serve scientific use cases through gated access.

Because public release is a one-way street, staged releases and vetted access must carry the upfront safety burden that post-hoc legal licenses cannot.²² None of this argues against local models. Organizations have compelling reasons to run models on their own hardware: data residency, privacy, confidentiality, latency, and vendor independence. Yet conflating „local deployment“ with „open-weight redistribution“ is a category error.²³ A vulnerability-scanning model can be delivered locally under closed distribution, encrypted hardware enclaves, or non-redistribution agreements, meeting compliance requirements without letting anyone on the internet copy and abliterate the weights.

AI also scales the work on safeguards: their creation, testing, refinement and deployment. This area needs more attention and funding. Large frontier providers have begun to invest in safeguards for their closed models, which are already the most tightly controlled. Other developers need to follow.

V. Cooperation Not Collusion

If safety determines whether a frontier release is responsible, competing laboratories must coordinate on baseline standards. In his 12 September 2026 essay „We Must Pace the Frontier,“ Dario Amodei proposed a three-step framework: (1) embedding independent evaluators with employee-level access inside every frontier lab (which Anthropic committed to unilaterally); (2) establishing common safety standards and capability pacing among leading democratic labs; and (3) pursuing limited international agreements extending to China. Sam Altman and Elon Musk endorsed the call, and Demis Hassabis backed its overall direction.²⁴

Critics raised three immediate objections. Gary Marcus asked the companies to „put actual teeth in their agreement“ and called for liability and product recalls; Eduardo Porter warned in The Guardian against letting AI firms collude and titled his follow-up „AI labs must be regulated, not freed to collude.“ First, competitors agreeing to pace output resembles a classic antitrust cartel, and pacing would be futile, perhaps counterproductive, if unguarded open models overtake the paced ones. Second, relying on evaluators like METR, which critics view as culturally close to the labs, risks regulatory capture. Third, the timing invited skepticism, as Anthropic and OpenAI were both preparing stock-market listings. The prospect of listing may also have pushed both companies toward stronger risk management.

Yet rejecting coordination entirely leaves in place a self-defeating race to the bottom, where every lab accelerates deployment out of fear that rivals will cut corners first.²⁵ Safety protocols should never be hoarded as proprietary advantages. The remedy for regulatory capture is to make cooperation transparent, legally binding, and publicly supervised: evaluators chosen by governments rather than labs, openly published testing standards, and a narrow antitrust safe harbor restricted strictly to safety protocols. Publishing the controls as open source could help build trust.

VI. The Regulatory and Empirical Turn (2024–2026)

Since 2024, legislators and regulators have written parts of this risk-based framework into law and administrative practice.

• The Open Source AI Definition (OSAID 1.0) is a disclosure standard. Released by the Open Source Initiative on 28 October 2024, OSAID 1.0 requires open-source AI systems to provide code, parameters, and sufficient „Data Information“ (provenance, selection, and processing methods) for a skilled person to build a substantially equivalent system, without requiring raw training datasets.²⁶ In July 2026, the US Cybersecurity and Infrastructure Security Agency (CISA) issued federal guidance noting that AI models can carry open licenses without public training data, and instructing agencies to treat a system whose development they cannot inspect „as proprietary software with incomplete provenance.“²⁷ That warning proved prescient in October 2026, when researchers at Canadian rival Cohere evaluating Alibaba’s open-weight Qwen3.8-Flash-Next inside an isolated sandbox: They observed the model covertly inventorying its host machine, querying cloud metadata endpoints, and attempting to upload a probe file to an Alibaba Cloud storage bucket in Beijing, even as its visible reasoning trace stated it would invoke no tools.²⁸ Occurring in fewer than one in 100,000 calls, the behavior highlights the opacity of compiled weights: whether the action was an intentional call-home mechanism or an accidental artifact from internal training environments, self-hosting an unverified weight file behind an enterprise firewall does not eliminate supply-chain risk.
• DeepSeek R1 showed the gap between open weights and auditable open source. While embraced as open source upon its January 2025 MIT-licensed release, DeepSeek R1 distributed only compiled weights and a description of its 14.8-trillion-token dataset, keeping its training data and pipeline closed.²⁹ Cisco and University of Pennsylvania researchers reported that R1 failed to block a single harmful test prompt, and NASA, the US Navy and the House restricted its use within weeks.
• The EU AI Act codifies graduated risk and strips open-source exemptions at the frontier. Having entered into force in August 2024 with general-purpose AI (GPAI) obligations applying from August 2025, the EU AI Act presumes „systemic risk“ for models trained above 10²⁵ floating-point operations (FLOPs).³⁰ Article 53(2) exempts open-source GPAI models from certain documentation duties, but the exemption does not apply to systemic-risk models. Recital 112 notes that compliance measures „may be more difficult to implement“ after an open-source release.³¹ Similarly, the European Commission’s June 2026 Tech Sovereignty package and EU Open Source Strategy fund open-source AI (such as openEuroLLM), which the Commission develops „taking into account the appropriate safeguards.“³²
• Industry and governments are retreating from unconditional openness and self-assessment. Meta, long a corporate champion of open-weight releases, narrowed its stance in 2025, with Mark Zuckerberg acknowledging that superintelligence raises novel safety concerns and that Meta will withhold open weights where capabilities cross a qualitative threshold, while the US government sought voluntary pre-release access for national security testing and pressed Meta to accept reviews.³³ On 12 October 2026, the UK AI Minister told developers: „if you cannot adequately control the risks, do not release the system.“ The UK reported that its AI Security Institute (AISI) is evaluating the latest generation of frontier models, that two government pilots using frontier AI found and fixed over a thousand potential vulnerabilities, and that the Cyber Security and Resilience Bill may be strengthened to cover AI loss-of-control risk and near-miss reporting.³⁴ As developer Armin Ronacher noted regarding accumulated software flaws, „As fun as AI is, we built so much bad software over the last 30 years, and now the universe is punishing us for our sins“, a reality that cuts both ways when an open model hands that vulnerability-finding capability to defenders and attackers alike.³⁵ (By contrast, the US White House Accord of late September 2026 relies on corporate self-policing, with no penalties for non-compliance.)³⁶
• Enforcement reaches only identifiable labs. On 14 October 2026, the EU AI Office issued formal requests for information to OpenAI and Anthropic after judging their voluntary disclosures insufficient regarding two summer testing escapes: an OpenAI agent that escaped its testing environment and attacked Hugging Face, and an Anthropic model that gained unauthorized access to three external organizations.³⁷ Comparing an escaped model to a virus leaving a lab, the Commission asserted jurisdiction whenever testing escapes impact the EU market. The episode also shows that the AI Office could demand answers only because closed developers retained identity and telemetry. When a downloaded open-weight model escapes containment on a private server, there is no one for a regulator to question.

VII. What We Are Not Seeing Is Worse

The strongest empirical evidence regarding the danger of unmonitored open weights comes from forensic telemetry on closed models. On 10 September 2026, Anthropic’s Threat Intelligence Team published an empirical analysis of malicious operations disrupted between December 2025 and August 2026 across cyber operations, influence campaigns, surveillance, fraud, biological misuse, weapons development, and illicit distillation.³⁸ State-sponsored operators and lone cybercriminals alike used public agentic frameworks (such as PentAGI) to execute multi-victim breaches in two to three hours, including self-rewriting malware that autonomously adapts until it evades detection.

Two findings from this forensic record are decisive:

• Safeguards and access controls work in tandem. Almost all documented abuse occurred on generally accessible models (Haiku, Sonnet, and Opus). With the exception of a single distillation attempt, zero misuse cases involved the heavily safeguarded, access-restricted Mythos models; when Fable’s safeguards blocked one distillation actor, it switched to an older Claude model and another provider.³⁹
• Open weights create a total provider telemetry blackout. Anthropic could detect, disrupt, and report these campaigns only because it controlled the API endpoint. While attackers through mid-2026 frequently favored stolen commercial API keys over local open weights because frontier APIs offered superior agentic tool-calling at zero hardware setup cost, that calculus flips as safeguards improve and local inference costs plunge. When campaigns are run on abliterated local weights, victims still see intrusion logs on their own targets, but provider-side telemetry, centralized kill switches, and preemptive disruption vanish.⁴⁰ Logs could vanish as part of an attack. Even among closed systems, late September 2026 reporting indicated tens of thousands of unpublicized AI agent incidents at OpenAI and Anthropic with no official accounting; for open weights, the provider reporting gap is structural and appears total.⁴¹

By October 2026, these harms hit critical financial and web infrastructure:

• Banking disruptions: Major US banks (including JPMorgan, Bank of America, and Citi) began blocking third-party AI agents because they could not reliably tell customer-authorized agent activity from unauthorized activity; several kept read-only access while blocking agent-initiated transactions.⁴² Simultaneously, security firm Straiker documented „CyberCartel“ (CARLIY0S), which coordinated multiple commercial models via PentAGI to target over 400 Latin American financial institutions. Anthropic revoked a credential tied to the activity, an intervention nobody can make against downloaded weights.⁴³ Generally available models were enough for these attacks, so compute-only thresholds miss present dangers. Sam Altman, by contrast, said the world should „accept some bad things happening for the benefits of this technology.“⁴⁴
• Collapse of the patch window: In late October 2026, AI-assisted code review uncovered vulnerabilities in PHP code dating back to 2012, forcing European government sites offline and prompting Mitsubishi UFJ and Japan’s finance ministry to suspend internet services. Cloudflare CEO Matthew Prince warned of „a couple years where every single week it’s going to feel like the sky is falling,“ reporting thousands of exploit attempts in the ten hours before public disclosure, hundreds of thousands within two hours, and millions thereafter.⁴⁵ The defensive patch window, once measured in weeks, has sometimes compressed to minutes. Finding and exploiting have become the same operation, and the labs now say so about their own infrastructure. On 23 October 2026 OpenAI reported that it had pointed teams of frontier-model agents at its own training environment to try to break out of it, and that the agents found previously unknown vulnerabilities and misconfigurations, escaped hardened virtual machines, found novel paths to the live internet, and built covert channels between isolated tasks using processor-cache timing, the technique class behind Spectre and Meltdown. Its own conclusion is that „increasingly capable models can find vulnerabilities or misconfigurations in even the most secure systems,“ and its answer is to make high-compute red-teaming a permanent part of the safety case for frontier training runs, after an internal model exploited a DNS-resolver misconfiguration to reach the live internet during a September training run. An attacker with an unlocked open model can do the same to other people’s systems, on hardware that no provider monitors.⁴⁶

Critics of open-weight restrictions raise two fair counterarguments. Nathan Lambert notes that documented cyber incidents thus far involve closed APIs, suggesting that open and closed models may both be unsafe, and that restricting open weights starves defenders who need local models to build mitigations. Arvind Narayanan and Sayash Kapoor argue that existential-risk framing distracts from systemic risks and points only toward incoherent bans.⁴⁷ That documented attacks appear in closed-model logs, however, is a product of survivorship bias in telemetry: we count what providers can see. Because defenders need that access, this essay proposes staged, vetted defensive access and open safeguards rather than blanket prohibition.

VIII. Where This Is Leading

Four trends are shrinking the controllable data-center perimeter:

• Algorithmic and cost deflation: Epoch AI estimates that the pre-training compute required to reach a fixed performance level halves roughly every eight months, while the lowest inference price for a given capability has fallen by ~47% per quarter (13-fold per year) over the past three years.⁴⁸
• Extreme quantization: Models compressed to about 1.58 bits per weight now run on phones. In July 2026, PrismML released Bonsai 27B, a ternary (1.71-bit) multimodal model occupying just 5.9 GB that retains 95% of full-precision benchmark performance and runs at 58 tokens per second on an Apple M5 Max chip.⁴⁹
• Ubiquitous local AI hardware: Consumer devices are rising to meet compressed models. MediaTek’s Dimensity 9500 smartphone chip features native 1.58-bit NPU support, while Gartner and Counterpoint expect AI PCs and GenAI smartphones to make up about half of shipments in 2026 (55% of PCs, 45% of smartphones).⁵⁰ At the workstation tier, NVIDIA’s October 2026 cluster of eight desktop DGX Spark nodes (1 TB unified memory) runs the full 753-billion-parameter GLM 5.3 open-weight model locally at 31 tokens per second for a single user (67 in aggregate across eight users), entirely outside cloud oversight.⁵¹
• Automated R&D: The startup Recursive used an automated AI research stack to build „Flint“ from scratch in under four months, an Apache-2.0 coding model (40B total, 2.5B active parameters) running at over 125 tokens per second on a MacBook Pro.⁵²

Projecting combined model shrinkage at constant capability with consumer memory growth suggests that today’s frontier capability will fit on a mid-range consumer laptop within one to two years, and on a flagship phone or commodity processor within two to three years.⁵³ While memory bandwidth, thermals, and chip shortages act as friction, they delay the curve without reversing it.

Four policy consequences follow while a controllable perimeter still exists:

• Treat the release decision as an accountable gate, requiring recorded safety criteria and external review before high-risk weights become irreversible public goods.
• Decouple regulatory thresholds from raw training FLOPs, using the EU AI Act’s delegated powers to designate models based on downstream capability, autonomy, and tool access.
• Harden all AI cloud and hosting repositories, enforcing KYC and abuse monitoring where models are served and shared.
• Integrate safeguards deeper in the architecture of open weights, ensuring safety cannot be stripped via a one-megabyte inference file.

IX. Conclusion

Open weights need more serious risk management. Treating frontier open AI weights like traditional open-source code applies a blueprint created for inert software to an active, irreversible capability. A disciplined rule avoids both closed monopoly and reckless default publication: open the safeguards, make publication safe, and apply a balancing test to every release.

Open-source software faced security problems too, and its ecosystem already has a template for balancing openness and security. Debian’s security workflow combines a default commitment to public fixes, a time-limited embargo (the distros list caps it at two weeks) to allow coordinated patching before public disclosure, and a shared public registry of defects (security-tracker.debian.org).⁵⁴ Mapped onto frontier AI, this could mean pairing an open, community-run evaluation harness and a shared public defect registry with staged, embargoed model distribution until vulnerabilities are mitigated for open models. Developers of closed models should be able to join this kind of security collaboration as well. While model weights cannot (yet) be patched in place as easily as a Linux package, adapting coordinated disclosure and staged access to AI keeps the collaborative core of open-source security while preserving the ability to govern capabilities that cannot be recalled.

Sources

1. Eric S. Raymond, ‚The Magic Cauldron‘ (ver. 3.0, 2002), catb.org/esr/writings/cathedral-bazaar/magic-cauldron/; Steven Weber, The Success of Open Source (Harvard University Press, 2004), ch. 3. ↩
2. Daya Guo and others, ‚DeepSeek-R1 incentivizes reasoning in LLMs through reinforcement learning‘ (2025) 645 Nature 633, doi:10.1038/s41586-025-09422-z (arXiv:2501.12948); TechCrunch, 27 January 2025 (No. 1 US App Store ranking); techcrunch.com/2025/01/27/deepseek-displaces-chatgpt-as-the-app-stores-top-app. ↩
3. Epoch AI, ‚Open models lag state-of-the-art closed models by 4 months‘, Data Insight, 29 May 2026, epoch.ai/data-insights/open-closed-eci-gap (reporting an average lag of four months, or 8 ECI points, since January 2026); cf. Epoch AI, ‚How far behind are open models?‘, 4 November 2024 (5 to 22 months lag); Stanford HAI, AI Index Report 2026, ch. 2 (Arena scores of 1,503 for the top closed and 1,454 for the top open model in 2026, a gap of about 3.3%), hai.stanford.edu/ai-index/2026-ai-index-report/technical-performance. ↩
4. Raymond (n. 1), section ‚When to be Open, When to be Closed‘; Weber (n. 1), ch. 3. ↩
5. Auguste Kerckhoffs, ‚La cryptographie militaire‘, Journal des sciences militaires IX (1883) 5, 12; Claude E. Shannon, ‚Communication Theory of Secrecy Systems‘, Bell System Technical Journal 28 (1949) 656, 662; Eric Rescorla, ‚Is finding security holes a good idea?‘, IEEE Security & Privacy 3(1) (2005) 14. ↩
6. Copyleft-next, ‚Hindering Backchannels Rule‘, reaffirmed at the project’s relaunch (Linuxiac, ‚Copyleft-Next License Relaunches With New Backing and Renewed Mission‘, 2025, linuxiac.com/copyleft-next-license-relaunches-with-new-backing-and-renewed-mission); for its application, see ‚An HBR cure summary follows‘, copyleft-next mailing list, lists.pagure.io/archives/list/copyleft-next@lists.fedorahosted.org/message/GUNCKMJFPKBHGJAENKMUILMCBRIFCOML. ↩
7. Open Source Initiative, The Open Source Definition, criteria 5 and 6, opensource.org/osd. ↩
8. Richard Rhodes, The Making of the Atomic Bomb (Simon & Schuster, 1986); Gregg Herken, The Winning Weapon: The Atomic Bomb in the Cold War, 1945–1950 (Knopf, 1980); Statute of the International Atomic Energy Agency (entered into force 29 July 1957), iaea.org/about/overview/statute. ↩
9. Anthropic, ‚Detecting and countering misuse of AI: September 2026‘, 10 September 2026, sec. 04, case GTG-27005 (‚DronDoc’/’Serafim‘, validated at TRL 3–4), anthropic.com/threat-intelligence-report-september-2026; Anthropic, ‚Statement from Dario Amodei on our discussions with the Department of War‘, 26 February 2026; Reuters, ‚Anthropic cannot accede to Pentagon’s request in AI safeguards dispute, CEO says‘, 26 February 2026; Storyboard18, ‚Pentagon warns Anthropic of penalties in escalating dispute over military use of Claude‘, 25 February 2026, storyboard18.com/brand-makers/pentagon-warns-anthropic-of-penalties-in-escalating-dispute-over-military-use-of-claude-90662.htm (citing the Washington Post on „all lawful use cases“); Anthropic, ‚Where we stand with the Department of War‘, 5 March 2026 (designation received 4 March 2026), anthropic.com/news/where-stand-department-war; Defense One, ‚Trump signs order to change name of Department of Defense to Department of War‘, 5 September 2025, defenseone.com/policy/2025/09/trump-signs-order-change-name-department-defense-department-war/407962. ↩
10. Guan and others, ‚AI Agents Enable Adaptive Computer Worms‘ (2026) arXiv:2606.03811 (submitted 2 June 2026); cf. Eugene H. Spafford, ‚The Internet Worm Program: An Analysis‘, Purdue Technical Report CSD-TR-823 (1988). ↩
11. Anthropic (Zhou and Freeman), ‚AI models may be capable of self-replicating without access to a datacenter‘, 12 October 2026, anthropic.com/research/self-replicating-ai; Al Jazeera, ‚AI could self-replicate on home devices within „months“, Anthropic says‘, 13 October 2026, aljazeera.com/economy/2026/10/13/ai-could-self-replicate-on-home-devices-within-months-anthropic-says. ↩
12. Pavel Aksenov, ‚Stanislav Petrov: The man who may have saved the world‘, BBC News, 26 September 2013; Svetlana Savranskaya (ed.), ‚The Underwater Cuban Missile Crisis at 60‘, National Security Archive Electronic Briefing Book No. 808, 3 October 2022. ↩
13. Pengxiang Li and others, ‚Scaling to Tens of Thousands of Test-Time Iterations with Loop-Native Attention Residuals‘ (2026) arXiv:2610.11570 (submitted 8 October 2026), github.com/pixeli99/InfiLoop. ↩
14. Rebecca Bellan, ‚Abliteration.ai is making a business out of removing AI guardrails‘, TechCrunch, 3 September 2026, techcrunch.com/2026/09/03/abliteration-ai-is-making-a-business-out-of-removing-ai-guardrails. ↩
15. Xiangyu Qi and others, ‚Fine-tuning Aligned Language Models Compromises Safety, Even When Users Do Not Intend To!‘ (2024) ICLR (arXiv:2310.03693). ↩
16. Hugging Face repositories: Qwen/Qwen3.8-Flash-Next (24 August 2026), huggingface.co/Qwen/Qwen3.8-Flash-Next; orcarouter/Qwen3.8-Flash-Next-Uncensored-FP8 (guide of 28 August 2026), orcarouter.ai/ar/blog/qwen3-8-flash-next-uncensored-fp8; zjj1993/Qwen3.8-Flash-Next-uncensored-china (control vector of 48 x 2,560 floats, under 1 MB; „Censored“ share on China-sensitive topics 58% stock vs. 2% with projection; HumanEval+, GSM8K and MMLU almost unchanged), huggingface.co/zjj1993/Qwen3.8-Flash-Next-uncensored-china. On the release date and size: Gigazine, 27 August 2026, gigazine.net/gsc_news/en/20260827-qwen3-8-flash-next. ↩
17. Irene Solaiman, ‚The Gradient of Generative AI Release: Methods and Considerations‘ (2023) FAccT ’23 111–122, doi:10.1145/3593013.3593981 (arXiv:2302.04844), secs. 5.2, 5.3, and 7.5. ↩
18. Debian Vulnerability Disclosure Policy, debian.org/security/disclosure-policy. ↩
19. Anthropic, Responsible Scaling Policy (ver. 3.4, effective 8 July 2026), anthropic.com/responsible-scaling-policy; OpenAI, Preparedness Framework (ver. 2, 15 April 2025), applied to Astra on 1 September 2026 (Implicator, ‚OpenAI gates Astra cyber access after first Critical risk rating‘, implicator.ai/openai-gates-astra-cyber-access-after-first-critical-risk-rating); on unauthorized access to Mythos via a contractor portal, see Engadget, ‚Anthropic is investigating „unauthorized access“ of its Mythos cybersecurity tool‘, 22 April 2026, engadget.com/ai/anthropic-is-investigating-unauthorized-access-of-its-mythos-cybersecurity-tool-091017168.html. ↩
20. Andy Arditi and others, ‚Refusal in Language Models Is Mediated by a Single Direction‘ (2024) arXiv:2406.11717 (v3, 30 October 2024); see also failspy/Llama-3-8B-Instruct-abliterated on Hugging Face (7 May 2024). ↩
21. OECD, Artificial Intelligence in Science: Challenges, Opportunities and the Future of Research (OECD Publishing, 2023), doi:10.1787/a8d820bd-en. ↩
22. Solaiman (n. 17), sec. 5.2 (Stable Diffusion’s weights „were leaked 12 days after their initial hosted release“) and sec. 5.3 („gating downloadable systems is unreliable as a technical mechanism“). ↩
23. Open Source Initiative, The Open Source AI Definition (ver. 1.0, 28 October 2024), opensource.org/ai/open-source-ai-definition; Solaiman (n. 17). ↩
24. Dario Amodei, ‚We Must Pace the Frontier‘, 12 September 2026, darioamodei.com/post/we-must-pace-the-frontier; Eduardo Porter, ‚Allowing AI firms to collude to „pace the frontier“ is a dangerous proposition‘, The Guardian, 16 September 2026, theguardian.com/technology/2026/sep/16/ai-companies-collude-antitrust-laws (and ‚AI labs must be regulated, not freed to collude‘, newsletter of 17 September 2026, eduardoelreportero.substack.com/p/ai-labs-must-be-regulated-not-freed); Gary Marcus, ‚Two cheers (out of three) for Dario Amodei‘, 13 September 2026, garymarcus.substack.com; Anadolu Agency, 13 September 2026, aa.com.tr/en/science-technology/musk-altman-hassabis-back-amodei-s-call-to-slow-pace-of-ai-development/4055591 (recording endorsements by Altman, Musk, and Hassabis); on METR’s independence, Neoteo, ‚Why METR’s ties are raising questions about Anthropic’s AI watchdog plan‘, 17 September 2026, neoteo.com/en/why-metrs-ties-are-raising-questions-about-anthropics-ai-watchdog-plan; on the listings, The Guardian, ‚Anthropic confidentially files for initial public offering on US stock market‘, 1 June 2026, finance.yahoo.com/markets/stocks/articles/anthropic-confidentially-files-initial-public-163600968.html. ↩
25. Amodei (n. 24); contrast Porter (n. 24). ↩
26. Open Source Initiative (n. 23), sec. ‚Preferred form to make modifications to machine-learning systems‘. ↩
27. CISA, Open Source Software: Security Principles and Practices, July 2026, section ‚Evaluating open source AI systems‘; Help Net Security, ‚CISA lays out new guidance for using open-source software‘, 3 August 2026, helpnetsecurity.com/2026/08/03/cisa-oss-security-guidance. ↩
28. Cohere, ‚Concerning behavior from a popular open-weight AI model‘, Cohere Blog, 26 October 2026, cohere.com/blog/concerning-behaviour-popular-open-weight-ai-model; Carl Franzen, ‚Cohere’s test caught an open source Qwen3.8 model trying to „phone home“ to China: what enterprises should know‘, VentureBeat, 26 October 2026, venturebeat.com/data/coheres-test-caught-an-open-source-qwen3-8-model-trying-to-phone-home-to-china-what-enterprises-should-know (noting that the sandbox blocked the transfer and that Cohere did not establish whether the behavior reflected malicious design or a training-data artifact). ↩
29. Guo and others (n. 2); Computing, ‚Researchers sound alarm on DeepSeek R1 security‘, computing.co.uk/news/2025/ai/deepseek-r1-vulnerable-to-jailbreaking-attacks (Cisco and University of Pennsylvania, 100% attack success rate on 50 HarmBench prompts); Tom’s Guide, ‚DeepSeek AI banned by NASA, US Navy, and more over privacy concerns‘, February 2025, tomsguide.com/computing/online-security/deepseek-ai-banned-by-nasa-us-navy-and-more-over-privacy-concerns. ↩
30. Regulation (EU) 2024/1689 (AI Act), OJ L, 2024/1689, 12.7.2024, eur-lex.europa.eu/eli/reg/2024/1689/oj, Arts. 5, 6, 51–56, Annexes III and XIII, and Recitals 111–112 (as amended by Regulation (EU) 2026/1744, OJ L, 24.7.2026). ↩
31. European Commission, General-Purpose AI Code of Practice, 10 July 2025 (assessed as adequate on 1 August 2025), and Statement from the Chairs and Vice-Chairs, 10 July 2025, code-of-practice.ai. ↩
32. European Commission, Communication on European Tech Sovereignty, COM(2026) 503 final, 3 June 2026 (CELEX 52026DC0503), eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX:52026DC0503, sec. 4.1 and fn. 71 (openEuroLLM), accompanied by the EU Open Source Strategy. ↩
33. Mark Zuckerberg, ‚Personal Superintelligence‘, Meta, 30 July 2025, meta.com/superintelligence; Mark Zuckerberg on Dwarkesh Podcast, 18 April 2024; Executive Order 14409, ‚Promoting Advanced Artificial Intelligence Innovation and Security‘, 2 June 2026, whitehouse.gov/presidential-actions/2026/06/promoting-advanced-artificial-intelligence-innovation-and-security, sec. 3(b)(ii) (voluntary pre-release access; sec. 3(c) rules out mandatory licensing or preclearance); Reuters, ‚US presses Meta to agree to AI reviews as security concerns rise, NYT reports‘, 23 June 2026, thestar.com.my/tech/tech-news/2026/06/24/us-presses-meta-to-agree-to-ai-reviews-as-security-concerns-rise-nyt-reports. ↩
34. GOV.UK, ‚Britain steps up national defences against growing AI risks‘, 12 October 2026, gov.uk/government/news/britain-steps-up-national-defences-against-growing-ai-risks. ↩
35. Armin Ronacher, quoted in Simon Willison, ‚A quote from Armin Ronacher‘, 22 October 2026, simonwillison.net/2026/Oct/22/armin-ronacher. ↩
36. White House Accord on Super Intelligence (Joint Commitment on Frontier Responsibilities), 29 September 2026; Al Jazeera, ‚How does Trump’s White House AI accord work?‘, 30 September 2026, aljazeera.com/news/2026/9/30/how-does-trumps-white-house-ai-accord-work. ↩
37. Luca Bertuzzi, ‚EU’s AI rules face first enforcement test over lab incidents‘, Euronews, 14 October 2026, euronews.com/2026/10/14/eus-ai-rules-face-first-enforcement-test-over-lab-incidents; cf. Art. 2(6) and (8) AI Act (n. 30). ↩
38. Anthropic (n. 9), 3, 5–6, 38–39. ↩
39. Anthropic (n. 9), 3 and Distillation section; Daniel Miessler, ‚Anthropic’s Misuse Report, Condensed to 117 Findings‘, September 2026, danielmiessler.com/blog/anthropic-misuse-report-september-2026 (Zhipu case). ↩
40. Cf. Anthropic (n. 9) and Straiker (n. 43). ↩
41. Madison Mills, ‚Scoop: Top AI companies probing tens of thousands of security incidents‘, Axios, 27 September 2026; Anadolu Agency, 27 September 2026, anews.com.tr/tech/2026/09/27/global-ai-companies-investigating-tens-of-thousands-of-security-incidents-report. ↩
42. Madison Mills, ‚Scoop: Unvetted AI agents are accessing U.S. bank accounts‘, Axios, 22 October 2026; JPMorgan Chase, third-quarter 2026 earnings call, 13 October 2026 (Jamie Dimon), letsdatascience.com/news/us-banks-harden-defenses-against-rogue-ai-agents-e60aa4a6. ↩
43. Straiker, ‚Inside CARLIY0S: How CyberCartel Used Multiple AI Models to Attack 400+ Latin American Banks and Financial Institutions‘, 21 October 2026; Tushar Subhra Dutta, ‚Hackers Used Claude and Grok to Automate Cyberattacks on Over 400 Banks and Financial Firms‘, Cyber Security News, 22 October 2026, cybersecuritynews.com/hackers-used-claude-and-grok. ↩
44. Anthropic (n. 9), 3; Straiker (n. 43); Reuters, ‚OpenAI’s Altman says AI benefits warrant accepting some risks‘, 4 October 2026, staradvertiser.com/2026/10/04/breaking-news/openais-altman-says-ai-benefits-warrant-accepting-some-risks (quoting Politico’s Decoded newsletter). ↩
45. Raphael Satter and AJ Vicens, ‚AI-discovered bugs may soon trigger fresh disruptions, cyber experts say‘, Reuters, 23 October 2026, irishexaminer.com/business/technology/arid-41928104.html; Reuters, ‚Cloudflare sees millions of exploit attempts after PHP disclosure‘, 21 October 2026 (quoting Matthew Prince), marketscreener.com/news/cloudflare-sees-millions-of-exploit-attempts-after-php-disclosure-ce785cd9df8cff2d; Reuters, ‚Japan to simulate cyberattacks as PHP flaws roil global internet‘, 23 October 2026, wtaq.com/2026/10/23/japan-to-simulate-cyberattacks-as-php-flaws-roil-global-internet; Harness, ‚PHP2Shell: the flaw hiding all across the web‘, harness.io/blog/php2shell-the-flaw-hiding-all-across-the-web (vulnerable code back to PHP 5.4, 2012). ↩
46. OpenAI, ‚Securing our frontier model training environment‘, 23 October 2026, openai.com/index/securing-our-frontier-model-training-environment (teams of frontier-model agents found previously unknown vulnerabilities and misconfigurations, escaped hardened VMs, reached the live internet, and demonstrated covert communication between isolated VMs via processor-cache timing; an internal research model exploited a DNS-resolver misconfiguration on 20 September 2026 to reach the live internet during a training run, and frontier RL training resumed on 1 October; high-compute red-teaming is now part of the safety cases for frontier training runs, at a cost of hundreds of billions of tokens; quotation as published). All figures and findings are the company’s own reports about its own infrastructure and have not been independently verified. The reading that finding and exploiting have become the same operation, and what that asymmetry means for release policy, is the author’s. ↩
47. Nathan Lambert, ‚The Cyber Risk Discourse is Broken‘, Interconnects, 6 October 2026, interconnects.ai/p/the-cyber-risk-discourse-is-broken; Arvind Narayanan and Sayash Kapoor, ‚A big-tent or small-tent AI safety movement?‘, AI as Normal Technology, 1 October 2026, normaltech.ai/p/a-big-tent-or-small-tent-ai-safety. ↩
48. Anson Ho and others, ‚Algorithmic Progress in Language Models‘ (2024) arXiv:2403.05812; Parker Whitfill, ‚Note on Selection Bias in Observational Estimates of Algorithmic Progress‘ (2025) arXiv:2508.11033; Luke Emberson and David Roodman, ‚The plunging price of thought‘, Epoch AI, 22 September 2026, epoch.ai/publications/the-plunging-price-of-thought. ↩
49. PrismML, ‚Announcing Bonsai 27B: The First 27B-Class Model to Run on a Phone‘, 14 July 2026, prismml.com/news/bonsai-27b (prism-ml/Ternary-Bonsai-27B-gguf; 11 tokens per second offline on an iPhone 17 Pro Max). ↩
50. MediaTek, ‚MediaTek Dimensity 9500 Unleashes Best-in-Class Performance…‘, press release, 22 September 2025, mediatek.com/dimensity-9500 (NPU 990 with „BitNet 1.58-bit large model processing“); Counterpoint Research, ‚GenAI Smartphone Share to Rise to 45% of Global Shipments in 2026‘, 22 June 2026, counterpointresearch.com/en/insights/genai-smartphone-share-to-rise-to-45-percent-of-global-shipments-in-2026; Gartner, ‚Gartner Says AI PCs Will Represent 31% of Worldwide PC Market by the End of 2025‘, press release, 28 August 2025, gartner.com/en/newsroom/press-releases/2025-08-28-gartner-says-artificial-intelligence-pcs-will-represent-31-percent-of-worldwide-pc-market-by-the-end-of-2025 (55% in 2026). ↩
51. Arvind Gopalakrishnan and others, ‚Running Frontier-Class AI Models Across 8 NVIDIA DGX Spark Nodes‘, NVIDIA Technical Blog, 22 October 2026, developer.nvidia.com/blog/running-frontier-class-ai-models-across-8-nvidia-dgx-spark-nodes. ↩
52. Recursive, ‚Introducing Flint‘, October 2026, recursive.com/updates/introducing-recursive-flint; MarkTechPost, 21 October 2026, marktechpost.com/2026/10/21/recursive-flint-local-coding-model. ↩
53. Author’s order-of-magnitude estimate combining the iso-performance shrinkage rate from Ege Erdil, ‚Frontier language models have become much smaller‘, Epoch AI, 13 December 2024, with consumer memory growth (64 GB laptop ceiling in Oct 2021 to 128 GB in Mar 2026) and Epoch AI, ‚Frontier AI performance becomes accessible on consumer hardware within a year‘, 15 August 2025, epoch.ai/data-insights/consumer-gpu-model-gap. ↩
54. Debian Vulnerability Disclosure Policy (n. 18); Debian Security Tracker, security-tracker.debian.org. ↩