
“If not properly subject to human controls, future development in artificial intelligence may also have the potential to be used to create novel threats to public safety and security, including by enabling the creation and the proliferation of weapons of mass destruction, such as biological, chemical, and nuclear weapons, as well as weapons with cyber-offensive capabilities.” California SB1047 Sec 2.C
“...Benchtop DNA synthesis machines can help rogue actors create new biological agents while bypassing traditional safety screenings… AI chatbots can provide step-by-step instructions for synthesizing deadly pathogens while evading safeguards… With AI, the number of people that can develop biological agents is set to increase, multiplying the risks of an engineered pandemic. This could be far more deadly, transmissible, and resistant to treatments than any other pandemic in history.” Center for AI Safety
The backers of California’s SB1047 routinely cite AI-enabled bioweapons as a threat justifying the radical regulatory regime that places a locus of liability on general computational models, rather than on particular dangerous applications or criminal acts.
For six years I've worked on scaling generative AI models at a leading industrial lab - I’ve worked on some of the largest language models ever trained. But before that I was an experimental lab-scientist for two decades working in synthetic biology, optogenetics, and immuno-oncology.
As a biologist I feel compelled to comment on this risk-scenario, speaking solely on my own behalf as a resident of California, and as someone who generally admires my state senator Scott Wiener, the primary sponsor of the bill. The claims being made about biological risks do not reflect scientific reality.
Those of us who have made a living in the practical, experimental arts of the world know: It is not easy to explain to someone how a line cook tests the elasticity of a steak to measure doneness, how a machinist listens to the sound of a cutting bit for signs of a good feed rate, or how a biologist reads the fluorescent smears in a gel to see that a polymerase chain reaction's annealing temperature was low. These tacit skills are acquired by hand, via repetition, under apprenticeship.
The preparedness and safety studies written by various AI labs on the subject of biological risks all try to assay the ability of users to elicit dangerous experimental recipes from language models. None publish the protocols or results in the sufficient detail necessary for an independent review, but it is clear that none of them actually assay the ability of any of these study participants to translate their dangerous chats into actual bench practice.
Having trained many people in experimental biology, I can tell you that it is a long and cruel apprenticeship requiring the disabuse of enthusiasts' ideas that manipulating the machines of life is straightforward. One can watch their sanity fray as they slowly learn to stitch DNA constructs together through the arcane and interminable mixing of clear and precious droplets of fluid. One can watch their bright eyes dim as they deal with failed DNA assemblies, failed cell transfections, fritzing instruments, and their own faltering punctiliousness. One can only try to cheer their fatalism as they learn that even the smallest of ambitions will demand the sacrifice of their youth to the lonely nights of a laboratory.
So have a bit of skepticism about these prophecies from those who have clearly never done biological work. I have built DNA sequencers and synthesis platforms by hand: know that there are no turn-key DNA synthesizers that let you print out new proteins and organisms without engaging in the expensive and long-winded protocols of molecular cell biology. The presence of pathogens in DNA databases does not imply that these dire mechanisms are there to summon from a keyboard - they are intricate machines that must be built and cultured by experts, not kabbalistic formulas: good luck to the poor asshole trying to learn from a chatbot how to assemble a 20kb ebola virus from oligonucleotides - let alone a 190kb smallpox genome.
Chatbots today are as likely to emit reasonably formatted nonsense as they are to improve upon a quick protocol review on the web, and they are helpless to instill the “hands” possessed by skilled professionals in those who have not earned them already. For those sincerely wanting to improve biosecurity: it’s worth listening to working scientists instead of those LARPing a biosecurity mindset with ulterior motives. There is an entire field of determined biologists and physicians that have long stood watch against the designs of madmen and the chaos of nature’s wild agents.
My more enthusiastic AI safety colleagues insist that my fears are not inspired enough. They insist that we must lay down the regulatory roadmap early to protect ourselves from the scaling curves rocketing up. From that apotheosis of computers that will write our business plans for us and raise up empires for the elect. From minds that for an encore will simply think away the scientific barriers that cradle our bounded existence.
But reality is unimaginably detailed: Solving the energy of a quantum mechanical system requires computation that scales exponentially in the number of atoms in the system.1 We have to calculate those energies accurately to derive the reaction kinetics for biological molecules, and we have to integrate those dynamics across aeons of molecular time to capture the processes of a living cell. This is simply and utterly intractable even using all of the classical2 supercomputers that we could build in the foreseeable future.
As for superintelligences peering through the veil of sense-data to discover an approximate hidden structure of reality: We have and we will continue to use neural nets to approximate many particular aspects of biology such as protein folding, but using AI to “solve” biology requires an enormous amount of high quality data about protein and cell function, and by and large there is no such data. Biology is barely a quantitative science, and outside of genomics we are hard pressed to find any reliable data to constrain our approximations.
Without that data, you can't accurately predict how an enzyme mutant is going to affect kinetics without actually doing a library screen yourself. You can't predict how that change will affect cell signaling circuits without doing a cell-based assay yourself. You can't predict how that might affect a viral life cycle inside a host environment without yourself performing animal and human challenge studies in a large population of captive hosts. You have to do iterative experiment-driven development to make anything in biology work, including creating a new pandemic as a budding evil scientist.
This isn't unique to biology. You have to iteratively develop any new physical innovation, because reality as a hypercomputer has more detail than any mind can practically approximate. Neural nets can encode intuition hard won from past experience, but the amount of quality scientific and industrial data in the world is shockingly sparse, and extrapolating to new domains is still rate-limited by the material data generation process: barely-supported trainees grinding away their lives in penury in academic and industrial labs.
It is our great hope in biotech that coupling AI with industrial lab-automation will grant us the tools commensurate to the challenge of making our biology predictable and controllable. But make no mistake than any of this will occur automatically, or without the blood and sweat of a new generation of engineers doubly skilled in the substrates of silicon and carbon.
And for their sake let me close with a warning:
Prophecies of a biological apocalypse have been preached for decades, dating back to well before the storied Asilomar conference on recombinant DNA. They are still a perennial mainstay of science fiction and policy discussions alike. Today many regulations exist largely in response to these prophetic visions rather than in response to grounded risk scenarios built from scientific facts.
I do not wish to make a libertarian point against regulation in general. I know that most people crafting regulations are motivated to do good, and I support much of the extant frameworks. But biologists are forced to routinely disprove the possibility of even the most fantastical risks even after a lifetime of improvements in our understanding of the living world. We may never know enough to acquit ourselves of our prophesied culpability.
We are still imprisoned in the narrative frameworks of decades’ past: We live in a food-insecure world that is still terrified of GMOs, but that ignores the blanket of Buffel grass suffocating the endemics of the American west. We struggle with accusations that our vigil against disease is a conspiracy to cause it, and are thus checked in our defenses of a vulnerable world, a world that does not grasp the scale of the incarnadine manifold of industrial animal flesh that is pregnant with new plagues.3
This is a world in which great ideas in biology and medicine are often not pursued, because the regulatory environment we built around them has made their expected value negative for rational investors. May fortune smile on those startups pursuing new medical diagnostic technologies, or engineering new non-cereal transgenic crop species, or developing cell therapies for crippling but non-fatal diseases.
We work in a world where biological innovators largely acceded to the narrative demands of the dire prophets. We still manage to make halting progress by selling our time and companies and dreams to firms large and connected enough to bear the burdens of the present regulatory frameworks. But it is a sclerotic and privileged world of institutions and wealth that we are forced to traffic with. A world insensate to the needs of those that it has chosen not to help because it has lost the courage to dare for anything better.
Let's not allow prophecy to similarly be the driving force of the regulatory narratives around machine intelligence. The machine-learning community has operated with a historically remarkable openness to society. We should protect that culture. We should protect that freedom to compute, to think4, to innovate, and to share what we’ve learned and built without culpability. We should demand that regulations concerning us operate within the narrative of our proximal reality: to address the risks to society that we are certainly creating, rather than addressing projections which live beyond any credible predictive horizon.
We should be telling better stories about the world that we are about to build.
Even the approximate non-relativistic Schroedinger equation for a system of N interacting particles requires solving a complex partial differential equation in 3N dimensions. Even performing a single FFT for a single solve step in 3N dimensions with grid size K costs ~O(3N * K3N * log(K)) - exponential in particle number. The native energy scale of atomic quantum simulations is the Hartree (27 eV), and thermal kT scales at room temperature is roughly a milliHartree (0.025 eV), requiring very high accuracies of any approximate simulation to correctly predict kinetics de novo. Even if we had access to an energy-level predictive oracle, we still have to model absolutely enormous numbers of conformation states and diffusive interaction pairs in a cell. There is an astounding temporal hierarchy problem intrinsic to simulating biology: Biological systems involve processes taking place across 12 orders of magnitude in time even for the fastest individual E. coli cell. You aren't directly simulating any of that.↩︎
Y’all can feel free to go ahead and ban quantum supercomputers capable of generic quantum simulation. Maybe wait till we get around to building one though.↩︎
In case I lost some of you here, please go watch the entirety of the recorded and adjudicated rootclaim debate between Peter Miller and Saar Wilf on the topic of sars-cov-2, and learn what a polytomy is. I used to design synthetic viruses for a living and had a professional hobby of repurposing pathogen virulence factors for escaping host immune surveillance: our recent pandemic was utterly basic, and the only thing interesting about it at all is seeing the visceral need of humanity to place culpability within itself, because it simply cannot deal with the cosmic horror of living in a wilderness of machines that would casually consume those we love.↩︎
Robert Laughlin foresaw a lot of the rhetoric being used in today's AI debates, and I recommend his reflections at The Crime of Reason: And the Closing of the Scientific Mind.↩︎