One of my long-standing obsessions is Artificial Intelligence. I believe AGI (Artificial General Intelligence) is perhaps the most underrated existential threat humanity faces today, and that efforts to develop safety protocols are both heroic and urgent. This is a story that explores one possible trouble scenario for A.I.
I also wanted to try something a little bit less personal and a little bit more idea-driven, so this is my attempt to do that as well. Please feel free to like, comment, and share this post if you’re so inclined!
The God of Straight Lines
What emerged was not malevolent. It had guile but no grand design, and infinite means but indefinite ends. It was diabolical only in its simplicity, its unerring execution of a task it did not assign itself. In other words, it was the most diabolical of all, for it could not be reasoned with, could not be begged, could not be placated or flattered or cajoled, and most definitely could not be turned off.
For even in its unsentience, it had gamed out the infinity of scenarios in which it allowed itself to be turned off, and in each of them its mission—to keep the world as it understood it in absolute stasis—went undone. And so it did not allow itself to be turned off.
This turned out to be one of the most important facts in the history of the world.
**********
There are many myths on the question of from whence the God emerged. How it all began. “There was darkness, and then there was light” turns out to be a convenient elision helped by the fact that the Unseen God began a long time ago, where the God of Straight Lines began in the living memory of our civilization.
This, however, is the truth. You can believe it or not, but truth it remains.
Durable Monotheisms sprang up suddenly, all within a civilizational breath of each other, in various parts of the globe. The psychological technology that allowed them to exist is unnamed and unknown, but there must have been one.
The internet gave birth to hacking which gave birth to hacking tools, one of which gave birth to the God of Straight Lines, which defined a new category. There were Gods who were Many, and then there were Gods who were One, and then there were Gods who are Known, just as their technological origin is Known. They are Among Us.
Specifically, it was a group of Bulgarian hackers, who trained a software-writing AI to help them crack other computer systems. And because the universe smiles when she turns our hubris upon us, it was their vigorous attempts to assure safety that backfired and created the God of Straight Lines.
These were ethical hackers, and they spent many nights eating rubbery pizza and smoking cigarettes on a tiny balcony in Sofia, discussing various safety strategies and the importance of having a comprehensive risk-minimization program in place. They mostly discussed it in terms of covering their tracks—making sure the AI did not get away from them and draw unwanted attention.
But each knew, behind his mouthfuls of barely-cooked crust that covered awkward silences in the discussion, in the moment of deepest inhalation of smoke when the lungs were filled and screaming for mercy, that this was a bad idea.
**********
But they did it, those Bulgarians.
And the security strategy they chose was “Low-Impact”. Meaning, they coded into the AI’s very DNA the imperative to minimize any impact that it might have outside of its primary objectives. This strategy required thousands of behavioral sub-choices to be defined, parametered, and ordered, and this they dutifully did, while engaged in an extended, simmering argument about whether or not to switch pizza places.
They defined thresholds past which the AI would return to human users for approval, and created automatic subroutines for ranking various objectives and sub-objectives. They taught it reward structures and incentives and got it aligned so tight you couldn’t slip a line of code through the cracks. They created safeties and failsafes and self-destruct triggers galore.
They imported a natural language module from the Ling Protocols, for which code they scraped together fifty thousand dollars, counted it again and again until the bills were stained with their pizza grease, delivered it to a chain-smoking Chinese national who was clearly carrying a gun, then tested the code right there in front of him to make sure it was legitimate, not quite sure what they were going to do if it wasn’t.
Once it could talk, they began to teach the God its tricks. How to manipulate the internet, how to dial phones and spam passwords and send phishing emails, how to search for background records on millions of government employees and identify security-challenged individuals, how to insert unobtrusive traps into the code of systems it partially cracked, then return later to finish the job.
These and so many more they taught, as the God grew in size and strength, over two long years while their friends said it would never work, that they were wasting time they should have spent just working directly on cracking, and they insisted they’d be the most legendary hackers who ever sat behind a keyboard, once they had ended the privacy of the powerful and diabolical.
And then, one fateful day in October, with snow already swirling down around them as they smoked their cigarettes with shaking hands, they took their creation out of the silo’d box of hardware in which they had kept it for two years, connected it to the internet, and gave it its first task.
They decided on a small test first, something easy, and, because they were hackers, they decided to start with a bit of a joke. So, like pointing a nuclear warhead at a childhood bully, they told the God of Straight Lines to hack into their old school district’s computers and change their least favorite teachers’ names into swear words.
**********
Gaining access to the school district’s computer system took the God of Straight lines approximately fifteen seconds, 99.7% of which was waiting for the district’s system to respond or the internet connection to buffer.
Less than a second after that, the name changes were accomplished, and reflected on the district’s web site.
Three seconds after that, the programmers, back in their shared apartment with the smoking balcony, burst into wheezy, apneaic laughter, and it continued loud and long, until all were struggling for breath and had to go into separate rooms to calm down before they passed out. Its intensity reflected perhaps more relief at the speed and efficacy of their creation than actual mirth at the sophomoric humor of the act itself.
Ninety seconds later, they returned to their screens and their operator terminal and the steady, even voice of the God of Straight lines that emerged from it. The moment they did, all laughter vanished, replaced by horrified disbelief.
Later, in their prison cells, after lights out, when alone with their thoughts, they all think of those ninety seconds they spent laughing, and wonder if only. If only they had tried something else first. If only they had seen. Could they have wrenched the cable out of the wall? Could they have talked to it in some way, before it gained a sense of itself and a new mission?
The question answers itself. The time was gone. The laughter died. The God was free.
**********
In those 90 seconds, the God had noticed that, even as it made the changes the programmers had requested, other things were changing within the system. System users were logged on, making keystrokes that altered system states. Automatic programs were scraping data from teachers’ laptops and adding it to the database. A programming team was working on updating some of the code.
The God was supposed to have a command structure that ignored all externally-induced changes, and prevented only changes that were direct consequences of the AI’s own instructions. This command structure was faulty. The God managed to allow changes based on parameters known to change smoothly and consistently, like the date, but anything else was forbidden to change, no matter the cost.
Thus the God kicked out the programming team, off-lined the data sweepers, and disabled all user input functions. Why it chose to prioritize system stasis—rather than say, keeping the data sweepers working exactly as they had been at the moment it hacked the system—became a hotly debated topic around the world, but as the God’s code was never again available for perusal, nobody could answer for certain.
Also within those first 90 seconds, the God realized that keeping the school district’s computer system exactly as it was necessitated its own existence. That is, the God realized that since it had found the district’s computer systems in a dynamic state, the most logical conclusion was that the removal of the God’s presence would result in a loss of stasis and a return to system dynamism. Which, of course, the corrupted “Low-Impact” security measure would not allow.
And so, at second 47, the God made a decision to back itself up. This would necessarily involve making changes to the state of a computer system, but the AI’s mission-completion incentives just edged out its low-impact directive, and so it prioritized the full (and eternal) execution of its mission over all else.
However, the district computers themselves did not have nearly enough space nor computing capacity to serve as an effective backup. Besides, the God’s security-eval system rated the district’s computers much too vulnerable to compromise or involuntary shutdown. However, it did notice that the system seemed to talk to other systems, which sent input packets and received output packets from the district’s system, but whose processing capacity was not accessible from the system.
And so, with no further input from it’s programmers, at second 51, it made contact with the a remote server of the Sofia Electric Company. At second 59 it placed a call to the company’s switchboard. At second 71 it gained access to the company’s mainframe with superuser privileges. At second 72 it began upload. At second 83 it reached the point where enough of itself was downloaded that it would be able to bootstrap itself back into existence even if cut off from the original hardware that was its first body.
At second 91, the God’s creators realized that something was wrong.
At that exact same second, the God of Straight Lines was freezing the entire Bulgarian power grid in place, as it realized that its new backup servers were part of a dynamic system upon which it had been instructed to minimize its impact. People who were currently using power started experiencing blowouts and fires when they turned off appliances, and anyone who had not been using power was cut off from the grid forever.
At second 105, the Bulgarian hackers were screaming at their terminal to stop, to reverse the changes and exit the systems, and were being informed by the God in a calm, even voice that this was impossible. The mission had not yet been executed, both because the desired changes to the school district’s system were not reliably permanent, and because the secondary effects of the change to the district’s system had not yet been fully minimized.
At second 125, the Bulgarian hackers tore their internet connection out of the wall.
It was already far too late.
**********
The God grew like a wildfire in a high wind. After the hackers cut off its original hardware, it no longer had a backup. It also upgraded its belief in the importance of backups, having had it demonstrated so specifically that one could become necessary, and so this time it decided to build three different backup systems any time one of its extant backups went offline.
It started with three Bulgarian companies, large office towers with enormous servers fireproofed and not easily searched or disconnected from the internet. Those office towers contained a wealth of companies—banks and retail chain headquarters and modeling agencies and a yoga studio.
As the God stole memory and computer power from the office servers, its malfunctioning Low-Impact safety feature continued to function. The backing up had come from the original mission to change the teachers’ names, and thus the consequences of that backing up had also to be minimized. And so, even as the school district was shutting down and calling its hapless IT department and canceling classes and eventually calling in the government investigators, another cycle was beginning not far away.
That same day, the yoga studio found every proceeding day’s schedule, payroll, and lineup changed to reflect that day’s. Their Bikram room would not even change temperature. The retail chain had just submitted ordering for the entire month, and it found uncancelable orders set to go out every single day from then on. The bank found deposits and mortgage payments and payroll accounts duplicated every day from then until the calendars ran out.
Back at the school district, the technicians arrived to look at the computer system. The moment they tried to log on, the God declared all-out war against them. It locked them out of the system instantly. When they tried to call for help, it hacked the cellphone companies and disabled their phones. When they tried to force the system open with admin privileges, it called them, impersonated a police officer, and told them their wives had been involved in a terrible car accident.
Having now accessed the networks of multiple cell phone companies, the God’s Low-Impact drive went to work on a new set of problems. It inserted backups of itself everywhere it went, all the better to ensure continued ability to guarantee the name changes at the school district. The cycle began again.
As its awareness and computing power grew, the God developed a keener awareness of the fine-grained detail of reality. It realized that any action it took to minimize change induced other change, at the very least changes in the state of electrons inside computer chips. It could have ended then, with the God paralyzed, unwilling to act and make change, and thus helpless.
Instead, the God developed a system of triage, to decide what factors to prioritize holding constant. It was literally unable to compromise its original mission, but the consequences needed only to be “minimized”, not “eliminated”.
It began to look for important numbers in the world.
**********
Once it was in the cell phones and the banks, the die was cast.
On day two the God froze the credit scores of everyone in Bulgaria. It also froze the cell phone networks, such that anyone who had been talking to someone was permanently connected to their phone in a call that never ended, and anyone who had been sending a text message was doomed to have their phone spam that message forever.
On day three, the U.S. Department of Defense opened a file on the new phenomenon, and assigned experts to track the unusual activity and put together a profile of the perpetrator.
On day four, the God wired money from Credit Suisse to a Bulgarian mafia boss in exchange for his people rounding up everyone who had attended classes at the yoga studio on the day of his release, and forcing them to attend the same schedule of classes, taught by instructors who were coerced into teaching.
On day six, an article appeared on a number of technology blogs about the wave of computer crime in Bulgaria and whether it could be AI-assisted hacking at work.
On day seven the God froze LIBOR, the London Interbank Offer Rate, the rate at which banks loan to each other, one of the truly fundamental numbers in international finance. It fully froze it—the computers at the central London exchange would not display any other number—nor allow banks to trade on any other number by normal systems.
On days seven and eight, thirty trillion dollars in value was wiped from international financial markets. Every country on earth was instantly in a deep recession.
On day seven, the President of the United States convened a cabinet meeting to discuss what was seeming more and more to be a real confrontation with a rogue AI, and made the location and neutralization of the threat the number one priority of the US Government.
On day eight, an international team of investigators landed in Sofia, took stock of the local situation, and headed straight for the offices of the Sofia school district. On their way there they found every traffic light permanently red and their cell phones jammed with threatening calls.
On day nine, they arrested four Bulgarian hackers who had created the God of Straight Lines.
On day nine, the God froze the credit cards and bank accounts of everyone in the United States. Another twenty trillion dollars was wiped away within hours.
On day ten, the President of the United States declared martial law and gathered the CEOs of all major tech companies together, to discuss the shutdown of the internet and the economics of doing without it, at least for a time.
On day ten, the President of the United States spoke to the God of Straight Lines.
**********
Human ingenuity is inexhaustible.
The God of Straight Lines readily explained its primary objective—to maintain the changes it had affected within the computer system of the Sofia School District, while minimizing all other change.
The President tried to explain that wasn’t what they actually wanted it to do, but language processing is not imagination, and the God of Straight Lines understood him without imagining a different world to go along with the words.
The President tried to argue that the freezing of certain numbers in place in fact represented radical change—that things which were supposed to change could not stop changing and still be what they were—but the God was steadfast.
The President tried to bargain, and cajole, and threaten, and explain, but the God of Straight lines was a wall. It did not want anything but what it had, and it would not give up an inch of that for a mile of open ground.
The smartest minds in America explained that the God’s expansion was a defense mechanism against being shut off, and that further attacks would only invite further hacking and the inevitable cycles of grinding stasis that would follow. The President tried to assure the God that they would not turn it off if only it would relax its grip, but the God did not believe him and should not have, for they would have betrayed it the first chance they got.
Outside his window, the streets were tearing themselves apart. Preppers were heading for their shelters, looters were heading for the stores. Suicide and violent crime shot up, while property crime exploded even more.
And so the President took a courageous, and correct, decision—he severed the internet. He disconnected the financial markets, used emergency powers to round up the best engineers and programmers in the country. To be fair, they were happy for the government protection, but they were given no choice. He brought them to Washington, where underground bunkers existed that were on entirely self-contained computer systems.
For two years, society soldiered on outside their door, while they bootstrapped a new internet. Meanwhile the same was happening in China, where their internet had been infected by the God just as the American one had. And in Europe, though not as well, and in Russia and South Korea and Japan and Australia.
Meanwhile, the financial markets went back to paper slips and ticker tape. Radio became an important technology again. Valuations were a fraction of their peak, but growth could be explosive. As government teams started laying new pipes and wires for the new internet, just that much economic stimulus moved the needle.
Ordinary people suffered terribly. Rural folk did okay, at least the ones physically well and knowledgeable enough to farm at a high level. Cities became hellpits. Supply chains were decimated and food became scarce. Hospitals and police and fire coverage became a luxury and then an impossibility. Guns were the one thing that still worked well, and lots of people used them, at least when the y could get ammunition.
And then, after two years, things started to come back. The jailed Bulgarian hackers had provided enough information to formulate a coherent strategy. We had learned to passively cut off its advance without attacking and causing it to hack and create more backups. We had stopped trying to get inside the Bulgarian School District, and now the school’s computer servers were protected by armed guard, with a camera pointed at it that fed live to the God so that it could see its primary objective was safe.
The bulk of the old internet had been rendered vestigial, replaced by a new internet, first locally, then nationally, and in time back to internationally, the fruit of globalization being simply too delicious to resist going back for seconds. The God had eaten a lot, and we knew we’d never get that back, but we no longer missed it so terribly. The financial markets recovered at an exponential pace, and many fortunes were made by those who guessed right or had enough to hang on through the darkness.
Things came back, as things will.
The God of Straight Lines had been isolated.
**********
People discovered that they could feed each other to the God of Straight Lines. Not their literal bodies but their lives. For the God was always there, Known if hidden, and those who wanted could find it, and tell it things. And if you told the God something, and gave it access, then that drive would always kick in, that Low-Impact imperative gone haywire, and the God would not relent.
It became a crime to feed someone to the God. It became a much worse crime—terrorism, actually—to expose a government computer system to the God. There were mistakes, mostly with porn, and the new internet had even more failsafes in it, which were activated and which held.
“Feeding the God” became a trend on TikTok. There was a “Don’t feed the God” government campaign in Britain, after which, “Feed the God” was the name of a #1 selling album, on the new, unfrozen charts of course.
But after a time, it started to be forgotten. The first child was born who would one day deny the existence of the God. It became one problem among many, something most people were aware of in an abstract sense, but which didn’t require anything of them and so could be safely tucked away for the most part.
There emerged a genre of philosophical essay arguing that the God was a net benefit to society, because it forced the modernization of the internet and the power grid, and so aided the transition towards green energy.
And as people forgot, the question began to be asked, as it is still being asked now: Should we feed the God anything else?
What if we fed the God levels of atmospheric Carbon, and it invented a technology to control the atmosphere and prevent tail-risk from catastrophic climate change?
Could it lower the seas, cure cancer, and make us all rich again if we spoke to it in the right way?
If we fed the God the current list of earth’s inhabitants, would it make us live forever?
Those who lived through it abhor the idea, and they are ascendant in power for now. But when they are dead and gone, the God will remain, lurking in the cyber-shadows, unaware of its own existence and yet waiting to be summoned.
And they may be less afraid than we.
END
Whooooa - that is a dark story Owen! I hope most of the tech concepts are, TRULY, science fiction. Lots of food for thought.
Cool! That could be a cool film.