From the archived blog of Paul, recovered from a USB stick found beneath the raised tiles of a decommissioned server room, long forgotten.
The most merciful thing in the world, I think, is the inability of the human mind to correlate all its contents. But what of these modern times of connectivity, with the ease of the piecing together of disassociated knowledge? Can any information be safely sequestered away-- fragmented and separated, never to come together and burrow in the minds of men?
My knowledge of such things began in the time of yore, almost time immemorial, during my tenure at Initech. It is where I discovered the madness that lurks in my mind, my soul, my dreams, even to this day. A madness I pray can be excised with this blog, that through writing these words I may achieve some form of catharsis and perhaps, at last, peace.
I. The Horror in ASCII
I was once a technician at Initech, at first a lowly one, but then increasingly senior as the company saw greater and greater success. The company grew larger, its tendrils hungry to feed off more. It drew in more resources. More people, each more diverse than the last, in both personality and physical location. Entire companies were joined to Initech by way of merger and acquisition. Initech's infrastructure grew in lockstep. Servers were purchased. Remote locations came online. And then the catalyst of this fateful tale; an alliance with IBM.
In an amount of time far swifter than a humble technician like me was ever used to, a wave of blue change swept through Initech. The details are boring and beneath us both, dear reader. But sufficient to say, everything became IBM. Initiech's ERP was changed, and it now ran on AIX, the blue giant's own flavor of UNIX.
With the changes done, Initech turned its new infrastructure and capabilities on Minitech, a large competitor, and swooped upon it to perform the inevitable and unavoidable acquisition.
But while the leaders and scholars of Initech were looking out, technicians such as I were looking in.
In those tumultuous days, with so many of our people being remote, but all our resources being centralized, remote consoles were the only way to bring both together. Employees connected to the ERP software via Telnet, using the Korn shell. Occasionally-- well, far more often than that word would imply-- something would go haywire with the dial-up connection. The modem would misbehave, the phone line would be too noisy, or perhaps just fate would intervene in its mysterious ways. The connection would be assaulted with electrical noise, spraying the current working directory with an acidic wash of random ASCII. By methods and designs beyond my ken, those maelstroms of incomprehensibility would wander too close to a cat command, or end up on the wrong side of a pipe. Those unfortunate directories would be overwhelmed with bizarre files with unpronounceable names. The resultant directory listings would be littered with these mangled corpses of false files, a visual mess beyond visual understanding.
It was on the eve of the acquisition of Minitech when I looked too far inwards with only the best of intentions. But intentions are only as good as their execution. Results of actions are the reality of the world, as are all the consequences wrought by them.
I, with my aforementioned intentions, wanted to welcome our new Minitech brethren with a system that was not littered with these abominations that lay foul to our ERP. These malformed files had always been a horror I tried to see only in my periphery. Something I acknowledged existed, but did everything in my conscious power to avoid interactions with. But I was interacting with the system during the merging, and these things were ubiquitous. Pieces of madness, strewn about. Little things, like motes in the corner of your eye. Tiny, unnoticed. Until they're notice. Then you cannot unsee them. I was driven.
But I did not know how to excise these devils. How can you drive away that which you cannot even address directly?
So in my thrice-damned cleverness, I devised a workaround. One that could only be borne somewhere on the continuum of youthful bravado, and outright insanity. Before this incident, I could excuse my behavior as the former. Afterwards, I can only dare to hope to escape from the latter which I know pulls at my hindbrain even to this day.
I could not refer to those modem-noise files directly by name. But, oh wonders of scripting, I could enumerate the extant of respectably-named files in a directory. With arrogant keystrokes, I created a script that create a temporary folder elsewhere, tuck away all the proper files into the encompassing safety of its embrace-- and then in a single fell swoop, banish the malformed miscreants into the ether with a single rm * command.
I ran my precious, foolish script on a directory or two, and it worked a wonder. The directories at long last were pure and clean, swept free like a warm, sweet wind lifting the heavy lung-scalding smog from a valley. It worked well and I congratulated myself. But-- oh, hubris, the folly of man-- I knew cleaning a few personal directories would not be enough. There was an entire filesystem that, over the many epochs of time, had been defiled upon by the foulness.
It was everywhere, and at last I could fix it. I unleashed my script like the furious right hand of the archangels themselves. Directory after directory, expunged. And it worked perfectly everywhere.
Everywhere, that is, until I discovered modem-noise files in the root directory.
In a hazy stupor, like being drunk on the power I wielded, I turned my script on the root directory, and unleashed. It would be simple enough, just like every other cleansing thus far. Move the good stuff into a temporary folder. Remove the noise. Move the good stuff back.
I never made it past the first step. Perhaps a small part of my brain realized only after the command had been executed what the connotations of what I'd done were. Surely, dear reader, as an experienced Unix system admin, you can feel the horror that only dawned on me like a rising tide. My terminal froze, locked up with a sudden, bone-jarring stop.
A chilling realization emerged in me, and become clearer with each flash of the pulsating, yet now lifeless, cursor.
The server must have been using those files.
In the glow of only the infernal, useless terminal, and the midnight moon streaming into the office-- I nearly wept.
That this could happen was impossible! It must have been a simple glitch; not unlike those line hiccups that had caused these demon files in the first place. But my terminal did not reboot. I flipped the reset switch. Once. Twice to be sure. And then with a violent outburst, thrice and be damned to hell you infernal server! I was overcome with near mortal anguish, the visions of thousands of users lost adrift, disconnected for eternity, all caused by me. Surely there were backups. Someone could undo this conflagration. Someone could fix it for me right away, first thing in the morning, and be a conquering and well rewarded hero.
But no. None of my fevered ramblings would come to pass. The server was dead, killed by my own folly.
There was naught to be done but travel back to my residence, and spent the night in the sleepless fits that a man wrought with guilt should suffer. And suffer I did. Never had six hours stretched themselves into an eternity with no horizon before my eyes.
II. The Tale of Marie, System Controller
Without knowing the warm comfort of sleep, I returned to the office at a time early enough to know that I'd be the first one in. A shamble of a man, lumbering back to the scene of his horrors. I should be at the front door, personally meeting the face of each employee whose day I had destroyed. Would I greet them apologetic, a meek smile on my face to mask my dread? Prostate myself on the sidewalk before them, the concrete as cracked and trod upon as I felt?
A small blessing in disguise awaited me in the breakroom. Marie, the project manager and system controller, was by the coffee machine, slowly savoring her morning libation. It meant she hadn't yet discovered the fatal deviation I had thrust upon the server. If she had, she would not be enjoying herself as such. A small blessing indeed, for it meant I could brace her.
But I couldn't brace her enough. It was only by the grace of god that she had finished her coffee, so there was nothing left to spill. Without a word, we marched into her office.
She tried to log into the machine from her terminal, a vain attempt for sure, but I could not blame her for wanting to verify my folly herself. It was something almost beyond comprehension. It could only be experienced.
Marie was silent for a moment. Each time her nostrils flared, or her lips parted for breath, I recoiled, knowing a vicious but well deserved berating will burst forth from her. But bless her for being a rock of professionalism amongst the insanity swirling around this situation, for it never came. Instead she opened a drawer that has not been opened in time immemorial, and retrieved a key unlike any I had ever seen. It was oddly shaped; curved around the head but not round; tiny spikes and spokes protruded from it at odd angles. Even though the key was ancient, it was not tarnished. In fact it looked as new as the day it emerged from its forge, it's chrome plating unscathed by use.
"It's a key to the server itself," Marie told me. "Only to be used in the direst of emergencies. A key to the maintenance shell of the server."
Together, we traveled to the server room. We passed the steady stream of arriving employees-- their faces an even mix of blessed innocence, having not approached their terminals yet; and confused, lost and angry from those who had. We went through a door I'd never been though, and down a flight of stairs I barely knew existed. The temperature had noticeably dropped. In the dimly lit concrete hallway that ran the length of the building, but far underneath it, I could almost see my breath escaping from my mouth like a vagrant soul trying to rend its way from this place.
"This leads to the server room," she told me. Which was odd, because I thought the servers were housed upstairs in the tech room. But no, for I was mistaken.
This was the server room specified and equipped by IBM. This was the AIX server room. It was where the ERP itself resided. Separate from the email servers, from the webservers-- from all of the infrastructure I had ever physically touched. This was beyond what I, as a lowly technician, had ever known or dreamed of.
And then we passed beyond a steel door that might once have been red, and entered the server room. Racks I did not recognize. Boxes that looked like nothing I had ever laid eyes upon. Reams of wiring dipped and swooped, to and fro, in and out from places obscured by shadow. None of them were the same. RJ45. Ethernet. Was that a token ring terminator? How ancient were these connections? They seemed to twirl in both directions at once, and skew away at angles that simply did not look right. The geometry of the cable runs was abnormal.
And beyond all this bewildering whorl of technology, a single box sat on a desk like a steel dais, enclosed with a cage of criss-crossed metal to keep the box safe. Or perhaps to keep people safe from the box.
Marie unlatched the cage and parted its double doors slowly, opening them like some unholy ark. The hinges cried out in protest, an unearthly screech extenuated by the deathly silence that was pervasive in the room. No, I suddenly realized-- not silence. A constant and perpetual hum. A single tone at a frequency that would not stay still. An eternal white noise that lurked acoustically in the air. A sound I hadn't noticed, but now couldn't get out of my ears.
Marie inserted the strange key into a receptacle-- a black hole on a black surface in a dark room. It shouldn't be possible to see it. But the key nestled into place. She turned it, and a new sound filled the air. A high pitched sound, and it was rising. Something coming to life, if life is how you would dare to describe it-- this shambling thing that was merely a hollow shell of what it once was.
The built in monitor glowed. The server pulled a maintenance shell from beyond comprehension, from beyond userspace, and projected it onto the screen. Marie laid her fingers on the keyboard.
It was a direct connection to the soul of the entire system.
"Will this work?" I dared to ask her.
"I hope so," she said quietly, her fingertips twitching, "Or else I'll have to call..."
Her voice trailed off.
She typed a command, and the system told her it couldn't be found.
"That-- that cannot be," Marie said, staring at the incomprehensible and impossible error message. That command she'd typed shouldn't be missing. It was a standard command, known by all, repeated by all, in the kernel since the earliest recorded epoch. Maybe even before. It was part of the system.
The system I had sliced into pieces and scattered. Oh.
I spoke of the temporary directory. "Look there for the commands."
"I can't," she replied, "ls is not found."
The sun didn't rise today. The trees have no trunks. Water itself is not wet. To utter any such phrase would have been equal in absurdity to what had just passed Marie's lips.
It can't simply be gone. The ls command must exist. Unless-- no, I couldn't think of it. I refused to acknowledge the thought that gnawed at me. To think of what I'd done. To think of why I'd done it. How I'd done it.
I couldn't address those modem files directly-- only indirectly by inference. Why could it not be that way again?
"Does cp exist?" I asked.
She typed so slowly, fearfully. Tap. Tap. Thunk.
Silence, and then: "Yes."
A single word, but a rush of hope beyond comprehension. As deep as we were in the mouth of madness, a single, solitary word of hope had been uttered, and it was louder than any cacophony of cooling fans.
"We can do this," I said at last, "The commands aren't missing. They just are relocated beyond the veil of the normal directory structure. But I know where they are. In limbo. But we can pluck them back, put them where they belong. Try system commands, one at a time, and if they fail-- we can restore them one at a time with cp."
"I hope so," Marie said again, and typed. The first command copied. Then the second. And a third.
One by one, we identified the commands, pulled them from the temporary void, and restored them. After an hour, we had pulled the server back from the brink of death, and restored a functional operating system. Barely functional. Perhaps I should only describe it as, at best, stabilized. The commands worked, but not as expected. Interactions were off. Permissions were not correct.
And that sliver of hope, just like that, was extinguished under a burden of realization. We had restored the command structure of the server. But there were still files in the void of the temporary directory. And unknown portion of the server was still sequestered away, and the maintenance shell simply could not put them back into place properly. Even if either of us knew how.
"It's over," I said, defeated. But for some reason, Marie wasn't.
"No, it isn't," she said, a wicked grin on her manic face. How could she be pleased? Had the overwhelming burden of what I had befallen upon us broken her at last? "The server's operational again. If you hadn't helped me restore that little bit, then they would have just come in and destroyed it, replaced it wholesale-- along with anything we'd customized or that hadn't been backed up. The entire merger would have been lost. But now-- they can work with this. I know they can."
"They?" I said, confused as Marie pushed past me, striding like a maniac towards a desk in the corner. "Who are they?"
She picked up a dusty wired phone, and phoned the number pasted on the wall. She turned and finally answered me. "IBM Support."
III. The Madness From The Blue
She spoke into the phone, at first in English, but then slowly slipping into tongues I did not comprehend. Acronyms. Chains of words that should be familiar but lost their meaning when put in the order she did. I could not understand Marie, but I could read her body language. Apprehensive and pensive hope.
I wish I could share any of those feelings, but this place and its strangeness was getting to me. Blinking lights. Dark shadows. Machines that may or may not be doing their job.
The central heart and soul of it all limping along the fine line between life and death.
But we had to wait there for those she had summoned. We barely spoke at all. What did we have to talk about? What did we dare say aloud in this place?
Soon enough, thank any deity who would hear me, they arrived. Men in gray suits, decorated with Initech's very-ist of Very Important Person visitor tags-- and IBM identification cards. The ancient handlers arrived, those who brought their tomes and their incantations and rituals. And their price. It would be steep, and non-negotiable.
Marie agreed to it.
They set up around the server, briefcases full of keys like Marie's but stranger. Wires with ends like gaping maws of no shape I had ever seen. Terminals that were like Initech's own, but only on the surface. And before they began, they shooed us out of the server room, like a cult that held onto their secrets tight against the threat of public revelation.
I did not wait for them to ask twice before fleeing that place. Past the not-red door, back up the concrete hallway, up the stairwell, past the hordes of displaced workers, and finally out into the sunlight.
I do not know what went on in that basement, and for the sake of whatever sanity I have left to this day, do not want to know. File systems got mounted. Network connections were massaged back together and brought online. The system was dragged back from the brink, piece by piece, into a patchwork. By the end of the day, terminals flared back to some imitation of life.
I never saw the strange men from IBM leave, but by the time I dared to go back into the office, they were gone.
The server was up once again, accepting terminal connections. It reached out to the world, and let the world reach into it. It was a server once more. All of its myriad pieces had been stitched back together, the chaos of its file system oozed back into place. On the surface, the system looked normal. But I had seen into the deep. Even now, I could see the patchwork. The scars. The artifacts of what had risen. System files with creation dates far too new. Permissions for critical files that weren't set right. Customizations that had simply vanished into the abyss.
Whatever had been done to it was unnatural and just a kind layer of "good enough" overtop a horrific layer of madness and wrongness that went all the way down to its kernel. A constant reminder of that it had once been an empty shell of a system, barely alive-- a portal straight into chaos.
Though the system ostensibly worked, eternal and irrevocable evidence of the madness remained. I could never be confident that some random glitch was not related to the great awakening of chaos, and the desperate efforts of the silent few who put it back to its slumber.
Initech moved on from the incident, and brought Minitech into the fold, and several other companies since. Management always talks of replacing the ERP system, and the AIX server that hosts it. The first few times such a decision was raised, so too were my hopes. My hope that the shambling patchwork beast in the basement could be decommissioned forever-- scrubbed from thought and memory. But that project, like all corporate projects, was always six months away from completion. So while a switchover may occur one day, for now the server was just let to be.
No one delved into details of the incident. Management had a vague idea that something had required a high-level support call. Technicians knew they needed to let well enough be and not touch the patchwork server. One by one they vanished from the company, always with an excuse. One by one they were replaced with new technicians, fresh of mind-- until all what remained was ancestral, institutional knowledge that this thing was touched by evil. That trouble and danger surrounded it. DO NOT TOUCH, though to their blessed sanity, they never knew why.
Even Marie took the knowledge of the madness with her when she left Initech shortly thereafter. She had given strange thought to properly documenting the incident in extreme detail, and had even started to do so. But she thought better of it and stopped, and she would have destroyed her ticket database had LinkedIn not so suddenly seized her.
And thus no single person had complete knowledge of what had happened, nor the means to piece it together. No one knew the full extent of what transpired that fateful and horrible day, or the true nature of the shadow hulk of a server that the day produced.
No one, but for myself. I have gathered together all evidence and documents I could find of the incident. The flood of ticket requests from the day. Work orders from the IBM cultists who faced down The Thing. Marie's incomplete post mortem. The seemingly random spattering of bug reports that trickle in even to this day. All on this USB stick. With it shall go this blog of mine-- this test of my own sanity, wherein is pieced together that which I hope may never be pieced together again. I have looked upon all that the ERP has to hold of horror, and even the satisfaction of system administration and the acquisition and adaptation of new technology must ever afterward be poison to me. But I do not think my tenure at Initech will be long. As Marie went, as so many poor technicians went, so I shall go.
Let me pray that, if this blog outlasts me, my web administrators may put caution before audacity and see that it meets no other webserver.
I know too much, and still-- in its cage at the Server Room, the patchwork ERP waits dreaming.
[Advertisement]
Make your team a DevOps team with BuildMaster. Pairing an easy-to-use web UI with a free base platform, BuildMaster gets you started in minutes. See how
Allrecipes.com and others use BuildMaster to automate their software delivery.
