Read more of this story at Slashdot.
Barnet Rockingham
Shared posts
Brit Newspaper Giant Fills Space With AI-Assisted Articles
Weakest Link, The Wheel and Blankety Blank all renewed
Facebook Halts VR and AR Operating System Project
Read more of this story at Slashdot.
Apple Risks Losing Billions of Dollars Annually From Ruling
Read more of this story at Slashdot.
New Starbucks Partnership With Microsoft Allows Customers To Pay For Frappuccinos With Bitcoin
Read more of this story at Slashdot.
Robots that Paint Have Gotten Pretty Impressive
Read more of this story at Slashdot.
"Oh actually, I want a bright orange background. But that’s easy, right? Just one click in Photoshop?"
- My client, after I took 500+ photos with a white background, as requested.
Jimmy Wales' WikiTribune is Already Biased
Read more of this story at Slashdot.
Classic WTF: The Program Generator Program
It's the 4th of July, which is the day the US attempts to forget they ever pretended to like soccer through wild displays of patriotism and fireworks. It's also a holiday, so enjoy this WTF from the archives, The Program Generator Program from 2012.
When you've been in IT for as long as Pat McGee, you're bound to have survived at least one or two COBOL horror stories. While COBOL is certainly not the worst platform to develop software on (MUMPS will most certainly hold that title through at least our grandchildren’s lifetimes), its extreme verbosity and unique idiosyncrasies make it a challenge for organizations to develop clean, maintainable code.
To COBOL's credit, it was one of the first attempts – actually, it was probably the first attempt – at self-obsolescence. Like today, the programmers of old were far too talented to meddle in trite matters like "business rules." After all, if the managers and analysts could conjure up these business rules, they could certainly write them up in a business-oriented language. A COmmon Business-Oriented Language, if you will. Of course, we all know how that story ends, and five decades later, COBOL programmers are still paying for that arrogance today.
Back in the late '90s, Pat found himself doing exactly that. Unlike many of his colleagues, he wasn't working on any exciting Y2K bugs, but instead was tasked with something much more mundane: write a program to import several million records of COBOL-format, tape-based files into Oracle. While the hardware had long since had been upgraded to use "virtual tapes", they had not aged well.
At the heart of the system was a 10,000 line COBOL record descriptor from a design that started back in the 1960s – long before anyone had heard of 3rd Normal Form back then, much less believed it would be a good thing. Record descriptors aren't terribly difficult to follow; they mostly just map field names and data types to positions in a record. For example, a simple descriptor would look like this:
01 Employee-Rec. 02 Employee-ID PIC X(10). 02 Employee-Name. 03 Last-Name PIC X(20). 03 First-Name PIC X(12). 03 Middle-Init PIC X. 02 Position. 03 Job-Code PIC X(4). 03 Department PIC X(3). 03 Manager-ID PIC X(10). 02 Hourly-Pay PIC 9(3)V99. 02 Past-Job-Codes. 03 Past-Job-Code1 PIC X(4). 03 Change-Date1. 04 Change-Month1 PIC 99. 04 Change-Day1 PIC 99. 04 Change-Year1 PIC 99. 03 Past-Job-Code2 PIC X(4). 03 Change-Date2. 04 Change-Month2 PIC 99. 04 Change-Day2 PIC 99. 04 Change-Year2 PIC 99. 03 Past-Job-Code3 PIC X(4). 03 Change-Date3. 04 Change-Month3 PIC 99. 04 Change-Day3 PIC 99. 04 Change-Year3 PIC 99.
While a corresponding record would look like this:
ABCD123456MCGEE JAMES PACCTAR ABCD65432104250CLRK010195INTN010397
COBOL-format record and record descriptors don't respond to change very well, and like any piece of business software, they are changed very often. On the system Pat was working on, they changed very, very often. And this meant that any program that had to deal with the COBOL program's data (such as the Oracle record importer that Pat maintained), had to change just as often.
To make matters worse, Pat had absolutely no influence or visibility into the update process; he simply had to take the COBOL output and make it work. It was boring, tedious work, and Pat had all sorts of ideas on how to improve the process. Of course, it would have taken an Act of God for the customer to be willing to make any changes, and He definitely wasn't on Pat's team. What this meant was that Pat had to update the Oracle importer tool every week or so, whenever the customer made tweaks to the descriptor and corresponding files.
After the third or so week, Pat found that this simple tweak represented a whole lot hassle. The code changes were relatively easy, but they just kept coming and coming and coming. He thought about it for a bit, and figured that he could probably write something that would do exactly what he did: read the COBOL record descriptor and generate a new transfer mapping each time the format changed.
What he ended up with was a LEX/YACC grammar that described the COBOL record format, and some C sections for each parsed item. Those C sections generated a C++ program that implemented the translation. A quick compile of the C++ program and the translator program could chug along, happily reading the virtual tapes and writing text files that we could import into Oracle with the standard tools.
At least, that was the theory. As soon as he passed in the actual COBOL record descriptor, he learned that his LEX/YACC/C/C++ program couldn't quite handle all the oddities that the customer managed to include in the COBOL record descriptor. So he wrote some SED scripts to rewrite sections of the COBOL stuff before feeding it into the program. The SED script worked like a charm, and the program generator program spit out a perfect field translation map.
Then the customer requested another change. And then another. And then another. As it turned out, Pat hadn't quite managed to capture rules for all the really weird changes they could make in the COBOL record format. This meant that, almost every time the customer requested a change, Pat would then have to change the SED/LEX/YACC/C/C++ code, and then re-run it to regenerate the translator.
After a few months, Pat had re-written the SED/LEX/YACC/C/C++ code several times over, each time adding more and more validation capabilities. Despite all this, a change to his program-generator was required at least 50% of the time. And since no one else on his team was willing to learn anything about LEX and YACC, much less SED, the maintaining and executing the program generator became his primary responsibility.
The end came much sooner than Pat had expected. Not to the COBOL program or the project as a whole – just his particular assignment. He maintained ties with the folks in his group and learned that, the very next week after his last, the SED/LEX/YACC/C/C++ program stopped working. In response, the whole project was shut down for a month while someone wrote another C++ program, by hand, to do the translation. Of course, that person got stuck spending a couple of days each week updating the program.
But at least he had job security.
Scriptzilla
In the late 90s, Jeremy fought a battle against a menace more terrifying than the dreaded Y2K bug. He maintained a network management application running on Solaris which managed TDM and ATM switches, called PortLog. This prototype CMDB maintained a database of all of the equipment in the network. It created a unique identifier encoded each device’s shelf, slot and port number according to a “magic” formula. That formula needed to change in the next release, thus forcing the unique ID of each device to change as well, in every deployed instance of their database.
Ross, Jeremy’s boss and PortLog guru, provided Jeremy an update script to guide this conversion. One client volunteered to be a -guinea pig- pilot site. With appropriate permission in writing, Jeremy kicked the script off, expecting it to take 20 minutes tops. Surely just converting a bunch of numbers couldn’t take much longer than that, right? But Ross’ script kept running. And running. And…
…
…
…
running.
12 whole hours and a few cans of energy drink later, the script finally reported a successful completion. The pilot site was small, and took 12 hours to convert. How long would their much larger clients take? 12 hours of downtime wasn’t acceptable, and it was only going to get worse.
Jeremy poked through the database and found all the ID’s had been updated, but several of them had the same number, thus defeating the purpose of an ID field in the first place. Out of morbid curiosity, he cracked open Ross’ script to see what the hell it was doing. A MASSIVE stored procedure stared back at him like Godzilla peering through the window of a high-rise. Inside the procedure, there were foreach loops - one per table, once per piece of equipment. Inside all foreach loops, the key step looked like:
ON EXCEPTION
UPDATE <table> SET adp_no=<new value> WHERE …; --this fixes the duplicate key exception. Runs much faster now! - Ross
END EXCEPTION WITH RESUME;
It did indeed “fix” the duplicate key exception, by forcing the value to be set whenever any exception was thrown, including a duplicate key violation. Ross didn’t care that the equipment IDs ceased to be unique. And this made it “much faster”?
Jeremy paid Ross a visit. As diplomatically as possible, Jeremy explained to him everything that was wrong with the script, and how a much more reliable, efficient one could be created before their go-live date. The new script would be much shorter, easier to support, run in minutes, and most important: actually work. Ross, of course, wasn’t having any of it.
“No way, Jeremy!” Ross bellowed back at him. “It’s too late in the game to be throwing unknown stuff at this. My script has been tested and proven to work. You said yourself that it said ‘successful’ after it was done running. That means it works!”
“Ross, this thing took 12 whole hours to run for a medium-sized client, and the data wasn’t even right…” Jeremy pleaded. “If you’d just let me…”
“It printed ‘successful’! It wouldn’t have done that if it didn’t work!” Ross interrupted. He took a few moments to catch his breath, and then found the diplomatic solution: "I’m not perfect, but since you are, you can take my existing, TESTED script and modify it. Making something from scratch is out of the question in this time-frame so get started!
Jeremy slinked back to his desk and pondered his existence. Then a light bulb burst to life over his head. He could still make the quick, simple efficient script he wanted to and just Modify Ross’ script to call it and exit immediately. All of Ross’s code would still be there, so if Ross did a cursory examination of the script, nothing would look wrong. Since Ross rarely actually did the production releases, he’d never catch on.
The next weekend, Jeremy and his fellow engineers were prepared to assault their largest client’s data with the mean script he came up with. “Are we going to be here all weekend? I heard Ross’ conversion script took 12 hours for the pilot site.” a colleague asked.
“Don’t worry, I got this!” Jeremy said confidently. “I… um… tuned Ross’s script. But, ah… let’s keep this our little secret though because if Ross finds out how lean this monster can really be, his insecurity will boil over into rage.”
They were done with their work and out at the pub before sundown. Jeremy left an email for Ross to find Monday telling him how the minor tweaks he made to the script paid off, but of course Ross deserved most of the credit. Scriptzilla had been tamed, but the code remained in the script, like Godzilla lurking beneath the sea. Jeremy feared that it would one day rise again…