Sixty Hours later...

I have to tell you a funny story. OK, two funny stories.

***

Everything was working fine with the software by late Wednesday night. Or early Thursday morning. I started an overnight test, went home, grabbed a few hours sleep, got back to work to check the results. Everything looked good. Seems like we were set for the Release.

Got a call from Configuration Management. Did I realize that the revision number was incorrect? Seems that the previous person in charge of this project never actually did a release, so it should've been Version 1.

Hmmm. That's interesting. The previous person in charge of this project had told me we were on release 8, which made this upcoming release version 9.

Checking into the release database, it turns out they had never actually released a PRE-PRODUCTION version, only EXPERIMENTAL versions. (There are four types of releases: development, experimental, pre-production, and production.) We are required to release a pre-production version for Qualification tests.

But ... they ran Qualification tests on the first version last year, right? Well, I guess not. At least, not with a pre-production version of the software.

Oops.

Since the version number is embedded in the software, I had to rebuild it and re-run all my tests. So the release was delayed til the afternoon. But it got done. Now we could all relax and let the Qual testing begin.


***

The build was ready, checked in, submitted to Configuration Management, and available for the Hardware folks to use in their pre-Qual tests.

It was going to be so nice; I was going to be going home on time tonight!

Then I got a call around 5 p.m. The Hardware pre-Qual tests failed. My presence was requested in the other building (where they were running the tests). So, instead of going home as I had planned, I detoured over to the other building.

The whole lot of them were gathered around the test station, wringing their hands and casting aspersions at the software. I reviewed the test failure log and shook my head; couldn't figure out how that error had occurred. Unless ...

Then one of the guys suddenly got this weird look on his face and said, "I think I know what's wrong." We all looked at him as he walked around behind the test station and starting fumbling around with the cables, muttering. He was looking for something and not finding it.

Turns out, the test station wasn't set up right. Some signals weren't getting through because a signal converter box was missing.

The software had actually detected a hardware failure!

And that took us til 7:30.

Can I go home now???

Late Late Late Night Benediction

So I'm sitting here at work, it's ten o'clock, there's a new build to test, and the lab computer goes belly-up, BSOD, down for the count. At least until reboot.

Why am I sitting here in the lab at ten o'clock at night?

Because the software has to go out tomorrow.

(Actually, it was supposed to go out last Friday, but that was a billion bugs ago, and there's only a million more to go.)

It was thought to be 'done' this afternoon around 4 p.m., when the tests passed on the Test Rig across the street, and everyone in the group laughed and smiled and threw up their hands in celebration, and went home.

But something in the log file nagged at me, something that should not have been in there. The test was reporting a Pass even though the log data showed very clearly that it had failed.

Why wasn't anyone else worried?

They're all so eager to just be done with it. This project has dragged on long enough, and we have to get something down to Chicago next week to throw in the thermal chambers, and right now they're probably willing to put the box in the chamber even if it doesn't boot up, just to get it out of the way.

But not me. I'm a bloody perfectionist.

So I'm slogging through the code, adding debug routines here and there to try and figure out what's going on (this is, after all, someone else's code, so it takes a while to figure out what was going on -- so much for documentation!!), loading up the box and re-running the tests, gathering more data, analyzing it, going back over the code, feeling like I'm banging my head into a brick wall -

When suddenly it becomes clear. There's a major disconnect between the guy who wrote the software, and the guys who expected to use it. The guy who wrote it, he expected the code to be run once, and continue running forever. The guys who expect to use it, they are going to be starting and stopping and re-starting this software throughout the whole test sequence. But it wasn't written in such a way as to handle that case. It's definitely a run-once kind of code.

Oops.

So the next question becomes, how can this code be massaged to work the right way, without having to rewrite the entire thing from scratch? I'd really like to go home now...

---
{much later, after much coding & caffeine}

Massaged the code for awhile, to no avail. Still failing. Ran it in both manual and automatic mode. Thought it was fixed because it was passing in manual mode, but then it failed in automatic mode. Weird. Tossed in some more debug code.

---
{a little while later}

Something odd going on here. It passes twice, then fails on the third try. And it seems to fail ONLY when restarting. Like there's something getting hosed up during the restart. But what's so special about the restart? The process is just supposed to suspend itself, not discard all the memory. But it's acting like something in memory is getting overwritten while the process is in a suspended state. So much for the partitioned system.

---
{really, really late}

It's really odd, but I don't have time for odd things right now. I'm tired and want to go home. Looking objectively at the problem, it's obvious that something is getting messed up during the first re-run of the test, something left in an unknown state so that it causes the comparison logic to fail. And the logic is failing in the middle of a loop, so it's a good guess that restarting the loop might reset the logic somehow. But there's no easy way to drop out of the loop ... or is there?

OK, so if I put a special variable in the failure-detection logic, I can use that to ignore any failures right after a restart. It gets set by the restart call, then when the comparison fails, it simply ignores the failure and keeps ignoring it until the loop finishes. Then the special variable gets reset, the comparison logic gets reset, and we get a 'clean slate'.

Let's see if it works...

---
{a little while later - midnight}

It works! Hooray!

Now let's zip this puppy up and get on home...

An Experiment

My head is still swimming from all the meetings which have taken place this week. It's all part of the "Getting in Over My Head" plan for the summer, where I play the part of "Befuddled and Confused" in this bizarre Morality Play.

For the past three years, this program has managed to survive despite the fact that we've been ignoring the proscribed software development processes, instead doing things on a panic-of-the-week basis.

In the normal software development process, we take a set of requirements which have been negotiated with the customer and design the software on paper, get approval from the customer on the design, transfer the paper design to an actual design, match the design against the requirements by running tests to verify that the requirements have been met, then release the software - and all the documentation that goes along with it (design documents, requirements documents, test documents, test reports).

In our "modified" process, the managers negotiate a delivery date with the customer, along with an 'incentive' payment that guarantees (by appealing to executive greed) that we'll make that date; then they turn around and demand that the engineers work 50-hour weeks (minimum) in order to meet that date; then the engineers, in order to meet the delivery date, ignore all the paperwork; and after much pain and torture, they make the delivery date, but have no idea what it is that they have delivered.

So why are they so surprised when it doesn't work as advertised?

**

The "mandate" for my new position is to ensure that the aforementioned 'correct' process is followed. This is what is known as "bucking the trend". I'm not sure if success is even possible at this late stage of the game, but it'll be interesting to find out.

After all, the worst they can do is to fire me.

Happy Birthday, Mom!

When I picture my Mom, she's always sitting on top of a tractor, out in some nameless field, with a big smile on her face and her eyes focused out somewhere far ahead, driving across with the plow hooked up and a huge cloud of dust behind her.

That's my Mom, the tractor-driver. It always impressed me that when she was a kid, she didn't want to be in the kitchen with her sisters, but out in the fields with her brothers, driving the tractor. Spunky, that. (Besides the fact that you don't have to do the dishes when you're helping outside, right?)

I think I drive like my Mom, too. Well, at least I drive like I imagine my Mom drove when she was growing up in Indiana back in the '50s. Fast. Not reckless, but fast, feeling the power of the engine and the feel of the road under the tires and the wind in my face and the shifting of the gears as my hand moves the stick...

There was this time, years ago, not too long after I'd gotten my license, and Mom was letting me drive the car back from camp, just the two of us. And we were heading down this country road out in Virginia, going along at a pretty good clip (within the speed limit, mind!), when suddenly this idiot pulls out in front of us. There was not time to stop, no time to think, barely time to react. I pulled the wheel smoothly to the left, veered into the other lane, zipped past the idiot, and then pulled back over again. Didn't even slow down. Of course, my adrenalin was pumping like crazy. And I thought for sure Mom was going to freak out, make me stop the car.

But she didn't. I'm not sure what thoughts were going through her head, but she just sat there seemingly calm as a cucumber. I think she said something like, "That was close!". But that was about it. And she let me keep driving. And we drove on home.

That's my Mom. Calm in a storm. Handles the tough times. Lots of strength in a tiny frame. And oh, so much fun to talk to!

Happy birthday, Mom! Can't wait to see you at the reunion!

When I'm Forty-Four

Each passing year, it becomes more difficult to deal with stress and stupidity at work, and not because my brain is falling apart (which it is, but that's another story). It's simply that life is too short to put up with idiots.

Forty-four means that in twenty years, I'll be sixty-four - no doubt singing that old Beatles' song, if I'm still able to sing.

And that is the question. Will I still be able to sing? Will I still be able to play guitar? Will I even be able to walk unassisted? Or will some disease or accident take away my ability to do those things which I take for granted now?

Seems like a waste of time to spend the best part of my life at work.

There is a lot of short-sightedness at work. Plans and schedules are made as though people are an inexhaustible resource, as though it is no big deal for people to work 50-hour weeks for months or years at a stretch without cracking. But people do not live to work. They work to live. Right?

Some people apparently enjoy working too much. And I don't mean the "fun" work of making things, creating things, putting together something tangible, something you can put in your hand as a final product and say, "I helped make this a reality"; it's the other kind of work, the supplemental work of creating budgets and schedules and spreadsheets and tracking matrices and cost accounts and weekly status reports and all the little paraphenalia that is supposed to allow us to keep on track, all the paperwork that, in the end, is put away in a file and never seen again, for which they sacrifice their evenings, their weekends, their families, their lives.

It is very difficult to take those seriously, especially when the data is so casually tossed aside when its usefulness is over. Most of it is round-filed the moment the status meetings are over. And someone stayed up all night long to prepare it.

There is a long-standing tradition at most big companies to make fun of the bean-counters, those little bespectacled minions who turn over rocks to count all the bugs underneath, and have their formulas for calculating everything from the number of hairs on their heads to the temperature of the coffee in their cups. They drive engineers crazy, talking about Earned Value and Cost At Completion and other fancy accounting terms, screaming for data which they can plug into some arcane mathematical structure to produce yet more data to prove that the engineers are having too much fun at the company's expense. But that data isn't real, it can't be seen or felt or touched or soldered or wired or switched; it is transient, and thus meaningless.

Yet this data is revered as Important, Critical, a High Priority; and woe unto those who enter the realm of the Status Meeting without it, for they shall be flogged severely.

So we spend hours a day preparing our status, and more hours attending meetings to talk about our status, and even more hours revising our plans so that we can have better status tomorrow; and all the while, the product languishes in its broken state, awaiting the few spare moments we might have in our busy day to actually add some functionality or fix something that's broken or test it in the lab.

And we end up spending long hours at the office trying to fit our fifteen-hour day into a ten-hour bag, and wonder why we're so tired by the end of it.

Meanwhile, the family is at home, going on with their lives, getting older by the moment, growing up, learning, expanding their horizons, figuring out what life is all about. Before we know it, the kids are grown, the memories are made, the world has moved on - and we're still sitting at our desks, trying to figure out how to handle the latest imaginary emergency with those intangible, unreal bean-counted numbers.

This is my forty-fifth year of life, having survived the first forty-four relatively unscathed. I think I'd like to spend less of it at work, and more at home enjoying my family. There are no guarantees on life or health. And the children are never going to get younger, only older, more independent.

Now if I could just figure out a way to spend less time at work for more pay...

Overcome by Events

There is a phrase we use here at work when a problem has been around for so long that it no longer becomes relevant, when so much of the software or hardware has changed with the passage of time that don't think the problem is actually a problem anymore. The problem is said to have been "Overcome by Events". OBE, for short.

This particular phrase applies not only to software and hardware anomalies, but also to documents and emails and discussion topics which linger too long in the "unresolved" category. Unfortunately, due to the overwhelming flood of information to which we are all subjected these days, it can truthfully be said that most of the hot topics of today will be OBE tomorrow.

So are they really worth worrying about?

I have learned over the past twenty-five years in (mostly) the aerospace industry that they are not. Most of them become irrelevant after a few days, a few weeks, a few months. Going through all my old emails, it amazes me how many of them were marked 'Critical' and then got lost in the shuffle of priorities, and months later, it is difficult to remember why it was that they were so important. Certainly they are not hot issues anymore.

(This also underlines the difficulty of determining what is really Critical when everything is deemed critical. "When everyone is Special, no one is Special.")

The problem is determining which issues are really important at the time they first come to light, so that they can be dealt with in a timely manner. And not just the ones which are important to the sender, but those which are important to everyone. Issues which have only a single champion, especially a champion who is unable to articulate clearly the reasons why this particular issue should get attention from others, are lost in the noise. Typically only the issues which are raised to the appropriate level get that attention.

But even with issues being raised to a high level of attention, sometimes there are just too many of them. The limits of human attention span are breached. Our ability to cope with the complexities of the issues is maxed out. Stress scrambles our brains.

It is then that our safety mechanism kicks in, and we decide to let Father Time do his best work, and we simply step away from the problem for a while to see if it is really as important as it seems.

A week later, after things have stewed in the pot, the problem does one of two things: it either comes to a furious boil, spilling over the top and scorching everyone in its path; or, in the more likely scenario, it falls off the radar and joins the million or so other 'Critical' problems that have been Overcome by Events.

Unending Tiredness

I was tired of work on Monday, so I left.

The weather is beautiful, the air is just hot enough to feel refreshing when it's being blown across the skin in a light breeze, the sunlight is bright enough to suggest shade but not demand it. The pool beckons.

I drove home and dragged (most of) the kids out of the house and took them to the pool and forced them to swim and splash and dive and play and have fun; then we high-tailed it home for dinner and piano practice.

Afterward, my eyes just didn't want to stay open, so I closed them, and napped.

It was a good nap.

Then it was time to wake up and go to bed.

The Weekend's Accomplishments

It's the first weekend in a long time which was not at least partially devoted to accomplishing work-related tasks, and there was a long list of home chores needing attention: hair maintenance, car maintenance, plumbing repairs, house cleaning, among other things. Most important, there was an important movie to watch.

But more on that in a moment.

First and foremost, it was time to catch up on the car maintenance. The van needed an oil change, and, since we are a DIY family, we did it ourselves. It took a bit longer than normal, owing to the fact that there were constant interruptions and distractions, like taking Adam down to the barber shop to get his hair cut. Then getting the boys started on the lawn maintenance (filling the gas tank, starting the mower, unravelling the cord for the weed-whacker, finding another cord for the backyard part of it). Then moving things around in the garage, unpacking the summer stuff from the upper storage area and figuring out what needs to get thrown out versus saved. Then vacuuming up the piles of sand brought in by the ants who were trying to make a nest underneath the concrete. Then tearing down all the extra boxes. Then helping Cheryl with her new laundry drying racks. Then trying to work on the sprinkler system. Then discovering that the vacuum backpressure valve (or whatever you call it) had blown out and needed replacing. Then discovering that one of the main water valves had cracked during a freeze and needed replacing.

Sneezing all the while. (Yes, the pollen is out, and in force!)

It was a long day, Saturday. Then Sunday came, and after church, after lunch, we headed out to the movie theater to (finally!) watch the new Pirates of the Caribbean flick.

We'd read that it was 'busy' - long on plot twists, short on character development. Somewhat true - there is a lot going on - but still quite a ride. The kids all enjoyed it immensely.

Afterward, it was home to rest up (and sneeze some more!) and eat the snacks we couldn't afford to buy at the theater, and then the kids went back to reading the books they'd borrowed from the library, and I went out to dig up the broken plumbing, and Cheryl worked on her studies.

Tomorrow, our summer home school starts. Something simple, I hear. Cheryl is going to have the kids write up a "what's next" scenario for the Pirates, to let them write their own ideas for movie #4.

And I'm going to try to catch up on work.

Last Day of School

Today is the Last Day of School.

It's a bit of a waste of time, really. What do they do for half a day? Sign yearbooks, say good-bye to those whom they won't see for the summer, chat with teachers cheerfully (for once) because there is no tension over missed assignments or bad grades or general unruliness, all the while looking forward to the final bell that signals The End.

Then they run off home to celebrate by doing the same things they would normally be doing on any other day off, namely eating and sleeping and playing.

It will be a few days before the novelty wears off, but wear off it will, and then they'll be getting bored.

Good thing we have a pool nearby...

5th-Grade Graduation

It is an odd thing, this 5th-grade graduation, a ceremony to celebrate the passage of a child from the elementary school into middle school. In my own personal history, fifth grade was just the year before sixth grade, which was still considered a part of the elementary school. Here, it is the end of elementary, and the beginning of that curious limbo-land of middle school, where the student is not quite grown up yet, but definitely too old for elementary behavior.

I'm not much on celebratory events (as most of you know). Too many times, an event which is significant only to a particular person or family gets blown all out of proportion, as though it were of great significance to the entire world. Certainly there is some merit in impressing upon a young person the magnitude of an achievement for which he/she has struggled mightily. Then again, there are some things which are achieved merely through the passage of time, requiring nothing more than mere survival (and avoidance of dangerous behavior patterns).

Which category does this falls under? I'm not sure. Perhaps my uncertainty has to do with the level of difficulty. How hard can it be to make it through fifth grade? Not as hard as getting through high school. Which is one of the reasons we make such a big deal out of a child's graduation from secondary school rather than elementary. (The other reason may have to do with the fact that, for many children, secondary school is as far as they will ever go.)

All that said, we're still proud of James. Not for the fact that he 'graduated' fifth grade, because that an expected event. And not for the fact that his teachers recognized his creative abilities (they tagged him with the 'future author' superlative, although he'd probably rather be known as a 'future artist'). We know he is talented. We also know that the teachers are required to give some kind of recognition to each child for the purpose of self-esteem, so that no one is "Left Behind" (although saying that someone has the 'prettiest eyes' is really pushing it!). Unfortunately, it still holds true that when everyone is special, no one is special.

Mainly we're proud of James because of the characteristics that we see every day: his gentleness, his compassion for others, his easy-going nature, his loyalty, his playfulness, his laugh, his talent for drawing, his willingness to eat whatever is in front of him, his silly voices, and his love for his family.

And he's just happy to have the summer off!

Game Called on accounta Rain

Didn't get a chance to work on the sprinklers after all. It rained most of the weekend; and what's the point of working on sprinkler's when it's already sprinkling?

We did manage to get into the pool ahead of the storm, though, and leave as things were starting to look a bit gray. Wasn't more than an hour or so before the rain started coming down in buckets, and the thunder roared, and the lightning flashed, and we sat in our nice, cozy little house and watched it all through the windows.

At some point - I forget when - we also managed to mosey over to the Library, where we took on a huge load of books. Plus a video or two. Kept the kids quiet for hours and hours after we got home.

Then on Sunday, we had the piano recital, where three of the kids (Adam, Deborah, Mary) played their pieces and then there was a reception and we ate cookies and drank punch (with sherbet ice cream in it) and went home to watch one of the movies. I can't remember which one. Hercules? or was it the Schoolhouse Rock video?

Man, my memory is getting bad.

Anyway, it was an exciting, action-packed weekend. Or not. At least there wasn't a lot of work going on. I got to catch up on some reading. And watch Schoolhouse Rock. And remember the 70s, fondly.

Hope your weekend was as much fun!

Success (?) at Last

It was a mad dash and a scramble, but we finally finished the effort to get the ATP software done.

There always seem to be last-minute concerns in a process like this, especially when it's all taking place on a Friday afternoon, and it only exacerbates the situation when half the team is in one building and the rest are in another. In our case, Engineering is over 'here', and Manufacturing is over 'there'. So after we declare our software good, they pack up all our hardware (and software) and truck it over to the other building, and run a bunch of tests in their Manufacturing Labs, and we talk on the phone for hours and hours, debating every little thing that comes up.

We had several hardware 'anomalies' which required explanation, but nothing that could be traced to the software. In some cases, we have to assume that the hardware tests have actually revealed hardware failures (the software can only be blamed for so much), and then the hardware engineers have to put their collective heads together and figure out if something is really wrong. But so far as we software weenies are concerned, we're done.

For now.

This whole push was simply to get the software in a state to support the ATP process, to allow Manufacturing to test the boards which are going out to the Customer's Training Lab, yet we know even now that there is still much that has not been tested. We are awaiting investigation of some areas which we simply could not complete due to bizarre, inexplicable integration failures. Once those problems are understood, we will have to revise our software again.

Well, someone will, anyway. I'm moving on to another assignment, another fire to put out. Our Certification activities are due for completion by next February, so for the next few months, that'll be my main focus. I'm passing this task off to another young engineer (who, incidentally, is just starting work on his Dissertation) and letting him have all the coding/testing fun, while it'll be documentation and planning processes and spreadsheets and budgetary concerns for me.

Oh, joy.

Meanwhile, this weekend looks like a good opportunity to put the lawn sprinkler system back together. The grass is looking a might poorly.

Disappointment

The left side of my throat has been hurting for over a week now. It started after my return from Seattle, beginning in the ear canal and sneaking downward til it got to the muscles in my throat, making it extremely painful to swallow.

Experimentation with various dosages of over-the-counter analgesics led me to self-prescribe two Excedrin migraine tablets every four hours. Anything less and it was impossible to deal with Life.

Consequently, my current daily prescription for Health and Livelihood is the abovementioned Excedrin plus one antihistamine tablet every four hours, switching to two antihistamine tablets at night right before bed. (That keeps me from waking up in the morning with severe sinus pain due to inhalation of cat fur during the night.)

My liver is probably shot by now.

The Good Doctor, who took a look at the interior of my ears and throat today, pronounced me fit as a fiddle, and proposed the thought that perhaps my malady was due to excessive drainage during the night, which irritates my throat and ear canal, causing pain. He gave me a sample container of Nasonex to try. The list of caveats for this medication is long and tedious.

It is always disappointing when the Doctor finds nothing seriously wrong in spite of the fact that the body is experiencing excruciating pain. It begs the question, Shouldn't there be some deep and dangerous meaning to all that pain? Can it really be caused by something so mundane as night drainage?

It would've been so much more satisfying had the Doctor said "Ah!", then leaned over and pulled a huge wood sliver out of my ear. It would've made more sense. We all know that slivers cause pain. And if it were long enough, it is theoretically possible that it could've stretched all the way down to the muscles of my throat to explain that pain as well.

(If this were an episode of House, they would've discovered some bizarre South American barbed tapeworm living in my Eustachian tube, feeding on ear wax and excreting hallucinogenic chemicals down my throat. Ewww!)

One side effect of all this medication is that I am exhausted all the time. And breathing in an asthmatic mode whenever attempting real exercise. Last night I attempted to run a quarter mile. My breath was gone well before the half-way point.

Such is the life of a software weenie.