Saturday, 19 October 2013

Praise the Fire Fighter, Damn the Safety Inspector

There's a sickness eating our industry.  It's a culture of Macho Programming.  In it's simplest form, it's the idea that if we just push harder, longer, stronger, then we will win.  Yet time and time again, experience shows that the way to win is by working smarter.


I was once working on a particular AAA title.  The game was running late.  We were in crunch, and had been for a while.  There was no end in sight.  I came across the following code:(1)

bool DetectCollision(Vector3 location, float radius){
  radius * 1.2f;

So that line of code makes the collisions a bit fatter.  It's what we call a 'Fudge Factor' - we don't know why a problem is occurring, so we fudge the numbers a little until it works.

But take a closer look.  The statement as written has no side-effects.  It doesn't actually modify the value of radius.  When the programmer wrote:

  radius * 1.2f;

they intended to write:

  radius *= 1.2f;

Let's review the facts:

  • It's the wrong fix to begin with (fudge factors are generally a bad idea)
  • It doesn't actually change the behaviour of the running program
  • The programmer didn't verify that their "fix" worked

And we can safely assume that the programmer didn't verify the original bug in the first place.

Compound Fail

It gets worse.  In this late stage of crunch, there were so many easily avoidable bugs coming in to the code, that the production team mandated every changelist required a second programmer to sign off.

That's right, not one, but two programmers, working together, managed to convince each other that this placebo changelist actually improved the game.  Together, they marked the bug as fixed and sent it back to QA, fully confident they had made the game better.

What other trivial mistakes did those programmers make that night?

What a colossal waste of time and resources, simply because those two programmers had inadequate sleep.  The project would have been much better off if those two programmers had just gone home at 18:00.  Or 17:00.  Or even 14:00.

[Edit: I just wanted to add, these are actually two really good programmers! I'd jump at the chance to have them on my team again.  The equation here is crunch + good programmers = too many careless mistakes.]

Praise The Fire Fighter ...

When you're deep in crunch, it's easy to see the heroic efforts, the mountains of caffeinated beverages, the change logs at 4am.  It's easy to point to the person who's working the hardest and say "We all need to be more like that guy."  Because when it's all falling apart, you need to do something.
(Hint: That something we need to do is to get more sleep.)

... Damn the Safety Inspector

And when you're at the beginning of the project, and the Safety Inspector is telling you that what you're building isn't up to code, that the schedules are unrealistic and will lead to crunch, slipped deadlines, hard-to-find bugs and adds unacceptable levels of risk to the project...  Well they're easy to dismiss "We're trying to build something here! Why are you trying to stop us?"
(Hint: The Safety Inspector is probably right.)

Macho Programming

As an industry, we should be working smarter than that. We should be rewarding measurable results and hard evidence, rather than effort and posturing.

To me, Macho Programming, is blundering onward in whichever way you possibly can, without regard to what's best for the project or the team, simply for the appearance of getting something done.

I think we can do better than that.  In the true Agile sense, as a team, we need to make the best decisions based on the best information we have, right now, and move forward in the best direction we can.

Who's with me?

(1) Some details have been changed to preserve anonymity.

No comments:

Post a Comment