Nobody Goes Home Until...

Have you ever had a manager use that phrase or something similar before? It's a downfall of development methodologies, or simply bad management.

Assigning a date to development has always been more than difficult. Estimates are made using best guesses, and in a perfect world they're based on similar work done before. That never means the estimate is guaranteed. Sometimes it takes less time, maybe because that task had in fact been done before. Sometimes it takes longer because of unforeseen events. Sometimes you get lucky and it's spot on.

I had a manager who was famous for using the phrase in the title of this post. I recall distinctly his use of that phrase four days in a row. Sadly, my team had discovered the problem and issued a fix at the zero hour on the first two days, leaving us feeling like we'd "trained" him to know he could get results by uttering those words. Never mind the arguments against carrying out such work. We would lose time on the planned work we had to do, the "fixes" weren't fully tested, and we had little idea of their long term effects. We knew we were making problematic software more problematic and falling behind. He knew he got results.

The reality is sometimes issues come up that weren't planned for. Projects have to be flexible enough to at least attempt to address those issues. If customer ABC cannot get their work done, you are failing them and need to be able to get them back on track. Sometimes, though, it's a problem of managing expectations. "We asked for the font to be pink two months ago! It's a simple change, why can't I have it now?"

I know of no development methodologies that solve this. Waterfall would have you put this issue in a next release. Agile would have you push this issue into a meeting where you decided when you could get to it, or which issues you would have to let slip in order to address it. Either way, something is going to have to wait, and sometimes that's not acceptable.

Steve Yegge described his view of Google's methodology on his blog a while back. Google seems to have more of a FIFO stack with an environment that nurtures good development. No real methodology to speak of, at least not one that's defined in books and expensive presentations. Really, what they've implemented seems to be more of a common sense approach.

We all know there is work to be done, and that expectations are that it does get done in a reasonable amount of time. Do we really need a rigid ruleset to follow in order to be sure it gets done?

When I'm asked what methodology we use, I think my answer is fairly typical. We borrow good ideas from methodologies and incorporate them into a hyrbid of sorts. I like the idea of scrum meetings, and we have a daily meeting where each developer on the team discusses what they did since last meeting and what they're working on now. That leads to another topic in the meeting I like... architecture. After our scrumish meeting, we discuss any problems that were brought to the table and possible solutions. This can't always be done in a single meeting, and if it takes longer, we'll either schedule a meeting to focus on that topic or table it for the next meeting to continue discussion.

We have milestones, but no hard dates. Hard dates promote hurried decisions and hurried coding. I want my developers to think about their solution and make sure it's right for our framework and architecture. The milestones we have merely give an order for incremental releases. That doesn't mean those items are all that is worked on until the milestone is hit, but they are the main focus. I like iterative releases to UAT. Users can test portions of the application while development moves on. What could be more efficient than that?

My goal on projects, after defining the project itself, is to allow developers to do their work in an environment that allows them to do it. I don't stand over their shoulders. I don't dictate what pieces of the project they work on in what order. I don't commit to dates that can't be met or are questionable.

I do report on progress. I do have a project plan that shows what items are being worked on by whom, and about how far along those items are. Dates have to be set because that's how project management software works, but they're slippery. I have confidence in my team and they're professional enough to continue work on their pieces of the puzzle, and that they'll come to me if that's problematic.

In a nutshell, I facilitate. I'm the middle guy between the folks who are getting the work done and the folks who are funding that work. Both sides want information, and both sides want to see progress. I don't believe it takes a particular methodology to accomplish that. It just takes professionals with some common sense.

Development vs. Manufacturing

A colleague pointed me to the this link which is lecture notes from an MIT class. It points out some common fallacies (derived from a study by Yuko Fujigaki) in running software development and cites the classic mistake as running software development as you would run manufacturing.

What is pointed out is that the time management techniques used in running manufacturing cannot be associated with software development without modifications. What are those techniques, then, and what modifications are required?

Taking from another article on a blog that I enjoy by Jeff Atwood, there are some parallels between software development and manufacturing, but they're often misapplied.

In manufacturing, you spend a whole bunch of time up-front doing design work and end up with a blueprint for what you want to build. You (loosely speaking) feed that blueprint into a bunch of machines who churn out your end product The traditional software-as-manufacturing metaphor maps "creating blueprints" to "writing a spec", "bunch of machines" to "developers", and "end product" to the code that those developers produce.

The revised metaphor instead maps "creating blueprints" to "writing code", "bunch of machines" to "compiler", and your end product is the binaries that you produce.

Using this metaphor better emphazises the challenges and efficiencies of software development as compared to manufacturing.

First, it casts the coding process as the creative design step that it is, rather than suggesting that developers are a bunch of automatons that take specs in at one end and churn out software at the other.

Second, it acknowledges the need for a detailed spec, but casts the code itself in that role. After all, what more detailed spec of how software works is there than the code itself? And unlike real manufacturing where screwing up the blueprints and starting to manufacture something incorrectly is extremely costly, actually "manufacturing" something from code is essentially free. Appreciating this frees you up to try different things with your blueprints to see which one produces the best end product.

So what does that mean? It simply means that the idea that the actual process of writing code is often misconstrued as being the easier part once specs are developed. The truth is, once a real spec (and what specs actually are is a topic in and of itself) is produced, the fun begins. With a spec in hand, a developer has a roadmap, but faces all of the challenges of working within the current codebase (or architecting if it's turnkey work), stumbling on functions that weren't written with scope and expansion in mind, data models, interface design, etc...

What this boils down to is the notion of time management in the software development process. The analogy I often use when discussing time management is asking a mechanic how long it will take to fix your car when the only description of the problem he/she has is "it's broken." You can narrow it down for them a bit with "it won't run" or even further with "it will start but won't move" but even with that detail you're not going to get a straight answer. It's not until the mechanic has had a chance to look at the vehicle that you're going to get anything resembling a time and cost to fix the problem.

For some reason, that's usually not acceptable in software development. Hard dates are demanded, and the developer has to live up to that date. And of course, when that date turns out to be short sighted, what happens is a stressed developer hacks in bad code at the last minute if they don't fight back on the deadline. That process leads to a common cycle in development where bad code is deployed, bugs are discovered, a client becomes unhappy or unhappier, more deadlines are set for the bug fixes, rinse/repeat.

What can be done to solve these problems? First, management of software development need to be aware that the development process is more art than science. There is no one way to solve any problem, and it's the thought process that gets to any of the solutions that takes the bulk of the time on the project. Without intimate knowledge of the codebase and the requirements, a hard date is impossible to define. It becomes less impossible when those two are known entities, but stumbling blocks, interruptions, scope creep, and any number of things can push that date out.

WWPD (What Would Paul (Graham) Do)

For those who don't know, Paul Graham has accomplished quite a lot in my industry, including cofounding Viaweb (now Yahoo Stores), and contributing to Lisp projects. More importantly to me, he offers tremendous insight into development through his website and blog.

I had yet another wonderful moment here at work last week. Late Thursday, as my team was trying to wrap up the unanalyzed, under-resourced project that affected all of our clients' ability to bill, another issue came up.

The way client issues are handled by this company are thus. A client calls, says they have an immense problem (which may or may not be the case), that it's a problem we've created (which may or may not be the case), and our senior management makes it the highest priority. Immediately all hands are on deck scrambling to figure out what the issue actually is, and all other work suffers.

This happened last Thursday. As it turns out, we fixed a problem with patient statements a few months ago. The previous development team did not tie patient statements to the account ledger, but instead used the source tables which tend not to know about adjustments, write-offs, etc... the usual information found in an account ledger. We completely rewrote them and they now work beautifully.

This particular client didn't use best practices for charge writeoffs and apparently didn't read the release notes nor bother to watch the training video we provided on the subjects. For a couple of months it seems they were sending patient statements to patients whose debt had already been written off. This apparently inundated the practice's call center with questions from these patients.

The MO here is usually that development, business analysis, training, and client services was wrong because the client said so.

After a few hours of analysis we determined what the problem was (our legacy versions' business analyst is unbelievably good at what she does) and reported it back up the chain and to the client. You can still hear the crickets chirping.

Meanwhile, the secret code was uttered from senior management to the product manageer. "Tell Jeff 'Nobody goes home until the EMC work is done.'" That was the project that was behind and pushed aside by this new important issue. And that is the code that senior management used four days in a row while they were at a conference and couldn't bear to tell clients we had a process for analyzing issues and would look into their particular "pink font" issue. That is the code that makes me want to quit my job and flip burgers for a living.

My response this time was "feel free to tell them I'll go home when it gets late, and if the work isn't done, I'll consider myself fired and never come back." In a fraction of a second I remembered a company policy regarding timely notice and lack of payout of accrued vacation pay. So I ammended the statement to "Wait.. you guys owe me an assload of vacation since I can never take one, so I'll come back with my letter of resignation."

Three people have stopped by my desk since to make sure I still worked here. I don't know if I really intended to quit on the spot, but the moral of the story is a developer/team lead/manager can only take so much of fireman mode. That mode causes the real work to suffer, and on the back end of that suffering are more fires to put out because the work was tied to a deadline and hurried or interrupted so often that the abstract vision was difficult to maintain.

The conclusion I came to over the weekend was that this incident only accelerates my desire to find work with some other company that mistreats its developers... only closer to home. And for better money. And maybe with a candy machine in the break room.