Politics
For years I traveled the continent as a consultant. As such, I assumed I was outside of the normal corporate politics. How could I be expected to know what hidden agendas were lurking in the minds of Management? Especially when I was brought in to do specific things or determine specific problems. Instead, I was often brought in to justify earlier decisions by having an "independent, outside consultant" come to the conclusion that the tools chosen were the right ones, that the processes developed were aligned with "Best Practices" or that the blame for the problems really was the "other" department. Instead of being asked if the tool choice was viable, the implementation sound or the processes and tools in alignment, I was often asked to "ignore what is already here and make a recommendation."

What I learned painfully over the years is that if you don't make some effort to both "read between the lines" and quietly make contacts within the organization in order to glean information, you will often find yourself working against expectations. This is easy to say, but difficult to do. For someone who is as pessimistic and paranoid as I am, I still find myself optimistically believing what I am told.

As an example from my consulting days, I was brought in to evaluate new Version Control and Defect Tracking solutions. I was specifically told to not include ClearCase and ClearQuest as they had already been eliminated. After doing surveys, literature searches, one-day evaluations, etc. I came up with a short list of tools that would do the job for an affordable price. I included the full list of products included in the evaluation as an appendix in the formal report. After it had been submitted, I found out that the Manager that had brought me in did not want to go with the corporate standard tool suite - ClearCase and ClearQuest. His managers wanted to know why the tools had not even made the first cut for evaluation and used that as justification for ignoring the report. He ended up in a defensive position and I took the heat.

I have found that this holds true even in permanent positions. If someone way up the hierarchy, even if they are not in your chain of command, holds firm beliefs, then you may well be tilting at windmills to get changes made - regardless of how logical and cost effective they are. Conversely, if they are ready to make a change, it may be your only opportunity to have a say in the direction your department will be going.

Culture
Corporate Culture is different from politics in that it is both more amorphous and less personal. There are many CM patterns that can be implemented, but the successful ones are those that people are willing to follow. As an example, I was working at a company who was attempting to transition from SIE-CMM Level 3 to Level 4. One of the classic patterns where CM and QA work closely together is for developers to check in code and potentially flag it somehow at three different phases:

1. When coding is done, but before trying to get it to cleanly compile,
2. When a clean compile is achieved, and
3. When developer testing is complete.

The difference between the first and second allows statistical analysis to be performed to determine the coding patterns that are most commonly made that result in compile errors. QA can then determine simple coding changes that will reduce the time necessary to get to a clean compile. The difference between the second and the third allows for similar statistical analysis to the types of logical errors programmers make during the implementation of a design.

This works fine - for the appropriate cultures! If the culture lends itself to Continuous Integration builds that always take the latest checked in code, then the delays between these stages becomes intolerable; especially for large teams. On the other hand, there are cultures that resist any changes made by another department.

I guess that could be summarized by saying, "choose your battles based on what you can win."

Over-Develop/Customize
On a more technical note, I find that I am a pessimistic programmer. By that, I mean that I develop code that is internally documented and inherently testable. Over the years I found that I preferred generic and data-driven coding to one-shot fixes. It allowed me to practice code reuse. As a part of this I also found that I needed to code so someone else could pick it up and maintain it after I was gone. Everyone agrees that this is a good thing - until it takes too long. As CMers, our job is not perceived to have a development component. Of course, triggers, workflow models and things like this often require custome coding.

What I have found out rather painfully, is that most organizations would just as soon get something up and running quickly, even if they don't understand it, than to take extra time. Errors, if they are never "posted," just don't rise to a level that anyone needs to be aware of. I have also found that people that pick up an implementation seem to prefer simple procedural coding to the more generic data-driven model. And it is easier to replace something small than figure out something large.

As an example of this, at one company I was implementing an interface between ClearCase and ClearQuest (and yes, I know they make one themselves - see Politics above) and determined that there were too many developers on too many projects to allow the interface to "block" waiting for an update to occur. At the same time, check-ins against inappropriate defects needed to be disallowed. Calling on my real-time, distributed processing background, I created daemons on each of the servers that communicated via TCP/IP. Each daemon queued its work and new requests could cause any of the queues to be reprioritized based the need for a fast response. They also allowed some level of data caching to be performed so API calls were not needed all of the time. This allowed very scalable growth, background ad-hoc queries and reporting and a decoupling of the release levels of the two products. What happened you ask? As soon as I was gone it was ripped out. There was no one there with the background to understand asynchronous communications and distributed processing, much less understand the requirements of daemons. They went back to a linear, start-to-finish processing for each of the triggers and learned to live with the performance penalties. Of course, it was all the fault of:That Consultant and IBM's products.

Tricky Code
This is a variation on the previous problem. After years of programming in just a couple of languages, especially scripting languages such as perl and sh/bash, you just get to know certain tricks that make your job easier. Just like making things too complex, saving time by assuming that others will have the same level of understanding of a language will have a maintenance cost. At the best, if you are a permanent employee, you will end up stuck maintaining the code long after you have moved on to other things. At the worst, the code will be modified by someone who thinks they know what it should do - and who breaks things in ways you never thought possible.

An example of this is processing XML files. The easiest way to do this in perl is to use nested hashes. Anyone who has ever had to reverse engineer nested hashes will appreciate that, while it is almost the only practical way to store the data, it will confuse just about everyone who tries to read the code. Iterating through nested keys is not for the faint of heart.

Version Controlling CM Artifacts
One of the first things I always do is set up a Version Control repository for CM artifacts, and I strongly recommend everyone in the CM arena do the same. These artifacts include any triggers, wrappers, bridges, daemons and custom documentation. The biggest mistake I made, however, was to use the same VC system as I was "developing" for. Why, you ask? Simple. If I make a mistake and deploy it, it may be impossible to undo it.

Most of the CM Artifacts do not require branching (you do make your implementations backward compatible, don't you?) and are not very large. Something like cvs or subversion meets the needs admirably. Just make sure you keep the implementation simple since it is likely that others who do not understand the "other" system will be able to figure it out otherwise.

Customization
Most tools, whether they are Commercial (COTS) or Open Source (FOSS), allow customization. In some cases this is no more that specifying local configuration settings, but in many it is possible to add custom fields, change workflows, enhance triggers, utilize APIs, etc. This is touted as a "Good Thing" - and it is, so long as you don't paint yourself into a corner. The more time and effort spent in customizing a tool or integrating a suite of tools, the more the result matches the way development occurs within the organization. Unfortunately, this means that you run into dependency problems when one or more of the tools are released with changes to either their behavior or their infrastructure/interface. The more work spent customizing things, the more work will be necessary to do upgrades. And if you need to replace one of the tools with a different one, the entire infrastructure will have to be either highly modified or replaced entirely.

Not too long ago, we were using cvs as our Version Control tool of choice in a rather large Agile development project. It became evident that we were having major problems controlling the merges from one branch to another, especially when someone refactored the code. I had several triggers in place along with a daemon or two that were responsible for doing pre-commit validation and periodic summary metrics. When we switched to AccuRev, the entire cvs infrastructure had to be reevaluated and the appropriate parts reimplemented for AccuRev; and not all of it was possible.

Once it was operational, AccuRev proved to be a good choice and solved a significant number of problems, some of which we did not even know we had! The bad news was that once again I had created triggers based on a "standard" stream structure and naming convention. One day, the Mangers of both Development and QA came in and said that they needed to change the process - and that the changes would require changes to both the stream structure and the naming convention. And I needed to have it operational within 12 hours. The fact that I succeeded is a testament to both AccuRev and my not having over-customized the implementation this time.

Expectations
Expectations are something we all have. Whenever two people get together, they each have expectations of how each other will behave and that our communications will be understood. Similarly, when we talk to "Users" of CM systems we have expectations that the system will be used in the intended fashion. "They," on the other hand, have expectations that may not match ours - such as the tools being integrated into their tools with little to no training required and no performance impact to them. Managing expectation is one of the hardest things to do since it requires active listening and open ended questions. These are the same skills required for Requirements Elicitation; and we all know how prevalent those skills are!

When the primary stakeholders have different expectations, and when those expectations are not understood by all parties, a major problem arises. This has been responsible for many project failures and the association of the proponents with the failures. Very many of these and one is looking for new avenues to explore.

Years ago, I subcontracted to a tool vendor to install a multi-platform Version Control tool with a GUI interface for one of their customers and to provide training on its use. It wasn't until the second day of training that I discovered what they thought they were getting was an IDE solution on Windows that would replace X Windows and NFS access to their UNIX systems so they could access their tools and source code transparently. This was not something the tools GUI could do! The sales people had not paid enough attention to what the customer was really asking, and the customer was either not asking the right questions or understanding what the sales people were saying. Regardless, the plug was pulled and I got to go home early.

Summary
There are many categories of problems in the world. As CMers, we get the chance to experience many of them - some we cause ourselves and some we get to fix after the fact. I have listed a few problems and mistakes I have been personally involved with over the years, but the list is by no means exhaustive. These may not be the biggest mistakes I have made, but they come close. And I am still making mistakes and causing myself problems. Someone once said something to the effect that one learns more from mistakes than from successes. Over the years, I have learned a lot and I try to pass some of that on so others can avoid at least some of my mistakes. While humans rarely learn from mistakes, misery does love company. Welcome to my table...

Trackback(0)

TrackBack URI for this entry

Comments (3)

Subscribe to this comment's feed

Anders Karlsson said:

	... Ben, Great article. These stories are both scary and amusing. Thanks for sharing /Anders :^)
	report abuse vote down vote up

March 12, 2009
Votes: +0

Ben Weatherall said:

	... Beto: Always glad to share. It is the only way we all learn. Thanks again, -Ben
	report abuse vote down vote up

March 09, 2009
Votes: +0

Beto Galassi said:

	... Very good text, thanks for share your experience!
	report abuse vote down vote up

March 04, 2009
Votes: +1

Write comment