Other than when the boss says to, when should tool evaluations be performed and what should they include? Other articles, both in this issue of CM Journal and in pass issues, describe how to gather the requirements, how to filter tools in or out of contention based on those requirements, how to weigh the vendors into the decision and whether it is better to go with COTS or FOSS tools. This article will focus on when it is appropriate to even start the evaluation process.

Are you in Pain?
One of the primary reasons to actually look at replacing tools is because they are too painful to continue using. The diagram below illustrates how we tend to tolerate increasing levels of pain until we get to some, albeit subjective, threshold value at which we do something to alleviate the pain. Shortly thereafter we start discovering things we don't like about the solution we implemented and the pain starts ramping up again.

Note that we normally do not get back to the level of comfort we had before we implement a solution.

Of course, if we switched tools every time things become painful, we would soon run out of tools to switch to. Before looking at new tools, see if the existing ones have upgrades available that would reduce the pain of using them, or if there are alternative ways of using them. As an example of this last, consider a Version Control tool that tracks internal element identifiers for every element. This makes merging easier when elements are renamed or relocated to somewhere else in the directory structure – unless users use the remove/add method of doing this instead. Now, the tool does not have a reference to link them and therefore cannot merge them. Users report that the tool is useless as merges take even longer than they did before they moved to a “smart” tool and ignore the fact that they are using it inappropriately. In this case, remedial training was the appropriate choice, not another new tool.

Tool End-of-Life
A really good reason to start the search for a new tool is the one currently in use has reached end-of-life. This can occur due to a number of reasons, including the vendor going out of business, the vendor being acquired by a competitor and being discontinued, or a newer product has been developed and the old tool is being phased out. In most cases, there is time to do a proper search for a replacement as the old tool can continue to be used until the replacement is found and implemented.

New Platform Identified
Everything is going along smoothly when Management decides that Platform X will be the wave of the future and Development is tasked with changing the systems, both existing and future, to support it. Unfortunately, the SCM tool chain does not support Platform X and at least one of the tools never will.

We pause for the panic attack. Take a deep breath. Now let it out slowly. Repeat 10 times.

Now that we can consider options other than new tools. Such things as, is cross-development an option? Is there a bridge that can be implemented easier or cheaper than switching tools? If there is no option other than one or more new tools, is it necessary to change them Enterprise wide? How long is the transition period? Can an intermediate tool be used while a better, long term solution is sought?

Implementation of New Development Methodologies
The biggest change in the recent past is the advent of Agile Methodologies at the Enterprise level. This, coupled with the Object Oriented languages like Java and the IDEs that support them, have led to the advent of increased code churn, not only in content, but in the namespace area as well. Many of the older Version Control tools do not support this effectively. But take the case of CVS where there are two reasonable upgrade paths – cvsnt and Subversion. While Subversion is normally considered preferable, cvsnt may offer the needed features without having to change the surrounding infrastructure.

Versions Control may not be the only SCM tool that requires a new look. The classic DIET (Defect, Issue and Enhancement Tracking) tools may not support the addition of Requirements, User Stories, Test Management, etc. though in many cases there are add-ons that will allow them to. Again, such a base as Bugzilla can be augmented by a whole list of add-ons that will allow it to support more “modern” methodologies without having to switch core tools and replace the corresponding infrastructure.

So What Needs Replacing?
We have already partially touched on this in the sections above, but consider that SCM tools form an entire tool-chain. Some vendors promote a single-source supply of the entire tool-chain from Requirements through Deployment, but most users are not comfortable with this since it locks them in and, in many cases, forces the use model of the tools. The figure below illustrates the typical SCM tool chain at a high level.

As mentioned in the figure, each of the high-level SCM tool sets (VC, DIET and Build Management) as well as Release Management can be composed of more than one tool. Rarely is it necessary to replace the functional tool set, and even more rare to replace the entire tool-chain. What one should do  is to make sure that only the parts that need to be replaced are changed. Of course, the requirements of any of the new tools may cause ripple effects within the rest of the tool-chain resulting in even more changes. This should be one of the evaluation criteria considered whenever a tool replacement becomes necessary.

Example
Let's assume that the SCM tool-chain to begin with consists of the following:

• Sun Java
• cvs 11.x series
• AnthillOS
• Bugzilla 3.0 series
• Custom interfaces
• Custom release scripts

The primary problem is that the Agile Development Team has grown from 5 to 25 developers with a corresponding increase in the number of testers. They have started doing multiple parallel development sprints plus a maintenance sprint to cover field reported defects. Each of the sprints operates on its own branch based off of the release branch it targets and each release branch is based off of the trunk. Initially, everything goes along pretty well, but it is noticed that code drift and refactoring is causing longer and longer merge cycles. At two days per merge, Management starts expressing concern and at 5 days, the concern is more along the lines of panic. Any acceleration in the schedule is canceled by the increasingly long merge times, not to mention that the merges are not picking up all of the changes (primarily due to refactoring) nor are all of the changes made correct (primarily caused by code drift coupled with an incomplete knowledge of the changes that were made).

After review and discussion, it was decided that just applying discipline to the problem, along with allocating the most knowledgeable personnel as Merge Meisters, was not a viable solution and the evaluation process for a new tool to replace cvs was started. At the time, Subversion was still in Beta with its ability to track namespace changes, so it was not an option. Cvsnt was in a similar position, but even further behind in its projected release. There were several viable COTS tools that met the criteria, but the critical nature of the problem required a fast selection and implementation. Review of vendor information reduced the number to 5, user community chatter further reduced it to 3. Of the three, only 2 stood any chance of fitting into the existing infrastructure without significant work. After immediate and long term cost analysis, discussions with vendor representatives and simple prototyping and demos, a final candidate was chosen.

Now that several years have passed would the same candidate be chosen today if the same situation existed? I don't know. The entire evaluation process would have to be repeated. I strongly suspect that the candidate chosen previously would still be a finalist.

Is a tool swap painless? No way! Often months later problems may be discovered with understanding what the tool could do or with how to use it effectively.

Summary
It is appropriate to start a tool evaluation when things become too painful to continue using the existing tool, or the tool is no longer effective in doing what it needs to. Make sure you know what the real problem is before looking at the SCM tool-chain as it is often not truly a tool problem at all. If the rest of the SCM tool-chain is working well enough, try to change out only the specific low-level tool that is “causing” the problem.

Finally, make haste as slowly as possible. There will always be times, as in the example above, where time is at a premium, but when there is time to do a good job, do so.

Ben Weatherall is currently based in Fort Worth, Texas where he practices Practical CM on a daily basis supporting a modified Agile-SCRUM development methodology. He uses a combination of AccuRev, CVS, Bugzilla and AnthillPro (as well as custom tools). He is a member of IEEE, ASEE (Association of Software Engineering Excellence – The SEI’s Dallas based SPIN Affiliate), FWLUG (Fort Worth Linux Uscers Group), NTLUG (North Texas Linux Users Group) and PLUG (Phoenix Linux Users Group).

Trackback(0)

TrackBack URI for this entry

Comments (3)

Subscribe to this comment's feed

Ashutosh Pandey said:

	... Thanx Ben, this was good... just what i need to convince my pointy haired boss with ...............:-)
	report abuse vote down vote up

July 06, 2010
Votes: +0

Ben Weatherall said:

	... Glad you enjoyed it.
	report abuse vote down vote up

June 30, 2010
Votes: +0

Praveen Kumar Reddy said:

	... Good one. Thank you.
	report abuse vote down vote up

June 29, 2010
Votes: +0

Write comment