How to Evaluate CM and ALM Tools


In his CM: the Next Generation series, Joe Farah gives us a glimpse into the trends that CM experts will need to tackle and master based upon industry trends and future technology challenges.


Joe Farah writes that evaluating and selecting configuration managment (CM) and application lifecycle management (ALM) tools is an opportunity both to ensure that you have a good process in place and to learn the present state and the state-of-the art in CM and ALM technology.

Evaluating and selecting configuration management (CM) and application lifecycle management (ALM) tools is an opportunity both to ensure that you have a good process in place and to learn the present state and the state-of-the art in CM and ALM technology. As with most technology, as you move from one generation to the next, the capabilities grow more rapidly, while the timeframe for each generation shrinks. Evaluation is a critical step in securing good process, and should be done in a manner that will ensure success. You don't want to start any project with the wrong requirements, nor with a dated understanding of technology.

First Things First
What's the best way to go about tool evaluation. Before you start evaluating, or at least as part of your initial preparation, make sure you equip yourself in the following three ways:

1) Know your users: talk to them. What would make their life easier. Your users don't know exactly what they need. But they might have some complaints, suggestions or other hints that will let you understand them. User complaints are normally high priority - and they should be. But some are objectively more urgent than others. Still, your users are your customers, so find out what you need to make them happy, and then plan to exceed their expectations.

2) Understand today's CM and ALM technology. It's not the same as ten years ago. It's not even the same as five years ago or three years ago. Yes, many tools stay the same, substantially, over long periods. But some are charging forward. And there are some new kids on the block too. Use the vendors to help you catch up. They say they've got some ground-breaking technology - make them tell you why in an hour or two. Read the CM Journal (especially the column CM: THE NEXT GENERATION).

<--pagebreak-->3) Understand CM. There are many definitions of CM. But the scope of CM is so wide and varied, you need to understand where you stand. Are you interested in Software CM? Good. For traditional or agile processes? Look at both. Ask why you can't apply that agile process to your traditional environment. You may be the world's foremost expert on merge tools, but you might discover that one of the keys to successful CM is to minimize branching and merging, not to cater to the most complex scenarios. Maybe you're not interested in software at all? Maybe you want to understand CMII ( or ITIL. Maybe you're interested in asset management. These are related but different areas. You need to understand the global scope so that you don't, as I have done on a few occasions, go down a whole 15 minute conversation only to find out that you're talking about two different things.

What Next?
Now it’s time to really take a look at what's out there. A lot of CM managers will look at a handful of tools, and then choose two or three to investigate in more detail. While you may get lucky this way, you can easily miss the tools best suited for your environment. Evaluation is a time-intensive process. Don't make a mistake up front that may cost you more time down the road.

Cast a wide net and read a paragraph or two about every tool. Pick 15 to 25 tools. Spend a half day or a day browsing through the web site for each tool, say 15 minutes per tool. Spend enough time to identify things you like and perhaps things you don't like. By the end of the day, you should be ready to start with a list of about six to eight vendors that you want to contact for more information. This may be a formal step, as in an RFI, or an informal step, as in, "Can you arrange a demo of your tool over the web?" If you're doing a formal RFI, spend the time FIRST to have some 1 or 2 hours web demos presented to you. In one week's time, you should be able to take your initial set of tools and have a good idea of which tools are candidates. But even more importantly, you'll have a better idea about what features and criteria are most important. This in turn will allow your RFI to be more intelligent.

<--pagebreak-->During this week of demos you're going to learn a lot about tools: How wide-ranging they are; how user interfaces can vary; how the scope of one tool is far more or less than another. You probably won't have a good feel yet for things like performance, administration and customization, but the basic CM/ALM functions, traceability, and ease-of-use should make themselves apparent during a demo. At the end of each demo, ask the following questions:

1) How much will it cost, per user, all in, for 25 developers and 25 occasional users? For all the ALM tools if possible.
2) Can I install the tool to evaluate it myself? Do I need any special infrastructure or training? How much?
3) How many support staff do you recommend for administration and customization per 50 users, as in (1)?


About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.