Fran:   Tool evaluations? We just did that last year!

Schiz:  Yeah, but they ignored our recommendations.

Noid:   For the most part, anyway…

Phren:  What do they want this time? Justification for what they’re going to purchase anyway?

Noid:   Actually, I think they have already bought new tools.

Schiz:  Why are we making this rip? It’s pretty expensive, not to mention time consuming, to do a whole round of evaluations if the end result has already been decided.

Ben:    So the first thing to determine is who and what is behind the request for a new tool. It might be either an actual or perceived problem, or it could be political in nature. Going into this blind could not only be painful, but potentially job threatening.

All:      Right.

Fran:   Let’s assume the need is real and TPTBs [1] have not already made up their minds. What should we look at first?

Schiz:  Requirements. We need to know what the tool should do.

Noid:   Process. We need to make sure the tool, whatever it is, can adapt to our process.

Para:   Current problems. If the pain the users are experiencing is not reduced, then we have achieved nothing.

Ben:     What if the current process is wrong? How do we know?

All:      That’s another subject.

Ben:    You don’t have to shout! But you know my signature line is, “Remember, Configuration Management is first an attitude, second a process, and only then a set of tools...” Process before tools. I have been preaching this for years. Are you ALL are telling me that it isn’t important?

Para:   Not really. It is just that in order to solve some problems, the tools may need to cause process modifications. You know you are a proponent of Continuous Process Improvement; in this case it may be that a tool change is just the trigger that causes process change instead of the other way around.

Ben:    Okay, but y’all have listed three “firsts” – Requirements, Process and Problems. I think all of these are related. The diagram below reflects this as three axes where you could plot your “pain” in order to determine where to focus.

Phren:  I think you are over thinking this… again…

Para:   He’s right. If you are stuck in the situation of evaluating new tools, assuming there is a known reason for doing this, then you already know which is the most important.

Ben:    So what you are saying is that we should talk about the methodology of evaluating tools and not the more esoteric process of assigning weighting factors to particular criteria.

Para:   Right. As long as we all agree that the reason for the trip is not political, then what we need to do is document a framework for generic evaluations.

Ben:    Okay. Let me see what I can come up with…

----------- ooo O ooo -----------

First Things First
Once you decide you need to pick a new tool, there are certain core areas to consider. With the exception of the first, these are, in no particular order:

• Features Available
• Support Infrastructure
• Environment
• Technology
• Interfaces
• Usage Models
• Overhead Costs
• Startup/Conversion Costs

Features Available
If you have never investigated tools of the kind you will be evaluating, or if it has been a while since you looks at what is available and what the current feature lists are, then you should probably take a quick peek at the current offerings. Pay particular attention to those features that might address problems you are having or allow information to be more easily presented to Management types.

Support Infrastructure
No tool exists in a vacuum. There needs to be a place for it to reside, execute and keep its data. This means that there are requirements for the physical platform, operating system (and release/patch level), available tool chains, communications paths (as well as mechanisms and security), etc. Consider the ramifications of what tool changes will mean. Especially consider if the tool(s) being replaced will need to continue to be available for legacy purposes or for other projects. Will new infrastructure be needed, or can the existing infrastructure be leveraged?

Environment
The environment a tool runs in consists of such things as paths, environment variables, permissions, account, etc. In many organizations, this is controlled by an IT department while in others CM has control. Know in advance what it will take to put server-side and companion processes [2] into place.

Technology
Consider the architecture of the tools under consideration. There are several key areas of that impact how a tool behaves and what supporting infrastructure is needed. These are:

• Storage – How the tool will store the primary data as well as its metadata. Common mechanisms are File System, Database and Hybrid. File based storage is easiest to backup and selectively restore, but is generally the slowest of the approaches. Database storage is fast and backup/restoration mechanisms are available, but are generally not selective in nature (they tend to be all or nothing restores). Additionally, there is generally an effective maximum size of objects that can be stored along with overall performance penalties when many large blobs are repositoried. Hybrid storage uses both Files and an adjunct Database (where the metadata is normally kept). This last model has all of the speed advantages of a database, but with the storage capacity of a file system. Unfortunately, backup and restoration problems occur when the two disjoint storage systems get out of synch.
• Client/Server – If a Client/Server model is used, then the two pieces have to be kept in synch somehow. Also consider whether the client side is a “thick” or “thin” client. Thick clients include more functionality, which is nice, but the disadvantages are that other interfaces (think IDEs and browsers) may not implement the same functionality – or at least not implement it in the same way. Thick clients also tend to require being in synch with the server release level. Thin clients minimize the added functionality and are basically only presentation layers. They do not suffer the same disadvantages as thick clients, but they also require much higher connectivity bandwidth to the server since they are not making the “decisions.” Neither is a bad approach, just be aware of what it will require down the line to provide the support, functional and performance requirements.
• Peer-to-Peer – This is a model that is decreasing in popularity, but it has several advantages the greatest being that it is inherently distributed in nature. The biggest disadvantage is that there is no central point of control for backups and restorations. Each peer is required to (1) perform their own backups, and (2) keep both tool release level and repository data in synch with the other peers.
• Client Only – This model uses central storage, but each client “talks” to it independent of all of the other clients. Like the peer-to-peer model, the clients need to maintain release level synchronization with each other at least when the storage schema changes.
• Cluster – This is a form of distributed resources for a centralized solution. It allows for such things as load leveling of requests, storage “snapshots” for backup purposes and near real-time clone storage for use by query, reporting and build tools. It also allows for scalable solutions where more nodes can be added to an existing cluster, often without downtime, when needs exceed current capacity and without having to migrate to a new infrastructure.
• Distributed (LAN and/or WAN) – Consider well whether you need a centralized solution or a distributed one. Centralized solutions typically allow for easier administration, greater control of security, better backup and restoration solutions, etc. Distributed solutions allow for faster local response at each of the distributed locations, but at the cost of administration, security, backup/restoration, etc. Many times this cost is well worth it, but when evaluating new tools make sure it is capable of doing what you need.

Interfaces
As was mentioned briefly above, consider whether there are interfaces to other tools, IDEs, third-party databases, etc. that must be supported (either now or in the foreseeable future). If there are, make sure the tools to be considered can support them and support them in a fashion that the users will accept. It is not uncommon that non-native tool interfaces are available but do not do what is needed or does it in a totally different way than the native interface. This can cause confusion at best and errors in usage at worst.

Usage Models
Consider how the tool will be used. Will it need to support such things as CI builds, Just-In-Time (JIT) merges, refactoring, etc. Make a list of user scenarios (user stories, use cases, call them what you will) and compare each tool being evaluated against this list to make sure it will work. If not, check to see if it is possible to change how people work. This might become part of a process improvement effort, but it will cost extra in terms of training and support.

Startup/Conversion Costs
Once a tool is selected, there will be costs, in terms of personnel, time and possibly money, to transition to the new tool. Startup costs include getting the infrastructure in place, having the administrators learn the internals of the tool, installing the tool, configuring it for proper use within the Organization and performing user training. Conversion costs include transferring the data from the old tool (even if it is just a file system) into the new tool, verifying it is both complete and correct and enabling the appropriate security.

Overhead Costs
Once a tool is in place there are additional costs. These can generally be broken own into categories such as high-level administration, disk space, backup systems/components, systems updates, tool updates, process management and day-to-day hand holding. These costs vary by tool, so it is good to keep this in mind during the evaluation process.

Doing the Evaluation
Now that you have a list of things to keep in mind and you have done a cursory looks at the features of the tools available, you are ready to start the evaluation process. The following steps should be taken in roughly the order presented, though there are always times when you can justify changing the order.

Gather Your Requirements
If you don’t know what you want a tool to do then you cannot make intelligent decisions about the relative merit of the tools under consideration. Start with a problem statement describing what you are trying to achieve. From there, elicit requirements based on the problems being experienced and the “pain” the various stakeholders are experiencing. Be prepared to adjust the requirements as the evaluation process proceeds. This is like anything else, scope creep must be managed, but so too should reasonable changes and/or additions to the requirements based on what is discovered during the investigation.

What to Do with the Requirements
Once you have your initial list of requirements, make a search for possible tools. Do a first pass through the literature on each tool and filter out the ones that do not meet the requirements or will not improve the existing situation. Keep a list of each of these tools, their release levels and a simple reason why they were excluded. There will always be someone who asks either, “Did you look at XYZ?” or “Were these the only tools you looked at?” Being able to answer, “I looked at a total of 32 tools, including XYZ, and eliminated 21 of them, also including XYZ, because they did not meet the requirements for the new tool,” will help you keep your sanity and cut off additional discussion before it gets out of hand.

There are two categories of tools to consider at this point and the approach to each varies: FOSS and Proprietary. FOSS tools generally do not have a vendor channel, so you will need to download them and try them out. Check for a support community and use it so you can see how responsive and helpful it is. Don’t spend an abnormal amount of time at this point as you are still in an elimination stage.

Proprietary tools have a vendor channel, so take advantage of it. As Joe Townsend has recommended repeatedly in the General CM forum, “give your requirements to the vendors you pick and make them demo the products. Never under any circumstances let any vendor say, ‘Oh yeah, our tool can do that’ without demonstrating that it can be done.” Make sure the demo is real and preferably using your actual data. While a remote demo is acceptable, don’t settle for a PowerPoint presentation. Make sure any interfaces you need supported are also demonstrated. Remember, demonstrations are to allow you to filter out unsuccessful candidates, not to select the tool to recommend.

As you did during the literature search, record the name, release level and a brief reason for rejecting each of the failed candidates. Since these candidates made it further in the evaluation process, the rejection description should be a little more detailed and reference which requirements were not met.

Prepare to Rank Your Candidates
By this time you should be down to a reasonably small number of candidates. You will want to rank them in a spreadsheet. Each row should list a specific requirement and each column should list a specific tool. Assign a value range for each row – everything from 1=Pass, 0=Failed to 0=Unsatisfactory/Unsupported to 9=Fully Compliant and Easy to Use. It may also be appropriate, before any tool columns are filled in, to assign weighting factors to specific requirements. These weighting factors should not be visible while filling in the data. I recommend that they be on a second worksheet that has the same row definitions as the data entry sheet and be used to generate a third sheet which applied the weighting factors to the entered data and reports the results. This third sheet is what will be presented to the Stakeholders along with the recommendations.

When you are creating your spreadsheet, make sure you create rows for subjective “feels” as well. You will want to have things like “Exciting to Use” and “Makes Me Want It” represented somewhere. If tools are equal on technical merits, the desire to use one over the other should win.

Pilot/Test Candidates
You may have already done some of this for FOSS tools. If so, enter the appropriate information in the spreadsheet. Regardless, this is the time to set things up “properly.” This may mean writing triggers and reports, testing integration with IDEs and other tools, setting up security and enabling any process/workflow tracking/management features. What you were looking at earlier was, “Does this look like I could use it?” What you are looking for this time is, “Does it do everything I need and do I really want to use it?”

Break the evaluation down into two parts: (1) CM use and (2) Stakeholder use. The first part is fairly obvious. Install, configure, import, lock down, implement triggers/wrappers and perform normal administration activities on a real mini-project – one where real changes can be made and real results observed. These should include deployment to users, updates (server, client and integrations as appropriate), backup, restoration (full and selective if possible) and use with administration rights enabled.

The second part involves getting Stakeholder participation. You will need to do some minor training so they can install clients and/or integrations, do any configuration necessary and exercise the tool in the planned fashion. Coordination of when the Stakeholders do this, for how long and to what level of detail is one of the things that should be done prior to the start of the evaluation. If the Stakeholders do not know what will be expected of them, you tend to get the, “You want me to do what?” response. This information also helps determine which evaluation philosophy you will use: (Evaluate Them All, or Evaluate Until One Passes).

The data you are trying to collect is two-fold:

1. Objective information on how long it takes to do certain thing, whether the use cases can actually be executed, etc., and
2. Subjective information on how hard it is to do certain tasks, how “easy” it is to use, how well it prevents mistakes, etc.
   

Regardless of which evaluation philosophy you are using, enter all of the results into your spreadsheet. If you are using the Evaluate Until One Passes philosophy, you will then need to meet with the Stakeholders and include Purchasing Authorities and Management (they generally delegate to more technical personnel for the actual evaluation). Present the results of the evaluation using the spreadsheet’s weighted results sheet. The intent is to get a consensus on whether or not the search should continue. It the decision is to continue the search, document the decision details in the spreadsheet and repeat the steps above. If you have run out of tools to check, then either find more tools or get a consensus that one of the ones already evaluated is the best of what is available at the time. Remember, staying with what you have is always an option. If a tool is found that everyone is happy with and that is affordable, stop your search and use the pilot/test project as a training ground.

If you are using the Evaluate Them All philosophy, repeat the evaluation steps above until you have exhausted the list, then convene the Stakeholders, Purchasing Authorities and Management to review the findings and choose a tool. This method allows everyone to feel that they have selected the best possible tool. And sometimes that feeling is the best chance that the tool rollout will be successful. The downside is that a lot of time will be spent during the evaluation period. It may even last so long that tools that were evaluated have evolved beyond what was tested.

In Summary
There are several threads in the CM Crossroads’ General CM forum that discuss performing tool evaluations. There are even one or two actual examples of what people went through doing so.

The consensus there is that the evaluation process is always going to take longer than you plan, will always grow beyond the time allowed to do it, will invariably leave out one or more Stakeholders and can rarely be successfully performed by only one person. Use vendors and support groups whenever possible to speed the process up, but do not believe everything they say without either seeing it in person or trying it yourself. And always remember that after the evaluation is over, you are the expert in all things related to the tool chosen.

I wish to give special thanks to Joe Townsend, Geoff Schradein, Joe Farah, and Stephen Baynes for their participation, clarification and expansion in the tool evaluation forum threads. You helped me clarify several areas in my own mind.

Trackback(0)

TrackBack URI for this entry

Comments (0)

Subscribe to this comment's feed

Write comment