An Evaluation Framework for Continuous Integration Tools

Tools that enable continuous integration are vital to any agile project. Learn how putting together a well-planned evaluation process for the selection of those tools enables your entire team to work more cohesively, while eliminating the waste and damage that ineffective tools can cause.

The Importance of Continuous Integration
As agile and other lean methodologies move to the forefront of software development, one thing is abundantly clear: continuous integration is the key process for making agile work. Without it, teams have no feedback as to the stability, quality or suitability-to-task of their software. Given the short cycle times in agile methods (2 week sprints, for example), this lack of feedback is the equivalent of driving on a dark, unfamiliar road with no headlights. By introducing continuous integration, teams can implement these agile micro-cycles more effectively, and ensure that the project is progressing appropriately.

Since continuous integration is so important, teams need to have a solid framework in place for evaluating tools. Choosing the wrong tool results in wasted time and effort, forcing teams to go back to square one in order to meet the goals of fast, effective build and test cycles. Conversely, when the right tool is introduced, developers become more productive, managers gain insight into the process, and business stakeholders (often resistant to agile) buy into the process.

The Evaluation Framework

With all of this in mind, I recently embarked on a continuous integration tool evaluation. You can read the first installment of the detailed results on my blog at Rather than rehashing the specific results of the evaluation, my goal in this article is to present the evaluation framework that I used, discuss why it was useful – and to save readers from investing the time and effort of developing their own framework.

At a high level, the framework facilitates evaluation of continuous integration tools along several vectors:

    • Installation
    • Configuration and maintenance
    • Running a simple job
    • Availability of results and metrics
    • Interacting with development tools
    • Automating complex build processes

The remainder of this article expands on each of these points and provides tips on how to evaluate tools in these areas. For each, I provide a table listing several specific features that you should look for. The table also has a ranking column, and I suggest that you assign a rank from 1 to 5, with 1 being a poor score and 5 a high score. In this way, some simple math at the end of your evaluation will give you an instant snapshot of how the various tools in your evaluation stack up.


Often overlooked, the ability to easily and properly install a continuous integration tool is important. Complex and failed installations waste time, and rightly sour the evaluation. If the tool installs quickly and intelligently on the other hand, you as an evaluator have cleared the first hurdle and already have a sense of the quality-consciousness of the people who developed the tool.

Continuous integration tools are typically architected as client/server products, with a web server back end and typically a web interface on the front end. Consequently, you should look at several things when installing the tool:

Installation Criteria

Ranking (1-5)

Ability to change default ports


Clear description of installation options


Ability to perform basic configuration as part of the installation


Configuration and Maintenance

Once the tool is installed, you may need to perform some configuration. My bias with respect to configuration is simple: less is better. By this I mean that the tool should be immediately useful once installed. Any tool that requires you to spend too much time understanding its concepts and tweaking settings before you can use it should be treated with suspicion: if it’s hard to configure, there’s a good chance that it will be hard to use.

Any tool in your shop ultimately needs to be maintained,

About the author

CMCrossroads is a TechWell community.

Through conferences, training, consulting, and online resources, TechWell helps you develop and deliver great software every day.