What’s wrong with the concept of Integrated Tool Suites? This has been touted as the holy grail of CM for years – right? Almost all of the commercial vendors of CM tools offer integrated tool suites that do at least Version Control and Defect Tracking, and most do more. Most of the IDEs, including Eclipse and MS-Visual Studio just to mention two, offer integration with other tools. So again, what’s wrong with this concept?

• Version Control (VC)
• Defect, Issue and Enhancement Tracking (DIET)
• Requirements Traceability (RT)
• Build Management (BM)
• Release Management (RM)

 Now let’s take a look at categories of tools that normally need to integrate with one or more of these components:
 

• Requirements are versioned either as sets, individually, or both
• Issues and/or Enhancements may be created as a result of new, changed or removed requirements
• Each requirement, its associated metadata and the elements it is tied to are tracked is some kind of RT tool

 Project Management Tools

• Project plans and schedules are versioned
• Time spent working on defects, issues and new development is tracked

 Development Tools

• Designs are versioned either as documents or models
• Source (code, DDL, etc.) generated to implement designs are versioned and tracked to the version of the requirements they are implemented under
• Unit Test source and data are versioned and tracked to the version of source they are executed against
• All source versions are tied to the Defect, Issue and/or Enhancement they fully or partially implement

 Build Management Tools

• Accesses versioned source to produce derived output (deliverables plus intermediates)
• Build identifiers are tied to the Defect, Issue and/or Enhancement they fully or partially implement
• Build logs, metadata, etc. are versioned and/or archived

 Packaging/Deployment Tools

• Build generated deliverables are captured along with other material for packaging and/or deployment
• Package identifiers are tied to the Defect, Issue and/or Enhancement they fully or partially implement

 Manual and Automated Test Tools

• Manual Test Plans, Scripts, Data, Expected Results, Actual Test Results etc. (Test Artifacts) are versioned
• All source versions are tied to the Defect, Issue and/or Enhancement they fully or partially implement
• Each requirement-based test is tied back to the associated requirement(s)

And finally, let’s list the presentation methods (interface types) that most organizations want to have available:

• Classic “fat client”
• Web
• IDE/Workbench
• API
• CLI (Command Line Interface)
• Email

Of course, we want this to be able to be used remotely in a secure fashion as well as when the user is disconnected from the network entirely. Speed is of no concern, so long as no delay for any single actions or aggregate action exceeds half a second. And don’t forget that each tool suite needs to support processes, both formal and informal, ranging from Agile pair-programming to classic Waterfall with formal review and approval cycles.

We now have enough for a three dimensional matrix, but even without going to that level of effort, this shows the scope of the problem. As a practical real-world example, I am currently involved in a modified Agile java development project. The core CM tools in use are cvs in pserver mode and a home grown DIET system. The cvs server is running under Linux and is release 1.11.14. The following “stand alone” cvs clients are being used in addition to the integrations into the Development IDEs: LinCVS, gnu cvs (command line), TortoiseCVS and WinCvs. ViewVC is also being used to present the cvs repository via a web interface. Development uses both Eclipse (v3.1 and v3.2) and Borland’s JBuilder (Developer and Enterprise, 2005 and 2006), for a combination of 6 development IDEs and 5 standalone interfaces. The build tools are Apache Ant and Urbancode’s Anthill OS, both of which access cvs via the CLI commands.

• Successful builds will cause the appropriate records in the DIET system to be updated with a build identifier and list of files that changed since the last successful build. This will provide traceablity and queue QC as to what DIET records to use in testing.
• Granular security down to the file level in the cvs repository. This allows only a restricted number of users to modify certain critical files.

• Special commit blockers to prevent commits without check-in comments that include DIET record identifiers.
• Special tag blockers to prevent the movement of tags by normal users once applied.
• Automatic deployment of Production Builds to the appropriate Download site.

The problems? The current list includes:

• Each IDE and client handles cvs errors differently; and all blocking actions are returned as errors.
• cvs does not do atomic commits, so some of the files committed in a change set may make it to the repository while others do not.
• Different IDEs do merges differently. Many used different algorithms and some of them do not catch “real” collisions while others produce many “false positives.”
• Different IDEs do Java Refactoring differently. In this case, both do so by doing removes and adds. This means there is no good way to track a file’s entire life cycle.
• Due to the above, many of the standard metrics reports are almost meaningless.
• Every process enforcement trigger or external interface that is implemented costs processing time that easily becomes humanly perceivable when a whole project’s worth of code is involved.

So why don’t we change to a more integrated tool suite? The reasons are almost as numerous as there are people involved. A few of the more important reasons are direct and indirect costs, schedule impacts, inertia (which can also be read as “comfort level”) and stakeholder resistance. Again, as an example we did a quick and dirty cost analysis and found that converting from cvs to Subversion would have a NRE (non-reoccurring engineering) cost of about $100,000 (US). And this is with both tools being free!

So having railed at the problems, what is the solution? First, let’s talk short term. If you can find an integrated tool suite that meets your needs from Development, QA, Business, Management and Process aspects and it is within your budget, get it! You will have problems, but you would anyway and at least once the suite is configured and deployed they should be manageable. There are two types of integrated tool suite vendors. The first is the one-stop-shop where everything is by a single vendor. The second is where a basic framework or IDE is advertised to work with a list of other third-party components of specific release levels. Both have the advantage of an active support organization.

If you need to roll your own integration, if you have to fit existing tools into an existing process or an existing set of tools, or if you need to “bend” an integrated tool suite into doing things it was never intended to do, your problems will resemble mine.

Now for a long term solution that I have proposed before: A consistent, open standard API for CM functions. Regardless of what one thinks of Microsoft, they at least came up with the SCC Interface that a lot of the tool suites work with. Unfortunately it is not only proprietary, but incomplete. An open standard interface specification is needed that addresses the CM components listed at the beginning of this article. For CM to truly be considered mature, we need to join the plug-and-play mindset. This should not mean that we should not be able to take advantage of specialized features in one or more of the component tools, it just means that we have the opportunity to be able to use them together when desired.

Ben Weatherall is currently based in Fort Worth, Texas where he practices Practical CM on a daily basis using a combination of CVS and custom tools to support a modified Agile-SCRUM development methodology. He is a member of IEEE, ASEE (Association of Software Engineering Excellence – The SEI’s Dallas based SPIN Affiliate), NTLUG (North Texas Linux Users Group), and PLUG (Phoenix Linux Users Group).

Trackback(0)

TrackBack URI for this entry

Comments (3)

Subscribe to this comment's feed

Ben Weatherall said:

... Mario and Robert, I am glad both of you liked the article. While I am aware of the Eclipse ALF project, and am watching it fairly closely, it is trying to solve the problem of giving the tool vendors a consistent and open interface to develop against. I applaud the effort and hope it catches on! The problem is that it requires the tool vendors, both proprietary and Open Source, to actively support it. At this point, many (read that most) do not. Now if we can get the other IDE vendors to pick up the Eclipse ALF standards... Robert, I was not aware of the Polarion offerings, but I did look at them this morning. It looks like a fairly good solution if one is using Subversion and indeed would be worth a closer look for a great number of projects. Unfortunately, it is not able to work in a plug-and-play fashion with other components. I could not use ClearCase version control with Doors requirements management, BugZilla defect tracking, Anthill OS/Ant build management and Tivoli release management tools. (Note: These tools were picked off of the top of my head and in no way reflect specific recommendations!) I think in both cases we are working towards a better solution. -Ben Weatherall

report abuse vote down vote up

September 25, 2006
Votes: +0

Mario Rousseau said:

Considering Aclipse ALF This paper is very interesting. I agree with a standard on CM interface API. Today, CM tool is an important piece in Software Development organizations as those who adopt CMMI compliance and Sabarney-Oxley. For CM interafce API, what you think about Eclipse ALF project (www.eclipse.org/alf). The goal of ALF is to have a set of APIs (web services) to let Software Development tools to talk together. And there is some demos which demonstrate the integration with different CM tools. And the switch is very easy if the web services is available for each CM tools. To be considered. Mario Rousseau Senior Software Engineer

report abuse vote down vote up

September 20, 2006
Votes: +0

Robert said:

... Hi Ben, Thank you for your interesting overview and summary. Well, what you are describing was our basic design goal. We are trying to come closer to most of your requirements or expectations. We disliked also this advertizing thing over years integrating separated islands of automation. Now we have chosen a new way to bring the topics you listed together. Best, Robert Neher, Polarion Software www.polarion.com

report abuse vote down vote up

September 20, 2006
Votes: +0

Write comment