Where are my unit tests?
I have seen many development teams who have implemented excellent Continuous Integration (CI) servers including CruiseControl, TeamCity and Anthill (to name just a few). Most of these efforts were very successful by setting up the CI Server to build automatically  every time a developer committed (e.g. checked in) his/her code back into the version control repository. Sometimes, the successful build is then deployed to a test environment where developers and QA personnel have ready access to a fresh build of the code, with the latest features, and can start discovering any bugs (or integration problems) early on in the development cycle. While this is all good, there is one area where I have consistently seen very little progress and that is in creating comprehensive and useful unit tests that automatically run upon a successful build. The CI Servers are designed to track builds, recent changes, as well as the success or failure of the Unit Tests. Unfortunately, when it comes to writing unit tests, many developers have trouble getting started. This is exactly where Standards can be a huge help.  

The IEEE 1008 Unit Testing Standard
The IEEE 1008 Unit Testing Standard (which is available via the IEEE Standards website (http://standards.ieee.org/reading/ieee/std_public/description/se/1008-1987_desc.html) provides a comprehensive framework for designing and implementing Unit Tests which can then be easily plugged into the build executed by your favorite CI Server. In CM Basics, we focus on providing a few quick tips and tricks. But in the future, I will be writing many more detailed CM Journal articles on IEEE Standards including practical guidance on how to implement these Software Development Standards effectively.

Here's (part of) a short outline:

Are standards really Rocket Science?
Standards are certainly filled with plain old common sense. Well written standards also  have the combined wisdom of many technology leaders who have taken the time to think through all of the items that they would include when performing a particular function. Standards save you time and help you to be more productive. They also show your auditors and your senior management that your software processes are based well accepted industry best practices. This is essential in organizations that need to show compliance with Section 404 of the Sarbanes Oxley Act of 2002.

We prefer working software
There has been much discussion about Agile developers preferring working software to volumes of (unused) documentation. I have personally seen large organizations write volumes of documentation on software lifecycles that simply became "shelf-ware" and were truly useless documents. In my practice, I have tried to be Agile enough to use just enough process and documentation to get the job done effectively. Sometimes that means reading a Standard so that I don't skip a step that otherwise, I would have included (that obviously saves a lot of time and effort). More importantly, it is following the collective wisdom of a lot of very smart people who have already been around the track a few times.

Is Agile growing into a comprehensive framework?
I have been swimming a little against the tide by suggesting that Agile needs to mature into a comprehensive framework. For some, SCRUM has been meeting this need. The IEEE is working on an Agile Standard (IEEE 1648) that will focus primarily on the Customer/Provider relationship. As a member of the IEEE S2ESC Management board, I am advocating that we form working groups to prepare standards to address a wide selection of Agile practices. Contact me offline if you would like more information on this effort (especially if you would like to serve on one of the working groups). For now you can get started improving your Continuous Integration practices by developing comprehensive unit Test Cases based upon the lessons learned by other technology professionals.

Bob Aiello is the Editor-in-Chief for CM Crossroads and an Software Engineer specializing in Software Process Improvement including Software Configuration and Release Management. Mr. Aiello has over 25 years experience as a technical manager in several top NYC Financial Services firms where he had had company-wide responsibility for CM, often providing hands-on technical support for enterprise Source Code Management tools, SOX/Cobit compliance, build engineering, continuous integration and automated application deployment. Bob is a long standing member of the Steering Committee of the NYC Software Process Improvement Network (CitySPIN), where he serves as the chair of the CM SIG. Mr. Aiello holds a Masters in Industrial Psychology from NYU and a B.S. in Computer Science and Math from Hofstra University. You may contact Mr. Aiello at raiello@acm.org or link with him via LinkedIn http://www.linkedin.com/in/bobaiello

Trackback(0)

TrackBack URI for this entry

Comments (10)

Subscribe to this comment's feed

Bob Aiello said:

	... Hey Curtis, you didn't say why though?? Bob
	report abuse vote down vote up

March 19, 2009
Votes: +0

Curtis Yanko said:

	... Agility is the ability to respond to changing stimuli, not stand up meetings. Stand up meetings is one KPI in an agile process. Having said that, I'm thinking CI needs some KPI's, not standards.
	report abuse vote down vote up

March 16, 2009
Votes: +0

Bob Aiello said:

... Hey Chris - the CMMI and ISO 9001:2000 are excellent frameworks for \"assessing\" the maturity of your processes. (9001 is used for assessing a quality management system.) I have been on the NYCSPIN (www.nycspin.org) Steering Committee for many years. (SPINS were setup by the SEI to promote the CMM way back when the CMM was indeed all the rage.) Both of these are \"assessment\" tools and do not (and never did) provide much information on how to actually setup the functions or processes in the first place. (Susan Land wrote a number of excellent books talking about using the IEEE Standards to actually stand-up these functions.) I agree with you that many Agile practitioners will not look outside their own community and consider other views/resources and frankly that really is a shame. However, Development Managers and Technology professionals need to use all available resources including Standards (e.g. IEEE/ISO) and frameworks (e.g. Cobit 4.1, ITIL v3). There is a tremendous amount of wisdom out there (much of it from the Agile community) and we should all make use of good ideas and share best practices. Thanks for your comments.

report abuse vote down vote up

March 05, 2009
Votes: +1

Chris Gardner said:

... Bob, Alistair Cockburn has much to say about the level of agility commensurate with risk (http://en.wikipedia.org/wiki/Alistair_Cockburn). Instead of trying to create standards for all agile processes, either create a new process or adopt (or adapt) Alistair's Crystal family of processes. You are certainly free to try to create standards, but don't be surprised if the adoption thereof is sparse. Remember when CMM and ISO 9001 were all the rage?

report abuse vote down vote up

March 05, 2009
Votes: +0

Bob Aiello said:

... Hi Chris, that would not be my first example of low hanging fruit. Although, if you are writing code for a life support system (or a large scale trading system or a missile guidance system) then perhaps that would be a good example. There are better examples though. The real point is that we have to "Right-size" our processes (usually based upon risk). I am suggesting that creating Standards for Agile would greatly help with it's adoption in large scale organizations (which have inherent compliance requirements). I have been at large defense contractors teaching/coaching CM where I spoke with them about adopting Agile practices (e.g. CI) - they still need to show that their documentation can pass a compliance audit.

report abuse vote down vote up

March 05, 2009
Votes: +0

Steve Freeman said:

	... You appear to be using the traditional meaning of Unit Test (sub-system) from what most of the Agile community use (class/component), which will lead to some confusion. It also looks like you're implying that all the tests should be worked out before coding, which would inhibit Test-Driven Development and incremental development in general.
	report abuse vote down vote up

March 05, 2009
Votes: +0

Chris Gardner said:

... Bob, I posted a follow-up comment on my blog: I just read your comment here about documentation. Are you advocating an IEEE standard for agile documentation? If so, why? In agile, tests and code comprise the primary documentation. Tests can take many forms, including xUnit, TestNG, FitNesse, Concordion, Cucumber among many others. Companies can figure out for themselves how leviathan SARBOX is and how best to handle it. Chris Gardner http://cgardner.blogspot.com/2009/03/continuous-integration-ci-needs-no.html

report abuse vote down vote up

March 05, 2009
Votes: +0

Bob Aiello said:

... Hey Jason, those of us in Industrial Psychology have been advocating and promoting self-organizing teams for some time. As for people over process and Face-to-face communication over written documentation - well the answer there is that it depends. There are times when you MUST have well written documentation or you will fail your audit and your company won't be in business. A number of firms have gotten into a lot of trouble for failing to comply with section 404 of the Sarbanes-Oxley act of 2002. The Agile Manifesto also notes that, "That is, while there is value in the items on the right, we value the items on the left more." In some circumstances - I would view this as having to choose between my right and my left hand (hint - I want both!

report abuse vote down vote up

March 05, 2009
Votes: +0

Bob Aiello said:

... I am loving the comments and feedback that is flying in. Here are the ground rules for publishing your comments. Please post or send me your comment to be posted. I do not just publish links to your blog or favorite website. I have done a great job already of creating my own bad links on CM Crossroads and I don\'t want to post links that we cannot control. I am ok with printing your link in your bio or signature. But please post or send me a complete and coherant comment. Why not just collaborate with me on another article and present your opposing point of view! Bob Aiello Editor in Chief http://www.linkedin.com/in/BobAiello

report abuse vote down vote up

March 05, 2009
Votes: +1

Jason Gorman said:

	... "Read on if you are ready to see how IEEE Standards can help you be more Agile!" Hmmmm. I'm not sure you've really grasped the whole Agile thing yet. Self-organising teams. People over process. Face-to-face communication over written documentation. There. I've said it :-)
	report abuse vote down vote up

March 04, 2009
Votes: +6

Write comment