- or -
The Pipe Dream!
A while ago, I presented a partial transcript of a
conversation a friend of mine was having - the Schizophrenic Paranoid. When I
first started to write this article, I sent out a request for a definition.
Over the wall, I started hearing yet another conversation that went something
like this...
Para: Building Reliable Applications. English syntax. ‘Applications' is the important part. He
wants to know what an application is. From the Merriam-Webster Online
Dictionary, ‘a program (as a word processor or a spreadsheet) that performs one
of the major tasks for which a computer is used'. So he wants to know how to
reliably build software programs.
Noid: English
syntax, yes; but the important part is Building. He has a set of reliable
applications that he wants to build. Simple - have him look at tools like make,
ant, Maven, or Anthill. Script everything and make sure it is all acquired from
a Version Control system.
Schiz: You are
both wrong, he wants to reliably build applications! Para,
just for you I looked up ‘reliable' in the Merriam-Webster Online Dictionary.
It states that it is an adjective meaning suitable or fit to be relied on,
dependable, or giving the same result on successive trials. He wants to be able
to reproducibly build applications.
Fran: You are all wrong. Or maybe you
are all right. It is just that the question is too vague. Para,
you know there is no such thing as a reliable application of any size or
complexity. There is no affordable way to ensure that it always does what it is
supposed to. Noid, of course he wants to build the
applications. Pieces parts only works if selling cheap
chicken. Schiz, you just want him to be able to
accomplish something he has control over. Ben, what did you mean?
Ben: <Thinking> Why am I hearing these guys?
</Thinking> I sent this out because I wasn't sure what was really being
asked either. Knowing the audience, I would assume the long title would really
be something like, ‘How to Reliably Build Supposedly Reliable Applications and
Ensure They Are as Reliable as Expected.' It just looks too long that way.
Para: Fran, you
were right - we were all right, only
we didn't go far enough! He wants reliable, reproducible, controlled automated
testing of the reliable, reproducible, consistent, automatically built
applications with automated post-build log analysis.
Ohhhhkkkaaaayyyy. This might exceed the scope of the article a little. I
hope. But let's see what we can cover...
First, let's talk a little about reliable applications in
the sense that they do both what they are supposed
to do and what they are expected to
do. The first implies that there are a set of requirements that the application
can be tested against. If all of the requirements are met, then the application
does what it is supposed to do. CM does not elicit, negotiate, document, review
or approve requirements. We do tend to get stuck with versioning them and
sometimes tracing them to the appropriate (versioned) test plans. Expectations
are intangibles that can make the difference between an accepted application
and one that is passed by and dies. CM has nothing to do with expectations,
though QA does and CM is responsible for ensuring the proper versioning of the
associated test plans.
Now reliable implies the applications execute the same way
every time. This means that unless a requirement changes or a bug is fixed, the
application should return the same results regardless of its release level. QC
generally accomplishes this through regression testing. If the regression
testing is automated and can be executed from the build system, then it should
be - at least on release candidates. If the tests can be performed fairly
quickly, then they can also be run as a part of "nightly" builds. This "early
return" can in turn allow for automated creation of Defect records. The faster
the notification of the developer, the less has been forgotten and the quicker
an appropriate fix can be made. It will also tend to introduce fewer additional
defects since the developer doesn't have to relearn the code in order to fix
it.
Assuming a stable build environment, the builds are
reproducible because everything is built from a version controlled repository
using the same tools and the same options. This also makes for consistent
builds. By keeping versioned "build environments" available it is possible to
extend the time a build is reproducible almost indefinitely. It is also a Buildmeister nightmare to keep track of, especially if the
environment selection is done as a part of the automated build procedure.
QA
proactively ensures quality by making sure low risk high quality processes and
standards are in place and followed throughout the development life cycle. For
the most part this requires both causal analysis and metrics. CM is the source
for a large part of QA's metrics. Some of these are:
- Elements added, removed and changed per release
- Elements added, removed and changed per Change
Request
- Average number of lines of code added, removed
and changed per element revision
- Number of times an element (or module - a group
of closely related elements) is changed between releases
- Number of times an element (or module - a group
of closely related elements) is changed per Change Request
- Number times an element (or module) is changed before
a Change Request is satisfied and passed by QC
This is by no means an exhaustive list, but it is a list
of reasonably easy to garner metrics. QA then does some further analysis to
determine if there are trends in the data or if there are spikes in the data.
Both initiate QA causal analyses to determine why the trends or spikes are
occurring. Note that the trends or spikes may be good things happening as well
as bad. At this point, QA only wants to know why so they can foster ever
increasingly "good" practices.
A classic example of a QA recommended change in C coding
practices is,
Do not code like this:
if( val
== 0 ) return 0;
The assignment statement is often
mistyped instead of the equality test
if( val
= 0 ) return 0;
This will always return a true
value and val will be reassigned a zero
value, regardless of what its previous value was. Instead, always put the
constant on the left side of the equality test. If this form is mistyped, the compiler will catch it and issue an
error.
if( 0 == val
) return 0;
As process and procedural changes are implemented,
subsequent metric collection provides the data that allows QA to determine if
the change itself was "good" or "bad." This overall process is called
Continuous Process Improvement.
We have already discussed running automated regression
tests as a part of the build process, but there are other types of tests, both
automated and manual, that QC generally runs. For reasonably stable
applications, automated end-to-end and functional tests are common. They augment
the simpler regression tests and more thoroughly test the overall functionality
of the application. Where regression tests determine of something changed that
should not have, these tests ensure that new functionality behaves as it should
and that external interfaces are working properly. Manual and ad-hoc tests are
also performed by testing personnel to ensure that "outside the box" use does
not cause unacceptable results - like a crash or corrupted data. A final
category is load and performance testing. These tests ensure that the system
can survive under abnormal load conditions and that they perform in a timely
fashion.
And last, but not least, we come back to automated build
log analysis. Just as QC's automated regression tests
look for expected and unexpected results, so should some post-build log
analysis tool. Unexpected warnings and errors, changes in the number of
elements processed, tasks skipped, gross changes in log file length, etc. are
all causes for human inspection. Why do this? Because if done to enough detail
to allow causal analysis, build logs are BIG! Creating and running filters
against the logs allow one to reduce the massive amount of data to a reasonable
amount of information.
So I guess we can at least touch on ways to ensuring reliable, reproducible, controlled
automated testing of the reliable, reproducible, consistent, automatically
built applications with automated post-build log analysis. Doing it requires
time, patience, automation, integration across disciplines and yet more time.
Not doing it allows for recalls, poor press, longer development cycles and
inconsistent quality. Plus time spent during off hours trying to find out what
went wrong and the added costs of finding problems later in the life cycle.
Why did I subtitle this "A Pipe Dream?" Because it seems
to be almost impossible to get Management blessing to produce applications this
way, regardless of the development methodology in use.
Ben Weatherall is currently based in Fort Worth, Texas where
he practices Practical CM on a daily basis using a combination of CVS
and custom tools to support a modified Agile-SCRUM development
methodology. He is a member of IEEE, ASEE (Association of Software
Engineering Excellence – The SEI’s Dallas based SPIN Affiliate), NTLUG
(North Texas Linux Users Group), and PLUG (Phoenix Linux Users Group).
Trackback(0)
Comments 
Write comment
 |
CM Journal Articles 
TrackBack URI for this entry
Subscribe to this comment's feed