Home 
CM Journal Articles Traceability and Auditability
CM Journal Articles Traceability and Auditability
|
| What is Traceability?
Deffinition Traceability, according to the American Heritage Dictionary, is the adverb form of the verb "trace," which has the following meanings among others: "To follow the course or trail of..." and "To ascertain the successive stages in the development or progress of..."[1] What we normally mean is the ability to trace the development of software from some point in its Product Development Lifecycle (PDLC or SDLC) to the point at which it is released for general use. Note that some organizations have two different "kinds" of traceability: Requirements and Changes. Requirements to Release Level The most common assumption is that traceability starts at the Requirements level and runs through the final Test level. An example of this is shown in the Software Development Lifecycle diagram below. 
In this model, each requirement is uniquely identified and its implementation is traced though the individual code elements to the tests that validate its implementation. Note that versioned requirements may appear and disappear during the life of a product's development and maintenance, so the traceability needs to be to this level of detail as well. In the diagram below, the files that are created and/or changed are traced to the requirements they implement and from there to the tests that specifically address them. It is up to a human to certify that the tests "prove" the requirement is satisfied. 
There is a more relaxed form of traceability from requirements that skips the source level entirely and just shows the relationship between the various tests and the requirements that are shown as satisfied by their execution. Change Management Level Traceability at the Change level tracks the source changes made to implement each Change Request. Change Requests may in turn be based on changes to requirements, enhancements or defects found. In many organizations, this is the only real traceability even attempted. It is especially common in organizations that misapply the Agile Manifesto by not documenting and tracking requirements as they evolve. Note that it is possible to do both Requirements and Change tracking as they are not supersets of each other. What is Auditability? Definition According to Wikipedia, "Auditability is a non-functional requirement (also known as one of the Ilities), and concerns the transparency of a system with regards to external audits."[2] It is the ability to audit, which, as Dictionary.com says, is "an official examination and verification of accounts and records."[3] There are several types of audits that software systems are subject to, the most commonly referenced being the FCA and the PCA. FCA Formal Configuration Audit - From Steven Kershaw, "is that the software was built in accordance with procedures and that it operates as planned. That proof (from the FCA) is the formal software test report." Even if it is not called this, most organizations do something like this. The ensure the deliverables are built properly and tested against the current requirements. PCA Physical Configuration Audit - From Steven Kershaw, "the artifacts leading up to and through formal test (peer review data, peer review audits, process audits, software baseline audits etc) all collectively make up the majority of a PCA." Many commercial organizations do not do this type of audit unless they are forced to. Other Other types of CM related audits or inspections include, but are not limited to:
Typically, no one wants to perform an audit. One reason is that an audit report is a quasi-legal document. The primary reason though is the amount of time and effort a good audit takes. There is some controversy as to whether the CM Manager (CMM) should be involved in the CM audit process. By doing the audit, the CMM will discover more actual problems than most other auditors; however it is conflict of interest. A better approach is for the CM team to ensure, via internal assessments, that the processes are being followed, things are being built correctly, etc. and then letting someone outside the group audit them. The QA group is a natural for this. Participants There should always be an audit leader who has overall responsibility for the audit process and takes legal responsibility for its completion, documentation, content and reporting. In addition to the audit leader, there are generally one or more subject matter experts (SMEs) that are changed with the actual performance of the interviews, inspections, and document reviews. Their expertise allows them to do more than just acknowledge that documents exist and people respond. They can delve deeper into any areas that appear questionable to determine if there are truly violations present or not. Documentation Each audit should result in an audit report that has a predefined structure and format. The format of the report should be defined as a part of each audit's definition, along with its scope, distribution list, level of sensitivity, etc. The final report should not be generally available in most cases. It is a controlled document that should be protected from unauthorized changes and any authorized changes must be identified and logged. Formality The very word "audit" carries a connotation of formality, and most audits are intended to be formal in nature. But this is not always the case. Part of defining an audit needs to be how formal it is. If it is to meet regulatory requirements or oversight, then it will be very formal. If it is to just make sure things are on track, then less formality may be reasonable and appropriate. Peer reviews are a good case in point; in commercial organizations, most of them are performed very informally and not documented at all. Others produce Peer Review Reports, but do not treat them as controlled documents. The goal is to have one or two audits more formal than the rest and perhaps use the results of less formal audits as source material. Unless it is necessary, performing formal audits is a great consumer of time and human resources that often occurs at the worst possible times. Periodicity How often should audits be performed? This should be decided up front as a part of defining the audits themselves along with the level of formality, audit leader, documentation to be produced and the distribution list to be used. The most common two choices are at milestone points in the lifecycle, such as just prior to or just after a major release, or at chronological intervals that are often enough to make the audit reasonably quick and far enough apart to not always be getting ready to do one. The goal of the audit is to find discrepancies and allow them to be corrected. To do this the interval between audits needs to be short enough that discrepancies can be addressed before additional "harm" can occur. Any quicker than this and nothing will be gained. Longer than this will turn the audit into a port-mortem report. Who Retains? Since audit reports are quasi-legal documents, the master copy should be retained by whoever is acting as the legal representative within the organization. At least one copy should be retained in off-site storage along with other backups and disaster recovery material. All other copies should be treated as controlled copies that can be recalled if necessary. For How Long? If the audits are mandated by a controlling authority or regulatory agency, the retention period will be specified by them. I recommend keeping them for at least one year beyond that for reference purposes. If the retention period is not mandated, then it should have been decided when the audit was being defined. If you are paranoid, seven years is a good round number. When is Too Much? The simple answer is, "When the cost of the audits exceeds the costs of any possible faults they may uncover." If they stand in the way of getting product to market in a timely fashion or if they are being performed just for the sake of performing them, then they are too much. CM itself is basically a combination of paranoid common sense, good management practices and risk assessment. Use these to determine ahead of time is a particular audit really has potential payback or not. Remember, audits have three primary purposes:
[1] Traceability. Dictionary.com. The American Heritage® Dictionary of the English Language, Fourth Edition. Houghton Mifflin Company, 2004. http://dictionary.reference.com/browse/Traceability (accessed: August 17, 2007). [2] Auditability. Reference.com. Wikipedia, the free encyclopedia. http://www.reference.com/browse/wiki/Auditability (accessed: August 17, 2007). [3] auditable. Dictionary.com. Dictionary.com Unabridged (v 1.1). Random House, Inc. http://dictionary.reference.com/browse/auditable (accessed: August 17, 2007). Ben Weatherall is currently based in Fort Worth, Texas where he practices Practical CM on a daily basis using a combination of CVS and custom tools to support a modified Agile-SCRUM development methodology. He is a member of IEEE, ASEE (Association of Software Engineering Excellence – The SEI’s Dallas based SPIN Affiliate), NTLUG (North Texas Linux Users Group), and PLUG (Phoenix Linux Users Group).
Set as favorite
Bookmark
Email this
Hits: 4870 Trackback(0)Comments (2)
|
TrackBack URI for this entry
Subscribe to this comment's feed |
... Hi Ben, you wrote that traceability may skip the source level, and requirements may refer to tests only. But what are tests? The test definitions and the scope of test execution is subject to the development process itself. As the source code evolves, there must be an according test development in parallel (or even better: tests should be written before the according feature development starts) that covers the new features. Therefore in fact, test definitions are sources. Maybe you meant that tracing covers the relationship between the requirements and the test results, that can be seen as part of the results of an overall build process. But nevertheless, a build is always coupled to a snapshot of sources or a source baseline. Hence, there will always be the need for a relationshiop to the sources. Thank you. |
O.S. Balaji
said:
|
... Hi Ben The importance of traceability has been explained quite well ! Just a small clarification , is FCA generally called as Formal Configuration Audit or Functional Configuration Audit?In either names ,though the intent is the same as you had explained , just wanted to know the general terminology used ! Thanks. |
Write comment
You must be logged in to post a comment. Please register if you do not have an account yet.
