Home 
CM Journal Articles Ten Points for Improving the Testing Process
CM Journal Articles Ten Points for Improving the Testing Process
|
| Testing is an integral part of the production lifecycle, including software and hardware design, and plays an important role in the overall quality assurance process that should optimally be part of every project, in every organization and business.
The following points are beneficial to consider when approaching the testing process. 1. Reinforce the Need for Testing Though testing is required, companies are often reluctant to allocate time and resources to the process. As a concept, testing may be regarded as evidence that there is an imperfection in the software manufacturing process, but it is the only means through which project managers can verify the actual quality of the work before it reaches its users. In addition, testing can often be subjected to the “far end” effect; that is, it is perceived to be at the “far end” of the project and therefore not given any attention until too late in the lifecycle. 2. Minimize Testing Requirements Although one measure of a successful test process is to demonstrate that errors can indeed be detected, if too many errors are found, emphasis needs to be placed on the development process that allowed errors to propagate throughout the life-cycle in the first place. In order to reduce design flaws, more and more development teams are taking advantage of graphical analysis and design notations such as the Specification and Description Language (SDL, ITU-T Recommendation Z.100) or the Unified Modeling Language (UML). These notations and the tools that support them allow improve quality in several ways, including the early verification of requirements through design test and simulation. 3. Allocate Adequate Resources at an Early Stage A common reason for not detecting errors at an early stage is that the test tool is inadequate or that the test team lacks the required skills or knowledge of the application under test. The result is that the quality assurance of the project is jeopardized. To prevent this from happening, the test team should have the resources to detect any subtle errors that may be present in any design. Do not rely on designers to detect errors as they normally can only detect a very small percentage of errors they themselves create. Make sure the testing team is properly trained and prepared and is independent of the design team (thus they will find a higher percentage of the errors). Remember that testing is part of the critical path of a project — therefore, it is as important to have skilled testing personnel as it is to have skilled designers, engineers, and managers. It is a well-known fact that the cost of fixing errors is dramatically reduced if the error can be detected early in the development lifecycle. A test team needs to be built up and maintained, as the requirements for a system are being understood (errors in the requirements are the cheapest to fix if caught early but the most expensive if caught too late). Testing should as early as possible, i.e., before development has been completed. 4. Plan for Testing A test plan should be available, not only covering what to test, but also how to test. Neglecting the latter may result in severe problems during the testing phase of a project. User interfaces are just as important as the services that need to be tested. Normally, this is not so much a problem for unit testing as it is for system or deployment testing. In this instance, the following should be asked early in the design phase: How do you test software when it is deployed on the target platform, and how do you trace any software bugs that show up only on the target? Two things are vitally important. 1) If you have no reason for a test then that test should not exist. This reason can only be documented by having clear traceability back to the original requirement of a system. 2) If you have no traceability information how do you know that your tests cover the requirements adequately? Not getting a high enough level of requirements coverage impacts the overall quality of the system that is produced. 5. Have Clear Objectives for Testing There are many objectives in testing: performance, regression, conformance, usability, acceptance, performance, and more. Each of these tests requires specific tools. Some tests may be purely algorithmic and automatic, while others require that the application actually be bundled on the target hardware. In some cases, manual testing is the most suitable method. Remember to analyze what tools are required for each of the testing objectives. There is not one single tool that solves all of the problems, but some, possibly in combination with others, will give a very robust testing platform. Always concentrate effort on a combination of testing the most important requirements with the most effective way of finding errors. 6. Use Standardized Test Suites Some telecommunication protocol standards contain standardized test suites. Always check the protocol standards to see if there are test suites for the application you are designing or testing. If so, use the standard test suite as a starting point for testing, as this saves time and gives the test engineer the additional benefit of using the standard tests with virtually no extra effort. If you are designing a new protocol, consider designing test suites as well. The European Telecommunication Standards Institute (ETSI), the Bluetooth Special Interest Group (SIG) and the ATM Forum are two organizations providing such test suites. Some of these suites have been implemented by commercial organizations and allow any company to purchase them in an executable format. However, be aware that these test suites are often updated as the standards evolve. 7. Design Tests for Reusability It is important to design software for reuse, and it is also important to be able to reuse tests. The standardized test suites provided by standards organizations are only one category of reusable tests. Test engineers may think that these standardized tests are only an outline of how the application is to behave (which is true for some early standardized test cases), but most of today’s standardized conformance test suites are also executable (after some adaptation to the implementation). These standardized tests provide a good outline for how to compose platform-independent and reusable tests, using a technology that has been successfully employed by European organizations for several years. It is also important to consider the maintainability of tests. 8. Use Commercial Tools for Testing Testing of complex applications cannot be conducted in a reliable and repeatable manner without the proper tools. The principles of software engineering apply to software testing as well — no one would ever consider creating a complex, real-world application without tools (CASE tools, compilers, etc.). Engineers need tools for testing as well as for development. 9. Use Non-Proprietary Test Languages Standardized languages are often better documented than their non-standardized counterparts. For example, a project manager may choose to develop a proprietary testing tool and language as part of a testing plan. This decision may lead to more development time spent on that subproject than on the actual main project. Needless to say, the outcome of the project would be less than successful, and the developed test tool would fall into oblivion. In order to alleviate this problem, some universities and companies offer courses in the use of standardized and formal languages. Non-proprietary test languages also have a far more rigorous level of scrutiny applied to them, considering such things as the language syntax to the detailed semantics of what should happen when a test executes. This experience should be reused when testing rather than attempting to reinvent the wheel by developing an in-house test language. 10. Use a Specialized Test Language There are specialized languages for expressing tests. It is as important to select the right language for testing as it is to select a suitable language for development. TTCN is a language with a long and rich history in the area of conformance testing of communications protocols, but has recently been updated to apply to a range of other testing problems, and certainly not just limited to the communications domain. The latest version is know as TTCN-3 (Test and Test Control Notation) and has been standardized by ETSI. Any test language should have adequate support for timing, data definitions, value definitions, concurrency and synchronous and asynchronous communication. Using a specialized test language such as TTCN-3 brings opportunities to outsource testing, to use commercial off-the-shelf test tools and to reuse or purchase tests from 3rd party sources. Conclusions Testing is a critical part of any development organization and needs to treated as such. Testing has its own life-cycle that parallels the design and development life-cycle – test planning, requirements analysis, design of test cases, writing of test cases, test execution and traceability – and can only be managed effectively when the right level of resources are dedicated to it. Lars Mats is a Senior Architect at Telelogic AB in Malmö, Sweden. He has been involved in the development of Telelogic’s tool family and in consulting for test suite deployment and implementation for various sites throughout Europe, the US and Australia. He received his MS in computer science from Uppsala University, Uppsala, Sweden, and can be reached by email at lars.mats@telelogic.com Matthew Graney is Director of Technical Marketing for Telelogic North America Inc. He joined Telelogic in October 1999, prior to which he worked with Motorola in Australia in the development of TTCN System Test solutions for GSM/GPRS cellular handsets. He graduated from the University of Adelaide, Australia in 1993 with an honors degree in Computer Systems Engineering, and can be reached by email at matthew.graney@telelogic.com
Set as favorite
Bookmark
Email this
Hits: 4076 Trackback(0)Comments (0)
|
| Last Updated on Wednesday, 26 July 2006 04:30 |
TrackBack URI for this entry
Subscribe to this comment's feed