In his article "Test Estimation" (STQE magazine, Nov./Dec. 2002), Rex Black describes the major elements of the test estimation process; for example, techniques for compromise between estimate extremes, project management tools to use, how to predict bug find and fix rates, and options to consider when the estimated test time exceeds Management's plan. The best estimation techniques fail, however, if no one considers the factors that influence the team's effort, time, and resources. In this companion piece, the author explains those factors that affect the test effort for good or ill.
Sometimes, even expert project managers-especially those who are unfamiliar with software testing projects-have real trouble estimating the time and resources to allocate for testing. Careful application of project estimation best practices is a good start, but there's more involved than following the rules of project management.
System engineering-including the testing-is a complex, high-risk human endeavor. As such, it's important to combine good estimation techniques with an understanding of the factors that can influence effort, time, dependencies, and resources. Some of these factors can act to slow down or speed up the schedule, while others, when present, can only slow things down.
Some of these factors arise from the process by which work is done. These include
- the extent to which testing activities pervade the project or are tacked on at the end
- clearly defined hand-offs between testing and the rest of the organization
- well-managed change control processes for project and test plans, product requirements, design, implementation, and testing
- the chosen system development or maintenance lifecycle, including the maturity of testing and project processes within that lifecycle
- timely and reliable bug fixes
- realistic and actionable project and testing schedules and budgets
- timely arrival of high-quality test deliverables
- proper execution of early test phases (unit, component, and integration)
Some of these factors are material and arise from the nature of the project, the tools at hand, the resources available, and so forth, including
- existing, assimilated, high-quality test and process automation and tools
- the quality of the test system, by which I mean the test environment, test process, test cases, test tools, and so forth
- an adequate, dedicated, and secure test environment
- a separate, adequate development debugging environment
- the availability of a reliable test oracle (so we can know a bug when we see one)
- available, high-quality (clear, concise, accurate, etc.) project documentation like requirements, designs, plans, and so forth
- reusable test systems and documentation from previous, similar projects
- the similarity of the project and the testing to be performed to previous endeavors
Some factors arise from the people on the team-and these can be the most important of all-including
- inspired and inspiring managers and technical leaders
- an enlightened management team who are committed to appropriate levels of quality and sufficient testing
- realistic expectations across all participants, including the individual contributors, the managers, and the project stakeholders
- proper skills, experience, and attitudes in the project team, especially in the managers and key players
- stability of the project team, especially the absence of turnover
- established, positive project team relationships, again including the individual contributors, the managers, and the project stakeholders
- competent, responsive test environment support
- project-wide appreciation of testing, release engineering, system administration, and other unglamorous but essential roles (i.e., not a "individual heroics" culture)
- use of skilled contractors and consultants to fill gaps
- honesty, commitment, transparency, and open, shared agendas among the individual contributors, the managers, and the project stakeholders
Finally, some complicating factors, when present, always increase schedule and effort, including:
- high complexity of the process, project, technology, organization, or test environment
- lots of stakeholders in the testing, the quality of the system, or project itself
- many subteams, especially when those teams are geographically separated
- the need to ramp up, train, and orient a growing test or project team
- the need to assimilate or develop new tools, techniques, or technologies at the testing or project levels
- the presence of custom hardware
- any requirement for new test systems, especially automated testware, as part of the testing effort
- any requirement to develop highly detailed, unambiguous test cases, especially to an unfamiliar standard of documentation
- tricky timing of component arrival, especially for integration testing and test development
- fragile test data, for example, data that is time sensitive
Process, material, people, and complications...On each project, specific aspects of the project in each category influence the resources and time required for various activities. When preparing a test estimate, it's important for the test manager and those on the test team who help with estimation to consider how each of these factors will affect the estimate.
Forgetting just one of these factors can turn a realistic estimate into an unrealistic one. Experience is often the ultimate teacher of these factors, but smart test managers can learn to ask smart questions-of themselves and the project team-about whether and how each factor will affect their project.
