Home 
CM Journal Articles SCM Assessment
CM Journal Articles SCM Assessment
|
 In past articles and forum threads I have discussed how "pain" drives tool changes and how to go about identifying and evaluating potential tool additions/replacements. This month, I want to cover how to assess your existing tools and processes to determine where you really need to focus. We always think we know what the problem is. That is why we often fix the symptoms and not the problems, but do we even know what all of the symptoms are? There are three cases that can be addressed by going directly into a selection process:
It has long been known that those in the trenches often know more about problems than their superiors. Or at least they perceive the problems differently. By surveying a wide, representative sample of personnel, it is possible to get past the symptoms to the real problems. The Survey One of the best ways to elicit this information is to conduct a survey, also called a questionnaire, where the identity of the person submitting the data is unknown. If you work within the organization being surveyed, you will either need to be very trusted and have permission from upper management that the source of the data collected may remain anonymous OR be a contractor hired to perform the survey who has confidentiality as one of the criteria spelled out in the contract. If this is not possible, then use whatever methods you have at your disposal to have the information collected be anonymous via some automated mechanism. The reason for anonymity is that it allows people to express opinions that are contrary to company policy. It even allows them to point fingers at superiors as the cause of problems. Note that it is also necessary to ensure that each person is surveyed only once. So, what should you ask? There are several schools of thought on this ranging from detailed, multiple choice questions tailored to elicit specific data to open ended questions that require the respondent to answer in paragraphs. I have found that that the open ended questionnaire yields the most information. It allows a respondent to provide data that was unexpected and unplanned for - data that may well point to real problems that were not anticipated. There is a link in the References section to a sample survey form I have used in the past with remarkably good results. It is only one page, so you might want to open it and refer to it during the rest of this section. The survey may be used in two ways:
There are four sections in the survey: Demographic, Scope of CM, Technology and Process, though none of these sections is identified as such within the questionnaire. There are several reasons for this - it prevents biasing the answers, it allows answers to one section to bleed over into other sections depending on how respondents understand the questions, and it prevents narrowing the "open endedness" of the questions. Section 1: Demographics The first few entries are self explanatory and are the only ones that are normally defined by the organizational entity (which may be the same as the company) being surveyed. The entry, "What role do you perform?" may be answered by a job title, but a list of roles is really more general. By asking if the respondent performs multiple roles, it allows them to detail what they really do within the organization. It is not uncommon to discover that the rest of the results are clustered around the demographics. This data should only be used to determine if there are perceptions, biases or "problems" that are experienced by specific groups of people. What is found may well not be what is expected. Even though the sample survey does not mention it, the Myers-Briggs personality type might be a good extension to the demographic data collected. So far as I know, this is not common practice, but it might be interesting to see if perceived problems are grouped by personality types rather than by other demographics. I have provided some references at the end of the article in case you are interested in adding this to your surveys. Section 2: Scope/Knowledge of CM The three entries starting with, "What is CM to you?" probe the level of understanding as to what function(s) CM actually performs. It is intended to find out if the information CM assumes everyone is aware of is truly known about, as well as the scope of control CM is expected to perform. The first question about CM tools can also yield insight into this as some thing the tools are Development's, some think they are QA's, etc. It is often perceived that CM is spending so much time "fixing others' tools." The question on CM level of support also yields some data for this section. Often CM is held accountable for tools, training and infrastructure that are "owned" by other groups. Section 3: Technology All of the questions relating to tools, both CM and non-CM, yield data for this section. One of the reasons for the number of questions in this section is that it is generally perceived that tools are both the problem and the solution to organizational problems. The purpose of these questions is to assist in determining if there is truly a need to replace or re-implement one or more tools. Stress in the instructions that if a respondent is unhappy with a tool, they should go into detail about why and even give examples if possible. Try to elicit whether it would make a difference if a "painful" tool was reconfigured or used in a different fashion. Sometimes a tool has acquired so poor a reputation that no matter what is done to make its use less painful, users will not accept it. They may have become permanently biased against it to the extent that it must be replaced even if it can do the job. Section 4: Process Though it might not seem like it, the questions regarding training are part of the process section. Since who creates training materials, who provides the expertise the training materials are based on and who performs the actual training are often different groups, it is often only a process that relates them to each other. Regardless, it is often necessary for CM to augment the official training with tips, tricks and traps than need to be presented in a less formal setting and/or at more appropriate times. It is also necessary to determine whether the pain being felt is due to the processes in effect. Even if that is true, it is also necessary to determine whether there is anything that can be done to change the process. As has been said before, it is often those in the trenches that know what does and what does not work and have good suggestions as to what to do to "fix things." Data Collection and Analysis In order for the results of a survey to be meaningful, a statistically significant number of surveys must be completed. There are all kinds of studies, both empirical and mathematical, that determine how many respondents are needed. What I have found from personal experience is that 10% of the available pool (with a minimum of 3 from each group) needs to be successfully surveyed. If the surveys are analyzed as they become available, it may be determined early that the results are converging and a smaller number may actually be necessary. Conversely, if the results are diverging, it may be necessary to expand the number of surveys. As the surveys are completed, they should be analyzed and entered into either a database or a spreadsheet to form a data repository. There is a link in the References section for a sample spreadsheet that I will use for the rest of this article. The first tab is used to track the status of those who will be personally interviewed. The ID may be a name (if no one other than the planner will see that tab) or an anonymous identifier the interviewer can use to distinguish one interview from another. As implied, this tab is primarily used for interview selection and planning purposes and should be separated into a separate spreadsheet or data table with a high security level. The second tab is the raw data as collected from the surveys. The ID field is only used to allow traceability back to the individual surveys themselves. The rest of the fields match the questions on the survey. If an individual answer is not easily quantifiable, or if it is desirable to capture more of the answer for explanatory purposes, add notes to the far right of the spreadsheet. In the sample, there is room for 4 notes per respondent. Feel free to add to or modify the fields, especially those associated with tools, training and processes, as data starts arriving. And yes, this means that some of the early respondents' data will need to be reevaluated and/or reclassified - so make provisions for this when you initially set up your data repository. You should start noticing fairly early that certain key words or phrases keep appearing. These should be noted in the data repository, even if new fields become necessary to capture them. The Next Steps As the survey approaches completion, start trying to "slice and dice" the data to detect patterns. You will want to use accepted statistical analysis methods in order to have the results acceptable to Management. The statistical mathematics need not be complex. It is reasonable to use simple percentages of respondents with specific answers. Graphing your numbers is a good way to determine if responses cluster, converge, diverge or have multiple spikes. Depending on how the data graphs, it may be possible to use standard deviation to throw out responses to individual questions that are at the ends of a bell curve. It may be that the initial assumptions are validated, but I can almost guarantee that you will learn something you did not expect. At the least, you will validate that you are on the right track. At the most, you will discover that you can save time and/or money by tweaking existing tools and/or processes, or by simply providing appropriate training. References
Ben Weatherall is currently based in Fort Worth, Texas where he practices Practical CM on a daily basis supporting a modified Agile-SCRUM development methodology. He uses a combination of AccuRev, CVS, Bugzilla and AnthillPro (as well as custom tools). He is a member of IEEE, ASEE (Association of Software Engineering Excellence – The SEI’s Dallas based SPIN Affiliate), FWLUG (Fort Worth Linux Users Group), NTLUG (North Texas Linux Users Group) and PLUG (Phoenix Linux Users Group).
Set as favorite
Bookmark
Email this
Hits: 2988 Trackback(0)Comments (0)
|
| Last Updated on Tuesday, 21 July 2009 15:07 |
TrackBack URI for this entry
Subscribe to this comment's feed