Conducting Effective Pilot Studies

1 Sponsored by the Department of Defense 2004 by Carnegie Mellon Universitypage 1 Pittsburgh, PA 15213-3890 Carnegie MellonSoftware Engineering InstituteConducting EffectivePilot StudiesMark KasunicSoftware Engineering Measurement and Analysis [SEMA]Software Engineering SEPG Conference 2003 by Carnegie Mellon Universitypage 2 Carnegie MellonSoftware Engineering InstituteThe ProblemQuite often, software/system improvements are made withoutmeasurement either before or after the change was , how do we know if the outcome was better or worsethan the original situation?In these cases, interpretation is based on opinion andimpressions but there is a lack of data to back it up!

2 Therefore, there is less confidence in the results of theinnovation interpretation is problematic and there s a risk thatconsensus about the results are not do we know if the change worked? 2003 by Carnegie Mellon Universitypage 3 Carnegie MellonSoftware Engineering InstituteNeed to Validate That Changes AreEffectiveAs-Is StateHypothesized Magnitude of ImprovementAs-Is StateTo-Be StateThe New As-Is StateActual Magnitude of ImprovementAre they the same? Or not?How do we know it worked? 2003 by Carnegie Mellon Universitypage 4 Carnegie MellonSoftware Engineering InstituteTypical Approaches to EvaluatingImprovementTypical approaches that failX represents the introduction of a changeO represents a measurable observationBefore the changeChange introducedAfter the ChangeApproach #1 XApproach #2 XOApproach #3XO 2003 by Carnegie Mellon Universitypage 5 Carnegie MellonSoftware Engineering InstituteHow Do You Know It Really Worked?

3 In all three approaches, there is no way to tell if the outcome fromthe change was better or worse than the original approaches that failBefore the changeChange introducedAfter the ChangeApproach #1 XApproach #1 XApproach #2 XOApproach #2 XOApproach #3 XOHow do you knowthe change worked? 2003 by Carnegie Mellon Universitypage 6 Carnegie MellonSoftware Engineering InstituteScientific Methods Do ExistResearch designs do exist for proper interpretation ofresults after innovations are introduced .. but they arerarely applied!Before the changeChange introducedAfter the ChangeXO2O1As-Is StateTo-Be StateTransition 2003 by Carnegie Mellon Universitypage 7 Carnegie MellonSoftware Engineering InstituteA Structured Approach to a Pilot StudyPlan and design the Pilot study1 Train personnel to accomplish change 2 Support and monitor Pilot study3 Evaluate Pilot results4 Make recommendations & improve5 2003 by Carnegie Mellon Universitypage 8 Carnegie MellonSoftware Engineering InstituteA Structured Approach to a Pilot study Define the problem How will you measuresuccess?

4 Where will youconduct the pilotstudy? Designing yourapproach usingscientific principles Writing down yourplanPlan and design the Pilot study1 Train personnel to accomplish change 2 Support and monitor Pilot study3 Evaluate Pilot results4 Make recommendations & improve5 2003 by Carnegie Mellon Universitypage 9 Carnegie MellonSoftware Engineering InstituteDefine the ProblemA problem that is clearly defined is the problem means identifying a gap between somedesired situation and the current important challenge is for the improvement team tocollect and use valid information to define the currentsituation instead of assuming that

5 It already has thenecessary valid problem statement implies no particular solutions and nopotential good problem statement states only the current anddesired situation. 2003 by Carnegie Mellon Universitypage 10 Carnegie MellonSoftware Engineering InstituteA Structured Approach to a Pilot StudySubtopics Define the problem How will you measuresuccess? Where will youconduct the pilotstudy? Designing yourapproach usingscientific principles Writing down yourplanPlan and design the Pilot study1 Train personnel to accomplish change 2 Support and monitor Pilot study3 Evaluate Pilot results4 Make recommendations & improve5 2003 by Carnegie Mellon Universitypage 11 Carnegie MellonSoftware Engineering InstituteDevelop Pilot study Success CriteriaYou will want to know:Did the solution component generate the outcome that it wasintended to achieve?

6 What are you hoping for in terms of performance changewhen using the solution component? Try to define performance standards that will help youdetermine this explicitly. Are there any historical data that canbe used to baseline the status quo?Did the users experience difficulty in its use? What are your expectations in terms of the solutioncomponent s impact on changing people s attitudes andbehaviors? Try to define qualitative measures that will provide objectiveassessment of job improvement for the users of the 2003 by Carnegie Mellon Universitypage 12 Carnegie MellonSoftware Engineering InstituteA Structured Approach to a Pilot StudySubtopics Define the problem How will you measuresuccess?

7 Where will youconduct the pilotstudy? Designing yourapproach usingscientific principles Writing down yourplanPlan and design the Pilot study1 Train personnel to accomplish change 2 Support and monitor Pilot study3 Evaluate Pilot results4 Make recommendations & improve5 2003 by Carnegie Mellon Universitypage 13 Carnegie MellonSoftware Engineering InstituteWill we be Able to Generalize Our Findings?Is there anything we can do to increase the probability thatwe can generalize the Pilot study results to the largerpopulation? Is this a typical program/project within theorganization? Is the experience and skill level of the Pilot studypersonnel typical of what one would find in otherprograms/projects in the organization?

8 Are there factors beyond our control that can confoundor influence the cause-and-effect relationship of thechange we are trying to evaluate?Making smart decisions about where (inyour organization) to conduct a pilotstudy improves confidence withgeneralizing the solution to other partsof the organization. 2003 by Carnegie Mellon Universitypage 14 Carnegie MellonSoftware Engineering InstituteUnderstanding the Pilot EnvironmentDuring Pilot study planning, you can mitigate the risk of misinterpreting orover-interpreting your eventual results if you can identify and characterizethe impact of potential influences.

9 How similar is the project/program environment of the candidate pilotproject to other projects in the organization (size, domain, etc.)? Will participants in the Pilot study embrace the proposed change (thatis being considered) or resist adopting it? What are the adoption characteristics of the Pilot project manager andthe project staff in general? Given the adoption characteristics of the Pilot participants, how muchrefinement (of the solution component) and support will be necessaryto test the potential effectiveness of change? How supportive is the project/program manager to the change that willbe piloted?

10 Are they enthusiastic about the idea of serving as a Pilot ? What kinds of pressure is the project/program already under?- Difficult schedule constraints?- New product or domain area?- Inexperienced staff? 2003 by Carnegie Mellon Universitypage 15 Carnegie MellonSoftware Engineering InstituteProject Categories ExampleProject TypeDescription on-time release is imperative 20-30 individuals on project staff using new object-oriented technologyCritical 5-10 individuals on project staff improvements to baseline productsProductenhancement < 5 individuals on project staff correction of bugs reported by usersMaintenance 5-20 individuals on project staff(dependent on need)