Behavioral characterization: Finding and using the influential factors in software process simulation models

Most software process simulation work has focused on the roles and uses of software process simulators, on the scope of models, and on simulation approaches. Consequently, the literature reflects a growing body of models that have recently been characterized by modeling purpose, scope, key result variables, and simulation method. While the software process simulation arena is maturing, little effort appears to have been given to statistical evaluation of model behavior through sensitivity analysis. Rather, most of software process simulation experimentation has examined selected factors for the sake of understanding their effects with regard to particular issues, such as the economics of quality assurance or the impact of inspections practice. In a broad sense, sensitivity analysis assesses the effect of each input on model outputs. Here, we discuss its use for behaviorally characterizing software process simulators. This paper discusses the benefits of using sensitivity analysis to characterize model behavior; the use of experimental design for this purpose; our procedure for using designed experiments to analyze deterministic simulation models; the application of this procedure to four published software process simulators; the results of our analysis; and the merits of this approach.