International Journal of Reliability, Risk and Safety: Theory and Application

International Journal of Reliability, Risk and Safety: Theory and Application

Software Defect Discovery and Resolution Modeling incorporating Severity

Document Type : Original Research Article

Authors
Maskura Nafreen 1
Ying Shi 2
Lance Fiondella 1
1 Department of Electrical and Computer Engineering, University of Massachusetts Dartmouth, North Dartmouth, Massachusetts, USA
2 Mission Software and Ground System Assurance Branch, NASA Goddard Space Flight Center, Greenbelt, Maryland, USA
10.22034/IJRRS.2024.7.1.8
Abstract
Traditional software reliability growth models only consider defect discovery data, yet the primary concern of software engineers is defect removal. Past attempts to model defect resolution emphasize approaches based on differential equations and queueing theory. However, these models do not explicitly identify the activities performed to remove defects and resources allocated to these activities according to their severity. Models should consider these practical factors to enable more detailed resource allocation and planning.
This paper presents a model to predict the number of defects resolved according to the discrete Cox proportional hazard model with covariates, demonstrating the approach with covariates on the number of low, medium, and high severity defects that were discovered but not resolved in successive intervals. A comparison with differential equation-based and distributional approaches reveals that the covariate model performs better on each goodness of fit measure considered and requires less time to apply. The covariate model also better tracks unresolved defects and exhibits low predictive error, even when as little as 10-20% of testing has been completed. These results suggest that collecting information on defect resolution activities and the corresponding effort dedicated could substantially improve defect resolution modeling to guide process improvement.
Keywords
Software defect resolution
Software defect severity
Software defect tracking
Software reliability
Software reliability modeling
Subjects
  1. Jelinski and P. Moranda, "SOFTWARE RELIABILITY RESEARCH," in Statistical Computer Performance Evaluation, W. Freiberger Ed.: Academic Press, 1972, pp. 465-484.
  2. F. Schneidewind, "Analysis of error processes in computer software," in Proceedings of the international conference on Reliable software, 1975, pp. 337-346, doi: https://doi.org/10.1145/800027.808456.
  3. -H. Lo and C.-Y. Huang, "An integration of fault detection and correction processes in software reliability analysis," Journal of Systems and Software, vol. 79, no. 9, pp. 1312-1323, 2006/09/01/ 2006, doi: https://doi.org/10.1016/j.jss.2005.12.006.
  4. Shibata, K. Rinsaka, and T. Dohi, "Metrics-Based Software Reliability Models Using Non-homogeneous Poisson Processes," in 2006 17th International Symposium on Software Reliability Engineering, 7-10 Nov. 2006 2006, pp. 52-61, doi: https://doi.org/10.1109/ISSRE.2006.28.
  5. Dohi, T. Matsuoka, and S. Osaki, "An infinite server queuing model for assessment of the software reliability," Electronics and Communications in Japan (Part III: Fundamental Electronic Science), vol. 85, no. 3, pp. 43-51, 2002/03/01 2002, doi: https://doi.org/10.1002/ecjc.1078.
  6. Dohi, S. Osaki, and K. S. Trivedi, "An infinite server queueing approach for describing software reliability growth: unified modeling and estimation framework," in 11th Asia-Pacific Software Engineering Conference, 30 Nov.-3 Dec. 2004 2004, pp. 110-119, doi: https://doi.org/10.1109/APSEC.2004.29.
  7. S. Gokhale and R. E. Mullen, "Queuing models for field defect resolution process," in 2006 17th International Symposium on Software Reliability Engineering, 2006: IEEE, pp. 353-362, doi: https://doi.org/10.1109/ISSRE.2006.38.
  8. N. Misra, "Software reliability analysis," IBM Systems Journal, vol. 22, no. 3, pp. 262-270, 1983, doi: https://doi.org/10.1147/sj.223.0262.
  9. Yamada, S. Osaki, and H. Narihisa, "A software reliability growth model with two types of errors," RAIRO-Operations Research, vol. 19, no. 1, pp. 87-104, 1985, doi: https://doi.org/10.1051/ro/1985190100871.
  10. Fiondella and S. S. Gokhale, "Software Reliability Models Incorporating Testing Effort," OPSEARCH, vol. 45, no. 4, pp. 351-368, 2008/12/01 2008, doi: https://doi.org/10.1007/BF03398825.
  11. Rinsaka, K. Shibata, and T. Dohi, "Proportional intensity-based software reliability modeling with time-dependent metrics," in 30th Annual International Computer Software and Applications Conference (COMPSAC'06), 2006, vol. 1: IEEE, pp. 369-376, doi: https://doi.org/10.1109/COMPSAC.2006.68.
  12. Nafreen, M. Luperon, L. Fiondella, V. Nagaraju, Y. Shi, and T. Wandji, "Connecting software reliability growth models to software defect tracking," in 2020 IEEE 31st International Symposium on Software Reliability Engineering (ISSRE), 2020: IEEE, pp. 138-147, doi: https://doi.org/10.1109/ISSRE5003.2020.00022.
  13. Sukhwani, J. Alonso, K. S. Trivedi, and I. Mcginnis, "Software reliability analysis of NASA space flight software: A practical experience," in 2016 IEEE International Conference on Software Quality, Reliability and Security (QRS), 2016: IEEE, pp. 386-397, doi: https://doi.org/10.1109/QRS.2016.50.
  14. Xie and M. Zhao, "The Schneidewind software reliability model revisited," in Proceedings Third International Symposium on Software Reliability Engineering, 7-10 October 1992 1992, pp. 184-192, doi: https://doi.ieeecomputersociety.org/10.1109/ISSRE.1992.285846.
  15. Ohba, "Inflection S-shaped software reliability growth model," in Stochastic Models in Reliability Theory: Proceedings of a Symposium Held in Nagoya, Japan, April 23–24, 1984, 1984: Springer, pp. 144-162, doi: https://doi.org/10.1007/978-3-642-45587-2_10.
  16. Yamada, M. Ohba, and S. Osaki, "S-shaped reliability growth modeling for software error detection," IEEE Transactions on reliability, vol. 32, no. 5, pp. 475-484, 1983, doi: https://doi.org/10.1109/TR.1983.5221735.
  17. Kapur and S. Younes, "Software reliability growth model with error dependency," Microelectronics Reliability, vol. 35, no. 2, pp. 273-278, 1995, doi: https://doi.org/10.1016/0026-2714(94)00054-R.
  18. Yamada, K. Tokuno, and S. Osaki, "Imperfect debugging models with fault introduction rate for software reliability assessment," International Journal of Systems Science, vol. 23, no. 12, pp. 2241-2252, 1992, doi: https://doi.org/10.1080/00207729208949452.
  19. S. Gokhale, P. N. Marinos, M. Lyn, and K. S. Trivedi, "Effect of repair policies on software reliability," in Proceedings of COMPASS'97: 12th Annual Conference on Computer Assurance, 1997: IEEE, pp. 105-116, doi: https://doi.org/10.1109/CMPASS.1997.613262.
  20. -Y. Huang, C.-T. Lin, S.-Y. Kuo, M. R. Lyu, and C.-C. Sue, "Software reliability growth models incorporating fault dependency with various debugging time lags," in Proceedings of the 28th Annual International Computer Software and Applications Conference, 2004. COMPSAC 2004., 2004: IEEE, pp. 186-191, doi: https://doi.org/10.1109/CMPSAC.2004.1342826.
  21. Ullah, M. Morisio, and A. Vetro, "A comparative analysis of software reliability growth models using defects data of closed and open source software," in 2012 35th Annual IEEE Software Engineering Workshop, 2012: IEEE, pp. 187-192, doi: https://doi.org/10.1109/SEW.2012.26.
  22. Liu, M. Xie, J. Yang, and M. Zhao, "A new framework and application of software reliability estimation based on fault detection and correction processes," in 2015 IEEE International Conference on Software Quality, Reliability and Security, 2015: IEEE, pp. 65-74, doi: https://doi.org/10.1109/QRS.2015.20.
  23. Yang, Y. Liu, M. Xie, and M. Zhao, "Modeling and analysis of reliability of multi-release open source software incorporating both fault detection and correction processes," Journal of Systems and Software, vol. 115, pp. 102-110, 2016/05/01/ 2016, doi: https://doi.org/10.1016/j.jss.2016.01.025.
  24. Cinque, D. Cotroneo, A. Pecchia, R. Pietrantuono, and S. Russo, "Debugging‐workflow‐aware software reliability growth analysis," Software Testing, Verification and Reliability, vol. 27, no. 7, p. e1638, 2017, doi: https://doi.org/10.1002/stvr.1638.
  25. Vizarreta et al., "Assessing the maturity of SDN controllers with software reliability growth models," IEEE Transactions on Network and Service Management, vol. 15, no. 3, pp. 1090-1104, 2018, doi: https://doi.org/10.1109/TNSM.2018.2848105.
  26. Vizarreta, E. Sakic, W. Kellerer, and C. M. Machuca, "Mining software repositories for predictive modelling of defects in sdn controller," in 2019 IFIP/IEEE Symposium on Integrated Network and Service Management (IM), 2019: IEEE, pp. 80-88.
  27. Xie, H. Qiu, Q. Zhai, and R. Peng, "A model of software fault detection and correction processes considering heterogeneous faults," Quality and Reliability Engineering International, vol. 39, no. 8, pp. 3428-3444, 2023, doi: https://doi.org/10.1002/qre.3172.
  28. Joe, "Statistical inference for general-order-statistics and nonhomogeneous-Poisson-process software reliability models," IEEE Transactions on Software Engineering, vol. 15, no. 11, pp. 1485-1490, 1989, doi: https://doi.org/10.1109/32.41340.
  29. -T. Lin, C.-Y. Huang, and C.-C. Sue, "Measuring and assessing software reliability growth through simulation-based approaches," in 31st Annual International Computer Software and Applications Conference (COMPSAC 2007), 2007, vol. 1: IEEE, pp. 439-448, doi: https://doi.org/10.1109/COMPSAC.2007.141.
  30. -Y. Huang and W.-C. Huang, "Software reliability analysis and measurement using finite and infinite server queueing models," IEEE Transactions on Reliability, vol. 57, no. 1, pp. 192-203, 2008.
  31. Zhang, G. Cui, and H.-w. Liu, "A finite queuing model with generalized modified Weibull testing effort for software reliability," in Proceedings of 2011 International Conference on Computer Science and Network Technology, 2011, vol. 1: IEEE, pp. 401-406, doi: https://doi.org/10.1109/ICCSNT.2011.6181985.
  32. Kapur, H. Pham, S. Anand, and K. Yadav, "A unified approach for developing software reliability growth models in the presence of imperfect debugging and error generation," IEEE Transactions on Reliability, vol. 60, no. 1, pp. 331-340, 2011, doi: https://doi.org/10.1109/TR.2010.2103590.
  33. -Y. Huang and T.-Y. Kuo, "Queueing-theory-based models for software reliability analysis and management," IEEE Transactions on Emerging Topics in Computing, vol. 5, no. 4, pp. 540-550, 2017, doi: https://doi.org/10.1109/TETC.2014.2388454.
  34. Tokuno, T. Nagata, and S. Yamada, "Stochastic software performability evaluation based on NHPP reliability growth model," International Journal of Reliability, Quality and Safety Engineering, vol. 18, no. 05, pp. 431-444, 2011, doi: https://doi.org/10.1142/S0218539311004172.
  35. Okamura and T. Dohi, "A generalized bivariate modeling framework of fault detection and correction processes," in 2017 IEEE 28th International Symposium on Software Reliability Engineering (ISSRE), 2017: IEEE, pp. 35-45, doi: https://doi.org/10.1109/ISSRE.2017.22.
  36. M. Khoshgoftaar and J. C. Munson, "Predicting software development errors using software complexity metrics," IEEE Journal on Selected Areas in Communications, vol. 8, no. 2, pp. 253-261, 1990, doi: https://doi.org/10.1109/49.46879.
  37. M. Khoshgoftaar, B. B. Bhattacharyya, and G. D. Richardson, "Predicting software errors, during development, using nonlinear regression models: a comparative study," IEEE Transactions on Reliability, vol. 41, no. 3, pp. 390-395, 1992, doi: https://doi.org/10.1109/24.159804.
  38. M. Khoshgoftaar, J. C. Munson, B. B. Bhattacharya, and G. D. Richardson, "Predictive modeling techniques of software quality from software measures," IEEE Transactions on Software Engineering, vol. 18, no. 11, pp. 979-987, 1992, doi: https://doi.org/10.1109/32.177367.
  39. Shi, M. Li, S. Arndt, and C. Smidts, "Metric-based software reliability prediction approach and its application," Empirical Software Engineering, vol. 22, pp. 1579-1633, 2017, doi: https://doi.org/10.1007/s10664-016-9425-9.
  40. M. Evanco and R. Lacovara, "A model-based framework for the integration of software metrics," Journal of Systems and Software, vol. 26, no. 1, pp. 77-86, 1994/07/01/ 1994, doi: https://doi.org/10.1016/0164-1212(94)90098-1.
  41. M. Evanco, "Poisson analyses of defects for small software components," Journal of Systems and Software, vol. 38, no. 1, pp. 27-35, 1997/07/01/ 1997, doi: https://doi.org/10.1016/S0164-1212(97)00063-0.
  42. L. Goel and K. Okumoto, "Time-dependent error-detection rate model for software reliability and other performance measures," IEEE transactions on Reliability, vol. 28, no. 3, pp. 206-211, 1979, doi: https://doi.org/10.1109/TR.1979.5220566.
  43. E. RamÍRez Cid and J. Alberto Achcar, "Software Reliability Considering the Superposition of Non-homogeneous Poisson Processes in the Presence of a Covariate," Statistics, vol. 36, no. 3, pp. 259-269, 2002/01/01 2002, doi: https://doi.org/10.1080/02331880212854.
  44. K. Ray, Z. Liu, and N. Ravishanker, "Dynamic reliability models for software using time-dependent covariates," Technometrics, vol. 48, no. 1, pp. 1-10, 2006, doi: https://doi.org/10.1198/004017005000000292.
  45. Gandy and U. Jensen, "A non‐parametric approach to software reliability," Applied Stochastic Models in Business and Industry, vol. 20, no. 1, pp. 3-15, 2004, doi: https://doi.org/10.1002/asmb.510.
  46. Ishii, T. Fujiwara, and T. Dohi, "Bivariate extension of software reliability modeling with number of test cases," International Journal of Reliability, Quality and Safety Engineering, vol. 15, no. 01, pp. 1-17, 2008, doi: https://doi.org/10.1142/S0218539308002897.
  47. Kapur, A. G. Aggarwal, and A. Tandon, "Two dimensional software reliability growth model with faults of different severity," Communications in Dependability and Quality Management, vol. 13, no. 4, pp. 98-110, 2010.
  48. Okamura, Y. Etani, and T. Dohi, "A multi-factor software reliability model based on logistic regression," in 2010 IEEE 21st International Symposium on Software Reliability Engineering, 2010: IEEE, pp. 31-40, doi: https://doi.org/10.1109/ISSRE.2010.14.
  49. Okamura, Y. Etani, and T. Dohi, "Quantifying the effectiveness of testing efforts on software fault detection with a logit software reliability growth model," in 2011 Joint Conference of the 21st International Workshop on Software Measurement and the 6th International Conference on Software Process and Product Measurement, 2011: IEEE, pp. 62-68, doi: https://doi.org/10.1109/IWSM-MENSURA.2011.26.
  50. Kuwa and T. Dohi, "Generalized Logit Regression-Based Software Reliability Modeling with Metrics Data," in 2013 IEEE 37th Annual Computer Software and Applications Conference, 2013: IEEE, pp. 246-255, doi: https://doi.org/10.1109/COMPSAC.2013.41.
  51. Kuwa, T. Dohi, and H. Okamura, "Generalized Cox proportional hazards regression-based software reliability modeling with metrics data," in 2013 IEEE 19th Pacific Rim International Symposium on Dependable Computing, 2013: IEEE, pp. 328-337, doi: https://doi.org/10.1109/PRDC.2013.55.
  52. Okamura and T. Dohi, "A novel framework of software reliability evaluation with software reliability growth models and software metrics," in 2014 IEEE 15th International Symposium on High-Assurance Systems Engineering, 2014: IEEE, pp. 97-104, doi: https://doi.org/10.1109/HASE.2014.22.
  53. Okamura and T. Dohi, "Towards comprehensive software reliability evaluation in open source software," in 2015 IEEE 26th International Symposium on Software Reliability Engineering (ISSRE), 2015: IEEE, pp. 121-129, doi: https://doi.org/10.1109/ISSRE.2015.7381806.
  54. Wiper, A. Palacios, and J. Marín, "Bayesian software reliability prediction using software metrics information," Quality Technology & Quantitative Management, vol. 9, no. 1, pp. 35-44, 2012, doi: https://doi.org/10.1080/16843703.2012.11673276.
  55. Torrado, M. P. Wiper, and R. E. Lillo, "Software reliability modeling with software metrics data via Gaussian processes," IEEE Transactions on Software Engineering, vol. 39, no. 8, pp. 1179-1186, 2012, doi: https://doi.org/10.1109/TSE.2012.87.
  56. Nagaraju, C. Jayasinghe, and L. Fiondella, "Optimal test activity allocation for covariate software reliability and security models," Journal of Systems and Software, vol. 168, p. 110643, 2020/10/01/ 2020, doi: https://doi.org/10.1016/j.jss.2020.110643.
  57. Yamada, J. Hishitani, and S. Osaki, "Software-reliability growth with a Weibull test-effort: a model and application," IEEE Transactions on Reliability, vol. 42, no. 1, pp. 100-106, 1993, doi: https://doi.org/10.1109/24.210278.
  58. Moranda, "Prediction of software reliability during debugging," in Proc. 1975 Annu. Reliab. Maintainance Symp., 1975.
  59. Farr, "Software reliability modeling survey," in Handbook of software reliability engineering: McGraw-Hill, Inc., 1996, pp. 71–117.
  60. J. Archdeacon, Correlation and regression analysis: a historian's guide. Univ of Wisconsin Press, 1994.
  61. Sharma, R. Garg, C. K. Nagpal, and R. K. Garg, "Selection of optimal software reliability growth models using a distance based approach," IEEE Transactions on Reliability, vol. 59, no. 2, pp. 266-276, 2010, doi: https://doi.org/10.1109/TR.2010.2048657.
  62. Akaike, "A new look at the statistical model identification," IEEE transactions on automatic control, vol. 19, no. 6, pp. 716-723, 1974, doi: https://doi.org/10.1109/TAC.1974.1100705.
  63. Schwarz, "Estimating the dimension of a model," The annals of statistics, pp. 461-464, 1978, doi: https://doi.org/10.1214/aos/1176344136.
International Journal of Reliability, Risk and Safety: Theory and Application
Volume 7, Issue 1
February 2024
Pages 59-72

  • Receive Date 28 February 2024
  • Revise Date 12 May 2024
  • Accept Date 25 May 2024