An Empirical Study of the Reliability of UNIX Utilities (1990)

This study examined the reliability of UNIX utilities using empirical testing procedures. The authors tested roughly 90 different utility programs on seven versions of UNIX, including a commercial version. The paper presents the tools used to test the UNIX programs, the tests, the results, and an analysis of the results. The authors, using the tools and testing procedures described, were able to crash more than 24 percent of the UNIX utility programs. The authors point out that the study is important because (1) it uncovers a large set of bugs that could be corrected, (2) it provides a means of identifying programming practices that could cause bugs or security holes, and (3) it simulates real crashes that could occur because of careless typing, noisy input, or noisy phone lines. A good portion of the paper discusses common mistakes made by programmers that caused the utilities to fail. This paper is a good example of how relatively simple techniques can uncover significant results. Because of the pervasive use of UNIX and the diverse set of environments examined in this study, many readers can benefit by the analysis of UNIX utility reliability. Tables included in the paper give specific names of utilities that failed; the analysis suggests reasons for the failures. -"Brett D. Fleisch"