Developers are all generally aware that better code documentation means a more easily maintained system. Many IT managers know that there is a lack in code or user documentation in the systems that they are responsible for. We’ve talked to hundreds of IT managers over the past 40 years (HCi was formed in 1981, just before MS-DOS came onto the market), and a lot of them have been uneasy about the level of documentation of their systems.

The response has always been: we know it’s probably worthwhile to do something about it, but how can we justify it? And how can we get the developers to actually do it?

Software engineering researchers have spent a lot of time (and money) asking questions like: will a code repository help with code reuse, or will the adoption of an OO language reduce build times. There is very very little research on documentation or its effects: software engineers just don’t seem very interested in it, even though it is probably one of the most critical factors in code reuse, for example.

One key piece of documentation research has been done in recent years, however, by Eirik Tryggeseth at the Norwegian University of Science and Technology. He carried out a carefully designed controlled experiment with over 30 test subjects, in which each of them had to analyse a modification request for a non-trivial application (2.7 KSLOC of uncommented C++).

Half of the subjects had:

  • requirements specification
  • design documents
  • test report
  • user manual

… and the other half did not. Both sets had:

  • the source code
  • a small sample of input and corresponding output
  • a presentation on the system
  • a demo of the system

Tryggeseth measured how long the subjects spent understanding the code, and how long writing pseudocode for the modification. He also measured their actual level of understanding, and the quality of the pseudocode.

The most important result of his research was: that the programmers who had no documentation spent 21.5% longer understanding the code. Translating this into the average IT maintenance environment means that undocumented code is more than 20% more expensive to analyse for changes, giving a direct potential saving in time.

How do you turn this into budget? These days, IT projects are increasingly driven by the need to show ROI. By estimating the cost of development of documentation (which we can do for you) and then comparing it with the current wasted cost of the additional 20% of maintenance over the expected life of the system, it is easy to see whether documenting a system is going to be worthwhile. And therefore easy to make a case for budget to do it.

Some of the other important results of the research were:

  • the developers with documentation gained a more thorough understanding of the system (measured objectively by a standardised test)
  • the developers with documentation did a better job of designing the change (again, measured objectively)

A particularly interesting result was that, without documentation, differences in skill levels between good and bad programmers disappeared. With documentation, good programmers did better work than poor programmers. Without documentation, they both did equally bad work. So spending money on the best people is a waste without documentation.

Reference: Tryggeseth, Eirik Report from an experiment: impact of documentation on maintenance in Journal of empirical software engineering. Kluwer Academic 2: 2, 201-207 1997 ISSN 1382-3256

See also: M. Leotta, F. Ricca, G. Antoniol, V. Garousi, J. Zhi and G. Ruhe, “A Pilot Experiment to Quantify the Effect of Documentation Accuracy on Maintenance Tasks,” 2013 IEEE International Conference on Software Maintenance, Eindhoven, Netherlands, 2013, pp. 428-431, doi: 10.1109/ICSM.2013.64.