New Initiative in Programming - Foundation for Open Project Documentation

Introduction

Simplicity needs projection and good taste. - L. Torvalds

It is a mistake to think that programmers wares are programs. Programmers have to produce trustworthy solutions and present it in the form of cogent arguments. Programs source code is just the accompanying material to which these arguments are to be applied to. - E. Dijkstra

In engineering practice, “project” assumes the development of project documentation.

Not long ago one of the authors watched how one notable programmer (participant of two ACM International Collegiate Programming Contest world finals) was wasting 15 minutes to understand a short program (6 lines of code). He knew that the program presented an iterative solution of classical problem “Hanoi towers”. After that we found in the Internet another solution, with rather good description of the algorithm .

Open Source Codes and Programs Understandability

In fact open source software does not ensure program understandability. The Leading analyst of BASF Corporation, professor of Fairleigb Dickinson University (NJ), N. Bezrukov, the author of the site http://www.softpanorama.org/, believes [1]: “The central question of practical programming is the question of source code understanding.

It is always good to have original code, but the problem is that in most cases it is not enough. Understanding of any non-trivial program requires additional documentation. This need grows exponentially with the size of the source code. Code analysis for recollection of initial project solutions and program understanding are two important branches of programming technology. For instance, make a try to understand structure of the compiler if you have no definition of formal language, which it compiles.

Everybody, who has participated in large-scale software reengineering projects, remembers the sense of helplessness, which occurs when you see the heap of badly documented (but, may be, very good written) source code. Availability of the source codes does not help when there is no access to key developers and ideas. If program is written, for example, in C programming language (relatively low-level one) and its documentation leaves much to be desired then all project solutions dissolve in the coding details. In such situations the value of high-level documentation like specifications, interfaces definitions and architecture description may raise the value of source code!

Lack of fit of original codes for program understanding results in appearing of methods, which unite code and documentation. One of the most famous attempts of such solutions was undertaken by D. Knuth in his book “Literate Programming” [2]. Probably, the most well-known prohibited book in the history of computer science was “Commentary on Unix: With Source Code” [3], which contains high-level explanation of source codes of Unix operating system even with description of used algorithms. This book has been copied and distributed illegally for more then twenty years form the moment of publication in 1977!

If you have not participated in project from its early stages then its complexity and size hides source codes from you. Understanding of “ancient” code is, probably, one of the most difficult sorts of programmers work if there is no documentation or initial developers.”

And one more opinion: “Does any worthy freeware program, whose [source codes] appearance does not arouse disgust, exist? Therefore there are rather few good freeware, in spite of its great amount” [4].

The possible result of this tendency was described by great Russian mathematician, L. S. Pontriagin: “Only well-done work brings pleasure! If it is done roughly, it causes aversion and bit by bit cultivates in the person immoral attitude to the labour” [5].

Why Are not Programs Projected?

So, working without source code is bad, but with source code it is not good also. There is lack of project documentation, which has to be made detailed, fine and accurate. Software source code is to be included in the project documentation as the component.

We can remember only three examples, which are produced commonly with no projection: children, works of art and, unfortunately, software.

It is also important that usage of documentation does not depend on the amount of production. Even the single dress in the home framework is sewed using patterns! Bridges, roads, skyscrapers are building according to documentation, but the software does not.

The situation, which is widely known in software developing, can be described by Weinberg’s Second law: “If builders built buildings the way programmers wrote programs, then the first woodpecker that came along would destroy civilization” [6].

Why there are a lot of documents, which are let out with devices and hardware? That documentation is quickly understandable and designed for specialists with average level of qualification. Such situation allows to rework and change device or hardware (with the documentation) easily even after the lapse of many years. For programs such sort of documentation usually is absent.

We see the cause of such state in the following. First of all, devices and hardware are manufactured for usage by exterior consumers. So the lack of documentation will force developer to spend rest of the life in the “factory”, but, we hope, he does not plan it. In software developing the situation is different. In most cases, the consumer of the source code (not of the final product, but only the code) is a developer (the same organization). The common opinion is that there are no reasons to create good quality documentation inside the organization.

Secondly, devices and hardware are “hard”, and software is “soft”. This means that it is much easier to make changes in the software, than in the hardware, but it does not mean that there is no need in the documentation of program. In fact the majority of programmers pathologically would not read and, even more, write documentation [7].

Experiments show that there are few young programmers, who are able to write program documentation. Regardless of the fact that in their universities they have passed examinations on large courses of mathematics. It did not give them ability to expound logically. For example, the same object can be named “lamp”, “Lamp”, “bulb” or “Bulb” in the different parts of documentation. The fantasy is unlimited! Developing environment helps to avoid such errors while programming, but when writing documentation there is no rescue.

Program developing has become similar to show-business with its pursuit of profits. Nobody is interested in projects future (especially, the far future). The main categories are “profitable-unprofitable” instead of “good-bad”. But in most cases the good technology is also the profitable one.

Unwillingness to write documentation may be caused by the fact that the more closed (undocumented) the project is, the more indispensable the author is!

Such work style, unfortunately, extended to software developing for especially crucial systems.

Programs are written, but not projected! While projecting, any techniques, which are more complicated than CRC-card [8] or usage diagram [9], are considered to be too intricate and are not used. The programmer can always refuse to use any technology, substantiating it for his chief saying that he could not do it in time” [10].

As a result, even users do not think erroneous program behavior is something extraordinary [11].

At present there exists general opinion in the society, that big houses should be projected and be well-documented, but for the software this is not needed. That’s why we can ask a question: why in the projects like “Digital House” it’s components are projected and documented with different level of quality?

In conclusion of this section let us note that present situation didn’t exist at the beginning of the era, when programmers used “big”, ancient machines, first computers. At that time programs were projected and written very thoroughly, because next attempt of its execution may be performed only in a day. So technical progress has resulted in less responsible developing.

Advantages of Project Documentation

Having well-done project documentation, software developer cannot “control” managers. After programmers dismissal, his place can be borrowed by anyone even with lower level of qualification (and salary), instead of person with higher qualification (as now).

Is it possible to teach how to project and implement programs by books? We think that it is. But now, unfortunately only with separate books: about projecting [11] and about implementing [12]. Unfortunately, there are almost no books, which cover both stages of software developing. Absence of such literature could be filled up by open projects, software with open project documentation, which allows experiencing advantages and disadvantages of made systems. It is similar to new kind of patterns [13]. Project documentation makes the refactoring [14] of the software easier.

Such is to contain, in particular, formal specification of programs logic, because “things, which have no formal specification, could not be verified and so could not be erroneous” [15]. “If there is no specification then there are no errors” J [16].

Moreover, project documentation is supposed to hold “protocols (histories of computations), which help to understand programs functionality; so theorists of programming are forming the opinion that set of protocols is characterizing program much better than the source code” [17].

Without project documentation, one of the main advantages of object-oriented programming - code reusing, may result in troubles [18].

And the main point. Project documentation is necessary, because we know from the algorithms theory (Rice’s theorem) that in common case it is impossible to prove truth of any non-trivial properties of computing function algorithmically, having only programs source code. “Non-trivial property” means that there are programs, which possess this property and which do not.

According to Turing, program understanding (opposite to its execution) requires “astuteness and inventiveness”. But source code analysis could not be automated and human analysis requires huge amount of time. So it is impossible to make any conclusions about programs properties!

Mathematicians found solution of similar problem in the antiquity. We mean writing proofs with the help of human language (documenting proofs). This method differs Greek mathematical school from the Egyptian school. In the Egyptian school, solutions of, for example, geometrical tasks were presented as a figure with “explanatory” legend: “Look!” It is similar to understand the program only with the help of its source code.

To allow others to understand what the program does and how it does it, there is a need to give a detailed description for the person with average level of qualification. This description has to cover programs developing and its static and dynamic properties. It actually represents project documentation and proves that the program computes necessary function (corresponds to requirements specification).

So, obviously, it is possible to assert that “project documentation” does not mean “operating documentation” (user’s manual, software development kit or something of this kind)

Basing on the explanation above, it is obvious that software project documentation is indispensable. For different projects it can be secret, partly open or open. But, at least, for the purposes of education, project documentation has to be open.

Variety of requirements for the project documentation

The following picture shows the documentation requirements for different kinds of projects.

Requirements for the project documentation

Let’s comment on the picture. Documentation for critical objects control software can be divided in two parts - basic, which should be created according to the standarts and extra, developed for the needs of customer. Other commercial projects are usually documented at customer’s request.

The first type of commercial projects must be well-documented, but it is not always so, because there is not much experience in writing such kind of documentation and people are not used to do it, unlike in IBM and some other companies.

Undertime and economy are often the reasons of poorly documented projects. The customer usually requests only some manuals and operating guides.

Non commercial software, which is not educational, is documented at discretion of the developer. In most of the cases, the developer doesn’t see the need to document the project, except for writing comments in the source code and making understandable names for functions and variables.

So we face joyless facts, especially in the cases, when non commercial software is used together with the commercial software, for which the customer has cut on the development of documentation.

In the described approach, the code should be based on the documentation, but not vice-versa. Meanwhile the documentation should describe not only the product itself, but also the creation process. It should be clear for the average user of the computer. Then average programmer after 11 hours of work will be able to understand it. That's why the documentation should be created to make the average user understand it. And it is desirable, that that user would sit with you. Such process (by analogy with extreme programming) can be named "extreme documentation writing".

Foundation for Open Project Documentation

One of the authors (Anatoly Shalyto) declared “Foundation for Open Project Documentation” on the opening of North-Eastern European semifinal competitions of ACM International Collegiate Programming Contest (Saint-Petersburg, November 2002). For support and propagation of the foundation site http://is.ifmo.ru was created.

At the Computer Technologies Department of Saint-Petersburg State University of Information Technologies, Mechanics and Optics the special pedagogical experiment began. Students were divided into nearly 40 groups (one or two persons in each group). Each group was to develop some project, using state-based programming technology (programming based on automata theory) [19]. Created systems with full project-documentation are to be published on http://is.ifmo.ru (section “Projects”). Many of them are already available.

Let us list some of finished projects or projects in developing:

1. concept of visualization of algorithms for teaching of discrete mathematics and programming;
   
2. realization of interactive scripts for educational animation using Macromedia Flash;
   
3. environment for teaching and testing students for arbitrary subjects (with example for lessons of English language);
   
4. combined usage of compiler developing theory and state-based programming;
   
5. skeleton animation;
   
6. usage of XML for describing appearance of video player (projects home site: http://www.crystalplayer.com/);
   
7. controlling systems for different devices (diesel generator, elevator, turnstile, coded lock, cash dispenser, traffic lights, coffee-machine, phone and many others);
   
8. bank security system;
   
9. client-server architecture, using state-based programming;
   
10. graphical user interfaces, using state-based programming;
    
11. SMTP-protocol implementation;
    
12. classical parallel problems: “Synchronization of the Chain of Shooters” and “Task about Philosophers Dinner”;
    
13. games: “Robocode” [20], “Terrarium”, “CodeRally”, “Sea Wars”, “Lines”, “Automatic Bomber”, “One-armed Bandit” and “Bank” (“Zavalinka”)).

Important note: among hundreds of tanks for the “Robocode” game, only our tank has full project documentation [21]. The same situation is with the “Terrarium” [22] game.

It is logically to assume that if the world is going to open source codes then there will be time for wide usage of open project documentation. This will allow to replace reading of source code by reading project documents.

The Tale at the End

While developing one of our projects (“Sea Wars” game), after eighth cycle correction (and that is not a record) projects author, student of Computer Technologies Department, lost his smile like a Cheshire cat. We were satisfied by the quality project. Program has good user interface, works well and was accurately documented.

But open project documentation shows both, advantages and disadvantages of the program. One notable programmer (prize-winner of competitions of ACM International Collegiate Programming Contest) looked through source code and projects author began to rework it again. This process (with breaks) continues more then half a year and we hope that in this project everything will be perfect like in Chekhov’s stories: the face (user interface), the clothes (documentation), the soul (source code) and thoughts (programs work).

Why the documentation should be opened?

The “open” project documentation means, that it, firstly, should exist and, secondly, it should be available for futher use and possibly for futher modification.

“Foundation for open project documentation” is free, anyone can support it. But it differs from “Free Software Foundation” and “Open Source Foundation”, because the ideas and concepts of open project documentation can be used not only in free software, but in commercial, secret and other projects.

Conclusion

The great amount of work to write good quality documentation does not allow to suppose that offered foundation will find acceptance in software developing show-business. Offered technology is “hard”, but now only “easy” and “agile” methods [23] become popular.

Nevertheless there are some fields of programming, where it is impossible to come without “hard” technologies. So new people, who need and like software with good project documentation, appear.

One of the students, after he saw project documentation, written according to our approach, for the first time, said that it was more detailed than TV-set’s documentation. He supposed that submarines were documented in same manner.

Even if usage of project documentation does not become popular, it is very significant from the pedagogical point of view. Developing of well documented projects is useful for the participants. It is also important for those, who just look through projects because of their cognitive and aesthetic value. You know that not all visitors of the museum are artists.

Let us finish with the quotation of one opinion about our approach: “There was a lack of Open Project Documentation. Almost all documentation for the commercial projects, unfortunately, serves as customers property and they do not hurry to proclaim it. That is the cause of such small amount of real projects in the Internet. There are a lot of sites with source code, but next to nothing with the project solutions. I looked into the section “Projects” on your site (http://is.ifmo.ru/) and compared my solutions with the offered. It is a pity that I have no access to these works earlier. I will not spend so much time for projecting and developing!” [24]

Finally, we would like to point out that the article reflects the tendency of the last years: the software development seems to start “growing up” - transforming from the art through the science [25, 26] to the engineering craft [27, 28].

One of the pioneers of the aircraft D. Douglas said once - "when the weight of the documentation becomes equal to the weight of the airplane, it is ready to fly", and A. Martin affirms, that the documentation for the famous Boeing 747 weighed more, than the plane itself. The same idea concerns the development of the software. Without the proper documentation the development process is out of control. Electronic formats of documentation have decreased the weight of the documentation, but it didn't change the fact of the matter.

This research is supported by Russian Fund for Fundamental Investigations (grant №02-07-90114, “Technology of Automata Programming Development”).

The materials of this paper will be reported on the Linux Summit 2004 (http://www.linuxsummit.org/summit2004_program.shtml).

References

1. Bezrukov N. Reiterated Look at “Council” and “Market” // BYTE/Russia. 2000. № 8.
2. Knuth D. Literate Programming. Stanford: Center for the Study of Language and Information, 1992.
3. Lions J. Commentary on UNIX: With Source Code. Annabooks, 1977.
4. Protasov P. From Below // Computerra. 2003. № 19.
5. Researcher of “Steering Wheel” // Informatics. 2003. № 11.
6. Bloch A. Murphy's Law // ECO. 1983. № 1-3.
7. Demin V. Problems of Working of Russian Developers on the West // PC Week/Russian Edition. 2001. № 32.
8. Badd T. Object-oriented Programming. SPb.: Piter, 1997.
9. Booch T., Rambo D., Jacobson A. UML. Users Manual. M.: DMK, 2000.
10. Fowler M. New Methods of Programming // http://www.spin.org.ua/.
11. Booch G. Future Creation // Open Systems (Otkrytye sistemy). 2001. № 12.
12. Stroustrup B. The C++ Programming Language. M.: Binomial (Binom), Addison-Wesley, 2000.
13. Gamma E., Helm R., Johnson R., Vlissidis J. Design Patterns: Elements of Reusable Object-Oriented Software. Addison-Wesley, 1995.
14. Fowler M., Beck K., Brant J., Opdyke W., Roberts D. Refactoring: Improving the Design of Existing Code. Addison-Wesley, 1999.
15. Zaitsev S. Description and Implementation of Computer Nets Protocols. M.: Science (Nauka), 1989.
16. Allen E. Bug Patterns in Java. APress, 2002.
17. Ershov A. Mixed Computations // In the World of Science (V mire nauki). 1984. № 6.
18. Telles M.A., Telles M., Hsieh U. The Science of Debugging. The Coriolis Group, 2001.
19. Shalyto A., Tukkel N. Programming with explicit state separation // World of PC (Mir PK). 2001, vol. 8, 9. http://is.ifmo.ru/ (section “Articles”).
20. Shalyto A., Tukkel N. Tanks and Automata // BYTE/Russia. 2003. № 2. http://is.ifmo.ru/ (section “Articles”).
21. Ozerov A. Four Tankmen and the Computer // Magic of PC (Magia PK). 2002. № 11. http://is.ifmo.ru/ (section “About Us”).
22. Markov S., Shalyto A. Controlling System for Herbivorous Animal for “Terrarium” Game. http://is.ifmo.ru/ (section “Project”).
23. Cockburn A. Agile Software Development. NJ: Addison-Wesley, 2001.
24. Trofimov S. E-mail: info@caseclub.ru.
25. Knuth D. The Art of Computer Programming. MA: Addison-Wesley, 1998.
26. Kazakov M., Korneev G., Shalyto A. Using Finite Automata in Developing Logic of Algorithm Visualizers // Telecommunication and Informatization of Education. 2003, vol.6, http://is.ifmo.ru/ (section “Articles”)
27. Sommervill I. Software Engineering MA: Addison-Wesley, 2001.
28. Braude E.J. Software Engineering: An Object Oriented Perspective. NY: John Wiley&Sons, 2001.
29. Fox J. Software and its Development. Englewood Cliffs. Prentice-Hall, 1982.