HEΛP is Coming for your Common Lisp Documentation Needs

HEΛP ("HELP") is yet another CL code documentation utility, which has been inspired by javadoc and doxygen.

HEΛP produces, by default, a (mostly) XHTML or HTML5 set of documents. The system strives to be simple to use by making a few choices, while maintaining a number of hooks that can be used to customize the system, e.g., by producing a different kind of output (e.g., in theory Texinfo).

The simplest way to get HEΛP (pun intended) is to use the function document, as in the example below, where a few assumptions are made.

We are in the directory where the code to be documented resides.
The actual documentation will end up in the directory (using POSIX syntax and conventions)
./docs/html/

  cl-prompt> (document *the-current-directory* :documentation-title "My Library" :format :html5)
  ;;; Several messages, some warnings and error messages.
  ;;; More messages, more warnings and error messages (these last ones ignored.).
  ;;; Yadda, yadda, yadda....
  NIL

As stated, the previous command will create the documentation (with an index.htm file in the ./docs/html/ folder. Alas, note that the *the-current-directory* must be obtained in an implementation-dependent way. Note also that the default format for the documentation is (mostly) HTML5.

The documentation thus generated is incomplete. Some HTML files are either skeletons or are not there (and your browser may complain accordingly). It is up to you to fill these files with appropriate content: a prominent example is the file named introduction.html which appears on the first page. It usually appears with a simple message telling you to edit said file.

A Few More Details

HEΛP rests on the following pillars: documentation strings and parameterized output formats. Also, there are a few assumptions that HEΛP makes about how it is run and how it produces documentation. There are also a few dependencies that must be taken into account for its installation and operation.

Documentation Strings

HEΛP reads and manipulates the documentantion strings contained in the source code. HEΛP can also read in the documentation strings from a simple-minded data-base of what are internally called doc-bits; as such data-base is a file directly readable by a Common Lisp, the actual documentation can be edited and built by manipulating the text contained therein.

How Documentation Strings Are Rendered

Each documentation string is rendered according to the resulting output format, most notably, (X)HTML(5). However, HEΛP strives to be reasonably "intelligent" and unobtrusive in its doc-string manipulation.

As previously stated, each doc-string is internally packaged up in a doc-bit, which is subclassed for each kind of "documentable" item in a code source. There are function doc-bits and there are, e.g., class doc-bits. Each doc-string is treated accordingly and its rendering tries to adhere, as much as possible, to the standard set by the Hyperspec. This means that the rendering of the doc-string is, at a minimum, put under a Descritpion: heading.

As per the Hyperspec, each doc-bit will contain, at a minumum, a name section, a Syntax: section, a Arguments and Values: section and a Description: section. Other sections appear depending on the doc-bit type. The Syntax: section is always automatically generated from the doc-bit associated signature (e.g., a function signature); the Arguments and Values: section may be automatically generated if it does not appear in the doc-string. The automatic generation of these sections tries to be as straighforward as possible. The system tries to take advantage of a few synctatic clues that can be present in each doc-string.

Documentation String Parsing

HEΛP tries to parse each doc string looking for standard Hyperspec formatting clues. This means that a doc-string is searched and deconstructed by looking for the following strings, appearing on isolated lines: Syntax:, Arguments and Values:, Description:, Examples:, See Also:, and Notes: (the list is not exaustive). If any of these "sections" is non empty, then it is rendered accordingly. Otherwise, everything is put under the Description: header.

The only contraints on the syntax are on the Arguments and Values: and Examples: sections. The first one must consist of one-liners with the parameter name separated from its description either by a "---" or by a ":". The second one should not contain anything that is not "code", i.e., actual code or listener interactions.

Consider the following example for a doc-string (the document function doc-string).

  "Produces the documentation for something.

The function is a wrapper for BUILD-DOCUMENTATION defaulting a few
parameters, in particular the output FORMAT (which defaults to HTML).

Arguments and Values:

FOR-WHAT --- what to document; can be a pathname or a 'system' (*).
DOCUMENTATION-TITLE --- a STRING which will appear as the documentation title.
FORMAT --- a SYMBOL designating the desired kind of output.
DESTINATION --- a (directory) PATHANME where the documentation will be produced.
ONLY-DOCUMENTED --- a BOOLEAN: whether to produce documentation only for documented items.
ONLY-EXPORTED --- a BOOLEAN: whether to produce documentation only for 'exported' items.
EVERYTHING --- a BOOLEAN: whether to produce documentation for everython, no matter what.
EXCLUDE-DIRECTORIES --- a LIST of directory pathnames not to be considered.
EXCLUDE-FILES --- a list of FILES not to be considered.
CLEAR-DOCUMENTATION-DB --- a KEYWORD stating if and when the documentation db should be cleared.

Notes:

(*) At the time of this ASDF and MK-DEFSYSTEM  are supported.

The arguments SOURCE and SUPERSEDE are, at the time of this writing,
effectively ignored.
"

The above is broken down into three sections and rendered as follows; note how the system adds the rendering of the arguments' list and other bits and pieces of information.

Function document

Package:

IT.UNIMIB.DISCO.MA.CL.HELambdaP

Syntax:

  document
        FOR-WHAT
        &KEY
        DOCUMENTATION-TITLE
        FORMAT
        LAYOUT
        SOURCE
        DESTINATION
        SUPERSEDE
        ONLY-DOCUMENTED
        ONLY-EXPORTED
        EVERYTHING
        EXCLUDE-DIRECTORIES
        EXCLUDE-FILES
        SPECIAL-METHODS-DEFS-FILES
        CLEAR-DOCUMENTATION-DB
        DEBUG-LEVEL
        INCLUDE-CL-USER-DEFINITIONS
        &ALLOW-OTHER-KEYS

    → result

Arguments and Values:

FOR-WHAT : what to document; can be a pathname or a 'system'.
DOCUMENTATION-TITLE : a STRING which will appear as the documentation title.
FORMAT : a SYMBOL designating the desired kind of output.
DESTINATION : a (directory) PATHANME where the documentation will be produced.
ONLY-DOCUMENTED : a BOOLEAN: whether to produce documentation only for documented items.
ONLY-EXPORTED : a BOOLEAN: whether to produce documentation only for 'exported' items.
EVERYTHING : a BOOLEAN: whether to produce documentation for everything, no matter what.
EXCLUDE-DIRECTORIES : a LIST of directory pathnames not to be considered.
EXCLUDE-FILES : a list of FILES not to be considered.
SPECIAL-METHODS-DEFS-FILES : a list of FILES to be LOADed before running the parsers.
CLEAR-DOCUMENTATION-DB : a KEYWORD stating if and when the documentation db should be cleared.
DEBUG-LEVEL : a number (between 0 and 42) or a string triggering more or less verbosity.

Description:

Produces the documentation for something.

The function is a wrapper for BUILD-DOCUMENTATION defaulting a few parameters, in particular the output FORMAT. The current default for FORMAT is :HTML, and experimental :HTML5 is available and it can be used to produce the documentation in HTML5 format.

The documentation, a set of HTML files and ancillary ones will be written in DESTINATION. What is saved as documentation is controlled by the arguments EVERYTHING, ONLY-DOCUMENTED and ONLY-EXPORTED. If EVERYTHING is non NIL, documentation pages for everything will be generated; otherwise non documented symbols first and non exported symbols next will be removed from the list of pages to be generated depending on the boolean values of ONLY-DOCUMENTED and ONLY-EXPORTED.

Notes:

At the time of this writing, ASDF and MK-DEFSYSTEM are supported.

The arguments SOURCE and SUPERSEDE are, at the time of this writing, effectively ignored.

The argument SPECIAL-METHODS-DEFS-FILES is a list of Common Lisp files that will be loaded before running the documentation parsers; it is assumed that these files will contain mostly DEFINE-DOCUMENTATION-EXTRACTOR and EXTRACT-NAMED-FORM-DOCUMENTATION definitions. The loading of these files is wrapped in an IGNORE-ERRORS form: failure to load one of them will not completely hamper the documentation procedure.

Note that EVERYTHING, ONLY-DOCUMENTED and ONLY-EXPORTED equal to NIL are equivalent to EVERYTHING non null.

If INCLUDE-CL-USER-DEFINITIONS is true (default is NIL), then documentation for doc bits defined in the CL-USER package will be included in the documentation.

Parameterized Output Formats

The HEΛP system uses CLOS dispatching to implement the production of different output formats. The functions involved are build-documentation and produce-documentation; they both take a FORMAT argument that designates the kind of output that must be produced.

At the time of this writing (May 2026), the currently supported values (May 2026) for the FORMAT parameter are

:hmtl, and
:html5

Future versions of HEΛP may provide, possibly a :texinfo output.

Format `:HTML`

The symbol :html, selects a mostly XHTML compliant output organized in a FRAMESET-based layout.

The decision to use FRAMESETs was taken because of the actual simplicity of the design, despite all the deprecation of the feature that can be found online and the fact that frames have been dropped by HTML5.

Format `:HTML5`

The symbol :html5, selects a mostly HMTL5 compliant output, which relies on the W3.CSS css styling framework.

Other Dependencies and Assumptions

The HEΛP system has a few dependencies and a few assumptions that must be taken into account.

HEΛP uses READ to process the code. This means that there are a number of dependencies on the actual runtime "package" state. The systems tries to be smart about creating packages as it encouters IN-PACKAGE and DEFPACKAGE forms, but the result may leave the runtime in a sub-optimal state.

HEΛP is sensitive to the "current directory". The system always tries to run in the current directory. The user is responsible to check this condition and the location of the output (which can be passed to the relevant functions).

HEΛP uses its own (X)HTML output system: XHTMΛ. The XHTMΛ library was constructed in order to deal with some of the author's idiosyncrasies. The system is still (after a lot of time) "under development" (this blog post is the only available public notice about it) and so there may be some unanticipated problems with the documentation system (X)HTML-based output.

Known Problems and Limitations

There are a few problems and limitations that can easily be avoided while using HEΛP.

`READ`ing and packages.

HEΛP can work on not-yet-loaded code; i.e., it can READ files that have not been seen by the CL environment yet. To achieve this feat, HEΛP makes several educated guesses when it encounters IN-PACKAGE and DEFPACKAGE forms by creating packages on the fly, while trying to handle nicknames.

While the above behavior is useful, it may fail in unexpected ways. To avoid such failures you can do two things.

Use DOCUMENT on systems. Hopefully the commands used will order the files with the "package" definition at the beginning. Note that at the time of this writing, ASDF has changed its internals and it may or may not work. MK:DEFSYSTEM works as advertised.
Call DOCUMENT on a code base that has been fully "loaded" in the CL environment; all packages should be availble and HEΛP should be able to READ all forms with no problems.

In the future, HEΛP may rely on Eclector to better handle package errors.

Exclude Files and Directories As Needed

As stated above, HEΛP uses READ to process code. This may cause some implementations (most notably, CCL) to bail out in unexpected ways, causing errors on some of the internal generic functions that may look strange. If that happens, you should first try to avoid processing that file using either the :exclude-files, the :exclude-file-types, or :exclude-directories) argument to DOCUMENT. Most of the times, the culprit file either has some syntax error or some unknown package qualifier deep into some SEXP.

CR/LF versus LF line ending

You may see another strange behavior when trying to process a Windows file on a UN*X platform and viceversa; that is, files with CR/LF versus LF for end of line. Different implementations may do different things in these cases and this may have an adverse effect on the parsing of documentation strings. You are advised to ensure that you are processing a file that is actually in the proper format for the platform you are running on.

Finally, the HTML5 documentation production is still a bit brittle.

Disclaimer

The code associated to these documents is not completely tested and it is bound to contain errors and omissions. This documentation may contain errors and omissions as well. Moreover, some design choices are not recognized as sub-optimal and may change in the future.

License

The file COPYING that accompanies the libray contains a MIT license. You are advised to use the code at your own risk. No warranty whatsoever is provided, the author will not be held responsible for any effect generated by your use of the library, and you can put here the scariest extra disclaimer you can think of.

Repository and Downloads

HEΛP is hosted at helambdap.sourceforge.net where its git repository resides.

The read only access to the git repository is available at either:

git://git.code.sf.net/p/helambdap/code, or
https://git.code.sf.net/p/helambdap/code

Just copy and paste these either link with to git clone.

The backend Sourceforge project is at http://sourceforge.net/projects/helambdap.