Long Docstrings

[Lawrence D'Oliveiro]

I often write long docstrings (not sure whether Lisp has its own name
for these). In Python, if the body of a class or function begins with
a string literal, then this becomes the docstring. And a REPL will
display this as help text if you type “help(«object»)”. E.g.

class Matrix :
"representation of a 3-by-2 affine homogeneous matrix. This does not" \
" actually use any Cairo routines to implement its calculations; these" \
" are done entirely using Python numerics. The from_cairo and to_cairo" \
" methods provide conversion to/from cairo_matrix_t structs. Routines" \
" elsewhere expect this Matrix type where the underlying Cairo routine" \
" wants a cairo_matrix_t, and return this type where the Cairo routine" \
" returns a cairo_matrix_t."

...

#end Matrix

This takes advantage of the fact that, like C and some C-derivatives,
Python supports implicit concatenation of adjacent string literals.
This allows me to construct very long string literals without messing
up the layout of my code.

Lisp doesn’t have such an implicit-concatenation convention. But it
has macros. And it easy enough to define a macro which does string
concatenation at macro-invocation time, e.g.

(defmacro mstr (&rest strs)
"lets me define a long string literal in pieces across lines, useful for docstrings."
(apply 'concat strs)
) ; defmacro mstr

And using this macro is equally easy:

(defun cur-line (ensure-newline)
(mstr
"returns list of two character positions, representing the"
" beginning and end of the selection if there is one, else"
" the beginning and end of the current line. ensure-newline"
" => ensures there is a newline at the end of the line."
)
...
) ; defun

[Madhu]

Post by Lawrence D'Oliveiro
I often write long docstrings (not sure whether Lisp has its own name
for these). In Python, if the body of a class or function begins with
a string literal, then this becomes the docstring. And a REPL will
display this as help text if you type “help(«object»)”. E.g.
"representation of a 3-by-2 affine homogeneous matrix. This does not" \
" actually use any Cairo routines to implement its calculations; these" \
" are done entirely using Python numerics. The from_cairo and to_cairo" \
" methods provide conversion to/from cairo_matrix_t structs. Routines" \
" elsewhere expect this Matrix type where the underlying Cairo routine" \
" wants a cairo_matrix_t, and return this type where the Cairo routine" \
" returns a cairo_matrix_t."
...
#end Matrix
This takes advantage of the fact that, like C and some C-derivatives,
Python supports implicit concatenation of adjacent string literals.
This allows me to construct very long string literals without messing
up the layout of my code.

Why do you want to split strings?

in python can't you just put the documentation in a string with embedded
newlines?
the length of each line will be under screenwidth.

""" representation of a 3-by-2 affine homogeneous matrix. This does not
actually use any Cairo routines to implement its calculations;
these[...] """

Post by Lawrence D'Oliveiro
Lisp doesn’t have such an implicit-concatenation convention. But it
has macros. And it easy enough to define a macro which does string
concatenation at macro-invocation time, e.g.

lisp strings (the ~ quoting convention) embed newlines by default.

Post by Lawrence D'Oliveiro
(defmacro mstr (&rest strs)
"lets me define a long string literal in pieces across lines, useful for docstrings."
(apply 'concat strs)
) ; defmacro mstr
(defun cur-line (ensure-newline)
(mstr
"returns list of two character positions, representing the"
" beginning and end of the selection if there is one, else"
" the beginning and end of the current line. ensure-newline"
" => ensures there is a newline at the end of the line."
)
...
) ; defun

You could just have said

(defun cur-line (ensure-newline)
"returns list of two character positions, representing the
beginning and end of the selection if there is one, else
the beginning and end of the current line. ensure-newline
ensures there is a newline at the end of the line."
)

[Lawrence D'Oliveiro]

Post by Madhu
in python can't you just put the documentation in a string with embedded
newlines?

You mean triple-quoted strings? And having to add extra blanks at the
start of each line, throwing the formatting off?

Post by Madhu
You could just have said

Which is why I said “without messing up the layout of my code.”

[Madhu]

Post by Lawrence D'Oliveiro

Post by Madhu
in python can't you just put the documentation in a string with embedded
newlines?

You mean triple-quoted strings? And having to add extra blanks at the
start of each line, throwing the formatting off?

Post by Madhu
You could just have said

Which is why I said “without messing up the layout of my code.”

Sorry I misunderstood, I thought you wanted (documentation function) to
yield formatted text and not a single line.

What Kaz said applies for emacs lisp too (`concat' gives it away) -- the
docstring has to be a string.

try add a declaration "(declare (speed 0))" for elisp
or "(declare (optimize (speed 0)))" after the docstring to see the
problem.

elisp will both complain
- Warning: docstring wider than 80
- diagnose a Warning: Stray ‘declare’ form: (declare (speed 0))

common lisp should blow up on compilation.
```
While compiling CUR-LINE :
The DECLARE expression (DECLARE
(OPTIMIZE
(SPEED 0))) is being treated as a form,
possibly because it's the result of macroexpansion. DECLARE expressions
can only appear in specified contexts and must be actual subexpressions
of the containing forms.
```

But you can get around it by using #.(mstr .... ) for the docstring
and evaluate it at read time.

[Lawrence D'Oliveiro]

elisp will both complain ...

I have used this successfully in elisp.

[Kaz Kylheku]

Post by Lawrence D'Oliveiro
I often write long docstrings (not sure whether Lisp has its own name
for these).

Lis has its own name: docstrings. It is Python that doesn't have its own name.

Post by Lawrence D'Oliveiro
(defmacro mstr (&rest strs)
"lets me define a long string literal in pieces across lines, useful for docstrings."
(apply 'concat strs)

What is concat?

Post by Lawrence D'Oliveiro
) ; defmacro mstr

Another solution would be to make the macro take a format argument.
Common Lisp has a mechanism inside format strings for removing a newline
and eating the whitespace which follows: tilde before a newline.

(format nil "this is a ~
multi-line ~
print job that ~
becomes one line.")

But ...

Post by Lawrence D'Oliveiro
(defun cur-line (ensure-newline)
(mstr
"returns list of two character positions, representing the"
" beginning and end of the selection if there is one, else"
" the beginning and end of the current line. ensure-newline"
" => ensures there is a newline at the end of the line."
)

I am concerned whether this is actually valid syntax for specifying
a docstring.

Note that according to ANSI CL, the syntax of defun is:

defun function-name lambda-list [[declaration* | documentation]] form*

You see how "declaration" and "form" are distinct? Further, the
specification says:

documentation---a string; not evaluated.

It's not evaluated, because it's not a form. The documentation
must be a string literal element in the defun syntax.

Your (mstr ...) therefore doesn't correspond to the "documentation"
element, it is a form.

Nowhere does the spec say that the forms enclosed in defun are
macroexpanded in order to look for docstrings.

It's indeed not working for me in CLISP.

If I define this:

(defun fn () "abc" 42)

the function has "abc" documentation. If I use this:

(defun fn () (mstr "abc") 42)

then it has NIL documentation.

[1]> (defmacro mstr () "abc")
MSTR
[2]> (mstr)
"abc"
[3]> (defun fn () (mstr) 42)
FN
[4]> (documentation 'fn 'function)
NIL
[5]> (defun fn () "abc" 42)
FN
[6]> (documentation 'fn 'function)
"abc"

You need to make your own defun-doc macro to make this work, or else
(yuck) rely on read-time #. evaluation.

;; using your concat

(defmacro defun-doc (name args docstring-list &body body)
`(defun ,name ,args ,(apply #'concat docstring-list) ,@body))

This macro sidesteps the problem by generating a defun which
a string object in the docstring position. defun nevers sees
anything but a string object there.

I would make the macro smart so that it checks for the missing
docstring-list, without which the first argument of the body will
be taken for that position. Like checking that docstring-list is
a proper list, all of whose elements are strings.

[David De La Harpe Golden]

Post by Kaz Kylheku

Post by Lawrence D'Oliveiro
I often write long docstrings (not sure whether Lisp has its own name
for these).

Lis has its own name: docstrings. It is Python that doesn't have its own name.

They're certainly normally called docstrings or documentation strings in
python. I expect python got the idea from lisp in the first place.

peps.python.org/pep-0257/#what-is-a-docstring
peps.python.org/pep-0008/#documentation-strings

Normal (and stated in the relevant PEPs!) python convention IS to use
python's triple-double-quoted """multiline-string""" literals for them
uniformly, though.

It's not idiomatic to use python's single-line-restricted
"double-quoted string" literals for them. Not made syntactically
illegal, but odd.

Common python documentation processors using python docstrings as input
do just expect the bit of excess whitespace that ends up embedded in the
docstrings, and generally just trim it away. There's, like, functions in
the stdlib for it

docs.python.org/3/library/inspect.html#inspect.cleandoc

It's also very common to write python docstrings in
Sphinx/reStructuredText syntax specifically (has the advantage of still
remaining fairly readable when viewed as plaintext) on the assumption
they'll also be pulled into the project's generated Sphinx documentation.

www.sphinx-doc.org/en/master/usage/quickstart.html#autodoc

Anyway.

[Lawrence D'Oliveiro]

It's not idiomatic to use python's single-line-restricted "double-quoted
string" literals for them. Not made syntactically illegal, but odd.

I have no idea why people design a language one way, then expect you to
use it a different way.

Common python documentation processors using python docstrings as input
do just expect the bit of excess whitespace that ends up embedded in the
docstrings, and generally just trim it away. There's, like, functions in
the stdlib for it

I wonder why you need to use them ...

[Kaz Kylheku]

Post by Lawrence D'Oliveiro

It's not idiomatic to use python's single-line-restricted "double-quoted
string" literals for them. Not made syntactically illegal, but odd.

I have no idea why people design a language one way, then expect you to
use it a different way.

Common python documentation processors using python docstrings as input
do just expect the bit of excess whitespace that ends up embedded in the
docstrings, and generally just trim it away. There's, like, functions in
the stdlib for it

I wonder why you need to use them ...

And why doesn't the Python compiler use them to sanitize the docstring
when the function definition is processed, to spare the tooling from
having to do it.

[David De La Harpe Golden]

Post by Kaz Kylheku

Post by Lawrence D'Oliveiro
I wonder why you need to use them ...

And why doesn't the Python compiler use them to sanitize the docstring
when the function definition is processed, to spare the tooling from
having to do it.

Well, sounds like they may well be about to do something like that for
Python 3.13 shortly (at least for compiled files) - pre-release notes -

docs.python.org/3.13/whatsnew/3.13.html

Post by Kaz Kylheku
Compiler now strip indents from docstrings. This will reduce the size

of bytecode cache (e.g. .pyc file). For example, cache file size for
sqlalchemy.orm.session in SQLAlchemy 2.0 is reduced by about 5%. This
change will affect tools using docstrings, like doctest.

Not sure about backward compat though, seems like it would necessarily
be quite a breaking change (seeing as docstrings have in fact preserved
line-leading whitespace all along, if only to be later typically but not
necessarily stripped)

Was debate ongoing 2 weeks ago whether they roll back on the change or not:

github.com/python/cpython/issues/81283#issuecomment-1878628233


Shrug. Not my circus not my monkeys etc.

[Kaz Kylheku]

Post by David De La Harpe Golden

Post by Kaz Kylheku

Post by Lawrence D'Oliveiro
I wonder why you need to use them ...

And why doesn't the Python compiler use them to sanitize the docstring
when the function definition is processed, to spare the tooling from
having to do it.

Well, sounds like they may well be about to do something like that for
Python 3.13 shortly (at least for compiled files) - pre-release notes -
docs.python.org/3.13/whatsnew/3.13.html

Post by Kaz Kylheku
Compiler now strip indents from docstrings. This will reduce the size

of bytecode cache (e.g. .pyc file). For example, cache file size for
sqlalchemy.orm.session in SQLAlchemy 2.0 is reduced by about 5%. This
change will affect tools using docstrings, like doctest.

Under my watch, this obvious thing would have been done 20 years ago.

(Well, if I didn't hate docstrings.)

The Python people are slow on the uptake.

Post by David De La Harpe Golden
Not sure about backward compat though, seems like it would necessarily
be quite a breaking change (seeing as docstrings have in fact preserved
line-leading whitespace all along, if only to be later typically but not
necessarily stripped)
github.com/python/cpython/issues/81283#issuecomment-1878628233

Undoubtedly, some of these nincompoops have managed to stow away
textual Python source inside docstrings, whose semantics is destroyed
by the stripping.