# A LISP function that expands macros fully: macroexpand-all .

This posting will assume that the reader is sufficiently familiar with LISP, to understand how its macros generally work. What such readers will often want to do, is to be able to expand macros, mainly for debugging purposes, before they get compiled into a function. This is because of the fact that, with many LISP implementations, functions are generally compiled in such a way, given actual LISP that acted as source code, and which the macro partially built, that the functions cannot be decompiled, nor the original LISP code recovered from them.

What Common LISP offers, are the functions ‘MACROEXPAND’ and ‘MACROEXPAND-1′, the first of which depends on the second. But, As the Common LISP Hyperspec already points out, ‘MACROEXPAND’ will only keep calling ‘MACROEXPAND-1′, until the second output from this subservient function returns as Nil. This will happen, when the argument is no longer a Macro Form. But, as is already mentioned, this second value output by the subservient function will return as Nil, even if the LISP expression still contains Macro Sub-Forms. And the reason this behaviour can be problematic is the fact that most LISP compilers will keep expanding the Macro Sub-Forms as well, when a macro has been put into a function. This is actually, what makes recursive macros fully possible.

And so, as a better debugging tool, what some LISP programmers might need, is a function that expands all the Macro Sub-Forms as well (even before the macro has been used in a function definition). The following LISP Function accomplishes this:


(defun macroexpand-lists (input)
(cond ((null (listp input)) input)
((null input) NIL)
((listp (car input)) (cons (macroexpand-all (car input))
(macroexpand-lists (cdr input)) ) )
(T (cons (car input) (macroexpand-lists (cdr input)) ) ) ) )

(defun macroexpand-all (input)
(cond ((null (listp input)) input)
((null input) NIL)
(T
(let ((expansion1 (macroexpand input)))
(cond ((null (listp expansion1)) expansion1)
((eq (car expansion1) 'cl:function) expansion1)
((listp (car expansion1))
(macroexpand-lists expansion1) )
(T (cons (car expansion1)
(macroexpand-lists (cdr expansion1)) ) ) ) ) ) ) )



Now, I suppose that what the reader may want to see next could be, how this function of mine performs, if I feed it a LISP Macro, Which I defined in an earlier posting. And the following snip illustrates this. The following snip assumes that the Macro, its subservient Function, and the Macro-Expansion Function were already loaded into a LISP session, as that session was started…

(Updated 7/20/2020, 15h05,,, )

(As of 7/16/2020, 9h00: )


dirk@Phosphene:~/Documents/Phoenix/Lisp$gcl -load AndOrMacro3.txt GCL (GNU Common Lisp) 2.6.12 CLtL1 Fri Apr 22 15:51:11 UTC 2016 Source License: LGPL(gcl,gmp), GPL(unexec,bfd,xgcl) Binary License: GPL due to GPL'ed components: (XGCL READLINE UNEXEC) Modifications of this banner must retain notice of a compatible license Dedicated to the memory of W. Schelter Use (help) to get some basic information on how to use GCL. Temporary directory for compiler files: /tmp/ >(defvar a 1) A >(defun test () (&& T T :IGN (setq a 2) NIL T)) TEST >a 1 >(test) NIL >a 2 >(macroexpand-all '(&& T T :IGN (setq a 2) NIL T)) (IF T (IF T (PROGN (EVAL-LIST '(SETQ A 2)) (IF NIL (IF T T))))) >(quit) dirk@Phosphene:~/Documents/Phoenix/Lisp$



It would seem that in this case, both the macro which is being tested, and the diagnostic aid which I wrote, work as they should.

(Update 7/16/2020, 12h00: )

Even though the first version of this diagnostic tool was able to expand all the Macro Sub-Forms which it was supposed to, it had as unwanted side effect, also to expand lists that contain the atom identifying the macro, in a later position. If given such a case, the earliest version of the function would eventually parse the list, as though it had begun with the macro-atom, and apply that macro to the following elements of the same list.

The reason I made this error was the fact, that it also led to the smallest code, that could scan lists of arbitrary complexity, for Macro Sub-Forms. However, this is not the behaviour that the LISP Compiler would apply, when compiling the function that contains a macro. Analogously to how atoms are treated, that identify functions, but somewhere in the middle of the list, such atoms should be ignored.

It’s only if, given a list of lists, that my function should scan each of the sub-lists for valid Macro Forms, since the underlying function already found all the forms that occurred, due to the macro’s identifying atom being the first in a given list.

Further, since my function is meant as a diagnostic tool, it should also point out such a special case to its user / programmer, as misplaced macro-atoms that will not be expanded. For all I know, such a programmer could have missed that this happens.

Therefore, I have now edited my function, resulting in two actual functions that include a helper-function, to duplicate how the LISP Compiler will reject such erroneous use of atoms that identify macros. The sample session shown last, still performs as it should. And the following sample now shows, what the updated version of my function does correctly, thus not expanding a macro which it shouldn’t:


dirk@Phosphene:~/Documents/Phoenix/Lisp$gcl -load AndOrMacro3.txt GCL (GNU Common Lisp) 2.6.12 CLtL1 Fri Apr 22 15:51:11 UTC 2016 Source License: LGPL(gcl,gmp), GPL(unexec,bfd,xgcl) Binary License: GPL due to GPL'ed components: (XGCL READLINE UNEXEC) Modifications of this banner must retain notice of a compatible license Dedicated to the memory of W. Schelter Use (help) to get some basic information on how to use GCL. Temporary directory for compiler files: /tmp/ >(macroexpand-all '(&& T T :IGN (setq a 2) NIL T)) (IF T (IF T (PROGN (EVAL-LIST '(SETQ A 2)) (IF NIL (IF T T))))) >(macroexpand-all '(&& T T && :IGN (setq a 2) NIL T)) (IF T (IF T (IF && (PROGN (EVAL-LIST '(SETQ A 2)) (IF NIL (IF T T)))))) >(quit) dirk@Phosphene:~/Documents/Phoenix/Lisp$



(Update 7/17/2020, 17h15: )

Oddly, I found another bug in an earlier version of these functions. If they were told ‘to macro-expand’ any expression, which contained a valid ‘LAMBDA’ expression, an endless loop ensued. And, as far as I could tell, the loop was as such:

• ‘MACROEXPAND’ was called first on the ‘LAMBDA’ expression,
• This resulted in an expression, prefixed by the dispatch macro ( #' ),
• The value of such an expression would have been a function, but not the same expression quoted, so,
• The ‘CADR’ of this expression was the same ‘LAMBDA’ expression,

Now, an incompetent way to try to patch this vulnerability would have been, to evaluate the expression first, and then to test it with the ‘FUNCTIONP’ function. However, an unwanted side effect of that would have been, that virtually every expression returned, would also be evaluated, before any other work was done on it! And so, the safer thing to do under ‘GNU Common LISP’ was, to test the ‘CAR’ of the expression, for whether it was equal to the atom ‘FUNCTION’. In this LISP implementation, this identified the expressions with the offending dispatch macro. In such a case, the expressions are to be returned without being processed further.

The patched code is shown above.

Other implementations of LISP might use a different, internal representation, of such things as:


'#'(LAMBDA (X Y) (BLOCK ADD (+ X Y)))



If that is so, then the simple test for the atom used above will fail to trap those. Some other means will then need to be found, first, by evaluating the ‘CAR’ of the expression above, in the given LISP console, and then proceeding.

(Update 7/18/2020, 11h20: )

Another question which the reader might have about me could be, ‘How did Dirk ever get so confused, as to think that the first evaluation of a macro, that produces a list – its expansion – and the subsequent evaluation of that list, both take place at run-time, and not, as the fact is established, that the expansion of the macro takes place when the function that contains it is being compiled, while the evaluation of the resulting list, as part of that function-definition, takes place at run-time, i.e., when the function is finally called?’

And the short answer to that question would be, ‘Because Dirk never compiled the functions which he was experimenting with.’

The way LISP works is that functions can perfectly-well be defined, without ever being compiled, in which case the format in which they are stored is similar, to the list-of-dotted-pairs format, that was also used to define them. Only, the way LISP is implemented, the exact way in which those lists are associated with the atoms that identify the function, must lead to the eventual evaluation of the function, when called.

If that is the usage, then the macros are actually expanded, during the same operation that interprets the function that used them. It’s only when the function is to be compiled, that the expansion of the macro takes place at an earlier point in time.

Similarly, macros can themselves be called as though they were functions, in which case both evaluations take place during the same operation.

The snip below, which is being used to analyze the function ‘EVAL-LIST’, illustrates the concept. I defined ‘EVAL-LIST’ elsewhere, and it is not an integral part of LISP, nor defined in this posting, nor a compiled function:


dirk@Phosphene:~/Documents/Phoenix/Lisp$gcl -load AndOrMacro3.txt GCL (GNU Common Lisp) 2.6.12 CLtL1 Fri Apr 22 15:51:11 UTC 2016 Source License: LGPL(gcl,gmp), GPL(unexec,bfd,xgcl) Binary License: GPL due to GPL'ed components: (XGCL READLINE UNEXEC) Modifications of this banner must retain notice of a compatible license Dedicated to the memory of W. Schelter Use (help) to get some basic information on how to use GCL. Temporary directory for compiler files: /tmp/ >(macroexpand '(defun eval-list (input) (cond ((null input) NIL) ((null (listp input)) (list (eval input)) ) ((null (listp (car input))) (list (eval input)) ) (T (cons (eval (car input)) (eval-list (cdr input)) ) ) ) )) (PROGN (SETF (SYMBOL-FUNCTION 'EVAL-LIST) #'(LAMBDA (INPUT) (BLOCK EVAL-LIST (IF (NULL INPUT) NIL (IF (NULL (LISTP INPUT)) (LIST (EVAL INPUT)) (IF (NULL (LISTP (CAR INPUT))) (LIST (EVAL INPUT)) (CONS (EVAL (CAR INPUT)) (EVAL-LIST (CDR INPUT))))))))) 'EVAL-LIST) T >#'eval-list (SYSTEM:LAMBDA-BLOCK EVAL-LIST (INPUT) (COND ((NULL INPUT) NIL) ((NULL (LISTP INPUT)) (LIST (EVAL INPUT))) ((NULL (LISTP (CAR INPUT))) (LIST (EVAL INPUT))) (T (CONS (EVAL (CAR INPUT)) (EVAL-LIST (CDR INPUT)))))) >(macroexpand #'eval-list) (SYSTEM:LAMBDA-BLOCK EVAL-LIST (INPUT) (COND ((NULL INPUT) NIL) ((NULL (LISTP INPUT)) (LIST (EVAL INPUT))) ((NULL (LISTP (CAR INPUT))) (LIST (EVAL INPUT))) (T (CONS (EVAL (CAR INPUT)) (EVAL-LIST (CDR INPUT)))))) NIL >(macroexpand-all #'eval-list) (SYSTEM:LAMBDA-BLOCK EVAL-LIST (INPUT) (IF (NULL INPUT) NIL (IF (NULL (LISTP INPUT)) (LIST (EVAL INPUT)) (IF (NULL (LISTP (CAR INPUT))) (LIST (EVAL INPUT)) (CONS (EVAL (CAR INPUT)) (EVAL-LIST (CDR INPUT))))))) >(quit) dirk@Phosphene:~/Documents/Phoenix/Lisp$



What this recent snip also demonstrates is the fact, that, if the built-in function ‘MACROEXPAND’ encounters a ‘LAMBDA’ expression, it not only expands the macros recursively that form the first atom of the expression, but also the macros in all the sub-forms. This is what I initially defined my ‘MACROEXPAND-ALL’ function to do differently from the built-in function, in case a ‘LAMBDA’ expression is not being expanded. This latter observation also reveals that LISP itself has no additional functions to expand macros, and that, if the user / programmer wants to have macros expanded fully, outside a ‘LAMBDA’ expression, he will need a function, that is at least similar to the function I’ve defined in this posting.

Only, if the function is to be defined, without being compiled, then by default, ‘MACROEXPAND’ is never called on it at all!

(Update 7/18/2020, 12h05: )

Another question which this latter observation explains, is, why the function ‘MACROEXPAND-ALL’ did not need to be forward-declared, when the function ‘MACROEXPAND-LISTS’ was being defined. In most programming languages, such a forward-declaration would be needed, because ‘MACROEXPAND-LISTS’ is to be compiled. But in my code example above, it is not needed. I suppose that, if the reader wanted to compile either function, the way to do that in LISP would be, Still Not To Compile ‘MACROEXPAND-LISTS’, but, when the time comes, to compile ‘MACROEXPAND-ALL’.

(Update 7/20/2020, 15h05: )

I ran into another conceptual issue with the definitions that I published above. It has to do with how the LISP ‘EQ’ function works. In LISP, this function will not always return ‘T’, just because two atoms are being supplied, and both are printed the same way. In general, this function will return T only if both supplied parameters do in fact evaluate to the same atom, in the case of atoms, which means, that both supplied parameters must refer to the same address. There exist more generic situations in LISP, where a form is not being supplied to a LISP function as a parameter, and in that case, some register nevertheless refers to an address, either of an atom, or of the first dotted pair in a list, etc..

This discourages me, when defining functions that are going to be compiled, from quoting a new atom literally, and then testing it for equality to a supplied atom, using ‘EQ’. On the other hand, I tend to look at the approach of quoting atoms and lists, in order to generate new content, as much more favourable.

In LISP, it can happen that a name is scanned in from text, to an unexpected symbol, especially if a function is being compiled as part of a different package, from the package it is going to be used in. However, I did understand all along, that it would be unlikely for the user / programmer actually to shadow one of the symbols that belong to the package ‘:COMMON-LISP’.

What some readers might think could be, that the use of ‘EQL’ or ‘EQUAL’ will circumvent such hazards. But alas, ‘EQL’ only builds on what ‘EQ’ does, by creating exceptions for numbers and text characters, while ‘EQUAL’ builds on what ‘EQL’ does, by traversing lists, text strings, etc., and then testing individual atoms found, with ‘EQL’ again…

Even though this thought bothered me for some time, I found that the best way to avoid such issues, was to use the form ‘CL:FUNCTION’ instead of just ‘FUNCTION’. Actually using package semantics seems a better approach, than trying to bypass package semantics.

Done.

Dirk