Declarative Languages
Lecture #11

Purpose: Evaluation rules and macros

11.1 Recall the evaluation rules for lists

We stated early on (section 3.1) that, to evaluate a list:

• if it is a function call then evaluate each of its arguments, in order, and call the function with the results of these evaluations. The result of the evaluation is whatever value the function returns.
• if it is not a function call, we treat it specially, according to the idiosyncratic definition of that particular non-function.
We are now in a position to return to this subject and cover it more precisely, recalling that there are two types of "not a function": macros and special operators. They share the properties that:
• each of these operators has its own individual syntax and, in particular,
• they typically don't evaluate all their arguments
They differ in that:
• macros always "stand for" other code, so by macroexpanding a macro call (possibly repeatedly) you can always get to the underlying non macro form.
• special operators are implemented "under the hood" - there is no way to "expand" a call to a special operator. [Not quite true - an implementation is free to implement a special operator as a macro if it wants - it would be more accurate for me to state that there is no guarantee that any special operator will "expand". For example, none of the special operators in LispWorks for Windows "expand".]
• you can define your own macros but the set of special operators is fixed.
So why you should bother with macros?. The answer to this is that macros result in code which is more:
• compact
• flexible
• maintainable
• rapidly developed

11.2 Review of special operators

Let's start with the special operators. There are 25 of these in the language and there is no way to add more of your own. The function special-operator-p called with one of these symbols returns true:
(special-operator-p 'setq)  =>  true
(special-operator-p 'setf)  =>  nil

We have met the following 10 special operators so far:

block
return-from
catch
throw
function
quote
if
let*
progn
unwind-protect
The following 13 (of which the first 5 will be more useful to you over the years than the rest) are outside the scope of this course:
eval-when  load-time-value  locally  the  labels  flet  macrolet  symbol-macrolet  go  tagbody  progv  multiple-value-call  multiple-value-prog1
That leaves two more which you might get by without ever using, but for completeness I do need to mention them here briefly:
• setq

• This is equivalent to the macro setf, but can only be used for setting a value into a variable / symbol. So (setf foo some-value) can be rewritten (setq foo some-value). A historical relic which probably wouldn't be in the language at all if it had been designed from scratch to support the more general purpose setf. Emacs lisp has setq but does not support setf.

• let

• This is similar to the special operator let* and has the same syntax. The only difference between the two is that let* processes its binding pairs in order (sequentially) but let processes them simultaneously (in parallel). For example:
(let* ((foo 99)
(bar (1+ foo)))
bar)                    =>  100
but
(let ((foo 99)
(bar (1+ foo)))
bar)                    =>  error
because let attempts to evaluate all the values in the variable-value pairs, i.e. here the values 99 and (1+ foo), before it sets any of the results into either of the variables foo or bar. So (1+ foo) cannot be calculated, as foo isn't available yet.

I personally tend to use let in preference to let* when there's only one binding pair, but this is purely a matter of style and has absolutely no operational significance. When there are two or more variables being bound, you will almost always get away with let* (either because it doesn't matter in your case, or because - as above - let is wrong in your case).

To my mind, let and let* are badly named (c.f. setf and its parallel counterpart psetf).

In the course of the first ten lectures we have met a fair number of lisp's 91 macros:

and  decf  defconstant  defparameter  defstruct  defun  dolist  dotimes  ignore-errors  incf  lambda  loop  or  pop  prog1  prog2  push  pushnew  return  setf  trace  untrace  with-open-file
Some are easily defined (e.g. and, return), others are more complex in their behaviour (the defining macros, for instance), and several (push) look OK from a distance but turn out to be slightly more fiendish when you get close up. Today we will
• take a look at how some of the simpler macros expand
• introduce a few more
• see how to add more macros to the language
Let's start with a very simple example: the macro return. We recall that this has syntax
(return &optional what)
and that it is shorthand for / equivalent to invoking the special operator return-from:
(return what) ==  (return-from nil what)

How is this equivalence managed?

Lisp provides the following assistance:

• the function macro-function, which takes a symbol and returns true if there is a macro associated with that symbol.
• the function macroexpand-1, which takes an entire lisp form and runs the macro-expander associated with the car of that form, if possible. It returns either the expanded form, or the original if no expander was defined.
So:
(macro-function 'return)            =>  true
(macroexpand-1 '(return))           =>  (return-from nil)
(macroexpand-1 '(return (foo bar))  =>  (return-from nil (foo bar))
The way you define this behaviour is via yet another (defining) macro: defmacro. For instance:
(defmacro return (&optional what)
(list 'return-from nil what))
To see how this works in practice, suppose we are running the following code:
(let ((count 0))
(dolist (element list)
(if (eq element thing)
(return count))
(incf count)))
and that we have arrived at the return form. The lisp system determines (by calling special-operator-p) that return is not a special operator. It then determines (by calling macro-function) that return is a macro. The function macroexpand-1 is then called with the entire lisp form constituting the macro call as its argument:
(macroexpand-1 '(return count))
That function invokes the macro definition for return, which generates a list with three elements: the symbols return-from, nil and count. It has therefore effectively generated a new lisp form:
(return-from nil count)
and this is evaluated in place of the original form.

You don't have to think all this through every time though. You can just say: it's a macro, so the form will be replaced by whatever the macro function expands it into.

Macros transform lisp source into new code. To quote Graham:

You define a macro by saying what a call should expand into.
Underneath, macros are just functions that return expressions.

Macros are an incredibly flexible and powerful way to expand the language. They beat #define into a cocked hat. Effectively, macros are programs which write more programs. They permit code which is compact, readable, maintainable. And because macros use lisp syntax to transform lisp into more lisp, you don't have to "leave the language" to use them.

11.4 Conditionals: when, unless and cond

Another simple macro is when. This has syntax
(when test-form form-1 form-2 ... form-n)
which is equivalent to
(if test-form (progn form-1 form-2 ... form-n))
In other words, when evaluates its first form. If that is non-nil, it evaluates all its remaining forms as an implicit progn and returns the value of the final form, otherwise when simply returns nil.

So

(if (char= next #\;)
(progn
(setf previous-position position)
(incf count)))
can be rewritten
(when (char= next #\;)
(setf previous-position position)
(incf count))
We can code its expander thus:
(defmacro when (test-form &rest forms)
(list 'if
test-form
(cons 'progn forms)))
Note that this is getting a little difficult to read (and to code correctly!) - you have to stop and think about when to use list and when to use cons, and this is still a fairly simple example. The trick is to use the backquote notation (section 4.5):
(defmacro when (test-form &rest forms)
`(if ,test-form
(progn ,@forms)))
Macro writing now becomes a whole load easier, because you simply write out the form you're trying to generate and place
• a backquote before the whole thing
• a comma before each form to be evaluated in the macro expander (we didn't want the symbol test-form in the result - what we did want was the lisp code associated with this variable)
• comma-at before each form to be evaluated and spliced in to the surrounding list - typically you splice variables supplied to the macro by &rest and typically these are forms to be evaluated as as implicit progn.
Another example: the macro unless which is like when but only runs the forms if the test-form evaluated to nil. The following would both work as macro definitions for unless. Note that one of them defines unless in terms of another macro (when), so that a call involving unless would have to be macroexpanded twice.
(defmacro unless (test-form &rest forms)
`(if ,test-form
nil
(progn ,@forms)))

(defmacro unless (test-form &rest forms)
`(when (not ,test-form)
,@forms))

Finally in this section, the macro cond removes the need for vast nested chains of if statements. The syntax is:
(cond clause-1 clause-2 ... clause-n)
where each clause looks like this:
(test-form form-1 ... form-n)
If the test in clause-1 is true then all the subforms in that clause are evaluated (implicit progn) and the value of the last form returned. Otherwise the next clause is tried, and so on in order until either one of the tests succeeds or they all fail.

This definition suggests a recursive implementation:

(defmacro cond (&rest clauses)
(when clauses
(let* ((this (first clauses))
(others (rest clauses))
(test (first this))
(forms (rest this)))
`(if ,test
(progn ,@forms)
(cond ,@others)))))
but an iterative one will do too:
(defmacro cond (&rest clauses)
(let ((expansion nil))
(dolist (this (reverse clauses))
(let ((test (first this))
(forms (rest this)))
(setf expansion
`(if ,test
(progn ,@forms)
,expansion))))
expansion))
Using the first expansion,
(cond (foo (bar) baz)
(wombat wibble)
(t 99))
expands to
(if foo
(progn (bar) baz)
(cond (baz wibble)
(t 99)))
using the second we get
(if foo
(progn (bar) baz)
(if wombat
(progn wibble)
(if t
(progn 99)
nil)))
Assuming I haven't made any mistakes in this, the forms are equivalent and either macro definition will do.

As you can see, writing macros can very rapidly become a complex task. However, the results tend to be rewarding, because the language is so much more powerful for the presence of macros.

11.5 Further examples of macros

• case

•

This has a syntax somewhat similar to cond:
(case value clause-1 clause-2 ... clause-n)
where each clause looks like this:
(keys form-1 ... form-n)
Each of the keys forms is a list of objects (not evaluated!). If value is eq to any of the keys for clause-1 then that clause's subforms are evaluated in the usual way; otherwise we try the next clause. The final clause may have one of the symbols t or otherwise (it doesn't matter which) instead of a list of keys - that means invoke this clause anyway if all else failed. Example:

(defun decode (x)
(case x
((i uno) 1)
((ii dos) 2)
((iii tres) 3)
((iv cuatro) 4)))
(defun add-em (x) (reduce '+ (mapcar 'decode x)))

Related macros include ecase which signals an error if none of the keys are matched, and typecase (along with etypecase) which dispatches on the type of value:

(defun satisfy (requirement state)
(etypecase requirement
(symbol ...)
(string ...)
(cons ...)))
• shiftf and rotatef

•

These two macros are probably more fun than use, except that rotatef is dead handy for swapping values around.

(shiftf place-1 place-2 ... place-n new-value)
assigns the old value of place-2 into place-1, the old value of place-3 into place-2, and so on to the end where new-value is assigned to place-n. The old value of place-1 is returned.

(rotatef place-1 place-2 ... place-n)
assigns the old value of place-2 into place-1, the old value of place-3 into place-2, and so on to the end where the old value of place-1 is assigned to place-n. The return value is nil. In particular,
(rotatef place-1 place-2)
is equivalent to

(let ((temp (place-1)))
(setf place-1 place-2
place-2 temp)
nil)

•

This macro has syntax:
(print-unreadable-object (object stream &key type identity) forms) => nil
It outputs a printed representation of object on stream, beginning with "<#" and ending with ">". Everything output to stream by the body forms is enclosed in the the angle brackets. If type is true, the output from forms is preceded by a brief description of the object's type and a space character. If identity is true, the output from forms is followed by a space character and a representation of the object's identity, typically a storage address.

If either type or identity is not supplied, its value defaults to nil. It is valid to omit the body forms. If type and identity are both true and there are no body forms, only one space character separates the type and the identity.

In this example, the precise form of the output is implementation-dependent.

``` (defmethod print-object ((obj airplane) stream)
(print-unreadable-object (obj stream :type t :identity t)
(format stream "~a" (tail-number obj))))

(format nil "~a" my-airplane)
=>  "#<Airplane NW0773 36000123135>"
OR =>  "#<FAA:AIRPLANE NW0773 17>"```
11.6 Revisiting the function evaluation rules

So, to evaluate a non empty list (seeing as the empty list is identical to the self-evaluating symbol nil), we look at the first element of that list.

1. If it is a symbol which names one of the 25 special operators, carry out the appropriate behaviour for that special operator.
2. Otherwise, if it is a symbol which has a macro definition, expand that macro and restart the evaluation (recursive loop!)
3. Otherwise, we must have a function. This may be either a symbol which is fboundp, or a list of the form
• (lambda (arg1 arg2 ...) ....)
We evaluate each argument in turn (from "left to right") and call the function with the results of these evaluations. The result of the evaluation is whatever value the function returns. Example of this rather arcane use of lambda:
((lambda (key value)
(setf (gethash key *table*) value))
(first pair)
(second pair))
but note that we could have done the same thing more legibly either by inserting a funcall
(funcall (lambda (key value)
(setf (gethash key *table*) value))
(first pair)
(second pair))
or [much better] by transforming the expression to use let*
(let* ((key (first pair))
(value (second pair)))
(setf (gethash key *table*) value))
and so you rarely see lambda used like this.

11.7 Practical session / Suggested activity

• Recall the macros push and pop:
• (setf words '(opposite of push))  =>  (opposite of push)
(setf same-words words)           =>  (opposite of push)
(push 'the words)                 =>  (the opposite of push)
words                             =>  (the opposite of push)
same-words                        =>  (opposite of push)
(pop same-words)                  =>  opposite
words                             =>  (the opposite of push)
same-words                        =>  (of push)
and implement them for yourself.

• Seeing as we're stuck with let in the language, use it to implement my-let*.

•
• Pick one or more of the macros discussed so far and attempt to implement them. Some are harder than others:
•  Easiest and  prog1  prog2 A little harder case  decf  dolist (in terms of loop)  dotimes (ditto)  incf  or  pushnew  rotatef  shiftf  trace (use the simplified syntax introduced in section 5.2)  untrace  with-open-file Why are these ones indvisable? defconstant  defparameter defstruct  defun  ignore-errors  lambda  loop  setf
• (Very difficult. Prize for first correct answer.) Evaluate the following forms in a lisp listener:
• (lambda (x) nil)
• (function-lambda-expression (lambda (x) nil))
and then consider the following cute puzzle among lispers: to come up with a self-reproducing consp form. A self-reproducing form is defined for the purposes of this question as one that evaluates to something equal to the original form. In other words, you need to come up with a form x such that
(and (consp x)
(equal x (eval x)))
is true.
It is considered cheating if your solution involves any permanent side effects. For example, we could trivially solve the problem by first defining a function foo, say, that returns (foo) as its value. But we disqualify this solution because defining a function counts as a permanent side effect. As you might no doubt suspect, the best known solution involves the use of lambda.
The solution that I am aware of is two lines of code. I would be curious to know if there are longer solutions (a family of them perhaps). Anybody with time on their hands to prove their research abilities is welcome to impress me.
• Graham chapter 10. Skip exercise 4 if you're not happy with the macro do (I never voluntaily use it, so why should you have to?).
• Answer no more than one of the following questions:
• Why should nobody voluntarily want to use do?
• What is it about the macro do which means you don't have to know how any of the other iteration macros work? Is this true?
• Write a macro my-if which translates calls of the form (my-if a then b) or (my-if a then b else c) into standard if forms. Go back through your work, find some function you've written using if and rewrite it to use my-if instead.
• Read the specification of print-unreadable-object above and then see if you can determine (before trying it out) what the following would print: