Purpose: Evaluation rules and macros

11.1 Recall the evaluation rules for lists

We stated early on (section 3.1) that, to evaluate a list:

• if it is a function call then evaluate each of its arguments, in order, and call the function with the results of these evaluations. The result of the evaluation is whatever value the function returns.
• if it is not a function call, we treat it specially, according to the idiosyncratic definition of that particular non-function.

• each of these operators has its own individual syntax and, in particular,
• they typically don't evaluate all their arguments

• macros always "stand for" other code, so by macroexpanding a macro call (possibly repeatedly) you can always get to the underlying non macro form.
• special operators are implemented "under the hood" - there is no way to "expand" a call to a special operator. [Not quite true - an implementation is free to implement a special operator as a macro if it wants - it would be more accurate for me to state that there is no guarantee that any special operator will "expand". For example, none of the special operators in LispWorks for Windows "expand".]
• you can define your own macros but the set of special operators is fixed.

• compact
• flexible
• natural / readable
• maintainable
• rapidly developed

11.2 Review of special operators

Let's start with the special operators. There are 25 of these in the language and there is no way to add more of your own. The function special-operator-p called with one of these symbols returns true:
    (special-operator-p 'setq)  =>  true
    (special-operator-p 'setf)  =>  nil

We have met the following 10 special operators so far:

block
return-from
catch
throw
function
quote
if
let*
progn
unwind-protect

eval-when  load-time-value  locally  the  labels  flet  macrolet  symbol-macrolet  go  tagbody  progv  multiple-value-call  multiple-value-prog1

• setq

This is equivalent to the macro setf, but can only be used for setting a value into a variable / symbol. So (setf foo some-value) can be rewritten (setq foo some-value). A historical relic which probably wouldn't be in the language at all if it had been designed from scratch to support the more general purpose setf. Emacs lisp has setq but does not support setf.
 
• let

This is similar to the special operator let* and has the same syntax. The only difference between the two is that let* processes its binding pairs in order (sequentially) but let processes them simultaneously (in parallel). For example:
    (let* ((foo 99)
           (bar (1+ foo)))
       bar)                    =>  100
but
    (let ((foo 99)
          (bar (1+ foo)))
       bar)                    =>  error
because let attempts to evaluate all the values in the variable-value pairs, i.e. here the values 99 and (1+ foo), before it sets any of the results into either of the variables foo or bar. So (1+ foo) cannot be calculated, as foo isn't available yet.

I personally tend to use let in preference to let* when there's only one binding pair, but this is purely a matter of style and has absolutely no operational significance. When there are two or more variables being bound, you will almost always get away with let* (either because it doesn't matter in your case, or because - as above - let is wrong in your case).

To my mind, let and let* are badly named (c.f. setf and its parallel counterpart psetf).

In the course of the first ten lectures we have met a fair number of lisp's 91 macros:

and  decf  defconstant  defparameter  defstruct  defun  dolist  dotimes  ignore-errors  incf  lambda  loop  or  pop  prog1  prog2  push  pushnew  return  setf  trace  untrace  with-open-file

• take a look at how some of the simpler macros expand
• introduce a few more
• see how to add more macros to the language

How is this equivalence managed?

Lisp provides the following assistance:

• the function macro-function, which takes a symbol and returns true if there is a macro associated with that symbol.
• the function macroexpand-1, which takes an entire lisp form and runs the macro-expander associated with the car of that form, if possible. It returns either the expanded form, or the original if no expander was defined.

(macro-function 'return)            =>  true
(macroexpand-1 '(return))           =>  (return-from nil)
(macroexpand-1 '(return (foo bar))  =>  (return-from nil (foo bar))

(defmacro return (&optional what)
  (list 'return-from nil what))

(let ((count 0))
  (dolist (element list)
    (if (eq element thing)
        (return count))
    (incf count)))

You don't have to think all this through every time though. You can just say: it's a macro, so the form will be replaced by whatever the macro function expands it into.

Macros transform lisp source into new code. To quote Graham:

Macros are an incredibly flexible and powerful way to expand the language. They beat #define into a cocked hat. Effectively, macros are programs which write more programs. They permit code which is compact, readable, maintainable. And because macros use lisp syntax to transform lisp into more lisp, you don't have to "leave the language" to use them.

11.4 Conditionals: when, unless and cond

Another simple macro is when. This has syntax
    (when test-form form-1 form-2 ... form-n)
which is equivalent to
    (if test-form (progn form-1 form-2 ... form-n))
In other words, when evaluates its first form. If that is non-nil, it evaluates all its remaining forms as an implicit progn and returns the value of the final form, otherwise when simply returns nil.

So

(if (char= next #\;)
    (progn
      (setf previous-position position)
      (incf count)))

(when (char= next #\;)
  (setf previous-position position)
  (incf count))

(defmacro when (test-form &rest forms)
  (list 'if
        test-form
        (cons 'progn forms)))

(defmacro when (test-form &rest forms)
  `(if ,test-form
       (progn ,@forms)))

• a backquote before the whole thing
• a comma before each form to be evaluated in the macro expander (we didn't want the symbol test-form in the result - what we did want was the lisp code associated with this variable)
• comma-at before each form to be evaluated and spliced in to the surrounding list - typically you splice variables supplied to the macro by &rest and typically these are forms to be evaluated as as implicit progn.

(defmacro unless (test-form &rest forms)
  `(if ,test-form
       nil
     (progn ,@forms)))

(defmacro unless (test-form &rest forms)
  `(when (not ,test-form)
     ,@forms))

This definition suggests a recursive implementation:

(defmacro cond (&rest clauses)
  (when clauses
    (let* ((this (first clauses))
           (others (rest clauses))
           (test (first this))
           (forms (rest this)))
      `(if ,test
           (progn ,@forms)
         (cond ,@others)))))

(defmacro cond (&rest clauses)
  (let ((expansion nil))
    (dolist (this (reverse clauses))
      (let ((test (first this))
            (forms (rest this)))
        (setf expansion
              `(if ,test
                   (progn ,@forms)
                 ,expansion))))
    expansion))

(cond (foo (bar) baz)
      (wombat wibble)
      (t 99))

(if foo
    (progn (bar) baz)
  (cond (baz wibble)
        (t 99)))

(if foo
    (progn (bar) baz)
  (if wombat
      (progn wibble)
    (if t
        (progn 99)
      nil)))

As you can see, writing macros can very rapidly become a complex task. However, the results tend to be rewarding, because the language is so much more powerful for the presence of macros.

11.5 Further examples of macros

• case

 

 

This has a syntax somewhat similar to cond:
    (case value clause-1 clause-2 ... clause-n)
where each clause looks like this:
    (keys form-1 ... form-n)
Each of the keys forms is a list of objects (not evaluated!). If value is eq to any of the keys for clause-1 then that clause's subforms are evaluated in the usual way; otherwise we try the next clause. The final clause may have one of the symbols t or otherwise (it doesn't matter which) instead of a list of keys - that means invoke this clause anyway if all else failed. Example:

(defun decode (x)
   (case x
     ((i uno) 1)
     ((ii dos) 2)
     ((iii tres) 3)
     ((iv cuatro) 4)))
(defun add-em (x) (reduce '+ (mapcar 'decode x)))

(add-em '(uno iii)) =>  4


Related macros include ecase which signals an error if none of the keys are matched, and typecase (along with etypecase) which dispatches on the type of value:

(defun satisfy (requirement state)
  (etypecase requirement
    (symbol ...)
    (string ...)
    (cons ...)))
• shiftf and rotatef

 

 

These two macros are probably more fun than use, except that rotatef is dead handy for swapping values around.

    (shiftf place-1 place-2 ... place-n new-value)
assigns the old value of place-2 into place-1, the old value of place-3 into place-2, and so on to the end where new-value is assigned to place-n. The old value of place-1 is returned.

    (rotatef place-1 place-2 ... place-n)
assigns the old value of place-2 into place-1, the old value of place-3 into place-2, and so on to the end where the old value of place-1 is assigned to place-n. The return value is nil. In particular,
    (rotatef place-1 place-2)
is equivalent to

 
(let ((temp (place-1)))
  (setf place-1 place-2
        place-2 temp)
  nil)
• print-unreadable-object

 

 

This macro has syntax:
    (print-unreadable-object (object stream &key type identity) forms) => nil
It outputs a printed representation of object on stream, beginning with "<#" and ending with ">". Everything output to stream by the body forms is enclosed in the the angle brackets. If type is true, the output from forms is preceded by a brief description of the object's type and a space character. If identity is true, the output from forms is followed by a space character and a representation of the object's identity, typically a storage address.

If either type or identity is not supplied, its value defaults to nil. It is valid to omit the body forms. If type and identity are both true and there are no body forms, only one space character separates the type and the identity.

In this example, the precise form of the output is implementation-dependent.

 (defmethod print-object ((obj airplane) stream)
   (print-unreadable-object (obj stream :type t :identity t)
     (format stream "~a" (tail-number obj))))

 (format nil "~a" my-airplane)
   =>  "#<Airplane NW0773 36000123135>"
OR =>  "#<FAA:AIRPLANE NW0773 17>"

 So, to evaluate a non empty list (seeing as the empty list is identical to the self-evaluating symbol nil), we look at the first element of that list.

1. If it is a symbol which names one of the 25 special operators, carry out the appropriate behaviour for that special operator.
2. Otherwise, if it is a symbol which has a macro definition, expand that macro and restart the evaluation (recursive loop!)
3. Otherwise, we must have a function. This may be either a symbol which is fboundp, or a list of the form
(lambda (arg1 arg2 ...) ....)
We evaluate each argument in turn (from "left to right") and call the function with the results of these evaluations. The result of the evaluation is whatever value the function returns. Example of this rather arcane use of lambda:
((lambda (key value)
   (setf (gethash key *table*) value))
 (first pair)
 (second pair))
but note that we could have done the same thing more legibly either by inserting a funcall
(funcall (lambda (key value)
           (setf (gethash key *table*) value))
         (first pair)
         (second pair))
or [much better] by transforming the expression to use let*
(let* ((key (first pair))
       (value (second pair)))
  (setf (gethash key *table*) value))
and so you rarely see lambda used like this.

11.7 Practical session / Suggested activity

• Recall the macros push and pop:
(setf words '(opposite of push))  =>  (opposite of push)
(setf same-words words)           =>  (opposite of push)
(push 'the words)                 =>  (the opposite of push)
words                             =>  (the opposite of push)
same-words                        =>  (opposite of push)
(pop same-words)                  =>  opposite
words                             =>  (the opposite of push)
same-words                        =>  (of push)
and implement them for yourself.
 
• Seeing as we're stuck with let in the language, use it to implement my-let*.

 
• Pick one or more of the macros discussed so far and attempt to implement them. Some are harder than others:

Easiest	and  prog1  prog2
A little harder	case  decf  dolist (in terms of loop)  dotimes (ditto)  incf  or  pushnew  rotatef  shiftf  trace (use the simplified syntax introduced in section 5.2)  untrace  with-open-file
Why are these ones indvisable?	defconstant  defparameter defstruct  defun  ignore-errors  lambda  loop  setf

• (Very difficult. Prize for first correct answer.) Evaluate the following forms in a lisp listener:
• (lambda (x) nil)
• (function-lambda-expression (lambda (x) nil))
and then consider the following cute puzzle among lispers: to come up with a self-reproducing consp form. A self-reproducing form is defined for the purposes of this question as one that evaluates to something equal to the original form. In other words, you need to come up with a form x such that
(and (consp x)
     (equal x (eval x)))
is true.
It is considered cheating if your solution involves any permanent side effects. For example, we could trivially solve the problem by first defining a function foo, say, that returns (foo) as its value. But we disqualify this solution because defining a function counts as a permanent side effect. As you might no doubt suspect, the best known solution involves the use of lambda.
The solution that I am aware of is two lines of code. I would be curious to know if there are longer solutions (a family of them perhaps). Anybody with time on their hands to prove their research abilities is welcome to impress me.

• Graham chapter 10. Skip exercise 4 if you're not happy with the macro do (I never voluntaily use it, so why should you have to?).
• Answer no more than one of the following questions:
• Why should nobody voluntarily want to use do?
• What is it about the macro do which means you don't have to know how any of the other iteration macros work? Is this true?
• Write a macro my-if which translates calls of the form (my-if a then b) or (my-if a then b else c) into standard if forms. Go back through your work, find some function you've written using if and rewrite it to use my-if instead.
• Read the specification of print-unreadable-object above and then see if you can determine (before trying it out) what the following would print:
(print-unreadable-object (*standard-output* *standard-output*))
When you do try it out, why do you see a nil?
• The definition of add-em in 11.5 above builds a list with mapcar and then throws it away soon after. Rewrite add-em using lambda to avoid this waste.
• Revisit your implementation of eval from week 9, and extend it in the light of this lecture.
• Revisit the last exercise in section 9.10. This should now look more elegant.