Using a symbol at the same time its package is created

[pjstirling]

Hi,

I've been doing some network programming this weekend and I've hit an
issue on which I'd like some suggestions.

I send a number of tyes of message between nodes ("Hi", "Here's a
job", "Success", etc), which means that each message sent needs to
carry a sentinel to identify what type it is, and, since debugging
issues from mismatched literals between the sending and receiving code
is vexing, I want to have a package with the shared constants that the
code for each side of the connection can use.

I started relatively simply (what passes for simple with me, anyway),
a DEFPACKAGE form with the exports, and a macro that created the
DEFCONSTANT forms.

This works, but has the ordinary issues related to the need to make
edits in two places.

It also has the extra problem that (since I'm working out protocol
issues as I go) sometimes one constant needs to become two, and sbcl
follows the spec and complains about package variance if the new
export list doesnt contain all of the symbols that were previously
exported.

Obviously what I'd like is a single macro form that would let me make
the edit in one place, and take care of the need to UNEXPORT any
symbols not in the current list, I'm just having issues figuring out
how to do so "correctly".

The problem is that the symbols for the DEFCONSTANT forms for the
PROGN must be interned in a package that won't exist until after the
PROGN is processed.

The situation is akin to the following code:

Code: Select all

(progn
  (defpackage #:transcode-commands
    (:use)
    (:export "+HELLO+"))
  (defconstant transcode-commands::+hello+ "HELLO"))

debugger invoked on a SB-INT:SIMPLE-READER-PACKAGE-ERROR in thread
#<THREAD "main thread" RUNNING {10028169A3}>:
Package TRANSCODE-COMMANDS does not exist.

The way I opted to do it was to invoke MAKE-PACKAGE from the
macro-function if FIND-PACKAGE returned NIL, like so (slightly
simplified):

Code: Select all

(defmacro commands-package (name &body syms)
  (let ((p (or (find-package name)
	       (make-package name))))
    `(progn
       (defpackage ,name
	 (:use)
	 (:export ,@syms))
       ,@ (mapcar (lambda (sym)
		    (let* ((name (symbol-name sym))
			   (shorn (subseq name
					  1
					  (1- (length name))))
			   (sym (intern name p)))
		      `(defconstant ,sym ,shorn)))
		  syms))))

This seems to work, even across loading from a FASL (on sbcl), but I'm
not sure that it should work. As I pointed out in another thread some
time ago, side-effects in macro-functions are a no-no, because they
may be run many times, in an interpreter, or none at all if the
expansion was saved in a FASL. (I suspect in this case the sbcl FASL
format has some way to refer to symbols from non-existent packages).

I'm wondering whether I should abandon this approach, require an empty
DEFPACKAGE that is left alone, and then a separate macro that does the
business with EXPORT, or perhaps another approach entirely.

Thoughts, suggestions?

[David Mullen]

Well, if you're using intern in the macro, that's already a side effect. So make-package is just one more side effect. It's like you said—if you need to make a defconstant form, then you can't do that without having a symbol at macro-expansion time. I came up with this macro for illustrative purposes:

Code: Select all

(defmacro commands-package (package-name &body syms)
  `(eval-when (:compile-toplevel :load-toplevel :execute)
     (defpackage ,package-name (:use) (:export ,@syms))
     ,@(mapcar (lambda (sym)
                 (let* ((name (symbol-name sym))
                        (shorn (subseq name 1 (1- (length name)))))
                   `(let ((symbol (intern ,name ',package-name)))
                      (setf (symbol-value symbol) ,shorn)
                      (proclaim (list 'special symbol)))))
               syms)))

Then the expansion would be like:

Code: Select all

(eval-when (:compile-toplevel :load-toplevel :execute)
  (defpackage #:transcode-commands (:use) (:export #:+hello+))
  (let ((symbol (intern "+HELLO+" '#:transcode-commands)))
    (setf (symbol-value symbol) "HELLO")
    (proclaim (list 'special symbol))))

So the macro I just wrote doesn't have side effects. But then the symbols won't be constants—they'll just be specials. The only alternative I can think of is to outsource the find-package/make-package dance to a reader macro—then when using the commands-package macro, you'd use the special syntax for the package form, which ensures that the package will exist when the macro gets expanded.

[pjstirling]

I'm sorry for not being more specific on the side-effects issue, it's
not that all side-effects should be banned in the host lisp, it's that
you must ensure that any side-effects are executed in the target lisp.

I came across this issue myself when in my html generation library I
wanted to save some metadata about what attributes were valid for a
particular element so that it could be used at run-time.

I started by populating a hash-table stored in a special variable from
the macro-function, this works the first time, but when I loaded from
a fasl the cache was empty.

Incidentally, this took me ages to track down, because I didn't
see it happen very often. I normally go weeks (or longer) between
restarting my lisp (generally when I reboot for kernel updates), and
on top of that, I usually rebuild sbcl from git at the same time
(which invalidates all fasls, and therefore meant that the next time I
loaded my web-apps it would be a full compile, rather than just a
load, and the issue wouldn't present itself).

The correct way to do it was to change the macro expansion to include
the appropriate SETF form.

Back on the subject of my current problem:

I figured out what sbcl is doing so that it successfully loads from
fasls: it looks through the top-level PROGN, and turns the
compiled DEFPACKAGE form into a separate fasl-operation to the
DEFCONSTANT forms, so by the time that it registers the constants the
package does exist in the target lisp.

Unfortunately I don't think that I can reasonably assume this is
portable

[David Mullen]

I appreciate the clarity, since my comment was intended to make the same point you're making about side effects. If you want something to "stick" in the FASL then you can't rely on side effects that don't stick across Lisp instances. Unlike you, I'm doing this on a laptop with Clozure CL, so I'm restarting it all the time. For that reason, obviously I'd notice if the stuff you're doing didn't work across FASL loads.

Of course, that's not what you're asking. You're asking if it's portable and works everywhere. Well, you have a package definition in a top-level progn before the constants are defined. By the Processing of Top Level Forms, those forms are also top-level and are processed sequentially. As you know, people tend to put the package definition in a separate file—which is recommended by the standard—but it's not technically required.