Packages

Discussion of Common Lisp
Post Reply
jsmcgd
Posts: 9
Joined: Sat Jun 28, 2008 4:43 am
Location: UK

Packages

Post by jsmcgd » Sat Jul 19, 2008 5:45 am

I've been tinkering with Lisp for a while now. I've read through 3 Lisp books. One thing that I still don't have pegged is how to use packages correctly. I still find myself having to load packages manually. I was wondering if someone could provide an example of some code that demonstrates good programming style for packages (ideally in SBCL).

It would be hugely appreciated.

Cheers.

ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: Packages

Post by ramarren » Sat Jul 19, 2008 10:30 am

jsmcgd wrote:I still find myself having to load packages manually.
Packages are not loaded (except in trivial sense, ie. in the same way all other Lisp code is). Packages are just namespaces, a way to associate strings with symbols. Most of the time you create a package, use other packages you want to be referred to unqualified, read all your symbols in it, mostly those forming source of your coe, and export those that are supposed to be used by other code.

Maybe you mean ASDF systems?

jsmcgd
Posts: 9
Joined: Sat Jun 28, 2008 4:43 am
Location: UK

Re: Packages

Post by jsmcgd » Sat Jul 19, 2008 3:12 pm

How perceptive! Yes I think I do mean ASDF systems.

How should one refer to something like, for example: lispbuilder-sdl? Is it a package, is it an ASDF system? Both?

EDIT: I found this which helps a great deal http://weitz.de/packages.html
If anyone could post some code where semantics of these terms become self evident I think it would be very helpful.

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Packages

Post by findinglisp » Sat Jul 19, 2008 7:40 pm

Let me take a crack at this and see if I can manage to not botch it up. I'll make use of some possible implementation details which are typically opaque to a programmer but which might help ground you a bit. I'll also simplify and cut corners, so don't be too hard on me if I'm not strictly accurate all the way through. :)

Okay, first you have to understand about the Lisp reader. Whenever a symbol is read into the system (at the REPL, for instance), the symbol must be interned. Roughly, this corresponds to entering the symbol into a lookup table with the string that names the symbol as the key. Whenever the reader reads that same symbol again, it first checks whether the symbol is already interned by looking up the symbol's name in the lookup table. If it's already there, the reader returns the previous symbol. If the symbol hasn't yet been interned, the reader creates a symbol object with the name that was read and enters the name -> symbol mapping into the lookup table so subsequent readings will return the same symbol.

Now, given that background, the trick with packages is that rather than have a single global lookup table for symbols read into the system, we have many. Each package is basically a data structure that contains a lookup table as well as a list of symbols that are exported from itself, and a list of symbols and packages that it imports symbols from. This package data structure is created with the DEFPACKAGE form. All that DEFPACKAGE does is create this data structure. Nothing more. One of the key points is that this is a first-class data structure in the system, however. It's not a figment of the compiler's imagination that disappears when code is compiled. It persists through the life of the running Lisp image or until the programmer removes it.

Now, when the reader reads in a source file, the question is, where should those symbols be interned? The reader has to intern them someplace. The answer is that it interns them in the package that is identified by the variable *PACKAGE*. The IN-PACKAGE form basically just sets *PACKAGE* to the specified package name. The rest of the symbols read by the reader after that point are then interned in the specified package until another IN-PACKAGE form is encountered or until end-of-file. A logical consequence of this is that a source file can contain symbols and definitions that are interned in more than one package. This isn't possible in a language like Java where a source file has a 1-to-1 correspondence with a class name and its position in the file system identifies which package it belongs to. If the Lisp behavior seems odd, try to envision the reader simply operating on the REPL, incrementally, rather than compilers and all the rest of it. Another consequence is that the various forms that make up the package can be scattered across multiple source files. As we need to do is execute an IN-PACKAGE form and keep reading.

An important point is that when a package uses another package, the reader first checks in all the used package to see if the symbol is a public symbol of that package before interning it in the current package. If so, the reader returns that symbol. That's why you can use symbols like CONS in every package that has used the COMMON-LISP package without having to write it out as COMMON-LISP:CONS all the time. If the reader doesn't find the symbols in any of the used packages, it interns it in the current package.

Now, there are several consequences of all this:
  1. We need to create the packages with the DEFPACKAGE form before we can use IN-PACKAGE to then start interning symbols into the package. That has to happen first, otherwise IN-PACKAGE can't find the package. The typical way to handle this is to put the DEFPACKAGE form into a separate file of its own and to make sure that file is loaded before any others in the package. By doing that, we can be sure we only enter the DEFPACKAGE once and that everything works correctly. If your package is small, you can also put the DEFPACKAGE form right at the front of the single source file for that package.
  2. Second, we need to first load all the packages that a given package uses before we start reading code that depends on the symbols from those other packages, otherwise the reader won't find them interned in the used packages and the symbols will get interned in the current package.
So, now imagine a large system that is composed of tens of different packages, possibly with hundreds of source files. Managing all those dependencies can be complicated. When you load everything, it all has to be done in the right order. If it's done in the wrong order, symbols will get interned into the wrong packages and lots of bad stuff will happen.

To manage all this complexity automatically, we use a system definition tool, or a defsystem. ASDF is one example of a defsystem, but there are others, too. The defsystem manages all the dependencies and ensures that packages are loaded before the packages that use them, all the way up to the top of the application.

I hope that helps.
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Packages

Post by findinglisp » Sun Jul 20, 2008 8:13 pm

It's probably also worth your time to read through the chapter on packages in Practical Common Lisp:
http://www.gigamonkeys.com/book/program ... mbols.html
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

Post Reply