Simple explanation of packages?

Discussion of Common Lisp
SideEffects
Posts: 4
Joined: Wed Aug 26, 2009 2:26 pm

Simple explanation of packages?

Post by SideEffects » Wed Aug 26, 2009 2:35 pm

Hi everyone, I'm new here so I hope I'm not asking something that has been answered before.

I'm trying to figure out how to use Common Lisp packages, in a very simple way.

Let's say I have code in 3 directories ~/a, ~/b and ~/c.

The code in ~/a and ~/b wants to use a shared "library" of functions stored in Lisp files in ~/c.

I'm not really concerned about namespaces, because I can guarantee that I haven't used the same names in ~/c as in ~/a and ~/b. I just want to be able to define a function in some file ~/c/C.cl like

(defun c-fun () ...)

and then be able to define a function in ~/a/A.cl that does something like

(defun a-fun () ... (c-fun) ...)

and another one in ~/b/B.cl that does

(defun b-fun () ... (c-fun) ...)

Can I make the stuff in ~/c into a "package" and export it to ~/a and ~/b. I couldn't find any good documentation on the web on how to do this.

Your help is appreciated.

S.E.

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Simple explanation of packages?

Post by findinglisp » Thu Aug 27, 2009 11:00 am

Okay, first you need to understand that unlike say Java or other languages, packages have nothing to do with directory structure. You can put all your source files into a single directory and use a million different packages. Or you can put your source code into a million different directories and use a single package. In short, a file is simply a file located in a directory. Common Lisp doesn't care about any of that.

Packages are data structures in a running Lisp image. Each symbol that is read by the Lisp reader must be interned into a package. In this sense, a package acts as a namespace, but it's really more than that since you can query a package for the list of symbols that are interned in it, manipulate the package at runtime, etc.

As the Lisp reader reads a source file, it will intern all the symbols it finds that aren't otherwise prefixed by a package name into the "current package." The current package is the value of the *PACKAGE* variable, and it typically set with the IN-PACKAGE form.

To use symbols from another package, you can:
  • Simply prefix the symbol from the other package with the package name. For instance, if you want to call C-FUN in package C from package A, simply invoke C:C-FUN (if the symbol is exported from package C) or C::C-FUN (if the symbol is not exported).
  • You can also use the USE-PACKAGE form to "import" all the symbols from a particular package into a different package. This allows you to invoke them without using the package name prefix. Only exported symbols are imported, so you'll still have to use C::C-FUN to call a non-exported symbol.
  • Finally, when you define package A, in the DEFPACKAGE form, you can use the :USE subform to import symbols from another package. This is essentially a more convenient way to issue a USE-PACKAGE form.
Note that using a package doesn't mean that the reader reads the package. All packages that you use must already be loaded separately. This is also a big difference from, say Ruby. Loading could be done using standard LOAD forms, or you could use a defsystem like ASDF to do that. The important thing is that loading has nothing to do with "using" a package.

I'd recommend that you read the chapter about packages and symbols from Practical Common Lisp: http://gigamonkeys.com/book/programming ... mbols.html
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

SideEffects
Posts: 4
Joined: Wed Aug 26, 2009 2:26 pm

Re: Simple explanation of packages?

Post by SideEffects » Thu Aug 27, 2009 1:50 pm

Okay, first you need to understand that unlike say Java or other languages, packages have nothing to do with directory structure. You can put all your source files into a single directory and use a million different packages. Or you can put your source code into a million different directories and use a single package. In short, a file is simply a file located in a directory. Common Lisp doesn't care about any of that.

Packages are data structures in a running Lisp image. Each symbol that is read by the Lisp reader must be interned into a package. In this sense, a package acts as a namespace, but it's really more than that since you can query a package for the list of symbols that are interned in it, manipulate the package at runtime, etc.

As the Lisp reader reads a source file, it will intern all the symbols it finds that aren't otherwise prefixed by a package name into the "current package." The current package is the value of the *PACKAGE* variable, and it typically set with the IN-PACKAGE form.

To use symbols from another package, you can:

* Simply prefix the symbol from the other package with the package name. For instance, if you want to call C-FUN in package C from package A, simply invoke C:C-FUN (if the symbol is exported from package C) or C::C-FUN (if the symbol is not exported).
* You can also use the USE-PACKAGE form to "import" all the symbols from a particular package into a different package. This allows you to invoke them without using the package name prefix. Only exported symbols are imported, so you'll still have to use C::C-FUN to call a non-exported symbol.
* Finally, when you define package A, in the DEFPACKAGE form, you can use the :USE subform to import symbols from another package. This is essentially a more convenient way to issue a USE-PACKAGE form.


Note that using a package doesn't mean that the reader reads the package. All packages that you use must already be loaded separately. This is also a big difference from, say Ruby. Loading could be done using standard LOAD forms, or you could use a defsystem like ASDF to do that. The important thing is that loading has nothing to do with "using" a package.

I'd recommend that you read the chapter about packages and symbols from Practical Common Lisp: http://gigamonkeys.com/book/programming ... mbols.html
Thanks, Dave, that is consistent with everything I've read about packages so far. The example in the Gigamonkeys book doesn't seem to clear up how the packages are actually loaded.

What I still fail to understand, and I apologize for being dense here, is how to tell the program in my file ~/a/A.cl to go look for a package in ~/c/C.cl. I understand that packages have nothing to do with directories, but you have to tell the program where to look somewhere right?

I'm assuming I need to put some markup like

(defpackage "c-package" (:use "common-lisp"))
(make-package :c-package)

in the ~/c/C.cl file and

(require :c-package)
(in-package :c-package)

in the ~/a/C.cl file. But how does Lisp know where to look for c-package?

I've tried things like putting
(load (compile-file "~/c/C.cl"))
at the top of my ~/a/A.cl file, but that doesn't seem right.

Confused,

S.E.

gugamilare
Posts: 406
Joined: Sat Mar 07, 2009 6:17 pm
Location: Brazil
Contact:

Re: Simple explanation of packages?

Post by gugamilare » Thu Aug 27, 2009 1:55 pm

There is a chapter dedicated to packages and symbols in the Practical Common Lisp book, in case you want a more complete explanation.

http://www.gigamonkeys.com/book/program ... mbols.html

ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: Simple explanation of packages?

Post by ramarren » Thu Aug 27, 2009 2:00 pm

SideEffects wrote:in the ~/a/C.cl file. But how does Lisp know where to look for c-package?
There is only one place where Lisp looks for packages: memory. As Dave said, packages do not have anything with files. REQUIRE function doesn't deal with packages, it deals with modules, but those are very weakly specified, so it is almost never used.

Usually compiling, loading and file dependencies are dealt with using a system definition facility like ASDF. There is a short introduction of Common Lisp project organization here.

SideEffects
Posts: 4
Joined: Wed Aug 26, 2009 2:26 pm

Re: Simple explanation of packages?

Post by SideEffects » Thu Aug 27, 2009 2:11 pm

Usually compiling, loading and file dependencies are dealt with using a system definition facility like ASDF. There is a short introduction of Common Lisp project organization here.
I guess ASDF is the piece I'm missing here. Groan, why does this have to be so complicated?

Harnon
Posts: 78
Joined: Wed Jul 30, 2008 9:59 am

Re: Simple explanation of packages?

Post by Harnon » Thu Aug 27, 2009 2:15 pm

You can try to download the Eclipse plugin for lisp, Cusp. It creates a template asd file and then you just have to right click to load. Unfortunately, currently only sbcl works. Actually, i have some patches that will allow clisp and ecl to work. Anyways, i think it is a great tool to start learning with.

Kompottkin
Posts: 94
Joined: Mon Jul 21, 2008 7:26 am
Location: München, Germany
Contact:

Re: Simple explanation of packages?

Post by Kompottkin » Thu Aug 27, 2009 2:18 pm

SideEffects wrote:I guess ASDF is the piece I'm missing here. Groan, why does this have to be so complicated?
One of the reasons is that the DEFSYSTEM proposal failed to be ratified by ANSI.

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Simple explanation of packages?

Post by findinglisp » Thu Aug 27, 2009 6:44 pm

SideEffects wrote:
Usually compiling, loading and file dependencies are dealt with using a system definition facility like ASDF. There is a short introduction of Common Lisp project organization here.
I guess ASDF is the piece I'm missing here. Groan, why does this have to be so complicated?
It doesn't necessarily have to be complicated. Worst case, just create a file that loads all your other files. I do this all the time when I don't want to deal with ASDF. If I was creating a large system or a library for release that had lots of other dependencies, I would use ASDF, but it doesn't have to be any more complex than:

Code: Select all

(load "a/A.lisp")
(load "c/C.lisp")
...etc...
Just make sure you load them in the right order. Then you'd have:

a/A.lisp

Code: Select all

(defpackage :a (:use :cl))
(in-package :a)

(defun foo ...)
(defun bar ...)
c/C.lisp

Code: Select all

(defpackage :c (:use :cl)) ;; could also (:use :cl :a) if you wanted to call FOO and BAR directly, without A:FOO and A:BAR

(in-package :c)

(a:foo 1 2 3)
(a:bar 1 3 3)
(defun baz ...) ;; BAZ is now interned in package C
Note that all these forms are also independent of files. The reason is that while the reader is reading a file (when you load it), it's simply building up state as it goes. It reads a form and then evaluates it. It then reads the next form and evaluates that. Etc., until it reaches end-of-file. Thus, you need to execute a DEFPACKAGE before you execute an IN-PACKAGE that references it, but there is no reason that those have to be in the same file. For instance, many people put a bunch of DEFPACKAGE forms into a single file called something like packages.lisp that gets loaded first. This defines all the packages up front. Then every subsequent file has an IN-PACKAGE form that tells the reader that everything after that point is to be read into the specified package.

Also, note that you could also literally concatenate all these files together in whatever load order was required to make it all work and load just that single file, like so:


bighonkingfile.lisp:

Code: Select all

(defpackage :a (:use :cl))
(in-package :a)

(defun foo ...)
(defun bar ...)

(defpackage :c (:use :cl)) ;; could also (:use :cl :a) if you wanted to call FOO and BAR directly, without A:FOO and A:BAR

(in-package :c)

(a:foo 1 2 3)
(a:bar 1 3 3)
(defun baz ...) ;; BAZ is now interned in package C
Then just (LOAD "bighonkingfile.lisp").

The reader would then process the DEFPACKAGE forms, then various IN-PACKAGE forms, etc. In other words, LOAD is simply like pointing the REPL at a file and everything in the file is processed effectively as if you had typed it at the REPL (this ignores compilation and EVAL-WHEN and some other nuances; that's the advanced course).

So, remember, DEFPACKAGE creates an in-memory data structure into which symbols are interned. The IN-PACKAGE form sets the *PACKAGE* variable which tells the reader which package is the current package. As the reader encounters unprefixed symbols, it interns them into the current package specified by *PACKAGE*. It's really that simple. The reader is independent of files; the reader operates at the REPL and with LOAD (LOAD is basically the equivalent of typing all the forms in the file into the REPL by hand, only automated). Forget files completely other than as blobs of program text which can be LOADed. For what it's worth, ASDF (and any other defsystem) is simply a complex way of specifying which files depend on other files so that the dependencies can be handled automatically to create the correct sequence of file loads. Think of ASDF as being sort of like apt-get or yum on a Linux box, but for file loading instead of handling package dependencies.

If all this seems pretty byzantine, it's only because it's not automated like it has been in many other languages. There is just some more machinery exposed here with Lisp that is otherwise handled automatically in Ruby, Java, etc. The mechanisms are all pretty simple, however.

I know that when I was going through this EXACT SAME confusion a few years ago, it didn't really make sense to me until I realized that a symbol isn't just a source-code thing--it's a first-class object that persists in the running image. So is a package. These are data structures at runtime, not just data structures in the compiler. When you write out a symbol name in a file, the reader basically converts that to a reference (a pointer) to a symbol data structure in the running image. But when you type "FOO" it has to know which "FOO" symbol you want to reference. To do that, it looks in a package data structure. You can think of a package data structure as a big hashtable that can map a symbol name (a string) to a symbol data structure (that's what INTERN does). Within a package, there is only one "FOO" symbol, but there can be another "FOO" symbol in another package. When the reader is reading along, it finds a block of source text that names a symbol, it then calls INTERN on it, using the current *PACKAGE* and gets back a reference to a symbol data structure. DEFPACKAGE creates the package (the hashtable). IN-PACKAGE simply sets the value of *PACKAGE*.
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

SideEffects
Posts: 4
Joined: Wed Aug 26, 2009 2:26 pm

Re: Simple explanation of packages?

Post by SideEffects » Fri Aug 28, 2009 5:35 am

Thank you Dave, that is the best explanation I've read yet. Until now I hadn't really understood the relationship between LOAD and packages. I had assumed that the IN-PACKAGE was somehow doing the loading, or that asdf was needed to load packages. I don't think this is made clear in Siebel's book either.

If I understand what you wrote, and I have (load "B.cl") inside A.cl, and then I enter (load (compile-file "A.cl")), then the (compile-file "A.cl") will also compile the code in B.cl, right?

S.E.

Post Reply