Trying to figure out ASDF

[psismondi]

I have been learning CL for several months now, loving it, and getting along without asdf or equivalent. I find the asdf documentation (from the asdf project page on commonlisp.net), ahem, horrible. (Most of the other material I have, from e.g. Peter Seibel, Paul Graham, etc. is great, so I don't think I'm a whiner about this. Maybe I am.)

If anyone knows of a good *complete* asdf tutorial point me to it.

I find this passage from what I take to be the official asdf manual quite confusing:

The files are located in the same directory as the file with the system definition. ASDF resolves symbolic links before loading the system definition file and stores its location in the resulting system 3 . This is a good thing because the user can move the system sources without having to edit the system definition.

First, what does "the file with the system definition" mean? Does it mean "the file containing the system definition" or "accompanied by" something or other? "With" is a weird choice of prepositions. And what symbolic links is he talking about? Symbolic links to what? I totally don't get it. And "stores its location" - WTF is "its"? This is a totally appalling mess of a sentence.

I'm sure it will all seem perfectly clear after I have figured it out.

BTW, those are actual questions - not just complaints about the sentence. Help me out if you can.

[nuntius]

On a posix system (anything but mswin), you can make symlinks to system1.asd, system2.asd, ... in one central directory (e.g. ~/.sbcl/systems). Then add this directory to asdf:*central-registry*. Due to the way ASDF resolves symlinks, it will look for source files relative to the location of the real system.asd file, not relative to the symlink.

[gugamilare]

The system definition file is normally a file with extension .asd that accompanies the library which uses it. It contains a system definition which tells asdf what are the source files, in which order they have to be compiled / loaded, how can one test it and so on.

Now, when you want to install a library, you need to tell ASDF where it is so it can load the library to you. You do that by inserting in the variable asdf:*central-registry* the path to the directory where the file .asd is located. But, instead of having to insert a new directory to this variable every time you want to use a new library, you just create a symbolic link in some directory.

For instance, suppose you downloaded a library named foo and installed it in the directory "/home/me/Lisp/libs/foo/". So there should be a file named "foo.asd" in that directory. One way to acknowledge asdf of that library is inserting

Code: Select all

(push #p"/home/me/Lisp/libs/foo/" asdf:*central-registry*)

into you Lisp init file (it is .sbclrc for SBCL, .clisprc for Clisp...). But that you would have to do with every library you install.

A simpler alternative would be to choose a directory, e.g. "/home/me/Lisp/systems/" and create a symbolic link there pointing to "/home/me/Lisp/libs/foo/foo.asd". Finally you tell asdf about that:

Code: Select all

(push #p"/home/me/Lisp/systems/" asdf:*central-registry*)

and you are done. Every time you download a new lib, you symlink the .asd file there

You can try using asdf-install, which downloads and installs libraries from Cliki automatically. SBCL already comes with it, you just need to

Code: Select all

(require :asdf-install)

In other systems, you will have to install it.

By the way, asdf and asdf-install are more or less a pain to work in Windows.

[psismondi]

Ok, thanks. Those replies help. I got the symbolic link thing working.

I don't know if I should post additional questions on this thread, or a new one. I guess I'll do it here.

After the initial simple example, the ASDF manual goes on like this:

Code: Select all

A more involved example

Let's illustrate some more involved uses of defsystem via a slightly convoluted example:

(defsystem "foo"  
  :version "1.0"  
  :components  
  ((:module "foo"  
    :components ((:file "bar")  
                 (:file "baz")  
                 (:file "quux"))  
    :perform (compile-op :after (op c) (do-something c))  
    :explain (compile-op :after (op c) (explain-something c)))  
   (:file "blah"))) 
The :perform tokens need explaining: essentially, these clauses:

:perform (compile-op :after (op c)  
           (do-something c))  
:explain (compile-op :after (op c)  
           (explain-something c)) 
have the effect of

 (defmethod perform :after ((op compile-op) (c (eql ...)))  
      (do-something c))  
 
 (defmethod explain :after ((op compile-op) (c (eql ...)))  
      (explain-something c)) 
where ... is the component in question.

Ok, this is totally opaque to me. He says that "the :perform tokens need explaining" but I don't see an actual explanation in what follows. The "tokens" apparently translate into an invocation of the defmethod macro. But what in the world does that mean in terms of its intended effect? When does the method get invoked, and what might it do when invoked? Some indication of what the "token" is trying to accomplish would have constituted an actual explanation.

[ramarren]

I don't think I have seen the explain thing ever used. Most of those more complex options of ASDF are quite rarely used. It is one of those things that if you cannot figure out from the source (ASDF is a single file of mostly portable Common Lisp), then you don't need.

In general one can hope that an operation is performed at point indicated by its name, that is, compile-op when compiling and load-op when loading, on all defined components, in order determined by the dependency network.

[wvxvw]

Hi. Sorry for maybe bringing up an old topic, but it seemed like the proper place to me. Pleas, feel free to move this post if you feel like.
I'm trying to figure out how to get several files compiled. I've followed several tutorials, but none worked (let alone following some illusionary good practices and avoiding deprecated API .
So, here I am, the code should explain my struggle, but if it doesn't, the explanation follows:

Code: Select all

(defpackage video-rounds-system
       (:use :common-lisp :asdf))
(in-package :video-rounds-system)

(defsystem "video-rounds"
	:description "rounds-web: a testing environment for Rounds project."
	:version "0.0.1"
	:author "wvxvw"
	:components ((:file "./rounds-web/packages.lisp") ;; defines (defpackage :rounds-web)
		(:file "./rounds-web/server.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/serialization.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/error-handlers.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/success-handlers.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/services.lisp" :depends-on ("./rounds-web/packages.lisp"))
		(:file "./rounds-web/snapshot.lisp" :depends-on ("./rounds-web/packages.lisp"))))
(asdf:oos :video-rounds)
(require :rounds-web)
(rounds-web:start-server)

The error:

Code: Select all

* (compile-file "start-server.lisp" :output-file "start-server")

; compiling file "/home/wvxvw/Projects/rounds/rounds-net/test-web-server/start-server.lisp" (written 02 MAY 2011 01:11:18 PM):
; compiling (DEFPACKAGE VIDEO-ROUNDS-SYSTEM ...)
; compiling (IN-PACKAGE :VIDEO-ROUNDS-SYSTEM)
; compiling (DEFSYSTEM "video-rounds" ...)
; compiling (OOS :VIDEO-ROUNDS)
; compiling (REQUIRE :ROUNDS-WEB); 
                                      ; compilation unit aborted
                                      ;   caught 1 fatal ERROR condition
; compilation aborted because of fatal error:
;   READ failure in COMPILE-FILE:
;     SB-INT:SIMPLE-READER-PACKAGE-ERROR at 901 (line 18, column 24) on #<SB-SYS:FD-STREAM for "file /home/wvxvw/Projects/rounds/rounds-net/test-web-server/start-server.lisp" {B0D9419}>:
;       package "ROUNDS-WEB" not found
; compilation aborted after 0:00:00.007
NIL
T
T
* 

Just to make sure I'm sane: this is the ./rounds-web/packages.lisp file

Code: Select all

(require :hunchentoot)
(defpackage :rounds-web
	(:use :cl :hunchentoot)
	(:export :start-server))

I have the file that defines a "system", which as far as my understanding permits means to tell SBCL to load specified files and compile them. After that I'm trying to require a package defined in those files, and whoops, no package... How do I even know what (defsystem) did? Am I at all understanding right what it was meant to do? I tried (operate-on-sytem) instead of (asdf:oos) with same results.
I've tried different ways of defpackage (the quote form and just a string, which I don't really understand the difference in the given context to no avail).

My Lisp is SBCL. This is probably not the newest version, but I would be thankful if the answer didn't involve updating and using some more libraries. I know I will have to update, but I'm just not able to do it now . If you may advise any reading on how to compile multiple files in the simplest way (I'm not doing anything complex!) that would help too.

Thanks in advance.

EDIT: For the future reference, I discovered that probably where it says :file, it doesn't really mean file, instead it only wants the file name (please correct me if I'm wrong, I'd really like to be wrong on that!) and the file must be in the same folder as *asd where you defined the system.
I finally got it working. But please tell me if I did it wrong! In this way:

Code: Select all

(in-package :cl-user)
(require :hunchentoot)
(require :cxml)
(require :cl-ppcre)
(require :cl-gd)
(defpackage video-rounds-system
       (:use :common-lisp :cl :asdf :hunchentoot :cxml :cl-ppcre :cl-gd))
(in-package :video-rounds-system)

(asdf:defsystem #:video-rounds
	:description "rounds-web: a testing environment for Rounds project."
	:version "0.0.1"
	:author "wvxvw"
	:components ((:file "packages") ;; defines (defpackage #:rounds-web)
		(:file "server" :depends-on ("packages"))
		(:file "serialization" :depends-on ("packages"))
		(:file "error-handlers" :depends-on ("packages"))
		(:file "success-handlers" :depends-on ("packages"))
		(:file "services" :depends-on ("packages"))
		(:file "snapshot" :depends-on ("packages"))))

Moved inside the ./rounds-web directory.

Code: Select all

(asdf:oos 'asdf:load-op :video-rounds)
(rounds-web:start-server)

also moved inside ./rounds-web .
I would be very happy to know how would I go about putting files in my project into different directories (there aren't too many of them, but I just like it to be in some order )

[ramarren]

The ".asd" file is loaded (that is, executed) by ASDF when scanning for systems and it should optimally not include anything other than (asdf:defsystem ...) form. The exception is when you are extending ASDF functionality in some way, but you shouldn't need that most of the time. The defpackage defining -system package is unnecessary, since ASDF loads the file in temporary package anyway. It appears in some older definitions, since it wasn't true in the past. But in any case if you have only the defsystem form composed from keywords and strings only it doesn't matter anyway, since no part of it depends on the package system, as long as the operator is given fully qualified form.

Code: Select all

(asdf:oos :video-rounds)
(require :rounds-web)
(rounds-web:start-server)

This lines are all not supposed to be placed in an asd file.

Furthermore, the first one is broken, since asdf:oos require operation specifier, typically 'asdf:load-op, and the preferred form in modern ASDF2 (installed by default by Quicklisp) is (asdf:load-system :system-name). Or for that matter (ql:quickload :system-name) if you are using Quicklisp, which you should.

The second line is mostly deprecated. REQUIRE is poorly specified. In SBCL it hooks into ASDF anyway. In general it shouldn't be used except as a shortcut in the REPL. It would also be redundant with the line above if it worked. Normally you shoudn't explicitly load systems from source files explicitly at all. The asd should specify dependencies on other systems.

The third line is wrong because you really should not start a server as a side effect of scanning system definitions. And it won't work anyway since neither REQUIRE nor ASDF:OOS have compile time effects, that is, the system wouldn't be loaded even if they were correct until the file was loaded, and it cannot be compiled because at compilation time the package doesn't exist.

Your packages.lisp file should contain REQUIRE, or any other file/system loading form, either. File and system dependencies are supposed to be specified by the asd file.

In asd file component names are names, from which pathnames are derived. It is possible to specify pathnames explicitly, but it is easier to just use something like (although I am fairly certain that you need more complex dependencies):

Code: Select all

(asdf:defsystem "video-rounds"
  :description "rounds-web: a testing environment for Rounds project."
  :version "0.0.1"
  :author "wvxvw"
  :depends-on (:hunchentoot)
  :components (:module "rounds-web"
                       :components
                       ((:file "packages") ;; defines (defpackage :rounds-web)
                        (:file "server" :depends-on ("packages"))
                        (:file "serialization" :depends-on ("packages"))
                        (:file "error-handlers" :depends-on ("packages"))
                        (:file "success-handlers" :depends-on ("packages"))
                        (:file "services" :depends-on ("packages"))
                        (:file "snapshot" :depends-on ("packages")))))

[nuntius]

I recommend against using ql:quickload, except when intentionally installing packages. It wraps asdf:load-system in a call that will optionally download some version of the package from Quicklisp if it is not found on your system. That may be proper behavior some times, but often it is not. I believe it is likely to result in frankenbuilds with subtle inconsistencies. YMMV.

[wvxvw]

Oh, thank you! Yes, I sort of figured it out that asd file has to only have the definition and maybe technical stuff related to locating the sources. Shame that's not what the docs say... I think documentation at times assumes too much about the reader
I'll know now to stay away from require too
One question though. I'm not sure if it worked by chance, or was intended. Once I called asdf:oos in the directory where *.asd file lives, it seems to find the system definition and load it. May I conclude that placing *.asd file in the directory with another file that calls asdf:oos is enough / is the right way to do it if that's just my code I'm working on?

[ramarren]

ASDF will scan some number of directories looking for asd files. Those directories are specified by, in old ASDF special variable *central-registry* and in ASDF2 by more complicated source registry mechanism. They will include the current directory, where current directory is defined by the *default-pathname-defaults*. At least in ASDF2 case it will be the value of that variable at the time source registry was initialized. It will not be the directory which contains the currently loaded file (unless those are the same). If you have many interdependent systems of your own you should add them to source registry.