Programming Style & (eval ...)

Discussion of Common Lisp
ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: Programming Style & (eval ...)

Post by ramarren » Fri Jul 25, 2008 12:10 pm

taylor_venable wrote:I have another related question. Since macros don't evaluate their arguments, is there a way to get an argument to a macro to evaluate without using eval?
The point of macros is that they happen at compile time, so all necessary information must be available at that time. If such information doesn't exist at such time, then obviously it cannot. If it does exist, then one can write a macro which will expand to a macro with arguments evaluated.

The problem here I think is that "code as data", while in theory a nice idea, when driven to its ultimate conclusion tends to lead to problems in practice. So in Common Lisp there is some separation, which can be broken by using eval/compile explicitly. And if you want to, nobody will stop you. But without that, macros can execute code they contain, and insert code to be executed at runtime, but cannot execute code they receive.
taylor_venable wrote:The specific instance which makes me ask this is that I've been working with cl-who, a package that takes lists and turns them into SGML. There's a macro with-html-output-to-string which takes an argument like (:p "hello") and turns it into <p>hello</p>.
Not exactly. Macros create code, not output, so in this case (:p "hello") is turned not into a string, but a function call (write-string "<p>hello</p>" stream). cl-who is a compiler, you can't pass it code-as-data without explicitly calling eval/compile, as above.
taylor_venable wrote:But what I need is to evaluate some code, then pass the result to with-html-output-to-string, like so:

Code: Select all

(cl-who:with-html-output-to-string ... (car '((:p "hello")))) => "<p>hello</p>"
But since the argument isn't evaluated, cl-who doesn't know what to do with (car ...) so it outputs nothing.
Actually, as documentation states "A form which is neither a string nor a keyword nor a list beginning with a keyword will be left as is (...)". So the cl-who knows what to do with (car ...), and it is executed, it just outputs nothing because well, it doesn't. What you might want is compile the code you wanted evaluated, either with another with-html-macro or cl-who htm specially treated symbol, and then use normal control flow rather than data manipulation.
taylor_venable wrote:(...) This works, but along the lines of avoiding eval I wondered if there was another way to escape from macro non-evaluation?
You can wrap the entire thing in another macro, as I mentioned in the beginning... this perhaps lacks something in elegance. I think I run into similar problem when I was learning Lisp, when autogenerating some FFI bindings, and I finally decided to output to another file and compile that. This works especially well if generated code can be expected to be fairly static.

nklein
Posts: 12
Joined: Sat Jun 28, 2008 9:13 am
Location: Minneapolis, Minnesota, USA
Contact:

Re: Programming Style & (eval ...)

Post by nklein » Fri Jul 25, 2008 2:35 pm

taylor_venable wrote:

Code: Select all

(defun all (f l)
  (let ((r (mapcar f l)))
    (eval `(and ,@r))))
I haven't read this whole thread, so I might be repeating something already mentioned. But, the every function has the advantage that it doesn't call the function f more times than it has to. To see the difference, think of how often #'evenp is called in these two macros:

Code: Select all

(defmacro applied-and-1 ( f l )
      `(and ,@(mapcar #'(lambda (v) (list f v)) l)))
(applied-and-1 evenp (1 2 3 4 5 6 7 8))

(defmacro applied-and-2 ( f l )
      `(and ,@(mapcar f l)))
(applied-and-2 evenp (1 2 3 4 5 6 7 8))

taylor_venable
Posts: 10
Joined: Tue Jul 15, 2008 4:50 am
Location: Fort Wayne, Indiana, United States
Contact:

Re: Programming Style & (eval ...)

Post by taylor_venable » Fri Jul 25, 2008 4:02 pm

@Ramarren

Sorry for my imprecise language, and thanks for your time in trying to explain it. This is probably just what you've already said, but I want to restate it, even if only for my own sake. I think I understand that when a macro is "called" (encountered at compile time) it is evaluated, and any result is expanded into its position in the surrounding code verbatim (so to speak), then at run time it (that is, the result of the expansion) is evaluated. So my question, stated with the most precision I can muster at this stage of my learning, is, given that a macro seems to at most be able to refer to the text (so to speak) of its arguments, does that not necessarily mean that once one is using a macro (let's call it m) which does not itself evaluate the arguments internally, that from there on "up" in the code the arguments to that macro m cannot be received in (and therefore expanded into) evaluated form without explicitly using eval in a macro that expands "around" m? So, per my situation currently, with-html-output-to-string is a macro that receives its argument as text, and as such it will never do what I am wanting unless I explicitly generate the result by calling eval and inject the result using a comma within a back-quote, for example, as I have shown above -- in other words, without using eval it will always either operate on "form" or on "(car '((:p "foo")))", neither one of which is what I want. Is this correct? I'm not particularly worried about the Lisp eval police :) so if this is the way I end up doing it, no problem. I just want to make sure I understand why I have to do it this way; if I'm not right on by now, hopefully I'm getting at least a little closer.

But it seems that this is the way things work, so now I'm wondering if there are any short-cuts to doing this, like a special form in CL that basically wraps a macro with a call to eval like this for you, or if it is my duty to write this code myself. It just seems odd to me that in the last week I've twice run into the situation where I wanted to evaluate the argument to a macro, and have it expand into code that contains the evaluated form, rather than the textual form. But perhaps it's because I'm new at using macros and don't yet know when and how it's appropriate to use them. In any case, thanks again for your help.

@findinglisp

Thanks for the reference, I'll definitely check this out tonight and over the weekend to try to get a better feel of how to do what I want to accomplish. I'm getting the feeling that there's a substantial gap in either my understanding of the system or how I'm thinking about the problem, so reading up on it and writing some more code should surely help.
"I have never let my schooling interfere with my education." -- Mark Twain

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Programming Style & (eval ...)

Post by findinglisp » Fri Jul 25, 2008 8:45 pm

taylor_venable wrote:I think I understand that when a macro is "called" (encountered at compile time) it is evaluated, and any result is expanded into its position in the surrounding code verbatim (so to speak), then at run time it (that is, the result of the expansion) is evaluated.
"encountered" is a good word. The best way to think about a macro is that it's a part of the compiler that you have written. It's a function that extends the compiler with your special syntax. When the compiler encounters code with your macro in it, it passes all that list structure that looks like parameters, unevaluated, to the macro. The macro hands back whatever list structure it wants, and the compiler then replaces the macro site with that replacement structure and keeps compiling. This can happen multiple times, recursively, until the compiler expands the whole Lisp form to something with all macros removed. At that point, the compiler can then compile everything similar to what it would do in a C compiler and such.

So, a macro is nothing more than a function that gets called at compile time to transform the input code to something that you want the compiler to then digest.
So my question, stated with the most precision I can muster at this stage of my learning, is, given that a macro seems to at most be able to refer to the text (so to speak) of its arguments, does that not necessarily mean that once one is using a macro (let's call it m) which does not itself evaluate the arguments internally, that from there on "up" in the code the arguments to that macro m cannot be received in (and therefore expanded into) evaluated form without explicitly using eval in a macro that expands "around" m?
I have to admit that I read the above several times and I don't quite understand what you're getting at. An example would be great here to help clarify what you're saying.

That said, let me see if I can give you a bit more information and see if that helps jiggle something.

A macro receives, through its parameters, the exact list structure that is located at those parameter spots. Being a function that is executed at compile-time, the macro can literally do anything with that list structure that it wants. The macro is a function running in the context of this large Lisp program called the compiler. So, the macro could, if it wanted, evaluate that list structure using eval, it could print it on the printer, it could open up a socket and post it to LispForum (please don't write a macro to do that ;) ), etc. Most of the time, a macro just manipulates the list structure to rearrange it and combine it with other list structure in some way to generate more efficient code or replace what would otherwise be a lot of repetitive typing, etc.

Now, if I understand what you mean by "see," then yes a single macro can't "see" any of the code in the list structure that surrounds the macro, in the same way that a function can't "see" any parameters passed to another function which calls it. Everybody can only "see" their local parameters. That said, you can write macros that work together or use conventions to pass data between themselves. For instance, you could have a global variable named *foo* that one or more macros set or read. Generally, this is bad and it's a sign that you're either up against a really tough problem or you're going about things the wrong way, but sometimes there is a need for it.
So, per my situation currently, with-html-output-to-string is a macro that receives its argument as text, and as such it will never do what I am wanting unless I explicitly generate the result by calling eval and inject the result using a comma within a back-quote, for example, as I have shown above -- in other words, without using eval it will always either operate on "form" or on "(car '((:p "foo")))", neither one of which is what I want. Is this correct? I'm not particularly worried about the Lisp eval police :) so if this is the way I end up doing it, no problem. I just want to make sure I understand why I have to do it this way; if I'm not right on by now, hopefully I'm getting at least a little closer.
I would check the library to see if it has a way to do what you want, which is include computed values. If it doesn't, then perhaps rather than twisting the library using EVAL, you should choose a new library, or hack the source code if you have it to do what you want.

In this case, CL-WHO is written by Edi Weitz. Edi is a great Lisp hacker, as you'll soon learn, and his libraries smoke! They're very high quality, have great documentation, and Edi usually thinks of just about everything.

I have never used CL-WHO myself, but I perused the docs online and came to this section:
http://www.weitz.de/cl-who/#syntax
Here, you can read that Edi has provided for just the situation that you are interested in. Specifically, he gives you the STR, FMT, ESC, and HTM escapes. These are tags in the arguments to WITH-HTML-OUTPUT-TO-STRING that are processed specially by the macro. Essentially, the macro walks through the list structure you give it as an argument and interprets it (it's literally a small interpreter running in the context of the macro). Typically, it interprets it by creating code that generates HTML output, but when it encounters one of these tags, it takes some list structure that you the user have included and injects it into the macro expansion. Then, when the macro hands all that back to the compiler, the compiler will arrange to have your forms called at just the right time in the middle of all that output.
But it seems that this is the way things work, so now I'm wondering if there are any short-cuts to doing this, like a special form in CL that basically wraps a macro with a call to eval like this for you, or if it is my duty to write this code myself. It just seems odd to me that in the last week I've twice run into the situation where I wanted to evaluate the argument to a macro, and have it expand into code that contains the evaluated form, rather than the textual form. But perhaps it's because I'm new at using macros and don't yet know when and how it's appropriate to use them. In any case, thanks again for your help.
Generally, whenever you reach for EVAL, it's the wrong tool. As a beginner, it's tempting, because it's right there. But once the light-bulb goes off, you'll realize that you almost never need it. About the only legitimate use of EVAL is when you're getting code from the outside world during the runtime of the program and you want to evaluate it (like a REPL, for instance). Other than that, if you have the code that you think you want to EVAL at compile time, EVAL is most probably the wrong tool.
@findinglisp

Thanks for the reference, I'll definitely check this out tonight and over the weekend to try to get a better feel of how to do what I want to accomplish. I'm getting the feeling that there's a substantial gap in either my understanding of the system or how I'm thinking about the problem, so reading up on it and writing some more code should surely help.
No problem. Once you have had a chance to digest all this and you think you're starting to "get it," I'd read On Lisp by Paul Graham for a heavy duty dosage of macrology.
http://www.paulgraham.com/onlisp.html
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

danb
Posts: 35
Joined: Sat Jun 28, 2008 1:05 pm
Location: Urbana, Illinois, US
Contact:

Re: Programming Style & (eval ...)

Post by danb » Sat Jul 26, 2008 1:14 am

findinglisp wrote:
So my question, stated with the most precision I can muster at this stage of my learning, is, given that a macro seems to at most be able to refer to the text (so to speak) of its arguments, does that not necessarily mean that once one is using a macro (let's call it m) which does not itself evaluate the arguments internally, that from there on "up" in the code the arguments to that macro m cannot be received in (and therefore expanded into) evaluated form without explicitly using eval in a macro that expands "around" m?
I have to admit that I read the above several times and I don't quite understand what you're getting at. An example would be great here to help clarify what you're saying.
It sounds like taylor_venable may want to do some fairly complex runtime generation of html. If the html-generating code can't be hardwired into a markup template, then maybe a functional html library would be more flexible. I think Ocsigen does something like that in OCaml, but I haven't seen anything comparable in CL.
Last edited by danb on Sat Jul 26, 2008 11:29 pm, edited 2 times in total.

dlweinreb
Posts: 41
Joined: Tue Jul 01, 2008 5:11 am
Location: Lexington, MA
Contact:

Re: Programming Style & (eval ...)

Post by dlweinreb » Sat Jul 26, 2008 1:26 am

taylor_venable wrote:
Ramarren wrote:
taylor_venable wrote:Which leads me to my question: is it considered acceptable style to use eval to do something like this? I typically shun the use of eval in other languages, because it's usually just a cheap way of getting something done which could be handled in more elegant ways, but with the data-is-code orientation of Lisp, is this a common tactic to employ?
In my (not very extensive, mind you) experience using eval is very rare, and it's use can be almost always replaced with something better.
I have another related question. Since macros don't evaluate their arguments, is there a way to get an argument to a macro to evaluate without using eval? The specific instance which makes me ask this is that I've been working with cl-who, a package that takes lists and turns them into SGML. There's a macro with-html-output-to-string which takes an argument like (:p "hello") and turns it into <p>hello</p>. But what I need is to evaluate some code, then pass the result to with-html-output-to-string, like so:

Code: Select all

(cl-who:with-html-output-to-string ... (car '((:p "hello")))) => "<p>hello</p>"
But since the argument isn't evaluated, cl-who doesn't know what to do with (car ...) so it outputs nothing. What technique is commonly employed to do this? By using eval I can write a macro like:

Code: Select all

(defmacro test (form)
  `(cl-who:with-html-output-to-string (*standard-output*) ,(eval form)))
This works, but along the lines of avoiding eval I wondered if there was another way to escape from macro non-evaluation?
What you do is to expand into the code that ought to get evaluated. (You definitely do not use eval!)

Either this is harder than I thought, or I'm just sleepy right now, but try this:

Code: Select all

(defmacro test (form)
  `(list 'cl-who:with-html-output-to-string '(*standard-output*) ,form))
and if that works (sorry, I don't have time to test it now), try:

Code: Select all

(defmacro test (form)
  ``(cl-who:with-html-output-to-string (*standard-output*) ,',form))
-- Dan

taylor_venable
Posts: 10
Joined: Tue Jul 15, 2008 4:50 am
Location: Fort Wayne, Indiana, United States
Contact:

Re: Programming Style & (eval ...)

Post by taylor_venable » Sat Jul 26, 2008 12:10 pm

findinglisp wrote:
taylor_venable wrote:So my question, stated with the most precision I can muster at this stage of my learning, is, given that a macro seems to at most be able to refer to the text (so to speak) of its arguments, does that not necessarily mean that once one is using a macro (let's call it m) which does not itself evaluate the arguments internally, that from there on "up" in the code the arguments to that macro m cannot be received in (and therefore expanded into) evaluated form without explicitly using eval in a macro that expands "around" m?
I have to admit that I read the above several times and I don't quite understand what you're getting at. An example would be great here to help clarify what you're saying.
I was just trying to state in as rigorous (and possibly also confusing!) a way as possible that macros cannot access the evaluated form of their arguments [e.g. "3" rather than "(car '(3 4 5 6))"] without explicitly doing the evaluation themselves, which in light of the explanations from you and Ramarren this makes perfect sense. So, a result of this is that once the text / unevaluated / list form of an argument like (car '(3 4 5 6)) is used as a parameter for a macro m, any other macros encountered during the expansion of m also receive the text / unevaluated / list form, unless m does the evaluation explicitly. Example:

Code: Select all

(defmacro foo (x) `(some-other-macro ,x))
(macroexpand-1 '(foo (car '(3 4 5 6)))) => (some-other-macro (car '(3 4 5 6)))

(defmacro bar (x) `(some-other-macro ,(eval x))
(macroexpand-1 '(bar (car '(3 4 5 6)))) => (some-other-macro 3)
Which follows logically from what you both have said. OK, so I think I get all that now. Still, I can see why using eval is not a good idea, because if you throw any variables into the expression, like:

Code: Select all

(let ((x '(3 4 5 6)))
  (macroexpand-1 '(bar (car x))) => error: unbound "x"
It seems as though eval operates in the "null lexical environment" (as the standard says) where x is unbound. Obviously, using eval is not a workable solution.

Even though I rely a good deal on Edi's code as-is (cl-ppcre is amazingly awesome), it looks like it may be a lot easier to add into cl-who the features I need, rather than trying to tie them in at a higher level. Basically, for anybody who may be interested (and at the admitted risk of getting away from the intent of the OP), here's what I'm doing. I've got my website which runs on Hunchentoot. Until recently it was running under Yaws using Xmerl to generate an RSS feed from individual XML files, which were also parsed with Xmerl to generate HTML blog pages on my site. But then my server suffered an unexpected motherboard failure and in wanting to play with Lisp more I moved to Hunchentoot. Rather than playing with XML I thought it would be cooler to take advantage of the fact that Lisp makes an excellent data storage system by itself and start doing my blog entries as Lisp files. They work like this (imagine this is a file at the path "/blog/2008/05/01" on the web server):

Code: Select all

((:title . "Title Goes Here")
(:description . "This is what shows up in the RSS feed.")
(:body . ((:p "Here is some text for the HTML page.")
          (:p "And here's another paragraph."))))
There's one chunk of Lisp like this (that is, one blog entry) per file, and there's one Lisp file (rss-content.lisp) accessible to the server that contains a list of all the files to put into the RSS feed, so something like:

Code: Select all

("2008/05/01" "2008/05/02")
Now, my RSS feed generator reads rss-content.lisp and generates the RSS <item> XML for each entry using cl-who. This I do by reading the file into a list content, then using cl-who:str on the part of the list that I want, which basically amounts to evaluating the argument [something like "(cdr (assoc :title content))"] and writing the result in the midst of whatever other processing was going on. This is fine for the RSS feed, because the title and description don't contain nested elements, so writing the content out as a string is fine. But... for the HTML, obviously, this won't work because trying to use cl-who:str gets you the printed version of the list. Closer than before, but still not quite what I wanted. :) So what I'm looking for is a way to read the entry file into content and then have (cdr (assoc :body content)) written out as transformed HTML. Now it seems like perhaps cl-who can't do this, but I'm going to ask on the mailing list to find out for sure.

@dlweinreb

Thanks for the idea, but neither of those seem to do what I need.
"I have never let my schooling interfere with my education." -- Mark Twain

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Programming Style & (eval ...)

Post by findinglisp » Sat Jul 26, 2008 8:00 pm

taylor_venable wrote:So, a result of this is that once the text / unevaluated / list form of an argument like (car '(3 4 5 6)) is used as a parameter for a macro m, any other macros encountered during the expansion of m also receive the text / unevaluated / list form, unless m does the evaluation explicitly.
Yes, you're working with code-as-data at that point, so everything is unevaluated. Most of the time that's just what you want because macros run at compile time, not at run time (ignoring interpretation vs. compilation issues here). Essentially, if you want something to run when the compiler runs, then you want macros. If you want something that runs when the program is run, you want functions.
<<snip...>>
Now, my RSS feed generator reads rss-content.lisp and generates the RSS <item> XML for each entry using cl-who. This I do by reading the file into a list content, then using cl-who:str on the part of the list that I want, which basically amounts to evaluating the argument [something like "(cdr (assoc :title content))"] and writing the result in the midst of whatever other processing was going on. This is fine for the RSS feed, because the title and description don't contain nested elements, so writing the content out as a string is fine. But... for the HTML, obviously, this won't work because trying to use cl-who:str gets you the printed version of the list. Closer than before, but still not quite what I wanted. :) So what I'm looking for is a way to read the entry file into content and then have (cdr (assoc :body content)) written out as transformed HTML. Now it seems like perhaps cl-who can't do this, but I'm going to ask on the mailing list to find out for sure.


Okay, I gotcha. I think it can, but I think you're going about it the wrong way. Essentially, CL-WHO is a sort of templating engine, where instead of storing the templates as HTML with embedded code (as ASP or JSP might do), you instead store the whole thing as a Lisp sexpr so you can include it right in the middle of all the other code you're writing. It's really designed to have a big template with cl-who:str and the other such functions being used to fill-in various pieces of data here and there. It sounds like you're wanting something more similar to XSLT, where you can transform the sexprs representing your data to HTML. CL-WHO might be able to help with some of that process, but if you want to go that way, what you really want to do is write a set of functions that walk your own particular data structures and spit out HTML appropriately.

You could create a template for your blog entries that uses CL-WHO, but you would essentially do something like this:

Code: Select all

(let ((entry (read-blog-entry)))
  (with-html-output (*standard-output*)
    (:html (:body (str (get-body entry))))))
I probably have that wrong as I have never used CL-WHO but essentially you can use CL-WHO:STR to embed some data from the lexical environment, or even global environment, into the CL-WHO template that you're using. The GET-BODY function would be something you would write and it would grab whatever data it needed out of the ENTRY variable and would format it as a string. If you want HTML markup around that, you could do that using appropriate CL-WHO markup around the exact string. If you macro-expand that whole thing, you'll see that CL-WHO generates code to generate the <HTML> and <BODY> tags in HTML, and them some code that would call your GET-BODY function on the ENTRY variable in the middle of all that. The call to GET-BODY would execute at *runtime* and would do whatever you need for that particular blog-entry based on the data in ENTRY.

I'd seriously recommend those chapters from Practical Common Lisp that I referenced before. They go into this exact sort of application and I think you'll learn a lot.
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

Post Reply