Need a *general purpose* way to APPLY macros and special ops

[vigil_ante]

I know about the (every #'identity mylist) trick to emulate an (apply #'and mylist), but is there a general solution? I'm a bit of a noob, but I think I've managed to construct a lovely little closure builder with only one drawback:

Code: Select all

(defun functional (this-function &rest other-functions)
  (lambda (&rest closure-args)
    (apply this-function (mapcar (rcurry #'apply closure-args) other-functions))))

(I used Graham's rcurry (ANSI Common Lisp page 110) to partially apply the argument list, because it seemed like the quickest way of mapcar-ing a function using the same arguments each time. Probably I am missing an obvious way to do it, but it isn't my primary concern at the moment.)

The idea is you have a number of other-functions that take the same kind of arguments. Let's say you want to call these functions on the same args and pass all of the results to another function, this-function, as a test or an aggragation or something. And let's say you want to pass around this prepackaged test/aggragator and use it whereever you want. So that instead of saying (this-function ((function1 myargs) (function2 myargs) [...]) every time you use it, you simply create a new function with (functional this-function function1 function2 [...]), then pass the resulting closure whatever args you wish. I think it has a lot of potential, at least for my pet project, but it'd be much nicer if it worked with special operators and/or macros. I'm talking primarily about this-function, though it'd be cool if other-functions could be special ops or macros as well.

Ideas?

[Ramarren]

Macros are expanded at macro expansion time and special operators have a special meaning to the compiler, so you cannot use them at runtime without invoking the compiler or at least the interpreter. While sort of possible the performance and complexity penalty would most likely make that useless.

[vigil_ante]

My understanding of the compilation/interpretation process is a bit limited, but I don't mind the performance penalty. This is for prototyping only. Hopefully it's possible to function or macro away the complexity. The goal here is expressiveness and ease of use.

Someone else suggested wrapping special ops or macros in lambdas and using reduce. That works for some cases like "and", but fails if you need more than 2 arguments at once, like an if-then-else. I can write custom wrappers I suppose, but that kinda defeats the purpose...

regardless, so what about special operators then? They don't have any runtime issues...

[Ramarren]

regardless, so what about special operators then? They don't have any runtime issues...

They do, what makes special operators special is that they are treated specially by the compiler/interpreter. You would have to construct the expression tree and then EVAL it for that to work, except that EVAL executes in null lexical context, so it wouldn't even be the same anyway.

Special operators and macros are supposed to implement syntax anyway, and I do not believe that applying them in this way makes sense in most cases. In the few cases where it might it is easy enough to convert to functional form.

[vigil_ante]

Why would it need to involve eval? Obviously I don't know the precise steps involved, but special operators can be called at runtime, so it's just a matter of bindings... so the compiler/intepreter is not build to handle late(r) bindings for special ops? Is that the issue?

I don't particularly care if I have to functionify the operators, which can necessarily entail lowering efficiency and/or breaking some uses that involve side effects (for instance, both the "then" and "else" clauses would be evaluated, which is fine unless there are side effects.) Though I suppose there are only what, 30 or so special operators, only a handful of which see any regular use, so it may not be worth the bother. "functional" seems to be a very expressive and convenient tool in my current project, much moreso than I originally anticipated, so if I could extend its domain beyond functions...even if that meant sacrificing performance and/or breaking side-effect usages...

[Ramarren]

special operators can be called at runtime

Special operators are not really "called" in the same sense that functions are. Special operators define a syntax for a certain operation. To construct the operation at runtime you have to build a code tree and then compile or interpret it.

Honestly, I am not even sure what you are trying to achieve. What special operator or macro other than possibly AND/OR would you ever want to APPLY? They are syntax abstractions, even if you could apply them, it would necessarily mangle the syntax and as far as I can see just make it an exercise in pointless code obscuration (which admittedly can be a fun exercise, but still). Functional abstractions should be used over functional interfaces, not syntax interfaces.

[gugamilare]

Why would it need to involve eval? Obviously I don't know the precise steps involved, but special operators can be called at runtime, so it's just a matter of bindings... so the compiler/intepreter is not build to handle late(r) bindings for special ops? Is that the issue?

I don't particularly care if I have to functionify the operators, which can necessarily entail lowering efficiency and/or breaking some uses that involve side effects (for instance, both the "then" and "else" clauses would be evaluated, which is fine unless there are side effects.) Though I suppose there are only what, 30 or so special operators, only a handful of which see any regular use, so it may not be worth the bother. "functional" seems to be a very expressive and convenient tool in my current project, much moreso than I originally anticipated, so if I could extend its domain beyond functions...even if that meant sacrificing performance and/or breaking side-effect usages...

Consider this problem:

Code: Select all

(if (zerop x) x (/ x))

That piece of code works without any errors (supposing, of course, that X is a number).

It's obvious that you can define IF as a function:

Code: Select all

(defun functional-if (test then &optional else)
  (if test then else))

... but you simply can't prevent FUNCTIONAL-IF from evaluating its arguments. For instance, the following code will signal an error:

Code: Select all

(functional-if (zerop x) x (/ x))

That is why IF is not (and can't be defined as) a function, because, when called, it doesn't evaluate its arguments.


If you care about it, there is a solution to this. It's called lazy evaluation - the functions are redefined in such a way that their arguments are not evaluated unless necessary. There are libraries for lazy evaluation in Common Lisp (e.g. CLAZY).

[vigil_ante]

I hadn't thought about using lazy evaluation in context of differentiating special ops/macros from functions, thanks.

Honestly, I am not even sure what you are trying to achieve. What special operator or macro other than possibly AND/OR would you ever want to APPLY? They are syntax abstractions, even if you could apply them, it would necessarily mangle the syntax and as far as I can see just make it an exercise in pointless code obscuration (which admittedly can be a fun exercise, but still). Functional abstractions should be used over functional interfaces, not syntax interfaces.

This is for a personal project with lots of data analysis and optimization. The idea is to obscure (I would use the word "encapsulate") an arbitrary number of aggragators, filters and flags, while preserving the underlying argument interface. Whether those aggragators, filters or flags are made up of functions, special ops and/or macros is irrelevant for my desired usage, which is to have an opaque closure I can pass around without knowing or caring about the logic it implements. The resulting closure can then be used as input to build yet another closure with 'functional'.

On a more general note, even beyond this particular project, the thing I enjoy most about CL is precisely the opaqueness/'obscurity' that bothers other people so much. I tend to share the opinion of Doug Hoyte regarding the importance of "dualities of syntax" and referential transparency being often highly undesirable. Lisp's uniformity of syntax (do-this to-this-stuff) allows you to divorce what you are doing from how you are doing it to a degree not possible in any other language I've encountered. For performance reasons I understand the necessity of separate implementations (which sadly leads to usage incompatibilities), but as far as expressivity is concerned I think it would be fabulous (at least while prototyping) if there were no difference between macro, special operator and function other than the evaluation of arguments and the resulting lexical environments. I had a rollicking debate on IRC about this last night; IMO there is an unhealthy obsession with implementation over usage, though someone mentioned fexpr as a possible solution for what I'm looking for. Though it feels more and more like I'll have to convert 'functional' into a macro... or just start writing function versions of / wrappers for the special ops and macros I need.

[gugamilare]

Make sure this will actually be needed in whatever you are doing. Premature optimization might be bad, but premature generalization is just as bad. Both might draw you away from what you are actually trying to achieve.

[vigil_ante]

So, it turns out making function versions of the stuff I need ain't so bad after all (since I do not mind automatic evaluation of arguments.)
Latest version of my closure builder:

Code: Select all

(defun argpass (control-function &rest inputs)
  (let ((curried-control control-function) (input-functions ()) (index-counter 0))
    (dolist (this-input inputs)
      (cond ((functionp this-input)
            (push this-input input-functions)
            (incf index-counter))
          (t (setf curried-control (ncurry index-counter curried-control this-input)))))
    (lambda (&rest input-function-args) (apply curried-control (mapcar (rcurry #'apply input-function-args) (nreverse input-functions))))))

Credit to pjb and sykopomp for creating the ncurry function (took 'em less than 2 minutes.)

I'm using the symbol "@" in my project for readability, but will call it argpass for conversation. The improvement in this version is that non-functional arguments will be detected and curried in. This removes the need to use any constant-returning functions. Remaining limitations include all input functions still needing to accept the same number and type of arguments (was intentional, though I'm thinking about making a version that allows the caller of ^ to specify that one of the input function's arguments should be dealt with separately), and of course the inability to use non-functions. Any feedback is welcome.

EDIT: fixed misplaced incf