What feature would you most like to see in Lisp?

[Unne]

Why doesn't someone do it then? It's not that these problems CAN'T be be solved in Common Lisp, it's that they haven't been and aren't being. I have to write programs today, I can't wait a decade while these things are fixed, assuming they ever will be.

A fair response is "Hey buddy, why don't YOU?" I don't have the time or the ability or the desire (from an "it'd be fun" perspective) to fix many of my perceived problems with CL, even though the language is powerful and flexible enough to let me do it, surely. One other possible motivation for me to do it would be necessity, but given Clojure (or even Ruby/Python which get you most of the way there) I don't even have necessity, because they already give me most or all of what I need, for which I am eternally grateful. Another motivation would be some kind of aesthetic or nostalgic devotion to Common Lisp that makes me solve the problem in spite of necessity or desire, but I don't have religious feelings about any language, and even though many people do, it seems not to motivate enough people to get us all the way there.

Lack of motivation and resources and effort in the community is the only problem, and it's a surmountable problem, but it's still a real show-stopper problem in the meantime.

[Ali Clark]

This thread was started in part to try to give an avenue to let off steam for exactly that. Its an informal note on the internet for the things that a modern lisp needs but doesn't currently have.

People should draw their own conclusions from the thread, but what I gathered was that there are more than one people out there who want:
a) GUI.
b) Threads.

On an optimistic note, CL appears to have outlets for both efforts. From what I've seen, McCLIM appears to be a fairly portable and supported GUI library. Threads unfortunately don't appear to have much appearance in open source implementations. But if SBCL threads can work on Windows as well as Linux, at least that could provide one free and cross-platform implementation to use threads on, which is a start.

Any replies with pointers to a standardised GUI library are welcome.
Any replies with pointers to a standardised threading library are welcome.
General replies on either of those are welcome.

And of course, if you can see other modern features that are missing from lisp, however small and however large, please post them here!
I (probably) won't write a library for any myself, but its good to make a note of it anyway.

[findinglisp]

But that's what I mean by theoretical concerns vs. practical concerns. Any implementation COULD have all the things I mentioned on my list, in theory, but how many do right now? I don't have a while to wait for things to sort out, so what are my options if I want to quickly and easily write a cross-platform multithreaded GUI app this afternoon? Not 2 or 10 years from now? That's why Clojure is exciting to me. Lack of those features is a self-inflicted problem in CL that could be fixed in theory, but it's still a problem, and Clojure solved it. Maybe it's not ideal, but at least it's a solution that exists right now

...

What are some other specific examples of functionality and syntax compromises besides loop/recur that are due to limitations of the JVM? Maybe there are some I'm not aware of. I don't know the JVM itself that well. I know that most of the many horrid limitations of Java don't exist in Clojure at all.

....

I think Clojure has potential to attract people from Java as much as attract people from other Lisps. The set of Java developers is a lot larger than the set of Lisp developers, so maybe Clojure has a chance even if it's rejected by Lispers. People using Java right now are the ones really in need of dire help anyways. It also has potential to attract people like me who desperately want to use Lisp but who find Common Lisp to be showing its age and lacking in many areas.

Okay, fair enough. I'll grant you that if you want to write a cross-platform GUI in a Lisp-like language today, your options are pretty limited. Certainly, there are implementations such as Lispworks that will allow you to do it in CL, but not the open source implementations today. Again, that's the taking advantage of the libraries thing and making the trade-offs thing.

The only thing I know about today in terms of a "wart" is the loop/recur thing. I honestly haven't done anything with Clojure other than read all the docs on the web site and all the various presentations that Rich has done that have been posted to the web. My real experience is admittedly limited. There may be performance issues that surface because of things about the JVM that can't be implemented as efficiently as possible (the way values are represented, boxed vs. unboxed, etc.). I don't know of anything, specifically, but every other Lisp I have seen implemented on the JVM seems to grumble about those things. Whether the performance is too poor for a given application is really the question. I'm generally not hung up on performance, per se, as I'm a big proponent that good enough is good enough and programmer time is much more valuable than machine cycles. For instance, I'm sure that Clojure is fine for GUIs and web applications which typically spend most of their time executing NOP instructions waiting for users to send them events. Whether Clojure would work for scientific computing, I have no idea.

Finally, whether Clojure survives or not is a very complex question to answer. I won't prognosticate one way or another ("It's way above my paygrade." ). As I said, I think either way Clojure is very interesting and Rich has done a great job. There are a lot of really cool ideas to learn from at a minimum. And I'm a big proponent of seeing where we can take Lisp in a post-standard-CL world. The main hang-up for me on Clojure is the insistence on the JVM. Certainly, it's a benefit, but it also brings limitations.

[metageek]

Certainly not a pipe-dream. You need a C header parser like cparse (http://common-lisp.net/project/cparse/), and the FFI ofcourse. From there you should be able to do something like (printf ...).

Another approach might to load a DSO and look for C++ functions; with C++ name mangling, you could probably get enough information to call them safely from Lisp. Name mangling is compiler-specific, of course, but it has the advantage of being a lot more restricted than C headers—no macros to worry about.

[jjgarcia]

Sorry for the late answer. I discovered Lisp forums quite late this month

It's a question of Clojure + JVM, or Common Lisp + Windows/Linux/OS X/others + a C compiler. Either way you're tied to and limited by your platform. How many Lisps have threading on Windows right now? How many even work cross-platform at all? How many libraries do? Look at ASDF for a good example of the bad side of using an operating system as your platform rather than a VM. The JVM has a very good compiler and garbage collector; how much time has been spent re-inventing those things for all the various Lisps? They're free in the JVM, and already mature and fast and stable and work on any OS.

C is not a limitation by any means. If anything, C is as low level as it gets without doing assembler and JVM's are written in C, btw. Then the other questions do affect me personally, so let me answer some of them.

Lisps do have a history and do have priorities and these do influence what features you find on them. Not many lisp programmers use threads or care about them, but that does not mean they do not exist. SBCL was one of the first free implementations to provide native threads, Clozure has them, and ECL as well.

Regarding portability, some lisps do have it easier than others. Those that implement their own native compilers have it more difficult, as they sometimes have assumptions about the OS and chips. ECL on the other hand is pretty much as portable as it gets, being completely built on top of C or C++. The only dependencies are the garbage collector, which is the Boehm-Demer-Weiser library, and the GMP bignum library, which can be compiled in pure C mode and is thus absolutely portable.

You talk about lisp implementations reinventing the wheel because we provide garbage collection? That is funny, because lisp implementations have had garbage collection way before the JVM ever did. SBCL and CMUCL have an impressive garbage collector, fast and accurate. ECL uses a well known and widely ported, the BDW, an example that a garbage collector can be reused. Others have different alternatives. All of them are much older than Sun's.

Java is definitely not the panacea. In my free time I do some stocks and derivatives trading and there you have two alternatives: Windows and Java. I must say the second one is not that nice in any platform other than Windows. Even some platforms do not work in some operating systems at all, such as OS X, because of the JVM they ship and the integration with the operating system that those programs have. Talk about portability.

[Unne]

Maybe not many Lisp programmers care about threads, but I do, and so does anyone who writes a GUI or web app. Which is a pretty huge slice of programming nowadays, in my world anyways. When we have 32- or 64-core CPUs in a few years, are Lispers still not going to care? Who's doing the work to keep all of the Lisps up to date with modern hardware and practices? Few have done it up to this point, and it's just going to get worse over time.

The JVM is written in C, so people building new languages on top of the JVM don't have to mess with C. That's the whole point. In SBCL you have this sad state of affairs: http://sbcl.sourceforge.net/platform-table.html Versions for some platforms lagging behind others, versions for some platforms non-existent because no one ported them. That is the limitation of using C and targeting an OS rather than a VM.

On the other hand for Clojure, most or all of that chart would be green, and all versions would be up to date. I don't have to wait and hope for someone to port Clojure to my OS. The hard work has been done in the VM and no one has to do it again.

I know that Lisp had garbage collection for a long time. My point was more about Lisps competing with each other. Does the SBCL GC do something substantially different than the ECL one or the Lispworks one or the CLISP one or the Allegro one? If not, why are they all still around, being maintained in parallel? Perhaps some give slightly different performance than others, but that doesn't matter nearly as much nowadays as it did in the 1980's when all of these Lisps were put together. Maybe this was a necessary state of affairs two decades ago. Today it's a waste of time. All you need is one good GC that everyone can use everywhere. This is debatable of course.

There are good and bad things about VMs. But the good things are relevant to me and to apparently most programmers in the world, given the immense popularity of .NET and Java.

(Please note, I'm not a Java fanatic. I use Ruby in my day to day life. But I can recognize the benefits of the JVM.)

[findinglisp]

The JVM is written in C, so people building new languages on top of the JVM don't have to mess with C. That's the whole point. In SBCL you have this sad state of affairs: http://sbcl.sourceforge.net/platform-table.html Versions for some platforms lagging behind others, versions for some platforms non-existent because no one ported them. That is the limitation of using C and targeting an OS rather than a VM.

On the other hand for Clojure, most or all of that chart would be green, and all versions would be up to date. I don't have to wait and hope for someone to port Clojure to my OS. The hard work has been done in the VM and no one has to do it again.

I know that Lisp had garbage collection for a long time. My point was more about Lisps competing with each other. Does the SBCL GC do something substantially different than the ECL one or the Lispworks one or the CLISP one or the Allegro one? If not, why are they all still around, being maintained in parallel? Perhaps some give slightly different performance than others, but that doesn't matter nearly as much nowadays as it did in the 1980's when all of these Lisps were put together. Maybe this was a necessary state of affairs two decades ago. Today it's a waste of time. All you need is one good GC that everyone can use everywhere. This is debatable of course.

I understand the point you're trying to make and to some extent I agree with you, but the facts actually undermine you here.

Specifically, Java has had just as rocky of a time getting JVM support everywhere. For years, the top Java enhancement request was "Port Java to Linux." Literally, that was it. It had thousands of votes from developers in Sun's bug database. Yes, that has gotten better since that time (Java obviously runs on Linux, even Sun's implementation these days, in addition to other open source versions now from Red Hat), but there are still many environments where Java won't run. And Clojure won't run there either until somebody ports Java. The people who port Java are not Clojure people, so you have this problem where you have to wait until an unrelated community decides that they want Java before you can get Clojure. Obviously, if they get excited about doing a Java port, there is benefit there because Clojure doesn't have to do the port, but you're stuck waiting until it happens.

Second, many of the Java implementations running on those different platforms use different technology. IBM's implementation of Java is different than Sun's. Specifically, the GCs are often a big source of difference in the implementation since you might want to trade off performance and memory usage for various operating environments (server vs. desktop vs. embedded, for instance). Thus, if you want to attack multiple CL implementations for reinventing the wheel (and there is some reason to call that out), don't use the GC as an example of that, particularly vs. Java.

That said, I won't argue with you further about Clojure. As I have always maintained, there is a lot to like in Clojure; my own opinion is that the use of the JVM isn't one of those things, however. I like the fact that Clojure exists. It's good for Lisp. It gets a lot of things right.

[jjgarcia]

Maybe not many Lisp programmers care about threads, but I do, and so does anyone who writes a GUI or web app. Which is a pretty huge slice of programming nowadays, in my world anyways. When we have 32- or 64-core CPUs in a few years, are Lispers still not going to care? Who's doing the work to keep all of the Lisps up to date with modern hardware and practices? Few have done it up to this point, and it's just going to get worse over time.

I am afraid the problem is in pessimistic views like this.

The JVM is written in C, so people building new languages on top of the JVM don't have to mess with C. That's the whole point. In SBCL you have this sad state of affairs: http://sbcl.sourceforge.net/platform-table.html Versions for some platforms lagging behind others, versions for some platforms non-existent because no one ported them. That is the limitation of using C and targeting an OS rather than a VM.

No, you are completely wrong about this. The fact that SBCL is lagging on some platforms is because you have to write a whole lot of things in assembler and C that the Sun people are giving to you for free. But that does not mean what you get from Sun is the best of the worlds. Or from IBM. Or from the GCJ people. You cannot blame an implementation for lack of developers and blame that on the fact of using C and porting things. Who writes your VM anyway?

On the other hand for Clojure, most or all of that chart would be green, and all versions would be up to date. I don't have to wait and hope for someone to port Clojure to my OS. The hard work has been done in the VM and no one has to do it again.

Note that I am not arguing about Clojure here. I do not know about this dialect, and right now I do not feel the need to learn it myself, but I just want to make it clear that you are spreading some misinformation about CL itself.

I know that Lisp had garbage collection for a long time. My point was more about Lisps competing with each other. Does the SBCL GC do something substantially different than the ECL one or the Lispworks one or the CLISP one or the Allegro one? If not, why are they all still around, being maintained in parallel? Perhaps some give slightly different performance than others, but that doesn't matter nearly as much nowadays as it did in the 1980's when all of these Lisps were put together. Maybe this was a necessary state of affairs two decades ago. Today it's a waste of time. All you need is one good GC that everyone can use everywhere. This is debatable of course.

Garbage collection is not a 100% solved problem, even though people may argue the opposite. Different implementations have different approaches. Have you heard about the difference between conservative vs. exact garbage collectors? Or between those that do not move and those that compactify? There are many choices. ECL cannot use a compactifying garbage collector without breaking the programs that embed it. SBCL does use a more sophisticated approach because it is in control of the memory.

There are good and bad things about VMs. But the good things are relevant to me and to apparently most programmers in the world, given the immense popularity of .NET and Java. (Please note, I'm not a Java fanatic. I use Ruby in my day to day life. But I can recognize the benefits of the JVM.)

I only entered this discussion to make your statements about CL more precise. I do think VM are nice and the fact that many mobile phones out there live out of java is a proof of that. I also do recognize the benefits of a VM, but this is nothing new and we still have to learn on that field. I myself program a lot in C/C++ for "a living" (if science can be considered so), and I do it with massive parallelization, and not only multithreading. Currently, VM will not help in this respect. And besides this, remember that just like somebody has to write your CL implementation, so does someone have to write the VM. The fact that one or another have more supporters does not make it neither the only alternative nor the best one.

[Unne]

No, you are completely wrong about this. The fact that SBCL is lagging on some platforms is because you have to write a whole lot of things in assembler and C that the Sun people are giving to you for free.

That is entirely my point. On one side, you need developers to port something. On the other you don't, because the work is already done and you can just use it for free. How can you say that requiring more developer time and effort is not a limitation?

People seem to be far too quick to dismiss manpower as a cost. In reality it's a huge cost, and if it's already been paid and you can take advantage, you are way ahead. This isn't pessimism, this is reality. If only SBCL had tons of developers with lots of time, it would exist on all platforms with libraries to rival any other language. If only I had a few million dollars, my house would be a mansion.

Garbage collection is not a 100% solved problem, even though people may argue the opposite. Different implementations have different approaches. Have you heard about the difference between conservative vs. exact garbage collectors? Or between those that do not move and those that compactify? There are many choices. ECL cannot use a compactifying garbage collector without breaking the programs that embed it. SBCL does use a more sophisticated approach because it is in control of the memory.

Yeah, it's an interesting subject, but it's largely academic. I know that it isn't a solved problem and there are different approaches, but any approach that works is good enough for me. Fast hardware and lots of memory means I don't have to care. I get by with Ruby after all, which is orders of magnitude slower than everything. ECL seems to be a sort of special case, given that it tries to interoperate with C. Rich has stated that the JVM's GC is pretty impressive nowadays anyways. He discusses it a bit here (video), I think perhaps at the end of part 2.

[jjgarcia]

That is entirely my point. On one side, you need developers to port something. On the other you don't, because the work is already done and you can just use it for free. How can you say that requiring more developer time and effort is not a limitation?

No, I think you do not get the point. Your argument is to jump on the bandwagon and leave everything aside when there is a project with a greater manpower. I strongly disagree with this. Diversity is the source of creativity. I furtherargue that there is room for everything and I will leave it here. I am happy that you are pretty excited about Clojure and all that, but please understand it as well that JVM is not everything and other people have other needs.

People seem to be far too quick to dismiss manpower as a cost. In reality it's a huge cost, and if it's already been paid and you can take advantage, you are way ahead. This isn't pessimism, this is reality. If only SBCL had tons of developers with lots of time, it would exist on all platforms with libraries to rival any other language. If only I had a few million dollars, my house would be a mansion.

Yes, but what is your point? That SBCL developers should stop doing what they are doing because there is the Java Virtual Machine? To promote other dialects? I sincerely hope you do not see the arrogance on that. You are asking them, and probably me, to stop supporting Common Lisp because you found something that makes you happy. I have seen this excitement too many times (C, C++, Sheme, Java, Haskell, Javascript...)

Garbage collection is not a 100% solved problem, even though people may argue the opposite.

Yeah, it's an interesting subject, but it's largely academic.

No, it is not. It is academic for you that get a very high level implementation, do not have serious needs other than threading and GUI, and do not worry about low level details. But perhaps somebody who works on a tight memory environment, or needs to port things to other OSs, or needs many languages to interoperate has to worry about this. Please understand that there are two levels of programmers: those that only work with high level libraries which are luckily supported everywhere, and those of us that have to deal with more complicated things. If I read all your posts, you are telling us we should be doing something else. But if all systems programmers disappear who will write your VMs? Who will research on the next big language? Do you realling think that a single dialect will settle everything? Or a VM for that matter?

I just want to warn you from your discourse of "do not waste effort on other things", "let's go with the right thing" because it can be harmful. I myself have experienced it on my own pet project, ECL, with GCL people asking to drop all effort on it and more recently Richard Fateman trying to convice Maxima developers to stop working on any port of Maxima to ECL. This is a discourse that can too easily become hate and in my own view it is not far away from that of people disregarding other lifestyles or beliefs.