Reading this thread there's a lot in it I'd like to respond to, but I'm not known for brevity in any case, and if I tried to respond to all of it I'd wind up writing something way too long to read. But...
First off I agree with findinglisp that Lispers ought not to gloss over the problems with CL. CL is very inconvenient to use in some respects. I mean- I use CL on both winxp and on Debian Linux (well, my home machine is Ubuntu these days, but I use Debian on my server), and I have been using CL on my own (i.e. not as part of a class or something, which would push the date back about 5 years) since around 2000. And I still get annoyed with some things. Not all of the difficulties are necessary consequences of the language itself, but when someone starts playing with CL they're not likely to make that kind of fine distinction.
That said, after 10 years of dabbling, and a couple years of actually using CL for some of my work, where I could get away with it, I pretty much know what to do to get things working in the implementations I use, and can figure out what to do, with some work, in new implementations. I started using ECL quite a bit about 1.5 years ago, and I think it's better than sliced bread, but there was a bit of pain at the beginning- I think it would be awfully difficult for someone to pick up as a first CL. Of course there's an easy answer to that, and it's the same answer the doctor gaveme when I told him it hurt my arms when I lifted them above my head: "Well then, don't do that."
I have to say that my opinion is that if you want to get started with CL, and if you want to have all the libraries, etc., (as opposed to just having a CL environment to learn to write CL in, in which case Lispworks Free edition on Windows is fine) you should be running some variant of Debian Linux, and using SBCL. But even then there are soe issues- my experience has been that using apt, or synaptic, or whatever package management tool to get sbcl and the libs you want is a losing proposition in the long run. And I'm not happy with any of the other means of managing libraries that I've tried (asdf-install, and the like), though I'll admit that I gave up on them a while ago, so maybe they are better now. At any rate, I manage my CL installations by hand. It's a bit of a pain, in the short term, but I find it tends to bite less in the long run.
I also keep archived copies for every library of every version I've ever had installed- if I were really bright I would actually keep my lisp libs directory under version control. A lot of the more important lisp libs are pretty much run from cvs or another verrsion control system, so you can't discount the possibility that someone 5 dependencies back is going to hose your entire tool chain with an ill-advised commit, if you choose to keep up-to-date. I just set up a new Ubuntu machine, and got all the newest versions of the libs I use- it took me most of a day to do it, one package at a time, and I will have to rewrite some of my code to realign it with the libs (hunchentoot in particular), but in the end that's not something I should make too big a deal of- it's the kind of thing I only do once every year or so.
But I can see how people just trying CL out might get a bit frustrated with all of that- and all I can say to them is: just get a base CL system and learn CL first. Worry about the libraries later. Historically speaking "Hello World" was printed to the console, not to a web page, and CL is ancient compared to Perl and Python and Ruby. My first "CL" (not standard-compliant, by any stretch) was poplog. There weren't really any libs available for it (and it was a pretty difficult environment in many respects), but that left you free to learn CL (or in poplog's case, to learn Cltl1
.) If you're just starting out in CL you should really concentrate on learning CL and not on learning to install CL libraries. A lot of people wind up giving up on CL before they've ever really programmed in it.
I've been intermittently playing with some code that spiders (a select part of) the web and does some document clustering, for a year or so now. There's also some frontend code involved- basically a web interface to it, and- well, now is not the time to explain it, but it sounds pretty similar to what findinglisp is talking about, in terms of the sorts of services you need. But it also has a lot of custom algorithmic stuff. Once I had all the libs in place, on SBCL under Debian, I didn't find using CL to be an impediment- in fact, I found it to be quite the opposite.
I would have had a hard time writing the algorithmic stuff in Ruby. This is partially because everything I know about Ruby I know from reading the Japanese docs (which are confusing- for instance I'm under the impression that Ruby is called Ruby because one day Matz was sitting around talking about birthstones with a co-worker when Godzilla burst into the room with an enormous Ruby held in his teeth- bear in mind that this was translated from Matz's English into Japanese, and then back into English by moi, so it might not be an entirely accurate account.) But basically this is where CL is strong- if _all_ you want to do is pull down some web pages and then present them to the user, then Ruby is probably easier. But if you want to pull down some web pages, process them intensively, and then present them to the user- well, for me, at least CL is actually a lot easier to use than Ruby, though that might be because I'm a lot more familiar with CL.At any rate, once I had all the libs installed I didn't run into the kinds of problems you seem to have run into.
Like I said earlier I started dabbling in CL about 10 years ago, maybe a little less. At that time there really weren't a lot of libraries available. When I look at how far CL has come in the last 10 years (no thanks to me, I might add- I have patched a couple of critical flaws in important libs, but that's about 20 lines of code, in all) I think CL is on the right track. I don't think CL is going to take over the world anytime soon (or ever, really), but I also think it's suitable for real work in the real world these days. Not everyone has to eat artisanal bread with good handmade cheese and brandy distilled from the tears of an angel for those things to have value. By the same token, it is not so much that CL has become less popular over the years, as much as it is that its appeal has become more selective
. It would be nice if it were easier to get paid to write CL, but... if you really want to promote CL then get yourself into a position of real power in some organization and then stake your career on CL. That's how languages get popular, I think, if they lack a wealthy sponsor.
Also, I think CL would benefit from not raising people's expectations so high... Paul Graham, bless him, is really guilty here. When you read something like "Beating the Averages".. well, you could be forgiven for thinking that if you just start programming in Lisp a Unicorn will shit under your pillow every night for a fortnight, and gnomes will come polish your... monitor. I'd like to see a law passed requiring everyone who cites "Beating the Averages" to have read and understood a pretty good portion of "On Lisp" first. Because programming, even, and maybe especially, in CL is not all unicorn shit and gnome blowjobs. CL has facilities that can make you a better programmer, IMHO, but being a decent CL programmer is pretty hard, actually. Thinking about code that writes code that writes code ad infinitum is not for the faint of heart.