Anyone using Lisp for web in production?

Discussion of Common Lisp
AlexPaes
Posts: 21
Joined: Sat Jun 28, 2008 3:38 am
Location: Lisbon, Portugal

Anyone using Lisp for web in production?

Post by AlexPaes » Mon Jul 07, 2008 9:54 am

Hi, does anyone around here been using Common Lisp for developing real web applications?

I've been using PHP for all my web programming but since i started learning Lisp i feel like i could really take advantage of this great language, so i started playing around with hunchentoot, cl-who, clsql and parenscript. But i'm worried about how it would work in a production environment. How well does CL scale in terms of performance, can i have different applications binded to different domains on the same server with a single lisp image or do i need multiple images running, etc... So anyone using CL for serious web work please post your experiences, issues and advices.

Thanks in advance to everyone.
CL-USER> (setf *boss* (make-instance 'smart-person))
NIL
CL-USER>

mike
Posts: 4
Joined: Wed Jul 09, 2008 1:25 pm
Location: San Francisco, CA
Contact:

Re: Anyone using Lisp for web in production?

Post by mike » Wed Jul 09, 2008 1:44 pm

I'm using SBCL for http://streamfocus.com/, a recently launched startup for online project and task management.

We're using some old web libs that work well enough, but if I were to start today, I would go with the ones you mentioned, hunchentoot, cl-who, etc. We do us cl-sql, but only the functions, not the CLOS stuff.

Since we're new, we unfortunately haven't hit scaling problems yet. My informal tests seem pretty good. I've gotten good use out of sbcl's built-in profiling functions, so when I find a page that's slow, I can easily find the offending code and fix it. Usually, it's been the normal high-level stuff like hitting the database too many times. In one case, it was pure code, but optimizing it with vectors and declaring integers fixed it up easily.

I'm running my lisp webserver on a different port, then using apache in front proxying to the lisp. The reason for this was SSL support, though you can get lisp to do ssl with the right tools. I don't know about binding multiple domains specifically, but certainly in this type of setup, you could make apache do it. And running a lisp webserver is usually a function call, not a config file for a process, so you can run as many as you want on different ports within the same process.

I find being able to update the running process though a REPL a wonderful advance over my previous experience with Java. However, I don't have a good way to replace an entire application in place like I can with a Java EAR. I'm not sure what the solution is, but I haven't ran into a real problem yet.

I love that html is generated from lisp code (like cl-who), rather than html templatized with code, but that does only work when it's programmers writing the web pages rather than designers.

-Mike

nklein
Posts: 12
Joined: Sat Jun 28, 2008 9:13 am
Location: Minneapolis, Minnesota, USA
Contact:

Re: Anyone using Lisp for web in production?

Post by nklein » Thu Jul 10, 2008 11:40 am

And, Mike? You're in a situation where you have a full server box or a big slice of one? or is it being served through some webhosting company?

Thanks,
Patrick

mike
Posts: 4
Joined: Wed Jul 09, 2008 1:25 pm
Location: San Francisco, CA
Contact:

Re: Anyone using Lisp for web in production?

Post by mike » Thu Jul 10, 2008 12:32 pm

nklein wrote:And, Mike? You're in a situation where you have a full server box or a big slice of one? or is it being served through some webhosting company?
We run on amazon ec2, Ubuntu specifically (http://groups.google.com/group/ec2ubuntu). There are cheaper options, but I liked the amount of control I had with ec2, and it's working well.

JamesF
Posts: 98
Joined: Thu Jul 10, 2008 7:14 pm

Re: Anyone using Lisp for web in production?

Post by JamesF » Thu Jul 10, 2008 7:52 pm

Not quite in production yet; give me a couple of days. Seriously, I've four sites that should be going beta tomorrow.

It's taken me a while to come up to speed, as my programming background before Lisp was SQL and scripting in shell, perl and PHP. Now that I've made progress up the learning curve, it's an awesome language to work in.
I'm using Hunchentoot, cl-who and CLSQL, with apache in front and PostgreSQL behind. Like Mike, I'm not using the CLOS stuff in CLSQL, and I'm looking at moving to postmodern. Oh, and mel-base is handy for email notifications, plus cl-pdf will be in use for prettier invoices before long. With a bit of refactoring, I've abstracted most of the moving parts into a couple of libraries of my own, and I can now spin off a new site in minutes flat - most of the incremental time for a new site is in the graphic design. Handling multiple domains with a single lisp image isn't a problem; I use apache to manage the domains and SSL termination from the outside, and run a separate hunchentoot server with its own port number and dispatch table for the active portion of each site. Because most of the functionality comes from a common library, adding a site adds very little memory overhead until it starts getting serious traffic. Of course, I say this without having seriously profiled or load-tested them :)

I looked at EC2, but still have an issue with Amazon's abuse of software patents (one-click checkout is patentable?!), so I'm reluctant to give them money. I've found http://www.flexiscale.com/, who provide a similar service. This allows me to scale up very quickly if (hopefully when) the sites mature a bit and come to wide public attention. For now, I want them to stay relatively low-profile, because the single worst thing that could happen is that thousands of potential customers visit in the early days, conclude (probably rightly) that it sucks, and never come back. I'm anticipating a lot of refinement right after they go live, and it's easier to do that when you don't have a large userbase to annoy.

The other thing that will help scaling is application architecture: each site shares a base library with the rest, but doesn't share its database or any state. So I can move any given application to a new server, update apache, and instantly handle more load. I haven't yet checked, but it should even be safe to run multiple instances of a given site against a common database server, and load-balance them via apache. The databases all use the same schema, but are separate, so again can be moved to a different database server if contention becomes an issue; this is much less trivial than moving an appserver, proportional to the amount of data that needs moving, but can still be done.
Initially, everything runs on a single server. As traffic ramps up, I'll move the most demanding server (web, app, database) onto its own virtual host. If one of the sites is hogging everything, it'll get its own set of hosts.

A handy hint about using virtual hosting: go for Xen. Virtuozzo doesn't play nice with SBCL; there's a problem with memory allocation. There was when I tried it, anyway. I've no idea about VMware, never having tried it.

Updating an app is relatively easy, with asdf. I use darcs for version control, which is great for producing tarballs of a release and gives me release versioning for free. I combine that with symlinks into asdf's system directory. Unzip a new version, update the new link, fire up a separate lisp image to pre-compile and smoke-test the new version (you never know for sure), then update the running process. To do that, I use screen: reattach to the image actually serving sites, (use-package :cl-user), and use asdf to reload whichever site that needs upgrading. Of course, I haven't tested what happens when you do this while the site's under load, so I may need to stop that application while I upgrade it (10 seconds' worth of interruption).

Disclaimer: I have a few years of experience as a web- and application-server admin in a high-traffic environment, so I learned a few tricks about architecture and version management :)

Apache is my choice because I've used it for years, I know it well, and I know from experience that it'll handle fairly high loads. I've heard good things about other webserver/proxy software, but will shop around if and when apache starts cracking under the strain of a single site. It's also battle-hardened, so I'm much more comfortable with that exposed to the internet, than I would be by sticking hunchentoot straight out there.


In case you're wondering: yes, there's an excellent chance that I'll release and open-source the libraries if it looks like a worthwhile contribution. However, they need a lot of refinement before I can do that with a straight face.

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Anyone using Lisp for web in production?

Post by findinglisp » Thu Jul 10, 2008 9:05 pm

JamesF wrote: A handy hint about using virtual hosting: go for Xen. Virtuozzo doesn't play nice with SBCL; there's a problem with memory allocation. There was when I tried it, anyway. I've no idea about VMware, never having tried it.
SBCL works find in VMware. I use it that way all the time. I could see how Virtuozzo might cause problems. There is some interesting memory mapping going on with Virtuozzo. With VMware, all that happens deeper under the covers so it's less of a worry.
...then update the running process. To do that, I use screen: reattach to the image actually serving sites, (use-package :cl-user), and use asdf to reload whichever site that needs upgrading. Of course, I haven't tested what happens when you do this while the site's under load, so I may need to stop that application while I upgrade it (10 seconds' worth of interruption).
What's your experience running things under screen? I use screen a lot for minor interactive use, but if the screen instance dies, all your attached processes die, too, right? Also, how do you handle unplanned reboots, for power outages, say? Does everything come back up correctly using rc.local or something like that, or do you have to restart it all by hand?

Finally, thanks for sharing your experience. Very interesting! 8-)
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

JamesF
Posts: 98
Joined: Thu Jul 10, 2008 7:14 pm

Re: Anyone using Lisp for web in production?

Post by JamesF » Thu Jul 10, 2008 10:07 pm

Haven't had a problem with screen so far, so I haven't worried about that. I'd prefer to use slime, but can't stand emacs; long-term, I'm hoping to equip climacs with vim bindings and use that. I've actually been running a lisp server that way for a few months now (demo version of one of the sites), and it's been fine.

Getting it to start automatically is something I haven't actually tried yet, I have to admit. There's an awful lot of work to be done after the initial go-live, and this is one of those little tasks. From a cursory look-over, it should be entirely possible to write an /etc/rc.d/ script that'll run as the appropriate user and kick off a detached screen session, which in turn calls the script that starts up the lisp process.

sbcl's quite happy to run arbitrary commands on startup; I have a script on my laptop that takes advantage of this, which makes life nice and easy for development. It kicks off a couple of xterms, starts sbcl in one and gets it to load the appropriate asdf system before calling (use-package) to get me into the package itself, then fires up gvim and tells it to load up whatever file I'm concentrating on. Lots of moving parts all over the place, which I'm not so happy about, but they're all small and have been reliable so far.

I'm still debating how to handle automated shutdown as well. The leading contender is yet another hunchentoot server process, which functions as a quick-n-dirty RPC server: call the right URL, and it'll shut down each of the servers before telling sbcl to quit. That one will be firewalled from everything except the local host, and invoked via something like curl or wget :)


I'm always happy to share what I've learned. Apart from anything else, I've gotten so much benefit from FOSS that it's one way of contributing in return.

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Anyone using Lisp for web in production?

Post by findinglisp » Thu Jul 10, 2008 10:30 pm

JamesF wrote:Getting it to start automatically is something I haven't actually tried yet, I have to admit. There's an awful lot of work to be done after the initial go-live, and this is one of those little tasks. From a cursory look-over, it should be entirely possible to write an /etc/rc.d/ script that'll run as the appropriate user and kick off a detached screen session, which in turn calls the script that starts up the lisp process.
Yea, I'd definitely make sure this works well. After the obligatory power outage takes down everything in the data center, you don't want to be faced with restarting oodles of Lisp-based applications all by hand. Not fun. :(

And yea, I know that the colo always has backup power, so that never happens :roll: , but just in case...
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

JamesF
Posts: 98
Joined: Thu Jul 10, 2008 7:14 pm

Re: Anyone using Lisp for web in production?

Post by JamesF » Thu Jul 10, 2008 10:45 pm

After the obligatory power outage takes down everything in the data center, you don't want to be faced with restarting oodles of Lisp-based applications all by hand.
No, no I certainly don't. I've refined it to a pretty quick and simple process, but it'll be automated before long. The reason it hasn't happened yet, and the reason it will before long, oddly enough, are the same: professional experience. Outages like that are unusual, and the Linux platform is nice and stable, so the actual urgency is low. However, I know from painful experience that it's easy to get complacent and forget, which bites hard when you do reboot the thing and find that nothing came back up. Also, the last thing you need to worry about is whether you remember to start up everything, and whether you started it all in the right order.

This last point is another one that can bite, too: getting things to start in the right order, when the dependencies are on different hosts. That's a case of getting the startup script to confirm that the dependency is running before it starts up its own application.

Monitoring and alerting systems are the next layer to go on, followed by performance monitoring and analysis... actually, all this infra stuff will probably be more useful to other people than the application code itself.

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: Anyone using Lisp for web in production?

Post by findinglisp » Thu Jul 10, 2008 10:56 pm

JamesF, yup, exactly.
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

Post Reply