ANN: mark-and-render

[JamesF]

Yes, it's yet another markup package for HTML. Because that's what the world needs most right now.

What I needed (OK, wanted) was an engine that would render HTML from wikimarkup. The back-story's a bit involved, but I started with s-expressions, found them unwieldy, realised I was nicely placed to separate the parsing and rendering, so worked out how to parse markup into an AST. Then I realised I had the general-purpose engine that I wanted for other web-based work, so ripped out the wiki-specific bit and in the process proved that it actually is as easily extensible as I'd hoped. It also helpfully auto-detects whether you're using s-expressions or wikimarkup in a given batch of input, and selects the parser accordingly.

The upshot is an engine that parses the incoming markup into an AST, then renders the AST as HTML, and has a mechanism for adding custom bits of rendering functionality for, say, adding an image link. That mechanism is in place and in use, but needs one more step and a bit of documentation before it's ready for the insertion of arbitrary {macro}content{/macro} processes. It also actually parses the input, rather than using regexes to transform things, which is a (not-so-)small point of pride

Version 0.1 is on github now, in its basic working form.

Credit goes to Peter Seibel, because I've shamelessly swiped the rendering code from Practical Common Lisp. Anything that's broken is entirely my fault.

[nuntius]

Thanks for sharing.

There have been several lisp-based wikis over the years, but none seem to have an "import from mediawiki" option...

[JamesF]

I could look into a Mediawiki importer, if that would be of use, and/or try writing a parser for that style of markup. However...

I'm a bit restricted in just what I can do with respect to an actual wiki, as I'm employed by a company that makes, sells and supports one - the potential conflict of interest is a bit of a killer.
The agreement we've reached is that I'm free to continue developing one in CL, and I can host public-facing instances for friends and family, so I'm clear to run one in the manner of Cliki, but they'll get very upset if I distribute the sources - that being where the greatest potential for conflict lies. This is partly why I separated this library into its own project, overriding the hyperlink function from within the wiki. It was actually a nice confirmation that I was on the right track with modularity, because the surgery involved was pretty minor.

So, while I could write an importer, and even run a public-facing wiki that was able to import, I'd be skating on thin ice to share the sourcecode for it. That said, it shouldn't be too hard to hack the wikimarkup parser to accept Mediawiki markup, and to modify the renderer to output a different kind of markup instead of HTML. I wonder how much trouble I'd get into for doing that myself and releasing it as part of mark-and-render?

[Warren Wilkinson]

By s-expr you actually mean 'a string whose contents resemble an s-expr' right? (or do you mean an s-expr built from lists).

A little web-app that could take one wiki's syntax and translate it into anothers might be an interesting project. A bit academic maybe, but it might save somebodies bacon a couple of times.

Does the wiki your company produces run in Lisp?

[JamesF]

By s-expr you actually mean 'a string whose contents resemble an s-expr' right? (or do you mean an s-expr built from lists).

Erm, kinda both. A section of cl-markup is a string whose contents bear a remarkable resemblance to a serialised s-expr; the "parser" uses read-from-string to transform that into the AST/s-expr that's handed to the renderer, in the classic cheat of using the Lisp reader in lieu of writing an actual parser. It passes the duck-test for me, but I'm willing to stand corrected. It may come down to the question of whether it's an s-expr when it's represented as a stream of characters, or only after it's been READ (READed?) - to my mind, the input's an s-expr and the result is an AST, hence me phrasing it that way.

A little web-app that could take one wiki's syntax and translate it into anothers might be an interesting project. A bit academic maybe, but it might save somebodies bacon a couple of times.

I think you'd be surprised at how often people find the need to import content into one wiki that was generated in another; we actually have a specialised utility for doing exactly that, though obviously its output is restricted to our own product. There's an excellent chance that I'll use this as a prompt to learn Clojure, because this approach might just be able to do a better job of being a universal wiki translator than the current one, and something that does run within the JVM is something that stands a chance of being worth adopting by my employer.
It's also worth noting that it wouldn't have to be a web-app - as long as the transformation engine is sufficiently well isolated and has a sufficiently useful and flexible approach to I/O, it can be embedded in all sorts of things, from a web-app that takes input from a text-box and displays the equivalent markup for another wiki, to a command-line app that walks the content of one wiki and uses the appropriate output method (write to file, remote API, direct SQL injection) to siphon it into another.

Does the wiki your company produces run in Lisp?

I wish! If it did, I'd be contributing directly to their codebase, but they're a Java house, so in that context I'm working as a mere system/network administrator. That's actually what mitigates the conflict of interest, because there's only so far you can go in terms of porting ideas between the two languages before you run into idioms in one that just don't really make sense in the other. That, and I'm one of three people in the entire outfit who are actually fluent in CL - and all of us are sysadmins. Go figure...