RSS feed

[AlexPaes]

Hi, i also could benefit from a simple RSS feed so i leave some more rss plugins i found on phpBB's mod site:

RSS Feed 2.0
RSS Syndication mod
Simple RSS mod for phpBB3
Simple syndication
Smartfeed

now it's a matter of finding the most convenient one, i will leave that to you guys. I will, however, suggest that if anyone has had any experience with any of these mods to go ahead and share it.

[findinglisp]

now it's a matter of finding the most convenient one, i will leave that to you guys. I will, however, suggest that if anyone has had any experience with any of these mods to go ahead and share it.

Thanks, Alex. That's helpful. I had done some research a couple of months ago, too, but the main thing stopping me from moving forward is that I found three or four different packages without having any one of them be a clear winner. The problem, I'm finding, with phpBB mods is that you basically have to hack them into the phpBB source at various different levels, which makes it a royal pain to try three or four of them to see which you like best.

Thus, if anybody has any experience with this stuff, as Alex said, it would be very helpful for me so that I don't spend any time trying out something that's a known dud.

[kroger]

Well, I really want to have rss on lispforum (I always miss discussions because I don't check the webpage often). How can I help you with that? I can install phpBB here, try the different plugins and send you a recipe of how to install the best one. Do you think that would be helpful?

[findinglisp]

Well, I really want to have rss on lispforum (I always miss discussions because I don't check the webpage often). How can I help you with that? I can install phpBB here, try the different plugins and send you a recipe of how to install the best one. Do you think that would be helpful?

Yea, that would actually be helpful. Unfortunately, with job and family activities, I'm just not having enough time to do the research. Every RSS package that I have looked at involves some amount of hacking the phpBB source to install, so I was sort of put off by that initially. There are also quite a few different ones that work more or less well, some that provide granularity at the individual forum level and some at the site level. I wanted something that would provide both site and forum level granularity, so somebody who was interested in Scheme only, say, could subscribe to just the Scheme forum feed, while somebody interested in the complete site could subscribe to the top-level. Anyway, if you have the time and the initiative to help, I would be grateful.

[schoppenhauer]

Just for fun I hacked a little script that parses the "active posts" (which should contain all new ones I think) and puts them into some RSS-Feed (hopefully - actually I didnt try it out yet, but if it doesnt validate, it should be easy to adapt):

Code: Select all

(defun http-page-to-string (path)
  (let*
      ((data (make-array 65535
          :adjustable nil))
       (response (trivial-http:http-get path))
       (stream (caddr response))
       (rd 0)
       (ret ""))
    (tagbody
       start
       (setf rd (read-sequence data stream))
       (setf ret (concatenate 'string
               ret
               (coerce (subseq data 0 rd) 'string)))
       (if (= rd 65535) (go start)))
    ret))

(defun get-entries (path)
  (prog ((scanner (cl-ppcre:create-scanner "<a href=\"([^\"]*)\" class=\"topictitle\">([^<]*)</a>"))
   (page (http-page-to-string path))
   (start 0)
   (ret nil))
       scan-it
       (multiple-value-bind
        (regstart end groupstarts groupends)       
      (cl-ppcre:scan scanner page :start start)
    (cond (regstart
      (push (cons
             (subseq page (elt groupstarts 0) (elt groupends 0))
             (subseq page (elt groupstarts 1) (elt groupends 1)))
            ret)
      (setf start end)
      (go scan-it))
          (t (return (nreverse ret)))))))

(defun output-rss (entries)
  (with-output-to-string (s)
    (format s "<?xml version=\"1.0\" encoding=\"utf-8\"?>
<rss version=\"2.0\">
<channel>
    <title>lispforum</title>
    <link>http://www.lispforum.com/</link>
    <description>Lisp Forum active Posts</description>
    <image>
      <url>http://www.lispforum.com/styles/prosilver/imageset/lispforum-logo4.png</url>
      <title>LispForum</title>
      <link>http://www.lispforum.com/index.php</link>
    </image>")
    (let
   ((scanner (cl-ppcre:create-scanner "t=([0-9]*)&")))
      (dolist (entry entries)
   (format s "<item><title>~a</title><link>~a</link><guid>~a</guid></item>"
      (cdr entry) (car entry)
      (multiple-value-bind
            (st en grst gren)
          (cl-ppcre:scan scanner (car entry))
        (declare (ignore st en))
        (subseq (car entry) (elt grst 0) (elt gren 0))))))
    (format s "</channel></rss>")))
 


(defun get-rss ()
  (output-rss (get-entries "http://www.lispforum.com/search.php?search_id=active_topics")))

It uses trivial-http and ppcre. Example:

Code: Select all

CL-USER> (get-rss)

"<?xml version=\"1.0\" encoding=\"utf-8\"?>
<rss version=\"2.0\">
<channel>
    <title>lispforum</title>
    <link>http://www.lispforum.com/</link>
    <description>Lisp Forum active Posts</description>
    <image>
      <url>http://www.lispforum.com/styles/prosilver/imageset/lispforum-logo4.png</url>
      <title>LispForum</title>
      <link>http://www.lispforum.com/index.php</link>
    </image><item><title>How Do You Teach Yourself Programming?</title><link>./viewtopic.php?f=20&amp;t=157&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>157</guid></item><item><title>Copying a multidimensional array</title><link>./viewtopic.php?f=2&amp;t=383&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>383</guid></item><item><title>Coercing to double-float</title><link>./viewtopic.php?f=2&amp;t=384&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>384</guid></item><item><title>Lisp Quiz #3</title><link>./viewtopic.php?f=32&amp;t=381&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>381</guid></item><item><title>Has anyone yet used Liskell?</title><link>./viewtopic.php?f=29&amp;t=385&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>385</guid></item><item><title>Which HTTP Server?</title><link>./viewtopic.php?f=2&amp;t=376&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>376</guid></item><item><title>sbcl + aserve</title><link>./viewtopic.php?f=2&amp;t=375&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>375</guid></item><item><title>Embedding PHP into Common Lisp</title><link>./viewtopic.php?f=2&amp;t=382&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>382</guid></item><item><title>Lisp and GPU computing</title><link>./viewtopic.php?f=2&amp;t=380&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>380</guid></item><item><title>Extending a simple array dynamically</title><link>./viewtopic.php?f=2&amp;t=377&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>377</guid></item><item><title>write-line</title><link>./viewtopic.php?f=29&amp;t=378&amp;sid=b0631e85f8b000488e3c069375992b68</link><guid>378</guid></item></channel></rss>"
CL-USER>


In the links it also grabs session-id's (which could be replaced easily by additional regexing), as guid it uses the t-parameters of the url's.

[Harleqin]

Some quick comments/questions:

- You actually use TAGBODY, PROG and GO in high level code?
- Reading a page into a string should be possible much easier.
- HTML is not a regular language. Using regular expressions to parse it can only be a brittle hack. There are some very nice XML/HTML parser libraries available for CL.

[schoppenhauer]

You actually use TAGBODY, PROG and GO in high level code?

Unfortunately using "do", I couldnt find a possibility to do something until some condition holds (i.e. do it when it holds once again). And I dont like loop. And it is no high-level-code, because it uses the lowlevel read-sequence-stuff.[/quote]

Reading a page into a string should be possible much easier.

I never used CL for HTTP-Access so far. This possibility seemed sufficient, but since I was sure that its possible easier, I put this part into an own function - feel free to modify it.

HTML is not a regular language. Using regular expressions to parse it can only be a brittle hack.

Actually, downloading this page to generate an RSS-Feed from it is a hack itself. If I wanted to have something "clean", I would have gone down to the databases. But well, its a hack, but it works well.

There are some very nice XML/HTML parser libraries available for CL.

I have tried s-xml and xmls. Both do not understand the entity "•", and both dont finish to parse it. Also cxml has problems. I would like to have cl-dom, but ... I couldnt find a download-link. Actually, there are at least a dozen of XML-Libraries for Common Lisp out there, but none really working. I think the best possibility would be to just bind the official DOM-Parser (using some java-binding or so), should be rather trivial, but I think noone did this so far.

[Harleqin]

OK, a quick refactoring (untested):

Code: Select all

(defun http-page-to-string (path)
  (with-open-stream (in-stream (third (trivial-http:http-get path)))
    (with-output-to-string (out-stream)
      (loop
        (let ((line (read-line in-stream nil)))
          (when (null line) (return))
          (write-line line out-stream)))))


Note that this uses the simple LOOP.

[Harleqin]

Regarding the HTML parsing: I haven't done this before, but I thought of Closure HTML.

[schoppenhauer]

Regarding the HTML parsing: I haven't done this before, but I thought of Closure HTML.

Try it (Its possible, but annoying)