RSS feed

Whatever is on your mind, whether Lisp related or not.
AlexPaes
Posts: 21
Joined: Sat Jun 28, 2008 3:38 am
Location: Lisbon, Portugal

Re: RSS feed

Post by AlexPaes » Tue Mar 10, 2009 11:37 am

Hi, i also could benefit from a simple RSS feed so i leave some more rss plugins i found on phpBB's mod site:

RSS Feed 2.0
RSS Syndication mod
Simple RSS mod for phpBB3
Simple syndication
Smartfeed

now it's a matter of finding the most convenient one, i will leave that to you guys. I will, however, suggest that if anyone has had any experience with any of these mods to go ahead and share it.
CL-USER> (setf *boss* (make-instance 'smart-person))
NIL
CL-USER>

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: RSS feed

Post by findinglisp » Wed Mar 11, 2009 12:34 pm

AlexPaes wrote:now it's a matter of finding the most convenient one, i will leave that to you guys. I will, however, suggest that if anyone has had any experience with any of these mods to go ahead and share it.
Thanks, Alex. That's helpful. I had done some research a couple of months ago, too, but the main thing stopping me from moving forward is that I found three or four different packages without having any one of them be a clear winner. The problem, I'm finding, with phpBB mods is that you basically have to hack them into the phpBB source at various different levels, which makes it a royal pain to try three or four of them to see which you like best.

Thus, if anybody has any experience with this stuff, as Alex said, it would be very helpful for me so that I don't spend any time trying out something that's a known dud.
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

kroger
Posts: 12
Joined: Mon Jul 28, 2008 2:38 am

Re: RSS feed

Post by kroger » Mon Jun 08, 2009 2:46 pm

Well, I really want to have rss on lispforum (I always miss discussions because I don't check the webpage often). How can I help you with that? I can install phpBB here, try the different plugins and send you a recipe of how to install the best one. Do you think that would be helpful?

findinglisp
Posts: 447
Joined: Sat Jun 28, 2008 7:49 am
Location: Austin, TX
Contact:

Re: RSS feed

Post by findinglisp » Tue Jun 09, 2009 8:50 am

kroger wrote:Well, I really want to have rss on lispforum (I always miss discussions because I don't check the webpage often). How can I help you with that? I can install phpBB here, try the different plugins and send you a recipe of how to install the best one. Do you think that would be helpful?
Yea, that would actually be helpful. Unfortunately, with job and family activities, I'm just not having enough time to do the research. Every RSS package that I have looked at involves some amount of hacking the phpBB source to install, so I was sort of put off by that initially. There are also quite a few different ones that work more or less well, some that provide granularity at the individual forum level and some at the site level. I wanted something that would provide both site and forum level granularity, so somebody who was interested in Scheme only, say, could subscribe to just the Scheme forum feed, while somebody interested in the complete site could subscribe to the top-level. Anyway, if you have the time and the initiative to help, I would be grateful.
Cheers, Dave
Slowly but surely the world is finding Lisp. http://www.findinglisp.com/blog/

schoppenhauer
Posts: 99
Joined: Sat Jul 26, 2008 2:30 pm
Location: Germany
Contact:

Re: RSS feed

Post by schoppenhauer » Wed Jun 17, 2009 7:47 pm

Just for fun I hacked a little script that parses the "active posts" (which should contain all new ones I think) and puts them into some RSS-Feed (hopefully - actually I didnt try it out yet, but if it doesnt validate, it should be easy to adapt):

Code: Select all

(defun http-page-to-string (path)
  (let*
      ((data (make-array 65535
			 :adjustable nil))
       (response (trivial-http:http-get path))
       (stream (caddr response))
       (rd 0)
       (ret ""))
    (tagbody
       start
       (setf rd (read-sequence data stream))
       (setf ret (concatenate 'string
			      ret
			      (coerce (subseq data 0 rd) 'string)))
       (if (= rd 65535) (go start)))
    ret))

(defun get-entries (path)
  (prog ((scanner (cl-ppcre:create-scanner "<a href=\"([^\"]*)\" class=\"topictitle\">([^<]*)</a>"))
	(page (http-page-to-string path))
	(start 0)
	(ret nil))
       scan-it
       (multiple-value-bind
	     (regstart end groupstarts groupends)       
	   (cl-ppcre:scan scanner page :start start)
	 (cond (regstart
		(push (cons
		       (subseq page (elt groupstarts 0) (elt groupends 0))
		       (subseq page (elt groupstarts 1) (elt groupends 1)))
		      ret)
		(setf start end)
		(go scan-it))
	       (t (return (nreverse ret)))))))

(defun output-rss (entries)
  (with-output-to-string (s)
    (format s "<?xml version=\"1.0\" encoding=\"utf-8\"?>
<rss version=\"2.0\">
<channel>
    <title>lispforum</title>
    <link>http://www.lispforum.com/</link>
    <description>Lisp Forum active Posts</description>
    <image>
      <url>http://www.lispforum.com/styles/prosilver/imageset/lispforum-logo4.png</url>
      <title>LispForum</title>
      <link>http://www.lispforum.com/index.php</link>
    </image>")
    (let
	((scanner (cl-ppcre:create-scanner "t=([0-9]*)&")))
      (dolist (entry entries)
	(format s "<item><title>~a</title><link>~a</link><guid>~a</guid></item>"
		(cdr entry) (car entry)
		(multiple-value-bind
		      (st en grst gren)
		    (cl-ppcre:scan scanner (car entry))
		  (declare (ignore st en))
		  (subseq (car entry) (elt grst 0) (elt gren 0))))))
    (format s "</channel></rss>")))
 


(defun get-rss ()
  (output-rss (get-entries "http://www.lispforum.com/search.php?search_id=active_topics")))
It uses trivial-http and ppcre. Example:

Code: Select all

CL-USER> (get-rss)

"<?xml version=\"1.0\" encoding=\"utf-8\"?>
<rss version=\"2.0\">
<channel>
    <title>lispforum</title>
    <link>http://www.lispforum.com/</link>
    <description>Lisp Forum active Posts</description>
    <image>
      <url>http://www.lispforum.com/styles/prosilver/imageset/lispforum-logo4.png</url>
      <title>LispForum</title>
      <link>http://www.lispforum.com/index.php</link>
    </image><item><title>How Do You Teach Yourself Programming?</title><link>./viewtopic.php?f=20&t=157&sid=b0631e85f8b000488e3c069375992b68</link><guid>157</guid></item><item><title>Copying a multidimensional array</title><link>./viewtopic.php?f=2&t=383&sid=b0631e85f8b000488e3c069375992b68</link><guid>383</guid></item><item><title>Coercing to double-float</title><link>./viewtopic.php?f=2&t=384&sid=b0631e85f8b000488e3c069375992b68</link><guid>384</guid></item><item><title>Lisp Quiz #3</title><link>./viewtopic.php?f=32&t=381&sid=b0631e85f8b000488e3c069375992b68</link><guid>381</guid></item><item><title>Has anyone yet used Liskell?</title><link>./viewtopic.php?f=29&t=385&sid=b0631e85f8b000488e3c069375992b68</link><guid>385</guid></item><item><title>Which HTTP Server?</title><link>./viewtopic.php?f=2&t=376&sid=b0631e85f8b000488e3c069375992b68</link><guid>376</guid></item><item><title>sbcl + aserve</title><link>./viewtopic.php?f=2&t=375&sid=b0631e85f8b000488e3c069375992b68</link><guid>375</guid></item><item><title>Embedding PHP into Common Lisp</title><link>./viewtopic.php?f=2&t=382&sid=b0631e85f8b000488e3c069375992b68</link><guid>382</guid></item><item><title>Lisp and GPU computing</title><link>./viewtopic.php?f=2&t=380&sid=b0631e85f8b000488e3c069375992b68</link><guid>380</guid></item><item><title>Extending a simple array dynamically</title><link>./viewtopic.php?f=2&t=377&sid=b0631e85f8b000488e3c069375992b68</link><guid>377</guid></item><item><title>write-line</title><link>./viewtopic.php?f=29&t=378&sid=b0631e85f8b000488e3c069375992b68</link><guid>378</guid></item></channel></rss>"
CL-USER> 
In the links it also grabs session-id's (which could be replaced easily by additional regexing), as guid it uses the t-parameters of the url's.
Sorry for my bad english.
Visit my blog http://blog.uxul.de/

Harleqin
Posts: 71
Joined: Wed Dec 17, 2008 5:18 am
Location: Bonn, Germany

Re: RSS feed

Post by Harleqin » Thu Jun 18, 2009 7:18 am

Some quick comments/questions:

- You actually use TAGBODY, PROG and GO in high level code?
- Reading a page into a string should be possible much easier.
- HTML is not a regular language. Using regular expressions to parse it can only be a brittle hack. There are some very nice XML/HTML parser libraries available for CL.
"Just throw more hardware at it" is the root of all evil.
Svante

schoppenhauer
Posts: 99
Joined: Sat Jul 26, 2008 2:30 pm
Location: Germany
Contact:

Re: RSS feed

Post by schoppenhauer » Thu Jun 18, 2009 8:26 am

Harleqin wrote:You actually use TAGBODY, PROG and GO in high level code?
Unfortunately using "do", I couldnt find a possibility to do something until some condition holds (i.e. do it when it holds once again). And I dont like loop. And it is no high-level-code, because it uses the lowlevel read-sequence-stuff.[/quote]
Harleqin wrote:Reading a page into a string should be possible much easier.
I never used CL for HTTP-Access so far. This possibility seemed sufficient, but since I was sure that its possible easier, I put this part into an own function - feel free to modify it.
Harleqin wrote:HTML is not a regular language. Using regular expressions to parse it can only be a brittle hack.
Actually, downloading this page to generate an RSS-Feed from it is a hack itself. If I wanted to have something "clean", I would have gone down to the databases. But well, its a hack, but it works well.
Harleqin wrote:There are some very nice XML/HTML parser libraries available for CL.
I have tried s-xml and xmls. Both do not understand the entity "&bull;", and both dont finish to parse it. Also cxml has problems. I would like to have cl-dom, but ... I couldnt find a download-link. Actually, there are at least a dozen of XML-Libraries for Common Lisp out there, but none really working. I think the best possibility would be to just bind the official DOM-Parser (using some java-binding or so), should be rather trivial, but I think noone did this so far.
Sorry for my bad english.
Visit my blog http://blog.uxul.de/

Harleqin
Posts: 71
Joined: Wed Dec 17, 2008 5:18 am
Location: Bonn, Germany

Re: RSS feed

Post by Harleqin » Thu Jun 18, 2009 10:59 am

OK, a quick refactoring (untested):

Code: Select all

(defun http-page-to-string (path)
  (with-open-stream (in-stream (third (trivial-http:http-get path)))
    (with-output-to-string (out-stream)
      (loop
        (let ((line (read-line in-stream nil)))
          (when (null line) (return))
          (write-line line out-stream)))))
Note that this uses the simple LOOP.
"Just throw more hardware at it" is the root of all evil.
Svante

Harleqin
Posts: 71
Joined: Wed Dec 17, 2008 5:18 am
Location: Bonn, Germany

Re: RSS feed

Post by Harleqin » Thu Jun 18, 2009 1:28 pm

Regarding the HTML parsing: I haven't done this before, but I thought of Closure HTML.
"Just throw more hardware at it" is the root of all evil.
Svante

schoppenhauer
Posts: 99
Joined: Sat Jul 26, 2008 2:30 pm
Location: Germany
Contact:

Re: RSS feed

Post by schoppenhauer » Thu Jun 18, 2009 6:11 pm

Harleqin wrote:Regarding the HTML parsing: I haven't done this before, but I thought of Closure HTML.
Try it :D (Its possible, but annoying)
Sorry for my bad english.
Visit my blog http://blog.uxul.de/

Post Reply