CL and a newbie trying to start ... :)

[mcc]

Hi,

I am using SBCL on AMD64 Gentoo Linux and Beaglebone Black (embedded Device ARMv7 OMAP8?TI), also Gentoo Linux).
Also I am using vim. I haven't done simv, quicklisp etc, because they doesn't want to work for me (compile/installation errors).
But this are spare problems for later decades...

Currently I am on the way to evaluate CL for me. Its "being different" makes me curious.

For the first, I want a "Shortwave broadcaster schedule" available from HFCC into a simplyfied database
(CL internal, no sqlite, postgres engine etc...).

The input file, which is ASCII and which I dont want to alter looks like this:

Code: Select all

; B14 ALL 25-mar-2015
; Global HF Schedule
; Processed on 25-mar-2015 at 14:50UTC
;                                 Timestamp: 1427295016
;----+----+----+------------------------------+---+----+-------+---+---+-------+------+------+-+-----+----------+---+---+---+-----+-+-----+-----+-----+-------
;FREQ STRT STOP CIRAF ZONES                    LOC POWR AZIMUTH SLW ANT DAYS    FDATE  TDATE MOD AFRQ LANGUAGE   ADM BRC FMO REQ# OLD ALT1 ALT2  ALT3  NOTES
;----+----+----+------------------------------+---+----+-------+---+---+-------+------+------+-+-----+----------+---+---+---+-----+-+-----+-----+-----+-------
 3185 0000 1300 4,9                            WRB  100 45        0 902 1234567 271014 290315 D       Eng        USA WRB FCC   430                     
 3195 0100 0400 2-4                            WRB  100 0         0 805 1234567 261014 011114 D       Eng        USA WRB FCC   431                     
 3195 0100 0400 2-4                            WRB  100 0         0 805 1234567 080315 290315 D       Eng        USA WRB FCC   432                     
 3195 0200 0500 2-4                            WRB  100 0         0 805 1234567 021114 070315 D       Eng        USA WRB FCC   433           

The input varies sometimes:

Code: Select all

5960 1900 2000 28E,29                         NAU  100 90        0 146 7       261014 310115 D  9600 Mul        D   CHW MBR 17321                     Z.351
 5960 2000 2100 27,28W                         CER  150 310       0 146 1234567 261014 290315 D       Eng        ALB CRI RTC  6328                     
 5960 2100 2200 27,28W                         CER  150 310       0 146 1234567 261014 290315 D       Eng        ALB CRI RTC  6329                     
 5960 2200 2300 43E,44                         SLA  250 60       25 211 1234567 280115 290315 D  7900 Eng        OMA BBC BAB 17150                     ENASE_F
 5960 2300 2400 5,8,9,11N,17,18,27,28W         EMR  500 310       0 215 1234567 261014 290315 D  9000 Eng        TUR TRT TRT  7577                     
 5960 2330 1800 42N                            URU  100 0         0 925 1234567 261014 290315 D       Zho        CHN CNR RTC  6330                     
 5965 0445 1015 41N                            JAM   50 0         0 700 1234567 261014 290315 D  6075 Mul        IND AIR AIR 11073                     
 5965 0900 1000 44NE,45N                       BEI  150 95        0 206 1234567 261014 290315 D  7700 Chn        CHN CRI RTC  6331     

Since I am a newbie to LISP (not to programming as such), I have a couple of questions here...

1.What is the recommended way to read this file with LISP?
2.How can I the columns into plists (property list) or should I use a hash table (how).
3.How can I store the stuff for later use?
4.Is it possible to read the line

;FREQ STRT STOP CIRAF ZONES LOC POWR AZIMUTH SLW ANT DAYS FDATE TDATE MOD AFRQ LANGUAGE ADM BRC FMO REQ# OLD ALT1 ALT2 ALT3 NOTES

as a setup information for the plist/hashtable, so that future changes of the input may be taken into account without the need of changing the Lisp code as such?

Thank you very much in advance for any help!
Best regards,
mcc

[edgar-rft]

Hi mcc!

rft = Radio-/Television Broadcast Technician (german abbreviation for Rundfunk-/Fernsehtechniker, don't try to find an english sense in it)

I don't have really much time at the moment, but shortwave HAM and radio is still a favourite hobby and I would be interested to help developing a database for the HFCC shortwave schedules. Common Lisp is not so terribly different when it comes to text processing. The general programming patterns for reading and writing text files are described in:

• Practical Common Lisp, Chapter 14: Files and File I/O

Also lists vs. hash-tables is the same like in nearly all other programming languages, hash-tables are much faster than property-lists with several thousand key/value pairs.

• Practical Common Lisp, Chapter 11: Collections, Section Hash-Tables

Perl-like regular expressions to split the lines and test the tokens are available via Edi Weitz's PPCRE, and the most easy way to install PPCRE is Xach Beane's Quicklisp. A general Common Lisp information source is the CLiki = Common Lisp Wiki.

I already had written a converter in Emacs Lisp for the shortwave schedules from the Bi Newsletter for DXers, but it turned out that the main work is to find out what you really can receive at home (what usually is only a tiny fraction of that list), that only can be found out by tuning and listening to a shortwave receiver, what takes lots of time. That's the main reason why my own database is still unfinished and incomplete.

Do you already have any particular application in mind for what you need the data base or is this a hobby, too?

...and in case you haven't noticed, the A15 is already out...

For people who don't know what we're talking about, see the "High Frequency Co-ordination Conference" (HFCC) Public Data Files archive. A15 means first half year of 2015. I just have downloaded the A15 zip-file and will try to write some simple Common Lisp code to read the file line-by-line to give an example how the whole thing could get started.

- edgar

[edgar-rft]

As promised, some whacky text file examples. I've used only built-in Common Lisp functions, no external libraries needed.

The basic construct to read a line from a file and return it as a string uses READ-LINE like this::

Code: Select all

(read-line stream nil 'eof)

stream must be a Common Lisp file-stream from that the line is read, nil and 'eof mean "dont't signal an error at end-of-file, return 'eof instead".

LOOP has its own language (it was an attempt to make writing loops easier for beginners), Practical Common Lisp has several chapters about LOOP.

(loop for variable = value-form ...) means "create a local variable, and update its value by evaluating value-form with every iteration anew".

(loop until condition ...) means "loop until condition is true".

(... finally final-form) means "evaluate final-form after the loop has terminated".

The loop stops at the first line with more than 4 characters, where the first five characters are ";FREQ", or at the end of the file if such a line could not be found. If a line was found, the complete line is returned, if no line was found 'eof is returned:

Code: Select all

(defun head-line (stream)
  "Return the headline as string."
  ;; skip lines until a line starts with ";FREQ"
  (loop for line = (read-line stream nil 'eof)
        until (if (stringp line)
                  (and (> (length line) 4)
                       (string= ";FREQ" (subseq line 0 5)))
                  (eq line 'eof))  ; break at end-of-file
        finally (return line)))

The same again, but this time all empty lines and all lines starting with a semicolon are skipped:

Code: Select all

(defun data-line (stream)
  "Return the next data line as string."
  (loop for line = (read-line stream nil 'eof)
        until (if (stringp line)
                  (and (> (length line) 0)          ; skip empty lines
                       (char/= (char line 0) #\;))  ; skip comment-lines
                  (eq line 'eof))                   ; break at end-of-file
        finally (return line)))

WITH-OPEN-FILE opens a file for reading or writing (or both) and implicitely closes the file either after the code has regularly finished and also if an error happens. This simplifies file-handling a lot. If no :direction argument is specified, the file is opened for :direction :input automatically. FORMAT is a second example for a built-in mini-language, where the format-string spefifies how the arguments shall be formatted in the output stream. Practical Common Lisp has lots of examples.

Code: Select all

;; open the input file
(with-open-file (hfcc-input-stream "/home/edgar/hfcc/A15all00.TXT")
  ;; open the output file
  (with-open-file (hfcc-output-stream "/home/edgar/hfcc/A15_filter.TXT"
                                      :direction :output
                                      :if-exists :supersede)  ; clobber old files
    ;; read the headline from the input stream
    (let ((head-line (head-line hfcc-input-stream)))
      (unless (eq head-line 'eof)
        ;; print the head-line to the output stream
        (format hfcc-output-stream "~a~%" head-line)
        ;; print all data-lines to the output-stream
        (loop for data-line = (data-line hfcc-input-stream)
              until (eq data-line 'eof)
              do (format hfcc-output-stream "~a~%" data-line))))))

This code writes a new file A15_filter.TXT containing only the head-line and all data-lines, while all empty lines and comment-lines are removed from the new file.

Another very common programming pattern seen above is using the same symbol-name for a variable as well as for a related function. This way it's not necessary to allocate a second symbol to temporarily store the output value of the function. LET binds the return value of the HEAD-LINE function to a local HEAD-LINE variable. The first HEAD-LINE after the LET is the variable, while (HEAD-LINE hfcc-input-stream) is the function call:

Code: Select all

(let ((head-line (head-line hfcc-input-stream)))
  (unless (eq head-line 'eof)
    ;; execute the code here only if the value of the HEAD-LINE variable is not 'eof ))

As you can see, text processing is no real science with Common Lisp.

Designing a database out of the lines is a different thing, where splitting the lines into tokens is the easiest part...

- edgar

[mcc]

Hi edgar,

before I really reply to your great postings let me say a
BIG THANK YOU!
Great!

(By the way I: Quicklisp works now...sbc-1.2.9/1.2.10 had done three tests which were marked as "win32" on Linux,
which failed and therefore "sb-bdsd-sockets" was not installed, which in turn is needed by quicklisp. After commenting
out those sbcl get installed fully and Quicklisp runs now. I sent a report to the bug-list for sbcl.)

(By the way II: A15 from the link gave me a 404...?!?!)

Have a nice Sunday!
Best regards,
mcc

[mcc]

Hi,

...now...

Currently I have a "design problem":

As the function, which reads the text, is basically a loop, the portion, which parses the text have to be inside
the loop...which in turn makes the whole program a single loop (and not very readable).

I would like a reader-function which does not evaluate the text in any way and -- if called -- only returns
one (the next) line of text.
Another way to implement this (an which I dont like that much) is a kind of callback function, which
is called from the reader-loop to parse the text. But this is again more like the big loop from the
beginning...

How can I do the reader-loop which, when called, only return "the next" line in Lisp?

Best regards,
mcc

[mcc]

Hi Edgar,

I googled a lot and assembled some lisp code from bits and pieces I found.
And -- as exspected -- it does not work: It loops forever.

The code is:

Code: Select all

(let (stream)
  (defun readsched (schedname)
    (if (eq stream nil )
      (with-open-file (stream schedname)))
    (read-line stream nil 'foo)))

(defun readme () 
  (loop for line =( readsched "a10.txt" )
        until (eq line 'foo)
        do (print line)))

The code compiles ok and a10.txt is in same directory as the code.
When started with

Code: Select all

(readme)

it goes into an endless loop.
Unfortunately I dont find the problem/bug...
What did I wrong here?

Best regards,
mcc

[edgar-rft]

I googled a lot and assembled some lisp code from bits and pieces I found.

Sorry, but this is a very bad method to learn Common Lisp.

There are several Lisp dialects (today mainly Common Lisp, Scheme, and Emacs Lisp) which have a common ancestor ("Lisp 1.5" from the year 1958), but differ very much in their behaviour. If you ask Google for Common Lisp programming problems you will get answers from lots of people who are confused about Common Lisp, because at school they have learned Scheme, what is significantly different. This is a very often observed phaenomenon. The majority of Common Lisp answers in the internet are pure bullshit. I've tried myself to learn Common Lisp that way and had to give up before getting mad.

There are two books where you can learn a lot more (and faster) than from Google:

• David Touretzky: Common Lisp: A Gentle Introduction to Symbolic Computation - not only for for people who have never programmed a computer before. David Touretzky explains Common Lisp from the very basics up to a level where you can write your own programs. The book doesn't cover object-oriented programming with Common Lisp.

• Peter Seibel: Practical Common Lisp - more practical examples than theory, but requires that you have a fundamental understanding how computer programming languages work. The code examples are from very easy at the beginning up to pretty advanced at the end of the book.

The CLiki lists several other books, but the two above have helped me most. Common Lisp is a rather complex language where every part of the language interacts with every other part, and all that cannot be learned within a few days. The good news is that we don't need the whole Common Lisp language to create a simple database, and we can look at more complicated things later.

One reason for the eternal loop is a parenthesis error, at the end of (with-open-file (stream schedname))) there is one closing paren too many:

Code: Select all

(let (stream)
  (defun readsched (schedname)
    (if (eq stream nil )
      (with-open-file (stream schedname)))
    (read-line stream nil 'foo)))

The closing paren must be moved to the end of the code, but even then the code wouldn't work. WITH-OPEN-FILE binds the STREAM via its own LET inside its body, what shadows the outer LET variable at the top of the code. Effectively the code would work like this:

Code: Select all

(let (stream)
  ;; (with-open-file (stream schedname)) produces the following code:
  (let ((stream (open schedname)))
    ;; inside here STREAM is a file-stream, but the outer STREAM is still NIL
    (unwind-protect
      ;; there was no BODY argument after (with-open-file (stream schedname))
      ;; so the BODY is NIL, what effectively does nothing
      nil
      ;; close the file-stream after evaluating the BODY argument
      (close schedname))))

The outer STREAM variable, established by LET at the top of the code, never gets the value of the file-stream, because the inner STREAM variable, that is automatically established by WITH-OPEN-FILE, shadows the outer STREAM variable. Even worse, the file-stream gets closed before the code does anything with it. READ-LINE in subsequent calls would not work because the outer STREAM variable is still NIL, what means that the file would be opened and closed over and over again.

The code above gives you an idea how you could open and read from the file-stream by hand:

Code: Select all

(let (stream)
  (defun readsched (schedname)
    (if (eq stream nil)
        (setf stream (open schedname))
        (read-line stream nil 'foo))))

With the first call the file gets opened and the file-stream is stored in the STREAM variable. With every subsequent call one line is read from the file and returned as a string, or 'foo is returned at the end of the file.

But now the calling code is responsible to close the file-stream after the last line has been read or it would produce a memory leak. This is no good design because the code that is responsible for opening and closing a file should be in one place and not littered elsewhere. Also this code would not have the automatic error handling that UNWIND-PROTECT provides in the expansion of WITH-OPEN-FILE.

Please look at the File-I/O chapter of Practical Common Lisp where all this is explained in detail.

Currently I have a "design problem":

As the function, which reads the text, is basically a loop, the portion, which parses the text have to be inside
the loop...which in turn makes the whole program a single loop (and not very readable).

The function(s), which do the parsing, must be called from inside the loop, but can be defined anywhere else. The only thing you must care about is that the variable holding the file-stream stays valid until the last line has been read from the file. Later today I will try to write a few examples how this would work.

- edgar

[mcc]

Hi edgar,

Thanks a lot for all your effort to get my struggling into walking !

I wanted to reply with a BIG THANK YOU before I am fully understanding,
what the problems of my code are and how to do it correctly.

I will reply fully, when I think I have at least understood enough to reply
meaningfully...

Best regards,
mcc

[mcc]

Hi edgar,

I finally got something stiched together, which
at least does not throw me into the debugger and
prints all lines of the file onto the terminal.

What do you think?

Code: Select all

(let (stream line)
  (defun readsched (schedname)
    (if (eq stream nil)
        (setf stream (open schedname)))
    (setf line (read-line stream nil nil))
    (if (eql line nil)
      (close stream))
    line
  ))

(let (text)
(defun reader (schedname)
  (loop 
    (setf text (readsched schedname))
    (print text)
    (when (eql text nil) (return nil)))))

Best regards
mcc