Page 1 of 1
Help me on Tokinzer
Posted: Sun Dec 02, 2012 3:54 am
by omarasl
Hello every body ,
I am very beginner in lisp , I need your help
I am trying to do some thing like dictionary .
I want an example of how I can take sentence like this "this is an example"
then take each word "this" "is" "an" "example"
thank you very much
Re: Help me on Tokinzer
Posted: Sun Dec 02, 2012 4:19 am
by Goheeca
Code: Select all
(defun split (string delimiters &optional (empty-strings nil))
(loop for start = 0 then (1+ finish)
for finish = (position-if #'(lambda (elem) (member elem delimiters)) string :start start)
for token = (subseq string start finish)
when (or empty-strings (/= 0 (length token))) collecting token
until (null finish)))
Usage:
Code: Select all
(split "Hello world, people!" '(#\Space #\, #\!))
I've edited/improved a snippet on
Rosetta Code. The part with
empty-strins is unavailing it could have an utilization only if I would collect tokens with the delimiters which are delimiting those certain tokens.
Re: Help me on Tokinzer
Posted: Sun Dec 02, 2012 5:08 am
by stackman
As far as I know, there is no ready-to-use splitting function in the cl standard.
However you can use the cl-split-sequence library, available through
quicklisp.
Assuming quicklisp is installed,
Code: Select all
CL-USER> (ql:quickload :split-sequence)
To load "split-sequence":
Load 1 ASDF system:
split-sequence
; Loading "split-sequence"
(:SPLIT-SEQUENCE)
CL-USER> (use-package :split-sequence)
T
CL-USER> (split-sequence #\space "This is an example.")
("This" "is" "an" "example.")
19
CL-USER>
The code for split-sequence can be read online at the following url:
http://common-lisp.net/project/clbuild/ ... uence.lisp