Help me on Tokinzer

Discussion of Common Lisp

Help me on Tokinzer

Postby omarasl » Sun Dec 02, 2012 3:54 am

Hello every body ,
I am very beginner in lisp , I need your help
I am trying to do some thing like dictionary .
I want an example of how I can take sentence like this "this is an example"
then take each word "this" "is" "an" "example"
thank you very much
omarasl
 
Posts: 12
Joined: Sun Nov 25, 2012 6:19 am

Re: Help me on Tokinzer

Postby Goheeca » Sun Dec 02, 2012 4:19 am

Code: Select all
(defun split (string delimiters &optional (empty-strings nil))
  (loop for start = 0 then (1+ finish)
        for finish = (position-if #'(lambda (elem) (member elem delimiters)) string :start start)
        for token = (subseq string start finish)
        when (or empty-strings (/= 0 (length token))) collecting token
        until (null finish)))

Usage:
Code: Select all
(split "Hello world, people!" '(#\Space #\, #\!))

I've edited/improved a snippet on Rosetta Code. The part with empty-strins is unavailing it could have an utilization only if I would collect tokens with the delimiters which are delimiting those certain tokens.
cl-2dsyntax is my attempt to create a Python-like reader. My mirror of CLHS (and the dark themed version).
User avatar
Goheeca
 
Posts: 198
Joined: Thu May 10, 2012 12:54 pm

Re: Help me on Tokinzer

Postby stackman » Sun Dec 02, 2012 5:08 am

As far as I know, there is no ready-to-use splitting function in the cl standard.
However you can use the cl-split-sequence library, available through quicklisp.
Assuming quicklisp is installed,
Code: Select all
CL-USER> (ql:quickload :split-sequence)
To load "split-sequence":
  Load 1 ASDF system:
    split-sequence
; Loading "split-sequence"

(:SPLIT-SEQUENCE)
CL-USER> (use-package :split-sequence)

T
CL-USER> (split-sequence #\space "This is an example.")
("This" "is" "an" "example.")
19
CL-USER>

The code for split-sequence can be read online at the following url:
http://common-lisp.net/project/clbuild/ ... uence.lisp
stackman
 
Posts: 28
Joined: Sat Oct 06, 2012 5:44 am


Return to Common Lisp

Who is online

Users browsing this forum: No registered users and 2 guests

cron