Problem with quote chars in filenames

Discussion of Common Lisp
Post Reply
eeeickythump
Posts: 9
Joined: Sun Jan 03, 2010 2:04 am

Problem with quote chars in filenames

Post by eeeickythump » Sun Jan 10, 2010 11:53 pm

Hi everyone
I have a problem with a program I have written that parses files for the game "Morrowind".
If anyone is interested the repository is at http://bitbucket.org/eeeickythump/esper/
Basically the problem is with filenames that have a single quote in them. Common lisp's pathname functions seem to insist on mangling these filenames by removing the quote (which is a perfectly legal character in Windows filesystems).

Using Clozure CL 1.4 on WinXP: (because the program must run on windows)

Code: Select all

CL-USER> (probe-file "c:/Morrowind/Data Files/Beryl's_Head_Replacer_v1.0.esm")  ; this is the actual filename
NIL
CL-USER> (probe-file #p"c:/Morrowind/Data Files/Beryl's_Head_Replacer_v1.0.esm")
NIL
CL-USER> (pathname "c:/Morrowind/Data Files/Beryl's_Head_Replacer_v1.0.esm")
#P"c:/Morrowind/Data Files/Beryls_Head_Replacer_v1'.0.esm"    ; note the mangling of single quotes
CL-USER> (probe-file *)
NIL
CL-USER> (pathname "c:/Morrowind/Data Files/Beryl\'s_Head_Replacer_v1.0.esm")
#P"c:/Morrowind/Data Files/Beryls_Head_Replacer_v1'.0.esm"
CL-USER> (pathname "c:/Morrowind/Data Files/Beryl\\'s_Head_Replacer_v1.0.esm")
#P"c:/Morrowind/Data Files/Beryl/s_Head_Replacer_v1'.0.esm"
CL-USER> (pathname "c:/Morrowind/Data Files/Beryl''s_Head_Replacer_v1.0.esm")   ; try 2 quotes
#P"c:/Morrowind/Data Files/Beryl''s_Head_Replacer_v1'.0.esm"    ; this fails too
I'm tearing my hair out. I'm going to have to release a version of the program that asks the users to rename any files with apostrophes in their names! This is unacceptable. Can someone enlighten me as to how I can stop lisp mangling these filenames?

nuntius
Posts: 538
Joined: Sat Aug 09, 2008 10:44 am
Location: Newton, MA

Re: Problem with quote chars in filenames

Post by nuntius » Mon Jan 11, 2010 10:00 am

That's ccl-specific behavior. Here's a working example with clisp on unix.

Code: Select all

# touch /tmp/quote\'here.txt
#clisp
[1]> (pathname "/tmp/quote'here.txt")
#P"/tmp/quote'here.txt"
[2]> (probe-file *)
#P"/tmp/quote'here.txt"
I would ask the ccl developer's list about a bugfix or workaround.

ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: Problem with quote chars in filenames

Post by ramarren » Mon Jan 11, 2010 11:25 am

Apparently, this is working as designed. It seems that on CCL on Windows single quote in pathnames is a metacharacter quote (because backslash is directory separator, I guess), which means that a double single quote is in fact a syntax for namestrings containing single quotes. And the spontaneous single quotes in your examples were to differentiate the file name dots from the name/extension separator dot.

eeeickythump
Posts: 9
Joined: Sun Jan 03, 2010 2:04 am

Re: Problem with quote chars in filenames

Post by eeeickythump » Mon Jan 11, 2010 3:45 pm

Ramarren wrote:Apparently, this is working as designed. It seems that on CCL on Windows single quote in pathnames is a metacharacter quote (because backslash is directory separator, I guess), which means that a double single quote is in fact a syntax for namestrings containing single quotes. And the spontaneous single quotes in your examples were to differentiate the file name dots from the name/extension separator dot.
Thanks! However double single quotes don't produce a "single single quote" in the pathname -- see the last example in my original post.
So there appears to be no way to get clozure to deal with filenames that contian single quotes.
I will open a ticket on the clozure site and see what eventuates.

ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: Problem with quote chars in filenames

Post by ramarren » Mon Jan 11, 2010 4:07 pm

eeeickythump wrote:Thanks! However double single quotes don't produce a "single single quote" in the pathname -- see the last example in my original post.
They do, it's just print/read consistency thing. It's like you name a symbol with a string containing a backslash:

Code: Select all

CL-USER> (intern (coerce '(#\A #\\ #\B) 'string))
|A\\B|
CL-USER> 'A\\B
|A\\B|
CL-USER> (coerce (symbol-name 'A\\B) 'list)
(#\A #\\ #\B)
The printed/read form contains pipes and double backslashes, but the actual name contains only one backslash and no pipes.

The filename containing a double single quote on the Lisp side refers to a filename containing a single single quote on the operating system side because the backslash cannot be used for this purpose because Windows uses it as directory separator (probably not really by now, but it still is everywhere in interface layers).

Try a (directory "*") in a directory containing files with quotes in their names, they will be translated to double single quotes when they are converted to pathnames. Single single quotes in pathname designators get eaten for the same reason single backslashes disappear in normal strings.

eeeickythump
Posts: 9
Joined: Sun Jan 03, 2010 2:04 am

Re: Problem with quote chars in filenames

Post by eeeickythump » Mon Jan 11, 2010 9:17 pm

Ramarren wrote:
eeeickythump wrote:Thanks! However double single quotes don't produce a "single single quote" in the pathname -- see the last example in my original post.
They do, it's just print/read consistency thing. It's like you name a symbol with a string containing a backslash:

Code: Select all

CL-USER> (intern (coerce '(#\A #\\ #\B) 'string))
|A\\B|
CL-USER> 'A\\B
|A\\B|
CL-USER> (coerce (symbol-name 'A\\B) 'list)
(#\A #\\ #\B)
The printed/read form contains pipes and double backslashes, but the actual name contains only one backslash and no pipes.

The filename containing a double single quote on the Lisp side refers to a filename containing a single single quote on the operating system side because the backslash cannot be used for this purpose because Windows uses it as directory separator (probably not really by now, but it still is everywhere in interface layers).

Try a (directory "*") in a directory containing files with quotes in their names, they will be translated to double single quotes when they are converted to pathnames. Single single quotes in pathname designators get eaten for the same reason single backslashes disappear in normal strings.
Ummmm... thanks but...

Code: Select all

CL-USER> (probe-file "c:/Morrowind/Data Files/Slof's Civilised Beasts.esp")
NIL
CL-USER> (probe-file "c:/Morrowind/Data Files/Slof''s Civilised Beasts.esp")
NIL
As it happened I received a quick reply on the clozure trac site. It seems it is a bug. A workaround is:

Code: Select all

(setq ccl::*pathname-escape-character* #\^Q)
Any other unlikely-to-be-used character can be substituted for the #\^Q.

ramarren
Posts: 613
Joined: Sun Jun 29, 2008 4:02 am
Location: Warsaw, Poland
Contact:

Re: Problem with quote chars in filenames

Post by ramarren » Mon Jan 11, 2010 11:48 pm

Oops, sorry. I thought I tried that, but since I only installed CCL on Windows to check that and tried it in the default console thing, I must have missed something. Well, at least you got to love that you can just go and rebind implementation internals.

Post Reply