Page 1 of 1

Clozure CL and unicode on Windows

Posted: Mon Oct 19, 2009 8:49 pm
by Eachother
Greetings!

I'm having problems with reading unicode files (using (with-open-file) and (read)) that contain byte-order-mark. It seems CCL just stops reading the file after the BOM and returns the BOM as the data it has read.

Here is the code:

First, create a unicode file:

Code: Select all

(with-open-file (f "f" :direction :output :external-format :utf-16) (write "abc" :stream f))
Second, read the data back:

Code: Select all

(setf x (with-open-file (f "f" :direction :input :external-format :utf-16) (read f)))
After I execute the above code I tell the CCL "x" and it replies "\→" instead of "abc".

If I remove the BOM from the file and reread it then x is what I wrote to the file, i.e. "abc".

The same example (writing and reading the same file) works fine under CLISP (with the exception of replacing ":utf-16" with "charsets:utf-16"): the file written is the same as from CCL (it starts with 0xFEFF), data is read back successfully, x is "abc".

Could anyone help me with with?

OS is Windows.
CCL is 1.3-r11936 (WindowsX8632)
CLISP is 2.48.

Thanks.

Re: Clozure CL and unicode on Windows

Posted: Tue Oct 20, 2009 11:33 am
by blandest
You could try to read the first two bytes (with #'read-byte) and just ignore them if they match the BOM signature.

Re: Clozure CL and unicode on Windows

Posted: Tue Oct 20, 2009 12:49 pm
by nuntius
That sounds vaguely familiar. You might check on Clozure's IRC or mailing list.
http://trac.clozure.com/openmcl#Support

I also think they are close to releasing 1.4; so you might see if a prerelease fixes this problem.

Re: Clozure CL and unicode on Windows

Posted: Fri Oct 23, 2009 12:56 pm
by rme
This is a Windows-specific Clozure CL bug. It's fixed in the trunk, and will be in the forthcoming 1.4 release.

If you want to try out a pre-release of 1.4, please see

http://trac.clozure.com/openmcl/wiki/ReleaseNotes/1.4

Note that you'll need to run (rebuild-ccl :clean t) after checking out a copy of the 1.4 prerelease in order to pick up the bug fix.

If you run into more CCL issues, please register on the Clozure CL Trac site (http://trac.clozure.com/openmcl) and create a ticket. I just happened to run across your report here by accident.