Binary Stream and Binary Types

[Kind&Deadly]

Im a newbie to programming in general,I'm trying to learn as a hobby mostly. I have toyed with C for a while tho.

This is frustrating.This is a rant.

After much reading and admittedly not enough practice, i got to play with files in CL.At first, i was bothered by the absence of a "posix-y" write function to "dump" a buffer.

Fine.I can live with it.
After all, CL has write/read-sequence, so in the end I should be able to... as long as anyhting I ever want to write is text or 8 bit integers.

WAIT... WHAT!?

Yeah right, http://www.gigamonkeys.com/book/practic ... files.html.

Fucking great.Thats not reinventing the wheel oh no... That's reinventing the wheel using wheeled machinery out of fucking wheeled machinery.

"Hey, you aren't allowed to dump anything but 8 bits form memory because it'snot portable machine and implementation wise."

Little insight... How about that given the fragility of FASL files my code is getting RECOMPILED ON EVERY SLIGHTLY DIFFERING MACHINE OR IMPLEMENTATION THAT EVER RUNS IT!!!
How is portability a matter there.

So the main question here is :

Is there any way to read/write an integer bigger than 8 bits WITHOUT playing the "imma summing mah bytes" game?

[JamesF]

Dude, if you think that's painful, wait 'til you look at the generalised machinery for handling filepaths. It was standardised at a time when there were Unix, DOS, VMS with its built-in file-versioning, other exotic systems floating about, and more being developed. Try handling that in a portable way that's both simple and elegant.

How portability a matter in file-writing code? In much the same way as C handled portability in an age of assembler: yes, you need to recompile it on each new platform but, as long as you don't build in any platform-specific tweaks, you don't need to rewrite it each time.

What I think you've just run into is the fact that CL spans the entire gamut from low-level bit-twiddling to high-level metaprogramming. There's a bit of overhead in exchange for the flexibility and scalability. It's also worth remembering that CL has a different imperative to, say, Python: it's for writing big, complex systems whose specs keep changing; beginner-friendliness just isn't a concern. This means that you're in for a hell of a learning curve but, once you've climbed it, the stuff of which awesome is made is at your command. This is also why it's hard to give a simple answer to the periodic question "so what do macros really buy you?" - like ramjets, they just don't make sense at the kind of scale that fits in a forum or mailing-list post.

To address your actual question: although I've not yet had cause to do this kind of thing myself, I'm pretty sure you can just open a file for writing, carefully make sure it's set up for the right kind of encoding, then fling bytes at it with abandon.

Btw: I love your comment about wheeled machinery. Beautifully put.

[Warren Wilkinson]

I just slurp the file into memory and access it with pointers. Something like this (in SBCL):

Code: Select all

(let ((data (read-sequence ...)))
   (sb-ext:with-pinned-objects (data)
       (let ((sap (sb-sys:vector-sap data)))
           (sb-sys:sap-ref-32 sap 0))))  ;; First word.

Files suck in every language, I've always found it easiest to define the data structures as methods to access raw memory. And then serialization is just streaming the raw bits to disk.

[Kind&Deadly]

Dude, if you think that's painful, wait 'til you look at the generalised machinery for handling filepaths. It was standardised at a time when there were Unix, DOS, VMS with its built-in file-versioning, other exotic systems floating about, and more being developed. Try handling that in a portable way that's both simple and elegant.

Yep, i've also tripped on that one.

What I think you've just run into is the fact that CL spans the entire gamut from low-level bit-twiddling to high-level metaprogramming. There's a bit of overhead in exchange for the flexibility and scalability. It's also worth remembering that CL has a different imperative to, say, Python: it's for writing big, complex systems whose specs keep changing; beginner-friendliness just isn't a concern. This means that you're in for a hell of a learning curve but, once you've climbed it, the stuff of which awesome is made is at your command. This is also why it's hard to give a simple answer to the periodic question "so what do macros really buy you?" - like ramjets, they just don't make sense at the kind of scale that fits in a forum or mailing-list post.

To address your actual question: although I've not yet had cause to do this kind of thing myself, I'm pretty sure you can just open a file for writing, carefully make sure it's set up for the right kind of encoding, then fling bytes at it with abandon.

So far its kind of frustrating it offers support for things that *dont* exist, and stuff as basic as binary output require you roll a frigging type system on top of the other one.

I got answered very politely on #lisp that what i was looking for was kind of complicated, but that, the functionality was already implemented in libraries like:
http://www.cl-user.net/asp/libs/binary-types
The implementation of binary parsing in practical common lisp chapter 24

Wow, again.
Lets see. In any other language I would be called a fucking retard for doing bulk IO byte by byte.But not in CL.
Apparently here its the only option,THAT, or rolling your own damn type system, but HEY! don't worry! you might say.
CL meta-programming capabilities sure can extend the language in the way you want... syntactically.

You dont have acess to memory ofc so lets use arrays for that!
But now... wait you cant use the data you just read, you need to play with it , a 32bit integer needs to be summed up to translate from 4byte array to
WHATEVER-BIT BENT OVER BACKWARDS your implementation happens to be using.

Fine, Fine , now that we operated with it lets just write it back OOOOOPPPPS we have to turn it back into a non-autistic format so the rest of the motherfucking world understands that is a 32 bit integer.

As i said doing anything similar in any other language would get me flamed but on lisp ITS FINE!
Whats the matter with efficiency? its FAST ENOUGH!
It's like shaving with a damn bowling pin, it might work if you try hard enough , but that doesn't make it the right way to shave nor the most efficient.

After all it frigging compiles to native code RIGHT?

Never mind that you got to be constantly converting out and into the autistic I'm-special-no1-understands-me bit representation right?

But its fine, apparently, after all when the fucking lisp machines come back CL will be prepared and ready, and wont have to play nice with absolutely fucking anything anymore right?

I just slurp the file into memory and access it with pointers. Something like this (in SBCL):

Code: Select all

(let ((data (read-sequence ...)))
   (sb-ext:with-pinned-objects (data)
       (let ((sap (sb-sys:vector-sap data)))
           (sb-sys:sap-ref-32 sap 0))))  ;; First word.

Files suck in every language, I've always found it easiest to define the data structures as methods to access raw memory. And then serialization is just streaming the raw bits to disk.

That is SBCL dependant.

i know you could do that,whats more , so far when i looked at retarded topics like sockets and this the only sanity came from implementation dependant api.

YET i find it ABSURD that the standard includes completely unnecessary bullshit like dunno, format printing numbers in English or roman notation ,path objects for decades extinct and yet to be invented filesystems, an object system that would make Java cry in a corner...

But no standard way to do IO.

The rage comes from the fact that i was so excited, CL was so great with all that powerful features...

I mean, if you are going to do whatever autistic fuck-you-all-im-special representation in memory, THE LEAST a language should do is provide a STANDARD way to interact with every one else.

BUT NO

Guess what? if people considered its fine to implement every commonly used feature yourself,it woudnt matter as long as the language is turing complete.

Oh ok, ill just go roll my own CLOS on to of BRAINFUCK...



Right now it is SO much more attractive to use the damn FFI and use the SANE clib posix interface, than rollling my fucking own typesystem on top of CL.

Remember that quote that said

"Every sufficiently complicated C program includes an incomplete, bug-ridden implementation of half of Common Lisp?"

Corollary:
"Every sufficiently complicated Common Lisp application includes a slow-as-fuck ,redundant and dependency-ridden implementation of half of POSIX"

The more I look into it the more I see libraries dont really do shit and are mostly FFI wrappers or just portability layers on top of the retarded, balkanized implementation specific API's.




Or give CL up altogether.
I dont know who to blame for it, design by committee, old age... whatever.The only thing im certain its that sane definitely doesnt describe CL.


So, there is not even a pack
http://docs.python.org/library/struct.html

or similar facility in the standard?

[Warren Wilkinson]

Yes, there is a pack.

Code: Select all

(defun pack (obj) (princ-to-string obj))

(defun unpack (str) (read-from-string str))

SBCL-only matters when you plan to run your code on other lisps. If you're only working with one, use the time saving devices it gives you.

[Kind&Deadly]

Yes, there is a pack.

Code: Select all

(defun pack (obj) (princ-to-string obj))

(defun unpack (str) (read-from-string str))

SBCL-only matters when you plan to run your code on other lisps. If you're only working with one, use the time saving devices it gives you.

(print/read)-(to/from)-string is the same as using print/read on a with-(output/input)-(to/from)-string form.

AKA text.

Nothing to do with binary I/O at all.

[Warren Wilkinson]

Yes its not binary. The link you gave says pack should return a string, and unpack take a string.

I'm guessing you want serialization of objects?

Yes this is tricky, and C does it pretty well, since in C you can access the raw bytes. You have a few options, pick your poison.

1. Use C data structures and the FFI.
2. Don't define the structures, just define accessor methods that access raw memory.
3. Hand write a serializer/deserializer that produces/reads-from a ub8 array or stream.
4. If you have a u32 array, conversion to a ub8 array should be pretty easy.

[nuntius]

FWIW, it sucks to process raw binary data in any language. Endian conversions, field padding, etc. always have to be done to get the data between internal (e.g. int32_t) and external (e.g. network byte order) formats. It must be done at the bit/byte level because that's where things are specified.

The only way around it is to use a definition-driven system. For example, see how IDL is used by Corba and DDS. See the XDR specification for a simpler variant.

CL's tendency to hide the memory representations of its data types makes this only slightly harder.

[Kind&Deadly]

No it doesnt suck in every language.

Mainly because :

A) They provide a sane standard way of dealing with it.

B)There are plenty of general or specialised libraries for it.


In CL it's more like

A) Reinvent the wheel yourself and maintain it forever because the standard is set on stone and never went nor is going to move from the lisp machines prophetic revival




B) Use some obscure library that was once programmed by one guy that noone has touched, maintained or dared to look at for the past 4 years.HAHA YOU USE IT YOU MAINTAIN IT.Nevermind it probably is trash by now and needs a rewrite, or that it was trash form the very start.Afterall its THE library, not that is there any more out there.



C) Ha Ha, you silly!, you dont DO things on CL you get to be a general Smug Lisp Weenie and complain noone accepts you.
Poor language misunderstood and hated for no reason, sure it hasnt got anything to do with its backwards design and its retarded

"Batteries incluede but not the ones that are actually useful and considered basic functionality in ever single other language, but the rotting carcass of a dead model that we are not going to change no matter how useless the standard turns to be beacuse we can brag how hip and different we are"

[Warren Wilkinson]

I've used a lot of languages, and I don't recall any of them having a sane standard way of dealing with binary data. If you point me towards some of them, I might be able to find useful CL libraries that work like that. Edi Weitz tends to have a useful library for everything. Actually, just looking at his libraries, he has one for binary streams with odd sized bytes (probably anything from 1 to 32 bit bytes): http://weitz.de/odd-streams/

And his stuff IS maintained.