C FILE* -> Lisp stream?

[hilbertastronaut]

I've got a C FILE pointer to an open file, and I want to access it as a Lisp stream -- any suggestions? I'm calling into an embedded Lisp (specifically, ECL) from the C world. It's my C program and I have full control of the FILE*, so I don't have to worry about other threads messing with it without my knowledge; the Lisp call happens entirely within the validity interval of the FILE*.

My thoughts were:

1. Surely somebody has made a Gray stream for this. I don't mind the performance hit as long as it doesn't cons too much.

2. If not #1, would making my own Gray stream be hard? Any suggestions for a good tutorial?

Many thanks in advance!

[hilbertastronaut]

I think I've figured out how to do it in an ECL-specific way: the public ECL API has a function to convert a POSIX file descriptor (an int) to a Lisp stream, and the POSIX function fileno() returns the file descriptor of a FILE*. It still would be nice, though, if someone had a Gray stream wrapper for FILE*, as fileno() breaks the C FILE* abstraction in a way that I don't fully understand. (I write shared-memory parallel code so this is of concern to me.)

[findinglisp]

If I remember right, there is an internal ECL function for converting between these two. Hmmm... or maybe that was between a raw Unix file descriptor and a stream, not a C FILE*. Look in the source files for ECL (not that large, actually), and you can probably find it. It's in the C section of the source.

[hilbertastronaut]

If I remember right, there is an internal ECL function for converting between these two. Hmmm... or maybe that was between a raw Unix file descriptor and a stream, not a C FILE*. Look in the source files for ECL (not that large, actually), and you can probably find it. It's in the C section of the source.

Thanks for the advice! Here's what I got:

Code: Select all

void
c_stream_to_lisp_stream (cl_object* lisp_stream,
                                       FILE* c_stream,
                                       int* const info)
{
  int fd = -1;

  /* Get integer file descriptor number for the given C stream.
     fileno must always return (under Linux at least), but may return
     -1 and set errno to EBADF if it can detect that the given stream
     (FILE*) is not valid. */
  errno = 0;
  fd = fileno (c_stream);
  if (errno == EBADF || fd == -1)
    {
      errno = 0;
      *info = -1;
      return;
    }   

  /* Make a Lisp stream from the given file descriptor.  Note: it had
     better be an input stream of some kind, or you'll be sorry!!! */
  *lisp_stream = ecl_make_stream_from_fd (Cnil, fd, smm_input);
  *info = 0;
}


(I work with LAPACK a lot, so the INFO mutable parameter is my usual way of returning errors in the C world.)

This should work (I haven't tested it yet), but it seems like a kludge. The ECL function ecl_make_stream_from_fd() actually calls fdopen() on the FILE* which lives inside the Lisp stream. It's silly to go from a FILE* to an int and back to a FILE* again. Also, I'm not sure if I have to call fflush() first on the original FILE*, before I push it into Lisp world, as FILE* may buffer or do some other nonobvious things that aren't expressed by the integer file descriptor. I'd much rather work directly with the FILE*. I could just extend the ECL API to create a Lisp stream directly from the FILE*, but I need to do some legal investigation to see whether that would make my code a derivative work of ECL. (ECL is LGPL, but some of my users prefer that my code stay BSD, perhaps because it's common for them to make derivative works out of my code rather than just to invoke it as a library, or perhaps because they need to placate some lawyers or managers.)

[nklein]

The FILE* buffers in user-space (your program). Any buffering that happens with the file descriptor (from fileno()) is in kernel-space (your operating system). You will run into trouble if your C code writes to the FILE* before you Lisp code is invoked without calling fflush() because there will be stuff in the FILE* buffer that writes to the fileno would completely ignore.

Also, if your C code reads a few bytes from the FILE*, it probably buffered 4k or 8k under the hood. If you then try to read from the file descriptor, you won't get the next bytes after the few that you read from the FILE*, you'll get the next bytes after what the FILE* buffered. fflush() won't help you out of that one.

Weirder things may happen to if you have the file open Read-Write.

So, this doesn't help you with the Gray streams, but hopefully, it dispels some of how the FILE* abstraction and the fileno() are at odds.

[hilbertastronaut]

The FILE* buffers in user-space (your program). Any buffering that happens with the file descriptor (from fileno()) is in kernel-space (your operating system). You will run into trouble if your C code writes to the FILE* before you Lisp code is invoked without calling fflush() because there will be stuff in the FILE* buffer that writes to the fileno would completely ignore.

Also, if your C code reads a few bytes from the FILE*, it probably buffered 4k or 8k under the hood. If you then try to read from the file descriptor, you won't get the next bytes after the few that you read from the FILE*, you'll get the next bytes after what the FILE* buffered. fflush() won't help you out of that one.

Weirder things may happen to if you have the file open Read-Write.

Ah, that is a big help, thank you : - ) The FILE* is an input stream that is entirely under my control in the C world, so I am free to call fflush() on it and not do anything with it before I pass it into C space, but your second paragraph above suggests that I would be safer to work with the FILE* directly -- both because the C world might do its own buffering, and because the FILE* in Lisp space might do its own buffering (so both Lisp and C could miss out on some data). But if C doesn't read from the FILE* at all before I feed its fd into Lisp, am I guaranteed that all the data will be there?

[nklein]

I think an implementation may be allowed to fill its buffer during the open call if you open the file for reading. I would bet most don't bother. But, I doubt there is a reason they can't.

[hilbertastronaut]

I think an implementation may be allowed to fill its buffer during the open call if you open the file for reading. I would bet most don't bother. But, I doubt there is a reason they can't.

Yeah, I think I'll just use a Gray stream, or maybe figure out if I can mess with ECL's internals without needing to make my code LGPL. I think it's ok because I'm not actually modifying ECL source, just #including an ECL header to get the definition of struct ecl_stream, and linking to ECL as a library -- that's the same use case as for GNU's libc, which is also LGPL.

I'm kind of surprised there aren't any Gray streams available that wrap FILE*. If there's enough interest and it works for me, I'll write one and release it. It might be slow but I'm mainly concerned about compatibility and safety.

[findinglisp]

I'd just modify ECL itself and post a patch to the ECL mailing list. Only that one little portion of your code (the patch) would be under LGPL. If you get into including header files into your other programs, then you start dancing around the line of derivative works and things get messy. Just create the function that you wish ECL had, put it in the ECL codebase, and send a patch to the ECL developers, and call that function from your code.

[hilbertastronaut]

I'd just modify ECL itself and post a patch to the ECL mailing list. Only that one little portion of your code (the patch) would be under LGPL. If you get into including header files into your other programs, then you start dancing around the line of derivative works and things get messy. Just create the function that you wish ECL had, put it in the ECL codebase, and send a patch to the ECL developers, and call that function from your code.

This approach was successful -- the functionality was trivial to add to ECL, so Juanjo just did it himself. Yay Juanjo! Yay ECL!

Thanks all for the advice : - )

The Gray stream would still be an interesting exercise, but I have other work to do too ; - )