I've made some test cases using a Java library (sorry but I can't remember it's name) and for small files the speed and memory usage seems better, but the problem comes when the file is large and I think I loose speed reading by byte.
This is something you should profile then. It might be the case that you lose speed vs the Java library from function call overhead, and it has nothing to do with buffering (HotSpot will eventually inline all the reading code for large files, where for all current CL compilers you have to make inline declarations yourself). Most CL implementations are going to have buffered file reading if for no other reason than their runtime uses libc for portability.
So do some profiling, get the source code for your implementation and see what it does (if you build your main lisp from source, you can use SLIME cross-referencing on system functions just like on any other code), and then you have the data to come up with some improvements either for your code or for the implementation you're using.