Please help my head get around 'deftype'

[tlareywi]

Hi all,

In general, I'm comfortable with the dynamic typing of CL 'except' in a case where I want to lock down interfaces to a package. That is, beyond exporting only the functions I want exported from the package, I want to specify the types of their arguments in the most precise way possible to make the code explicit for users of the package.

It looks like deftype can be used to build up type abstractions but I'm really having a heck of a time with it. The Graham book only briefly touches the topic. There are two main things I'm trying to define.

1. A type 'somthing-list' that represents a simple list where all elements of that list are of type something.

2. A way to define a function signature type. For instance, a type that represents the class of all functions that take two number arguments.

Are either of these even possible using deftype? Maybe there's an entirely different, more Lispy, way of achieving the same sort of package encapsulation effect? Any examples or suggestions would be greatly appreciated.

Thanks in advance!

[sburson]

I'm afraid this is an area where CL is really weak. I don't believe there's a way to define the types you want -- and I don't know what CL would do with them if you did. You certainly can describe them in your interface documentation; after that, your best bet is to check the supplied arguments at runtime, perhaps with CHECK-TYPE. But you won't be able to check that a supplied function takes two numbers, or even that it takes two arguments; you'll just have to call it and take your chances. It's the Lisp way: don't worry, be happy

[qbg]

You can do #1 with a SATISFIES type. You create a predicate function, say something-list-p, then the type would be (satisfies something-list-p).

Unfortunately, it probably wouldn't help the compiler and it has to run the function each time you want to check an object for that type...

[findinglisp]

Maybe I'm not following you, but why do you care about typing? From your original posting, it sounds like you just need it for documentation purposes, to communicate with users of your API. If that's true, then I'd suggest that you just use well-written doc strings and be done with it. Typically, typing is used in Lisp to try to make things really easy on the compiler such that it can generate better code, not for documentation purposes. If you aren't trying to make a particular bit of code go faster, I'd just concentrate on doc strings and not on types. If you are trying to make things go faster, then DEFTYPE might help you reduce the number of characters you have to type (no pun intended) when you write our your type declarations. But those should really only be necessary around inner loops, etc. And as people have already said, there are limitations to what can be expressed with the Lisp type system, though those limits are at least as large as what you can do with many other statically typed languages (e.g. C). For instance, developing a type like "all numbers between 17 and 32" or "all fixnums" or "a string or a character" are pretty easy. Things like "all prime numbers less than 4 billion" would be hard.

Summary: try to figure out whether you really need to do that. If not, then don't. If you still do, then look at declarations and DEFTYPE.

[tlareywi]

Maybe I'm not following you, but why do you care about typing? From your original posting, it sounds like you just need it for documentation purposes, to communicate with users of your API. If that's true, then I'd suggest that you just use well-written doc strings and be done with it.

It is true that this serves as a sort of documentation. However, it goes beyond that because documentation isn't part of the runtime system. A person may or may not read the documentation. Even if they do, they may still pass something unintended into the package that doesn't generate problems immediately but rather down the road during runtime at some point, making it difficult for them to determine a root cause. The motivation for typing interface points between packages is so that you can make some solid statement, enforced by the system, regarding what you define as valid input. If the interfaces are open to any input then you've effectively created a situation where you must anticipate any possible input a user of your package may provide. By typing your interfaces, you're not letting them pass in things that you don't anticipate and hence increase your code's robustness. Granted, this doesn't always matter but it does sometimes.

As a little background, I come from mainly a C++/COM environment building large applications. So some of my ideas I'm sure carry over from that. However, for the past year I've worked mainly with Lua, an interactive scripting language with functional features and dynamic typing (I've been learning Lisp in hopes that I could get the best of both worlds, and then some). A significant percentage of defects we've encountered in this code base arises from developers calling interface points, where the arguments are of dynamic type, with an unpredicted 'type' of argument. There are plenty of cases where the documentation is read but not well understood.

Anyway, I really hope I don't spawn a type system war (pointless) as that isn't my intention at all. There's a place for many different kinds of type systems. I just wish they'd bend to my will

[Exolon]

Hi, have you looked at Qi? It has a static typing system (and has some nice features like pattern matching).

[tlareywi]

Hi, have you looked at Qi? It has a static typing system (and has some nice features like pattern matching).

A bit. Qi was mentioned in another thread. Having just got my head around Lisp fundamentals, I'm a bit reluctant to stack another language on top but it may very well be the way to go. Thanks all!

[Exolon]

I'm in the exact same boat, and am going to put off even investigating Qi properly until I've done a few mini-projects in common-or-garden CL first

[findinglisp]

It is true that this serves as a sort of documentation. However, it goes beyond that because documentation isn't part of the runtime system. A person may or may not read the documentation. Even if they do, they may still pass something unintended into the package that doesn't generate problems immediately but rather down the road during runtime at some point, making it difficult for them to determine a root cause. The motivation for typing interface points between packages is so that you can make some solid statement, enforced by the system, regarding what you define as valid input. If the interfaces are open to any input then you've effectively created a situation where you must anticipate any possible input a user of your package may provide. By typing your interfaces, you're not letting them pass in things that you don't anticipate and hence increase your code's robustness. Granted, this doesn't always matter but it does sometimes.

So, I'm still not sure what you are after, exactly:

1. Documentation
2. API enforcement
3. Performance optimization

If you want the first, then doc strings are fine. And doc strings are available in the runtime system. This is a big difference from other static languages or things like Javadoc that use special comment formats. Lisp doc strings are stored with the functions and variables that they document. Type "(DESCRIBE 'SYMBOL)" in a typical Lisp where SYMBOL is one of your symbols with a known doc string and notice the output. At least with SBCL, it's quite robust. Note that often the base CL API is compiled with doc strings removed to save space. We don't need the CLHS repeated online. The point is, doc strings work quite well to document APIs. There are even a few packages out there to automatically generate other forms of API documentation (e.g. HTML) from doc strings.

If you're interested in doing the second option, you can always check types with your own code. You don't need to declare anything to do so. In other words, you can write something like:

Code: Select all

(defun foo (bar)
  (unless (integerp bar)
    (error "BAR must be an integer"))
  ...)

Finally, if you want performance optimization, that's when it really benefits you to declare types.

As a little background, I come from mainly a C++/COM environment building large applications. So some of my ideas I'm sure carry over from that. However, for the past year I've worked mainly with Lua, an interactive scripting language with functional features and dynamic typing (I've been learning Lisp in hopes that I could get the best of both worlds, and then some). A significant percentage of defects we've encountered in this code base arises from developers calling interface points, where the arguments are of dynamic type, with an unpredicted 'type' of argument. There are plenty of cases where the documentation is read but not well understood.

So I still don't see the benefit. In a C++/COM world, type mismatches cause big problems because operations are not type-checked. In Lisp, you can't accidentally add a string and an integer. When you try, you'll generate a condition and be dropped into the debugger. At that point you typically have a full stack trace and can quickly identify where things went wrong. If the same thing happens in C++/COM, you just get a mysterious crash and if you're lucky a core file to debug after the fact. If you're really up tight, you could type check every parameter in an API right at the first entry to the API, but I'm really not sure that's too useful in practice. If it was such a big problem, you'd see it being done in Lisp code all over the place, and that just isn't the case.

Anyway, I really hope I don't spawn a type system war (pointless) as that isn't my intention at all. There's a place for many different kinds of type systems. I just wish they'd bend to my will

I'm not hung up on type systems, so you aren't offending me at all. I'd simply suggest that if you really want to learn Lisp, that you try to not impose your C++/COM background on it. Instead, read lots of well-written Lisp code (I'd recommend Paradigms of Artificial Intelligence Programming as a good start) and see what good Lisp programmers do. Programmers have been using Lisp for nearly a half-century and there isn't much that they haven't tried during that time, I would surmise. I have been constantly amazed that when I start thinking Lisp needs such and such, I quickly realize that either it already has it, buried in some corner of the CLHS, or I don't really need it.

Put another way, whenever you learn to speak another language, whether human or programming language, you know you have arrived when you master idioms. Idioms from one human language rarely translate well to another human language. Indeed, whenever you use an idiom from a foreign language, you typically speak it in the foreign language. I submit that's true of programming languages, too. So, rather than trying to translate C++/COM idioms into Lisp, try learning Lisp's idioms first.

[tlareywi]

So, I'm still not sure what you are after, exactly:

1. Documentation
2. API enforcement
3. Performance optimization

API enforcement is what I'm after in this case.

So I still don't see the benefit. In a C++/COM world, type mismatches cause big problems because operations are not type-checked. In Lisp, you can't accidentally add a string and an integer. When you try, you'll generate a condition and be dropped into the debugger. At that point you typically have a full stack trace and can quickly identify where things went wrong.

Consider a case where the thing being passed into the API is a function and the function is stored off in a slot. As long as you give lisp some function, no error condition will arise immediately. Instead, assuming the function provided does not accept the expected arguments, you wont get an error until the function is actually invoked from somewhere. This could be immediately, much later, never, etc. If your code, the API client, is not the one doing the invocation, this could be quite surprising and confusing.

If it was such a big problem, you'd see it being done in Lisp code all over the place, and that just isn't the case.

As I mentioned earlier, I don't believe it is a pervasive problem. I don't think explicit type checking needs to be littered throughout most code. I'm focusing on module level entry points...APIs if you like.

I'm not hung up on type systems, so you aren't offending me at all. I'd simply suggest that if you really want to learn Lisp, that you try to not impose your C++/COM background on it.

I'm trying I'm not a fan of COM in general, but I think the general concept of componentization is valuable for any language that aims to implement large systems. I'd be interested to hear from folks that have worked on mid to large Lisp programs, say 100k+ lines of code, and how they tackled API/interface sorts issues between code modules owned by different devs, etc.