Michael Vanier: LFM and LFSP

(This is from an email sent by Mike Vanier. It is such a well-written description of the problems hackers face that I thought everyone should see it.)

My Ph.D. research involved writing very complex simulations of nervous systems. I used a simulation package that was written in C and had its own scripting language, all written in-house. I extended the hell out of it, but the experience was so painful I don't think I can ever work on a large C project again.

Since I want to continue working in this field, and since I love to hack, I want to re-write the simulator "the right way". However, I've been dithering on the choice of language. It's pretty clear that the core simulation objects have to be written in C++. C is too painful, and anything else is going to give an unacceptable hit in speed (simulation is one of those rare fields where it is impossible to have too much speed). But this is probably less than 50% of the code, maybe much less. The rest is infrastructure; scripting interface as well as a lot of support code.

For scripting I want to use Scheme or some Lisp dialect. But the language choice for infrastructure is unclear. I could use C++, but that's unnecessarily painful especially since the infrastructure is not speed-critical. So I'd decided to use Java; it's fast enough, there are a lot of libraries, and a lot of people know it so I could conceivably get others to work on it as well. After making this decision, my interest waned and I started another (unrelated) project.

In the process of working on that other project (which involves Scheme and Objective Caml (Ocaml), an ML dialect), it occurred to me that Ocaml would be a better choice than Java for the intermediate layer. It's faster, has better type-checking, is much more powerful, and can even be used as its own scripting language because of the type inference and interactive REPL. If necessary, I could write a simple Lisp-like language on top of Ocaml with little difficulty. The C interface to Ocaml is also quite mature, and there is a good-sized standard library (though nothing like the enormous Java libraries). Also, it's much lighter weight than Java. But here is the most important reason: it's a hell of a lot more fun to program in than Java.

Writing Java code, though not particulary painful in the sense that C is painful (core dumps etc.), puts me to sleep. Writing Ocaml (which is a "language designed for smart people" if there ever was one) is exciting. My motivation to tackle the project has tripled overnight. The interesting question is: why is Ocaml so much more fun than Java? Why are "languages designed for smart people" (LFSPs) so much more fun to program in than "languages designed for the masses" (LFMs)?

One possibility is that LFSPs tend to be more unusual, and hence are more novel. I'll admit that this is part of the answer, but it misses the main point. Any new language is going to be novel, but the novelty usually wears off quickly. The real point is that LFSPs have a much greater support for abstraction, and in particular for defining your own abstractions, than LFMs.

This is not accidental; LFMs deliberately restrict the abstractive power of the language, because of the feeling that users "can't handle" that much power. This means that there is a glass ceiling of abstraction; your designs can only get this abstract and no more. This is reassuring to Joe Average, because he knows that he isn't going to see any code he can't understand. It is reassuring to Joe Boss, because he knows that he can always fire you and hire another programmer to maintain and extend your code. But it is incredibly frustrating to Joe Wizard Hacker, because he knows that his design can be made N times more general and more abstract but the language forces him to do the Same Old Thing again and again.

This grinds you down after a while; if I had a nickel for every time I've written "for (i = 0; i < N; i++)" in C I'd be a millionaire. I've known several programmers who after only a few years of hardcore hacking get burned out to the point where they say they never want to code again. This is really tragic, and I think part of it is that they're using LFMs when they should be using LFSPs.