Re: Revenge of the Nerds

Revenge of the Nerds has generated a lot of additional discussion, which I summarize here for anyone interested in going deeper into the issues raised in it.

Trevor Blackwell wrote that Lisp is only a win for certain classes of project. His mail was so articulate that I will just include it verbatim:

I think it would seem more persuasive if you limited the scope of where Lisp is supposed to be better. It probably isn't better, for example, in:

- communications, interface, format conversion
- real-time control
- numerically intensive code
- operating system implementation
- quick little scripts to munge files
The above actually constitute a large fraction of all software written.

It is better for information processing: running very complex algorithms across complex data sets. It's only recently, in the WWW age, that people actually do a lot of this.

Before the rise of the Web, I think only a very small minority of software contained complex algorithms: sorting is about as complex as it got. Can you think of a complex algorithm in FreeBSD? The 50 million lines of code in a Nortel switch probably didn't even contain a sort. It was all about managing hardware and resources and handling failures gracefully. Cisco routers also have millions of lines of software, but not many complex algorithms.

The sort of software that ITA wrote is really very rare. 10 years ago, about the only software with really complex algorithms were CAD design & synthesis, and they did use Lisp.

That a shift in what kinds of software is being written makes Lisp resurface as excellent choice is a more believable statement than that the vast majority of programmers have been boneheads for 40 years.

I agree that Lisp might not be the language to use if you wanted to do something really low level, like move bits around, and processor time was worth more to you than programmer time. For that you'd want C or even assembly language. But I think such applications are rare and getting rarer (see Moore's Law). I know of several places using Lisp for real-time applications, including Rod Brooks' robot R&D lab.

I also agree that Lisp might not be the ideal language to use to write code that works closely with programs written in C, but that isn't Lisp's fault. Nor is C the ideal language to use to write programs that work closely with Lisp.

I disagree that Lisp is bad for writing quick little scripts. Its interactive nature makes it especially good for that.

I also disagree that it is not believable that the vast majority of programmers have been boneheads for 40 years. It seems to me entirely possible. Measured simply by numbers of users, the current leaders in any field of technology (indeed, almost any field at all) will be mostly mediocre. Look at Windows.

Technical innovations regularly take decades to spread. Volkswagen started building cars with unibodies in the 1930s. By the 1980s, practically all new cars were designed that way. Was it simply that the vast majority of car designers were boneheads for 40 years? Yep, though I think "conservative" is the preferred euphemism.

Bengt Richter came up with an additional Python solution: def foo(n): def bar(i): bar.s += i return bar.s bar.s = n return bar but I think Python hackers still consider defining a new class to be the canonical solution.

Several Python users have written to tell me that the reason you can't write the Lisp/Perl/Smalltalk/Javascript program in Python is that lexical variables in Python aren't mutable, and that "anonymous functions" can only contain a single expression. When I ask why you can't just write def foo(n): lambda i: n += i in Python, I'm not asking what it is in the current definition of Python that prevents this. I know that Python currently imposes these restrictions. What I'm asking is what these restrictions buy you. How does it make Python a better language if you can't change the value of a variable from an outer scope, or put more than one expression in a lambda? What does it buy you to distinguish between expressions and statements?

The restriction on what you can say in an anonymous function seems particularly indefensible. In fact, the whole phrase "anonymous function" suggests limited thinking. Would you call a hash table that wasn't the value of a variable an "anonymous hash table?" If functions are a data type in your language, they should be treated just like other data types. A named function should just be a function that happens to be the value of a variable. Restricting what you can put in a function that isn't the value of a variable makes about as much sense as restricting what you can do with a hash table or string that isn't the value of a variable.

Thomas Herchenroeder wrote to stress that languages had to be suitable for ordinary programmers, not just super-hackers:

I believe you are thinking of an excellent hacker whose code gets read once in a while by fellow excellent hackers who marvel at his fine algorithms and coding style. They are interested to learn new things and eager to understand everything for the pure pleasure of the intellectual challenge. Unfortunately, that's not the world I (and a lot of other people) live in. My co-workers ... want to get along with my code with minimal effort. So the power of a language becomes related to what an average IT professional can easily digest, which in turn is something that is commonly taught at university courses, explained in easily available books and discussed in popular articles. It boils down to mainstream knowledge about programming, languages, algorithms and patterns. To a degree, you simply have to go with the crowd. Again Python: It implements mainstream concepts in a succinct and elegant way.

So, maybe we are back at the point where we need a "powerful" language for the power users, the hackers. And a "powerful" (in a different sense) language for everyday people in everyday companies.

I agree that there is a role for languages designed for novice and less-motivated programmers. The only thing I disagree about is whether "powerful" is the word to use to describe this quality. I think there is already a word for this proposed second sense of "powerful", and it is "accessible." A Ferrari is powerful. A VW Golf is accessible.

Why stretch the language to find some metaphorical sense in which you can call the Golf powerful, when there is already a word for the quality you mean? Plus, if you blur the word "powerful" by using it too broadly, you no longer have a name for the very definite quality possessed by the Ferrari.

I think it is important that some of us, at least, keep our focus on power in the Ferrari sense. Someone has to, because that is the source of the next generation of "mainstream knowledge". In 1980, university courses and easily available books were advanced if they talked about structured programming. Now they talk about topics like garbage collection and dynamic typing, which would in 1980 have been considered esoteric stuff.

Paul Prescod wrote that I chose an example that was deliberately Lisp-biased. If I had wanted to do that I would have written a macro, not a function. If the example I chose is biased, it is Lisp/Perl/Smalltalk/Javascript biased, because the code is similarly simple in all four (and many others).

I was actually surprised at how badly Python did. I had never realized, for example, that a Python lambda-expression couldn't contain the same things as a named function, or that variables from enclosing scopes are visible but not modifiable. Neither Lisp nor Perl nor Smalltalk nor Javascript impose either restriction.

I can't see what advantage either restriction brings you. I can see how Python's gradual, ongoing (= incomplete) evolution would have produced them. So Occam's Razor implies that the latter is the reason Python is this way. I.e. these restrictions are bugs, not features.

Paul has written a rebuttal to Revenge of the Nerds, On the Relationship between Python and Lisp. (Peter Norvig has also written about this topic in Python for Lisp Programmers.)

There has been a lot of discussion on LL1, about whether succinctness is a good design goal for programming languages, as I implied in the appendix to Revenge of the Nerds. That question is handled separately in Succinctness is Power.