duck-season
Duck Season
On March 31, John Doe commented (about Tin Foil Ducks):
> heh you got to love the 'i wont try to sell you lisp' rhetoric :), yet
> you haven't failed to mention it in all your blogs (I dont mind, but
> it's just so ironic that you'd say that :).
>
> who has done performance studies that measure productivity between
> fluent Java (or any other language, let's say) and Lisp programmers?
> What were the gains by using one over the other for the same type of
> task? Just curious. Lisp has my interest, and even though I'm not apt
> at it, I'd be willing to learn it if I can be convinced it has
> something in it for me that will allow me to stay lazy - or lazier
> than I am. :).
Oooooh, no. No, no. I'm not taking the bait. I know what game you're playing at. Wabbit season! Duck season! Wabbit season!! *Wabbit* season!! Duck season! *BANG*
You can't fool me into trying to sell you on Lisp. :)
Besides, you'd be mad at me. I haven't found any Lisps out there that are even remotely what you'd call "convenient". It's totally a hobbyist/tinkerer language. Like how some people love to work on car engines — they spend all their time in their garage, covered head to foot in engine grease. That's how working with Lisp feels to me. It's a lot of dirty work before you can drive it around town, let alone take it to the track.
An even better analogy might be Linux, or Cygwin. Both of them have a high cost of ownership. If you use Windows or a Mac, a lot of stuff is just "handled" for you. Most people prefer that — arguably even most developers. They want to get work done, without needing to spend all their time trying to build utility packages from source, or find the right combination of flags to pass to ssh to get colors working correctly on their man-page display.
I mean, even if you do get the colors working, who really wants to use "man"?
The Unwinnable Argument
How would you convince a Windows fan(atic) that they should switch to Linux on their desktop? In most cases, you probably couldn't. You might be able to convince them that they should switch to a Mac, because Macs are sexy. But with Linux... you're trying to tell them that they should use a command prompt? That's so, like, 1980's.
You can tell them they're really not as productive with (raw) Windows, and that if they'd just spend the years necessary to become an amateur Linux administrator and developer, then they can become far more productive at grepping through text files. See how far you get with that argument.
You Linux folks know, deep down, that you're more productive than developers who use Windows. You Windows folks know, deep down, that you're more productive than developers who use Linux. And me and a small minority of Windows+Cygwin (or Linux+Wine) users know, deep down, that we have the best and worst of both worlds, but we're still the most productive.
My guess is that it's probably the Mac folks who really are the most productive. I want one of those things. I have 17"-PowerBook envy.
And of course we've already been round and round on the exact same issue, except with editors instead of operating systems. There are GUI lovers who like blinking lights and widgets and that heady feeling of massive productivity that you get when you realize that you have literally hundreds of tabs open, and you can click on any of them whenever you feel like it. It's a real rush.
And then there are the crufty old terminal-window lovers, who get that their massive-productivity rush when they've just completed a command sequence with nineteen consecutive keystrokes for the first time without messing up halfway through and having to re-type it.
It's a trade-off. A fundamental one. You have to decide whether you want control or convenience. You can't maximize them both. Control requires flexibility (via configurability), and the more of that you add in, the thicker the user manual becomes, and the steeper the learning curve gets.
It's just human nature to be attracted to the easy solutions: to go for the easy gains, even if you know (at some level) that you'll achieve greater gains by taking the harder path. Why should I spend years learning martial arts when I can just pack a gun and spend a couple hours a week in the gym?
And it's not as if Lisp and co. get to hog the "I'm the cool martial-artist" side of the metaphor, either. If you choose a higher-level language, you're making the control/convenience trade-off in two totally different ways:
You're sacrificing control over pointers, memory management, and other "low-level" stuff, in return for the convenience of being able to express computations elegantly. I.e. you're putting more load on the computers to put less load on the engineers, under the assumption that people cost more than computers.
You're sacrificing the convenience of using a popular language (with all the attendant benefits of having lots of libraries, documentation, fancy installers, plentiful interview candidates, online resources, etc.) in order to gain more control over your language environment, and ultimately, your programs.
Either way, you're winning in some ways and losing in others.
I hope you're wondering: "Can't there be a language that gives you supreme flexibility and control, and comes with docs and installers and libraries and candidates so on?"
That's what I'm wondering, anyway. But the question is really more complicated than that, because there are in fact other dimensions that you also want to optimize. And some of them appear to be mutually exclusive, or nearly so.
Shootout
To make this (hopefully) a bit clearer, compare it to an Equalizer, which presents you with a set of sliders that adjust the frequency components of a signal in different ranges, from bass to treble. There's no single setting that works best for all types of music (or all types of people). If there were, EQs wouldn't exist, obviously.
There are various settings available for programming languages; imagine them as sliders on an EQ. Every language makes its own choices for each setting. This isn't a complete list, but here are some of the settings you might find on a "Language Equalizer":
Run-time speed: how fast the resulting executables are, amortized over the entire body of code produced by the practitioners of the language. (I.e., how "naturally fast" the language is, when you express things in ways that are reasonably idiomatic for that language.)
Memory usage: what's the memory footprint of programs created in this language? Obviously people are interested in both the maximum and the average or expected-case.
Static type-safety: how much the language checks for possible errors before ever running your code. C++ and Java are fairly middle-of-the-road here. Languages like Ada and OCaml and Nice are much stricter. For reasons that I still only dimly understand, static type-safety is painful to us human beings — more painful than getting runtime errors. So each language has to choose very carefully how far to turn this dial.
Run-time safety: how hard the language tries, at run-time, to keep your program from doing something "bad" — e.g. things that could crash the program, corrupt data, maliciously take control of the program, or whatever the language designer happens to think is "bad". (Normally people think of this as simply present or not, but you can also have security checks, bytecode verification, etc.)
Cycle time: how fast can you make changes to your code and see them in the running program? This has a huge impact on your quality-of-life as a developer. Having a fast compiler helps, but not having to restart at all is even better.
Introspection: how much metadata the language includes about its own internal structures, to let you view, query, manipulate, and potentially modify them (all programmatically, of course.)
Expressiveness: how easy is it to "say" things in the language? Examples include: defining your own data types, creating and using containers, expressing program control flow, interfacing with the operating system, defining and/or using modules and namespaces, issuing pragmas or instructions/hints to the compiler, defining and coordinating concurrent activities... it's a very list long list, and one that's growing over time.
Extensibility: what facilities the language provides for changing it to suit your needs, or your tastes. This includes things like preprocessors, template systems, macros, metaprogramming, meta-object protocols, aspect weavers, and their ilk.
Familiarity: how close is the language to some other language that people already know? It's much easier to learn languages that resemble the ones you use — that's why it's easier for English speakers to learn the germanic or romance languages than asian languages, for instance. A language could maximize every other dimension, and still fail by being too alien for most people.
Even with this short, incomplete list of features, we can already see some pairs that are natural enemies in the wild:
Introspection conflicts with memory usage, because you need runtime data structures in order to implement it.
Static type-safety often conflicts with cycle time, because you have to wait for the compiler to do all that checking. (It's often the case that there are static checks the compiler writer would love to do, but they're simply impractical to implement efficiently.)
Static type-safety usually also conflicts with expressiveness to some extent. In languages like Java that have no type inference, you wind up having to enter a bunch of redundant type information, so your program becomes rather verbose. In languages like Haskell that have type inference, you can often leave out the type tags, but in order to make type inference practical, they make it illegal to say certain very natural things.
Run-time safety usually conflicts with run-time speed, because run-time safety requires computing things at run-time, and they don't come for free.
Expressiveness usually conflicts with run-time speed (or memory usage, or both), because modern compiler technology still isn't very good.
Expressiveness also often conflicts with familiarity. Take regular expressions, for instance — they're incredibly expressive, but they don't look very familiar, and it takes some time to learn them — or even to be convinced that they're worth learning, for that matter.
So every language winds up making trade-offs, and those trade-offs make the language good for some domains and bad for others — and also good for some people and bad for others.
So which trade-offs do you want? Take your pick. Here's an impromptu stab at showing some of the values for various languages, based on my own gut feelings, on a scale of 1=awful to 5=great:
Why looky, Java wins by a nose! Of course, ask me on a different day, I'll give you different numbers. Everyone has different opinions about what the numbers would be, and also about the relative importance of each category (most people would agree they probably aren't all equally important, but everyone would pick different favorites.)
And not all the categories are even listed. Portability isn't in there, or C++'s already miserable score would sink even further behind all the rest.
The point isn't to get the numbers right, or even the overall rankings. The point is to show that every language makes very different choices about how to deal with these fundamental trade-offs.
I didn't set out to "fudge" the numbers — I just put my first guess at a value for each language in each category, and left them like that. But look: every language has at least one "1" and at least one "5". Funny how it worked out that way.
I'll show you funny, you ****
Yeah, I know. It's not funny at all. In fact, it just plain sucks that every single language choice you have available to you today is awful in at least one important category:
C is blissfully unaware of its own existence, can crash your service at will, and makes it hard to say almost everything.
C++ is inconsistent and unsafe (a dangerous combination), and the development cycle time is as bad as it gets in this world, period.
Java is a memory pig, and its lack of extensibility, combined with only middling expressiveness, means it's painfully verbose.
Perl is, well, Perl. It used to be so cool, too.
Ruby and Python are slow, lack good concurrency options, and aren't as cool in the library department as Perl or Java.
Lisp and Scheme don't have built-in reflection, lack many important libraries, and they look like oatmeal.
ML and Haskell are both from Planet Weird. And the people of that planet evidently do not believe in libraries.
Face it: we live in the Stone Age of programming. OK, that's a slight exaggeration. I used to program in the Stone Age of programming, and it's definitely gotten somewhat better than it was when I was using Turbo Pascal 5.0 — and even that was pretty sweet compared to back in the early days of FORTRAN. I suppose we're in the Bronze Age, or maybe the Late Stone Age. Either way, we still have the Iron age, the Renaissance, the Industrial Revolution, the Information Age, and the Dot Com Bubble to look forward to.
In the meantime, what's a person to do?
Use a good language at work. To me, that means Java plus either Ruby or Python, your choice. I'd slightly prefer Ruby, but it wouldn't matter.
You should tinker with Lisp in your garage. Eric Raymond's right — it will make you a better programmer. Really, really. And if you'd like to help get us out of the Stone Age a little faster, then Adopt a Lisp (or any functional language).
I'm adopting Gambit Scheme and Kawa Scheme — by which I mean, any tinkering I do will be to help advance those languages in the hope that someday, one or both will be usable for real work. You know, reporting bugs, adding libraries, documenting them, whatever. We're talking about a very small time commitment, but if a lot of people do that, it can add up in a hurry.
If you don't like any of the existing functional languages, then adopt Ruby, since it's awfully close. With a bigger community, Ruby will get to version 2.0 ("Rite", a virtual machine for Ruby) faster, which will enable it to have native threads, better performance, and will quite possibly make it the coolest language on the planet for many tasks.
I'm also adopting Ruby, as an "honorary functional language", one that also happens to kick ass in the OO, reflection, and consistency departments. I'll continue to use it as my scripting language of choice, and I'll be trying to use it for larger projects as well.
And of course, for the forseeable future, the vast majority of all of my work will be in Java, as it has been for the past 8 years or so. But there's nothing wrong with that. Java 5 ain't so bad, all things considered. It's marginally acceptable, for a Stone Age language.
All in all, though, I sometimes wish we'd been born 1000 years from now...
(Published April 01, 2005)
Comments
I don't think "the unwinnable argument" is as futile as you make it out to be.
You ask "How would you convince a Windows fan(atic) that they should switch to Linux on their desktop?" Let's adjust it a bit and ask "how would you convince a VIM fan(atic) that they should switch to EMACS?"
The answer to that question is simple. Give a presentation where you demonstrate how ruthlessly efficient your editor makes you.
I've been using VIM pretty solidly for about seven years. I've also tried to make friends with EMACS at least twice before, including buying "Learning GNU Emacs." Despite this history, indicating that Emacs is not for me, your presentation still made me invest some serious effort into trying again (including writing elisp functions to emulate a few VIM behaviors I couldn't live without).
In the end I couldn't stick with it — I couldn't live without the modality, and I couldn't live with having to use ALT on a regular basis. But I live knowing that in some respects, I'm lacking ultimate power in my editing experience.
What I'm getting at is this. If your language/editor/religion is as good as you think it is, it should be possible for you to demonstrate its awesomeness in a compelling way. If you could write some crazy-cool LISP program, that would really force a competent programmer to stop and say "wow, that really amazing and useful thing you just did is really not possible or elegant in my favorite language X", I think that would be much more effective advocacy than making arguments that sound more like "dude, take a hit of this and you'll see this whole other world."
Posted by: Josh H. at April 1, 2005 07:12 AM
"wow, that really amazing and useful thing you just did is really not possible or elegant in my favorite language X"
Where do we have software that needs to change its behavior on the fly? That's where I'd expect to see Greenspun's Rule in action. One example that springs to mind right now is changing the configuration of a running FLApplication.
Also, what sort of reflection are you thinking of, that Lisp and Scheme don't have?
The Common Lisp spec mandates being able to traverse data structures interactively, possibly modifying them For Schemes, it's up to the implementation to provide them.
The Common Lisp Meta-Object Protocol lets you examine your classes and redefine them on the fly. The MOP is also the basis for a bunch of the Scheme object systems.
Posted by: Derek U. at April 1, 2005 06:51 PM
Interesting thoughts about adopting a language. I know, you said "adopt a Lisp", but I'm stretching a bit. A couple months ago I adopted Rebol. Not because it's the perfect language (it really, really isn't), but because I can play with it and tinker with it and write articles about it to my heart's desire, free from consequence. I don't need to evangelize it, because I don't think it's even reached enough mass where evangelizing would improve its situation. Nobody has come up to me yet saying "Why should I use Rebol?", because not enough people have heard of it. I can play, steal ideas for other projects, and apply what I've learned in my daily work. My articles make it a little easier for new users to look at it once they've heard of it, which has had the effect of adding a couple new users, bringing it a little closer to that critical mass. More importantly, it gets newbies familiar with languages that look a little funny, which will make Lisp less scary when they finally see it.
Meanwhile, I get to enjoy myself because I'm just having fun.
Posted by: Brian W. at April 1, 2005 07:32 PM
Derek — Lisp provides some reasonable facilities for building reflective systems. But that's a far cry from the standard out-of-the-box support you find in Java, Python, and Ruby (in increasing order of reflective power).
The implementation-dependence of the reflection support in both Lisp and Scheme is a problem. If you write code that relies on the implementation-dependent behavior, you've written non-portable code.
The MOP is only useful in cases where you're using classes and objects. In Ruby and Java, every bit of code is associated with a class, and you can reflect on it. For instance, in Lisp, how do you reflectively get a list of all the functions that operate on a particular data type, in a standardized way? Take Strings, for instance. In Ruby, you can do this:
"".methods.sort.grep(/!/)
=> ["capitalize!", "chomp!", "chop!", "delete!", "downcase!", "gsub!",
"lstrip!", "next!", "reverse!", "rstrip!", "slice!", "squeeze!",
"strip!", "sub!", "succ!", "swapcase!", "tr!", "tr_s!", "upcase!"]
et voila. I've got a list of all the String methods that work in-place with side-effects. And Ruby classes are open, so if I want to add new methods to class String, it's trivial:
class String
def endsWith suffix
self =~ /#{suffix}$/
end
end
"foobar".endsWith("bar")
=> 3
Reflection only starts to become truly powerful and useful when the support for it is standardized and applies to every construct in the language.
If all of Lisp were written in CLOS, its reflection capabilities would be just as good as Ruby's, and it would be just as slow as Ruby.
This is what I mean by "fundamental tradeoffs".
Lisp is still cool, don't get me wrong. You're 100% on the mark about us having followed Greenspun's Tenth Rule — using C++ for anything bigger than a few thousand lines starts pushing in this direction. But Lisp is *desperately* in need of an update to modernize it. This kind of thing gets harder and harder with committees, and smaller Lisps with a single maintainer tend to move a lot faster.
That's why I'm leaning more towards Scheme(s). You can influence them more.
Posted by: Steve Yegge at April 1, 2005 09:02 PM
Nice write up — definitely more than I expected. Perhaps Arc, which is _the_ language that maybe we're all hoping for will score 5 on your matrix, on all counts. Where I also think Lisp got it wrong is not supplying the same type of libraries as Java did. If it was 1/2 as loaded with the same amount of libraries, we might've been Lisp-ing now instead of contracting Java-nitis...
Posted by: John Doe at April 1, 2005 09:26 PM
I don't think the Lisp library scene is as bad as you make it out to be. Granted, it's far from complete, but it's really tearing along now.
If you use Common Lisp, ASDF is where it's at:
(require 'asdf)
(require 'asdf-install)
(asdf-install:install 'cl-pcre)
(asdf-install:install 'clsql)
There, just like with CPAN, I installed a (kick-ass) regex lib and a DBI-like system.
ASDF is what made me finally swing to CL in the great CL-Scheme debate. Here's a list of the libraries that are currently available: http://www.cliki.net/asdf-install
As far as how to convince non-Lisp programmers that they're missing something, here's a great example that I just saw this morning: http://www.cs.brown.edu/~sk/Publications/Papers/Published/sk-automata-macros
And, if you can convince someone that Lisp is the One True Language(TM), then the Vim-Emacs debate just falls over as a consequence (I was a Vim user until about a year ago, when Lisp and Haskell moved from distractions to obsessions).
Now I just wish that CL would collapse its namespaces and require TCO. Then I would stop pining for Scheme.
Posted by: Greg P. at April 6, 2005 10:25 PM
You said:
> But look: every language has at least one "1" and
> at least one "5". Funny how it worked out that
> way.
Look again, Steve...you didn't give Perl any 5's...not even for its libraries (CPAN, anyone)? Your anti-Perl bias shows, sir....but you know what? The more I read of your blogs and the more I study and work with other languages, the more I'm starting to develop my own anti-Perl bias.
Well, maybe not anti Perl, but let's just say that I'm really interested in working less with Perl than I have been over the past 5 years.
-dan
Posted by: Dan K. at April 10, 2005 09:35 PM
Sorry Dan. You're right. Perl used to have a 5 in runtime type-safety, and then for some reason I bumped it (and Ruby and Python) down a step, so Perl lost its only 5. I can't even remember why I did it. Maybe because... nah. Can't remember. Like I said, ask me on a different day, I'll give you different numbers.
Posted by: Steve Yegge at April 11, 2005 10:13 PM
I would add another category to use when evaluating languges, "I/O and concurrency". Of course, only Gambit Scheme and Erlang would get a 5 in this category :-)
I/O and concurrency often get short shrift from most language designers (IMHO), presumably because they're considered too real-world and architecture-dependent: "just implement it in a library". But in fact I/O and concurrency should be 100% first-class concepts, and be inexpensive (i.e., "threads" should be ultra-lightweight, and I/O should have zero-copy semantics).
Posted by: Jacob G. at April 13, 2005 11:29 PM
Why tinker with Lisp in the garage? Why not tinker with Smalltalk? I've been playing with a smalltalk environment on my Mac, Squeak. Also is available for other OSes I think (but I don't care). It's really cool, it's fun and interesting, and just way different.
Posted by: Ryan R. at April 18, 2005 11:11 PM