The Cathedral and the Bazaar - unterstein.net

1 The Cathedral and the BazaarEric Steven Raymondcf text and copyright ~esr/writingsAbstractI anatomize a successful open-source project, fetchmail, thatwas run as a deliberate test of some surprising theories aboutso=ware engineering suggested by the history of Linux. I dis-cuss these theories in terms of two fundamentally di:erentdevelopment styles, the Cathedral model of most of the com-mercial world versus the Bazaar model of the Linux show that these models derive from opposing assumptionsabout the nature of the so=ware-debugging task. I then make asustained argument from the Linux experience for the proposi-tion that Given enough eyeballs, all bugs are shallow , suggestproductive analogies with other self-correcting systems of self-ish agents, and conclude with some exploration of the implica-tions of this insight for the future of so= The Cathedral and the BazaarLinux is subversive.

2 Who would have thought even five years ago(1991) that a world-class operating system could coalesce as if bymagic out of part-time hacking by several thousand developers scat-tered all over the planet, connected only by the tenuous strands ofthe Internet?Certainly not I. By the time Linux swam onto my radar screenin early 1993, I had already been involved in Unix and open-sourcedevelopment for ten years. I was one of the firstgnucontributorsin the mid-1980s. I had released a good deal of open-source so=-ware onto the net, developing or co-developing several programs(nethack, Emacs svcandgudmodes, xlife, and others) that are stillin wide use today. I thought I knew how it was overturned much of what I thought I knew.

3 I had beenpreaching the Unix gospel of small tools, rapid prototyping and evo-122 THE MAIL MUST GET THROUGH lutionary programming for years. But I also believed there was acertain critical complexity above which a more centralized, a prioriapproach was required. I believed that the most important so=ware(operating systems and really large tools like the Emacs program-ming editor) needed to be built like cathedrals, carefully cra=ed byindividual wizards or small bands of mages working in splendidisolation, with no beta to be released before its Torvalds s style of development release early and o=en,delegate everything you can, be open to the point of promiscuity came as a surprise. No quiet, reverent Cathedral -building here rather, the Linux community seemed to resemble a great babblingbazaar of di:ering agendas and approaches (aptly symbolized bythe Linux archive sites, who d take submissions fromanyone) out ofwhich a coherent and stable system could seemingly emerge only bya succession of fact that this Bazaar style seemed to work, and work well,came as a distinct shock.

4 As I learned my way around, I workedhard not just at individual projects, but also at trying to under-stand why the Linux world not only didn t fly apart in confusionbut seemed to go from strength to strength at a speed barely imag-inable to mid-1996 I thought I was beginning to understand. Chancehanded me a perfect way to test my theory, in the form of an open-source project that I could consciously try to run in the Bazaar I did and it was a significant is the story of that project. I ll use it to propose some apho-risms about e:ective open-source development. Not all of these arethings I first learned in the Linux world, but we ll see how the Linuxworld gives them particular point.

5 If I m correct, they ll help you un-derstand exactly what it is that makes the Linux community such afountain of good so=ware and, perhaps, they will help you becomemore productive The Mail Must Get ThroughSince 1993 I d been running the technical side of a small free-accessInternet service provider called Chester County InterLink(ccil)inWest Chester, Pennsylvania. I co-foundedcciland wrote our uniquemultiuserbulletin-boardso=ware Today it supports almost three thousand userson thirty lines. The job allowed me 24-hour-a-day access to the netthroughccil s 56K line in fact, the job practically demanded it!I had gotten quite used to instant Internet email. I found havingto periodically telnet over tolocketo check my mail annoying.

6 WhatI wanted was for my mail to be delivered on snark (my home system)so that I would be notified when it arrived and could handle it usingall my local snativemailforwardingprotocol,smtp(Simpl eMailTransfer Protocol), wouldn t suit, because it works best when ma-chines are connected full-time, while my personal machine isn t al-ways on the net, and doesn t have a staticipaddress. What I neededwasaprogramthatwouldreachoutovermy intermittentdialupcon-nection and pull across my mail to be delivered locally. I knew suchthings existed, and that most of them used a simple application pro-tocol calledpop(Post O?ce Protocol).popis now widely supportedby most common mail clients, but at the time, it wasn t built-in tothe mail reader I was needed apop3client.

7 So I went out on the net and found , I found three or four. I used one of them for a while, but itwas missing what seemed an obvious feature, the ability to hack theaddresses on fetched mail so replies would work problem was this: suppose someone named joe onlockesent me mail. If I fetched the mail to snark and then tried to reply toit, my mailer would cheerfully try to ship it to a nonexistent joe onsnark. Hand-editing reply addresses to tack on quickly gotto be a serious was clearly something the computer ought to be doing forme. But none of the existingpopclients knew how! And this bringsus to the first lesson:# 1 Every good work of so=ware starts by scratching a developer spersonal this should have been obvious (it s long been proverbialthat Necessity is the mother of invention ) but too o=en so=waredevelopers spend their days grinding away for pay at programs theyneither need nor love.

8 But not in the Linux world which may ex-plain why the average quality of so=ware originated in the Linuxcommunity is so THE MAIL MUST GET THROUGHSo, did I immediately launch into a furious whirl of coding upa brand-newpop3client to compete with the existing ones? Not onyour life! I looked carefully at thepoputilities I had in hand, askingmyself which one is closest to what I want? Because# 2 Good programmers know what to write. Great ones know whatto rewrite (and reuse).While I don t claim to be a great programmer, I try to imitate important trait of the great ones is constructive laziness. Theyknow that you get an A not for e:ort but for results, and that it salmost always easier to start from a good partial solution than fromnothing at Torvalds, for example, didn t actually try to write Linuxfrom scratch.

9 Instead, he started by reusing code and ideas fromMinix, a tiny Unix-like operating system forpcclones. Eventuallyall the Minix code went away or was completely rewritten butwhile it was there, it provided sca:olding for the infant that wouldeventually become the same spirit, I went looking for an existingpoputility thatwas reasonably well coded, to use as a development source-sharing tradition of the Unix world has always beenfriendly to code reuse (this is why thegnuproject chose Unix asa baseos, in spite of serious reservations about theositself). TheLinux world has taken this tradition nearly to its technological limit;it has terabytes of open sources generally spending timelooking for some else s almost-good-enough is more likely to give yougood results in the Linux world than anywhere it did for me.

10 With those I d found earlier, my second searchmade up a total of nine candidates fetchpop, PopTart, get-mail, gw-pop, pimp, pop-perl, popc, popmail and upop. The one I first settledon was fetchpop by Seung-Hong Oh. I put my header-rewrite fea-ture in it, and made various other improvements which the authoraccepted into his few weeks later, though, I stumbled across the code for pop-client by Carl Harris, and found I had a problem. Though fetch-pop had some good original ideas in it (such as its background-daemon mode), it could only handlepop3and was rather amateur-ishly coded (Seung-Hong was at that time a bright but inexperiencedprogrammer, and both traits showed). Carl s code was better, quite5professional and solid, but his program lacked several importantand rather tricky-to-implement fetchpop features (including thoseI d coded myself).