Loading ...
Sorry, an error occurred while loading the content.

Reading Open Source Code

Expand Messages
  • Shlomi Fish
    Hi all! How much do you read the source code of open source programs you are interested in? I know some people recommend doing so. Paul Graham recommended
    Message 1 of 19 , Dec 26, 2003
    • 0 Attachment
      Hi all!

      How much do you read the source code of open source programs you are
      interested in? I know some people recommend doing so. Paul Graham
      recommended doing so in his "Hackers and Painters" article:

      http://www.paulgraham.com/hp.html

      He says that at his time there wasn't too much code available to learn
      from, while today there's plenty of it out there, so hackers can benefit.

      ESR, also recommends doing so in TAOUP.

      I, personally, did not get too much to reading the code of programs, even
      those that interest me a great deal. I did read the source code of some
      programs, but it was mostly to achieve a certain end. For instance:

      1. I analyzed the code of PySol and GNOME AisleRiot to write programs that
      can generate their boards for input into Freecell Solver.

      2. Having realized that GIMP did not have support for editing gradients
      from scripts, I modified the code (several times for several different
      versions), to do so, and naturally read it in the process.

      3. I once tried to understand why a pam module was giving me trouble, and
      read its code. (only to realize it was fixed in the CVS version, which was
      released as a package for RedHat, but not as an official upgraded version)

      4. Many times I delved deep into the KDE code trying to analyze why a
      certain thing did not work as it should and what I should do to fix it.
      (it's errors can be a bit cryptic).

      5. I recently modified the code of some CPAN modules and read it in the
      process.

      6. Once Mark Jason Dominus gave a task to implement a "patch" program on
      his Perl Quiz of the Week. To find out something, I read the source code
      of GNU patch, and am not ashamed to admit that I couldn't make heads nor
      tails of it.

      7. I read the source code of Subversion while patching it, but naturally,
      not always understood what to do correctly.

      (maybe I exaggeratd in these examples).

      The point is that I don't usually read code for fun, only when I wish to
      achieve a certain end. I'm not saying it's OK that I don't, just that I
      usually prefer writing code (and coming up with new techniques) than
      spending time reading and understanding other people's code in the hope of
      finding new techniques.

      I do know of some people who regularly read the code of other people's
      programs. One distinguished hacker I know, even commented on my code (but
      only said she found it hard to understand). I could testify that I
      found reading the code of various programs I did not write very difficult
      at times as well.

      One thought I have is that clueful hackers can infer the implementation of
      many programs from their behaviour. Thus, reading the code is not always
      time-worthy. However, there are many programs which I don't yet fully know
      how they work (albeit may have a vague idea), and did not read the code so
      far.

      Regards,

      Shlomi Fish

      ----------------------------------------------------------------------
      Shlomi Fish shlomif@...
      Home Page: http://t2.technion.ac.il/~shlomif/

      Writing a BitKeeper replacement is probably easier at this point than getting
      its license changed.

      Matt Mackall on OFTC.net #offtopic.
    • Muli Ben-Yehuda
      ... Quite a lot, thank you. And how are you today? ... he who does not understand history, is bound to reinvent a square wheel . ... Code reading should be a
      Message 2 of 19 , Dec 26, 2003
      • 0 Attachment
        On Fri, Dec 26, 2003 at 08:36:52PM +0200, Shlomi Fish wrote:

        > How much do you read the source code of open source programs you are
        > interested in?

        Quite a lot, thank you. And how are you today?

        > The point is that I don't usually read code for fun, only when I wish to
        > achieve a certain end. I'm not saying it's OK that I don't, just that I
        > usually prefer writing code (and coming up with new techniques) than
        > spending time reading and understanding other people's code in the hope of
        > finding new techniques.

        "he who does not understand history, is bound to reinvent a square
        wheel".

        > One thought I have is that clueful hackers can infer the implementation of
        > many programs from their behaviour. Thus, reading the code is not always
        > time-worthy. However, there are many programs which I don't yet fully know
        > how they work (albeit may have a vague idea), and did not read the code so
        > far.

        Code reading should be a required class in computer science.

        Cheers,
        Muli
        --
        Muli Ben-Yehuda
        http://www.mulix.org | http://mulix.livejournal.com/

        "the nucleus of linux oscillates my world" - gccbot@#offtopic
      • Shlomi Fish
        ... Fine, thanks. Thanks for the verbosity of your answer. ... He who does not understand history, is bound to reinvent an inferior wheel, and gradually
        Message 3 of 19 , Dec 26, 2003
        • 0 Attachment
          On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:

          > On Fri, Dec 26, 2003 at 08:36:52PM +0200, Shlomi Fish wrote:
          >
          > > How much do you read the source code of open source programs you are
          > > interested in?
          >
          > Quite a lot, thank you. And how are you today?
          >

          Fine, thanks. Thanks for the verbosity of your answer.

          > > The point is that I don't usually read code for fun, only when I wish to
          > > achieve a certain end. I'm not saying it's OK that I don't, just that I
          > > usually prefer writing code (and coming up with new techniques) than
          > > spending time reading and understanding other people's code in the hope of
          > >finding new techniques.
          >
          > "he who does not understand history, is bound to reinvent a square
          > wheel".
          >

          "He who does not understand history, is bound to reinvent an inferior
          wheel, and gradually improve it, until it's better than its predecessors".

          Seriously, that may or may not be the case. I'm all for code reuse and
          starting from a working implementation, refactoring it or adapting it to
          your needs, etc. etc. In this case, you do have to study the code or at
          least its interface.

          Now, if you want to create something similar to what already exists, you
          are right that it is a good idea to ask around, or to inspect the code of
          the similar implementations. But reading the code of programs that has
          nothing to do with what you need to achieve, may or may not help you in
          the future.

          As an example of inventing a better wheel, I can (while risking your
          corrolary of Godwin's Law) give Freecell Solver. I did not look for the
          existence of previous efforts, and as a result believed (mistanely, BTW)
          that I need to use meta-moves to solve Freecell with. Meta moves are a
          series of more than one move used to achieve a certain goal in mind. (done
          as one by the solver).

          Now, meta-moves were not implemented in any previous solver, so it made my
          solver quite unique. Furthermore, they later enabled configuring FCS to
          solve a board in a multi-tude of different heuristics. Finally, a
          certain human player who watched the solutions play said they were quite
          innovative and interesting.

          Had I simply looked for previous effort, I would have understood that a
          less interesting approach would have sufficed and so the benefit of doing
          it would have been lost.

          > > One thought I have is that clueful hackers can infer the implementation of
          > > many programs from their behaviour. Thus, reading the code is not always
          > > time-worthy. However, there are many programs which I don't yet fully know
          > > how they work (albeit may have a vague idea), and did not read the code so
          > > far.
          >
          > Code reading should be a required class in computer science.
          >

          Do you mean that people should be trained to read the code that other
          people wrote (even if it's relatively uninteresting)? Or that people
          should be given tasks to read and understand the source code of prominent,
          high-quality, possibly very unique open-source packages? (or something
          else entirely?)

          Regards,

          Shlomi Fish

          ----------------------------------------------------------------------
          Shlomi Fish shlomif@...
          Home Page: http://t2.technion.ac.il/~shlomif/

          Writing a BitKeeper replacement is probably easier at this point than getting
          its license changed.

          Matt Mackall on OFTC.net #offtopic.
        • Muli Ben-Yehuda
          ... Don t mention it. I m on a touch-typing-diet. ... Yes. ... Yes. What I meant is that code is written for humans, not machines. Forcing students to suffer
          Message 4 of 19 , Dec 26, 2003
          • 0 Attachment
            On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:

            > Fine, thanks. Thanks for the verbosity of your answer.

            Don't mention it. I'm on a touch-typing-diet.

            > Do you mean that people should be trained to read the code that other
            > people wrote (even if it's relatively uninteresting)?

            Yes.

            > Or that people
            > should be given tasks to read and understand the source code of prominent,
            > high-quality, possibly very unique open-source packages? (or something
            > else entirely?)

            Yes.

            What I meant is that code is written for humans, not machines. Forcing
            students to suffer through some bad good, while also hopefully getting
            them to enjoy good code, should make that point clear. I am so sick
            and tired of "professional" programmers who write code that if
            translated to Hebrew would be the equivalent of "Dana Kama. Dana
            Holechet. Yofi Dana!", if not complete gibberish. Code is poetry, and
            should be treated as such. It should also be correct, unlike poetry,
            but that's a different aspect I do not wish to discuss right now. I
            hope that was verbose enough ;-)

            Cheers,
            Muli
            --
            Muli Ben-Yehuda
            http://www.mulix.org | http://mulix.livejournal.com/

            "the nucleus of linux oscillates my world" - gccbot@#offtopic
          • Shlomi Fish
            ... Well, have an easy diet, which will hopefully last as long as it is needed. (and no longer than that) ... Maybe you meant that it is written primarily for
            Message 5 of 19 , Dec 26, 2003
            • 0 Attachment
              On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:

              > On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:
              >
              > > Fine, thanks. Thanks for the verbosity of your answer.
              >
              > Don't mention it. I'm on a touch-typing-diet.
              >

              Well, have an easy diet, which will hopefully last as long as it is
              needed. (and no longer than that)

              > > Do you mean that people should be trained to read the code that other
              > > people wrote (even if it's relatively uninteresting)?
              >
              > Yes.
              >
              > > Or that people
              > > should be given tasks to read and understand the source code of prominent,
              > > high-quality, possibly very unique open-source packages? (or something
              > > else entirely?)
              >
              > Yes.
              >
              > What I meant is that code is written for humans, not machines.

              Maybe you meant that it is written primarily for humans and secondly for
              machines? (paraphrasing on Abelson & Sussman from SICP)

              > Forcing
              > students to suffer through some bad code, while also hopefully getting
              > them to enjoy good code, should make that point clear.

              A better idea may be to tell them to implement a new feature, re-organize
              the code, or whatever based on an already working implementation. That way
              they will have to read the code, but will also make them appreciate how
              well or how bad it is written as.

              > I am so sick
              > and tired of "professional" programmers who write code that if
              > translated to Hebrew would be the equivalent of "Dana Kama. Dana
              > Holechet. Yofi Dana!", if not complete gibberish.

              Actually, I'm not ashamed to admit that some of the Perl and shell scripts
              I wrote are not better than "Dana Kama, etc.". But you're in luck. Some of
              the code I encountered would be translated to Hebrew as "Asiti eth
              hatheatron hakeilo nehmad kazeh, veraiti sham shalosh anashim osim ehad
              lasheni shikha kazoth".

              > Code is poetry, and
              > should be treated as such.

              Well, not all code. Production code, yes. But throwaway code, or such
              scripts that just accomplish one thing well, can be left as pretty ugly.

              > It should also be correct, unlike poetry,
              > but that's a different aspect I do not wish to discuss right now. I
              > hope that was verbose enough ;-)
              >

              Yes, it was. Thanks.

              Regards,

              Shlomi Fish

              ----------------------------------------------------------------------
              Shlomi Fish shlomif@...
              Home Page: http://t2.technion.ac.il/~shlomif/

              You are banished! You are banished! You are banished!

              Hey? I'm just kidding!
            • Tzahi Fadida
              ... I think the issue is indeed writting for other people. since we are a long time away in computer years from assembly(which was quite fun at at the time),
              Message 6 of 19 , Dec 26, 2003
              • 0 Attachment
                > -----Original Message-----
                > From: Shlomi Fish [mailto:shlomif@...]
                > Sent: Friday, December 26, 2003 10:33 PM
                > To: hackers-il@yahoogroups.com
                > Subject: Re: [hackers-il] Re: Reading Open Source Code
                >
                >
                > On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:
                >
                > > On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:
                > >
                > > > Fine, thanks. Thanks for the verbosity of your answer.
                > >
                > > Don't mention it. I'm on a touch-typing-diet.
                > >
                >
                > Well, have an easy diet, which will hopefully last as long as it is
                > needed. (and no longer than that)
                >
                > > > Do you mean that people should be trained to read the code that other
                > > > people wrote (even if it's relatively uninteresting)?
                > >
                > > Yes.
                > >
                > > > Or that people
                > > > should be given tasks to read and understand the source code of prominent,
                > > > high-quality, possibly very unique open-source packages? (or something
                > > > else entirely?)
                > >
                > > Yes.
                > >
                > > What I meant is that code is written for humans, not machines.
                >
                > Maybe you meant that it is written primarily for humans and secondly for
                > machines? (paraphrasing on Abelson & Sussman from SICP)

                I think the issue is indeed writting for other people. since we are a long
                time away in computer years from assembly(which was quite fun at at
                the time), nowdays you should think of what the next person that will
                maintain or extend on your code will need.
                I think that this issue is not given its rightful place at computer sciences.
                I would refer you to "concepts of programming languages", SEBETA.
                Aside from that, i think many open source projects would benefit from
                a good design. the thing is, its not fun for a lot of people to design first
                based on requirements, using a methdology. so they use XP "kinds" of
                organic evolution to their software cycles.
                However, this projects suffer from problems that comes from mistakes just
                like that. Mistakes, that are apparent when the project needs to grow
                beyond the 12 people stage.
                I didn't suggest this before because i don't have the time, and the usual
                response would be to do it yourself but i think that someone should
                round up a group of people that would design projects based on
                requirements for teams that do open source projects.

                anyway, back to the main issue. I think that reading an open source
                would be much easier if it was well designed using a proven methodology
                and "enough" documented so that you can at least pinpoint the
                whereabout of a glitch and fix it in half the time it took you.

                >
                > > Forcing
                > > students to suffer through some bad code, while also hopefully getting
                > > them to enjoy good code, should make that point clear.
                >
                > A better idea may be to tell them to implement a new feature, re-organize
                > the code, or whatever based on an already working implementation. That way
                > they will have to read the code, but will also make them appreciate how
                > well or how bad it is written as.
                >
                > > I am so sick
                > > and tired of "professional" programmers who write code that if
                > > translated to Hebrew would be the equivalent of "Dana Kama. Dana
                > > Holechet. Yofi Dana!", if not complete gibberish.
                >
                > Actually, I'm not ashamed to admit that some of the Perl and shell scripts
                > I wrote are not better than "Dana Kama, etc.". But you're in luck. Some of
                > the code I encountered would be translated to Hebrew as "Asiti eth
                > hatheatron hakeilo nehmad kazeh, veraiti sham shalosh anashim osim ehad
                > lasheni shikha kazoth".
                >
                > > Code is poetry, and
                > > should be treated as such.
                >
                > Well, not all code. Production code, yes. But throwaway code, or such
                > scripts that just accomplish one thing well, can be left as pretty ugly.

                they can be maybe left ugly but things like "use strict" in perl are a must
                if you want to take the code more seriously.

                >
                > > It should also be correct, unlike poetry,
                > > but that's a different aspect I do not wish to discuss right now. I
                > > hope that was verbose enough ;-)
                > >
                >
                > Yes, it was. Thanks.
                >
                > Regards,
                >
                > Shlomi Fish
                >
                > ----------------------------------------------------------------------
                > Shlomi Fish shlomif@...
                > Home Page: http://t2.technion.ac.il/~shlomif/
                >
                > You are banished! You are banished! You are banished!
                >
                > Hey? I'm just kidding!
                >
                > To unsubscribe from this group, send an email to:
                > hackers-il-unsubscribe@egroups.com
                >
                >
                >
                >
                > Yahoo! Groups Links
                >
                > To visit your group on the web, go to:
                > http://groups.yahoo.com/group/hackers-il/
                >
                > To unsubscribe from this group, send an email to:
                > hackers-il-unsubscribe@yahoogroups.com
                >
                > Your use of Yahoo! Groups is subject to:
                > http://docs.yahoo.com/info/terms/
                >
                >
                >
              • Shlomi Fish
                ... Correctly. ... Right again. Reading code and starting from a working codebase only to extend it is rarely done in programming schools, which is a shame,
                Message 7 of 19 , Dec 26, 2003
                • 0 Attachment
                  On Fri, 26 Dec 2003, Tzahi Fadida wrote:

                  > > -----Original Message-----
                  > > From: Shlomi Fish [mailto:shlomif@...]
                  > > Sent: Friday, December 26, 2003 10:33 PM
                  > > To: hackers-il@yahoogroups.com
                  > > Subject: Re: [hackers-il] Re: Reading Open Source Code
                  > >
                  > >
                  > > On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:
                  > >
                  > > > On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:
                  > > >
                  > > > > Fine, thanks. Thanks for the verbosity of your answer.
                  > > >
                  > > > Don't mention it. I'm on a touch-typing-diet.
                  > > >
                  > >
                  > > Well, have an easy diet, which will hopefully last as long as it is
                  > > needed. (and no longer than that)
                  > >
                  > > > > Do you mean that people should be trained to read the code that other
                  > > > > people wrote (even if it's relatively uninteresting)?
                  > > >
                  > > > Yes.
                  > > >
                  > > > > Or that people
                  > > > > should be given tasks to read and understand the source code of prominent,
                  > > > > high-quality, possibly very unique open-source packages? (or something
                  > > > > else entirely?)
                  > > >
                  > > > Yes.
                  > > >
                  > > > What I meant is that code is written for humans, not machines.
                  > >
                  > > Maybe you meant that it is written primarily for humans and secondly for
                  > > machines? (paraphrasing on Abelson & Sussman from SICP)
                  >
                  > I think the issue is indeed writting for other people. since we are a long
                  > time away in computer years from assembly(which was quite fun at at
                  > the time), nowdays you should think of what the next person that will
                  > maintain or extend on your code will need.

                  Correctly.

                  > I think that this issue is not given its rightful place at computer sciences.

                  Right again. Reading code and starting from a working codebase only to
                  extend it is rarely done in programming schools, which is a shame, because
                  it's a very needed skill.

                  > I would refer you to "concepts of programming languages", SEBETA.

                  What is it? A book? An article? Can you give a link? What is "SEBETA"?

                  > Aside from that, i think many open source projects would benefit from
                  > a good design.

                  Naturally. But so would a great deal of in-house or even commercial
                  software, some of which was heavily designed and written with a lot of
                  "red tape". If your project suffers from bad design, you should refactor
                  it, so it will have a good one.

                  Otherwise, (I may have misinterpreted you) it is always a good idea to
                  design and plan a change before you start hacking on it. Unless of course
                  we are talking about mindless logical transformations, where you just
                  change stuff in the code.

                  > the thing is, its not fun for a lot of people to design first
                  > based on requirements, using a methdology. so they use XP "kinds" of
                  > organic evolution to their software cycles.

                  I'm not entirely familiar with Extreme Programming (= XP). I only heard
                  about it and attended the first half of (a very good) lecture about it.
                  What I do know is that in the open source world, the recommended way of
                  organizing and designing a software project is the "Bazaar" way as
                  described by ESR in "The Cathedral and the Bazaar" (which is an excellent
                  read even if you disagree with it). The Bazaar model seems very similar to
                  XP, but naturally accepts the constraints of an Internet-wide development
                  and that open source software is such that needs to be deployed on a large
                  number of variant systems.

                  I don't think either XP or the Bazaar model tell you not to design.
                  Rather, they tell you to do incremental design. I.e: if you want to add a
                  feature, then mentally plan how to integrate it into the core code without
                  breaking its modularity. If you feel the code is unmodular, refactor it
                  according to a planned design. In XP, there is a mandatory one design
                  meeting per day.

                  > However, this projects suffer from problems that comes from mistakes just
                  > like that. Mistakes, that are apparent when the project needs to grow
                  > beyond the 12 people stage.

                  Luckily, most projects don't need to grow beyond the 12 people stage. Most
                  projects have a core team of 1 or 2 developrs, and a halo of a bit more.
                  I'm personally involved in Subversion, where the number of contributors is
                  very large, and yet it's in pretty good shape. Still, the number of people
                  doing most of the work is relatively small.

                  I think open source has proved that its projects can scale to a large
                  number of developers, without grinding to a halt, or its code
                  deteriorating into a very low quality mess. The reasons why this is the
                  case were discussed in the "Go! Linux" panel, and also in many other
                  places and occasions.

                  > I didn't suggest this before because i don't have the time, and the usual
                  > response would be to do it yourself but i think that someone should
                  > round up a group of people that would design projects based on
                  > requirements for teams that do open source projects.
                  >

                  I don't quite understand this last sentence, but I'll try. I think you
                  mean that they would design tools to aid in quality control, software
                  management, etc. of open source projects. Like SourceForge/GForge, only on
                  steroids. Well, that may be a good idea, but of course will involve a lot
                  of work.

                  If you want these tools to be programming language-aware, then you'll need
                  to handle all the different languages out there. Good luck to whoever
                  choose to go this way.

                  > anyway, back to the main issue. I think that reading an open source
                  > would be much easier if it was well designed using a proven methodology
                  > and "enough" documented so that you can at least pinpoint the
                  > whereabout of a glitch and fix it in half the time it took you.
                  >

                  "proven methodology" - do such beasts exist? I think you've spent too much
                  time being an industrial engineer. Have you ever read "The Mythical
                  Man-Month"?

                  Also, define "adequately documented"? Do you want a Literate Programming
                  Style book documenting the program? Do you want a detailed architecture
                  document? Is a user's guide enough? Do you agree with what XP says that
                  even comments should usually be unnecessary and the code should be
                  self-explanatory?

                  I'm not saying a code should not be documented at all. However,
                  documenting a working code, and keeping the documentation in sync with the
                  code is very difficult. The best way to understand a code is to ask a
                  programmer who worked on it extensively.

                  I did encounter some times where code I did not write was very easy to
                  understand. One such case was a Perl module I needed for a CGI script I
                  maintain. The code proved to be not particularly extensible, but I was
                  able to tweak it to do what I needed quite easily. It was heavily
                  commented, but I felt that the comments kind of prevented the good flow
                  of reading it and were redundant. (so eat your hearts out - Python
                  bigots!)

                  > > > Code is poetry, and
                  > > > should be treated as such.
                  > >
                  > > Well, not all code. Production code, yes. But throwaway code, or such
                  > > scripts that just accomplish one thing well, can be left as pretty ugly.
                  >
                  > they can be maybe left ugly but things like "use strict" in perl are a must
                  > if you want to take the code more seriously.
                  >

                  Correct. I use "use strict" and "-w" on every serious code I write. I
                  don't usually on "perl -e " command line one liners, though, but they
                  fall into the second category.


                  ----------------------------------------------------------------------
                  Shlomi Fish shlomif@...
                  Home Page: http://t2.technion.ac.il/~shlomif/

                  You are banished! You are banished! You are banished!

                  Hey? I'm just kidding!
                • Tzahi Fadida
                  ... a book http://www.amazon.com/exec/obidos/tg/detail/-/0201752956/qid=1072484373/sr=1-1/r ef=sr_1_1/104-7844860-9557568?v=glance&s=books ... yeah, before,
                  Message 8 of 19 , Dec 26, 2003
                  • 0 Attachment
                    > -----Original Message-----
                    > From: Shlomi Fish [mailto:shlomif@...]
                    > Sent: Saturday, December 27, 2003 1:40 AM
                    > To: hackers-il@yahoogroups.com
                    > Subject: RE: [hackers-il] Re: Reading Open Source Code
                    >
                    >
                    > On Fri, 26 Dec 2003, Tzahi Fadida wrote:
                    >
                    > > > -----Original Message-----
                    > > > From: Shlomi Fish [mailto:shlomif@...]
                    > > > Sent: Friday, December 26, 2003 10:33 PM
                    > > > To: hackers-il@yahoogroups.com
                    > > > Subject: Re: [hackers-il] Re: Reading Open Source Code
                    > > >
                    > > >
                    > > > On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:
                    > > >
                    > > > > On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:
                    > > > >
                    > > > > > Fine, thanks. Thanks for the verbosity of your answer.
                    > > > >
                    > > > > Don't mention it. I'm on a touch-typing-diet.
                    > > > >
                    > > >
                    > > > Well, have an easy diet, which will hopefully last as long as it is
                    > > > needed. (and no longer than that)
                    > > >
                    > > > > > Do you mean that people should be trained to read the code that other
                    > > > > > people wrote (even if it's relatively uninteresting)?
                    > > > >
                    > > > > Yes.
                    > > > >
                    > > > > > Or that people
                    > > > > > should be given tasks to read and understand the source code
                    > of prominent,
                    > > > > > high-quality, possibly very unique open-source packages? (or something
                    > > > > > else entirely?)
                    > > > >
                    > > > > Yes.
                    > > > >
                    > > > > What I meant is that code is written for humans, not machines.
                    > > >
                    > > > Maybe you meant that it is written primarily for humans and secondly for
                    > > > machines? (paraphrasing on Abelson & Sussman from SICP)
                    > >
                    > > I think the issue is indeed writting for other people. since we are a long
                    > > time away in computer years from assembly(which was quite fun at at
                    > > the time), nowdays you should think of what the next person that will
                    > > maintain or extend on your code will need.
                    >
                    > Correctly.
                    >
                    > > I think that this issue is not given its rightful place at computer
                    > sciences.
                    >
                    > Right again. Reading code and starting from a working codebase only to
                    > extend it is rarely done in programming schools, which is a shame, because
                    > it's a very needed skill.
                    >
                    > > I would refer you to "concepts of programming languages", SEBETA.
                    >
                    > What is it? A book? An article? Can you give a link? What is "SEBETA"?

                    a book
                    http://www.amazon.com/exec/obidos/tg/detail/-/0201752956/qid=1072484373/sr=1-1/r
                    ef=sr_1_1/104-7844860-9557568?v=glance&s=books

                    >
                    > > Aside from that, i think many open source projects would benefit from
                    > > a good design.
                    >
                    > Naturally. But so would a great deal of in-house or even commercial
                    > software, some of which was heavily designed and written with a lot of
                    > "red tape". If your project suffers from bad design, you should refactor
                    > it, so it will have a good one.
                    >
                    > Otherwise, (I may have misinterpreted you) it is always a good idea to
                    > design and plan a change before you start hacking on it. Unless of course
                    > we are talking about mindless logical transformations, where you just
                    > change stuff in the code.

                    yeah,
                    before, during and make sure that the maintainers also maintain the
                    design.

                    >
                    > > the thing is, its not fun for a lot of people to design first
                    > > based on requirements, using a methdology. so they use XP "kinds" of
                    > > organic evolution to their software cycles.
                    >
                    > I'm not entirely familiar with Extreme Programming (= XP). I only heard
                    > about it and attended the first half of (a very good) lecture about it.
                    > What I do know is that in the open source world, the recommended way of
                    > organizing and designing a software project is the "Bazaar" way as
                    > described by ESR in "The Cathedral and the Bazaar" (which is an excellent
                    > read even if you disagree with it). The Bazaar model seems very similar to
                    > XP, but naturally accepts the constraints of an Internet-wide development
                    > and that open source software is such that needs to be deployed on a large
                    > number of variant systems.
                    >
                    > I don't think either XP or the Bazaar model tell you not to design.
                    > Rather, they tell you to do incremental design. I.e: if you want to add a
                    > feature, then mentally plan how to integrate it into the core code without
                    > breaking its modularity. If you feel the code is unmodular, refactor it
                    > according to a planned design. In XP, there is a mandatory one design
                    > meeting per day.

                    XP is against the traditional sens of design. I was attending a seminar
                    where one of the doctoral works was presented. it was on behavioural
                    stuff so that part isn't important here, however, they talked about
                    how they do it in companies and what are the limitations and offers to
                    make it better etc...
                    Now, here comes the interesting part - While i liked it in general for
                    small to medium term projects the inhouse staff of Information Systems
                    almost kicked them down the stairs. I myself feel that of course, for
                    big and "important" projects where you need a good design to refer
                    to where there is a problem this kind of methodology is out of place.
                    However, the throwaway with overall design and start programming
                    and document a bit inside the programm and in the end is what
                    most software houses do already and it works for the most simple
                    and non life threatening or most reliable projects.
                    The excuse is of course that the design is supposed to be recycled
                    several time during the software life time and no one does that because
                    of budgetery concerns. To the scientific sense this of course stinks but
                    the commercial world do what they want and what is considered better
                    for them.
                    I must stress here though that there are many militairy and proven "safe"
                    systems that must have a good design. one example would be the black
                    boxes in airplanes or in the robots they send to mars. This can be done
                    using mathematical models that can be checked by a computer. Of course
                    it would be impossible for large scale projects to be built like this.
                    So, the next best thing is to take something like GAIA for agents,
                    or UML for small projects, etc... to each appropriate field of software,
                    and make sure it works.

                    THE MAIN PROBLEM i see here is that in CS and any other faculty in the
                    technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
                    1999). this code is like the doctors - not to do harm and negligence.
                    see a Book:
                    Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1


                    >
                    > > However, this projects suffer from problems that comes from mistakes just
                    > > like that. Mistakes, that are apparent when the project needs to grow
                    > > beyond the 12 people stage.
                    >
                    > Luckily, most projects don't need to grow beyond the 12 people stage. Most
                    > projects have a core team of 1 or 2 developrs, and a halo of a bit more.
                    > I'm personally involved in Subversion, where the number of contributors is
                    > very large, and yet it's in pretty good shape. Still, the number of people
                    > doing most of the work is relatively small.
                    >
                    > I think open source has proved that its projects can scale to a large
                    > number of developers, without grinding to a halt, or its code
                    > deteriorating into a very low quality mess. The reasons why this is the
                    > case were discussed in the "Go! Linux" panel, and also in many other
                    > places and occasions.

                    I'll have to check that. I guess good will and "this is my baby and i know
                    anything about it" is a good part of that.

                    >
                    > > I didn't suggest this before because i don't have the time, and the usual
                    > > response would be to do it yourself but i think that someone should
                    > > round up a group of people that would design projects based on
                    > > requirements for teams that do open source projects.
                    > >
                    >
                    > I don't quite understand this last sentence, but I'll try. I think you
                    > mean that they would design tools to aid in quality control, software
                    > management, etc. of open source projects. Like SourceForge/GForge, only on
                    > steroids. Well, that may be a good idea, but of course will involve a lot
                    > of work.
                    >
                    > If you want these tools to be programming language-aware, then you'll need
                    > to handle all the different languages out there. Good luck to whoever
                    > choose to go this way.

                    Though a really good idea, and i don't think its really possible using
                    regular methodologies like UML for reasons that would take a course time
                    to explain. Mainly due to the many models. Orthogonality plays a crucial
                    roles in this matters.
                    Anyway the IE IS area has a new methodology called OPM, which
                    I must state here first that its patented and thus i have personal feelings
                    about it, though it maybe continues our last discussion of universities
                    internal funding sources. this methodology is the new wave suppose
                    to supercede UML using one model for all, and supposed to be able
                    to one model - > every programming language.
                    And I for one, believe it is possible.
                    Ok, comming back to the issue, i was talking about OUTSOURCING
                    open source style. i.e. people with design skills at different areas
                    with different methodologies would design projects with the base
                    requirements and maybe a meeting place between designers and
                    project developers that will commision designs and refactoring from
                    them.
                    The main problem i see that this is supposed to solve, is the experienced
                    designer issues, and the knowledge time - to - develop issues.

                    >
                    > > anyway, back to the main issue. I think that reading an open source
                    > > would be much easier if it was well designed using a proven methodology
                    > > and "enough" documented so that you can at least pinpoint the
                    > > whereabout of a glitch and fix it in half the time it took you.
                    > >
                    >
                    > "proven methodology" - do such beasts exist? I think you've spent too much
                    > time being an industrial engineer. Have you ever read "The Mythical
                    > Man-Month"?

                    yes there are, though rare. mathematical models which are "safe" can
                    be proved, however these are for small scale. Mostly a few lines of code.
                    No, i was talking about how much time they exists and if they have a
                    record track of successful usage.

                    >
                    > Also, define "adequately documented"? Do you want a Literate Programming
                    > Style book documenting the program? Do you want a detailed architecture
                    > document? Is a user's guide enough? Do you agree with what XP says that
                    > even comments should usually be unnecessary and the code should be
                    > self-explanatory?

                    don't count of self-explanatory, this rarely happen when a multitude of
                    people write different bits of code.
                    i can't define here what is adequate precisely because every project
                    is different. I can only say that it should be adequate in the sense
                    that further down the line the code could be maintained using this
                    documentation.
                    This is the XP way of thinking - "adequate enough".
                    I am still ambivalent about this so i can't be accurate.

                    >
                    > I'm not saying a code should not be documented at all. However,
                    > documenting a working code, and keeping the documentation in sync with the
                    > code is very difficult. The best way to understand a code is to ask a
                    > programmer who worked on it extensively.

                    yes. again you are quoting XP, and i'll requote traditional methodologies.

                    >
                    > I did encounter some times where code I did not write was very easy to
                    > understand. One such case was a Perl module I needed for a CGI script I
                    > maintain. The code proved to be not particularly extensible, but I was
                    > able to tweak it to do what I needed quite easily. It was heavily
                    > commented, but I felt that the comments kind of prevented the good flow
                    > of reading it and were redundant. (so eat your hearts out - Python
                    > bigots!)

                    yes, if the code is self explenatory around the 200 lines or so, then
                    there is no need to comment every line. java's javadoc is very good
                    to my thinking.

                    >
                    > > > > Code is poetry, and
                    > > > > should be treated as such.
                    > > >
                    > > > Well, not all code. Production code, yes. But throwaway code, or such
                    > > > scripts that just accomplish one thing well, can be left as pretty ugly.
                    > >
                    > > they can be maybe left ugly but things like "use strict" in perl are a must
                    > > if you want to take the code more seriously.
                    > >
                    >
                    > Correct. I use "use strict" and "-w" on every serious code I write. I
                    > don't usually on "perl -e " command line one liners, though, but they
                    > fall into the second category.

                    right

                    >
                    >
                    > ----------------------------------------------------------------------
                    > Shlomi Fish shlomif@...
                    > Home Page: http://t2.technion.ac.il/~shlomif/
                    >
                    > You are banished! You are banished! You are banished!
                    >
                    > Hey? I'm just kidding!
                    >
                    > To unsubscribe from this group, send an email to:
                    > hackers-il-unsubscribe@egroups.com
                    >
                    >
                    >
                    > Yahoo! Groups Links
                    >
                    > To visit your group on the web, go to:
                    > http://groups.yahoo.com/group/hackers-il/
                    >
                    > To unsubscribe from this group, send an email to:
                    > hackers-il-unsubscribe@yahoogroups.com
                    >
                    > Your use of Yahoo! Groups is subject to:
                    > http://docs.yahoo.com/info/terms/
                    >
                    >
                    >
                  • Nadav Har'El
                    ... At this stage, I don t normally say to myself Wow, this Mozilla program is nice and I use it everyday, why don t I just read its entire code . But there
                    Message 9 of 19 , Dec 27, 2003
                    • 0 Attachment
                      On Fri, Dec 26, 2003, Shlomi Fish wrote about "[hackers-il] Reading Open Source Code":
                      > Hi all!
                      >
                      > How much do you read the source code of open source programs you are
                      > interested in? I know some people recommend doing so. Paul Graham
                      > recommended doing so in his "Hackers and Painters" article:

                      At this stage, I don't normally say to myself "Wow, this Mozilla program is
                      nice and I use it everyday, why don't I just read its entire code". But there
                      are good reasons why I have read source code in the past, and continue to do
                      so once in a while:

                      1. To learn.

                      I started learning Unix by reading source code - or actually, by *typing*
                      source code, because no network was available to me and the only source
                      code I had access to was printed in books (most notably, Kernighan & Pike's
                      "The Unix Programming Environment" and Kernighan & Ritchie's "The C
                      Programming Language".
                      After reading (and typing) all the programs in those books, I sometimes
                      took source code of other Unix programs (which was available at the time)
                      to learn from it techniques I didn't yet know, such as how to write programs
                      for the graphical terminal we had back then (The AT&T 5620, a.k.a. the
                      jerq). Today I still read available code (usually free software, of course)
                      when I want to learn new techniques - in addition to reading the relevant
                      manuals, of course.

                      2. To change.

                      The most important reason to read someone else's code is if you want to
                      change it. My first experience of this sort was 18 years ago (wow, time
                      flies...) when I thought that BASIC was too lacking a language, and I
                      thought I knew better and wanted to invent a new feature in BASIC. So I
                      found the source code (in C, running on UNIX) of a BASIC interpreter, and
                      with some help from my father (I was a lousy programmer at the time) I
                      added a new feature to that BASIC interpreter. That interpreter was not
                      free software, so I had nothing to do with my "patch", and it was lost
                      in the mists of time (or in my infamous "rm" accident).
                      More recent experiences of this sort involved me sending patches (and
                      them being accepted) for problems that bothered me in Gzip, Apache, and
                      OpenSSL. Fixing these problems naturally required reading these projects'
                      source code - usually not the entire source code but only certain (but
                      sometimes substantial) parts.

                      3. To do better.

                      Think you can write a faster web-server from Apache? Think you can write
                      a version of OpenOffice which takes less memory? You better look into the
                      code and check if your hunches are write, or whether the tricks you were
                      thinking of implementing in your new super-duper project are already in
                      the familiar code, and they are not the silver bullets you hoped for.

                      --
                      Nadav Har'El | Saturday, Dec 27 2003, 2 Tevet 5764
                      nyh@... |-----------------------------------------
                      Phone: +972-53-790466, ICQ 13349191 |My password is my dog's name. His name is
                      http://nadav.harel.org.il |a#j!4@h, but I change it every month.
                    • Orna Agmon
                      ... I believe most people follow this rational, if they actually do reas source code. And the question which bothers me now is: We have an often-assumed
                      Message 10 of 19 , Dec 27, 2003
                      • 0 Attachment
                        On Sat, 27 Dec 2003, Nadav Har'El wrote:

                        > On Fri, Dec 26, 2003, Shlomi Fish wrote about "[hackers-il] Reading Open Source Code":
                        > > Hi all!
                        > >
                        > > How much do you read the source code of open source programs you are
                        > > interested in? I know some people recommend doing so. Paul Graham
                        > > recommended doing so in his "Hackers and Painters" article:
                        >
                        > At this stage, I don't normally say to myself "Wow, this Mozilla program is
                        > nice and I use it everyday, why don't I just read its entire code". But there
                        > are good reasons why I have read source code in the past, and continue to do
                        > so once in a while:
                        >
                        > 1. To learn.
                        >
                        > I started learning Unix by reading source code - or actually, by *typing*
                        > source code, because no network was available to me and the only source
                        > code I had access to was printed in books (most notably, Kernighan & Pike's
                        > "The Unix Programming Environment" and Kernighan & Ritchie's "The C
                        > Programming Language".
                        > After reading (and typing) all the programs in those books, I sometimes
                        > took source code of other Unix programs (which was available at the time)
                        > to learn from it techniques I didn't yet know, such as how to write programs
                        > for the graphical terminal we had back then (The AT&T 5620, a.k.a. the
                        > jerq). Today I still read available code (usually free software, of course)
                        > when I want to learn new techniques - in addition to reading the relevant
                        > manuals, of course.
                        >
                        > 2. To change.
                        >
                        > The most important reason to read someone else's code is if you want to
                        > change it. My first experience of this sort was 18 years ago (wow, time
                        > flies...) when I thought that BASIC was too lacking a language, and I
                        > thought I knew better and wanted to invent a new feature in BASIC. So I
                        > found the source code (in C, running on UNIX) of a BASIC interpreter, and
                        > with some help from my father (I was a lousy programmer at the time) I
                        > added a new feature to that BASIC interpreter. That interpreter was not
                        > free software, so I had nothing to do with my "patch", and it was lost
                        > in the mists of time (or in my infamous "rm" accident).
                        > More recent experiences of this sort involved me sending patches (and
                        > them being accepted) for problems that bothered me in Gzip, Apache, and
                        > OpenSSL. Fixing these problems naturally required reading these projects'
                        > source code - usually not the entire source code but only certain (but
                        > sometimes substantial) parts.
                        >
                        > 3. To do better.
                        >
                        > Think you can write a faster web-server from Apache? Think you can write
                        > a version of OpenOffice which takes less memory? You better look into the
                        > code and check if your hunches are write, or whether the tricks you were
                        > thinking of implementing in your new super-duper project are already in
                        > the familiar code, and they are not the silver bullets you hoped for.

                        I believe most people follow this rational, if they actually do reas
                        source code. And the question which bothers me now is: We have an
                        often-assumed assumption, which we take for granted:

                        If it is open-source, it must be safe, it cannot have Trojans or otherwise
                        be malicious (to the one running it), because somewhere out there someone
                        who understands the code other than the original programmer would have
                        noticed and screamed about it to the world. (I do not count mistakes here-
                        errors happen).

                        If everybody behaves like this, then this assumption is totally out of
                        order.

                        Orna.
                      • Muli Ben-Yehuda
                        ... It really depends on the project, the number of active developers, their attitude towards reviewing each other s source code and the ammount of community
                        Message 11 of 19 , Dec 27, 2003
                        • 0 Attachment
                          On Sat, Dec 27, 2003 at 12:03:36PM +0200, Orna Agmon wrote:

                          > If it is open-source, it must be safe, it cannot have Trojans or otherwise
                          > be malicious (to the one running it), because somewhere out there someone
                          > who understands the code other than the original programmer would have
                          > noticed and screamed about it to the world. (I do not count mistakes here-
                          > errors happen).
                          >
                          > If everybody behaves like this, then this assumption is totally out of
                          > order.

                          It really depends on the project, the number of active developers,
                          their attitude towards reviewing each other's source code and the
                          ammount of community involvement. Compare and contrast:

                          - someone tried to inject a backdoor to the Linux kernel. It was
                          immediately discovered.

                          - there was a database (informix?) that was released as open source a
                          few years ago. Only after a couple of years of it being "open" did
                          someone discover that the original programmer left a huge backdoor.

                          Therefore, just assuming that if something is open it contains no
                          surprises is dangerous.

                          Cheers,
                          Muli
                          --
                          Muli Ben-Yehuda
                          http://www.mulix.org | http://mulix.livejournal.com/

                          "the nucleus of linux oscillates my world" - gccbot@#offtopic
                        • Shlomi Fish
                          ... Heh, me too. Use the source, Luke! ... You mean the course that used to teach XINU and now User-mode-Linux? I suppose that does qualify. However, in the EE
                          Message 12 of 19 , Dec 27, 2003
                          • 0 Attachment
                            On Sat, 27 Dec 2003, guy keren wrote:

                            >
                            > On Sat, 27 Dec 2003, Nadav Har'El wrote:
                            >
                            > > 1. To learn.
                            > >
                            > > 2. To change.
                            > >
                            > > 3. To do better.
                            >
                            > add to that "4. to find features". i often find myself reading the source
                            > code in order to find undocumented features. and with open-source - there
                            > is often rather sparse documentation, and hence a lot of undocumented
                            > features.
                            >
                            > this also includes understanding how some feature works, in order to find
                            > why my program does not work well or properly.
                            >
                            > this could include reading the kernel's source in order to find what
                            > exactly some poorly documented ioctl() do, or reading the source of a
                            > networking library in order to find why it sometimes aborts the main loop.
                            >

                            Heh, me too. Use the source, Luke!

                            > i got into the habbit of reading source to the leevl that i sometimes do
                            > that without first checking if the FAQ contains the question i want an
                            > answer for.
                            >



                            > and who said there is no teaching of "reading source code" in the academi?
                            > what is the "intorodction to operating systems" course, of the technion's
                            > CS department, if not a course in reading source code, finding bugs and
                            > adding features?

                            You mean the course that used to teach XINU and now User-mode-Linux? I
                            suppose that does qualify. However, in the EE department, there's a
                            different corresponding course, in which we just have to write various
                            user-land programs that use the system facilities. (a shell, a malloc/free
                            implementation, etc.) It's not a bad course, but we don't get to do any
                            kernel hacking when starting from a working code base.

                            I still think it's not enough. The purpose of the OS course is to teach
                            about OS design, and its side-effect is reading and extending code. But we
                            could probably use a dedicated course for just that.

                            Regards,

                            Shlomi Fish


                            ----------------------------------------------------------------------
                            Shlomi Fish shlomif@...
                            Home Page: http://t2.technion.ac.il/~shlomif/

                            You are banished! You are banished! You are banished!

                            Hey? I'm just kidding!
                          • guy keren
                            ... add to that 4. to find features . i often find myself reading the source code in order to find undocumented features. and with open-source - there is
                            Message 13 of 19 , Dec 27, 2003
                            • 0 Attachment
                              On Sat, 27 Dec 2003, Nadav Har'El wrote:

                              > 1. To learn.
                              >
                              > 2. To change.
                              >
                              > 3. To do better.

                              add to that "4. to find features". i often find myself reading the source
                              code in order to find undocumented features. and with open-source - there
                              is often rather sparse documentation, and hence a lot of undocumented
                              features.

                              this also includes understanding how some feature works, in order to find
                              why my program does not work well or properly.

                              this could include reading the kernel's source in order to find what
                              exactly some poorly documented ioctl() do, or reading the source of a
                              networking library in order to find why it sometimes aborts the main loop.

                              i got into the habbit of reading source to the leevl that i sometimes do
                              that without first checking if the FAQ contains the question i want an
                              answer for.

                              and who said there is no teaching of "reading source code" in the academi?
                              what is the "intorodction to operating systems" course, of the technion's
                              CS department, if not a course in reading source code, finding bugs and
                              adding features? ofcourse, you got to want to learn that well, in order to
                              learn that well. and you need to have good homework partners to do that
                              "standard questions" for you, in order to have the time to look for the
                              more subtle bugs.

                              --
                              guy

                              "For world domination - press 1,
                              or dial 0, and please hold, for the creator." -- nob o. dy
                            • Shlomi Fish
                              ... OK, I may check it out eventually. Is it available in Technion libraries? ... Or make it even more modular to form an even better design. Software is not
                              Message 14 of 19 , Dec 27, 2003
                              • 0 Attachment
                                On Sat, 27 Dec 2003, Tzahi Fadida wrote:

                                > > -----Original Message-----
                                > > From: Shlomi Fish [mailto:shlomif@...]
                                > > Sent: Saturday, December 27, 2003 1:40 AM
                                > > To: hackers-il@yahoogroups.com
                                > > Subject: RE: [hackers-il] Re: Reading Open Source Code
                                > >
                                > >
                                > > On Fri, 26 Dec 2003, Tzahi Fadida wrote:
                                > >
                                > > > > -----Original Message-----
                                > > > > From: Shlomi Fish [mailto:shlomif@...]
                                > > > > Sent: Friday, December 26, 2003 10:33 PM
                                > > > > To: hackers-il@yahoogroups.com
                                > > > > Subject: Re: [hackers-il] Re: Reading Open Source Code
                                > > > >
                                > > > >
                                > > > > On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:
                                > > > >
                                > > > > > On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:
                                > > > > >
                                > > > > > > Fine, thanks. Thanks for the verbosity of your answer.
                                > > > > >
                                > > > > > Don't mention it. I'm on a touch-typing-diet.
                                > > > > >
                                > > > >
                                > > > > Well, have an easy diet, which will hopefully last as long as it is
                                > > > > needed. (and no longer than that)
                                > > > >
                                > > > > > > Do you mean that people should be trained to read the code that other
                                > > > > > > people wrote (even if it's relatively uninteresting)?
                                > > > > >
                                > > > > > Yes.
                                > > > > >
                                > > > > > > Or that people
                                > > > > > > should be given tasks to read and understand the source code
                                > > of prominent,
                                > > > > > > high-quality, possibly very unique open-source packages? (or something
                                > > > > > > else entirely?)
                                > > > > >
                                > > > > > Yes.
                                > > > > >
                                > > > > > What I meant is that code is written for humans, not machines.
                                > > > >
                                > > > > Maybe you meant that it is written primarily for humans and secondly for
                                > > > > machines? (paraphrasing on Abelson & Sussman from SICP)
                                > > >
                                > > > I think the issue is indeed writting for other people. since we are a long
                                > > > time away in computer years from assembly(which was quite fun at at
                                > > > the time), nowdays you should think of what the next person that will
                                > > > maintain or extend on your code will need.
                                > >
                                > > Correctly.
                                > >
                                > > > I think that this issue is not given its rightful place at computer
                                > > sciences.
                                > >
                                > > Right again. Reading code and starting from a working codebase only to
                                > > extend it is rarely done in programming schools, which is a shame, because
                                > > it's a very needed skill.
                                > >
                                > > > I would refer you to "concepts of programming languages", SEBETA.
                                > >
                                > > What is it? A book? An article? Can you give a link? What is "SEBETA"?
                                >
                                > a book
                                > http://www.amazon.com/exec/obidos/tg/detail/-/0201752956/qid=1072484373/sr=1-1/r
                                > ef=sr_1_1/104-7844860-9557568?v=glance&s=books
                                >

                                OK, I may check it out eventually. Is it available in Technion libraries?

                                > >
                                > > > Aside from that, i think many open source projects would benefit from
                                > > > a good design.
                                > >
                                > > Naturally. But so would a great deal of in-house or even commercial
                                > > software, some of which was heavily designed and written with a lot of
                                > > "red tape". If your project suffers from bad design, you should refactor
                                > > it, so it will have a good one.
                                > >
                                > > Otherwise, (I may have misinterpreted you) it is always a good idea to
                                > > design and plan a change before you start hacking on it. Unless of course
                                > > we are talking about mindless logical transformations, where you just
                                > > change stuff in the code.
                                >
                                > yeah,
                                > before, during and make sure that the maintainers also maintain the
                                > design.
                                >

                                Or make it even more modular to form an even better design. Software is
                                not static and what is an adequate design for a software that does A,B and
                                C, may not necessarily be good for a software that needs to do A,B,C and
                                D.

                                > >
                                > > > the thing is, its not fun for a lot of people to design first
                                > > > based on requirements, using a methdology. so they use XP "kinds" of
                                > > > organic evolution to their software cycles.
                                > >
                                > > I'm not entirely familiar with Extreme Programming (= XP). I only heard
                                > > about it and attended the first half of (a very good) lecture about it.
                                > > What I do know is that in the open source world, the recommended way of
                                > > organizing and designing a software project is the "Bazaar" way as
                                > > described by ESR in "The Cathedral and the Bazaar" (which is an excellent
                                > > read even if you disagree with it). The Bazaar model seems very similar to
                                > > XP, but naturally accepts the constraints of an Internet-wide development
                                > > and that open source software is such that needs to be deployed on a large
                                > > number of variant systems.
                                > >
                                > > I don't think either XP or the Bazaar model tell you not to design.
                                > > Rather, they tell you to do incremental design. I.e: if you want to add a
                                > > feature, then mentally plan how to integrate it into the core code without
                                > > breaking its modularity. If you feel the code is unmodular, refactor it
                                > > according to a planned design. In XP, there is a mandatory one design
                                > > meeting per day.
                                >
                                > XP is against the traditional sens of design. I was attending a seminar
                                > where one of the doctoral works was presented. it was on behavioural
                                > stuff so that part isn't important here, however, they talked about
                                > how they do it in companies and what are the limitations and offers to
                                > make it better etc...
                                > Now, here comes the interesting part - While i liked it in general for
                                > small to medium term projects the inhouse staff of Information Systems
                                > almost kicked them down the stairs. I myself feel that of course, for
                                > big and "important" projects where you need a good design to refer
                                > to where there is a problem this kind of methodology is out of place.

                                Perhaps. Albeit a great deal of large scale open source projects have
                                emerged recently and have been developed in a Bazaar way. Usually, you
                                divide such projects into sub-components and define well-defined
                                interfaces for making use of each component. Then the internal design of
                                each component is left as the "private zone" of its core developers, and
                                the other developers of the project need not be concerned about it.

                                > However, the throwaway with overall design and start programming
                                > and document a bit inside the programm and in the end is what
                                > most software houses do already and it works for the most simple
                                > and non life threatening or most reliable projects.

                                True.

                                > The excuse is of course that the design is supposed to be recycled
                                > several time during the software life time and no one does that because
                                > of budgetery concerns. To the scientific sense this of course stinks but
                                > the commercial world do what they want and what is considered better
                                > for them.

                                Right.

                                > I must stress here though that there are many militairy and proven "safe"
                                > systems that must have a good design. one example would be the black
                                > boxes in airplanes or in the robots they send to mars. This can be done
                                > using mathematical models that can be checked by a computer. Of course
                                > it would be impossible for large scale projects to be built like this.
                                > So, the next best thing is to take something like GAIA for agents,
                                > or UML for small projects, etc... to each appropriate field of software,
                                > and make sure it works.
                                >
                                > THE MAIN PROBLEM i see here is that in CS and any other faculty in the
                                > technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
                                > 1999). this code is like the doctors - not to do harm and negligence.
                                > see a Book:
                                > Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1
                                >

                                I saw the IEEE code of ethics once in writing. Can't remember it all.
                                However, I'm not sure it says anything specific about which
                                software engineering methodologies to use.

                                >
                                > >
                                > > > However, this projects suffer from problems that comes from mistakes just
                                > > > like that. Mistakes, that are apparent when the project needs to grow
                                > > > beyond the 12 people stage.
                                > >
                                > > Luckily, most projects don't need to grow beyond the 12 people stage. Most
                                > > projects have a core team of 1 or 2 developrs, and a halo of a bit more.
                                > > I'm personally involved in Subversion, where the number of contributors is
                                > > very large, and yet it's in pretty good shape. Still, the number of people
                                > > doing most of the work is relatively small.
                                > >
                                > > I think open source has proved that its projects can scale to a large
                                > > number of developers, without grinding to a halt, or its code
                                > > deteriorating into a very low quality mess. The reasons why this is the
                                > > case were discussed in the "Go! Linux" panel, and also in many other
                                > > places and occasions.
                                >
                                > I'll have to check that. I guess good will and "this is my baby and i know
                                > anything about it" is a good part of that.
                                >

                                Some of the points that were raised (by the Go! Linux Panelers and
                                others):

                                1. By splitting the code into components it is possible to divide the work
                                and make sure it scales.
                                2. A great deal of code is not automatically accepted into a project and
                                its correctness, style and quality is criticized by many reviewers before
                                it is accepted.
                                3. In many software shops, programmers are terriorial about their code.
                                Those shops in which programmers encourage others to take a look at their
                                code, look for potential bugs and suggest or implement improvements,
                                eventually see a boast in their productivity. Open Source is like that
                                only on steroids.
                                4. (my own) In many open source projects there is less "red tape" than in
                                those projects that are run by software houses. Thus, a programmer can
                                achieve more as he is not restricted by factors induced by his workplace.
                                5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
                                etc.), actually reduce the output of programmers, and workers of other
                                professions in which there is a genuine mental effort involved in the
                                work. In fact, what can encourage them is letting them work on things
                                they desire to work on. Thus, open source hackers who receive little
                                monetary incentive and work on what they like embody the ideal of this
                                approach.

                                > >
                                > > > I didn't suggest this before because i don't have the time, and the usual
                                > > > response would be to do it yourself but i think that someone should
                                > > > round up a group of people that would design projects based on
                                > > > requirements for teams that do open source projects.
                                > > >
                                > >
                                > > I don't quite understand this last sentence, but I'll try. I think you
                                > > mean that they would design tools to aid in quality control, software
                                > > management, etc. of open source projects. Like SourceForge/GForge, only on
                                > > steroids. Well, that may be a good idea, but of course will involve a lot
                                > > of work.
                                > >
                                > > If you want these tools to be programming language-aware, then you'll need
                                > > to handle all the different languages out there. Good luck to whoever
                                > > choose to go this way.
                                >
                                > Though a really good idea, and i don't think its really possible using
                                > regular methodologies like UML for reasons that would take a course time
                                > to explain. Mainly due to the many models. Orthogonality plays a crucial
                                > roles in this matters.
                                > Anyway the IE IS area has a new methodology called OPM, which
                                > I must state here first that its patented and thus i have personal feelings
                                > about it, though it maybe continues our last discussion of universities
                                > internal funding sources. this methodology is the new wave suppose
                                > to supercede UML using one model for all, and supposed to be able
                                > to one model - > every programming language.
                                > And I for one, believe it is possible.
                                > Ok, comming back to the issue, i was talking about OUTSOURCING
                                > open source style. i.e. people with design skills at different areas
                                > with different methodologies would design projects with the base
                                > requirements and maybe a meeting place between designers and
                                > project developers that will commision designs and refactoring from
                                > them.

                                This could be a problem for one good reason: as ESR identifies in:

                                http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar01s10.html

                                You have to start a project by writing a working version of the code. It
                                may not have to do everything the project needs to do in its final
                                version, but it has to prove that the source code has a future in it. Most
                                projects that start by forming a mailing list and writing a future arch
                                doc or whitepaper wither away into oblivion, without having too much to
                                show for.

                                I can testify from experience that this was the case for my projects.
                                Thus, I always make sure that if I wish to release a project into the
                                wild, I first sit down and write a version that achieves a small subset of
                                the planned featureset.

                                However, your idea may hold for re-designing projects, refactoring them or
                                improving them. I know that Subversion started from a vapourware design
                                that a certain RedHat employee designed, and let others implement. The
                                latter is well-known for this tendency.

                                > The main problem i see that this is supposed to solve, is the experienced
                                > designer issues, and the knowledge time - to - develop issues.
                                >
                                > >
                                > > > anyway, back to the main issue. I think that reading an open source
                                > > > would be much easier if it was well designed using a proven methodology
                                > > > and "enough" documented so that you can at least pinpoint the
                                > > > whereabout of a glitch and fix it in half the time it took you.
                                > > >
                                > >
                                > > "proven methodology" - do such beasts exist? I think you've spent too much
                                > > time being an industrial engineer. Have you ever read "The Mythical
                                > > Man-Month"?
                                >
                                > yes there are, though rare. mathematical models which are "safe" can
                                > be proved, however these are for small scale. Mostly a few lines of code.
                                > No, i was talking about how much time they exists and if they have a
                                > record track of successful usage.
                                >

                                Well, the Bazaar model so far proved itself spectacularily, at least
                                according to the eyes of observers. I don't know about XP, though there
                                are reports it had been successful for certain projects in its
                                target niche. It reminds me of the book "Facts and Fallacies of Software
                                Engineering" which aims to analyze which trends increase productivity and
                                which don't based on research and statistics:

                                http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftware/102-6670515-1781755

                                > >
                                > > Also, define "adequately documented"? Do you want a Literate Programming
                                > > Style book documenting the program? Do you want a detailed architecture
                                > > document? Is a user's guide enough? Do you agree with what XP says that
                                > > even comments should usually be unnecessary and the code should be
                                > > self-explanatory?
                                >
                                > don't count of self-explanatory, this rarely happen when a multitude of
                                > people write different bits of code.

                                Probably true. That's why many software projects have style guidelines and
                                review every contribution before its inclusion.

                                > i can't define here what is adequate precisely because every project
                                > is different. I can only say that it should be adequate in the sense
                                > that further down the line the code could be maintained using this
                                > documentation.
                                > This is the XP way of thinking - "adequate enough".
                                > I am still ambivalent about this so i can't be accurate.
                                >

                                OK. Eventually, by experiencing enough with the code, you can eventually
                                understand even the most obfuscated one. Even a code with tons of
                                comments, meaningful variable names, a consistent style convention, an
                                organized architecture document, API documentation, etc. may not
                                necessarily be easy to understand. And like I said I many times
                                encountered code that had only a small subset of these things, and was
                                very clear and easy to modify.

                                I cannot testify I've been a saint in this regard. Between version 2.4.x
                                and 2.6.x of Freecell Solver, I added a great deal of optimizations
                                some of which I feel may have decreased readability a great deal. But they
                                did made the code 1.5 times faster, which is still significant.

                                > >
                                > > I'm not saying a code should not be documented at all. However,
                                > > documenting a working code, and keeping the documentation in sync with the
                                > > code is very difficult. The best way to understand a code is to ask a
                                > > programmer who worked on it extensively.
                                >
                                > yes. again you are quoting XP, and i'll requote traditional methodologies.
                                >

                                I am not quoting XP as I'm not entirely familiar with it. I'm speaking
                                from experience and intuition. Maybe my opinions may seem XPish to you,
                                but maybe that's what my experience has taught me.

                                > >
                                > > I did encounter some times where code I did not write was very easy to
                                > > understand. One such case was a Perl module I needed for a CGI script I
                                > > maintain. The code proved to be not particularly extensible, but I was
                                > > able to tweak it to do what I needed quite easily. It was heavily
                                > > commented, but I felt that the comments kind of prevented the good flow
                                > > of reading it and were redundant. (so eat your hearts out - Python
                                > > bigots!)
                                >
                                > yes, if the code is self explenatory around the 200 lines or so, then
                                > there is no need to comment every line. java's javadoc is very good
                                > to my thinking.
                                >

                                I've seen some pretty huge codebases that were self-explanatory. I found
                                the GIMP (or at least its gradient editing functionality which is it what
                                I've focused on for the time being) very legible and easy to modify. And
                                that was without reading any description of its internals.

                                So, to conclude: some code will be perfectly legible for experienced
                                programmers even with little documentation. Some code will be very hard to
                                understand even with mountains of documentation. (which many programmers
                                will feel like they don't have time to read). Documentation is not an
                                excuse to write hard-to-understand code, and cannot compensate for its
                                lack of readability. However, it could help sometimes.

                                What goes without saying is that a software needs to have proper user/API
                                documentation, and that its behaviour should correspond to the user model.
                                This is sadly something that a great deal of open source software does not
                                yet implement to an adequate level.

                                Regards,

                                Shlomi Fish

                                ----------------------------------------------------------------------
                                Shlomi Fish shlomif@...
                                Home Page: http://t2.technion.ac.il/~shlomif/

                                You are banished! You are banished! You are banished!

                                Hey? I'm just kidding!
                              • Tzafrir Cohen
                                ... Actually: it means that if this a project with its source code avilable then sneaking in backdoors is more difficult. But then again, I can make trojaned
                                Message 15 of 19 , Dec 27, 2003
                                • 0 Attachment
                                  On Sat, Dec 27, 2003 at 12:22:20PM +0200, Muli Ben-Yehuda wrote:
                                  > On Sat, Dec 27, 2003 at 12:03:36PM +0200, Orna Agmon wrote:
                                  >
                                  > > If it is open-source, it must be safe, it cannot have Trojans or otherwise
                                  > > be malicious (to the one running it), because somewhere out there someone
                                  > > who understands the code other than the original programmer would have
                                  > > noticed and screamed about it to the world. (I do not count mistakes here-
                                  > > errors happen).

                                  Actually: it means that if this a project with its source code avilable
                                  then sneaking in backdoors is more difficult.

                                  But then again, I can make trojaned binary of my program avilable on the
                                  web site without changing the source code.

                                  > >
                                  > > If everybody behaves like this, then this assumption is totally out of
                                  > > order.
                                  >
                                  > It really depends on the project, the number of active developers,
                                  > their attitude towards reviewing each other's source code and the
                                  > ammount of community involvement. Compare and contrast:
                                  >
                                  > - someone tried to inject a backdoor to the Linux kernel. It was
                                  > immediately discovered.

                                  Actually: it was never officially submitted, and in the place it was
                                  added it wouldn't have had much effect. So it was not cought before
                                  the "many eyeballs" of kernel programmers would have had a chance to
                                  spot it.

                                  >
                                  > - there was a database (informix?) that was released as open source a
                                  > few years ago. Only after a couple of years of it being "open" did
                                  > someone discover that the original programmer left a huge backdoor.

                                  The database is firebird, and the back-door was spotted about half a
                                  year after the release of the code.

                                  >
                                  > Therefore, just assuming that if something is open it contains no
                                  > surprises is dangerous.

                                  Fully agree.

                                  --
                                  Tzafrir Cohen +---------------------------+
                                  http://www.technion.ac.il/~tzafrir/ |vim is a mutt's best friend|
                                  mailto:tzafrir@... +---------------------------+
                                • Tzahi Fadida
                                  ... maybe, i myself bought it. maybe you can find something similar. among others its about readability writability and how you can define criterias to judge
                                  Message 16 of 19 , Dec 28, 2003
                                  • 0 Attachment
                                    > -----Original Message-----
                                    > From: Shlomi Fish [mailto:shlomif@...]
                                    > Sent: Saturday, December 27, 2003 8:49 PM
                                    > To: hackers-il@yahoogroups.com
                                    > Subject: RE: [hackers-il] Re: Reading Open Source Code
                                    >
                                    >
                                    > On Sat, 27 Dec 2003, Tzahi Fadida wrote:
                                    >
                                    > > > -----Original Message-----
                                    > > > From: Shlomi Fish [mailto:shlomif@...]
                                    > > > Sent: Saturday, December 27, 2003 1:40 AM
                                    > > > To: hackers-il@yahoogroups.com
                                    > > > Subject: RE: [hackers-il] Re: Reading Open Source Code
                                    > > >
                                    > > >
                                    > > > On Fri, 26 Dec 2003, Tzahi Fadida wrote:
                                    > > >
                                    > > > > > -----Original Message-----
                                    > > > > > From: Shlomi Fish [mailto:shlomif@...]
                                    > > > > > Sent: Friday, December 26, 2003 10:33 PM
                                    > > > > > To: hackers-il@yahoogroups.com
                                    > > > > > Subject: Re: [hackers-il] Re: Reading Open Source Code
                                    > > > > >
                                    > > > > >
                                    > > > > > On Fri, 26 Dec 2003, Muli Ben-Yehuda wrote:
                                    > > > > >
                                    > > > > > > On Fri, Dec 26, 2003 at 09:02:41PM +0200, Shlomi Fish wrote:
                                    > > > > > >
                                    > > > > > > > Fine, thanks. Thanks for the verbosity of your answer.
                                    > > > > > >
                                    > > > > > > Don't mention it. I'm on a touch-typing-diet.
                                    > > > > > >
                                    > > > > >
                                    > > > > > Well, have an easy diet, which will hopefully last as long as it is
                                    > > > > > needed. (and no longer than that)
                                    > > > > >
                                    > > > > > > > Do you mean that people should be trained to read the
                                    > code that other
                                    > > > > > > > people wrote (even if it's relatively uninteresting)?
                                    > > > > > >
                                    > > > > > > Yes.
                                    > > > > > >
                                    > > > > > > > Or that people
                                    > > > > > > > should be given tasks to read and understand the source code
                                    > > > of prominent,
                                    > > > > > > > high-quality, possibly very unique open-source packages?
                                    > (or something
                                    > > > > > > > else entirely?)
                                    > > > > > >
                                    > > > > > > Yes.
                                    > > > > > >
                                    > > > > > > What I meant is that code is written for humans, not machines.
                                    > > > > >
                                    > > > > > Maybe you meant that it is written primarily for humans and
                                    > secondly for
                                    > > > > > machines? (paraphrasing on Abelson & Sussman from SICP)
                                    > > > >
                                    > > > > I think the issue is indeed writting for other people. since we
                                    > are a long
                                    > > > > time away in computer years from assembly(which was quite fun at at
                                    > > > > the time), nowdays you should think of what the next person that will
                                    > > > > maintain or extend on your code will need.
                                    > > >
                                    > > > Correctly.
                                    > > >
                                    > > > > I think that this issue is not given its rightful place at computer
                                    > > > sciences.
                                    > > >
                                    > > > Right again. Reading code and starting from a working codebase only to
                                    > > > extend it is rarely done in programming schools, which is a shame, because
                                    > > > it's a very needed skill.
                                    > > >
                                    > > > > I would refer you to "concepts of programming languages", SEBETA.
                                    > > >
                                    > > > What is it? A book? An article? Can you give a link? What is "SEBETA"?
                                    > >
                                    > > a book
                                    > >
                                    > http://www.amazon.com/exec/obidos/tg/detail/-/0201752956/qid=107248437
                                    > 3/sr=1-1/r
                                    > > ef=sr_1_1/104-7844860-9557568?v=glance&s=books
                                    > >
                                    >
                                    > OK, I may check it out eventually. Is it available in Technion libraries?

                                    maybe, i myself bought it. maybe you can find something similar. among
                                    others its about readability writability and how you can define criterias to
                                    judge programming languages when you come to evaluate if you need
                                    to use it.

                                    >
                                    > > >
                                    > > > > Aside from that, i think many open source projects would benefit from
                                    > > > > a good design.
                                    > > >
                                    > > > Naturally. But so would a great deal of in-house or even commercial
                                    > > > software, some of which was heavily designed and written with a lot of
                                    > > > "red tape". If your project suffers from bad design, you should refactor
                                    > > > it, so it will have a good one.
                                    > > >
                                    > > > Otherwise, (I may have misinterpreted you) it is always a good idea to
                                    > > > design and plan a change before you start hacking on it. Unless of course
                                    > > > we are talking about mindless logical transformations, where you just
                                    > > > change stuff in the code.
                                    > >
                                    > > yeah,
                                    > > before, during and make sure that the maintainers also maintain the
                                    > > design.
                                    > >
                                    >
                                    > Or make it even more modular to form an even better design. Software is
                                    > not static and what is an adequate design for a software that does A,B and
                                    > C, may not necessarily be good for a software that needs to do A,B,C and
                                    > D.

                                    well, scalability is always good. the reason UML, which is the most widely
                                    used, doesn't seem to be loved by many is because they fail to
                                    understand it does not scale. the modularity issue is just an illusion,
                                    and the "package" attempt at one of the models is pathetic.
                                    A good scaling must come with more attention to details.
                                    again, i don't say to use it with open source because of patents and
                                    licensing issue, but see OPCAT at the IE IS department.
                                    I wish we had something more free, or maybe there is and i don't know
                                    about it. if not, please enlighten me.

                                    >
                                    > > >
                                    > > > > the thing is, its not fun for a lot of people to design first
                                    > > > > based on requirements, using a methdology. so they use XP "kinds" of
                                    > > > > organic evolution to their software cycles.
                                    > > >
                                    > > > I'm not entirely familiar with Extreme Programming (= XP). I only heard
                                    > > > about it and attended the first half of (a very good) lecture about it.
                                    > > > What I do know is that in the open source world, the recommended way of
                                    > > > organizing and designing a software project is the "Bazaar" way as
                                    > > > described by ESR in "The Cathedral and the Bazaar" (which is an excellent
                                    > > > read even if you disagree with it). The Bazaar model seems very similar to
                                    > > > XP, but naturally accepts the constraints of an Internet-wide development
                                    > > > and that open source software is such that needs to be deployed on a large
                                    > > > number of variant systems.
                                    > > >
                                    > > > I don't think either XP or the Bazaar model tell you not to design.
                                    > > > Rather, they tell you to do incremental design. I.e: if you want to add a
                                    > > > feature, then mentally plan how to integrate it into the core code without
                                    > > > breaking its modularity. If you feel the code is unmodular, refactor it
                                    > > > according to a planned design. In XP, there is a mandatory one design
                                    > > > meeting per day.
                                    > >
                                    > > XP is against the traditional sens of design. I was attending a seminar
                                    > > where one of the doctoral works was presented. it was on behavioural
                                    > > stuff so that part isn't important here, however, they talked about
                                    > > how they do it in companies and what are the limitations and offers to
                                    > > make it better etc...
                                    > > Now, here comes the interesting part - While i liked it in general for
                                    > > small to medium term projects the inhouse staff of Information Systems
                                    > > almost kicked them down the stairs. I myself feel that of course, for
                                    > > big and "important" projects where you need a good design to refer
                                    > > to where there is a problem this kind of methodology is out of place.
                                    >
                                    > Perhaps. Albeit a great deal of large scale open source projects have
                                    > emerged recently and have been developed in a Bazaar way. Usually, you
                                    > divide such projects into sub-components and define well-defined
                                    > interfaces for making use of each component. Then the internal design of
                                    > each component is left as the "private zone" of its core developers, and
                                    > the other developers of the project need not be concerned about it.

                                    yeah, the blackbox idea. however, this is an ad hoc methods. i.e.
                                    if for example someone decided to use one driver, and then someone
                                    decided to use a different one, and now you have to install 2 drivers.
                                    etc... you need to have an overall look. you can design using for example,
                                    the class model in UML and as you probably know, its not getting to
                                    specifics there, and so 90% is left for the "private zone" to design.

                                    >
                                    > > However, the throwaway with overall design and start programming
                                    > > and document a bit inside the programm and in the end is what
                                    > > most software houses do already and it works for the most simple
                                    > > and non life threatening or most reliable projects.
                                    >
                                    > True.
                                    >
                                    > > The excuse is of course that the design is supposed to be recycled
                                    > > several time during the software life time and no one does that because
                                    > > of budgetery concerns. To the scientific sense this of course stinks but
                                    > > the commercial world do what they want and what is considered better
                                    > > for them.
                                    >
                                    > Right.
                                    >
                                    > > I must stress here though that there are many militairy and proven "safe"
                                    > > systems that must have a good design. one example would be the black
                                    > > boxes in airplanes or in the robots they send to mars. This can be done
                                    > > using mathematical models that can be checked by a computer. Of course
                                    > > it would be impossible for large scale projects to be built like this.
                                    > > So, the next best thing is to take something like GAIA for agents,
                                    > > or UML for small projects, etc... to each appropriate field of software,
                                    > > and make sure it works.
                                    > >
                                    > > THE MAIN PROBLEM i see here is that in CS and any other faculty in the
                                    > > technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
                                    > > 1999). this code is like the doctors - not to do harm and negligence.
                                    > > see a Book:
                                    > > Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1
                                    > >
                                    >
                                    > I saw the IEEE code of ethics once in writing. Can't remember it all.
                                    > However, I'm not sure it says anything specific about which
                                    > software engineering methodologies to use.

                                    of course not, its a code of ethics, but it certainly defines what is
                                    acceptable.
                                    it even starts: "those who contribute by direct participation or by
                                    teaching, to the analysis, specification, design, development,certification,
                                    maintainence...."
                                    http://www.acm.org/serving/se/code.htm

                                    >
                                    > >
                                    > > >
                                    > > > > However, this projects suffer from problems that comes from
                                    > mistakes just
                                    > > > > like that. Mistakes, that are apparent when the project needs to grow
                                    > > > > beyond the 12 people stage.
                                    > > >
                                    > > > Luckily, most projects don't need to grow beyond the 12 people stage. Most
                                    > > > projects have a core team of 1 or 2 developrs, and a halo of a bit more.
                                    > > > I'm personally involved in Subversion, where the number of contributors is
                                    > > > very large, and yet it's in pretty good shape. Still, the number of people
                                    > > > doing most of the work is relatively small.
                                    > > >
                                    > > > I think open source has proved that its projects can scale to a large
                                    > > > number of developers, without grinding to a halt, or its code
                                    > > > deteriorating into a very low quality mess. The reasons why this is the
                                    > > > case were discussed in the "Go! Linux" panel, and also in many other
                                    > > > places and occasions.
                                    > >
                                    > > I'll have to check that. I guess good will and "this is my baby and i know
                                    > > anything about it" is a good part of that.
                                    > >
                                    >
                                    > Some of the points that were raised (by the Go! Linux Panelers and
                                    > others):
                                    >
                                    > 1. By splitting the code into components it is possible to divide the work
                                    > and make sure it scales.
                                    > 2. A great deal of code is not automatically accepted into a project and
                                    > its correctness, style and quality is criticized by many reviewers before
                                    > it is accepted.
                                    > 3. In many software shops, programmers are terriorial about their code.
                                    > Those shops in which programmers encourage others to take a look at their
                                    > code, look for potential bugs and suggest or implement improvements,
                                    > eventually see a boast in their productivity. Open Source is like that
                                    > only on steroids.
                                    > 4. (my own) In many open source projects there is less "red tape" than in
                                    > those projects that are run by software houses. Thus, a programmer can
                                    > achieve more as he is not restricted by factors induced by his workplace.
                                    > 5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
                                    > etc.), actually reduce the output of programmers, and workers of other
                                    > professions in which there is a genuine mental effort involved in the
                                    > work. In fact, what can encourage them is letting them work on things
                                    > they desire to work on. Thus, open source hackers who receive little
                                    > monetary incentive and work on what they like embody the ideal of this
                                    > approach.

                                    the last,5, is a known phenomena in psychology, or at lest that's what
                                    we learned in one of the cognitive courses i took.
                                    I am a bit skeptic about the conclusion, because it does not solve anything
                                    but avoid the phenomena. on the same respect, if you lower a person's
                                    salary or cancel a bonus, you will get the effect of more productivity.
                                    should that mean that we should start dissing the work of open source
                                    developers? :)


                                    >
                                    > > >
                                    > > > > I didn't suggest this before because i don't have the time, and
                                    > the usual
                                    > > > > response would be to do it yourself but i think that someone should
                                    > > > > round up a group of people that would design projects based on
                                    > > > > requirements for teams that do open source projects.
                                    > > > >
                                    > > >
                                    > > > I don't quite understand this last sentence, but I'll try. I think you
                                    > > > mean that they would design tools to aid in quality control, software
                                    > > > management, etc. of open source projects. Like SourceForge/GForge, only on
                                    > > > steroids. Well, that may be a good idea, but of course will involve a lot
                                    > > > of work.
                                    > > >
                                    > > > If you want these tools to be programming language-aware, then you'll need
                                    > > > to handle all the different languages out there. Good luck to whoever
                                    > > > choose to go this way.
                                    > >
                                    > > Though a really good idea, and i don't think its really possible using
                                    > > regular methodologies like UML for reasons that would take a course time
                                    > > to explain. Mainly due to the many models. Orthogonality plays a crucial
                                    > > roles in this matters.
                                    > > Anyway the IE IS area has a new methodology called OPM, which
                                    > > I must state here first that its patented and thus i have personal feelings
                                    > > about it, though it maybe continues our last discussion of universities
                                    > > internal funding sources. this methodology is the new wave suppose
                                    > > to supercede UML using one model for all, and supposed to be able
                                    > > to one model - > every programming language.
                                    > > And I for one, believe it is possible.
                                    > > Ok, comming back to the issue, i was talking about OUTSOURCING
                                    > > open source style. i.e. people with design skills at different areas
                                    > > with different methodologies would design projects with the base
                                    > > requirements and maybe a meeting place between designers and
                                    > > project developers that will commision designs and refactoring from
                                    > > them.
                                    >
                                    > This could be a problem for one good reason: as ESR identifies in:
                                    >
                                    > http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar
                                    > 01s10.html
                                    >
                                    > You have to start a project by writing a working version of the code. It
                                    > may not have to do everything the project needs to do in its final
                                    > version, but it has to prove that the source code has a future in it. Most
                                    > projects that start by forming a mailing list and writing a future arch
                                    > doc or whitepaper wither away into oblivion, without having too much to
                                    > show for.

                                    well, this is a murphy's law kind of thing. usually inexperienced or
                                    experienced people from different reasons don't have the time or
                                    streangth to continue a project and what the quoted reason means is
                                    that the left overs can be continued by someone else, but if there aren't
                                    any then its a problem.
                                    I think its negligable, the same can be argued for someone who leaves
                                    a message about an idea without implementing it because he doesn't have
                                    the time. (like i did)

                                    >
                                    > I can testify from experience that this was the case for my projects.
                                    > Thus, I always make sure that if I wish to release a project into the
                                    > wild, I first sit down and write a version that achieves a small subset of
                                    > the planned featureset.

                                    i know the feeling, but its a psychological thing and not a scientific reason.
                                    i even skipped a few myself, but not before i told them in some mailinglist
                                    so others, if they have the streangth, can implement.
                                    an example, is the idea that not using an open source documents
                                    and software in govrenment facilities is against the freedom of information
                                    law before the committe about this thing convened for the first time. i
                                    corresponded with rabia and others and even got the law copy.
                                    but of course i didn't try to get on the amuta wagon when a few of us
                                    raised the issue of amuta several times in linux-il. i don't have the time, but
                                    it doesn't mean someone else can't do it.

                                    >
                                    > However, your idea may hold for re-designing projects, refactoring them or
                                    > improving them. I know that Subversion started from a vapourware design
                                    > that a certain RedHat employee designed, and let others implement. The
                                    > latter is well-known for this tendency.

                                    well, my idea is about who ever wants a design for whatever reason can
                                    commision one from the community. so yeah.

                                    >
                                    > > The main problem i see that this is supposed to solve, is the experienced
                                    > > designer issues, and the knowledge time - to - develop issues.
                                    > >
                                    > > >
                                    > > > > anyway, back to the main issue. I think that reading an open source
                                    > > > > would be much easier if it was well designed using a proven methodology
                                    > > > > and "enough" documented so that you can at least pinpoint the
                                    > > > > whereabout of a glitch and fix it in half the time it took you.
                                    > > > >
                                    > > >
                                    > > > "proven methodology" - do such beasts exist? I think you've spent too much
                                    > > > time being an industrial engineer. Have you ever read "The Mythical
                                    > > > Man-Month"?
                                    > >
                                    > > yes there are, though rare. mathematical models which are "safe" can
                                    > > be proved, however these are for small scale. Mostly a few lines of code.
                                    > > No, i was talking about how much time they exists and if they have a
                                    > > record track of successful usage.
                                    > >
                                    >
                                    > Well, the Bazaar model so far proved itself spectacularily, at least
                                    > according to the eyes of observers. I don't know about XP, though there
                                    > are reports it had been successful for certain projects in its
                                    > target niche. It reminds me of the book "Facts and Fallacies of Software
                                    > Engineering" which aims to analyze which trends increase productivity and
                                    > which don't based on research and statistics:
                                    >
                                    > http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftwar
                                    > e/102-6670515-1781755

                                    i personally doubt the empirical research of such things because they
                                    tend to be biased to a certain methodology. usually because, for example,
                                    XP was concived by companies (I think IBM or HP, can't remember) for
                                    companies. Companies usually like to cut corners, and so i would refer
                                    those companies to the ACM code of ethics.

                                    >
                                    > > >
                                    > > > Also, define "adequately documented"? Do you want a Literate Programming
                                    > > > Style book documenting the program? Do you want a detailed architecture
                                    > > > document? Is a user's guide enough? Do you agree with what XP says that
                                    > > > even comments should usually be unnecessary and the code should be
                                    > > > self-explanatory?
                                    > >
                                    > > don't count of self-explanatory, this rarely happen when a multitude of
                                    > > people write different bits of code.
                                    >
                                    > Probably true. That's why many software projects have style guidelines and
                                    > review every contribution before its inclusion.

                                    right.

                                    >
                                    > > i can't define here what is adequate precisely because every project
                                    > > is different. I can only say that it should be adequate in the sense
                                    > > that further down the line the code could be maintained using this
                                    > > documentation.
                                    > > This is the XP way of thinking - "adequate enough".
                                    > > I am still ambivalent about this so i can't be accurate.
                                    > >
                                    >
                                    > OK. Eventually, by experiencing enough with the code, you can eventually
                                    > understand even the most obfuscated one. Even a code with tons of
                                    > comments, meaningful variable names, a consistent style convention, an
                                    > organized architecture document, API documentation, etc. may not
                                    > necessarily be easy to understand. And like I said I many times
                                    > encountered code that had only a small subset of these things, and was
                                    > very clear and easy to modify.
                                    >
                                    > I cannot testify I've been a saint in this regard. Between version 2.4.x
                                    > and 2.6.x of Freecell Solver, I added a great deal of optimizations
                                    > some of which I feel may have decreased readability a great deal. But they
                                    > did made the code 1.5 times faster, which is still significant.

                                    right.

                                    >
                                    > > >
                                    > > > I'm not saying a code should not be documented at all. However,
                                    > > > documenting a working code, and keeping the documentation in sync with the
                                    > > > code is very difficult. The best way to understand a code is to ask a
                                    > > > programmer who worked on it extensively.
                                    > >
                                    > > yes. again you are quoting XP, and i'll requote traditional methodologies.
                                    > >
                                    >
                                    > I am not quoting XP as I'm not entirely familiar with it. I'm speaking
                                    > from experience and intuition. Maybe my opinions may seem XPish to you,
                                    > but maybe that's what my experience has taught me.

                                    XP was developed from experience rather than science, and so it seems
                                    to integrate perfectly into one's practicies. however, it should be carefully
                                    checked when used.

                                    >
                                    > > >
                                    > > > I did encounter some times where code I did not write was very easy to
                                    > > > understand. One such case was a Perl module I needed for a CGI script I
                                    > > > maintain. The code proved to be not particularly extensible, but I was
                                    > > > able to tweak it to do what I needed quite easily. It was heavily
                                    > > > commented, but I felt that the comments kind of prevented the good flow
                                    > > > of reading it and were redundant. (so eat your hearts out - Python
                                    > > > bigots!)
                                    > >
                                    > > yes, if the code is self explenatory around the 200 lines or so, then
                                    > > there is no need to comment every line. java's javadoc is very good
                                    > > to my thinking.
                                    > >
                                    >
                                    > I've seen some pretty huge codebases that were self-explanatory. I found
                                    > the GIMP (or at least its gradient editing functionality which is it what
                                    > I've focused on for the time being) very legible and easy to modify. And
                                    > that was without reading any description of its internals.
                                    >
                                    > So, to conclude: some code will be perfectly legible for experienced
                                    > programmers even with little documentation. Some code will be very hard to
                                    > understand even with mountains of documentation. (which many programmers
                                    > will feel like they don't have time to read). Documentation is not an
                                    > excuse to write hard-to-understand code, and cannot compensate for its
                                    > lack of readability. However, it could help sometimes.
                                    >
                                    > What goes without saying is that a software needs to have proper user/API
                                    > documentation, and that its behaviour should correspond to the user model.
                                    > This is sadly something that a great deal of open source software does not
                                    > yet implement to an adequate level.

                                    right.

                                    >
                                    > Regards,
                                    >
                                    > Shlomi Fish
                                    >
                                    > ----------------------------------------------------------------------
                                    > Shlomi Fish shlomif@...
                                    > Home Page: http://t2.technion.ac.il/~shlomif/
                                    >
                                    > You are banished! You are banished! You are banished!
                                    >
                                    > Hey? I'm just kidding!
                                    >
                                    > To unsubscribe from this group, send an email to:
                                    > hackers-il-unsubscribe@egroups.com
                                    >
                                    >
                                    >
                                    > Yahoo! Groups Links
                                    >
                                    > To visit your group on the web, go to:
                                    > http://groups.yahoo.com/group/hackers-il/
                                    >
                                    > To unsubscribe from this group, send an email to:
                                    > hackers-il-unsubscribe@yahoogroups.com
                                    >
                                    > Your use of Yahoo! Groups is subject to:
                                    > http://docs.yahoo.com/info/terms/
                                    >
                                    >
                                    >
                                  • Shlomi Fish
                                    ... I felt UML was too nitty-gritty as well. Too many diagrams to represent the micro-management of the code instead of the really important stuff. Thanks for
                                    Message 17 of 19 , Dec 28, 2003
                                    • 0 Attachment
                                      On Sun, 28 Dec 2003, Tzahi Fadida wrote:

                                      > > > >
                                      > > > > > Aside from that, i think many open source projects would benefit from
                                      > > > > > a good design.
                                      > > > >
                                      > > > > Naturally. But so would a great deal of in-house or even commercial
                                      > > > > software, some of which was heavily designed and written with a lot of
                                      > > > > "red tape". If your project suffers from bad design, you should refactor
                                      > > > > it, so it will have a good one.
                                      > > > >
                                      > > > > Otherwise, (I may have misinterpreted you) it is always a good idea to
                                      > > > > design and plan a change before you start hacking on it. Unless of course
                                      > > > > we are talking about mindless logical transformations, where you just
                                      > > > > change stuff in the code.
                                      > > >
                                      > > > yeah,
                                      > > > before, during and make sure that the maintainers also maintain the
                                      > > > design.
                                      > > >
                                      > >
                                      > > Or make it even more modular to form an even better design. Software is
                                      > > not static and what is an adequate design for a software that does A,B and
                                      > > C, may not necessarily be good for a software that needs to do A,B,C and
                                      > > D.
                                      >
                                      > well, scalability is always good. the reason UML, which is the most widely
                                      > used, doesn't seem to be loved by many is because they fail to
                                      > understand it does not scale. the modularity issue is just an illusion,
                                      > and the "package" attempt at one of the models is pathetic.
                                      > A good scaling must come with more attention to details.
                                      > again, i don't say to use it with open source because of patents and
                                      > licensing issue, but see OPCAT at the IE IS department.
                                      > I wish we had something more free, or maybe there is and i don't know
                                      > about it. if not, please enlighten me.
                                      >

                                      I felt UML was too nitty-gritty as well. Too many diagrams to represent
                                      the micro-management of the code instead of the really important stuff.
                                      Thanks for enlightening me in this regard!

                                      But you can do good design without any official diagrams and with only
                                      a-posteriori diagrams that are given to illustrate various points (and
                                      need not necessary be anything official like UML).

                                      > > > where one of the doctoral works was presented. it was on behavioural
                                      > > > stuff so that part isn't important here, however, they talked about
                                      > > > how they do it in companies and what are the limitations and offers to
                                      > > > make it better etc...
                                      > > > Now, here comes the interesting part - While i liked it in general for
                                      > > > small to medium term projects the inhouse staff of Information Systems
                                      > > > almost kicked them down the stairs. I myself feel that of course, for
                                      > > > big and "important" projects where you need a good design to refer
                                      > > > to where there is a problem this kind of methodology is out of place.
                                      > >
                                      > > Perhaps. Albeit a great deal of large scale open source projects have
                                      > > emerged recently and have been developed in a Bazaar way. Usually, you
                                      > > divide such projects into sub-components and define well-defined
                                      > > interfaces for making use of each component. Then the internal design of
                                      > > each component is left as the "private zone" of its core developers, and
                                      > > the other developers of the project need not be concerned about it.
                                      >
                                      > yeah, the blackbox idea. however, this is an ad hoc methods. i.e.
                                      > if for example someone decided to use one driver, and then someone
                                      > decided to use a different one, and now you have to install 2 drivers.
                                      > etc... you need to have an overall look. you can design using for example,
                                      > the class model in UML and as you probably know, its not getting to
                                      > specifics there, and so 90% is left for the "private zone" to design.
                                      >

                                      Well, this is one problem that is prominent in the open source world any
                                      way you look at it. There's duplication of work anywhere. What's good for
                                      Linux needs to be duplicated in the BSDs. What's good for GNOME needs to
                                      be implemented in KDE as well. We have a gazillion Wikis, window
                                      managers, editors, soundwave editors, bug tracking systems, CMSes, etc.
                                      etc. Of course, this problem is not restricted to the open source world.
                                      For instance Microsoft have released a large number of database packages:

                                      ODBC, RDO, DAO, ADO, OLEDB, ADO.NET

                                      All by the same company under the same roof. So I guess you can't really
                                      avoid it, albeit can do your best to minimize it. Perhaps the only true
                                      way to do so is to break backwards compatiblity if necessary.

                                      > > > systems that must have a good design. one example would be the black
                                      > > > boxes in airplanes or in the robots they send to mars. This can be done
                                      > > > using mathematical models that can be checked by a computer. Of course
                                      > > > it would be impossible for large scale projects to be built like this.
                                      > > > So, the next best thing is to take something like GAIA for agents,
                                      > > > or UML for small projects, etc... to each appropriate field of software,
                                      > > > and make sure it works.
                                      > > >
                                      > > > THE MAIN PROBLEM i see here is that in CS and any other faculty in the
                                      > > > technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
                                      > > > 1999). this code is like the doctors - not to do harm and negligence.
                                      > > > see a Book:
                                      > > > Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1
                                      > > >
                                      > >
                                      > > I saw the IEEE code of ethics once in writing. Can't remember it all.
                                      > > However, I'm not sure it says anything specific about which
                                      > > software engineering methodologies to use.
                                      >
                                      > of course not, its a code of ethics, but it certainly defines what is
                                      > acceptable.
                                      > it even starts: "those who contribute by direct participation or by
                                      > teaching, to the analysis, specification, design, development,certification,
                                      > maintainence...."
                                      > http://www.acm.org/serving/se/code.htm
                                      >

                                      Right. But I'm not sure XP will be frowned upon it in much regard. After
                                      all it has test-driven development, pair programming (and so catching a
                                      lot of bugs, and good peer review), refactoring and other elements that
                                      increase the quality of the code. I'm not sure if it's entirely suitable
                                      for developing software to guide missiles, but it does stress quality.

                                      > >
                                      > > >
                                      > > > >
                                      > > > > > However, this projects suffer from problems that comes from
                                      > > mistakes just
                                      > > > > > like that. Mistakes, that are apparent when the project needs to grow
                                      > > > > > beyond the 12 people stage.
                                      > > > >
                                      > > > > Luckily, most projects don't need to grow beyond the 12 people stage. Most
                                      > > > > projects have a core team of 1 or 2 developrs, and a halo of a bit more.
                                      > > > > I'm personally involved in Subversion, where the number of contributors is
                                      > > > > very large, and yet it's in pretty good shape. Still, the number of people
                                      > > > > doing most of the work is relatively small.
                                      > > > >
                                      > > > > I think open source has proved that its projects can scale to a large
                                      > > > > number of developers, without grinding to a halt, or its code
                                      > > > > deteriorating into a very low quality mess. The reasons why this is the
                                      > > > > case were discussed in the "Go! Linux" panel, and also in many other
                                      > > > > places and occasions.
                                      > > >
                                      > > > I'll have to check that. I guess good will and "this is my baby and i know
                                      > > > anything about it" is a good part of that.
                                      > > >
                                      > >
                                      > > Some of the points that were raised (by the Go! Linux Panelers and
                                      > > others):
                                      > >
                                      > > 1. By splitting the code into components it is possible to divide the work
                                      > > and make sure it scales.
                                      > > 2. A great deal of code is not automatically accepted into a project and
                                      > > its correctness, style and quality is criticized by many reviewers before
                                      > > it is accepted.
                                      > > 3. In many software shops, programmers are terriorial about their code.
                                      > > Those shops in which programmers encourage others to take a look at their
                                      > > code, look for potential bugs and suggest or implement improvements,
                                      > > eventually see a boast in their productivity. Open Source is like that
                                      > > only on steroids.
                                      > > 4. (my own) In many open source projects there is less "red tape" than in
                                      > > those projects that are run by software houses. Thus, a programmer can
                                      > > achieve more as he is not restricted by factors induced by his workplace.
                                      > > 5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
                                      > > etc.), actually reduce the output of programmers, and workers of other
                                      > > professions in which there is a genuine mental effort involved in the
                                      > > work. In fact, what can encourage them is letting them work on things
                                      > > they desire to work on. Thus, open source hackers who receive little
                                      > > monetary incentive and work on what they like embody the ideal of this
                                      > > approach.
                                      >
                                      > the last,5, is a known phenomena in psychology, or at lest that's what
                                      > we learned in one of the cognitive courses i took.
                                      > I am a bit skeptic about the conclusion, because it does not solve anything
                                      > but avoid the phenomena. on the same respect, if you lower a person's
                                      > salary or cancel a bonus, you will get the effect of more productivity.
                                      > should that mean that we should start dissing the work of open source
                                      > developers? :)
                                      >

                                      Maybe you should read "Homesteading the Noosphere" to get more
                                      understanding. Lowering a salary of a programmer who gets paid to do
                                      programming, will likely be a demotivator as well, because he'll feel that
                                      he's no good. An increase in salary, can in fact motivate if presented
                                      correctly, or if done as part of a desireable promotion.

                                      I agree that when ESR is taking this phenomenon to extreme, it may not
                                      necessarily be a correct thing to do scientifically. But it's might as
                                      well have a grain of truth to it.

                                      >
                                      > >
                                      > > > >
                                      > > > > > I didn't suggest this before because i don't have the time, and
                                      > > the usual
                                      > > > > > response would be to do it yourself but i think that someone should
                                      > > > > > round up a group of people that would design projects based on
                                      > > > > > requirements for teams that do open source projects.
                                      > > > > >
                                      > > > >
                                      > > > > I don't quite understand this last sentence, but I'll try. I think you
                                      > > > > mean that they would design tools to aid in quality control, software
                                      > > > > management, etc. of open source projects. Like SourceForge/GForge, only on
                                      > > > > steroids. Well, that may be a good idea, but of course will involve a lot
                                      > > > > of work.
                                      > > > >
                                      > > > > If you want these tools to be programming language-aware, then you'll need
                                      > > > > to handle all the different languages out there. Good luck to whoever
                                      > > > > choose to go this way.
                                      > > >
                                      > > > Though a really good idea, and i don't think its really possible using
                                      > > > regular methodologies like UML for reasons that would take a course time
                                      > > > to explain. Mainly due to the many models. Orthogonality plays a crucial
                                      > > > roles in this matters.
                                      > > > Anyway the IE IS area has a new methodology called OPM, which
                                      > > > I must state here first that its patented and thus i have personal feelings
                                      > > > about it, though it maybe continues our last discussion of universities
                                      > > > internal funding sources. this methodology is the new wave suppose
                                      > > > to supercede UML using one model for all, and supposed to be able
                                      > > > to one model - > every programming language.
                                      > > > And I for one, believe it is possible.
                                      > > > Ok, comming back to the issue, i was talking about OUTSOURCING
                                      > > > open source style. i.e. people with design skills at different areas
                                      > > > with different methodologies would design projects with the base
                                      > > > requirements and maybe a meeting place between designers and
                                      > > > project developers that will commision designs and refactoring from
                                      > > > them.
                                      > >
                                      > > This could be a problem for one good reason: as ESR identifies in:
                                      > >
                                      > > http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar
                                      > > 01s10.html
                                      > >
                                      > > You have to start a project by writing a working version of the code. It
                                      > > may not have to do everything the project needs to do in its final
                                      > > version, but it has to prove that the source code has a future in it. Most
                                      > > projects that start by forming a mailing list and writing a future arch
                                      > > doc or whitepaper wither away into oblivion, without having too much to
                                      > > show for.
                                      >
                                      > well, this is a murphy's law kind of thing. usually inexperienced or
                                      > experienced people from different reasons don't have the time or
                                      > streangth to continue a project and what the quoted reason means is
                                      > that the left overs can be continued by someone else, but if there aren't
                                      > any then its a problem.
                                      > I think its negligable, the same can be argued for someone who leaves
                                      > a message about an idea without implementing it because he doesn't have
                                      > the time. (like i did)
                                      >

                                      Right.

                                      > >
                                      > > I can testify from experience that this was the case for my projects.
                                      > > Thus, I always make sure that if I wish to release a project into the
                                      > > wild, I first sit down and write a version that achieves a small subset of
                                      > > the planned featureset.
                                      >
                                      > i know the feeling, but its a psychological thing and not a scientific reason.
                                      > i even skipped a few myself, but not before i told them in some mailinglist
                                      > so others, if they have the streangth, can implement.

                                      Well, you are right that it is mostly psychological and not scientific.
                                      Some projects that started with a mailing list and a planningware
                                      announcement are flourishing now. Many others which still have a working
                                      codebase are now neglected.

                                      In any case, be aware that as nice ideas are. They don't require a lot of
                                      effort. An individual needs much more effort in developing a good product
                                      out of it, mass producing it and mass distributing it. (or whatever is the
                                      equivalent for software).

                                      > an example, is the idea that not using an open source documents
                                      > and software in govrenment facilities is against the freedom of information
                                      > law before the committe about this thing convened for the first time. i
                                      > corresponded with rabia and others and even got the law copy.
                                      > but of course i didn't try to get on the amuta wagon when a few of us
                                      > raised the issue of amuta several times in linux-il. i don't have the time, but
                                      > it doesn't mean someone else can't do it.
                                      >

                                      Well, if you're not going to do it, either someone else will and get most
                                      of the credit, or it won't get done at all. We all share the same
                                      existence.

                                      > >
                                      > > However, your idea may hold for re-designing projects, refactoring them or
                                      > > improving them. I know that Subversion started from a vapourware design
                                      > > that a certain RedHat employee designed, and let others implement. The
                                      > > latter is well-known for this tendency.
                                      >
                                      > well, my idea is about who ever wants a design for whatever reason can
                                      > commision one from the community. so yeah.
                                      >

                                      Well, I suppose you can post a message to mailing lists or web forums
                                      asking for a good design for a program that does A,B and C. Many people
                                      will be happy to suggest and discuss such good designs.

                                      If you want to set up a site, then a rudimentary one using some web-log
                                      system (Slash, Post-Nuke, Scoop, etc.) should be very easy to do and
                                      you'll only need hosting. It may actually prove to be quite popular.

                                      > >
                                      > > > The main problem i see that this is supposed to solve, is the experienced
                                      > > > designer issues, and the knowledge time - to - develop issues.
                                      > > >
                                      > > > >
                                      > > > > > anyway, back to the main issue. I think that reading an open source
                                      > > > > > would be much easier if it was well designed using a proven methodology
                                      > > > > > and "enough" documented so that you can at least pinpoint the
                                      > > > > > whereabout of a glitch and fix it in half the time it took you.
                                      > > > > >
                                      > > > >
                                      > > > > "proven methodology" - do such beasts exist? I think you've spent too much
                                      > > > > time being an industrial engineer. Have you ever read "The Mythical
                                      > > > > Man-Month"?
                                      > > >
                                      > > > yes there are, though rare. mathematical models which are "safe" can
                                      > > > be proved, however these are for small scale. Mostly a few lines of code.
                                      > > > No, i was talking about how much time they exists and if they have a
                                      > > > record track of successful usage.
                                      > > >
                                      > >
                                      > > Well, the Bazaar model so far proved itself spectacularily, at least
                                      > > according to the eyes of observers. I don't know about XP, though there
                                      > > are reports it had been successful for certain projects in its
                                      > > target niche. It reminds me of the book "Facts and Fallacies of Software
                                      > > Engineering" which aims to analyze which trends increase productivity and
                                      > > which don't based on research and statistics:
                                      > >
                                      > > http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftwar
                                      > > e/102-6670515-1781755
                                      >
                                      > i personally doubt the empirical research of such things because they
                                      > tend to be biased to a certain methodology. usually because, for example,
                                      > XP was concived by companies (I think IBM or HP, can't remember) for
                                      > companies. Companies usually like to cut corners, and so i would refer
                                      > those companies to the ACM code of ethics.
                                      >

                                      Well, this book is supposed to be based on a great deal of researches
                                      conducted. I did not read it, so I cannot testify for how well it is
                                      based.

                                      > >
                                      > > > >
                                      > > > > Also, define "adequately documented"? Do you want a Literate Programming
                                      > > > > Style book documenting the program? Do you want a detailed architecture
                                      > > > > document? Is a user's guide enough? Do you agree with what XP says that
                                      > > > > even comments should usually be unnecessary and the code should be
                                      > > > > self-explanatory?
                                      > > >
                                      > > > don't count of self-explanatory, this rarely happen when a multitude of
                                      > > > people write different bits of code.
                                      > >
                                      > > Probably true. That's why many software projects have style guidelines and
                                      > > review every contribution before its inclusion.
                                      >
                                      > right.
                                      >
                                      > >
                                      > > > i can't define here what is adequate precisely because every project
                                      > > > is different. I can only say that it should be adequate in the sense
                                      > > > that further down the line the code could be maintained using this
                                      > > > documentation.
                                      > > > This is the XP way of thinking - "adequate enough".
                                      > > > I am still ambivalent about this so i can't be accurate.
                                      > > >
                                      > >
                                      > > OK. Eventually, by experiencing enough with the code, you can eventually
                                      > > understand even the most obfuscated one. Even a code with tons of
                                      > > comments, meaningful variable names, a consistent style convention, an
                                      > > organized architecture document, API documentation, etc. may not
                                      > > necessarily be easy to understand. And like I said I many times
                                      > > encountered code that had only a small subset of these things, and was
                                      > > very clear and easy to modify.
                                      > >
                                      > > I cannot testify I've been a saint in this regard. Between version 2.4.x
                                      > > and 2.6.x of Freecell Solver, I added a great deal of optimizations
                                      > > some of which I feel may have decreased readability a great deal. But they
                                      > > did made the code 1.5 times faster, which is still significant.
                                      >
                                      > right.
                                      >
                                      > >
                                      > > > >
                                      > > > > I'm not saying a code should not be documented at all. However,
                                      > > > > documenting a working code, and keeping the documentation in sync with the
                                      > > > > code is very difficult. The best way to understand a code is to ask a
                                      > > > > programmer who worked on it extensively.
                                      > > >
                                      > > > yes. again you are quoting XP, and i'll requote traditional methodologies.
                                      > > >
                                      > >
                                      > > I am not quoting XP as I'm not entirely familiar with it. I'm speaking
                                      > > from experience and intuition. Maybe my opinions may seem XPish to you,
                                      > > but maybe that's what my experience has taught me.
                                      >
                                      > XP was developed from experience rather than science, and so it seems
                                      > to integrate perfectly into one's practicies. however, it should be carefully
                                      > checked when used.
                                      >

                                      "experience rather than science". Isn't science supposed to be
                                      experimental? If someone has a theory, he needs to prove it actually works
                                      in the real world. This is very difficult to do in software engineering
                                      and team management where the success of the team is dependant on a great
                                      deal of many factors. But it is still necessary to show it is.

                                      I don't really care if a theory was deduced from experiments and then
                                      explained theoretically or vice versa. I know that in physics, both thing
                                      s were witnessed. And XP was eventually explained why it is working.
                                      Albeit, with such sciences that involve a lot of human interaction, it is
                                      hard to satisfy all researchers.

                                      Regards,

                                      Shlomi Fish

                                      ----------------------------------------------------------------------
                                      Shlomi Fish shlomif@...
                                      Home Page: http://t2.technion.ac.il/~shlomif/

                                      You are banished! You are banished! You are banished!

                                      Hey? I'm just kidding!
                                    • Tzahi Fadida
                                      ... well, this is often misleading. I am now using methodologies for about 10 years and i can t live without them now. at the start ERD and DFD was all that i
                                      Message 18 of 19 , Dec 29, 2003
                                      • 0 Attachment
                                        > -----Original Message-----
                                        > From: Shlomi Fish [mailto:shlomif@...]
                                        > Sent: Sunday, December 28, 2003 3:54 PM
                                        > To: Hackers-IL
                                        > Subject: RE: [hackers-il] Re: Reading Open Source Code
                                        >
                                        >
                                        > On Sun, 28 Dec 2003, Tzahi Fadida wrote:
                                        >
                                        > > > > >
                                        > > > > > > Aside from that, i think many open source projects would
                                        > benefit from
                                        > > > > > > a good design.
                                        > > > > >
                                        > > > > > Naturally. But so would a great deal of in-house or even commercial
                                        > > > > > software, some of which was heavily designed and written with a lot of
                                        > > > > > "red tape". If your project suffers from bad design, you
                                        > should refactor
                                        > > > > > it, so it will have a good one.
                                        > > > > >
                                        > > > > > Otherwise, (I may have misinterpreted you) it is always a good idea to
                                        > > > > > design and plan a change before you start hacking on it.
                                        > Unless of course
                                        > > > > > we are talking about mindless logical transformations, where you just
                                        > > > > > change stuff in the code.
                                        > > > >
                                        > > > > yeah,
                                        > > > > before, during and make sure that the maintainers also maintain the
                                        > > > > design.
                                        > > > >
                                        > > >
                                        > > > Or make it even more modular to form an even better design. Software is
                                        > > > not static and what is an adequate design for a software that does A,B and
                                        > > > C, may not necessarily be good for a software that needs to do A,B,C and
                                        > > > D.
                                        > >
                                        > > well, scalability is always good. the reason UML, which is the most widely
                                        > > used, doesn't seem to be loved by many is because they fail to
                                        > > understand it does not scale. the modularity issue is just an illusion,
                                        > > and the "package" attempt at one of the models is pathetic.
                                        > > A good scaling must come with more attention to details.
                                        > > again, i don't say to use it with open source because of patents and
                                        > > licensing issue, but see OPCAT at the IE IS department.
                                        > > I wish we had something more free, or maybe there is and i don't know
                                        > > about it. if not, please enlighten me.
                                        > >
                                        >
                                        > I felt UML was too nitty-gritty as well. Too many diagrams to represent
                                        > the micro-management of the code instead of the really important stuff.
                                        > Thanks for enlightening me in this regard!
                                        >
                                        > But you can do good design without any official diagrams and with only
                                        > a-posteriori diagrams that are given to illustrate various points (and
                                        > need not necessary be anything official like UML).

                                        well, this is often misleading. I am now using methodologies for about
                                        10 years and i can't live without them now.
                                        at the start ERD and DFD was all that i needed and all that i knew about.
                                        after that, i discovered UML class diagram, and now i mostly use
                                        the UML business class diagram with the three tier model as stereotypes.
                                        these are very handy nowdays.
                                        Anyway, i since started to work on Agent technology and so i evaluated
                                        several agent design methodologies. GAIA strikes out as the best for
                                        a good role design, however it has many shortcommings. others are
                                        lagging behind but more are comming.

                                        >
                                        > > > > where one of the doctoral works was presented. it was on behavioural
                                        > > > > stuff so that part isn't important here, however, they talked about
                                        > > > > how they do it in companies and what are the limitations and offers to
                                        > > > > make it better etc...
                                        > > > > Now, here comes the interesting part - While i liked it in general for
                                        > > > > small to medium term projects the inhouse staff of Information Systems
                                        > > > > almost kicked them down the stairs. I myself feel that of course, for
                                        > > > > big and "important" projects where you need a good design to refer
                                        > > > > to where there is a problem this kind of methodology is out of place.
                                        > > >
                                        > > > Perhaps. Albeit a great deal of large scale open source projects have
                                        > > > emerged recently and have been developed in a Bazaar way. Usually, you
                                        > > > divide such projects into sub-components and define well-defined
                                        > > > interfaces for making use of each component. Then the internal design of
                                        > > > each component is left as the "private zone" of its core developers, and
                                        > > > the other developers of the project need not be concerned about it.
                                        > >
                                        > > yeah, the blackbox idea. however, this is an ad hoc methods. i.e.
                                        > > if for example someone decided to use one driver, and then someone
                                        > > decided to use a different one, and now you have to install 2 drivers.
                                        > > etc... you need to have an overall look. you can design using for example,
                                        > > the class model in UML and as you probably know, its not getting to
                                        > > specifics there, and so 90% is left for the "private zone" to design.
                                        > >
                                        >
                                        > Well, this is one problem that is prominent in the open source world any
                                        > way you look at it. There's duplication of work anywhere. What's good for
                                        > Linux needs to be duplicated in the BSDs. What's good for GNOME needs to
                                        > be implemented in KDE as well. We have a gazillion Wikis, window
                                        > managers, editors, soundwave editors, bug tracking systems, CMSes, etc.
                                        > etc. Of course, this problem is not restricted to the open source world.
                                        > For instance Microsoft have released a large number of database packages:
                                        >
                                        > ODBC, RDO, DAO, ADO, OLEDB, ADO.NET
                                        >
                                        > All by the same company under the same roof. So I guess you can't really
                                        > avoid it, albeit can do your best to minimize it. Perhaps the only true
                                        > way to do so is to break backwards compatiblity if necessary.

                                        right, but you can at least anticipate this in advance and provide
                                        appropriate api's instead of years of angry mails to developers.

                                        >
                                        > > > > systems that must have a good design. one example would be the black
                                        > > > > boxes in airplanes or in the robots they send to mars. This can be done
                                        > > > > using mathematical models that can be checked by a computer. Of course
                                        > > > > it would be impossible for large scale projects to be built like this.
                                        > > > > So, the next best thing is to take something like GAIA for agents,
                                        > > > > or UML for small projects, etc... to each appropriate field of software,
                                        > > > > and make sure it works.
                                        > > > >
                                        > > > > THE MAIN PROBLEM i see here is that in CS and any other faculty in the
                                        > > > > technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
                                        > > > > 1999). this code is like the doctors - not to do harm and negligence.
                                        > > > > see a Book:
                                        > > > > Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1
                                        > > > >
                                        > > >
                                        > > > I saw the IEEE code of ethics once in writing. Can't remember it all.
                                        > > > However, I'm not sure it says anything specific about which
                                        > > > software engineering methodologies to use.
                                        > >
                                        > > of course not, its a code of ethics, but it certainly defines what is
                                        > > acceptable.
                                        > > it even starts: "those who contribute by direct participation or by
                                        > > teaching, to the analysis, specification, design, development,certification,
                                        > > maintainence...."
                                        > > http://www.acm.org/serving/se/code.htm
                                        > >
                                        >
                                        > Right. But I'm not sure XP will be frowned upon it in much regard. After
                                        > all it has test-driven development, pair programming (and so catching a
                                        > lot of bugs, and good peer review), refactoring and other elements that
                                        > increase the quality of the code. I'm not sure if it's entirely suitable
                                        > for developing software to guide missiles, but it does stress quality.

                                        I am just saying its no good for big projects like SAPing :) the navy force
                                        or a govrenement branch.

                                        >
                                        > > >
                                        > > > >
                                        > > > > >
                                        > > > > > > However, this projects suffer from problems that comes from
                                        > > > mistakes just
                                        > > > > > > like that. Mistakes, that are apparent when the project
                                        > needs to grow
                                        > > > > > > beyond the 12 people stage.
                                        > > > > >
                                        > > > > > Luckily, most projects don't need to grow beyond the 12
                                        > people stage. Most
                                        > > > > > projects have a core team of 1 or 2 developrs, and a halo of
                                        > a bit more.
                                        > > > > > I'm personally involved in Subversion, where the number of
                                        > contributors is
                                        > > > > > very large, and yet it's in pretty good shape. Still, the
                                        > number of people
                                        > > > > > doing most of the work is relatively small.
                                        > > > > >
                                        > > > > > I think open source has proved that its projects can scale to a large
                                        > > > > > number of developers, without grinding to a halt, or its code
                                        > > > > > deteriorating into a very low quality mess. The reasons why
                                        > this is the
                                        > > > > > case were discussed in the "Go! Linux" panel, and also in many other
                                        > > > > > places and occasions.
                                        > > > >
                                        > > > > I'll have to check that. I guess good will and "this is my baby
                                        > and i know
                                        > > > > anything about it" is a good part of that.
                                        > > > >
                                        > > >
                                        > > > Some of the points that were raised (by the Go! Linux Panelers and
                                        > > > others):
                                        > > >
                                        > > > 1. By splitting the code into components it is possible to divide the work
                                        > > > and make sure it scales.
                                        > > > 2. A great deal of code is not automatically accepted into a project and
                                        > > > its correctness, style and quality is criticized by many reviewers before
                                        > > > it is accepted.
                                        > > > 3. In many software shops, programmers are terriorial about their code.
                                        > > > Those shops in which programmers encourage others to take a look at their
                                        > > > code, look for potential bugs and suggest or implement improvements,
                                        > > > eventually see a boast in their productivity. Open Source is like that
                                        > > > only on steroids.
                                        > > > 4. (my own) In many open source projects there is less "red tape" than in
                                        > > > those projects that are run by software houses. Thus, a programmer can
                                        > > > achieve more as he is not restricted by factors induced by his workplace.
                                        > > > 5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
                                        > > > etc.), actually reduce the output of programmers, and workers of other
                                        > > > professions in which there is a genuine mental effort involved in the
                                        > > > work. In fact, what can encourage them is letting them work on things
                                        > > > they desire to work on. Thus, open source hackers who receive little
                                        > > > monetary incentive and work on what they like embody the ideal of this
                                        > > > approach.
                                        > >
                                        > > the last,5, is a known phenomena in psychology, or at lest that's what
                                        > > we learned in one of the cognitive courses i took.
                                        > > I am a bit skeptic about the conclusion, because it does not solve anything
                                        > > but avoid the phenomena. on the same respect, if you lower a person's
                                        > > salary or cancel a bonus, you will get the effect of more productivity.
                                        > > should that mean that we should start dissing the work of open source
                                        > > developers? :)
                                        > >
                                        >
                                        > Maybe you should read "Homesteading the Noosphere" to get more
                                        > understanding. Lowering a salary of a programmer who gets paid to do
                                        > programming, will likely be a demotivator as well, because he'll feel that
                                        > he's no good. An increase in salary, can in fact motivate if presented
                                        > correctly, or if done as part of a desireable promotion.
                                        >
                                        > I agree that when ESR is taking this phenomenon to extreme, it may not
                                        > necessarily be a correct thing to do scientifically. But it's might as
                                        > well have a grain of truth to it.

                                        Any psychology prof. will tell you not to take their studies too seriously as
                                        i can assure you many companies do. i.e. many extend psychological
                                        principals from lab experiments to real life when they should ultimately
                                        conclude that it only worked in the lab, how can they be sure it would
                                        work in other situation in a non controled environment...

                                        >
                                        > >
                                        > > >
                                        > > > > >
                                        > > > > > > I didn't suggest this before because i don't have the time, and
                                        > > > the usual
                                        > > > > > > response would be to do it yourself but i think that someone should
                                        > > > > > > round up a group of people that would design projects based on
                                        > > > > > > requirements for teams that do open source projects.
                                        > > > > > >
                                        > > > > >
                                        > > > > > I don't quite understand this last sentence, but I'll try. I think you
                                        > > > > > mean that they would design tools to aid in quality control, software
                                        > > > > > management, etc. of open source projects. Like
                                        > SourceForge/GForge, only on
                                        > > > > > steroids. Well, that may be a good idea, but of course will
                                        > involve a lot
                                        > > > > > of work.
                                        > > > > >
                                        > > > > > If you want these tools to be programming language-aware,
                                        > then you'll need
                                        > > > > > to handle all the different languages out there. Good luck to whoever
                                        > > > > > choose to go this way.
                                        > > > >
                                        > > > > Though a really good idea, and i don't think its really possible using
                                        > > > > regular methodologies like UML for reasons that would take a course time
                                        > > > > to explain. Mainly due to the many models. Orthogonality plays a crucial
                                        > > > > roles in this matters.
                                        > > > > Anyway the IE IS area has a new methodology called OPM, which
                                        > > > > I must state here first that its patented and thus i have
                                        > personal feelings
                                        > > > > about it, though it maybe continues our last discussion of universities
                                        > > > > internal funding sources. this methodology is the new wave suppose
                                        > > > > to supercede UML using one model for all, and supposed to be able
                                        > > > > to one model - > every programming language.
                                        > > > > And I for one, believe it is possible.
                                        > > > > Ok, comming back to the issue, i was talking about OUTSOURCING
                                        > > > > open source style. i.e. people with design skills at different areas
                                        > > > > with different methodologies would design projects with the base
                                        > > > > requirements and maybe a meeting place between designers and
                                        > > > > project developers that will commision designs and refactoring from
                                        > > > > them.
                                        > > >
                                        > > > This could be a problem for one good reason: as ESR identifies in:
                                        > > >
                                        > > > http://www.catb.org/~esr/writings/cathedral-bazaar/cathedral-bazaar/ar
                                        > > > 01s10.html
                                        > > >
                                        > > > You have to start a project by writing a working version of the code. It
                                        > > > may not have to do everything the project needs to do in its final
                                        > > > version, but it has to prove that the source code has a future in it. Most
                                        > > > projects that start by forming a mailing list and writing a future arch
                                        > > > doc or whitepaper wither away into oblivion, without having too much to
                                        > > > show for.
                                        > >
                                        > > well, this is a murphy's law kind of thing. usually inexperienced or
                                        > > experienced people from different reasons don't have the time or
                                        > > streangth to continue a project and what the quoted reason means is
                                        > > that the left overs can be continued by someone else, but if there aren't
                                        > > any then its a problem.
                                        > > I think its negligable, the same can be argued for someone who leaves
                                        > > a message about an idea without implementing it because he doesn't have
                                        > > the time. (like i did)
                                        > >
                                        >
                                        > Right.
                                        >
                                        > > >
                                        > > > I can testify from experience that this was the case for my projects.
                                        > > > Thus, I always make sure that if I wish to release a project into the
                                        > > > wild, I first sit down and write a version that achieves a small subset of
                                        > > > the planned featureset.
                                        > >
                                        > > i know the feeling, but its a psychological thing and not a
                                        > scientific reason.
                                        > > i even skipped a few myself, but not before i told them in some mailinglist
                                        > > so others, if they have the streangth, can implement.
                                        >
                                        > Well, you are right that it is mostly psychological and not scientific.
                                        > Some projects that started with a mailing list and a planningware
                                        > announcement are flourishing now. Many others which still have a working
                                        > codebase are now neglected.
                                        >
                                        > In any case, be aware that as nice ideas are. They don't require a lot of
                                        > effort. An individual needs much more effort in developing a good product
                                        > out of it, mass producing it and mass distributing it. (or whatever is the
                                        > equivalent for software).

                                        right

                                        >
                                        > > an example, is the idea that not using an open source documents
                                        > > and software in govrenment facilities is against the freedom of information
                                        > > law before the committe about this thing convened for the first time. i
                                        > > corresponded with rabia and others and even got the law copy.
                                        > > but of course i didn't try to get on the amuta wagon when a few of us
                                        > > raised the issue of amuta several times in linux-il. i don't have
                                        > the time, but
                                        > > it doesn't mean someone else can't do it.
                                        > >
                                        >
                                        > Well, if you're not going to do it, either someone else will and get most
                                        > of the credit, or it won't get done at all. We all share the same
                                        > existence.

                                        I have a rule, if i am not going to do it, i don't care someone else will
                                        take the credit. again, the psychological thing.

                                        >
                                        > > >
                                        > > > However, your idea may hold for re-designing projects, refactoring them or
                                        > > > improving them. I know that Subversion started from a vapourware design
                                        > > > that a certain RedHat employee designed, and let others implement. The
                                        > > > latter is well-known for this tendency.
                                        > >
                                        > > well, my idea is about who ever wants a design for whatever reason can
                                        > > commision one from the community. so yeah.
                                        > >
                                        >
                                        > Well, I suppose you can post a message to mailing lists or web forums
                                        > asking for a good design for a program that does A,B and C. Many people
                                        > will be happy to suggest and discuss such good designs.
                                        >
                                        > If you want to set up a site, then a rudimentary one using some web-log
                                        > system (Slash, Post-Nuke, Scoop, etc.) should be very easy to do and
                                        > you'll only need hosting. It may actually prove to be quite popular.

                                        yes, it may be so.
                                        I just have to find the time. maybe we could integrate this into haifux.
                                        i.e. close in person development support for open source projects
                                        or even wannabe, etc..

                                        >
                                        > > >
                                        > > > > The main problem i see that this is supposed to solve, is the
                                        > experienced
                                        > > > > designer issues, and the knowledge time - to - develop issues.
                                        > > > >
                                        > > > > >
                                        > > > > > > anyway, back to the main issue. I think that reading an open source
                                        > > > > > > would be much easier if it was well designed using a proven
                                        > methodology
                                        > > > > > > and "enough" documented so that you can at least pinpoint the
                                        > > > > > > whereabout of a glitch and fix it in half the time it took you.
                                        > > > > > >
                                        > > > > >
                                        > > > > > "proven methodology" - do such beasts exist? I think you've
                                        > spent too much
                                        > > > > > time being an industrial engineer. Have you ever read "The Mythical
                                        > > > > > Man-Month"?
                                        > > > >
                                        > > > > yes there are, though rare. mathematical models which are "safe" can
                                        > > > > be proved, however these are for small scale. Mostly a few
                                        > lines of code.
                                        > > > > No, i was talking about how much time they exists and if they have a
                                        > > > > record track of successful usage.
                                        > > > >
                                        > > >
                                        > > > Well, the Bazaar model so far proved itself spectacularily, at least
                                        > > > according to the eyes of observers. I don't know about XP, though there
                                        > > > are reports it had been successful for certain projects in its
                                        > > > target niche. It reminds me of the book "Facts and Fallacies of Software
                                        > > > Engineering" which aims to analyze which trends increase productivity and
                                        > > > which don't based on research and statistics:
                                        > > >
                                        > > > http://www.amazon.com/exec/obidos/tg/detail/-/0321117425/joelonsoftwar
                                        > > > e/102-6670515-1781755
                                        > >
                                        > > i personally doubt the empirical research of such things because they
                                        > > tend to be biased to a certain methodology. usually because, for example,
                                        > > XP was concived by companies (I think IBM or HP, can't remember) for
                                        > > companies. Companies usually like to cut corners, and so i would refer
                                        > > those companies to the ACM code of ethics.
                                        > >
                                        >
                                        > Well, this book is supposed to be based on a great deal of researches
                                        > conducted. I did not read it, so I cannot testify for how well it is
                                        > based.
                                        >
                                        > > >
                                        > > > > >
                                        > > > > > Also, define "adequately documented"? Do you want a Literate
                                        > Programming
                                        > > > > > Style book documenting the program? Do you want a detailed
                                        > architecture
                                        > > > > > document? Is a user's guide enough? Do you agree with what XP
                                        > says that
                                        > > > > > even comments should usually be unnecessary and the code should be
                                        > > > > > self-explanatory?
                                        > > > >
                                        > > > > don't count of self-explanatory, this rarely happen when a multitude of
                                        > > > > people write different bits of code.
                                        > > >
                                        > > > Probably true. That's why many software projects have style guidelines and
                                        > > > review every contribution before its inclusion.
                                        > >
                                        > > right.
                                        > >
                                        > > >
                                        > > > > i can't define here what is adequate precisely because every project
                                        > > > > is different. I can only say that it should be adequate in the sense
                                        > > > > that further down the line the code could be maintained using this
                                        > > > > documentation.
                                        > > > > This is the XP way of thinking - "adequate enough".
                                        > > > > I am still ambivalent about this so i can't be accurate.
                                        > > > >
                                        > > >
                                        > > > OK. Eventually, by experiencing enough with the code, you can eventually
                                        > > > understand even the most obfuscated one. Even a code with tons of
                                        > > > comments, meaningful variable names, a consistent style convention, an
                                        > > > organized architecture document, API documentation, etc. may not
                                        > > > necessarily be easy to understand. And like I said I many times
                                        > > > encountered code that had only a small subset of these things, and was
                                        > > > very clear and easy to modify.
                                        > > >
                                        > > > I cannot testify I've been a saint in this regard. Between version 2.4.x
                                        > > > and 2.6.x of Freecell Solver, I added a great deal of optimizations
                                        > > > some of which I feel may have decreased readability a great deal. But they
                                        > > > did made the code 1.5 times faster, which is still significant.
                                        > >
                                        > > right.
                                        > >
                                        > > >
                                        > > > > >
                                        > > > > > I'm not saying a code should not be documented at all. However,
                                        > > > > > documenting a working code, and keeping the documentation in
                                        > sync with the
                                        > > > > > code is very difficult. The best way to understand a code is to ask a
                                        > > > > > programmer who worked on it extensively.
                                        > > > >
                                        > > > > yes. again you are quoting XP, and i'll requote traditional
                                        > methodologies.
                                        > > > >
                                        > > >
                                        > > > I am not quoting XP as I'm not entirely familiar with it. I'm speaking
                                        > > > from experience and intuition. Maybe my opinions may seem XPish to you,
                                        > > > but maybe that's what my experience has taught me.
                                        > >
                                        > > XP was developed from experience rather than science, and so it seems
                                        > > to integrate perfectly into one's practicies. however, it should be
                                        > carefully
                                        > > checked when used.
                                        > >
                                        >
                                        > "experience rather than science". Isn't science supposed to be
                                        > experimental? If someone has a theory, he needs to prove it actually works
                                        > in the real world. This is very difficult to do in software engineering
                                        > and team management where the success of the team is dependant on a great
                                        > deal of many factors. But it is still necessary to show it is.
                                        >
                                        > I don't really care if a theory was deduced from experiments and then
                                        > explained theoretically or vice versa. I know that in physics, both thing
                                        > s were witnessed. And XP was eventually explained why it is working.
                                        > Albeit, with such sciences that involve a lot of human interaction, it is
                                        > hard to satisfy all researchers.

                                        however, many researchers, even prominent ones, sometimes after
                                        experimenting conclude they found a life principle and immideately
                                        goes on to write an article about it. next thing you know, a company
                                        was adviced to use it. next thing you know **get ready for exageration**
                                        pal kal is falling. but you know what i mean.
                                        its like if you have a million black horses in a continent and thus conclude
                                        that using induction that the base 0..1000 are all black horses, and goes
                                        on to prove that all horses are black. meanwhile in another continent
                                        all the horses are white.

                                        >
                                        > Regards,
                                        >
                                        > Shlomi Fish
                                        >
                                        > ----------------------------------------------------------------------
                                        > Shlomi Fish shlomif@...
                                        > Home Page: http://t2.technion.ac.il/~shlomif/
                                        >
                                        > You are banished! You are banished! You are banished!
                                        >
                                        > Hey? I'm just kidding!
                                        >
                                        > To unsubscribe from this group, send an email to:
                                        > hackers-il-unsubscribe@egroups.com
                                        >
                                        >
                                        >
                                        >
                                        > Yahoo! Groups Links
                                        >
                                        > To visit your group on the web, go to:
                                        > http://groups.yahoo.com/group/hackers-il/
                                        >
                                        > To unsubscribe from this group, send an email to:
                                        > hackers-il-unsubscribe@yahoogroups.com
                                        >
                                        > Your use of Yahoo! Groups is subject to:
                                        > http://docs.yahoo.com/info/terms/
                                        >
                                        >
                                        >
                                      • Shlomi Fish
                                        Hi Tzahi! For next time, please prune sub-threads that you do not wish to comment on. Otherwise these messages become very long. ... Well, to each his own. If
                                        Message 19 of 19 , Jan 4, 2004
                                        • 0 Attachment
                                          Hi Tzahi!

                                          For next time, please prune sub-threads that you do not wish to comment
                                          on. Otherwise these messages become very long.

                                          On Mon, 29 Dec 2003, Tzahi Fadida wrote:

                                          > > -----Original Message-----
                                          > > From: Shlomi Fish [mailto:shlomif@...]
                                          > > Sent: Sunday, December 28, 2003 3:54 PM
                                          > > To: Hackers-IL
                                          > > Subject: RE: [hackers-il] Re: Reading Open Source Code
                                          > >
                                          > >
                                          > > On Sun, 28 Dec 2003, Tzahi Fadida wrote:
                                          > >
                                          > > > > > >
                                          > > > > > > > Aside from that, i think many open source projects would
                                          > > benefit from
                                          > > > > > > > a good design.
                                          > > > > > >
                                          > > > > > > Naturally. But so would a great deal of in-house or even commercial
                                          > > > > > > software, some of which was heavily designed and written with a lot of
                                          > > > > > > "red tape". If your project suffers from bad design, you
                                          > > should refactor
                                          > > > > > > it, so it will have a good one.
                                          > > > > > >
                                          > > > > > > Otherwise, (I may have misinterpreted you) it is always a good idea to
                                          > > > > > > design and plan a change before you start hacking on it.
                                          > > Unless of course
                                          > > > > > > we are talking about mindless logical transformations, where you just
                                          > > > > > > change stuff in the code.
                                          > > > > >
                                          > > > > > yeah,
                                          > > > > > before, during and make sure that the maintainers also maintain the
                                          > > > > > design.
                                          > > > > >
                                          > > > >
                                          > > > > Or make it even more modular to form an even better design. Software is
                                          > > > > not static and what is an adequate design for a software that does A,B and
                                          > > > > C, may not necessarily be good for a software that needs to do A,B,C and
                                          > > > > D.
                                          > > >
                                          > > > well, scalability is always good. the reason UML, which is the most widely
                                          > > > used, doesn't seem to be loved by many is because they fail to
                                          > > > understand it does not scale. the modularity issue is just an illusion,
                                          > > > and the "package" attempt at one of the models is pathetic.
                                          > > > A good scaling must come with more attention to details.
                                          > > > again, i don't say to use it with open source because of patents and
                                          > > > licensing issue, but see OPCAT at the IE IS department.
                                          > > > I wish we had something more free, or maybe there is and i don't know
                                          > > > about it. if not, please enlighten me.
                                          > > >
                                          > >
                                          > > I felt UML was too nitty-gritty as well. Too many diagrams to represent
                                          > > the micro-management of the code instead of the really important stuff.
                                          > > Thanks for enlightening me in this regard!
                                          > >
                                          > > But you can do good design without any official diagrams and with only
                                          > > a-posteriori diagrams that are given to illustrate various points (and
                                          > > need not necessary be anything official like UML).
                                          >
                                          > well, this is often misleading. I am now using methodologies for about
                                          > 10 years and i can't live without them now.
                                          > at the start ERD and DFD was all that i needed and all that i knew about.
                                          > after that, i discovered UML class diagram, and now i mostly use
                                          > the UML business class diagram with the three tier model as stereotypes.
                                          > these are very handy nowdays.
                                          > Anyway, i since started to work on Agent technology and so i evaluated
                                          > several agent design methodologies. GAIA strikes out as the best for
                                          > a good role design, however it has many shortcommings. others are
                                          > lagging behind but more are comming.
                                          >

                                          Well, to each his own. If I can find a good tool for creating and
                                          manipulating diagrams, I may give it a try.

                                          > >
                                          > > > > > systems that must have a good design. one example would be the black
                                          > > > > > boxes in airplanes or in the robots they send to mars. This can be done
                                          > > > > > using mathematical models that can be checked by a computer. Of course
                                          > > > > > it would be impossible for large scale projects to be built like this.
                                          > > > > > So, the next best thing is to take something like GAIA for agents,
                                          > > > > > or UML for small projects, etc... to each appropriate field of software,
                                          > > > > > and make sure it works.
                                          > > > > >
                                          > > > > > THE MAIN PROBLEM i see here is that in CS and any other faculty in the
                                          > > > > > technion, i never heard of the IEEE/ACM Code of Ethics(c IEEE/ACM
                                          > > > > > 1999). this code is like the doctors - not to do harm and negligence.
                                          > > > > > see a Book:
                                          > > > > > Software Engineering, IAM SOMMERVILLE 6th edition,chapter 1
                                          > > > > >
                                          > > > >
                                          > > > > I saw the IEEE code of ethics once in writing. Can't remember it all.
                                          > > > > However, I'm not sure it says anything specific about which
                                          > > > > software engineering methodologies to use.
                                          > > >
                                          > > > of course not, its a code of ethics, but it certainly defines what is
                                          > > > acceptable.
                                          > > > it even starts: "those who contribute by direct participation or by
                                          > > > teaching, to the analysis, specification, design, development,certification,
                                          > > > maintainence...."
                                          > > > http://www.acm.org/serving/se/code.htm
                                          > > >
                                          > >
                                          > > Right. But I'm not sure XP will be frowned upon it in much regard. After
                                          > > all it has test-driven development, pair programming (and so catching a
                                          > > lot of bugs, and good peer review), refactoring and other elements that
                                          > > increase the quality of the code. I'm not sure if it's entirely suitable
                                          > > for developing software to guide missiles, but it does stress quality.
                                          >
                                          > I am just saying its no good for big projects like SAPing :) the navy force
                                          > or a govrenement branch.
                                          >

                                          Right. I'm not sure it was ever tried in these contexts, or that its
                                          authors claim it is suitable for that.

                                          > >
                                          > > > >
                                          > > > > >
                                          > > > > > >
                                          > > > > > > > However, this projects suffer from problems that comes from
                                          > > > > mistakes just
                                          > > > > > > > like that. Mistakes, that are apparent when the project
                                          > > needs to grow
                                          > > > > > > > beyond the 12 people stage.
                                          > > > > > >
                                          > > > > > > Luckily, most projects don't need to grow beyond the 12
                                          > > people stage. Most
                                          > > > > > > projects have a core team of 1 or 2 developrs, and a halo of
                                          > > a bit more.
                                          > > > > > > I'm personally involved in Subversion, where the number of
                                          > > contributors is
                                          > > > > > > very large, and yet it's in pretty good shape. Still, the
                                          > > number of people
                                          > > > > > > doing most of the work is relatively small.
                                          > > > > > >
                                          > > > > > > I think open source has proved that its projects can scale to a large
                                          > > > > > > number of developers, without grinding to a halt, or its code
                                          > > > > > > deteriorating into a very low quality mess. The reasons why
                                          > > this is the
                                          > > > > > > case were discussed in the "Go! Linux" panel, and also in many other
                                          > > > > > > places and occasions.
                                          > > > > >
                                          > > > > > I'll have to check that. I guess good will and "this is my baby
                                          > > and i know
                                          > > > > > anything about it" is a good part of that.
                                          > > > > >
                                          > > > >
                                          > > > > Some of the points that were raised (by the Go! Linux Panelers and
                                          > > > > others):
                                          > > > >
                                          > > > > 1. By splitting the code into components it is possible to divide the work
                                          > > > > and make sure it scales.
                                          > > > > 2. A great deal of code is not automatically accepted into a project and
                                          > > > > its correctness, style and quality is criticized by many reviewers before
                                          > > > > it is accepted.
                                          > > > > 3. In many software shops, programmers are terriorial about their code.
                                          > > > > Those shops in which programmers encourage others to take a look at their
                                          > > > > code, look for potential bugs and suggest or implement improvements,
                                          > > > > eventually see a boast in their productivity. Open Source is like that
                                          > > > > only on steroids.
                                          > > > > 4. (my own) In many open source projects there is less "red tape" than in
                                          > > > > those projects that are run by software houses. Thus, a programmer can
                                          > > > > achieve more as he is not restricted by factors induced by his workplace.
                                          > > > > 5. (Homesteading the Noosphere). It was shown that incentive pay (bonuses,
                                          > > > > etc.), actually reduce the output of programmers, and workers of other
                                          > > > > professions in which there is a genuine mental effort involved in the
                                          > > > > work. In fact, what can encourage them is letting them work on things
                                          > > > > they desire to work on. Thus, open source hackers who receive little
                                          > > > > monetary incentive and work on what they like embody the ideal of this
                                          > > > > approach.
                                          > > >
                                          > > > the last,5, is a known phenomena in psychology, or at lest that's what
                                          > > > we learned in one of the cognitive courses i took.
                                          > > > I am a bit skeptic about the conclusion, because it does not solve anything
                                          > > > but avoid the phenomena. on the same respect, if you lower a person's
                                          > > > salary or cancel a bonus, you will get the effect of more productivity.
                                          > > > should that mean that we should start dissing the work of open source
                                          > > > developers? :)
                                          > > >
                                          > >
                                          > > Maybe you should read "Homesteading the Noosphere" to get more
                                          > > understanding. Lowering a salary of a programmer who gets paid to do
                                          > > programming, will likely be a demotivator as well, because he'll feel that
                                          > > he's no good. An increase in salary, can in fact motivate if presented
                                          > > correctly, or if done as part of a desireable promotion.
                                          > >
                                          > > I agree that when ESR is taking this phenomenon to extreme, it may not
                                          > > necessarily be a correct thing to do scientifically. But it's might as
                                          > > well have a grain of truth to it.
                                          >
                                          > Any psychology prof. will tell you not to take their studies too seriously as
                                          > i can assure you many companies do. i.e. many extend psychological
                                          > principals from lab experiments to real life when they should ultimately
                                          > conclude that it only worked in the lab, how can they be sure it would
                                          > work in other situation in a non controled environment...
                                          >

                                          Right.

                                          > > > an example, is the idea that not using an open source documents
                                          > > > and software in govrenment facilities is against the freedom of information
                                          > > > law before the committe about this thing convened for the first time. i
                                          > > > corresponded with rabia and others and even got the law copy.
                                          > > > but of course i didn't try to get on the amuta wagon when a few of us
                                          > > > raised the issue of amuta several times in linux-il. i don't have
                                          > > the time, but
                                          > > > it doesn't mean someone else can't do it.
                                          > > >
                                          > >
                                          > > Well, if you're not going to do it, either someone else will and get most
                                          > > of the credit, or it won't get done at all. We all share the same
                                          > > existence.
                                          >
                                          > I have a rule, if i am not going to do it, i don't care someone else will
                                          > take the credit. again, the psychological thing.
                                          >

                                          So do I. However, I was trying to point at the fact that it is ultimately
                                          you who is responsible for taking an idea forward. (yours or someone
                                          else's)

                                          > >
                                          > > > >
                                          > > > > However, your idea may hold for re-designing projects, refactoring them or
                                          > > > > improving them. I know that Subversion started from a vapourware design
                                          > > > > that a certain RedHat employee designed, and let others implement. The
                                          > > > > latter is well-known for this tendency.
                                          > > >
                                          > > > well, my idea is about who ever wants a design for whatever reason can
                                          > > > commision one from the community. so yeah.
                                          > > >
                                          > >
                                          > > Well, I suppose you can post a message to mailing lists or web forums
                                          > > asking for a good design for a program that does A,B and C. Many people
                                          > > will be happy to suggest and discuss such good designs.
                                          > >
                                          > > If you want to set up a site, then a rudimentary one using some web-log
                                          > > system (Slash, Post-Nuke, Scoop, etc.) should be very easy to do and
                                          > > you'll only need hosting. It may actually prove to be quite popular.
                                          >
                                          > yes, it may be so.
                                          > I just have to find the time. maybe we could integrate this into haifux.
                                          > i.e. close in person development support for open source projects
                                          > or even wannabe, etc..
                                          >

                                          Again, it's up to you. Note that Scoop (the kuro5hin back-end) boasts the
                                          fact that it needs minimal maintainance. What it does is let users vote
                                          for what will appear and get submitted.

                                          > > > >
                                          > > > > > >
                                          > > > > > > I'm not saying a code should not be documented at all. However,
                                          > > > > > > documenting a working code, and keeping the documentation in
                                          > > sync with the
                                          > > > > > > code is very difficult. The best way to understand a code is to ask a
                                          > > > > > > programmer who worked on it extensively.
                                          > > > > >
                                          > > > > > yes. again you are quoting XP, and i'll requote traditional
                                          > > methodologies.
                                          > > > > >
                                          > > > >
                                          > > > > I am not quoting XP as I'm not entirely familiar with it. I'm speaking
                                          > > > > from experience and intuition. Maybe my opinions may seem XPish to you,
                                          > > > > but maybe that's what my experience has taught me.
                                          > > >
                                          > > > XP was developed from experience rather than science, and so it seems
                                          > > > to integrate perfectly into one's practicies. however, it should be
                                          > > carefully
                                          > > > checked when used.
                                          > > >
                                          > >
                                          > > "experience rather than science". Isn't science supposed to be
                                          > > experimental? If someone has a theory, he needs to prove it actually works
                                          > > in the real world. This is very difficult to do in software engineering
                                          > > and team management where the success of the team is dependant on a great
                                          > > deal of many factors. But it is still necessary to show it is.
                                          > >
                                          > > I don't really care if a theory was deduced from experiments and then
                                          > > explained theoretically or vice versa. I know that in physics, both thing
                                          > > s were witnessed. And XP was eventually explained why it is working.
                                          > > Albeit, with such sciences that involve a lot of human interaction, it is
                                          > > hard to satisfy all researchers.
                                          >
                                          > however, many researchers, even prominent ones, sometimes after
                                          > experimenting conclude they found a life principle and immideately
                                          > goes on to write an article about it. next thing you know, a company
                                          > was adviced to use it. next thing you know **get ready for exageration**
                                          > pal kal is falling. but you know what i mean.
                                          > its like if you have a million black horses in a continent and thus conclude
                                          > that using induction that the base 0..1000 are all black horses, and goes
                                          > on to prove that all horses are black. meanwhile in another continent
                                          > all the horses are white.
                                          >

                                          True.

                                          Regards,

                                          Shlomi Fish

                                          ----------------------------------------------------------------------
                                          Shlomi Fish shlomif@...
                                          Home Page: http://t2.technion.ac.il/~shlomif/

                                          You are banished! You are banished! You are banished!

                                          Hey? I'm just kidding!
                                        Your message has been successfully submitted and would be delivered to recipients shortly.