Loading ...
Sorry, an error occurred while loading the content.

Re: [ttlug] Linux training

Expand Messages
  • Richard Bailey
    You re right Khyron, this does seem to be above the level that I m looking at. I ll be at the Meet n Greet tomorrow in J Malone s. Hope to see some of you
    Message 1 of 20 , Jan 27, 2010
    • 0 Attachment
      You're right Khyron, this does seem to be above the level that I'm looking
      at.

      I'll be at the Meet 'n Greet tomorrow in J Malone's. Hope to see some of you
      guys there so we can talk linux training in Trinidad.

      Thanks for the contacts so far. I'll hit you guys up when I get an
      opportunity.

      On Wed, Jan 27, 2010 at 4:13 PM, Khyron <khyron4eva@...> wrote:

      > To be clear, all of this Linux training from the LF appears to be focused
      > on
      >
      > the kernel -- which Linux is, a kernel -- with emphasis on the development
      > and kernel internals. Good stuff, indeed. (Hard to learn it on your own
      > on
      >
      > a timeline, speaking from experience.) Yet, with the exception of the
      > "Linux
      > Administration 101" webinar, the material is targeting people who have
      > prior
      >
      > experience in that arena, just maybe not with how things work in the Linux
      > world.
      >
      > For example, Git only makes sense to people who [1] have used revision
      > control, which far too developers have and [2] have used revision control
      > systems such as Perforce, TeamWare and BitKeeper.
      >
      > So the question one must ask themselves, and have clarity about, is what
      > level of knowledge do they have and do they not have? If you're trying to
      > learn how to administer Linux systems, Linux kernel internals is not the
      > place you want to go. Internals is an advanced topic that comes later in
      > the timeline of learning how to administer and manage systems.
      >
      > Just my observation.
      >
      > On Wed, Jan 27, 2010 at 13:54, Vasudev Seeram <cavguy101@...> wrote:
      >
      > >
      > >
      > > Would they be interested in a free Linux training webinar?
      > >
      > >
      > http://www.linux.com/news/featured-blogs/158:jim-zemlin/280635:linux-market-needs-more-talent
      > >
      > > V.
      > >
      > > --- On Mon, 1/25/10, Richard Bailey <rmjb@... <rmjb%40mail.com>>
      > > wrote:
      > >
      > > From: Richard Bailey <rmjb@... <rmjb%40mail.com>>
      > > Subject: [ttlug] Linux training
      > >
      > > To: "ttlug" <TTLUG@yahoogroups.com <TTLUG%40yahoogroups.com>>
      > > Date: Monday, January 25, 2010, 4:14 PM
      > >
      > >
      > >
      > >
      > > Hello,
      > >
      > > Anyone knows of local Linux training available? The aim is to expose
      > > current
      > >
      > > windows admins to the basics of linux usage and administration. I need to
      > >
      > > get load off myself. SBCS, who had a course running last year doesn't
      > have
      > >
      > > any so far for this year.
      > >
      > > --
      > >
      > > "Asking the wrong questions is the leading cause of wrong answers."
      > >
      > > [Non-text portions of this message have been removed]
      > >
      > > [Non-text portions of this message have been removed]
      > >
      > >
      > >
      >
      >
      >
      > --
      > "You can choose your friends, you can choose the deals." - Equity Private
      >
      > Blog - http://whatderass.blogspot.com/
      > Twitter - @khyron4eva
      >
      >
      > [Non-text portions of this message have been removed]
      >
      >
      >
      > ------------------------------------
      >
      > Help build TTLUG by forwarding this to anyone who is interested in the
      > subject matter or would otherwise benefit from joining the mailing list.
      >
      > Trinidad and Tobago Linux Users Group http://groups.yahoo.com/group/ttlug
      > To subscribe, send an email to_______ TTLUG-subscribe@yahoogroups.com
      > To unsubscribe, send an email to_____ TTLUG-unsubscribe@yahoogroups.com
      > List owner/moderator Richard Jobity__ TTLUG-owner@yahoogroups.com
      > Yahoo! Groups Links
      >
      >
      >
      >


      --
      "Asking the wrong questions is the leading cause of wrong answers."


      [Non-text portions of this message have been removed]
    • Allan Samaroo
      I m not sure I understand your last statement as you repeat the phrase learn how to administer and it conflicts with the previous statement. Why would an
      Message 2 of 20 , Jan 27, 2010
      • 0 Attachment
        I'm not sure I understand your last statement as you repeat the phrase
        'learn how to administer' and it conflicts with the previous statement.

        Why would an administrator, on any platform, ever want to know how the
        internals work? That's the interest of a programmer.

        In my day as an administrator I needed only to know about using the
        technology, not how it works. Even recompiling the kernel doesn't
        require knowledge of the internals.....



        > So the question one must ask themselves, and have clarity about, is what
        > level of knowledge do they have and do they not have? If you're trying to
        > learn how to administer Linux systems, Linux kernel internals is not the
        > place you want to go. Internals is an advanced topic that comes later in
        > the timeline of learning how to administer and manage systems.
      • Khyron
        I think this comment almost made my head explode. It s probably the most absurd thing I ve heard in 9 hours (since I heard some absurd stuff from a few
        Message 3 of 20 , Jan 28, 2010
        • 0 Attachment
          I think this comment almost made my head explode. It's probably the most
          absurd thing I've heard in 9 hours (since I heard some absurd stuff from a
          few co-workers about 7 - 8.5 hours ago).

          Any administrator with an interest or need in doing anything more than the
          most rudimentary work needs to know how his system works internally. It
          is common to have to performance tune a system. It is common to have to
          script or program a solution to a problem, such as when I once had to
          debug an X Windows problem and the only way to do so was to write a
          simple X Windows program in C to test the behavior of the X server. Any
          admin I've ever known who did at least understand how their system worked
          from top to bottom could not be any more than an intermediate level admin.
          Period. I have former bosses, former colleagues, and friends who would
          agree with that. (This reminds me of a former co-worker, now at Google,
          who gave one of my favorite former bosses the best answer to the interview
          question "explain the boot process of an X86 PC". The guy gave him the
          CPU registers that got loaded at various points, and with what, among other
          details! I love it.)

          I can't even begin to explain how infuriating "Why would an administrator,
          on
          any platform, ever want to know how the internals work? That's the interest

          of a programmer" is to me. My first thought is that whoever would make this

          statement is clearly not versed in the Unix tradition. The first
          administrators
          WERE programmers, because they were developing the system they used
          and creating a system friendly to them. The BEST administrators now are
          programmers, because to do the most interesting things with systems, you
          have to be able to write code. (Scripting falls into this programming
          domain
          as well.) The BEST administrators are presented with problems which are
          scarcely tractable without writing code on some level.

          The other piece to this is that most programmers don't know and don't care
          about the impact of their code on the people who will use it. That's why
          their code sucks. Such programmers also don't RESPECT what the sysadmins
          who have to use their crappy software are dealing with. If those programmers

          have to interact with the sysadmins at some point (say, in a corporate IT
          dept.
          or say your local web startup), they will get an earful from those sysadmins

          that they threw under the bus with their garbage code. It is hard to gain
          the
          respect of the people running and managing your code if your code is crappy.

          This leads to the systems guys undermining the programmers, and leads to
          the programmers hearing "no" more than they want. It also leads to things
          such as the systems guys being overworked fixing broken code in production
          and without support from their management and eventually leaving, or
          sysadmins who "go off" on the developers. There are so many other things
          which can and do happen when the developers and sysadmins don't speak
          each other's language and don't care about how each other's worlds work.

          Sysadmins who write code or programmers who sysadmin are able to look at
          such situations from the point of view of both. There may be a creative,
          systems level solution which does not require code (which supports the goals

          of simplicity and maintainability). If there is a need to write code, not
          only
          do they have the ability to do so, but they can address the problem with the

          least amount of code, building the systems components around the code as
          required. Finally, these people are able to bridge the divide better
          between
          operations (sysadmins) and engineering (programmers) who typically DON'T
          understand what each other does and how they work, which leads to all
          types of problems which have been documented by people way smarter than
          me.[1] I'll link to some of the thoughts on the problems of programmers and

          sysadmins who don't grok each other's worlds. The Flickr guys did a great
          presentation on this at Velocity 2009.[2][3]

          A few examples:

          - One of the best systems guys I know owns a company which consults with
          many well known (and lesser known) companies. 2 of his clients contracted
          him to design an image retrieval system which operated in large scale in
          real
          time. Think Facebook Images, Flickr or SmugMug. Only, he had to deliver
          this system at the lowest cost. He wrote about 1700 lines of Perl and close

          to 800 lines of C to build this system, which can run on Linux and Solaris.
          (The demo is deployed on OpenSolaris.) I guarantee that Facebook Images
          and Flickr took a lot longer to build, and required more people to build
          them.
          Now, this is essentially a storage system, but so is FB Images.[1] Since
          you
          will likely run into his name if you read this entire post and the
          references I
          provide, I'll go ahead and say it. He's Theo Schlossnagle of OmniTI[4] and
          I've sent out stuff by Theo before about ZFS and (Open)Solaris.

          - Ben Rockwood, a contributor to the Enlightenment project, is the Director
          of Systems at Joyent. While he blogs and presents about systems admin
          topics, he has submitted code to the OpenSolaris project as well as the
          Enlightenment project.[5][6][7]

          - An acquaintance of mine who previously worked at Ning and most recently
          at Twitter. His background is hardcore computer science, but at Earthlink
          and Ning, his duties were a mix of systems admin/engineering and software
          engineering. At Ning in particular, he both performed sysadmin and software

          engineering duties, and performed on-call duties supporting in his software
          when it broke. Much of his work involved performance, and making the
          Java backend of Ning perform optimally, diagnosing and troubleshooting
          code problems in the systems environment. Usually, there are entire types
          of problems which only appear in production environments, and with web
          companies, the level of traffic and demand placed upon them commonly
          cannot be reproduced under test or QA environments. This is why Facebook
          performs what they call "dark launches". I've linked to his thoughts on the
          divide between operations and engineering; the best way to bridge that
          divide is for people on both sides to transcend it.[8]

          - Small companies (a la web startups or capital efficient early stage tech
          companies) do not have the resources to invest in both sysadmins AND
          programmers. So you take the time and care to invest in someone who
          either can already do these things, or has the ability and desire to learn.
          Employee #4 at Facebook, the first non-founder employee, is such a person.
          (I happen to know him, as well as the VP of Operations.) Apparently he's
          the lead reliability engineer @ Blizzard's battle.net now.[9]

          - When I worked for a storage startup, I dealt with the capital efficiency
          issue. Though I am not a programmer, I spent a lot of time hacking on the
          Linux kernel (yes, even adding some code) to do things which the kernel
          developers in their finite wisdom had not bothered to add. All of this was
          because I needed to know what the kernel was doing in the deployment
          scenario envisioned by the guys building the company's product. They
          weren't sysadmins either, and it showed because the product didn't work.
          It didn't work (generally) but even more importantly, it didn't work the way

          most people wanted to use it.

          (Hint: The US government will not buy a backup system that can't delete
          because their higher security data deletion standards, and not just those
          developed by the national security infrastructure [CIA, NSA, DISA, etc.],
          require you to be able to confirm that the data is gone. The backup
          product my company was developing had the property that it could keep
          data forever. The government wasn't interested. That's a HUGE customer
          that spends over $1,000,000,000,000 US annually.)

          Anyway, this was a problem because we weren't building a product that our
          customers would actually want and have a use for. It showed that we did
          not understand their pains. The CTO, a really smart guy, was what Joel
          Spolsky calls an "architecture astronaut".[10] Not understanding your
          customers, what they want and need, what their pain is, is a sure way to
          die as a company. This was a contributor to the we had problems securing
          a second round of institutional venture financing, and ended up laying off a

          lot of people in early 2002.

          - More times than I can count, I've told some Windows "admin" how their
          OS works, or how to solve some application problem, because I understand
          the fundamental computer science, computer engineering and electrical
          engineering underlying that system. Windows admins -- and apparently
          more Linux people than I would think -- don't get how all of these things
          fit together. Once you know the fundamental concepts, they can be applied
          to any situation and system.

          - Google created the role of the "site reliability engineer (SRE)" which has

          been copied by many organizations.[11] (See above about Facebook and
          Blizzard.) The SRE's role specifically is to transcend the barriers between

          engineering (programmers) and operations (sysadmins) by helping the
          developers make their code work in the systems environment, whether
          that involves actual debugging, coding, scripting, or just working with the
          developers directly so they understand the operational environment and
          processes. They also interact with the operations side, assisting devs to
          optimize their software for deployment into the operational environment,
          then managing the deployment, monitoring, and maintaining the running
          systems. In the best operations environments and practices, the dev
          who wrote the broken code is an escalation point for resolving problems
          due to their bad code. That means that dev will be paged if their code
          breaks in production. The way it should be; you get a new respect for
          what others go through if you can't just throw your garbage code over the
          wall and let others clean up the mess. If you have to get out of bed and
          spend 2 hours of your time fixing the problem you created, you're going
          to be inclined to not create problems. Facebook, Flickr, Yahoo!, and many
          other online organizations are embracing the SRE role for this reason.

          - An sysadmin (ops guy) develops software (engineering) to perform ops
          better. The UNIX tradition at work.[12]

          - If there wasn't a need for practitioners to understand internals, none of
          the following would exist. Engineering/programmers can't exist idly in
          their
          own world w/o interacting with, and having accountability to, their
          customers. ACM Queue was founded for this reason.[13] Now, the list:

          http://www.amazon.com/exec/obidos/ASIN/0201750392/richteer-20

          (A review of the above by Ben Rockwood can be found at
          http://cuddletech.com/articles/ssp_review/)

          http://www.amazon.com/gp/product/0131482092?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131482092

          http://www.amazon.com/gp/product/0131568191?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131568191

          http://www.amazon.com/gp/product/0130952494?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0130952494

          http://www.amazon.com/gp/product/0131411551?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131411551

          (I own the above, but there are 2 more in the series. Stevens died FAR too
          soon, but his contributions are legendary.)

          http://www.amazon.com/gp/product/0131876716?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131876716

          (I own Volume 1 of the 3rd edition, but I've just added the 5th edition to
          my Amazon Wish List.)

          http://www.amazon.com/gp/product/0735625301?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0735625301

          - Briefly, I started a project with my current job to update monitoring. In

          exploring this, I worked very closely with the guy who is primarily
          responsible for writing the monitoring tests for things like CPU
          utilization.
          (He's a former Unix sysadmin, but now he focuses on developing monitoring.
          Lots of Perl and C code. Having been an admin allows him some appreciation
          for what impact bad monitors have on my group; too bad his management
          has not understood this.) Anyway, in order to reduce the number of bogus
          CPU monitoring alerts (e.g. Oracle backups running nightly, or NetBackup
          backups, or some custom process developed by the customer), I was required
          to understand what his monitoring tests were doing. That meant reading
          Perl code. The code uses internal Solaris kernel variables - poorly - too
          calculate values. Using this information, and the OpenSolaris code online
          and open for reading, I began tracking how the Solaris kernel measures
          CPU utilization. I still have to spend time with it, but I can quantify
          just
          how poorly his Perl code is working. Additionally, I may have found a bug
          in the Solaris code in how it calculates CPU utilization. This would be a
          20+
          year old logic bug. More verification is required, but you're full of it if
          you
          think a sysadmin shouldn't know the internals of their OS.

          Computers are systems. Networks are systems. People and society are
          systems. All of these things have to come together harmoniously into a
          larger system in order to make effective use of the resources. Lacking an
          understanding of how all of the components work with each other, from
          the top to the bottom, is a recipe for suboptimal use and even failure. If
          you are going to call yourself a computer engineer, you will have to merge
          knowledge of software with knowledge of how that software interacts
          with the hardware it runs on and the networks (hardware and software)
          that it will traverse. That means understanding how the application uses
          the OS; how the OS uses the hardware; what happens in the hardware
          when the code is running on it; how the protocols fit into the picture and
          interact with the apps and OS; how the data fits on the wire, or over the
          airwaves; and whatever other components are involved.

          Anyway, I'm done. I encourage all to read the references, especially the
          ones I did not mention above. They add color to the discussion. I have
          been writing this e-mail for far too long as it is, and I don't want to
          start
          repeating myself. There is plenty of reason for sysadmins to understand
          how the code works. You can't fix (properly) what you don't understand.
          Read the joke "X Marks the Spot" at http://bit.ly/c8Af04 for a great
          example. Some of the references also talk about how software development
          methodologies, particularly Agile Development, can be applied for sysadmins
          and in operational environments. I include this for all of you programmers
          on this list.


          References:

          1. http://www.agileweboperations.com/partitions-and-warfare/

          2.
          http://www.youtube.com/watch?v=LdOe18KhtT4&feature=PlayList&p=99E1757A2C010F4B&index=23

          3. http://www.kitchensoap.com/2009/06/23/slides-for-velocity-talk-2009/

          4. http://lethargy.org/~jesus/about.html<http://lethargy.org/%7Ejesus/about.html>

          5. http://cuddletech.com/tools/index.shtml

          6. http://cuddletech.com/articles/index.shtml

          7. http://cuddletech.com/blog/

          8.
          http://daemons.net/~clay/2009/04/02/engineering-and-operations-bridging-the-divide/<http://daemons.net/%7Eclay/2009/04/02/engineering-and-operations-bridging-the-divide/>

          9. http://www.taner.net/resume.html

          10. http://www.joelonsoftware.com/articles/fog0000000018.html

          11. http://research.google.com/archive/LinuxWorld-07-describeSRE.pdf

          12.
          http://www.agileweboperations.com/configuration-management-remixed-introducing-carpet/

          13. http://blogs.sun.com/bmc/entry/queue_cacm_and_the_rebirth

          14. http://www.agileweboperations.com/devopsdays-2009/

          15. http://www.kitchensoap.com/category/capacity-planning/

          16. http://www.kitchensoap.com/category/webops/

          17.
          http://www.kitchensoap.com/2009/01/14/mechanical-analogies-to-web-stuff-part-1/

          18.
          http://www.kitchensoap.com/2009/05/06/mechanical-analogies-to-web-stuff-part-2/

          19.
          http://radar.oreilly.com/2009/06/jonathan-heiliger-facebook-velocity-webops.html

          20.
          http://www.agileweboperations.com/the-12-principles-behind-the-agile-manifesto-adapted-to-web-operations/

          21.
          http://gigaom.com/2009/06/25/facebooks-jonathan-heiliger-talks-infrastructure-and-usernames/

          On Wed, Jan 27, 2010 at 21:54, Allan Samaroo <samaroo@...> wrote:

          >
          >
          > I'm not sure I understand your last statement as you repeat the phrase
          > 'learn how to administer' and it conflicts with the previous statement.
          >
          > Why would an administrator, on any platform, ever want to know how the
          > internals work? That's the interest of a programmer.
          >
          > In my day as an administrator I needed only to know about using the
          > technology, not how it works. Even recompiling the kernel doesn't
          > require knowledge of the internals.....
          >
          >
          > > So the question one must ask themselves, and have clarity about, is what
          > > level of knowledge do they have and do they not have? If you're trying to
          > > learn how to administer Linux systems, Linux kernel internals is not the
          > > place you want to go. Internals is an advanced topic that comes later in
          > > the timeline of learning how to administer and manage systems.
          >
          >



          --
          "You can choose your friends, you can choose the deals." - Equity Private

          Blog - http://whatderass.blogspot.com/
          Twitter - @khyron4eva


          [Non-text portions of this message have been removed]
        • Hassan Voyeau
          What is your definition of system internals? ... [Non-text portions of this message have been removed]
          Message 4 of 20 , Jan 28, 2010
          • 0 Attachment
            What is your definition of system internals?

            On 28 January 2010 07:04, Khyron <khyron4eva@...> wrote:

            >
            > Any administrator with an interest or need in doing anything more than the
            > most rudimentary work needs to know how his system works internally.


            [Non-text portions of this message have been removed]
          • Khyron
            I hate it when this happens BUT I include these for completeness... http://velocityconference.blip.tv/ Here you ll find video from all of Velocity 2009,
            Message 5 of 20 , Jan 28, 2010
            • 0 Attachment
              I hate it when this happens BUT I include these for completeness...

              http://velocityconference.blip.tv/

              Here you'll find video from all of Velocity 2009, including the Allspaw talk

              on "10+ Deploys Per Day". Allspaw is the former Operations chief at
              Flickr, now the head of such at Etsy. Heiliger's keynote is up there as
              well.

              Additionally, I may not have finished my thoughts on computer engineering.
              In my opinion, there has never been a division between hardware and
              software. The software doesn't have a platform without the hardware. The
              hardware has no purpose without the software. A true master, or someone
              seeking mastery, has to be able to deal with both. I consider myself such a

              seeker.

              The following video is from the Chief JVM Architect, Cliff Click, at Azul
              Systems (which hopefully will keep the interest of you Java devs). He came
              from Sun where he focused on increasing the performance of the JVM and
              developed the HotSpot Server Compiler. We all know and love HotSpot, I
              would think. Anyway, he does a deep dive on how modern x86 hardware
              operates, and the implications for the highly parallel software world we're
              moving too. The audience is all Java developers. So, why is this important

              to them? Because once you have the code working, if you're lucky, you'll
              need to make it performant. (Thanks to Sun and my best friend, a former
              Sun SE, for that word.) If you don't know how the system interprets and
              runs you code, at best, you can do nothing. At worst, you'll make your bad
              system insufferable. Not a recipe for success. This requires understanding

              the hardware. But wait! These are software guys; they don't need to know
              anything about the hardware. Right, Allan?

              I would argue that if you're a craftsman, seeking mastery, then the hardware

              matters to you. You may not start off dealing directly with the
              performance.
              However, it will eventually come up, after you've ensured the correctness of

              your software. In this modern era, you cannot count on CPUs to just run
              your code faster. Those days are over; single threaded performance fixes
              have run their course. Look at the Sun UltraSPARC T1/T2/T2+ processors.
              The single threaded performance is weak, but the CPUs are designed to
              deliver high throughput not exceptional single threaded performance. How?
              By running lots of threads in parallel. If you've got a multithreaded
              Apache
              backing your website, or other modern web servers (nginx, Varnish,
              lighttpd, etc.), this might matter to you as a sysadmin.

              Yes, you have to have a grasp of all the pieces and how they fit together.

              I also forgot to mention that people who can cross both worlds - the Ops
              and Eng worlds - are EXTREMELY rare. Which means, they can make a
              lot of money and get to work on interesting problems, because there is so
              much that they bring to the table in one person, and that includes holistic
              understanding of software, hardware, networks, client-side issues, people,
              and how all the pieces work together.

              (The most interesting systems problems to me include those where the
              solution is to adjust someone's misguided expectations. No "technical"
              solution involved, but a solution none the less.)

              Cheers!

              On Thu, Jan 28, 2010 at 06:04, Khyron <khyron4eva@...> wrote:

              > I think this comment almost made my head explode. It's probably the most
              > absurd thing I've heard in 9 hours (since I heard some absurd stuff from a
              > few co-workers about 7 - 8.5 hours ago).
              >
              > Any administrator with an interest or need in doing anything more than the
              > most rudimentary work needs to know how his system works internally. It
              > is common to have to performance tune a system. It is common to have to
              > script or program a solution to a problem, such as when I once had to
              > debug an X Windows problem and the only way to do so was to write a
              > simple X Windows program in C to test the behavior of the X server. Any
              > admin I've ever known who did at least understand how their system worked
              > from top to bottom could not be any more than an intermediate level admin.
              > Period. I have former bosses, former colleagues, and friends who would
              > agree with that. (This reminds me of a former co-worker, now at Google,
              > who gave one of my favorite former bosses the best answer to the interview
              > question "explain the boot process of an X86 PC". The guy gave him the
              > CPU registers that got loaded at various points, and with what, among other
              >
              > details! I love it.)
              >
              > I can't even begin to explain how infuriating "Why would an administrator,
              > on
              > any platform, ever want to know how the internals work? That's the
              > interest
              > of a programmer" is to me. My first thought is that whoever would make
              > this
              > statement is clearly not versed in the Unix tradition. The first
              > administrators
              > WERE programmers, because they were developing the system they used
              > and creating a system friendly to them. The BEST administrators now are
              > programmers, because to do the most interesting things with systems, you
              > have to be able to write code. (Scripting falls into this programming
              > domain
              > as well.) The BEST administrators are presented with problems which are
              > scarcely tractable without writing code on some level.
              >
              > The other piece to this is that most programmers don't know and don't care
              > about the impact of their code on the people who will use it. That's why
              > their code sucks. Such programmers also don't RESPECT what the sysadmins
              > who have to use their crappy software are dealing with. If those
              > programmers
              > have to interact with the sysadmins at some point (say, in a corporate IT
              > dept.
              > or say your local web startup), they will get an earful from those
              > sysadmins
              > that they threw under the bus with their garbage code. It is hard to gain
              > the
              > respect of the people running and managing your code if your code is
              > crappy.
              > This leads to the systems guys undermining the programmers, and leads to
              > the programmers hearing "no" more than they want. It also leads to things
              > such as the systems guys being overworked fixing broken code in production
              > and without support from their management and eventually leaving, or
              > sysadmins who "go off" on the developers. There are so many other things
              > which can and do happen when the developers and sysadmins don't speak
              > each other's language and don't care about how each other's worlds work.
              >
              > Sysadmins who write code or programmers who sysadmin are able to look at
              > such situations from the point of view of both. There may be a creative,
              > systems level solution which does not require code (which supports the
              > goals
              > of simplicity and maintainability). If there is a need to write code, not
              > only
              > do they have the ability to do so, but they can address the problem with
              > the
              > least amount of code, building the systems components around the code as
              > required. Finally, these people are able to bridge the divide better
              > between
              > operations (sysadmins) and engineering (programmers) who typically DON'T
              > understand what each other does and how they work, which leads to all
              > types of problems which have been documented by people way smarter than
              > me.[1] I'll link to some of the thoughts on the problems of programmers
              > and
              > sysadmins who don't grok each other's worlds. The Flickr guys did a great
              > presentation on this at Velocity 2009.[2][3]
              >
              > A few examples:
              >
              > - One of the best systems guys I know owns a company which consults with
              > many well known (and lesser known) companies. 2 of his clients contracted
              > him to design an image retrieval system which operated in large scale in
              > real
              > time. Think Facebook Images, Flickr or SmugMug. Only, he had to deliver
              > this system at the lowest cost. He wrote about 1700 lines of Perl and
              > close
              > to 800 lines of C to build this system, which can run on Linux and Solaris.
              >
              > (The demo is deployed on OpenSolaris.) I guarantee that Facebook Images
              > and Flickr took a lot longer to build, and required more people to build
              > them.
              > Now, this is essentially a storage system, but so is FB Images.[1] Since
              > you
              > will likely run into his name if you read this entire post and the
              > references I
              > provide, I'll go ahead and say it. He's Theo Schlossnagle of OmniTI[4] and
              > I've sent out stuff by Theo before about ZFS and (Open)Solaris.
              >
              > - Ben Rockwood, a contributor to the Enlightenment project, is the Director
              >
              > of Systems at Joyent. While he blogs and presents about systems admin
              > topics, he has submitted code to the OpenSolaris project as well as the
              > Enlightenment project.[5][6][7]
              >
              > - An acquaintance of mine who previously worked at Ning and most recently
              > at Twitter. His background is hardcore computer science, but at Earthlink
              > and Ning, his duties were a mix of systems admin/engineering and software
              > engineering. At Ning in particular, he both performed sysadmin and
              > software
              > engineering duties, and performed on-call duties supporting in his software
              >
              > when it broke. Much of his work involved performance, and making the
              > Java backend of Ning perform optimally, diagnosing and troubleshooting
              > code problems in the systems environment. Usually, there are entire types
              > of problems which only appear in production environments, and with web
              > companies, the level of traffic and demand placed upon them commonly
              > cannot be reproduced under test or QA environments. This is why Facebook
              > performs what they call "dark launches". I've linked to his thoughts on the
              >
              > divide between operations and engineering; the best way to bridge that
              > divide is for people on both sides to transcend it.[8]
              >
              > - Small companies (a la web startups or capital efficient early stage tech
              > companies) do not have the resources to invest in both sysadmins AND
              > programmers. So you take the time and care to invest in someone who
              > either can already do these things, or has the ability and desire to learn.
              >
              > Employee #4 at Facebook, the first non-founder employee, is such a person.
              > (I happen to know him, as well as the VP of Operations.) Apparently he's
              > the lead reliability engineer @ Blizzard's battle.net now.[9]
              >
              > - When I worked for a storage startup, I dealt with the capital efficiency
              > issue. Though I am not a programmer, I spent a lot of time hacking on the
              > Linux kernel (yes, even adding some code) to do things which the kernel
              > developers in their finite wisdom had not bothered to add. All of this was
              >
              > because I needed to know what the kernel was doing in the deployment
              > scenario envisioned by the guys building the company's product. They
              > weren't sysadmins either, and it showed because the product didn't work.
              > It didn't work (generally) but even more importantly, it didn't work the
              > way
              > most people wanted to use it.
              >
              > (Hint: The US government will not buy a backup system that can't delete
              > because their higher security data deletion standards, and not just those
              > developed by the national security infrastructure [CIA, NSA, DISA, etc.],
              > require you to be able to confirm that the data is gone. The backup
              > product my company was developing had the property that it could keep
              > data forever. The government wasn't interested. That's a HUGE customer
              > that spends over $1,000,000,000,000 US annually.)
              >
              > Anyway, this was a problem because we weren't building a product that our
              > customers would actually want and have a use for. It showed that we did
              > not understand their pains. The CTO, a really smart guy, was what Joel
              > Spolsky calls an "architecture astronaut".[10] Not understanding your
              > customers, what they want and need, what their pain is, is a sure way to
              > die as a company. This was a contributor to the we had problems securing
              > a second round of institutional venture financing, and ended up laying off
              > a
              > lot of people in early 2002.
              >
              > - More times than I can count, I've told some Windows "admin" how their
              > OS works, or how to solve some application problem, because I understand
              > the fundamental computer science, computer engineering and electrical
              > engineering underlying that system. Windows admins -- and apparently
              > more Linux people than I would think -- don't get how all of these things
              > fit together. Once you know the fundamental concepts, they can be applied
              > to any situation and system.
              >
              > - Google created the role of the "site reliability engineer (SRE)" which
              > has
              > been copied by many organizations.[11] (See above about Facebook and
              > Blizzard.) The SRE's role specifically is to transcend the barriers
              > between
              > engineering (programmers) and operations (sysadmins) by helping the
              > developers make their code work in the systems environment, whether
              > that involves actual debugging, coding, scripting, or just working with the
              >
              > developers directly so they understand the operational environment and
              > processes. They also interact with the operations side, assisting devs to
              > optimize their software for deployment into the operational environment,
              > then managing the deployment, monitoring, and maintaining the running
              > systems. In the best operations environments and practices, the dev
              > who wrote the broken code is an escalation point for resolving problems
              > due to their bad code. That means that dev will be paged if their code
              > breaks in production. The way it should be; you get a new respect for
              > what others go through if you can't just throw your garbage code over the
              > wall and let others clean up the mess. If you have to get out of bed and
              > spend 2 hours of your time fixing the problem you created, you're going
              > to be inclined to not create problems. Facebook, Flickr, Yahoo!, and many
              > other online organizations are embracing the SRE role for this reason.
              >
              > - An sysadmin (ops guy) develops software (engineering) to perform ops
              > better. The UNIX tradition at work.[12]
              >
              > - If there wasn't a need for practitioners to understand internals, none of
              >
              > the following would exist. Engineering/programmers can't exist idly in
              > their
              > own world w/o interacting with, and having accountability to, their
              > customers. ACM Queue was founded for this reason.[13] Now, the list:
              >
              > http://www.amazon.com/exec/obidos/ASIN/0201750392/richteer-20
              >
              > (A review of the above by Ben Rockwood can be found at
              > http://cuddletech.com/articles/ssp_review/)
              >
              >
              > http://www.amazon.com/gp/product/0131482092?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131482092
              >
              >
              > http://www.amazon.com/gp/product/0131568191?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131568191
              >
              >
              > http://www.amazon.com/gp/product/0130952494?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0130952494
              >
              >
              > http://www.amazon.com/gp/product/0131411551?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131411551
              >
              > (I own the above, but there are 2 more in the series. Stevens died FAR too
              >
              > soon, but his contributions are legendary.)
              >
              >
              > http://www.amazon.com/gp/product/0131876716?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131876716
              >
              > (I own Volume 1 of the 3rd edition, but I've just added the 5th edition to
              > my Amazon Wish List.)
              >
              >
              > http://www.amazon.com/gp/product/0735625301?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0735625301
              >
              > - Briefly, I started a project with my current job to update monitoring.
              > In
              > exploring this, I worked very closely with the guy who is primarily
              > responsible for writing the monitoring tests for things like CPU
              > utilization.
              > (He's a former Unix sysadmin, but now he focuses on developing monitoring.
              >
              > Lots of Perl and C code. Having been an admin allows him some appreciation
              >
              > for what impact bad monitors have on my group; too bad his management
              > has not understood this.) Anyway, in order to reduce the number of bogus
              > CPU monitoring alerts (e.g. Oracle backups running nightly, or NetBackup
              > backups, or some custom process developed by the customer), I was required
              > to understand what his monitoring tests were doing. That meant reading
              > Perl code. The code uses internal Solaris kernel variables - poorly - too
              > calculate values. Using this information, and the OpenSolaris code online
              > and open for reading, I began tracking how the Solaris kernel measures
              > CPU utilization. I still have to spend time with it, but I can quantify
              > just
              > how poorly his Perl code is working. Additionally, I may have found a bug
              > in the Solaris code in how it calculates CPU utilization. This would be a
              > 20+
              > year old logic bug. More verification is required, but you're full of it
              > if you
              > think a sysadmin shouldn't know the internals of their OS.
              >
              > Computers are systems. Networks are systems. People and society are
              > systems. All of these things have to come together harmoniously into a
              > larger system in order to make effective use of the resources. Lacking an
              > understanding of how all of the components work with each other, from
              > the top to the bottom, is a recipe for suboptimal use and even failure. If
              >
              > you are going to call yourself a computer engineer, you will have to merge
              > knowledge of software with knowledge of how that software interacts
              > with the hardware it runs on and the networks (hardware and software)
              > that it will traverse. That means understanding how the application uses
              > the OS; how the OS uses the hardware; what happens in the hardware
              > when the code is running on it; how the protocols fit into the picture and
              > interact with the apps and OS; how the data fits on the wire, or over the
              > airwaves; and whatever other components are involved.
              >
              > Anyway, I'm done. I encourage all to read the references, especially the
              > ones I did not mention above. They add color to the discussion. I have
              > been writing this e-mail for far too long as it is, and I don't want to
              > start
              > repeating myself. There is plenty of reason for sysadmins to understand
              > how the code works. You can't fix (properly) what you don't understand.
              > Read the joke "X Marks the Spot" at http://bit.ly/c8Af04 for a great
              > example. Some of the references also talk about how software development
              > methodologies, particularly Agile Development, can be applied for sysadmins
              >
              > and in operational environments. I include this for all of you programmers
              >
              > on this list.
              >
              >
              > References:
              >
              > 1. http://www.agileweboperations.com/partitions-and-warfare/
              >
              > 2.
              > http://www.youtube.com/watch?v=LdOe18KhtT4&feature=PlayList&p=99E1757A2C010F4B&index=23
              >
              > 3. http://www.kitchensoap.com/2009/06/23/slides-for-velocity-talk-2009/
              >
              > 4. http://lethargy.org/~jesus/about.html<http://lethargy.org/%7Ejesus/about.html>
              >
              > 5. http://cuddletech.com/tools/index.shtml
              >
              > 6. http://cuddletech.com/articles/index.shtml
              >
              > 7. http://cuddletech.com/blog/
              >
              > 8.
              > http://daemons.net/~clay/2009/04/02/engineering-and-operations-bridging-the-divide/<http://daemons.net/%7Eclay/2009/04/02/engineering-and-operations-bridging-the-divide/>
              >
              > 9. http://www.taner.net/resume.html
              >
              > 10. http://www.joelonsoftware.com/articles/fog0000000018.html
              >
              > 11. http://research.google.com/archive/LinuxWorld-07-describeSRE.pdf
              >
              > 12.
              > http://www.agileweboperations.com/configuration-management-remixed-introducing-carpet/
              >
              > 13. http://blogs.sun.com/bmc/entry/queue_cacm_and_the_rebirth
              >
              > 14. http://www.agileweboperations.com/devopsdays-2009/
              >
              > 15. http://www.kitchensoap.com/category/capacity-planning/
              >
              > 16. http://www.kitchensoap.com/category/webops/
              >
              > 17.
              > http://www.kitchensoap.com/2009/01/14/mechanical-analogies-to-web-stuff-part-1/
              >
              > 18.
              > http://www.kitchensoap.com/2009/05/06/mechanical-analogies-to-web-stuff-part-2/
              >
              > 19.
              > http://radar.oreilly.com/2009/06/jonathan-heiliger-facebook-velocity-webops.html
              >
              > 20.
              > http://www.agileweboperations.com/the-12-principles-behind-the-agile-manifesto-adapted-to-web-operations/
              >
              > 21.
              > http://gigaom.com/2009/06/25/facebooks-jonathan-heiliger-talks-infrastructure-and-usernames/
              >
              >
              > On Wed, Jan 27, 2010 at 21:54, Allan Samaroo <samaroo@...> wrote:
              >
              >>
              >>
              >> I'm not sure I understand your last statement as you repeat the phrase
              >> 'learn how to administer' and it conflicts with the previous statement.
              >>
              >> Why would an administrator, on any platform, ever want to know how the
              >> internals work? That's the interest of a programmer.
              >>
              >> In my day as an administrator I needed only to know about using the
              >> technology, not how it works. Even recompiling the kernel doesn't
              >> require knowledge of the internals.....
              >>
              >>
              >> > So the question one must ask themselves, and have clarity about, is what
              >> > level of knowledge do they have and do they not have? If you're trying
              >> to
              >> > learn how to administer Linux systems, Linux kernel internals is not the
              >> > place you want to go. Internals is an advanced topic that comes later in
              >> > the timeline of learning how to administer and manage systems.
              >>
              >>
              >
              >
              >
              > --
              > "You can choose your friends, you can choose the deals." - Equity Private
              >
              > Blog - http://whatderass.blogspot.com/
              > Twitter - @khyron4eva
              >
              >


              --
              "You can choose your friends, you can choose the deals." - Equity Private

              Blog - http://whatderass.blogspot.com/
              Twitter - @khyron4eva


              [Non-text portions of this message have been removed]
            • Khyron
              Hmmm, didn t expect this question. Internals would be all the stuff in the black box . For a sysadmin, it s deeper than for say, the sysadmin s boss or a
              Message 6 of 20 , Jan 28, 2010
              • 0 Attachment
                Hmmm, didn't expect this question.

                Internals would be all the stuff in the "black box". For a sysadmin, it's
                deeper than for say, the sysadmin's boss or a user/customer.

                Kernel level; libraries, particularly the C library; syscalls; CPU
                architecture;
                memory architecture and layout; bus issues; firmware; application level
                code; software execution environment, including how the virtual machine
                (e.g. JVM for example) works, if present; network protocols, including how
                the standards for them are defined; application protocols; network hardware
                issues, including in some cases the CPU architecture of the networking
                device; APIs and ABIs. ALL of that is internals. Understand, this may not
                be a complete list. After 22 hours, I need sleep.

                Hope that answers the question.

                On Thu, Jan 28, 2010 at 06:28, Hassan Voyeau <hassan.voyeau@...>wrote:

                >
                >
                > What is your definition of system internals?
                >
                >
                > On 28 January 2010 07:04, Khyron <khyron4eva@...<khyron4eva%40gmail.com>>
                > wrote:
                >
                > >
                > > Any administrator with an interest or need in doing anything more than
                > the
                > > most rudimentary work needs to know how his system works internally.
                >
                > [Non-text portions of this message have been removed]
                >
                >
                >



                --
                "You can choose your friends, you can choose the deals." - Equity Private

                Blog - http://whatderass.blogspot.com/
                Twitter - @khyron4eva


                [Non-text portions of this message have been removed]
              • Khyron
                And data structures. How I could forget data structures is beyond me. ... -- You can choose your friends, you can choose the deals. - Equity Private Blog -
                Message 7 of 20 , Jan 28, 2010
                • 0 Attachment
                  And data structures. How I could forget data structures is beyond me.

                  On Thu, Jan 28, 2010 at 06:39, Khyron <khyron4eva@...> wrote:

                  > Hmmm, didn't expect this question.
                  >
                  > Internals would be all the stuff in the "black box". For a sysadmin, it's
                  > deeper than for say, the sysadmin's boss or a user/customer.
                  >
                  > Kernel level; libraries, particularly the C library; syscalls; CPU
                  > architecture;
                  > memory architecture and layout; bus issues; firmware; application level
                  > code; software execution environment, including how the virtual machine
                  > (e.g. JVM for example) works, if present; network protocols, including how
                  > the standards for them are defined; application protocols; network hardware
                  >
                  > issues, including in some cases the CPU architecture of the networking
                  > device; APIs and ABIs. ALL of that is internals. Understand, this may not
                  >
                  > be a complete list. After 22 hours, I need sleep.
                  >
                  > Hope that answers the question.
                  >
                  >
                  > On Thu, Jan 28, 2010 at 06:28, Hassan Voyeau <hassan.voyeau@...>wrote:
                  >
                  >>
                  >>
                  >> What is your definition of system internals?
                  >>
                  >>
                  >> On 28 January 2010 07:04, Khyron <khyron4eva@...<khyron4eva%40gmail.com>>
                  >> wrote:
                  >>
                  >> >
                  >> > Any administrator with an interest or need in doing anything more than
                  >> the
                  >> > most rudimentary work needs to know how his system works internally.
                  >>
                  >> [Non-text portions of this message have been removed]
                  >>
                  >>
                  >>
                  >
                  >
                  >
                  > --
                  > "You can choose your friends, you can choose the deals." - Equity Private
                  >
                  > Blog - http://whatderass.blogspot.com/
                  > Twitter - @khyron4eva
                  >
                  >


                  --
                  "You can choose your friends, you can choose the deals." - Equity Private

                  Blog - http://whatderass.blogspot.com/
                  Twitter - @khyron4eva


                  [Non-text portions of this message have been removed]
                • Allan Samaroo
                  Wow! What a response! Did you think I was trying to insult you? This list is to share knowledge and to assist. Your response is more of an attack and is a
                  Message 8 of 20 , Jan 28, 2010
                  • 0 Attachment
                    Wow! What a response! Did you think I was trying to insult you? This
                    list is to share knowledge and to assist. Your response is more of an
                    attack and is a display of poor behaviour. If you can't answer a simple
                    question with a simple response then don't. This is not a hardcore tech
                    list.

                    Clearly you do not realise that the roles of an administrator and a
                    programmer have evolved and therefore the knowledge now required for
                    each role is not the same. This happened for a reason. Sure one can be
                    both but that choice is theirs not yours. Saying that it should be so
                    because that's how it was historically is to me as absurd as you
                    interpreted my question.

                    I'm sure the material in your post is interesting and may be valuable to
                    some so I thank you for that. I wish I had the time to read them.

                    You still didn't explain your conflicting statements and you continue in
                    a similar manner. I'm not sure you actually know what you're talking
                    about. Perhaps you're confusing systems administrator with systems
                    engineer? It's difficult to tell with the way you write.





                    Khyron wrote:
                    > I think this comment almost made my head explode. It's probably the most
                    > absurd thing I've heard in 9 hours (since I heard some absurd stuff from a
                    > few co-workers about 7 - 8.5 hours ago).
                    >
                    > Any administrator with an interest or need in doing anything more than the
                    > most rudimentary work needs to know how his system works internally. It
                    > is common to have to performance tune a system. It is common to have to
                    > script or program a solution to a problem, such as when I once had to
                    > debug an X Windows problem and the only way to do so was to write a
                    > simple X Windows program in C to test the behavior of the X server. Any
                    > admin I've ever known who did at least understand how their system worked
                    > from top to bottom could not be any more than an intermediate level admin.
                    > Period. I have former bosses, former colleagues, and friends who would
                    > agree with that. (This reminds me of a former co-worker, now at Google,
                    > who gave one of my favorite former bosses the best answer to the interview
                    > question "explain the boot process of an X86 PC". The guy gave him the
                    > CPU registers that got loaded at various points, and with what, among other
                    > details! I love it.)
                    >
                    > I can't even begin to explain how infuriating "Why would an administrator,
                    > on
                    > any platform, ever want to know how the internals work? That's the interest
                    >
                    > of a programmer" is to me. My first thought is that whoever would make this
                    >
                    > statement is clearly not versed in the Unix tradition. The first
                    > administrators
                    > WERE programmers, because they were developing the system they used
                    > and creating a system friendly to them. The BEST administrators now are
                    > programmers, because to do the most interesting things with systems, you
                    > have to be able to write code. (Scripting falls into this programming
                    > domain
                    > as well.) The BEST administrators are presented with problems which are
                    > scarcely tractable without writing code on some level.
                    >
                    > The other piece to this is that most programmers don't know and don't care
                    > about the impact of their code on the people who will use it. That's why
                    > their code sucks. Such programmers also don't RESPECT what the sysadmins
                    > who have to use their crappy software are dealing with. If those programmers
                    >
                    > have to interact with the sysadmins at some point (say, in a corporate IT
                    > dept.
                    > or say your local web startup), they will get an earful from those sysadmins
                    >
                    > that they threw under the bus with their garbage code. It is hard to gain
                    > the
                    > respect of the people running and managing your code if your code is crappy.
                    >
                    > This leads to the systems guys undermining the programmers, and leads to
                    > the programmers hearing "no" more than they want. It also leads to things
                    > such as the systems guys being overworked fixing broken code in production
                    > and without support from their management and eventually leaving, or
                    > sysadmins who "go off" on the developers. There are so many other things
                    > which can and do happen when the developers and sysadmins don't speak
                    > each other's language and don't care about how each other's worlds work.
                    >
                    > Sysadmins who write code or programmers who sysadmin are able to look at
                    > such situations from the point of view of both. There may be a creative,
                    > systems level solution which does not require code (which supports the goals
                    >
                    > of simplicity and maintainability). If there is a need to write code, not
                    > only
                    > do they have the ability to do so, but they can address the problem with the
                    >
                    > least amount of code, building the systems components around the code as
                    > required. Finally, these people are able to bridge the divide better
                    > between
                    > operations (sysadmins) and engineering (programmers) who typically DON'T
                    > understand what each other does and how they work, which leads to all
                    > types of problems which have been documented by people way smarter than
                    > me.[1] I'll link to some of the thoughts on the problems of programmers and
                    >
                    > sysadmins who don't grok each other's worlds. The Flickr guys did a great
                    > presentation on this at Velocity 2009.[2][3]
                    >
                    > A few examples:
                    >
                    > - One of the best systems guys I know owns a company which consults with
                    > many well known (and lesser known) companies. 2 of his clients contracted
                    > him to design an image retrieval system which operated in large scale in
                    > real
                    > time. Think Facebook Images, Flickr or SmugMug. Only, he had to deliver
                    > this system at the lowest cost. He wrote about 1700 lines of Perl and close
                    >
                    > to 800 lines of C to build this system, which can run on Linux and Solaris.
                    > (The demo is deployed on OpenSolaris.) I guarantee that Facebook Images
                    > and Flickr took a lot longer to build, and required more people to build
                    > them.
                    > Now, this is essentially a storage system, but so is FB Images.[1] Since
                    > you
                    > will likely run into his name if you read this entire post and the
                    > references I
                    > provide, I'll go ahead and say it. He's Theo Schlossnagle of OmniTI[4] and
                    > I've sent out stuff by Theo before about ZFS and (Open)Solaris.
                    >
                    > - Ben Rockwood, a contributor to the Enlightenment project, is the Director
                    > of Systems at Joyent. While he blogs and presents about systems admin
                    > topics, he has submitted code to the OpenSolaris project as well as the
                    > Enlightenment project.[5][6][7]
                    >
                    > - An acquaintance of mine who previously worked at Ning and most recently
                    > at Twitter. His background is hardcore computer science, but at Earthlink
                    > and Ning, his duties were a mix of systems admin/engineering and software
                    > engineering. At Ning in particular, he both performed sysadmin and software
                    >
                    > engineering duties, and performed on-call duties supporting in his software
                    > when it broke. Much of his work involved performance, and making the
                    > Java backend of Ning perform optimally, diagnosing and troubleshooting
                    > code problems in the systems environment. Usually, there are entire types
                    > of problems which only appear in production environments, and with web
                    > companies, the level of traffic and demand placed upon them commonly
                    > cannot be reproduced under test or QA environments. This is why Facebook
                    > performs what they call "dark launches". I've linked to his thoughts on the
                    > divide between operations and engineering; the best way to bridge that
                    > divide is for people on both sides to transcend it.[8]
                    >
                    > - Small companies (a la web startups or capital efficient early stage tech
                    > companies) do not have the resources to invest in both sysadmins AND
                    > programmers. So you take the time and care to invest in someone who
                    > either can already do these things, or has the ability and desire to learn.
                    > Employee #4 at Facebook, the first non-founder employee, is such a person.
                    > (I happen to know him, as well as the VP of Operations.) Apparently he's
                    > the lead reliability engineer @ Blizzard's battle.net now.[9]
                    >
                    > - When I worked for a storage startup, I dealt with the capital efficiency
                    > issue. Though I am not a programmer, I spent a lot of time hacking on the
                    > Linux kernel (yes, even adding some code) to do things which the kernel
                    > developers in their finite wisdom had not bothered to add. All of this was
                    > because I needed to know what the kernel was doing in the deployment
                    > scenario envisioned by the guys building the company's product. They
                    > weren't sysadmins either, and it showed because the product didn't work.
                    > It didn't work (generally) but even more importantly, it didn't work the way
                    >
                    > most people wanted to use it.
                    >
                    > (Hint: The US government will not buy a backup system that can't delete
                    > because their higher security data deletion standards, and not just those
                    > developed by the national security infrastructure [CIA, NSA, DISA, etc.],
                    > require you to be able to confirm that the data is gone. The backup
                    > product my company was developing had the property that it could keep
                    > data forever. The government wasn't interested. That's a HUGE customer
                    > that spends over $1,000,000,000,000 US annually.)
                    >
                    > Anyway, this was a problem because we weren't building a product that our
                    > customers would actually want and have a use for. It showed that we did
                    > not understand their pains. The CTO, a really smart guy, was what Joel
                    > Spolsky calls an "architecture astronaut".[10] Not understanding your
                    > customers, what they want and need, what their pain is, is a sure way to
                    > die as a company. This was a contributor to the we had problems securing
                    > a second round of institutional venture financing, and ended up laying off a
                    >
                    > lot of people in early 2002.
                    >
                    > - More times than I can count, I've told some Windows "admin" how their
                    > OS works, or how to solve some application problem, because I understand
                    > the fundamental computer science, computer engineering and electrical
                    > engineering underlying that system. Windows admins -- and apparently
                    > more Linux people than I would think -- don't get how all of these things
                    > fit together. Once you know the fundamental concepts, they can be applied
                    > to any situation and system.
                    >
                    > - Google created the role of the "site reliability engineer (SRE)" which has
                    >
                    > been copied by many organizations.[11] (See above about Facebook and
                    > Blizzard.) The SRE's role specifically is to transcend the barriers between
                    >
                    > engineering (programmers) and operations (sysadmins) by helping the
                    > developers make their code work in the systems environment, whether
                    > that involves actual debugging, coding, scripting, or just working with the
                    > developers directly so they understand the operational environment and
                    > processes. They also interact with the operations side, assisting devs to
                    > optimize their software for deployment into the operational environment,
                    > then managing the deployment, monitoring, and maintaining the running
                    > systems. In the best operations environments and practices, the dev
                    > who wrote the broken code is an escalation point for resolving problems
                    > due to their bad code. That means that dev will be paged if their code
                    > breaks in production. The way it should be; you get a new respect for
                    > what others go through if you can't just throw your garbage code over the
                    > wall and let others clean up the mess. If you have to get out of bed and
                    > spend 2 hours of your time fixing the problem you created, you're going
                    > to be inclined to not create problems. Facebook, Flickr, Yahoo!, and many
                    > other online organizations are embracing the SRE role for this reason.
                    >
                    > - An sysadmin (ops guy) develops software (engineering) to perform ops
                    > better. The UNIX tradition at work.[12]
                    >
                    > - If there wasn't a need for practitioners to understand internals, none of
                    > the following would exist. Engineering/programmers can't exist idly in
                    > their
                    > own world w/o interacting with, and having accountability to, their
                    > customers. ACM Queue was founded for this reason.[13] Now, the list:
                    >
                    > http://www.amazon.com/exec/obidos/ASIN/0201750392/richteer-20
                    >
                    > (A review of the above by Ben Rockwood can be found at
                    > http://cuddletech.com/articles/ssp_review/)
                    >
                    > http://www.amazon.com/gp/product/0131482092?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131482092
                    >
                    > http://www.amazon.com/gp/product/0131568191?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131568191
                    >
                    > http://www.amazon.com/gp/product/0130952494?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0130952494
                    >
                    > http://www.amazon.com/gp/product/0131411551?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131411551
                    >
                    > (I own the above, but there are 2 more in the series. Stevens died FAR too
                    > soon, but his contributions are legendary.)
                    >
                    > http://www.amazon.com/gp/product/0131876716?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131876716
                    >
                    > (I own Volume 1 of the 3rd edition, but I've just added the 5th edition to
                    > my Amazon Wish List.)
                    >
                    > http://www.amazon.com/gp/product/0735625301?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0735625301
                    >
                    > - Briefly, I started a project with my current job to update monitoring. In
                    >
                    > exploring this, I worked very closely with the guy who is primarily
                    > responsible for writing the monitoring tests for things like CPU
                    > utilization.
                    > (He's a former Unix sysadmin, but now he focuses on developing monitoring.
                    > Lots of Perl and C code. Having been an admin allows him some appreciation
                    > for what impact bad monitors have on my group; too bad his management
                    > has not understood this.) Anyway, in order to reduce the number of bogus
                    > CPU monitoring alerts (e.g. Oracle backups running nightly, or NetBackup
                    > backups, or some custom process developed by the customer), I was required
                    > to understand what his monitoring tests were doing. That meant reading
                    > Perl code. The code uses internal Solaris kernel variables - poorly - too
                    > calculate values. Using this information, and the OpenSolaris code online
                    > and open for reading, I began tracking how the Solaris kernel measures
                    > CPU utilization. I still have to spend time with it, but I can quantify
                    > just
                    > how poorly his Perl code is working. Additionally, I may have found a bug
                    > in the Solaris code in how it calculates CPU utilization. This would be a
                    > 20+
                    > year old logic bug. More verification is required, but you're full of it if
                    > you
                    > think a sysadmin shouldn't know the internals of their OS.
                    >
                    > Computers are systems. Networks are systems. People and society are
                    > systems. All of these things have to come together harmoniously into a
                    > larger system in order to make effective use of the resources. Lacking an
                    > understanding of how all of the components work with each other, from
                    > the top to the bottom, is a recipe for suboptimal use and even failure. If
                    > you are going to call yourself a computer engineer, you will have to merge
                    > knowledge of software with knowledge of how that software interacts
                    > with the hardware it runs on and the networks (hardware and software)
                    > that it will traverse. That means understanding how the application uses
                    > the OS; how the OS uses the hardware; what happens in the hardware
                    > when the code is running on it; how the protocols fit into the picture and
                    > interact with the apps and OS; how the data fits on the wire, or over the
                    > airwaves; and whatever other components are involved.
                    >
                    > Anyway, I'm done. I encourage all to read the references, especially the
                    > ones I did not mention above. They add color to the discussion. I have
                    > been writing this e-mail for far too long as it is, and I don't want to
                    > start
                    > repeating myself. There is plenty of reason for sysadmins to understand
                    > how the code works. You can't fix (properly) what you don't understand.
                    > Read the joke "X Marks the Spot" at http://bit.ly/c8Af04 for a great
                    > example. Some of the references also talk about how software development
                    > methodologies, particularly Agile Development, can be applied for sysadmins
                    > and in operational environments. I include this for all of you programmers
                    > on this list.
                    >
                    >
                    > References:
                    >
                    > 1. http://www.agileweboperations.com/partitions-and-warfare/
                    >
                    > 2.
                    > http://www.youtube.com/watch?v=LdOe18KhtT4&feature=PlayList&p=99E1757A2C010F4B&index=23
                    >
                    > 3. http://www.kitchensoap.com/2009/06/23/slides-for-velocity-talk-2009/
                    >
                    > 4. http://lethargy.org/~jesus/about.html<http://lethargy.org/%7Ejesus/about.html>
                    >
                    > 5. http://cuddletech.com/tools/index.shtml
                    >
                    > 6. http://cuddletech.com/articles/index.shtml
                    >
                    > 7. http://cuddletech.com/blog/
                    >
                    > 8.
                    > http://daemons.net/~clay/2009/04/02/engineering-and-operations-bridging-the-divide/<http://daemons.net/%7Eclay/2009/04/02/engineering-and-operations-bridging-the-divide/>
                    >
                    > 9. http://www.taner.net/resume.html
                    >
                    > 10. http://www.joelonsoftware.com/articles/fog0000000018.html
                    >
                    > 11. http://research.google.com/archive/LinuxWorld-07-describeSRE.pdf
                    >
                    > 12.
                    > http://www.agileweboperations.com/configuration-management-remixed-introducing-carpet/
                    >
                    > 13. http://blogs.sun.com/bmc/entry/queue_cacm_and_the_rebirth
                    >
                    > 14. http://www.agileweboperations.com/devopsdays-2009/
                    >
                    > 15. http://www.kitchensoap.com/category/capacity-planning/
                    >
                    > 16. http://www.kitchensoap.com/category/webops/
                    >
                    > 17.
                    > http://www.kitchensoap.com/2009/01/14/mechanical-analogies-to-web-stuff-part-1/
                    >
                    > 18.
                    > http://www.kitchensoap.com/2009/05/06/mechanical-analogies-to-web-stuff-part-2/
                    >
                    > 19.
                    > http://radar.oreilly.com/2009/06/jonathan-heiliger-facebook-velocity-webops.html
                    >
                    > 20.
                    > http://www.agileweboperations.com/the-12-principles-behind-the-agile-manifesto-adapted-to-web-operations/
                    >
                    > 21.
                    > http://gigaom.com/2009/06/25/facebooks-jonathan-heiliger-talks-infrastructure-and-usernames/
                    >
                    > On Wed, Jan 27, 2010 at 21:54, Allan Samaroo <samaroo@...> wrote:
                    >
                    >>
                    >> I'm not sure I understand your last statement as you repeat the phrase
                    >> 'learn how to administer' and it conflicts with the previous statement.
                    >>
                    >> Why would an administrator, on any platform, ever want to know how the
                    >> internals work? That's the interest of a programmer.
                    >>
                    >> In my day as an administrator I needed only to know about using the
                    >> technology, not how it works. Even recompiling the kernel doesn't
                    >> require knowledge of the internals.....
                    >>
                    >>
                    >>> So the question one must ask themselves, and have clarity about, is what
                    >>> level of knowledge do they have and do they not have? If you're trying to
                    >>> learn how to administer Linux systems, Linux kernel internals is not the
                    >>> place you want to go. Internals is an advanced topic that comes later in
                    >>> the timeline of learning how to administer and manage systems.
                    >>
                    >>
                    >
                    >
                    >
                  • Allan Samaroo
                    ... At what point did I ever say this? Asking a question does not construe making a statement. Listen your experiences are different from mine and I
                    Message 9 of 20 , Jan 28, 2010
                    • 0 Attachment
                      > the hardware. But wait! These are software guys; they don't need to know
                      > anything about the hardware. Right, Allan?


                      At what point did I ever say this? Asking a question does not construe
                      making a statement.

                      Listen your experiences are different from mine and I appreciate that.
                      You don't know me and I don't know you yet you somehow feel the need to
                      (rather aggressively) educate me to your way of thinking. I don't know
                      what you believe you need to prove but you've certainly proven something.
                    • Khyron
                      True, you never said that. I was being sarcastic. ... -- You can choose your friends, you can choose the deals. - Equity Private Blog -
                      Message 10 of 20 , Jan 28, 2010
                      • 0 Attachment
                        True, you never said that. I was being sarcastic.

                        On Thu, Jan 28, 2010 at 09:40, Allan Samaroo <samaroo@...> wrote:

                        >
                        >
                        >
                        > > the hardware. But wait! These are software guys; they don't need to know
                        > > anything about the hardware. Right, Allan?
                        >
                        > At what point did I ever say this? Asking a question does not construe
                        > making a statement.
                        >
                        > Listen your experiences are different from mine and I appreciate that.
                        > You don't know me and I don't know you yet you somehow feel the need to
                        > (rather aggressively) educate me to your way of thinking. I don't know
                        > what you believe you need to prove but you've certainly proven something.
                        >
                        >
                        >



                        --
                        "You can choose your friends, you can choose the deals." - Equity Private

                        Blog - http://whatderass.blogspot.com/
                        Twitter - @khyron4eva


                        [Non-text portions of this message have been removed]
                      • Khyron
                        Hmmm. Interesting that you would feel attacked by my response. I think I addressed one remark at you specifically. As for the rest, I stated just how
                        Message 11 of 20 , Jan 28, 2010
                        • 0 Attachment
                          Hmmm. Interesting that you would feel attacked by my response. I think
                          I addressed one remark at you specifically. As for the rest, I stated just
                          how
                          incredulous I was at the statement you made, and offered modern examples
                          of how and why I thought it misguided. I pose the question to the other
                          members of this list, as to whether they think I intentionally attacked you
                          with this post. What poor behaviour would the community accuse me of?

                          Consider, Allan, that whatever made you feel attacked -- and apparently
                          makes you feel threatened by me -- is unreal. Had I meant to attack you,
                          I assure you I wouldn't bother doing it in a backhanded fashion. I don't
                          have time to waste on such antics; I fight dirty. So, no, Allan I was not
                          attacking you nor intending for your to feel attacked. I apologize if you
                          did.
                          I don't apologize for what I said, but I apologize that you would be so
                          blinded by something that happened in your past that your past feelings
                          were dug up as you read my e-mail.

                          I answered your question. I attempted to answer it thoroughly. I don't
                          answer question in a half-ass fashion. I like to be complete.

                          As for this not being a hardcore tech list, are you so sure about that?
                          Maybe it is not currently, but from my interactions with some people
                          privately, they seek a higher quality, more "hardcore" level of discourse
                          on this list. Let's be real -- this is the mailing list of a Linux user
                          group. If
                          this list can't be hardcore, then what list can? I think you're the only
                          person to actually *complain* that a thorough, technical answer was
                          provided on a technical mailing list about Linux. Think about it.

                          As for the roles of sysadmin and programmer (which is distinct from
                          developer, IMO), yes, they have evolved. In complexity. In levels of
                          abstraction, which intentionally keep "programmers" mindlessly coding
                          without respect for the implications and impact of their code. Developers,
                          by nature, take these things into account. (For more on leakiness of
                          abstractions and why programmers should know what is going on at the
                          hardware level -- thus, becoming developers -- I think it is best summed
                          up by http://bit.ly/97h5CH)

                          And you're right. I didn't address your earlier comments. So I'll do that
                          now so you don't feel so cheated.

                          I said:

                          "If you're trying to learn how to administer Linux systems, Linux kernel
                          internals is not the place you want to go. Internals is an advanced topic
                          that comes later in the timeline of learning how to administer and manage
                          systems."

                          So, first, I ask what exactly is unclear here? Also, what is conflicting in

                          these 2 sentences?

                          In the first sentence, I said clearly that if you are learning Linux system
                          administration, you don't want to start with Linux kernel internals. I then

                          clarified that statement in the next sentence, where I said that internals
                          is advanced subject matter. Someone who is still learning basic or
                          intermediate system administration would be diving into the deep end to
                          pick up Linux kernel internals. It's running before crawling. Kernel
                          internals can be difficult, requires reading lots of code, spending lots of
                          time modifying/building/running code, asking technical questions on
                          technical mailing lists and IRC, reading documentation and books, and
                          generally tinkering with parts of the system which are normally the
                          domain of developers. Do I think a sysadmin worth his salt can and has
                          worked in the domain of kernel internals? Yes. Do I think a beginning
                          sysadmin, just learning the basics of his craft, should be spending time
                          in Linux kernel internals? No. (I hope those were simple enough answers.)
                          A beginning/junior/intermediate level sysadmin should be learning the
                          fundamentals of their craft, the history of their field, and computer
                          science
                          basics to go with all of that. They should be focused on concepts and
                          how they work. Once they have mastered understanding of the concepts,
                          they would be well suited to examine how Linux implements those
                          concepts, where it differs from what's described in textbooks, and other
                          activities associated with studying "Linux kernel internals".

                          Is that not clear?

                          So what say you, community members? Have I overstepped my bounds?
                          I throw myself on your mercy, and accept your fair justice.

                          On Thu, Jan 28, 2010 at 09:21, Allan Samaroo <samaroo@...> wrote:

                          >
                          >
                          > Wow! What a response! Did you think I was trying to insult you? This
                          > list is to share knowledge and to assist. Your response is more of an
                          > attack and is a display of poor behaviour. If you can't answer a simple
                          > question with a simple response then don't. This is not a hardcore tech
                          > list.
                          >
                          > Clearly you do not realise that the roles of an administrator and a
                          > programmer have evolved and therefore the knowledge now required for
                          > each role is not the same. This happened for a reason. Sure one can be
                          > both but that choice is theirs not yours. Saying that it should be so
                          > because that's how it was historically is to me as absurd as you
                          > interpreted my question.
                          >
                          > I'm sure the material in your post is interesting and may be valuable to
                          > some so I thank you for that. I wish I had the time to read them.
                          >
                          > You still didn't explain your conflicting statements and you continue in
                          > a similar manner. I'm not sure you actually know what you're talking
                          > about. Perhaps you're confusing systems administrator with systems
                          > engineer? It's difficult to tell with the way you write.
                          >
                          >
                          > Khyron wrote:
                          > > I think this comment almost made my head explode. It's probably the most
                          > > absurd thing I've heard in 9 hours (since I heard some absurd stuff from
                          > a
                          > > few co-workers about 7 - 8.5 hours ago).
                          > >
                          > > Any administrator with an interest or need in doing anything more than
                          > the
                          > > most rudimentary work needs to know how his system works internally. It
                          > > is common to have to performance tune a system. It is common to have to
                          > > script or program a solution to a problem, such as when I once had to
                          > > debug an X Windows problem and the only way to do so was to write a
                          > > simple X Windows program in C to test the behavior of the X server. Any
                          > > admin I've ever known who did at least understand how their system worked
                          > > from top to bottom could not be any more than an intermediate level
                          > admin.
                          > > Period. I have former bosses, former colleagues, and friends who would
                          > > agree with that. (This reminds me of a former co-worker, now at Google,
                          > > who gave one of my favorite former bosses the best answer to the
                          > interview
                          > > question "explain the boot process of an X86 PC". The guy gave him the
                          > > CPU registers that got loaded at various points, and with what, among
                          > other
                          > > details! I love it.)
                          > >
                          > > I can't even begin to explain how infuriating "Why would an
                          > administrator,
                          > > on
                          > > any platform, ever want to know how the internals work? That's the
                          > interest
                          > >
                          > > of a programmer" is to me. My first thought is that whoever would make
                          > this
                          > >
                          > > statement is clearly not versed in the Unix tradition. The first
                          > > administrators
                          > > WERE programmers, because they were developing the system they used
                          > > and creating a system friendly to them. The BEST administrators now are
                          > > programmers, because to do the most interesting things with systems, you
                          > > have to be able to write code. (Scripting falls into this programming
                          > > domain
                          > > as well.) The BEST administrators are presented with problems which are
                          > > scarcely tractable without writing code on some level.
                          > >
                          > > The other piece to this is that most programmers don't know and don't
                          > care
                          > > about the impact of their code on the people who will use it. That's why
                          > > their code sucks. Such programmers also don't RESPECT what the sysadmins
                          > > who have to use their crappy software are dealing with. If those
                          > programmers
                          > >
                          > > have to interact with the sysadmins at some point (say, in a corporate IT
                          > > dept.
                          > > or say your local web startup), they will get an earful from those
                          > sysadmins
                          > >
                          > > that they threw under the bus with their garbage code. It is hard to gain
                          > > the
                          > > respect of the people running and managing your code if your code is
                          > crappy.
                          > >
                          > > This leads to the systems guys undermining the programmers, and leads to
                          > > the programmers hearing "no" more than they want. It also leads to things
                          > > such as the systems guys being overworked fixing broken code in
                          > production
                          > > and without support from their management and eventually leaving, or
                          > > sysadmins who "go off" on the developers. There are so many other things
                          > > which can and do happen when the developers and sysadmins don't speak
                          > > each other's language and don't care about how each other's worlds work.
                          > >
                          > > Sysadmins who write code or programmers who sysadmin are able to look at
                          > > such situations from the point of view of both. There may be a creative,
                          > > systems level solution which does not require code (which supports the
                          > goals
                          > >
                          > > of simplicity and maintainability). If there is a need to write code, not
                          > > only
                          > > do they have the ability to do so, but they can address the problem with
                          > the
                          > >
                          > > least amount of code, building the systems components around the code as
                          > > required. Finally, these people are able to bridge the divide better
                          > > between
                          > > operations (sysadmins) and engineering (programmers) who typically DON'T
                          > > understand what each other does and how they work, which leads to all
                          > > types of problems which have been documented by people way smarter than
                          > > me.[1] I'll link to some of the thoughts on the problems of programmers
                          > and
                          > >
                          > > sysadmins who don't grok each other's worlds. The Flickr guys did a great
                          > > presentation on this at Velocity 2009.[2][3]
                          > >
                          > > A few examples:
                          > >
                          > > - One of the best systems guys I know owns a company which consults with
                          > > many well known (and lesser known) companies. 2 of his clients contracted
                          > > him to design an image retrieval system which operated in large scale in
                          > > real
                          > > time. Think Facebook Images, Flickr or SmugMug. Only, he had to deliver
                          > > this system at the lowest cost. He wrote about 1700 lines of Perl and
                          > close
                          > >
                          > > to 800 lines of C to build this system, which can run on Linux and
                          > Solaris.
                          > > (The demo is deployed on OpenSolaris.) I guarantee that Facebook Images
                          > > and Flickr took a lot longer to build, and required more people to build
                          > > them.
                          > > Now, this is essentially a storage system, but so is FB Images.[1] Since
                          > > you
                          > > will likely run into his name if you read this entire post and the
                          > > references I
                          > > provide, I'll go ahead and say it. He's Theo Schlossnagle of OmniTI[4]
                          > and
                          > > I've sent out stuff by Theo before about ZFS and (Open)Solaris.
                          > >
                          > > - Ben Rockwood, a contributor to the Enlightenment project, is the
                          > Director
                          > > of Systems at Joyent. While he blogs and presents about systems admin
                          > > topics, he has submitted code to the OpenSolaris project as well as the
                          > > Enlightenment project.[5][6][7]
                          > >
                          > > - An acquaintance of mine who previously worked at Ning and most recently
                          > > at Twitter. His background is hardcore computer science, but at Earthlink
                          > > and Ning, his duties were a mix of systems admin/engineering and software
                          > > engineering. At Ning in particular, he both performed sysadmin and
                          > software
                          > >
                          > > engineering duties, and performed on-call duties supporting in his
                          > software
                          > > when it broke. Much of his work involved performance, and making the
                          > > Java backend of Ning perform optimally, diagnosing and troubleshooting
                          > > code problems in the systems environment. Usually, there are entire types
                          > > of problems which only appear in production environments, and with web
                          > > companies, the level of traffic and demand placed upon them commonly
                          > > cannot be reproduced under test or QA environments. This is why Facebook
                          > > performs what they call "dark launches". I've linked to his thoughts on
                          > the
                          > > divide between operations and engineering; the best way to bridge that
                          > > divide is for people on both sides to transcend it.[8]
                          > >
                          > > - Small companies (a la web startups or capital efficient early stage
                          > tech
                          > > companies) do not have the resources to invest in both sysadmins AND
                          > > programmers. So you take the time and care to invest in someone who
                          > > either can already do these things, or has the ability and desire to
                          > learn.
                          > > Employee #4 at Facebook, the first non-founder employee, is such a
                          > person.
                          > > (I happen to know him, as well as the VP of Operations.) Apparently he's
                          > > the lead reliability engineer @ Blizzard's battle.net now.[9]
                          > >
                          > > - When I worked for a storage startup, I dealt with the capital
                          > efficiency
                          > > issue. Though I am not a programmer, I spent a lot of time hacking on the
                          > > Linux kernel (yes, even adding some code) to do things which the kernel
                          > > developers in their finite wisdom had not bothered to add. All of this
                          > was
                          > > because I needed to know what the kernel was doing in the deployment
                          > > scenario envisioned by the guys building the company's product. They
                          > > weren't sysadmins either, and it showed because the product didn't work.
                          > > It didn't work (generally) but even more importantly, it didn't work the
                          > way
                          > >
                          > > most people wanted to use it.
                          > >
                          > > (Hint: The US government will not buy a backup system that can't delete
                          > > because their higher security data deletion standards, and not just those
                          > > developed by the national security infrastructure [CIA, NSA, DISA, etc.],
                          > > require you to be able to confirm that the data is gone. The backup
                          > > product my company was developing had the property that it could keep
                          > > data forever. The government wasn't interested. That's a HUGE customer
                          > > that spends over $1,000,000,000,000 US annually.)
                          > >
                          > > Anyway, this was a problem because we weren't building a product that our
                          > > customers would actually want and have a use for. It showed that we did
                          > > not understand their pains. The CTO, a really smart guy, was what Joel
                          > > Spolsky calls an "architecture astronaut".[10] Not understanding your
                          > > customers, what they want and need, what their pain is, is a sure way to
                          > > die as a company. This was a contributor to the we had problems securing
                          > > a second round of institutional venture financing, and ended up laying
                          > off a
                          > >
                          > > lot of people in early 2002.
                          > >
                          > > - More times than I can count, I've told some Windows "admin" how their
                          > > OS works, or how to solve some application problem, because I understand
                          > > the fundamental computer science, computer engineering and electrical
                          > > engineering underlying that system. Windows admins -- and apparently
                          > > more Linux people than I would think -- don't get how all of these things
                          > > fit together. Once you know the fundamental concepts, they can be applied
                          > > to any situation and system.
                          > >
                          > > - Google created the role of the "site reliability engineer (SRE)" which
                          > has
                          > >
                          > > been copied by many organizations.[11] (See above about Facebook and
                          > > Blizzard.) The SRE's role specifically is to transcend the barriers
                          > between
                          > >
                          > > engineering (programmers) and operations (sysadmins) by helping the
                          > > developers make their code work in the systems environment, whether
                          > > that involves actual debugging, coding, scripting, or just working with
                          > the
                          > > developers directly so they understand the operational environment and
                          > > processes. They also interact with the operations side, assisting devs to
                          > > optimize their software for deployment into the operational environment,
                          > > then managing the deployment, monitoring, and maintaining the running
                          > > systems. In the best operations environments and practices, the dev
                          > > who wrote the broken code is an escalation point for resolving problems
                          > > due to their bad code. That means that dev will be paged if their code
                          > > breaks in production. The way it should be; you get a new respect for
                          > > what others go through if you can't just throw your garbage code over the
                          > > wall and let others clean up the mess. If you have to get out of bed and
                          > > spend 2 hours of your time fixing the problem you created, you're going
                          > > to be inclined to not create problems. Facebook, Flickr, Yahoo!, and many
                          > > other online organizations are embracing the SRE role for this reason.
                          > >
                          > > - An sysadmin (ops guy) develops software (engineering) to perform ops
                          > > better. The UNIX tradition at work.[12]
                          > >
                          > > - If there wasn't a need for practitioners to understand internals, none
                          > of
                          > > the following would exist. Engineering/programmers can't exist idly in
                          > > their
                          > > own world w/o interacting with, and having accountability to, their
                          > > customers. ACM Queue was founded for this reason.[13] Now, the list:
                          > >
                          > > http://www.amazon.com/exec/obidos/ASIN/0201750392/richteer-20
                          > >
                          > > (A review of the above by Ben Rockwood can be found at
                          > > http://cuddletech.com/articles/ssp_review/)
                          > >
                          > >
                          > http://www.amazon.com/gp/product/0131482092?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131482092
                          > >
                          > >
                          > http://www.amazon.com/gp/product/0131568191?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131568191
                          > >
                          > >
                          > http://www.amazon.com/gp/product/0130952494?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0130952494
                          > >
                          > >
                          > http://www.amazon.com/gp/product/0131411551?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131411551
                          > >
                          > > (I own the above, but there are 2 more in the series. Stevens died FAR
                          > too
                          > > soon, but his contributions are legendary.)
                          > >
                          > >
                          > http://www.amazon.com/gp/product/0131876716?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0131876716
                          > >
                          > > (I own Volume 1 of the 3rd edition, but I've just added the 5th edition
                          > to
                          > > my Amazon Wish List.)
                          > >
                          > >
                          > http://www.amazon.com/gp/product/0735625301?ie=UTF8&tag=alph01-20&linkCode=as2&camp=1789&creative=9325&creativeASIN=0735625301
                          > >
                          > > - Briefly, I started a project with my current job to update monitoring.
                          > In
                          > >
                          > > exploring this, I worked very closely with the guy who is primarily
                          > > responsible for writing the monitoring tests for things like CPU
                          > > utilization.
                          > > (He's a former Unix sysadmin, but now he focuses on developing
                          > monitoring.
                          > > Lots of Perl and C code. Having been an admin allows him some
                          > appreciation
                          > > for what impact bad monitors have on my group; too bad his management
                          > > has not understood this.) Anyway, in order to reduce the number of bogus
                          > > CPU monitoring alerts (e.g. Oracle backups running nightly, or NetBackup
                          > > backups, or some custom process developed by the customer), I was
                          > required
                          > > to understand what his monitoring tests were doing. That meant reading
                          > > Perl code. The code uses internal Solaris kernel variables - poorly - too
                          > > calculate values. Using this information, and the OpenSolaris code online
                          > > and open for reading, I began tracking how the Solaris kernel measures
                          > > CPU utilization. I still have to spend time with it, but I can quantify
                          > > just
                          > > how poorly his Perl code is working. Additionally, I may have found a bug
                          > > in the Solaris code in how it calculates CPU utilization. This would be a
                          > > 20+
                          > > year old logic bug. More verification is required, but you're full of it
                          > if
                          > > you
                          > > think a sysadmin shouldn't know the internals of their OS.
                          > >
                          > > Computers are systems. Networks are systems. People and society are
                          > > systems. All of these things have to come together harmoniously into a
                          > > larger system in order to make effective use of the resources. Lacking an
                          > > understanding of how all of the components work with each other, from
                          > > the top to the bottom, is a recipe for suboptimal use and even failure.
                          > If
                          > > you are going to call yourself a computer engineer, you will have to
                          > merge
                          > > knowledge of software with knowledge of how that software interacts
                          > > with the hardware it runs on and the networks (hardware and software)
                          > > that it will traverse. That means understanding how the application uses
                          > > the OS; how the OS uses the hardware; what happens in the hardware
                          > > when the code is running on it; how the protocols fit into the picture
                          > and
                          > > interact with the apps and OS; how the data fits on the wire, or over the
                          > > airwaves; and whatever other components are involved.
                          > >
                          > > Anyway, I'm done. I encourage all to read the references, especially the
                          > > ones I did not mention above. They add color to the discussion. I have
                          > > been writing this e-mail for far too long as it is, and I don't want to
                          > > start
                          > > repeating myself. There is plenty of reason for sysadmins to understand
                          > > how the code works. You can't fix (properly) what you don't understand.
                          > > Read the joke "X Marks the Spot" at http://bit.ly/c8Af04 for a great
                          > > example. Some of the references also talk about how software development
                          > > methodologies, particularly Agile Development, can be applied for
                          > sysadmins
                          > > and in operational environments. I include this for all of you
                          > programmers
                          > > on this list.
                          > >
                          > >
                          > > References:
                          > >
                          > > 1. http://www.agileweboperations.com/partitions-and-warfare/
                          > >
                          > > 2.
                          > >
                          > http://www.youtube.com/watch?v=LdOe18KhtT4&feature=PlayList&p=99E1757A2C010F4B&index=23
                          > >
                          > > 3. http://www.kitchensoap.com/2009/06/23/slides-for-velocity-talk-2009/
                          > >
                          > > 4. http://lethargy.org/~jesus/about.html<http://lethargy.org/%7Ejesus/about.html>
                          > <http://lethargy.org/%7Ejesus/about.html>
                          >
                          > >
                          > > 5. http://cuddletech.com/tools/index.shtml
                          > >
                          > > 6. http://cuddletech.com/articles/index.shtml
                          > >
                          > > 7. http://cuddletech.com/blog/
                          > >
                          > > 8.
                          > >
                          > http://daemons.net/~clay/2009/04/02/engineering-and-operations-bridging-the-divide/<http://daemons.net/%7Eclay/2009/04/02/engineering-and-operations-bridging-the-divide/>
                          > <
                          > http://daemons.net/%7Eclay/2009/04/02/engineering-and-operations-bridging-the-divide/
                          > >
                          >
                          > >
                          > > 9. http://www.taner.net/resume.html
                          > >
                          > > 10. http://www.joelonsoftware.com/articles/fog0000000018.html
                          > >
                          > > 11. http://research.google.com/archive/LinuxWorld-07-describeSRE.pdf
                          > >
                          > > 12.
                          > >
                          > http://www.agileweboperations.com/configuration-management-remixed-introducing-carpet/
                          > >
                          > > 13. http://blogs.sun.com/bmc/entry/queue_cacm_and_the_rebirth
                          > >
                          > > 14. http://www.agileweboperations.com/devopsdays-2009/
                          > >
                          > > 15. http://www.kitchensoap.com/category/capacity-planning/
                          > >
                          > > 16. http://www.kitchensoap.com/category/webops/
                          > >
                          > > 17.
                          > >
                          > http://www.kitchensoap.com/2009/01/14/mechanical-analogies-to-web-stuff-part-1/
                          > >
                          > > 18.
                          > >
                          > http://www.kitchensoap.com/2009/05/06/mechanical-analogies-to-web-stuff-part-2/
                          > >
                          > > 19.
                          > >
                          > http://radar.oreilly.com/2009/06/jonathan-heiliger-facebook-velocity-webops.html
                          > >
                          > > 20.
                          > >
                          > http://www.agileweboperations.com/the-12-principles-behind-the-agile-manifesto-adapted-to-web-operations/
                          > >
                          > > 21.
                          > >
                          > http://gigaom.com/2009/06/25/facebooks-jonathan-heiliger-talks-infrastructure-and-usernames/
                          > >
                          > > On Wed, Jan 27, 2010 at 21:54, Allan Samaroo <samaroo@...<samaroo%40gmx.net>>
                          > wrote:
                          > >
                          > >>
                          > >> I'm not sure I understand your last statement as you repeat the phrase
                          > >> 'learn how to administer' and it conflicts with the previous statement.
                          > >>
                          > >> Why would an administrator, on any platform, ever want to know how the
                          > >> internals work? That's the interest of a programmer.
                          > >>
                          > >> In my day as an administrator I needed only to know about using the
                          > >> technology, not how it works. Even recompiling the kernel doesn't
                          > >> require knowledge of the internals.....
                          > >>
                          > >>
                          > >>> So the question one must ask themselves, and have clarity about, is
                          > what
                          > >>> level of knowledge do they have and do they not have? If you're trying
                          > to
                          > >>> learn how to administer Linux systems, Linux kernel internals is not
                          > the
                          > >>> place you want to go. Internals is an advanced topic that comes later
                          > in
                          > >>> the timeline of learning how to administer and manage systems.
                          > >>
                          > >>
                          > >
                          > >
                          > >
                          >
                          >



                          --
                          "You can choose your friends, you can choose the deals." - Equity Private

                          Blog - http://whatderass.blogspot.com/
                          Twitter - @khyron4eva


                          [Non-text portions of this message have been removed]
                        • Allan Samaroo
                          ???? Khyron I do not feel attacked nor threatened by you. Your initial response had a distinct tone. The length of the response also didn t help. ... This
                          Message 12 of 20 , Jan 28, 2010
                          • 0 Attachment
                            ???? Khyron I do not feel attacked nor threatened by you. Your initial
                            response had a distinct tone. The length of the response also didn't help.


                            > I don't apologize for what I said, but I apologize that you would be so
                            > blinded by something that happened in your past that your past feelings
                            > were dug up as you read my e-mail.

                            This is out of line.

                            Say what you will in response but I am no longer going to be replying to
                            this thread.
                          • Stephen Sankarsingh
                            Do I think a beginning sysadmin, just learning the basics of his craft, should be spending time in Linux kernel internals? No. As a sysadmin who didn t
                            Message 13 of 20 , Jan 28, 2010
                            • 0 Attachment
                              "Do I think a beginning sysadmin, just learning the basics of his craft,
                              should be spending time in Linux kernel internals? No. "

                              As a sysadmin who didn't benefit from any formal training in computer
                              science, I think that over time I have learnt many very complex things along
                              with some of the basics.

                              I am sure there are lots of basic things that I do not know, simply because
                              I never needed to use them. Khyron pointed out in a previous thread that I
                              was using 'cat' and 'grep' in the wrong way. He didn't follow-up by saying
                              how I should use them so I still have no idea how to do so or even what to
                              use in-place of them. I do know that the way I currently use them usually
                              works for me!

                              Though nothing stops me from learning things as I go along, my sieve-like
                              brain (wife's description, not mine) usually forgets them again until the
                              next time I need to know how to use them. I have to agree that if I learnt
                              Linux in a structured way instead of in the haphazard way in which it
                              presented itself to me that I may have been a more knowledgeable sysadmin,
                              then again I may also have become bored in the process.

                              To answer the question, I have to admit that if over time I didn't learn
                              something about the internals of the operating system (i.e. the black box)
                              that it would be virtually impossible for me to troubleshoot wtf's going on
                              with an application.

                              Also, from the relatively few opportunities I've had to view other people
                              using the Linux CLI, it is apparent that they use it in vastly different
                              ways.

                              /Stephen


                              [Non-text portions of this message have been removed]
                            • Khyron
                              For those who have still followed along, thanks for listening. I also notice that no one besides Allan called me out for offending them or disrespecting the
                              Message 14 of 20 , Jan 29, 2010
                              • 0 Attachment
                                For those who have still followed along, thanks for listening. I also
                                notice
                                that no one besides Allan called me out for offending them or disrespecting
                                the community. Complaints still welcome. If I've done something wrong by
                                the community's standards, then I will make amends if I can. I'm not a
                                complete asshole, no matter what certain people may think, but I am very
                                pedantic.

                                One last thought with this thread.

                                http://www.facebook.com/video/video.php?v=208561675468&ref=mf

                                Within this presentation, besides there being lots of cool engineering talk
                                which I love, there is discussion of new ways of doing business due to the
                                server (hardware & systems) guys and datacenter guys at Facebook and
                                how closely they work. I point this out because again, it shows that when
                                professionals in one discipline have an understanding of how professionals
                                in complimentary disciplines work, the two disciplines can collaborate to
                                solve problems that would be difficult or impossible for either group to
                                solve
                                alone. If those 2 disciplines occur in 1 physical person, even better (in
                                some
                                cases).

                                This is also a good example of why it makes sense for people to have a
                                holistic view of their field. "Server guys" who know what is happening
                                electrically and mechanically within the servers can address the scale
                                issues
                                that these large server deployments have, in terms of power, cooling and
                                cost. "Systems guys" who understand what the electrical, mechanical, power
                                and cost issues of servers are can focus on optimizing the operating systems

                                and applications to increase the overall efficiency of the servers. The
                                software guys can also optimize their code to increase the efficiency of the

                                servers, and also to help the server and datacenter guys deploy the systems
                                in the most effective ways (e.g. server sizing, power/cooling requirements,
                                etc.).

                                Stephen, to address your points...

                                Learning informally is the history of this industry. Junior sysadmins have
                                traditionally been "apprenticed" to senior sysadmins, and developed that
                                way, and eventually becoming the mentors. There are few formal programs
                                in system administration. (Although, they are coming online. The first
                                Ph.D.
                                in sysadmin was granted a few years ago. That guy is a professor at the
                                university from which he received his doctorate.) However, I think it is
                                possible, even within an informal structure, to bring into use formal
                                education and knowledge. I benefitted because I was studying in an EE
                                program, surrounded by other EE students and professors as well as other
                                disciplines (computer science, mechanical engineering), learning the basics
                                -
                                the fundamentals - as I apprenticed to more senior sysadmins (who were
                                usually students too). It is definitely not the only way to go, but it
                                helped
                                because again, those fundamentals are always in play, whether you know
                                and understand them or not. The laws of physics still exist even if you
                                don't know how to to apply them or how they are manipulated or impacted
                                by the things you do at the macro level.

                                (Interestingly, ALL of the guys who trained me were West Indians, and many
                                were Trinidadians. They're still friends and colleagues to this day.)

                                I believe in hacking learning. It's like "The Matrix", right? "There are
                                rules.
                                Some of these rules can be bent. Some of them can be broken." Of course,
                                before you can bend or break the rules, you have to KNOW the rules, when
                                they are intended to be applied, and the reasons for those rules existence.
                                Trying to bend/break rules before having that understanding can be
                                dangerous.

                                Stephen, as for using "cat" and "grep", that's a convo we can have offline
                                sometime, if you'd like. I think I hinted at it but I did not come out and
                                say
                                specifically WHY you want to do things differently with those tools. Also,
                                as
                                for your "sieve-like brain", I always tell my students to keep and refer to
                                a
                                notebook. If you have a good notebook, you can record those things which
                                you don't use all the time and reference them later. Over years, you have
                                a body of knowledge accumulated. Review the notebook every so often and
                                voila! It is far more important to know HOW to learn vs. knowing answers.
                                I know, when I have hired in the past, that is the key trait I have looked
                                for,
                                is does the person know how to learn, how to figure things out, in a
                                structured manner. Having answers is not necessarily a plus if you don't
                                know how to learn what you don't know in a crunch.

                                Anyway, I hope someone learned something or found some use in all of that.
                                As I said to Allan, I believe in being thorough and complete. If anyone has
                                a
                                problem with that, let me know and I'll refrain from doing so in the future.

                                On Thu, Jan 28, 2010 at 11:49, Stephen Sankarsingh <stephentnt@...>wrote:

                                >
                                >
                                > "Do I think a beginning sysadmin, just learning the basics of his craft,
                                > should be spending time in Linux kernel internals? No. "
                                >
                                > As a sysadmin who didn't benefit from any formal training in computer
                                > science, I think that over time I have learnt many very complex things
                                > along
                                > with some of the basics.
                                >
                                > I am sure there are lots of basic things that I do not know, simply because
                                > I never needed to use them. Khyron pointed out in a previous thread that I
                                > was using 'cat' and 'grep' in the wrong way. He didn't follow-up by saying
                                > how I should use them so I still have no idea how to do so or even what to
                                > use in-place of them. I do know that the way I currently use them usually
                                > works for me!
                                >
                                > Though nothing stops me from learning things as I go along, my sieve-like
                                > brain (wife's description, not mine) usually forgets them again until the
                                > next time I need to know how to use them. I have to agree that if I learnt
                                > Linux in a structured way instead of in the haphazard way in which it
                                > presented itself to me that I may have been a more knowledgeable sysadmin,
                                > then again I may also have become bored in the process.
                                >
                                > To answer the question, I have to admit that if over time I didn't learn
                                > something about the internals of the operating system (i.e. the black box)
                                > that it would be virtually impossible for me to troubleshoot wtf's going on
                                > with an application.
                                >
                                > Also, from the relatively few opportunities I've had to view other people
                                > using the Linux CLI, it is apparent that they use it in vastly different
                                > ways.
                                >
                                > /Stephen
                                >
                                >
                                > [Non-text portions of this message have been removed]
                                >
                                >
                                >



                                --
                                "You can choose your friends, you can choose the deals." - Equity Private

                                Blog - http://whatderass.blogspot.com/
                                Twitter - @khyron4eva


                                [Non-text portions of this message have been removed]
                              • Falina Baksh
                                Hi All, Please be reminded of TTLUG s mailing list policy . Khyron your post was most interesting and very
                                Message 15 of 20 , Jan 29, 2010
                                • 0 Attachment
                                  Hi All,

                                  Please be reminded of TTLUG's mailing list policy
                                  <http://ttlug.pbworks.com/Mailing-List-Policy>.

                                  Khyron your post was most interesting and very thorough. It did seem to me
                                  however, while reading your post that you did "throw some words" out at
                                  Allan. It may not have been your intention but it did come across in that
                                  fashion.

                                  It was great that you apologized for any attack Allan may have felt however
                                  you followed your apology with the statements below. This was much to
                                  personal.

                                  I don't apologize for what I said, but I apologize that you would be so
                                  > blinded by something that happened in your past that your past feelings
                                  > were dug up as you read my e-mail.
                                  >

                                  Also - everyone, lets try to stay away from the coarse language when posting
                                  to the list.

                                  Back to the topic:

                                  I've been one who had to learn on my own - and there are definitely gaps in
                                  my knowledge. There are always multiple methods to doing anything. Most
                                  methods always seem to get the task at hand done but using the wrong way
                                  always leaves room for problems in the future.

                                  Without an understanding of the entire environment you're administrating,
                                  you may not be fully aware of the implications of using one method vs the
                                  other. This means that as Administrator's we always have to be on-top of our
                                  game.

                                  I don't think that we need a thorough understanding of every single
                                  component of the system (though that would be ideal) but we need to have at
                                  least a basic understanding of what role each component plays and how they
                                  interact with each other. This way when your Dba and your application
                                  programmers come to you with a task you have the capacity to pose educated
                                  questions / challenges about the implications their task will have on your
                                  environment.

                                  As a systems administrator you pretty much "own" the entire system.
                                  Regardless of who else does what, at the end of the day the ball is usually
                                  in your court to find / resolve the issue.

                                  Just my two pence.

                                  Fali


                                  On Fri, Jan 29, 2010 at 12:10 PM, Khyron <khyron4eva@...> wrote:

                                  > For those who have still followed along, thanks for listening. I also
                                  > notice
                                  > that no one besides Allan called me out for offending them or disrespecting
                                  > the community. Complaints still welcome. If I've done something wrong by
                                  > the community's standards, then I will make amends if I can. I'm not a
                                  > complete asshole, no matter what certain people may think, but I am very
                                  > pedantic.
                                  >
                                  > One last thought with this thread.
                                  >
                                  > http://www.facebook.com/video/video.php?v=208561675468&ref=mf
                                  >
                                  > Within this presentation, besides there being lots of cool engineering talk
                                  > which I love, there is discussion of new ways of doing business due to the
                                  > server (hardware & systems) guys and datacenter guys at Facebook and
                                  > how closely they work. I point this out because again, it shows that when
                                  > professionals in one discipline have an understanding of how professionals
                                  > in complimentary disciplines work, the two disciplines can collaborate to
                                  > solve problems that would be difficult or impossible for either group to
                                  > solve
                                  > alone. If those 2 disciplines occur in 1 physical person, even better (in
                                  > some
                                  > cases).
                                  >
                                  > This is also a good example of why it makes sense for people to have a
                                  > holistic view of their field. "Server guys" who know what is happening
                                  > electrically and mechanically within the servers can address the scale
                                  > issues
                                  > that these large server deployments have, in terms of power, cooling and
                                  > cost. "Systems guys" who understand what the electrical, mechanical, power
                                  > and cost issues of servers are can focus on optimizing the operating
                                  > systems
                                  >
                                  > and applications to increase the overall efficiency of the servers. The
                                  > software guys can also optimize their code to increase the efficiency of
                                  > the
                                  >
                                  > servers, and also to help the server and datacenter guys deploy the systems
                                  > in the most effective ways (e.g. server sizing, power/cooling requirements,
                                  > etc.).
                                  >
                                  > Stephen, to address your points...
                                  >
                                  > Learning informally is the history of this industry. Junior sysadmins have
                                  > traditionally been "apprenticed" to senior sysadmins, and developed that
                                  > way, and eventually becoming the mentors. There are few formal programs
                                  > in system administration. (Although, they are coming online. The first
                                  > Ph.D.
                                  > in sysadmin was granted a few years ago. That guy is a professor at the
                                  > university from which he received his doctorate.) However, I think it is
                                  > possible, even within an informal structure, to bring into use formal
                                  > education and knowledge. I benefitted because I was studying in an EE
                                  > program, surrounded by other EE students and professors as well as other
                                  > disciplines (computer science, mechanical engineering), learning the basics
                                  > -
                                  > the fundamentals - as I apprenticed to more senior sysadmins (who were
                                  > usually students too). It is definitely not the only way to go, but it
                                  > helped
                                  > because again, those fundamentals are always in play, whether you know
                                  > and understand them or not. The laws of physics still exist even if you
                                  > don't know how to to apply them or how they are manipulated or impacted
                                  > by the things you do at the macro level.
                                  >
                                  > (Interestingly, ALL of the guys who trained me were West Indians, and many
                                  > were Trinidadians. They're still friends and colleagues to this day.)
                                  >
                                  > I believe in hacking learning. It's like "The Matrix", right? "There are
                                  > rules.
                                  > Some of these rules can be bent. Some of them can be broken." Of course,
                                  > before you can bend or break the rules, you have to KNOW the rules, when
                                  > they are intended to be applied, and the reasons for those rules existence.
                                  > Trying to bend/break rules before having that understanding can be
                                  > dangerous.
                                  >
                                  > Stephen, as for using "cat" and "grep", that's a convo we can have offline
                                  > sometime, if you'd like. I think I hinted at it but I did not come out and
                                  > say
                                  > specifically WHY you want to do things differently with those tools. Also,
                                  > as
                                  > for your "sieve-like brain", I always tell my students to keep and refer to
                                  > a
                                  > notebook. If you have a good notebook, you can record those things which
                                  > you don't use all the time and reference them later. Over years, you have
                                  > a body of knowledge accumulated. Review the notebook every so often and
                                  > voila! It is far more important to know HOW to learn vs. knowing answers.
                                  > I know, when I have hired in the past, that is the key trait I have looked
                                  > for,
                                  > is does the person know how to learn, how to figure things out, in a
                                  > structured manner. Having answers is not necessarily a plus if you don't
                                  > know how to learn what you don't know in a crunch.
                                  >
                                  > Anyway, I hope someone learned something or found some use in all of that.
                                  > As I said to Allan, I believe in being thorough and complete. If anyone
                                  > has
                                  > a
                                  > problem with that, let me know and I'll refrain from doing so in the
                                  > future.
                                  >
                                  > On Thu, Jan 28, 2010 at 11:49, Stephen Sankarsingh <stephentnt@...
                                  > >wrote:
                                  >
                                  > >
                                  > >
                                  > > "Do I think a beginning sysadmin, just learning the basics of his craft,
                                  > > should be spending time in Linux kernel internals? No. "
                                  > >
                                  > > As a sysadmin who didn't benefit from any formal training in computer
                                  > > science, I think that over time I have learnt many very complex things
                                  > > along
                                  > > with some of the basics.
                                  > >
                                  > > I am sure there are lots of basic things that I do not know, simply
                                  > because
                                  > > I never needed to use them. Khyron pointed out in a previous thread that
                                  > I
                                  > > was using 'cat' and 'grep' in the wrong way. He didn't follow-up by
                                  > saying
                                  > > how I should use them so I still have no idea how to do so or even what
                                  > to
                                  > > use in-place of them. I do know that the way I currently use them usually
                                  > > works for me!
                                  > >
                                  > > Though nothing stops me from learning things as I go along, my sieve-like
                                  > > brain (wife's description, not mine) usually forgets them again until the
                                  > > next time I need to know how to use them. I have to agree that if I
                                  > learnt
                                  > > Linux in a structured way instead of in the haphazard way in which it
                                  > > presented itself to me that I may have been a more knowledgeable
                                  > sysadmin,
                                  > > then again I may also have become bored in the process.
                                  > >
                                  > > To answer the question, I have to admit that if over time I didn't learn
                                  > > something about the internals of the operating system (i.e. the black
                                  > box)
                                  > > that it would be virtually impossible for me to troubleshoot wtf's going
                                  > on
                                  > > with an application.
                                  > >
                                  > > Also, from the relatively few opportunities I've had to view other people
                                  > > using the Linux CLI, it is apparent that they use it in vastly different
                                  > > ways.
                                  > >
                                  > > /Stephen
                                  > >
                                  > >
                                  > > [Non-text portions of this message have been removed]
                                  > >
                                  > >
                                  > >
                                  >
                                  >
                                  >
                                  > --
                                  > "You can choose your friends, you can choose the deals." - Equity Private
                                  >
                                  > Blog - http://whatderass.blogspot.com/
                                  > Twitter - @khyron4eva
                                  >
                                  >
                                  > [Non-text portions of this message have been removed]
                                  >
                                  >
                                  >
                                  > ------------------------------------
                                  >
                                  > Help build TTLUG by forwarding this to anyone who is interested in the
                                  > subject matter or would otherwise benefit from joining the mailing list.
                                  >
                                  > Trinidad and Tobago Linux Users Group http://groups.yahoo.com/group/ttlug
                                  > To subscribe, send an email to_______ TTLUG-subscribe@yahoogroups.com
                                  > To unsubscribe, send an email to_____ TTLUG-unsubscribe@yahoogroups.com
                                  > List owner/moderator Richard Jobity__ TTLUG-owner@yahoogroups.com
                                  > Yahoo! Groups Links
                                  >
                                  >
                                  >
                                  >


                                  --
                                  Rgds,
                                  Falina


                                  [Non-text portions of this message have been removed]
                                Your message has been successfully submitted and would be delivered to recipients shortly.