Loading ...
Sorry, an error occurred while loading the content.
 

Re: [agileDatabases] Re: [AM] The DAMA crowd misses the point yet again

Expand Messages
  • Scott Ambler
    ... Perhaps. Then again, you might want to read the archives of the dm-discuss@yahoogroups.com list. That community has a history of misrepresenting agile,
    Message 1 of 10 , May 5, 2007
      --- Clifford Heath <clifford.heath@...> wrote:

      >
      > You read it very differently that I did. When he
      > says
      > "Scott says.... the implication is....", and I read
      > it as
      > "Scott says.... and so people do....". He's
      > definitely
      > getting stuck in, but not on the basis of what you
      > say... rather on the basis of what people do. I

      Perhaps. Then again, you might want to read the
      archives of the dm-discuss@yahoogroups.com list. That
      community has a history of misrepresenting agile,
      misrepresenting my work, and personally attacking me.
      The quality of the interaction on that list is
      horrendous and I think you'd be hard pressed to find
      an IT mailing list where the behavior is worse.

      So, when I see the sort of material that Dan has
      published coming from someone in the DM community I am
      no longer willing to give them the benefit of the
      doubt. That community burned any good will that it
      has with me long ago.


      > can't
      > reason why he blames you, but my only surmisal is
      > that he picked you as a representative of Agile in
      > the whole, and he's seen lazy incompetents (such as

      In a private email Dan indicated that he wasn't
      willing to get involved with either the AM or the AD
      lists, so it's difficult to have a discussion about
      what he was thinking.

      > populate a lot of the software development world)
      > justifying their predilections by calling themselves
      > Agile.

      Yes. And because he doesn't seem to be willing to
      interact with the actual Agile community he'll
      continue to struggle to understand the actual message
      and be able to distiguish between the agilists and the
      code&fixers. Worse yet, he's only reinforced these
      obvious misunderstandings in a community which very
      clearly needs help.

      > IOW, he blames you for excuses made by other
      > people about their inadequacy, because he thinks
      > that Agile has legitimised those excuses. And to

      If someone is going to get up in a public forum and
      discuss Agile, or discuss anything for that matter,
      then they should have a reasonable idea of what
      they're talking about. Dan very clearly didn't do his
      homework, and I suspect that he wasn't motivated to.
      He had an anti-agile axe to grind and so he ground it.
      It was consistent throughout his slides. Just look
      at slide 50, is that something that anyone who was
      interested in being fair end a presentation with? Or
      51 for that matter?

      The AM and AD communities are open. Dan pulled
      material off of web pages which include links to pages
      describing how to sign up to the mailing lists. He
      had every opportunity to first discuss and share his
      ideas and observations with the relevant communities.
      He choose not to. He has every opportunity to get
      involved with those communities now. He still chooses
      not to.

      > some
      > extent, I think he's right that it has - I've seen
      > that
      > behaviour too. That doesn't mean that either you
      > or Agile *deserves* blame...

      Exactly. This is yet another example of the
      dysfunction that we see in the DM community.

      >
      > Nevertheless, the behaviour is real, and the Agile
      > movement must deal - as it tries to - with human
      > weakness, not just instruct the strong and willing.
      > If you start with a sufficiently dim view of human
      > nature, you'll soon see why Dan blames Agile.

      Good point. Most of the literatue within the DM
      community does seem to have a dim view of human
      behavior. It may explain why that community continues
      to struggle to be effective.

      - Scott


      Be smarter than spam. See how smart SpamGuard is at giving junk email the boot with the All-new Yahoo! Mail at http://mrd.mail.yahoo.com/try_beta?.intl=ca
    • Jon Kern
      Scott, well, my friend, it simply boils down to this: folks can say whatever they want. the audience can choose to listen, to seek to understand, or to simply
      Message 2 of 10 , May 5, 2007
        Scott,

        well, my friend, it simply boils down to this:

        folks can say whatever they want. the audience can choose to listen, to
        seek to understand, or to simply be lemmings.

        when someone casts disparaging remarks on another, it merely reflects
        poorly on them. the ego is a wicked task master for many!

        you can never control your reputation. just continue doing the right
        things. other people's opinions do not matter.

        i assume dan is doing the best he can and thought this is the right
        message to deliver. it reflects only his opinion, such as it is. so be it.

        agile methods are possibly best left to those who are interested at a
        deeper level to try and do the best they can with the resources given.
        it is not perfect just as democracy is not perfect.

        jon
        blog: http://technicaldebt.com



        Scott Ambler said the following on 5/5/07 12:39 PM:
        >
        >
        > --- Clifford Heath <clifford.heath@...
        > <mailto:clifford.heath%40gmail.com>> wrote:
        >
        > >
        > > You read it very differently that I did. When he
        > > says
        > > "Scott says.... the implication is....", and I read
        > > it as
        > > "Scott says.... and so people do....". He's
        > > definitely
        > > getting stuck in, but not on the basis of what you
        > > say... rather on the basis of what people do. I
        >
        > Perhaps. Then again, you might want to read the
        > archives of the dm-discuss@yahoogroups.com
        > <mailto:dm-discuss%40yahoogroups.com> list. That
        > community has a history of misrepresenting agile,
        > misrepresenting my work, and personally attacking me.
        > The quality of the interaction on that list is
        > horrendous and I think you'd be hard pressed to find
        > an IT mailing list where the behavior is worse.
        >
        > So, when I see the sort of material that Dan has
        > published coming from someone in the DM community I am
        > no longer willing to give them the benefit of the
        > doubt. That community burned any good will that it
        > has with me long ago.
        >
        > > can't
        > > reason why he blames you, but my only surmisal is
        > > that he picked you as a representative of Agile in
        > > the whole, and he's seen lazy incompetents (such as
        >
        > In a private email Dan indicated that he wasn't
        > willing to get involved with either the AM or the AD
        > lists, so it's difficult to have a discussion about
        > what he was thinking.
        >
        > > populate a lot of the software development world)
        > > justifying their predilections by calling themselves
        > > Agile.
        >
        > Yes. And because he doesn't seem to be willing to
        > interact with the actual Agile community he'll
        > continue to struggle to understand the actual message
        > and be able to distiguish between the agilists and the
        > code&fixers. Worse yet, he's only reinforced these
        > obvious misunderstandings in a community which very
        > clearly needs help.
        >
        > > IOW, he blames you for excuses made by other
        > > people about their inadequacy, because he thinks
        > > that Agile has legitimised those excuses. And to
        >
        > If someone is going to get up in a public forum and
        > discuss Agile, or discuss anything for that matter,
        > then they should have a reasonable idea of what
        > they're talking about. Dan very clearly didn't do his
        > homework, and I suspect that he wasn't motivated to.
        > He had an anti-agile axe to grind and so he ground it.
        > It was consistent throughout his slides. Just look
        > at slide 50, is that something that anyone who was
        > interested in being fair end a presentation with? Or
        > 51 for that matter?
        >
        > The AM and AD communities are open. Dan pulled
        > material off of web pages which include links to pages
        > describing how to sign up to the mailing lists. He
        > had every opportunity to first discuss and share his
        > ideas and observations with the relevant communities.
        > He choose not to. He has every opportunity to get
        > involved with those communities now. He still chooses
        > not to.
        >
        > > some
        > > extent, I think he's right that it has - I've seen
        > > that
        > > behaviour too. That doesn't mean that either you
        > > or Agile *deserves* blame...
        >
        > Exactly. This is yet another example of the
        > dysfunction that we see in the DM community.
        >
        > >
        > > Nevertheless, the behaviour is real, and the Agile
        > > movement must deal - as it tries to - with human
        > > weakness, not just instruct the strong and willing.
        > > If you start with a sufficiently dim view of human
        > > nature, you'll soon see why Dan blames Agile.
        >
        > Good point. Most of the literatue within the DM
        > community does seem to have a dim view of human
        > behavior. It may explain why that community continues
        > to struggle to be effective.
        >
        > - Scott
        >
        > Be smarter than spam. See how smart SpamGuard is at giving junk email
        > the boot with the All-new Yahoo! Mail at
        > http://mrd.mail.yahoo.com/try_beta?.intl=ca
        > <http://mrd.mail.yahoo.com/try_beta?.intl=ca>
        >
        >
      • Jon Kern
        re: Slide 49 Don t Bet Better still, don t bet! If you have planned, architected and designed properly, you are much less likely to have to throw any work
        Message 3 of 10 , May 5, 2007
          re: Slide 49 "Don't Bet"

          "Better still, don't bet! If you have planned, architected and designed
          properly, you are much
          less likely to have to throw any work away"

          reminds me of something my dad used to say:
          "If the dog hadn't stopped to take a ____, he'd a caught the rabbit"

          Where is it in agile dogma that says
          Three Principles:
          - plan poorly
          - architect like a lunatic
          - and design like a newbie

          DUH. Check the size of that "IF" next time...

          jon
          blog: http://technicaldebt.com



          Scott Ambler said the following on 5/5/07 12:39 PM:
          >
          >
          > --- Clifford Heath <clifford.heath@...
          > <mailto:clifford.heath%40gmail.com>> wrote:
          >
          > >
          > > You read it very differently that I did. When he
          > > says
          > > "Scott says.... the implication is....", and I read
          > > it as
          > > "Scott says.... and so people do....". He's
          > > definitely
          > > getting stuck in, but not on the basis of what you
          > > say... rather on the basis of what people do. I
          >
          > Perhaps. Then again, you might want to read the
          > archives of the dm-discuss@yahoogroups.com
          > <mailto:dm-discuss%40yahoogroups.com> list. That
          > community has a history of misrepresenting agile,
          > misrepresenting my work, and personally attacking me.
          > The quality of the interaction on that list is
          > horrendous and I think you'd be hard pressed to find
          > an IT mailing list where the behavior is worse.
          >
          > So, when I see the sort of material that Dan has
          > published coming from someone in the DM community I am
          > no longer willing to give them the benefit of the
          > doubt. That community burned any good will that it
          > has with me long ago.
          >
          > > can't
          > > reason why he blames you, but my only surmisal is
          > > that he picked you as a representative of Agile in
          > > the whole, and he's seen lazy incompetents (such as
          >
          > In a private email Dan indicated that he wasn't
          > willing to get involved with either the AM or the AD
          > lists, so it's difficult to have a discussion about
          > what he was thinking.
          >
          > > populate a lot of the software development world)
          > > justifying their predilections by calling themselves
          > > Agile.
          >
          > Yes. And because he doesn't seem to be willing to
          > interact with the actual Agile community he'll
          > continue to struggle to understand the actual message
          > and be able to distiguish between the agilists and the
          > code&fixers. Worse yet, he's only reinforced these
          > obvious misunderstandings in a community which very
          > clearly needs help.
          >
          > > IOW, he blames you for excuses made by other
          > > people about their inadequacy, because he thinks
          > > that Agile has legitimised those excuses. And to
          >
          > If someone is going to get up in a public forum and
          > discuss Agile, or discuss anything for that matter,
          > then they should have a reasonable idea of what
          > they're talking about. Dan very clearly didn't do his
          > homework, and I suspect that he wasn't motivated to.
          > He had an anti-agile axe to grind and so he ground it.
          > It was consistent throughout his slides. Just look
          > at slide 50, is that something that anyone who was
          > interested in being fair end a presentation with? Or
          > 51 for that matter?
          >
          > The AM and AD communities are open. Dan pulled
          > material off of web pages which include links to pages
          > describing how to sign up to the mailing lists. He
          > had every opportunity to first discuss and share his
          > ideas and observations with the relevant communities.
          > He choose not to. He has every opportunity to get
          > involved with those communities now. He still chooses
          > not to.
          >
          > > some
          > > extent, I think he's right that it has - I've seen
          > > that
          > > behaviour too. That doesn't mean that either you
          > > or Agile *deserves* blame...
          >
          > Exactly. This is yet another example of the
          > dysfunction that we see in the DM community.
          >
          > >
          > > Nevertheless, the behaviour is real, and the Agile
          > > movement must deal - as it tries to - with human
          > > weakness, not just instruct the strong and willing.
          > > If you start with a sufficiently dim view of human
          > > nature, you'll soon see why Dan blames Agile.
          >
          > Good point. Most of the literatue within the DM
          > community does seem to have a dim view of human
          > behavior. It may explain why that community continues
          > to struggle to be effective.
          >
          > - Scott
          >
          > Be smarter than spam. See how smart SpamGuard is at giving junk email
          > the boot with the All-new Yahoo! Mail at
          > http://mrd.mail.yahoo.com/try_beta?.intl=ca
          > <http://mrd.mail.yahoo.com/try_beta?.intl=ca>
          >
          >
        • Clifford Heath
          ... They need sufficient insight and experience to know they re allowing it, and to evaluate the costs accurately. Nice story about your conversion to Agile,
          Message 4 of 10 , May 5, 2007
            On 06/05/2007, at 12:19 AM, Dave Rooney wrote:
            > Technical debt is only created when the people involved allow it to be
            > created.

            They need sufficient insight and experience to know they're allowing it,
            and to evaluate the costs accurately.

            Nice story about your conversion to Agile, BTW.

            > I have worked on Agile teams that routinely and effectively
            > refactored the database in a very efficient and collaborative manner.

            Likewise. It depends on the application though... I have worked
            almost exclusively in a *products* environment, until the last few
            years providing software development tools for commercial apps
            (which is where I got vicarious experience of *other* people's
            processes). More recently, I worked with a smart Agile team on
            a suite of a half-dozen products that shared a common database,
            and extended it with their own tables, with quite tight coupling,
            and a lot of very hard-core hand-crafted SQL traversing up to 100
            GB of data while the systems were serving up to 100 TPS 24x7.
            The core DB was about 100 tables, and the total was 270-ish.
            Each sub-product got upgraded separately, and needed to work
            with multiple versions of the base and other sub-products. The
            configuration risk was absolutely frightening, the number of
            deployed customers over one hundred, and the cost of a foul-up
            enormous - this was *not* an in-house application that could be
            quickly patched! And yours truly was in the DB hot seat :-) We did
            some pretty cunning things to make the process work.

            In an environment like that, you can't afford to make many mistakes,
            because the cost of fixing them is so high. Of course they happened
            occasionally, but our support received superlative reviews spanning
            a decade. So I know it's possible... but Agile did increase the risks.
            Being a product suite, we had no customer on-site either... mostly the
            work was being done for people in another continent, another timezone.
            It makes it hard to know you're building the right thing...

            > I have also worked on teams where the data model was considered
            > sacrosanct, i.e. the modelers had figured that before development
            > started they knew everything that they would ever need to know
            > about the
            > problem domain.

            Well, that's just plain stupid, I agree totally.

            > Nope, not at all. The underlying theme is the people! Good people
            > can
            > make changes as required to both code and the data model
            > effectively in
            > order to support the business.

            That's it. But good tools can reduce the cost of change, and the
            trouble with the kind of environment I described above is that there
            really aren't any refactoring tools that can tell you whether a change
            will break the performance of a critical query that took a week to tune.

            > With regards to your response points above, I agree wholeheartedly but
            > would add a fourth point:
            > * Educate people about the costs of NOT refactoring data.

            Yes, very true. The whole thing is a cost minimisation exercise,
            and that needs accurate cost prediction based on hard experience.

            >> The right answer to the problems that occur on both sides of
            >> this divide is to find ways to create and validate information
            >> models *before* making any decisions about aggregation.
            > See my last paragraph above. Sometimes you just don't know all the
            > aggregations before you start. I would also suggest that trying to
            > know
            > all the aggregations is a very inefficient use of the development
            > team's
            > time, or at least there are diminishing returns the longer that the
            > team
            > spends attempting to do so.

            Sorry, I probably shouldn't have said "aggregations", it has a
            UML meaning that's subtly different from what I meant.

            In ER modeling, the main purpose of normalisation is to dig out
            hidden entity types, and group (aggregate) their attributes in
            separate tables. That way they don't jump out and bite so much
            in the form of required refactoring.

            In object modeling, we use use cases and behaviour models
            to work out what information is associated with each role player,
            by looking at what information is contained in a message. So
            the entities in an object model reflect a behavioural aggregation.

            In Object Role Modeling, aggregation isn't allowed, so you can't
            get it wrong. Models are enforced to be elementary, there is no
            manual composition. The tools then compose either or both of
            an object model and an ER model. The composition is correct
            if the model was correct, and it's fully normalised and as efficient
            as the conceptual model allows.

            > How much time would you suggest spending on these activities?

            As much as it takes, and as often as it's needed, of course.
            The problem with previous Object Role Modeling tools is that
            they generate large amounts of code but have limited refactoring
            support, so the original model gets orphaned. My approach of
            using executable models avoids that.

            > Perhaps
            > I'm wrong, but I feel that you're making the assumption that you can
            > know far more about the problem that you are actually able up front.

            I wasn't talking about only what you can do up front. Whichever bit
            of a model you're constructing today can be more easily validated
            against today's understanding, and more easily mapped into an O-O
            or relational or O-O structure that's correct today. Sorry for not
            making
            that clearer.

            >> I've been using this technique for more than twenty years and
            >> I can't believe it hasn't taken off like wildfire, it's just so
            >> effective.
            > I've been using agile techniques for 20 years (though only under the
            > banner of 'agile' for the last 7), and they are taking off like
            > wildfire!

            I'm talking about fact-based modeling in general and Object Role
            Modeling in particular, not Agile techniques.

            Clifford Heath, Data Constellation.
          • Clifford Heath
            ... If your refactorings on objects reflect changes in the underlying information model, then you re probably talking about the same sorts of impacts. The
            Message 5 of 10 , May 5, 2007
              On 06/05/2007, at 7:57 AM, Jon Kern wrote:
              > I'm not quite sure why refactoring data models is any more costly than
              > refactoring object models. My experience on doing new projects does
              > not
              > show data models to be any more or less expensive than other parts

              If your refactorings on objects reflect changes in the underlying
              information model, then you're probably talking about the same
              sorts of impacts.

              The point is that the information model pervades every part of the
              application, from the choice of UI paradigm, through every interface,
              the logic, and down through the persistence layers, and right into the
              SQL, if you have to hand-craft queries in order to get where you're
              going. That means there's a *lot* of duplication, and a lot of impact
              (refactoring cost), of any change. And as I said in my other message,
              refactoring the SQL isn't easy, especially when the unit test criteria
              include scalability constraints on multi-GB datasets. It's not as though
              the SQL engines raise exceptions saying "hang on, you seem to have
              a cross-join between two tables which your design says will grow to
              a million records each" :-).

              No-one else I know even writes down the expected size of database
              tables, expected fan-out statistics on relationships, let alone uses
              that data to predict performance of critical queries. Yet this up-front
              analysis can often be the most agile way to avoid these problems.

              Clifford Heath, Data Constellation.
            • Scott Ambler
              ... This sort of misunderstanding is one of the fundamental challenges that the agile community faces with the more traditional aspects of the IT world. They
              Message 6 of 10 , May 6, 2007
                --- Jon Kern <jonkern@...> wrote:

                > re: Slide 49 "Don't Bet"
                >
                > "Better still, don't bet! If you have planned,
                > architected and designed
                > properly, you are much
                > less likely to have to throw any work away"
                >
                > reminds me of something my dad used to say:
                > "If the dog hadn't stopped to take a ____, he'd
                > a caught the rabbit"
                >
                > Where is it in agile dogma that says
                > Three Principles:
                > - plan poorly
                > - architect like a lunatic
                > - and design like a newbie

                This sort of misunderstanding is one of the
                fundamental challenges that the agile community faces
                with the more traditional aspects of the IT world.
                They struggle to understand agile because it really is
                different that what they're used to. The underlying
                philosophies have changed. Many of the assumptions of
                the traditional world, such as needing to think
                everything through up front, have been examined and
                found wanting.

                The data management community is a perfect example of
                a group of people that don't seem to understand what
                it is that we're doing, that don't seem willing to
                learn new techniques, and that unfortunately seem to
                be very willing to misrepresent and twist our message
                into something that conforms to their very obvious
                prejudices. Considering their incredibly poor track
                record, the Data Warehouse Institute estimates that
                data quality problems potentially result in $600
                billion annual productivity loss worldwide, you would
                think that they'd be interested in finding new ways to
                help address quality problems. Concrete practices
                such as database regression testing and database
                refactoring come to mind, practices which they choose
                to denigrate whenever possible. Instead their
                solution is to throw more bureaucracy at the problem.
                That community is so out of sync with everyone else
                it's incredible. Unfortunately they're burdened with
                thought leaders whose heads are clearly in the 70s and
                organizations such as DAMA which do little more than
                promote the existing insularness. It's a real shame
                that the DM community is unable to think outside of
                it's self-inflicted box.

                - Scott



                Ask a question on any topic and get answers from real people. Go to Yahoo! Answers and share what you know at http://ca.answers.yahoo.com
              • Curt Sampson
                ... Oh, I can only hope that my competitors subscribe to this set of beliefs. ... I cannot for the life of me see why any data group managers with the
                Message 7 of 10 , May 6, 2007
                  On Fri, 4 May 2007, Scott Ambler wrote:

                  > http://www.dama-nj.org/presentations/FragileData%20DanPaolini.pdf

                  Oh, I can only hope that my competitors subscribe to this set of beliefs.

                  On Sun, 6 May 2007, Scott Ambler wrote:

                  > Considering their incredibly poor track record... you would think
                  > that they'd be interested in finding new ways to help address quality
                  > problems.

                  I cannot for the life of me see why any data group managers with the
                  slightest shred of political sense would like agile. They'd be able to
                  all of their work with half the staff or less, and many of those staff
                  would be more responsible to the development groups than the data group.
                  If you're trying to destroy a data empire, giving Agile a serious try is
                  certainly a good way to do it.

                  On Sun, 6 May 2007, Clifford Heath wrote:

                  > He's definitely getting stuck in, but not on the basis of what you
                  > say... rather on the basis of what people do.

                  I wonder. Is this really behaviour he's seeing in the real world on a
                  consistent basis, or is he just expressing his fears, biased by his
                  point of view?

                  cjs
                  --
                  Curt Sampson +81 90 7737 2974
                  <cjs@...> http://www.starling-software.com
                  The power of accurate observation is commonly called cynicism
                  by those who have not got it. --George Bernard Shaw
                • Johan Stuyts
                  ... They have pros and cons, and for me OR mappers work better than interacting with the database directly. I have been thinking about a layer that is strong
                  Message 8 of 10 , Aug 8 11:22 PM
                    > > Your technique looks very useful but I am using an object-relational
                    > > mapper. Have you applied this technique when using an ORM?
                    > If so, how?
                    >
                    > That could well be part of the problem: OR mappers tend to
                    > take problems
                    > solved quite easily with relational logic and make them much more
                    > difficult.

                    They have pros and cons, and for me OR mappers work better than
                    interacting with the database directly. I have been thinking about a
                    layer that is strong for both objects and relational, but have not been
                    able to come up with a satisfactory solution.

                    > But basically, just fix your concept of the candidate key of
                    > this table
                    > to include "this information valid at this particular time," and it
                    > probably won't be too hard to move on from there.

                    Doh, I did not think about replacing the deletion number field with a
                    date field. Thanks.

                    Johan
                  Your message has been successfully submitted and would be delivered to recipients shortly.