Loading ...
Sorry, an error occurred while loading the content.

Re: Agile Implementation of Netezza/Cognos Implementation

Expand Messages
  • bobcorrick@btopenworld.com
    Boma, It sounds as though your situation may be improved through evolutionary delivery rather than a big bang. For example: what are your BI clients using now
    Message 1 of 5 , Sep 14, 2011
    • 0 Attachment
      Boma,
      It sounds as though your situation may be improved through evolutionary delivery rather than a big bang.
      For example: what are your BI clients using now to inform their work? Spreadsheets, perhaps? So what might you provide soon and often that would help them?
      Longer term, the data quality issue may bite hardest.
      My 2c.
      Bob.

      --- In agileDatabases@yahoogroups.com, "bomasamudram" <bomasamudram@...> wrote:
      >
      > I am a managing a Netezza/Cognos implementation effort and there is a hard deadline to deliver a quality BI release (BI: Business Intelligence). The following challenges are impacting the implementation.
      >
      > * Dimensional Warehouse
      > * Dynamic Attribute & Flexible Hierarchy
      > * Staging Models
      > * Processing Models
      > * Metadata Models
      > * Semantic Layer
      > * Data Quality Model
      >
      > The report developer and analyst capability cost driver is rated approx Average (around 50th percentile), factoring in the Netezza and Cognos learning curve. The definition of "Report Dev Complete" is a moving target that is causing the project schedule to slip. The word "refactoring" is being fashionably used to punt any efforts to establish rigor. To illustrate ths point:
      >
      > PM (asks in the standup meeting): Joe, how much more work is remaining to achieve code complete (milestone) of Report Foo?
      > Joe: It depends, I am refactoring the report. It can take anywhere between 2 hours and 3 days. You also need to bear in mind that we are impacted by report complexity, technology learning curve, simultaneous refactoring happening in the datamart and ETL layers as well as data quality issues.
      >
      > The snapshot of this conversation is a vivid illustration of the challenge that I am wrestling with. There is a need to urgently "agilify" this project.
      >
      > In addition, I welcome any pointers to design patterns or blueprints corresponding to the following:
      >
      > DATA MODELS AND DATA STRUCTURES
      > * Dimensional Warehouse
      > * Subscriber Profiles
      > * Segmentation Schemes
      > * Dynamic Attribute & Flexible Hierarchy
      > * Staging Models
      > * Reporting
      > * Processing Models
      > * Metadata Models
      > * Semantic Layer
      > * Data Quality Model
      >
      > FACTORY MANAGEMENT
      > * Scheduling
      > * Workflow
      > * Dimension Processing
      > * Fact Processing
      > * Alerts & Messaging
      > * Quality Profiling /Monitoring
      > * ABaC
      > * Process Logging
      > * Quality Measures
      > * 3rd Party Data Cleansing
      >
      >
      > ON-STREAM DATA MANAGEMENT AND OPERATIONAL ANALYTICS
      > * CDC: Change Data Capture
      > * On-Stream Analytics
      > * Key Management (SK vs. NK)
      > * Pivot (Attributes & Hierarchies)
      > * Latent Fact Handling (Late Poll / Adjust)
      > * SQL Generation & Validation: Selection + Filtering
      >
      >
      > ANALYTICS, REPORTING & PERFORMANCE ACCELERATORS
      > * Like Application Grouping On Spokes
      > * Data Movement Attributes Between systems
      > * Interactive Reporting
      > * Automated "Spoke Creation"
      > * Metadata tool integration & management
      > * Automated Star or Snowflake Model Reporting Model Generation
      >
      > Thanks,
      >
      > Boma
      >
    • Gabriel Tanase
      IMHO the project is trying to chew too much at the same time. I feel that Joe spoke the truth: how can a developer know how long it would take to finalize
      Message 2 of 5 , Sep 14, 2011
      • 0 Attachment
        IMHO the project is trying to "chew" too much at the same time.

        I feel that "Joe" spoke the truth: how can a developer know how long it
        would take to finalize a report when there is continuous simultaneous
        refactoring in the database schema and ETL, plus data quality issues?

        Agile or not, technological and data dependencies cannot really be
        sidestepped by 'refactoring'.

        Perhaps the PM should have asked "Joe, how much more work is remaining to
        achieve code complete (milestone) of Report Foo using the database schema
        and data provided through ETL and frozen as of zz/zz/zzzz ?"


        In a non-BI project I have been working in, the code iterations work with a
        stable database schema and data set, which had been delivered by a database
        design & ETL iteration scheduled and executed prior to the the code
        iteration starting.
        Only on an exception basis, and only after evaluating the impact as minor
        and the benefits as sufficient to outweigh the negative impacts, the
        database schema changes while a code iteration is in train.
        Yes, this may ocassionaly leave functional or technical debt at the end of a
        code iteration. So be it, that's what agile prioritization is for at the
        beginning of the next iteration(s).
        While one or more code iterations are taking place in paralle, the data
        design and ETL are working on their iteration of the db schema and data,
        which will be consumed by later code iterations.

        As for some blueprints: I found Kimball's books and the books that Chris
        Adamson authored or co-authored (
        http://www.amazon.com/Christopher-Adamson/e/B001H6O2Z0) useful.


        Kind regards,
        Gabriel
        ----------
        http://ie.linkedin.com/in/gabrieltanase



        On 13 September 2011 23:55, bomasamudram <bomasamudram@...> wrote:

        > **
        >
        >
        > I am a managing a Netezza/Cognos implementation effort and there is a hard
        > deadline to deliver a quality BI release (BI: Business Intelligence). The
        > following challenges are impacting the implementation.
        >
        > * Dimensional Warehouse
        > * Dynamic Attribute & Flexible Hierarchy
        > * Staging Models
        > * Processing Models
        > * Metadata Models
        > * Semantic Layer
        > * Data Quality Model
        >
        > The report developer and analyst capability cost driver is rated approx
        > Average (around 50th percentile), factoring in the Netezza and Cognos
        > learning curve. The definition of "Report Dev Complete" is a moving target
        > that is causing the project schedule to slip. The word "refactoring" is
        > being fashionably used to punt any efforts to establish rigor. To illustrate
        > ths point:
        >
        > PM (asks in the standup meeting): Joe, how much more work is remaining to
        > achieve code complete (milestone) of Report Foo?
        > Joe: It depends, I am refactoring the report. It can take anywhere between
        > 2 hours and 3 days. You also need to bear in mind that we are impacted by
        > report complexity, technology learning curve, simultaneous refactoring
        > happening in the datamart and ETL layers as well as data quality issues.
        >
        > The snapshot of this conversation is a vivid illustration of the challenge
        > that I am wrestling with. There is a need to urgently "agilify" this
        > project.
        >
        > In addition, I welcome any pointers to design patterns or blueprints
        > corresponding to the following:
        > [...]
        >
        > Thanks,
        >
        > Boma
        >
        > _
        >


        [Non-text portions of this message have been removed]
      • Keith Ray
        Have you already read up on the various books about database refactoring? http://goo.gl/6ZBT7 Note that refactoring is not supposed to change behavior, so once
        Message 3 of 5 , Sep 14, 2011
        • 0 Attachment
          Have you already read up on the various books about database refactoring?
          http://goo.gl/6ZBT7

          Note that refactoring is not supposed to change behavior, so once a report
          is written, additional refactoring should not make that report stop working.

          --
          C. Keith Ray

          Coach, Trainer, and Developer at Industrial logic, Inc.
          http://industriallogic.com "Amplify Your Agility"
          Coaching and Live and Web-based Training
          On Tue, Sep 13, 2011 at 3:55 PM, bomasamudram <bomasamudram@...>wrote:

          > **
          >
          >
          > I am a managing a Netezza/Cognos implementation effort and there is a hard
          > deadline to deliver a quality BI release (BI: Business Intelligence). The
          > following challenges are impacting the implementation.
          >
          > * Dimensional Warehouse
          > * Dynamic Attribute & Flexible Hierarchy
          > * Staging Models
          > * Processing Models
          > * Metadata Models
          > * Semantic Layer
          > * Data Quality Model
          >
          > The report developer and analyst capability cost driver is rated approx
          > Average (around 50th percentile), factoring in the Netezza and Cognos
          > learning curve. The definition of "Report Dev Complete" is a moving target
          > that is causing the project schedule to slip. The word "refactoring" is
          > being fashionably used to punt any efforts to establish rigor. To illustrate
          > ths point:
          >
          > PM (asks in the standup meeting): Joe, how much more work is remaining to
          > achieve code complete (milestone) of Report Foo?
          > Joe: It depends, I am refactoring the report. It can take anywhere between
          > 2 hours and 3 days. You also need to bear in mind that we are impacted by
          > report complexity, technology learning curve, simultaneous refactoring
          > happening in the datamart and ETL layers as well as data quality issues.
          >
          > The snapshot of this conversation is a vivid illustration of the challenge
          > that I am wrestling with. There is a need to urgently "agilify" this
          > project.
          >
          > In addition, I welcome any pointers to design patterns or blueprints
          > corresponding to the following:
          >
          > DATA MODELS AND DATA STRUCTURES
          > * Dimensional Warehouse
          > * Subscriber Profiles
          > * Segmentation Schemes
          > * Dynamic Attribute & Flexible Hierarchy
          > * Staging Models
          > * Reporting
          > * Processing Models
          > * Metadata Models
          > * Semantic Layer
          > * Data Quality Model
          >
          > FACTORY MANAGEMENT
          > * Scheduling
          > * Workflow
          > * Dimension Processing
          > * Fact Processing
          > * Alerts & Messaging
          > * Quality Profiling /Monitoring
          > * ABaC
          > * Process Logging
          > * Quality Measures
          > * 3rd Party Data Cleansing
          >
          > ON-STREAM DATA MANAGEMENT AND OPERATIONAL ANALYTICS
          > * CDC: Change Data Capture
          > * On-Stream Analytics
          > * Key Management (SK vs. NK)
          > * Pivot (Attributes & Hierarchies)
          > * Latent Fact Handling (Late Poll / Adjust)
          > * SQL Generation & Validation: Selection + Filtering
          >
          > ANALYTICS, REPORTING & PERFORMANCE ACCELERATORS
          > * Like Application Grouping On Spokes
          > * Data Movement Attributes Between systems
          > * Interactive Reporting
          > * Automated "Spoke Creation"
          > * Metadata tool integration & management
          > * Automated Star or Snowflake Model Reporting Model Generation
          >
          > Thanks,
          >
          > Boma
          >
          >
          >


          [Non-text portions of this message have been removed]
        • Scott Ambler
          GT Agile or not, technological and data dependencies cannot really be sidestepped by refactoring . But, they can often be addressed through mocks or
          Message 4 of 5 , Sep 17, 2011
          • 0 Attachment
            GT > Agile or not, technological and data dependencies cannot really be sidestepped by 'refactoring'.

            But, they can often be addressed through mocks or stubs.  Of course, you should probably do some initial requirements and architectural envisioning to identify them in the first place.
             
            GT > Only after evaluating the impact as minor and the benefits as sufficient to outweigh the negative impacts, the database schema changes while a code iteration is in train.

            Agreed.  This is exactly the first step of the process of database refactoring, doin a reality check to determine if it makes sense to do so.
             
            GT > While one or more code iterations are taking place in paralle, the data design and ETL are working on their iteration of the db schema and data, which will be consumed by later code iterations.

            Better yet, this could occur in a "whole team" manner where data-experienced people are embedded in the actual team.  This can improve productivity by reducing overall overhead.  Unfortunately this can be difficult in many companies due to the organizational complexities resulting from the cultural impedance mismatch between data and development professionals.

            - Scott 
            Scott W. Ambler
            Chief Methodologist for Agile/Lean, IBM Rational
            Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
            Follow me on Twitter: http://twitter.com/scottwambler


            [Non-text portions of this message have been removed]
          Your message has been successfully submitted and would be delivered to recipients shortly.