Loading ...
Sorry, an error occurred while loading the content.
 

Re: Agile Implementation of Netezza/Cognos Implementation

Expand Messages
  • bobcorrick@btopenworld.com
    Boma, It sounds as though your situation may be improved through evolutionary delivery rather than a big bang. For example: what are your BI clients using now
    Message 1 of 5 , Sep 14, 2011
      Boma,
      It sounds as though your situation may be improved through evolutionary delivery rather than a big bang.
      For example: what are your BI clients using now to inform their work? Spreadsheets, perhaps? So what might you provide soon and often that would help them?
      Longer term, the data quality issue may bite hardest.
      My 2c.
      Bob.

      --- In agileDatabases@yahoogroups.com, "bomasamudram" <bomasamudram@...> wrote:
      >
      > I am a managing a Netezza/Cognos implementation effort and there is a hard deadline to deliver a quality BI release (BI: Business Intelligence). The following challenges are impacting the implementation.
      >
      > * Dimensional Warehouse
      > * Dynamic Attribute & Flexible Hierarchy
      > * Staging Models
      > * Processing Models
      > * Metadata Models
      > * Semantic Layer
      > * Data Quality Model
      >
      > The report developer and analyst capability cost driver is rated approx Average (around 50th percentile), factoring in the Netezza and Cognos learning curve. The definition of "Report Dev Complete" is a moving target that is causing the project schedule to slip. The word "refactoring" is being fashionably used to punt any efforts to establish rigor. To illustrate ths point:
      >
      > PM (asks in the standup meeting): Joe, how much more work is remaining to achieve code complete (milestone) of Report Foo?
      > Joe: It depends, I am refactoring the report. It can take anywhere between 2 hours and 3 days. You also need to bear in mind that we are impacted by report complexity, technology learning curve, simultaneous refactoring happening in the datamart and ETL layers as well as data quality issues.
      >
      > The snapshot of this conversation is a vivid illustration of the challenge that I am wrestling with. There is a need to urgently "agilify" this project.
      >
      > In addition, I welcome any pointers to design patterns or blueprints corresponding to the following:
      >
      > DATA MODELS AND DATA STRUCTURES
      > * Dimensional Warehouse
      > * Subscriber Profiles
      > * Segmentation Schemes
      > * Dynamic Attribute & Flexible Hierarchy
      > * Staging Models
      > * Reporting
      > * Processing Models
      > * Metadata Models
      > * Semantic Layer
      > * Data Quality Model
      >
      > FACTORY MANAGEMENT
      > * Scheduling
      > * Workflow
      > * Dimension Processing
      > * Fact Processing
      > * Alerts & Messaging
      > * Quality Profiling /Monitoring
      > * ABaC
      > * Process Logging
      > * Quality Measures
      > * 3rd Party Data Cleansing
      >
      >
      > ON-STREAM DATA MANAGEMENT AND OPERATIONAL ANALYTICS
      > * CDC: Change Data Capture
      > * On-Stream Analytics
      > * Key Management (SK vs. NK)
      > * Pivot (Attributes & Hierarchies)
      > * Latent Fact Handling (Late Poll / Adjust)
      > * SQL Generation & Validation: Selection + Filtering
      >
      >
      > ANALYTICS, REPORTING & PERFORMANCE ACCELERATORS
      > * Like Application Grouping On Spokes
      > * Data Movement Attributes Between systems
      > * Interactive Reporting
      > * Automated "Spoke Creation"
      > * Metadata tool integration & management
      > * Automated Star or Snowflake Model Reporting Model Generation
      >
      > Thanks,
      >
      > Boma
      >
    • Keith Ray
      Have you already read up on the various books about database refactoring? http://goo.gl/6ZBT7 Note that refactoring is not supposed to change behavior, so once
      Message 2 of 5 , Sep 14, 2011
        Have you already read up on the various books about database refactoring?
        http://goo.gl/6ZBT7

        Note that refactoring is not supposed to change behavior, so once a report
        is written, additional refactoring should not make that report stop working.

        --
        C. Keith Ray

        Coach, Trainer, and Developer at Industrial logic, Inc.
        http://industriallogic.com "Amplify Your Agility"
        Coaching and Live and Web-based Training
        On Tue, Sep 13, 2011 at 3:55 PM, bomasamudram <bomasamudram@...>wrote:

        > **
        >
        >
        > I am a managing a Netezza/Cognos implementation effort and there is a hard
        > deadline to deliver a quality BI release (BI: Business Intelligence). The
        > following challenges are impacting the implementation.
        >
        > * Dimensional Warehouse
        > * Dynamic Attribute & Flexible Hierarchy
        > * Staging Models
        > * Processing Models
        > * Metadata Models
        > * Semantic Layer
        > * Data Quality Model
        >
        > The report developer and analyst capability cost driver is rated approx
        > Average (around 50th percentile), factoring in the Netezza and Cognos
        > learning curve. The definition of "Report Dev Complete" is a moving target
        > that is causing the project schedule to slip. The word "refactoring" is
        > being fashionably used to punt any efforts to establish rigor. To illustrate
        > ths point:
        >
        > PM (asks in the standup meeting): Joe, how much more work is remaining to
        > achieve code complete (milestone) of Report Foo?
        > Joe: It depends, I am refactoring the report. It can take anywhere between
        > 2 hours and 3 days. You also need to bear in mind that we are impacted by
        > report complexity, technology learning curve, simultaneous refactoring
        > happening in the datamart and ETL layers as well as data quality issues.
        >
        > The snapshot of this conversation is a vivid illustration of the challenge
        > that I am wrestling with. There is a need to urgently "agilify" this
        > project.
        >
        > In addition, I welcome any pointers to design patterns or blueprints
        > corresponding to the following:
        >
        > DATA MODELS AND DATA STRUCTURES
        > * Dimensional Warehouse
        > * Subscriber Profiles
        > * Segmentation Schemes
        > * Dynamic Attribute & Flexible Hierarchy
        > * Staging Models
        > * Reporting
        > * Processing Models
        > * Metadata Models
        > * Semantic Layer
        > * Data Quality Model
        >
        > FACTORY MANAGEMENT
        > * Scheduling
        > * Workflow
        > * Dimension Processing
        > * Fact Processing
        > * Alerts & Messaging
        > * Quality Profiling /Monitoring
        > * ABaC
        > * Process Logging
        > * Quality Measures
        > * 3rd Party Data Cleansing
        >
        > ON-STREAM DATA MANAGEMENT AND OPERATIONAL ANALYTICS
        > * CDC: Change Data Capture
        > * On-Stream Analytics
        > * Key Management (SK vs. NK)
        > * Pivot (Attributes & Hierarchies)
        > * Latent Fact Handling (Late Poll / Adjust)
        > * SQL Generation & Validation: Selection + Filtering
        >
        > ANALYTICS, REPORTING & PERFORMANCE ACCELERATORS
        > * Like Application Grouping On Spokes
        > * Data Movement Attributes Between systems
        > * Interactive Reporting
        > * Automated "Spoke Creation"
        > * Metadata tool integration & management
        > * Automated Star or Snowflake Model Reporting Model Generation
        >
        > Thanks,
        >
        > Boma
        >
        >
        >


        [Non-text portions of this message have been removed]
      • Scott Ambler
        GT Agile or not, technological and data dependencies cannot really be sidestepped by refactoring . But, they can often be addressed through mocks or
        Message 3 of 5 , Sep 17, 2011
          GT > Agile or not, technological and data dependencies cannot really be sidestepped by 'refactoring'.

          But, they can often be addressed through mocks or stubs.  Of course, you should probably do some initial requirements and architectural envisioning to identify them in the first place.
           
          GT > Only after evaluating the impact as minor and the benefits as sufficient to outweigh the negative impacts, the database schema changes while a code iteration is in train.

          Agreed.  This is exactly the first step of the process of database refactoring, doin a reality check to determine if it makes sense to do so.
           
          GT > While one or more code iterations are taking place in paralle, the data design and ETL are working on their iteration of the db schema and data, which will be consumed by later code iterations.

          Better yet, this could occur in a "whole team" manner where data-experienced people are embedded in the actual team.  This can improve productivity by reducing overall overhead.  Unfortunately this can be difficult in many companies due to the organizational complexities resulting from the cultural impedance mismatch between data and development professionals.

          - Scott 
          Scott W. Ambler
          Chief Methodologist for Agile/Lean, IBM Rational
          Agile at Scale blog: http://www.ibm.com/developerworks/blogs/page/ambler
          Follow me on Twitter: http://twitter.com/scottwambler


          [Non-text portions of this message have been removed]
        Your message has been successfully submitted and would be delivered to recipients shortly.