Loading ...
Sorry, an error occurred while loading the content.
 

RE: [XP] etl tools (particularly DataStage) supporting agile practices

Expand Messages
  • Bayley, Alistair
    ... difficult... ... There is a moderately active AgileDatabases group, where you may get more mileage with your questions (and more information than I have to
    Message 1 of 2 , Mar 20, 2008
      > From: extremeprogramming@yahoogroups.com
      > [mailto:extremeprogramming@yahoogroups.com] On Behalf Of
      > martin_j_andrews
      >
      > Yesterday, I sat in a demonstration from IBM of its ETL tool called
      > DataStage. It's supposed to be a market leader in this space. I
      > asked some simple (or so I thought) questions about unit/integration
      > testing, source control & continuous integration. The answers turned
      > out to be remarkably bad.
      >
      > Source control was the worst....
      >
      > Automated unit testing was almost as bad. Everything seems to be GUI
      > driven in the fat client app, which makes automation rather
      difficult...
      >
      > I was flabbergasted at how badly this turned out, particularly in
      > relation to source control.
      >
      > Does anyone have some experience with other ETL that differs? Is
      > there some viable alternatives in this space that would support the
      > agility of development better? Maybe I'm even giving DataStage a poor
      > review because I don't understand the 'normal' development style?


      There is a moderately active AgileDatabases group, where you may get
      more mileage with your questions (and more information than I have to
      offer).

      I'm not offering much positive here; just adding to your perspective. We
      recently looked at using Informatica (which sounds very similar to
      DataStage) to implement new feeds into an existing datawarehouse.
      Existing feeds are written in Oracle PL/SQL, with which we can use
      source code management, automated testing, etc. Our presentation was not
      from a vendor, but from someone on one of our other internal teams who
      is already using it.

      I came away with the same impressions that you did:
      - has it's own internal repository; no source code management
      - no scripting/automation capability (for releases, for example).
      Releases are done manually (!?)
      - everything is done via the GUI

      I get the impression that the vendors (well, at least these two) don't
      view what you're doing as writing code, so it's not subject to the same
      requirements and constraints.


      Alistair
      *****************************************************************
      Confidentiality Note: The information contained in this message,
      and any attachments, may contain confidential and/or privileged
      material. It is intended solely for the person(s) or entity to
      which it is addressed. Any review, retransmission, dissemination,
      or taking of any action in reliance upon this information by
      persons or entities other than the intended recipient(s) is
      prohibited. If you received this in error, please contact the
      sender and delete the material from any computer.
      *****************************************************************
    Your message has been successfully submitted and would be delivered to recipients shortly.