Loading ...
Sorry, an error occurred while loading the content.

Re: is XP adapted for Business Intelligence applications

Expand Messages
  • strazhce
    Hi, Steve. ... This is an interesting approach. I imagine there would be considerable resistance to developing a conversion tool (I m an opponent of storage
    Message 1 of 18 , Jan 4, 2010
    • 0 Attachment
      Hi, Steve.

      --- In extremeprogramming@yahoogroups.com, Steven Gordon <sgordonphd@...> wrote:
      >
      > Why does the code have to be done in PL SQL?
      >
      > Is it because it has always been done this way, or because it is the
      > skill your team has, or because any other language would be too slow
      > for the large data sets?
      >
      > Would it be feasible to harness a flexible language like Ruby
      This is an interesting approach. I imagine there would be considerable resistance to developing a conversion tool (I'm an opponent of storage procedures programming, but I would think twice before doing this).

      1. What are benefits of this approach?
      2. How this promotes better testability/maintainability?
      - how do you test such code?
      - how do you refactor such code?

      Thanks.
      Oleg
    • Steven Gordon
      ... [Non-text portions of this message have been removed]
      Message 2 of 18 , Jan 4, 2010
      • 0 Attachment
        On Mon, Jan 4, 2010 at 5:59 AM, strazhce <infobox.oleg@...> wrote:

        >
        >
        > Hi, Steve.
        >
        >
        > --- In extremeprogramming@yahoogroups.com<extremeprogramming%40yahoogroups.com>,
        > Steven Gordon <sgordonphd@...> wrote:
        > >
        > > Why does the code have to be done in PL SQL?
        > >
        > > Is it because it has always been done this way, or because it is the
        > > skill your team has, or because any other language would be too slow
        > > for the large data sets?
        > >
        > > Would it be feasible to harness a flexible language like Ruby
        > This is an interesting approach. I imagine there would be considerable
        > resistance to developing a conversion tool (I'm an opponent of storage
        > procedures programming, but I would think twice before doing this).
        >
        > 1. What are benefits of this approach?
        > 2. How this promotes better testability/maintainability?
        > - how do you test such code?
        > - how do you refactor such code?
        >
        > Thanks.
        > Oleg
        >
        >
        >


        [Non-text portions of this message have been removed]
      • Steven Gordon
        ... Testability and maintainability. ... Unit test? You can unit test the code in Ruby by executing it. I believe the embedded SQL can be directly executed
        Message 3 of 18 , Jan 4, 2010
        • 0 Attachment
          On Mon, Jan 4, 2010 at 5:59 AM, strazhce <infobox.oleg@...> wrote:

          >
          >
          > Hi, Steve.
          >
          >
          > --- In extremeprogramming@yahoogroups.com<extremeprogramming%40yahoogroups.com>,
          > Steven Gordon <sgordonphd@...> wrote:
          > >
          > > Why does the code have to be done in PL SQL?
          > >
          > > Is it because it has always been done this way, or because it is the
          > > skill your team has, or because any other language would be too slow
          > > for the large data sets?
          > >
          > > Would it be feasible to harness a flexible language like Ruby
          > This is an interesting approach. I imagine there would be considerable
          > resistance to developing a conversion tool (I'm an opponent of storage
          > procedures programming, but I would think twice before doing this).
          >
          > 1. What are benefits of this approach?
          >

          Testability and maintainability.


          > 2. How this promotes better testability/maintainability?
          > - how do you test such code?
          >

          Unit test? You can unit test the code in Ruby by executing it. I believe
          the embedded SQL can be directly executed (might be slow for large data
          sets, but fine for small data sets used for unit testing). Mocking for code
          isolation can be accomplished by mocking in Ruby.

          Acceptance testing would be best done by generating target code and
          deploying it.


          > - how do you refactor such code?
          >

          It might not be automated, but the unit testing facilitates refactoring,
          even if it is manual.

          The issue is whether the cost and maintenance of the evolving target-code
          generator is worth the ability to better support isolated unit-testing and
          refactoring of the code. I am not totally convinced on this trade-off, so
          it is just "a thought experiment".


          >
          > Thanks.
          > Oleg
          >
          >
          >
          >


          [Non-text portions of this message have been removed]
        • William Pietri
          ... I think that s a point worth highlighting. Automated refactoring tools are undeniably awesome. But even if you have to refactor manually, it s still much
          Message 4 of 18 , Jan 4, 2010
          • 0 Attachment
            On 01/04/2010 07:08 AM, Steven Gordon wrote:
            > It might not be automated, but the unit testing facilitates refactoring,
            > even if it is manual.
            >

            I think that's a point worth highlighting. Automated refactoring tools
            are undeniably awesome. But even if you have to refactor manually, it's
            still much better than the alternatives.

            William
          • JeffGrigg
            [After all these years, it does seem that my posts are now moderated. How interesting. I do think that my contributions are constructive.] ... Yes. Anything
            Message 5 of 18 , Jan 5, 2010
            • 0 Attachment
              [After all these years, it does seem that my posts are now moderated. How interesting. I do think that my contributions are constructive.]

              > That said, it is certainly possible to do TDD for procedural
              > code including PL/SQL.

              Yes. Anything is possible. Some things are easier than others.

              For xUnit testing of PL/SQL in PL/SQL, consider the following:
              http://www.c2.com/cgi/wiki?PlUnit
              and
              http://www.c2.com/cgi/wiki?PlSqlUnit

              But I suspect that given the tight binding of PL/SQL to the database and lack of support for injecting mock implementations, something that I would be likely to try for testing of complex PL/SQL-based applications is to do all testing in a development database where you have complete runtime control over the data and PL/SQL procedures, and be willing to have the tests repopulate the database and replace stored procedure code.

              > Response : OK good advice but this seem to me that this is not truly
              > unit testing because we can't test in isolation.
              >
              > Ex: procedure P1 make some work and uses procedure P2 and P3.
              > In JAVA I would stub the P2 and P3 data needed in P1 and so
              > test P1 in isolation from P2 and P3. In PL SQL this is not
              > possible so if there is a bug in P2 the « unit test » on P2
              > will fail but also the « unit test » on P1.

              Unit testing is over-emphasized. Do integration testing.

              IE: Given that the database is in a certain state and you call P1, expect the database to now be in a new state, and certain expected result sets are returned.

              Also, if testing in a development database you own, you can replace procedures P2 and P3 with mock versions, call P1, and then restore the old versions of P2 and P3.


              > Response : thanks for the reference. There is another problem:
              > In JAVA we have refactoring tools in eclipse that helps. I
              > don't know such tools for PL SQL or SQL.

              You have the most powerful refactoring tool known to the industry: The Human Mind.

              Yes, you can refactor SQL and PL/SQL. The lack of helpful tooling may be an annoyance, but that shouldn't stop you.

              What would you do if you needed to do a refactoring in Java, but it didn't appear on the IDE's menu of supported refactorings?
            • John Roth
              ... I checked, and you ve got the same status as all other members. I m certainly not seeing any moderation messages for you, or any other member outside of
              Message 6 of 18 , Jan 5, 2010
              • 0 Attachment
                JeffGrigg wrote:
                >
                >
                > [After all these years, it does seem that my posts are now moderated.
                > How interesting. I do think that my contributions are constructive.]
                >
                I checked, and you've got the same status as all other members.
                I'm certainly not seeing any moderation messages for you, or
                any other member outside of the usual first post moderation.

                We seem to have a problem with timely delivery of messages. I've
                noticed it on this and on other Yahoo groups I'm a member of.
                I've seen some messages take as long as a couple of days to
                clear whatever process Yahoo uses, and it's been going on for
                several months.

                John Roth
                Moderator.
              • Ron Jeffries
                Hello, JeffGrigg. On Tuesday, January 5, 2010, at 8:02:21 AM, you ... Me too, and I m not aware of any moderation (on my part). Did you say F#$% or something?
                Message 7 of 18 , Jan 5, 2010
                • 0 Attachment
                  Hello, JeffGrigg. On Tuesday, January 5, 2010, at 8:02:21 AM, you
                  wrote:

                  > [After all these years, it does seem that my posts are now
                  > moderated. How interesting. I do think that my contributions are constructive.]

                  Me too, and I'm not aware of any moderation (on my part). Did you
                  say F#$% or something? :)

                  Ron Jeffries
                  www.XProgramming.com
                  www.xprogramming.com/blog
                  If it is more than you need, it is waste. -- Andy Seidl
                • JeffGrigg
                  ... No. But I might, if it takes several days to post a message! ;- Perhaps my previous posts just got lost in some bit-bucket. Bother!
                  Message 8 of 18 , Jan 5, 2010
                  • 0 Attachment
                    --- John Roth <JohnRoth1@...> wrote:
                    > I've seen some messages take as long as a couple of days to
                    > clear whatever process Yahoo uses, and it's been going on for
                    > several months.

                    --- Ron Jeffries <ronjeffries@...> wrote:
                    > Me too, and I'm not aware of any moderation (on my part).
                    > Did you say F#$% or something? :)

                    No. But I might, if it takes several days to post a message! ;->

                    Perhaps my previous posts just got lost in some bit-bucket. Bother!
                    :-[
                  • D.André Dhondt
                    ... In my experience with DSS and T-SQL, TDD doesn t make a lot of sense for a 4GL like SQL, which doesn t really specify what we re going to do in a unit way.
                    Message 9 of 18 , Jan 6, 2010
                    • 0 Attachment
                      >
                      > > That said, it is certainly possible to do TDD for procedural
                      > > code including PL/SQL.
                      >

                      In my experience with DSS and T-SQL, TDD doesn't make a lot of sense for a
                      4GL like SQL, which doesn't really specify what we're going to do in a unit
                      way. We added tests, but they don't have the same kind of coverage as UTs.
                      Scaled-down dev/test databases don't make much sense either, since we
                      didn't really know what records the query was going to fetch, nor what the
                      performance implications would be, until we ran it in the production
                      environment. Even the execution paths turned out different on tables with
                      different statistics.

                      One of the aims of TDD is to make sure our units are right--that they are
                      what we intended. What we found is that by using smaller chunks, e.g.,
                      'extracting method' in our server-side code as much as possible, we
                      clarified our intent, we could add some integration tests/functional
                      tests/FIT tests, and basically we got more confidence in our database-side
                      code. We also found that by clarifying intent, we could simplify the
                      existing code, and that made significant gains in performance. Regression
                      testing was almost entirely manual, though.

                      Still, the biggest concern with db-side changes had to do with performance,
                      and we just couldn't find a way to provide sufficient test coverage for
                      that. So our strategy (there is a book written on this subject, db
                      refactoring or something, but I never read it) ended up being a strangler
                      pattern--isolate the functionality that needed to be on the database, add
                      some functional tests, and then just don't touch it unless it's important
                      enough to test on the production servers....





                      --
                      D. André Dhondt
                      mobile: 001 33 671 034 984
                      http://dhondtsayitsagile.blogspot.com/

                      Support low-cost conferences -- http://agiletour.org/
                      If you're in the area, join Agile Philly http://www.AgilePhilly.com
                      Mentor/be mentored: the Agile Skills Project
                      https://sites.google.com/site/agileskillsprojectwiki/


                      [Non-text portions of this message have been removed]
                    • Phlip
                      ... This is why one Ward Cunningham just tweeted Estimating is the non-problem that know-nothings spent decades trying to solve. I would have been more
                      Message 10 of 18 , Jan 9, 2010
                      • 0 Attachment
                        loicmidy wrote:

                        > Our current methodology is faterfall. Here are ours main problems with it:
                        > 1 : we have a long study phase at the beginning of each project : between 1 year and 1 year and a half. Consequences : the duration of the project is long + the managers like me have little visibility during this phase. I hope to slash this phase to 6 month at maximum.
                        > 2 : our customer are our statisticiens and they often ask for features that in fact they don't nead (we see that in the end). By giving frequent feedback to ours statisticiens and by prioritizinf features I hope we won't developp useless features.

                        This is why one Ward Cunningham just tweeted "Estimating is the non-problem that
                        know-nothings spent decades trying to solve."

                        I would have been more gentle to your waterfallers...
                      • Adam Sroka
                        ... Yeah. I had to retweet that one myself. ... I sat in on a meeting yesterday (Fri) where one of the teams my colleague is coaching was trying to understand
                        Message 11 of 18 , Jan 9, 2010
                        • 0 Attachment
                          On Sat, Jan 9, 2010 at 10:14 PM, Phlip <phlip2005@...> wrote:
                          >
                          >
                          >
                          > loicmidy wrote:
                          >
                          > > Our current methodology is faterfall. Here are ours main problems with it:
                          > > 1 : we have a long study phase at the beginning of each project : between 1 year and 1 year and a half. Consequences : the duration of the project is long + the managers like me have little visibility during this phase. I hope to slash this phase to 6 month at maximum.
                          > > 2 : our customer are our statisticiens and they often ask for features that in fact they don't nead (we see that in the end). By giving frequent feedback to ours statisticiens and by prioritizinf features I hope we won't developp useless features.
                          >
                          > This is why one Ward Cunningham just tweeted "Estimating is the non-problem that
                          > know-nothings spent decades trying to solve."
                          >

                          Yeah. I had to retweet that one myself.

                          > I would have been more gentle to your waterfallers...
                          >

                          I sat in on a meeting yesterday (Fri) where one of the teams my
                          colleague is coaching was trying to understand how to estimate. It was
                          very painful. I made the point that they needed to do their best
                          according to what they already seemed to understand, that their
                          estimates would still be wrong, but that it would all work out over
                          time. They looked at me like I was speaking some alien language (Which
                          I suppose isn't far from the truth.)

                          >
                        Your message has been successfully submitted and would be delivered to recipients shortly.