Loading ...
Sorry, an error occurred while loading the content.

Datatable, Datasource and dynamically retrieving MORE data through pagination?

Expand Messages
  • Michael Hasenstein
    Hi, Does anyone know of a good example for $subject? I know the DT (and SDT) YUI source itself very well by now, but before looking at three large components
    Message 1 of 16 , Mar 26, 2009
    • 0 Attachment
      Hi,

      Does anyone know of a good example for $subject?

      I know the DT (and SDT) YUI source itself very well by now, but before looking at three large components to find my answers...

      I already built a (pretty large, dynamically self-generating based upon metadata from the server) subclass of SDT, I my need to do the same for the paginator (overwriting some methods with my own), but hopefully not for the Datasource too.

      I've a server that initially only sends a small subset of the data to my datasource (attached to a datatable). Encoded in the returned JSON is information how to retrieve the next record. Initially the DT shows the first set of data fully, and pressing "next" in the paginator should cause the Datasource to ask the server for the next set of records, etc.

      Okay, a simpler example will suffice - one where not all data is retrieved by the Datasource at once, but where more requests are made to the server as the user scrolls through the data.

      Thanks
      Michael
    • Michael Hasenstein
      I should add that I have found generateRequest in DataTable, but it never gets called. I set the paginator like this (no difference when placed in the config
      Message 2 of 16 , Mar 26, 2009
      • 0 Attachment
        I should add that I have found generateRequest in DataTable, but it never gets called.

        I set the paginator like this (no difference when placed in the config object for the constructor, by the way):

        this.setAttributes({
        paginator : new YAHOO.widget.Paginator({
        rowsPerPage: 2,
        containers: Dom.getElementsByClassName('browsetablefooter', 'div', this.getTfootEl()),
        totalRecords: YAHOO.widget.Paginator.VALUE_UNLIMITED
        })
        });

        The problem: I still see only two pages to paginate through instead of "unlimited" (also tried numeric setting of 100).

        Why does my paginator only show the few records received in the initial request even though I'm telling it there are more?




        --- In ydn-javascript@yahoogroups.com, I wrote:
        >
        > Hi,
        >
        > Does anyone know of a good example for $subject?
        >
        > I know the DT (and SDT) YUI source itself very well by now, but before looking at three large components to find my answers...
      • Michael Hasenstein
        (Subject was: Re: Datatable, Datasource and dynamically retrieving MORE data through pagination? ) I just found something in datatable.js that seems to make
        Message 3 of 16 , Mar 26, 2009
        • 0 Attachment
          (Subject was: "Re: Datatable, Datasource and dynamically retrieving MORE data through pagination?")

          I just found something in datatable.js that seems to make no sense at all, because it makes the while idea of dynamically loading data through datasource initiated by pagination impossible!

          Function _handleDataReturnPayload, which is called each time the datasource delivers data to the datatable, overwrites the "totalRecords" property of the paginator with the amount of records actually held.

          But that would mean I can NEVER have more records than the datatable already sees! So, if I have 10000 records total but only 100 have been loaded to the datatable, it is impossible to show to the user that there are 900 more???

          What am I missing?
        • Michael Hasenstein
          By the way, while if option dynamicData is true the totalRecords of the payload are used, that option is NOT what I want. It removes all previous records.
          Message 4 of 16 , Mar 26, 2009
          • 0 Attachment
            By the way, while if option "dynamicData" is true the totalRecords of the payload are used, that option is NOT what I want. It removes all previous records.

            What I want is to lazy-load data from the server into the DT and that seems impossible by design? That would be a MAJOR disappointment, what is such a device for after all.. :-(



            --- In ydn-javascript@yahoogroups.com, "Michael Hasenstein" <hasenstein@...> wrote:
            >
            > (Subject was: "Re: Datatable, Datasource and dynamically retrieving MORE data through pagination?")
            >
            > I just found something in datatable.js that seems to make no sense at all, because it makes the while idea of dynamically loading data through datasource initiated by pagination impossible!
            >
            > Function _handleDataReturnPayload, which is called each time the datasource delivers data to the datatable, overwrites the "totalRecords" property of the paginator with the amount of records actually held.
            >
            > But that would mean I can NEVER have more records than the datatable already sees! So, if I have 10000 records total but only 100 have been loaded to the datatable, it is impossible to show to the user that there are 900 more???
            >
            > What am I missing?
            >
          • Frank Dietrich
            ... You can add, modify, delete records at any time.
            Message 5 of 16 , Mar 26, 2009
            • 0 Attachment
              >By the way, while if option "dynamicData" is true the totalRecords of the
              >payload are used, that option is NOT what I want. It removes all previous
              >records.
              >
              >What I want is to lazy-load data from the server into the DT and that
              >seems impossible by design? That would be a MAJOR disappointment, what is
              >such a device for after all.. :-(


              You can add, modify, delete records at any time.
            • Michael Hasenstein
              ... That is not my question. Of course I can do it *manually*. My scenario is a lazy-loading of data through a datasource object into the datatable - and
              Message 6 of 16 , Mar 26, 2009
              • 0 Attachment
                > You can add, modify, delete records at any time.


                That is not my question. Of course I can do it *manually*. My scenario is a "lazy-loading" of data through a datasource object into the datatable - and that is not far-fetched but a pretty obvious use IMHO.
              • Frank Dietrich
                ... Things we need often seem to be highly important and obviously usefull. However, watching this group for about a year now, I can t remember having seen
                Message 7 of 16 , Mar 27, 2009
                • 0 Attachment
                  >> You can add, modify, delete records at any time.
                  >
                  >
                  >That is not my question. Of course I can do it *manually*. My scenario is
                  >a "lazy-loading" of data through a datasource object into the datatable -
                  >and that is not far-fetched but a pretty obvious use IMHO.

                  Things we need often seem to be highly important and obviously usefull.

                  However, watching this group for about a year now, I can't remember having
                  seen this question just once yet. Other questions are coming up over and over
                  again.

                  Surely it would be a nice feature to load the first ten records and fetch the other
                  9000 in the background. But how about sorting, pagination etc.? Probably no factor
                  in Your case, but issues that would need addressing for a component supporting this
                  kind of dataloading. Rendering already is a big thing etc.

                  If You need it for Your specific case, it's not big thing to write a function that does it.
                  YUI gives You all the necessary low-level methods at hand.
                • agentorangejdm
                  ... I don t see why this is so hard? on initial load load the first 10, then on the data return event fire off to get the next 5000 or whatever, and append
                  Message 8 of 16 , Mar 27, 2009
                  • 0 Attachment
                    --- In ydn-javascript@yahoogroups.com, "Frank Dietrich" <fdietrich@...> wrote:
                    >
                    > >> You can add, modify, delete records at any time.
                    > >
                    > >
                    > >That is not my question. Of course I can do it *manually*. My scenario is
                    > >a "lazy-loading" of data through a datasource object into the datatable -
                    > >and that is not far-fetched but a pretty obvious use IMHO.
                    >
                    > Things we need often seem to be highly important and obviously usefull.
                    >
                    > However, watching this group for about a year now, I can't remember having
                    > seen this question just once yet. Other questions are coming up over and over
                    > again.
                    >
                    > Surely it would be a nice feature to load the first ten records and fetch the other
                    > 9000 in the background. But how about sorting, pagination etc.? Probably no factor
                    > in Your case, but issues that would need addressing for a component supporting this
                    > kind of dataloading. Rendering already is a big thing etc.
                    >
                    > If You need it for Your specific case, it's not big thing to write a function that does it.
                    > YUI gives You all the necessary low-level methods at hand.
                    >


                    I don't see why this is so hard? on initial load load the first 10, then on the data return event fire off to get the next 5000 or whatever, and append that data to the table? The only problem i would see with that is that you might be doing extra work when the user hits a sort, then the table will try to append, but then get the next request to load (and flush the table) the new sorted data, which may not be such a huge issue if the other 5000 is also paginated etc ..
                  • Lucas Smith
                    ... Michael, If I m understanding your request correctly, it s not hard. Just new to the list. Try issuing a dataSource.sendRequest from a dataReturnEvent
                    Message 9 of 16 , Mar 27, 2009
                    • 0 Attachment
                      --- In ydn-javascript@yahoogroups.com, "agentorangejdm" <jarret.minkler@...> wrote:
                      >
                      > --- In ydn-javascript@yahoogroups.com, "Frank Dietrich" <fdietrich@> wrote:
                      > >
                      > > >> You can add, modify, delete records at any time.
                      > > >
                      > > >
                      > > >That is not my question. Of course I can do it *manually*. My scenario is
                      > > >a "lazy-loading" of data through a datasource object into the datatable -
                      > > >and that is not far-fetched but a pretty obvious use IMHO.
                      > >
                      > > Things we need often seem to be highly important and obviously usefull.
                      > >
                      > > However, watching this group for about a year now, I can't remember having
                      > > seen this question just once yet. Other questions are coming up over and over
                      > > again.
                      > >
                      > > Surely it would be a nice feature to load the first ten records and fetch the other
                      > > 9000 in the background. But how about sorting, pagination etc.? Probably no factor
                      > > in Your case, but issues that would need addressing for a component supporting this
                      > > kind of dataloading. Rendering already is a big thing etc.
                      > >
                      > > If You need it for Your specific case, it's not big thing to write a function that does it.
                      > > YUI gives You all the necessary low-level methods at hand.
                      > >
                      >
                      >
                      > I don't see why this is so hard? on initial load load the first 10, then on the data return event fire off to get the next 5000 or whatever, and append that data to the table? The only problem i would see with that is that you might be doing extra work when the user hits a sort, then the table will try to append, but then get the next request to load (and flush the table) the new sorted data, which may not be such a huge issue if the other 5000 is also paginated etc ..
                      >

                      Michael,

                      If I'm understanding your request correctly, it's not hard. Just new to the list.

                      Try issuing a dataSource.sendRequest from a dataReturnEvent subscriber. Pass to the sendRequest a custom callback handler that turns off the DataTable's dynamicData flag (since presumably you have all the data now) and calls addRows(all_the_new_stuff) to append the rest of the data to the DataTable's RecordSet.

                      Here's an example:
                      http://gist.github.com/86823

                      Hope this helps,
                      Luke
                    • starvingprogrammer
                      I am not sure if this is what you are asking, but if this dynamic data is coming from your own source then you can have it return the total records (not just
                      Message 10 of 16 , Mar 27, 2009
                      • 0 Attachment
                        I am not sure if this is what you are asking, but if this dynamic data
                        is coming from your own source then you can have it return the total
                        records (not just the records returned in this page request) in the meta
                        data. Then overload DataTable.handleDataReturnPayload like this:
                        myDataTable.handleDataReturnPayload = function(oRequest, oResponse,
                        oPayload) {
                        oPayload.totalRecords = oResponse.meta.totalRecords;
                        return oPayload;
                        }

                        In regards to the lazy loading, you can have your DataSource return more
                        rows than what your Paginator puts on a single page. So if you have 15
                        rows per page, the DataSource can return 150. The key here is that
                        these extra rows will be ignored unless you overload
                        DataSource.addToCache. The implementation is just loop through the
                        results splicing off page-sized chunks from oResponse.results while
                        spoofing the oRequest lines so when you add each chunk to aCache, the
                        Paginator will continue to work as expected.

                        Brian

                        --- In ydn-javascript@yahoogroups.com, "Michael Hasenstein"
                        <hasenstein@...> wrote:
                        >
                        > By the way, while if option "dynamicData" is true the totalRecords of
                        the payload are used, that option is NOT what I want. It removes all
                        previous records.
                        >
                        > What I want is to lazy-load data from the server into the DT and that
                        seems impossible by design? That would be a MAJOR disappointment, what
                        is such a device for after all.. :-(
                        >
                        >
                        >
                        > --- In ydn-javascript@yahoogroups.com, "Michael Hasenstein"
                        hasenstein@ wrote:
                        > >
                        > > (Subject was: "Re: Datatable, Datasource and dynamically retrieving
                        MORE data through pagination?")
                        > >
                        > > I just found something in datatable.js that seems to make no sense
                        at all, because it makes the while idea of dynamically loading data
                        through datasource initiated by pagination impossible!
                        > >
                        > > Function _handleDataReturnPayload, which is called each time the
                        datasource delivers data to the datatable, overwrites the "totalRecords"
                        property of the paginator with the amount of records actually held.
                        > >
                        > > But that would mean I can NEVER have more records than the datatable
                        already sees! So, if I have 10000 records total but only 100 have been
                        loaded to the datatable, it is impossible to show to the user that there
                        are 900 more???
                        > >
                        > > What am I missing?
                        > >
                        >
                      • Michael Hasenstein
                        ... Found something interesting in datatable.js, hang on... I can t, the database cannot give me total nr. Don t ask me why, I ve no access to the backend and
                        Message 11 of 16 , Mar 27, 2009
                        • 0 Attachment
                          --- In ydn-javascript@yahoogroups.com, "starvingprogrammer"  wrote:
                          > I am not sure if this is what you are asking, but if this dynamic data
                          > is coming from your own source then you can have it return the total
                          > records
                          >....

                          Found something interesting in datatable.js, hang on...

                          I can't, the database cannot give me total nr. Don't ask me why, I've no access to the backend and only see my own little piece. I'm not too concerned, it's only a (well-paid) 3-week job for a customer who produces software for municipalities. Anyway, thanks for the code, but for that reason it won't work. There are more complications that I won't go into, the customer handed me a lengthy list of things they'd like to have and it is stretching what's possible to do starting with the standard DataSource, ScrollingDataTable and Paginator widgets in just three weeks... (but I'm not concerned or in a rush)

                          By the way guys, it IS complicated, and I know the YUI code very well by now and don't hesitate to call myself "expert" - and to proof it. Doesn't mean I know every corner of the YUI, I only look into it as needed. I discovered something very interesting today about $subject:
                          (In datatable.js) there are four  different handlers that could be used to handle data coming from the DataSource:
                          • onDataReturnReplaceRows
                          • onDataReturnAppendRows
                          • onDataReturnInsertRows
                          • onDataReturnUpdateRows
                          • onDataReturnSetRows
                          Only the last one is ever used. It is hard-wired in the datatable and there's no way (except by overwriting the methods or changing the code) to use the other ones. They are not used anywhere else. In sortColumn(), in onPaginatorChangeRequest(), onDataReturnInitializeTable() the one handler is used.

                          There is no documentation of this. So what I see here is a very much work-in-progress code.

                          Please make no mistake, I definitely don't complain! It's more like taking stock of where we are in JS library development right now. I still love YUI, whenever I use jQuery I'm amazed at how great the core-jQuery is for small in-page things - and how much I dislike the jQuery-UI stuff compared to YUI's widget quality. And I do participate in improving YUI, thus far all my bug reports for various components have always been accepted (and I try to include a solution whenever I can) :-)

                          I haven't looked at those handlers in details or thought about it since I've other things to do right now, but thought it might be interesting for someone. Maybe there even is a person here who knows more about this "dead code"?


                          > In regards to the lazy loading, you can have your DataSource return more
                          > rows than what your Paginator puts on a single page. So if you have 15
                          > rows per page, the DataSource can return 150. The key here is that
                          > these extra rows will be ignored unless you overload
                          > DataSource.addToCache.

                          Thanks, interesting pointer. I'll keep that in mind! The last two weeks I dissected datatable.js, tried to avoid looking at even more code all at once :-)  It's the first time I cannot just use the standard DT features.


                          > The implementation is just loop through the
                          > results splicing off page-sized chunks from oResponse.results while
                          > spoofing the oRequest lines so when you add each chunk to aCache, the
                          > Paginator will continue to work as expected.

                          Thanks!

                          Michael
                        • Lucas Smith
                          ... [snip] The dynamicData config relies on you being able to tell the Paginator what the totalRecords is. Paginator includes a static VALUE_UNLIMITED that
                          Message 12 of 16 , Mar 28, 2009
                          • 0 Attachment
                            --- In ydn-javascript@yahoogroups.com, "Michael Hasenstein" <hasenstein@...> wrote:
                            >
                            > --- In ydn-javascript@yahoogroups.com, "starvingprogrammer" wrote:
                            > > I am not sure if this is what you are asking, but if this dynamic data
                            > > is coming from your own source then you can have it return the total
                            > > records
                            > >....
                            >
                            > Found something interesting in datatable.js, hang on...
                            >
                            > I can't, the database cannot give me total nr. Don't ask me why, I've no
                            > access to the backend and only see my own little piece.
                            [snip]

                            The dynamicData config relies on you being able to tell the Paginator what the totalRecords is. Paginator includes a static VALUE_UNLIMITED that you can pass for totalRecords, and the UI components will render and respond as if there were no (known) end to the incoming records.

                            In your DataTable instance's handleDataReturnPayload, set payload.totalRecords to YAHOO.widget.Paginator.VALUE_UNLIMITED until you can determine that you've reached the end of the server's data set.

                            > (In datatable.js) there are four different handlers that could be used
                            > to handle data coming from the DataSource:
                            >
                            > * onDataReturnReplaceRows
                            > * onDataReturnAppendRows
                            > * onDataReturnInsertRows
                            > * onDataReturnUpdateRows
                            > * onDataReturnSetRows
                            >
                            > Only the last one is ever used. It is hard-wired in the datatable and
                            > there's no way (except by overwriting the methods or changing the code)
                            > to use the other ones. They are not used anywhere else. In sortColumn(),
                            > in onPaginatorChangeRequest(), onDataReturnInitializeTable() the one
                            > handler is used.
                            >
                            > There is no documentation of this. So what I see here is a very much
                            > work-in-progress code.
                            >
                            [snip]

                            In addition to the four[sic] you've listed, there is also onDataReturnInitializeTable. These individual callbacks are provided as convenience methods for common record operations performed by implementers via DataSource sendRequest's callback param. Generally, they don't make sense in the context of paging or sorting.

                            The infrastructure for the data io was in place before some were added, and there has been little call (not zero, mind you) to add a level of configurability to the DataSource sendRequest io bottled up in the sorting and paging methods. If you'd like to add your voice, please submit a feature request (http://yuilibrary.com/projects/yui2/newticket) or If you'd prefer, you can fork the code from github and submit a patch.

                            http://developer.yahoo.com/yui/articles/reportingbugs/

                            [snip]
                            > And I do participate
                            > in improving YUI, thus far all my bug reports for various components
                            > have always been accepted (and I try to include a solution whenever I
                            > can) :-)

                            Thank you very much for your participation! I hope you're enjoying YUI's move to github. Have you considered signing a CLA and contributing via fork, patch, pull request?

                            Hope this helps,
                            Luke
                          • Michael Hasenstein
                            Thanks, I ll look into it some more next week. I have considered contributing, and I sure do like the move to git and github. (Git) I just hope they get the
                            Message 13 of 16 , Mar 29, 2009
                            • 0 Attachment
                              Thanks, I'll look into it some more next week.

                              I have considered contributing, and I sure do like the move to git and
                              github. (Git) I just hope they get the performance and stability issues
                              sorted out with the new YUI website. When I submitted a bugreport last
                              week I often couldn't get to the tickets, or it took a long time to
                              load.

                              Thanks for the pointer to the CLA, I might consider it. I learned to
                              love Javascript, it's so amazingly powerful and soooo much nicer and
                              orders of magnitude more elegant than Ruby (to compare it with one of
                              the other languages out there that have some mass-appeal and have strong
                              functional elements); and the YUI seems to me the best JS library for
                              large projects by far, for reasons of its design philosophy and
                              implementation. YUI3 looks appealing in that respect too.

                              Michael



                              --- In ydn-javascript@yahoogroups.com, "Lucas Smith" wrote:
                              [...]
                              > The dynamicData config relies on you being able to tell the Paginator
                              > what the totalRecords is. Paginator includes a static VALUE_UNLIMITED
                              > that you can pass for totalRecords, and the UI components will render
                              and
                              > respond as if there were no (known) end to the incoming records.
                              >
                              > In your DataTable instance's handleDataReturnPayload, set
                              payload.totalRecords
                              > to YAHOO.widget.Paginator.VALUE_UNLIMITED until you can determine that
                              > you've reached the end of the server's data set.
                              [...]
                              > The infrastructure for the data io was in place before some were
                              added, and there has
                              > been little call (not zero, mind you) to add a level of
                              configurability to the DataSource
                              > sendRequest io bottled up in the sorting and paging methods. If you'd
                              like to add your
                              > voice, please submit a feature request
                              (http://yuilibrary.com/projects/yui2/newticket)
                              > or If you'd prefer, you can fork the code from github and submit a
                              patch.
                              >
                              > http://developer.yahoo.com/yui/articles/reportingbugs/
                              [...]
                              > Thank you very much for your participation! I hope you're enjoying
                              YUI's move to
                              > github. Have you considered signing a CLA and contributing via fork,
                              patch, pull request?
                            • jennykhan
                              Hi Michael, As you already know, the dynamicData feature is fundamentally not what you are looking for. Since you don t have access to the total number of
                              Message 14 of 16 , Mar 30, 2009
                              • 0 Attachment
                                Hi Michael,

                                As you already know, the "dynamicData" feature is fundamentally not what you are looking for. Since you don't have access to the total number of records *and* you want to accumulate records in memory, you'll need to be wary of overloading the browser. That said, I recommend exploring the ideas presented below as a simplified solution. You can, of course, feel free to fully customize the behavior to suit your specific needs as you described earlier.

                                Cheers,
                                Jenny



                                --- In ydn-javascript@yahoogroups.com, "agentorangejdm" <jarret.minkler@...> wrote:
                                >
                                > I don't see why this is so hard? on initial load load the first 10, then on the data return event fire off to get the next 5000 or whatever, and append that data to the table? The only problem i would see with that is that you might be doing extra work when the user hits a sort, then the table will try to append, but then get the next request to load (and flush the table) the new sorted data, which may not be such a huge issue if the other 5000 is also paginated etc ..
                                >
                              • Michael Hasenstein
                                Thanks Jenny - but there is no below in your message, at least I can t see anything :-) Yes, JS-caching without knowing how much memory there is is a
                                Message 15 of 16 , Mar 31, 2009
                                • 0 Attachment
                                  Thanks Jenny - but there is no "below" in your message, at least I can't see anything :-)

                                  Yes, JS-caching without knowing how much memory there is is a (potentially deadly) shot in the dark. Well, I could ask the user some questions about their hardware on startup ;-)


                                  --- In ydn-javascript@yahoogroups.com, "jennykhan" <jennyhan@...> wrote:
                                  > Hi Michael,
                                  >
                                  > As you already know, the "dynamicData" feature is fundamentally not what you are looking for. Since you don't have access to the total number of records *and* you want to accumulate records in memory, you'll need to be wary of overloading the browser. That said, I recommend exploring the ideas presented below as a simplified solution. You can, of course, feel free to fully customize the behavior to suit your specific needs as you described earlier.
                                  >
                                  > Cheers,
                                  > Jenny
                                • jennykhan
                                  Oops, I had meant to paste in agentorangejdm s suggestion: on initial load load the first 10, then on the data return event fire off to get the next 5000 or
                                  Message 16 of 16 , Mar 31, 2009
                                  • 0 Attachment
                                    Oops, I had meant to paste in agentorangejdm's suggestion:

                                    "on initial load load the first 10, then on the data return event fire off to get the next 5000 or whatever, and append that data to the table? The only problem i would see with that is that you might be doing extra work when the user hits a sort, then the table will try to append, but then get the next request to load (and flush the table) the new sorted data, which may not be such a huge issue if the other 5000 is also paginated etc .."



                                    --- In ydn-javascript@yahoogroups.com, "Michael Hasenstein" <hasenstein@...> wrote:
                                    >
                                    > Thanks Jenny - but there is no "below" in your message, at least I can't see anything :-)
                                    >
                                    > Yes, JS-caching without knowing how much memory there is is a (potentially deadly) shot in the dark. Well, I could ask the user some questions about their hardware on startup ;-)
                                    >
                                    >
                                    > --- In ydn-javascript@yahoogroups.com, "jennykhan" <jennyhan@> wrote:
                                    > > Hi Michael,
                                    > >
                                    > > As you already know, the "dynamicData" feature is fundamentally not what you are looking for. Since you don't have access to the total number of records *and* you want to accumulate records in memory, you'll need to be wary of overloading the browser. That said, I recommend exploring the ideas presented below as a simplified solution. You can, of course, feel free to fully customize the behavior to suit your specific needs as you described earlier.
                                    > >
                                    > > Cheers,
                                    > > Jenny
                                    >
                                  Your message has been successfully submitted and would be delivered to recipients shortly.