Loading ...
Sorry, an error occurred while loading the content.

Re: Alternative storage drivers?

Expand Messages
  • alitchfield
    Hi, We have just found and fallen in love with Taffy DB it s superb, fast and intuitive and almost exactly what we need. However the 5mb limit of localstorage
    Message 1 of 9 , May 3, 2012
    • 0 Attachment
      Hi,

      We have just found and fallen in love with Taffy DB it's superb, fast and intuitive and almost exactly what we need.

      However the 5mb limit of localstorage is holding us back - we'd like to go to approx 35mb.

      Regarding the previous posts are there any plans percolating?

      Also - where's the donate button!?

      Thanks,
      Andrew

      --- In taffydb@yahoogroups.com, taffydb-owner@yahoogroups.com wrote:
      >
      > Comments:
      >
      > 1. Right now the localStorage is kept in sync with JavaScript memory but Taffy only uses it for storage and not for any queries or the like.
      >
      > 2. Taffy 2.0 has a callback() method which is can optionally be used when writing a query to let the query be pushed to the end of the call stack via setTimeout. In theory one could take advantage of other kinds of async storage via the callback(), especially since it doesn't make sense to load many MBs of data into the JavaScript memory space.
      >
      > Example: db({id:33}).callback(function () {
      > //this is record set
      > })
      >
      > 3. I'm torn between whether to try and go the persistJS road and support a lot of different storage formats out of the box or instead if there should just be an API for registering storage formats.
      >
      > Something like:
      >
      > TAFFY.extend("store","localStorage",{
      > storageInit:function (name) {
      > // setup the storage with the "name" provided by the dev
      > },
      > onDBInit:function () {
      > // load data from localStorage
      > },
      > onDataChange:function () {
      > // save changes to localStorage
      > }
      > });
      >
      >
      >
      >
      > --- In taffydb@yahoogroups.com, Arieh Glazer <arieh.glazer@> wrote:
      > >
      > > Hey
      > > What I wanted to know is what are the requierments of such a driver- what
      > > methods it must have, how do I register it etc.
      > > When I approached this earlier this year, I did this:
      > >
      > > - I had a basic object using localStorage, and then extended it with new
      > > drivers.
      > > - Each driver had a test variable that would say if the browser
      > > supported it
      > > - I had a driver stack, so on initialization, if no options was set, I
      > > would iterate and choose the first that passed it's test
      > >
      > > The main issue to note though is that unlike localStorage, all other
      > > storage solutions are async - which means they return their results through
      > > callbacks - so if the API isn't async it won't work (requires quite a
      > > change to the API).
      > >
      > > On Thu, Nov 10, 2011 at 8:38 PM, <taffydb-owner@yahoogroups.com> wrote:
      > >
      > > > **
      > > >
      > > >
      > > > Hi Arieh,
      > > >
      > > > Supporting other types of storage would be awesome! Right now the
      > > > localstorage is there because it was very easy to add and requested but I
      > > > suspect it isn't widely used if at all. Ideally "storage" would be a good
      > > > candidate to extra either as a plugin or at least another .js file.
      > > >
      > > > The interface from a developer standpoint is very simple right now:
      > > >
      > > > db.store("name");
      > > >
      > > > The internals are pretty simple as well. Let me know what you are
      > > > thinking. Perhaps we could a traffy-store.js file that would be could be
      > > > included to add storage options to your script and then .store() can take
      > > > either the name you are using (.store("name")) which will use the best
      > > > guess at storage or you can specify directly
      > > > (.store("localStorage","name")).
      > > >
      > > > I'd also love to have support for storage options on nodejs. At the very
      > > > least memory or the file system but perhaps there is an elegant way to
      > > > extend it to databases as well.
      > > >
      > > > Looking forward to your comments,
      > > >
      > > > Ian
      > > >
      > > >
      > > > --- In taffydb@yahoogroups.com, Arieh Glazer <arieh.glazer@> wrote:
      > > > >
      > > > > Hey
      > > > > I was wondering if there were plans to support alternative storage
      > > > drivers
      > > > > other than localStorage (such as SQLite and IndexedDB).
      > > > > I love the interface Taffy provides, but find that 5MB is a very limited
      > > > > amount of data for a live application (considering most browsers offer
      > > > much
      > > > > more).
      > > > >
      > > > > I would love to help write such drivers btw, just not sure how to plug it
      > > > > in (if there even a support for this in the core). Is there an interface
      > > > > that needs to be provided?
      > > > >
      > > > > --
      > > > > Arieh Glazer
      > > > > 052-5348-561
      > > > > http://www.arieh.co.il
      > > > > http://www.link-wd.co.il
      > > > >
      > > >
      > > >
      > > >
      > >
      > >
      > >
      > > --
      > > Arieh Glazer
      > > ארי×" ×'לזר
      > > 052-5348-561
      > > http://www.arieh.co.il
      > > http://www.link-wd.co.il
      > >
      >
    • tacoman_cool
      Hey Andrew, Can you give us a little more detail about what you are trying to do that will take up to 35mbs? You could, in theory, break up the data somehow
      Message 2 of 9 , May 4, 2012
      • 0 Attachment
        Hey Andrew,

        Can you give us a little more detail about what you are trying to do that will take up to 35mbs? You could, in theory, break up the data somehow but it sounds like a lot for a browser by itself to be doing.

        Ian

        --- In taffydb@yahoogroups.com, "alitchfield" <ajl236@...> wrote:
        >
        > Hi,
        >
        > We have just found and fallen in love with Taffy DB it's superb, fast and intuitive and almost exactly what we need.
        >
        > However the 5mb limit of localstorage is holding us back - we'd like to go to approx 35mb.
        >
        > Regarding the previous posts are there any plans percolating?
        >
        > Also - where's the donate button!?
        >
        > Thanks,
        > Andrew
        >
        > --- In taffydb@yahoogroups.com, taffydb-owner@yahoogroups.com wrote:
        > >
        > > Comments:
        > >
        > > 1. Right now the localStorage is kept in sync with JavaScript memory but Taffy only uses it for storage and not for any queries or the like.
        > >
        > > 2. Taffy 2.0 has a callback() method which is can optionally be used when writing a query to let the query be pushed to the end of the call stack via setTimeout. In theory one could take advantage of other kinds of async storage via the callback(), especially since it doesn't make sense to load many MBs of data into the JavaScript memory space.
        > >
        > > Example: db({id:33}).callback(function () {
        > > //this is record set
        > > })
        > >
        > > 3. I'm torn between whether to try and go the persistJS road and support a lot of different storage formats out of the box or instead if there should just be an API for registering storage formats.
        > >
        > > Something like:
        > >
        > > TAFFY.extend("store","localStorage",{
        > > storageInit:function (name) {
        > > // setup the storage with the "name" provided by the dev
        > > },
        > > onDBInit:function () {
        > > // load data from localStorage
        > > },
        > > onDataChange:function () {
        > > // save changes to localStorage
        > > }
        > > });
        > >
        > >
        > >
        > >
        > > --- In taffydb@yahoogroups.com, Arieh Glazer <arieh.glazer@> wrote:
        > > >
        > > > Hey
        > > > What I wanted to know is what are the requierments of such a driver- what
        > > > methods it must have, how do I register it etc.
        > > > When I approached this earlier this year, I did this:
        > > >
        > > > - I had a basic object using localStorage, and then extended it with new
        > > > drivers.
        > > > - Each driver had a test variable that would say if the browser
        > > > supported it
        > > > - I had a driver stack, so on initialization, if no options was set, I
        > > > would iterate and choose the first that passed it's test
        > > >
        > > > The main issue to note though is that unlike localStorage, all other
        > > > storage solutions are async - which means they return their results through
        > > > callbacks - so if the API isn't async it won't work (requires quite a
        > > > change to the API).
        > > >
        > > > On Thu, Nov 10, 2011 at 8:38 PM, <taffydb-owner@yahoogroups.com> wrote:
        > > >
        > > > > **
        > > > >
        > > > >
        > > > > Hi Arieh,
        > > > >
        > > > > Supporting other types of storage would be awesome! Right now the
        > > > > localstorage is there because it was very easy to add and requested but I
        > > > > suspect it isn't widely used if at all. Ideally "storage" would be a good
        > > > > candidate to extra either as a plugin or at least another .js file.
        > > > >
        > > > > The interface from a developer standpoint is very simple right now:
        > > > >
        > > > > db.store("name");
        > > > >
        > > > > The internals are pretty simple as well. Let me know what you are
        > > > > thinking. Perhaps we could a traffy-store.js file that would be could be
        > > > > included to add storage options to your script and then .store() can take
        > > > > either the name you are using (.store("name")) which will use the best
        > > > > guess at storage or you can specify directly
        > > > > (.store("localStorage","name")).
        > > > >
        > > > > I'd also love to have support for storage options on nodejs. At the very
        > > > > least memory or the file system but perhaps there is an elegant way to
        > > > > extend it to databases as well.
        > > > >
        > > > > Looking forward to your comments,
        > > > >
        > > > > Ian
        > > > >
        > > > >
        > > > > --- In taffydb@yahoogroups.com, Arieh Glazer <arieh.glazer@> wrote:
        > > > > >
        > > > > > Hey
        > > > > > I was wondering if there were plans to support alternative storage
        > > > > drivers
        > > > > > other than localStorage (such as SQLite and IndexedDB).
        > > > > > I love the interface Taffy provides, but find that 5MB is a very limited
        > > > > > amount of data for a live application (considering most browsers offer
        > > > > much
        > > > > > more).
        > > > > >
        > > > > > I would love to help write such drivers btw, just not sure how to plug it
        > > > > > in (if there even a support for this in the core). Is there an interface
        > > > > > that needs to be provided?
        > > > > >
        > > > > > --
        > > > > > Arieh Glazer
        > > > > > 052-5348-561
        > > > > > http://www.arieh.co.il
        > > > > > http://www.link-wd.co.il
        > > > > >
        > > > >
        > > > >
        > > > >
        > > >
        > > >
        > > >
        > > > --
        > > > Arieh Glazer
        > > > ארי×" ×'לזר
        > > > 052-5348-561
        > > > http://www.arieh.co.il
        > > > http://www.link-wd.co.il
        > > >
        > >
        >
      • alitchfield
        Hi, We have a mobile HTML5 app that helps our field engineers view and edit data offline in areas often without 3G data. 35MB is a lot of data however HTML5
        Message 3 of 9 , May 5, 2012
        • 0 Attachment
          Hi,

          We have a mobile HTML5 app that helps our field engineers view and edit data offline in areas often without 3G data.

          35MB is a lot of data however HTML5 WebSQL works very well and the app has been setup of hundreds of Apple and Android smartphones and tablets over the last two years.

          HTML5 WebSQL is now deprecated and Apple is not yet supporting IndexedDB so I need another solution.

          It's my understanding that HTML5 key/value localstorage that TaffyDB uses supports 5Mb of storage but unfortuneately our business logic requires the 35MB.

          Thanks,


          --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
          >
          > Hey Andrew,
          >
          > Can you give us a little more detail about what you are trying to do that will take up to 35mbs? You could, in theory, break up the data somehow but it sounds like a lot for a browser by itself to be doing.
          >
          > Ian
          >
          > --- In taffydb@yahoogroups.com, "alitchfield" <ajl236@> wrote:
          > >
          > > Hi,
          > >
          > > We have just found and fallen in love with Taffy DB it's superb, fast and intuitive and almost exactly what we need.
          > >
          > > However the 5mb limit of localstorage is holding us back - we'd like to go to approx 35mb.
          > >
          > > Regarding the previous posts are there any plans percolating?
          > >
          > > Also - where's the donate button!?
          > >
          > > Thanks,
          > > Andrew
          > >
          > > --- In taffydb@yahoogroups.com, taffydb-owner@yahoogroups.com wrote:
          > > >
          > > > Comments:
          > > >
          > > > 1. Right now the localStorage is kept in sync with JavaScript memory but Taffy only uses it for storage and not for any queries or the like.
          > > >
          > > > 2. Taffy 2.0 has a callback() method which is can optionally be used when writing a query to let the query be pushed to the end of the call stack via setTimeout. In theory one could take advantage of other kinds of async storage via the callback(), especially since it doesn't make sense to load many MBs of data into the JavaScript memory space.
          > > >
          > > > Example: db({id:33}).callback(function () {
          > > > //this is record set
          > > > })
          > > >
          > > > 3. I'm torn between whether to try and go the persistJS road and support a lot of different storage formats out of the box or instead if there should just be an API for registering storage formats.
          > > >
          > > > Something like:
          > > >
          > > > TAFFY.extend("store","localStorage",{
          > > > storageInit:function (name) {
          > > > // setup the storage with the "name" provided by the dev
          > > > },
          > > > onDBInit:function () {
          > > > // load data from localStorage
          > > > },
          > > > onDataChange:function () {
          > > > // save changes to localStorage
          > > > }
          > > > });
          > > >
          > > >
          > > >
          > > >
          > > > --- In taffydb@yahoogroups.com, Arieh Glazer <arieh.glazer@> wrote:
          > > > >
          > > > > Hey
          > > > > What I wanted to know is what are the requierments of such a driver- what
          > > > > methods it must have, how do I register it etc.
          > > > > When I approached this earlier this year, I did this:
          > > > >
          > > > > - I had a basic object using localStorage, and then extended it with new
          > > > > drivers.
          > > > > - Each driver had a test variable that would say if the browser
          > > > > supported it
          > > > > - I had a driver stack, so on initialization, if no options was set, I
          > > > > would iterate and choose the first that passed it's test
          > > > >
          > > > > The main issue to note though is that unlike localStorage, all other
          > > > > storage solutions are async - which means they return their results through
          > > > > callbacks - so if the API isn't async it won't work (requires quite a
          > > > > change to the API).
          > > > >
          > > > > On Thu, Nov 10, 2011 at 8:38 PM, <taffydb-owner@yahoogroups.com> wrote:
          > > > >
          > > > > > **
          > > > > >
          > > > > >
          > > > > > Hi Arieh,
          > > > > >
          > > > > > Supporting other types of storage would be awesome! Right now the
          > > > > > localstorage is there because it was very easy to add and requested but I
          > > > > > suspect it isn't widely used if at all. Ideally "storage" would be a good
          > > > > > candidate to extra either as a plugin or at least another .js file.
          > > > > >
          > > > > > The interface from a developer standpoint is very simple right now:
          > > > > >
          > > > > > db.store("name");
          > > > > >
          > > > > > The internals are pretty simple as well. Let me know what you are
          > > > > > thinking. Perhaps we could a traffy-store.js file that would be could be
          > > > > > included to add storage options to your script and then .store() can take
          > > > > > either the name you are using (.store("name")) which will use the best
          > > > > > guess at storage or you can specify directly
          > > > > > (.store("localStorage","name")).
          > > > > >
          > > > > > I'd also love to have support for storage options on nodejs. At the very
          > > > > > least memory or the file system but perhaps there is an elegant way to
          > > > > > extend it to databases as well.
          > > > > >
          > > > > > Looking forward to your comments,
          > > > > >
          > > > > > Ian
          > > > > >
          > > > > >
          > > > > > --- In taffydb@yahoogroups.com, Arieh Glazer <arieh.glazer@> wrote:
          > > > > > >
          > > > > > > Hey
          > > > > > > I was wondering if there were plans to support alternative storage
          > > > > > drivers
          > > > > > > other than localStorage (such as SQLite and IndexedDB).
          > > > > > > I love the interface Taffy provides, but find that 5MB is a very limited
          > > > > > > amount of data for a live application (considering most browsers offer
          > > > > > much
          > > > > > > more).
          > > > > > >
          > > > > > > I would love to help write such drivers btw, just not sure how to plug it
          > > > > > > in (if there even a support for this in the core). Is there an interface
          > > > > > > that needs to be provided?
          > > > > > >
          > > > > > > --
          > > > > > > Arieh Glazer
          > > > > > > 052-5348-561
          > > > > > > http://www.arieh.co.il
          > > > > > > http://www.link-wd.co.il
          > > > > > >
          > > > > >
          > > > > >
          > > > > >
          > > > >
          > > > >
          > > > >
          > > > > --
          > > > > Arieh Glazer
          > > > > ארי×" ×'לזר
          > > > > 052-5348-561
          > > > > http://www.arieh.co.il
          > > > > http://www.link-wd.co.il
          > > > >
          > > >
          > >
          >
        • alitchfield
          In-memory databases are of course different to disk based databases however I wanted to try using TaffyDB to achieve my business requirements. I wanted to do
          Message 4 of 9 , May 16, 2012
          • 0 Attachment
            In-memory databases are of course different to disk based databases however I wanted to try using TaffyDB to achieve my business requirements. I wanted to do this out of necessity and partly for fun to see if modern mobile browsers could handle the load, here's an update of the progress.

            After trial and error we now have 200,000 json objects (~20MB) split over an array of 70 TaffyDB's with a query time of a quarter second on iPhone 4S or HTC One X. Initially all 200,000 were in one TaffyDB but the query time was too slow, approx 5 seconds.

            We don't use the .store() method as the TaffyDB's will be initialised from IndexedDB/WebSQL/LocalStorage when the program starts and data will then be persisted back as required. The initial large dataset is downloaded via a web service.

            The program works on iOS5, Android and many more browsers than our previous deprecated HTML5 WebSQL solution. Whilst work and testing on this solution continues the outlook seems good at this stage. TaffyDB rocks.
          • Ziyahan Fakir ALBENiZ
            Hi; Did you try sqlite? ... -- Ziyahan Fakir Albeniz Bilgisayar Programcýsý *GSM :* +90 533 637 1572 *Skype :* ziyahanalbeniz *Web :*
            Message 5 of 9 , May 16, 2012
            • 0 Attachment
              Hi;
              Did you try sqlite?

              2012/5/17, alitchfield <ajl236@...>:
              >
              >
              > In-memory databases are of course different to disk based databases however
              > I wanted to try using TaffyDB to achieve my business requirements. I wanted
              > to do this out of necessity and partly for fun to see if modern mobile
              > browsers could handle the load, here's an update of the progress.
              >
              > After trial and error we now have 200,000 json objects (~20MB) split over an
              > array of 70 TaffyDB's with a query time of a quarter second on iPhone 4S or
              > HTC One X. Initially all 200,000 were in one TaffyDB but the query time was
              > too slow, approx 5 seconds.
              >
              > We don't use the .store() method as the TaffyDB's will be initialised from
              > IndexedDB/WebSQL/LocalStorage when the program starts and data will then be
              > persisted back as required. The initial large dataset is downloaded via a
              > web service.
              >
              > The program works on iOS5, Android and many more browsers than our previous
              > deprecated HTML5 WebSQL solution. Whilst work and testing on this solution
              > continues the outlook seems good at this stage. TaffyDB rocks.
              >
              >
              >


              --
              Ziyahan Fakir Albeniz
              Bilgisayar Programcısı

              *GSM :* +90 533 637 1572
              *Skype :* ziyahanalbeniz
              *Web :* http://ziyahanalbeniz.blogspot.com
            Your message has been successfully submitted and would be delivered to recipients shortly.