Loading ...
Sorry, an error occurred while loading the content.

Re: Taffy DB 1.6 Feature Wishlist

Expand Messages
  • berg.matt
    Ian, Great job on the 1.5 release. Your support for Taffy DB has been excellent. My main wishlist feature is something I think you have talked about
    Message 1 of 8 , Jul 1, 2008
    • 0 Attachment
      Ian,

      Great job on the 1.5 release. Your support for Taffy DB has been
      excellent.

      My main wishlist feature is something I think you have talked about
      previously, a "primary index". At least with my usage, I am
      constantly needing to get a record by the ID. Most of the the time I
      keep track of the Taffy index along with the ID, but when I delete
      things, obviously the indexes are not current anymore.

      Another idea would be the option to use public utitlity's (isNumeric,
      JSON, etc) from other frameworks (YUI, Dojo, etc) instead of including
      your own. Could cut down on file size. Definitely not a big deal though.

      Matt

      --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
      >
      > Hey All,
      >
      > 1.5 bugs are fixed so I'm looking ahead to 1.6 and adding in a few
      > requested features as well as deprecating some aspects that are
      > confusing.
      >
      > I'd like comments if anyone has any as well as additional "wish list"
      > items for this version or future versions. Here is what I'm thinking
      > about for 1.6:
      >
      > 1. Add support for breaking our of forEach loops.
      >
      > Presumably you'd pass back a variable and when the forEach loop sees
      > that variable it will stop looping. My thought is to create a new
      > variable called TAFFY.exit and if you return TAFFY.exit it will exit
      > the forEach. This would also work when you pass a function to find(),
      > although it may not really have any uses in that case.
      >
      > 2. Add setTemplate, removeTemplate, and applyTemplate methods to the
      > collection.
      >
      > The idea with a template is that it would be an object which would be
      > "merged" with every record in your database. Think of it as default
      > values. If you called setTemplate({columnClass:"blue"}) it would add a
      > columnClass:"Blue" attribute to every record that didn't already
      > contain a columnClass attribute. setTemplate would add/replace a
      > template to run with every new insert, removeTemplate would remove
      > that template, and applyTemplate would let you apply a template to all
      > records but not leave the template in place for future inserts.
      >
      > 3. Support for OR/AND logic in find method by using arrays.
      >
      > Right now the find method takes two arguments. The first is the object
      > to use to create a lookup, and the second is an array of records to
      > search within. Every item within your lookup object is treated as an
      > AND expression and all have to match.
      >
      > There are three limitations with the current system: A. no way to do
      > OR selections. B. No way to do multiple filters on a single column
      > (example: x:{gt:1} AND x:{NOT:70}). C. You have to populate the
      > second argument with an array either through another find call or
      > through your own code.
      >
      > The new system would work something like this. Every argument would be
      > treated like an AND filter unless the argument is an array, in which
      > case it would be treated like an OR filter.
      >
      > So find({x:{gt:70}},{x:{lt:90}}) would return rows with a value of x
      > between 70 and 90. That would be an AND lookup.
      >
      > For OR loopups you would use an array (not the square brackets)
      > find([{x:7},{x:{gt:30}}]) which would return rows with a value of x
      > that is 7 or above 30.
      >
      > You could still use a filter array as well:
      > find({x:{gt:100}},[0,1,2,3,4]) which would search records 0 through 4
      > for x values greater than 100.
      >
      > You could also string together many arguments:
      >
      > find({x:{gt:1}},{x:{NOT:10}},{x:{gt:50},status:"active:})
      >
      > This would return rows where x is greater than 1, is not 10, and is
      > greater than 50 (but only if the status is active).
      >
      > In theory this makes the find function much more powerful although it
      > remains to be seen if you really need that kind of expressive power in
      > one function call.
      >
      > Thoughts?
      >
      > 4. Remove arraycontains support (which was deprecated in the last
      > release in favor of has).
      >
      > 5. Deprecate myCollection.RAW and myCollection.length.
      >
      > RAW is confusing. It is currently the data you passed into TAFFY()
      > when you created your collection, not the current state of your
      > collection. RAW would be removed in 1.7. You can get the current state
      > of your collection by calling get().
      >
      > myCollection.length would be deprecated in favor of a getLength()
      > method. Currently you could change the length value externally which
      > could really screw up your scripts (although TAFFY should still know
      > the correct length of your collection for searching, etc).
      >
      > 6. Add a getLength method (see above)
      >
      > That's it. Any thoughts from the TaffyDB community?
      >
      > Ian
      >
    • tacoman_cool
      Hey Darren, Wow! Awesome list of suggestions. Let me try an address some of them below. 1. I agree the filter API is getting a little complex and is missing
      Message 2 of 8 , Jul 1, 2008
      • 0 Attachment
        Hey Darren,

        Wow! Awesome list of suggestions. Let me try an address some of them
        below.

        1. I agree the filter API is getting a little complex and is missing
        some vital elements like negative operators. I like your idea of using
        ! to make it negative. In theory we could eliminate almost half of the
        options by supporting the bang ! operator.

        2. The score search is really cool. I'll do some more research on it.
        Thanks for the link.

        3. I really like your template suggestion. Maybe applyTemplate would
        take a filter like other functions. setTemplate and removeTemplate
        will still apply to the collection as a whole and have only one
        argument. The first argument for apply would be the template and the
        second the optional filter. So you could do something like
        people.applyTemplate({likesCounty:true},{state:"TX"})

        Keep up the good ideas. I'll try and filter a semi-final feature list
        before I start on 1.6.

        Ian
      • tacoman_cool
        Hey Matt, Thanks for the excellent suggestions. Couple of quick comments: 1. As for a primary index, what would you think about some kind of UUID? A wish list
        Message 3 of 8 , Jul 1, 2008
        • 0 Attachment
          Hey Matt,

          Thanks for the excellent suggestions. Couple of quick comments:

          1. As for a primary index, what would you think about some kind of
          UUID? A wish list feature that I have is to add in support for
          generating UUIDs that could act as the primary key for your
          collection. Probably it would work by calling TAFFY.newID(). Maybe it
          would just make sense to assign a number in the insert that would stay
          fixed for that collection (different than the array index). The insert
          could return the created number. Thoughts?

          2. I'd love to use whatever library is installed instead of including
          my own copy of duplicate code. That said, TaffyDB needs many of those
          feature in order to work correctly and forcing users to use a certain
          platform would be a pain. Maybe once we are out of the "new features"
          phase it would be possible to create plugin versions for all the
          popular libraries that would use as much of their code as possible.
          Good way to get more folks using Taffy DB too.

          Thanks again for your continued feedback. It is very nice to not be
          working in a vacuum.

          Ian
        • berg.matt
          With regards to #1, I think the main issue is quickly accessing the records by primary key. For example, I already include my own ID from the database in the
          Message 4 of 8 , Jul 1, 2008
          • 0 Attachment
            With regards to #1, I think the main issue is quickly accessing the
            records by primary key. For example, I already include my own ID from
            the database in the collection. But everytime that I want to get that
            record by ID, Taffy has to loop to find it.

            I guess it probably just isn't plausible with Javascript to be able to
            access a record directly by the ID without loops. Unless a separate
            index was created along side the Taffy collection, which kept track of
            that information. But then there is the issue of maintaining that
            index on inserts and deletes.

            Matt

            --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
            >
            > Hey Matt,
            >
            > Thanks for the excellent suggestions. Couple of quick comments:
            >
            > 1. As for a primary index, what would you think about some kind of
            > UUID? A wish list feature that I have is to add in support for
            > generating UUIDs that could act as the primary key for your
            > collection. Probably it would work by calling TAFFY.newID(). Maybe it
            > would just make sense to assign a number in the insert that would stay
            > fixed for that collection (different than the array index). The insert
            > could return the created number. Thoughts?
            >
            > 2. I'd love to use whatever library is installed instead of including
            > my own copy of duplicate code. That said, TaffyDB needs many of those
            > feature in order to work correctly and forcing users to use a certain
            > platform would be a pain. Maybe once we are out of the "new features"
            > phase it would be possible to create plugin versions for all the
            > popular libraries that would use as much of their code as possible.
            > Good way to get more folks using Taffy DB too.
            >
            > Thanks again for your continued feedback. It is very nice to not be
            > working in a vacuum.
            >
            > Ian
            >
          • tacoman_cool
            Hey Matt, I see your point. In the index testing I ve done I ve found that you really need 2500 or more records before you start getting a performance
            Message 5 of 8 , Jul 2, 2008
            • 0 Attachment
              Hey Matt,

              I see your point. In the index testing I've done I've found that you
              really need 2500 or more records before you start getting a
              performance advantage from an index (given that insert, update,
              delete, and order by statements are all slower due to index updates).
              If you are working with less than that it is generally faster to just
              loop. That said, a specific primary key index might be faster if it
              used an object to do lookups rather than an array loop. I think it is
              still a drawing board idea to speed things up in a later version (if
              it can be made to work without too much craziness).

              Ian

              --- In taffydb@yahoogroups.com, "berg.matt" <berg.matt@...> wrote:
              >
              > With regards to #1, I think the main issue is quickly accessing the
              > records by primary key. For example, I already include my own ID from
              > the database in the collection. But everytime that I want to get that
              > record by ID, Taffy has to loop to find it.
              >
              > I guess it probably just isn't plausible with Javascript to be able to
              > access a record directly by the ID without loops. Unless a separate
              > index was created along side the Taffy collection, which kept track of
              > that information. But then there is the issue of maintaining that
              > index on inserts and deletes.
              >
              > Matt
              >
              > --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@> wrote:
              > >
              > > Hey Matt,
              > >
              > > Thanks for the excellent suggestions. Couple of quick comments:
              > >
              > > 1. As for a primary index, what would you think about some kind of
              > > UUID? A wish list feature that I have is to add in support for
              > > generating UUIDs that could act as the primary key for your
              > > collection. Probably it would work by calling TAFFY.newID(). Maybe it
              > > would just make sense to assign a number in the insert that would stay
              > > fixed for that collection (different than the array index). The insert
              > > could return the created number. Thoughts?
              > >
              > > 2. I'd love to use whatever library is installed instead of including
              > > my own copy of duplicate code. That said, TaffyDB needs many of those
              > > feature in order to work correctly and forcing users to use a certain
              > > platform would be a pain. Maybe once we are out of the "new features"
              > > phase it would be possible to create plugin versions for all the
              > > popular libraries that would use as much of their code as possible.
              > > Good way to get more folks using Taffy DB too.
              > >
              > > Thanks again for your continued feedback. It is very nice to not be
              > > working in a vacuum.
              > >
              > > Ian
              > >
              >
            • darren_kovalchik
              I was also thinking that it might be nice to have a removeTemplate option as well. For example to remove the favorite food property for everyone living in
              Message 6 of 8 , Jul 8, 2008
              • 0 Attachment
                I was also thinking that it might be nice to have a removeTemplate
                option as well. For example to remove the favorite food property for
                everyone living in california because it was overrun by robots.

                In regards to primary index keys: I have personally been using Taffy
                objects with less than 100 entries for most of my tests. Any more than
                that, and I think I'd start breaking up the data with calls to the
                server database for more information when necessary. However, that's
                certainly not the only way to go about things, and I can see where
                having a larger TaffyDB would come in handy to speed up an application
                without having to make slow server calls.

                I know you were worried about the code size increase the addition of
                primary indexes might cause. I wonder if you could extend the class
                with an optional second download file taffy-indexes.js or something. I
                dunno.

                Anyway, I think indexes are worth pursuing, but not at the top of my
                list, at any rate.

                I like the idea of porting TaffyDB to popular frameworks once it's
                feature complete. It could certainly reduce the code size by
                eliminating duplicate functions. However, it would make it more
                difficult to add additional features or even fix bugs beyond that
                point because you'd have to apply the changes to four or five
                versions. Also something for farther down the line I guess.

                --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
                >
                > Hey Darren,
                >
                > Wow! Awesome list of suggestions. Let me try an address some of them
                > below.
                >
                > 1. I agree the filter API is getting a little complex and is missing
                > some vital elements like negative operators. I like your idea of using
                > ! to make it negative. In theory we could eliminate almost half of the
                > options by supporting the bang ! operator.
                >
                > 2. The score search is really cool. I'll do some more research on it.
                > Thanks for the link.
                >
                > 3. I really like your template suggestion. Maybe applyTemplate would
                > take a filter like other functions. setTemplate and removeTemplate
                > will still apply to the collection as a whole and have only one
                > argument. The first argument for apply would be the template and the
                > second the optional filter. So you could do something like
                > people.applyTemplate({likesCounty:true},{state:"TX"})
                >
                > Keep up the good ideas. I'll try and filter a semi-final feature list
                > before I start on 1.6.
                >
                > Ian
                >
              Your message has been successfully submitted and would be delivered to recipients shortly.