Loading ...
Sorry, an error occurred while loading the content.

Re: Taffy DB 1.6 Feature Wishlist

Expand Messages
  • darren_kovalchik
    Great job on the 1.5 release. I ve been using it with no problems. I like the new has filter and feel like everything runs just a bit snappier. I wrote down
    Message 1 of 8 , Jun 30, 2008
    • 0 Attachment

      Great job on the 1.5 release. I've been using it with no problems. I like the new "has" filter and feel like everything runs just a bit snappier. I wrote down some of my thoughts about things I've noticed and wanted, as well as responded to a few of your ideas above. Hope some of it is helpful.


      ·    I'd like to see matching negative filters for all positive filters.

      o       "notHas" or "doesNotHave" as the opposite of "has" (would be especially useful I think)

      o       "doesNotStartWith" and "doesNotEndWith."

      o       "notLength"

       

      ·    I really like the new "has" filter. It would be nice if "equal" and "notequal" could also accept objects and arrays, which could then deprecate "isSameObject," "isSameArray", etc.

       

      ·    I was thinking about an option for a ranked search. This could be useful if you were allowing the user to search the TaffyDB from a search input to retrieve results which are relevantly close.

       

      o       Interestingly, I ran across this article today: http://orderedlist.com/articles /live-search-with-quicksilver -style which uses a ranking algorithm based off quicksilver: http://rails-oceania.googlecode .com/svn/lachiecox/qs_score /trunk/qs_score.js

      o       The quicksilver port is just hosted on google code without much information on who made it, but it's a good example at least. The ranking algorithm is surprisingly small, so it seems that something like this could possibly be added fairly easily and would add a nice functionality to TaffyDB without increasing the code base too much.

       

      ·    I've noticed the camel case for the filters is a little inconsistent. No big deal.

       

      ·    It seems to me the "notWhatever" syntax can get a little confusing, especially when dealing with things like "has" where you would say "doesNotHave." Would it be possible to create a "not" filter (different from notequal shorthand), in which you could enclose any of the other filters to find their negative? For example: data.find({favoriteFoods: {has: {not: [`pizza']}}}); or data.find({favoriteFoods: {not: {has: [`pizza']}}});  or something along those lines. However, that does add yet another filter object, which gets a bit confusing. My other thought was the ability to put a "!" on the front of filters to indicate the negative. Like: data.find({ favoriteFoods: {`!has': [`pizza']}}); Of course, then you'd need to enclose those filters in quotes or an error will be thrown, but it might be worth a try.

       

      ·    I like your idea for templates. I think it would also be useful to be able to merge a template only with a given set of results, for example: people.find(age:{gt:40},gender:'male').setTemplate({senior:'yes'});

       

      ·    I like your ideas on advanced AND and OR searching. So far, I've only been using pretty basic lookups like matching values or finding a value in an array, but I definitely feel like this could be a useful feature, especially if dealing with complex data sent from a server database. I say it could be an extremely useful feature given the right circumstances.

    • berg.matt
      Ian, Great job on the 1.5 release. Your support for Taffy DB has been excellent. My main wishlist feature is something I think you have talked about
      Message 2 of 8 , Jul 1, 2008
      • 0 Attachment
        Ian,

        Great job on the 1.5 release. Your support for Taffy DB has been
        excellent.

        My main wishlist feature is something I think you have talked about
        previously, a "primary index". At least with my usage, I am
        constantly needing to get a record by the ID. Most of the the time I
        keep track of the Taffy index along with the ID, but when I delete
        things, obviously the indexes are not current anymore.

        Another idea would be the option to use public utitlity's (isNumeric,
        JSON, etc) from other frameworks (YUI, Dojo, etc) instead of including
        your own. Could cut down on file size. Definitely not a big deal though.

        Matt

        --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
        >
        > Hey All,
        >
        > 1.5 bugs are fixed so I'm looking ahead to 1.6 and adding in a few
        > requested features as well as deprecating some aspects that are
        > confusing.
        >
        > I'd like comments if anyone has any as well as additional "wish list"
        > items for this version or future versions. Here is what I'm thinking
        > about for 1.6:
        >
        > 1. Add support for breaking our of forEach loops.
        >
        > Presumably you'd pass back a variable and when the forEach loop sees
        > that variable it will stop looping. My thought is to create a new
        > variable called TAFFY.exit and if you return TAFFY.exit it will exit
        > the forEach. This would also work when you pass a function to find(),
        > although it may not really have any uses in that case.
        >
        > 2. Add setTemplate, removeTemplate, and applyTemplate methods to the
        > collection.
        >
        > The idea with a template is that it would be an object which would be
        > "merged" with every record in your database. Think of it as default
        > values. If you called setTemplate({columnClass:"blue"}) it would add a
        > columnClass:"Blue" attribute to every record that didn't already
        > contain a columnClass attribute. setTemplate would add/replace a
        > template to run with every new insert, removeTemplate would remove
        > that template, and applyTemplate would let you apply a template to all
        > records but not leave the template in place for future inserts.
        >
        > 3. Support for OR/AND logic in find method by using arrays.
        >
        > Right now the find method takes two arguments. The first is the object
        > to use to create a lookup, and the second is an array of records to
        > search within. Every item within your lookup object is treated as an
        > AND expression and all have to match.
        >
        > There are three limitations with the current system: A. no way to do
        > OR selections. B. No way to do multiple filters on a single column
        > (example: x:{gt:1} AND x:{NOT:70}). C. You have to populate the
        > second argument with an array either through another find call or
        > through your own code.
        >
        > The new system would work something like this. Every argument would be
        > treated like an AND filter unless the argument is an array, in which
        > case it would be treated like an OR filter.
        >
        > So find({x:{gt:70}},{x:{lt:90}}) would return rows with a value of x
        > between 70 and 90. That would be an AND lookup.
        >
        > For OR loopups you would use an array (not the square brackets)
        > find([{x:7},{x:{gt:30}}]) which would return rows with a value of x
        > that is 7 or above 30.
        >
        > You could still use a filter array as well:
        > find({x:{gt:100}},[0,1,2,3,4]) which would search records 0 through 4
        > for x values greater than 100.
        >
        > You could also string together many arguments:
        >
        > find({x:{gt:1}},{x:{NOT:10}},{x:{gt:50},status:"active:})
        >
        > This would return rows where x is greater than 1, is not 10, and is
        > greater than 50 (but only if the status is active).
        >
        > In theory this makes the find function much more powerful although it
        > remains to be seen if you really need that kind of expressive power in
        > one function call.
        >
        > Thoughts?
        >
        > 4. Remove arraycontains support (which was deprecated in the last
        > release in favor of has).
        >
        > 5. Deprecate myCollection.RAW and myCollection.length.
        >
        > RAW is confusing. It is currently the data you passed into TAFFY()
        > when you created your collection, not the current state of your
        > collection. RAW would be removed in 1.7. You can get the current state
        > of your collection by calling get().
        >
        > myCollection.length would be deprecated in favor of a getLength()
        > method. Currently you could change the length value externally which
        > could really screw up your scripts (although TAFFY should still know
        > the correct length of your collection for searching, etc).
        >
        > 6. Add a getLength method (see above)
        >
        > That's it. Any thoughts from the TaffyDB community?
        >
        > Ian
        >
      • tacoman_cool
        Hey Darren, Wow! Awesome list of suggestions. Let me try an address some of them below. 1. I agree the filter API is getting a little complex and is missing
        Message 3 of 8 , Jul 1, 2008
        • 0 Attachment
          Hey Darren,

          Wow! Awesome list of suggestions. Let me try an address some of them
          below.

          1. I agree the filter API is getting a little complex and is missing
          some vital elements like negative operators. I like your idea of using
          ! to make it negative. In theory we could eliminate almost half of the
          options by supporting the bang ! operator.

          2. The score search is really cool. I'll do some more research on it.
          Thanks for the link.

          3. I really like your template suggestion. Maybe applyTemplate would
          take a filter like other functions. setTemplate and removeTemplate
          will still apply to the collection as a whole and have only one
          argument. The first argument for apply would be the template and the
          second the optional filter. So you could do something like
          people.applyTemplate({likesCounty:true},{state:"TX"})

          Keep up the good ideas. I'll try and filter a semi-final feature list
          before I start on 1.6.

          Ian
        • tacoman_cool
          Hey Matt, Thanks for the excellent suggestions. Couple of quick comments: 1. As for a primary index, what would you think about some kind of UUID? A wish list
          Message 4 of 8 , Jul 1, 2008
          • 0 Attachment
            Hey Matt,

            Thanks for the excellent suggestions. Couple of quick comments:

            1. As for a primary index, what would you think about some kind of
            UUID? A wish list feature that I have is to add in support for
            generating UUIDs that could act as the primary key for your
            collection. Probably it would work by calling TAFFY.newID(). Maybe it
            would just make sense to assign a number in the insert that would stay
            fixed for that collection (different than the array index). The insert
            could return the created number. Thoughts?

            2. I'd love to use whatever library is installed instead of including
            my own copy of duplicate code. That said, TaffyDB needs many of those
            feature in order to work correctly and forcing users to use a certain
            platform would be a pain. Maybe once we are out of the "new features"
            phase it would be possible to create plugin versions for all the
            popular libraries that would use as much of their code as possible.
            Good way to get more folks using Taffy DB too.

            Thanks again for your continued feedback. It is very nice to not be
            working in a vacuum.

            Ian
          • berg.matt
            With regards to #1, I think the main issue is quickly accessing the records by primary key. For example, I already include my own ID from the database in the
            Message 5 of 8 , Jul 1, 2008
            • 0 Attachment
              With regards to #1, I think the main issue is quickly accessing the
              records by primary key. For example, I already include my own ID from
              the database in the collection. But everytime that I want to get that
              record by ID, Taffy has to loop to find it.

              I guess it probably just isn't plausible with Javascript to be able to
              access a record directly by the ID without loops. Unless a separate
              index was created along side the Taffy collection, which kept track of
              that information. But then there is the issue of maintaining that
              index on inserts and deletes.

              Matt

              --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
              >
              > Hey Matt,
              >
              > Thanks for the excellent suggestions. Couple of quick comments:
              >
              > 1. As for a primary index, what would you think about some kind of
              > UUID? A wish list feature that I have is to add in support for
              > generating UUIDs that could act as the primary key for your
              > collection. Probably it would work by calling TAFFY.newID(). Maybe it
              > would just make sense to assign a number in the insert that would stay
              > fixed for that collection (different than the array index). The insert
              > could return the created number. Thoughts?
              >
              > 2. I'd love to use whatever library is installed instead of including
              > my own copy of duplicate code. That said, TaffyDB needs many of those
              > feature in order to work correctly and forcing users to use a certain
              > platform would be a pain. Maybe once we are out of the "new features"
              > phase it would be possible to create plugin versions for all the
              > popular libraries that would use as much of their code as possible.
              > Good way to get more folks using Taffy DB too.
              >
              > Thanks again for your continued feedback. It is very nice to not be
              > working in a vacuum.
              >
              > Ian
              >
            • tacoman_cool
              Hey Matt, I see your point. In the index testing I ve done I ve found that you really need 2500 or more records before you start getting a performance
              Message 6 of 8 , Jul 2, 2008
              • 0 Attachment
                Hey Matt,

                I see your point. In the index testing I've done I've found that you
                really need 2500 or more records before you start getting a
                performance advantage from an index (given that insert, update,
                delete, and order by statements are all slower due to index updates).
                If you are working with less than that it is generally faster to just
                loop. That said, a specific primary key index might be faster if it
                used an object to do lookups rather than an array loop. I think it is
                still a drawing board idea to speed things up in a later version (if
                it can be made to work without too much craziness).

                Ian

                --- In taffydb@yahoogroups.com, "berg.matt" <berg.matt@...> wrote:
                >
                > With regards to #1, I think the main issue is quickly accessing the
                > records by primary key. For example, I already include my own ID from
                > the database in the collection. But everytime that I want to get that
                > record by ID, Taffy has to loop to find it.
                >
                > I guess it probably just isn't plausible with Javascript to be able to
                > access a record directly by the ID without loops. Unless a separate
                > index was created along side the Taffy collection, which kept track of
                > that information. But then there is the issue of maintaining that
                > index on inserts and deletes.
                >
                > Matt
                >
                > --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@> wrote:
                > >
                > > Hey Matt,
                > >
                > > Thanks for the excellent suggestions. Couple of quick comments:
                > >
                > > 1. As for a primary index, what would you think about some kind of
                > > UUID? A wish list feature that I have is to add in support for
                > > generating UUIDs that could act as the primary key for your
                > > collection. Probably it would work by calling TAFFY.newID(). Maybe it
                > > would just make sense to assign a number in the insert that would stay
                > > fixed for that collection (different than the array index). The insert
                > > could return the created number. Thoughts?
                > >
                > > 2. I'd love to use whatever library is installed instead of including
                > > my own copy of duplicate code. That said, TaffyDB needs many of those
                > > feature in order to work correctly and forcing users to use a certain
                > > platform would be a pain. Maybe once we are out of the "new features"
                > > phase it would be possible to create plugin versions for all the
                > > popular libraries that would use as much of their code as possible.
                > > Good way to get more folks using Taffy DB too.
                > >
                > > Thanks again for your continued feedback. It is very nice to not be
                > > working in a vacuum.
                > >
                > > Ian
                > >
                >
              • darren_kovalchik
                I was also thinking that it might be nice to have a removeTemplate option as well. For example to remove the favorite food property for everyone living in
                Message 7 of 8 , Jul 8, 2008
                • 0 Attachment
                  I was also thinking that it might be nice to have a removeTemplate
                  option as well. For example to remove the favorite food property for
                  everyone living in california because it was overrun by robots.

                  In regards to primary index keys: I have personally been using Taffy
                  objects with less than 100 entries for most of my tests. Any more than
                  that, and I think I'd start breaking up the data with calls to the
                  server database for more information when necessary. However, that's
                  certainly not the only way to go about things, and I can see where
                  having a larger TaffyDB would come in handy to speed up an application
                  without having to make slow server calls.

                  I know you were worried about the code size increase the addition of
                  primary indexes might cause. I wonder if you could extend the class
                  with an optional second download file taffy-indexes.js or something. I
                  dunno.

                  Anyway, I think indexes are worth pursuing, but not at the top of my
                  list, at any rate.

                  I like the idea of porting TaffyDB to popular frameworks once it's
                  feature complete. It could certainly reduce the code size by
                  eliminating duplicate functions. However, it would make it more
                  difficult to add additional features or even fix bugs beyond that
                  point because you'd have to apply the changes to four or five
                  versions. Also something for farther down the line I guess.

                  --- In taffydb@yahoogroups.com, "tacoman_cool" <ian@...> wrote:
                  >
                  > Hey Darren,
                  >
                  > Wow! Awesome list of suggestions. Let me try an address some of them
                  > below.
                  >
                  > 1. I agree the filter API is getting a little complex and is missing
                  > some vital elements like negative operators. I like your idea of using
                  > ! to make it negative. In theory we could eliminate almost half of the
                  > options by supporting the bang ! operator.
                  >
                  > 2. The score search is really cool. I'll do some more research on it.
                  > Thanks for the link.
                  >
                  > 3. I really like your template suggestion. Maybe applyTemplate would
                  > take a filter like other functions. setTemplate and removeTemplate
                  > will still apply to the collection as a whole and have only one
                  > argument. The first argument for apply would be the template and the
                  > second the optional filter. So you could do something like
                  > people.applyTemplate({likesCounty:true},{state:"TX"})
                  >
                  > Keep up the good ideas. I'll try and filter a semi-final feature list
                  > before I start on 1.6.
                  >
                  > Ian
                  >
                Your message has been successfully submitted and would be delivered to recipients shortly.