Loading ...
Sorry, an error occurred while loading the content.

session data slowdown

Expand Messages
  • raystormphd
    Hello, In an attempt to offload SELECT queries from our database and decrease the time spent gathering data, we have begun to aggressively cache data in users
    Message 1 of 4 , Jun 5, 2003
    • 0 Attachment
      Hello,

      In an attempt to offload SELECT queries from our database and decrease
      the time spent gathering data, we have begun to aggressively cache
      data in users' sessions (StateDB=MLDBM::Sync::SDBM_File,
      StateSerializer=Data::Dumper). Unfortunately, when the data set is
      too large the process of reading/thawing the data becomes painfully
      slow. Switching to DB_File didn't seem to help. We haven't tried
      Storable but my inclination is that it wouldn't make that much of a
      difference. The session files are on the order of a 100k.

      1) Is the thawing of the data to blame?

      2) How might we speed this process up or are we heading down a blind
      alley?

      3) Lastly, I have heard good things about IPC:MM (fastest gun in the
      west when it comes to using Shared memory for a cache). Could
      this be used instead of MLDBM::Sync::SDBM_File?

      Any hints or help would be appreciated!

      Christian


      ---------------------------------------------------------------------
      To unsubscribe, e-mail: asp-unsubscribe@...
      For additional commands, e-mail: asp-help@...
    • Josh Chamas
      ... Storable might help. It might be twice as fast as Dumper for large data sets like this. ... Yes. This really is too much data I would say. You should
      Message 2 of 4 , Jun 5, 2003
      • 0 Attachment
        raystormphd wrote:
        > Hello,
        >
        > In an attempt to offload SELECT queries from our database and decrease
        > the time spent gathering data, we have begun to aggressively cache
        > data in users' sessions (StateDB=MLDBM::Sync::SDBM_File,
        > StateSerializer=Data::Dumper). Unfortunately, when the data set is
        > too large the process of reading/thawing the data becomes painfully
        > slow. Switching to DB_File didn't seem to help. We haven't tried
        > Storable but my inclination is that it wouldn't make that much of a
        > difference. The session files are on the order of a 100k.
        >

        Storable might help. It might be twice as fast as Dumper for large
        data sets like this.

        > 1) Is the thawing of the data to blame?
        >

        Yes. This really is too much data I would say. You should really
        just try to store small bits in $Session. Any global data might be
        better to cache in a global data structure, or even use the CacheDB
        and includes caching.

        > 2) How might we speed this process up or are we heading down a blind
        > alley?
        >

        Try SessionSerialize setting, similar to calling $Session->Lock in the
        Script_OnStart. Try not accessing keys too often. You might try to
        make a read only copy of data like this:

        use vars qw($SessionReadOnly);
        sub Script_OnStart {
        $SessionReadOnly = { %$Session };
        }

        so if you are accessing same data multiple times, this might be faster.

        > 3) Lastly, I have heard good things about IPC:MM (fastest gun in the
        > west when it comes to using Shared memory for a cache). Could
        > this be used instead of MLDBM::Sync::SDBM_File?
        >

        No, but you could have your own cache layer reference per user
        by $Session->SessionID. You might also try using Tie::TextDir either
        for StateDB or for CacheDB ( if you start includes caching ).
        Note that if you have a really fast local database like MySQL, you
        might be better off not caching at all & lettings most queries hit
        the database.

        > Any hints or help would be appreciated!
        >

        My hints were very generic. A lot of it has to do with exactly how you are
        using $Session programmatically.

        Regards,

        Josh

        ________________________________________________________________
        Josh Chamas, Founder phone:925-552-0128
        Chamas Enterprises Inc. http://www.chamas.com
        NodeWorks Link Checking http://www.nodeworks.com


        ---------------------------------------------------------------------
        To unsubscribe, e-mail: asp-unsubscribe@...
        For additional commands, e-mail: asp-help@...
      • raystormphd
        Josh, Thank you for your insite into our problem. We have some more ammunition to play with now. ... blind ... the ... faster. This brings up another set of
        Message 3 of 4 , Jun 6, 2003
        • 0 Attachment
          Josh,

          Thank you for your insite into our problem. We have some more
          ammunition to play with now.

          > > 2) How might we speed this process up or are we heading down
          blind
          > > alley?
          > >
          >
          > Try SessionSerialize setting, similar to calling $Session->Lock in
          the
          > Script_OnStart. Try not accessing keys too often. You might try to
          > make a read only copy of data like this:
          >
          > use vars qw($SessionReadOnly);
          > sub Script_OnStart {
          > $SessionReadOnly = { %$Session };
          > }
          >
          > so if you are accessing same data multiple times, this might be
          faster.

          This brings up another set of questions I have had regarding Session
          data. I have tried finding the answer in posts/docs/etc. but haven't
          found a definitive answer.

          How does a series of reads or writes to a Session object hit the disk
          and the StateSerializer?

          For instance, let's say I did this:

          $Session->{'key1'}=data1
          $Session->{'key2'}=data2
          $Session->{'key3'}=data3

          or

          data1=$Session->{'key1'}
          data2=$Session->{'key2'}
          data3=$Session->{'key3'}

          Does each statement cause a freeze/thaw of the whole Session data
          structure to/from disk?

          In your code, you make one read only copy. Does this cause only one
          thaw? This is why your proposed trick would help, right?

          Cheers,

          Christian


          ---------------------------------------------------------------------
          To unsubscribe, e-mail: asp-unsubscribe@...
          For additional commands, e-mail: asp-help@...
        • Josh Chamas
          ... No, each one does its own freeze/thaw just for that key / value pair. ... The idea behind that is if you are just using it to look up lots of global values
          Message 4 of 4 , Jun 6, 2003
          • 0 Attachment
            raystormphd wrote:
            >
            > data1=$Session->{'key1'}
            > data2=$Session->{'key2'}
            > data3=$Session->{'key3'}
            >
            > Does each statement cause a freeze/thaw of the whole Session data
            > structure to/from disk?
            >

            No, each one does its own freeze/thaw just for that key / value pair.

            > In your code, you make one read only copy. Does this cause only one
            > thaw? This is why your proposed trick would help, right?
            >

            The idea behind that is if you are just using it to look up lots of
            global values & db data, which is not recommended anyway, you might
            just init an in memory hash to reference in your code instead of
            going to the $Session each time, so perhaps better to...

            $Session->Lock;
            $SessionCopy = { %$Session };
            $Session->UnLock;
            ...
            $SessionCopy->{KEY};
            $SessionCopy->{KEY};
            $SessionCopy->{KEY};

            than to:

            $Session->{KEY}
            $Session->{KEY}
            $Session->{KEY}

            but really better is to do this...

            my $value = $Session->{KEY};
            $value;
            $value;

            Like I said, a lot of it depends on how you are programming. But overall,
            you want to make as few calls getting as little data from $Session as possible.

            Regards,

            Josh

            ________________________________________________________________
            Josh Chamas, Founder phone:925-552-0128
            Chamas Enterprises Inc. http://www.chamas.com
            NodeWorks Link Checking http://www.nodeworks.com


            ---------------------------------------------------------------------
            To unsubscribe, e-mail: asp-unsubscribe@...
            For additional commands, e-mail: asp-help@...
          Your message has been successfully submitted and would be delivered to recipients shortly.