Loading ...
Sorry, an error occurred while loading the content.

Re: [govtrack] also, offsite cache?

Expand Messages
  • Joshua Tauberer / GovTrack.us
    ... You re talking about caching GovTrack pages, right? Mostly, I hope to just fix the problem... Hopefully some improvements will be made in the next few
    Message 1 of 4 , Jul 27, 2007
    • 0 Attachment
      Nathan Hawks wrote:
      > Another thought, what about a cache?
      > - Script that uploads cached pages offsite after the thomas grab
      > - Instead of the Site Offline message, do a JS pull from the offsite
      > cache
      > - Would have to insert a big message div at top saying that this was
      > cached at date X time Y and dynamic content will be available again

      You're talking about caching GovTrack pages, right?

      Mostly, I hope to just fix the problem... Hopefully some improvements
      will be made in the next few days.

      But, for caching, it's very problematic because there are thousands of
      potential pages GovTrack could serve. Most don't ever get requested.
      5,000 subject terms, some 50,000 bills, the thousands of people that
      have ever served in Congress, etc. (It wouldn't be impossible, but it's
      not trivial.)

      - Josh
    • Nathan Hawks
      A rolling cache, perhaps? Everything that ever gets clicked on, each time it s updated, gets submitted to the morning upload queue? Sans wikis and
      Message 2 of 4 , Jul 27, 2007
      • 0 Attachment
        A rolling cache, perhaps? Everything that ever gets clicked on, each
        time it's updated, gets submitted to the morning upload queue? Sans
        wikis and purely-user-generated content, of course.

        Even once you fix the underlying problem, it'd be nice to imagine and
        work for a day when tens of millions of people are hitting the page 100+
        times a day :)

        I agree, probably not trivial.

        In any event good luck with the bughunting :)

        Nathan

        On Fri, 2007-07-27 at 12:52 -0400, Joshua Tauberer / GovTrack.us wrote:
        > Nathan Hawks wrote:
        > > Another thought, what about a cache?
        > > - Script that uploads cached pages offsite after the thomas grab
        > > - Instead of the Site Offline message, do a JS pull from the offsite
        > > cache
        > > - Would have to insert a big message div at top saying that this was
        > > cached at date X time Y and dynamic content will be available again
        >
        > You're talking about caching GovTrack pages, right?
        >
        > Mostly, I hope to just fix the problem... Hopefully some improvements
        > will be made in the next few days.
        >
        > But, for caching, it's very problematic because there are thousands
        > of
        > potential pages GovTrack could serve. Most don't ever get requested.
        > 5,000 subject terms, some 50,000 bills, the thousands of people that
        > have ever served in Congress, etc. (It wouldn't be impossible, but
        > it's
        > not trivial.)
        >
        > - Josh
        >
        >
        >
        >
      • Josh Tauberer
        ... Actually, I already cache pages that have no user-specific content (if the user comes with no GovTrack cookies set), and I wonder if I could just spew
        Message 3 of 4 , Aug 3 12:55 PM
        • 0 Attachment
          Nathan Hawks wrote:
          > Another thought, what about a cache?
          > - Script that uploads cached pages offsite after the thomas grab
          > - Instead of the Site Offline message, do a JS pull from the offsite
          > cache
          > - Would have to insert a big message div at top saying that this was
          > cached at date X time Y and dynamic content will be available again
          > ASAP.

          Actually, I already cache pages that have no user-specific content (if
          the user comes with no GovTrack cookies set), and I wonder if I could
          just spew those back during down time. That's an interesting idea.

          OTOH, either today was a light day, or upgrading to the latest Mono
          fixed something. There haven't been many server issues today.
          Nevertheless, next week I plan to buy a better server and will probably
          have the site moved over in a few weeks. (These days the advertising on
          the site generates enough money to pay a normal price for a good
          dedicated server.)

          --
          - Josh Tauberer

          http://razor.occams.info

          "Yields falsehood when preceded by its quotation! Yields
          falsehood when preceded by its quotation!" Achilles to
          Tortoise (in "Gödel, Escher, Bach" by Douglas Hofstadter)
        Your message has been successfully submitted and would be delivered to recipients shortly.