Loading ...
Sorry, an error occurred while loading the content.

Vote API disabled for the moment

Expand Messages
  • Josh Tauberer
    Someone s queries to the votes API were bringing down my whole server, so I ve disabled the two votes endpoints for the moment. MySQL wasn t optimizing a query
    Message 1 of 10 , Feb 20 10:28 AM
    • 0 Attachment
      Someone's queries to the votes API were bringing down my whole server,
      so I've disabled the two votes endpoints for the moment.

      MySQL wasn't optimizing a query very well, to say the least.

      --
      - Josh Tauberer (@JoshData)

      http://razor.occams.info
    • maplight_kent
      Can we do anything to help get it back online? We have some MySQL folks who might be able to optimize the culprit queries. If it was our calls, I ll
      Message 2 of 10 , Feb 21 12:26 PM
      • 0 Attachment
        Can we do anything to help get it back online? We have some MySQL folks who might be able to optimize the culprit queries.

        If it was our calls, I'll double-check that we're not pulling more data than necessary.

        Kent

        --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
        >
        > Someone's queries to the votes API were bringing down my whole server,
        > so I've disabled the two votes endpoints for the moment.
        >
        > MySQL wasn't optimizing a query very well, to say the least.
        >
        > --
        > - Josh Tauberer (@JoshData)
        >
        > http://razor.occams.info
        >
      • Josh Tauberer
        Hey, Kent. On the up-side, it wasn t you. The culprit was paying attention and sent an email soon after the API started returning 404s. :) I m afraid to turn
        Message 3 of 10 , Feb 21 1:00 PM
        • 0 Attachment
          Hey, Kent.

          On the up-side, it wasn't you. The culprit was paying attention and sent an email soon after the API started returning 404s. :)

          I'm afraid to turn the v1 API back on because the framework I used for it (Tastypie in Django) doesn't give me much control over the queries.

          You can switch to the v2 API which works mostly the same. Just substitute v2 for v1, and check the field names still line up. In the v2 API, I've blocked the sort of query that was the problem.

          Basically, when you have an OFFSET that's too large MySQL decides it doesn't need to worry about query optimization anymore, and it executes the query without the use of the key that it uses normally. The problem was triggered by any offset >= 157191. Unfortunately since the SQL itself is still generated by Django, I can't do any further optimization (like adding FORCE INDEX, which does work).

          I've made it so the v2 API won't let you set a high offset, so at least this problem is avoided.

          The v2 API also has better error checking. In fact, when there's an error it reports the error to the caller rather than sending an email to me. (Improvement!) It is also more cautious about what to allow to prevent other runaway queries, so some previously possible queries with complex filters or unusual ordering may by rejected by v2.

          This issue only affects the two votes-related APIs because those were the only APIs that were backed by MySQL. Bills and people APIs are backed by Solr (and so probably have different undiscovered issues).

          Feedback on v2 welcome, etc.

          - Josh Tauberer (@JoshData)
          
          http://razor.occams.info
          On 02/21/2013 03:26 PM, maplight_kent wrote:
          
          
          Can we do anything to help get it back online?  We have some MySQL folks who might be able to optimize the culprit queries.
          
          If it was our calls, I'll double-check that we're not pulling more data than necessary.
          
          Kent
          
          --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
          
          Someone's queries to the votes API were bringing down my whole server, 
          so I've disabled the two votes endpoints for the moment.
          
          MySQL wasn't optimizing a query very well, to say the least.
          
          -- 
          - Josh Tauberer (@JoshData)
          
          http://razor.occams.info
          
          
          
          
          
          ------------------------------------
          
          Yahoo! Groups Links
          
          <*> To visit your group on the web, go to:
              http://groups.yahoo.com/group/govtrack/
          
          <*> Your email settings:
              Individual Email | Traditional
          
          <*> To change settings online go to:
              http://groups.yahoo.com/group/govtrack/join
              (Yahoo! ID required)
          
          <*> To change settings via email:
              govtrack-digest@yahoogroups.com 
              govtrack-fullfeatured@yahoogroups.com
          
          <*> To unsubscribe from this group, send an email to:
              govtrack-unsubscribe@yahoogroups.com
          
          <*> Your use of Yahoo! Groups is subject to:
              http://docs.yahoo.com/info/terms/
          

        • maplight_kent
          Thanks. I looked on the GovTrack site, but couldn t find docs for V2. If there are, would you mind pointing me to them? I m having trouble filtering. URLs
          Message 4 of 10 , Feb 21 4:32 PM
          • 0 Attachment
            Thanks.

            I looked on the GovTrack site, but couldn't find docs for V2. If there are, would you mind pointing me to them?

            I'm having trouble filtering.

            URLs like this are returning "Invalid field name for filter: congress"

            * http://www.govtrack.us/api/v2/vote/?congress=113&related_bill__number=273&related_bill__bill_type=house_bill&limit=50

            * http://www.govtrack.us/api/v2/vote/?congress=113

            * http://www.govtrack.us/api/v2/vote/?congress=1

            * http://www.govtrack.us/api/v2/vote/?related_bill__congress=113

            * http://www.govtrack.us/api/v2/vote/?number=1

            Bottom line is I'm looking for votes for specific bills, and for latest roll votes (so, filtered by roll number or date >= xyz).
          • Josh Tauberer
            ... Haven t done it yet. ... Instead of filtering on bill congress/type/number, just use the bill ID: http://www.govtrack.us/api/v2/vote?related_bill=73152 You
            Message 5 of 10 , Feb 22 6:40 AM
            • 0 Attachment
              I looked on the GovTrack site, but couldn't find docs for V2.  If there are, would you mind pointing me to them?
              

              Haven't done it yet.

              Bottom line is I'm looking for votes for specific bills

              Instead of filtering on bill congress/type/number, just use the bill ID:

              http://www.govtrack.us/api/v2/vote?related_bill=73152

              You should be able to get the bill ID from the bill API first by filtering on congress/type/number.

              The original is not permitted in V2 because it's not smart enough to know the database can answer that quickly.


              , and for latest roll votes (so, filtered by roll number or date >= xyz).

              Sorting by date should work the same as before:

              http://www.govtrack.us/api/v2/vote?created__gt=2013-01-01

              I've just fixed it so you can filter by the congress field:

              http://www.govtrack.us/api/v2/vote?congress=113

              number still doesn't work. I'll have to look at it next week. To be smart, it should only allow number if congress is also used as a filter --- otherwise the db query won't use an index.

              Sorry about the confusion.

              - Josh Tauberer (@JoshData)
              
              http://razor.occams.info
              On 02/21/2013 07:32 PM, maplight_kent wrote:
              
              
              Thanks.
              
              I looked on the GovTrack site, but couldn't find docs for V2.  If there are, would you mind pointing me to them?
              
              I'm having trouble filtering.
              
              URLs like this are returning "Invalid field name for filter: congress"
              
              * http://www.govtrack.us/api/v2/vote/?congress=113&related_bill__number=273&related_bill__bill_type=house_bill&limit=50
              
              * http://www.govtrack.us/api/v2/vote/?congress=113
              
              * http://www.govtrack.us/api/v2/vote/?congress=1
              
              * http://www.govtrack.us/api/v2/vote/?related_bill__congress=113
              
              * http://www.govtrack.us/api/v2/vote/?number=1
              
              Bottom line is I'm looking for votes for specific bills, and for latest roll votes (so, filtered by roll number or date >= xyz).
              
              
              
              ------------------------------------
              
              Yahoo! Groups Links
              
              <*> To visit your group on the web, go to:
                  http://groups.yahoo.com/group/govtrack/
              
              <*> Your email settings:
                  Individual Email | Traditional
              
              <*> To change settings online go to:
                  http://groups.yahoo.com/group/govtrack/join
                  (Yahoo! ID required)
              
              <*> To change settings via email:
                  govtrack-digest@yahoogroups.com 
                  govtrack-fullfeatured@yahoogroups.com
              
              <*> To unsubscribe from this group, send an email to:
                  govtrack-unsubscribe@yahoogroups.com
              
              <*> Your use of Yahoo! Groups is subject to:
                  http://docs.yahoo.com/info/terms/
              

            • mcafeeryan92
              Hi, What s going on with the vote_voter functionality? Is there some equivalent available in v2, or is v1 s functionality going to work soon or is it changed?
              Message 6 of 10 , Feb 23 10:33 AM
              • 0 Attachment
                Hi,

                What's going on with the vote_voter functionality? Is there some equivalent available in v2, or is v1's functionality going to work soon or is it changed? Thanks so much!

                -Ryan

                --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
                >
                > Someone's queries to the votes API were bringing down my whole server,
                > so I've disabled the two votes endpoints for the moment.
                >
                > MySQL wasn't optimizing a query very well, to say the least.
                >
                > --
                > - Josh Tauberer (@JoshData)
                >
                > http://razor.occams.info
                >
              • Maurizio
                Hello Josh, first, thank you for your awesome work. I wanted to ask if you ll enable vote_voter for v2 as well :) Cheers, Maurizio
                Message 7 of 10 , Feb 24 8:48 AM
                • 0 Attachment
                  Hello Josh,

                  first, thank you for your awesome work.
                  I wanted to ask if you'll enable vote_voter for v2 as well :)

                  Cheers,
                  Maurizio

                  --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
                  >
                  > Hey, Kent.
                  >
                  > On the up-side, it wasn't you. The culprit was paying attention and sent
                  > an email soon after the API started returning 404s. :)
                  >
                  > I'm afraid to turn the v1 API back on because the framework I used for
                  > it (Tastypie in Django) doesn't give me much control over the queries.
                  >
                  > You can switch to the v2 API which works mostly the same. Just
                  > substitute v2 for v1, and check the field names still line up. In the v2
                  > API, I've blocked the sort of query that was the problem.
                  >
                  > Basically, when you have an OFFSET that's too large MySQL decides it
                  > doesn't need to worry about query optimization anymore, and it executes
                  > the query without the use of the key that it uses normally. The problem
                  > was triggered by any offset >= 157191. Unfortunately since the SQL
                  > itself is still generated by Django, I can't do any further optimization
                  > (like adding FORCE INDEX, which does work).
                  >
                  > I've made it so the v2 API won't let you set a high offset, so at least
                  > this problem is avoided.
                  >
                  > The v2 API also has better error checking. In fact, when there's an
                  > error it reports the error to the caller rather than sending an email to
                  > me. (Improvement!) It is also more cautious about what to allow to
                  > prevent other runaway queries, so some previously possible queries with
                  > complex filters or unusual ordering may by rejected by v2.
                  >
                  > This issue only affects the two votes-related APIs because those were
                  > the only APIs that were backed by MySQL. Bills and people APIs are
                  > backed by Solr (and so probably have different undiscovered issues).
                  >
                  > Feedback on v2 welcome, etc.
                  >
                  > - Josh Tauberer (@JoshData)
                  >
                  > http://razor.occams.info
                  >
                  > On 02/21/2013 03:26 PM, maplight_kent wrote:
                  > >
                  > >
                  > > Can we do anything to help get it back online? We have some MySQL folks who might be able to optimize the culprit queries.
                  > >
                  > > If it was our calls, I'll double-check that we're not pulling more data than necessary.
                  > >
                  > > Kent
                  > >
                  > > --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
                  > >> Someone's queries to the votes API were bringing down my whole server,
                  > >> so I've disabled the two votes endpoints for the moment.
                  > >>
                  > >> MySQL wasn't optimizing a query very well, to say the least.
                  > >>
                  > >> --
                  > >> - Josh Tauberer (@JoshData)
                  > >>
                  > >> http://razor.occams.info
                  > >>
                  > >
                  > >
                  > >
                  > > ------------------------------------
                  > >
                  > > Yahoo! Groups Links
                  > >
                  > >
                  > >
                  >
                • Josh Tauberer
                  Ok it s enabled now. The option filter doesn t work the same. You can pass the ID of an option object, which you can get from the vote API. I ll try to add
                  Message 8 of 10 , Feb 24 10:50 AM
                  • 0 Attachment
                    Ok it's enabled now.

                    The 'option' filter doesn't work the same. You can pass the ID of an
                    option object, which you can get from the vote API.

                    I'll try to add documentation this week.

                    - Josh Tauberer (@JoshData)

                    http://razor.occams.info

                    On 02/24/2013 11:48 AM, Maurizio wrote:
                    > Hello Josh,
                    >
                    > first, thank you for your awesome work.
                    > I wanted to ask if you'll enable vote_voter for v2 as well :)
                    >
                    > Cheers,
                    > Maurizio
                    >
                    > --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
                    >> Hey, Kent.
                    >>
                    >> On the up-side, it wasn't you. The culprit was paying attention and sent
                    >> an email soon after the API started returning 404s. :)
                    >>
                    >> I'm afraid to turn the v1 API back on because the framework I used for
                    >> it (Tastypie in Django) doesn't give me much control over the queries.
                    >>
                    >> You can switch to the v2 API which works mostly the same. Just
                    >> substitute v2 for v1, and check the field names still line up. In the v2
                    >> API, I've blocked the sort of query that was the problem.
                    >>
                    >> Basically, when you have an OFFSET that's too large MySQL decides it
                    >> doesn't need to worry about query optimization anymore, and it executes
                    >> the query without the use of the key that it uses normally. The problem
                    >> was triggered by any offset >= 157191. Unfortunately since the SQL
                    >> itself is still generated by Django, I can't do any further optimization
                    >> (like adding FORCE INDEX, which does work).
                    >>
                    >> I've made it so the v2 API won't let you set a high offset, so at least
                    >> this problem is avoided.
                    >>
                    >> The v2 API also has better error checking. In fact, when there's an
                    >> error it reports the error to the caller rather than sending an email to
                    >> me. (Improvement!) It is also more cautious about what to allow to
                    >> prevent other runaway queries, so some previously possible queries with
                    >> complex filters or unusual ordering may by rejected by v2.
                    >>
                    >> This issue only affects the two votes-related APIs because those were
                    >> the only APIs that were backed by MySQL. Bills and people APIs are
                    >> backed by Solr (and so probably have different undiscovered issues).
                    >>
                    >> Feedback on v2 welcome, etc.
                    >>
                    >> - Josh Tauberer (@JoshData)
                    >>
                    >> http://razor.occams.info
                    >>
                    >> On 02/21/2013 03:26 PM, maplight_kent wrote:
                    >>>
                    >>> Can we do anything to help get it back online? We have some MySQL folks who might be able to optimize the culprit queries.
                    >>>
                    >>> If it was our calls, I'll double-check that we're not pulling more data than necessary.
                    >>>
                    >>> Kent
                    >>>
                    >>> --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
                    >>>> Someone's queries to the votes API were bringing down my whole server,
                    >>>> so I've disabled the two votes endpoints for the moment.
                    >>>>
                    >>>> MySQL wasn't optimizing a query very well, to say the least.
                    >>>>
                    >>>> --
                    >>>> - Josh Tauberer (@JoshData)
                    >>>>
                    >>>> http://razor.occams.info
                    >>>>
                    >>>
                    >>>
                    >>> ------------------------------------
                    >>>
                    >>> Yahoo! Groups Links
                    >>>
                    >>>
                    >>>
                    >
                    >
                    >
                    > ------------------------------------
                    >
                    > Yahoo! Groups Links
                    >
                    >
                    >
                  • Maurizio
                    Thanks Josh, I can see it now. You rock. Maurizio
                    Message 9 of 10 , Feb 24 12:59 PM
                    • 0 Attachment
                      Thanks Josh, I can see it now. You rock.

                      Maurizio

                      --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
                      >
                      > Ok it's enabled now.
                      >
                      > The 'option' filter doesn't work the same. You can pass the ID of an
                      > option object, which you can get from the vote API.
                      >
                      > I'll try to add documentation this week.
                      >
                      > - Josh Tauberer (@JoshData)
                      >
                      > http://razor.occams.info
                      >
                      > On 02/24/2013 11:48 AM, Maurizio wrote:
                      > > Hello Josh,
                      > >
                      > > first, thank you for your awesome work.
                      > > I wanted to ask if you'll enable vote_voter for v2 as well :)
                      > >
                      > > Cheers,
                      > > Maurizio
                      > >
                      > > --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
                      > >> Hey, Kent.
                      > >>
                      > >> On the up-side, it wasn't you. The culprit was paying attention and sent
                      > >> an email soon after the API started returning 404s. :)
                      > >>
                      > >> I'm afraid to turn the v1 API back on because the framework I used for
                      > >> it (Tastypie in Django) doesn't give me much control over the queries.
                      > >>
                      > >> You can switch to the v2 API which works mostly the same. Just
                      > >> substitute v2 for v1, and check the field names still line up. In the v2
                      > >> API, I've blocked the sort of query that was the problem.
                      > >>
                      > >> Basically, when you have an OFFSET that's too large MySQL decides it
                      > >> doesn't need to worry about query optimization anymore, and it executes
                      > >> the query without the use of the key that it uses normally. The problem
                      > >> was triggered by any offset >= 157191. Unfortunately since the SQL
                      > >> itself is still generated by Django, I can't do any further optimization
                      > >> (like adding FORCE INDEX, which does work).
                      > >>
                      > >> I've made it so the v2 API won't let you set a high offset, so at least
                      > >> this problem is avoided.
                      > >>
                      > >> The v2 API also has better error checking. In fact, when there's an
                      > >> error it reports the error to the caller rather than sending an email to
                      > >> me. (Improvement!) It is also more cautious about what to allow to
                      > >> prevent other runaway queries, so some previously possible queries with
                      > >> complex filters or unusual ordering may by rejected by v2.
                      > >>
                      > >> This issue only affects the two votes-related APIs because those were
                      > >> the only APIs that were backed by MySQL. Bills and people APIs are
                      > >> backed by Solr (and so probably have different undiscovered issues).
                      > >>
                      > >> Feedback on v2 welcome, etc.
                      > >>
                      > >> - Josh Tauberer (@JoshData)
                      > >>
                      > >> http://razor.occams.info
                      > >>
                      > >> On 02/21/2013 03:26 PM, maplight_kent wrote:
                      > >>>
                      > >>> Can we do anything to help get it back online? We have some MySQL folks who might be able to optimize the culprit queries.
                      > >>>
                      > >>> If it was our calls, I'll double-check that we're not pulling more data than necessary.
                      > >>>
                      > >>> Kent
                      > >>>
                      > >>> --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
                      > >>>> Someone's queries to the votes API were bringing down my whole server,
                      > >>>> so I've disabled the two votes endpoints for the moment.
                      > >>>>
                      > >>>> MySQL wasn't optimizing a query very well, to say the least.
                      > >>>>
                      > >>>> --
                      > >>>> - Josh Tauberer (@JoshData)
                      > >>>>
                      > >>>> http://razor.occams.info
                      > >>>>
                      > >>>
                      > >>>
                      > >>> ------------------------------------
                      > >>>
                      > >>> Yahoo! Groups Links
                      > >>>
                      > >>>
                      > >>>
                      > >
                      > >
                      > >
                      > > ------------------------------------
                      > >
                      > > Yahoo! Groups Links
                      > >
                      > >
                      > >
                      >
                    • Josh Tauberer
                      Documentation for the v2 API is up: http://www.govtrack.us/developers/api FYI, v2 also adds full text search for bills (and people). - Josh Tauberer
                      Message 10 of 10 , Feb 27 1:23 PM
                      • 0 Attachment
                        Documentation for the v2 API is up:

                        http://www.govtrack.us/developers/api

                        FYI, v2 also adds full text search for bills (and people).
                        - Josh Tauberer (@JoshData)
                        
                        http://razor.occams.info
                        On 02/24/2013 03:59 PM, Maurizio wrote:
                        Thanks Josh, I can see it now. You rock.
                        
                        Maurizio
                        
                        --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
                        
                        Ok it's enabled now.
                        
                        The 'option' filter doesn't work the same. You can pass the ID of an 
                        option object, which you can get from the vote API.
                        
                        I'll try to add documentation this week.
                        
                        - Josh Tauberer (@JoshData)
                        
                        http://razor.occams.info
                        
                        On 02/24/2013 11:48 AM, Maurizio wrote:
                        
                        Hello Josh,
                        
                        first, thank you for your awesome work.
                        I wanted to ask if you'll enable vote_voter for v2 as well :)
                        
                        Cheers,
                        Maurizio
                        
                        --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
                        
                        Hey, Kent.
                        
                        On the up-side, it wasn't you. The culprit was paying attention and sent
                        an email soon after the API started returning 404s. :)
                        
                        I'm afraid to turn the v1 API back on because the framework I used for
                        it (Tastypie in Django) doesn't give me much control over the queries.
                        
                        You can switch to the v2 API which works mostly the same. Just
                        substitute v2 for v1, and check the field names still line up. In the v2
                        API, I've blocked the sort of query that was the problem.
                        
                        Basically, when you have an OFFSET that's too large MySQL decides it
                        doesn't need to worry about query optimization anymore, and it executes
                        the query without the use of the key that it uses normally. The problem
                        was triggered by any offset >= 157191. Unfortunately since the SQL
                        itself is still generated by Django, I can't do any further optimization
                        (like adding FORCE INDEX, which does work).
                        
                        I've made it so the v2 API won't let you set a high offset, so at least
                        this problem is avoided.
                        
                        The v2 API also has better error checking. In fact, when there's an
                        error it reports the error to the caller rather than sending an email to
                        me. (Improvement!) It is also more cautious about what to allow to
                        prevent other runaway queries, so some previously possible queries with
                        complex filters or unusual ordering may by rejected by v2.
                        
                        This issue only affects the two votes-related APIs because those were
                        the only APIs that were backed by MySQL. Bills and people APIs are
                        backed by Solr (and so probably have different undiscovered issues).
                        
                        Feedback on v2 welcome, etc.
                        
                        - Josh Tauberer (@JoshData)
                        
                        http://razor.occams.info
                        
                        On 02/21/2013 03:26 PM, maplight_kent wrote:
                        
                        Can we do anything to help get it back online?  We have some MySQL folks who might be able to optimize the culprit queries.
                        
                        If it was our calls, I'll double-check that we're not pulling more data than necessary.
                        
                        Kent
                        
                        --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
                        
                        Someone's queries to the votes API were bringing down my whole server,
                        so I've disabled the two votes endpoints for the moment.
                        
                        MySQL wasn't optimizing a query very well, to say the least.
                        
                        -- 
                        - Josh Tauberer (@JoshData)
                        
                        http://razor.occams.info
                        
                        
                        
                        ------------------------------------
                        
                        Yahoo! Groups Links
                        
                        
                        
                        
                        
                        
                        ------------------------------------
                        
                        Yahoo! Groups Links
                        
                        
                        
                        
                        
                        
                        
                        ------------------------------------
                        
                        Yahoo! Groups Links
                        
                        <*> To visit your group on the web, go to:
                            http://groups.yahoo.com/group/govtrack/
                        
                        <*> Your email settings:
                            Individual Email | Traditional
                        
                        <*> To change settings online go to:
                            http://groups.yahoo.com/group/govtrack/join
                            (Yahoo! ID required)
                        
                        <*> To change settings via email:
                            govtrack-digest@yahoogroups.com 
                            govtrack-fullfeatured@yahoogroups.com
                        
                        <*> To unsubscribe from this group, send an email to:
                            govtrack-unsubscribe@yahoogroups.com
                        
                        <*> Your use of Yahoo! Groups is subject to:
                            http://docs.yahoo.com/info/terms/
                        

                      Your message has been successfully submitted and would be delivered to recipients shortly.