I am doing some work for a Non-Profit that requires the ability to determine who
there Congress members are by ZIP+4.  It's great that www.govtrak.us has added
the ability to do so, but I am having a difficult time finding this
functionality in the API.

Could someone please point me to where this is documented?  I noticed that other
sites like www.opencongress.org (which rely on GovTrak) have implemented this
functionality.  Unfortunately, they do not provide documentation in their API
either.

Thank you in advance for any help or advice.

- Josh Tauberer (@JoshData)
http://razor.occams.info

I am doing some work for a Non-Profit that requires the ability to determine who there Congress members are by ZIP+4. It's great that www.govtrak.us has added the ability to do so, but I am having a difficult time finding this functionality in the API.
Could someone please point me to where this is documented? I noticed that other sites like www.opencongress.org (which rely on GovTrak) have implemented this functionality. Unfortunately, they do not provide documentation in their API either.
Thank you in advance for any help or advice.
------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/govtrack/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/govtrack/join
(Yahoo! ID required)
<*> To change settings via email:
govtrack-digest@yahoogroups.com govtrack-fullfeatured@yahoogroups.com
<*> To unsubscribe from this group, send an email to:
govtrack-unsubscribe@yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/

Hey Josh,

You have helped me greatly!  I really appreciate the advice and guidance.  I
ended up going w/ MapQuest rather than Google, since there isn't an explicit
requirement that you MUST display a map -- although I definitely need to bring
this up w/ my Legal Department to confirm.  I am purely interested in
determining the coordinates and storing them in a DB to be later used for
GovTrak's API so that I can determine the Congressional District.

The only thing I have left is to map my the Districts to the corresponding
Representatives. I'll browse through your code to see how you have implemented
it.

Just out of curiosity, do you know if you guys plan on offering a Person Search
by District in your API at all?  The reason I ask is that (if at all possible),
I would like to obtain this information from a group that actually maintains it
-- rather than me having to make sure I update my data when needed.

Thanks again for the extremely valuable help.

Joe

--- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
>
> Hi, Joseph.
>
> I use a combination of the Google Maps geocoder (address to lat/lng)
> plus a GIS system based on work by the Chicago Tribune and OpenNorth
> (lat/lng to congressional district).
>
> I've just posted documentation for the 2nd part of the API here:
>
> 2010 districts:
> http://gis.govtrack.us/map/demo/2010-cd/?nocache=1
>
> 2012 districts:
> http://gis.govtrack.us/map/demo/cd-2012/?nocache=1
>
> ZIP+4s change regularly and there is no open ZIP+4 data to my knowledge
> (the USPS sells it). In a previous life I used the CDYNE service, which
> is paid, but relatively cheap, and gives good street address to district
> info.
>
> Full source code and deployment info:
> https://github.com/tauberer/boundaries_us
>
> - Josh Tauberer (@JoshData)
>
> http://razor.occams.info
>
> On 10/21/2012 06:46 AM, joseph_duero wrote:
> > I am doing some work for a Non-Profit that requires the ability to determine
who there Congress members are by ZIP+4.  It's great that www.govtrak.us has
added the ability to do so, but I am having a difficult time finding this
functionality in the API.
> >
> > Could someone please point me to where this is documented?  I noticed that
other sites like www.opencongress.org (which rely on GovTrak) have implemented
this functionality.  Unfortunately, they do not provide documentation in their
API either.
> >
> > Thank you in advance for any help or advice.
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> >
>

- Josh Tauberer (@JoshData)
http://razor.occams.info

Hey Josh,
You have helped me greatly! I really appreciate the advice and guidance. I ended up going w/ MapQuest rather than Google, since there isn't an explicit requirement that you MUST display a map -- although I definitely need to bring this up w/ my Legal Department to confirm. I am purely interested in determining the coordinates and storing them in a DB to be later used for GovTrak's API so that I can determine the Congressional District.
The only thing I have left is to map my the Districts to the corresponding Representatives. I'll browse through your code to see how you have implemented it. Just out of curiosity, do you know if you guys plan on offering a Person Search by District in your API at all? The reason I ask is that (if at all possible), I would like to obtain this information from a group that actually maintains it -- rather than me having to make sure I update my data when needed.
Thanks again for the extremely valuable help.
Joe
--- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:

Hi, Joseph.
I use a combination of the Google Maps geocoder (address to lat/lng) plus a GIS system based on work by the Chicago Tribune and OpenNorth (lat/lng to congressional district).
I've just posted documentation for the 2nd part of the API here:
2010 districts:
http://gis.govtrack.us/map/demo/2010-cd/?nocache=1
2012 districts:
http://gis.govtrack.us/map/demo/cd-2012/?nocache=1
ZIP+4s change regularly and there is no open ZIP+4 data to my knowledge (the USPS sells it). In a previous life I used the CDYNE service, which is paid, but relatively cheap, and gives good street address to district info.
Full source code and deployment info: https://github.com/tauberer/boundaries_us
- Josh Tauberer (@JoshData)
http://razor.occams.info
On 10/21/2012 06:46 AM, joseph_duero wrote:

I am doing some work for a Non-Profit that requires the ability to determine who there Congress members are by ZIP+4. It's great that www.govtrak.us has added the ability to do so, but I am having a difficult time finding this functionality in the API.
Could someone please point me to where this is documented? I noticed that other sites like www.opencongress.org (which rely on GovTrak) have implemented this functionality. Unfortunately, they do not provide documentation in their API either.
Thank you in advance for any help or advice.
------------------------------------
Yahoo! Groups Links

------------------------------------
Yahoo! Groups Links
<*> To visit your group on the web, go to:
http://groups.yahoo.com/group/govtrack/
<*> Your email settings:
Individual Email | Traditional
<*> To change settings online go to:
http://groups.yahoo.com/group/govtrack/join
(Yahoo! ID required)
<*> To change settings via email:
govtrack-digest@yahoogroups.com govtrack-fullfeatured@yahoogroups.com
<*> To unsubscribe from this group, send an email to:
govtrack-unsubscribe@yahoogroups.com
<*> Your use of Yahoo! Groups is subject to:
http://docs.yahoo.com/info/terms/

-- - Josh Tauberer (@JoshData)
http://razor.occams.info

I wanted to offer some lessons I learned in putting this data up in Filemaker,
as I would think others probably use that platform. (Hopefully this is the
appropriate forum for this).

1. It might be helpful to people to describe the general transformation that
needs to take place to load the data into Filemaker:

a. The bulk data comes from govtrack as field/value pairs. To load into
Filemaker, it needs to get translated into a list of field declarations (at the
beginning), followed by all the data values (without field labels) in the same
order. The data values are delimited by Filemaker-specific XML code. (See
http://www.filemaker.com/help/html/import_export.16.30.html, or Google
FMPXMLRESULT).

b. There is some Filemaker specific static text that has to go into the file;
it's documented in the link above.

c. The Role data is presented by  govtrack as if there could be multiple roles
per person (because in the full history file there are). But as there is just
one role per person in the Session-specific  file (people.112), I
flattened the role data into the same record as the person fields. There needs
to be some extra XSLT code to accomplish this which you will see in my XSLT
file.

d. The following article, from a Filemaker book (QUE CORPORATION - Using
Filemaker pro X, 2003, p.193), helped me understand the basics of what I needed
to do, although the code I needed to write differed in important ways:
http://flylib.com/books/en/4.353.1.193/1/.

The key difference is this example accesses data in XML elements. govtrack data
is placed in XML attributes, which you access differently. See my XSLT file.

e. Working with XSLT is massively easier if you use a test bench site to test
the code. This one, unlike all the others I tried, works well for this
application because it displays the resulting XML rather than trying to execute
it as web display code:
http://xslttest.appspot.com

That way you can compare it to what Filemaker says you need.

2. I'll post my XSLT file separately. I'm sure it could be massively improved
upon but it works.

I'd welcome any comments or suggestions. Next I want to figure out how to update
a joined table of current and historical data with the full historical people
file.

So here is the text of the XSLT file I created (I can see that others have
posted file attachments, but I can't see a button to do it here. But who cares,
it's just text.) So here ya go:


<?xml version="1.0" encoding="UTF-8" ?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output indent="yes" method="xml"/>

<xsl:template match="people">

<FMPXMLRESULT xmlns="http://www.filemaker.com/fmpxmlresult">

<ERRORCODE>0</ERRORCODE>

<PRODUCT BUILD="10-09-2012" NAME="FileMaker" VERSION="Pro 12.0v3"/>

<DATABASE DATEFORMAT="M/d/yyyy" LAYOUT="" NAME="people.112" RECORDS="{@count}" 
TIMEFORMAT="h:mm:ss a"/>

<METADATA>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="id" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="lastname" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="firstname" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="middlename" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="birthday" TYPE="DATE"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="gender" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="religion" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="pvsid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="osid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="bioguideid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="metavidid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="youtubeid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="twitterid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="icpsrid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="facebookgraphid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="thomasid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="name" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="title" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="type" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="startdate" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="enddate" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="party" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="state" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="district" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="url" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="address" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="current" TYPE="TEXT"/>

</METADATA>

      <RESULTSET FOUND="{@count}">

        <xsl:for-each select="person">

          <ROW MODID="0" RECORDID="1">

            <COL><DATA><xsl:value-of select="@id"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@lastname"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@firstname"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@middlename"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@birthday"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@gender"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@religion"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@pvsid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@osid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@bioguideid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@metavidid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@youtubeid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@twitterid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@icpsrid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@facebookgraphid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@thomasid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@name"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@title"/></DATA></COL>

            <xsl:for-each select="role">

              <COL><DATA><xsl:value-of select="@type"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@startdate"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@enddate"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@party"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@state"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@district"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@url"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@address"/></DATA></COL>

              <COL><DATA><xsl:value-of select="@current"/></DATA></COL>

            </xsl:for-each>

          </ROW>

        </xsl:for-each>

      </RESULTSET>

    </FMPXMLRESULT>

  </xsl:template>

</xsl:stylesheet>

Oops! The origianl XSLT file I posted did not transform the birthday as is
necessary to load dates into Filemaker. So here's another XSLT file (this one is
to load the historical people and role data, so it's not for exactly the same
destination database in Filemaker, but it should show how to transform the dates
correctly.)  I've checked and this approach produces valid dates in Filemaker.
Cheers!

<?xml version="1.0" encoding="UTF-8" ?>

<xsl:stylesheet version="1.0" xmlns:xsl="http://www.w3.org/1999/XSL/Transform">

<xsl:output indent="yes" method="xml"/>

<xsl:template name="formatDate">
	 <xsl:param name="FMPdate" />
	 <xsl:variable name="year" select="substring-before($FMPdate, '-')" />
	 <xsl:variable name="day" select="substring-before(substring-after($FMPdate,
'-'), '-')" />
	 <xsl:variable name="month" select="substring-after(substring-after($FMPdate,
'-'), '-')" />
	 <xsl:value-of select="concat($day, '/', $month, '/', $year)" />
</xsl:template>

<xsl:template match="people">

<FMPXMLRESULT xmlns="http://www.filemaker.com/fmpxmlresult">

<ERRORCODE>0</ERRORCODE>

<PRODUCT BUILD="10-09-2012" NAME="FileMaker" VERSION="Pro 12.0v3"/>

<DATABASE DATEFORMAT="M/d/yyyy" LAYOUT="" NAME="people.112" RECORDS="{@count}" 
TIMEFORMAT="h:mm:ss a"/>

<METADATA>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="id" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="lastname" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="firstname" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="middlename" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="birthday" TYPE="DATE"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="gender" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="religion" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="pvsid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="osid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="bioguideid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="metavidid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="youtubeid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="twitterid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="icpsrid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="facebookgraphid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="thomasid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="lismemberid" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="name" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="title" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="state" TYPE="TEXT"/>

<FIELD EMPTYOK="YES" MAXREPEAT="1" NAME="district" TYPE="TEXT"/>


</METADATA>

      <RESULTSET FOUND="{@count}">

        <xsl:for-each select="person">

          <ROW MODID="0" RECORDID="1">

            <COL><DATA><xsl:value-of select="@id"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@lastname"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@firstname"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@middlename"/></DATA></COL>

            <COL><DATA>
                   <xsl:call-template name="formatDate">
                       <xsl:with-param name="FMPdate" select="@birthday" />
                   </xsl:call-template>
            </DATA></COL>

            <COL><DATA><xsl:value-of select="@gender"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@religion"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@pvsid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@osid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@bioguideid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@metavidid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@youtubeid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@twitterid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@icpsrid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@facebookgraphid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@thomasid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@lismemberid"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@name"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@title"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@state"/></DATA></COL>

            <COL><DATA><xsl:value-of select="@district"/></DATA></COL>

            <xsl:for-each select="role">



            </xsl:for-each>

          </ROW>

        </xsl:for-each>

      </RESULTSET>

    </FMPXMLRESULT>

  </xsl:template>

</xsl:stylesheet>

--- In govtrack@yahoogroups.com, "cb299y" <cb299y@...> wrote:
>
> I wanted to offer some lessons I learned in putting this data up in Filemaker,
as I would think others probably use that platform. (Hopefully this is the
appropriate forum for this).
>
> 1. It might be helpful to people to describe the general transformation that
needs to take place to load the data into Filemaker:
>
> a. The bulk data comes from govtrack as field/value pairs. To load into
Filemaker, it needs to get translated into a list of field declarations (at the
beginning), followed by all the data values (without field labels) in the same
order. The data values are delimited by Filemaker-specific XML code. (See
http://www.filemaker.com/help/html/import_export.16.30.html, or Google
FMPXMLRESULT).
>
> b. There is some Filemaker specific static text that has to go into the file;
it's documented in the link above.
>
> c. The Role data is presented by  govtrack as if there could be multiple roles
per person (because in the full history file there are). But as there is just
one role per person in the Session-specific  file (people.112), I
> flattened the role data into the same record as the person fields. There needs
to be some extra XSLT code to accomplish this which you will see in my XSLT
file.
>
> d. The following article, from a Filemaker book (QUE CORPORATION - Using
Filemaker pro X, 2003, p.193), helped me understand the basics of what I needed
to do, although the code I needed to write differed in important ways:
> http://flylib.com/books/en/4.353.1.193/1/.
>
> The key difference is this example accesses data in XML elements. govtrack
data is placed in XML attributes, which you access differently. See my XSLT
file.
>
> e. Working with XSLT is massively easier if you use a test bench site to test
the code. This one, unlike all the others I tried, works well for this
application because it displays the resulting XML rather than trying to execute
it as web display code:
> http://xslttest.appspot.com
>
> That way you can compare it to what Filemaker says you need.
>
> 2. I'll post my XSLT file separately. I'm sure it could be massively improved
upon but it works.
>
> I'd welcome any comments or suggestions. Next I want to figure out how to
update a joined table of current and historical data with the full historical
people file.
>

-- - Josh Tauberer (@JoshData)
http://razor.occams.info

 

Hi, all. Here are my current plans for the 113th Congress.

New Members
-------------------

I hope to get info up on the new Members of Congress right after the current Congress adjourns. If I recall, that's usually around the 20th. It'll go up in the main people.xml file as well as in people.xml in the soon-to-be-created 113 directory.

Deprecating people.xml
------------------------------

With the start of the 113th Congress, the people.xml files will be deprecated (meaning still available but not recommended). The recommended way to get that info will be through the new congress-legislators project on github (https://github.com/unitedstates/congress-legislators) which is a collaboration between Eric Mill (Sunlight), Derek Willis, and myself (and anyone else who would like to help out).

You can read more about that in Alex Howard's article today on it:
http://radar.oreilly.com/2012/12/the-united-states-code-is-on-github.html

The existing people.xml files will probably be deleted at the end of 2013, giving you a year to switch to the new project.

Possible New Bill XML
----------------------------

I'm not sure, but starting with the 113th Congress I may replace my old Perl-based THOMAS scrapers for bill data with the new congress project ( https://github.com/unitedstates/congress; same collaborators). The new scrapers should produce bill status XML (i.e. 113/bills/*) in exactly the same format as the existing XML. But there may be bugs and some small changes. I'll write again when I know more about that.

If that goes well, the existing bill XML will actually be deprecated in 2014 in favor of the new scraper's JSON output, and with any luck the current XML format will be permanently retired starting with the 114th Congress. (I'll continue to make available existing bill data indefinitely.)

Anyway, more on that later.

Undocumented Data
---------------------------

The undocumented repstats and repstats.person will not be created for the 113th Congress. Instead I'll be creating a new stats directory with better information (e.g. missed votes %'s). The existing directories will probably be deleted by summer 2013.

If you're using anything else undocumented, you might want to drop me a line to check on its status for next year as I'm doing a ton of internal cleanups.

Thanks,

-- - Josh Tauberer (@JoshData)
http://razor.occams.info

- Josh Tauberer (@JoshData)
http://razor.occams.info

Hi Josh:

    Have you considered adding a list of Caucuses, Taskforces, Coalitions, etc. (eg, Blue Dogs) a congressman belongs to, to his/her description page?

    Wikipedia has a list of Caucuses and their officers, but not members.

    It would really be helpful for understanding their legislative philosophies.

Harvey

On 12/6/2012 11:39 AM, Josh Tauberer wrote:

 

Hi, all. Here are my current plans for the 113th Congress.

New Members
-------------------

I hope to get info up on the new Members of Congress right after the current Congress adjourns. If I recall, that's usually around the 20th. It'll go up in the main people.xml file as well as in people.xml in the soon-to-be-created 113 directory.

Deprecating people.xml
------------------------------

With the start of the 113th Congress, the people.xml files will be deprecated (meaning still available but not recommended). The recommended way to get that info will be through the new congress-legislators project on github (https://github.com/unitedstates/congress-legislators) which is a collaboration between Eric Mill (Sunlight), Derek Willis, and myself (and anyone else who would like to help out).

You can read more about that in Alex Howard's article today on it:
http://radar.oreilly.com/2012/12/the-united-states-code-is-on-github.html

The existing people.xml files will probably be deleted at the end of 2013, giving you a year to switch to the new project.

Possible New Bill XML
----------------------------

I'm not sure, but starting with the 113th Congress I may replace my old Perl-based THOMAS scrapers for bill data with the new congress project ( https://github.com/unitedstates/congress; same collaborators). The new scrapers should produce bill status XML (i.e. 113/bills/*) in exactly the same format as the existing XML. But there may be bugs and some small changes. I'll write again when I know more about that.

If that goes well, the existing bill XML will actually be deprecated in 2014 in favor of the new scraper's JSON output, and with any luck the current XML format will be permanently retired starting with the 114th Congress. (I'll continue to make available existing bill data indefinitely.)

Anyway, more on that later.

Undocumented Data
---------------------------

The undocumented repstats and repstats.person will not be created for the 113th Congress. Instead I'll be creating a new stats directory with better information (e.g. missed votes %'s). The existing directories will probably be deleted by summer 2013.

If you're using anything else undocumented, you might want to drop me a line to check on its status for next year as I'm doing a ton of internal cleanups.

Thanks,

-- - Josh Tauberer (@JoshData)
http://razor.occams.info

I would like to learn some more about the files contained in th "db" directory:
database.sql and database.people.sql.

* Are these files actively being used either by GovTrack itself or some second
party consumer of GovTrack data?
* How often are they updated? The latest modification date as of today
(12/12/2012) is 7/9/2012.
* Is the script or program that created these files available?

I would be interested in working on creating a relational database of a subset
of GovTrack data -- something that could be kept current with the latest updates
of GovTrack's data.

Hi, John.

> * Are these files actively being used either by GovTrack itself or some second
party consumer of GovTrack data?

They're not really in the 'supported' set of files, and they will
probably disappear in a few months when I transition to the new
https://github.com/unitedstates/congress and
https://github.com/unitedstates/congress-legislators backends.

The best way to get info on Members of Congress is the new
https://github.com/unitedstates/congress-legislators project.

For other data, the best way to get it is via the raw data XML
(http://www.govtrack.us/developers/data) or the beta API
(http://www.govtrack.us/developers/api) or the new scraper project
https://github.com/unitedstates/congress.

- Josh Tauberer (@JoshData)

http://razor.occams.info

On 12/12/2012 08:38 PM, johnpankowicz wrote:
> I would like to learn some more about the files contained in th "db"
directory: database.sql and database.people.sql.
>
> * Are these files actively being used either by GovTrack itself or some second
party consumer of GovTrack data?
> * How often are they updated? The latest modification date as of today
(12/12/2012) is 7/9/2012.
> * Is the script or program that created these files available?
>
> I would be interested in working on creating a relational database of a subset
of GovTrack data -- something that could be kept current with the latest updates
of GovTrack's data.
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>

Hi Josh,

Thank for the info. Are you aware of any work being done to populate data from
the github projects into a relational database?

John

--- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
>
> Hi, John.
>
> > * Are these files actively being used either by GovTrack itself or some
second party consumer of GovTrack data?
>
> They're not really in the 'supported' set of files, and they will
> probably disappear in a few months when I transition to the new
> https://github.com/unitedstates/congress and
> https://github.com/unitedstates/congress-legislators backends.
>
> The best way to get info on Members of Congress is the new
> https://github.com/unitedstates/congress-legislators project.
>
> For other data, the best way to get it is via the raw data XML
> (http://www.govtrack.us/developers/data) or the beta API
> (http://www.govtrack.us/developers/api) or the new scraper project
> https://github.com/unitedstates/congress.
>
> - Josh Tauberer (@JoshData)
>
> http://razor.occams.info
>
> On 12/12/2012 08:38 PM, johnpankowicz wrote:
> > I would like to learn some more about the files contained in th "db"
directory: database.sql and database.people.sql.
> >
> > * Are these files actively being used either by GovTrack itself or some
second party consumer of GovTrack data?
> > * How often are they updated? The latest modification date as of today
(12/12/2012) is 7/9/2012.
> > * Is the script or program that created these files available?
> >
> > I would be interested in working on creating a relational database of a
subset of GovTrack data -- something that could be kept current with the latest
updates of GovTrack's data.
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> >
>

Thanks Eric,

I will read through your code before I start working on mine. I first need to
learn a bit of Ruby. I've programmed in many languages but haven't yet worked
with Ruby. I've been wanting to learn it anyway.

John

--- In govtrack@yahoogroups.com, Eric Mill <eric@...> wrote:
>
> Hi John,
>
> I'm not loading it into a relational database, but I am loading data on
> people, committees, and bills from that data on
> github.com/unitedstatesinto a non-relational schema-less database
> (MongoDB) for serving as part of
> Sunlight's Congress API.
>
> So if it's any help to you, here are real-world examples of processing and
> transforming that data:
>
>
https://github.com/sunlightlabs/congress/blob/master/tasks/legislators/legislato\
rs.rb
>
https://github.com/sunlightlabs/congress/blob/master/tasks/committees/committees\
.rb
> https://github.com/sunlightlabs/congress/blob/master/tasks/bills/bills.rb
>
> -- Eric
>
>
> On Thu, Dec 13, 2012 at 10:35 AM, johnpankowicz
> <JohnPankowicz@...>wrote:
>
> > Hi Josh,
> >
> > Thank for the info. Are you aware of any work being done to populate data
> > from the github projects into a relational database?
> >
> > John
> >
> > --- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@> wrote:
> > >
> > > Hi, John.
> > >
> > > > * Are these files actively being used either by GovTrack itself or
> > some second party consumer of GovTrack data?
> > >
> > > They're not really in the 'supported' set of files, and they will
> > > probably disappear in a few months when I transition to the new
> > > https://github.com/unitedstates/congress and
> > > https://github.com/unitedstates/congress-legislators backends.
> > >
> > > The best way to get info on Members of Congress is the new
> > > https://github.com/unitedstates/congress-legislators project.
> > >
> > > For other data, the best way to get it is via the raw data XML
> > > (http://www.govtrack.us/developers/data) or the beta API
> > > (http://www.govtrack.us/developers/api) or the new scraper project
> > > https://github.com/unitedstates/congress.
> > >
> > > - Josh Tauberer (@JoshData)
> > >
> > > http://razor.occams.info
> > >
> > > On 12/12/2012 08:38 PM, johnpankowicz wrote:
> > > > I would like to learn some more about the files contained in th "db"
> > directory: database.sql and database.people.sql.
> > > >
> > > > * Are these files actively being used either by GovTrack itself or
> > some second party consumer of GovTrack data?
> > > > * How often are they updated? The latest modification date as of today
> > (12/12/2012) is 7/9/2012.
> > > > * Is the script or program that created these files available?
> > > >
> > > > I would be interested in working on creating a relational database of
> > a subset of GovTrack data -- something that could be kept current with the
> > latest updates of GovTrack's data.
> > > >
> > > >
> > > >
> > > > ------------------------------------
> > > >
> > > > Yahoo! Groups Links
> > > >
> > > >
> > > >
> > >
> >
> >
> >
> >
> > ------------------------------------
> >
> > Yahoo! Groups Links
> >
> >
> >
> >
>
>
> --
> Developer | sunlightfoundation.com
>

Hello!

Most of you will probably get this like ten times, but-

The next International Open Data Day is February 23, 2013.

Eric Mill and I are organizing a hackathon in DC. It'll be one of several dozen similar events throughout the world happening on that day (see www.opendataday.org).

This will be DC's second open data day hackathon. Last year we worked on coding and data analysis projects related to government data. This year we'll be doing that, plus design challenges, and we'll also be holding an introductory tutorial on open data. You don't need any particular skills to get something out of this hackathon.

    Register now: http://opendataday2013dc.eventbrite.com/

Hope to see you in February.

-- - Josh Tauberer (@JoshData)
http://razor.occams.info

-- - Josh Tauberer (@JoshData)
http://razor.occams.info

Now that GovTrack is switching over to a new set of scrapers co-authored
by Eric Mill at Sunlight, I've had to take the legacy site down since it
was no longer getting new data. I've also taken down the legacy Google
Maps layers.

Legacy Site
-----------

The legacy site was the pre-2.0 govtrack running at legacy.govtrack.us.

If you were using any legacy URLs and need help switching to the current
site, please let me know how I can help.

If you were using any old APIs, please drop me a line. You can try using
the new API at govtrack.us/developers/api, but I have a 2.0 API in the
works and I can give you pointers on using that instead.

Sorry for the late notice. It just occurred to me that the legacy site
would not know about the data from the new scrapers.

Legacy Maps
-----------

This was the map tile layer at govtrack.us/perl/wms/wms.cgi. This is now
gone. It now gives 403s.

The replacement is http://gis.govtrack.us/map/demo/cd-2012/.

It is very easy to deploy.

--
- Josh Tauberer (@JoshData)

http://razor.occams.info

The GovTrack people database has had entries for vice presidents and
other people who served neither as a Members of Congress nor as a president.

These folks have had <person> elements in people.xml but no <role> elements.

Is anyone using those people IDs?

Those entries are liable to dissapear. If VPs ever come back, they'll
get issued new IDs. Unless this poses a problem for anyone.

--
- Josh Tauberer (@JoshData)

http://razor.occams.info

Just an FYI, 78 people in the people.xml for 113 are missing bioguide ids.
Possibly when file was updated they didn't have bioguide ids, but at least some
of them have them now. (e.g. Sen. Brian Schatz
http://bioguide.congress.gov/scripts/biodisplay.pl?index=S001194 )

unitedstates/congress-legislators seems to have them, though.

On Jan 2, 2013, at 8:24 PM, Josh Tauberer wrote:

> The roster of the 113th Congress is up.
>
> * It's in the people.xml files, e.g.:
http://govtrack.us/data/us/113/people.xml
>
> * As well as the GovTrack API.
>
> * And additionally the congress-legislators project on github, currently in
the 113th_Congress branch:
https://github.com/unitedstates/congress-legislators/tree/113th_Congress which
will be merged with master when the 112th Congress adjourns.
>
> Note that the 112th Congress will adjourn tomorrow and the 113th Congress will
also possibly begin tomorrow, so end dates of reelected Members's previous term
may have the same value as their next start dates.
>
> As I mentioned previously, the people.xml files will disappear at the end of
2013 (or 2014? I forget what I said.). The new way to get people data is one of
the other methods listed above. Although the API v1 is going to be deprecated
soon too when I finish v2.
>
> Best,
>
> --
> - Josh Tauberer (@JoshData)
>
>
> http://razor.occams.info
>
>

--
Francis Avila
Senior Developer
Dancing Mammoth, Inc.
(Formerly PJ Doland Web Design, Inc.)
P: 703.621.0990
E: favila@...
http://dancingmammoth.com

Thanks, yeah.

I've just posted updated people.xml (us and us/113) that draws in data
from the new congress-legislators db rather than my legacy db, and this
fills in all of the missing attributes. Sorry for the delay.

Thanks to Gordon who wrote the script to do the conversion
(https://github.com/govtrack/legacy-conversion/blob/master/convert_people.py).

Just as a reminder, as mentioned in a previous email, the people.xml
files will continue to be updated throughout 2013 but are now deprecated
and will disappear at the end of 2013.

I can now retire my legacy people database, which feels good.

- Josh Tauberer (@JoshData)

http://razor.occams.info

On 01/09/2013 02:53 AM, Francis Avila wrote:
> Just an FYI, 78 people in the people.xml for 113 are missing bioguide ids.
Possibly when file was updated they didn't have bioguide ids, but at least some
of them have them now. (e.g. Sen. Brian Schatz
http://bioguide.congress.gov/scripts/biodisplay.pl?index=S001194 )
>
> unitedstates/congress-legislators seems to have them, though.
>
> On Jan 2, 2013, at 8:24 PM, Josh Tauberer wrote:
>
>> The roster of the 113th Congress is up.
>>
>> * It's in the people.xml files, e.g.:
http://govtrack.us/data/us/113/people.xml
>>
>> * As well as the GovTrack API.
>>
>> * And additionally the congress-legislators project on github, currently in
the 113th_Congress branch:
https://github.com/unitedstates/congress-legislators/tree/113th_Congress which
will be merged with master when the 112th Congress adjourns.
>>
>> Note that the 112th Congress will adjourn tomorrow and the 113th Congress
will also possibly begin tomorrow, so end dates of reelected Members's previous
term may have the same value as their next start dates.
>>
>> As I mentioned previously, the people.xml files will disappear at the end of
2013 (or 2014? I forget what I said.). The new way to get people data is one of
the other methods listed above. Although the API v1 is going to be deprecated
soon too when I finish v2.
>>
>> Best,
>>
>> --
>> - Josh Tauberer (@JoshData)
>>
>>
>> http://razor.occams.info
>>
>>

I believe GovTrack would be keeping them, but assigning them new IDs?

-- Eric

- Josh Tauberer (@JoshData)
http://razor.occams.info

I believe GovTrack would be keeping them, but assigning them new IDs?

-- Eric

On Jan 9, 2013 10:46 AM, "Daniel Schuman" <dschuman@...> wrote:

VPs, as presidents of the Senate, may be worth keeping. I can't speak to the others.

On Tue, Jan 8, 2013 at 8:32 PM, Josh Tauberer <tauberer@...> wrote:

vTrack people database has had entries for vice presidents and
other people who served neither as a Members of Congress nor as a president.

These folks have had elements in people.xml but no elements.

Daniel

Daniel Schuman
Director | Advisory Committee on Transparency
Policy Counsel | The Sunlight Foundation
o: 202-742-1520 x 273 | c: 202-713-5795 | @danielschuman

I was wondering how nicknames are assigned to people? I see that it's Bob Casey,
but Charles Schumer. And it's Michael Crapo, not Mike Crapo, even though his
official website refers to him as Mike (http://www.crapo.senate.gov/).

Appreciate the insight....

jason

It's a little haphazard for senators, and in some cases these may go
back to the database I started with in 2003. In the House the data now
is based on how Members of Congress are referred to by the House Clerk,
nickname and all. Starting this week, it's all via the
https://github.com/unitedstates/congress-legislators project (patches
welcome).

Members of Congress are very often referred to in different ways in
different places, and they change how they want to be referred to over
time, so in the past I didn't typically update these things until there
was an obvious problem.

On GovTrack, nicknames typically appear in quotes, but on the votes page
I use nicknames alone in some places so the names fit in the small space
allocated for them.

- Josh Tauberer (@JoshData)

http://razor.occams.info

On 01/10/2013 02:45 PM, Jason B wrote:
> I was wondering how nicknames are assigned to people? I see that it's Bob
Casey, but Charles Schumer. And it's Michael Crapo, not Mike Crapo, even though
his official website refers to him as Mike (http://www.crapo.senate.gov/).
>
> Appreciate the insight....
>
> jason
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>

--- In govtrack@yahoogroups.com, Josh Tauberer  wrote:
>
> It's a little haphazard for senators, and in some cases these may go
> back to the database I started with in 2003. In the House the data now
> is based on how Members of Congress are referred to by the House Clerk,
> nickname and all. Starting this week, it's all via the
> https://github.com/unitedstates/congress-legislators project (patches
> welcome).
>

Appreciate the reply. I'm trying to match against an existing database, which I
believe has the "common" name listed, but the matching is all over the place.

jason

For current Members of Congress, state & district (House) or state &
lastname (Senate) often works well --- as long as the list is definitely
up to date. (I'm assuming you have no IDs on the list used in any other
public project.)

- Josh Tauberer (@JoshData)

http://razor.occams.info

On 01/10/2013 03:10 PM, Jason B wrote:
> --- In govtrack@yahoogroups.com, Josh Tauberer  wrote:
>> It's a little haphazard for senators, and in some cases these may go
>> back to the database I started with in 2003. In the House the data now
>> is based on how Members of Congress are referred to by the House Clerk,
>> nickname and all. Starting this week, it's all via the
>> https://github.com/unitedstates/congress-legislators project (patches
>> welcome).
>>
> Appreciate the reply. I'm trying to match against an existing database, which
I believe has the "common" name listed, but the matching is all over the place.
>
> jason
>
>
>
> ------------------------------------
>
> Yahoo! Groups Links
>
>
>