Mark Dilger wrote:
> Could anyone point me to a list of elected officials
> over time.  I expect that no single list has
> everything I want, but the ideal list would have all
> senators, representatives, presidents, etc. etc. with
> their names, positions, and such going back in history
> as far as possible.  Computer readable lists (xml, tab
> delimited, csv, postgres or mysql dumps) are
> preferable, but I can write the scripts to parse HTML,
> PDF or whatever if I need to.

GovTrack's people.xml file (see the about -> source data page) has all
congresspeople ever elected, based on bioguide.congress.gov.

> I am looking for the same data from all countries.

That's very ambitious!

--
- Josh Tauberer

http://razor.occams.info

"Yields falsehood when preceded by its quotation!  Yields
falsehood when preceded by its quotation!" Achilles to
Tortoise (in "Gödel, Escher, Bach" by Douglas Hofstadter)

Joe Germuska wrote:
> Has there been any discussion of a machine tag convention for GovTrack
> "nodes"?  (see http://machinetags.org/ or
> http://en.wikipedia.org/wiki/Machine_tag
> <http://en.wikipedia.org/wiki/Machine_tag>)  If not, is there any interest?

Hi, Joe.

I'm not a particularly big fan of tags. I see them as a very short term,
unorganized solution to a much larger problem. As time goes on, the
likelihood of cashes of tag names grows and grows (i.e. ambiguity of
what the tag was meant to refer to), and as people try to push more
information inside of a tag, we get stuck trying to parse the
information back out, not knowing whether it was intended or not.

My preferred approach is using semantic web methods, like URIs for
identifying things and RDF for encoding structured data.

> I could imagine a lot of value to the community of GovTrack users
> tagging del.icio.us <http://del.icio.us> bookmarks or whatnot with tags
> that could be linked back to specific bills.

And so from that perspective, I'm not opposed to adding things like that
to the site, but people would have to point me in the right direction
for what to do.

> govtrack:bill=h110-2206

In the ideal world, I would use a globally unique identifier for each bill.

> One could also suggest a format for "people" tags, like
> govtrack:person=300038 (Dick Durbin)  Not sure if there is any usable
> unique ID that would make it more "shared" than govtrack.

There are IDs assigned with the Library of Congress (see
bioguide.congress.gov) which are neutral ground, although they have some
issues like multiple identifiers assigned to the same individual ---
after women change their name (at least in their historical data, which
might just be unintentional).

> For people like Dick Durbin, the likelihood of users selecting the same
> tag intuitively is much higher than for tagging a bill.

And the likelihood that other Dick Durbin's in the world get conflated
with our congressional guy also is higher, which is why I don't like
tags much.

But like I said, I'm not deeply opposed to using them on the website.

- Josh

Personally, I love the idea of tagging, though not necessarily as
something that would be on the front page. It's on my todo list at
critter watch, though admittedly it's pretty far down the list.

But yeah, there are a number of problems with it. And some good things.
I think letting folks tag items, and other users can optionally view
them (or perhaps view the most popular tags for some items) would be a
fun experiment.

--B

Hello,

Does anyone know where I can get data that has the congressional
districts, and corresponding cities and zips within each district?

Thanks much,

Taylor

Hello,

Does anyone know where I can get data that has the congressional
districts, and corresponding cities and zips within each district?

Thanks much,

Taylor

Thanks much for the links to the specific data.

The zip code link leads to another link below that is not working.

http://www.census.gov/geo/www/bob/bdy_files.html

I think it's referencing this page...

http://www.census.gov/geo/www/tiger/zip1999.html

Looks like the most recent zip data is 1999.

The post office charges $900 to get the most recent! :o

Any other ideas?

Taylor



The link for the zip code file

--- In govtrack@yahoogroups.com, "Jeremy Dunck" <jdunck@...> wrote:
>
> districts: http://www.census.gov/geo/www/cob/cd_metadata.html
> cities: http://www.census.gov/geo/www/tiger/
> zip codes:
>
http://postgis.refractions.net/pipermail/postgis-users/2006-July/012671.html
>
> On 6/22/07, tay199 <tay199@...> wrote:
> >
> >   Hello,
> >
> > Does anyone know where I can get data that has the congressional
> > districts, and corresponding cities and zips within each district?
> >
> > Thanks much,
> >
> > Taylor
> >
> >
> >
>

Try the 2006 Tiger Census Data, I have it downloaded.  Check here:
http://www.census.gov/geo/www/tiger.  I would send them to ya, but it's
well over 650GB.

Thought some might be interested.

- Josh

-------- Original Message --------
Subject:  [openhouseproject] New Registry of State agency databases
Date:  Sat, 30 Jun 2007 11:20:55 -0900
From:  Daniel Cornwall <daniel.cornwall@...>
Reply-To:  openhouseproject@googlegroups.com
To:  <openhouseproject@googlegroups.com>



Hello,

I realize this group is focused on federal level information, but I
thought some of you might interested in a new American Library
Association Government Documents Roundtable project that will attempt to
provide annotated links to public databases produced by the fifty state
governments plus the District of Columbia.

The main project page can be found at
http://wikis.ala.org/godort/index.php/State_Agency_Databases. At this
moment in time, only the Alaska page is close to being finished, but
viewing it will give you a flavor for the proposed project. The Alaska
page is at http://wikis.ala.org/godort/index.php/Alaska.

If you'd like to help, please adopt a state and start adding databases
using the Alaska page as a template. If you don't like the Alaska page,
click on the "talk" tab of
http://wikis.ala.org/godort/index.php/State_Agency_Databases and suggest
a better format.

Otherwise, just watch the space because it will become progressively
more useful. - Daniel Cornwall, Free Government Information,
dnlcornwall@...

--~--~---------~--~----~------------~-------~--~----~
You received this message because you are subscribed to the Google
Groups "Open House Project" group.
To post to this group, send email to openhouseproject@googlegroups.com
To unsubscribe from this group, send email to
openhouseproject-unsubscribe@googlegroups.com
For more options, visit this group at
http://groups.google.com/group/openhouseproject?hl=en
-~----------~----~----~----~------~----~------~--~---


--
- Josh Tauberer

http://razor.occams.info

"Yields falsehood when preceded by its quotation!  Yields
falsehood when preceded by its quotation!" Achilles to
Tortoise (in "Gödel, Escher, Bach" by Douglas Hofstadter)

Group,

I've publicly posted some of the screen-scraping Perl scripts GovTrack
uses to collect its data. I'll be adding my scripts to the Subversion
repository below over time with the purpose of making it possible for
others to maintain, extend, and improve them.

The scripts are not released under an open source license for a variety
of reasons, among which is the fact that open source licenses don't
enforce the type of open information model I'd want the *fruits* of the
scripts to be covered under. Since wide distribution of the scripts
isn't necessary to benefit from them, an open source license would be
teethless.

The repository is browsable here:
http://razor.occams.info/code/repo/?/govtrack/gather/us

I don't know quite what it would take to make any of the scripts
actually work for you, though. Certainly you'll need a 'data' directory
in the right place, and possibly MySQL tables in the right place, Perl
modules, etc. If you try to get them working and can't, or make
progress, post any notes here, or better yet post patches to make set-up
easier. Also, the code is heavily *un*commented, and well... that's just
the way it will be.

All patches are welcome.

--
- Josh Tauberer

http://razor.occams.info

"Yields falsehood when preceded by its quotation!  Yields
falsehood when preceded by its quotation!" Achilles to
Tortoise (in "Gödel, Escher, Bach" by Douglas Hofstadter)

--- In govtrack@yahoogroups.com, Josh Tauberer <tauberer@...> wrote:
> The repository is browsable here:
> http://razor.occams.info/code/repo/?/govtrack/gather/us

Thank you Josh as always.

Any way you could just document what each file does?

Here's list:
database.people.sql
database.tables2.sql
database.tables.sql
db.pl
general.pl
indexing.pl
parse_record.pl
parse_rollcall.pl
parse_status.pl
personaldb.pl
sql.pl
util.pl

(p.s. EXCELLENT article on XML.com... That's how I found your site)

Also could you maybe send us more or less the pages you scrape?
I'll probably get them from reading the script, but I'm a php guy, not
perl...but a programmer is a programmer right? I should be able to
figure it out.

--- In govtrack@yahoogroups.com, "damianmont" <photoca@...> wrote:
> Any way you could just document what each file does?
>
> Here's list:

Sure.

> database.people.sql
MySQL table schema and data for "people" table of all people that have
served in Congress (name, birthday, etc.), and "people_roles" table
for every role in Congress each person has served (role type
(senator/representative), start/end date, party, etc.).

> database.tables2.sql
> database.tables.sql
MySQL table schemas for other tables that are filled in by the
scripts, mostly for indexing bills. The people tables are the only
ones I edit by hand and are not automatically generated from some
other source.

You'll need to pipe these to mysql, or otherwise load them, for any of
the scripts to work. (The indexing tables aren't strictly necessary if
you disable the indexing routines one way or another, but the people
tables are pretty critical for all of the parsing scripts.)

> db.pl
Utility script for opening the MySQL database.

> general.pl
Really old utility functions that I don't really use.

> indexing.pl
Subroutines that update the indexing MySQL tables based on the
contents of a bill or vote file.

> parse_record.pl
Parses the Congressional Record from THOMAS.

> parse_rollcall.pl
Parses roll call pages from the House and Senate websites.

> parse_status.pl
Parses bill status pages from THOMAS.

> personaldb.pl
Converts a name of a representative into an ID. Considers a date, role
type, and state/district info to disambiguate names when it's ambiguous.

> sql.pl
Utility functions for dealing with MySQL (preparing SQL statements
programmatically).

> util.pl
A ton of utility functions used throughout.

> (p.s. EXCELLENT article on XML.com... That's how I found your site)

Thanks!

> Also could you maybe send us more or less the pages you scrape?
> I'll probably get them from reading the script, but I'm a php guy, not
> perl...but a programmer is a programmer right? I should be able to
> figure it out.

That's a long list. Maybe another time!

- Josh

I suspect anyone on this list using Python will be interested in this:
http://holovaty.com/blog/archive/2007/07/06/0128

Can I have your opinion Joshua (and anyone else that wants to chime in)

Your excellent code points to you checking the most recent bill action at:
http://thomas.loc.gov/cgi-bin/query/B?r110:@FIELD(FLD003+h)+@FIELD(DDATE+2007062\
9)
(say for the "house" and for the date: 06/29/2007)

What about looking in the following directory:
http://thomas.loc.gov/home/gpoxmlc110/

They seem to have an update list all in xml format.

Your opinion on it's quality. Had you seen it before? do you think
it'll be there for a while?

--- In govtrack@yahoogroups.com, "damianmont" <photoca@...> wrote:
>
> Can I have your opinion Joshua (and anyone else that wants to chime in)
>
> Your excellent code points to you checking the most recent bill
action at:
>
http://thomas.loc.gov/cgi-bin/query/B?r110:@FIELD(FLD003+h)+@FIELD(DDATE+2007062\
9)
> (say for the "house" and for the date: 06/29/2007)
>
> What about looking in the following directory:
> http://thomas.loc.gov/home/gpoxmlc110/
>
> They seem to have an update list all in xml format.
>
> Your opinion on it's quality. Had you seen it before? do you think
> it'll be there for a while?
>

I happen to write something about that a few days ago:
http://www.theopenhouseproject.com/2007/07/04/legislative-xml-what-we-have-and-w\
hat-were-seeking/

The first part of that post describes the (gpoxml) files you found.

- Josh

YEAH! Thanks Josh. I'll keep the group posted on any patches or things
of value we find and can contribute.

Taylor

Great, thanks Josh...

I didn't do much scripting (proper) for http://www.whereabill.org/,
but since Josh has started the ball rolling, I'll add in the one
script I did write: an XSLT file that takes the XML version of Josh's
people database
(http://www.govtrack.us/data/us/110/repstats/people.xml) and extracts
the people/roles relevant to a single specified Congressional session.

The problem (for my purposes, which include sending this information
to the browser client on each request) is that the people.xml file is
large (6.3MB), and includes lots of dead people. :) So I use this
script to get only the information for a particular Congress.

Kevin


bioseparate.xml:

<?xml version="1.0" encoding="UTF-8"?>
<xsl:stylesheet version='1.0'
xmlns:xsl='http://www.w3.org/1999/XSL/Transform'
xmlns:exsl="http://exslt.org/common">
<xsl:output method="xml" version="1.0" encoding="ISO-8859-1"
indent="yes"/>

<xsl:param name="congress">110</xsl:param>

<xsl:template match="people">
	 <xsl:copy>
	 <xsl:variable name="striprole">
		 <xsl:for-each select="person">
			 <xsl:copy>
			 <xsl:copy-of select="@*"/>
			 <xsl:copy-of select="role[(((number($congress)*2 + 1787) >=
number(substring-before(@startdate,'-'))) and ((number($congress)*2 +
1787) <= number(substring-before(@enddate,'-')))) or
(((number($congress)*2 + 1788) >=
number(substring-before(@startdate,'-'))) and ((number($congress)*2 +
1788) <= number(substring-before(@enddate,'-'))))]"/>
			 </xsl:copy>
		 </xsl:for-each>
	 </xsl:variable>
	 <xsl:for-each select="exsl:node-set($striprole)/person[role]">
		 <xsl:copy-of select="."/>
	 </xsl:for-each>
	 </xsl:copy>
</xsl:template>

</xsl:stylesheet>

Kevin,

Love that www.WhereaBill.org site.
You use the information from josh's xml files?

Love the mashup, very well done.

--- In govtrack@yahoogroups.com, "Kevin Henry" <k@...> wrote:
>
> Great, thanks Josh...
>
> I didn't do much scripting (proper) for http://www.whereabill.org ,
> but since Josh has started the ball rolling, I'll add in the one
> script I did write: an XSLT file that takes the XML version of Josh's
> people database
> (http://www.govtrack.us/data/us/110/repstats/people.xml) and extracts
> the people/roles relevant to a single specified Congressional session.

Kevin,

Love that www.WhereaBill.org site.
You use the information from josh's xml files?

Love the mashup, very well done.

--- In govtrack@yahoogroups.com, "Kevin Henry" <k@...> wrote:
>
> Great, thanks Josh...
>
> I didn't do much scripting (proper) for http://www.whereabill.org ,
> but since Josh has started the ball rolling, I'll add in the one
> script I did write: an XSLT file that takes the XML version of Josh's
> people database
> (http://www.govtrack.us/data/us/110/repstats/people.xml) and extracts
> the people/roles relevant to a single specified Congressional session.

Peggy and Damian,

Thanks, glad you like the site.

I'm getting all the data from govtrack. Specifically, I'm using the
following files:

- the bill status data (www.govtrack.us/data/us/*/bills/*.xml)
- the roll vote data (/data/us/*/rolls/*.xml)
- the people database (/data/us/110/repstats/people.xml)
- the popularity listing (/data/us/bills.technorati.xml)
- the search service (/congress/billsearch_api.xpd)

I keep a copy of all the files on my server, and do a daily rsync (as
Josh describes here: http://www.govtrack.us/source.xpd) to stay current.

Basically, when the server gets a request for a certain bill, it
retrieves the status data and goes through the action items, parsing
them into the steps that will be represented in the "driving
directions". It also retrieves any relevant roll vote data, and then
sends that information (along with the summary, the titles, the list
of sponsors, and the biographical data for that session of Congress)
back to the client, which renders everything (with the help of the
Google Maps API).

So it's really a (sort-of) UI sitting on top of govtrack's (sort-of) API.

Let me know if you need any more information...


Regards,
Kevin
http://www.whereabill.org/

Hi all,

Just wanted to drop a quick line of introduction.  I was turned on to
the list at Josh's recommendation. I am a 30-year-old PHP programmer and
writer living in Akron, Ohio.

Once upon a time I tried to start a site that treated the various
Indymedias as news wires, but after a few months (and this was before
RSS was as well-developed as it is today) I burned out and quit.

I have a few ideas on an issue-tracking site similar to GovTrack and
soonishly, I'll formulate them into a proposal for open source
development.

Good to meet you all!

Nathan

Hello,

Alaska is taking over my map!!!
http://72.32.197.213/govit/Pages/search/mapchartforgovleader.aspx

I'd really like to decrease the size of Alaska, and move Alaska and
Hawaii under the mainland.

My boundary files are from... http://www.census.gov/geo/www/cob/cd110.html

Can anyone help me by telling me how I can adjust the size and
locations of Alaska and Hawaii?

Thank you,

Taylor

Howdy,

Nice to meet you too. My name is Taylor and I'm a web project manager
from San Francisco. That means I was not a good enough coder to be a
programmer, or an artist to be a designer. ;)

Looking forward to hearing about your project.

Cheers,

Taylor