https://rawgithub.com/The-Politico/src/master/images/logo/badge.png

Welcome to django-politico-civic’s documentation!

Why this?

At POLITICO, civic data is a key component of our report. We record the results of every federal election in the country. We track the movements of prospective presidential candidates. We collect campaign finance data. All of these data tasks require the same foundation of data. Thus, the POLITICO Civic project was born.

If you work in a newsroom or with any civic data project, you might have similar problems to solve. Newsrooms across the United States spend many months every two years (at least) building the same piece of technology: a system to ingest election results as quickly as possible and display them with fancy data visualizations. By sharing POLITICO Civic, we hope to deescalate that arms race.

What is it?

POLITICO Civic is a Django project composed of a number of pluggable Django apps. Each of the pluggable apps contains models around a particular facet of civic data and standard serializers that allow us to pass data around through JSON. At the bottom of the app tree, the apps contain front-end applications for live results, election calendars and other data-driven displays.

To give an illustration of how POLITICO Civic works, here is the election results app tree and its dependency structure described in a big scary dependency diagram.

_images/apptree.png

Don’t run away! I promise this makes sense!

Benefits

Modularity: We designed our project this way to make each component of civic data easier to reason about. And when we start supporting other types of civic data, we don’t have to add bloat to another Django application. We’ve designed in a way that allows us to start that app from scratch and pull in the dependencies we need.

Predictable structure: During high-stress live events such as election nights, having a strong schema and foundation for all of the underlying data that goes into an election night — information about political offices, geography, election cycles, primary conditions, and more — lets us breathe a little easier and focus more on what is new about that night: the results.

Reusability: Some of these applications are useful outside the context of POLITICO Civic. For example, Geography contains all of the geographic data for political divisions in the United States. It comes with a bootstrap process built-in that grabs the latest geodata for states, counties and Congressional districts from the U.S. Census Bureau. It can also compress that data into topojson and dump it to Amazon S3. That is useful for any newsroom that might make maps of the United States.

Core technologies

POILTICO Civic is based on several key pieces of technology:

Our model architecture took inspiration from a couple inspired projects:

Using component apps in Django

The component apps in politico-civic are designed to be plug-and-play. You can install any of them in your own Django project and they should work within your project and install all their necessary dependencies. Each app contains its own bootstrap management command that will seed your models with real data.

For example, let’s install politico-civic-vote in a Django project. You can follow these steps for any of POLITICO Civic’s component apps.

First, you need to set up your Django project with a PostgreSQL database. Read the Django docs on databases if you don’t know how to do this.

Then, install the component application.

$ pip install politico-civic-vote

In your Django settings, add the app and its dependencies to your INSTALLED_APPS section. Consult the dependency diagram in the quickstart section to determine your dependencies.

INSTALLED_APPS = [
    ...
    "rest_framework",
    "entity",
    "geography",
    "government",
    "election",
    "vote",
]

Then, migrate your database.

$ python manage.py migrate

No matter which component app you choose to install, you can use a Django management command to seed your database with real data. For politico-civic-vote, the command is bootstrap_vote. The naming convention extends to whichever app you isntalled. Each component app will seed its own data and the data of the apps it depends on.

Run the management command like this:

$ python manage.py bootstrap_vote

Note

If you use anything depending on politico-civic-government, you will need an API key from the ProPublica Congress API. Export it into your environment as PROPUBLICA_CONGRESS_API_KEY.

That’s it! Open your Django admin and see your seed data.

Quickstart

Use these docs if you’re trying to install the entire politico-civic project. If you don’t work at POLITICO, you probably don’t want this. Instead, install the component apps you want in your own Django project.

  1. Install global dependencies for the project:
$ brew install jq
$ pip install pipenv

Get Terraform from the project website.

  1. Create local PostgreSQL database
$ createdb civic
  1. Fill out your .env file
DATABASE_URL=“postgresql://username:password@localhost:5432/civic”
...
(get all of our API keys from someone on the team)
  1. Install local dependencies for the project:
$ pipenv install
$ pipenv shell
$ python setup.py develop
  1. Bootstrap database
$ python manage.py bootstrap_electionnight
  1. Check it out!
$ python manage.py runserver

Models

aploader

AP Loader leverages elex, a tool created by NPR and the New York Times, to get election results from the AP Elections API.

APElectionMeta

class aploader.models.APElectionMeta(*args, **kwargs)

Election information corresponding to AP election night.

Parameters:
  • id (AutoField) – Id
  • election (OneToOneField to Election) – Election
  • ballot_measure (OneToOneField to BallotMeasure) – Ballot measure
  • ap_election_id (CharField) – Ap election id
  • called (BooleanField) – Called
  • tabulated (BooleanField) – Tabulated
  • override_ap_call (BooleanField) – Override ap call
  • override_ap_votes (BooleanField) – Override ap votes
  • precincts_reporting (PositiveIntegerField) – Precincts reporting
  • precincts_total (PositiveIntegerField) – Precincts total
  • precincts_reporting_pct (DecimalField) – Precincts reporting pct

ChamberCall

class aploader.models.ChamberCall(*args, **kwargs)

Calls for chambers of Congress

Parameters:
  • id (AutoField) – Id
  • body (ForeignKey to Body) – Body
  • cycle (ForeignKey to ElectionCycle) – Cycle
  • party (ForeignKey to Party) – Party
  • call_time (DateTimeField) – Call time

demography

Demography collects and aggregates Census variables by the political divisions defined in Geography.

CensusEstimate

class demography.models.CensusEstimate(*args, **kwargs)

Individual census series estimates.

Parameters:
  • id (AutoField) – Id
  • division (ForeignKey to Division) – Division
  • variable (ForeignKey to CensusVariable) – Variable
  • estimate (FloatField) – Estimate

CensusLabel

class demography.models.CensusLabel(*args, **kwargs)

Custom labels for census variables that allow us to aggregate variables.

Parameters:
  • id (AutoField) – Id
  • label (CharField) – Label
  • aggregation (CharField) – Aggregation
  • table (ForeignKey to CensusTable) – Table

CensusTable

class demography.models.CensusTable(*args, **kwargs)

A census series.

Parameters:
  • id (AutoField) – Id
  • series (CharField) – Series
  • year (CharField) – Year
  • code (CharField) – Code
  • title (CharField) – Title

CensusVariable

class demography.models.CensusVariable(*args, **kwargs)

Individual variables on census series to pull, e.g., “001E” on ACS table 19001, the total for household income.

Parameters:
  • id (AutoField) – Id
  • code (CharField) – 3 digit code for variable and ‘E’, e.g., 001E.
  • table (ForeignKey to CensusTable) – Table
  • label (ForeignKey to CensusLabel) – Label

election

Election models information about races for particular offices. It also models candidate information, which inherits people from Entity and attaches them to races in Election.

BallotAnswer

class election.models.BallotAnswer(*args, **kwargs)

An answer to a ballot question.

Parameters:
  • id (UUIDField) – Id
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • answer (TextField) – Answer
  • winner (BooleanField) – Winner
  • ballot_measure (ForeignKey to BallotMeasure) – Ballot measure

BallotMeasure

class election.models.BallotMeasure(*args, **kwargs)

A ballot measure.

Parameters:
  • uid (CharField) – Uid
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • question (TextField) – Question
  • division (ForeignKey to Division) – Division
  • number (CharField) – Number
  • election_day (ForeignKey to ElectionDay) – Election day

Candidate

class election.models.Candidate(*args, **kwargs)

A person who runs in a race for an office.

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • race (ForeignKey to Race) – Race
  • person (ForeignKey to Person) – Person
  • party (ForeignKey to Party) – Party
  • ap_candidate_id (CharField) – Ap candidate id
  • incumbent (BooleanField) – Incumbent
  • top_of_ticket (ForeignKey to Candidate) – Top of ticket
  • prospective (BooleanField) – The candidate has not yet declared her candidacy.

CandidateElection

class election.models.CandidateElection(*args, **kwargs)

A CandidateElection represents the abstract relationship between a candidate and an election and carries properties like whether the candidate is uncontested or whether we aggregate their vote totals.

Parameters:
  • id (UUIDField) – Id
  • candidate (ForeignKey to Candidate) – Candidate
  • election (ForeignKey to Election) – Election
  • aggregable (BooleanField) – Aggregable
  • uncontested (BooleanField) – Uncontested

Election

class election.models.Election(*args, **kwargs)

A specific contest in a race held on a specific day.

Parameters:
  • uid (CharField) – Uid
  • election_type (ForeignKey to ElectionType) – Election type
  • race (ForeignKey to Race) – Race
  • party (ForeignKey to Party) – Party
  • election_day (ForeignKey to ElectionDay) – Election day
  • division (ForeignKey to Division) – Division
  • candidates (ManyToManyField) – Candidates

ElectionCycle

class election.models.ElectionCycle(uid, slug, name)
Parameters:
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • name (CharField) – Name

ElectionDay

class election.models.ElectionDay(*args, **kwargs)

A day on which one or many elections can be held.

Parameters:
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • date (DateField) – Date
  • cycle (ForeignKey to ElectionCycle) – Cycle

ElectionEvent

class election.models.ElectionEvent(*args, **kwargs)

A statewide election event

Parameters:
  • id (AutoField) – Id
  • slug (SlugField) – Slug
  • label (CharField) – Label
  • event_type (CharField) – Event type
  • dem_primary_type (CharField) – Dem primary type
  • gop_primary_type (CharField) – Gop primary type
  • election_day (ForeignKey to ElectionDay) – Election day
  • division (ForeignKey to Division) – Division
  • early_vote_start (DateField) – Early vote start
  • early_vote_close (DateField) – Early vote close
  • vote_by_mail_application_deadline (DateField) – Vote by mail application deadline
  • vote_by_mail_ballot_deadline (DateField) – Vote by mail ballot deadline
  • online_registration_deadline (DateField) – Online registration deadline
  • registration_deadline (DateField) – Registration deadline
  • poll_closing_time (DateTimeField) – Poll closing time

ElectionType

class election.models.ElectionType(*args, **kwargs)

e.g., “General”, “Primary”

Parameters:
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • ap_code (CharField) – Ap code
  • number_of_winners (PositiveSmallIntegerField) – Number of winners
  • winning_threshold (DecimalField) – Winning threshold

Race

class election.models.Race(*args, **kwargs)

A race for an office comprised of one or many elections.

Parameters:
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • description (TextField) – Description
  • office (ForeignKey to Office) – Office
  • cycle (ForeignKey to ElectionCycle) – Cycle
  • special (BooleanField) – Special

electionnight

Election Night builds live results pages based on AP data and models the text content needed on those pages.

PageContent

class electionnight.models.PageContent(*args, **kwargs)

A specific page that content can attach to.

Parameters:
  • id (UUIDField) – Id
  • content_type (ForeignKey to ContentType) – Content type
  • object_id (CharField) – Object id
  • election_day (ForeignKey to ElectionDay) – Election day
  • division (ForeignKey to Division) – Division
  • special_election (BooleanField) – Special election
  • parent (ForeignKey to PageContent) – Parent
  • featured (ManyToManyField) – Featured

PageContentBlock

class electionnight.models.PageContentBlock(*args, **kwargs)

A block of content for an individual page.

Parameters:
  • id (UUIDField) – Id
  • page (ForeignKey to PageContent) – Page
  • content_type (ForeignKey to PageContentType) – Content type
  • content (MarkdownField) – Content
  • created (DateTimeField) – Created
  • updated (DateTimeField) – Updated

PageContentType

class electionnight.models.PageContentType(*args, **kwargs)

The kind of content contained in a content block. Used to serialize content blocks.

Parameters:
  • slug (SlugField) – Slug
  • name (CharField) – Name

PageType

class electionnight.models.PageType(*args, **kwargs)

A type of page that content can attach to.

Parameters:
  • id (UUIDField) – Id
  • model_type (ForeignKey to ContentType) – Model type
  • election_day (ForeignKey to ElectionDay) – Election day
  • division_level (ForeignKey to DivisionLevel) – Set for all page types except generic election day
  • jurisdiction (ForeignKey to Jurisdiction) – Only set jurisdiction for federal pages
  • body (ForeignKey to Body) – Only set body for senate/house pages
  • office (ForeignKey to Office) – Only set office for the presidency

entity

Entity houses models for people and organizations. For example, the Republican Party is an organization, and Mitt Romney is a person. Their roles as political parties and candidates will come in downstream apps, but Entity houses the base level information about them.

ImageTag

class entity.models.ImageTag(*args, **kwargs)

Tags represent a type of image, which is used to serialize it.

Parameters:
  • id (AutoField) – Id
  • name (SlugField) – Name

Organization

class entity.models.Organization(*args, **kwargs)

An org.

Generally follows the Popolo spec: http://www.popoloproject.com/specs/organization.html

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • name (CharField) – Name
  • identifiers (JSONField) – Identifiers
  • classification (ForeignKey to OrganizationClassification) – Classification
  • parent (ForeignKey to Organization) – Parent
  • national_headquarters (CountryField) – National headquarters
  • founding_date (DateField) – Founding date
  • dissolution_date (DateField) – Dissolution date
  • summary (CharField) – A one-line biographical summary.
  • description (TextField) – A longer-form description.
  • links (ArrayField) – External web links, comma-separated.
  • created (DateTimeField) – Created
  • updated (DateTimeField) – Updated

OrganizationClassification

class entity.models.OrganizationClassification(id, name)
Parameters:
  • id (AutoField) – Id
  • name (CharField) – Name

OrganizationImage

class entity.models.OrganizationImage(*args, **kwargs)

Image attached to a person, which can be serialized by a tag.

Parameters:
  • id (AutoField) – Id
  • organization (ForeignKey to Organization) – Organization
  • tag (ForeignKey to ImageTag) – Used to serialize images.
  • image (ImageField) – Image
  • created (DateTimeField) – Created
  • updated (DateTimeField) – Updated

Person

class entity.models.Person(*args, **kwargs)

A real human being.🎵

Generally follows the Popolo spec: http://www.popoloproject.com/specs/person.html

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • last_name (CharField) – Last name
  • first_name (CharField) – First name
  • middle_name (CharField) – Middle name
  • suffix (CharField) – Suffix
  • full_name (CharField) – Full name
  • identifiers (JSONField) – Identifiers
  • gender (GenderField) – Gender
  • race (RaceField) – Race
  • nationality (CountryField) – Nationality
  • state_of_residence (StateField) – If U.S. resident.
  • birth_date (DateField) – Birth date
  • death_date (DateField) – Death date
  • summary (CharField) – A one-line biographical summary.
  • description (TextField) – A longer-form description.
  • links (ArrayField) – External web links, comma-separated.
  • created (DateTimeField) – Created
  • updated (DateTimeField) – Updated

PersonImage

class entity.models.PersonImage(*args, **kwargs)

Image attached to a person, which can be serialized by a tag.

Parameters:
  • id (AutoField) – Id
  • person (ForeignKey to Person) – Person
  • tag (ForeignKey to ImageTag) – Used to serialize images.
  • image (URLField) – Image
  • created (DateTimeField) – Created
  • updated (DateTimeField) – Updated

geography

Geography houses models for all of the geographic political divisions in the United States. It contains bootstrap scripts that get shapefiles from the Census Bureau for states, counties and congressional districts and load them into your database. It also creates a simplified geography for each of those objects.

Division

class geography.models.Division(*args, **kwargs)

A political or administrative geography.

For example, a particular state, county, district, precinct or municipality.

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • name (CharField) – Name
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • parent (ForeignKey to Division) – Parent
  • level (ForeignKey to DivisionLevel) – Level
  • code (CharField) – Code representing a geography: FIPS code for states and counties, district number for districts, precinct number for precincts, etc.
  • code_components (JSONField) – Component parts of code
  • effective (BooleanField) – Effective
  • effective_start (DateTimeField) – Effective start
  • effective_end (DateTimeField) – Effective end
  • intersecting (ManyToManyField) – Intersecting divisions intersect this one geographically but do not necessarily have a parent/child relationship. The relationship between a congressional district and a precinct is an example of an intersecting relationship.

DivisionLevel

class geography.models.DivisionLevel(*args, **kwargs)

Level of government or administration at which a division exists.

For example, federal, state, district, county, precinct, municipal.

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • name (CharField) – Name
  • parent (ForeignKey to DivisionLevel) – Parent

Geometry

class geography.models.Geometry(*args, **kwargs)

The spatial representation (in topoJSON) of a Division.

Parameters:
  • id (UUIDField) – Id
  • division (ForeignKey to Division) – Division
  • subdivision_level (ForeignKey to DivisionLevel) – Subdivision level
  • simplification (FloatField) – Minimum quantile of planar triangle areas for simplfying topojson.
  • topojson (JSONField) – Topojson
  • source (URLField) – Link to the source of this geography data.
  • series (CharField) – Year of boundary series, e.g., 2016 TIGER/Line files.
  • effective (BooleanField) – Effective
  • effective_start (DateField) – Effective start
  • effective_end (DateField) – Effective end

IntersectRelationship

class geography.models.IntersectRelationship(*args, **kwargs)

Each IntersectRelationship instance represents one side of a paired relationship between intersecting divisions.

The intersection field represents the decimal proportion of the to_division that intersects with the from_division. It’s useful for apportioning counts between the areas, for example, population statistics from census data.

Parameters:
  • id (AutoField) – Id
  • from_division (ForeignKey to Division) – From division
  • to_division (ForeignKey to Division) – To division
  • intersection (DecimalField) – The portion of the to_division that intersects this division.

government

Government contains information about political jurisdictions, bodies, and offices. For example, the United States Federal Government is a jurisdiction, the U.S. Senate is a body, and the Class 1 Senate seat from Texas is an office. It also contains the modeling for political parties.

Body

class government.models.Body(*args, **kwargs)

A body represents a collection of offices or individuals organized around a common government or public service function.

For example: the U.S. Senate, Florida House of Representatives, Columbia City Council, etc.

Note

Duplicate slugs are allowed on this model to accomodate states, for example:

  • florida/senate/
  • michigan/senate/
Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Customizable slug. Defaults to Org slug without stopwords.
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • organization (OneToOneField to Organization) – Organization
  • jurisdiction (ForeignKey to Jurisdiction) – Jurisdiction
  • parent (ForeignKey to Body) – Parent

Jurisdiction

class government.models.Jurisdiction(*args, **kwargs)

A Jurisdiction represents a logical unit of governance, comprised of a collection of legislative bodies, administrative offices or public services.

For example: the United States Federal Government, the Government of the District of Columbia, Columbia Missouri City Government, etc.

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • name (CharField) – Name
  • division (ForeignKey to Division) – Division
  • parent (ForeignKey to Jurisdiction) – Parent

Office

class government.models.Office(*args, **kwargs)

An office represents a post, seat or position occuppied by an individual as a result of an election.

For example: Senator, Governor, President, Representative.

In the case of executive positions, like governor or president, the office is tied directlty to a jurisdiction. Otherwise, the office ties to a body tied to a jurisdiction.

Note

Duplicate slugs are allowed on this model to accomodate states, for example:

  • florida/house/seat-2/
  • michigan/house/seat-2/
Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Slug
  • name (CharField) – Name
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • senate_class (CharField) – Senate class
  • division (ForeignKey to Division) – Division
  • jurisdiction (ForeignKey to Jurisdiction) – Jurisdiction
  • body (ForeignKey to Body) – Body

Party

class government.models.Party(*args, **kwargs)

A political party.

Parameters:
  • id (UUIDField) – Id
  • uid (CharField) – Uid
  • slug (SlugField) – Customizable slug. Defaults to slugged Org name.
  • label (CharField) – Label
  • short_label (CharField) – Short label
  • organization (OneToOneField to Organization) – All parties except Independent should attach to an Org.
  • ap_code (CharField) – Ap code
  • aggregate_candidates (BooleanField) – Determines whether to globally aggregate vote totals of this party’s candidates during an election.

vote

Vote models various types of voting that happens in elections.

Delegates

class vote.models.Delegates(*args, **kwargs)

Pledged delegates.

Parameters:
  • id (UUIDField) – Id
  • division (ForeignKey to Division) – Division
  • count (PositiveIntegerField) – Count
  • pct (DecimalField) – Pct
  • total (PositiveIntegerField) – Total
  • candidate_election (ForeignKey to CandidateElection) – Candidate election
  • superdelegates (BooleanField) – Superdelegates

ElectoralVotes

class vote.models.ElectoralVotes(*args, **kwargs)

Electoral votes.

Parameters:
  • id (UUIDField) – Id
  • division (ForeignKey to Division) – Division
  • count (PositiveIntegerField) – Count
  • pct (DecimalField) – Pct
  • total (PositiveIntegerField) – Total
  • candidate_election (ForeignKey to CandidateElection) – Candidate election
  • winning (BooleanField) – Winning

Votes

class vote.models.Votes(*args, **kwargs)

Popular votes.

Parameters:
  • id (UUIDField) – Id
  • division (ForeignKey to Division) – Division
  • count (PositiveIntegerField) – Count
  • pct (DecimalField) – Pct
  • total (PositiveIntegerField) – Total
  • candidate_election (ForeignKey to CandidateElection) – Candidate election
  • ballot_answer (ForeignKey to BallotAnswer) – Ballot answer
  • winning (BooleanField) – Winning
  • runoff (BooleanField) – Runoff

Servers

Civic provides a cli called onespot that handles server management for you.

To get it installed on your path, make sure your virtual environment is activated, and run python setup.py develop.

IMPORTANT: Each onespot command takes a --target=production argument in order to make these commands run on the production server. By default, the commands go to staging.

You will also need to ensure that you have environment files for the servers in your project. These are gitignored because they contain API keys that we cannot leak to the public. In both the terraform/staging and terraform/production folders, you will need both a .env file and a terraform.tfvars file. Talk to Tyler if you don’t have these.

You can always run onespot help for information on the command line.

Provisioning

Run these commands when you need to create new servers or push new code to the servers.

Destroy server

onespot server destroy

This command will completely remove the server and its corresponding security groups from AWS.

Provision new server

onespot server launch

This command will create a new EC2 instance according to the size defined in terraform.tfvars, and associate it with the elastic IP defined in terraform.tfvars.

Setup new server

onespot server setup

This command will install an SSL certificate, setup logging, and install your nginx and uwsgi configuration files to an existing server. Run this after you have launched a new server.

Updating existing server

onespot server update

This command will grab the latest from the master branch of this repo on Github and put it on the server. Then, it will reinstall requirements, migrate the database if necessary, and collect static files.

Indices and tables