OpenBlock¶

Don't forget to stop it!¶

Remember that AWS bills by the hour. Especially if you're not on the free plan, be sure to stop or terminate any instances you're not using when done with them.

Change Settings¶

The OpenBlock config file will be at /home/openblock/openblock/src/myblock/myblock/settings.py. Edit that file as per Configuring OpenBlock.

(Text editors nano and vim are installed; you can of course install emacs or whatever else you like.)

Security warning: it is especially important that you change the PASSWORD_CREATE_SALT and PASSWORD_RESET_SALT settings.

Note that anytime you change settings, or updater your openblock code, you'll want to run this command before you can see your changes take effect on your site:

$  touch /home/openblock/openblock/wsgi/myblock.wsgi

Make an Admin User¶

Your instance does not come with an administrative django user, because of course we don't want other people who clone the AMI to know your password. You can create one with this command:

$ django-admin.py createsuperuser

Now you can log in at http://<your public DNS>/admin.

What's Next¶

You'll want to start Loading Geographic Data.

Then you'll want to add some custom content types to your site, and write some scraper scripts to populate them.

Don't forget ldconfig!¶

Typically after installing libraries, you will need to run this command:

$ sudo ldconfig

... in order for new libraries to be found while building software.

PostGIS template setup¶

Regardless of whether you run postgresql locally or on another host, you'll want a PostGIS template database. Some platforms install this automatically for you, some don't.

You (or your database admin) should follow the instructions for Creating a Spatial Database Template for PostGIS in the GeoDjango documentation and be sure to heed the Note about varying names and locations of the relevant files.

Database Access Settings¶

The following instructions (and the default settings) assume that there is an openblock database user which can create a database for use with openblock. You can create an openblock user by running:

$ sudo -u postgres createuser --createrole --createdb openblock

Depending on your database security setup, you may need to adjust the instructions, settings of postgres and/or settings of openblock.

Postgres administration is beyond the scope of these instructions, but as a quickstart, you can disable postgres security for local users by changing the pg_hba.conf file under etc (the precise location varies, but for postgresql 8.4 on Ubuntu it's /etc/postgresql/8.4/main/pg_hba.conf), comment out any line that starts with local all, and add a line like this:

local   all    all  trust

Then restart postgresql. This is not suitable for production.

See Postgres pg_hba.conf documentation or the postgres wiki for more information.

$ createdb -U openblock test_ob_access
$ dropdb -U openblock test_ob_access

Installing Stable Packages¶

The latest stable releases of ebpub, ebdata, and obadmin can be found on the Python Package Index. To install from these packages, we will publish a consolidated pip requirements file that will install all the necessary python packages. These requirements files will be listed for each release at http://openplans.github.com/openblock/ .

For example, the 1.1 release is at: http://openplans.github.com/openblock/requirements/openblock-requirements-1.1.0.txt and can be installed with this command:

$ $VIRTUAL_ENV/bin/pip install -r http://openplans.github.com/openblock/requirements/openblock-requirements-1.1.0.txt

If you encounter errors during package installation, please see Common Installation Problems.

Installing Development Code¶

Download the openblock software:

$ cd $VIRTUAL_ENV
$ mkdir -p src/
$ git clone git://github.com/openplans/openblock.git src/openblock

It takes a few more Pip commands to install for development, like so commands:

$ cd $VIRTUAL_ENV/src/openblock
$ pip install -r ebpub/requirements.txt
$ pip install -e ebpub
$ pip install -r ebdata/requirements.txt
$ pip install -e ebdata
$ pip install -r obadmin/requirements.txt
$ pip install -e obadmin

If you encounter errors during package installation, please see Common Installation Problems.

Installing the easy way¶

It's easiest to install your platform's package for lxml globally, if it has one. For example, on ubuntu:

$ sudo apt-get install python-lxml

(Note that if you want to take this approach, you must not run virtualenv with the --no-site-packages option, as that will prevent your virtualenv from being able to use this package.)

The slightly harder way¶

If your platform doesn't have a ready-made lxml package, or if you prefer to build your own, you'll need the libxml2 and libxslt development libraries, and then install lxml yourself. For example, on ubuntu you can do:

$ sudo apt-get install libxml2 libxml2-dev libxslt libxslt-dev

And once you have those, on any platform you can do:

$ sudo ldconfig
$ sudo pip install lxml

Installing the easy way¶

GDAL installation isn't covered in detail by the GeoDjango install docs.

The easiest thing to do is check if your operating system already provides a ready-made python GDAL package. For example, on Ubuntu, this will work:

$ sudo apt-get install python-gdal

(Note that if you want to take this approach, you must not run virtualenv with the --no-site-packages option, as that will prevent your virtualenv from being able to use this package.)

GDAL the hard way¶

TODO: see if we can contribute this upstream?

Installing GDAL by hand can be a little tricky, because you have to be careful about which version you install, and in some cases it may not install properly without a few extra arguments.

First, get the GDAL development library. On Ubuntu, this can be installed like:

$ sudo apt-get install libgdal libdal1-dev
$ sudo ldconfig

Next, make sure you are in your openblock environment and it is activated:

$ cd <path_to_openblock>
$ source bin/activate

Next, determine which version of the Python GDAL package you need. Try this command:

$ gdal-config --version

The output will be a version number like "1.6.3". Your Python GDAL package version number needs to match the first two digits. So if gdal-config --version tells you "1.6.3", then you would need a version of Python GDAL that's at least 1.6.0, but less than 1.7. Or if gdal-config tells you that you have 1.7.0, then you would need version 1.7.something of the Python GDAL package. You get the idea. You can use pip to find an appropriate version, like this:

$ pip install --no-install "GDAL>=1.6,<1.7a"  # adjust version as needed

Next, remove the bogus setup.cfg file, if any:

$ rm -f $VIRTUAL_ENV/build/GDAL/setup.cfg

Build the python package with some extra options, determined as described below:

$ cd $VIRTUAL_ENV/build/GDAL
$ python setup.py build_ext --gdal-config=gdal-config \
    --library-dirs=/usr/lib \
    --libraries=gdal1.6.0 \
    --include-dirs=/usr/include/gdal \
  install

The correct value for --library-dirs can be determined by running gdal-config --libs and looking for any output starting with -L. The correct value for --libraries can be determined with the same command but looking for output beginning with -l. The correct value for --include-dirs can be determined by running gdal-config --cflags and looking for output beginning with -I.

Still no luck?¶

If you get an error like /usr/include/gdal/ogr_p.h:94: fatal error: swq.h: No such file or directory, that's because of a bug in GDAL. (See http://trac.osgeo.org/gdal/ticket/3468 .)

The workaround is to manually install swq.h in the same directory that contains ogr_p.h, typically somewhere like /usr/include/gdal. You can get swq.h for GDAL 1.7 here: http://svn.osgeo.org/gdal/branches/1.7/gdal/ogr/swq.h

Then try the preceding setup.py build_ext command again.

Basic Setup¶

First, follow all the instructions in the Preparing Your System document and Installing the Openblock Software

If you followed the Installing the Openblock Software instructions properly, you've already got a virtualenv ready. Go into it and activate it, if you haven't yet:

$ cd path/to/your/virtualenv
$ source bin/activate

Installing obdemo - stable release¶

obdemo is included as part of Installing Stable Packages.

You'll then want to make a copy of the skeleton settings file for editing, which lives at $VIRTUAL_ENV/lib/python2.*/site-packages/obdemo/settings.py.in.

Installing obdemo for development¶

You can work on the latest development code of obdemo and its dependencies like this, assuming you've already taken care of Installing Development Code:

$ cd $VIRTUAL_ENV/src/openblock
$ pip install -r obdemo/requirements.txt
$ pip install -e obdemo

Editing Settings¶

You'll want to edit the demo's django settings at this point, or at least look at it to get an idea of what can be configured. There is also some configuration documentation you should look at.

obdemo doesn't come with a settings.py; it comes with a settings.py.in template that you can copy and edit.

(If you've installed OpenBlock sources from git, this file will be at $VIRTUAL_ENV/src/openblock/obdemo/obdemo/settings.py.in. If you've installed a stable release from pypi, it will be at $VIRTUAL_ENV/lib/python2.*/site-packages/obdemo/settings.py.in.)

$ cd path/to/obdemo
$ cp settings.py.in settings.py
$ favorite_editor settings.py

At minimum, you should change the values of:

• PASSWORD_CREATE_SALT - this is used when users create a new account.
• PASSWORD_RESET_SALT - this is used when users reset their passwords.
• STAFF_COOKIE_VALUE - this is used for allowing staff members to see some parts of the site that other users cannot, such as types of news items that you're still working on.

You'll also want to think about Choosing Your Map Base Layer.

Database Initialization¶

You should already have taken care of Database Setup. Create the (empty) database with this command:

$ sudo -u postgres createdb -U openblock --template template_postgis openblock

Now initialize your database tables:

$ export DJANGO_SETTINGS_MODULE=obdemo.settings
$ django-admin.py syncdb --migrate

(The --migrate option is important; it loads some initial data that openblock depends on including stored procedures, and some default Schemas that you can try out, modify, and delete as needed.)

This will also prompt you to create an administrative user, which is a good idea.

Starting the Test Server¶

Run these commands to start the test server:

$ export DJANGO_SETTINGS_MODULE=obdemo.settings
$ django-admin.py runserver
  ...
  Development server is running at http://127.0.0.1:8000/

You can now visit http://127.0.0.1:8000/ in your Web browser to see the site in action (with no data). You can log in to view the administrative site at http://127.0.0.1:8000/admin/ .

Loading Demo Data¶

OpenBlock is pretty boring without data! You'll want to load some geographic data and some local news. We've included some example data for Boston, MA, and scraper scripts you can use to start with if you don't have all of your local data on hand yet.

Set your DJANGO_SETTINGS_MODULE environment variable before you begin:

$ export DJANGO_SETTINGS_MODULE=obdemo.settings

First you'll want to load Boston geographies. This will take several minutes:

$ django-admin.py import_boston_zips
$ django-admin.py import_boston_hoods
$ django-admin.py import_boston_blocks

Then fetch some news from the web, this will take several minutes:

$ django-admin.py import_boston_news

For testing with random data you might also want to get the misc directory from the OpenBlock source code, and try the random-news script like so:

$ src/openblock/misc/bin/random_news.py 10 local-news

... where 10 is the number of random articles to generate, and 'local-news' is a Schema slug. You must first have some blocks in the database; it will assign randomly generated local news articles to randomly chosen blocks.

Basic Setup¶

First, follow all the instructions in the Preparing Your System document and Installing the Openblock Software

If you followed the Installing the Openblock Software instructions properly, you've already got a virtualenv ready. Go into it and activate it, if you haven't yet:

$ cd path/to/your/virtualenv
$ source bin/activate

Create Custom App Package¶

Now do the following to create a new OpenBlock project. Note: Your project name should be suitable for use as a python module name; i.e. no spaces etc. Here we assume the project name is myblock:

$ cd $VIRTUAL_ENV/src
$ paster create -t openblock myblock

After answering a few questions, this will create a bare-bones Django project in the folder you specified. Next, install the project into your environment:

$ cd myblock
$ python setup.py develop
...

What You Get¶

The created package is a minimal Django project that includes:

• settings.py you can edit.
• urls.py that wraps ebpub's URLs; you can override individual views here, add custom views and other Django apps, etc.
• manage.sh, a tiny wrapper around manage.py / django-admin.py that saves you the trouble of activating your virtualenv or exporting DJANGO_SETTINGS_MODULE.
• templates/homepage.html, an example of overriding one of ebpub's templates, edit as you like.
• wsgi/<projectname>.wsgi, suitable for deploying your project under Apache and mod_wsgi. It takes care of finding the containing virtualenv and the DJANGO_SETTINGS_MODULE automatically.

Adjust Django Settings¶

Your django settings are located in settings.py within your project. You should review these and make adjustments based on your setup:

$ favorite_editor myblock/settings.py
...

Read more about important settings you can/should customize.

If you plan to use a remote database or have other changes to database connection information, make sure you change them in your settings.py. See Database Access Settings and be sure everything works before you proceed.

Create and Initialize the Database¶

Now, as usual with Django projects, you'll need to create and initialize your database. If you haven't changed the default database settings, and if you've followed the PostGIS template setup instructions, then the database creation command would simply be:

$ sudo -u postgres createdb -U openblock --template template_postgis openblock_myblock

If you have a different postgresql setup, for example you're using a different user than openblock, just change the -U option accordingly.

Now initialize your database tables:

$ export DJANGO_SETTINGS_MODULE=myblock.settings
$ django-admin.py syncdb --migrate

(The --migrate option is important; it loads some initial data that openblock depends on including stored procedures, and some default Schemas that you can try out, modify, and delete as needed.)

This will also prompt you to create an administrative user, which is a good idea.

Starting the Test Server¶

Run django's test server using your project's settings and visit http://127.0.0.1:8000/ in your Web browser to see the site in action (with no data):

$ export DJANGO_SETTINGS_MODULE=myblock.settings
$ django-admin.py runserver
...
Development server is running at http://127.0.0.1:8000/

You can now visit http://127.0.0.1:8000/ in your Web browser to see the site in action (with no data). You can log in to view the administrative site at http://127.0.0.1:8000/admin/ .

Default: OpenStreetMap tiles hosted by OpenGeo¶

This is the default setting in ebpub/ebpub/settings_default.py. It is a fairly clean design inspired by everyblock.com, and was derived from OpenStreetMap data. It is free for use for any purpose, but note that there have been some reliability issues with this server in the past.

Other Publicly Available Layers¶

It's easy to use any base layer supported by olwidget. More specifically:

Custom or Other Base Layers¶

Do you have your own tile server running, or have a URL to something else not in the above list? Great! You can use that with a few extra settings. This option takes a little more work; you will have to know which OpenLayers layer subclass is appropriate, and what parameters to pass to it.

In fact, this is how our default OpenGeo / OpenStreetMap layer is configured, so let's use that as an example:

MAP_BASELAYER_TYPE = 'custom.opengeo_osm'

MAP_CUSTOM_BASE_LAYERS = {
   'opengeo_osm':  # to use this, set MAP_BASELAYER_TYPE='custom.opengeo_osm'
       {"class": "WMS",  # The OpenLayers.Layer subclass to use.
        "args": [  # These are passed as arguments to the constructor.
           "OpenStreetMap (OpenGeo)",
           "http://maps.opengeo.org/geowebcache/service/wms",
           {"layers": "openstreetmap",
            "format": "image/png",
            "bgcolor": "#A1BDC4",
            },
           {"wrapDateLine": True
            },
           ],
        }
}

short_name¶

This is how OpenBlock knows which dictionary in METRO_LIST to use. It must exactly match the value of settings.SHORT_NAME.

extent¶

This is a list of (leftmost longitude, lower latitude, rightmost longitude, upper latitude).

One way to find these coordinates would be to use Google Maps to zoom to your region, then point at the lower left corner of your area, right-click, and select "Drop LatLng Marker". You will see a marker that displays the latitude,longitude of that point on the map. Then do the same in the upper right corner.

This defines a bounding box - the range of latitudes and longitudes that are relevant to your area. It is used in many views as the default bounding box when searching for relevant NewsItems. It is also used by some data-loading scripts to filter out data that's not relevant to your area.

multiple_cities¶

Set multiple_cities to True if you want one OpenBlock site to serve multiple cities or towns in the same region.

For example, you might be setting it up for a county. In this example you could use the county name for city_name and metro_name. Or you might be somewhere like the San Francisco Bay Area and wanting to include San Francisco, Oakland, Berkeley, and so on. So city_name might be 'San Francisco' and metro_name might be something like 'Bay Area'.

If multiple_cities is True, you must also set city_location_type, see below.

This option affects numerous URLs on the site; users will be able to browse first by city, then by street, then by block, and so on. If it's False, the city browsing page will be left out of the site structure.

city_location_type¶

You only need this if multiple_cities is True. In that case you will need to create a LocationType for cities, and city_location_type should be set to that LocationType's slug.

You will then want to create a Location for each city in your region. See Loading Location Data for more.

When would you put more than one dictionary in METRO_LIST?¶

The only dictionary in METRO_LIST that has any effect is the one whose short_name matches settings.SHORT_NAME.

The purpose of having more than one metro dictionary in METRO_LIST would be to run multiple OpenBlock sites for multiple metro areas with some shared configuration. You are probably not doing this.

The idea is that you could have one settings file containing the master METRO_LIST, and then for each site you'd have its own settings file that imports METRO_LIST (and any other shared stuff you like) from the master settings file. Each site-specific settings file would also set settings.SHORT_NAME to match the 'short_name' key of one of the dictionaries.

Most people will probably not be doing that. This feature serves the needs of everyblock.com, which runs separate sites for many cities across the USA.

Don't have an SMTP Server?¶

You may be able to use an appropriate account on Gmail or another public mail service. See for example this blog post).

Background Tasks¶

Several of the things you can do in the admin UI can potentially take a long time, so they are launched as background tasks. If you want to use these admin UI features, be sure to read this section.

You'll need to do this once at server start time:

$ export DJANGO_SETTINGS_MODULE=myblock.settings  # change as needed
$ django-admin.py process_tasks

Like the runserver command, this won't immediately exit. It will sit quietly until there are background jobs to process for installing geographic data.

NB. If you are installed on EC2, then django-admin.py process_tasks is already run via a cron job.

Load ZIP Codes via Admin UI¶

If you have a list of the ZIP codes you'd like to install, just be sure the background task daemon is running. Then you can surf to http://<your domain>/admin/db/location/ and click the link "Import ZIP Shapefiles". You can pick your state, paste your list of ZIPs, and wait for the import to finish.

You can do this several times if your area crosses state lines.

When this is done, skip down to Verifying ZIP Codes below.

(TODO: screen shot?)

Load ZIP Codes via Command Line¶

Alternately, you can use command-line scripts to install ZIP codes.

The US Census Bureau has shapefiles for all USA zip codes. Go to http://www2.census.gov/cgi-bin/shapefiles2009/national-files, select your state from the drop-down, and submit. Toward the bottom of the file list, you should see one labeled "5-Digit ZIP Code Tabulation Area (2002)".

Download that file. It should have a name that looks like tl_2009_36_zcta5.zip where 36 is a state ID (in this case, 36 is for New York).

Unzip the file. It should contain a number of files like this:

$ unzip tl_2009_36_zcta5.zip
inflating: tl_2009_36_zcta5.dbf
inflating: tl_2009_36_zcta5.prj
inflating: tl_2009_36_zcta5.shp
inflating: tl_2009_36_zcta5.shp.xml
inflating: tl_2009_36_zcta5.shx

The ZIP code file you downloaded is for an entire state. You're probably not setting up OpenBlock for an entire state, so you'll need to filter out those ZIP codes that are irrelevant to your area of interest. The zip import script allows you to do that, if you have configured your metro extent.

$ import_zips_tiger -v -b /path/to/where/you/unzipped/the/files/

The -b option tells it to filter out zip codes outside your metro extent, and -v tells it to give verbose output.

It will tell you which ZIP codes were skipped, and at the end, print a count of how many were created.

Verifying ZIP Codes¶

To verify that your ZIP codes loaded, point your browser at the home page. There should be a link to view "61 ZIP codes", or however many you loaded. Follow that to see a list of them all, and click on one to see a page about that ZIP code.

If you want to have a look "under the hood", you can use the django admin UI to do so. Browse to http://localhost:8000/admin , and enter your admin username / password when prompted.

Navigate to "Db" -> "Location Types". You should see that there is a Location Type called "ZIP Code" in the system now.

Navigate back to "Db", then go to "Db" -> "Locations". You should see a number of ZIP codes in the list. If you click on one, you should see an edit form that contains a map, showing you the borders of this ZIP code.

(TODO: screen shot?)

Finding Blocks Data¶

In the US, the Census Bureau's TIGER data website is a good source of data. From http://www2.census.gov/cgi-bin/shapefiles2009/national-files, you will need several files. First select the State you're interested in. Download the file labeled "Place (Current)".

Next, select the County you're interested in. From the county's page, download the files labeled "All Lines", "Topological Faces (Polygons With All Geocodes)", and "Feature Names Relationship File".

Loading Blocks using the Admin UI¶

It's easy to use the admin UI to load US Census shapefiles. First, be sure the background task daemon is running.

Then you can surf to http://<your domain>/admin/streets/blocks/ and click the link "Import Block Shapefiles". Type in the city name that these blocks are in, upload the four zip files you downloaded above, click "Import" and wait for it to finish.

(It is likely to take several minutes - more or less, depending on your hardware; this is the most computationally intensive thing that OpenBlock ever does.)

Streets, Intersections, and BlockIntersections will be done automatically.

You can repeat this process if your area spans multiple shapefiles. (It tends to get slower as the number of intersections grows.)

When done, skip down to Verifying Blocks.

(TODO: screen shot?)

Loading Blocks using the Command Line¶

You don't have to use the admin UI if you're happy at the command line. It takes several steps.

$ import_blocks_tiger --city=BOSTON --filter-bounds \
  tl_2009_25025_edges.shp tl_2009_25025_featnames.dbf \
  tl_2009_25025_faces.dbf tl_2009_25_place.shp

$ import_blocks_tiger --filter-location="cities:cambridge" \
  --filter-location="cities:newton" ...

$ import_blocks_tiger  --help
Usage: import_blocks_tiger edges.shp featnames.dbf faces.dbf place.shp

Options:
 -h, --help            show this help message and exit
 -v, --verbose
 -c CITY, --city=CITY  A city name to filter against
 -f, --fix-cities      Whether to override "city" attribute of blocks and
                       streets by finding an intersecting Location of a city-
                       ish type. Only makes sense if you have configured
                       multiple_cities=True in the METRO_LIST of your
                       settings.py, and after you have created some
                       appropriate Locations.
 -b, --filter-bounds   Whether to skip blocks outside the metro extent from
                       your settings.py. Default True.
 -e ENCODING, --encoding=ENCODING
                       Encoding to use when reading the shapefile

$ populate_streets -v -v -v -v streets
$ populate_streets -v -v -v -v block_intersections
$ populate_streets -v -v -v -v intersections

Verifying Blocks¶

Try starting up django and browsing or searching some blocks:

$ django-admin.py runserver

Now browse http://localhost:8000/streets/ and have a look around. You should see a comprehensive list of streets on that page, and each should link to a list of blocks. On the list of blocks, each block should link to a detail page that includes a map of a several-block radius.

You should also be able to search. In the search bar at top right, type in some addresses or intersections that you know should exist, and verify that they're found.

What kinds of locations?¶

Aside from ZIP codes, what kinds of geographic regions are you interested in?

OpenBlock can handle any number of types of locations. You can use the admin UI to create as many LocationTypes as you want, by visiting http://localhost:8000/admin/db/locationtype/ and click "Add". Fill out the fields as desired. You'll want to enable both 'is_browsable' and 'is_significant'.

(Note also that the shapefile import scripts described below can create LocationTypes for you automatically, so you may not need to do anything in the admin UI.)

You're limited only by the data you have available. Some suggestions: try looking for neighborhoods/districts/wards, police precincts, school districts, political districts...

Drawing Locations by Hand¶

If you don't have shapefiles available, it's always possible to hand-draw locations in the admin UI. This is a great option for relatively simple shapes where you don't need to be very precise with the edges. This might also be appropriate for areas whose boundaries are informal. For example, often local residents will have a general sense of where neighborhoods begin and end, but there may not be "official" boundaries published anywhere.

Just browse to /admin/db/locations, click "Add location", drag and zoom the map as desired, select a location type, and start clicking away on the map.

When happy with your polygon, double-click on the last point to stop drawing.

To modify it, click the "Modify features" icon in the map toolbar and then you can click and drag individual points, or click a point and hit the Delete key to remove a point.

There are Undo and Redo buttons, although the history will be forgotten once you click the Save button on the form.

(TODO: screen shots?)

For precise complex shapes, it's just not practical to draw a 500-point polygon in our admin UI.

Finding Location Data¶

The trouble with loading local place data is that, at least in the USA, there is no central agency responsible for all of it, and no standards for how local governments should publish their geospatial data. This means it's scattered all over the web, and we can't just tell you where to find it.

Try googling for the name of your area plus "shapefiles".

Loading Location Data¶

Once you have one or more Location Types defined, you can start populating them, either via the command line or the admin UI.

$ import_locations  --help

Usage: import_locations [options] type_slug /path/to/shapefile

Options:
 -h, --help            show this help message and exit
 -n NAME_FIELD, --name-field=NAME_FIELD
                       field that contains location's name
 -i LAYER_ID, --layer-index=LAYER_ID
                       index of layer in shapefile
 -s SOURCE, --source=SOURCE
                       source metadata of the shapefile
 -v, --verbose         be verbose
 -b, --filter-bounds   exclude locations not within the lon/lat bounds of
                       your metro's extent (from your settings.py) (default
                       false)
 --type-name=TYPE_NAME
                       specifies the location type name
 --type-name-plural=TYPE_NAME_PLURAL
                       specifies the location type plural name

$ import_neighborhoods  --help
Usage: import_neighborhoods [options] /path/to/shapefile

Options:
 -h, --help            show this help message and exit
 -n NAME_FIELD, --name-field=NAME_FIELD
                       field that contains location's name
 -i LAYER_ID, --layer-index=LAYER_ID
                       index of layer in shapefile
 -s SOURCE, --source=SOURCE
                       source metadata of the shapefile
 -v, --verbose         be verbose
 -b, --filter-bounds   exclude locations not within the lon/lat bounds of
                       your metro's extent (from your settings.py) (default
                       false)

Can I load KML, GeoJSON, OpenStreetMap XML, or other kinds of files?¶

No, at this time the only files we can directly import are shapefiles. Try using tools like ogr2ogr to convert your data into shapefiles.

Creating the Schema¶

The first step is to create an ebpub.db.models.Schema to represent the Crime Report type:

>>> from ebpub.db.models import Schema
>>> crime_report = Schema()

This object will contain metadata about all Crime Reports, like what its title is and how to pluralize it:

>>> crime_report.indefinite_article = 'a'
>>> crime_report.name = "Crime Report"
>>> crime_report.plural_name = "Crime Reports"

The slug is the unique identifier for this Schema that will be used in URLs on the site. It should be brief and contain URL safe characters:

>>> crime_report.slug = 'crimereport'

The min_date field can be used to limit how far back the user can navigate when viewing crime reports. For now, we'll just assume that everything is in the future:

>>> from datetime import datetime
>>> crime_report.min_date = datetime.utcnow()

The last_updated field tracks when we last checked for new crime reports. We'll also just stub this out to the current time for now:

>>> crime_report.last_updated = datetime.utcnow()

The has_newsitem_detail field controls whether this item has a page hosted on this site, or whether it has its own external url. We'll host these ourselves:

>>> crime_report.has_newsitem_detail = True

The is_public field controls whether or not NewsItems of this type are visible to anybody other than administrators on the site. Normally you should wait until the type is set up and loaded with news before "turning it on". We'll just make it available immediately:

>>> crime_report.is_public = True

The is_event field controls whether or not NewsItems of this type are announcements of future events, rather than news that happened in the past. For more details on how to do this, see Event-like News Types This doesn't apply to crime reports, so we'll leave it set to False:

>>> crime_report.is_event = False

There are a few additional fields you can explore (see the code in ebpub.db.models.Schema), but this will be good enough to start with. So let's save it and move on:

>>> crime_report.save()

At this point you will be able to see the type listed on your site's front page, and reach an (empty) listing using your slug by visiting http://localhost:8000/crimereport assuming you are running the web server.

Making Maps Prettier¶

If you want your different NewsItem types to stand out from each other on maps, you have two options.

You can set your schema's map_color to a hex color code (eg. #FF0000), and markers for that news type will be filled with that color.

Or, you can set your schema's map_icon_url to the URL of an image to use for its markers. Should be roughly 35x35 pixels. (If you are hosting your own map icons, it's fine to use a relative URL here.)

OpenBlock does not currently ship with any map icons. One source of good free (Creative Commons 3.0 BY-SA) map icons is http://mapicons.nicolasmollet.com/ .

Adding Custom Fields¶

As mentioned earlier, we will add the following custom fields:

• Responding officer's name
• Type of crime (in english)
• Police code for crime

We will create an ebpub.db.models.SchemaField to describe each custom field. Let's start with the reporting officer:

>>> from ebpub.db.models import SchemaField
>>> officer = SchemaField()
>>> officer.schema = crime_report
>>> officer.pretty_name = "Reporting Officer's Name"
>>> officer.pretty_name_plural = "Reporting Officer's Names"

The values of all the custom fields for a particular NewsItem will be stored in a single ebpub.db.models.Attribute object. The Attribute object has a fixed set of fields which can be used for custom attributes. The fields are named according to their type, and numbered:

Each SchemaField will map onto one of the fields of the Attribute class. We'll map the reporting officer onto the first varchar field varchar01 by setting the real_name attribute:

>>> officer.real_name = 'varchar01'

When working with a crime report NewsItem, we'll want to have an alias for this attribute in the code, so we don't always have to remember what 'varchar01' means for crime reports. This is set using the name field of the SchemaField. We'll call it officer, and move on:

>>> officer.name = 'officer'

That's the important stuff. There are a bunch of mandatory display-related fields; we'll just gloss over these for now:

>>> officer.display = True
>>> officer.display_order = 10
>>> officer.is_searchable = True
>>> officer.is_lookup = False
>>> officer.is_filter = False
>>> officer.is_charted = False

Now we can save this SchemaField:

>>> officer.save()

The name of the crime works the same way, but we'll need to store it in a different field. We'll use the second varchar field, varchar02:

>>> crime_name = SchemaField()
>>> crime_name.schema = crime_report
>>> crime_name.real_name = "varchar02"
>>> crime_name.pretty_name = "Crime Type"
>>> crime_name.pretty_plural_name = "Crime Types"
>>> crime_name.name = "crime_type"
>>> crime_name.display = True
>>> crime_name.display_order = 10
>>> crime_name.is_searchable = True
>>> crime_name.is_lookup = False
>>> crime_name.is_filter = False
>>> crime_name.is_charted = False
>>> crime_name.save()

For the crime code, we'll use an integer field:

>>> crime_code = SchemaField()
>>> crime_code.schema = crime_report
>>> crime_code.real_name = "int01"
>>> crime_code.pretty_name = "Crime Code"
>>> crime_code.pretty_plural_name = "Crime Codes"
>>> crime_code.name = "crime_code"
>>> crime_code.display = True
>>> crime_code.display_order = 10
>>> crime_code.is_searchable = True
>>> crime_code.is_lookup = False
>>> crime_code.is_filter = False
>>> crime_code.is_charted = False
>>> crime_code.save()

Phew, okay we just designed a NewsItem type!

Creating a NewsItem with the Schema¶

Now we can finally start churning out amazing crime reports. We start by making a basic news item with our schema and filling out the basic fields:

>>> from ebpub.db.models import NewsItem
>>> report = NewsItem()
>>> report.schema = crime_report
>>> report.title = "Hooligans causing disturbance downtown"
>>> report.location_name = "123 Fakey St."
>>> report.item_date = datetime.utcnow()
>>> report.pub_date = datetime.utcnow()
>>> report.description = "Blah Blah Blah"
>>> report.save()

Great, now (any only now) we can set the extra fields, which are weirdly immediately set when accessing the special attributes dictionary on the NewsItem. (There is some python magic going on, see the code in ebpub.db.models.) We use the names that we assigned when we were designing the schema:

>>> report.attributes['officer'] = "John Smith"
>>> report.attributes['crime_type'] = "Disturbing The Peace"
>>> report.attributes['crime_code'] = 187

If you visit the crime reports page at http://localhost:8000/crimereport it should list your new item. You can click its link to view the custom details you added.

Hooray!

Testing the daemon¶

Give it a try:

$ python ebdata/ebdata/retrieval/updaterdaemon/runner.py --config=/path/to/config.py  start

If it works, nothing obvious should happen :) It's running in the background. You shouldn't expect anything to happen until the next multiple of 10 minutes. When it's time, check the log file to see if anything's happening:

$ tail -f /tmp/updaterdaemon.log

(Hit Ctrl-C to get out of that.)

If there's nothing in the main log, check the error log:

$ less /tmp/updaterdaemon.err

To stop the daemon, do this:

$ python ebdata/ebdata/retrieval/updaterdaemon/runner.py stop

Installing the init script¶

UpdaterDaemon also comes with a script suitable for putting in /etc/init.d, so it will be restarted whenever the system is rebooted. To install this script, copy it from ebdata/retrieval/updaterdaemon/initscript into something like /etc/init.d/openblock-updaterdaemon. It is known to work on Ubuntu; let us know if you have trouble with it on other linux systems.

After copying, edit the script, setting a few crucial environment variables:

HERE should point to the virtualenv where you installed OpenBlock.

CONFIG should point to a config file as described in the previous sections.

SU_USER should be the name of the user account to use for running the daemon.

You might also want to set LOGFILE and ERRLOGFILE to control where the logs go.

Now try running the script as root:

$ sudo /etc/init.d/openblock-updaterdaemon start

Check the log files to make sure it's working.

Purpose¶

Support simple widgets, mashups, frontends based on OpenBlock content and filtering capability. We would appreciate any feedback you have on how to improve the usefulness and usability of this API.

Caveat¶

This is a preliminary work-in-progress API and may be changed substantially in future versions.

See Also¶

The OpenBlock API uses several standards for formats and protocols. Please see the (externally maintained) documentation focused on the particular formats for more details. These include GeoJSON, Atom, and JSONP. Some helpful links:

Examples / Quickstart¶

Grab some 'articles' about Roxbury

curl "http://bos.openblock.org/api/dev1/items.json?type=articles&locationid=neighborhoods/roxbury&limit=3" > items.json

URL prefix¶

All calls to the OpenBlock API referenced in this document are prefixed by:

/api/dev1/

Authentication and API Keys¶

Authentication for those methods which require it may currently be accomplished in two ways:

• HTTP Basic Auth headers (any normal user account registered with OpenBlock will work)
• sending your API key in the X-Openblock-Key HTTP header

An API key may be obtained by logging in and visiting your preferences page, if the OpenBlock site makes that available to you; or the site administrators can create keys via the admin UI at admin/key/apikey/.

Support for Cross Domain Access¶

To enable widgets and mashups in the browser from domains other than the host OpenBlock instance, the API supports the JSONP convention.

Unless otherwise noted, all portions of the API using the http GET method support JSONP by providing the "jsonp" query parameter. The "jsonp" parameter may only contain letters, numbers, and underscores; other characters will be removed.

GET methods supporting Atom output may also provide the "jsonp" parameter. In this case the output is JSONP-X.

Rate Limits, AKA Throttling¶

The API may be configured to limit the number of API requests a user can perform during a certain time period. This limit applies across all resources provided by the API.

By default, the limit is 150 requests per hour.

If the user is not authenticated and provides no API key, the user's IP address will be used.

If this limit is reached, the server will return a 503 SERVICE UNAVAILABLE response, with a Retry-After header indicating the number of seconds after which the user can try again.

The site administrator controls throttling by changing several values in settings.py:

API_THROTTLE_AT=150  # max requests per timeframe.
API_THROTTLE_TIMEFRAME = 60 * 60  # Default 1 hour.
# How long to retain the times the user has accessed the API. Default 1 week.
API_THROTTLE_EXPIRATION = 60 * 60 * 24 * 7

# NOTE in order to enable throttling, you MUST also configure
# CACHES['default'] to something other than a DummyCache. Example:
CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.locmem.LocMemCache',
    }
}

GET [api prefix]¶

GET items.json¶

{"type": "FeatureCollection",
 "features": [
    {"type": "Feature",
     "properties": {
        "title": "An Article About Roxbury",
        "url": "...",
        "type": "articles",
        "description": "Test Roxbury",
        ...
     },
     "geometry": {
       "type": "Point",
       "coordinates": [-71.086787000000001, 42.314782999999998]
     }
    },
 ...
]}

GET items.atom¶

FIXME example

GET items/<id>.json¶

GET geocode¶

"type": "FeatureCollection",
"features": [
 {
  "geometry": {
   "type": "Point",
   "coordinates": [
    -71.086787000000001,
    42.314782999999998
   ]
  },
  "type": "Feature",
  "properties": {
   "city": "BOSTON",
   "type": "neighborhoods",
   "name": "Roxbury",
   "query": "Roxbury"
  }
 }]}

GET items/types.json¶

[{'elvis-sightings': {
   'indefinite_article': 'an',
   'name': 'Elvis Sighting',
   'plural_name': 'Elvis Sightings',
   'slug': 'elvis-sightings',
   'last_updated': '2011-02-22',
   'attributes': {
     'verified': {
       'pretty_name': 'Verified Really Elvis',
       'type': 'bool'
    }
  }
}]

GET locations.json¶

[
 {
  "city": "YOUR CITY",
  "description": "",
  "url": "/api/dev1/locations/zipcodes/02108.json",
  "type": "zipcodes",
  "slug": "02108",
  "name": "02108"
 },
 {
  "city": "YOUR CITY",
  "description": "",
  "url": "/api/dev1/locations/neighborhoods/allstonbrighton.json",
  "type": "neighborhoods",
  "slug": "allstonbrighton",
  "name": "Allston/Brighton"
 }
]

GET locations/<locationid>.json¶

{ "type": "Feature",
 "geometry": {
   "type": "Polygon",
   "coordinates": [
     [102.0, 0.0], [103.0, 1.0], [104.0, 0.0], [105.0, 1.0], ...
     ]
   },
 "properties": {
   "type": "zipcode",
   "city": "boston",
   "name": "02115",
   "slug": "02115",
   "description": "lorem ipsum blah blah",
   "centroid": "POINT (101.0 0.5)",
   "area": 3633354.76,
   "source": "http://example.com/zip_codes_or_something",
   "population": null,
   }
 },

GET locations/types.json¶

{
 "towns": {"name": "Town",
           "plural_name": "Towns",
           "scope:" "boston"},
 "zipcodes": { ... }
}

GET places/types.json¶

{
    "poi": {
        "name": "Point of Interest",
        "plural_name": "Points of Interest",
        "geojson_url": "/api/dev1/places/poi.json"
    },
    "police": {
        "name": "Police Station",
        "plural_name": "Police Stations",
        "geojson_url": "/api/dev1/places/police.json"
    }
}

GET places/<placetype>.json¶

{
 "type": "FeatureCollection",
 "features": [
  {
   "geometry": {
    "type": "Point",
    "coordinates": [
     -71.052149999999997,
     42.332369999999997
    ]
   },
   "type": "Feature",
   "properties": {
    "type": "poi",
    "name": "Fake Monument",
    "address": ""
   }
  },
  {
   "geometry": {
    "type": "Point",
    "coordinates": [
     -71.052149999999997,
     42.332369999999997
    ]
   },
   "type": "Feature",
   "properties": {
    "type": "poi",
    "name": "Fake Yards",
    "address": ""
   }
  }
 ]
}

Spatial Filtering¶

Spatial filters allow the selection of items based on geographic areas. At most one spatial filter may be applied per API request.

Other Filters¶

POST items/¶

{
  "errors": {
    "url": [
      "This field is required."
    ],
    "item_date": [
      "Enter a valid date."
    ]
  }
}

NewsItem JSON Format¶

A NewsItem is represented by a GeoJSON Feature containing:

• a "geometry" attribute representing its specific location, generally a Point.
• a "type" attribute, which is always "Feature".
• a "properties" attribute containing details of the news item according to its schema.

See the GeoJSON specification for additional information on GeoJSON: http://geojson.org/geojson-spec.html

Example:

{
  "geometry": {
   "type": "Point",
   "coordinates": [
    -71.055719999999994, 42.359819999999999
   ]
  },
  "type": "Feature",
  "properties": {
    "title": "Looked kind of like Elvis",
    "type": "elvis-sightings",
    "description": "Witnesses reported someone who looked just like Elvis except eight feet tall and with long red hair and green skin.",
    "url": "http://example.com/elvis123",
    "item_date": "2010-12-10",
    "pub_date": "2010-12-10T16:55:01-06:00",
    "verified": false,
    "location_name": "123 Main St, Springfield, MA",
  }
 }

NewsItem Atom Format¶

Generally follows the Atom specification. Location information is specified with GeoRSS-Simple.

Extended schema attributes are specified in the "http://openblock.org/ns/0" namespace.

Example:

<?xml version="1.0" encoding="utf8"?>
<feed xmlns="http://www.w3.org/2005/Atom"
      xmlns:openblock="http://openblock.org/ns/0"
      xmlns:georss="http://www.georss.org/georss">
   <title>openblock news item atom feed</title>
   <link href="/api/dev1/items.json" rel="alternate"></link>
   <link href="/api/dev1/items.atom" rel="self"></link>
   <id>/api/dev1/items.atom</id>
   <updated>2010-12-10T16:55:01-06:00</updated>
   <entry>
      <title>Looked kind of like Elvis</title>
      <link href="http://example.com/elvis123" rel="alternate"></link>
      <updated>2010-12-10T16:55:01-06:00</updated>
      <id>...</id>
      <summary type="html">Witnesses reported someone who looked just
         like Elvis except eight feet tall and with long red hair and green skin.
      </summary>
      <georss:point>42.3598199999999991 -71.0557199999999938</georss:point>
      <georss:featureName>4 S. Market St.</georss:featureName>
      <openblock:type>elvis-sightings</openblock:type>
      <openblock:attributes>
         <openblock:attribute type="bool" name="verified">False</openblock:attribute>
      </openblock:attributes>
   </entry>
</feed>

Adding a new Widget¶

A new widget can be created by visiting the OpenBlock admin site and selecting the Add link next to Widgets in the Widgets section.

Configuring a Widget¶

The configuration of a widget combines criteria for selecting items to display with an output template for controlling the display of those items.

Currently items are always ordered by date.

Embedding Content¶

The administrative page for each widget provides two ways you can embed the widget on an external site.

Creating A Template¶

A new template can be created by visiting the OpenBlock admin site and selecting the Add link next to Templates in the Widgets section.

Item Info¶

The variable items contains a list of items that should be displayed by the widget. This list is generally looped over using the for template tag, eg:

{% for item in items %}
   <!-- output something about the item -->
   {{ item.title }}
{% endfor %}

Each item in the list contains a basic set of fields, and may include several extension fields that are particular to the type (schema) of the item.

{{ item.attributes_by_name.crime_code.value }}

{% for attribute in item.attributes %}
  {{ attribute.value }}
{% endfor %}

Widget Info¶

The context variable widget provides information about the widget. The widget variable has the following fields:

Check the Release Notes¶

Database Migrations¶

When upgrading your copy of the OpenBlock code, there may sometimes be updates to Model code which require corresponding changes to your existing database.

You can do this with one command (it's prudent to make a database backup first):

django-admin.py syncdb --migrate

To see what migrations exist and which ones you've already run, you can do:

django-admin.py migrate --list

Under the hood, this uses South to automate these database migrations. If you really want to see what the migrations do, or need to write your own, you'll want to read the South documentation to understand how they work. Then look in the migrations/ subdirectories located under the various app directories, notably ebpub/ebpub/db/migrations/ to see what the existing migration scripts look like.

Upgrade Notes¶

• Due to an oversight, both obdemo and the generated custom applications prior to this release didn't include the TIME_ZONE setting. Please set settings.TIME_ZONE to an appropriate value.

• As usual, install all dependencies, eg if you are upgrading a source checkout:

  pip install -r ebpub/requirements.txt
  pip install -e ebpub
  pip install -r ebdata/requirements.txt
  pip install -e ebdata
  pip install -r obadmin/requirements.txt
  pip install -e obadmin
  pip install -r obdemo/requirements.txt
  pip install -e obdemo
  

• As usual, sync and migrate the database:

  django-admin.py syncdb
  django-admin.py migrate
  

Backward Incompatibilities¶

• updaterdaemon is now deprecated. It has not been removed, but we no longer recommend or support its use. Use cron or your favorite cron alternative instead for running scrapers regularly. See the "Running Scrapers" docs. This allows us to close a bunch of tickets (#194, #51, #87, #180, #199, #222).
• removed redundant get_metro_bbox function.
• remove unused template tags: SHORT_NAME, STATE_ABBREV, EB_SUBDOMAIN.

New Features in 1.1¶

• Big shareable maps: "Explore these items on a larger map" link on all type-specific news lists. For example, http://demo.openblockproject.org/photos/filter/locations=neighborhoods,financial-district/ links to http://bit.ly/njmZT6 which is shareable via permalink. (There is also undocumented support for embedding these via iframes.)
• Comments on NewsItems. Requires logging in, and the Schema must have allow_comments=True and has_detail=True. Needs docs.
• User-contributed "Neighbor Messages" and "Neighbor Events" news types, in the ebpub.neighbornews package. Needs docs.
• Better support for running in a multi-city area:
  • new get_city_locations() function to get a list of all Locations whose LocationType matches the 'city_location_type' from settings.METRO_LIST.
  • --fix-cities option to block import scripts (and admin UI) that allows fixing imported blocks so block.city matches an existing overlapping city-ish Location.
  • clean out intersections and streets on import, so they're regenerated safely. Optionally skip regeneration.
  • some related URL bugfixes.
• Import Places from a CSV file via the admin UI. Needs docs.
• Date and time picker widgets on forms, where relevant. (#186)
• Block import supports filtering by your default metro extent, not just city name. #160
• Support for future events, not just recent news. Several scrapers support this: the ma/boston/events scraper, and the general/meetups/ scraper, and the neighbornews package. See docs in docs/packages/ebdata.rst. (Ticket #246)
• Added a scraper for Meetup.com, in ebdata/scrapers/general/meetups. It's zero-configuration: it just loops over your zip codes and finds all meetups for those. It's at ebdata/scrapers/general/meetup/meetup_retrieval.py and the associated schema can be loaded like so: django-admin.py loaddata ebdata/scrapers/general/meetup/meetup_schema.json You'll need to set MEETUP_API_KEY in settings.py. (Ticket #208)
• Add a --reset option to update_aggregates script, deletes all aggregates and starts over. (ticket #221)
• Add an ebpub/bin/delete_newsitems.py script, useful during schema development: wipes all newsitems and attributes and lookups of a given schema.
• Also add --quiet, --verbose, --dry-run, and --help command-line options to update_aggregates.
• Email alerts can now be sent via a command-line script. (related to ticket #65). Includes docs for how to set it up with cron.
• Email alert signup can be disabled by removing 'ebpub.alerts' from settings.INSTALLED_APPS. (refs ticket #65).
• obdemo includes flickr and meetup in default news types.
• Flickr scraper (ticket #26). It's at ebdata/scrapers/general/flickr/flickr_retrieval.py and the associated schema can be loaded like so: django-admin.py loaddata ebdata/scrapers/general/flickr/photos_schema.json You'll need to set FLICKR_API_KEY and FLICKR_API_SECRET in settings.py.
• Import locations from shapefiles in the admin UI (ticket #59). With documentation (ticket #234).
• Import blocks from shapefiles in the admin UI. Also populates streets, blockintersections, and intersections. (ticket #215)
• You can now set the default location type via settings.DEFAULT_LOCTYPE_SLUG. (#148)
• Add --verbose and --quiet options to a bunch of command-line scripts and scrapers.
• Don't email scraper errors by default. That's just not nice, and cron already does that.
• All provided scrapers now log to settings.SCRAPER_LOGFILE_NAME.
• Custom apps generated via paster create -t openblock now include a wsgi file for use with mod_wsgi, an alternative settings file for use with django-admin process_tasks, a skeleton cron config, executable manage.sh and manage.py files. Also, manage.sh is now better at automatically finding and activating the virtualenv.
• obdemo also includes an example cron config file, a manage.sh file, and the alt. settings file. And no longer has an example updaterdaemon config.
• Our Amazon EC2 AMI will now use cron rather than updaterdaemon. Lots of other fixes in the EC2 scripts too.

Bugs fixed¶

• Fixed broken map on feeds page, ticket #237.
• Added missing links to the password change form.
• CSRF protection everywhere, ticket #185. (As a side-effect we are now using JQuery 1.5.2.)
• Block import: generated names now sort numerically correctly (eg. "12-100 Main St" rather than "100-12 Main St")
• Block import: Don't try to guess right_from, right_to if not provided; that typically means there really is nothing on that side of the street.
• Boston demo: restaurant inspections scraper fixed to accomodate markup changes.
• De-hardcoded "neighborhoods" from various URLs. (#148)
• Zip code import UI has no default state (to avoid selecting Alabama by mistake).
• Zip code import now sets creation date (#233)
• Removed confusing NewsItem "About" page. (#228)
• Removed map from NewsItem list in admin UI, was too slow. (#219)
• SavedPlace now enforces that it has either a Block or a Location but not both. (#213)
• Items shown on map on schema filter page now use same filters as the items on the page. (#121)
• Support 2010 US Census tiger files (ticket #147). Use them for the Boston demo.
• Georeport / open311 scraper: support unofficial 'page' parameter (ticket #245); also, use the 'address' field for location_name if provided.
• Seeclickfix scraper: allow city & state params, don't hardcode to boston; ticket #243.
• place_detail_overview wasn't actually filtering by place.
• ajax date charts would blow up if no results found.
• Fix ticket #77: Now filtering news by item_date instead of pub_date since that's the date that's shown and used for aggregates.
• Fix "show/hide" buttons on place detail page and account page. (tickets #204, #115, 236)
• Fixed bug that caused many "Unknown" locations in location charts. (ticket #192). And removed "unknowns" entirely from the chart.
• Locations weren't capitalized on some pages. (ticket #202)
• Several bounds-related errors in Location import fixed (thanks to Bret Walker).
• Scrapers that create timezone-aware datetimes no longer blow up.
• GeoReport scraper: scrape a reasonable amount of days, not 60 every darn time. And do pagination (ticket #245)
• Georss scraper: Had the forwards / backwards coordinate test reversed :-
• Georss scraper: Skip items with no location_name.
• Fix some migration ordering bugs.
• parse_date no longer blows up if you feed it a date or datetime instance.
• CSS fixes for ajax date charts on location overview page.

Documentation¶

• Lots more docs about loading geographic data.
• Document email configuration. (ticket #205)
• Document what you get when doing paster create -t openblock.
• More docs about running on Amazon EC2.
• Describe differences from Everyblock
• More help_text added to several Model fields, so admin UI is slightly more self-documenting.
• Many many minor updates and tweaks.

Other¶

• Upgraded to OpenLayers 2.11. (ticket #250)
• Upgraded to Django 1.3.1.
• Upgraded to JQuery 1.5.2.
• Removed some unused template tags (SHORT_NAME, STATE_ABBREV, EB_SUBDOMAIN).
• Removed old version of map popups code.

Documentation¶

• Added docs for cloning an EC2 instance from our Amazon AMI.
• Document the scrapers that ship out-of-the-box with ebdata.
• Remove nonexistent --city option from geodata docs.
• Changed docs theme for easier navigation.

Upgrade Notes¶

• If you originally installed, or already upgraded to, OpenBlock 1.0 Beta, you may skip the rest of this section.

• If you have an existing database that was built with 1.0a1 or earlier, you'll need to run this command to deal with the removal of the "django-apikey" dependency:

  django-admin.py migrate apikey 0001 --fake
  

• Many data-loading scripts that were scattered all over the source tree are now installed into your environment's bin directory, so they should be on your $PATH. Documentation has been updated accordingly.

• As usual, you should always run after upgrading:

  django-admin.py syncdb --migrate
  

  If you were unlucky and had last migrated with a git checkout including migrations that later got renamed or removed, you may get errors from migrating. In that case try adding the --delete-ghost-migrations option.

• Production webserver configurations will need a line added to get the django-olwidget javascript and CSS to show up. For example, for Apache you'd add a line like (adjust path as needed):

  Alias /olwidget/ /home/openblock/openblock/src/django-olwidget/
  

• We now require Django 1.3. This probably doesn't have any impact on you. (ticket #155).

• Settings changes:

  • MAP_BASELAYER_TYPE can now be any base layer supported by olwidget, eg. "google.streets". (Some require other settings for eg. API keys; see ebpub/settings_default.py for comments and examples.)

  • You can add custom base layers to your maps by creating the dictionary settings.MAP_CUSTOM_BASE_LAYERS. See ebpub/settings_default.py for an example.

    This replaces the WMS_URL setting from openblock 1.0a1 which is no longer supported.

New Features in 1.0¶

• ticket #33: Different map icons for different news item types. To use this, you can use the admin UI to configure "map icon url" or "map color" for a Schema.
• ticket #85: Added streets.PlaceType model for categorizing Places. These also can have individual colors or icon URLs on the /maps/ view. (Original ticket title was "'Landmark' location type")
• ticket #142: JSON push API for news items. See docs/main/api.rst
• ticket #187: REST API standard features: API key provisioning; require keys (or auth) for POST / DELETE; throttling
• Import US Zip Codes as Locations, via the admin UI.
• Work-in-progress: user-submitted content. See code in the ebpub/neighbornews app.
• Work-in-progress: Maps you can share just by copy/pasting a URL. For a sneak preview, browse to /maps/.
• Much better admin UI maps. (ticket #140: Bad admin UI for GeometryFields)
• ticket #72: unify NewsItem.attributes and NewsItem.attribute_values
• ticket #52: Proper validation for Street Misspellings in admin
• ticket #157: fill in normalized name automatically
• ticket #123: Configurable base layer should apply to admin UI maps too

Bug fixes¶

This is not a complete list; not all bugs fixed in this release were ticketed.

• Fix #172: schema_detail view blows up (TypeError) if there are no NewsItems in the last 30 days, but there is a matching AggregateLocation. (That shouldn't happen, but evidently did with some boston demo schemas; also fixed a related possible off-by-one error that may have been a factor.)
• Schema filter page: don't say 'You might want to try...' if there's nothing to try.
• Fix bug where scrapers that create timezone-aware datetimes blow up
• Fix errors in bounds checking in location importers, thanks to Bret Walker.
• Fix missing import in Place admin
• Fixed several bugs where django-nose (optional) would try to run some things that aren't tests.
• ticket #110: Fix infinite loop if newsitem.schema.has_newsitem_detail is False but newsitem.url is empty; give 404 instead.
• Importers should now not blow up if run more than once.
• ticket #22: Scraper scripts in everyblock/cities/boston mostly don't work OOTB
• ticket #79: Geotagging oddity
• ticket #188: items.json doesn't include location_name
• ticket #200: "obdemo bin scripts are documented, but don't get installed when installing obdemo non-editable"

Documentation¶

• ticket #80: Documentation for Street Misspellings
• ticket #162: Document pip / easy_install workarounds
• ticket #139: Document adding database user / granting database access
• ticket #198: version number in documentation
• ticket #197: documentation for deploying static media
• Fix outdated paths to example scrapers.
• Fix location of get_or_create_lookup
• Note differences from everyblock

Other¶

• ticket #181: Prepare packages for distribution on pypi.
• ticket #83: Split out non-core packages into a separate download (ebblog, ebwiki, ebgeo, ebinternal, and everyblock are now at https://github.com/openplans/openblock-extras )
• ticket #156: Removing lots of clustering code that's totally unused.
• remove a redundant get_metro_bbox function from ebpub.utils.geodjango; use get_default_bounds(), does the same thing.

Highlights¶

• Out-of-the-box theme, with maps.
• REST API
• Enable the Django admin UI
• Embeddable widgets that you can configure via the admin UI
• Lots more documentation