pycsw 1.8.1 Documentation¶

Standards Support¶

Supported Operations¶

Supported Output Formats¶

• XML (default)
• JSON

Supported Output Schemas¶

• Dublin Core
• ISO 19139
• FGDC CSDGM
• NASA DIF
• Atom

Supported Sorting Functionality¶

• ogc:SortBy
• ascending or descending
• aspatial (queryable properties)
• spatial (geometric area)

Supported Filters¶

System Requirements¶

pycsw is written in Python, and works with (tested) version 2.6 and 2.7

pycsw requires the following Python supporting libraries:

• lxml for XML support
• SQLAlchemy for database bindings
• pyproj for coordinate transformations
• Shapely for spatial query / geometry support
• OWSLib for CSW client and metadata parser

Installing from Source¶

Download the latest stable version or fetch from Git.

$ virtualenv pycsw && cd pycsw && . bin/activate
$ git clone https://github.com/geopython/pycsw.git && cd pycsw
$ pip install -e . && pip install -r requirements-standalone.txt
$ cp default-sample.cfg default.cfg
$ vi default.cfg
# adjust paths in
# - server.home
# - repository.database
# set server.url to http://localhost:8000/
$ python csw.wsgi
$ curl http://localhost:8000/?service=CSW&version=2.0.2&request=GetCapabilities

$ git clone git://github.com/geopython/pycsw.git

<Location /pycsw/>
 Options FollowSymLinks +ExecCGI
 Allow from all
 AddHandler cgi-script .py
</Location>

$ git clone git://github.com/geopython/pycsw.git
$ python setup.py build
$ python setup.py install

Installing from the Python Package Index (PyPi)¶

# easy_install or pip will do the trick
$ easy_install pycsw
# or
$ pip install pycsw

Installing from OpenSUSE Build Service¶

In order to install the OBS package in openSUSE 12.3, one can run the following commands as user root:

# zypper -ar http://download.opensuse.org/repositories/Application:/Geo/openSUSE_12.3/ GEO
# zypper -ar http://download.opensuse.org/repositories/devel:/languages:/python/openSUSE_12.3/ python
# zypper refresh
# zypper install python-pycsw pycsw-cgi

For earlier openSUSE versions change 12.3 with 12.2. For future openSUSE version use Factory.

An alternative method is to use the One-Click Installer.

Installing on Ubuntu/Xubuntu/Kubuntu¶

In order to install pycsw to an Ubuntu based distribution, one can run the following commands:

# sudo add-apt-repository ppa:pycsw/stable
# sudo apt-get update
# sudo apt-get install python-pycsw pycsw-cgi

An alternative method is to use the OSGeoLive installation script located in pycsw/etc/dist/osgeolive:

# cd pycsw/etc/dist
# sudo ./install_pycsw.sh

The script installs the dependencies (Apache, lxml, sqlalchemy, shapely, pyproj) and then pycsw to /var/www.

Running on Windows¶

For Windows installs, change the first line of csw.py to:

#!/Python27/python -u

Security¶

By default, default.cfg is at the root of the pycsw install. If pycsw is setup outside an HTTP server’s cgi-bin area, this file could be read. The following options protect the configuration:

• move default.cfg to a non HTTP accessible area, and modify csw.py to point to the updated location
• configure web server to deny access to the configuration. For example, in Apache, add the following to httpd.conf:

<Files ~ "\.(cfg)$">
 order allow,deny
 deny from all
</Files>

Running on WSGI¶

pycsw supports the Web Server Gateway Interface (WSGI). To run pycsw in WSGI mode, use csw.wsgi in your WSGI server environment. Below is an example of configuring with Apache:

WSGIDaemonProcess host1 home=/var/www/pycsw processes=2
WSGIProcessGroup host1
WSGIScriptAlias /pycsw-wsgi /var/www/pycsw/csw.wsgi
<Directory /var/www/pycsw>
  Order deny,allow
  Allow from all
</Directory>

or use the WSGI reference implementation:

$ python ./csw.wsgi
Serving on port 8000...

which will publish pycsw to http://localhost:8000/

Alternate Configurations¶

By default, pycsw loads default.cfg at runtime. To load an alternate configuration, modify csw.py to point to the desired configuration. Alternatively, pycsw supports explicitly specifiying a configuration by appending config=/path/to/default.cfg to the base URL of the service (e.g. http://localhost/pycsw/csw.py?config=tests/suites/default/default.cfg&service=CSW&version=2.0.2&request=GetCapabilities). When the config parameter is passed by a CSW client, pycsw will override the default configuration location and subsequent settings with those of the specified configuration.

This also provides the functionality to deploy numerous CSW servers with a single pycsw installation.

Alias /pycsw/csw-foo.py /var/www/pycsw/csw.py
SetEnvIf Request_URI "/pycsw/csw-foo.py" PYCSW_CONFIG=/var/www/pycsw/csw-foo.cfg

#!/bin/sh

export PYCSW_CONFIG=/var/www/pycsw/csw-foo.cfg

/var/www/pycsw/csw.py

Metadata Repository Setup¶

pycsw supports the following databases:

• SQLite3
• PostgreSQL
• PostgreSQL with PostGIS enabled
• MySQL

To expose your geospatial metadata via pycsw, perform the following actions:

• setup the database
• import metadata
• publish the repository

Supported Information Models¶

By default, pycsw supports the csw:Record information model.

Setting up the Database¶

$ pycsw-admin.py -c setup_db -f default.cfg

This will create the necessary tables and values for the repository.

The database created is an OGC SFSQL compliant database, and can be used with any implementing software. For example, to use with OGR:

$ ogrinfo /path/to/records.db
INFO: Open of 'records.db'
using driver 'SQLite' successful.
1: records (Polygon)
$ ogrinfo -al /path/to/records.db
# lots of output

Loading Records¶

$ pycsw-admin.py -c load_records -f default.cfg -p /path/to/records

This will import all *.xml records from /path/to/records into the database specified in default.cfg (repository.database). Passing -r to the script will process /path/to/records recursively.

Exporting the Repository¶

$ pycsw-admin.py -c export_records -f default.cfg -p /path/to/output_dir

This will write each record in the database specified in default.cfg (repository.database) to an XML document on disk, in directory /path/to/output_dir.

Optimizing the Database¶

$ pycsw-admin.py -c optimize_db -f default.cfg

Database Specific Notes¶

Mapping to an Existing Repository¶

pycsw supports publishing metadata from an existing repository. To enable this functionality, the default database mappings must be modified to represent the existing database columns mapping to the abstract core model (the default mappings are in pycsw/config.py:MD_CORE_MODEL).

To override the default settings:

• define a custom database mapping based on etc/mappings.py
• in default.cfg, set repository.mappings to the location of the mappings.py file:

[repository]
...
mappings=path/to/mappings.py

See the GeoNode Configuration and Open Data Catalog Configuration for further examples.

Scenario: Federated Search¶

pycsw deployment with 3 configurations (CSW-1, CSW-2, CSW-3), subsequently providing three (3) endpoints. Each endpoint is based on an opaque metadata repository (based on theme/place/discipline, etc.). Goal is to perform a single search against all endpoints.

pycsw realizes this functionality by supporting alternate configurations, and exposes the additional CSW endpoint(s) with the following design pattern:

CSW-1: http://localhost/pycsw/csw.py?config=CSW-1.cfg

CSW-2: http://localhost/pycsw/csw.py?config=CSW-2.cfg

CSW-3: http://localhost/pycsw/csw.py?config=CSW-3.cfg

...where the *.cfg configuration files are configured for each respective metadata repository. The above CSW endpoints can be interacted with as usual.

To federate the discovery of the three (3) portals into a unified search, pycsw realizes this functionality by deploying an additional configuration which acts as the superset of CSW-1, CSW-2, CSW-3:

CSW-all: http://localhost/pycsw/csw.py?config=CSW-all.cfg

This allows the client to invoke one (1) CSW GetRecords request, in which the CSW endpoint spawns the same GetRecords request to 1..n distributed CSW endpoints. Distributed CSW endpoints are advertised in CSW Capabilities XML via ows:Constraint:

<ows:OperationsMetadata>
...
    <ows:Constraint name="FederatedCatalogues">
        <ows:Value>http://localhost/pycsw/csw.py?config=CSW-1.cfg</ows:Value>
        <ows:Value>http://localhost/pycsw/csw.py?config=CSW-2.cfg</ows:Value>
        <ows:Value>http://localhost/pycsw/csw.py?config=CSW-3.cfg</ows:Value>
    </ows:Constraint>
...
</ows:OperationsMetadata>

...which advertises which CSW endpoint(s) the CSW server will spawn if a distributed search is requested by the client.

in the CSW-all configuration:

[server]
...
federatedcatalogues=http://localhost/pycsw/csw.py?config=CSW-1.cfg,http://localhost/pycsw/csw.py?config=CSW-2.cfg,http://localhost/pycsw/csw.py?config=CSW-3.cfg

At which point a CSW client request to CSW-all with distributedsearch=TRUE, while specifying an optional hopCount. Query network topology:

     AnyClient
         ^
         |
         v
      CSW-all
         ^
         |
         v
  /-------------\
  ^      ^      ^
  |      |      |
  v      v      v
CSW-1  CSW-2  CSW-3

As a result, a pycsw deployment in this scenario may be approached on a per ‘theme’ basis, or at an aggregate level.

All interaction in this scenario is local to the pycsw installation, so network performance would not be problematic.

A very important facet of distributed search is as per Annex B of OGC:CSW 2.0.2. Given that all the CSW endpoints are managed locally, duplicates and infinite looping are not deemed to present an issue.

Description Document¶

To generate an OpenSearch Description Document:

$ cd /path/to/pycsw
$ export PYTHONPATH=`pwd`
$ python-admin.py -c gen_opensearch_description -f default.cfg -o /path/to/opensearch.xml

This will create the document which can then be autodiscovered.

OpenSearch support is enabled by default. HTTP requests must be specified with mode=opensearch in the base URL for OpenSearch requests, e.g.:

http://localhost/pycsw/csw.py?mode=opensearch&service=CSW&verison=2.0.2&request=GetRecords&elementsetname=brief&typenames=csw:Record&resulttype=results

Supported Resource Types¶

For transactions and harvesting, pycsw supports the following metadata resource types by default:

Additional metadata models are supported by enabling the appropriate Profile Plugins.

$ mkdir /path/data
$ chmod 777 /path/data
$ chmod 666 test.db
$ mv test.db /path/data

For CSW-T deployments, it is strongly advised that this directory reside in an area that is not accessible by HTTP.

Harvesting¶

pycsw supports the CSW-T Harvest operation. Records which are harvested require to setup a cronjob to periodically refresh records in the local repository. A sample cronjob is available in etc/harvest-all.cron which points to pycsw-admin.py (you must specify the correct path to your configuration). Harvest operation results can be sent by email (via mailto:) or ftp (via ftp://) if the Harvest request specifies csw:ResponseHandler.

Transactions¶

pycsw supports 3 modes of the Transaction operation (Insert, Update, Delete):

• Insert: full XML documents can be inserted as per CSW-T
• Update: updates can be made as full record updates or record properties against a csw:Constraint
• Delete: deletes can be made against a csw:Constraint

Transaction operation results can be sent by email (via mailto:) or ftp (via ftp://) if the Transaction request specifies csw:ResponseHandler.

The Tester contain CSW-T request examples.

Scenario: One Database, Many Views¶

Imagine a sample database table of records (subset below for brevity):

A default pycsw instance (with no repository.filters option) will always process CSW requests against the entire table. So a CSW GetRecords filter like:

<ogc:Filter>
    <ogc:PropertyIsEqualTo>
        <ogc:PropertyName>apiso:Title</ogc:PropertyName>
        <ogc:Literal>foo1</ogc:Literal>
    </ogc:PropertyIsEqualTo>
</ogc:Filter>

...will return:

Suppose you wanted to deploy another pycsw instance which serves metadata from the same database, but only from a specific subset. Here we set the repository.filter option:

[repository]
database=sqlite:///records.db
filter=pycsw:ParentIdentifier = '33'

The same CSW GetRecords filter as per above then yields the following results:

Another example:

[repository]
database=sqlite:///records.db
filter=pycsw:ParentIdentifier != '33'

The same CSW GetRecords filter as per above then yields the following results:

The repository.filter option accepts all core queryables set in the pycsw core model (see pycsw.config.StaticContext.md_core_model for the complete list).

Overview¶

pycsw allows for the implementation of profiles to the core standard. Profiles allow specification of additional metadata format types (i.e. ISO 19139:2007, NASA DIF, INSPIRE, etc.) to the repository, which can be queried and presented to the client. pycsw supports a plugin architecture which allows for runtime loading of Python code.

All profiles must be placed in the pycsw/plugins/profiles directory.

Requirements¶

pycsw/
  plugins/
  __init__.py # empty
  profiles/ # directory to store profiles
    __init__.py # empty
    profile.py # defines abstract profile object (properties and methods) and functions to load plugins
    apiso/ # profile directory
      __init__.py # empty
      apiso.py # profile code
      ... # supporting files, etc.

Abstract Base Class Definition¶

All profile code must be instantiated as a subclass of profile.Profile. Below is an example to add a Foo profile:

from pycsw.plugins.profiles import profile

class FooProfile(profile.Profile):
    profile.Profile.__init__(self,
        name='foo',
        version='1.0.3',
        title='My Foo Profile',
        url='http://example.org/fooprofile/docs',
        namespace='http://example.org/foons',
        typename='foo:RootElement',
        outputschema=http://example.org/foons',
        prefixes=['foo'],
        model=model,
        core_namespaces=namespaces,
        added_namespaces={'foo': 'http://example.org/foons'}
        repository=REPOSITORY['foo:RootElement'])

Your profile plugin class (FooProfile) must implement all methods as per profile.Profile. Profile methods must always return lxml.etree.Element types, or None.

Enabling Profiles¶

All profiles are disabled by default. To specify profiles at runtime, set the server.profiles value in the Configuration to the name of the package (in the pycsw/plugins/profiles directory). To enable multiple profiles, specify as a comma separated value (see Configuration).

Testing¶

Profiles must add examples to the Tester interface, which must provide example requests specific to the profile.

ISO Metadata Application Profile (1.0.0)¶

INSPIRE Extension¶

CSW-ebRIM Registry Service - Part 1: ebRIM profile of CSW¶

Overview¶

pycsw allows for extending the implementation of output schemas to the core standard. outputschemas allow for a client to request metadata in a specific format (ISO, Dublin Core, FGDC, NASA DIF and Atom are default).

All outputschemas must be placed in the pycsw/plugins/outputschemas directory.

Requirements¶

pycsw/
  plugins/
  __init__.py # empty
  outputschemas/
    __init__.py # __all__ is a list of all provided outputschemas
    atom.py # default
    dif.py # default
    fgdc.py # default

Implementing a new outputschema¶

Create a file in pycsw/plugins/outputschemas, which defines the following:

• NAMESPACE: the default namespace of the outputschema which will be advertised
• NAMESPACE: dict of all applicable namespaces to outputschema
• XPATH_MAPPINGS: dict of pycsw core queryables mapped to the equivalent XPath of the outputschema
• write_record: function which returns a record as an lxml.etree.Element object

Add the name of the file to __init__.py:__all__. The new outputschema is now supported in pycsw.

Testing¶

New outputschemas must add examples to the Tester interface, which must provide example requests specific to the profile.

GeoNode Setup¶

pycsw is enabled and configured by default in GeoNode, so there are no additional steps required once GeoNode is setup. See the CATALOGUE and PYCSW settings.py entries at http://docs.geonode.org/en/latest/developers/reference/django-apps.html#id1 for customizing pycsw within GeoNode.

Open Data Catalog Setup¶

Open Data Catalog provides CSW functionality using pycsw out of the box (installing ODC will also install pycsw). Settings are defined in https://github.com/azavea/Open-Data-Catalog/blob/master/OpenDataCatalog/settings.py#L165.

At this point, pycsw is able to read from the Open Data Catalog repository using the Django ORM.

CKAN Setup¶

Installation and configuration Instructions are provided as part of the ckanext-spatial documentation.

OGC CITE¶

Compliance benchmarking is done via the OGC Compliance & Interoperability Testing & Evaluation Initiative. The pycsw wiki documents testing procedures and status.

Tester¶

The pycsw tests framework (in tests) is a collection of testsuites to perform automated regession testing of the codebase. Test are run against all pushes to the GitHub repository via Travis CI.

$ cd /path/to/pycsw
# run all tests (starts up http://localhost:8000)
$ paver test
# run tests only against specific testsuites
$ paver test -s apiso,fgdc
# run all tests, including harvesting (this is turned off by default given the volatility of remote services/data testing)
$ paver test -r

$ paver gen_tests_html

CSW Clients¶

• Geoportal CSW Clients
• OWSLib
• qgcsw (QGIS plugin)

CSW Servers¶

• deegree
• eXcat
• GeoNetwork opensource

Metadata Editing Tools¶

• CatMDEdit
• EUOSME
• GIMED
• Metatools (QGIS plugin)

Community¶

Please see the Community page for information on the pycsw community, getting support, and how to get involved.

GitHub¶

Code, tests, documentation, wiki and issue tracking are all managed on GitHub. Make sure you have a GitHub account.

Code Overview¶

• the pycsw wiki documents an overview of the codebase

Documentation¶

• documentation is managed in docs/, in reStructuredText format
• Sphinx is used to generate the documentation
• See the reStructuredText Primer on rST markup and syntax.

Bugs¶

pycsw’s issue tracker is the place to report bugs or request enhancements. To submit a bug be sure to specify the pycsw version you are using, the appropriate component, a description of how to reproduce the bug, as well as what version of Python and platform. For convenience, you can run pycsw-admin.py -c get_sysprof and copy/paste the output into your issue.

Forking pycsw¶

Contributions are most easily managed via GitHub pull requests. Fork pycsw into your own GitHub repository to be able to commit your work and submit pull requests.

Development¶

# setup a virtualenv
$ virtualenv mypycsw && cd mypycsw
$ . ./bin/activate
# clone the repository locally
$ git clone git@github.com:USERNAME/pycsw.git
$ cd pycsw
$ pip install -e . && pip install -r requirements-standalone.txt
# add the main pycsw master branch to keep up to date with upstream changes
$ git remote add upstream https://github.com/geopython/pycsw.git
$ git pull upstream master
# create a local branch off master
# The name of the branch should include the issue number if it exists
$ git branch 72-foo
$ git checkout 72-foo
#
# make code/doc changes
#
$ git commit -am 'fix xyz (#72-foo)'
$ git push origin 72-foo

$ git checkout master
$ git pull upstream master

GitHub Commit Access¶

• proposals to provide developers with GitHub commit access shall be emailed to the pycsw-devel mailing list. Proposals shall be approved by the pycsw development team. Committers shall be added by the project admin
• removal of commit access shall be handled in the same manner
• each committer shall be listed in https://github.com/geopython/pycsw/blob/master/COMMITTERS.txt