DKAN Open Data Portal

This is the central site for technical/developer documentation of DKAN. DKAN is a Drupal-based open data portal and catalog developed by Granicus (previously NuCivic and GovDelivery).

DKAN Overview

DKAN is a Drupal-based open data tool with a full suite of cataloging, publishing and visualization features that allows governments, nonprofits and universities to easily publish data to the public. DKAN is maintained by GovDelivery.

About this documentation

What follows is a style guide for the DKAN documentation. Use it both to follow the conventions used throughout the site, and for your own contributions. DKAN’s docs are written in a combination of Markdown (specifiically, CommonMark) and ReStructuredText (RST), and built with Sphynx. The docs live in the /docs folder of the DKAN Project; to suggest modifications, submit a pull request as you would for any suggested code change.

File types

Index files should always be in RST, to render correctly in the sidebar when built. Additional files can be in markdown or RST format depending on your preference. Currently, most DKAN documentation is in Markdown, mainly for historical reasons.

In some cases, README.md files are pulled into the docs site from elsewhere in the repository. This is accomplished with symbolic links in the docs folder.

Images

Screenshots should be taken at standard desktop resolution (no retina!) and avoid showing any browser chrome. If necessary they may contain arrows and annotations in red with sans-serif typeface.

Text conventions

Modules

Module names are written in Title Case with no additional styling. Quotes can be used if needed for clarity – for instance, it might be confusing to talk about how the “Data” module affects data on the site without quote marks. When possible, a module name is linked to its home page (on Drupal.org or Github) on its first mention in a page.

Entities and bundles

A specific content type or other entity bundle is written in italics, as in referring to a dataset node or a chloropleth visualization. Entity types, like “node,” require no additional styling.

Files

Filenames are written as inline code as in this example: thisfile.txt will do the trick.

Terminal commands

Terminal commands should be expressed in a full code block, with each line starting with$:

$ first -i "run" this-command
$ ../then.this --one
Code blocks

Code blocks are also expressed as... code blocks:

/**
 * Adds declared endpoint to list.
 *
 * This and hook_open_data_schema_map_load() are necessary so that modules can
 * declare more than one endpoint.
 */
function hook_open_data_schema_map_endpoints_alter(&$records) {
  $records[] = 'my_machine_name';
}
Code objects

When referring to $variables, function_names() and classNames inline, use bold inline code style. This can be achieved in markdown like this:

**`This text`** will be code-styled and bold

Catalog Basics

Open data catalogs - or portals - are simple in purpose but may appear complicated at first glance. A homepage will often feature a slideshow, list of blog posts, or a visualization using data from the underlying catalog; these homepage assets can easily obscure the underlying structure of the data.

Understanding the structure of an open data catalog can help users, developers, and stakeholders make the best use of open data tools.

Catalog

The catalog is a wrapper containing information, sorted into datasets. A catalog should provide the answers to basic questions such as: “Who is providing this data?” and “Under what conditions?”

DCAT - an RDF vocabulary designed to facilitate interoperability between data catalogs published on the Web - recommends providing the following fields: catalog record, dataset, description, homepage, language, license, publisher, release date, rights, spatial, themes, title and update date.

Dataset

A dataset contains individual resources as well as metadata. Metadata is the “Who, What, When, Where, Why” of each dataset. There are a number of specifications for dataset metadata and DKAN currently complies with the DCAT and Project Open Data schemas.

It is possible to add new fields to DKAN to conform to additional specifications or custom requirements.

Resource

Resources are the actual files, APIs or links that are being shared. Resource types include csv, html, xls, json, xlsx, doc, docx, rdf, txt, jpg, png, gif, tiff, pdf, odf, ods, odt, tsv, geojson and xml files. If the resource is an API, it can be used as a live source of information for building a site or application.

Datasets and Resources within DKAN and CKAN

The following images display how datasets and resources are structured within DKAN and CKAN. Screenshots are from Data.gov which is powered by CKAN. DKAN follows CKAN conventions where possible.

_images/DatasetResourceDiagram2.png
_images/datasets-resources.png
Metadata

Comparing DKAN and CKAN

CKAN is an open data catalog that has powered many high-profile portals, including the main open data portals for both the United Kingdom and the United States, among others. The makers of DKAN have enjoyed contributing to and deploying CKAN. So why DKAN?

Ultimately, DKAN is a complimentary offering to CKAN in the effort to make data more open and accessible.

Drupal and PHP Ecosystems

PHP powers a significant percentage of web pages and Drupal powers an estimated 2% of the Internet as a whole. This percentage is even higher among government entities who choose to publish open data. DKAN offers an easy option for those who have already adopted PHP or Drupal. DKAN can also be enabled in existing Drupal sites so that anyone using Drupal can easily start to publish open data in standards compliant ways.

One of the design goals of DKAN is to make it easy for anyone with an inexpensive hosting environment to create an open data catalog. Thanks to the popularity of Drupal there are many resources to help install and host Drupal sites like DKAN.

Integrated Content Management System

CKAN has powerful publishing, auditing, and harvesting features for open datasets. Those using CKAN often choose to pair it with Drupal, Wordpress, Django, or other content management systems (CMS) or web publishing platforms to create pages, blogs and other content.

DKAN takes a different approach by integrating open data catalog features into an existing CMS. Datasets are treated as content that can unlock rich workflows. Drupal also provides a user interface for many site management activities. In turn, teams managing content only need to be trained on one system instead of two.

The fact that DKAN provides a single codebase is another benefit. Again, DKAN is a complementary effort to CKAN in enabling people to publish open data using open source tools.

Get DKAN

DKAN is open source and flexible: You can download it for free and run it on your own server or choose from one of our hosting partners below.

Download and run DKAN on your server

DKAN is based on the open source Drupal content and application framework and runs almost anywhere Drupal is supported. Users unfamiliar with Drupal may be more comfortable trying one of the hosted options listed below, or contacting us to obtain a private demonstration instance. There is extensive information on how to install DKAN on your own in the the developers/installation section of this site.

Hosting Partners

DKAN is open source and flexible: you can download it for free and run it on your own server (see installation instructions) or choose from one of our hosting partners below.

Acquia

Click here to install DKAN on Acquia for free.

Acquia offers a number of hosting tools built specifically for best maintaining Drupal sites. These include integrations with 3rd party systems like New Relic and Blaze Meter as well as reports on module updates, performance, and security reviews. Most importantly, Acquia offers a dashboard that makes it easy to move code (hosted by git), media files, and the database between development, testing, and production environments:

Acquia Dashboard screenshot

These tools allow a single site builder or team of developers to follow best practices, scale up if needed, and follow a rigorous QA process all without ever touching a server.

Single-click Installation

Acquia offers a “single-click” installation of DKAN. While this is labelled as a “Test drive,” the environment offers the same dashboard tools as a full, paid account. Visit Acquia’s website for complete instructions.

Maintaining a DKAN Site on Acquia

Updates to DKAN are released frequently. Acquia will not push these updates to your instance automatically, but you can keep your codebase up-to-date using your own workflow, or following our general Upgrade Instructions.

Pantheon

Click here to install DKAN on Pantheon for free.

Pantheon provides reliable Drupal cloud hosting with a powerful development tools and web-based user interface designed to facilitate and encourage best development practices. With essentially a single click, you can spin up a new DKAN instance on Pantheon and log in to your new site in just a few minutes.

You can register for a free account here. Once you have access, create a new site:

Pantheon new site form

Choose to make a new site from scratch, and to use the DKAN distribution:

DKAN distribution link

Pantheon will then build your new based site on the latest DKAN release. You will go through a normal Drupal install process, explained in detail in the installation instructions.

Managing updates

Pantheon uses a modified version of Drupal Pressflow, which is publicly available on GitHub. Whenever a new version of the DKAN distribution is released, the changes are merged into a version of DKAN special-built for Pantheon, also available on GitHub.

However, Pantheon provides an easy way to update your instance of DKAN (or any Drupal distribution hosted with them). Each time the DKAN’s Pantheon build is updated, an alert will appear in your Pantheon dashboard:

Pulling in upstream changes via UI

Usually, you will be able to use the “Apply Updates” button to merge those “upstream” changes directly into your copy of the codebase, alongside any changes you have already made to it. If you are developing locally using git, the next time you pull from your Pantheon repository, you’ll receive the DKAN updates locally as well.

If you have modified any of the files included with DKAN, merging in upstream changes may produce conflicts. Pantheon’s dashboard provides instructions for how to do the merge locally, to give you more control over resolving potential conflicts.

Troubleshooting

An error like the following is often seen at the end of the install process on Pantheon:

An AJAX HTTP error occurred. HTTP Result Code: 502 Debugging information follows...

This will hopefully be fixed on future releases. However, the resulting site should still be fully installed and functional.

Granicus Data Subscription

Granicus, the company leading DKAN’s development, offers a turn-key software as a service (SaaS) product called Granicus Data Enterprise. Hosted on Acquia Cloud, Granicus Data Enterprise meets US government security standards, comes pre-configured for out-of-the-box open data standards compliance and advanced data visualization techniques, and includes 24-hour, enterprise-level support. Contact Granicus for more information.

Installation

This document contains instructions for installing the DKAN open data publishing software on your webserver. If you’re not comfortable installing and maintaining server software, you may wish to Deploy a Ready-to-Run DKAN Instance instead.

Please note that we are in the process of revamping our installation and upgrade guide. The instructions here will work, but please bear with us as we develop better documentation and processes.

Before getting started, it’s recommended that you familiarize yourself with:

What you will find in the main DKAN Repository is a Drupal installation profile. To set up a working website using DKAN, you will need to acquire or build a full DKAN distribution of Drupal.

Tip

DKAN Starter is project containing a prebuilt version of DKAN and the tools Granicus uses for our own implementations and deployments. Learn more advanced workflows in that project’s documentation.

Requirements

Operating Environment

DKAN is based on Drupal software and – generally – runs anywhere Drupal is supported. This document assumes installation on a Linux-based Apache webserver using MySQL as a back-end database (aka LAMP server). For other environments, please see our Alternative Environment Support.

  • MySQL: minimum version 5.0.15+ with PDO
  • before installation, please create one MySQL database and associated user.
  • PHP: minimum version 5.3.x
  • Apache: minimum version 2.x
  • Git
Hardware

DKAN has been successfully tested in limited-resource environments, such as Amazon’s “micro” AWS instance, for development.

  • Minimum RAM: 1GB for development, 2GB or more recommended for production.
  • Minimum Disk: 64M for base installation, recommended 1GB or more for production.

DKAN is based on Drupal and follows the same basic installation procedure as any Drupal distribution. More information about various requirements can be located in the Drupal Installation Guide.

Pre Installation

Using fully made version

At the moment, our supported fully-made DKAN codebase is the DKAN DROPS-7 repository, which is optimized to run on the Pantheon platform. You can build a DKAN site with a single click on Pantheon here. (We also offer one-click installation on Acquia)

Download and unzip the latest version of the “DKAN DROPS” codebase on your server webroot.

if you want to do this with git instead:

$ git clone --branch master https://github.com/nuams/dkan-drops-7.git dkan
Build your own

This “builds” a full DKAN website codebase from the bleeding-edge development version of DKAN, by downloading Drupal and all the additional modules that DKAN needs to run. You may want to use this method to get recent changes that have not yet been included in an official release, or to use a branch or forked version of the DKAN profile.

Note that rsync is used to copy the DKAN profile inside the Drupal /profiles folder. You may wish to modify this process to fit your own development practices.

Requires drush version 8.x.

$ git clone --branch 7.x-1.x https://github.com/NuCivic/dkan.git
$ cd dkan
$ drush make --prepare-install drupal-org-core.make webroot --yes
$ rsync -av . webroot/profiles/dkan --exclude webroot
$ drush -y make --no-core --contrib-destination=./ drupal-org.make webroot/profiles/dkan --no-recursion
$ cd webroot

You can also build from a specific release of DKAN by checkout out the correct tag after cloning. For instance:

$ git clone --branch 7.x-1.x https://github.com/NuCivic/dkan.git
$ git checkout tags/7.x-1.11
...

The automated software builder will download and configure the latest version of DKAN and prepare it for installation. When complete, proceed to “Installing the DKAN Software” section below.

Note before proceeding: Recline previews require clean URLs

Installation

Once you’ve downloaded the DKAN software, it’s time to install it. If you’ve previously installed Drupal, this process will be very similar.

With drush
$ drush site-install dkan --db-url="mysql://DBUSER:DBPASS@localhost/DBNAME"

You can add the --verbose switch if you want to see every step. The installation should end with drush creating an admin account with a random password, which will be output in a message to the terminal.

With the web installer
  • Open a web browser and visit http://YOURDKANSITE/install.php:
Installation Screen

Installation Screen

  • The first installation screen is a language selection menu. Although DKAN does provide limited multi-language support, installation must currently be performed in English. Continue.
  • At this point, your server resources and capabilities are checked to ensure they meet DKAN installation requirements. All errors must be corrected before installation can proceed. Instructions for correcting each error condition are provided.
Installation Screen - database config

Installation Screen

  • Once your server meets all installation requirements, you’ll be presented with the database configuration screen. Enter your MySQL database name, database username, and database password, then click “Save to Continue” to proceed.
Progress Bar

Progress Bar

  • The installation will proceed, displaying a progress bar on the screen. Depending on your server resources, this may take several minutes.
Configuration Screen

Configuration

  • When installation is complete, the site configuration screen will be displayed. Follow the prompts to set your administrative username, email address, site name, time zone, and other default settings as shown. If the final configuration completes without error, you’ll see a short congratulatory message and you’ll be prompted to access your new site.
  • Proceed to “DKAN - Data Management” or “DKAN - User Management ” next to continue setting up your new DKAN server.

Install for development

This method is particularly useful for people who want to work on the DKAN project itself, as it preserves Git versioning information in every profile, theme and module directory. The core developers use this method when developing and testing DKAN.

Grab Development version
$ git clone --branch 7.x-1.x https://github.com/nuams/dkan.git
$ cd dkan
Build dkan
$ drush make --prepare-install drupal-org-core.make webroot --yes
$ rsync -av . webroot/profiles/dkan --exclude webroot
$ drush -y make --no-core --working-copy --contrib-destination=./ drupal-org.make webroot/profiles/dkan --no-recursion --concurrency=3
$ cd webroot

Updating and Maintaining DKAN

There are several strategies for maintaining your DKAN site. Maintaining a DKAN site does not differ substantially from maintaining other Drupal distributions.

Drupal distributions consist of a script that runs at the time of the installation as well as a set of modules, themes, and libraries that exist at profiles/MY_DISTRIBUTION directory. These modules, themes, and libraries work the same as any other modules, themes, or libraries that are added to Drupal sites. They are packaged together in the profiles directory to make it easier to install and maintain.

Tip

DKAN Starter is project containing a prebuilt version of DKAN and the tools Granicus uses for our own implementations and deployments. Learn more advanced workflows in that project’s documentation.

Filesystem Conventions

With Drupal’s inheritance model mentioned above, it should not be necessary to place custom code or modules in the profiles/dkan directory. Additional modules, themes, or libraries, or newer versions of ones already present in profiles/dkan, can be placed in sites/all.

If it is necessary or expedient to overwrite files in the profiles/dkan directory, it is recommended to keep a patch of the changes. A patch will make it possible to re-apply changes once a newer version of DKAN is added to the ‘profiles/dkan’ directory.

If DKAN’s extensions and customizations of core Drupal are isolated in profiles/DKAN, and your site’s particular configuration, files, and overrides and customizations of DKAN are isolated in sites/, maintaining your DKAN site will be much easier.

Primary Maintenance Tasks

By “maintenance” we mean three specific tasks

  • Upgrading DKAN to receive new features and bug-fixes
  • Adding additional modules or features
  • Overriding current modules or functionally

Getting DKAN Updates

DKAN uses Drupal versioning standards, with one modification. Minor upgrades to DKAN are released approximately every 4-6 weeks. For instance, a minor release would move from DKAN 7.x-1.11 to 7.x-1.12. Starting with version 7.x-1.12, we are adding patch releases for security and bug fixes. For instance, the first patch release between 7.x-1.12 and 7.x-1.13 will be 7.x-1.12.1.

Please note you can not use drush up with DKAN. This is because DKAN is not packaged on Drupal.org.

Basic Upgrades

The least complex way to update your DKAN codebase is similar to an update of Drupal itself.

  1. Back up your database (just in case!)
  2. Copy your sites folder somewhere safe.
  3. Replace your entire codebase with the latest fully built version of DKAN from DKAN DROPS-7.
  4. Check the new versions’ release notes to see if there are any special instructions for updating. (If you are several releases behind, you may need to follow instructions for several releases).
  5. Replace the sites folder in your new codebase with your old sites folder.
  6. Now navigate to http://yoursite.com/update.php or run drush updatedb.
  7. Clear caches by visiting /admin/performance or running drush cache-clear all.
  8. Revert all features by visiting /admin/structure/features or running drush features-revert-all (Use with caution, as this may overwrite any DKAN configuration you have overridden and not exported to code; see Features for more information.)

Note: Occasionally a DKAN component will be moved to a new directory. This should be explained in the release notes for that version. But if you get errors related to incorrect location of module files, you may want to try rebuilding the registry.

Using drush make

We are developing an easier workflow to update DKAN on the command line. For the time being, the recommended method for updating using the drush make instructions described in the Installation Instructions is similar to the process described above.

Assuming you have followed the instructions for drush make and have a webroot folder inside a main clone of the DKAN repo:

  1. Back up your database
  2. Copy your sites folder somewhere safe.
  3. Remove your webroot folder: rm -rf webroot (use with caution!)
  4. Check out the new version of DKAN you want to update to: git checkout tags/7.x-1.12
  5. drush make drupal-org-core.make webroot --yes
  6. rsync -av . webroot/profiles/dkan --exclude webroot
  7. drush make --no-core --contrib-destination=./ drupal-org.make webroot/profiles/dkan --no-recursion --yes
  8. Replace the sites folder in your new codebase with your old sites folder.
  9. Check the new versions’ release notes to see if there are any special instructions for updating. (If you are several releases behind, you may need to follow instructions for several releases).
  10. drush updatedb.
  11. drush cache-clear all
  12. drush features-revert-all (use with caution).

You can also use this method to upgrade to the most recent “bleeding-edge” development version of DKAN. Instead of checking out a specific tag, check out the 7.x-1.x branch in step 3.

Features Module

DKAN packages much of its configuration using the Features module.

After DKAN is upgraded DKAN site maintainers may wish to revert some features in order to take advantage of new functionality. We recommend using the Features Override module to capture overridden features elements to make it easier to revert Features from DKAN when desired. More documentation on this to come.

Advanced Workflows

Using a Custom Make file

DKAN is “built” using a make file and drush make. The drupal-org.make file in DKAN contains a list of most of the modules installed in DKAN.

When developing a website for production, it is recommended to keep a make file for all custom modules added to DKAN. Instead of using drush pm-download or other means of downloading and adding modules to sites/all, a make file is kept that has a list of the sites modules. This enforces some best practices about not overwriting contributed modules, maintaining patches, and reusability. This make file along with DKAN’s makefiles also provide a reusable recipe for your site.

More documentation and automation scripts regarding this process are under active development and can be viewed here: DKAN Starter Documentation.

Adding additional modules or features

New modules, themes, or libraries should be added to the ‘sites/all’ directory. For modules or themes it is often useful to differentiate “custom” modules from “community” modules. We often have a directory structure for modules like:

Location Contents
sites/all/modules/contrib community or contributed modules
sites/all/modules/custom custom modules
sites/all/libraries Additional libraries
Overriding current DKAN modules or functionality

Drupal has an inheritance model that makes it easy to override modules added to distributions as well as the functionality of other modules.

Any modules or themes added to sites/all will override the same named module as one that is placed in profiles/dkan/.

If a DKAN site maintainer wishes to update a module supplied by DKAN that module can be placed in “sites/all”. For example if one wished to update the Date module, if there is a security update or new release with a certain functionality, add it to sites/all:

Location Version
profiles/dkan/modules/contrib/date 7.x-1.4
sites/all/modules/contrib/date 7.x-1.5

In this case, DKAN will use the version 7.x-1.5 and ignore 7.x-1.4.

If, later, you update your site to a version of DKAN that uses Date v. 7.x-1.5, the version in sites/all should be removed. Be careful to review your overrides in sites/all after every DKAN update to ensure you are not missing important module updates.

Note that moving to a different location for an existing, installed module will require a Registry Rebuild to prompt Drupal to refresh all module paths.

Major Components

This section contains the documentation for each of the major modules and other components that make up DKAN.

With the exception of the modules described in the last two items in this table of contents (Open Data Schema Map and Visualization Entity), and of the Recline module which is described inside the Datasets section, all this functionality is provided by the modules that ship with the DKAN profile.

DKAN Dataset Module and Sub-Modules

DKAN’s core functionality around datasets, their metadata and resources, is defined in “DKAN Dataset” (dkan_dataset) and its submodules (in the modules folder):

  • DKAN Dataset Content Types contains the actual Features exports for the Dataset and Resource content types and fields.
  • DKAN Dataset Rest API defines a REST endpoint via the Services module, exposing full CRUD operations to authenticated 3rd-party apps and services. See the Dataset REST API documentation for more information.
  • DKAN Dataset Groups provides Organic Groups functionality in DKAN, which groups both dataset content and site users into discreet groups with separate branding and granular access permissions. Usually used to allow for multiple data publishers (for instance, sub-agencies sharing a single data portal).

Usage

Creating Datasets and Resources

DKAN’s data publishing model is based on the concept of datasets and resources. A dataset is a container for one or more resources; a resource is the actual “data” being published, such as a CSV table, a GeoJSON data file, or a TIFF aerial image.

The dataset and resource content types in DKAN are provided by the DKAN Dataset module.

In our example, we’ll be adding a dataset with Wisconsin polling places to a DKAN site. The data may look familiar; it’s one of the sample datasets provided with DKAN upon installation.

Step 1: Create the Dataset

By default, only authenticated (“logged-in”) users can add new Datasets and Resources to a DKAN website.The default DKAN user permissions allows Site managers, Editors, and Content Creators access to the administration menu. From here a user may navigate to the Content » Add Content » Dataset link to access the “Create Dataset” form.

The Dataset is the container for the actual data resource files and contains basic information about the data, such as title, description, category tags, and license. Once we’ve entered information about the data, we can click the “Next: Add data” button to begin adding data.

Step 2: Add one or more Resources to the Dataset

After creating a dataset, we’re prompted to add one or more data resources to it. There are three types of Resources that can be added to a Dataset, depending on the type and location of the Resource:

Upload:This option allows publishers to upload data files to the DKAN site. As in the “link to a file” option, the data within the file will be imported into your DKAN site’s Datastore for preview and analysis by your users. See The DKAN Datastore for more information.
API or Website URL:
 Some data resources aren’t standalone files but queryable online databases; the interface to these databases is known as an API. Adding links to these types of online database interfaces to your DKAN data catalog can be very useful for developers interested in working with your data.
Remote file:This option allows publishers to create a link to a data file published on another Internet website. Although the file itself will remain on the other site, the data within the file can be imported into your DKAN site’s Datastore for preview and analysis by your users. See DKAN Datastore for more information.

Note

To provide previews for your resources, they must contain either a local or remote file (Link to a file or Upload a file). If you use API or Website URL your link will be displayed in an iFrame but not further previewing will be possible.

To continue with our Wisconsin Polling Places example, we’ll add one resource file to the Dataset we created in Step 1. Our resource file is a CSV, that is, comma-separated values format; this is a popular file format for exchanging tabular data. Let’s explore the example resource shown here and the various fields within:

Resource / Choose File:
 

upload a file from your local hard drive.

Resource / Recline Views:
 

DKAN’s “Data Preview” feature allows visitors to preview published data in three views:

  • Map - data with latitude and longitude columns or GeoJSON files can be previewed in a map interface
  • Graph - tabular (spreadsheet) data can be graphed by users, letting them create their own meaningful visualizations
  • Grid - by default, tabular data is presented in a spreadsheet view, with filter, sort, and search capabilities
Title:

this is the title of the individual data file, not the parent dataset container.

Description:

a rich-text editor field is provided so publishers can offer detailed and useful descriptions

Format:

entering the file format here will allow users the ability to search for data by specific format

Dataset:

this is the parent dataset container; this field should already be populated if you’re adding a Resource subsequent to adding a Dataset

At the bottom of the Add Resource page, we can choose:

Save:Save progress on this resource and immediately return to it for further editing
Save and add another:
 Save this resource and add another resource to the same dataset
Next: Additional Info:
 Save this resource and move to the third stage in adding a complete dataset, entering optional metadata about the dataset

In our example, we’re only adding a single resource, so we’ll click “Next: Additional Info” to move onto Step 3. If we had more than one resource to add to this dataset, we would choose the “Save and add another” option. Simply clicking “Save” would end the Dataset creation process and save the dataset, for now, with no additional metadata.

Step 3: Adding Metadata to a Dataset

We now come to a third form which allows us to add additional metadata to the dataset. All these fields are optional, but provide valuable information about your dataset to both human visitors to the website and machines discovering your dataset through one of DKAN’s public APIs.

Let’s take a closer look at some of the metadata fields available on this form:

Author:The Dataset’s author, in plain text.
Spatial / Geographical Coverage Area:
 Lets us define what region the data applies to. In this case, the US State of Wisconsin. You can use the map widget to draw an outline around the state borders, or, click the “Add data manually” button if you already have a GeoJSON string you can paste in.
Spatial / Geographical Coverage Location:
 The region the data applies to, written in plain text. This can be used instead of or in addition to the Coverage Area field.
Frequency:How often is this dataset updated? We might expect our list of polling places to be updated every year, so we could select “annually.” However, often we don’t expect the data to be updated (even in this case, perhaps we plan to post the next version of the data as a _separate_ dataset), in which case we can leave this blank.
Temporal Coverage:
 Like Geographic Coverage, this field lets us give some context to the data, but now for the relevant time period. Here we could enter the year or years for which our polling places data is accurate.
Granularity:This is a somewhat open-ended metadata field that lets you describe the granularity or accuracy of your data. For instance: “Year”. Note, this field is depreciated in DCAT and Project Open Data, and may be removed from DKAN.
Data Dictionary:
 This should be a URL to a resource that provides some sort of description that helps understanding the data. See Project Open Data data dictionary for more info.
Additional Info:
 Lets us arbitrarily define other metadata fields. See Additional Info field for more information.
Resources:This field is a reference to the resources you have already added.

After you click “Save”, the metadata we enter will appear on the page for this Dataset:

Configuration
Adding or Removing Allowed Resource File Types

Any type of file can be uploaded to Resources (though only CSV files can be imported to the Datastore.

File types are controlled at “/admin/structure/types/manage/resource/fields/field_upload”

To add or remove file types navigate as an ‘administrator’ and enter extensions into the “Allowed file extensions” field.

Advanced Metadata Features

A Dataset is a container for storing files, APIs, or other resources as well as the metadata about those resources. The metadata in a DKAN Dataset is structured specifically for describing Open Data.

The metadata in a DKAN Dataset is culled from the DCAT standard as well as Project Open Data. For more information on the default Dataset fields view the Open Data Field Comparison Tables.

The Dataset form allows users to create Datasets and add appropriate metadata:

_images/add-dataset-screen-1.png

The DKAN Dataset API exposes Dataset metadata for individual datasets as well an entire catalog.

_images/data-json.png
Custom metadata

It is easy to add new fields to DKAN which will show up on the Dataset form, make available as search facets, and be available to output in one of the Dataset APIs.

If there is information that only pertains to one or more datasets then it is possible to use the “Additional Info” field. This allows content editors to add unique field / value entries that exist only on a single dataset:

https://cloud.githubusercontent.com/assets/512243/4188796/57b53a52-3776-11e4-97f6-61e18e3cd90d.png

Globally-available custom fields can also be added through Drupal’s Fields UI and added to public APIs using the Open Data Schema Mapper.

Data Extent

The “Data Extent” block is a visual representation of the “Spatial / Geographical Coverage Area”.

_images/data-extent-block.png

The “Spatial / Geographical Coverage Area” field is a geojson representation of the area a Dataset covers. This can be a point, box, or other representation.

DKAN provides a widget so that a spatial area can be drawn if desired:

_images/spacial-geographical.png
Revision History

DKAN Datasets and Resources track revisions in order to log and display changes, using Drupal’s built-in revision system.

User Interface

Revision log entries can be added through the user interface by clicking “Revision information” in the dataset or resource edit form and can be viewed by clicking “Revisions” on a Dataset or Resource page:

_images/revision.png
Loading Revision information Programmatically

Revision comments generated in code can be viewed by loading a Dataset or Resource and viewing the log: $node = node_load(‘dataset node id’); echo $node->log

Revision List API

A list of recent revisions are available through the revision_list API at “/api/3/action/revision_list”

File Revisions

Copies are kept of files from previous revisions that can be compared manually by a user. Diffs of individual files are not available by default, but could be implemented with some custom code using Apache Solr and the Diff module, or a similar strategy.

Data Preview Features

Resources include powerful preview functionality via the Recline module. See Visualizations/Data Previews

Groups in DKAN

Groups allow you to group together datasets under an organization (i.e. Parks and Recreation Department, Department of Education) or category (e.g. Transport Data, Health Data) in order to make it easier for users to browse datasets by theme.

As a best practice, datasets and resources that are added to a Group should share a common publisher.

Essentially, Groups are both a way to collect common Datasets and enable an additional workflow on DKAN. On the outward-facing side, site visitors are able to browse and search Datasets published by a specific Group, which is the common publisher of a number of Datasets.

Behind the scenes, Groups add an additional set of roles and permissions that ensure quality and security when publishing data. Group roles and permissions ensure that Content Creators can add new data but only to their assigned Group. This is especially important for large sites that may have several working groups publishing data to the site. By adding users to the Group’s membership roster, you can create and manage a community based around the data within the group.

How to use Groups
Adding a new Group

When adding a new Group, the form has fields for basic information about the Group itself that should tell site visitors what to expect from the Datasets in the Group.

Title:Name your Group to reflect the agency or whoever the common data publisher is for the datasets that will belong to the Group.
Image:The image here acts like the logo for your Group. It appears on the overview Groups page as well as the individual page of the Group itself. It’s best to choose a square image to fit the dimensions of the thumbnail. Whether you choose an image, a logo, or an icon you can use any image that meets the size and file type requirements. As a Site Manager, you may want to add generic icons to the Groups you add if a current logo is unavailable.
Description:This text is the full description for your Group similar to an “About” page. The description includes details about the agency, its goals, and information about the data it publishes. While you want to include all the relevant information of the Group, the best descriptions are 1-2 paragraphs long and include a link to the agency’s main web page for more details.
Summary text:You can use the Summary to create unique text for your Group. This text appears as a snippet under the Group image on the Group overview page. If left blank the first portion of the body text will be used (about 100 words). Including a summary can be useful in adding more key search terms or using a different tone to intrigue site visitors to learn more.
_images/group.png
Managing Groups and Members

Once you’ve added a new Group, you can assign Datasets (and their Resources) to that Group. You can also manage the members of a Group, adding new members and giving certain members different roles. Members of a Group are bound by the permissions of their role and restricted to the content in their Group. As a Site Manager you have access to all Groups and are not limited by the permissions of the Group.

Roles and Permissions

With large sites there is often a need to have special permissions for a group of users to handle a specific set of content. Think of a large agency or department with sub-departments or programs that produce content. On the one hand these users shouldn’t have the ability to manage or edit content for the entire site or other Groups. On the other hand it would be impractical for Editors or Site Managers to handle content for a large number of users. To keep content organized and in the hands of its owners without introducing the risk of inadvertent (and sometimes irreversible) actions, Group-level permissions give users the ability to do things they couldn’t necessarily do on the site outside of the Group.

Within Groups there are different levels of access a user can have, which determines another level of permissions. Any user who belongs to a group falls into one of two types: Member or Administrator. Users not in the group are considered Nonmembers.

Non-Member:

A Non-Member is any user on the site who does not belong to the Group. This role can request membership in the Group and view Group members and content.

Member:

A Member is a basic user within the Group who is mostly adding and editing their own content for the Group. As Datasets are added they can be assigned to a Group. Members of a Group can add Datasets to their Group and edit those Datasets from the Group page. Content within a Group can only be edited by members of that Group, so it’s important to only associate Datasets with Groups that the user belongs to.

Administrator:

An Administrator of a Group plays a similar role to that of an Editor but for their particular Group rather than for the entire site. Conversely, Editors outside of a Group are not able to manage content published within a Group.

Administrators of Groups are able to add and remove Group members and manage (create/edit/delete) all content within the Group. It’s good practice to have only 1 or 2 users in this role for any given Group.

Adding users

Adding users to a Group is a straightforward process. Click on the “Group” tab on the group page and click “Add people”. Begin typing the username of an existing user into the “User name” field and select from the list of autocomplete options. A user must already have an account to be added to a Group, so if a person needs to be added you should first create an account for them with the appropriate role. By default a user will only have a Member role in the Group. To give the user an Administrator role and permissions, check the administrator member box.

Users may also request membership in a Group. If you or the Group Administrator directly add a user to the Group, then you don’t need to add any text in the Request message box. If the user requested membership, then their request message will appear here as part of the member profile (only visible to the Site Manager and Administrator).

People:

Site Managers and Administrators of Groups can edit the Group details as well as the members and their roles. On the People page under the Group tab, you can access the overview of Group members, edit or delete individual member profiles, and take bulk actions on a group of members.

Remove a user from the Group:
 

If a user is removed from the Group, they can no longer access Datasets added to that Group to edit them. All non-members can see Datasets that belong to a Group, but non-members can’t edit those Datasets.

You can remove a single user from the Group with the remove link in the far-right column next to the user’s profile details to access and change the details of their profile. Alternatively, you can remove a group of users from the Group with a bulk action by checking off multiple users and selecting the Remove from group option from the drop-down menu in the Operations box.

Block a user from the Group:
 

You can keep a user from joining a Group by blocking that user. If a user is blocked they won’t see the option to request subscription for the Group. In order to keep a user blocked, that user must technically be a member of the Group, so it’s different from removing a user entirely.

You can block a single user from the Group with the edit link in the far-right column next to the user’s profile details to access and change the details of their profile. Alternatively, you can block a group of users from the Group with a bulk action by checking off multiple users and selecting the Modify membership status option from the drop-down menu in the Operations box.

Change the member’s Group role:
 

In general, you shouldn’t need to change a user’s role often. Most users will be added as members and stay members. For any Group it’s best to limit Administrator roles to just one or two people.

If you need to change a user’s role you can use the edit link in the far-right column next to the user’s profile details to access and change the details of their profile. Alternatively, you can change a Group member’s role for a group of users in the Group with a bulk action by checking off multiple users and selecting the Modify OG member roles option from the drop-down menu in the Operations box.

_images/group-members.png
More on Group Membership

In DKAN, “subscribing” to a Group is synonymous with becoming a member of the Group. Nonmembers can submit a membership request to the Group Administrator to become a Group member. Members have privileges to access and edit Datasets associated to the Group, so membership requires moderation. As a Site Manager, you can join any Group without moderation. Any other role must first submit a request to subscribe and then be approved by the Group Administrator.

Active members, both Members and Administrators, can leave the Group by clicking the Unsubscribe From Group link on the Group home page. Once members are removed or leave the Group they no longer have access to the Datasets associated with the Group.

As users author Datasets, it’s important that they associate Datasets only with Groups that they belong to. Once a Dataset is associated with a Group, that Dataset can only be edited by a person in the Group, either the Administrator or a member who authored the Dataset. So if a nonmember authors a Dataset and then associates it to a Group, the author won’t be able to access and edit that Dataset any longer. The Group Administrator can either remove the Dataset from the Group or add the nonmember to the Group as a Member.

DKAN Datastore

DKAN Datastore bundles a number of modules and configuration to allow users to upload CSV files, parse them and save them into the native database as flat tables, allowing users to query them through a public API.

Drupal Architecture

The DKAN Datastore’s importer is a wrapper around the Feeds module. The custom Feeds Flatstore Processor and Feeds Field Fetcher plugins were created the file uploaded to the resource form a feed item.

The Data module is used to manage datastore tables’ schema.

The Datastore API uses the Services module to provide an endpoint, although nearly all the underlying functionality is overridden and provided directly by the DKAN Datastore API module.

Getting Started

When you create a dataset with resources, you have data in DKAN which you can display and store in several ways. However, DKAN is still reading this data directly from the file or API you added as a resource.

To get the fullest functionality possible out of your datasets, you should add your CSV resources to the datastore.

If you are exploring a resource that is not yet in the datastore, you will see a message advising you of this.

_images/datastore-message.png

Click the “Manage Datastore” button at the top of the screen. On the “Manage Datastore” page, confirm that the delimiter and file encoding options are correct, then use the “Import” button at the bottom of the page to import the data from your file or API into DKAN’s local datastore.

_images/datastore-resource.png

Your data is now ready to use via the API! Click the “Data API” button at the top of the resource screen for specific instructions.

Processing Options

By default Resource files are added to the DKAN Datastore manually. This can be changed to:

  • Import upon form submission
  • Import in the background
  • Import periodically
Changing Default Datastore Import Behavior

Default behavior for linked and uploaded files is controlled through the Feeds module. To access the Feeds administrative interface, enable the Feeds Admin UI module (which is included but not enabled by default in DKAN). Once turned on you can access the Feeds UI at /admin/structure/feeds. You should see two Feeds Importers by default:

_images/datastore-feeds-importers.png
Import on submission

To import a Resource file upon saving the resource, click Import on submission in the settings section for each importer:

_images/datastore-import-submission.png

This is not recommended for large imports as a batch screen will be triggered that will not stop until the entire file is imported.

Process in background

This setting means that once an import has started, it will be processed in 50 row increments in the background. Processing will occur during cron. The queue of imports is managed by the Job Schedule module. Each cron run will process a maximum of 200 jobs in a maximum of 30 seconds. Note that an import won’t be started by saving the Resource form. This will only be triggered by clicking “Import” on the “Manage Datastore” page or if triggered programatically. This setting can be used in addition to “Import on submission” option to start imports that will be imported in the background.

Periodic import

Importing items on a periodic basis makes the most sense if you have a file you are linking to that you want to periodically re-import. This setting requires that cron is running on a regular schedule.

Geocoder

DKAN’s native Datastore can use the Drupal Geocoder module to add latitude/longitude coordinates to resources that have plain-text address information. This means that datasets containing plain-text addresses can be viewed on a map using the Data Preview or other map-based data visualizations. It is not included by default with DKAN but can be downloaded here.

Instructions
  1. Install and enabling the geocoder module.
  2. Click the Manage Datastore tab on any resource with address information.
  3. Check the “Geolocate” box.
  4. Select the Geolocation Service you will be using.
  5. In the Geolocate Addressses field enter the field or fields from the file that make up the address to geolocate.
  6. Click the Import button
_images/datastore-geolocate.png
Geolocation Services

Geolocation services offered are

Note that Nominatim is a driven by Open Street Map data, which is the most open of the options offered.

Geolocation Limits

The number of rows that can be geolocated is determined by the service you select. Google, for example, allows you to geolocate up to 2500 times per day before paying.

Adding Service API Keys

The Geocoder module supports adding API keys for the Yahoo and Google services. Users can sign up for those services and, in Google’s case, geocode up to 100,000 addressees per day.

Managing datastores with Drush

To create a datastore from a local file:

drush dsc (path-to-local-file)

To update a datastore from a local file:

drush dsu (datastore-id) (path-to-local-file)

To delete a datastore file (imported items will be deleted as well):

drush dsfd (datastore-id)

To get the URI of the datastore file:

drush dsfuri (datastore-id)

Using the Fast Import Option

DKAN Datastore’s “fast import” allows for importing huge CSV files into the datastore at a fraction of the time it would take using the regular import.

When a CSV is imported using the regular import, this is what it happens under the hood:

  1. PHP interpreter reads the file line-by-line from the disk
  2. Each time a line is parsed it sends a query to the database
  3. The database receives the query and parses it
  4. The database creates a query execution plan
  5. The database excecutes the plan (i.e., inserts a new row)

Note

Steps 3, 4 and 5 are executed for each row in the CSV.

The Datastore Fast Import was designed to remove as many steps as possible from the previous list. It performs the following steps:

  1. PHP interpreter sends a LOAD DATA query to the database
  2. The database receive the query and parses it
  3. The database reads and imports the whole file into a table

Only one query is executed, so the amount of time required to import a big dataset is drastically reduced. On a multi-megabyte file, this could mean the difference between an import time of hours to minutes.

Requirements
  • A MySQL / MariaDB database
  • MySQL database should support PDO::MYSQL_ATTR_LOCAL_INFILE and PDO::MYSQL_ATTR_USE_BUFFERED_QUERY flags.
  • Cronjob or similar to execute periodic imports.
  • Drush

Note

Because of the above requirements, which may not be available on all hosting environments, this module is disabled by default in DKAN.

Installation
  • Inside your settings.php add a pdo element to your database configuration. For example:

    <?php
    $databases['default']['default'] = array (
      'database' => 'drupal',
      'username' => 'drupal',
      'password' => '123',
      'host' => '172.17.0.11',
      'port' => '',
      'driver' => 'mysql',
      'prefix' => '',
      'pdo' => array(
         PDO::MYSQL_ATTR_LOCAL_INFILE => 1,
         PDO::MYSQL_ATTR_USE_BUFFERED_QUERY => 1,
       )
    );
    
  • Go to /admin/modules, turn on DKAN Datastore Fast Import and press Save configuration. Alternatively you can use drush to enable this module: drush en dkan_datastore_fast_import.

  • Make sure you do not see this message at the top of the page:

    Required PDO flags for dkan_datastore_fast_import were not found. This module requires PDO::MYSQL_ATTR_LOCAL_INFILE and PDO::MYSQL_ATTR_USE_BUFFERED_QUERY
    
  • Set up the following command to run periodically using a cronjob or similar: drush queue-run dkan_datastore_queue

Configuration

To configure how Fast Import behaves go to admin/dkan/datastore.

There are 3 basic configurations that control the Fast Import functionality:

Use regular import as default:
 Use Fast Import checkbox is uncheked by default on the resource’s datastore import form so files are imported using the normal dkan datastore import. However you can still enable fast import for any resource by clicking that checkbox.
Use fast import as default:
 Use Fast Import checkbox is cheked by default so files are imported using DKAN Fast Import. Like the previous setting, you can uncheck Use Fast Import on the resource-specific datastore import form to use the normal import instead.
Use fast import for files with a weight over:
 From this setting you obtain a refined control about when Use Fast Import should be checked. This option reveals an additional setting: “File size threshold.” “Use Fast Import” will be checked on the datastore import form for all the files over this size threshold. A size expressed as a number of bytes with optional SI or IEC binary unit prefix (e.g. 2, 3K, 5MB, 10G, 6GiB, 8 bytes, 9mbytes)

Either of the two “Use fast import” options will also reveal the following additional settings:

Load Data Statement:
 Some hostings doesn’t support LOAD DATA LOCAL INFILE. If that’s your case you can switch to LOAD DATA INFILE.
Queue Filesize Threshold:
 If a file is small enough, you can avoid waiting until the drush queue runs by configuring this threshold. Files with a size under this value won’t be queued and will rather imported during the request. The time to perform the import should fit into the php request timeout, or your import could be aborted.
Usage

To import a resource using Fast Import:

  • Create a resource using a CSV file (node/add/resource) or edit an existing one.
  • Click on Manage Datastore
  • Make sure the status says No imported items (You can use the Drop Datastore link if needed).
  • Check Use Fast Import checkbox
  • Press import
  • If you get an error like SQLSTATE[28000]: invalid authorization specification: 1045 access denied for user 'drupal'@'%' (using password: yes) you will need to grant FILE permissions to your MYSQL user. To do so use this command: GRANT FILE ON *.* TO 'user-name'

Note

If you are using the docker-based development environment described in the DKAN Starter documentation, you will need to execute the following commands (take note that admin123 is the password of the admin user in that mysql environment):

ahoy docker exec db bash
mysql -u root -padmin123
GRANT FILE ON *.* TO 'drupal';

When the option “Use Fast Import” is checked, some other options become visible that affect how MySQL will parse your file:

  • Quote delimiters: the character that encloses the fields in your CSV file.
  • Lines terminated by: the character that works as line terminator in your CSV file.
  • Fields escaped by: the character used to escape other characters in your CSV file.

Also, you can choose if the empty cells will be read as NULL or zeros by checking the box for “Read empty cells as NULL”.

Datastore API

Once processed, Datastore information is available via the Datastore API. For more information, see the Datastore API page.

DKAN Harvest

DKAN Harvest is a module that provides a common harvesting framework for DKAN. It supports custom extensions and adds drush commands and a web UI to manage harvesting sources and jobs. To “harvest” data is to use the public feed or API of another data portal to import items from that portal’s catalog into your own. For example, Data.gov harvests all of its datasets from the data.json files of hundreds of U.S. federal, state and local data portals.

DKAN Harvest is built on top of the widely-used Migrate framework for Drupal. It follows a two-step process to import datasets:

  1. Process a source URI and save resulting data locally to disk as JSON
  2. Perform migrations into DKAN with the locally cached JSON files, using mappings provided by the DKAN Migrate Base module.

Harvest Sources

Harvest Sources are nodes that store the source’s URI and some additional configuration. Users with the administrator or site manager role will be able to create and manage harvest sources.

Create a new harvest source
  1. Go to node/add/harvest-source or Content > Add content > Harvest Source, and fill out the form.
Title:Administrative title for the source.
Description:Administrative description or notes about the source.
Source URI:The full Uniform Resource Identifier, such as http://demo.getdkan.com/data.json.
Type:data.json or data.xml - If the harvest source type you are looking for is not available, please refer to the Define a new Harvest Source Type section below.

The form includes four multi-value fields to help you fine tune the results of your harvest.

Filters:Filters restrict the datasets imported by a particular field. For instance, if you are harvesting a data.json source and want only to harvest health-related datasets, you might add a filter with “keyword” in the first text box, and “heatlh” in the second.
Excludes:Excludes are the inverse of filters. For example, if you know there is one publisher listed on the source whose datasets you do not want to bring into your data portal, you might add “publisher” with value “Governor’s Office of Terrible Data”
Overrides:Overrides will replace values from the source when you harvest. For instance, if you want to take responsibility for the datasets once harvested and add your agency’s name as the publisher, you might add “publisher” with your agency’s name as the value.
Defaults:Defaults work the same as overrides, but will only be used if the relevant field is empty in the source

Project Open Data (as well as most metadata APIs) includes many fields that are not simple key-value pairs. If you need to access or modify nested array values you can use this dot syntax to specify the path: key.nested_key.0.other_nested_key. For example, the Publisher field in Project Open Data is expressed like this:

"publisher": {
  "@type": "org:Organization",
  "name": "demo.getdkan.com"
},

To access the name property for filtering or overriding, you can set publisher.name in the first text box and the value you want to use in the second one.

_images/harvest-filters.png
  1. Click Save, the datasets from the source will be cached, and you will see a preview of what will be imported. This preview page shows a list of dataset titles and identifiers now in the harvest cache, allowing you to perform a basic check on your source configuration. If it does not look right, click the Edit tab to make adjustments.
  2. Click Harvest Now. The datasets that were cached will now be imported into your site.
_images/harvest-search.png

Harvest Source nodes are viewable by the public and provide some basic metadata to the user.

Managing Harvest Sources

From the admin menu, navigate to DKAN > DKAN Harvest Dashboard to view harvest sources. The DKAN Harvest Dashboard provides site managers a quick overview of harvest sources, when they were last updated, number of datasets, and status of the source. The dashboard also allows site managers to perform manual harvest operations:

Harvest (cache and migrate):
 This will cache the source data locally and migrate that source data into your site content.
Cache source(s):
 This will fetch the source data, apply the source configuration (filters, excludes, etc.) and cache the data locally without migrating. You may wish to do this to check for errors, or to refresh the preview.
Migrate source(s):
 This will migrate the current cache for the selected sources, no matter how old it is.
_images/harvest-dashboard.png

Click on the title of a harvest source from the dashboard to see the details of that source. Administrative tasks can be accomplished from the tabs across the top.

_images/harvest-source-node.png
View:View the harvest source node.
Edit:Click to make changes to the configuration of the harvest source.
Preview:Click to pull the latest data from the source endpoint into the cache.
Manage Datasets:
 An administrative view that lets you sort and filter the datasets from this source. The most powerful function on this page is to filter by orphan status. When a dataset that was harvested into your system previously is no longer provided in the source, it is considered “orphaned” on your site and unpublished. From the Manage Datasets screen, you can either permanently delete or re-publish orphan datasets.
Events:Event Log that provides historical data on all harvests run on this source. The information is managed by the core dkan_harvest via a per-harvest source migrate_log table that tracks the number of datasets created, updated, failed, orphaned, and unchanged and status. If the value for the field Status is Error then you can click on the text to see the log error and identify the problem.
Errors:Error log that shows a list of all errors recorded during harvesting on the source.

Harvest Drush Commands

DKAN Harvest provides multiple drush commands to manage harvest sources and control harvest jobs. In fact, once your sources are properly configured, running harvests from Drush on a cron job or other scheduling system like Jenkins is highly reccomended.

It is recommanded to pass the --user=1 drush option to harvest operation (especially harvest migration jobs) to make sure that the entities created have a proper user as author.

List Harvest sources available
# List all available Harvest Sources
$ drush --user=1 dkan-harvest-status
# Alias
$ drush --user=1 dkan-hs
Run a full harvest (Cache & Migration)
# Harvest data and run migration on all the harvest sources available.
$ drush --user=1 dkan-harvest
# Alias
$ drush --user=1 dkan-h

# Harvest specific  harvest source.
$ drush --user=1 dkan-harvest test_harvest_source
# Alias
$ drush --user=1 dkan-h test_harvest_source
Run a harvest cache
# Run a harvest cache operation on all the harvest sources available.
$ drush --user=1 dkan-harvest-cache
# Alias
$ drush --user=1 dkan-hc

# Harvest cache specific harvest source.
$ drush --user=1 dkan-harvest-cache test_harvest_source
# Alias
$ drush --user=1 dkan-hc test_harvest_source
Run a harvest migration job
# Run a harvest migrate operation on all the harvest sources available.
$ drush --user=1 dkan-harvest-migrate
# Alias
$ drush --user=1 dkan-hm

# Harvest migrate specific harvest source.
$ drush --user=1 dkan-harvest-migrate test_harvest_source
# Alias
$ drush --user=1 dkan-hm test_harvest_source
Extending DKAN Harvest

DKAN developers can use the api provided by DKAN Harvest to add support for additioanl harvest source types. The dkan_harvest_datajson module encapsulate the reference implementation providing support for POD type sources.

If you need to harvest from an end point type other then POD. You can extend the DKAN Harvest APIs to implement said support by following a simple checklist:

  • Define a new Harvest Source Type via hook_harvest_source_types.
  • Implement the Harvest Source Type cache callback.
  • Implement the Harvest Source Type Migration Class.
  • (Optional) Write tests for your source type implementation.
Define a new Harvest Source Type

DKAN Harvest leverages Drupal’s hook system to provide a way to extend the Source types that DKAN Harvest supports. To add a new harvest source type the we return their definitions as array items via the hook_harvest_source_types() hook.

/**
 * Implements hook_harvest_source_types().
 */
function dkan_harvest_test_harvest_source_types() {
  return array(
    'harvest_test_type' => array(
      'machine_name' => 'harvest_test_type',
      'label' => 'Dkan Harvest Test Type',
      'cache callback' => 'dkan_harvest_cache_default',
      'migration class' => 'HarvestMigration',
    ),

    // Define another harvest source type.
    'harvest_another_test_type' => array(
      'machine_name' => 'harvest_another_test_type',
      'label' => 'Dkan Harvest Another Test Type',
      'cache callback' => 'dkan_harvest_cache_default',
      'migration class' => 'HarvestMigration',
    ),
  );
}

Each array item defines a single harvest source type. Each harvest source item consists of an array with 4 keyed values:

machine_name:Unique string identifying the harvest source type.
label:This label wil be used on the harvest add node form.
cache callback:Cache function to perform; takes HarvestSource object and timestamp as arguments) and returns a HarvestCache object
migration class:
 A registered Migrate class to use for this source type
Cache callbacks
/**
 * @param HarvestSource $source
 * @param $harvest_updatetime
 *
 * @return HarvestCache
 */
function dkan_harvest_datajson_cache(HarvestSource $source, $harvest_updatetime)

This callback takes care of downloading/filtering/altering the data from the source end-point to the local file directory provided by the HarvestSource::getCacheDir() method. The recommended folder structure for cached data is to have one dataset per uniqely named file. The actual migration is then performed on the cached data, not on the remote source itself.

sh
$ tree
.
├── 5251bc60-02e2-4023-a3fb-03760551ab4a
├── 80756f84-894f-4796-bb52-33dd0a54164e
├── 846158bd-1821-48d8-80c8-bb23a98294a9
└── 84cada83-2382-4ba2-b9be-97634b422a07

0 directories, 4 files

$ cat 84cada83-2382-4ba2-b9be-97634b422a07
/* JSON content of the cached dataset data */

The harvest cache function needs to support the modifications to the source available from the harvest source via the Filter, Excludes, Overrides and Default fields. Each of these configurations is available from the HarvestSource object via the HarvestSource::filters, HarvestSource::excludes, HarvestSource::overrides, HarvestSource::defaults methods.

Migration Classes

The common harvest migration logic is encapsulated in the HarvestMigration class, (which extends the MigrateDKAN class provided via the DKAN Migrate Base module. DKAN Harvest will support only migration classes extended from HarvestMigration. This class is responsible for consuming the downloaded data during the harvest cache step to create the DKAN dataset and associated nodes.

Implementing a Harvest Source Type Migration class is the matter of checking couple of boxes:

  • Wire the cached files on the HarvestMigration::__construct() method.
  • Override the fields mapping on the HarvestMigration::setFieldMappings() method.
  • Add alternate logic for existing default DKAN fields or extra logic for custom fields on the HarvestMigration::prepareRow() and the HarvestMigration::prepare().

Working on the Migration Class for Harvest Source Type should be straitforward, but a good knowladge on how migrate works is a big help.

HarvestMigration::__construct()

Setting the HarvestMigrateSourceList is the only logic required during the construction of the extended HarvestMigration. During the harvest migration we can’t reliably determin and parse the type of cache file (JSON, XML, etc..) so we still need to provide this information to the Migration class via the MigrateItem variable. the Migrate module provide different helpful class for different input file parsing (MigrateItemFile, MigrateItemJSON, MigrateItemXML). For the the POD dkan_harvest_datajson reference implementation we use the MigrateItemJSON class to read the JSON files downloaded from data.json end-points.

public function __construct($arguments) {
  parent::__construct($arguments);
  $this->itemUrl = drupal_realpath($this->dkanHarvestSource->getCacheDir()) .
    '/:id';

  $this->source = new HarvestMigrateSourceList(
    new HarvestList($this->dkanHarvestSource->getCacheDir()),
    new MigrateItemJSON($this->itemUrl),
    array(),
    $this->sourceListOptions
  );
}
HarvestMigration::setFieldMappings()

The default Mapping for all the default DKAN fields and properties is done on the HarvestMigration::setFieldMapping() method. Overriding one or many field mapping is done by overrrding the setFieldMapping() in the child class and add/update the new/changed fields.

For example to override the mapping for the og_group_ref field.

public function setFieldMappings() {
  parent::setFieldMappings();
  $this->addFieldMapping('og_group_ref', 'group_id');
Resources import

The base HarvestMigration class will (by default) look for a $row->resources objects array that should contain all the data needed for constructing the resource node(s) associated with the dataset. the helper method HarvestMigration::prepareResourceHelper() should make creating the resources array items more streamlined.

Example code snippet:

/**
 * Implements prepareRow.
 */
public function prepareRow($row) {
  // Redacted code

  $row->resources = $this->prepareRowResources($row->xml);

  // Redacted code
}
Harvest and DKAN Workflow support

By default, DKAN Harvest will make sure that the harvested dataset node will be set to the published moderation state if the DKAN Workflow module is enabled on the DKAN site. This can be changed at the fields mapping level by overriding the workbench_moderation_state_new field.

DKAN Workflow

DKAN Workflow is a Workflow implementation for DKAN based on the Workbench family of modules.

The goal of this component is help various organizations adhere to an editorial workflow for metadata publishing by providing:

  • Content state tracking and revisioning
  • State oriented management UI
  • Access control
_images/dkan_workflow_screenshot.png

DKAN Workflow main administration interface.

Requirements

The DKAN workflow component comes in the form of three modules:

  • DKAN Workflow
  • DKAN Workflow Permissions
  • Views Workflow List

In addition to these core modules, DKAN Workflow depends on multiple Drupal contrib modules

Outside of the direct Workbench add-ons, DKAN Workflow needs additional Drupal contrib modules to provide extra functionality (Menu and link badges, etc).

All those dependencies are declared in the drupal-org.make file.

Installation

DKAN workflow is included in the core DKAN install but it’s not enabled by default. It can be enabled either from the Modules management page or by using drush.

drush en dkan_workflow -y

Enabling DKAN workflow should enable all the dependencies modules and update the user roles (more information available in the Workflow Roles section).

You may see a message instructing you to rebuild permissions, click the “Rebuild permissions” link to update the node access settings.

_images/rebuild-permissions-message.png

Metadata Moderation States

There are three default moderations states available by default in DKAN:

Draft:This is the starter state that the metadata (be it dataset or resource) is in when first created by the “Workflow Contributor” ( defined in the Workflow Roles section). The node can be updated and have multiple iteration (or revision in the Drupal jargon) without the need to change the state. After the author evaluate the content is ready for being reviewed. The node moderation state can be set to “Needs Review”.
Needs Review:When the content author consider the work to be good enough to be reviewed by a Moderator, the node(s) can be set to the Needs Review. This will signal to available “Workflow Moderator” users that the data is ready to be looked at by peers (more information in the Workflow Roles section).
Published:When the content is judged being ready for public consumption. The qualified moderator (Take a look at the Workflow Roles section) can set it to the Published state. This will make the current revision of the metadata to be accessible by all the site visitor and the dataset/resources will be added to the search index.

Content Moderation UI

Controlling the moderation state of the various core content types provided by DKAN can be done from various places.

My Workbench

The main moderation interface is available from the My Workbench link from the navigation bar, or accessible directly via admin/workbench.

_images/dkan_workflow_main_interface.png
  1. Moderation Tabs.
    My content:This tab is the only tab without the moderation table and provides quick links to content creation forms.
    My drafts:This will display the draft content authored by the logged in user.
    Needs review:This will display the content with the moderation state set to Needs Review depending on the Workflow role of the current user (This behavior is detailed in the Workflow Roles section).
    Stale drafts:This moderation tab is equivalent to My drafts tabs except that it holds all the draft content that was not updated in the last 72 hours. This tab is only accessible by Workflow Supervisor (see Workflow Roles).
    Stale reviews:This moderation tab is equivalent to Needs review tabs except that it holds all the Needs Review content that was not updated in the last 48 hours. This tab is only accessible by Workflow Supervisor (see Workflow Roles).
  2. Content Filters. Users can filter through the moderated content by Title, Type (Dataset, Resource, Data Story, etc), and Groups.
  3. Bulk updates. Certain operations like publishing or rejection can be applied to all or a selected subset of the content available on the moderation tab.
  4. Moderated content Table. The table will list all the moderated content relevant to the tab currently selected. Supports displaying dataset without resource or with all it’s resources published (5), moderated dataset with moderated child resource (6), and even child moderated resource(s) with published parent dataset (7).
Node Edit Page

Changing the moderation state for individual nodes (be it a dataset or a resource) is available via the node edit form at the bottom of the edit page under the Publishing options sidebar. Authors and reviewers can change the moderation state and add a note about the change via the Moderation notes text area.

_images/workflow_node_edit.png

Workflow Roles

DKAN workflow permissions provides 3 Drupal roles:

Workflow Contributor:
 This is the lowest level role desgined with “Content Creator” users in mind, with access only to the workflow menu and limited set of admininstration pages. The only transitions granted for this role is from “Draft” to “Needs Review” and the opposite way from “Needs Review” to “Draft”. The only tabs available for the “Workflow Contributor” role are the “My Draft” tab and “Needs Review tab”. Accros all the tabs, a user with this role have access only to the content that was authored by him/her.
Workflow Moderator:
 This is a more advanced role desgined for “Editor” role. In addition of all the capabilities of the “Workflow Contributor” role, A “Workflow Moderator” can move content from “Needs review” to “Published”. “Workflow Moderator” users have access to all the content that is associated to the same Groups that they belong to (checkout Organic Groups integration for more information).
Workflow Supervisor:
 This is the role associated with “Site Manager” users. In addition to being able to view and act upon all the content available on all the tabs (more information available in the Organic Groups integration), this role is the only role that have access to the “Stale Drafts” and “Stale Review” tabs.
Automatic User Role Assignment

Users with only workflow roles won’t be able to do much in DKAN and need to be associated to its equivalent core role. The Roles form on the User edit page supports adding the suited core role when only a Workflow role is checked.

_images/dkan_workflow_autorole.gif

Automatic core role assignment with workflow roles.

Organic Groups integration
Content viewing
What a user will see My drafts Needs review
Workflow Contributor
  • Only content that they submitted.
  • Can see only content they have submitted.
Workflow Moderator
  • The content submitted to their organic group.
  • Their own content.
  • The content submitted to their organic group.
  • Thier own content.
Workflow Supervisor
  • Only content that they submitted.
  • All the “Needs review” content.
Emails

For each state transition (for example from Draft to Needs Review, from Needs Review to Draft, etc) a set of users with workflow roles will be notified by an email notification. The users will be selected following those rules:

  1. Email original content author.
  2. Email “Workflow Moderators” that are members of a group that the content have been associated to.
  3. Email all “Workflow Supervisors”.

Emails will have the context triggering the notification with links to the updated content.

Extending DKAN Workflow

Tweaking the Email template

Changing the email template being sent when a moderation operation is applied can be done via the admin/config/workbench/email configuration page. For more in-depth documentation please Review the Workbench Modules Docs.

Workbench Modules Docs

For more advanced edge case writing custom code may be needed. For more information please refer to the workflow modules documentation.

DKAN Topics

While DKAN includes a free-tagging tags/keywords field for datasets, many data portals organize datasets into more predefined categories by subject matter. These are usually a small collection of subjects with logos that are incorporated into the site navigation. Neither DKAN’s tags or “groups” (which are designed for grouping user permissions and usually represent organizational divisions) are exactly appropriate for this task.

The DKAN Topics module adds a “topics” vocabulary to DKAN, and corresponding functionality throughout the site. It adds a facet to the search/datasets page, and a pane to the default homepage. Topics can be administered through the standard Drupal taxonomy interface.

The included DKAN Default Topics module will add, on enable, a set of default civic topics using the Taxonomy Fixtures module.

DKAN Topics is enabled by default on new DKAN installations, with default terms loaded into the vocabulary. The module can be disabled and uninstalled, and all existing topics will be removed.

Permissions

  • Users with the Site Manager or Editor role can add and edit topic terms.
  • Users with the Administrator role can add new icons.

Adding a new topic term

From the Administration menu, navigate to Site Configuration > Taxonomy > Topics > Add term

Name:Enter the term for your new topic.
Description:This field is not currently displayed publicly.
Icon Type:DKAN Topics comes with a default set of font icons that can be used with your terms, you can upload your own font icons if desired. See Adding new icons. Or you may select to use image icons, when you toggle the image option, an image upload input field will appear.
Icon:If using font icons, select the icon you want to associate with your topic term.
Icon Color:Icons will display the same color as text on datasets unless a specific color is selected here.

Editing topic terms

  1. From the Administration menu, navigate to Site Configuration > Taxonomy > Topics
  2. You will see a list of current topic terms, click the ‘edit’ link under Operations that corresponds to the term you would like to edit.
  3. Make changes and click “Save”.

Removing Topics from the main menu

  1. Navigate to Site Configuration > Menus
  2. On the Menus screen, click “list links”
  3. Uncheck the box under “Enabled” for Topics
  4. Click “Save configuration”

Adding new icons

The font used for Topics can only be changed if there are NO default icon values in use, only one icon font can be used at a time.

  1. Navigate to Configuration > Content Authoring > Font Icon Select Options
  2. Click “Upload New Library”
  3. Enter a title for your new font option
  4. Upload the four files for your font
  5. Click “Save”
  6. Navigate to Structure > Taxonomy > Topics > Manage Fields > Icon
  7. Select your font from the font dropdown in the Icon field settings section.
  8. Click “Save settings”

Known Issues

  • This module adds a main menu link for “Topics”. If you want a different word in place of “Topics”, you can change the name in the main menu configuration but the icons in the dropdown will stop working. If you use String Overrrides you can change the Menu link title and the icons will continue to work, however the facet block title and the dataset form field title will still display as ‘Topics’.
  • Adding a new icon font for use with topics needs work to keep the icon functionality in facets and menus from breaking.

Fixtures and Default Content

Fixtures are a programming concept for default data that is included with application code for testing or other purposes. The data is provided in a structured format like XML or JSON, and imported into the database as part of a build process.

In DKAN, fixtures are used to provide datasets and other supporting content out of the box. The most visible use case for this will be DKAN’s default content, which showcases DKAN’s various capabilities. The fixtures themselves for default content are packaged in a separate sub-module, DKAN Default Content.

The DKAN Fixtures module provides tools to easily export all the content that lives inside a DKAN site into JSON fixture files, following a defined schema. Currently the content supported by the module are Groups, Resources, Datasets, Data Dashboards, Data Stories and Pages. Visualization Entites are also supported.

The module also provides basic Migrate classes that can be used to import content easily on a DKAN site.

Default Content Module

DKAN Default Content is the module that holds all the default content delivered with DKAN. All content is imported through the fixtures that can be found inside the /data directory. DKAN Fixtures was used to generate the default content fixtures and to migrate all the data using the migration clases that are provided.

Updating the fixtures

The default content fixtures can be updated easily through the DKAN Default Content module using the following steps:

  1. All the content present on the DB is going to be exported, so be sure to clean all the content first.
  2. Run drush dkan-save-data
  3. All default content fixtures should be updated and saved inside the ‘data’ directory in the dkan_default_content module.

Please note that some rules should be followed when preparing the default content:

  • Only published datasets are going to be exported.
  • Only resources associated with datasets are going to be exported.
  • Only groups that have associated datasets are going to be exported.
  • The size of of the dkan_default_content module should be kept as small as possible, so small files and images should be used.
  • When using internal visualizations as visualization embeds be sure to use the ‘Local’ option on the visualization embed settings, so that embeds are not pointing to a specific domain.
Importing default content

All the default content is imported automatically as soon as the DKAN Default Content module is enabled. Enable the DKAN Default Content module in the browser via admin/modules, or on the command line via drush:

drush en dkan_default_content

Removing the default content

All the default content is automatically removed as soon as the DKAN Default Content module is disabled with the exception of pages (Homepage, About page, etc). Disable the DKAN Default Content module in the browser via admin/modules, or on the command line via drush:

drush dis dkan_default_content

Upgrading pages from ctools panel pages to page nodes

Starting with DKAN 7.x-1.13, the default homepage has been converted from a Panel Page into a common page node. Page nodes now have panelizer enabled, so the full layout of a panel page can be reproduced in a simple page node. DKAN now provides a function that can be used to perform this conversion automatically. The provided function will:

  • Generate an exact copy of the specified panel page automatically.
  • Generate a new panelized page node and disable (not delete) the old ctools panel page homepage
  • The new node page will be set as the site homepage if the $is_homepage parameter is set as ‘true’.

Please note that some CSS adjustments might be needed in order for the node page to look exactly like the panel page, as CSS IDs and classes might be different.

The function can be found in the dkan_sitewide module and can be used as follows:

drush php-eval "dkan_sitewide_convert_panel_page(<page-name>);"

About paths

Pathauto is disabled for content created using dkan_fixtures because performance reasons. Instead, paths should be added to the fixtures using the path key.

Taxonomy Fixtures

A similar system exists for importing and exporting taxonomy terms as default content. The NuCivic-created Taxonomy Fixtures module ships with DKAN but is available for us in other Drupal projects.

Federal Extras

The Open Data Federal Extras Module module extends DKAN Dataset to include selected Project Open Data and other federal requirements. See the Project Open Data Schema for more information (this module essentially adds the fields marked “USG”). It includes a list of U.S. federal bureau and program codes, with a script to keep program codes up-to-date.

Additional Fields

  • Bureau Code
  • Program Code
  • Data Quality
  • primary IT Investment UII
  • System of Records

Enabling the module will add these fields to your Dataset content type. Note that disabling the module will not remove them. To remove the fields completely (which will permanently delete all data in those fields), uninstall the module from the module administration screen or via drush pm-uninstall.

Updating the Program Code list
  1. Go to http://project-open-data.github.io/schema/#programCode
  2. Download “Federal Program Inventory”
  3. Export in csv to fed_program_code_list
  4. cd ‘fed_program_code_list’
  5. Make sure filenames at beginning of list_to_array.php are correct.
  6. Run php -r " require 'list_to_array.php'; opfe_list_to_array_process();
The Bureau Code List

The Bureau Code list is built from a CSV downloaded from seanherron/OMB-Agency-Bureau-and-Treasury-Codes. This repository has not been updated recently and new sources for this data are currently being considered.

Theme

DKAN takes advantage of Drupal’s well-developed and flexible theming system, allowing administrators complete graphical control over the entire DKAN site. Features such as responsive page templates, accessible design elements, and built-in media management provide the latest in design and presentation technologies.

Default DKAN Theme

In Drupal and DKAN, the collection of page templates, fonts, colors, images, and other “look and feel” elements are known as a “theme.” NuBoot Radix is the default theme for DKAN and is a subtheme of Radix which uses bootstrap styles and is compatible with panelized sites. This theme has some basic customization features built in, for many client sites, these configurations will be all that is necessary to meet the client expectations. The configurable items are:

  • Logo (svg logos will display better on retina displays)
  • Front page Hero image OR you can set a solid color background for the hero region.
  • Favicon.
  • Copyright text.
  • Color options via the Colorizer screen
  • Fonts: use font-your-face to use alternate fonts

DKAN administrators have the choice of customizing the existing DKAN Theme through theme settings, implementing an entirely new theme, or creating a subtheme of nuboot_radix.

By default, the DKAN Theme is located in: [SITEROOT]/themes/nuboot_radix

Theme and Appearance Settings

The DKAN Theme does provide a few simple customizations that administrators can use to change the default appearance of the site from the theme settings screen. If logged in as an administrative user, navigate to Appearance >> Settings

Site name and slogan

From the settings screen, you can toggle on/off the site name and slogan, simply check the box next to the elements you want to use.

_images/Appearance_DKAN.png
Shortcut icon

If you would like to use a different favicon image, uncheck the ‘Use the default’ checkbox, and upload your own.

Hero background image/color

The Hero Unit is the background image that displays on the front page. To use a different photo than the one supplied, click the ‘Choose file’ button to upload a new image. This image will expand to cover the full width of the browser so be sure to upload a horizontal/landscape image and not a vertical/portrait image.

_images/Appearance_DKAN_3.png
Color scheme

To use the colorizer option, you must use the default theme as the admin theme. Navigate to [SITEROOT]/admin/appearance and scroll to the bottom. Confirm that the Admin theme is set to ‘Default Theme’.

Now navigate to [SITEROOT]/admin/appearance/colorizer by clicking on the ‘Colorizer’ tab. Here you will see the color scheme options. There are a few default options you can select from the drop down, or you can enter hex values to create a custom color scheme, be sure to click ‘Save Configuration’ when finished. Your new colors are saved to a css file in your files directory. If you do not see your changes you may need to clear the colorizer cache by clicking the ‘Clear Colorizer Cache’ button. These colors will override all other styles in the theme.

_images/Appearance_DKAN_4.png
Fonts

On the Appearance page, you will see a sub-menu item for @font-your-face. Navigate to [SITEROOT]/admin/appearance/fontyourface.

By default, the Google fonts api is enabled. If you click on the Browse all fonts tab, you will be able to select which google fonts to add to your site. Once you have made your selections, click on the Enabled fonts tab to view the font settings screen. Click on the By CSS selector, here you can select the css tag and what font should be applied to it using the table. To add fonts to specific css selectors, add them to the open text field at the bottom of the list, select a font from the dropdown next to it and click the Save applied fonts button For more information on how to use @fontyourface visit the project page.

Apply fonts Select google fonts
Creating a new subtheme

To create a Nuboot Radix subtheme, run these commands

drush en radix

drush vset theme_default radix

drush radix "MyThemeName" --kit=https://github.com/NuCivic/radix-kit-nuboot/archive/master.zip

drush vset theme_default MyThemeName

drush dis radix

OR if using Ahoy:

ahoy dkan theme new-from-kit [new-theme-name]

ahoy dkan theme setup

ahoy dkan theme watch

Your new subtheme will be placed in to the /sites/all/themes/ directory, it will contain only the directory structure, add your overrides where appropriate.

Theming Tools

Install Node and npm. You will use gulp for compiling the sass files. To get your local environment set up, follow these steps:

  1. Install Node and npm. You can read a guide on how to install node here
  2. Install bower: npm install -g bower.
  3. Go to the root of your theme and run the following commands: npm run setup.
  4. Update browserSyncProxy in config.json
  5. Edit the files under the scss and js directory, these will be compiled into the assets directory. Run the following command to compile Sass and watch for changes: gulp.

Icon Fonts

The Nuboot Radix theme ships with two icon fonts:

dkan-flaticon

This font is used for file types (csv, pdf, xls, etc) designed by Freepik

The font files and the css are inside the Nuboot Radix theme dkan/themes/nuboot_radix. If you would like to use your own file type icons you can override the dkan-flaticon css by creating a custom theme. OR, if you would like to use the dkan-flaticon icons but NOT use Nuboot Radix as your base theme, you will need to copy the dkan-flaticon fonts and the dkan-flaticon.css into the theme you are using.

dkan/themes/nuboot_radix/assets/fonts/dkan-flaticon.eot
dkan/themes/nuboot_radix/assets/fonts/dkan-flaticon.svg
dkan/themes/nuboot_radix/assets/fonts/dkan-flaticon.ttf
dkan/themes/nuboot_radix/assets/fonts/dkan-flaticon.woff
dkan/themes/nuboot_radix/assets/css/dkan-flaticon.css
dkan-topics

This font is used for the Content Type and Topics icons, see Streamline Icons

If you would like to use your own icon font for Topics, use the steps outlined here.

DKAN Roles and Permissions

Roles categorize types of users. Permissions are assigned to these roles and represent functions the user can perform across the site.

Below is a list of the roles and general permissions included in the DKAN Permissions module. The descriptions should help show what different user types are able to do on the site. Whan adding new users, site managers will assign the appropriate role(s) to match the tasks the user is expected to perform.

There will also be cases where you need users to have different permissions in the context of a particular group (for instance, to be able to modify content belonging to their agency but not other agencies on the same website). Read more about group permissions here.

The DKAN Permissions module

The DKAN Permissions module provides default roles and permissions for the DKAN distribution. It uses the export method provided by the Features Roles Permissions module so you can examine the specific roles and permissions provided by reviewing dkan_permissions.features.roles_permissions.inc.

Drupal Core Roles

Anonymous User
  • General site visitors that are not logged in.
  • Can view and search published content.
Authenticated User

These users have an account on the site, can log in, and have profile information that can be verified (and authenticated). This type has the lowest level of permissions.

Permissions:

  • Can log in.
  • Can edit their user profile.
  • Can view and search published content.
Administrator

The administrator role holds every permission and requires high technical competency. This role has the ability to cause serious damage, so it’s generally reserved for a single web professional. Administrators hold the highest level of all roles and permissions.

Permissions:

  • Enable/disable modules and features.
  • Change the appearance of the site with alternate themes.
  • Create and edit user roles and permissions.
  • Create views, blocks, features, content types.
  • Access any UI configuration.

DKAN Core Roles

Content Creator

Content Creators can add content to the site. This will be someone working in your organization who helps by adding to the data catalogue but is not responsible for anything more. This level of access takes users into the production side of the site, but gives little freedom to move outside of creating and adding certain content types. Limiting this role is critical for avoiding inadvertent damage to site content.

Permissions:

  • Create dataset, resource, data story, and data dashboard content.
  • Create chart visualizations.
  • Edit own content (can not edit content added by another user).
  • View own unpublished content and revision history of all published content.
  • Add and view files to site library.
Editor

This will typically be a person handling the content on a frequent basis. Someone in your organization with expertise on the subject-matter that is expansive as well as in-depth. An editor role is similar to a content creator role because the focus is on content, however, an editor will have the ability to manage and edit content added by others. The editor role does not go further into administrative functions.

Permissions:

  • Create page, dataset, resource, data story, and data dashboard content.
  • Create chart visualizations.
  • Edit, delete, and manage versions of content added by other users.
  • Add and view files to site library.
Site Manager

The highest level for non-technical users. A site manager is mostly concerned with the admin functions of the site. Typically this will fall to someone in a supervisory role. The site manager takes a high-level view of the site, its content, and the users on the site. This person is able to make general configurations to the site and assigns roles to new users but does not deal with the technical configuration of the backend.

Permissions:

  • Create, edit, delete all content types created by any user.
  • Create, edit and manage Harvest Source content.
  • Assigns roles to all user levels, but cannot create new roles/perms.
  • Create and manage groups.
  • Manage site logo, name, slogan, copyright, colors, fonts, main menu, recline config, open data schema mapper, dataset forms and previews.

DKAN Workflow Roles

If your organizaton needs an editorial workflow for managing content creation and editing, DKAN also includes a feature called DKAN Workflow that adds three more roles to establish a content approval process. Read more about that here.

Installation/upgrade notes

On new DKAN installs, the DKAN Permissions module is enabled by default. The older (“sidewide”-namespaced) permissions module will still be included in the codebase to support existing sites, but will be disabled by default on new installs. For _existing_ sites, the opposite is true - updating your code will _not_ cause the newer module to be enabled automatically or disable the older permissions module.

If you have been using the older DKAN Sitewide Roles and Permissions module on an existing site and do upgrade, we do recommend you disable it and enable the new DKAN Permissions module. The newer module has an improved set of roles and permissions designed around what we consider the most general use-cases. However, this will likely mean reviewing all of the user accounts on your site and ensuring that they have the roles that they should.

You also of course have the option of disabling both modules, setting your own prefered roles and permissions and exporting those to a custom feature.

Content and Storytelling Tools

DKAN uses Panels to allow site editors to create customized layouts and manage content through a drag and drop interface. For more on how to use the editing tools view the IPE section.

Content Types

Page

Use the Page content type for basic informational content that describes the purpose of your website. Examples would include the front page, an about page, an FAQ page, etc.

Create a page

  1. From the administration menu, navigate to Content > Add Content > Page.
  2. Fill in the title and body fields.
  3. Choose the desired layout.
  4. Click Save.
  5. Now add additional content by clicking the + buttons in the regions where you want the new content to display. Learn more about the In-Place Editor (IPE) here.
  6. Click Save at the bottom of the page.
Data Stories

Use the Data Story content type for narrative content. Narrative content is designed to tell a story and show the impact data has on the everyday lives of citizens. By adding context and synthesis, you can bring out the personal elements of data, and help highlight the investment and value of open data that might not otherwise be apparent.

Create a data story

  1. From the administration menu, navigate to Content > Add Content > Data Story.
  2. Fill in the title, add a cover photo, add tags, topics, and a description.
  3. Choose the desired layout.
  4. Click Save.
  5. Now add additional content by clicking the + buttons in the regions where you want the new content to display. Learn more about the In-Place Editor (IPE) here.
  6. Click Save at the bottom of the page.
Data Dashboard

Mix videos, images, slide shows, visualizations, text, tables, and maps to most effectively deliver your content. With more than 20 responsive layouts to choose from and the easy to use drag and drop interface, any user can create compelling data-powered content within minutes.

Create a data dashboard

  1. From the administration menu, navigate to Content > Add Content > Data Dashboard.
  2. Fill in the data dashboard title.
  3. Choose the desired layout.
  4. Click Save.
  5. Now add content by clicking the + buttons in the regions where you want the new content to display. Learn more about the In-Place Editor (IPE) here.
  6. Click Save at the bottom of the page.

Using the in-place editor (IPE)

The In-Place Editor (IPE) interface at its core it is a drag and drop content manager that lets you visually design a layout and place content within that layout.

DKAN’s customized IPE interface

DKAN has simplified the list of IPE options to show only the most used elements, this makes adding and editing content a faster and less daunting process.

Editing existing panels-based content
  1. Go to a panelized page (in this case the DKAN homepage) and click “Customize this page”:
_images/customize-this-page.png
  1. Use the drag and drop controls to rearrange a given pane, click and hold the directional arrows button on the top right corner of the panel, drag it below the region name you want it to display in, wait until you see a yellow space light up before you let go :
_images/drag-n-drop.png
  1. Save changes by clicking the save button at the bottom:

    _images/ipe-save.png
Adding new panes to Panels-based pages
  1. Go to a page managed by a panel layout and click the Customize this page button at the bottom:

  2. Click the “plus” sign (“+”) to add a new pane to any given panel region.

  3. Select the type of content you would like to add:

    _images/ipe-list.png
  4. Fill in the form that is presented, click ‘Finish’.

  5. Remember to click ‘Save’ at the bottom when you are done adding content to the page.

Altering the layout of an existing page
  1. Go to the page you want to change the layout for and click Change Layout:

    _images/change-layout.png
  2. Select new layout:

    _images/layout-options.png
  3. Choose where existing panes belong in the new layout:

    _images/arrange-content.png
  4. Click Save (or Save as Custom) and enjoy the new page:

    _images/ipe-save.png

Open Data Schema Map

This module provides a flexible way to expose your Drupal content via APIs following specific Open Data schemas. Currently, the CKAN, Project Open Data and DCAT-AP schemas are provided, but new schemas can be easily added through your own modules. A user interface is in place to create endpoints and map fields from the chosen schema to Drupal content using tokens.

This module was developed as part of the DKAN project, but will work on an Drupal 7 site. A separate module exists for DKAN-specific implementation.

Note that serious performance issues can result if you do not follow recommendations in the ODSM File Cache section.

Basic concepts

Schema

A schema is a list of field definitions, usually representing a community specification for presenting machine-readable data. The core Open Data Schema Map module does not include any schemas; they are provided by additional modules. A schema module includes:

  • a standard Drupal .module file – with an implementation of hook_open_data_schema() to expose the schema to the core Open Data Schema Map module, plus _alter functions for any needed modifications of the UI form or the data output itself.
  • the schema itself, expressed as a .json file. For instance, see the Project Open Data schema file to see how these schema are defined in JSON
API

An API in this module is a configuration set that exposes a specific set of machine-readable data at a specific URL (known as the API’s endpoint). This module allows you to create multiple APIs that you save as database records and/or export using Features. An API record will contain:

  • an endpoint URL
  • a schema (chosen from the available schemas provided by the additional modules as described above)
  • a mapping of fields defined in that schema to Drupal tokens (usually referencing fields from a node)
  • optionally, one or more arguments passed through the URL to filter the result set

Usage

Installation

Enable the main Open Data Schema Map module as usual, and additionally enable any schema modules you will need to create your API.

Creating APIs

Navigate to admin/config/services/odsm and click “Add API.”

screen shot 2014-07-14 at 3 24 03 pm

Give the API a title, machine name, choose which entity type (usually node) and bundle (in DKAN, this is usually Dataset).

screen shot 2014-07-14 at 3 46 39 pm

You will need to create the API record before adding arguments and mappings.

Arguments

The results of the API call can be filtered by a particular field via arguments in the URL. To add an argument, first choose the schema field then, if you are filtering by a custom field API field (ie, a field whose machine name begins with “field_”), identify the database column that would contain the actual argument value. Leave off the field name prefix; for instance, if filtering by a DKAN tag (a term reference field), the correct column is field_tags_tid, so you would enter “tid”. Which Drupal field to use will be extrapolated from the token you map to that schema field.

Screen Shot 2014-07-14 at 3.55.49 PM.png | uploaded via ZenHub

Field Mapping

The API form presents you with a field for each field in your schema. Map the fields using Drupal’s token system. Note: using more than one token in a single field may produce unexpected results and is not recommended.

Multi-value fields

For Drupal multi-value entity reference fields, the schema can use an array to instruct the API to iterate over each value and map the referenced data to multiple schema fields. For instance, in the CKAN schema, tags are described like this in schema_ckan.json:

      "tags": {
      "title":"Tags",
      "description":"",
      "anyOf": [
        {
          "type": "array",
          "items": {
            "type": "object",
            "properties": {
              "id": {
                "title": "UUID",
                "type": "string"
              },
              "vocabulary_id": {
                "title": "Vocaulary ID",
                "type": "string"
              },
              "name": {
                "title": "Name",
                "type": "string"
              },
              "revision_timestamp": {
                "title": "Revision Timestamp",
                "type": "string"
              },
              "state": {
                "title": "state",
                "description": "",
                "type": "string",
                "enum": ["uncomplete", "complete", "active"]
              }
            }
          }
        }
      ]
    },

You can choose which of the available multivalue fields on your selected bundle to map to the “tags” array, exposing all of the referenced “tag” entities (taxonomy terms in this example) to use as the context for your token mappings on the schema fields within that array. First, simply choose the multivalue field, leaving the individual field mappings blank, and save the form.

screen shot 2014-07-16 at 12 14 29 am

When you return to the tags section of the form after saving, you will now see a special token navigator you can use to find tokens that will work with this iterative approach (using “Nth” in place of the standard delta value in the token):

screen shot 2014-07-16 at 12 22 00 am

Customizing

Adding new schemas

You are not limited by the schemas included with this module; any Open Data schema may be defined in a custom module. Use the open_data_schema_ckan module as a model to get started.

Date format

Date formats can be chanaged manually by changing the “Medium” date time format in “admin/config/regional/date-time” or in code by using one of the alter hooks: screen shot 2014-09-04 at 11 15 01 am

A Note on XML Output

Open Data Schema Map provides an XML output format. This is provided via a separate submodule in the modules/ folder for historical reasons, but should be refactored into the main ODSM module in a future release.

XML endpoints still require a schema defined in JSON. Defining your own XML endpoint may be less than intuitive for the time beind, but take a look at the DCAT schema module for a model.

The ODSM File Cache

Open Data Schema Map endpoints that list a large number of entities – Project Open Data (data.json), the CKAN Package List (/api/3/action/package_list) and DCAT-AP Catalog (catalog.xml) – perform a full entity load for each record listed in order to perform the token replacements. This can cause a major performance hit each time any of these URLs is hit on a site with more than a few dozen datasets, and on a site with thousands the response time can be two minutes or more.

Open Data Schema Map includes a file caching function to save a snapshot of any endpoint as a static file to be served up quickly, with very few hits to the database.

File caches at present can only be generated by a Drush command. The recommended usage on a production website is to set up a cron job or use a task runner like Jenkins to regenerate the file caches for your performance-intensive endpoints daily, at whatever time your site experiences the least amount of traffic. The trade-off of course is that any additions or changes to your site will not be reflected on these endpoints until they are regenerated.

An administrative UI to regenerate a file cache manually may be included in a future release.

Use

The Drush command supplied by Open Data Schema Map is odsm-filecache (also available simply as the alias odsmfc). This command takes as its argument the machine name for an ODSM endpoint. For example:

drush odsm-filecache data_json_1_1

This will render the full data_json_1_1 endpoint (which is the data.json implementation that ships with DKAN) to the filesystem, saving it to:

public://odsm_cache_data_json_1_1

Now a hit to /data.json will be routed to this file, which in most cases will actually live at /sites/default/files/odsm_cache_data_json_1_1.

Schema Validation

Both the Project Open Data and DCAT-AP schemas ship with validation tools you can access from the Drupal admin menu. More documentation on this feature coming soon...

Community

We are accepting issues for Open Data Schema Map in the DKAN issue queue only. Please label your issue as “Component: ODSM” after submitting so we can identify problems and feature requests faster.

If submitting a pull request to this project, please try to link your PR to the corresponding issue in the DKAN issue thread.

Visualizations

DKAN provides visualization functionality in two primary ways: through the automatically-created “data previews” that are displayed with data resources, and the more-complex and manually-build visualization “entities”:

DKAN Data Preview Features

DKAN allows users to have a preview of their data when uploaded to or linked to a resource. Which preview type is displayed for a particular resource depends on the data format selected. If no data format is provided, DKAN will attempt to auto-detect the format from the file’s metadata; re-editing the resource and correcting the format field manually may be necessary if the auto-detection is not successful.

This functionality is provided via the Recline module, which is not part of the core DKAN repository but is a basic dependency of it (and will be included when building the distribution via drush make).

Recline.js

DKAN, like CKAN, offers an integration with the Recline Javascript library. Recline allows site visitors to preview tabular data visually. The preview works for CSV and XLS [*] files that are uploaded to the DKAN site or hosted remotely and linked to, as well as for data stored in DKAN’s local SQL-based datastore.

Grid View

All tabular data can be rendered as spreadsheet-style rows and columns:

_images/csv-preview.png
[*]For xls files be sure to fill in the format field to see previews of the data
_images/xls-format.png
Map View

Visitors can preview data that contains either coordinates or GeoJSON on a Leaflet.js -based map:

_images/map-preview.png
Graph View

If enabled, visitors can chose one column of your data as an X-axis, one or more as Y-axis data, and preview your data as a bar, point or line graph.

_images/graph-preview.png
File size limits

Files can only be previewed if they are well formatted and small enough to render in the browser.

If files are too large to preview within 1 second you will get the following message “File was too large or unavailable for preview.”

Files that are too large to preview in the browser can be previewed by adding them to the datastore. Once a file is in the datastore the preview is only asking for the first 25 rows of the data. Thus large datasets can be previewed.

Additional Preview Types

DKAN provides preview formats for several additional file types beyond what is supported by Recline.js, these include: JSON, geojson, XML, ArcGIS REST, WMS, images, PDF, and ZIP files. These additional preview formatters are defined in a forked version of Recline

Zip files

DKAN offers the ability to preview the files and folders locked in ZIP files. DKAN will display a list of contents for ZIP files uploaded as resources on datasets.

_images/zip-preview.png
Image files

Image files (JPG, PNG or GIF) uploaded as resources will be displayed directly on the resource page.

_images/image-preview.png
Web Map Service (WMS)

DKAN can use Leaflet to display a preview of a WMS server, provided endpoint.

_images/wms.png

WMS support in DKAN is still somewhat experimental and your results may vary.

ArcGIS REST

An ESRI/ArcGIS REST endpoint may also be displayed in a Leaflet preview.

_images/arcgis.png
JSON files
_images/json-preview.png
GeoJSON files
_images/geojson-preview.png
XML files
_images/xml-preview.png
PDF files
_images/pdf-preview.png
External Previews

Starting with version 7.x-1.10, DKAN supports previewing/opening resources in external services that offer simple URL-based integrations. For instance, the CartoDB mapping service offers an Open in CartoDB service. Enabling this for CSV files will result in a dataset display like this:

_images/external-preview.png

External preview functionality can be enabled and configured in the “DKAN Dataset Previews” administration page (/admin/dkan/dataset_preview).

Configuration

By default previews are available for resources with files below 3MB of size. However you can customize this limit in the recline configuration page (/admin/dkan/recline).

_images/recline-configuration.png

Visualization Entity

Visualization Entity is included in the DKAN distribution.

This module aims to provide:

  • A base drupal entity to extend and create visualization bundles.
  • A set of visualization bundles that provide functionality out of the box for the module and indicate a good example for extending this entity.
  • A set of permissions common to all visualization bundles.
  • A common “iframe view” shared between all visualization bundles.
  • A common “embed” functionality shared between all visualization bundles.

Visualization Chart

Visualization Entity Charts is enabled by default in DKAN. This modules provides the ability to create embeddable NVD3 charts.

Usage

New chart entities can be created by going to /admin/structure/entity-type/visualization/ve_chart/add. A multi-step process will guide you through the creation of a chart based on an uploaded data file.

Step One - Choose a Resource

Use the form to upload a data file OR link to an existing data resource. Valid source data include: * CSV * Google Spreadsheet * Data Proxy

_images/chart-step-1.png
Step Two - Define Variables

Choose a single x-field and one or more y-fields (series) for the visualization.

_images/chart-step-2.png
Step Three - Choose Chart Type

NOTE: X and Y Axis Fields are not supported by the Pie Chart type.

_images/chart-step-3.png
Step Four - Preview and Adjust

You can adjust colors, margins, include a goal, labels, tick values, and more.

_images/chart-step-4.png
Query Editor

Click the ‘+’ on the query editor to add a query parameter to specify the data used for the chart.

_images/chart-query-editor.png
Filter Editor

Click the ‘+’ on the filter editor to add one or more filters to limit the data used for the chart. Multiple filters can be applied to data, the operator is ‘OR’.

  • Select the field you would like to filter by.
  • Select the type of filter (value, range, geo distance)
  • Click Add
  • Fill in the fields to complete the filter.

To delete a filter, click the ‘x’ next to the filter name.

_images/chart-filter-editor.png
Chart Configuration
X Axis
  • Format Select an appropriate format for the X Axis labels.
  • Axis Label will provide a custom label for the x axis.
  • Note: Axis labels do not display for Pie Charts.
  • Label rotation will change angle of label values.
  • Tick Values with step value will update the number of ticks for the X axis. NOTE: If the range set for tick values is smaller than the range of complete data represented, the chart will be abbreviated.
Y Axis
  • Axis Label Provides a custom label for the y axis.
  • Note: Axis labels do not display for Pie Charts.
  • Adjust the distance field until Label is visible on the chart.
  • Adjust Tick Values with step value to change the number of ticks for the Y axis. NOTE: If the range set for tick values is smaller than the range of complete data represented, the chart will be abbreviated.
General
Color:Set the color the chart is drawn in. Use either a HEX color code or a valid css color name
Transition Time:
 Time in ms it takes for graph to animate.
Goal:Overlay a goal or target line on the chart.
Margin:Enter value of margin in the order: top, right, bottom, left
Show Title:Display the title you entered on step 1.
Show Controls:Whether to show extra controls or not. Extra controls include things like making multiBar charts stacked or side by side.
Show Legend:Display a legend for the chart.
Group By X Field:
 For multiple series values Y will show values grouped by X
Show Tooltips:Shows data and label on hover.
Reduce Ticks:Reduces the number of axis values displayed.
Recline

The bundle also includes an integration with the Recline module. If you have a content type with a recline file field, you can add a Recline Field Reference field to your chart bundle. This field type is defined in a module that comes bundled with Visualization Entity. The included DKAN integration module adds a Recline Field Reference pointing specifically at DKAN’s Resource content type. In this case, entering an existing Resource node in the reference field will automatically populate the resource file into the chart entity’s file field.

GeoJSON

Warning

Under Development. Do not use on production.

Enable the geojson bundle:

$ drush -y visualization_entity_geojson_bundle
$ drush cc all
Create Visualization
  • Look for Content -> Add Content -> Resource in the admin menu and click on it.
  • Upload a geojson data file for the resource
_images/geojson-step-01.png
  • Fill the required fields, enter ‘geojson’ in the format field, and save the resource
_images/geojson-step-03.png
  • Look for Structure -> Entity Types -> Visualization -> Geojson Visualization -> Add Geojson Visualization in the admin menu and click on it.
  • Set a title
  • Select the resource containing the geojson data file you uploaded
_images/geojson-step-04.png
  • Click Save & Enjoy!
_images/geojson-step-05.png
Choropleth

Warning

Under Development. Do not use on production.

Enable the choropleth bundle:

$ drush en -y visualization_entity_choropleth_bundle
$ drush cc all
Examples Files

Two example files are provided in the examples folder:

africa.geojson
africa-data.csv
Create Visualization
  • Look for Content -> Add Content -> Resource in the admin menu and click on it.
  • Upload a africa-data.csv file from the examples folder for the resource.
_images/choropleth-step-00.png
  • Fill the required fields, enter ‘geojson’ in the format field, and save the resource
_images/choropleth-step-01.png
  • Look for Structure -> Entity Types -> Geo File -> geojson -> Add geojson in the admin menu and click on it.
_images/choropleth-step-02.png
  • Set Title
  • Upload a geojson file
  • Fill name attribute with the column name in the data (csv resource) that will match the name property for the features in the geojson file.
_images/choropleth-step-03.png
  • Click Save.
  • You’ll get a preview for the geojson file you just uploaded.
_images/choropleth-step-04.png
  • Look for Structure -> Entity Types -> Visualization -> Choropleth Visualization -> Add Choropleth Visualization in the admin menu and click on it.
_images/choropleth-step-05.png
  • Fill Title
  • Select the geojson file we created for the geojson field.
  • Select the resource file we created for the resource field.
_images/choropleth-step-06.png
  • Select the colors you like to use for the choropleth map.
  • Fill data column with the column in the csv data you’ll like to pick as the source of numerical data for the polygon coloring. If you leave this field blank, you’ll get a list of radio buttons to pick up the column when the visualization gets rendered.
  • Fill the data breakpoints with comma separated numbers. If you leave this field blank, breakpoints will be calculated for you based on the data.
_images/choropleth-step-07.png
  • Click Save & Enjoy!
_images/choropleth-step-08.png

This module is still on early development.

Additional visualization functionality can be found in the following projects, which are not included in the core DKAN project and are still in a relatively expiremental state:

Note

The three modules mentioned above that are not distributed with DKAN continue to be maintained in separate repositories because they work independently of DKAN and could be installed in non-DKAN Drupal sites.

Extending and Customizing DKAN

Much additional functionality can be added simply by installing one of the tens of thousands of contrib modules from the Drupal community. However, as a Drupal Distribution DKAN is a flexible framework which developers can also build off of and add to.

DKAN consists of of a distribution profile which manages the initial installation, 3rd party libraries and drupal modules, and DKAN specific modules.

Below is a simplified version of where the DKAN code sits within the fully packaged version:

profiles/
    dkan/
      libraries/ (3rd party libraries)
      modules/
         dkan/ (dkan modules)
         contrib/ (3rd party module dependencies)
      themes/ (dkan themes)
sites/
   all/
      libraries/ (your libraries)
      modules/ (your modules)
      themes/ (your themes)

After installing DKAN additional functionality should be added to the “sites” directory.

In the future, this section will feature more detailed information on developing custom extentions to DKAN. For now, read additional information about:

Customizing the License Field

In order to add options to the existing ones you need to implement hook_license_subscribe in the following fashion:

// Let's assume we want to do this as part of the fictitious license_options_extra module
function license_options_extra_license_subscribe() {
  return array(
    'tcl' => array(
      'label' => 'Talis Community License (TCL)',
      'uri' => 'http://opendefinition.org/licenses/tcl/',
    ),
  );
}

The code above add the Talis Community License (TCL) license referencing it to the tcl key. It also provides a link to the license (optional). You can provide as many options as you want through the array being returned.

Removing License Options

In order to remove options from the existing ones you need to implement hook_license_unsubscribe in the following fashion:

// Let's assume we want to do this as part of the fictitious license_options_extra module
function license_options_extra_license_unsubscribe() {
  return array(
    'notspecified',
  );
}

The code above removes the notspecified option. You can provide as many options as you want through the array being returned.

Additional notes about the behavior of both hooks

  • The options provided through the license drupal field configuration are COMPLETELY ignored.
  • hook_license_subscribe() implementations are of course called before hook_license_unsubscribe() implementations.
  • Options subscribed through hook_license_subscribe() are processed as they come through the order of modules provided by the drupal registry.
  • If multiple options are provided using the same key then it grabs the first one that comes in and ignores the rest
  • If you want to replace and item that already exists, unsubscribe the existing key and provided an alternative one for your option

References to the code

  • Hooks are invoked here
  • Field formatter implementation for the license field is in here

Community Contributions

What follows is a list of modules from the community that extend DKAN’s functionality. Many are developed by the DKAN team themselves but are still in an expiremental stage and not part of the core DKAN software. Contact us to add your mdodule to this list!

API Guide

DKAN includes a number of APIs to allow it to communicate with external applications.

Dataset REST API

The DKAN Dataset REST API uses the Services module to create CRUD endpoint at api/dataset/node. By default, this endpoint provides full CRUD access to a website’s content nodes, and limited access to users (to allow authentication). The endpoint can be customized at /admin/structure/services/list/dkan_dataset_api/resources.

Services Documentation

The DKAN Dataset API module is only a light wrapper around the Services module, which has extensive documentation. Here are some entries of interest:

The Sessions module also has a thriving community on the Drupal Stack Exchange.

Server Types

DKAN Dataset REST API comes with a REST Server. Other server types are also incldued in the Services module but not turned on. Those include:

  • OAUTH
  • XML-RPC

Repsonse Formats

  • bencode
  • json
  • jsonp
  • php
  • xml
  • yaml

Request Parser Types

  • application/json
  • application/vnd.php.serialized
  • application/x-www-form-urlencoded
  • application/x-yaml
  • application/xml
  • multipart/form-data
  • text/xml

Authentication Modes

Session Authentication

Session authentication is enabled by default. With session authentication an inital request is made to the user login to requet a session cookie. That session cookie is then stored locally and sent with a request in the X-CSRF-Token header to authenticate the request.

Token Authentication

Token authenticaion is not currently available out-of-the-box. However, it can be enabled by adding the Services Token Access module to your site. This is less secure but is easier for community members to use, and may be added to the DKAN distribution in a future release.

Authentication Permissions

The permissions with which a user is granted depend on the user role. User roles and permissions are easily configured in the user administration screen at admin/people, and DKAN comes with a number of pre-configured default roles via the DKAN Permissions module.

Examples

Below you can find examples in PHP for the most common use cases, using session authentication.

For an example of a fully-functional python-based client to the DKAN REST API, see the pydkan project.

Get the CSRF Token
// Setup request URL.
$request_url = 'http://example.com/services/session/token';

// Setup request.
$curl = curl_init($request_url);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Accept: application/json')); // Accept JSON response.
curl_setopt($curl, CURLOPT_POST, 1); // Do a regular HTTP POST.
curl_setopt($curl, CURLOPT_COOKIE, "$cookie_session"); // Send the cookie session that we got after login.
curl_setopt($curl, CURLOPT_HEADER, FALSE);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_FAILONERROR, TRUE);

// Execute request and save CSRF Token.
$csrf_token = curl_exec($curl);
Create a Resource
// Set up request URL.
$request_url = 'http://example.com/api/dataset/node';

// Setup resource data.
// A great explanation on how to target each node field can be found on the 'Identifying field names' article linked on the 'Documentation' section.
$resource_data = array(
    'type' => 'resource',
    'title' => 'Example resource',
    'status' => 1,
    'body[und][0][value]' => 'The description'
);
$resource_data = http_build_query($resource_data);

// Setup request.
$curl = curl_init($request_url);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Accept: application/json', 'X-CSRF-Token: ' . $csrf_token));
curl_setopt($curl, CURLOPT_POST, 1); // Do a regular HTTP POST.
curl_setopt($curl, CURLOPT_POSTFIELDS, $resource_data); // Set POST data.
curl_setopt($curl, CURLOPT_COOKIE, "$cookie_session");
curl_setopt($curl, CURLOPT_HEADER, FALSE);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_FAILONERROR, TRUE);

// Execute request and get response.
$response = curl_exec($curl);
Attach a file to a resource
// Set up request URL.
$request_url = 'http://example.com/api/dataset/node/' . $resource_id . '/attach_file';

// Setup file data.
$file_data = array(
    'files[1]' => curl_file_create($file),
    'field_name' => 'field_upload',
    'attach' => 1
);

// Set up request.
$curl = curl_init($request_url);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Content-Type: multipart/form-data','Accept: application/json', 'X-CSRF-Token: ' . $csrf_token));
curl_setopt($curl, CURLOPT_POST, 1); // Do a regular HTTP POST.
curl_setopt($curl, CURLOPT_POSTFIELDS, $file_data); // Set POST data.
curl_setopt($curl, CURLOPT_COOKIE, "$cookie_session");
curl_setopt($curl, CURLOPT_HEADER, FALSE);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_FAILONERROR, TRUE);

// Execute request and get response.
$response = curl_exec($curl);
Create a Dataset
// Set up request URL.
$request_url = 'http://example.com/api/dataset/node';

// Set up dataset data.
// A great explanation on how to target each node field can be found on the 'Identifying field names' article linked on the 'Documentation' section.
$dataset_data = array(
    'type' => 'dataset',
    'title' => 'Example dataset',
    'status' => 1,
    'body[und][0][value]' => 'The description',
    'field_resources[und][0][target_id]' => 'Madison Polling Places (5)' // Resource title plus node id
    'field_author[und][0][value]' => 'Bob Lafollette'
);
$dataset_data = http_build_query($dataset_data);

// Set up request.
$curl = curl_init($request_url);
curl_setopt($curl, CURLOPT_HTTPHEADER, array('Accept: application/json', 'X-CSRF-Token: ' . $csrf_token));
curl_setopt($curl, CURLOPT_POST, 1); // Do a regular HTTP POST.
curl_setopt($curl, CURLOPT_POSTFIELDS, $dataset_data); // Set POST data.
curl_setopt($curl, CURLOPT_COOKIE, "$cookie_session");
curl_setopt($curl, CURLOPT_HEADER, FALSE);
curl_setopt($curl, CURLOPT_RETURNTRANSFER, TRUE);
curl_setopt($curl, CURLOPT_FAILONERROR, TRUE);

// Execute request and get response.
$response = curl_exec($curl);

Testing in the terminal

If you want to quickly test that the functionality is working, you can run the following commands from a terminal.

Replace the domain, username and password in the commands below to match your development environment, and then replace the token value with the token from the response to the authentication request.

Users
Authentication (login)
curl -X POST -i -H "Content-type: application/json" -H "Accept: application/json" -c cookies.txt -X POST http://demo.getdkan.com/api/dataset/user/login -d '{
  "username":"admin",
  "password":"password"
}'

This will return the cookie and the CSRF token that we need to reuse for all the authenticated user iteration via the API.

Content (Datasets, resource)
Retrive dataset
curl http://demo.getdkan.com/api/dataset/node/22.json

Example response:

{
  "vid": "52",
  "uid": "1",
  "title": "Wisconsin Polling Places",
  "log": "Update to resource 'Madison Polling Places'",
  "status": "1",
  "comment": "0",
  "promote": "0",
  "sticky": "0",
  "vuuid": "30daa43f-aa4a-477a-b011-047ce3d5007e",
  "nid": "22",
  "type": "dataset",
  "language": "und",
  "created": "1360541580",
  "changed": "1477369101",
  "tnid": "0",
  "translate": "0",
  "uuid": "934400f2-a5dc-4abf-bf16-3f17335888d3",
  "revision_timestamp": "1477369101",
  "revision_uid": "1",
  "body": {
    "und": [
      {
        "value": "<p>Polling places in the state of Wisconsin.</p>\n",
        "summary": null,
        "format": "html",
        "safe_value": "<p>Polling places in the state of Wisconsin.</p>\n",
        "safe_summary": ""
      }
    ]
  },
  "field_additional_info": [],
  "field_author": {
    "und": [
      {
        "value": "Wisconsin Board of Elections",
        "format": null,
        "safe_value": "Wisconsin Board of Elections"
      }
    ]
  },
  "field_conforms_to": [],
  "field_contact_email": {
    "und": [
      {
        "value": "datademo@nucivic.com",
        "format": null,
        "safe_value": "datademo@nucivic.com"
      }
    ]
  },
  "field_contact_name": {
    "und": [
      {
        "value": "Couch, Aaron",
        "format": null,
        "safe_value": "Couch, Aaron"
      }
    ]
  },
  "field_data_dictionary": [],
  "field_data_dictionary_type": [],
  "field_frequency": {
    "und": [
      {
        "value": "5"
      }
    ]
  },
  "field_granularity": [],
  "field_harvest_source_ref": [],
  "field_is_part_of": [],
  "field_landing_page": [],
  "field_language": [],
  "field_license": {
    "und": [
      {
        "value": "cc-by",
        "format": null,
        "safe_value": "cc-by"
      }
    ]
  },
  "field_harvest_source_issued": [],
  "field_harvest_source_modified": [],
  "field_pod_theme": [],
  "field_public_access_level": {
    "und": [
      {
        "value": "public"
      }
    ]
  },
  "field_related_content": [],
  "field_resources": {
    "und": [
      {
        "target_id": "4"
      }
    ]
  },
  "field_rights": [],
  "field_spatial": {
    "und": [
      {
        "wkt": "POLYGON ((-90.415429 46.568478, -90.229213 46.508231, -90.119674 46.338446, -89.09001 46.135799, -88.662808 45.987922, -88.531362 46.020784, -88.10416 45.922199, -87.989145 45.796229, -87.781021 45.675736, -87.791975 45.500474, -87.885083 45.363551, -87.649574 45.341643, -87.742682 45.199243, -87.589328 45.095181, -87.627666 44.974688, -87.819359 44.95278, -87.983668 44.722749, -88.043914 44.563917, -87.928898 44.536533, -87.775544 44.640595, -87.611236 44.837764, -87.403112 44.914442, -87.238804 45.166381, -87.03068 45.22115, -87.047111 45.089704, -87.189511 44.969211, -87.468835 44.552964, -87.545512 44.322932, -87.540035 44.158624, -87.644097 44.103854, -87.737205 43.8793, -87.704344 43.687607, -87.791975 43.561637, -87.912467 43.249452, -87.885083 43.002989, -87.76459 42.783912, -87.802929 42.493634, -88.788778 42.493634, -90.639984 42.510065, -90.711184 42.636034, -91.067185 42.75105, -91.143862 42.909881, -91.176724 43.134436, -91.056231 43.254929, -91.204109 43.353514, -91.215062 43.501391, -91.269832 43.616407, -91.242447 43.775238, -91.43414 43.994316, -91.592971 44.032654, -91.877772 44.202439, -91.927065 44.333886, -92.233773 44.443425, -92.337835 44.552964, -92.545959 44.569394, -92.808852 44.750133, -92.737652 45.117088, -92.75956 45.286874, -92.644544 45.440228, -92.770513 45.566198, -92.885529 45.577151, -92.869098 45.719552, -92.639067 45.933153, -92.354266 46.015307, -92.29402 46.075553, -92.29402 46.667063, -92.091373 46.749217, -92.014696 46.705401, -91.790141 46.694447, -91.09457 46.864232, -90.837154 46.95734, -90.749522 46.88614, -90.886446 46.754694, -90.55783 46.584908))",
        "geo_type": "polygon",
        "lat": "44.635",
        "lon": "-90.0142",
        "left": "-92.8855",
        "top": "46.9573",
        "right": "-87.0307",
        "bottom": "42.4936",
        "srid": null,
        "accuracy": null,
        "source": null
      }
    ]
  },
  "field_spatial_geographical_cover": {
    "und": [
      {
        "value": "Wisconsin, United States",
        "format": null,
        "safe_value": "Wisconsin, United States"
      }
    ]
  },
  "field_tags": {
    "und": [
      {
        "tid": "9"
      }
    ]
  },
  "field_temporal_coverage": [],
  "og_group_ref": {
    "und": [
      {
        "target_id": "1"
      }
    ]
  },
  "field_topic": [],
  "field_orphan": {
    "und": [
      {
        "value": "0"
      }
    ]
  },
  "rdf_mapping": {
    "rdftype": [
      "sioc:Item",
      "foaf:Document"
    ],
    "title": {
      "predicates": [
        "dc:title"
      ]
    },
    "created": {
      "predicates": [
        "dc:date",
        "dc:created"
      ],
      "datatype": "xsd:dateTime",
      "callback": "date_iso8601"
    },
    "changed": {
      "predicates": [
        "dc:modified"
      ],
      "datatype": "xsd:dateTime",
      "callback": "date_iso8601"
    },
    "body": {
      "predicates": [
        "content:encoded"
      ]
    },
    "uid": {
      "predicates": [
        "sioc:has_creator"
      ],
      "type": "rel"
    },
    "name": {
      "predicates": [
        "foaf:name"
      ]
    },
    "comment_count": {
      "predicates": [
        "sioc:num_replies"
      ],
      "datatype": "xsd:integer"
    },
    "last_activity": {
      "predicates": [
        "sioc:last_activity_date"
      ],
      "datatype": "xsd:dateTime",
      "callback": "date_iso8601"
    }
  },
  "path": "http://demo.getdkan.com/dataset/wisconsin-polling-places",
  "name": "admin",
  "picture": "0",
  "data": "b:0;"
}
Create a new dataset

This will need an authenticated user with appropriate permissions. The headers include the user credentials (cookie and CSRF token).

curl -X POST -i -H "Content-type: application/json" -H "X-CSRF-Token: 8RniaOCwrsK8Mvue0al_C6EMAraTg26jzklDdLLgvns" -b cookies.txt -X POST http://demo.getdkan.com//api/dataset/node -d '{
  "title":"A node created via DKAN REST API",
  "type":"dataset",
  "body": {
    "und": [{"value": "This should be the description"}]
  }
}'
Update dataset title

To update content we use the PUT HTTP method. This will add the word “UPDATED” to the title:

curl -X PUT -i -H "Content-type: application/json" -H "X-CSRF-Token: 8RniaOCwrsK8Mvue0al_C6EMAraTg26jzklDdLLgvns" -b cookies.txt http://demo.getdkan.com//api/dataset/node/22 -d '{
  "title":"A node created with services 3.x and REST server - UPDATED"
}'
Update a dataset field

Titles are a core property for content in Drupal. Updating additional content-type-specific fields requires a slightly more complex data structure. To update a dataset’s frequency, for instance:

curl -X PUT -i -H "Content-type: application/json" -H "X-CSRF-Token: 8RniaOCwrsK8Mvue0al_C6EMAraTg26jzklDdLLgvns" -b cookies.txt http://demo.getdkan.com/api/dataset/node/22 -d '{
  "field_frequency": {"und":{"value": 6}}
}'

Because the REST API runs input through the dataset node form for validation, the data structure may differ for different fields. For instance, because it is a “Select or license” field, the structure for changing the License field on a dataset to “cc-nc” (Creative Commons Non-Commercial) would be:

{
  "field_license": {"und": {"select": {"value": "cc-nc"}}}
}

See the Services documentation on custom fields for more detailed information.

Add new resource to dataset

This is a two-step process with the API:

  1. Create the resource node:
curl -X POST -i -H "Content-type: application/json" -H "X-CSRF-Token: 8RniaOCwrsK8Mvue0al_C6EMAraTg26jzklDdLLgvns" -b cookies.txt -X POST http://demo.getdkan.com/api/dataset/node -d '{
  "title":"A resource created via the DKAN REST API",
  "type":"resource",
  "body": {"und": [{"value": "This should be the description for the resource."}]},
  "field_link_api": {"und": [{"url": "http://data.worldbank.org/"}]}
}'
  1. Attach the newly created resource node to the parent dataset. Use the node ids that match the dataset and resource created by the commands above.
curl -X PUT -i -H "Content-type: application/json" -H "X-CSRF-Token: 8RniaOCwrsK8Mvue0al_C6EMAraTg26jzklDdLLgvns" -b cookies.txt http://demo.getdkan.com/api/dataset/node/43 -d '{
  "field_resources": {"und": [{"target_id": "A resource created via the DKAN REST APIs (45)"}]}
}'

Note

The provided value (A resource create via the DKAN REST API (45)) is the value expected from the dataset entry form, with “45” being the resource node id.

Query for url/values of previous revision of file.

The assumption in this example is that the file is stored remotely and we are looking to get the link as it was set in a previous revision of the resource node.

Versions/revisions are tracked via Durpal’s vid identifier. We can query a specific node revision (for example vid 89) using the vid as parameter

curl -X GET -gi -H "Content-type: application/json" -H "X-CSRF-Token: 8RniaOCwrsK8Mvue0al_C6EMAraTg26jzklDdLLgvns" -b cookies.txt 'http://demo.getdkan.com/api/dataset/node.json?parameters[vid]=89'
Known issues
  • Datasets and other content nodes can only be queried via node id or other entity. UUID support pending.
  • Upon attaching a file to a resource via the API, DKAN will immediately import this file to the Datastore if it is a valid CSV. This may not always be the desired behavior; more control over datastore behavior should be available to API clients.

Datastore API

DKAN offers a Datastore API as a custom endpoint for the Drupal Services module.

This API is designed to be as compatible as possible with the [CKAN Datastore API] (http://ckan.readthedocs.org/en/latest/maintaining/datastore.html).

Parameters

  • resource_id (mixed) – id (string) or ids (array) of the resource(s) to be searched against.
  • filters (mixed) – array or string of matching conditions to select
  • q (string) – full text query
  • offset (int) – offset this number of rows
  • limit (int) – maximum number of rows to return (default: 100)
  • fields (array or comma separated string) – fields to return (default: all fields in original order)
  • sort (string) – comma separated field names with ordering
  • join (array) – array of fields to join from multiple tables
  • group_by (array) – array of fields to group by

Aggregation functions

  • sum (string) – field to compute the sum
  • avg (string) – field to compute the average
  • min (string) – field to compute the maximum
  • max (string) – field to compute the minimum
  • std (string) – field to compute the standard deviation
  • variance (string) – field to compute the variance

URL format

Parameters passed by URL share a common format:

param_name[resource_alias][field_name]=value,value1
  • param_name: the param you are using (e.g. offset)
  • resource_alias(optional): an alias to reference an specific resource in further params.
  • field_name(optional): a field name used by the param name.
  • value: a list of values divided by commas

Note that resource_alias and field_name arguments are optional and depend on what you want to query. For example, if you need to limit the number of records, you need to use the limit parameter. However, it doesn’t make sense to specify an alias or a field in such a case. You only need to provide the number of records you need to retrieve:

...&limit=5

There is one exception: Even when the sort parameter shares the above syntax, it also accepts an alternative format:

...&sort=field1,field2 desc

Multiple queries

Sometimes you want to do mutiple datastore queries in one network request (e.g., to feed a data dashboard). In that case you can post a JSON object to http://EXAMPLE.COM/api/action/datastore/search.json with all the queries to perform.

The request body should have a format similar to this:

Request body
{
  "my_query": {
    "resource_id": {
      "states": "d2142282-9838-4cca-972f-f1741410417b",
      "gold_prices":"d3c099c6-1340-4ee5-b030-8faf22b4b424"
    },
    "limit": 5
  },
  "my_query1": {
    "resource_id": {
      "gold_prices": "d3c099c6-1340-4ee5-b030-8faf22b4b424"
    },
    "limit": 5
  }
}
Response
{
  "my_query": {
    "help": "Search a datastore table. :param resource_id: id or alias of the data that is going to be selected.",
    "success": true,
    "result": {
      "fields": [
        {
          "id": "nombre",
          "type": "text"
        },
        {
          "id": "state_id",
          "type": "int"
        }
      ],
      "resource_id": {
        "states": "d2142282-9838-4cca-972f-f1741410417b",
        "gold_prices": "d3c099c6-1340-4ee5-b030-8faf22b4b424"
      },
      "limit": 1,
      "total": 5,
      "records": [
        {
          "nombre": "Alabama",
          "state_id": "1",
          "feeds*flatstore_entry*id": "1",
          "timestamp": "1466096874",
          "feeds*entity*id": "13"
        }
      ]
    }
  },
  "my_query1": {
    "help": "Search a datastore table. :param resource_id: id or alias of the data that is going to be selected.",
    "success": true,
    "result": {
      "fields": [
        {
          "id": "date",
          "type": "datetime"
        },
        {
          "id": "price",
          "type": "float"
        },
        {
          "id": "state_id",
          "type": "int"
        }
      ],
      "resource_id": {
        "gold_prices": "d3c099c6-1340-4ee5-b030-8faf22b4b424"
      },
      "limit": 1,
      "total": 748,
      "records": [
        {
          "date": "1950-01-01",
          "price": "34.73",
          "state_id": "1",
          "feeds*flatstore_entry*id": "1",
          "timestamp": "1466036208",
          "feeds*entity*id": "12"
        }
      ]
    }
  }
}7

Response formats

Requests can be sent over HTTP. Data can be returned as JSON, XML, or JSONP. To retrieve data in a different format, change the extension in the url.

Instead of using this:

http://EXAMPLE.COM/api/action/datastore/search.json

Use this:

http://EXAMPLE.COM/api/action/datastore/search.xml

Or this:

http://EXAMPLE.COM/api/action/datastore/search.jsonp

Limitations

  • The q parameter doesn’t work in combination with the join parameter.
  • Filters don’t work with float (decimals) values

Examples

The following is a simple example with two resources that contain four records each. Note that the resource id would be a UUID not single digit number in real scenario.

Resource 1:

country population id timestamp
US 315,209,000 1 1359062329
CA 35,002,447 2 1359062329
AR 40,117,096 3 1359062329
JP 127,520,000 4 1359062329

Resource 2:

country squarekm id timestamp
US 9,629,091 1 1359062713
CA 9,984,670 2 1359062713
AR 2,780,400 3 1359062713
JP 377,930 4 1359062713
Simple query example
http://EXAMPLE.COM/api/dataset/search?resource_id=d3c099c6-1340-4ee5-b030-8faf22b4b424&filters[country]=AR,US&fields=country,population,timestamp&sort[country]=asc

Returns the country, population, and timestamp fields for US and AR from dataset 1 sorting by the country in ascending order.

Joins

If you wish to query multiple tables, indicate the table as an array key in the following fields:

http://example.com/api/dataset/search?resource_id[pop]=d3c099c6-1340-4ee5-b030-8faf22b4b424&resource_id[size]=d3c099c6-1340-4ee5-b030-8faf22b4b424&filters[pop][country]=US,AR&join[pop]=country&join[size]=country

Returns the country, population, squarekm and id for “US” and “AR” from datasets 11 and 13.

Caching

GET and POST request are cached by Drupal. The params passed through the request are used to create a cache id to store the data to be retrieved on further requests.

Since Datastore API uses the Drupal cache system under the hood, the Datastore API cache will be cleared at the same time as the rest of the Drupal cache. This coule be when the cache is wiped manually, or when the cache lifetime ends.

All this options can be configured at admin/config/development/performance

CKAN Dataset API

DKAN provides a number of public, read-only APIs that are designed to provide catalog and dataset information as well as updates that allow observers to track and pull in changes. These public APIs are specifically designed to allow CKAN sites to harvest from DKAN based off of the APIs used for the CKAN Harvester.

All the APIs listed on this page are provided via the Open Data Schema Map module.

Open Data APIs

In addition to the Drupal and CKAN-based APIs supplied with DKAN, two major open data standards are supported. Both are supplied by and configurable through the Open Data Schema Map module.

Project Open Data

DKAN provides full support and mapping for U.S. Project Open Data, in both its federal and non-federal variants, with a data.json endpoint. The optional Open Data Federal Extras module is needed for full federal POD compliance. See a demo here.

DCAT-AP

DKAN also provides endpoints and configurable field mappings for DCAT-AP, the application profile for data portals in Europe, developed by the European Commission. DCAT-AP is of course based on the Data Catalog Vocabulary (DCAT), but provides stricter definitions of catalogs, datasets, distributions and other objects. Through Open Data Schema Map, DKAN provides both a catalog endpoint (see demo) and individual RDF endoints for each dataset (see by going to any dataset on http://demo.getdkan.com/ and clicking the “RDF” link on the lefthand sidebar).

Field Comparison

Catalog
Dataset
DKAN Field/Property DCAT-AP property POD property
title dct:title title
body dct:description description
field_tags dcat:keyword keyword
field_license   license
field_author    
field_spatial_geographical_area    
field_spatial_geographical_cover dct:spatial spatial
field_frequency dct:accrualPeriodicity accrualPeriodicity
og_group_ref dct:publisher publisher
field_temporal_coverage dct:temporal temporal
field_granularity    
field_data_dictionary   dataDictionary
field_contact_name dcat:contactPoint.vcard:fn contactPoint
field_contact_email dcat:contactPoint.vcard:hasEmail mbox
field_public_access_level dct:accessRights accessLevel
field_additional_info    
field_resources dcat:distribution distribution
field_related_content   references
uuid dct:identifier identifier
modified_date dct:modified modified
release_date dct:issued issued

The following properties are provided by the Open Data Federal Extras module and have no equivilant in DCAT. They are only relevant to U.S. federal agencies.

DKAN Field/Property POD property
field_odfe_bureau_code bureauCode
field_odfe_data_quality dataQuality
field_odfe_investment_uii primaryITInvestmentUII
field_odfe_program_code programCode
field_odfe_system_of_records systemOfRecords
Resource / Distribution
DKAN Field/Property DCAT-AP property POD property  
  dcat:accessURL accessURL  
  dct:conformsTo conformsTo  
    describedBy  
    describedByType  
body dct:description description  
field_link_remote_file || field_upload dcat:downloadURL downloadURL  
field_format   format  
field_upload:mime dcat:mediaType mediaType  
title title dct:title  

Releases

Release Notes

Release notes here will be identical to the releases kept in the Github repository’s releases section.

DKAN 1.13.3

This is a “patch” release of DKAN, containing bug fixes and minor updates, but adding no new functionality.

Improvements in this release
  • Groups permissions were given better logic. Editors will now become administrators of any group they are added to, giving them permission to edit or moderate any content in that group. Also, Site Manager was given permissions to edit the order of featured groups.
  • We now provide parsing options on the Datastore Fast Import. This means a user can set what delimiters and line terminators are used when importing a CSV file, avoiding some import errors users were experiencing.
  • Some errors were fixed that caused installing DKAN with the browser (using /install.php rather than drush). Also, special characters were removed from default content filenames, fixing an installation bug on Windows systems.
  • The Rules module was patched to prevent a “cache rebuild lock” (see https://www.drupal.org/node/2851567)
  • Several bugs in the Harvest module were fixed:
    • The Site Manager can now use the main dashboard view to initiate harvest actions in bulk
    • Inconsistencies in date fields (Harvest source “modified” vs Drupal’s “modified” dates) were addressed by adding new fields to the dataset content type.
    • A bug preventing the “temporal coverage” field from being harvested was fixed.
  • The restws and media modules were updated to latest versions
  • Several other smaller bug fixes and improvements; see the CHANGELOG for more information.
Special Notes
Changes to Harvester’s date handling

This update changes the way the Harvest module reflects the issued and modified dates of harvested datasets. Project Open Data and most other metadata standards provide an “issued” or “created” date for datasets, as well as a “modified” date. The original release of Harvester simply overwrote Drupal’s created and modified node properties with the source’s dates for these fields, but we’ve run into two problems with this:

  1. A node’s modified date is easily overwritten by other actions in Drupal
  2. It can be useful to store both the date that a dataset was issued in its source, and the date it was added to the portal harvesting it.

An existing field, field_modified_source_date, was already handeling some of this but we decided to scrap that and start from scratch. Starting with this release, a source’s issued and modified dates will be stored in the new fields field_harvest_source_issued and field_harvest_source_modified. When these two fields are present, those will be the dates shown on a dataset’s landing page on DKAN and in the site’s data.json and DCAT RDF feeds.

All sources should be re-harvested after updating to this patch release to ensure that all date fields and properties are accurate.

DKAN 1.13.2 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates, but adding no new functionality.

See full 1.13 release notes here.

Improvements in this release
  • Updated documentation.
  • DKAN Datastore: fixed null values being imported as 0.
  • Fixed errors related to the open_data_schema_apis_features_rebuild() function.
  • Fixed errors when users with content creator role are editing or adding resources.
  • Fixed access to the POD validation screen for site managers.
  • Fixed access to the featured groups block for anonymous users.
  • DKAN Harvest: fixed permissions for the site manager role, they now have access to the cache, delete, harvest, and migrate bulk operations from the Harvest Dashboard.
  • DKAN Harvest: added support for importing contact name and contact email.
  • Updated contrib modules: services, visualization_entity and views.

DKAN 1.13.1 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 1.13, but adding no new functionality. It was released very shortly after 1.13 to address bugs that surfaced during deployments and upgrades. If you have not yet upgraded to 1.13, upgrade directly to this release and skip 1.13; if you have already upgraded, we recommend updating to 1.13.1 immediately.

See full 1.13 release notes here.

Improvements in this release
  • Fix validation page permission check using wrong permission name odsm.
  • Fixed a bug in the home page conversion function
  • Fixed the page title (<head><title>) so that it’s just the site name (not the node or panel title)
  • Updated the DKAN API link on dataset pages to use the new documentation site page.
  • Fixed error messages appearing on homepage after upgrade
  • Fixed panelizer permissions to hide the “Customize Display” button for Site Managers.
  • Additional minor bug fixes to code and tests

DKAN 1.13 Release notes

Read up on our latest release, 1.13! This is a huge release and we have new features and enhancements that will make using DKAN better than ever. Major new features include a Harvester, API improvements, new data previews, and DCAT-AP support.

Have questions or thoughts? Let us know on our public DKAN Repo in the issues queue or chat with us in our Gitter room.

What’s new
DKAN Harvest

DKAN’s new Harvest module gives DKAN administrators and site managers the ability to harvest datasets from other sources for their DKAN site. Built on the widely-used Drupal Migrate system, the Harvest module can fetch data and publish it to a DKAN site.

DKAN Harvest is fully configurable via the DKAN web interface. Harvest sources can be added like any other type of content, and data can be overridden or manipulated before being published. Migrations can be managed via an intuitive dashboard interface or triggered with a cron job.

Get more details on DKAN Harvest by reading its documentation pages.

DKAN Dataset REST API

Previous DKAN APIs have been read-only and focused on CKAN compatibility. A new RESTful API for DKAN datasets provides an endpoint that allows other applications to interact with DKAN, including creating and modifying datasets and resources.

Get more details on the Dataset REST API.

DCAT-AP Compliance

DKAN’s DCAT RDF publishing has been re-engineered to be more consistent with its other open-data standards implementation, and, more importantly, to implement the new DCAT-AP protocol. DKAN’s fields can be easily mapped to DCAT-AP fields using the Open Data Schema Mapper. Now, DKAN meets US standards (Project Open Data) as well as European Union standards (DCAT-AP).

New Default Content

On a basic install of DKAN, the site came with only a few example Datasets and Resources. We added more default content in order to demonstrate the full capabilities on DKAN and give new users a better experience in testing out various DKAN features. The default content module is part of core DKAN, so all new installs of DKAN contain the content. Simply disabling the “DKAN Default Content” module will remove all the default content to give you a blank slate to start building.

What we’ve improved

The functionality isn’t new, but it’s working even better than before. These improvements make DKAN better on the whole including user experience, efficiency, and scale.

Site Managers can assign roles

The Site Manager role is part of a core group of user roles on DKAN. Site Managers have a high degree of control over the entire site, but are expected to be less technical than a full admin. Previously, it wasn’t possible for Site Managers to assign roles to new users. By adding the RoleAssign module we have provided this more fine-grained permission.

Site Managers can add pages

One of the most basic content types on DKAN is a page. And though the content type is straightforward it can have implications for the structure and appearance of the site. Originally, Site Managers were not able to add or manage Pages; in DKAN 1.13 Site Managers now have this permission.

Even more permissions for Site Managers
  • Theme Settings: Includes page elements and ability to add custom logo and favicon from “Site Configuration > Theme Settings” at /admin/appearance/settings
  • Colorizer: Ability to create custom color schemes for a DKAN site from “Site Configuration > Colorizer” at /admin/appearance/colorizer
  • Open Data Schema Mapper (ODSM): Ability to add, edit, and delete APIs and their mappings to DKAN fields from “Site Configuration > Open Data Schema Mapper” at /admin/config/services/odsm
  • Menus: Permission to manage the main menu links by adding, editing, and deleting links from “Site Configuration > Menu” at /admin/structure/menu
  • Topics Icons: When adding or editing terms in the Topics taxonomy at path /admin/structure/taxonomy/dkan_topics, site managers can choose an “Icon Type” of either “Font Icon” or “Image.” If they choose “Font Icon,” the edit form displays a list of available icons from which to choose. If they choose “Image,” the form lets them upload an image.
  • Enabling External Previews: Site Managers can enable previews so that site visitors can look at Resource contents with visualization tools, Carto and ArcGIS. To enable External Previews, go “DKAN > Data Previews” at /admin/dkan/dataset_preview
  • DCAT and POD validation: Site Managers can select the settings for how Datasets are validated against Project Open Data and DCAT-AP standards.
  • Site managers can confirm that the website’s data.json feed is working correctly by using the POD online validator at https://labs.data.gov/dashboard/validate
  • Site managers can confirm that the website’s DCAT feed is working correctly from Site Configuration > Open Data Schema Mapper > DCAT Validation at /admin/config/services/odsm/validate/dcat or by using the DCAT online validator at http://www.dcat.be/validator/
  • Order of Groups if using the featured groups block: Site Managers can arrange the order in which Groups appear in the featured group block from “DKAN > Featured Groups Sort Order” at /admin/dkan/featured-groups-sort-order
  • Recline size configuration: Manage size constraints for Recline, which powers internal previews, from “DKAN > Recline Configuration” at /admin/dkan/recline
Open Data Federal Extras (ODFE) is now part of core DKAN

Project Open Data (POD) sets a standard for the information about datasets (metadata) included when the data is published. DKAN collects metadata with fields on the Dataset form when the data is published. For most agencies, the fields reflect basic requirements. However US federal agencies are required to provide additional information about the data published. The extra fields appear as part of the Dataset form with our ODFE module.

We’ve moved this module into core DKAN so that it’s part of every install, though not enabled by default. When enabled, the ODFE module makes federal agencies compliant with POD standards.

Get more details on Federal Extras on DKAN.

In-place Editor for Site Managers + look and feel

Previously, the In-place Editor (IPE, an interface for changing page layouts) for Site Managers included several options that weren’t appropriate for their level of permissions. This UI was confusing, so we improved the interface to only include options that are relevant to non-admin user roles. We also improved the overall look and feel of the IPE to be more user-friendly.

Improved Data Previews

The Data Previews feature is designed to let site visitors take a sneak peek at the content of a Resource before downloading the file. In this release, we’ve improved Data Previews to support more file formats beyond CSV and XLS. Now previews also support JSON, geojson, XML, ArcGIS REST, WMS, images, PDF, and ZIP data formats.

Improvements to Data Previews also offer better support for Resources that are hosted remotely. Previously, Resources that were linked to a web source could only be previewed if they were first imported into the DKAN Datastore (which only supported CSV files). In DKAN 1.13, linked Resources can be previewed with the Data Previews feature.

Improved Datastore API

The DKAN Datastore API makes it possible to query for contents of Resources uploaded to the Datastore as detailed as a cell within a table. The improved Datastore API is enhanced to open greater possibilities of requesting complex queries from contents within a Datastore on a DKAN site as well as multiple queries in a single request.

Get more details on the Datastore API

New Dataset fields added

A number of metadata fields that are common requirements for open data standards like Project Open Data and DCAT-AP have been added to the basic dataset content type in DKAN. These fields are:

  • Data Standard
  • Data Dictionary Type
  • Homepage URL
  • Data Standard (conformsTo)
  • Rights
  • Language

The following fields are also included but hidden. This allows the harvester to ingest metadata with these fields and present them in data.json but they are not yet supported in the UI for locally-created datasets:

See the Project Open Data schema for more information on these fields.

NuBoot Radix now in core DKAN

Previously, the NuBoot Radix theme on DKAN was not part of the core repository. The old Omega-based theme and its tools (the Delta module) have been removed, and NuBoot Radix is no longer maintained in a separate repository. Legacy themes will not be maintained or shipped with DKAN and require manual installation if you want to use the Omega theme. Alternatively, you can make customizations to the default theme.

Additionally, the admin theme setting is set to use the default theme rather than the NuBoot Radix theme. Because the Colorizer tool requires both the admin and default to be the same, setting the admin theme setting to use the default theme enables Colorizer to work normally for both the NuBoot subtheme and other custom subthemes without admins needing to manually override this setting.

Added POD-based validation on Datasets

A new option was added on DKAN Dataset Forms at /admin/dkan/dataset_forms in order to enable form validation based on Project Open Data. If “Validate dataset form according to Project Open Data” is enabled, then all POD required fields will be also required in DKAN. See the POD schema for details of the validation standards: https://project-open-data.cio.gov/v1.1/schema/

Minor improvements
  • We added more frequency update options to the Dataset form to comply with Project Open Data standards
  • Added icons to the Topics drop-down menu.
  • Renamed the default HTML text format from “HTML” to “Markdown HTML.” This is also reflected in the UI when adding new content.
  • Made URLs in the Additional info section of Datasets display properly and made them clickable
  • Removed the Conditional Fields module from DKAN, which was adding bloat and creating lots of unwanted artifacts in features, and only used by two fields. A few lines of Form API code was enough to reproduce the functionality.
  • Several improvements were made to enhance accessibility and 508 compliance on DKAN including labels, alt text in the UI, table headers and more.
  • The group field has been simplified in the dataset node form and removed from the resource node form (resources always inherit their datasets‘ groups.)
What we fixed

Some DKAN features weren’t working as expected and causing issues. We fixed those, so you can keep using DKAN as expected.

  • Data proxy for Google spreadsheets was broken. This meant that the data in a Resource could not be parsed properly, and these files could not be used to generate Charts with the DKAN Visualization Entity. We fixed this so Resources that are not internal to DKAN can be used to create a Chart on DKAN.
  • The DKAN API endpoints for JSON and RDF outputs did not generate the proper file when called. We fixed this so that the JSON endpoint works normally. With our new DCAT-AP compliance feature, the RDF output will generate an XML file.
  • Several bugs in the Visualization entity resulted in a frustrating user experience when attempting to create a Chart. A number of fixes were made to result in the expected behavior of the Chart.
  • Only certain roles had permission to use the “Markdown HTML” text format, which resulted in many users unable to edit text to content which had been created using that format. We fixed that so now all roles can add text to a text box regardless of the text format.
  • The toolbar that appears at the top of the text area when using the “Markdown” text format used to include some buttons for styling options that DKAN did not actually support, which created confusion. We’ve removed the unsupported buttons from the toolbar so it only contains supported options (headings, bold, italic, lists, blockquote, link, image, line break).
Upgrading to the latest version
Removing EVA module

WARNING: The EVA is removed with this release. If you are using EVA in your project you need to add it to your sites/all/modules folder.

One view previously used EVA, user_profile_search. You can avoid an error by reverting the feature dkan_sitewide_user. Note that we recommend reverting all features drush fra -y for the upgrade.

Features Reverts

The following commands should be run for the upgrade:

drush rr
drush fr dkan_dataset_content_types -y
drush fr dkan_permissions -y
If ODFE is enabled then: ahoy drush fr open_data_federal_extras -y
If ODFE is enabled then: rm -rf sites/all/modules/open_data_federal_extras
drush fr open_data_schema_map_dkan -y
drush updatedb -y
drush en dkan_ipe -y
drush en dkan_harvest_dashboard -y
drush en menu_admin_per_menu -y
drush fra -y
drush rr
New Documentation location

We have moved DKAN’s technical documentation from the Drupal-based site where it used to live to the /docs folder within the main DKAN repo. Documentation changes will now be part of code commits and pull requests. The documentation is built on ReadTheDocs, but lives at the same URL as the old site: http://docs.getdkan.com/

Special Notes

Front Page - Upgrade Instructions

Front page configuration has been removed from features and the dkan_sitewide_demo_front feature has been deprecated. To save existing front page configuration, run the following command drush php-eval "dkan_sitewide_convert_panel_page('<page-name>', TRUE);" This will convert the front page to a panelized node.

Also note that if you use any of the default DKAN blocks for your front page, some may dissapear after the upgrade. A user with the administrator role should be able to restore them by editing the panel and finding them in the Miscellaneous list.

Added Modules

  • DKAN Harvest
  • DKAN Harvest Dashboard
  • DKAN IPE
  • Menu Admin per Menu

Removed Modules - The following modules were removed from the code base and upgrading will remove the files and db tables associated with them.

  • Conditional Fields
  • Entity RDF
  • RDF UI
  • RDF Extensions
  • Delta
  • EVA

Removed Themes

  • Omega

Consolidation - External repos have been merged into DKAN core. This means that dkan_dataset, dkan_workflow, dkan_datastore and nuboot_radix are all included in the main dkan profile repo.

To remove newly untracked files run:

git clean -f

Known issues
  • Site managers do not have permission to see the Project Open Data validator screen, which tells them if their site is currently producing a valid data.json. This will be fixed in 1.13.1. The equivalant screen is availabe for DCAT-AP, under the configuration menu. If you need to check your site’s data.json feed in the meantime, you can use Data.gov’s online POD validator at https://labs.data.gov/dashboard/validate.

DKAN 1.12.14 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Added css for chart visualizaitons, fixes issue in IE.
  • Updates drupal core to 7.53
  • Upgrade visualization_entity to 7.x-1.1
  • Upgrade services module (via datastore module) to 3.19

DKAN 1.12.13 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Fixed broken recline (resource CSV) preview embeds caused by how recline module loads bootstrap.
  • Added patch to remote_stream_wrapper to avoid memory exhausted errors on big files.
  • Fixed publisher token in open_data_schema_map_dkan. It was showing only URL rather than publisher name.
  • Remote linked files are now proxied through a DKAN site so that we don’t run into CORS issues for the visualizations. That can cause memory issues with large files. This only proxies files smaller than 50MB.

DKAN 1.12.12 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Group assignment cleanup update

This release contains a database update that may take some time to complete on larger sites. As described in #1267, on older DKAN sites it was very easy to add resources to datasets that would not be added to the parent dataset’s group.

Many sites may have this issue and not even realize, but it will create problems for organizations that use DKAN’s out-of-box group roles and permissions; it give users permission to edit content in their own group but not content outside of it. If a resource is not assigned to a group, this could block a group user from editing their groups’ own datasets.

This patch release contains a database update that will identify datasets with resources missing their group assignments, and correct them. See pull request #1491 for the fix itself.

Other improvements in this release
  • Data previews can be captured in a certain state for embedding into other web pages as an iframe, but these previews were not cached by Drupal. This has been fixed; embedded previews are now cached, which is a significant performance boost for some websites.
  • Upgraded Drupal core to 7.52, ctools to 1.11, and Media to 2.0-beta13.
  • Fixed links to group pages from group node teasers that broke when site has clean urls disabled
  • Fixed some issues with certain filters/facets on the search page being unexpectedly collapsed.
  • Fixed a bug introduced on a previous patch release that hid the body field on page nodes.
  • Improved the access check and security on datastore pages - unauthorized users could perform certain datastore functions by guessing URLs.
  • Fixed “page not found” errors when clicking certain topics links with spaces or special characters in them.

DKAN 1.12.11 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Fixed a bug in Recline regarding file objects and node forms that caused errors when using the “view changes” button on Dataset or Resource edit form
  • Center group images in “Groups” page and group node page sidebar
  • Add “2x,” “3x” etc to Dataset teasers when more than one resource of a particular format present.
  • Update the default jquery library setting from 1.7 to 1.10
  • Fixed topics menu and facet links if special characters are used in topic terms.
  • Fixed dataset form redirect when validation fails, was sending user to node/add/dataset rather than node/%/edit
  • Patch fontyourface to remove from the “standard text” selector, to prevent unpredictable results from this option
  • Fixed issue in open_data_schema_map with “Data Dictionary” field not displaying URL in data.json file

DKAN 1.12.10 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Fixes an issue causing JavaScript errors in IE browsers, preventing recline previews from displaying. Bug was introduced in 7.x-1.12.7.
  • Provides a better upgrade path for Markdown text format and bueditor improvements added in #1085

DKAN 1.12.9 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Fix to a bug introduced in the Recline module in 1.12.7. Previews were not displaying if the resource file was smaller than max preview size, even if loading the resource from datastore.
  • Fix a problem with paths for image icons for topics. Uploaded image icons will now display correctly in the Topics menu.
  • Fixed the “data dictionary” field to use HTML input format. This field, at least in U.S. Project Open Data, is really intended to have only URL values, and using HTML ensures that links will be clickable. Also made improvements to the field’s display and help text.
  • Updated the DKAN Workflow content type legend to include all content types with correct icons. This is mostly to prepare for future content types’ inclusion in Workflow but also to support sites that already have additional content types in Workflow.

DKAN 1.12.8 Release Notes

Special note: This release contains security updates and should be applied immediately.

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release

DKAN 1.12.7 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Updates contrib modules (uuid, services, manualcrop, markdown, panelizer, panopoly_widgets and panopoly_images).
  • Fixes a bug that produced a “Cannot use a scalar value” error when users trying to add a content pane to a data story node in panels.
  • Changes theming on resources to remove iframe to “API or link URL” if remote file field also populated. The only real use-case this applies to is when harvesting datasets from a data.json that includes both an accessURL and a downloadURL for a single distribution; both these fields should not be used at once on resouces created using the node form.
  • Updates the options in the Dataset “frequency” field to reflect ISO 8601 standards for DCAT and Project Open Data
  • Provides a simple file proxy to serve remote CSVs through local domain and resolve cross-origin issues with previews n these resources
  • Somes changes to the Resource node form to improve the UX of links and file attachments, especially by replacing the label “Link to an API” to “API or Website URL.”
  • Fix bug on relative paths for links in theme template files, which caused some links to break when DKAN lives at a subdirectory rather than a website’s domain root.
  • Fixes a minor typo in the DKAN topics menu link that was exposed by the XSS fix in the 7.x-1.12.6 release.

DKAN 1.12.6 Release Notes

Special note: This release contains security updates and should be applied immediately.

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Adds validation to search parameters. Without this, text can be passed into search parameters and passed unsanitized to the browser. The way the facets are themed allowed Javascript to be injected into the attributes of any facet items with icons, (such as content types). The fix validates all facet input and returns a “page not found” error if a search parameter passed in the URL arguments does not match available options. See #1271
  • Starts sanitizing output to the facets on the search page, as an additional safeguard against malicious input from the search parameters.
  • Adds icons to the topics drop-down menu rather than just the title of the topic.
  • Added a new module called Role Assign, which gives site managers the ability to assign roles to other users without giving them access to the entire Permissions module, which meant previously only admins could assign roles.
  • Made links and emails in dataset metadata clickable in certain places that they hadn’t been for a better user experience.

DKAN 1.12.5 Release Notes

Special note: This release contains security updates and should be applied immediately.

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
Special Note

This is a fairly big update to Drupal and you may want to familiarize yourself with it by reading the release announcement.

Of note to DKAN users is that “Administer fields” is now a separate permission. This means, for instance that site managers and editors, who by default have “administer taxonomy vocabularies” permissions, will no longer be able to add, remove, reorder or otherwise modify fields in a taxonomy vocabulary.

Only administrators will be able to do administer fields. We see this as a welcome improvement, as making minor changes to, say, the topics vocabulary should not require giving a site manager permission to alter the entire data architecture.

DKAN 1.12.4 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Fixes an oversight that broke RDF endpoints for Datasets (the “RDF” link in the left sidebar of a dataset page). The DKAN Permissions module did not give the “Access the resource node” RESTFull Web Services permission to anonymous users.
Upgrade steps

Only necessary if you are using DKAN Permissions module:

$ drush features-revert dkan_permissions

DKAN 1.12.3 Release Notes

Special note: This release contains security updates and should be applied immediately.

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • Open Data Schema Map update: Improves performance of package_show endpoint, which became nearly unresponsive on sites with thousands of datasets.
  • Point to new release of Visualization Entity, which cleans up some makefiles and brings visualization_entity_charts into the same project repository.
  • Fix editor permissions to allow access to visualizations list from admin menu
  • Upgrade Views to 7.14
  • Upgrade Drupal core to 7.44 (security update)
  • Add <br>, <h2>. <h3>, <center>, <iframe> to allowed html tags in “Martkdown HTML” format
  • Renamed the default HTML text format to “Markdown HTML”
  • Improved markdown text editor toolbar
  • Several improvements aimed at removing unstable configuration like menus from features:
    • Stories main menu link removed from stories view
    • Groups main menu link removed from page manager groups page config
    • Main menu links added on install function rather than through features
  • Add body field to Dashboard content type
  • Front page page manager (panels) config moved to the dkan_sitewide_demo_front feature
  • Front page group views moved to the dkan_dataset_groups feature
  • Add functions to convert between iso and dkan fields #241
Upgrade steps

$ drush fr dkan_dataset_content_types -y

DKAN 1.12.2 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release

DKAN 1.12.1 Release Notes

This is a “patch” release of DKAN, containing bug fixes and minor updates to DKAN 7.x-1.12, but adding no new functionality. Upgrading should be straightforward.

Improvements in this release
  • New version of Visualization Entity Charts fixes a javascript error that was causing multiple UI bugs, and restores support for Google Sheets and Data Proxy data backends
  • Fix XSS vulnerability by adding sanitization for titles on workbench view
  • Update of a number of contrib modules: Colorizer, Admin Menu Source, Honey Pot, Panopoly Widgets, Panopoly Images, Pathauto, Rules, Restws, Manual Crop, Image Cache Actions, Features, Search API, Organic Groups, Chosen, Date, Entity, Facet API, Facet API Bonus, Facet API Pretty Paths, UUID, Views, Select or other, Remote Stream Wrapper, Link, Libraries, Beauty Tips, Gravatar, OG Extras, Services and Workbench Email.
  • Patch colorfield module to avoid incorrect status error. See https://www.drupal.org/node/2696505

DKAN 1.12 Release Notes

Check out what’s new in the latest version of DKAN 7.x-1.12! Have questions or thoughts? Let us know on our public DKAN Repo with issues or chat with us in our Gitter room.

New Features and Enhancements
New DKAN Workflow Module

DKAN Workflow is a new module that ships with DKAN but is not enabled by default. Once the module is enabled, DKAN Workflow creates an editorial publishing workflow on DKAN for moderating content. Within this module is a secondary set of permissions and roles layered on core DKAN roles and permissions. This module can be very useful to establish more controls among large teams managing content.

Read more about Workflow in the main DKAN Documentation.

Check out the blog post: http://www.nucivic.com/new-feature-alert-dkan-workflow/

Large File Support and Fast Import

Through the file_resup module we’ve added the ability to support large file uploads on DKAN. There are no limits technically imposed by the software, but file sizes limited up to few gigabytes will produce the best results. For large files, we also added a mySQL statement to quickly import the files into the DKAN datastore. We’re calling this DKAN Data Datastore Fast Import module and it appears as such in the module configuration on DKAN. The module is disabled by default, and must be enabled and then configured to be utilized in DKAN. OOB DKAN defaults to using the Drupal file importer that also processes the data (better for smaller files). In configuration, the mySQL statement may replace the Drupal file importer or appear as an additional option for importing a file into the DKAN Datastore. Rather than processing the data, the mySQL statement favors a faster load time by importing directly to a mySQL database.

Large file support also makes for a more robust DKAN Datastore API. Files imported into the DKAN Datastore are included in the public DKAN Datastore API.

Read more details about the DKAN Datastore and the Datastore API on our blog post.

Topics

DKAN Topics is a new module added into DKAN core with the 7.x-1.12 release. Topics are preset by an administrator that reflect the areas of interest to the target audience. Topics are configured by default to appear as a panel on the homepage of a DKAN site.

In DKAN, Topics are a taxonomy with a vocabulary (also set by the admin) and can be associated to datasets by lower-access users as they contribute data to DKAN. Additionally admins can set which icons may or may not be used within a font set for Topics by other users. OOB DKAN includes a font set of over 100 icons to choose from, and new font sets can easily be added. Alternatively, images may be uploaded for Topics icons.

Read more details about Topics on our blog post.

Minor Upgrades and Improvements
  • Changed UI of DKAN Charts to include “Tick Values” “Step” and “Goal” line in chart. This gives users added functionality in what they can do with Charts. The UI is also more organized to tell users how the options affect which parts of the Charts.
  • Upgrade Drupal core to 7.43 to fix security vulnerabilities.
  • Security update for Features
  • Security update for Fieldable Panels Panes
  • Move permissions dependency of from features_roles_permissions to dkan_permissions module.
  • Remove export of administrator permissions when new modules are enabled, so that dkan_permissions does not appear as overridden. With the admin_role module, admin roles automatically receive permissions for modules as they are enabled. By removing the export of administrator permissions, overrides will only appear when true changes to the dkan_permissions module have been made.
User experience improvements
  • Enable Pretty Paths Facet API to make URL paths on DKAN pages human-readable and SEO optimized. This module is combined with the Drupal Search API module.
  • With the new search page redesign in DKAN 7.x-1.11, this release includes optimized search functionality. Previously only datasets were displayed in search results; in version 1.12 all content appears in search results. The /search path will lead to the main search page and the /dataset page will redirect to the /search?type=dataset for backward compatibility, pre-filtered for dataset content search results.
  • The dkan_sitewide_profile_page module was removed and new code added to existing modules (detailed documentation as code comments). With the removal of this module, the DKAN command center no longer exists and profiles of any user role appear the same.
  • A “Data dashboard” creation link was added on user admin menu.
  • Facet titles between user, search and group pages were standarized.
Patches and Bug Fixes
  • Added permission to allow anonymous users to see any piece of content created with the visualization entity module such charts and maps.
  • Created patch to solve bug that kept fonts enabled after being turned off.
  • Fixed regex used to rewrite font file paths on Dkan Topics, which was breaking filepaths in data.json.
Upgrade Notes

If you upgrade an existing site to 7.x-1.12 and enable DKAN Topics, you may encounter issues with your search index. For troubleshooting tips, see the DKAN Topics Readme.

Upgrade instructions
  1. drush updatedb -y
  2. drush en dkan_featured_topics -y
  3. Enable ‘field_topics’ on dataset index (admin/config/search/search_api/index/datasets/fields)
  4. drush fr dkan_sitewide_search_db -y
  5. drush sapi-c datasets
  6. drush sapi-r datasets
  7. drush sapi-i datasets
  8. drush fra -y
  9. Remove the ‘Add Dataset’ link from main menu
  10. Remove the ‘Datasets’ link from main menu that point to the old /dataset url for search
  11. drush rr

DKAN 1.11 Release Notes

Check out what’s new in the latest version of DKAN 7.x-1.11! Have questions or thoughts? Let us know on our public DKAN Repo with issues or chat with us in our Gitter room.

New Features and Enhancements
New roles and permissions module

We made a new module (“DKAN Permissions”) to replace the previous “Sitewide Roles and Permissions” module. The previous feature largely left it to the administrator to determine user roles and what each role could do. While administrators will still have the ability to customize user roles and permissions, we wanted to make it easier to use DKAN right after installation. We have a detailed post about the standard roles in this new feature, DKAN roles and permissions made simple. Read up on how this standard DKAN feature makes user management easier out-of-the-box.

Group roles and permissions in a new module

The old “Sitewide Roles and Permissions” module also included group roles and permissions. As the new “DKAN Permissions” replaces the old module, groups were also updated. The capabilities of groups, the roles and permissions within, and how groups interact with the rest of a DKAN site have been moved to a new module, “DKAN Dataset Group Perms”. You can read more detailed user-documentation about Group roles and permissions as part of DKAN roles and permissions made simple.

Re-designed dataset/search page

In this version of DKAN, we improved the layout and visual appearance of the datasets page to make it easier for site visitors to browse, search and find content. In previous versions, the search box on the homepage was the only way for site visitors to search all content types. The re-designed datasets page directs site visitors to a page with results pre-selected to filter the datasets content type. Site visitors have the option to add and remove filters on the datasets page in order to browse all content types as well use the search bar at the top of the page to find specific content.

Minor Upgrades and Improvements
User experience improvements
  • Dataset page layout: The dataset page now has a cleaner layout and overall better visual design that is a more intuitive user experience. We also added icons for content types that are standard across the site, so that site visitors can quickly recognize what type of content they are clicking on.
  • User profile image UX: In this release we removed the confusion in the user experience of changing a user’s profile image including the existing Gravatar integration.
  • Groups view: We changed the default number of groups displayed on a page from 10 to 12. This keeps the layout of groups evenly separated in neat rows of 3.
Patches and Bug Fixes
  • Colorizer feature bug fix: fix to the colorizer feature on DKAN so that style changes don’t disappear after running the Poormanscron Drupal module.
  • Group node page bug fix: a quick fix to remove an extra page on the Groups page.
  • Warning message bug fix: a quick fix to remove redundant error messages for admins upon installing DKAN.
  • Fontyourface module patch: makes it easier to see fonts changes on your DKAN site by updating as soon as changes saved.
  • FacetAPI module patch: patched a potential security breach that appeared through cross-site scripting.
Special Notes
Permissions/roles upgrade

As mentioned above, this release includes a new module called DKAN Permissions (dkan_permissions), which is meant to replace the old DKAN Sitewide Roles and Permissions (dkan_sitewide_roles_perms). New installs of DKAN will enable this module and ignore the old one, while existing sites will see no change (but are recommended to upgrade). See more information in the module’s README file.

The command line method, including deleting the now-deprecated “storyteller” role, would look like this:

$ drush dis dkan_sitewide_roles_perms -y
$ drush rdel 'storyteller' 
$ drush en dkan_permissions -y
$ drush fra -y
$ drush fr dkan_permissions -y

Group-level permissions have been moved to DKAN Dataset Groups Permissions. To start using them, enable the new module, and revert it using Features. Do this via the UI or on the command line with Drush:

$ drush en dkan_dataset_groups_perms -y
$ drush fr dkan_dataset_groups_perms -y
Avoiding PHP errors after upgrade

In some cases, changes to the codebase may cause PHP errors when the Drupal bootstrap process looks for a file that no longer exists. As always, backing up your db before upgrading is recommended. In addition, if you get to a state in which Drupal will not bootstrap successfully due to an issue with views_autocomplete_filters, try some combination of:

$ drush sql-query 'DELETE FROM registry WHERE filename LIKE "%views_autocomplete_filters_handler_filter_string.inc%"';
$ drush cc all
$ drush rr
Search page redesign

This release of DKAN includes a redesigned search page, which is also the page used to browse datasets. This redesign required some changes to the included search indexes. Use the following drush commands to rebuild your search indexes, or go to admin/config/search/search_api/index/datasets and disable, re-enable and re-index the datasets search index.

$ drush search-api-disable datasets -y 
$ drush search-api-enable datasets -y
$ drush sapi-r -y
$ drush sapi-i -y
DKAN Dataset

See the DKAN Dataset release notes for 7.x-1.11 for notes specific to the DKAN Dataset module.

DKAN 1.10 Release Notes

Check out what’s new in the latest version of DKAN 7.x-1.10! Have questions or thoughts? Let us know on our public DKAN Repo with issues or chat with us in our Gitter room.

DKAN Distribution
  • A number of improvements to our test infrastructure
  • Improved user experience for user photos and Gravatar image fallback
  • Improvements to views on group pages
  • Enabled and improved UX of tools for adding existing visualizations directly into panels layouts
  • Fixed an extremely annoying bug in the Colorizer module that blew away colorizer CSS every time Drupal cron ran (sites using external/system cron were unaffected)
  • Added better HTTPS by loading certain external images over HTTPS
  • Upgraded to Drupal version 7.x-1.41
  • Added a CONTRIBUTING.md file to provide community contribution guidelines for DKAN project.
DKAN Dataset Module

https://github.com/NuCivic/dkan_dataset

  • Various improvements to dataset teaser displays.
  • Support for external previews (opening catalogued resources instantly in third-party visualization services, * including ArcGIS and CartoDB)
  • Support for Flaticon at module level, so vector icons work on any theme
  • Minor improvements and bugfixes
NuBoot Radix Theme

https://github.com/NuCivic/nuboot_radix

  • Style improvements for “open with” button
  • Fix default logo path when svg not available

DKAN 1.9 Release Notes

Check out what’s new in the latest version of DKAN 7.x-1.9! Have questions or thoughts? Let us know on our public DKAN Repo with issues or chat with us in our Gitter room.

New Features and Enhancements
  • Added “Data Dashboards” and “Data Stories” content types with customizable layouts
  • Added Panopoly Widgets for use in Dashboards, Stories, the front page and panel pages
  • Added new custom DKAN widgets for use on panel pages
  • The Visualization Entity module, along with the additional Charts bundle, are now included in DKAN core
  • Added a “command center” to the user’s own profile page to more easily find common functions
  • @font-your-face module added w/configuration for easier font admin
  • Multiple security updates and other contrib module and theme updates.
  • Drupal now on version 7.39.
  • Many bug fixes and code cleanups
  • See release notes for nuboot_radix theme and individual DKAN modules for additional release notes
Search API Database upgrade issue

This release updates the Search API DB module, which, if you are using the database backend, may break your search indexes, resulting in either a) nothing, or b) all nodes regardless of type on your search/datasets pages.

To fix, follow these steps after updating your database:

  1. Go to /admin/config/search/search_api/index/datasets/edit to edit the database node index
  2. Set the Server to “< No Server >” and Save. Your index will now show as “disabled.”
  3. Edit it again (by clicking “Edit” next to “Database Node Index”) and set the Server back to “Database Server” and Save.
  4. Click the “enable” link while viewing the resulting page, or edit the index again and check the “Enabled” box.
  5. Once the index is enabled, you should see “0/[total] Indexed” on the index page. Click “Index Now” at the bottom of the page to re-index all your datasets.
  6. Repeat for any other indexes you have running on a database server.

This issue has been identified and documented for other Drupal distributions using Search API. See release notes for the Panopoly and OpenAtrium distros on Drupal.org.

DKAN 1.8 Release Notes

Special note: This release contains security updates and should be applied immediately.

Check out what’s new in the latest version of DKAN 7.x-1.8! Have questions or thoughts? Let us know on our public DKAN Repo with issues or chat with us in our Gitter room.

New Features and Enhancements
  • Drupal core update to 7.36
  • Some tweaks to display properly drupal admin pages using nuboot_radix
  • Removed unnecessary drupal warning messages (colorizer, updates, etc)
  • Tweaks to behat tests
  • Security updates for contrib modules

Changelog

7.x-1.13.4

  • #1899 Override default error message on duplicate path aliases with more information.
  • NuCivic/visualization_entity#72 Switch to relative path for uploaded resources on visualization entity charts
  • #1836 Update language codes to use dashes rather than underscores for POD compliance.
  • #1836 Fix the POD and DCAT language output to use key values rather than label values.
  • #1891 Expose publishing options for content creators who do not belong to a group.
  • #1891 Disabled authoring information fields for content creators who do belong to a group.
  • #1901 Upgrade media module to 2.1
  • #1897 Fixed handling of dataset language setting to pass DCAT validation.
  • #1896 Fixed bug on open_data_schema_map: Multiple urls cause data.json describedBy field to be truncated.
  • #1890 Fixed the chosen widget exclusion settings to not apply to group search filters.
  • #1887 Removed unnecessary ‘Select’ button from the remote file input field on the resource form.
  • #1869 Add validation check on the API or Website URL field to require a full url.
  • #1883 Removed required status from the Rights field if the public access level is set to ‘none’.
  • #1871 Better handling of remote resources with BOM.
  • #1873 Fix dkan_workflow_permissions: Allow Workflow Contributors and Workflow Moderators to view stale drafts and Workflow Moderators to view stale reviews.
  • #1858 Fixed default panelizer setting for page node title.
  • #1874 Fix dkan_update_7013 to not use drush commands.
  • #1876 Removed hard coded colors in the menu.scss file.
  • #1825 Adapt the ‘dataQuality’ input value druing POD harvest.
  • #1817 Load empty cells as null in fast import.
  • #1846 Adds a hook_update to exclude the data dictionary field from using the markdown toolbar.
  • #1813 Update the groups page view to sort alphabetically rather than by post date.
  • NuCivic/recline#89 Only load previews for resources using files; API/website links should always display iframe.
  • #1841 Fixed mis-named function on dkan_dataset_content_types.api.php
  • #1814 Update dkan_workflow_permissions to rebuild permissions on enable/disable.
  • #1810 Stop throwing exception on tables with only numeric columns, to prevent preview breaking.
  • #1834 Use last day of year (December 31) for 4 digits end year during temporal POD field import.
  • #1832 Fallback to modified date for field_harvest_source_issued if not available during POD Harvest.
  • #1828 Field Frequency Harvest POD integration is missing support for “irregular” value.
  • #1820 Use “accessURL” during resources harvest if “downloadURL” field is not available.
  • #1857 Fixed publishing options not accessible when dkan_workflow is enabled.

7.x-1.13.3-RC1

  • #1810 Stop throwing exception on tables with only numeric columns, to prevent preview breaking.

7.x-1.13.3 2017-04-18

  • #1863 Update restws module to v2.7
  • #1859 Fixed update hooks to correctly set jquery version and remove old modified source date field
  • #1864 Update media to 2.0 and remove patch 2534724.
  • #1829 Fixed missing properties on warning message during datajson harvest cache.
  • #1802 Better support for Issued and Updated dataset properties from harvested sources.
  • #1821 Remove redundant CSS load in dkan_dataset.
  • #1726 Fixed broken links in search results to nodes without aliases
  • #1792 Remove Windows reserved characters from default content filenames for better Windows support
  • #1815 Support importing Temporal Coverage during POD Harvest.
  • #1785 Fix error messages when installing DKAN with the UI.
  • #1786 Remove horizontal tab field from the group content type, fix margin on group block.
  • #1786 Add hook to auto-assign editors the og administrator role.
  • #1752 Added option for using quote delimiters when fast import is enabled for dkan datastore. Fixed error message when an import fails on DKAN Datastore Fast Import.
  • #1703 Refactor Datastore API module, fixing some caching issues and improving joins
  • #1804 Add project = dkan to dkan_sitewide.info to fix errors on update screen.
  • #1811 Apply patches to the rules module that can prevent unnecessary DB lockouts.

7.x-1.13.2 2017-03-16

  • #1803 Fix broken access to featured groups sort order view.
  • #1796 Fix Harvest support for contact name and contact email.
  • #1795 Update front page test on topics.feature with @customizable.
  • #1783 Update services to 3.19
  • #1794 Update visualization_entity to 1.1
  • #1781 Update views to 3.15 and remove patch 1388684.
  • #1751 Add POD validation link to command center menu for site manager access.
  • #1762 Re-number update hooks that were mixed up during release integration.
  • #1709 Changed function dkan sitewide conversion homepage to fix problem with link entity attribute. This attribute need to be a boolean.
  • #1742 Fix home page HTML <head><title> so that it’s just the site name (not the node or panel title)
  • #1747 Update DKAN API link to use the RTD documentation page.
  • #1730 Fix logic error for front page in theme causing error messages on homepage
  • #1728 Added a tag @defaultHomepage to a topic test which relies on homepage content.
  • #5807 Fix panelizer permissions to hide ‘Customize Display’ button
  • #1744 Grant Site Manager role Harvest Dashboard actions.
  • #1776 Fix 500 errors when linking or uploading geojson files in resources.
  • #1767 Better handling of empty values by datastore_api. Also see NuCivic/feeds_flatstore_processor#9

7.x-1.13

  • #1719 Added site details to settings nuboot_radix to allow change site name, slogan, e-amail address for site manager.
  • #1717 Upgrading Drupal to 7.54
  • #1705 Added css for chart visualizaitons, fixes issue in IE.
  • #1695 Upgrade better_exposed_filters to v3.4.
  • #1702 Update branding.
  • #1369 Added ReadTheDocs integrations and docs folder for centralized documentation
  • #1701 Renamed test files.
  • #1691 #1682 Update resource tests to work with client site variations.
  • #1694 Moved functions from FeatureContext to DKANExtension.
  • #1687 Added a function in dkan_sitewide to check if a specific page is the front page.
  • #1693 Improve module and user cleanup inside dkan workflow context after run tests
  • #1669 Update leaflet library to v1.0.2 and leaflet markercluster to v1.0.0. Unift leaflet libraries (was using a separate version for recline module)
  • #1684 Security update for autocomplete_deluxe
  • #1670 Upgraded to new 1.0 release of Visualization Entity module
  • #1663 Add install hook to DKAN Workflow to force a Features revert of dkan_sitewide_menu, to make sure Workflow links added correctly
  • #1348 Remove Panels IPE from dataset and search pages; in DKAN, we only want to show IPE for panelizer layouts, not templates or other “sitewide” pages.
  • #1046 Update user profile page search to be consistant with the rest of the site and moves user info to sidebar block.
  • #1096 Fixes typo in “add data story” link in command center menu
  • #1069 Add topic icons to the drop down menu
  • #1085 Renamed the default HTML text format to ‘Markdown HTML’
  • #1440 Improved markdown text editor toolbar
  • #1110 URLs on dataset’s additional info are now properly displayed as links.
  • #1130 Removed warnings about undefined permissions when visualization_entity_choropleth_bundle module is enabled.
  • #1130 Removed warnings about undefined permissions when visualization_entity_geojson_bundle module is enabled.
  • #1130 Removed specific visualization entity charts permissions from dkan_extension.
  • #1130 Added test feature to validate that users with Editor, Content Creator and Site Manager roles are not able to access to admin pages.
  • #1259 Default content was improved and now it’s generated based on fixtures.
  • #1152 Removed omega theme.
  • #1152 Removed delta module.
  • #1152 Moved Nuboot Radix theme into core build.
  • #1152 Change admin theme setting to use default theme rather than nuboot radix
  • #1221 Page content type added to panelizer and now defined in features, with in-place editor enabled for page content.
  • #1222 Page link added to the command center menu.
  • #1301 Added Open Data Federal Extras module into DKAN Core.
  • #1297 Fix for relative paths to ensure links still work when a site is installed into a subdirectory.
  • #1358 Removed field mapping warnings during import of default content.
  • #1376 Removed conditional_fields from dkan_topics.
  • #1387 Upgrade Features to 7.x-2.9
  • #1387 Make date facet use “day” granularity.
  • #1387 Remove ARC2 library.
  • #1439 Disable pathauto for content created using dkan_fixtures.
  • #1403 Removed warning message when try edit resource without dataset.
  • #1468 Added PHP Unit tests configurations to make them run on DKAN.
  • #1463 Added hook_uninstall to federal extra module for remove fields after uninstall module.
  • #1459 Removed redundant definition for CKAN package_list endpoint from open_data_schema_map_dkan, allowing that feature to be in default state after install.
  • #1456 Update ctools to version 1.10, which fixes some PHP7 incompatibilities
  • #1440 Added links to ODSM page and DCAT validation under Site Configuration in the command center menu
  • #1367 Update open data schema module to render a valid output passing validation (dcat and rdf).
  • Added upgrade hook to clean DB after removal of ‘conditional_fields’, ‘entity_rdf’, ‘rdfui’ and ‘rdfx’.
  • #1527 Added upgrade hook to remove deprecated test and theme directories.
  • #1537 Fixed popular tags view.
  • #1685 Moved front page search block from dkan_sitewide_demo_front to dkan_sitewide.
  • #1562 Fixed link to group page on group node teaser when site has clean urls disabled.
  • #1534 Added update hook to disable dkan_default_content on upgrades.
  • #1565 Updated visualization_entity.
  • #1556 Added fix to display the ‘Request membership’ link only to users that are logged in.
  • #1582 Remove content padding on data extent and social blocks.
  • #1606 Added DKAN Extension.
  • #1508 Upgrade workbench_moderation to v3.0.
  • #1595 Upgrade entity to 1.8
  • #1598 Upgrade manualcrop to 1.6
  • #1598 Upgrade markdown to 1.5
  • #1598 Upgrade panels to 3.8
  • #1592 Upgrade ctools to 1.12
  • #1592 Upgrade entityreference to 1.2
  • #1592 Upgrade libraries to 2.3
  • #1592 Upgrade services to 3.17
  • #1592 Upgrade simple_gmap to 1.3
  • #1592 Upgrade tablefield to 2.5
  • #1592 Upgrade entityreference_filter to 1.7
  • #1622 Upgrade fieldable_panels_panes to 1.11
  • #1596 Upgrade radix to 3.5
  • #1598 Upgrade search_api to 1.20
  • #1603 Upgrade media to 2.0-beta13
  • #1603 Upgrade panopoly_widgets to 1.41
  • #1603 Upgrade panopoly_images to 1.41
  • #1604 Upgrade file_entity to 2.0-beta3
  • #1605 Upgrade link_iframe_formatter to 1.1
  • #1654 Assign workflow supervisor role to site manager users when dkan workflow is enabled.
  • #1645 Switch to forked version of the spectrum library to address accessibility issues.
  • #1642 Removed eva module.
  • Fixed CSV column validation on visualization entity.
  • Fixed retrieve of version information on chroma.js.
  • #1678 Add patch to fix panopoly_widgets overrides OOB.
DKAN Datastore:
  • #1387 Added DKAN Datastore module into DKAN Core.
  • #1599 Greatly expand the datastore API to support aggregation functions, better joins, multiple queries. See dkan_datastore_api’s README file.
  • #1599 Fixed bugs when running the datastore via cron.
DKAN Harvest:
  • #1676 Harvested content is published even if dkan_workflow is enabled.
  • #1287 Added DKAN Harvest module into DKAN Core.
  • Added UI to add harvest sources
  • Added UI to manage harvest sources and datasets
  • #1446 Fix dkan harvest feature overridden after install.
  • #1440 Add link for site managers to create harvest source in command center menu.
  • #1472 Added Batch API on ‘Harvest now’ action.
  • #1482 Added Batch API on the ‘Preview’ page.
  • #1531 Added ‘Add Source’ shortcut on Harvest Dashboard pages.
  • #1531 Fixed breadcrumb on harvest sources pages: Preview, Manage Datasets, Events, Errors.
  • #1434 Fixed harvesting of resources with remote files with redirects.
  • #1588 Fix “Manage Datastore” tab leaking to the harvest source node.
  • #1588 Add icons for harvest source tabs.
  • #1634 Fixed the dataset count displayed on the harvest preview message.
  • #1640 Added support for –limit, –instruments, –skiphash and –idlist options on drush commands.
  • #1658 Fix 508 compliance errors on admin and node view pages.
  • #1650 Removed links to Harvest Source nodes from main menu.
  • #1672 Prevent harvest source machine name from containing forward slash character.
  • Improved PHP Unit tests.
  • #1468 Add support “Compound” fields on Filters/Overrides/Excludes/Default in DKAN Harvest.
  • #1488 Replace field notes by field body in harvest source content type.
DKAN Migrate Base:
  • #1387 Added DKAN Migrate Base module into DKAN Core.
  • #1462 Removed row from migration map table when a dataset is deleted.
DKAN Workflow:
  • Add patch for workbench_moderation to avoid php shutdown function errors.
  • #1690 Added patch for workbench_moderation: Invalid argument supplied for foreach() 2360973.
  • #1712 Added a filter in the workflow.feature to avoid issues for the amount of already existing nodes.
  • #1707 Isolate dkan workflow tests related to emails.#1069
  • #1715 Moved the update of the roleassign_roles variable from dkan_workflow_permissions to dkan_workflow.
  • #1677 Fixed panels-related bug where, if a dataset had both a “published” and “draft” version, the published would show in the draft tab.
  • #1438 Updated DKAN workflow vbo customizations to not affect other vbo forms.
  • #1481 Updated workbench_email from 3.9 to 3.11
  • #1667 Fixed admin menu source when dkan_workflow is enabled.
  • #1663 Moved workbench links from dkan_sitewide_menu to dkan_workflow.
DKAN Dataset:
  • #1696 Disable/uninstall dkan_dataset_api if enabled because it is deprecated in favor open_data_schema_map.
  • Leaflet draw widget usability improvements
  • #1636 Fix validation on resource forms when multiple resource type fields are populated
  • #1626 Fix 508 compliance issues for leafleat draw widget
  • #1387 Added DKAN Dataset module into DKAN Core.
  • #1589 Fix geofield map button toggle function.
  • #1345 Added default image for groups.
  • #1377 Add preview support for resources of many more formats, outside of recline.js. Support added for JSON, geojson, XML, ArcGIS REST, WMS, images, PDF, ZIP
  • #1377 Fix bugs in resource mimetypes and previews.
  • Add more frequency update options to the dataset creation form https://project-open-data.cio.gov/iso8601_guidance/#accrualperiodicity
  • #1301 Groups field on Dataset form was modified to be a single select with all group options and it was moved from the tabs section to the main section of the form.
  • #1301 Groups field was hidden on the resource form since groups are assigned automatically based on the parent datasets.
  • #1301 Moved and renamed the following fields from ODFE to DKAN Core: field_is_part_of, field_data_dictionary_type, field_landing_page, field_pod_theme, field_conforms_to, field_rights, field_language.
  • #1301 Added hook updates to handle the renaming of the old ODFE fields removing the ‘odfe’ namespacing.
  • #1301 Added hook update to migrate the content from field_odfe_category to field_pod_theme on Datasets.
  • #1301 Added new option on DKAN Dataset Forms to enable POD based validation on Dataset form.
  • #1301 Added new option on DKAN Dataset Forms to enable Groups field validation on Dataset form.
  • #1280 Fixes to groups UX: Group body field now labeled “Description,” longer group descriptions do not get cut off after 200 characters on group page, and extraneous “about” tab on Group node edit form removed
  • Add new “modified date” field, hidden in form, to datasets. This is for dkan_harvest compatibility; saves sources dates into a separate field so they aren’t affected by node changes. ODSM mappings also updated.
  • Fix keywords/tags creating broken links when containing spaces, and add missing keywords to default content.
  • #1477 Fix ODSM permissions for non-admin roles.
  • #1532 Fixed text on ‘Download All’ button to not display HTML.
  • #1570 Removed ‘Back to dataset’ button on standalone resources.
  • Better support of downloading remote files from the resource view page “Download” button.
  • #1553 Removed warning when a resource is created without title.
  • #1591 Add limit to proxy resources.
  • #1576 Improved download of remote files.
  • #1670 Fix 508 compliance issues for visualization_entity_charts.
  • #1669 Fix 508 compliance issues for leaflet draw widget.
  • #1434 Add patch to remote_stream_wrapper to fix memory exhausted errors.
  • #1652 Renamed ‘filefield_remotefile’ as ‘filefield_dkan_remotefile’.
  • #1671 Add patch to field_group to avoid DOM-based cross-site scripting vulnerabilities.
DKAN Topics:
  • #1159 Added a test for creating a topic term.
  • #1486 Make the icon field required if the icon type is set to ‘font’.
  • #1656 Fix topics icon selector functionality.

7.x-1.12.13 2017-01-04

  • Fix broken recline (resource CSV) preview embeds caused by how recline module loads bootstrap
  • Fix publisher token in open_data_schema_map_dkan - was showing only URL rather than publisher name
  • Add patch to remote_stream_wrapper to avoid memory exhausted on big files.

7.x-1.12.12 2016-12-15

  • Start caching recline “embed” pages (previews rendered with no headers/sidebars for iframe) (recline module)
  • Upgrade Drupal core to 7.52
  • Update ctools to 1.11
  • Fix link to group page on group node teaser when site has clean urls disabled
  • Fix “format” facet collapsed even when selected on the search page.
  • Fix resources not synced with datasets when upgrade from 1.11
  • Update media to 2.0-beta13
  • Fix hidden body field on page content type
  • Improve access check/security on datastore pages - unauthorized users could perform certain datastore functions
  • Use dashes in links to tags to avoid “page not found” errors
DKAN Datastore:
  • Add limit to proxy resources.
  • Better support of downloading remote files from the resource view page “Download” button.

7.x-1.12.11 2016-10-20

  • Fixed a bug in Recline regarding file objects and node forms that caused errors when using the “view changes” button on Dataset or Resource edit form
  • Center group images in “Groups” page and group node page sidebar
  • Add “2x” “3x” etc to datset teasers when more than one resource of a particular format present.
  • Update the default jquery library setting from 1.7 to 1.10
  • Fix topics menu and facet links if special characters are used in topic terms.
  • Fixed dataset form redirect when validation fails, was sending user to node/add/dataset rather than node/%/edit
  • Patch fontyourface to remove <div> from the “standard text” selector, to prevent unpredictable results from this option
  • Fixed issue in open_data_schema_map with “Data Dictionary” field not displaying URL in data.json file

7.x-1.12.10 2016-09-07

  • Fix JS error on IE browsers preventing previews from appearing
  • Provide upgrade paths for older sites upgrading to newer BU editor config

7.x-1-12.9 2016-08-31

  • Fix data dictionary field to use html input format and improvements to the field’s display and help text.
  • Update dkan_workflow content type legend to include all content types with correct icons.
  • Fix to new bug introduced in recline module in 1.12.7; previews would not display if file smaller than max preview size, even if loading from datastore.
  • Fix a problem with paths for image icons for topics. Uploaded image icons will now display correctly in the Topics menu.

7.x-1.12.8 2016-08-19

  • Patch panelizer module to correct bug introduced in previous release. See release notes for details.
  • Add validation to Group form to prevent duplicate groups from being created
  • Update and patch panels and panelizer to fix critical security issue. See release notes.
  • Patch panelizer module to correct bug introduced in previous release. See release notes.

7.x-1-12.7 2016-08-09

  • Update contrib modules uuid, services, manualcrop, markdown, panelizer, panopoly_widgets, panopoly_images
  • Fix bug that produced a “Cannot use a scalar value” error when trying to add a content pane to a data story node in panels.
  • Change theming on resource to remove iframe to “API or link URL” if remote file field also populated
  • Update options in Dataset “frequency” field to reflect standards for DCAT and Project Open Data
  • Provide “data proxy” to serve remote CSVs through local domain and resolve cross-origin issues with previews n these resources
  • Some changes to the Resource node form to improve UX of links and file attachments, especially by replacing the label “Link to an API” to “API or Website URL”
  • Fix bug on relative paths for links in theme template files.
  • Fix minor typo in DKAN topics menu link

7.x-1.12.6 2016-07-26

  • Sanitize theme output for facets to avoid any security issue from search input
  • Fix a potential XSS vulnerability in search facets by adding some validation hooks for facet input
  • Theme updates back ported from dev branch, including topics icons in topics drop-down menu
  • Add RoleAssign module, update DKAN Permissions to give site managers permission to assign roles.
  • Make links and emails in metadata pages clickable

7.x-1.12.5 2016-07-13

  • Update restws to 7.x-2.6 (critical security update)
  • Update to latest version of recline.js to fix map tiles (Mapquest discontinued open access)
  • Upgrade Drupal core to 7.50

7.x-1.12.4 2016-07-12

  • Update permissions in DKAN Permissions module to allow anonymous users to access RDF endpoints for individual datasets.

7.x-1.12.3 2016-06-30

  • Open Data Schema Map update: improves performance of package_show endpoint
  • Point to new release of visualization entity, which cleans up some make files and brings visualization_entity_charts into the same project repository.
  • Fix editor permissions to allow access to visualizations list from admin menu
  • Upgrade Views to 7.14
  • Upgrade Drupal core to 7.44
  • Add <br><h2><h3><center><iframe> to allowed html tags
  • Renamed the default HTML text format to ‘Markdown HTML’
  • Improved markdown text editor toolbar
  • Stories main menu link removed from stories view
  • Groups main menu link removed from page manager groups page config
  • Main menu links added on install function rather than through features
  • Add body field to data_dashboard content types
  • Front page page manager config moved to the dkan_sitewide_demo_front feature
  • Front page group views moved to the dkan_dataset_groups feature
  • Add dkan_ipe feature to simplify the in-place editor interface, includes addition of the panels_curator module

7.x-1.12.2 2016-06-09

  • Fix problems in visualization entity charts creation due to version of CSV.js referenced in recline.make

7.x-1.12.1 2016-06-07

  • New version of Visualization Entity Charts fixes a number of UI bugs and restores support for Google Sheets and Data Proxy
  • Fix XSS vulnerability by adding sanitization for titles on workbench view
  • Upgrade of: Colorizer, Admin Menu Source, Honey Pot, Panopoly Widgets, Panopoly Images, Pathauto, Rules, Restws, Manual Crop, Image Cache Actions, Features, Search API, Organic Groups, Chosen, Date, Entity, Facet API, Facet API Bonus, Facet API Pretty Paths, UUID, Views, Select or other, Remote Stream Wrapper, Link, Libraries, Beauty Tips, Gravatar, OG Extras, Services and Workbench Email.
  • Upgrade of Panopoly Widgets, Panopoly Images and Fieldable Panels Panes.
  • Patch colorfield module to avoid incorrect status error. See https://www.drupal.org/node/2696505

7.x-1.12 2016-04-20

  • Fixed regex used to rewrite font file paths on Dkan Topics, which was breaking filepaths in data.json
  • Rename dkan_featured_topics to dkan_topics before release
  • Add “data dashboard” creation link to user admin menu (in dkan_sitewide_menu)
  • Patch DKAN core file module to fix managed file problem https://www.drupal.org/node/1903010#comment-10118508
  • Standardize facet titles between user, search and group pages
  • Add dkan_featured_topics module and relevant updates to dkan_dataset and nuboot_radix
  • Upgrade Drupal core to 7.43
  • Enable pretty paths for search page
  • Change search page path from /dataset to /search and redirect /dataset to /search for backward compatibility
  • Add big file upload support through file_resup module
  • Add support to import big files using mysql statement load data infile
  • Added Dkan Workflow module
  • Upgrade Fieldable Panels Panes to 1.8
  • Added hook_update to remove ‘Add Dataset’ and ‘Datasets’ links from main menu. The ‘Datasets’ link is now added by the dkan_sitewide_search_db feature.
  • Removed ‘taxonomy_menu_vocab_parent_dkan_topics’ variable from info file to get the ‘DKAN Featured Topics’ feature back into ‘Default’ state.
  • Upgrade of Panopoly Widgets, Panopoly Images and Fieldable Panels Panes.

7.x-1.12 2015-04-01

DKAN_Datastore:
  • Add update function enable field_hidden module

7.x-1.11 2016-02-02

DKAN Dataset:
  • Fix some bugs breaking the resource links on dataset pages
  • Re-arranged group links on group landing pages; membership links now in sidebar rather than tabs
  • For long fields in the “additional info” custom metadata table on dataset pages, limit the row height for fields displaying extremely long text values
  • Add a list of related visualizations (if using the [Visualization Entity](https://github.com/NuCivic/visualization_entity) module) to resource pages
  • Move group-level permission export into new dkan_dataset_groups_perms module
  • Fixed counting of datasets, broken when a dataset belongs to more than one group
  • Hide “add resource” button on datasets from users who do not have permission to do so
  • Numerous small improvements and bug fixes

7.x-1.11 2016-02-01

  • Re-designed dataset/search page. “Datasets” link on default menu bar now goes to a page that lets you browse and search all site content, not just datasets, but does filter by dataset. Search box will search all content by default. See note below.
  • Add new dkan_permissions module, refactoring default roles and permissions and using new export method. See note below.
  • Moved group permissions from old dkan_sitewide_roles_perms into new dkan_dataset_groups_perms
  • Patch fontyourface module to make font changes happen instantly
  • Fix adminrole implementation to avoid warning on install
  • Fix (again) a bug that would make colorizer styles disappear after running “poor man’s cron”
  • Add to resource node page a list of visualizations built with that resource
  • Number of contrib module updates
  • Change the view of groups to show 12 instead of 10 nodes per page, to fit with the layout of three per line
  • Patch the FacetAPI module to avoid a cross-site scripting vulnerability
  • Add integration with ProboCI (http://probo.ci/) for QA builds
  • Fix a bug that showed an extra pager on the group page
  • Major refactor of Behat tests, including introduction of new [DKAN extension](https://github.com/NuCivic/dkanextension).
  • Improve layout/ordering of blocks in the sidebar for dataset pages
  • Improvement in the UX of user pictures, including Gravatar integration
  • Numerous other small improvements and bugfixes

Notice: Avoiding PHP errors after upgrade

In some cases, changes to the codebase may cause PHP errors when the Drupal bootstrap process looks for a file that no longer exists. As always, backing up your db before upgrading is recommended. In addition, if you get to a state in which Drupal will not bootstrap successfully due to an issue with views_autocomplete_filters, try some combination of:

$ drush sql-query ‘DELETE FROM registry WHERE filename LIKE “%views_autocomplete_filters_handler_filter_string.inc%”’; $ drush cc all $ drush rr

Search page redesign

This release of DKAN includes a redesigned search page, which is also the page used to browse datasets. This redesign required some changes to the included search indexes. Use the following drush commands to rebuild your search indexes, or go to admin/config/search/search_api/index/datasets and disable, re-enable and re-index the _datasets_ search index.

$ drush search-api-disable datasets -y $ drush search-api-enable datasets -y $ drush sapi-r -y $ drush sapi-i -y

Permissions/roles upgrade

As mentioned above, this release includes a new module called DKAN Permissions (dkan_permissions), which is meant to replace the old DKAN Sitewide Roles and Permissions (dkan_sitewide_roles_perms). New installs of DKAN will enable this module and ignore the old one, while existing sites will see no change (but are recommended to upgrade). See more information in the module’s README file.

The command line method, including deleting the now-deprecated “storyteller” role, would look like this:

$ drush dis dkan_sitewide_roles_perms -y $ drush rdel ‘storyteller’ $ drush en dkan_permissions -y $ drush fra -y $ drush fr dkan_permissions -y

Group-level permissions have been moved to DKAN Dataset Groups Permissions. To start using them, enable the new module, and revert it using Features. Do this via the UI or on the command line with Drush:

$ drush en dkan_dataset_groups_perms -y $ drush fr dkan_dataset_groups_perms -y

DKAN Dataset: - See the DKAN Dataset release notes for 7.x-1.11 for notes specific to the DKAN Dataset module.

7.x-1.10 2015-11-10

DKAN Distribution
  • A number of improvements to our test infrastructure
  • Improved user experience for user photos and Gravatar image fallback
  • Improvements to views on group pages
  • Enabled and improved UX of tools for adding existing visualizations directly into panels layouts
  • Fixed an extremely annoying bug in the Colorizer module that blew away colorizer CSS every time Drupal cron ran (sites using external/system cron were unaffected)
  • Added better HTTPS by loading certain external images over HTTPS
  • Upgraded to Drupal version 7.x-1.41
  • Added a CONTRIBUTING.md file to provide community contribution guidelines for DKAN project.
DKAN Dataset Module
  • NOTE: 7.x-1.10 Was re-released on 2015-11-18 to address bugs in the teaser preview links.
  • Various improvements to dataset teaser displays.
  • Support for external previews (opening catalogued resources instantly in third-party visualization services, * including ArcGIS and CartoDB)
  • Support for Flaticon at module level, so vector icons work on any theme
  • Minor improvements and bugfixes
DKAN Datastore
  • Upgrade Feeds module to latest dev version
NuBoot Radix Theme
  • Style improvements for “open with” button
  • Fix default logo path when svg not available

7.x-1.9 2015-09-17

  • Added “Data Dashboards” and “Data Stories” content types with customizable layouts
  • Added Panopoly Widgets for use in Dashboards, Stories, the front page and panel pages
  • Added new custom DKAN widgets for use on panel pages
  • The Visualization Entity module, along with the additional Charts bundle, are now included in DKAN core
  • Added a “command center” to the user’s own profile page to more easily find common functions
  • @font-your-face module added w/configuration for easier font admin
  • Multiple security updates and other contrib module and theme updates.
  • Drupal now on version 7.39.
  • Many bug fixes and code cleanups
  • See release notes for nuboot_radix theme and individual DKAN modules for additional release notes
DKAN Datastore:
DKAN Dataset:
  • Fixed issues around the group/publisher field in search indexes and facets
  • When multiple resources are available for a dataset, new option to download all as zip file
  • Improvements for resource display in dataset pages and teasers
  • New hooks to allow additional license options https://github.com/NuCivic/dkan/issues/447
  • Numerous small improvements and bugfixes

Search API Database upgrade issue

This release updates the Search API DB module, which, if you are using the database backend, may break your search indexes, resulting in either a) nothing, or b) all nodes regardless of type on your search/datasets pages.

To fix, follow these steps after updating your database:

  1. Go to /admin/config/search/search_api/index/datasets/edit to edit the database node index
  2. Set the Server to “< No Server >” and Save. Your index will now show as “disabled.”
  3. Edit it again (by clicking “Edit” next to “Database Node Index”) and set the Server back to “Database Server” and Save.
  4. Click the “enable” link while viewing the resulting page, or edit the index again and check the “Enabled” box.
  5. Once the index is enabled, you should see “0/[total] Indexed” on the index page. Click “Index Now” at the bottom of the page to re-index all your datasets.
  6. Repeat for any other indexes you have running on a database server.

This issue has been identified and documented for other Drupal distributions using Search API. See release notes for the Panopoly (https://www.drupal.org/node/2425263) and OpenAtrium (https://www.drupal.org/node/2443025) distros on Drupal.org.

7.x-1.8 2015-04-02

  • Drupal core update to 7.36
  • Some tweaks to display properly drupal admin pages using nuboot_radix
  • Removed unnecessary drupal warning messages (colorizer, updates, etc)
  • Tweaks to behat tests
  • Security updates for contrib modules
  • DKAN_Datastore: Adding CRUD functions to Datastore.inc class
  • DKAN_Datastore: Adding drush interface to CRUD functions in order to create instances from the cli
DKAN Dataset:
  • Several recline.js tweaks
  • Updated open_data_schema_map to include module that allows to hook your own custom output formatters (xml example included)
  • Security updates for contrib modules

7.x-1.7 2015-02-20

DKAN Dataset:

7.x-1.6

  • core update to 7.34
  • fix issues with javascript in behat tests
  • freezing behat drupal extension to 1.0.2
  • dkan_sitewide modules now report to github
DKAN_Datastore:
  • Fix services module reference at dkan_datastore.make file
DKAN Dataset:
  • Creating taxonomies during setup test
  • Updating ref_field patch
  • Replacing dkan_dataset_api with open_data_schema_map

7.x-1.5 2014-10-15

  • Drupal core security update. Upgrades to 7.32

7.x-1.4 2014-10-10

  • Replaced tid hardcoding during profile install with proper taxonomy_vocabulary_machine_name_load
  • Removed duplicates in dkan.make already present in dkan_dataset.make
  • Footer branding updated
  • Editable dkan blocks
  • Sort on search
  • New rebuild script to keep dkan updated on custom installs
DKAN_Datastore:
  • Many warnings fixed with proper isset calls
  • Added dkan_datastore_api_count implementation
  • Optimisations on GET parameters processing
DKAN Dataset:
  • Many warnings fixed with proper isset calls
  • Changes in make file (fixed versions on some components)
  • Additional Info block on dataset page
  • Added docx as allowed format for field_upload

7.x-1.3

  • Issue #192 drupal security update

7.x-1.2

  • Issue #184 drupal security update
  • Updating makefile back to 7.x-1.x state
    • 1.1 Release commit + changing git urls with https urls in order to travis to not fail during make
  • Changing git protocol to https for dkan_dataset, dkan_datastore and nuboot in order for travis build to success
  • Issue #166 updating nuboot for iframe link theme
  • Update dataset.inc fixed #150
  • Update README.md
  • Updating nuams references to NuCivic
  • Updating nuams references to NuCivic
  • Updating nuams references to Nucivic
  • Updating nuams references to NuCivic
  • Updating nuams references to NuCivic
  • Updating drupal-org.make to point to Nucivic
  • Updating drupal-org-dev.make to point to NuCivic
  • Moving to using URLs for make file instead of copying locally
  • Fixes issue #112: “downloald” typo in drupal-org.make
  • Changed ‘The goal of the project combine...’ to ‘The goal of the project is to combine...’
  • Update .travis.yml
  • Adding information about google group
  • Adding key to drupal core to play nice with buildmanager
  • adding multi channel notifications for dkan
  • adding slack integration for travis
  • Update to make files

7.x-1.1 2013-06-27

  • Moved dkan development to Nucivic’s github: https://github.com/NuCivic/dkan
  • Grabbing dkan_dataset.make and dkan_datastore.make fiels from github cdn
  • Several bugs fixed.
  • Moving to using URLs for make file instead of copying locally #160
  • Issue nuams/dkan#112: “downloald” typo in drupal-org.make #159
  • Fixing #151: Admin role defaults to “Editor” #158
  • Fix typo on About page #156
  • Using hook_post_features_revert() for setting user_admin_role. #145
  • Issue nuams/dkan#138 update recline in dkan_dataset.make to fix negative lat/lon #135
  • Update to make files #131
  • POD changes #128
  • Issue #108 Add default email address for sample content #126
  • Fix license block #123
  • Issue #111 Fix facetapi_pretty_paths makefile definition #97
  • Update drupal-org.make #84
DKAN_Datastore:
DKAN Dataset:
  • Change references to NuCivic organization on github
  • Several bug fixes
  • Some changes for Project Open Data compliance, including data.json self-reference

7.x-1.0

7.x-1.0-beta

  • First tagged release. Further tags will include a changelog.

Roadmap

Coming soon!

Additional resources