Introduction and Goals¶

This is a project to allow scientists to create research compendium [1] comprising all relevant narrative, code, and data to make their research truly reproducible. Our goal is allow and teach researchers to document the computational portions of their research methods as thoroughly as they would document a tabletop experiment. We want our tools to fulfill these goals:

The application has the following goals.

• We will make it possible to archive all of the data, codes, documentation, parameters, and environmental settings linked with published research in a versioned form.
• We will support the verification and validation processes by providing for the execution of shared code and the visualization of results.
• We want to help and encourage researchers to manage their research in a way that makes it mixable and executable.
• Most of all we wish to make these tools heavily automated, and easy to access and utilize to lessen the exertion required from already overburdened academic researchers in the process of publishing fully reproducible work.

Imagine if all the materials in a research project could be continuously packaged and deployed with no snags preventing use and refinement by anyone. We could help make research accessible to everyone.

Project Structure¶

This is a django project with the following structure.

• home: this handles the landing page, faq, and similar concerns that don’t call for separate apps.
• users: this handles users and profiles by using django-allauth and cookiecutter-django’s user template
• compendia: this handles the archiving and representation of a compendium.
• lib: this holds code that does not call for an app
• api: this handles our service apis.

Resources¶

• Free software: MIT License
• Technical Documentation: http://researchcompendia.readthedocs.org
• Issue tracker: https://github.com/researchcompendia/researchcompendia/issues
• Wiki: https://github.com/researchcompendia/researchcompendia/wiki
• IRC: #researchcompendia

Acknowledgements¶

Make a separate acknowledgements page?

References¶

Compendia¶

For this prototype, we’ve started with a few simple models to represent research compendia and related metadata. Here are a few of the most salient models.

The starting point for a compendium is an Article. An Article contains high level information about a compendium.

• reference information to the original material for use in citations
• a “code and data abstract” description
• links to archives of compendium materials such as code archives, data archives, and documentation
• link to archive of the verification package. This is only editable by admins.
• DOI for the original material as well as DOIs for the compendium materials.
• taxonomic information
• licenses for code and data

Articles with verification packages can be run via the verification service API. Each run of an article results in a Verification record that links back to its Article. Our prototype for now stores only a few fields. In the future, ResearchCompendia will use Sumatra as a service and will consume serialized Sumatra records for each run. At which point we will deprecate our own verification model.

API¶

{"verifications":
  [
    {
      "archive_file_url": "/media/results/025a8ae48fdbd220fccccb879f4d1b4e-2014-04-03-15-42-37/verification.zip",
      "archive_info": {
        "output_files": [ { "bytes": 101, "file": "pizza_order.json", "size": "101B" } ]
      },
      "created": "2014-04-03T20:42:37.288Z",
      "id": 52,
      "parameters": {},
      "requestid": "messageidnotusedyet",
      "status": "unknown",
      "stderr": "",
      "stdout": "{'attending': 33, 'pizzas': {'cheese': 3, 'meat': 3, 'vegan': 1, 'veg': 4}}\n\n    33 people will show up (guess)\n    3 cheese pizzas\n    3 meat pizzas\n    4 vegetarian pizzas\n    1 vegan pizzas\n    \n"
    },
    // and so on ...
  ]
}

Contributing Code¶

Before you get started, remember that it is easier for us to accept pull requests that are narrow in scope and easy to review. Ideally we want to see pull requests with a commit that is a logical changeset so plan to work in this fashion.

When committing a change, please include the issue number in your commit comment. This helps us track the progress in the related issue. This works like so

$ git commit CONTRIBUTING.rst -m ‘improved the contributing docs for #1’

TODO: define some guidelines for what makes something easy to review.

Getting Started with Development

Reviewing Code and Concepts¶

We are always learning. Review code in our repository and suggest improvements and alternatives to our approaches.

Look through pull requests and review the changes.

Help us by discussing issues we’ve tagged with the brainstorming label

Reporting Bugs¶

Report bugs in our issue tracker. https://github.com/researchcompendia/researchcompendia/issues.

If you are reporting a bug, please include:

• Your browser information.
• Any details about your local setup that might be helpful in troubleshooting.
• Detailed steps to reproduce the bug.

Give us feeback¶

Talk to us about features you’d like. Let us know how we are doing. You can send us email from the contact form or let us know by filing an issue in the issue tracker.

Write Documentation¶

We could always use more documentation, whether as part of the official docs, in docstrings, or even on the web in blog posts, articles, and such.

Getting Code¶

Fork the repository and check out your fork and add our repo as a remote:

$ git clone https://github.com/YOURACCOUNT/researchcompendia.git
$ cd researchcompendia
$ git remote add parent https://github.com/researchcompendia/researchcompendia.git

Installing Dependencies¶

1. Install your local copy into a virtualenv. Here is one way to do it:

   $ cd researchcompendia/
   $ virtualenv venv
   $ source venv/bin/activate
   $ pip install -r requirements/ci.txt
   

   You can also use virtualenvwrapper if you have it is installed. It is convenient, though not required.

2. Set up environment variables. Once you have the environment varialbes set up, you are ready to set up the database.

3. We use Postgres as our database backend. We haven’t written docs to walk though setting up Postgres yet. If you are unfamiliar with this process, you could consider using sqlite locally event thought our preferences is to use postgres in all environments. If you’d like to use sqlite, set up your DATABASE_URL environment variable to have a path to a file name that will hold your database. for example:

   'sqlite:////path/to/my/site/root/researchcompendia.db'
   

Setting up the Environment¶

These are the environment variables that are used in the site settings listed along with their defaults. For any default that doesn’t apply to you, make an environment variable with your preferred setting.

You are required to create SECRET_KEY environment variable.

If you use fabric to provision a vagrant box, it will generate a SECRET_KEY for you.

Environment Variable	Default Setting
ADMINS	compendia@example.com
AWS_ACCESS_KEY_ID	‘’
AWS_SECRET_ACCESS_KEY	‘’
AWS_STORAGE_BUCKET_NAME	compendiaexamplebucket
DATABASE_URL	postgres://:5432/researchcompendia
DJANGO_BROKER_URL	hamqp://guest:guest@localhost:5672//
DJANGO_CELERY_DISABLE_RATE_LIMITS	True
DJANGO_CELERY_RESULT_BACKEND	cache+memcached://127.0.0.1:11211/
DJANGO_CELERY_RESULT_SERIALIZER	json
DJANGO_CELERY_TASK_SERIALIZER	json
DJANGO_CELERY_TIMEZONE	US/Central
CROSSREF_PID	‘’
DEBUG	True
DEFAULT_FILE_STORAGE	django.core.files.storage.FileSystemStorage
DEFAULT_FROM_EMAIL	compendia@example.com
DISQUS_API_KEY	‘none’
DISQUS_WEBSITE_SHORTNAME	researchcompendiaorg
BONSAI_URL	http://127.0.0.1:9200
EMAIL_BACKEND	django.core.mail.backends.console.EmailBackend
MAILGUN_ACCESS_KEY	‘’
MAILGUN_SERVER_NAME	‘’
MEDIA_ROOT	normpath(join(PROJECT_ROOT, ‘media’))
MEDIA_URL	if using the s3boto storages then ‘%s/media/’ % S3_URL, otherwise /media/
REMOTE_DEBUG	False
SECRET_KEY	no default. will blow up if not set
SITE_ID	1
STATICFILES_STORAGE	django.contrib.staticfiles.storage.StaticFilesStorage
STATIC_ROOT	normpath(join(PROJECT_ROOT, ‘staticfiles’))
STATIC_URL	if using the s3boto storages then ‘%s/static/’ % S3_URL, otherwise /static/

There are a couple of places where settings are hard-coded in to templates. These need to be fixed. Meanwhile, you will need to change or remove the google analytics and addthis pubid codes.

• Our google analytics tracking code is hardcoded in templates/base.html.
• Our addthis pubid is hardcoded in templates/compendia/detail.html.

Preparing the Database¶

Set up the database by running:

$ cd companionpages
$ ./manage.py syncdb --migrate
$ ./manage.py loaddata fixtures/*

Launching ResearchCompendia¶

Once the database is set up, you can start the app:

$ ./manage.py runserver

Or perhaps you would like to have detailed stacktraces and messages:

$ ./manage.py runserver --traceback -v 3

Making Changes¶

Now that you have the code, a virtualenv, and the proper environment variables, you are ready to make your changes locally.

1. Make a topic branch for your changes. For example, if you wanted to add twitter logins to the site, you could make a branch named twitterlogin:

   $ git checkout -b twitterlogin
   

2. Periodically update your branch from the parent develop branch. Use git rebase (not git merge):

   $ git fetch parent
   $ git rebase parent/develop
   

   We prefer a pull request with one commit rather than many small commits. To avoid making a request with many commits, you can do an interactive rebase and use fixup.:

   $ git rebase -i parent/develop
   

3. Check that your changes pass style check and automated tests:

   $ make test
   

4. Demonstrate your changes. It can be helpful to share work you are running locally from your own machine so that other people can help test. PageKite is a free/libre open source software project that can do this for you. This QuickStart shows how.

5. Commit your changes and push your branch to up to your fork on GitHub.:

   $ git add .
   $ git commit -m "Adds twitter login for #123"
   $ git push origin twitterlogin
   

Now you are ready to make a pull request.

Reviewing Changes¶

Submit a pull request through the GitHub website to submit it for review. Before you submit a pull request, check that it meets these guidelines:

0. The pull request should be easy to review.
1. The pull request should include tests
2. Check https://travis-ci.org/researchcompendia/researchcompendia/pull_requests and make sure that the tests pass
3. If the pull request adds functionality, the docs and/or comments should be updated.

Trying out Vagrant¶

If you want to use Vagrant clone the researchcompendia-deployment repo. It contains fabric files and a Vagrantfile that pulls down a debian wheezy VM from vagrantcound:

$ git clone https://github.com/researchcompendia/researchcompendia-deployment.git
$ cd researchcompendia-deployment
$ vagrant up
$ fab vagrant provision

Provision is not idempotent, so running it twice will probably fail in interesting ways. If you want to start over need to run vagrant destroy first.

Provision will set up the vagrant box in the same way that a production box is set up.

Initial Setup¶

Clone the researchcompendia-deployment repo:

$ git clone https://github.com/researchcompendia/researchcompendia-deployment.git
$ cd researchcompendia-deployment
$ virtualenv venv
$ source venv/bin/activate
$ pip install -r requirements.txt

The top of the fabric.py file has settings for hostnames, port numbers, etc. Check these to verify that they apply to you. If they do not, change them as appropriate. Each fabric command should be prefaced with the name of the environment, dev, staging, prod, vagrant are available environments. If this is the first time you are setting up a box, run the provision task. This example provisions staging:

(venv)$ fab staging provision

Until the deployment and configuration process completely automated, there are manual steps to go through for a first install and deployment. You’ll want to create a django superuser. Log in to the box, sudo su tyler, source the environment variables, run the createsuperuser command:

./manage.py createsuperuser

Creating A Release¶

Releases are created with the git flow release subcommands (part of git-flow).

Version numbers: We use a semantic versioning scheme.

All of the changes you want to go in to a release should be in the development branch. Once everything is there run (the version for this example is 1.0.0):

$ git flow release start 1.0.0

Once you’ve started a release, edit the HISTORY.rst file and bump the version in __init__.py and commit the changes. Then run:

$ git flow release finish 1.0.0

This will merge everything in develop to master and create a tag for you. Once that is done, you need to push the changes:

$ git checkout master
$ git push
$ git push --tags
$ git checkout develop
$ git push

Deploying a Release¶

Check release notes for any required updates to environment variables, database migrations, static files changes. Activate your researchcompendia-deployment virtualenv and run the deploy command:

(venv)$ fab staging deploy:1.0.0

That command is for the simplest case of a change. It doesn’t migrate the database, for example. There is a fab command for that, migrate

Miscellanea¶

Convenient packages like htop and multitail are installed. Use sudo htop for a handy way to observe and control running processes.

The provision task creates a directory layout in the tyler user’s home directory organized in the following way:

$ tree -L 2 site
site
├── bin
│   ├── celeryworker.sh
│   ├── check_downloads.sh
│   ├── environment.sh
│   └── runserver.sh
├── logs
│   ├── log_files.yml
└── tyler

Logs for nginx, celery, gunicorn, supervisor, cron, django are in the logs/ directory.:

logs
├── celery_worker.log         logs for celery and tasks
├── cron_checkdownloads.log   logs to see that the download link checker was called
├── gunicorn_supervisor.log   gunicorn/django console logs
├── log_files.yml             papertrail remote_syslog config file
├── tyler.access.log          nginx access log
└── tyler.error.log           nginx error log

Remote logging, the webapp, and celery are controled by supervisor. run sudo supervisorctl status to see a list of statuses.:

$ sudo supervisorctl status
celery                           EXITED     Jan 16 11:21 PM
remote_syslog                    RUNNING    pid 13411, uptime 1 day, 0:05:17
researchcompendia                RUNNING    pid 13828, uptime 1 day, 0:01:17

Core Team¶

• Sheila Miguez <sheila@researchcompendia.org>
• Jennifer Seiler
• Victoria Stodden

Contributors¶

• @benmarwick
• You?

1.3.1 (2014-06-03)¶

• Fixes #196, /search/ without query parameters caused a 500.

1.3.0¶

• Adds a Demos page where we can list ongoing demos. Currently we have a Table of Contents demo and some executability demos.
• The demo Table of Contents demo allows for an admin to generate a table of contents based on entries that categorize compendia types. The current demo shows a result card style with no stripped down informationc ompared to the main site result card style.
• Adds facets based on compendium type and primary research fields. These are stackable in the url, but the UI only drills down via links for now.
• Upgrades insecure requirements. Started tracking requirements via the https://requires.io service.
• Adds microformat to our header for rel-vcs as specified by https://joeyh.name/rfc/rel-vcs/
• template refactoring – pulled out some browse, facet, and pagination code in to separate files to be included in other templates. DRY
• minor style changes

1.2.1 (2014-05-12)¶

1.2.0 (2014-04-14)¶

• First pull request from an external contributor! #168 fixes two typos in the FAQ. Thanks @benmarwick.
• First iteration with execution with some limited ability to do parameter passing, with execution history
• DOI minting for data and code

1.1.0 (2014-03-06)¶

1.0.0 (2013-12-18)¶

First talk! Now that we’ve had the first talk about this, let’s have 1.0.0!

Not a lot of user-facing changes for this release. We’ve Renamed project ResearchCompendia from the pre-release tyler name, and renamed the repo to go with that.

• We have tagging on creation, but tags are not yet used for search and browsing.
• We have simple text search that searches through authors, title, abstract
• We have a simple compendium creation form with DOI autocompletition.
• We have some prelimary developer docs that discuss contributions and development.
• We have user profile pages so that users can view a list of their compendia.