Prerequisites¶

Redhat:

sudo yum install cyrus-sasl-ldap cyrus-sasl-devel openldap-devel libxslt libxslt-devel libxslt-python

Debian/Ubuntu:

sudo apt-get install libssl-dev libsasl2-dev libldap-2.4-2 libldap2-dev libxslt1.1 libxslt1-dev python-libxslt1 libexiv2-dev

Configuration¶

Configuring MyTARDIS is done through a standard Django settings.py file there are some extra configuration options that are specific to MyTARDIS.

tardis.settings_changeme.FILE_STORE_PATH¶

The location of the file store.

initialising¶

When creating a new database the syncdb command will need to be called to initialise the schema and instert the initial data fixtures.

migrating¶

Some of the upgrades to MyTARDIS will require that the database schema be upgraded to match the internal datamodel. This tool migrates data from old database schemas to the current one. It detects which version of the database you are currently running and will automatically migrate to the current version.

backup¶

The backupdb command allows to backup and to restore of the MyTARDIS database. The command uses the corresponding database tool to facilitate this task. Currently implemented are PostgreSQL and MySQL. In case of backup, a directory called backups is created (if it does not exists) in the working directory. In case of restore, the database for storing the tablespace must already exist before loading the backup file into the database.

Managing Schema¶

MyTARDIS currently relies on the in-built Django administrative interface for managing schema. The administrative interface is accesible from a link similar to:

http://localhost/admin/

Schema definitions are the combination of two tables, Schema and ParameterName.

The Schema record identifies the schema (namespace), with a ParameterName record for each attribute of the schema. Other attributes of the parameter include the type, e.g. string, number, URL, date, enumerated list, etc., whether the parameter is searchable or not, and if it is an enumerated list, the valid choices.

Note: Creating a proper schema administration page is scheduled to be completed by December 2011.

METS¶

The Meta-data Encoding and Transmission Standard was recommended by Monash Librarians and ANU in 2008 as the XML description format for MyTARDIS datasets.

Ingestion Script¶

Metadata may be easily ingested using a simple script and POST request:

#!/bin/bash

file="$1"
username="localdb_admin"
password="secret"
host="http://localhost:8000"
owner="$username"

curl -F username=$username -F password=$password -F xmldata=@${file} -F experiment_owner=$owner "$host/experiment/register/"

To use this script paste it into a new file called, e.g. register.sh, chmod +x register.sh then can call it using ./register.sh file.xml. There are several example XML and METS files within the tardis test suite.

Post Processing¶

MyTARDIS takes advantage of the Django signal framework to provide post processing of files. The only default post processing step that is enabled by default operates on newly created Dataset Files.

AuthService Objects¶

class tardis.tardis_portal.auth.AuthService(settings=<django.conf.LazySettings object at 0x30ffbd0>)¶

The AuthService provides an interface for querying the auth(n|z) framework within MyTARDIS. The auth service works by reading the class path to plugins from the settings file.

Parameters:	settings (django.conf.settings) – the settings object that contains the list of user and group plugins.

authenticate(authMethod, **credentials)¶

Try and authenticate the user using the auth type he/she specified to use and if authentication didn’t work using that method, try each Django AuthProvider.

Parameters:	authMethod (string) – the shortname of the auth method. **credentials – the credentials as expected by the auth plugin

getGroups(request)¶

Return a list of tuples containing pluginname and group id

Parameters:	request (django.http.HttpRequest) – a HTTP Request instance

getGroupsForEntity(entity)¶

Return a list of the groups an entity belongs to

Parameters:	entity (string) – the entity to earch for, user or group.

The groups will be reurned as a list similar to:

[{'name': 'Group 456', 'id': '2'},
{'name': 'Group 123', 'id': '1'}]

getUser(user_dict)¶

Return a user model based on the user dict.

This function is responsible for creating the user within the Django DB and returning the resulting user model.

searchEntities(filter)¶

Return a list of users and/or groups

searchGroups(**kw)¶

Return a list of users and/or groups

Parameters:	id – the value of the id to search for name – the value of the displayname to search for max_results – the maximum number of elements to return sort_by – the attribute the users should be sorted on plugin – restrict the search to the specific group provider

searchUsers(filter)¶

Return a list of users and/or groups

Auth Plugins¶

• tardis.tardis_portal.auth.localdb_auth
• tardis.tardis_portal.auth.ldap_auth
• tardis.tardis_portal.auth.ip_auth

Filter Plugins¶

• tardis.tardis_portal.filters.exif.make_filter

EXIF Filter¶

tardis.tardis_portal.filters.exif.make_filter(name='', schema='', tagsToFind=[], tagsToExclude=[])¶

This filter provides simple metadata extraction of EXIF tags from images.

If a white list is specified then it takes precidence and all other tags will be ignored.

Parameters:	name (string) – the short name of the schema. schema (string) – the name of the schema to load the EXIF data into. tagsToFind (list of strings) – a list of the tags to include. tagsToExclude (list of strings) – a list of the tags to exclude.

Component Architecture¶

The diagram below shows the major components of a typical MyTARDIS installation.

Web Server

MyTARDIS is typically deployed with the standard Apache Http + WSGI + Django + Python stack.

RDBMS

Any of the Django supported databases may be used, typically Postgres.

Data Archive

Normal file server, e.g. NFS, SAMBA.

Browser

Typically Firefox

Ingestion

Ingestion is normally site specific.

Functional Architecture¶

MyTARDIS follows the standard Django architecture, as shown below [JCDA]:

The major functional modules in MyTARDIS are:

Auth

Authentication and Authorisation.

Download

Data download functionality.

Filters

Processing of metadata as data is uploaded, e.g. extract EXIF metadata from JPEG files.

Management

Additional commands for the Django CLI. The backup utility is implemented as a Django command.

Migrations

South database migration code.

Publish

RIF-CS metadata publication.

Information on the individual modules is available from the Module Index.

Scalability Model¶

At the component / module level, performance and Scalability within MyTARDIS is achieved by:

• Allowing long lived or compute intensive operations to be hosted on separate machines from the main web server.
  • E.g. by running multiple web servers and sending long lived operations, such as ingestion, to a server reserved for such operations.
• Performance and Scalability of the database is achieved by a combination of 1) optimising the SQL requests issued by MyTARDIS, and 2) database specific scalability, please refer to the appropriate database documentation.
• Performance and Scalability of the web server is provided through the normal mechanisms for apache / wsgi hosted python applications, e,g,:
  • Increasing individual Server capability
    • Individual Server performance / utilization may be managed by controlling the number of python (django) processes active at any one time.
  • Deploying multiple web servers
    • Multiple web servers may be deployed using standard mechanisms, e.g. hardware load balancers. State (session) information is distributed using Django’s standard session model.
• The Data Archive is a normal file system, e.g. NFS, SAMBA, etc., with performance and scalability dependent on the implementation and deployment.
• Extraction and formatting of metadata for ingestion is up to the client and may be distributed across any number of machines.

Persistance and Data Architecture¶

An introduction to how Tardis models Experiments, Datasets and Datafiles is provided in the Overview.

Django provides an Object-Relational-Model that maps Django Models to the underlying relational database. The models as defined in Django are:

An additional model diagram including the base Django models is also available: MyTARDIS & Django Models.

The underlying SQL schema generated by Django is not normally referenced directly as it is managed through Django’s ORM, however a version of the schema is available: MyTARDIS & Django SQL.

Security Model¶

Class Diagram¶

Unless the application has a very rich business model (which MyTARDIS does not), a Class diagram doesn’t convey much information in Django applications (it ends up being a flat list of classes). To understand the context in which the MyTARDIS classes are created, please refer to the Django documentation, http://docs.djangoproject.com/ on the responsibilities of models, views, managers and templates.

2.0 - Unreleased¶

• Auth/Auth redesign [Gerson, Uli, Russel]

  • Authorisation. Support for several pluggable authorisation plugins (Django internal, LDAP, VBL). The added AuthService middleware provides a mechanism to query all available auth modules to determine what group memberships a users has.

  • Alternative authorisation. Rule based experiment access control engine was implemented with the following access attributes for indivdual users and groups: canRead, canWrite, canDelete, isOwner. Additionally, a time stamp can be specified for each access rule.

    Further information can be found at the wiki: Authorisation Engine design

• Metadata Editing [Steve, Grischa]

• New METS parser & METS exporter [Gerson]

• Dist/Buildout infrastructure [Russell]

• Through the web creation and editing of experiments [Steve, Russell]

• Through the web upload of files [Steve]

• Download protocol handler [Russel, Uli]

• Logging framework [Uli]

• Django 1.3

1.07 - 01/06/2010¶

• Publish to tardis.edu.au interface created, though not implemented, pending legal text

1.06 - 15/03/2010¶

• Parameter import interface for creation of new parameter/schema definitions
• iPhone Interface

1.05 - 01/03/2010¶

• Images as parameters supported
• Data / metadata transfer from synchrotron is now ‘threaded’ using asynchronous web service transfers.

1.0 - 01/02/2010¶

• MyTARDIS created from existin MyTARDIS python / django codebase
• Allows private data to be stored
• Open key/value parameter model, replacing current crystallography one
• Internal data store for data
• LDAP Login
• Pagination of files
• Creation of synchrotron-tardis from MyTARDIS codebase including specific code for the VBL login service and data transfer to MyTARDIS deployments.
• Web server changed to apache and mod_wsgi

0.5 - 2009¶

• Re-wrote federated index (python / django)
• Federated stores are now simple web server based with optional FTP access
• Runs on Jython / Tomcat

0.1 - 2007¶

• Federated index (php) running on Apache HTTP Server
• Crystallography data deposition and packaging tools for Fedora Commons (java swing desktop)
• Search Interface via web

Development¶

svn checkout http://mytardis.googlecode.com/svn/trunk/ mytardis-read-only

svn add path/to/new/file

svn diff > feature_name_or_bugfix.diff

./bin/sphinxbuilder

#!/bin/bash
rm dist/*
[ -d egg-cache ] || mkdir -p egg-cache
export PYTHON_EGG_CACHE=`pwd`/egg-cache
python setup.py clean
python bootstrap.py
./bin/buildout
./bin/django-admin.py test --with-xunit --with-coverage --cover-package=tardis.tardis_portal --with-xcoverage
python setup.py sdist

nosetest.xml

**/coverage.xml