Welcome¶
Welcome to Open Library developer documentation.
This place is intended for developers of Open Library project.
If you are new to Open Library development, you can get started by Setting up a dev instance.
Table of Contents¶
Setting up a dev instance¶
Open Library supports dev instance based on Vagrant. This document will step you though the installation process.
The vagrant setup uses Ubuntu 14.04 LTS operating system. Make sure you have at least 1GB of RAM in the virtual machine.
Getting the source¶
Open Library uses git for version control and the code repository is hosted on github.
You can get the source code from there using:
$ git clone git://github.com/internetarchive/openlibrary.git
$ cd openlibrary
In case you don’t have git installed already, you can install it on Ubuntu using:
$ sudo apt-get install git-core
and on Mac OS X using:
$ brew install git
Starting the dev-instance¶
The Open Library dev-instance can be started using:
$ vagrant up
This will setup a virtual machine with Ubuntu 14.04, installs all dependencies, setup database and loads sample data.
Once, the virtual machine is up, you’ll be able to access the website at:
An admin user with the following credentials is created as part of the installation.
<pre> username: openlibrary password: openlibrary </pre>
Known Issues¶
It is known that the following issues exist:
- Stats on the home page is not working
- /admin is failing
- subject search is not working
Getting Started with Development¶
Working with JavaScript¶
Introduction¶
The Open Library website makes heavy use of jQuery libraries.
Except jquery and jquery-ui, all other third-party javascript libraries are combined and included as vendor.js.
All the custom javascripts are combined and includes as all.js.
vendor.js and third party libraries¶
All third-party javascripts are added in the vendor/js directory in the repository and static/build/vendor.js is generated by combining these javascripts. The files included in static/build/vendor.js are specified in a shell script static/js/vendor.jsh.
To include a new third-party library:
Add that library in vendor/js in the repository
Add an entry in static/js/vendor.jsh
Generate vendor.js by running make
$ make js
Commit vendor.jsh and the library added to the repository
all.js and custom javascripts¶
All the custom javascript files are put in the repo at openlibrary/plugins/openlibrary/js. All these javascript files are combined to generate build/js/all.js.
The order in which these files are included is determined by the sort order of the names. In general, it is a bad idea to depend on the order of files loaded.
If you make any changes to any of the javascript files, run make js to regenerate build/js/all.js.
Working with CSS¶
All stylesheets are in static/css. They are combined to generate build/css/all.css, which is included in all the web pages.
It’s a good idea to break CSS into multiple logical files, instead of putting it in one monolithic file.
If you make changes to any CSS, run make css to regenerate build/css/all.css.
Architecture of the dev instance¶
This page contains details on how the dev instance is designed, how the various pieces run and interact with each other.
- Services used
- How they interact
- How are they managed
- Logging
- Config files
Design Notes¶
Contains design and implementation details of complex parts of the system.
Mostly about specific pieces like lists (not overall)
Dev Instance¶
Installing Dependencies¶
Installing CouchDB¶
On Linux, installing CouchDB involves installing so many dependencies, some of them are incompatible with ubuntu/debian installations. Fortunately, a binary distribution with all the binaries is provided by CouchOne.
The problem with that distribution is that it is interactive and it won’t be possible to automate the installation of that. To overcome this problem, the binaries after running the interactive installation are taken and bundled. The ol install script, downloads the bundle, unpacks it and runs a script to updates the path in some scripts and config files.
The created bundle couchdb-1.0.1-linux-binaries.tgz is uploaded to http://www.archive.org/details/ol_vendor.
On Max OS X, installing couchdb is somewhat easier than installing on Linux. However to make the installations on both mac and linux similar, binaries are taken from CouchDBX app.
Unlike Linux, where bin and etc are available in the top-level, mac binaries have them at 2 level deep. To make both distributions identical, a bin/couchdb script is added and etc is symlinked from couchdb_1.0.1/etc. Also the mac couchdb script expects the base directory should be the working directory. The above bin/couchdb script, takes of this too.
Lists¶
This document explain how lists feature is implemented.
Feature Flags¶
Feature flags is a way to have features under development live on production and visible only to admins/beta-users.
The idea of Feature Flags came from Flicker. They manage their development on a single branch using feature flags.
http://code.flickr.com/blog/2009/12/02/flipping-out/
Using Feature Flags¶
Feature flags are used in templates and in controller classes.
To make some part of the template visible only if a feature-flag is enabled:
$if "lists" in ctx.features:
<h3>Lists</h3>
$for list in page.get_lists():
...
To enable a url only if a feature flag is enabled:
class home(delegate.page):
path = "/"
def is_enabled(self):
return "home-v2" in web.ctx.features
def GET(self):
return render_template("home")
Setting Feature Flags¶
In Open Library, the feature flags are specified in the openlibrary.yml file as follows:
features:
merge-authors: enabled
lists: admin
lending_v2:
filter: usergroup
usergroup: beta-users
The value of a feature flag is called a filter. A filter can be specified either as its name or as a dict containing its name and parameters. For example, the following 2 example mean the same.
features:
lists: admin
features:
lists:
filter: admin
Available filters are:
enabled
Enabled for all users.
disabled
Disabled for all users.
loggedin
Enabled only for logged-in users.
admin
Enabled for admin users.
usergroup
Enabled for the users part of the specified usergroup.
lending_v2: filter: usergroup usergroup: beta-users
queryparam
Enabled only if the url has a specified query parameter.
debug: filter: queryparam name: debug value: true
Appendices¶
Configuration files¶
List of all the relevant config files, their syntaxes and what the control.
setup.py commands¶
Most openlibrary operations are modeled as setup.py commands.
bootstrap¶
Bootstraps the dev instance.
$ python setup.py bootstrap
Bootstrap command is explained in bootstrap.
start¶
Starts all the OL services.:
$ python setup.py start
This command starts supervisord using configuration from conf/supervisor/linux.ini or conf/supervisor/macosx.ini based on the platform.
Logs of the services are written to var/log.
shell¶
Start a bash shell with the env required for running all OL scripts.
$ python setup.py shell
This starts a bash shell with conf/bashrc as rc file.
test¶
Runs all the test cases.
$ python setup.py test
Behind the scenes it runs scripts/runtests.sh.
build_sphinx¶
Builds the sphinx docs.
$ python setup.py build_sphinx
The docs will be generated in build/sphinx/html/.
The custom setup.py commands are implemented in setup_commands.py.
To know more about how to add custom setuptools commands, see:
http://tarekziade.wordpress.com/2007/09/30/extending-setuptools-adding-a-new-command/
Database Migrations¶
Ocationally, new tables get added to openlibrary database and some existing tables get altered. Scripts are provided to migrate the existing dev instances to new schema.
To migrate an existing dev instance:
$ python setup.py shell
$ python scripts/migrate_db.py
This will look the current database schema and identifies its version and upgrades it to the latest version.
Dependencies¶
The system requires a lot of 3rd party programs as well as a whole bunch of python libraries. We have an installation script which you can run that will setup the entire thing for you but in order to run this script, the following packages have to be installed first.
- Git
- For getting the source repository
- PostgreSQL 8.2 or later (psql)
- This is where we store our data
- Python 2.5 or later (python)
- The application is written in python
- Java Runtime (tested with openjdk-6-jre)
- The indexer (solr) is a Java application
- Python virtualenv (python-virtualenv)
- Necessary to create “virtual” installations of Python so that we can install packages without touching your system distribution. More details at the virtualenv PyPI page.
- Apache Solr (installed by the installation script)
- The search engine.
On Linux, you will also have to install the following dev packages python-dev, libpq-dev and libxslt-dev.
Some Python packages are also required, but they will be automatically installed by the installation script.