Welcome to Zamboni’s documentation!¶

Need help?¶

Come talk to us on irc://irc.mozilla.org/marketplace if you have questions, issues, or compliments.

On OS X¶

The best solution for installing UNIX tools on OS X is Homebrew.

The following packages will get you set for zamboni:

brew install python libxml2 mysql openssl swig304 jpeg pngcrush redis

On Ubuntu¶

The following command will install the required development files on Ubuntu or, if you’re running a recent version, you can install them automatically:

sudo aptitude install python-dev python-virtualenv libxml2-dev libxslt1-dev libmysqlclient-dev libssl-dev swig openssl curl pngcrush redis-server

Services¶

Zamboni has three dependencies you must install and have running:

• MySQL should require no configuration.
• Redis should require no configuration.
• Seee elasticsearch for setup and configuration.

git clone --recursive git://github.com/mozilla/zamboni.git
cd zamboni

git submodule update --init --recursive

curl -sL https://raw.githubusercontent.com/brainsik/virtualenv-burrito/master/virtualenv-burrito.sh | $SHELL

mkvirtualenv zamboni

workon zamboni  # requires virtualenvwrapper

mkvirtualenv --python=/usr/local/bin/python2.7 zamboni

mkvirtualenv --no-site-packages zamboni

make update_deps

Environment settings¶

Out of the box, zamboni should work without any need for settings changes. Some settings are configurable from the environment. See the marketplace docs for information on the environment variables and how they affect zamboni.

./manage.py migrate
./manage.py loaddata init

Database Migrations¶

Each incremental change we add to the database is done with Django migrations. To keep your local DB fresh and up to date, run migrations like this:

./manage.py migrate

Loading Test Apps¶

Fake apps and feed collections can be created by running:

./manage.py generate_feed

Specific example applications can be loaded by running:

./manage.py generate_apps_from_spec data/apps/test_apps.json

See Fake App Data for details of the JSON format.

If you just want a certain number of public apps in various categories to be created, run:

./manage.py generate_apps N

where N is the number of apps you want created in your database.

./manage.py runserver

http://localhost:2600/services/monitor

./manage.py addusertogroup <your email> 1

npm install
python manage.py compress_assets

TEMPLATE_DEBUG = True

make full_update

MySQL¶

On your dev machine, MySQL probably needs some tweaks. Locate your my.cnf (or create one) then, at the very least, make UTF8 the default encoding:

[mysqld]
character-set-server=utf8

Here are some other helpful settings:

[mysqld]
default-storage-engine=innodb
character-set-server=utf8
skip-sync-frm=OFF
innodb_file_per_table

On Mac OS X with homebrew, put my.cnf in /usr/local/Cellar/mysql/5.5.15/my.cnf then restart like:

launchctl unload -w ~/Library/LaunchAgents/com.mysql.mysqld.plist
launchctl load -w ~/Library/LaunchAgents/com.mysql.mysqld.plist

Here are more tips for optimizing MySQL on your dev machine.

Memcached¶

By default zamboni uses an in memory cache. To install memcached libmemcached-dev on Ubuntu and libmemcached on OS X. Alter your local settings file to use:

CACHES = {
    'default': {
        'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
        'LOCATION': ['localhost:11211'],
        'TIMEOUT': 500,
    }
}

RabbitMQ and Celery¶

By default zamboni automatically processes jobs without needing Celery.

See the Celery page for installation instructions. The example settings set CELERY_ALWAYS_EAGER = True. If you’re setting up RabbitMQ and want to use celery worker you will need to alter your local settings file to set this up.

See Celery for more instructions.

Node.js¶

Node.js is needed for Stylus and LESS, which in turn are needed to precompile the CSS files.

If you want to serve the CSS files from another domain than the webserver, you will need to precompile them. Otherwise you can have them compiled on the fly, using javascript in your browser, if you set LESS_PREPROCESS = False in your local settings.

First, we need to install node and npm:

brew install node
curl http://npmjs.org/install.sh | sh

Optionally make the local scripts available on your path if you don’t already have this in your profile:

export PATH="./node_modules/.bin/:${PATH}"

Not working?

• If you’re having trouble installing node, try http://shapeshed.com/journal/setting-up-nodejs-and-npm-on-mac-osx/. You need brew, which we used earlier.
• If you’re having trouble with npm, check out the README on https://github.com/isaacs/npm

Stylus CSS¶

Learn about Stylus at http://learnboost.github.com/stylus/

cd zamboni
npm install

In your settings_local.py (or settings_local_mkt.py) ensure you are pointing to the correct executable for stylus:

STYLUS_BIN = path('node_modules/stylus/bin/stylus')

Installation¶

brew install rabbitmq

# On a Mac, you can find this in System Preferences > Sharing
export HOSTNAME='<laptop name>.local'

# Set your host up so it's semi-permanent
sudo scutil --set HostName $HOSTNAME

# Update your hosts by either:
# 1) Manually editing /etc/hosts
# 2) `echo 127.0.0.1 $HOSTNAME >> /etc/hosts`

# RabbitMQ insists on writing to /var
sudo rabbitmq-server -detached

# Setup rabitty things (sudo is required to read the cookie file)
sudo rabbitmqctl add_user zamboni zamboni
sudo rabbitmqctl add_vhost zamboni
sudo rabbitmqctl set_permissions -p zamboni zamboni ".*" ".*" ".*"

./manage.py celery worker -Q priority,devhub,images,limited  $OPTIONS

Celery Tasks¶

Any python function can be set as a celery task. For example, let’s say we want to update our current_version but we don’t care how quickly it happens, just that it happens. We can define it like so:

@task(rate_limit='2/m')
def _update_addons_current_version(data, **kw):
    task_log.debug("[%s@%s] Updating addons current_versions." %
                   (len(data), _update_addons_current_version.rate_limit))
    for pk in data:
        try:
            addon = Webapp.objects.get(pk=pk)
            addon.update_version()
        except Webapp.DoesNotExist:
            task_log.debug("Missing addon: %d" % pk)

@task is a decorator for Celery to find our tasks. We can specify a rate_limit like 2/m which means celery worker will only run this command 2 times a minute at most. This keeps write-heavy tasks from killing your database.

If we run this command like so:

from celery.task.sets import TaskSet

all_pks = Webapp.objects.all().values_list('pk', flat=True)
ts = [_update_addons_current_version.subtask(args=[pks])
      for pks in mkt.site.utils.chunked(all_pks, 300)]
TaskSet(ts).apply_async()

All the Webapps with ids in pks will (eventually) have their current_versions updated.

@cronjobs.register
def update_addons_current_version():
    """Update the current_version field of the addons."""
    d = Webapp.objects.valid().values_list('id', flat=True)

    with establish_connection() as conn:
        for chunk in chunked(d, 1000):
            print chunk
            _update_addons_current_version.apply_async(args=[chunk],
                                                       connection=conn)

During Development¶

celery worker only knows about code as it was defined at instantiation time. If you change your @task function, you’ll need to HUP the process.

However, if you’ve got the @task running perfectly you can tweak all the code, including cron jobs that call it without need of restart.

Installation¶

You can download the Elasticsearch code and run elasticsearch directly from this folder. This makes it easy to upgrade or test new versions as needed. Optionally you can install Elasticsearch using your preferred system package manager.

We are currently using Elasticsearch version 1.6.2. You can install by doing the following:

curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-1.6.2.tar.gz
tar -xvzf elasticsearch-1.6.2.tar.gz
cd elasticsearch-1.6.2

For running Marketplace you must install the ICU Analysis Plugin:

./bin/plugin -install elasticsearch/elasticsearch-analysis-icu/2.6.0

For more about the ICU plugin, see the ICU Github Page.

Settings¶

cluster.name: wooyeah

# Don't try to cluster with other machines during local development.
# Remove the following 3 lines to enable default clustering.
network.host: localhost
discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["localhost"]

script.disable_dynamic: false

path:
  logs: /usr/local/var/log
  data: /usr/local/var/data

We use a custom analyzer for indexing add-on names since they’re a little different from normal text.

To get the same results as our servers, configure Elasticsearch by copying the scripts/elasticsearch/elasticsearch.yml (available in the scripts/elasticsearch/ folder of your install) to your system.

For example, copy it to the local directory so it’s nearby when you launch Elasticsearch:

cp /path/to/zamboni/scripts/elasticsearch/elasticsearch.yml .

If you don’t do this your results will be slightly different, but you probably won’t notice.

Launching and Setting Up¶

Launch the Elasticsearch service:

./bin/elasticsearch -Des.config=elasticsearch.yml

Zamboni has commands that sets up mappings and indexes for you. Setting up the mappings is analagous to defining the structure of a table, indexing is analagous to storing rows.

It is worth noting that the index is maintained incrementally through post_save and post_delete hooks.

Use this to create the apps index and index apps:

./manage.py reindex --index=apps

Or you could use the makefile target (using the settings_local.py file):

make reindex

If you need to use another settings file and add arguments:

make SETTINGS=settings_other ARGS='--force' reindex

Querying Elasticsearch in Django¶

We use Elasticsearch DSL, a Python library that gives us a search API to elasticsearch.

On Marketplace, apps use mkt/webapps/indexers:WebappIndexer as its interface to Elasticsearch:

query_results = WebappIndexer.search().query(...).filter(...).execute()

Testing with Elasticsearch¶

All test cases using Elasticsearch should inherit from mkt.site.tests.ESTestCase. All such tests will be skipped by the test runner unless:

RUN_ES_TESTS = True

This is done as a performance optimization to keep the run time of the test suite down, unless necessary.

Troubleshooting¶

I got a CircularReference error on .search() - check that a whole object is not being passed into the filters, but rather just a field’s value.

I indexed something into Elasticsearch, but my query returns nothing - check whether the query contains upper-case letters or hyphens. If so, try lowercasing your query filter. For hyphens, set the field’s mapping to not be analyzed:

'my_field': {'type': 'string', 'index': 'not_analyzed'}

Installing through pip¶

You can get a development environment with

pip install --no-deps -r requirements/dev.txt

Adding new packages¶

Note: this is deprecated, all packages should be added in requirements.

The vendor repo was seeded with

pip install --no-install --build=vendor/packages --src=vendor/src -I -r requirements/dev.txt

Then I added everything in /packages and set up submodules in /src (see below). We’ll be keeping this up to date through Hudson, but if you add new packages you should seed them yourself.

If we wanted to add a new dependency called cheeseballs to zamboni, you would add it to requirements/prod.txt or requirements/dev.txt and then do

pip install --no-install --build=vendor/packages --src=vendor/src -I cheeseballs

Then you need to update vendor/zamboni.pth. Python uses .pth files to dynamically add directories to sys.path (docs).

I created zamboni.pth with this:

find packages src -type d -depth 1 > zamboni.pth

html5lib and selenium are troublesome, so they need to be sourced with packages/html5lib/src and packages/selenium/src. Hopefully you won’t hit any snags like that.

for f in src/*
    pushd $f >/dev/null && REPO=$(git config remote.origin.url) && popd > /dev/null && git submodule add $REPO $f

M2Crypto installation¶

If you are on a Linux box and get a compilation error while installing M2Crypto like the following:

SWIG/_m2crypto_wrap.c:6116:1: error: unknown type name ‘STACK’

... snip a very long output of errors around STACK...

SWIG/_m2crypto_wrap.c:23497:20: error: expected expression before ‘)’ token

   result = (STACK *)pkcs7_get0_signers(arg1,arg2,arg3);

                    ^

error: command 'gcc' failed with exit status 1

It may be because of a few reasons:

• comment the line starting with M2Crypto in requirements/compiled.txt

• install the patched package from the Debian repositories (replace x86_64-linux-gnu by i386-linux-gnu if you’re on a 32bits platform):

  DEB_HOST_MULTIARCH=x86_64-linux-gnu pip install -I --exists-action=w "git+git://anonscm.debian.org/collab-maint/m2crypto.git@debian/0.21.1-3#egg=M2Crypto"
  pip install --no-deps -r requirements/dev.txt
  

• revert your changes to requirements/compiled.txt:

  git checkout requirements/compiled.txt
  

Pillow¶

As of Mac OS X Mavericks, you might see this error when pip builds Pillow:

clang: error: unknown argument: '-mno-fused-madd' [-Wunused-command-line-argument-hard-error-in-future]

clang: note: this will be a hard error (cannot be downgraded to a warning) in the future

error: command 'cc' failed with exit status 1

You can solve this by setting these environment variables in your shell before running pip install ...:

export CFLAGS=-Qunused-arguments
export CPPFLAGS=-Qunused-arguments
pip install ...

More info: http://stackoverflow.com/questions/22334776/installing-pillow-pil-on-mavericks/22365032

Image processing isn’t working¶

If adding images to apps or extensions doesn’t seem to work then there’s a couple of settings you should check.

--------------------------------------------------------------------
*** TKINTER support not available
--- JPEG support available
--- ZLIB (PNG/ZIP) support available
*** FREETYPE2 support not available
*** LITTLECMS support not available
--------------------------------------------------------------------

sudo ln -s /usr/lib/i386-linux-gnu/libz.so /usr/lib
sudo ln -s /usr/lib/i386-linux-gnu/libjpeg.so /usr/lib

sudo ln -s /usr/lib/x86_64-linux-gnu/libz.so /usr/lib
sudo ln -s /usr/lib/x86_64-linux-gnu/libjpeg.so /usr/lib

pip install -I PIL

ES is timing out¶

This can be caused by number_of_replicas not being set to 0 for the local indexes.

To check the settings run:

curl -s localhost:9200/_cluster/state\?pretty | fgrep number_of_replicas -B 5

If you see any that aren’t 0 do the following:

Set ES_DEFAULT_NUM_REPLICAS to 0 in your local settings.

To set them to zero immediately run:

curl -XPUT localhost:9200/_settings -d '{ "index" : { "number_of_replicas" : 0 } }'

Running Tests¶

To run the whole shebang use:

python manage.py test

There are a lot of options you can pass to adjust the output. Read the docs for the full set, but some common ones are:

• -P to prevent nose adding the lib directory to the path.
• --noinput tells Django not to ask about creating or destroying test databases.
• --logging-clear-handlers tells nose that you don’t want to see any logging output. Without this, our debug logging will spew all over your console during test runs. This can be useful for debugging, but it’s not that great most of the time. See the docs for more stuff you can do with nose and logging.

Our continuous integration server adds some additional flags for other features (for example, coverage statistics). To see what those commands are check out the .travis.yml file.

There are a few useful makefile targets that you can use:

Run all the tests:

make test

If you need to rebuild the database:

make test_force_db

To fail and stop running tests on the first failure:

make tdd

If you wish to add arguments, or run a specific test, overload the variables (check the Makefile for more information):

make SETTINGS=settings_mkt ARGS='--verbosity 2 zamboni.mkt.site.tests.test_url_prefix:MiddlewareTest.test_get_app' test

Those targets include some useful options, like the --with-id which allows you to re-run only the tests failed from the previous run:

make test_failed

REUSE_DB=1 python manage.py test

Writing Tests¶

We support two types of automated tests right now and there are some details below but remember, if you’re confused look at existing tests for examples.

Why Tests Fail¶

Tests usually fail for one of two reasons: The code has changed or the data has changed. An third reason is time. Some tests have time-dependent data usually in the fixtues. For example, some featured items have expiration dates.

We can usually save our future-selves time by setting these expirations far in the future.

Localization Tests¶

If you want test that your localization works then you can add in locales in the test directory. For an example see devhub/tests/locale. These locales are not in the normal path so should not show up unless you add them to the LOCALE_PATH. If you change the .po files for these test locales, you will need to recompile the .mo files manually, for example:

msgfmt --check-format -o django.mo django.po

Sending actual email¶

In some circumstance you want to still recieve some emails, even when REAL_EMAIL is False. To allow addresses to receive emails, rather than be redirected to /admin/mail, use mkt.zadmin.models.set_config to set the real_email_allowed_regex key to a comma separated list of valid emails in regex format:

from mkt.zadmin.models import set_config
set_config('real_email_allowed_regex', '.+@mozilla\.com$,you@who\.ca$')

The Perfect Git Configuration¶

We’re going to talk about two git repositories:

• origin will be the main zamboni repo at http://github.com/mozilla/zamboni.
• mine will be your fork at http://github.com/:user/zamboni.

There should be something like this in your .git/config already:

[remote "origin"]
    url = git://github.com/mozilla/zamboni.git
    fetch = +refs/heads/*:refs/remotes/origin/*

Now we’ll set up your master to pull directly from the upstream zamboni:

[branch "master"]
    remote = origin
    merge = master
    rebase = true

This can also be done through the git config command (e.g. git config branch.master.remote origin) but editing .git/config is often easier.

After you’ve forked the repository on github, tell git about your new repo:

git remote add -f mine git@github.com:user/zamboni.git

Make sure to replace user with your name.

git checkout -b my-bug master

git fetch origin && git rebase origin/master

git checkout master && git pull && git checkout @{-1} && git rebase master

git push mine my-bug

Local Branches¶

Most new code is developed in local one-off branches, usually encompassing one or two patches to fix a bug. Upstream doesn’t care how you do local development, but we don’t want to see a merge commit every time you merge a single patch from a branch. Merge commits make for a noisy history, which is not fun to look at and can make it difficult to cherry-pick hotfixes to a release branch. We do like to see merge commits when you’re committing a set of related patches from a feature branch. The rule of thumb is to rebase and use fast-forward merge for single patches or a branch of unrelated bug fixes, but to use a merge commit if you have multiple commits that form a cohesive unit.

Here are some tips on Using topic branches and interactive rebasing effectively.