Kozmic CI¶

Step 1: Install Docker¶

If you use Digital Ocean, you can just create a droplet from an image with pre-installed Docker:

If you use Ubuntu 13.04 or later, installing Docker is just as simple as that:

apt-key adv --keyserver keyserver.ubuntu.com --recv-keys 36A1D7869245C8950F966E92D8576A8BA88D21E9
echo deb http://get.docker.io/ubuntu docker main > /etc/apt/sources.list.d/docker.list
apt-get update
apt-get install -f lxc-docker

If you use another OS, take a look at Docker installation instructions.

Step 2: Register a new application on GitHub¶

Go to https://github.com/settings/applications/new and create a new application.

Set the homepage URL to http://my-server-ip-or-addr and the authorization callback URL to http://my-server-ip-or-addr/_auth/auth-callback.

Step 3: Start Kozmic CI¶

Create a directory for Kozmic CI logs:

mkdir -p $HOME/kozmic-ci/log

Create a data-only container that will be used to persist the Kozmic CI data:

docker run -v /var/lib/docker -v /var/lib/mysql --name kozmic-data ubuntu:12.04 true

Run Kozmic CI:

JOB=$(docker run -e=SECRET_KEY=xxxxx \
                 -e=GITHUB_CLIENT_ID=xxxxxxxxxxxxxxxxxxxx \
                 -e=GITHUB_CLIENT_SECRET=xxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxxx \
                 -e=SERVER_NAME=xxx.xxx.xxx.xxx \
                 -p=80:80 \
                 -p=8080:8080 \
                 --volumes-from kozmic-data \
                 -v=$HOME/kozmic-ci/log:/var/log \
                 --privileged -d aromanovich/kozmic)
docker logs $JOB

A few comments:

• SECRET_KEY must be set to a unique, unpredictable value. It must be kept the same if you are restarting or updating Kozmic CI container.
• GITHUB_CLIENT_ID and GITHUB_CLIENT_SECRET must contain the OAuth client id and secret of your GitHub application.
• SERVER_NAME must contain an IP address or domain name of the machine. It must be accessible from the outside Internet.
• -p=80:80 -p=8080:8080 binds the container ports to the host system.
• -v=$HOME/kozmic-ci/log:/var/log mounts the directory from the host into the container which make is possible to see what’s going on inside.
• --privileged key is required for running Docker within Docker.

After starting the container, take a look at the logs directory content and make sure that it doesn’t say any errors. That’s it!

MySQL and Python¶

Suppose the project is written in Python and uses MySQL. Here’s an example of a hook configuration.

Docker base image: kozmic/ubuntu:12.04.

Install script:

#!/bin/bash
set -e  # Exit if any command returns a non-zero status

sudo su <<EOF
pip install -r ./requirements/basic.txt
pip install -r ./requirements/dev.txt
EOF

Tracked files:

requirements/basic.txt
requirements/dev.txt

Build script:

#!/bin/bash
set -e

sudo su <<EOF
/usr/bin/mysqld_safe &
sleep 3  # Give it time to start
mysql -e 'create database rsstank_test character set utf8 collate utf8_general_ci;'
EOF

cp ./rsstank/config_local.py-kozmic ./rsstank/config_local.py
./test.sh

We run pip from root because it sets up packages system-wide.

MySQL is already set up in the kozmic/ubuntu:12.04 image. It has to be started manually before the tests, because Docker doesn’t use Ubuntu’s init system.

MongoDB and Python¶

Here’s another example for a project that uses MongoDB.

Docker base image: kozmic/debian:wheezy.

Install script:

#!/bin/bash
set -e  # Exit if any command returns a non-zero status

echo 'deb http://downloads-distro.mongodb.org/repo/debian-sysvinit dist 10gen' | \
    sudo tee /etc/apt/sources.list.d/mongodb.list
sudo apt-get update
sudo apt-get install -y --force-yes mongodb-10gen

sudo pip install -r requirements/devel.txt

Tracked files:

requirements/basic.txt
requirements/dev.txt

Build script:

#!/bin/bash
set -e  # Exit if any command returns a non-zero status

sudo /etc/init.d/mongodb start

py.test --cov=adlift --cov-report=term

kozmic.models¶

class kozmic.models.RepositoryBase[source]¶

A base repository class to be used by HasRepositories mixin.

gh_id = Column(None, Integer(), table=None, nullable=False)¶

GitHub id

gh_name = Column(None, String(length=200), table=None, nullable=False)¶

GitHub name (i.e., kozmic)

gh_full_name = Column(None, String(length=200), table=None, nullable=False)¶

GitHub full name (i.e., aromanovich/kozmic)

gh_ssh_clone_url = Column(None, String(length=200), table=None, nullable=False)¶

SSH clone url

gh_https_clone_url = Column(None, String(length=200), table=None, nullable=False)¶

HTTPS clone url

is_public = Column(None, Boolean(), table=None, nullable=False)¶

Is the repository public?

classmethod from_gh_repo(gh_repo)[source]¶

Constructs an instance of cls from gh_repo.

class kozmic.models.HasRepositories[source]¶

Mixin that adds repositories relationship to the model. Repositories are stored in separate tables for each parent. Repository attribute contains model (inherited from RepositoryBase) mapped to the parent’s repositories table.

This pattern is well described in “Hand Coded Applications with SQLAlchemy” presentation by Mike Bayer.

class kozmic.models.User(**kwargs)[source]¶

User account.

repositories¶

Set of user repositories.

organizations¶

Set of user organizations in which user has admin rights to at least one repository (see Organization).

gh_id¶

GitHub user id

gh_login¶

GitHub user login

gh_name¶

Human-readable GitHub name

gh_access_token¶

OAuth access token

gh_avatar_url¶

GitHub avatar URL

repos_last_synchronized_at¶

The last time when the user’s repositories and organizations were synced with GitHub

email¶

E-mail address

get_identity()[source]¶

Returns user’s flask.ext.principal.Identity.

get_available_projects(annotate_with_latest_builds=False)[source]¶

Returns list of Projects that user has access to. If annotate_with_latest_builds is specified, returns list of pairs (Projects, Build) where the second element is the latest project build or None if the project was never built.

gh[source]¶

An authenticated GitHub session for this user.

Type:	github3.github.GitHub

get_gh_org_repos()[source]¶

Retrieves data from GitHub API and returns a pair of values:

1. set of github3.orgs.Organization in which the current user has at least one repository with admin rights;
2. dict mapping these organization’ ids to lists of github3.repo.Repository to which the current user has admin access.

get_gh_repos()[source]¶

Retrieves data from GitHub API and returns a list of the user owned repositories.

Return type:	list of github3.repo.Repository

sync_memberships_with_github()[source]¶

Does the same as Project.sync_memberships_with_github(), but for the user. Returns True if there were not any GitHub errors; False otherwise.

class kozmic.models.Organization(**kwargs)[source]¶

Stores a set of organization repositories that a user has admin access to.

Different Kozmic users, but members of the same GitHub organization, will have their own Organization entries with possibly different sets of repositories (because they are possibly members of different teams).

repositories¶

gh_id¶

GitHub organization id

gh_login¶

GitHub organization login

gh_name¶

Human-readable GitHub name

user¶

User whose admin rights is reflected by this organization

class kozmic.models.DeployKey(passphrase, key_size=2048)[source]¶

An RSA deploy key pair.

gh_id¶

GitHub deploy key id

rsa_private_key¶

RSA private deploy key in PEM format encrypted with the app secret key

rsa_public_key¶

RSA public deploy key in OpenSSH format

ensure()[source]¶

If the corresponding GitHub deploy key does not exist, creates it. Returns True if there weren’t GitHub API errors; False otherwise.

delete()[source]¶

Deletes the public key from GitHub. Returns True if it has been successfully deleted (or was missing); False otherwise.

class kozmic.models.Project(**kwargs)[source]¶

Project is a GitHub repository that is being watched by Kozmic CI.

gh_id¶

GitHub repo id

gh_name¶

GitHub repo name (i.e., kozmic)

gh_full_name¶

GitHub repo full name (i.e., aromanovich/kozmic)

gh_login¶

GitHub repo owner (user or organization) login

gh_ssh_clone_url¶

SSH repo clone url

gh_https_clone_url¶

HTTPS clone url

is_public¶

Is the project’s repository public?

deploy_key¶

Deploy key

members¶

Project members

owner¶

Project owner

gh[source]¶

Project’s GitHub.

Type:	github3.repos.Repository

delete()[source]¶

Deletes the project and it’s corresponding GitHub entities such as hooks, deploy key, etc. Returns True if they all have been successfully deleted (or were missing); False otherwise.

get_latest_build(ref=None)[source]¶

Return type:	Build

sync_memberships_with_github()[source]¶

Synchronizes project members with GitHub.

GitHub _repository members_ with admin and push rights become project managers, other _repository members_ become project members.

Returns True if there were not any GitHub errors; False otherwise.

class kozmic.models.Hook(**kwargs)[source]¶

Reflects a GitHub hook.

gh_id¶

GitHub hook id

title¶

Title

install_script¶

Install script

build_script¶

Script to be run at hook call

docker_image¶

Name of a Docker image to run build script in (for example, “ubuntu” or “aromanovich/ubuntu-kozmic”). Specified docker image is pulled from index.docker.io before build

project¶

Project

ensure()[source]¶

If the corresponding GitHub hook does not exist, creates it. If it exists, but has wrong configuration, re-configures it. Returns True if there weren’t GitHub API errors; False otherwise.

delete()[source]¶

Deletes the project hook. Returns True if it’s corresponding GitHub hook is missing or has been successfully deleted; False otherwise.

class kozmic.models.TrackedFile(**kwargs)[source]¶

Reflecs a tracked file.

path¶

Path within git repository

hook¶

Hook

class kozmic.models.Build(**kwargs)[source]¶

Reflects a project commit that triggered a project hook.

number¶

Build number (within a project)

gh_commit_ref¶

Commit reference (branch on which the commit was pushed)

gh_commit_sha¶

Commit SHA

gh_commit_author¶

Commit author

gh_commit_message¶

Commit message

created_at¶

Created at

status¶

Build status, one of the following strings: ‘enqueued’, ‘success’, ‘pending’, ‘failure’, ‘error’

project¶

Project

calculate_number()[source]¶

Computes and sets number.

started_at[source]¶

Time the first job has started or None if there is no started jobs yet.

finished_at[source]¶

Time the last job has finished or None if there is no finished jobs yet.

set_status(status, target_url='', description='')[source]¶

Sets status and posts it on GitHub.

class kozmic.models.HookCall(**kwargs)[source]¶

Reflects a fact that GitHub triggered a project hook.

created_at¶

Created at

gh_payload¶

JSON payload from a GitHub webhook request

hook¶

Hook

build¶

Build

class kozmic.models.Job(**kwargs)[source]¶

A job that caused by a hook call.

started_at¶

Time the job has started or None

finished_at¶

Time the job has finished or None

return_code¶

Return code

stdout¶

Job log

task_uuid¶

uuid of a Celery task that is running a job

build¶

Build

hook_call¶

HookCall

get_cache_id()[source]¶

Returns a string that can be used for tagging a Docker image built from the install script. A cache id changes whenever the base Docker image, the install script or any of the tracked files is changed.

Note

Requires that Docker is running and Docker base image (self.hook_call.hook.docker_image) is pulled.

started()[source]¶

Sets started_at and updates build status. Must be called when the job is started.

finished(return_code)[source]¶

Sets finished_at and updates build status. Must be called when the job is finished.

tailer_url[source]¶

URL of a websocket that streams a job log in realtime.

permanent_url[source]¶

A permanent URL of the job.

is_finished()[source]¶

Is the job finished?

status[source]¶

One of the following values: ‘enqueued’, ‘success’, ‘pending’, ‘failure’, ‘error’.

kozmic.perms¶

kozmic.perms.project_owner = <functools.partial object at 0x3fbadb8>¶

Project owner need

kozmic.perms.project_manager = <functools.partial object at 0x403ddb8>¶

Project manager need

kozmic.perms.project_member = <functools.partial object at 0x403de10>¶

Project member need

kozmic.perms.delete_project(id)[source]¶

Returns a Permission to delete the project identified by id.

kozmic.perms.manage_project(id)[source]¶

Returns a Permission to manage the project identified by id.

kozmic.perms.view_project(id)[source]¶

Returns a Permission to view the project identified by id.

kozmic.builds.tasks¶

kozmic.builds.tasks.do_job(hook_call_id)[source]¶

A Celery task that does a job specified by a hook call.

Creates a Job instance and executes a build script prescribed by a triggered Hook. Also sends job output to Job.task_uuid Redis pub-sub channel and updates build status.

Parameters:	hook_call_id – int, HookCall identifier

kozmic.builds.tasks.restart_job(id)[source]¶

A Celery task that restarts a job.

Parameters:	id – int, Job identifier

class kozmic.builds.tasks.Publisher(redis_client, channel)[source]¶

Parameters:	redis_client – Redis client channel (str) – pub/sub channel name

class kozmic.builds.tasks.Tailer(log_path, publisher, container, kill_timeout=600)[source]¶

A daemon thread that waits for additional lines to be appended to a specified log file. Once there is a new line, it does the following:

1. Translates ANSI sequences to HTML tags;
2. Sends the line to a Redis pub/sub channel;
3. Pushes it to Redis list of the same name.

If the log file does not change for kill_timeout seconds, specified Docker container will be killed and corresponding message will be appended to the log file.

Once the thread has finished, has_killed_container tells whether the :param:`container` has stopped by itself or been killed by a timeout.

Parameters:	log_path (str) – path to the log file to watch publisher (Publisher) – publisher container (dictionary returned by docker.Client.create_container()) – container to kill kill_timeout (int) – number of seconds since the last log append after which kill the container

class kozmic.builds.tasks.Builder(docker, message_queue, docker_image, script, working_dir, clone_url, commit_sha, deploy_key=None)[source]¶

A thread that starts a script in a container and waits for it to complete.

One of the following attributes is not None once the thread has finished:

return_code¶

Integer, a build script’s return code if everything went well.

exc_info¶

exc_info triple (type, value, traceback) if something went wrong.

Parameters:

docker (docker.Client) – Docker client message_queue (Queue.Queue) – a queue to which put an identifier of the started Docker container. Identifier is a dictionary returned by docker.Client.create_container(). Builder will block until the message is acknowledged by calling Queue.Queue.task_done(). deploy_key (2-tuple of strings) – a pair of strings (private key, passphrase) docker_image (str) – a name of Docker image to be used for build_script execution. The image has to be already pulled from the registry. working_dir (str) – path of the directory to be mounted in container’s /kozmic path clone_url (str) – SSH clone URL commit_sha (str) – SHA of the commit to be checked out

kozmic¶

kozmic.__init__.create_app(config=None)[source]¶

Returns a fully configured Flask application.

Parameters:	config – a config object or it’s name. Will be passed directly to flask.config.Config.from_object(). If not specified, the value of KOZMIC_CONFIG environment variable will be used. If KOZMIC_CONFIG is not specified, 'kozmic.config.DefaultConfig' will be used.

kozmic.accounts¶

kozmic.accounts.bp¶

flask.Blueprint that gives users a means to manage their account settings.

kozmic.auth¶

kozmic.auth.bp¶

flask.Blueprint that implements an authentication through GitHub.

kozmic.builds¶

kozmic.builds.bp¶

flask.Blueprint that implements webhooks to be triggered by GitHub and serves status badges.

Note

Does not require authentication.

kozmic.projects¶

kozmic.projects.bp¶

flask.Blueprint that provides all the means for managing and viewing projects.

kozmic.repos¶

kozmic.repos.bp¶

flask.Blueprint that gives users the abilities to:

1. View list of GitHub repositories they have admin access to
2. Create kozmic.models.Project for any of them