A bunch of Fabric commands we use all the time¶

Sample fabfile¶

Below is a fabfile.py.in buildout template that uses commands from Server group to set up a Projects server (based on Ubuntu 10.04).

import os
from fabric.api import env
from niteoweb.fabfile.server import *

env.path = os.getcwd()
env.hosts = ['${ips:server}']
env.hostname = '${config:hostname}'
env.shortname = '${config:shortname}'
env.temp_root_pass = '${pass:temp_root}'
env.server_ip = '${ips:server}'
env.hq_ip = '${ips:hq}'
env.bacula_ip = '${ips:bacula}'
env.office1_ip = '${ips:office1}'
env.office2_ip = '${ips:office2}'

env.email = 'maintenance@company.com'
env.admins = ['bob', 'jane', ]

env.rules = [
    # allow SSH access from our offices
    'ufw allow from %(office1_ip)s to any port ssh' % env,
    'ufw allow from %(office2_ip)s to any port ssh' % env,

    # allow access to Bacula File Deamon from our backup server
    'ufw allow from %(bacula_ip)s to any port bacula-fd' % env,

    # allow access to Munin from HQ server
    'ufw allow from %(hq_ip)s to any port munin' % env,

    # allow HTTP from everywhere
    ufw allow http
    ufw allow https
]

def deploy():
    """The highest-level meta-command for deploying Projects
    server. Use this command only on a vanilla Ubuntu 10.04 server."""

    with settings(user='root', password=env.temp_root_pass):
        create_admin_accounts(default_password='secret123')

    create_projects_group()

    # security
    harden_sshd()
    install_ufw()
    disable_root_login()

    # bootstrap server
    set_hostname()
    set_system_time()
    install_unattended_upgrades()
    raid_monitoring()
    install_rkhunter()

    # install software stack
    install_system_libs()
    install_nginx()
    install_sendmail()

    # install python
    install_python_26()
    install_python_24()
    configure_egg_cache()

    # monitoring, backup, etc.
    install_munin_node()
    install_bacula_client()
    configure_hetzner_backup()

Sample buildout.cfg¶

This fabfile.py template has a dependency on the niteoweb.fabfile package and also expects to find certain buildout values and config files in certain directories. Here’s a sample buildout.cfg that you can use to prepare an environment for using this fabfile.py.in. Save the fabfile.py.in in etc/ directory in your buildout.

[buildout]
unzip = true
newest = false
extensions = buildout.dumppickedversions
prefer-final = true

parts =
    fabric
    fabfile
    bacula-fd-conf
    bacula-master-conf
    duplicity-sh

# Configuration constants
[config]
# domain on which this server runs
hostname = zulu.company.com

# server's name
shortname = zulu

# Ports of services running on this server
# (besides Nginx running on port 80 and 443)
[ports]
ssh = 22
munin = 4949
bacula = 9102

# Various IPs needed for deployment
[ips]
server = ?.?.?.?
hq = ?.?.?.?
bacula = ?.?.?.?
office1 = ?.?.?.?
office2 = ?.?.?.?

# Passwords
[pass]
bacula = strong_password_here
duplicity = strong_password_here
hetzner_ftp_user = whatever_hetzner_gives_you
hetzner_ftp_pass = whatever_hetzner_gives_you
temp_root = root_password_that_hetzner_gives_you_for_a_new_server
# temp_root password is changed and disabled later on in deployment

# Prepare Fabric
[fabfile]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/fabfile.py.in
output = ${buildout:directory}/fabfile.py

[fabric]
recipe = zc.recipe.egg
eggs =
    Fabric
    niteoweb.fabfile

# Generate config files from templates in ./etc
[bacula-fd-conf]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/bacula-fd.conf.in
output = ${buildout:directory}/etc/bacula-fd.conf

[bacula-master-conf]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/bacula-master.conf.in
output = ${buildout:directory}/etc/bacula-master.conf

[duplicity-sh]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/duplicity.sh.in
output = ${buildout:directory}/etc/duplicity.sh

Config files¶

Samples of config files that you need to put inside etc/ directory in your buildout:

• bacula-fd.conf.in.
• bacula-master.conf.in.
• duplicity.sh.in.
• duplicityfilelist.conf.
• nginx.conf.

Prerequisities¶

You router needs to forward ports 500 and 4500 to your IPsec server.

Sample fabfile¶

Below is a fabfile.py.in buildout template that uses commands from Server group to set up an IPsec server (based on Ubuntu 10.04).

import os
from fabric.api import env
from fabric.api import settings
from fabric.api import sudo

from niteoweb.fabfile.server import *

env.path = os.getcwd()
env.hosts = ['${ips:server}']
env.server_ip = '${ips:server}'
env.shortname = '${config:shortname}'
env.hostname = '${config:hostname}'
env.temp_root_pass = '${pass:temp_root}'

env.email = 'maintenance@company.com'
env.admins = ['bob', 'jane', ]

def deploy():
    """The highest-level meta-command for deploying Plone to the server.
    Use this command only on a fresh and clean server."""

    with settings(user='root', password=env.temp_root_pass):
        create_admin_accounts(default_password='secret123')

    # security
    harden_sshd()
    disable_root_login()

    # bootstrap server
    set_hostname()
    set_system_time()
    install_unattended_upgrades()
    install_sendmail()
    install_rkhunter()

    # install software stack
    install_ipsec()

Sample buildout.cfg¶

This fabfile.py template has a dependency on the niteoweb.fabfile package and also expects to find certain buildout values and config files in certain directories. Here’s a sample buildout.cfg that you can use to prepare an environment for using this fabfile.py.in. Save the fabfile.py.in in etc/ directory in your buildout.

[buildout]
unzip = true
newest = false
extensions = buildout.dumppickedversions
prefer-final = true

parts =
    fabfile
    fabric
    racoon.conf
    psk.txt

[config]
# Project shortname
shortname = ipsec

# Main domain on which this project runs on
hostname = ipsec.company.com

# Various IPs needed for deployment
[ips]
server = ?.?.?.?

[pass]
# Temporary root password assigned to us by hosting provider
temp_root = some_password_here
ipsec = strong_password_here

# Prepare Fabric
[fabric]
recipe = zc.recipe.egg
eggs =
    Fabric
    niteoweb.fabfile

[fabfile]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/fabfile.py.in
output = ${buildout:directory}/fabfile.py

# Generate config files from templates in ./etc
[racoon.conf]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/racoon.conf.in
output = ${buildout:directory}/etc/racoon.conf

[psk.txt]
recipe = collective.recipe.template
input = ${buildout:directory}/etc/psk.txt.in
output = ${buildout:directory}/etc/psk.txt

Config files¶

Samples of config files that you need to put inside etc/ directory in your buildout:

• racoon.conf.in.
• psk.txt.in.

Client configuration¶

Configuring a client to use this IPsec server is fairly easy. For iOS, go to Settings -> Network -> VPN and add a new IPsec VPN with the following settings:

• Description: whatever
• Server: Public IP of your router behind which the IPsec server sits
• Account: a Linux user on the machine that is in the sudo group
• Group name: sudo (it’s specified in racoon.conf)
• Secret: secret set for group sudo in psk.txt

Project commands¶

niteoweb.fabfile.project.configure_nginx(shortname=None)[source]¶

Upload Nginx configuration for this site to /etc/nginx/sites-available and enable it so it gets included in the main nginx.conf.

niteoweb.fabfile.project.download_code(shortname=None, prod_user=None, svn_params=None, svn_url=None, svn_repo=None, svn_dir=None)[source]¶

Pull project code from code repository.

niteoweb.fabfile.project.download_data()[source]¶

Download Zope’s Data.fs from the server.

niteoweb.fabfile.project.enable_nginx_config(shortname=None)[source]¶

Make a link from sites-available/ to sites-enabled/ and reload Nginx.

niteoweb.fabfile.project.prepare_buildout(prod_user=None, python_version=None, production_cfg=None)[source]¶

Prepare zc.buildout environment so we can use bin/buildout -c production.cfg to build a production environment.

niteoweb.fabfile.project.run_buildout(prod_user=None, production_cfg=None)[source]¶

Run bin/buildout -c production.cfg in production user’s home folder on the production server.

niteoweb.fabfile.project.start_supervisord(prod_user=None)[source]¶

Start supervisord process monitor which in turn starts Zope and optionally others (Varnish, HAProxy, etc.).

niteoweb.fabfile.project.supervisorctl(*cmd)[source]¶

Runs an arbitrary supervisorctl command.

niteoweb.fabfile.project.upload_blobs(prod_user=None, path=None)[source]¶

Upload BLOB part of Zope’s data to the server.

niteoweb.fabfile.project.upload_data(prod_user=None)[source]¶

Upload Zope’s data to the server.

niteoweb.fabfile.project.upload_nginx_config(shortname=None, nginx_conf=None)[source]¶

Upload Nginx configuration to /etc/nginx/sites-available.

niteoweb.fabfile.project.upload_sphinx(hq_ip=None, sphinx_dir=None, path=None)[source]¶

Uploads HTML files generated by Sphinx.

niteoweb.fabfile.project.upload_zodb(prod_user=None, path=None)[source]¶

Upload ZODB part of Zope’s data to the server.

Server commands¶

niteoweb.fabfile.server.add_to_bacula_master(shortname=None, path=None, bacula_host_string=None)[source]¶

Add this server’s Bacula client configuration to Bacula master.

niteoweb.fabfile.server.configure_bacula_client(path=None)[source]¶

Upload configuration for Bacula File Deamon (client) and restart it.

niteoweb.fabfile.server.configure_bacula_master(path=None)[source]¶

Upload configuration files for Bacula Master.

niteoweb.fabfile.server.configure_egg_cache()[source]¶

Configure a system-wide egg-cache so we have a local cache of eggs that we use in order to add speed and reduncancy to zc.buildout.

niteoweb.fabfile.server.configure_hetzner_backup(duplicityfilelist=None, duplicitysh=None)[source]¶

Hetzner gives us 100GB of backup storage. Let’s use it with Duplicity to backup the whole disk.

niteoweb.fabfile.server.configure_nginx(nginx_conf=None)[source]¶

Upload Nginx configuration and restart Nginx so this configuration takes effect.

niteoweb.fabfile.server.configure_postgres()[source]¶

Upload Postgres configuration from etc/ and restart the server.

niteoweb.fabfile.server.configure_racoon(racoonconf=None, psktxt=None)[source]¶

Upload racoon configuration files and restart the service.

niteoweb.fabfile.server.configure_ufw(rules=None)[source]¶

Configure Uncomplicated Firewall.

niteoweb.fabfile.server.create_admin_account(admin, default_password=None)[source]¶

Create an account for an admin to use to access the server.

niteoweb.fabfile.server.create_admin_accounts(admins=None, default_password=None)[source]¶

Create admin accounts, so admins can access the server.

niteoweb.fabfile.server.create_project_user(prod_user)[source]¶

Add a user for a single project so the entire project can run under this user.

niteoweb.fabfile.server.create_projects_group()[source]¶

Create a group that will hold all project users -> users that are dedicated for running one project.

niteoweb.fabfile.server.disable_root_login()[source]¶

Disable root login for even more security. Access to root account is now possible by first connecting with your dedicated maintenance account and then running sudo su -.

niteoweb.fabfile.server.generate_selfsigned_ssl(hostname=None)[source]¶

Generate self-signed SSL certificates and provide them to Nginx.

niteoweb.fabfile.server.harden_sshd()[source]¶

Security harden sshd.

niteoweb.fabfile.server.initialize_postgres()[source]¶

Initialize the main database.

niteoweb.fabfile.server.install_bacula_client()[source]¶

Install and configure Bacula backup client, which listens for instructions from Bacula master and backups critical data when told to do so.

niteoweb.fabfile.server.install_bacula_master()[source]¶

Install and configure Bacula Master.

niteoweb.fabfile.server.install_ipsec(racoonconf=None, psktxt=None, server_ip=None)[source]¶

Install and configure IPsec server.

niteoweb.fabfile.server.install_java()[source]¶

Install java from webupd8 repository.

niteoweb.fabfile.server.install_munin_node(add_to_master=True)[source]¶

Install and configure Munin node, which gathers system information and sends it to Munin master.

niteoweb.fabfile.server.install_mysql(default_password=None)[source]¶

Install MySQL database server.

niteoweb.fabfile.server.install_nginx(nginx_conf=None)[source]¶

Install and configure Nginx webserver.

niteoweb.fabfile.server.install_php()[source]¶

Install FastCGI interface for running PHP scripts via Nginx.

niteoweb.fabfile.server.install_postgres()[source]¶

Install and configure Postgresql database server.

niteoweb.fabfile.server.install_python_24()[source]¶

Install Python 2.4 and tools for it.

niteoweb.fabfile.server.install_python_26()[source]¶

Install Python 2.6 and tools for it.

niteoweb.fabfile.server.install_python_27()[source]¶

Install Python 2.7 and tools for it.

niteoweb.fabfile.server.install_rkhunter(email=None)[source]¶

Install and configure RootKit Hunter.

niteoweb.fabfile.server.install_sendmail(email=None)[source]¶

Prepare a localhost SMTP server for sending out system notifications to admins.

niteoweb.fabfile.server.install_system_libs(additional_libs=None)[source]¶

Install a bunch of stuff we need for normal operation such as gcc, rsync, vim, libpng, etc.

niteoweb.fabfile.server.install_ufw(rules=None)[source]¶

Install and configure Uncomplicated Firewall.

niteoweb.fabfile.server.install_unattended_upgrades(email=None)[source]¶

Configure Ubuntu to automatically install security updates.

niteoweb.fabfile.server.normalize_rackspace()[source]¶

docstring for normalize_rackspace

niteoweb.fabfile.server.raid_monitoring(email=None)[source]¶

Configure monitoring of our RAID-1 field. If anything goes wrong, send an email!

niteoweb.fabfile.server.set_hostname(server_ip=None, hostname=None)[source]¶

Set server’s hostname.

niteoweb.fabfile.server.set_system_time(timezone=None)[source]¶

Set timezone and install ntp to keep time accurate.