svSite

svSite is a project to create reusable website code that is usable for small or medium sized associations. It will be usable for communication within the organisation, creating and promoting events, displaying news and static information, etc.

Status

Under development; not usable yet. Please stand by for release v1.0!

License

The code is available under the BSD License. You can free (and encouraged) to use it as you please, but at your own risk.

Using

Installation

svSite should provide a fully functional site with minimal work.

Although later on you may want to personalize the look (info), which will take time. That’s inevitable.

To get svSite running, follow the steps in the appropriate section.

Linux / bash

Installing dependencies

For this to work, you will need python3-dev including pip and a database (sqlite3 is default and easy, but slow). Things will be easier and better with virtualenv or pew and git, so probably get those too. You’ll also need libjpeg-dev and the dev version of Python because of pillow. You can install them with:

sudo apt-get install python3.4-dev sqlite3 git libjpeg-dev python-pip
sudo apt-get install postgresql libpq-dev       # for postgres, only if you want that database
sudo apt-get install mysql-server mysql-client  # for mysql, only if you want that database

Make sure you use the python3.X-dev that matches your python version (rather than python3-dev). If there are problems, you might need these packages.

Now get the code. The easiest way is with git, replacing SITENAME:

git clone https://github.com/mverleg/svsite.git SITENAME

Enter the directory (cd SITENAME).

Starting a virtual environment is recommended (but optional), as it keeps this project’s Python packages separate from those of other projects. If you know how to do this, just do it your way. This is just one of the convenient ways:

sudo pip install -U pew
pew new --python=python3 sv

If you skip this step, everything will be installed system-wide, so you need to prepend sudo before any pip command. Also make sure you’re installing for Python 3.

Install the necessary Python dependencies through:

pip install -r dev/requires.pip
pip install psycopg2     # for postgres, only if you want that database
pip install mysqlclient  # for mysql, only if you want that database
Development

If you want to run tests, build the documentation or do anything other than simply running the website, you should install (otherwise skip it):

pip install -r dev/requires_dev.pip  # optional
Database

We need a database. SQLite is used by default, which you could replace now or later (see local settings) for a substantial performance gain. To create the structure and an administrator, type this and follow the steps:

python3 source/manage.py migrate
python3 source/manage.py createsuperuser
Static files

Then there are static files we need, which are handles by bower by default [1]. On Ubuntu, you can install bower using:

sudo apt-get install nodejs
npm install bower

After that, install the static files and connect them:

python3 source/manage.py bower install
python3 source/manage.py collectstatic --noinput
Starting the server

Then you can start the test-server. This is not done with the normal runserver command but with

python3 source/manage.py runsslserver localhost.markv.nl:8443 --settings=base.settings_development

We use this special command to use a secure connection, which is enforced by default. In this test mode, an unsigned certificate is used, so you might have to add a security exception.

You can replace the url and port. You can stop the server with ctrl+C.

Next time

To (re)start the server later, go to the correct directory and run:

pew workon sv  # only if you use virtualenv
python3 source/manage.py runsslserver localhost.markv.nl:8443 --settings=base.settings_development

This should allow for easy development and testing.

Footnotes

[1]If you don’t want to install node and bower, you can easily download the packages listed in dev/bower/json by hand and put them in env/bower. Make sure they have a dist subdirectory where the code lives. You still need to run the collectstatic command if you do this.

Automatic tests

There are only few automatic tests at this time, but more might be added. You are also more than welcome to add more yourself. The tests use py.test with a few addons, which are included in dev/requires_dev.pip. If you installed those packages, you can run the tests by simply typing py.test in the root directory. It could take a while (possibly half a minute).

Going live

Everything working and ready to launch the site for the world to see? Read Going live!

Machine-specific settings

Some settings are machine-dependent, so you need to create local.py containing these settings. This file should be in the same directory as settings.py, so typically source/local.py.

At least, your local settings should contain:

from os.path import dirname, join

BASE_DIR = dirname(dirname(__file__))

SITE_URL = 'svleo.markv.nl'  #todo: update url
ALLOWED_HOSTS = [SITE_URL, 'localhost']

SITE_DISP_NAME = 'svSite'  #todo: update site name and tagline
SITE_DISP_TAGLINE = 'Make your own website for your group!'

SECRET_KEY = ''  #todo: generate a long random string

DATABASES = {  #todo: choose some database settings
        'default': {
                'ENGINE': 'django.db.backends.postgresql_psycopg2',
                'NAME': 'database',
                'USER': 'username',
                'PASSWORD': 'PASSWORD',
                'HOST': '127.0.0.1',
                'CONN_MAX_AGE': 120,
        }
}
# alternatively, as a deveopment database:
# DATABASES = {
#       "default": {
#               "ENGINE": "django.db.backends.sqlite3",
#               "NAME": join(BASE_DIR, 'dev', 'data.sqlite3'),
#       }
# }

MEDIA_ROOT = join('data', 'media', 'svleo')
STATIC_ROOT = join('data', 'static', 'svleo')
CMS_PAGE_MEDIA_PATH = join(MEDIA_ROOT, 'cms')
SV_THEMES_DIR = join(BASE_DIR, 'themes')

You can create a secret key using random.org (join both together), or generate a better one yourself with bash:

</dev/urandom tr -dc '1234567890!@#$%&*--+=__qwertQWERTasdfgASDFGzxcvbZXCVB' | head -c 32

You might also want to have a look at some of these:

SV_DEFAULT_THEME = 'standard'

TIME_ZONE = 'Europe/Amsterdam'
LANGUAGE_CODE = 'nl'
LANGUAGES = (
        ('nl', ('Dutch')),  # using gettext_noop here causes a circular import
        ('en', ('English')),
)

CACHES = {
        'default': {
                'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
                'LOCATION': '127.0.0.1:11211',
        }
}

# You have to redifine TEMPLATES if you want to add a template path or change TEMPLATE_DEBUG

INTERNAL_IPS = []  # these ips are treated differently if a problem occurs

# more info: https://docs.djangoproject.com/en/dev/topics/logging/#configuring-logging
LOGGING = {
        'version': 1,
        'disable_existing_loggers': False,
        'handlers': {
                'file': {
                        'level': 'DEBUG',
                        'class': 'logging.FileHandler',
                        'filename': '/path/to/django/debug.log',  # change this path
                },
        },
        'loggers': {
                'django': {
                        'handlers': ['file'],
                        'level': 'DEBUG',
                        'propagate': True,
                },
        },
}

SESSION_COOKIE_SECURE = CSRF_COOKIE_SECURE = False

DEBUG = FILER_DEBUG = False

You can change other Django settings, particularly it might be worthwhile to have a look at globalization settings.

Going live

There are many ways to make a Django site accessible to the world. The method in the official documentation is using Apache and mod_wsgi. You’re free to use any method. After setting up the server with https, don’t forget to check the second part of the document on how to set up a secure``https`` connection.

Server (no https yet)

wsgi

This is the “normal” wsgi way. It is explained in the official documentation. This section is just an example of a complete Apache configuration file. Where to put this depends on your operating system and setup; for Ubuntu it’s in /etc/apache2/sites-available/svsite (then do a2ensite svsite and sudo service apache2 reload):

<VirtualHost *:80>
        ServerName domain.com
        ServerAlias *.domain.com

        Options -Indexes

        # PLACE HTTPS STUFF HERE. More on that later.

        # Run WSGI in daemon mode (separate process for each Django site)
        # python-path should point to your virtual environment
        # /live/svsite should be the location of your code
        WSGIDaemonProcess svsite python-path=/path_to_virtualenv/lib/python3.4/site-packages
        WSGIProcessGroup svsite
        WSGIScriptAlias / /live/svsite/source/wsgi.py

        # This is for static files, which should be served by Apache without Django's help
        # There is some cache stuff, which you can turn off by removing it
        Alias /static/ /data/static/svsite/
        <Directory /data/static/svsite/>
                ExpiresActive On
                ExpiresDefault "access plus 1 day"
                Header append Cache-Control "public"
                Options -Indexes
                Order deny,allow
                Allow from all
        </Directory>

        # Media is similar to static, but without cache
        # (note that anyone can access any files if they have the url)
        Alias /media/ /data/media/svsite/
        <Directory /data/media/svsite/>
                Options -Indexes
                Order deny,allow
                Allow from all
        </Directory>

        # If you want to protect files but let Apache serve them, use Xsend
        # XSendFile On
        # XSendFilePath /data/media/svsite/

        # Apache logs (don't forget to set up logging in Django settings)
        LogLevel info
        ErrorLog ${APACHE_LOG_DIR}/svsite-error.log
        CustomLog ${APACHE_LOG_DIR}/svsite-access.log common
</VirtualHost>

A downside of this method is that all your websites must use the same python; you can’t have one using python2 and another using python3. It also allows you to restart Django without root privileges. These might be unimportant, but if they are, use another method, like the next one.

wsgi-express

This alternative method is a variation on the official one. It also uses Apache and mod_wsgi, but mod_wsgi is part of Python instead of Apache. An advantage of this setup is that you can have different websites with different Python versions, which is not otherwise possible.

This relies on mod_wsgi, which should already be installed in your virtual environment (otherwise, remember the pew stuff? Do that and pip install mod_wsgi).

You can test that it works with (user and group are optional, it’s safe to use the correct permissions when live later though):

python source/manage.py runmodwsgi --log-to-terminal --user www-data --group devs --host=localhost --port 8081 --pythonpath=/path-to-virtualenv/lib/python3.4/site-packages source/wsgi.py

which should let you visit localhost:8081 and see the site. If it does not work, have a look at mod_wsgi pypi page.

The idea is to run a wsgi server, and let Apache proxypass the requests to it. Here is an example of the Apache settings for a non-HTTPS setup (which should be added later). Depending on your setup, this might belong in /etc/apache2/sites-available/svsite:

<VirtualHost *:80>
        ServerName domain.com
        ServerAlias *.domain.com

        Options -Indexes

        # PLACE HTTPS STUFF HERE. More on that later.

        # This is for static files, which should be served by Apache without Django's help
        # There is some cache stuff, which you can turn off by removing it.
        # Need to make a ProxyPass exception, since ProxyPass is handled
        # before Alias so it swallows everything otherwise.
        Alias /static /data/static/svsite
        ProxyPass /static !
        <Directory /data/static/svsite/>
                ExpiresActive On
                ExpiresDefault "access plus 1 day"
                Header append Cache-Control "public"
                Options -Indexes
                Order deny,allow
                Allow from all
        </Directory>

        # Media is similar to static, but without cache.
        # (note that anyone can access any files if they have the url)
        Alias /media /data/media/svsite
        ProxyPass /media !
        <Directory /data/media/svsite/>
                Options -Indexes
                Order deny,allow
                Allow from all
        </Directory>

        # This is the core part: all the non-static traffic is just sent to wsgi.
        # `retry=0` causes Apache to retry to contact wsgi every time, even if it got no response last time
        ProxyPass / http://localhost:8081/ retry=0
        ProxyPassReverse / http://localhost:8081/

        # Apache logs (don't forget to set up logging in Django settings).
        LogLevel info
        ErrorLog ${APACHE_LOG_DIR}/svsite-error.log
        CustomLog ${APACHE_LOG_DIR}/svsite-access.log common
</VirtualHost>

Use a2ensite svsite and sudo service apache2 reload.

Then we need to make sure that the wsgi server is always running. There are many ways. On Ubuntu and possibly other related systems, one can use Upstart. Here is an example configuration file, which should go in /etc/init/svsite:

description "Always run the wsgi daemon for svsite website"

# automatically start on boot
start on filesystem or runlevel [2345]

# automatically stop on shutdown
stop on shutdown or runlevel [!2345]

# restart if it stops for any reason other than you manually stopping it
respawn

# this is the code that starts the process (update the parths and user/group)
script
        cd /live/svsite
        /path_to_virtualenv/bin/python3.4 source/manage.py runmodwsgi --log-to-terminal --user www-data --group devs --host=localhost --port 8081 --pythonpath=/path_to_virtualenv/svsite/lib/python3.4/site-packages source/wsgi.py
end script

# make sure the wsgi process is gone, otherwise you can't restart
post-stop script
    kill $(cat /var/run/svleo.pid)
    rm -f /var/run/svleo.pid
end script

After saving this, you can use these self-explanatory commands:

sudo service svsite status
sudo service svsite start
sudo service svsite stop

If both svsite and apache2 are running, you should then be able to visit your site! What happens is that you visit it on port 80 (or 443 after the next section) and it arrives at Apache. In case of static or media files, Apache sends the files (possibly with caching headers). Otherwise, it asks the wsgi server on port 8081 for the page, which Django responds.

The server should not be reachable on port 8081 (http://domain.com:8081/) from the outside words. You might also want to check that the wsgi server (and apache and the database) automatically start on reboot (by rebooting).

Secure connection (https)

After using one of the setup methods, it’s highly recommended that you set up a secure connection. Now that letsencrypt offers free certificates (donations appreciated), there are few good excuses left not to. One method will be documented, but there are many.

Apache & letsencrypt

This section will explain how to do it for Apache with letsencrypt, so it can be used with either of the above setups. There are other options, which are documented online.

First, generate a certificate (more details here) by running the following commands), answering as appropriate. This will place letsencrypt in the current directory, so move to the directory where you want it first.:

# get the code and stop Apache
git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
sudo service apache2 stop
# request the certificate (change the domains)
sudo ./letsencrypt-auto certonly --standalone -d domain.com -d www.domain.com

The certificate files should be stored in /etc/letsencrypt/live/domain.com/ (with your domain). If the above command reports another location, use that.

Now we need to update the Apache configuration. First, change the port in the first line from 80 to 443:

<VirtualHost *:80>   # old one
<VirtualHost *:443>  # new one

Place the below (with updated paths) in your Apache config inside the <VirtualHost *:443> (as marked with a comment above):

SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/chain.pem
# Header always set Strict-Transport-Security "max-age=2600000; preload"

And at the bottom add (if you want all requests to be secure):

<VirtualHost *:80>
        ServerName domain.com
        ServerAlias *.domain.com

        Redirect permanent / https://domain.com/
</VirtualHost>

The last line tells browsers to not access your site through http for a long time. Only enable it when you are confident things are working and will keep working! It’s good for security, making it hard for attackers to divert traffic to http, but it’ll make your site inaccessible if https stops working.

Now just restart Apache and see if things work:

sudo service apache2 restart

You can update local.py with (at least):

SESSION_COOKIE_SECURE = CSRF_COOKIE_SECURE = False

You’ll need to refresh your https certificates every few months. Don’t forget to do that!

Layout

svSite should provide a fully functional site (with minimal work), but you may want to personalize the look, which will take time. That’s inevitable. Here is some information on how to do it.

Theme requirements

To create your own theme, these are the requirements:

  • The files should be organized into directories templates, static and info:

    • The templates directory should contain base.html holding the theme body and optionally head.html holding anything in <head> (you might want to include default_head.html). Except for that, it can contain any templates you want (these are only used if you explicitly include them).
    • The static directory should contain any static files you use (see below on how to use them).
    • The info directory can contain any of these files: readme.rst, description.rst, credits.rst and license.txt. Other files can be included but nothing special happens with them.
  • Include static css/js files using:

    {% load addtoblock from sekizai_tags %}
    {% addtoblock "css" %}
            <link rel="stylesheet" href="{% static THEME_PREFIX|add:'/css/style.css' %}">
    {% endaddtoblock "css" %}
    

    and other static files:

    {% load static from staticfiles %}
    <img src="{% static THEME_PREFIX|add:'/logo.png' %}" />
    

    You can also hard-code {% static 'theme_name/logo.png' %}. This behaves differently in case another theme extends this one.

  • For the base.html template:

    • It should not extend anything (it is itself included).

    • It should define precisely these placeholders:

      {% placeholder "header" %}
      {% placeholder "top-row" %}
      {% placeholder "content" %}
      {% placeholder "sidebar" %}
      {% placeholder "bottom-row" %}
      
    • It should {% include include_page %} if it’s set, e.g. a structure like this:

      {% if page_include %}
              {% include page_include %}
      {% else %}
              {% placeholder "content" %}
      {% endif %}
      
    • You do not need to define {% block %} . You won’t be able to extend them since Django doesn’t let you extend blocks from included templates.

Extending

Making changes

svSite should let you get a site running with a (somewhat) limited amount of work. But perhaps there are still some details you want to change. And you can! Given some knowledge of Django (features) and/or html/css/js (layout), you can create your copy of the code and occasionally get updates from the main project.

And perhaps your changes turn out great, and you want to share them. That is greatly appreciated, and this page tells you how to do it!

Contribute

You can contribute by adding to the project! This includes:

Making changes

You will need to make sure you can push code to Github by setting up ssh keys. Then fork the svsite repository and follow these steps.

You will need python3, pip, a database (sqlite3 is default and easy, but slow), virtualenv, git, elasticsearch and some SSL packages. Just type:

sudo apt-get install python3 sqlite3 python-virtualenv git build-essential libssl-dev libffi-dev python-dev elasticsearch

Get your copy of the svsite code:

git clone git@github.com:YOUR_SVSITE_FORK.git

Go to the directory, start a virtualenv and install:

virtualenv -p python3 env
source env/bin/activate
pip install --requirement dev/pip_freeze.txt
pip install --no-deps --editable .

To create the database and superuser:

python3 source/manage.py migrate
python3 source/manage.py createsuperuser

You might want to run the tests:

py.test tests

Then you can start the server on localhost:

python3 source/manage.py runserver_plus --settings=base.settings_development

You can now open the site in a browser. It is running on localhost over https on port 8443.

If www. is prepended, you can use a domain that works with that prefix (not localhost:8443). For example,

This refers to your localhost (127.0.0.1). The first time you will probably need to add a security exception, as this is a debug SSL certificate.

Now you are ready to make your chances!

After you are done and have tested your changes (and converted space-indents to tabs), you can suggest it for inclusion into svsite by means of a pull request

External services

There is a minimal api for building external services, which is described in Integration API. Such additions are welcome, you’re encouraged to notify us when you complete one!

Automated tests

Automatic testing is currently very limited for the project. We use py.test ones, which can be stored in /test/ or source/$app/test. It’s greatly appreciated if you add more, for your own additions or for existing code. It’ll help ensure the quality of the codebase!

A general note

Good luck! Why we never forget our fellow coders

Integration API

There is a https-json api for integrating external tools. It provides read-only access to information about users (member) and groups (team).

Server setup

To allow client services access to some member and team info through the http api, you only need to change a few settings.

These settings should go in local.py, since they are installation dependent and should not be accessible to outsiders.

  • Add setting INTEGRATION_KEYS, which should be a list of keys:

    INTEGRATION_KEYS = [
      'abc123',      # php widget
      'password!',   # android app
    ]
    

    It is advisable to add a new key for each service, so that you can revoke them individually, if the need arises. Generate keys at random.org .

  • Add setting INTEGRATION_ALLOW_EMAIL which can be True or False (the default). Services can only request a list of email addresses if this is True (option email=yes). Otherwise, services can only get a user’s email when that user logs in to the service.

  • Restart the server.

Also note that, though https is always important, it is even more important with this api, since both api keys and user credentials and info will be sent over plain http if you don’t have a secure connection.

External tool setup

To get the information, send a POST request to one of the API urls. The urls for your server can be found at an info page for the server, which is usually /$intapi/. This is the info send to each of the urls:

Info Default url Input Output Note
(info page) /$intapi/members/ (nothing) url & config info can use GET
Member list /$intapi/members/ key, optionally email=yes list of usernames if email, dict username->email
Member /$intapi/member/ key, username, password user info map can authenticate if successful
Team list /$intapi/teams/ key team name list no hidden teams
Team /$intapi/team/ key, teamname team info map works for hidden teams

The option email=yes only works if the server has INTEGRATION_ALLOW_EMAIL set to True.

If all goes well, the result will be a string containing a JSON list or map. Otherwise you will get an error message and a non-200 status code.

It is recommended that you associate the relevant user data with that user’s session in a safe way (rather than store it in a database), as you will get a fresh copy each time the users logs in.

As an example (Bash terminal, but others like PHP should be similar):

$ curl --show-error --request POST 'https://domain.com/$intapi/members/' --data-urlencode "key=abc123"
[
  "mark",
  "henk"
]
$ curl --show-error --request POST 'https://domain.com/$intapi/members/' --data-urlencode "key=abc123" --data-urlencode "email=yes"
{
  "mark": "mark@spam.la",
  "henk": ""
}
$ curl --show-error --request POST 'https://domain.com/$intapi/member/' --data-urlencode "key=abc123" --data-urlencode "username=mark" --data-urlencode "password=drowssap"
{
  "username": "mark",
  "first_name": "Mark",
  "last_name": "V",
  "email": "mark@spam.la",
  "birthday": null,
  "teams": {
        "Tokkies": "Mastersjief"
  }
}
$ curl --show-error --request POST 'https://domain.com/$intapi/teams/' --data-urlencode "key=abc123"
[
  "Tokkies"
]
$ curl --show-error --request POST 'https://domain.com/$intapi/team/' --data-urlencode "key=abc123" --data-urlencode "teamname=Tokkies"
{
  "hidden": false,
  "teamname": "Tokkies",
  "description": "You know, from TV?",
  "leaders": [
        "mark"
  ],
  "members": {
        "mark": "Mastersjief"
  }
}

Good luck!

Models

The models and their relations can be seen in this graph:

With graphviz and django-extensions you can generate this image yourself:

python source/manage.py graph_models --all --settings=base.settings_development | grep -v '^  //' | grep -v '^[[:space:]]*$$' > images/models.dot

Design notes (hacks)

Some parts are less than elegant. Although, at the time of writing, it seems there may not be a better way, it warrants a warning anyway.

Migrating

Clean migrations don’t quite work for some cms addons. Find the migration info in the installation documentation.

Themes

Djangocms seems not designed to handle dynamic templates, so a fixed template is used that dynamically includes the theme template based on a context variable.

Since djangocms uses sekizai, which must have it’s render_block be in the top template, it is necessary to have the <head> and <body> in this top template, and to include only the rest of the content of these tags.

Furthermore, the CMS does some kind of pre-render without context to find the placeholders to be filled. This means placeholders cannot depend on the theme (=context). Placeholders are defined in default_body.html and themes should match those.

Special pages

This relates to those pages (e.g. search results) that should not be plugins in the CMS, but should be integrated into it anyway (to be in the menu, be moved and allow placeholders).

What I would have preferred to do would be to have such pages (as apphooks) extend the main template and overwrite {% block content %}. However, because of themes, {% block content %} is necessarily defined in an {% include %} file. Django cannot extend blocks defined in included files (regrettably) since they are each rendered separately (not so much ‘included’), making the block useless.

The ‘solution’ used is to force templates to include a dynamic template instead of the content placeholder for such pages.

{% if page_include %}
        {% include page_include %}
{% else %}
        {% placeholder "content" %}
{% endif %}

There is a special version of render, namely base.render_cms_special, that you can use like this:

def my_view(request):
        value = 'do some query or something'
        return render_cms_special(request, 'my_template.html', dict(
                key=value,
        ))

It is important to note that my_template.html in this example should render just the content part, not the full page. Don’t {% extend %} the base template (or anything, for that matter); this is done automatically.

In order for placeholderes to show up and for things to be integrated into the cms, you will need to add this view/app as an app-hook (this is the normal way; the only difference is that you should use render_cms_special).

Users & groups

#todo - built-in Django groups - CMS users

More things

Frequently...

...asked questions

How can I contribute?

Okay, this one hasn’t been asked “frequently” in the strict meaning of the word, but anyway. Glad you’re interested! Your help is welcome! Please check the contribute section.

...encountered problems

  • After updating, you might get:

    KeyError at /en/stuff/
    'SomethingPlugin'
    

    This means a plugin was removed but is still in the database. Just run:

    python source/manage.py cms delete_orphaned_plugins --noinput
    

    if you were using the plugin that was removed, then those use cases will be gone. The alternative is reverting the update.

ElasticSearch / Haystack can’t connect

You can test that elasticsearch is running using a http request on port 9200, like so:

curl -X GET http://127.0.0.1:9200

If it isn’t, there could be a number of reasons.

In my case, I had to set START_DAEMON=true in /etc/default/elasticsearch (source)

You might also have the wrong version of the Python binding, see here