svSite¶
svSite is a project to create reusable website code that is usable for small or medium sized associations. It will be usable for communication within the organisation, creating and promoting events, displaying news and static information, etc.
Status¶
Under development; not usable yet. Please stand by for release v1.0!
Resources¶
- Documentation: http://svsite.readthedocs.org/
- Bugs, features: https://github.com/mverleg/svsite/issues
License¶
The code is available under the BSD License. You can free (and encouraged) to use it as you please, but at your own risk.
Using¶
Installation¶
svSite should provide a fully functional site with minimal work.
Although later on you may want to personalize the look (info), which will take time. That’s inevitable.
To get svSite running, follow the steps in the appropriate section.
Linux / bash¶
Installing dependencies¶
For this to work, you will need python3-dev
including pip
and a database (sqlite3
is default and easy, but slow). Things will be easier and better with virtualenv
or pew
and git
, so probably get those too. You’ll also need libjpeg-dev
and the dev version of Python because of pillow
. You can install them with:
sudo apt-get install python3.4-dev sqlite3 git libjpeg-dev python-pip
sudo apt-get install postgresql libpq-dev # for postgres, only if you want that database
sudo apt-get install mysql-server mysql-client # for mysql, only if you want that database
Make sure you use the python3.X-dev
that matches your python version (rather than python3-dev
). If there are problems, you might need these packages.
Now get the code. The easiest way is with git, replacing SITENAME
:
git clone https://github.com/mverleg/svsite.git SITENAME
Enter the directory (cd SITENAME
).
Starting a virtual environment is recommended (but optional), as it keeps this project’s Python packages separate from those of other projects. If you know how to do this, just do it your way. This is just one of the convenient ways:
sudo pip install -U pew
pew new --python=python3 sv
If you skip this step, everything will be installed system-wide, so you need to prepend sudo
before any pip command. Also make sure you’re installing for Python 3.
Install the necessary Python dependencies through:
pip install -r dev/requires.pip
pip install psycopg2 # for postgres, only if you want that database
pip install mysqlclient # for mysql, only if you want that database
Development¶
If you want to run tests, build the documentation or do anything other than simply running the website, you should install (otherwise skip it):
pip install -r dev/requires_dev.pip # optional
Database¶
We need a database. SQLite is used by default, which you could replace now or later (see local settings) for a substantial performance gain. To create the structure and an administrator, type this and follow the steps:
python3 source/manage.py migrate
python3 source/manage.py createsuperuser
Static files¶
Then there are static files we need, which are handles by bower by default [1]. On Ubuntu, you can install bower using:
sudo apt-get install nodejs
npm install bower
After that, install the static files and connect them:
python3 source/manage.py bower install
python3 source/manage.py collectstatic --noinput
Starting the server¶
Then you can start the test-server. This is not done with the normal runserver
command but with
python3 source/manage.py runsslserver localhost.markv.nl:8443 --settings=base.settings_development
We use this special command to use a secure connection, which is enforced by default. In this test mode, an unsigned certificate is used, so you might have to add a security exception.
You can replace the url and port. You can stop the server with ctrl+C
.
Next time¶
To (re)start the server later, go to the correct directory and run:
pew workon sv # only if you use virtualenv
python3 source/manage.py runsslserver localhost.markv.nl:8443 --settings=base.settings_development
This should allow for easy development and testing.
Footnotes
[1] | If you don’t want to install node and bower, you can easily download the packages listed in dev/bower/json by hand and put them in env/bower. Make sure they have a dist subdirectory where the code lives. You still need to run the collectstatic command if you do this. |
Automatic tests¶
There are only few automatic tests at this time, but more might be added. You are also more than welcome to add more yourself. The tests use py.test
with a few addons, which are included in dev/requires_dev.pip
. If you installed those packages, you can run the tests by simply typing py.test
in the root directory. It could take a while (possibly half a minute).
Going live¶
Everything working and ready to launch the site for the world to see? Read Going live!
Machine-specific settings¶
Some settings are machine-dependent, so you need to create local.py
containing these settings. This file should be in the same directory as settings.py
, so typically source/local.py
.
At least, your local settings should contain:
from os.path import dirname, join
BASE_DIR = dirname(dirname(__file__))
SITE_URL = 'svleo.markv.nl' #todo: update url
ALLOWED_HOSTS = [SITE_URL, 'localhost']
SITE_DISP_NAME = 'svSite' #todo: update site name and tagline
SITE_DISP_TAGLINE = 'Make your own website for your group!'
SECRET_KEY = '' #todo: generate a long random string
DATABASES = { #todo: choose some database settings
'default': {
'ENGINE': 'django.db.backends.postgresql_psycopg2',
'NAME': 'database',
'USER': 'username',
'PASSWORD': 'PASSWORD',
'HOST': '127.0.0.1',
'CONN_MAX_AGE': 120,
}
}
# alternatively, as a deveopment database:
# DATABASES = {
# "default": {
# "ENGINE": "django.db.backends.sqlite3",
# "NAME": join(BASE_DIR, 'dev', 'data.sqlite3'),
# }
# }
MEDIA_ROOT = join('data', 'media', 'svleo')
STATIC_ROOT = join('data', 'static', 'svleo')
CMS_PAGE_MEDIA_PATH = join(MEDIA_ROOT, 'cms')
SV_THEMES_DIR = join(BASE_DIR, 'themes')
You can create a secret key using random.org (join both together), or generate a better one yourself with bash:
</dev/urandom tr -dc '1234567890!@#$%&*--+=__qwertQWERTasdfgASDFGzxcvbZXCVB' | head -c 32
You might also want to have a look at some of these:
SV_DEFAULT_THEME = 'standard'
TIME_ZONE = 'Europe/Amsterdam'
LANGUAGE_CODE = 'nl'
LANGUAGES = (
('nl', ('Dutch')), # using gettext_noop here causes a circular import
('en', ('English')),
)
CACHES = {
'default': {
'BACKEND': 'django.core.cache.backends.memcached.MemcachedCache',
'LOCATION': '127.0.0.1:11211',
}
}
# You have to redifine TEMPLATES if you want to add a template path or change TEMPLATE_DEBUG
INTERNAL_IPS = [] # these ips are treated differently if a problem occurs
# more info: https://docs.djangoproject.com/en/dev/topics/logging/#configuring-logging
LOGGING = {
'version': 1,
'disable_existing_loggers': False,
'handlers': {
'file': {
'level': 'DEBUG',
'class': 'logging.FileHandler',
'filename': '/path/to/django/debug.log', # change this path
},
},
'loggers': {
'django': {
'handlers': ['file'],
'level': 'DEBUG',
'propagate': True,
},
},
}
SESSION_COOKIE_SECURE = CSRF_COOKIE_SECURE = False
DEBUG = FILER_DEBUG = False
You can change other Django settings, particularly it might be worthwhile to have a look at globalization settings.
Going live¶
There are many ways to make a Django site accessible to the world. The method in the official documentation is using Apache
and mod_wsgi
. You’re free to use any method. After setting up the server with https, don’t forget to check the second part of the document on how to set up a secure``https`` connection.
Server (no https yet)¶
wsgi¶
This is the “normal” wsgi way. It is explained in the official documentation. This section is just an example of a complete Apache configuration file. Where to put this depends on your operating system and setup; for Ubuntu it’s in /etc/apache2/sites-available/svsite
(then do a2ensite svsite
and sudo service apache2 reload
):
<VirtualHost *:80>
ServerName domain.com
ServerAlias *.domain.com
Options -Indexes
# PLACE HTTPS STUFF HERE. More on that later.
# Run WSGI in daemon mode (separate process for each Django site)
# python-path should point to your virtual environment
# /live/svsite should be the location of your code
WSGIDaemonProcess svsite python-path=/path_to_virtualenv/lib/python3.4/site-packages
WSGIProcessGroup svsite
WSGIScriptAlias / /live/svsite/source/wsgi.py
# This is for static files, which should be served by Apache without Django's help
# There is some cache stuff, which you can turn off by removing it
Alias /static/ /data/static/svsite/
<Directory /data/static/svsite/>
ExpiresActive On
ExpiresDefault "access plus 1 day"
Header append Cache-Control "public"
Options -Indexes
Order deny,allow
Allow from all
</Directory>
# Media is similar to static, but without cache
# (note that anyone can access any files if they have the url)
Alias /media/ /data/media/svsite/
<Directory /data/media/svsite/>
Options -Indexes
Order deny,allow
Allow from all
</Directory>
# If you want to protect files but let Apache serve them, use Xsend
# XSendFile On
# XSendFilePath /data/media/svsite/
# Apache logs (don't forget to set up logging in Django settings)
LogLevel info
ErrorLog ${APACHE_LOG_DIR}/svsite-error.log
CustomLog ${APACHE_LOG_DIR}/svsite-access.log common
</VirtualHost>
A downside of this method is that all your websites must use the same python; you can’t have one using python2 and another using python3. It also allows you to restart Django without root privileges. These might be unimportant, but if they are, use another method, like the next one.
wsgi-express¶
This alternative method is a variation on the official one. It also uses Apache
and mod_wsgi
, but mod_wsgi
is part of Python instead of Apache. An advantage of this setup is that you can have different websites with different Python versions, which is not otherwise possible.
This relies on mod_wsgi
, which should already be installed in your virtual environment (otherwise, remember the pew
stuff? Do that and pip install mod_wsgi
).
You can test that it works with (user and group are optional, it’s safe to use the correct permissions when live later though):
python source/manage.py runmodwsgi --log-to-terminal --user www-data --group devs --host=localhost --port 8081 --pythonpath=/path-to-virtualenv/lib/python3.4/site-packages source/wsgi.py
which should let you visit localhost:8081 and see the site. If it does not work, have a look at mod_wsgi pypi page.
The idea is to run a wsgi server, and let Apache proxypass the requests to it. Here is an example of the Apache settings for a non-HTTPS setup (which should be added later). Depending on your setup, this might belong in /etc/apache2/sites-available/svsite
:
<VirtualHost *:80>
ServerName domain.com
ServerAlias *.domain.com
Options -Indexes
# PLACE HTTPS STUFF HERE. More on that later.
# This is for static files, which should be served by Apache without Django's help
# There is some cache stuff, which you can turn off by removing it.
# Need to make a ProxyPass exception, since ProxyPass is handled
# before Alias so it swallows everything otherwise.
Alias /static /data/static/svsite
ProxyPass /static !
<Directory /data/static/svsite/>
ExpiresActive On
ExpiresDefault "access plus 1 day"
Header append Cache-Control "public"
Options -Indexes
Order deny,allow
Allow from all
</Directory>
# Media is similar to static, but without cache.
# (note that anyone can access any files if they have the url)
Alias /media /data/media/svsite
ProxyPass /media !
<Directory /data/media/svsite/>
Options -Indexes
Order deny,allow
Allow from all
</Directory>
# This is the core part: all the non-static traffic is just sent to wsgi.
# `retry=0` causes Apache to retry to contact wsgi every time, even if it got no response last time
ProxyPass / http://localhost:8081/ retry=0
ProxyPassReverse / http://localhost:8081/
# Apache logs (don't forget to set up logging in Django settings).
LogLevel info
ErrorLog ${APACHE_LOG_DIR}/svsite-error.log
CustomLog ${APACHE_LOG_DIR}/svsite-access.log common
</VirtualHost>
Use a2ensite svsite
and sudo service apache2 reload
.
Then we need to make sure that the wsgi server is always running. There are many ways. On Ubuntu and possibly other related systems, one can use Upstart. Here is an example configuration file, which should go in /etc/init/svsite
:
description "Always run the wsgi daemon for svsite website"
# automatically start on boot
start on filesystem or runlevel [2345]
# automatically stop on shutdown
stop on shutdown or runlevel [!2345]
# restart if it stops for any reason other than you manually stopping it
respawn
# this is the code that starts the process (update the parths and user/group)
script
cd /live/svsite
/path_to_virtualenv/bin/python3.4 source/manage.py runmodwsgi --log-to-terminal --user www-data --group devs --host=localhost --port 8081 --pythonpath=/path_to_virtualenv/svsite/lib/python3.4/site-packages source/wsgi.py
end script
# make sure the wsgi process is gone, otherwise you can't restart
post-stop script
kill $(cat /var/run/svleo.pid)
rm -f /var/run/svleo.pid
end script
After saving this, you can use these self-explanatory commands:
sudo service svsite status
sudo service svsite start
sudo service svsite stop
If both svsite
and apache2
are running, you should then be able to visit your site! What happens is that you visit it on port 80 (or 443 after the next section) and it arrives at Apache. In case of static or media files, Apache sends the files (possibly with caching headers). Otherwise, it asks the wsgi server on port 8081 for the page, which Django responds.
The server should not be reachable on port 8081 (http://domain.com:8081/) from the outside words. You might also want to check that the wsgi server (and apache and the database) automatically start on reboot (by rebooting).
Secure connection (https)¶
After using one of the setup methods, it’s highly recommended that you set up a secure connection. Now that letsencrypt offers free certificates (donations appreciated), there are few good excuses left not to. One method will be documented, but there are many.
Apache & letsencrypt¶
This section will explain how to do it for Apache
with letsencrypt
, so it can be used with either of the above setups. There are other options, which are documented online.
First, generate a certificate (more details here) by running the following commands), answering as appropriate. This will place letsencrypt
in the current directory, so move to the directory where you want it first.:
# get the code and stop Apache
git clone https://github.com/letsencrypt/letsencrypt
cd letsencrypt
sudo service apache2 stop
# request the certificate (change the domains)
sudo ./letsencrypt-auto certonly --standalone -d domain.com -d www.domain.com
The certificate files should be stored in /etc/letsencrypt/live/domain.com/
(with your domain). If the above command reports another location, use that.
Now we need to update the Apache configuration. First, change the port in the first line from 80
to 443
:
<VirtualHost *:80> # old one
<VirtualHost *:443> # new one
Place the below (with updated paths) in your Apache config inside the <VirtualHost *:443>
(as marked with a comment above):
SSLEngine on
SSLCertificateFile /etc/letsencrypt/live/domain.com/cert.pem
SSLCertificateKeyFile /etc/letsencrypt/live/domain.com/privkey.pem
SSLCertificateChainFile /etc/letsencrypt/live/domain.com/chain.pem
# Header always set Strict-Transport-Security "max-age=2600000; preload"
And at the bottom add (if you want all requests to be secure):
<VirtualHost *:80>
ServerName domain.com
ServerAlias *.domain.com
Redirect permanent / https://domain.com/
</VirtualHost>
The last line tells browsers to not access your site through http for a long time. Only enable it when you are confident things are working and will keep working! It’s good for security, making it hard for attackers to divert traffic to http, but it’ll make your site inaccessible if https stops working.
Now just restart Apache and see if things work:
sudo service apache2 restart
You can update local.py
with (at least):
SESSION_COOKIE_SECURE = CSRF_COOKIE_SECURE = False
You’ll need to refresh your https certificates every few months. Don’t forget to do that!
Layout¶
svSite should provide a fully functional site (with minimal work), but you may want to personalize the look, which will take time. That’s inevitable. Here is some information on how to do it.
Theme requirements¶
To create your own theme, these are the requirements:
The files should be organized into directories
templates
,static
andinfo
:- The
templates
directory should containbase.html
holding the theme body and optionallyhead.html
holding anything in<head>
(you might want to includedefault_head.html
). Except for that, it can contain any templates you want (these are only used if you explicitly include them). - The
static
directory should contain any static files you use (see below on how to use them). - The
info
directory can contain any of these files:readme.rst
,description.rst
,credits.rst
andlicense.txt
. Other files can be included but nothing special happens with them.
- The
Include static css/js files using:
{% load addtoblock from sekizai_tags %} {% addtoblock "css" %} <link rel="stylesheet" href="{% static THEME_PREFIX|add:'/css/style.css' %}"> {% endaddtoblock "css" %}
and other static files:
{% load static from staticfiles %} <img src="{% static THEME_PREFIX|add:'/logo.png' %}" />
You can also hard-code
{% static 'theme_name/logo.png' %}
. This behaves differently in case another theme extends this one.For the
base.html
template:It should not extend anything (it is itself included).
It should define precisely these placeholders:
{% placeholder "header" %} {% placeholder "top-row" %} {% placeholder "content" %} {% placeholder "sidebar" %} {% placeholder "bottom-row" %}
It should
{% include include_page %}
if it’s set, e.g. a structure like this:{% if page_include %} {% include page_include %} {% else %} {% placeholder "content" %} {% endif %}
You do not need to define
{% block %}
. You won’t be able to extend them since Django doesn’t let you extend blocks from included templates.
Extending¶
Making changes¶
svSite should let you get a site running with a (somewhat) limited amount of work. But perhaps there are still some details you want to change. And you can! Given some knowledge of Django (features) and/or html/css/js (layout), you can create your copy of the code and occasionally get updates from the main project.
And perhaps your changes turn out great, and you want to share them. That is greatly appreciated, and this page tells you how to do it!
Contribute¶
You can contribute by adding to the project! This includes:
Making changes¶
You will need to make sure you can push code to Github by setting up ssh keys. Then fork the svsite repository and follow these steps.
You will need python3
, pip
, a database (sqlite3
is default and easy, but slow), virtualenv
, git
, elasticsearch
and some SSL packages. Just type:
sudo apt-get install python3 sqlite3 python-virtualenv git build-essential libssl-dev libffi-dev python-dev elasticsearch
Get your copy of the svsite code:
git clone git@github.com:YOUR_SVSITE_FORK.git
Go to the directory, start a virtualenv and install:
virtualenv -p python3 env
source env/bin/activate
pip install --requirement dev/pip_freeze.txt
pip install --no-deps --editable .
To create the database and superuser:
python3 source/manage.py migrate
python3 source/manage.py createsuperuser
You might want to run the tests:
py.test tests
Then you can start the server on localhost:
python3 source/manage.py runserver_plus --settings=base.settings_development
You can now open the site in a browser. It is running on localhost over https on port 8443.
If www.
is prepended, you can use a domain that works with that prefix (not localhost:8443
). For example,
This refers to your localhost (127.0.0.1). The first time you will probably need to add a security exception, as this is a debug SSL certificate.
Now you are ready to make your chances!
After you are done and have tested your changes (and converted space-indents to tabs), you can suggest it for inclusion into svsite by means of a pull request
External services¶
There is a minimal api for building external services, which is described in Integration API. Such additions are welcome, you’re encouraged to notify us when you complete one!
Automated tests¶
Automatic testing is currently very limited for the project. We use py.test
ones, which can be stored in /test/
or source/$app/test
. It’s greatly appreciated if you add more, for your own additions or for existing code. It’ll help ensure the quality of the codebase!
A general note¶
Good luck! Why we never forget our fellow coders
Integration API¶
There is a https-json
api for integrating external tools. It provides read-only access to information about users (member
) and groups (team
).
Server setup¶
To allow client services access to some member and team info through the http api, you only need to change a few settings.
These settings should go in local.py
, since they are installation dependent and should not be accessible to outsiders.
Add setting
INTEGRATION_KEYS
, which should be a list of keys:INTEGRATION_KEYS = [ 'abc123', # php widget 'password!', # android app ]
It is advisable to add a new key for each service, so that you can revoke them individually, if the need arises. Generate keys at random.org .
Add setting
INTEGRATION_ALLOW_EMAIL
which can beTrue
orFalse
(the default). Services can only request a list of email addresses if this isTrue
(optionemail=yes
). Otherwise, services can only get a user’s email when that user logs in to the service.Restart the server.
Also note that, though https
is always important, it is even more important with this api, since both api keys and user credentials and info will be sent over plain http
if you don’t have a secure connection.
External tool setup¶
To get the information, send a POST
request to one of the API urls. The urls for your server can be found at an info page for the server, which is usually /$intapi/
. This is the info send to each of the urls:
Info | Default url | Input | Output | Note |
---|---|---|---|---|
(info page) | /$intapi/members/ |
(nothing) | url & config info | can use GET |
Member list | /$intapi/members/ |
key , optionally email=yes |
list of usernames | if email, dict username->email |
Member | /$intapi/member/ |
key , username , password |
user info map | can authenticate if successful |
Team list | /$intapi/teams/ |
key |
team name list | no hidden teams |
Team | /$intapi/team/ |
key , teamname |
team info map | works for hidden teams |
The option email=yes
only works if the server has INTEGRATION_ALLOW_EMAIL
set to True
.
If all goes well, the result will be a string containing a JSON
list or map. Otherwise you will get an error message and a non-200 status code.
It is recommended that you associate the relevant user data with that user’s session in a safe way (rather than store it in a database), as you will get a fresh copy each time the users logs in.
As an example (Bash terminal, but others like PHP should be similar):
$ curl --show-error --request POST 'https://domain.com/$intapi/members/' --data-urlencode "key=abc123"
[
"mark",
"henk"
]
$ curl --show-error --request POST 'https://domain.com/$intapi/members/' --data-urlencode "key=abc123" --data-urlencode "email=yes"
{
"mark": "mark@spam.la",
"henk": ""
}
$ curl --show-error --request POST 'https://domain.com/$intapi/member/' --data-urlencode "key=abc123" --data-urlencode "username=mark" --data-urlencode "password=drowssap"
{
"username": "mark",
"first_name": "Mark",
"last_name": "V",
"email": "mark@spam.la",
"birthday": null,
"teams": {
"Tokkies": "Mastersjief"
}
}
$ curl --show-error --request POST 'https://domain.com/$intapi/teams/' --data-urlencode "key=abc123"
[
"Tokkies"
]
$ curl --show-error --request POST 'https://domain.com/$intapi/team/' --data-urlencode "key=abc123" --data-urlencode "teamname=Tokkies"
{
"hidden": false,
"teamname": "Tokkies",
"description": "You know, from TV?",
"leaders": [
"mark"
],
"members": {
"mark": "Mastersjief"
}
}
Good luck!
Models¶
The models and their relations can be seen in this graph:
With graphviz and django-extensions you can generate this image yourself:
python source/manage.py graph_models --all --settings=base.settings_development | grep -v '^ //' | grep -v '^[[:space:]]*$$' > images/models.dot
Design notes (hacks)¶
Some parts are less than elegant. Although, at the time of writing, it seems there may not be a better way, it warrants a warning anyway.
Migrating¶
Clean migrations don’t quite work for some cms addons. Find the migration info in the installation documentation.
Themes¶
Djangocms seems not designed to handle dynamic templates, so a fixed template is used that dynamically includes the theme template based on a context variable.
Since djangocms uses sekizai, which must have it’s render_block be in the top template, it is necessary to have the <head> and <body> in this top template, and to include only the rest of the content of these tags.
Furthermore, the CMS does some kind of pre-render without context to find the placeholders to be filled. This means placeholders cannot depend on the theme (=context). Placeholders are defined in default_body.html
and themes should match those.
Special pages¶
This relates to those pages (e.g. search results) that should not be plugins in the CMS, but should be integrated into it anyway (to be in the menu, be moved and allow placeholders).
What I would have preferred to do would be to have such pages (as apphooks) extend the main template and overwrite {% block content %}
.
However, because of themes, {% block content %}
is necessarily defined in an {% include %}
file.
Django cannot extend blocks defined in included files (regrettably) since they are each rendered separately (not so much ‘included’), making the block useless.
The ‘solution’ used is to force templates to include a dynamic template instead of the content placeholder for such pages.
{% if page_include %}
{% include page_include %}
{% else %}
{% placeholder "content" %}
{% endif %}
There is a special version of render, namely base.render_cms_special, that you can use like this:
def my_view(request):
value = 'do some query or something'
return render_cms_special(request, 'my_template.html', dict(
key=value,
))
It is important to note that my_template.html
in this example should render just the content part, not the full page. Don’t {% extend %}
the base template (or anything, for that matter); this is done automatically.
In order for placeholderes to show up and for things to be integrated into the cms, you will need to add this view/app as an app-hook (this is the normal way; the only difference is that you should use render_cms_special
).
Users & groups¶
#todo - built-in Django groups - CMS users
More things¶
Frequently...¶
...asked questions¶
How can I contribute?¶
Okay, this one hasn’t been asked “frequently” in the strict meaning of the word, but anyway. Glad you’re interested! Your help is welcome! Please check the contribute section.
...encountered problems¶
After updating, you might get:
KeyError at /en/stuff/ 'SomethingPlugin'
This means a plugin was removed but is still in the database. Just run:
python source/manage.py cms delete_orphaned_plugins --noinput
if you were using the plugin that was removed, then those use cases will be gone. The alternative is reverting the update.
ElasticSearch / Haystack can’t connect¶
You can test that elasticsearch is running using a http request on port 9200, like so:
curl -X GET http://127.0.0.1:9200
If it isn’t, there could be a number of reasons.
In my case, I had to set START_DAEMON=true
in /etc/default/elasticsearch (source)
You might also have the wrong version of the Python binding, see here