Klein, a Web Micro-Framework

https://travis-ci.org/twisted/klein.png?branch=master

Klein is a micro-framework for developing production-ready web services with Python. It’s built on widely used and well tested components like Werkzeug and Twisted, and has near-complete test coverage.

Klein is developed by a team of contributors on GitHub. We’re also commonly in #twisted.web on Freenode.

Introduction to Klein

This is an introduction to Klein, going through from creating a simple web site to something supporting AJAX and more.

Introduction – Getting Started

Klein is a micro-framework for developing production-ready web services with Python, built off Werkzeug and Twisted. The purpose of this introduction is to show you how to install, use, and deploy Klein-based web applications.

This Introduction

This introduction is meant as a general introduction to Klein concepts.

Everything should be as self-contained, but not everything may be runnable (for example, code that shows only a specific function).

Installing

Klein is available on PyPI. Run this to install it:

pip install klein

Note

Since Twisted is a Klein dependency, you need to have the requirements to install that as well. You will need the Python development headers and a working compiler - installing python-dev and build-essential on Debian, Mint, or Ubuntu should be all you need.

Hello World

The following example implements a web server that will respond with “Hello, world!” when accessing the root directory.

from klein import run, route

@route('/')
def home(request):
    return 'Hello, world!'

run("localhost", 8080)

This imports run and route from the Klein package, and uses them directly. It then starts a Twisted Web server on port 8080, listening on the loopback address.

This works fine for basic applications. However, by creating a Klein instance, then calling the run and route methods on it, you are able to make your routing not global.

from klein import Klein
app = Klein()

@app.route('/')
def home(request):
    return 'Hello, world!'

app.run("localhost", 8080)

By not using the global Klein instance, you can have different Klein routers, each having different routes, if your application requires that in the future.

Adding Routes

Add more decorated functions to add more routes to your Klein applications.

from klein import Klein
app = Klein()

@app.route('/')
def pg_root(request):
    return 'I am the root page!'

@app.route('/about')
def pg_about(request):
    return 'I am a Klein application!'

app.run("localhost", 8080)

Variable Routes

You can also make variable routes. This gives your functions extra arguments which match up with the parts of the routes that you have specified. By using this, you can implement pages that change depending on this – for example, by displaying users on a site, or documents in a repository.

from klein import Klein
app = Klein()

@app.route('/user/<username>')
def pg_user(request, username):
    return 'Hi %s!' % (username,)

app.run("localhost", 8080)

If you start the server and then visit http://localhost:8080/user/bob, you should get Hi bob! in return.

You can also define what types it should match. The three available types are string (default), int and float.

from klein import Klein
app = Klein()

@app.route('/<string:arg>')
def pg_string(request, arg):
    return 'String: %s!' % (arg,)

@app.route('/<float:arg>')
def pg_float(request, arg):
    return 'Float: %s!' % (arg,)

@app.route('/<int:arg>')
def pg_int(request, arg):
    return 'Int: %s!' % (arg,)

app.run("localhost", 8080)

If you run this example and visit http://localhost:8080/somestring, it will be routed by pg_string, http://localhost:8080/1.0 will be routed by pg_float and http://localhost:8080/1 will be routed by pg_int.

Route Order Matters

But remember: order matters! This becomes very important when you are using variable paths. You can have a general, variable path, and then have hard coded paths over the top of it, such as in the following example.

from klein import Klein
app = Klein()

@app.route('/user/<username>')
def pg_user(request, username):
    return 'Hi %s!' % (username,)

@app.route('/user/bob')
def pg_user_bob(request):
    return 'Hello there bob!'

app.run("localhost", 8080)

The later applying route for bob will overwrite the variable routing in pg_user. Any other username will be routed to pg_user as normal.

Static Files

To serve static files from a directory, set the branch keyword argument on the route you’re serving them from to True, and return a t.w.static.File with the path you want to serve.

from twisted.web.static import File
from klein import Klein
app = Klein()

@app.route('/', branch=True)
def pg_index(request):
    return File('./')

app.run("localhost", 8080)

If you run this example and then visit http://localhost:8080/, you will get a directory listing.

Deferreds

Since it’s all just Twisted underneath, you can return Deferreds, which then fire with a result.

import treq
from klein import Klein
app = Klein()

@app.route('/', branch=True)
def google(request):
    d = treq.get('https://www.google.com' + request.uri)
    d.addCallback(treq.content)
    return d

app.run("localhost", 8080)

This example here uses treq (think Requests, but using Twisted) to implement a Google proxy.

Return Anything

Klein tries to do the right thing with what you return. You can return a result (which can be regular text, a Resource, or a Renderable) synchronously (via return) or asynchronously (via Deferred). Just remember not to give Klein any unicode, you have to encode it into bytes first.

Onwards

That covers most of the general Klein concepts. The next chapter is about deploying your Klein application using Twisted’s tap functionality.

Introduction – Using twistd to Start Your Application

twistd (pronounced “twist-dee”) is an application runner for Twisted applications. It takes care of starting your app, setting up loggers, daemonising, and providing a nice interface to start it.

Using the twistd web Plugin

Exposing a valid IResource will allow your application to use the pre-existing twistd web plugin.

To enable this functionality, just expose the resource object of your Klein router:

from klein import Klein
app = Klein()

@app.route('/')
def hello(request):
    return "Hello, world!"

resource = app.resource

Then run it (in this example, the file above is saved as twistdPlugin.py:

$ twistd -n web --class=twistdPlugin.resource

The full selection of options you can give to twistd web can be found in its help page. Here are some relevant entries in it:

  -n, --notracebacks      Do not display tracebacks in broken web pages.
                          Displaying tracebacks to users may be security risk!
  -p, --port=             strports description of the port to start the server
                          on.
  -l, --logfile=          Path to web CLF (Combined Log Format) log file.
      --https=            Port to listen on for Secure HTTP.
  -c, --certificate=      SSL certificate to use for HTTPS. [default:
                          server.pem]
  -k, --privkey=          SSL certificate to use for HTTPS. [default:
                          server.pem]
      --class=            Create a Resource subclass with a zero-argument
                          constructor.

Using HTTPS via the twistd web Plugin

The twistd web plugin has inbuilt support for HTTPS, assuming you have TLS support for Twisted.

As an example, we will create some self-signed certs – for the second command, the answers don’t really matter, as this is only a demo:

$ openssl genrsa > privkey.pem
$ openssl req -new -x509 -key privkey.pem -out cert.pem -days 365

We will then run our plugin, specifying a HTTPS port and the relevant certificates:

$ twistd -n web --class=twistdPlugin.resource -c cert.pem -k privkey.pem --https=4433

This will then start a HTTPS server on port 4433. Visiting https://localhost:4433 will give you a certificate error – if you add a temporary exception, you will then be given the “Hello, world!” page. Inspecting your browser’s URL bar should reveal a little lock – meaning that the connection is encrypted!

Of course, in production, you’d be using a cert signed by a certificate authority – but self-signed certs have their uses.

Klein Examples

These are examples that show how to use different parts of Klein, or use things with Klein.

Example – Serving Static Files

Helpfully you can also return a t.w.resource.IResource such as t.w.static.File. If branch=True is passed to route the returned IResource will also be allowed to handle all children path segments. So http://localhost:8080/static/img.gif should return an image and http://localhost:8080/static/ should return a directory listing.

from twisted.web.static import File
from klein import run, route

@route('/static/', branch=True)
def static(request):
    return File("./static")

@route('/')
def home(request):
    return '<img src="/static/img.gif">'

run("localhost", 8080)

Example – Using Twisted.Web Templates

You can also make easy use of Twisted’s templating system by returning anything that implements IRenderable. For example, returning a t.w.template.Element will make it be rendered, with the result sent as the response body:

from twisted.web.template import Element, XMLString, renderer
from klein import run, route

class HelloElement(Element):
    loader = XMLString((
        '<h1 '
        'xmlns:t="http://twistedmatrix.com/ns/twisted.web.template/0.1"'
        '>Hello, <span t:render="name"></span>!</h1>'))

    def __init__(self, name):
        self._name = name

    @renderer
    def name(self, request, tag):
        return self._name


@route('/hello/<string:name>')
def home(request, name='world'):
    return HelloElement(name)

run("localhost", 8080)

Example – Using Deferreds

And of course, this is Twisted. So there is a wealth of APIs that return a Deferred. A Deferred may also be returned from handler functions and their result will be used as the response body.

Here is a simple Google proxy, using treq (think Requests, but using Twisted):

import treq
from klein import Klein
app = Klein()

@app.route('/', branch=True)
def google(request):
    d = treq.get('https://www.google.com' + request.uri)
    d.addCallback(treq.content)
    return d

app.run("localhost", 8080)

Example – Using twistd

Another important integration point with Twisted is the twistd application runner. It provides rich logging support, daemonization, reactor selection, profiler integration, and many more useful features.

To provide access to these features (and others like HTTPS) klein provides the resource function which returns a valid IResource for your application.

Here is our “Hello, World!” application again in a form that can be launched by twistd:

from klein import resource, route

@route('/')
def hello(request):
    return "Hello, world!"

To run the above application we can save it as helloworld.py and use the twistd web plugin:

twistd -n web --class=helloworld.resource

Example – Handling POST

The route decorator supports a methods keyword which is the list of HTTP methods as strings. For example, methods=['POST'] will cause the handler to be invoked when an POST request is received. If a handler can support multiple methods the current method can be distinguished with request.method.

Here is our "Hello, world!" example extended to support setting the name we are saying Hello to via a POST request with a name argument.

This also demonstrates the use of the redirect method of the request to redirect back to '/' after handling the POST.

The most specific handler should be defined first. So the POST handler must be defined before the handler with no methods.

from twisted.internet.defer import succeed
from klein import run, route

name='world'

@route('/', methods=['POST'])
def setname(request):
    global name
    name = request.args.get('name', ['world'])[0]
    request.redirect('/')
    return succeed(None)

@route('/')
def hello(request):
    return "Hello, {0}!".format(name)

run("localhost", 8080)

The following curl command can be used to test this behaviour:

curl -v -L -d name='bob' http://localhost:8080/

Example – Subroutes

The routes decorator lets you set up different routes easily, but it can be cumbersome to write many similar routes. If you need to write similar routes, the subroute function can help. subroute is a context manager within whose scope all invocations of route will have a prefix added.

Here is an example app that has routes for /branch/lair, /branch/crypt, and /branch/swamp all defined in a with app.subroute() block.

from klein import Klein

app = Klein()

with app.subroute("/branch") as app:
    @app.route("/lair")
    def lair(request):
        return b"These stairs lead to the lair of beasts."

    @app.route("/crypt")
    def crypt(request):
        return b"These stairs lead to an ancient crypt."

    @app.route("/swamp")
    def swamp(request):
        return b"A stair to a swampy wasteland."

app.run("localhost", 8080)

Example – Using Non-Global State

For obvious reasons it may be desirable for your application to have some non-global state that is used by your route handlers.

Below we have created a simple ItemStore class that has an instance of Klein as a class variable app. We can now use @app.route to decorate the methods of the class.

import json

from klein import Klein


class ItemStore(object):
    app = Klein()

    def __init__(self):
        self._items = {}

    @app.route('/')
    def items(self, request):
        request.setHeader('Content-Type', 'application/json')
        return json.dumps(self._items)

    @app.route('/<string:name>', methods=['PUT'])
    def save_item(self, request, name):
        request.setHeader('Content-Type', 'application/json')
        body = json.load(request.content)
        self._items[name] = body
        return json.dumps({'success': True})

    @app.route('/<string:name>', methods=['GET'])
    def get_item(self, request, name):
        request.setHeader('Content-Type', 'application/json')
        return json.dumps(self._items.get(name))


if __name__ == '__main__':
    store = ItemStore()
    store.app.run('localhost', 8080)

Example – Handling Errors

It may be desirable to have uniform error-handling code for many routes. We can do this with Klein.handle_errors.

Below we have created a class that will translate NotFound exceptions into a custom 404 response.

from klein import Klein

class NotFound(Exception):
    pass


class ItemStore(object):
    app = Klein()

    @app.handle_errors(NotFound)
    def notfound(self, request, failure):
        request.setResponseCode(404)
        return 'Not found, I say'

    @app.route('/droid/<string:name>')
    def droid(self, request, name):
        if name in ['R2D2', 'C3P0']:
            raise NotFound()
        return 'Droid found'

    @app.route('/bounty/<string:target>')
    def bounty(self, request, target):
        if target == 'Han Solo':
            return '150,000'
        raise NotFound()


if __name__ == '__main__':
    store = ItemStore()
    store.app.run('localhost', 8080)

The following cURL commands (and output) can be used to test this behaviour:

curl -L http://localhost:8080/droid/R2D2
Not found, I say

curl -L http://localhost:8080/droid/Janeway
Droid found

curl -L http://localhost:8080/bounty/things
Not found, I say

Contributing

If you’d like to help out, here’s some material to help you get started!

Contributing to Klein

Klein is an open source project that welcomes contributions of all kinds coming from the community, including:

Getting started

Here is a list of shell commands that will install the dependencies of Klein, run the test suite with Python 2.7 and the current version of Twisted, compile the documentation, and check for coding style issues with pyflakes.

pip install --user tox
tox -e py27-twcurrent
tox -e docs
tox -e pyflakes

Tox makes a virtualenv, installs Klein’s dependencies into the virtualenv, and then runs a set of commands based on the -e (environment) argument. This strategy allows one to make and test changes to Klein without needing to change system-level Python packages.

Next steps

Here are some suggestions to make the contributing process easier for everyone:

Code
  • Use Twisted’s coding standards as a guideline for code changes you make. Some parts of Klein (eg. klein.resource.ensure_utf8_bytes) do not adhere to the Twisted style guide, but changing that would break public APIs, which is worse than breaking the style guidelines. Similarly, if you change existing code, following the Twisted style guide is good, but is less important than not breaking public APIs.
  • Compatibility across versions is important: here are `Twisted’s compatibility guidelines<https://twistedmatrix.com/trac/wiki/CompatibilityPolicy>`_, which Klein shares.
  • If you’re adding a new feature, please add a file with an example and some explanation to the examples directory, then add your example to /docs/index.rst.
  • Please run tox -e pyflakes to check for style issues in changed code. PyFlakes and similar tools expose many small-but-common errors early enough that it’s easy to remedy the problem.
  • Code changes should have tests: untested code is buggy code. Klein uses Twisted Trial and tox for its tests. The command to run the full test suite is tox with no arguments. This will run tests against several versions of Python and Twisted, which can be time-consuming. To run tests against only one or a few versions, pass a -e argument with an environment from the envlist in tox.ini: for example, tox -e py33-tw150 will run tests with Python 3.3 and Twisted 15.0. To run only one or a few specific tests in the suite, add a filename or fully-qualified Python path to the end of the test invocation: for example, tox klein.test.test_app.KleinTestCase.test_foo will run only the test_foo() method of the KleinTestCase class in klein/test/test_app.py. These two test shortcuts can be combined to give you a quick feedback cycle, but make sure to check on the full test suite from time to time to make sure changes haven’t had unexpected side effects.
  • Show us your code changes through pull requests sent to Klein’s GitHub repo. This is the best way to make your code visible to others and to get feedback about it.
  • If your pull request is a work in progress, please put [WIP] in its title.
  • Add yourself to AUTHORS. Your contribution matters.
Documentation

Klein uses Epydoc for docstrings in code and uses Sphinx for standalone documentation.

  • In documents with headers, use this format and order for headers:

    ========
    Header 1
    ========
    
    Header 2
    ========
    
    Header 3
    --------
    
    Header 4
    ~~~~~~~~
    
  • In prose, please use gender-neutral pronouns or structure sentences such that using pronouns is unecessary.

  • It’s best to put each sentence on a different line: this makes diffs much easier to read. Sentences that are part of list items need to be indented to be considered part of the same list item.

Reviewing

All code changes added to Klein must be reviewed by at least one other person who is not an author of the code being added. This helps prevent bugs from slipping through the net, gives another source for improvements, and makes sure that the code productively follows guidelines.

Reviewers should read Glyph’s mailing list post on reviewing – code and docs don’t have to be perfect, only better.

Releasing Klein

Klein is released on a time-dependent basis, similar to Twisted.

Each version is numbered with the major portion being the last two digits of the year, and the minor portion being the zero-indexed release number. That is, the first release of 2016 would be 16.0, and the second would be 16.1.

Doing a Release

  1. Change the version number and commit it.
  2. Clear the directory of any other changes using git clean -f -x -d  .
  3. Tag the release using git tag -s <release> -m "Tag <release> release"
  4. Push up the tag using git push --tags.
  5. Make a pull request for this changes. Continue when it is merged.
  6. Generate the tarball and wheel using python setup.py sdist bdist_wheel.
  7. Upload the tarball and wheel using twine upload dist/klein-*.