Turq user manual

Warning

Turq is no longer maintained nor used by its author. Try mitmproxy with scripting instead. See also https://gist.github.com/vfaronov/3f7848932ed96a264c382902262ce7b3

Turq is a small HTTP server that can be scripted in a Python-based language. Use it to set up mock HTTP resources that respond with the status, headers, and body of your choosing. Turq is designed for quick interactive testing, but can be used in automated scenarios as well.

User guide

Quick start

To run Turq, you need Python 3.4 or higher. Once you have that, install the turq package with pip:

$ pip3 install turq

Start Turq:

$ turq

You should see something like this:

18:22:19  turq  new rules installed
18:22:19  turq  mock on port 13085 - try http://pergamon:13085/
18:22:19  turq  editor on port 13086 - try http://pergamon:13086/
18:22:19  turq  editor password: QGOf9Y9Eqjvz4XhY4JA3U7hG (any username)

As you can see, Turq starts two HTTP servers. One is the mock server for the mocks you define. The other is the optional rules editor that makes writing mocks easier.

First you probably want to open the editor. By default, Turq listens on all network interfaces, so you can open the editor at http://localhost:13086/ in your Web browser. Turq also tries to guess and print a URL that doesn’t include localhost, which is useful when you run Turq on some remote machine via SSH.

Turq will ask you for the password that it generated and printed for you. You can leave the username field blank, it is ignored.

Warning

Anybody with access to the Turq editor can execute arbitrary code in the Turq process. The default password protection should keep you safe in most cases, but doesn’t help against an active man-in-the-middle. If that’s a problem, limit Turq to loopback with --bind localhost, or run without the editor.

In the editor, you define your mock by writing rules in the big code area, using the examples on the right as your guide. The default rules are just error(404), which means that the mock server will respond with 404 (Not Found) to every request. Let’s check that with curl:

$ curl -i http://pergamon:13085/some/page.html
HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8
date: Tue, 04 Apr 2017 15:33:55 GMT
transfer-encoding: chunked

Error! Nothing matches the given URI

Keep an eye on the system console where you launched turq — all requests and responses are logged there:

19:01:30  turq.connection.1  new connection from 127.0.0.1
19:01:30  turq.request.1  > GET /some/page.html HTTP/1.1
19:01:30  turq.request.1  < HTTP/1.1 404 Not Found

When you are done, stop Turq by pressing Ctrl+C in the console.

That’s it, basically. Check turq --help for command-line options, or read on for more hints on how to use Turq.

Programmatic use

Turq was designed for interactive use; it trades precision for convenience and simplicity. However, you can use it non-interactively if you like:

$ turq --no-editor --rules /path/to/rules.py

Give it a second to spin up, or just loop until you can connect() to it. Shut it down with SIGTERM like any other process:

$ pkill turq

It goes without saying that Turq can’t be used anywhere near production.

Using mitmproxy with Turq

Put mitmproxy in front of Turq to:

  • enable TLS (https) access to the mock server;
  • inspect all requests and responses in detail;
  • validate them with HTTPolice; and more.

Assuming Turq runs on the default port, use a command like this:

$ mitmproxy -p 13185 --mode reverse:http://localhost:13085

Then tell your client to connect to port 13185 (http or https) instead of 13085.

Known issues

Password protection in the rules editor does not work well in some browsers. For example, you may randomly get “Connection error” in Internet Explorer. To avoid this, you can disable password protection with -P "", but be sure to have some other protection instead.

The mock server doesn’t send any cache-related headers by default. As a result, some browsers may cache your mocks, leading to strange results. You can disable caching in your rules:

add_header('Cache-Control', 'no-store')

Turq has limited options to control the addresses it listens on. You can forward its ports manually with socat or mitmproxy.

Examples

The rules language of Turq is documented only by example.

Basics

# This is a comment. Normal Python syntax.
if path == '/hello':
    header('Content-Type', 'text/plain')
    body('Hello world!\r\n')
else:
    error(404)

“RESTful” routing

if route('/v1/products/:product_id'):
    if GET or HEAD:
        json({'id': int(product_id),
              'inStock': True})
    elif PUT:
        # Pretend that we saved it
        json(request.json)
    elif DELETE:
        status(204)   # No Content

HTML pages

To get a simple page:

html()

If you want to change the contents of the page, the full Dominate library is at your service (dominate.tags imported as H):

with html():
    H.h1('Welcome to our site')
    H.p('Have a look at our ',
        H.a('products', href='/products'))

To change the <head>:

with html() as document:
    with document.head:
        H.style('h1 {color: red}')
    H.h1('Welcome to our site')

Request details

if request.json:     # parsed JSON body
    name = request.json['name']
elif request.form:   # URL-encoded or multipart
    name = request.form['name']
elif query:     # query string parameters
    name = query['name']
else:
    raw_name = request.body     # raw bytes
    name = raw_name.decode('utf-8')

# Header names are case-insensitive
if 'json' in request.headers['Accept']:
    json({'hello': name})
else:
    text('Hello %s!\r\n' % name)

Response headers

header() replaces the given header, so this will send only max-age:

header('Cache-Control', 'public')
header('Cache-Control', 'max-age=3600')

To add a header instead:

add_header('Set-Cookie', 'sessionid=123456')
add_header('Set-Cookie', '__adtrack=abcdef')

Custom status code and reason

status(567, 'Server Fell Over')
text('Server crashed, sorry!\r\n')

Redirection

if path == '/':
    redirect('/index.html')
elif path == '/index.html':
    html()

redirect() sends 302 (Found) by default, but you can override:

redirect('/index.html', 307)

Authentication

To demand basic authentication:

basic_auth()
with html():
    H.h1('Super-secret page!')

This sends 401 (Unauthorized) unless the request had Authorization with the Basic scheme (credentials are ignored).

Similarly for digest:

digest_auth()

And for bearer:

bearer_auth()

Body from file

text(open('/etc/services'))

Inspecting requests

To see what the client sends, including headers (but not the raw body), put debug() somewhere early in your rules:

debug()

and watch the console output. Alternatively, for even more diagnostics, run Turq with the --verbose option.

Or use mitmproxy.

Forwarding requests

Turq can act as a gateway or “reverse proxy”:

forward('httpbin.org', 80,  # host, port
        target)             # path + query string
# At this point, response from httpbin.org:80
# has been copied to Turq, and can be tweaked:
delete_header('Server')
add_header('Cache-Control', 'max-age=86400')

Turq uses TLS when connecting to port 443, but ignores certificates. You can override TLS like this:

forward('develop1.example', 8765,
        '/v1/articles', tls=True)

Cross-origin resource sharing

cors() adds the right Access-Control-* headers, and handles preflight requests automatically:

cors()
json({'some': 'data'})

For legacy systems, JSONP is also supported, reacting automatically to a callback query parameter:

json({'some': 'data'}, jsonp=True)

Compression

Call gzip() after setting the body:

with html():
    # 100 paragraphs of text
    for i in range(100):
        H.p(lorem_ipsum())
gzip()

Random responses

if maybe(0.1):    # 10% probability
    error(503)
else:
    html()

Response framing

By default, if the client supports it, Turq uses Transfer-Encoding: chunked and keeps the connection alive.

To use Content-Length instead of Transfer-Encoding, call content_length() after you’ve set the body:

text('Hello world!\r\n')
content_length()

To close the connection after sending the response:

add_header('Connection', 'close')
text('Hello world!\r\n')

Streaming responses

header('Content-Type', 'text/event-stream')
sleep(1)        # 1 second delay
chunk('data: my event 1\r\n\r\n')
sleep(1)
chunk('data: my event 2\r\n\r\n')
sleep(1)
chunk('data: my event 3\r\n\r\n')

Once you call chunk(), the response begins streaming. Any headers you set after that will be sent in the trailer part:

header('Content-Type', 'text/plain')
header('Trailer', 'Content-MD5')
chunk('Hello, ')
chunk('world!\n')
header('Content-MD5', '746308829575e17c3331bbcb00c0898b')

Handling Expect: 100-continue

with interim():
    status(100)

text('Resource updated OK')

In the above example, 100 (Continue) is sent immediately after the interim() block, but the final 200 (OK) response is sent only after reading the full request body.

If instead you want to send a response before reading the request body:

error(403)      # Forbidden
flush()

Custom methods

if method != 'FROBNICATE':
    error(405)   # Method Not Allowed
    header('Allow', 'FROBNICATE')

Switching protocols

if request.headers['Upgrade'] == 'QXTP':
    with interim():
        status(101)   # Switching Protocols
        header('Upgrade', 'QXTP')
        header('Connection', 'upgrade')
    send_raw('This is no longer HTTP!\r\n')
    send_raw('This is QXTP now!\r\n')

Anything else

In the end, Turq rules are just Python code that is not sandboxed, so you can import and use anything you like. For example, to send random binary data:

import os
header('Content-Type', 'application/octet-stream')
body(os.urandom(128))

History of changes

0.3.1 - 2017-04-04

Packaging fixes.

0.3.0 - 2017-04-04

  • Complete rewrite. Only the most notable changes are listed below.
  • Requires Python 3.4 or higher.
  • The rules language is completely different, simpler and more powerful.
  • Notable new features of the rules language:
    • forwarding to other servers (“reverse proxy”);
    • easy “RESTful” routing with path segments;
    • easy construction of arbitrary HTML pages (using Dominate);
    • CORS support now handles preflight requests automatically;
    • control over finer aspects of the protocol: streaming, 1xx responses, Content-Length, Transfer-Encoding, keep-alive.
  • On the other hand, some features have been removed for now:
    • alternating responses (first(), next(), then());
    • shortcuts for JavaScript and XML responses (js(), xml()).
  • You can now choose which network interface Turq listens on, including IPv6.
  • The Turq editor (formerly known as “console”) now has automatic indentation and syntax highlighting.
  • The Turq editor is now optional, listens on a separate port, and is protected with a password by default.
  • Turq can now print more information to the (system) console, including request and response headers.
  • Initial rules may now be read from a file at startup. This provides a simple way to use Turq programmatically.
  • Turq can now handle multiple concurrent requests.

0.2.0 - 2012-12-09

  • Stochastic responses (maybe(), otherwise()).
  • Various features of the response can now be parametrized with lambdas.
  • body_file() now expands tilde to the user’s home directory.

0.1.0 - 2012-11-17

Initial release.