habitat documentation

Contents:

Introduction

habitat is a system for uploading, processing, storing and displaying telemetry and related information transmitted from high altitude balloons.

Typically this telemetry takes the form of a GPS position and potentially other data, accompanied by information on who received the data, and is displayed by means of a map or chart (or both) showing the path a balloon took and the historic trend of sensor readings.

Internally, configuration and data is stored in a CouchDB database. The back end is written in Python and is responsible for parsing incoming data and storing it in the database, while the frontend is written independently in JavaScript and HTML and communicates with CouchDB directly to obtain data and display it.

This documentation covers setting up a habitat system, describes the format used to store data in CouchDB, and provides reference documentation for the habitat source code.

Useful habitat links:

Installing

habitat is currently still in early development and is not ready to be installed.

Check back here later for installation help when the first release is out.

Configuration

Command Line Configuration

habitat daemons takes zero or one command line arguments: an optional filename specifying the configuration to use, or ”./habitat.yml” by default:

./bin/parser
# or
./bin/parser /path/to/config.yml

Configuration File

The configuration file is written in YAML as several key: value pairs. Various habitat components may require certain pieces of configuration; where possible these are all documented here.

habitat-wide Configuration

couch_uri: "http://localhost:5984"
couch_db: habitat
log_stderr_level: DEBUG
log_file_level: INFO

couch_uri and couch_db specify how to connect to the CouchDB database. The URI may contain authentication details if required.

log_stderr_level and log_file_level set the log levels for a log file and the stderr output and may be “NONE”, “ERROR”, “WARN”, “INFO” or “DEBUG”.

parser configuration

parser:
    certs_dir: "/path/to/certs"
    modules:
        - name: "UKHAS"
          class: "habitat.parser_modules.ukhas_parser.UKHASParser"
parserdaemon:
    log_file: "/path/to/parser/log"

Inside the parser and parserdaemon objects:

  • certs_dir specifies where the habitat certificates (used for code signing) are kept
  • log_file specifies where the parser daemon should write its log file to
  • modules gives a list of all the parser modules that should be loaded, with a name (that must match names used in flight documents) and the Python path to load.

This configuration is used by habitat.parser and habitat.parser_daemon.

loadable_manager configuration

loadables:
    - name: "sensors.base"
      class: "habitat.sensors.base"
    - name: "sensors.stdtelem"
      class: "habitat.sensors.stdtelem"
    - name: "filters.common"
      class: "habitat.filters"

Inside the loadables object is a list of modules to load and the short name they should be loaded against. This is used by habitat.loadable_manager.

Database information

Schema

habitat stores information in a CouchDB database. At present five types of document are stored, identified by a type key:

  • Flight documents detailing a balloon flight (type: "flight")
  • Payload Configuration documents containing settings for one payload, such as radio transmission data and format (type: "payload_configuration").
  • Payload Telemetry documents containing parsed information from a telemetry message transmitted by a payload and associated with a Flight (type: "payload_telemetry")
  • Listener telemetry documents containing position data on someone listening to a payload (type: "listener_telemetry")
  • Listener information documents containing metadata on a listener such as name and radio (type: "listener_information")

The schema are described using JSON Schema and the latest version may be browsed online via jsonschema explorer.

Database documents are typically managed through the various web interfaces and are uploaded and retrieved using the CouchDB API.

Views, Filters & Validation Functions

Documents in the habitat CouchDB are indexed and accessed using CouchDB views, which are pre-calculated sets of results that are updated automatically and may be paged and searched through.

A selection of generic views are provided, but it’s entirely likely that a custom view would be required for a given application.

Three types of function may be defined in a CouchDB design document. Views consist of a map and optionally a reduce and are typically used to query stored data. Filters selectively include certain documents in a stream from the database, for example to the parser. Validation functions check all new incoming documents to ensure they meet whatever requirements are imposed, making sure that only valid documents are stored in the database.

For more comprehensive information, please refer to the CouchDB documentation.

Included Views

For documentation on the views currently included with habitat, please refer to the source documentation for each: habitat.views.

Using Views: Example

Simple Example

In this simple example we just fetch the list of payload_configuration documents and print out payload names. The full configuration document is available as payload[“doc”].

Python
import couchdbkit
db = couchdbkit.Server("http://habitat.habhub.org")["habitat"]
payloads = db.view("payload_configuration/name_time_created", include_docs=True)
for payload in payloads:
    print "Payload: {0}".format(payload["key"][0])
Javascript
<script src="jquery.couch.js"></script>
db = $.couch.db("habitat")
db.view("payload_configuration/name_time_created", {include_docs: true, success: function(data) {
    for(payload in data.rows) {
        console.log("Payload: " + data.rows[payload].key[0]);
    }
}});

View Collation

A more complicated example, which demonstrates view collation. The flight/launch_time_including_payloads view returns both flight documents and the associated payload_configuration documents, indicated by the second item in the key (0 for flight, 1 for payload_configuration). This code snippet fetches all the flights and prints their name and launch time and the name of each payload in them.

Python
import couchdbkit
db = couchdbkit.Server("http://habitat.habhub.org")["habitat"]
flights = db.view("flight/launch_time_including_payloads", include_docs=True)
for flight in flights:
    if flight["key"][1] == 0:
        print "Flight {0} launches at {1}!".format(
            flight["doc"]["name"], flight["doc"]["launch"]["time"])
        print "  Payloads:"
    else:
        print "    {0}".format(flight["doc"]["name"])

Filters

Filter Levels

There are three points in the message flow at which a filter can act: before pre-parsing happens, after the pre-parse but before the main parse, and after the main parse.

Pre-parsing extracts a callsign from a string and uses it to look up the rest of the payload’s configuration, so pre-parse filters are specified per module and will act on everything that module receives. These are called pre-filters and are specified in the parser configuration document.

Intermediate filters act after a callsign has been found but before the message is parsed for data, so they can correct particular format errors a certain payload might be transmitting. Post-parse filters act after the data parsing has happened, so can tweak the output data. Both intermediate and post filters are specified in the payload section of a flight document.

Filter Syntax

Two types of filters are supported: normal and hotfix. Normal filters give a callable object, which must be in the Python path, and optionally may give a configuration object which will be passed as the second argument to the callable. Hotfix filters just supply some Python code, which is used as the body of a function given the incoming data as its sole argument. In either case, the filter must return the newly processed data.

Example of a normal filter:

{
    "type": "normal",
    "callable": "habitat.filters.upper_case"
}
# habitat/filters.py
def upper_case(data):
    return data.upper()

A normal filter with a configuration object:

{
    "type": "normal",
    "callable": "habitat.filters.daylight_savings",
    "config": {"time_field": 7}
}
# habitat/filters.py
def daylight_savings(data, config):
    time = data[config['time_field']]
    hour = int(time[0:2])
    data[config['time_field']] = str(hour + 1) + time[2:]
    return data

A hotfix filter:

{
    "type": "hotfix",
    "code": "parts = data.split(',')\nreturn '.'.join(parts)\n"
}

Which would be assembled into:

def f(data):
    parts = data.split(',')
    return '.'.join(parts)

A more complete hotfix example, to fix non-zero-padded time values:

from habitat.utils.filtertools import UKHASChecksumFixer

parts = data.split(",")
timestr = parts[2]
timeparts = timestr.split(":")
timestr = "{0:02d}:{1:02d}:{2:02d}".format(*[int(part) for part in timeparts])
parts[2] = timestr
newdata = ",".join(parts)

with UKHASChecksumFixer('xor', {"data": data}) as fixer:
    fixer["data"] = newdata

    return fixer["data"]

Filter Utils

Please refer to habitat.utils.filtertools for information on available filter tools such as UKHASChecksumFixer used above.

Certificates

The certificates folder contains x509 certificate files which are used to verify the authenticity of hotfix code.

As hotfix code is run live from the CouchDB database, habitat uses certificates to check that the code can be trusted. The habitat developers maintain a certificate authority, whose certificate is included as ca/habitat_ca_cert.pem, which is used to sign code-signing certificates.

Hotfix code then has its SHA-256 digest signed by the developer’s private key, and this is verified by habitat before the code is executed.

You can add new CA certificates to the ca folder, and new code-signing certificates to the certs folder, as you please. Hotfix code references a certificate filename found in the certs folder.

Generating a Private Key

$ openssl genrsa -des3 4096 > private.pem
$ openssl req -new -key private.pem -out req.csr

Now send req.csr to us and we can sign it with the habitat CA and give you the signed certificate.

Generating a Certificate Authority

This is a fair bit more complex. Consider using tools such as tinyca or gnomint.

Signing Code

$ vi my_hotfix_code.py
$ habitat/bin/sign_hotfix my_hotfix_code.py ~/my_rsa_key.pem

The printed result is a JSON object which can be placed into the filters list on a flight document.

UKHAS Parser Configuration

Introduction

The UKHAS protocol is the most widely used at time of writing, and is implemented by the UKHAS parser module. This document provides information on how what configuration settings the UKHAS parser module expects.

Parser module configuration is given in the “sentence” dictionary of the payload dictionary in a flight document.

Generating Payload Configuration Documents

The easiest and recommended way to generate configuration documents is using the web tool genpayload.

Standard UKHAS Sentences

A typical minimum UKHAS protocol sentence may be:

$$habitat,123,13:16:24,51.123,0.123,11000*ABCD

This sentence starts with a double dollar sign ($$) followed by the payload name (here habitat), several comma-delimited fields and is then terminated by an asterisk and four-digit CRC16 CCITT checksum (*ABCD).

In this typical case, the fields are a message ID, the time, a GPS latitude and longitude in decimal degrees, and the current altitude.

However, both the checksum algorithm in use and the number, type and order of fields may be configured per-payload.

Parser Module Configuration

The parser module expects to be given the callsign, the checksum algorithm, the protocol name (“UKHAS”) and a list of fields, each of which should at least specify the field name and data type.

Checksum Algorithms

Three algorithms are available:

  • CRC16 CCITT (crc16-ccitt):

    The recommended algorithm, uses two bytes transmitted as four ASCII digits in hexadecimal. Can often be calculated using libraries for your payload hardware platform. In particular, note that we use a polynomial of 0x1021 and a start value of 0xFFFF, without reversing the input. If implemented correctly, the string habitat should checksum to 0x3EFB.

  • XOR (xor):

    The simplest algorithm, calculating the one-byte XOR over all the message data and transmitting as two ASCII digits in hexadecimal. habitat checksums to 0x63.

  • Fletcher-16 (fletcher-16):

    Not recommended but supported. Uses a modulus of 255 by default, if modulus 256 is required use fletcher-16-256.

In all cases, the checksum is of everything after the $$ and before the *.

Field Names

Field names may be any string that does not start with an underscore. It is recommended that they follow the basic pattern of prefix[_suffix[_suffix[...]]] to aid presentation: for example, temperature_internal and temperature_external could then be grouped together automatically by a user interface.

In addition, several common field names have been standardised on, and their use is strongly encouraged:

Field Name To Use Notes
Sentence ID (aka count, message count, sequence number) sentence_id An increasing integer
Time time Something like HH:MM:SS or HHMMSS or HHMM or HH:MM.
Latitude latitude Will be converted to decimal degrees based on format field.
Longitude longitude Will be converted to decimal degrees based on format field.
Altitude altitude In, or converted to, metres.
Temperature temperature Should specify a suffix, such as _internal or _external. In or converted to degrees Celsius.
Satellites In View satellites  
Battery Voltage battery Suffixes allowable, e.g., _backup, _cutdown, but without the suffix it is treated as the main battery voltage. In volts.
Pressure pressure Suffixes allowable, e.g., _balloon. Should be in or converted to Pa.
Speed speed For speed over the ground. Should be converted to m/s (SI units).
Ascent Rate ascentrate For vertical speed. Should be m/s.

Standard user interfaces will use title case to render these names, so flight_mode would become Flight Mode and so on. Some exceptions may be made in the case of the common field names specified above.

Field Types

Supported types are:

  • string: a plain text string which is not interpreted in any way.
  • float: a value that should be interpreted as a floating point number. Transmitted as a string, e.g., “123.45”, rather than in binary.
  • int: a value that should be interpreted as an integer.
  • time: a field containing the time as either HH:MM:SS or just HH:MM. Will be interpreted into a time representation.
  • time: a field containing the time of day, in one of the following formats: HH:MM:SS, HHMMSS, HH:MM, HHMM.
  • coordinate: a coordinate, see below

Coordinate Fields

Coordinate fields are used to contain, for instance, payload latitude and longitude. They have an additional configuration parameter, format, which is used to define how the coordinate should be parsed. Options are:

  • dd.dddd: decimal degrees, with any number of digits after the decimal point. Leading zeros are allowed.
  • ddmm.mm: degrees and decimal minutes, with the two digits just before the decimal point representing the number of minutes and all digits before those two representing the number of degrees.

In both cases, the number can be prefixed by a space or + or - sign.

Please note that the the options reflect the style of coordinate (degrees only vs degrees and minutes), not the number of digits in either case.

Units

Received data may use any convenient unit, however it is strongly recommended that filters (see below) be used to convert the incoming data into SI units. These then allow for standardisation and ease of display on user interface layers.

Filters

See Filters

habitat code documentation

habitat The top level habitat package.

Indices and tables