Configuring the Server¶

Once Natural Log is installed, you need to create a server configuration file, which uses the JSON format. Here is an example ln_local.json config file for a local test server we will use elsewhere in the documentation:

{
    "host" : "127.0.0.1",
    "port" : 6283,
    "url_base" : "http://localhost:6823/",

    "resampling_intervals" : [1, 60, 3600, 86400],

    "storage" : {
                    "backend" : "memory"
                }
}

The required fields in the server configuration are:

Storage Backends¶

Natural Log supports multiple storage backends as a way to experiment with different ways to store time series data. Each backend and its configuration parameters are described below.

Starting the Server¶

Before starting the Natural Log server, we first need to initialize the storage backend:

$ ln-server -c ln_local.json init
Natural Log 0.1
Initializing "memory" storage backend...
Done.

Then we can start the server:

$ ln-server -c ln_local.json start
Natural Log 0.1
Opening "memory" storage backend...
Listening on 127.0.0.1:6283
Base URL is http://localhost:6283/

Data Series Attributes¶

servers.node01.cpu_temp
servers.node01.disk_temp
servers.node02.cpu_temp
servers.node02.disk_temp

int8
int16
int32
int64
float32
float64

int16[100]
float32[2,4]
int32[10,10,2]

blob:image/png
blob:application/pdf
blob:binary/octet-stream

Creating a Data Series¶

We assume that you have the Natural Log server running as described in Configuring the Server. The examples below use the Requests module for Python to make the REST API calls:

import requests
url_base = 'http://localhost:6283/'

To create a data series, we make a dictionary of the desired attributes and send it to the create URL. Here is an example of creating a temperature data series:

t = {
    'name' : 'node01.cpu_temp',
    'type' : 'float32',
    'reduction' : 'mean',
    'interpolation' : 'linear',
    'description' : 'Temperature of CPU in node01',
    'unit' : 'deg F'
}

r = requests.post(url_base + 'create/', data=t)
assert r.status_code == 200  # Check for success

An integer commit counter:

t = {
    'name' : 'commits',
    'type' : 'int8',
    'reduction' : 'sum',
    'interpolation' : 'middle',
    'description' : 'Number of commits to repository',
}

r = requests.post(url_base + 'create/', data=t)
assert r.status_code == 200  # Check for success

An array data series:

t = {
    'name' : 'channel_crc_errors',
    'type' : 'int32[100]',
    'reduction' : 'sum',
    'interpolation' : 'previous',
    'description' : 'Number of CRC errors for each data channel.',
}

r = requests.post(url_base + 'create/', data=t)
assert r.status_code == 200  # Check for success

And finally, a blob data series:

t = {
    'name' : 'cameras.entrance',
    'type' : 'blob:image/jpeg',
    'reduction' : 'middle',
    'interpolation' : 'none',
    'description' : 'Webcam aimed at lab entrance'
}

r = requests.post(url_base + 'create/', data=t)
assert r.status_code == 200  # Check for success

Server-side Timestamping¶

Recording data is done by POST requests to the data/[source name]/ URL. If no time is given, then the time of the request is used as the time of the data point. This avoids the need to worry about time synchronization on data collection clients.

Here is an example that populate the temperature data source with some random values by pausing between each point:

import time, random
post_url = url_base + 'data/node01.cpu_temp/'
for i in xrange(10):
    v = { 'value' : str(random.uniform(80, 120)) }
    r = requests.post(post_url, data=v)
    assert r.status_code == requests.codes.ok  # Check for errors

Client-side Timestamping¶

A timestamp can also be provided with the data point. The restriction is that each data source is append-only, so data must always be sent to the database in chronological order.

To avoid ambiguity, the interface of Natural Log describes times in ISO 8601 format, and times are always in UTC. Here is an example:

post_url = url_base + 'data/commits/'
for i in xrange(1, 15):
    v = {
        'time'  : '2013-07-%02d 12:00:31.503' % i,
        'value' : '1'
    }
    r = requests.post(post_url, data=v)
    assert r.status_code == requests.codes.ok  # Check for errors

Array Data Sources¶

Sources with array data types are updated in the same way, just with value set to a JSON string representation of a list. Here is an example with the CRC error data source:

import json
post_url = url_base + 'data/channel_crc_errors/'
channel_data = [ 1 for i in xrange(100) ]
v = { 'value' : json.dumps(channel_data) }
r = requests.post(post_url, data=v)
assert r.status_code == requests.codes.ok  # Check for errors

Multi-dimensional arrays can be represented with nested lists of lists.

Blob Data Sources¶

Blob data sources are handled slightly differently in order to transmit binary data. We have to use a multipart form encoding to hold the value field. In Requests, this is very straightforward:

# First fetch Google's Euler doodle image
r = requests.get('http://lh5.ggpht.com/Npa8E2JNHZHzrfQCutzmqqxD3WyQiiLibcaAvR4rR0hEs7LJDY-ahWf5SRBN5Jj7oDhRiZKk7Ca_rCn4rEAEFt_HC3Ho2OImBYDZKPg')
assert r.status_code == requests.codes.ok
image = r.content

# Next, upload
post_url = url_base + 'data/cameras.entrance/'
import datetime
t = { 'time' : datetime.utcnow().isoformat(' ') }
v = { 'value' : image }
r = requests.post(post_url, data=t, files=v)
assert r.status_code = requests.codes.ok

How Resampling Works¶

Let’s suppose we have a time-ordered series of raw data points (floats, in this case) called X:

A query comes in asking for 4 data points from t=20 to t=50 seconds. First, the storage backend decides if it needs to adjust this request for performance reasons. Assuming it makes no changes to the request, it will need to decide the value of the data series at t=20, 30, 40, and 50 seconds, indicated by the red lines:

To estimate the value of the data series at the given times, the query engine groups the raw data points into equally-spaced bins around each of the evaluation times:

In this example, there are some bins with multiple raw points, and some bins with no points.

Natural Log uses the following rules to decide the value of each evaluation point:

1. If the bin contains 1 raw point, that value in the bin is the value of that raw point.
2. If the bin contains multiple raw points, the reduction strategy is used to combine the raw points into a single value.
3. If the bin contains 0 raw points, the interpolation strategy is used to generate a value from adjacent bins.

In the above example, suppose the reduction strategy is mean and the interpolation strategy is linear, the result of the query will be:

Making a Query¶

The examples in this section assume the Natural Log server is running with the configuration described in Configuring the Server. We continue to use the Requests module:

import requests
url_base = 'http://localhost:6283/'

For this example, let’s make two new data series, temperature and humidity:

t = {
    'name' : 'temperature',
    'type' : 'float32',
    'reduction' : 'mean',
    'interpolation' : 'linear',
    'description' : 'Outside Temperature',
    'unit' : 'deg C'
}

r = requests.post(url_base + 'create/', data=t)
assert r.status_code == 200  # Check for success

t = {
    'name' : 'humidity',
    'type' : 'float32',
    'reduction' : 'mean',
    'interpolation' : 'linear',
    'description' : 'Relative humidity',
    'unit' : '%'
}

r = requests.post(url_base + 'create/', data=t)
assert r.status_code == 200  # Check for success

And let’s fill them with some hourly data:

post_url = url_base + 'data/temperature/'
for i in xrange(1, 15):
    v = {
        'time'  : '2013-07-24 %02d:00:00' % i,
        'value' : 25.0 + 0.5 * i
    }
    r = requests.post(post_url, data=v)
    assert r.status_code == requests.codes.ok  # Check for errors

post_url = url_base + 'data/humidity/'
for i in xrange(1, 15):
    v = {
        'time'  : '2013-07-24 %02d:00:00' % i,
        'value' : 55.0 - 0.2 * i
    }
    r = requests.post(post_url, data=v)
    assert r.status_code == requests.codes.ok  # Check for errors

Now we can create a query for the database that resamples to 4 points and send it:

q = {
    'selectors' : ['temperature', 'humidity'],
    'first' : '2013-07-24 01:00:00'
    'last'  : '2013-07-24 15:00:00'
    'npoints' : 4,
}

r = requests.post(post_url, data=t)
assert r.status_code == requests.codes.ok  # Check for errors
result = r.json()

Depending on the particular backend settings, the contents of result could be something like:

{
    'times' : ['2013-07-24 01:00:00',
               '2013-07-24 05:00:00',
               '2013-07-24 09:00:00',
               '2013-07-24 14:00:00'],
    'values' : [
        [ 25.75, 54.7],
        [ 27.25, 54.1],
        [ 29.25, 53.3],
        [ 31.25, 52.5]
        ]
}

If a series is an array type, the entries in the values array will be lists (possibly nested, if the array has multiple dimensions), and if the series is a blob type, the value will be a string containing a URL to retrieve the appropriate blob from the Natural Log server.

The default strategies for the series (mean and linear) were used in the above query, but we can also decide to override them in the selector using the forms name:reduction (leaving the interpolation strategy to be the default) or name:reduction:interpolation (overriding both strategies).

For example, this query:

q = {
    'selectors' : ['temperature:closest', 'humidity:min'],
    'first' : '2013-07-24 01:00:00'
    'last'  : '2013-07-24 15:00:00'
    'npoints' : 4,
}

r = requests.post(post_url, data=t)
assert r.status_code == requests.codes.ok  # Check for errors
result = r.json()

returns the following result:

{
    'times' : ['2013-07-24 01:00:00',
               '2013-07-24 05:00:00',
               '2013-07-24 09:00:00',
               '2013-07-24 14:00:00'],
    'values' : [
        [ 25.5, 54.6],
        [ 27.5, 53.8],
        [ 29.5, 53.0],
        [ 31.5, 52.2]
        ]
}

Continuous Queries¶

Rather than polling for new data, you can subscribe to changes and receive updates pushed from the server via server-sent events (SSE). In this case, the results are sent as a series of (time, value) pairs rather than a list of times and a list of values. This mode is selected by leaving the parameter last out of the query.

GET /¶

Get a list of data series.

POST /create¶

Create a new data series.

GET /data/[series name]¶

Get the original values recorded for this data series, without resampling.

GET /data/[series name]/[index]¶

Get a single raw value from a series. This is primarily used to fetch the contents of a blob with the mimetype set in the response.

POST /data/[series name]¶

Record a new value for this data series.

GET /data/[series name]/config¶

Get the configuration information for this data series.

POST /data/[series name]/config¶

Modify the configuration information for this data series. Only the unit and description of the series can be changed this way.

GET /query¶

Resample the selected data series and return the result. The query engine may return results with slightly different first and last times, as well as a different number of points. If last is omitted, the request is interpreted as a continuous query and the requested results and any future results are pushed via a persistent server-sent events (SSE) connection.