logtag¶

logtag accepts lines of log input on STDIN, applies a filter chain to the log data, and then prints the resulting log entries to STDOUT.

By default, logtag has a filter chain of init_txt,add_timestamp,add_source_host. This means that it will treat incoming log data as plain text, and will add a timestamp and hostname data to the log entries.

To use logtag in its default configuration, simply pass data on STDIN, calling logtag without arguments:

$ program_to_log | logtag
{"@timestamp": "2013-05-10T16:06:53.441811Z", "@source_host": "lynx.local", "@message": "my log entry"}
{"@timestamp": "2013-05-10T16:06:53.442090Z", "@source_host": "lynx.local", "@message": "another one"}

To tag each log entry with “foo” and parse Lograge format logs into the @fields field:

$ program_to_log | logtag --filters-append add_tags:foo,parse_lograge
{"@timestamp": "2013-05-10T16:08:26.999239Z", ..., "@message": "a log with=some key=value pairs", "@tags": ["foo"], "@fields": {"with": "some", "key": "value"}}
{"@timestamp": "2013-05-10T16:08:26.999505Z", ..., "@message": "another=one", "@tags": ["foo"], "@fields": {"another": "one"}}

If your program emits JSON objects, you might want to use a different filter chain that parses the incoming data as JSON:

$ program_emitting_json | logtag --filters init_json,add_timestamp
{"age": 67, "name": "Bob Dole", "@timestamp": "2013-05-10T16:13:49.558039Z"}
{"age": 43, "name": "John Doe", "@timestamp": "2013-05-10T16:13:49.558452Z"}

logship¶

logship works just like logtag, but instead of printing the results to STDOUT, it ships the resulting log entries to one or more “shippers.” Shippers include:

• redis: a shipper that pushes log entries onto a Redis list.
• statsd: a shipper that submits data to Statsd on the basis of log contents.
• stdout: a shipper that simply prints to STDOUT. Mainly used for debugging.

To ship log data to a Redis server, you might invoke logship as follows:

$ program_to_log | logship --shipper redis,redis://localhost:6379

You can configure filters with the --filters-append and --filters options, just as you can with logtag:

$ program_emitting_json | logship --filters init_json,add_timestamp --shipper redis,redis://localhost:6379

The Redis shipper can take multiple server addresses. It will round-robin between the list of servers. If one server is down, or out-of-memory, or otherwise unable to accept log entries, logship will attempt to submit the log entry to one of the other servers.

$ program_to_log | logship --shipper redis,redis://redis-1.local:6379,redis://redis-2.local:6379

For further shipper documentation, see the API documentation for the tagalog.shipper package and its submodules (e.g. tagalog.shipper.redis, tagalog.shipper.statsd).

logtext¶

logtext accepts a stream of newline-delimited Logstash-format JSON documents on STDIN, and prints the log messages on STDOUT. If the log entries have a timestamp, the lines will be prefixed with the timestamp:

$ echo '{"@message":"hello"}\n{"@message":"world"}' | logtext
hello
world

logstamp¶

logstamp accepts lines of input on STDIN, and emits each line prefixed with a high-precision ISO8601 timestamp on STDOUT:

$ echo "hello\nworld" | logstamp
2013-05-13T08:59:50.750691Z hello
2013-05-13T08:59:50.751132Z world

filters Module¶

The filters Module contains the definitions for the filters supported by logtag and logship, as well as a number of functions used to construct the filter chains.

A filter is a generator function which takes an iterable as its first argument, and optionally takes additional configuration arguments. It returns a generator yielding the filtered log lines.

tagalog.filters.add_fields(iterable, **kw_fields)¶

Add fields to each item in iterable. Each key=value pair provided is merged into the @fields object, which will be created if required.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_fields(data_in, foo='bar', baz='qux')
>>> [x for x in data_out]
[{'@fields': {'foo': 'bar', 'baz': 'qux'}, '@message': 'one message'},
 {'@fields': {'foo': 'bar', 'baz': 'qux'}, '@message': 'another message'}]

tagalog.filters.add_source_host(iterable, override=False)¶

Add the FQDN of the current machine to the @source_host field of each item in iterable.

By default, existing @source_host fields will not be overwritten. This behaviour can be toggled with the override argument.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_source_host(data_in)
>>> [x for x in data_out]
[{'@source_host': 'lynx.local', '@message': 'one message'},
 {'@source_host': 'lynx.local', '@message': 'another message'}]

tagalog.filters.add_tags(iterable, *taglist)¶

Add tags to each item in iterable. Each tag is added to the @tags array, which will be created if required.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_tags(data_in, 'foo', 'bar')
>>> [x for x in data_out]
[{'@message': 'one message', '@tags': ['foo', 'bar']},
 {'@message': 'another message', '@tags': ['foo', 'bar']}]

tagalog.filters.add_timestamp(iterable, override=False)¶

Compute an accurate timestamp for each item in iterable, adding an accurate timestamp to each one when received. The timestamp is a usecond-precision ISO8601 string added to the @timestamp field.

By default, existing @timestamp fields will not be overwritten. This behaviour can be toggled with the override argument.

>>> data_in = [{'@message': 'one message'}, {'@message': 'another message'}]
>>> data_out = add_timestamp(data_in)
>>> [x for x in data_out]
[{'@timestamp': '2013-05-13T10:37:56.766743Z', '@message': 'one message'},
 {'@timestamp': '2013-05-13T10:37:56.767185Z', '@message': 'another message'}]

tagalog.filters.build(description)¶

Build a filter chain from a filter description string

tagalog.filters.get(name, args=[])¶

Get a filter function from a filter name and a list of unparsed arguments

tagalog.filters.init_json(iterable)¶

Read lines of JSON text from iterable and parse each line as a JSON object. Yield dicts for each line that successfully parses as a JSON object. Unparseable events will be skipped and raise a warning.

>>> data_in = ['{"@message": "one message"}', '{"@message": "another message"}']
>>> data_out = init_json(data_in)
>>> [x for x in data_out]
[{u'@message': u'one message'}, {u'@message': u'another message'}]

tagalog.filters.init_txt(iterable)¶

Read lines of text from iterable and yield dicts with the line data stored in the @message field.

>>> data_in = ["hello\n", "world\n"]
>>> data_out = init_txt(data_in)
>>> [x for x in data_out]
[{'@message': 'hello'}, {'@message': 'world'}]

tagalog.filters.parse_lograge(iterable)¶

Attempt to parse each dict or dict-like object in iterable as if it were in lograge format, (e.g. “status=200 path=/users/login time=125ms”), adding key-value pairs to '@fields‘ for each matching item.

>>> data_in = [{'@message': 'path=/foo/bar status=200 time=0.060'},
...            {'@message': 'path=/baz/qux status=503 time=1.651'}]
>>> data_out = parse_lograge(data_in)
>>> [x for x in data_out]
[{'@fields': {'status': '200', 'path': '/foo/bar', 'time': '0.060'}, '@message': 'path=/foo/bar status=200 time=0.060'},
 {'@fields': {'status': '503', 'path': '/baz/qux', 'time': '1.651'}, '@message': 'path=/baz/qux status=503 time=1.651'}]

tagalog.filters.pipeline(*functions)¶

Construct a filter pipeline from a list of filter functions

Given a list of functions taking an iterable as their only argument, and which return a generator, this function will return a single function with the same signature, which applies each function in turn for each item in the iterable, yielding the results.

That is, given filter functions a, b, and c, pipeline(a, b, c) will return a function which yields c(b(a(x))) for each item x in the iterable passed to it. For example:

>>> def a(iterable):
...     for item in iterable:
...         yield item + ':a'
...
>>> def b(iterable):
...     for item in iterable:
...         yield item + ':b'
...
>>> pipe = pipeline(a, b)
>>> data_in = ["foo", "bar"]
>>> data_out = pipe(data_in)
>>> [x for x in data_out]
['foo:a:b', 'bar:a:b']

io Module¶

tagalog.io.lines(fp)¶

Read lines of UTF-8 from the file-like object given in fp, making sure that when reading from STDIN, reads are at most line-buffered.

UTF-8 decoding errors are handled silently. Invalid characters are replaced by U+FFFD REPLACEMENT CHARACTER.

Line endings are normalised to newlines by Python’s universal newlines feature.

Returns an iterator yielding lines.

Subpackages¶