dossier.fc — feature collections¶

Collections of named features.

This module provides dossier.fc.FeatureCollection and a number of supporting classes. A feature collection is a dictionary mapping a feature name to a feature representation. The representation is typically a dossier.fc.StringCounter, an implementation of collections.Counter, which is fundamentally a mapping from a string value to an integer.

This representation allows multiple values to be stored for multiple bits of information about some entity of interest. The weights in the underlying counters can be used to indicate how common a particular value is, or how many times it appears in the source documents.

president = FeatureCollection()
president['NAME']['Barack Obama'] += 1
president['entity_type']['PER'] += 1
president['PER_ADDRESS']['White House'] += 1
president['PER_ADDRESS']['1600 Pennsylvania Ave.'] += 1

Feature collections can have representations other than the basic string-counter representation; these representations may not preserve the source strings, but will still be suitable for machine-learning applications.

Feature collections can be serialized to RFC 7049 CBOR format, similar to a binary JSON representation. They can also be stored sequentially in flat files using dossier.fc.FeatureCollectionChunk as an accessor.

class dossier.fc.FeatureCollection(data=None, read_only=False)[source]¶

Bases: _abcoll.MutableMapping

A collection of features.

This is a dictionary from feature name to a collections.Counter or similar object. In typical use callers will not try to instantiate individual dictionary elements, but will fall back on the collection’s default-value behavior:

fc = FeatureCollection()
fc['NAME']['John Smith'] += 1

The default default feature type is StringCounter.

Feature collection construction and serialization:

__init__(data=None, read_only=False)[source]¶

Creates a new empty feature collection.

If data is a dictionary-like object with a structure similar to that of a feature collection (i.e., a dict of multisets), then it is used to initialize the feature collection.

classmethod loads(data)[source]¶

Create a feature collection from a CBOR byte string.

dumps()[source]¶

Create a CBOR byte string from a feature collection.

classmethod from_dict(data, read_only=False)[source]¶

Recreate a feature collection from a dictionary.

The dictionary is of the format dumped by to_dict(). Additional information, such as whether the feature collection should be read-only, is not included in this dictionary, and is instead passed as parameters to this function.

to_dict()[source]¶

Dump a feature collection’s features to a dictionary.

This does not include additional data, such as whether or not the collection is read-only. The returned dictionary is suitable for serialization into JSON, CBOR, or similar data formats.

static register_serializer(feature_type, obj)[source]¶

This is a class method that lets you define your own feature type serializers. tag should be the name of the feature type that you want to define serialization for. Currently, the valid values are StringCounter, Unicode, SparseVector or DenseVector.

Note that this function is not thread safe.

obj must be an object with three attributes defined.

obj.loads is a function that takes a CBOR created Python data structure and returns a new feature counter.

obj.dumps is a function that takes a feature counter and returns a Python data structure that can be serialized by CBOR.

obj.constructor is a function with no parameters that returns the Python type that can be used to construct new features. It should be possible to call obj.constructor() to get a new and empty feature counter.

Feature collection values and attributes:

read_only¶

Flag if this feature collection is read-only.

When a feature collection is read-only, no part of it can be modified. Individual feature counters cannot be added, deleted, or changed. This attribute is preserved across serialization and deserialization.

generation¶

Get the generation number for this feature collection.

This is the highest generation number across all counters in the collection, if the counters support generation numbers. This collection has not changed if the generation number has not changed.

DISPLAY_PREFIX = '#'¶

Prefix on names of features that are human-readable.

Processing may convert a feature name to a similar feature #name that is human-readable, while converting the original feature to a form that is machine-readable only; for instance, replacing strings with integers for faster comparison.

EPHEMERAL_PREFIX = '_'¶

Prefix on names of features that are not persisted.

to_dict() and dumps() will not write out features that begin with this character.

Feature collection computation:

__add__(other)[source]¶

Add features from two FeatureCollections.

>>> fc1 = FeatureCollection({'foo': Counter('abbb')})
>>> fc2 = FeatureCollection({'foo': Counter('bcc')})
>>> fc1 + fc2
FeatureCollection({'foo': Counter({'b': 4, 'c': 2, 'a': 1})})

Note that if a feature in either of the collections is not an instance of collections.Counter, then it is ignored.

__sub__(other)[source]¶

Subtract features from two FeatureCollections.

>>> fc1 = FeatureCollection({'foo': Counter('abbb')})
>>> fc2 = FeatureCollection({'foo': Counter('bcc')})
>>> fc1 - fc2
FeatureCollection({'foo': Counter({'b': 2, 'a': 1})})

Note that if a feature in either of the collections is not an instance of collections.Counter, then it is ignored.

__mul__(coef)[source]¶

__imul__(coef)[source]¶

In-place multiplication by a scalar.

total()[source]¶

Returns sum of all counts in all features that are multisets.

merge_with(other, multiset_op, other_op=None)[source]¶

Merge this feature collection with another.

Merges two feature collections using the given multiset_op on each corresponding multiset and returns a new FeatureCollection. The contents of the two original feature collections are not modified.

For each feature name in both feature sets, if either feature collection being merged has a collections.Counter instance as its value, then the two values are merged by calling multiset_op with both values as parameters. If either feature collection has something other than a collections.Counter, and other_op is not None, then other_op is called with both values to merge them. If other_op is None and a feature is not present in either feature collection with a counter value, then the feature will not be present in the result.

Parameters:	other (FeatureCollection) – The feature collection to merge into self. multiset_op (fun(Counter, Counter) -> Counter) – Function to merge two counters other_op (fun(object, object) -> object) – Function to merge two non-counters
Return type:	FeatureCollection

class dossier.fc.StringCounter(*args, **kwargs)[source]¶

Bases: collections.Counter

Simple counter based on exact string matching.

This is a subclass of collections.Counter that includes a generation counter so that it can be used in a cache.

StringCounter is the default feature type in a feature collection, so you typically don’t have to instantiate a StringCounter explicitly:

fc = FeatureCollection()
fc['NAME']['John Smith'] += 1

But instantiating directly works too:

sc = StringCounter()
sc['John Smith'] += 1

fc = FeatureCollection({'NAME': sc})
fc['NAME']['John Smith'] += 1
assert fc['NAME']['John Smith'] == 2

Note that instances of this class support all the methods defined for a collections.Counter, but only the ones unique to StringCounter are listed here.

__init__(*args, **kwargs)[source]¶

Initialize a StringCounter with existing counts:

>>> sc = StringCounter(a=4, b=2, c=0)
>>> sc['b']
2

See the documentation for collections.Counter for more examples.

truncate_most_common(*args, **kwargs)[source]¶

Sorts the counter and keeps only the most common items up to truncation_length in place.

read_only¶

Flag indicating whether this collection is read-only.

This flag always begins as False, it cannot be set via the constructor for compatibility with collections.Counter. If this flag is set, then any operations that mutate it will raise ReadOnlyException.

generation¶

Generation number for this counter instance.

This number is incremented by every operation that mutates the counter object. If two collections are the same object and have the same generation number, then they are identical.

Having this property allows a pair of id(sc) and the generation to be an immutable hashable key for things like memoization operations, accounting for the possibility of the counter changing over time.

>>> sc = StringCounter({'a': 1})
>>> cache = {(id(sc), sc.generation): 1}
>>> (id(sc), sc.generation) in cache
True
>>> sc['a']
1
>>> (id(sc), sc.generation) in cache
True
>>> sc['a'] += 1
>>> sc['a']
2
>>> (id(sc), sc.generation) in cache
False

class dossier.fc.SparseVector[source]¶

Bases: object

An abstract class for sparse vectors.

Currently, there is no default implementation of a sparse vector.

Other implementations of sparse vectors must inherit from this class. Otherwise they cannot be used inside a dossier.fc.FeatureCollection.

class dossier.fc.DenseVector[source]¶

Bases: object

An abstract class for dense vectors.

Currently, there is no default implementation of a dense vector.

Other implementations of dense vectors must inherit from this class. Otherwise they cannot be used inside a dossier.fc.FeatureCollection.

dossier.fc.FeatureCollectionChunk¶

alias of <Mock id='140505177601168'>

class dossier.fc.ReadOnlyException[source]¶

Bases: dossier.fc.exceptions.BaseException

Code attempted to modify a read-only feature collection.

This occurs when adding, deleting, or making other in-place modifications to a FeatureCollection that has its read_only flag set. It also occurs when attempting to make changes to a StringCounter contained in such a collection.

class dossier.fc.SerializationError[source]¶

Bases: dossier.fc.exceptions.BaseException

A problem occurred serializing or deserializing.

This can occur if a FeatureCollection has an unrecognized feature type, or if a CBOR input does not have the correct format.

dossier.store — store feature collections¶

A simple storage interface for feature collections.

mod:dossier.store provides a convenient interface to a mod:kvlayer table for storing dossier.fc.FeatureCollection. The interface consists of methods to query, search, add and remove feature collections from the store. It also provides functions for defining and searching indexes.

Using a storage backend in your code requires a working kvlayer configuration, which is usually written in a YAML file like so:

kvlayer:
  app_name: store
  namespace: dossier
  storage_type: redis
  storage_addresses: ["redis.example.com:6379"]

And here’s a full working example that uses local memory to store feature collections:

from dossier.fc import FeatureCollection
from dossier.store import Store
import kvlayer
import yakonfig

yaml = """
kvlayer:
  app_name: store
  namespace: dossier
  storage_type: local
"""
with yakonfig.defaulted_config([kvlayer], yaml=yaml):
    store = Store(kvlayer.client())

    fc = FeatureCollection({u'NAME': {'Foo': 1, 'Bar': 2}})
    store.put([('1', fc)])
    print store.get('1')

See the documentation for yakonfig for more details on the configuration setup.

Another example showing how to store, retrieve and delete a feature collection:

fc = dossier.fc.FeatureCollection()
fc[u'NAME'][u'foo'] += 1
fc[u'NAME'][u'bar'] = 42

kvl = kvlayer.client()
store = dossier.store.Store(kvl)

store.put('{yourid}', fc)
assert store.get('{yourid}')[u'NAME'][u'bar'] == 42
store.delete('{yourid}')
assert store.get('{yourid}') is None

Here is another example that demonstrates use of indexing to enable a poor man’s case insensitive search:

fc = dossier.fc.FeatureCollection()
fc[u'NAME'][u'foo'] += 1
fc[u'NAME'][u'bar'] = 42

kvl = kvlayer.client()
store = dossier.store.Store(kvl)

# Index transforms must be defined on every instance of `Store`.
# (The index data is persisted; the transforms themselves are
# ephemeral.)
store.define_index(u'name_casei',
                   create=feature_index(u'NAME'),
                   transform=lambda s: s.lower().encode('utf-8'))

store.put('{yourid}', fc)  # `put` automatically updates indexes.
assert list(store.index_scan(u'name_casei', 'FoO'))[0] == '{yourid}'

class dossier.store.Store(kvlclient, impl=None, feature_indexes=None)[source]¶

A feature collection database.

A feature collection database stores feature collections for content objects like profiles from external knowledge bases.

Every feature collection is keyed by its content_id, which is a byte string. The value of a content_id is specific to the type of content represented by the feature collection. In other words, its representation is unspecified.

__init__(kvlclient, impl=None, feature_indexes=None)[source]¶

Connects to a feature collection store.

This also initializes the underlying kvlayer namespace.

Parameters:	kvl (kvlayer.AbstractStorage) – kvlayer storage client
Return type:	Store

get(content_id)[source]¶

Retrieve a feature collection from the store. This is the same as get_many([content_id])

If the feature collection does not exist None is returned.

Return type:	dossier.fc.FeatureCollection

get_many(content_id_list)[source]¶

Yield (content_id, data) tuples for ids in list.

As with get(), if a content_id in the list is missing, then it is yielded with a data value of None.

Return type:	yields tuple(str, dossier.fc.FeatureCollection)

put(items, indexes=True)[source]¶

Add feature collections to the store.

Given an iterable of tuples of the form (content_id, feature collection), add each to the store and overwrite any that already exist.

This method optionally accepts a keyword argument indexes, which by default is set to True. When it is True, it will create new indexes for each content object for all indexes defined on this store.

Note that this will not update existing indexes. (There is currently no way to do this without running some sort of garbage collection process.)

Parameters:	items (iterable) – iterable of (content_id, FeatureCollection).

delete(content_id)[source]¶

Delete a feature collection from the store.

Deletes the content item from the store with identifier content_id.

Parameters:	content_id (str) – identifier for the content object represented by a feature collection

delete_all()[source]¶

Deletes all storage.

This includes every content object and all index data.

scan(*key_ranges)[source]¶

Retrieve feature collections in a range of ids.

Returns a generator of content objects corresponding to the content identifier ranges given. key_ranges can be a possibly empty list of 2-tuples, where the first element of the tuple is the beginning of a range and the second element is the end of a range. To specify the beginning or end of the table, use an empty tuple ().

If the list is empty, then this yields all content objects in the storage.

Parameters:	key_ranges – as described in kvlayer._abstract_storage.AbstractStorage()
Return type:	generator of (content_id, dossier.fc.FeatureCollection).

scan_ids(*key_ranges)[source]¶

Retrieve content ids in a range of ids.

Returns a generator of content_id corresponding to the content identifier ranges given. key_ranges can be a possibly empty list of 2-tuples, where the first element of the tuple is the beginning of a range and the second element is the end of a range. To specify the beginning or end of the table, use an empty tuple ().

If the list is empty, then this yields all content ids in the storage.

Parameters:	key_ranges – as described in kvlayer._abstract_storage.AbstractStorage()
Return type:	generator of content_id

scan_prefix(prefix)[source]¶

Returns a generator of content objects matching a prefix.

The prefix here is a prefix for content_id.

Return type:	generator of (content_id, dossier.fc.FeatureCollection).

scan_prefix_ids(prefix)[source]¶

Returns a generator of content ids matching a prefix.

The prefix here is a prefix for content_id.

Return type:	generator of content_id

Methods for indexing:

index_scan(idx_name, val)[source]¶

Returns ids that match an indexed value.

Returns a generator of content identifiers that have an entry in the index idx_name with value val (after index transforms are applied).

If the index named by idx_name is not registered, then a KeyError is raised.

Parameters:	idx_name (unicode) – name of index val (unspecified (depends on the index, usually unicode)) – the value to use to search the index
Return type:	generator of content_id
Raises:	KeyError

index_scan_prefix(idx_name, val_prefix)[source]¶

Returns ids that match a prefix of an indexed value.

Returns a generator of content identifiers that have an entry in the index idx_name with prefix val_prefix (after index transforms are applied).

If the index named by idx_name is not registered, then a KeyError is raised.

Parameters:	idx_name (unicode) – name of index val_prefix – the value to use to search the index
Return type:	generator of content_id
Raises:	KeyError

index_scan_prefix_and_return_key(idx_name, val_prefix)[source]¶

Returns ids that match a prefix of an indexed value, and the specific key that matched the search prefix.

Returns a generator of (index key, content identifier) that have an entry in the index idx_name with prefix val_prefix (after index transforms are applied).

If the index named by idx_name is not registered, then a KeyError is raised.

Parameters:	idx_name (unicode) – name of index val_prefix – the value to use to search the index
Return type:	generator of (index key, content_id)
Raises:	KeyError

define_index(idx_name, create, transform)[source]¶

Add an index to this store instance.

Adds an index transform to the current FC store. Once an index with name idx_name is added, it will be available in all index_* methods. Additionally, the index will be automatically updated on calls to put().

If an index with name idx_name already exists, then it is overwritten.

Note that indexes do not persist. They must be re-defined for each instance of Store.

For example, to add an index on the boNAME feature, you can use the feature_index helper function:

store.define_index('boNAME',
                   feature_index('boNAME'),
                   lambda s: s.encode('utf-8'))

Another example for creating an index on names:

store.define_index('NAME',
                   feature_index('canonical_name', 'NAME'),
                   lambda s: s.lower().encode('utf-8'))

Parameters:

idx_name (unicode) – The name of the index. Must be UTF-8 encodable. create – A function that accepts the transform function and a pair of (content_id, fc) and produces a generator of index values from the pair given using transform. transform – A function that accepts an arbitrary value and applies a transform to it. This transforms the stored value to the index value. This must produce a value with type str (or bytes).

dossier.store.feature_index(*feature_names)[source]¶

Returns a index creation function.

Returns a valid index create function for the feature names given. This can be used with the Store.define_index() method to create indexes on any combination of features in a feature collection.

Return type:	(val -> index val) -> (content_id, FeatureCollection) -> generator of [index val]

dossier.label — store ground truth data as labels¶

A simple storage interface for labels (ground truth data).

dossier.label provides a convenient interface to a kvlayer table for storing ground truth data, otherwise known as “labels.” Each label, at the highest level, maps two things (addressed by content identifiers) to a coreferent value. This coreferent value is an indication by a human that these two things are “the same”, “not the same” or “I don’t know if they are the same.” Sameness in this case is determined by the human doing the annotation.

Each label also contains an annotator_id, which identifies the human that created the label. A timestamp (in milliseconds since the Unix epoch) is also included on every label.

kvlayer:
  app_name: store
  namespace: dossier
  storage_type: redis
  storage_addresses: ["redis.example.com:6379"]

from dossier.label import Label, LabelStore, CorefValue
import kvlayer
import yakonfig

yaml = """
kvlayer:
  app_name: store
  namespace: dossier
  storage_type: local
"""
with yakonfig.defaulted_config([kvlayer], yaml=yaml):
    label_store = LabelStore(kvlayer.client())

    lab = Label('a', 'b', 'annotator', CorefValue.Positive)
    label_store.put(lab)

    assert lab == label_store.get('a', 'b', 'annotator')

dossier.models — Active learning¶

dossier.web — DossierStack web services¶