Welcome to Topik’s documentation!¶

Top-level API (basic, quick experiments)¶

Basic functions are provided as immediate children of the topik module:

>>> from topik import TopikProject, read_input, tokenize, vectorize, run_model

These provide defaults, and allow pass-through of any other arguments as keyword arguments.

Package structure¶

This is likely the most useful API for Python power users. Import functions directly from their packages. This will show you their complete list of available arguments, and give you autocompletion of those arguments.

>>> from topik.vectorizers.tfidf import tfidf
>>> from topik.models.plsa import plsa

Registered functions¶

This API layer exists primarily for creating extensible GUIs. All functions for each processing step are registered with a Borg-pattern dictionary upon import. This dictionary should be useful in modularly exposing available functionality, simply by querying keys. What is not (yet) implemented is any sort of type hinting that would let you also create appropriate input widgets for all inputs.

Output formats¶

Output formats are how your data are represented to further processing and modeling. Projects use a particular output format to store your intermediate results. To ensure a uniform interface, output formats implement the interface described by OutputInterface. Presently, two such backends are implemented: InMemoryOutput and ElasticSearchOutput. Available outputs can be examined by checking the keys of the topik.fileio.registered_outputs dictionary:

>>> from topik.fileio import registered_outputs
>>> list(registered_outputs.keys())

The default output is the InMemoryOutput. No additional arguments are necessary. InMemoryOutput stores everything in a Python dictionary. As such, it is memory intensive. All operations done with a InMemoryOutput block until complete. InMemoryOutput is the simplest to use, but it will ultimately limit the size of analyses that you can perform.

The ElasticSearchOutput can be specified to TopikProject using the output_type argument. It must be accompanied by another keyword argument, output_args, which should be a dictionary containing connection details and any additional arguments.

>>> output_args = {"source": "localhost", "index": "destination_index", content_field="text"}
>>> project = TopikProject("my_project", output_type="ElasticSearchOutput",
                           output_args=output_args)

ElasticSearchOutput stores everything in an Elasticsearch instance that you specify. Operations do not block, and have “eventual consistency”: the corpus will eventually have all of the documents you sent available, but not necessarily immediately after the read_input function returns. This lag time is due to Elasticsearch indexing the data on the server side.

Saving and loading projects¶

Projects are designed to help you go back to some earlier state. There are several dictionary-like objects accessible on the project object:

>>> project = TopikProject("my_project")
>>> project.output.tokenized_corpora
>>> project.output.vectorized_corpora
>>> project.output.modeled_corpora

These are more quickly accessible as selected properties of the project:

>>> project.selected_filtered_corpus
>>> project.selected_tokenized_corpus
>>> project.selected_vectorized_corpus
>>> project.selected_modeled_corpus

These selected properties keep track of the last-used technique, and give you the corresponding data.

You can change the selected state using the select_tokenized_corpus(), select_modeled_corpus(), and select_modeled_corpus() methods.

Project objects also persist their state to disk. This is done in two or more files, dependent on the output backend in use. There will always be two files:

• a .topikproject file, describing the project metadata and how to load the project
• a .topikdata file, containing or describing how to obtain the data contained in the project.

Each of the above files are JSON format. Additional files may store data in binary format. If you move your outputs on disk, make sure to move all of them, or Topik will not be able to load your results.

If using the project with a context manager, data is saved and connections are closed when the context ends. Otherwise, call the save() to write data to disk, or the close() method to write data to disk and close connections.

Loading projects is achieved by providing simply the project name that you provided when creating the project. Additional connection details will be loaded from disk automatically.

>>> project = TopikProject("my_project")

JSON additional options¶

For JSON stream and “large JSON” inputs, an additional keyword argument may be passed, json_prefix, which is the period-separated path leading to the single content field. This is for content fields not at the root level of the JSON document. For example, given the JSON content:

[ {"nested": {"dictionary": {"text": "I am the text you're looking for."} } } ]

You would read using the following json_prefix argument:

>>> corpus = read_input(source="data_file.json", json_prefix="nested.dictionary")

Elasticsearch additional options and notes¶

The Elasticsearch importer expects a full string specifying the Elasticsearch server. This string at a minimum must contain both the server address and the index to access (if any). All results returned from the Elasticsearch query contain only the contents of the ‘_source’ field returned from the query.

>>> corpus = read_input(source="https://localhost:9200", index="test_index")

Extra arguments passed by keyword are passed to the Elasticsearch instance creation. This can be used to pass additional login parameters, for example, to use SSL:

>>> corpus = read_input(source="https://user:secret@localhost:9200",
                        index="test_index", use_ssl=True)

The source argument for Elasticsearch also supports multiple servers, though this requires that you manually specify the ‘elastic’ source_type:

>>> corpus = read_input(source=["https://server1", "https://server2"],
                        index="test_index", source_type="elastic")

For more information on server options, please refer to Elasticsearch’s documentation.

Extra keyword arguments are also passed to the scroll helper that returns results. Of special note here, an additional query keyword argument can be passed to limit the records imported from the server. This query must follow the Elasticsearch query DSL. For more information on Elasticsearch query DSL, please refer to Elasticsearch’s DSL docs.

>>> query = "{"filtered": {"query": {"match": { "tweet": "full text search"}}}}"
>>> corpus = read_input(source="https://localhost:9200", index="test_index", query=query)

Tracking documents¶

One important aspect that hasn’t come up here is that documents are tracked by hashing their contents. Projects do this for you automatically:

>>> project = TopikProject("my_project")
>>> project.read_input("./reviews/", content_field="text")

If you do not use Topik’s project feature, then you need to create these id’s yourself. Tokenization and all subsequent steps expect data that has these id’s, with the idea that any future parallelism will use these id’s to keep track of data during and after processing. One way to get id’s is below:

>>> content_field = "text"
>>> raw_data = ((hash(item[content_field]), item[content_field]) for item in corpus)

Available methods¶

The tokenize method accepts a few arguments to specify a tokenization method and control behavior therein. The available tokenization methods are available in the topik.tokenizers.registered_tokenizers dictionary. The presently available methods are:

• “simple”: (default) lowercases input text and extracts single words. Uses Gensim.
• “ngrams”: Collects bigrams and trigrams in addition to single words. Uses NLTK.
• “entities”: Extracts noun phrases as entities. Uses TextBlob.
• “mixed”: first extracts noun phrases as entities, then follows up with simple tokenization for single words. Uses TextBlob.

All methods accept a keyword argument stopwords, which are words that will be ignored in tokenization. These are words that add little content value, such as prepositions. The default, None, loads and uses gensim’s STOPWORDS collection.

Collocation tokenization¶

Collocation tokenization collects phrases of words (pairs and triplets, bigrams and trigrams) that occur together often throughout your collection of documents.

To obtain the bigram and trigram patterns, use the ngrams() function:

>>> from topik.tokenizers import ngrams
>>> tokens = ngrams(corpus, freq_bounds=[(5,10000), (3, 10000)])

Tweakable parameters are:

• top_n: limit results to a maximum number
• min_length: the minimum length that any single word can be
• freq_bounds: list of tuples of [(min_freq, max_freq)]. Min_freq is the minimum number of times that a pair occurs before being considered. The first entry in this list is bigrams. Presently, only bigrams and trigrams are supported.

For small bodies of text, you’ll need small freq values, but this may be correspondingly “noisy.”

Entities tokenization¶

We refer to entities as noun phrases, as extracted by the TextBlob library. Topik provides the entities() function.

You can tweak noun phrase extraction with a minimum and maximum occurrence frequency. This is the frequency across your entire corpus of documents.

>>> from topik.tokenizers import entities
>>> tokens = entities(corpus, min_length=1, freq_min=4, freq_max=10000)

Mixed tokenization¶

mixed() tokenization employs both the entities tokenizer and the simple tokenizer, for when the entities tokenizer is overly restrictive, or for when words are interesting both together and apart. Usage is similar to the entities tokenizer:

>>> from topik.tokenizers import mixed
>>> tokens = mixed(corpus, min_length=1, freq_min=4, freq_max=10000)

Bag of Words vectorization¶

Bag of words is simply a word count. Words are collected from the tokenized input, assigned numeric identifiers, and counted.

bag_of_words() is the default method applied by the vectorize() function:

>>> from topik import vectorize
>>> vector_output = vectorize(tokenized_corpus)

TF-IDF vectorization¶

tfidf() (Term-Frequency-Inverse-Document-Frequency) is a scheme that weights words based not only on how much they occur in a single document, but also how much they occur across the entire corpus. Words that occur many times in one document, but not much over the corpus are deemed to be more important under this scheme.

NOTE: TF-IDF is NOT compatible with LDA models as Bag-Of-Words vectorization is a fundamental assumption of LDA.

>>> from topik import vectorize
>>> vector_output = vectorize(tokenized_corpus, method="tfidf")

Vectorizer output¶

Vectorization is the step at which conversion from text to numerical identifiers happens. For this reason, the output of the vectorizer is an object that contains both the vectorized corpus, and a dictionary mapping the text tokens to their numeric identifiers. This object is defined in VectorizerOutput. These output objects are passed directly to modeling functions.