Installation¶

Install Databuild:

$ pip install databuild

Quickstart¶

Run databuild using a buildfile:

$ data-build.py buildfile.json

buildfile.json contains a list of operations to be performed on data. Think of it as a script for a spreadsheet.

An example of build file could be:

[
  {
    "operation": "sheets.import_data",
    "description": "Importing data from csv file",
    "params": {
      "sheet": "dataset1",
      "format": "csv",
      "filename": "dataset1.csv",
      "skip_last_lines": 1
    }
  },
  {
    "operation": "columns.add_column",
    "description": "Calculate the gender ratio",
    "params": {
      "sheet": "dataset1",
      "name": "Gender Ratio",
      "expression": {
        "language": "python",
        "content": "return float(row['Male Total']) / float(row['Female Total'])"
      }
    }
  },
  {
    "operation": "sheets.export_data",
    "description": "save the data",
    "params": {
      "sheet": "dataset1",
      "format": "csv",
      "filename": "dataset2.csv"
    }
  }
]

YAML buildfiles are also supported. Databuild will guess the type based on the extension.

Philosophy¶

Databuild is an alternative to more complex and complete packages like pandas, numpy, and R.

It’s aimed at users that are not necessarily data scientist and are looking to a simpler alternative to such softwares.

It’s admittely less performant than those and is not optimized for huge datasets. But Databuild is much easier to get started with.

The general principles in Databuild are:

• Low entry barrier
• Easy to install
• Easy to grasp
• Extensible

Databuild can be useful for scenarios such as:

• Documenting data transformations in your infoviz project
• Automate data processing in a declarative way

Buildfiles¶

A buildfile contains a list of operations to be performed on data. Think of it as a script for a spreadsheet.

JSON and YAML format are supported. databuild will guess the format based on the file extension.

An example of build file could be:

[
  {
    "operation": "sheets.import_data",
    "description": "Importing data from csv file",
    "params": {
      "sheet": "dataset1",
      "format": "csv",
      "filename": "dataset1.csv",
      "skip_last_lines": 1
    }
  },
  {
    "operation": "columns.add_column",
    "description": "Calculate the gender ratio",
    "params": {
      "sheet": "dataset1",
      "name": "Gender Ratio",
      "expression": {
        "language": "python",
        "content": "return float(row['Male Total']) / float(row['Female Total'])"
      }
    }
  },
  {
    "operation": "sheets.export_data",
    "description": "save the data",
    "params": {
      "sheet": "dataset1",
      "format": "csv",
      "filename": "dataset2.csv"
    }
  }
]

The same file in yaml:

- operation: sheets.import_data
  description: Importing data from csv file
  params:
    sheet: dataset1
    format: csv
    filename: dataset1.csv
    skip_last_lines: 1
- operation: columns.add_column
  description: Calculate the gender ratio
  params:
    sheet: dataset1
    name: Gender Ratio
    expression:
      language: python
      content: "return float(row['Male Total']) / float(row['Female Totale'])"
- operation: sheets.export_data
  description: save the data
  params:
    sheet: dataset1
    format: csv
    filename: dataset2.csv

Python API¶

Databuild can be integrated in your python project. Just import the build function:

from databuild.builder import build

build('buildfile.json')

Supported arguments:

• build_file Required. Path to the buildfile.
• settings Optional. Python module path containing the settings. Defaults to datbuild.settings
• echo Optional. Set this to True if you want the operations’ description printed to the screen. Defaults to False.

Operation Functions¶

Operations functions are regular Python function that perform actions on the book. Examples of operations are: sheets.import_data, columns.add_column, columns.update_column, and more.

They have a function name that identifies them, an optional description and a number of parameters that they accept. Different operation functions accept different parameters.

Custom Operation¶

You can add your custom operation and use them in your buildfile.

An Operation is just a regular python function. The first arguments has to be the workbook, but the remaining arguments will be pulled in from the params property of the operation in the buildfile.

def myoperation(workbook, foo, bar, baz):
    pass

Operations are defined in modules, which are just regulare Python files.

As long as your operation modules are in your PYTHONPATH, you can add them to your OPERATION_MODULES setting (see operation-modules-setting) and then call the operation in your buildfile by referencing its import path:

[
    ...,
    {
        "operation": "mymodule.myoperation",
        "description": "",
        "params": {
            "foo": "foos",
            "bar": "bars",
            "baz": "bazes"
        }
    }
]

Expressions¶

Expressions are objects encapsulating code for situations such as filtering or calculations.

An expression has the following properties:

• language: The name of the environment where the expression will be executed, as specified in settings.LANGUAGES. See LANGUAGES).
• content: The actual code to run, or
• path: path to a file containing the code to run

The expression will be evaluated inside a function and run against every row in the datasheet. The following context variables will be avalaible:

* ``row``: A dictionary representing the currently selected row.

Environments¶

Expressions are evaluated in the environment specified by their language property.

The value maps to a specific environment as specified in settings.LANGUAGES (See the LANGUAGES setting).

Functions¶

Functions are additional methods that can be used inside Expressions.

def myfunction(environment, book, **kwargs)

return environment.copy(my_return_value)

Settings¶

LANGUAGES = {
    'python': 'databuild.environments.python.PythonEnvironment',
}

FUNCTION_MODULES = (
    'databuild.functions.data',
)

OPERATION_MODULES = (
    "databuild.operations.sheets",
    "databuild.operations.columns",
)