ReproZip’s Documentation¶

Required Software Packages¶

Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow reprounzip to work, but some features will not be available [1]. If you don’t have Python on your machine, you can get it from python.org [2]; you should prefer a 2.x release to a 3.x one. You will also need the pip installer.

Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:

Installing reprozip¶

To install the reprozip component, simply run the following command:

$ pip install reprozip

To update the software, use the flag -U:

$ pip install -U reprozip

Installing reprounzip¶

To install the reprounzip component, simply run the following command:

$ pip install reprounzip

To update the software, use the flag -U:

$ pip install -U reprounzip

The additional plugins for reprounzip can also be installed using the same command:

$ pip install reprounzip-docker reprounzip-vagrant

Alternatively, you can install reprounzip with all the available plugins using:

$ pip install reprounzip[all]

Binaries¶

An installer containing Python 2.7, reprounzip, and all the plugins can be downloaded from GitHub.

Required Software Packages¶

Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow reprounzip to work, but some features will not be available [6]. If you don’t have Python on your machine, you can get it from python.org; you should prefer a 2.x release to a 3.x one. You will also need the pip installer.

Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:

Installing reprounzip¶

First, be sure to upgrade setuptools:

$ pip install -U setuptools

To install the reprounzip component, simply run the following command:

$ pip install reprounzip

To update the software, use the flag -U:

$ pip install -U reprounzip

The additional plugins for reprounzip can also be installed using the same command:

$ pip install reprounzip-docker reprounzip-vagrant

Alternatively, you can install reprounzip with all the available plugins using:

$ pip install reprounzip[all]

Binaries¶

A 32-bit installer containing Python 2.7, reprounzip, and all the plugins can be downloaded from GitHub.

Required Software Packages¶

Python 2.7.3 or greater is recommended to run ReproZip. Older versions should allow reprounzip to work, but some features will not be available [9]. If you don’t have Python on your machine, you can get it from python.org; you should prefer a 2.x release to a 3.x one. You will also need the pip installer.

Besides Python and pip, each component or plugin to be used may have additional dependencies that you need to install (if you do not have them already installed in your environment), as described below:

Installing reprounzip¶

To install the reprounzip component, simply run the following command:

$ pip install reprounzip

To update the software, use the flag -U:

$ pip install -U reprounzip

The additional plugins for reprounzip can also be installed using the same command:

$ pip install reprounzip-vagrant
$ pip install reprounzip-docker

Alternatively, you can install reprounzip with all the available plugins using:

$ pip install reprounzip[all]

Packing Multiple Command Lines¶

As mentioned before, ReproZip allows multiple runs (i.e., command lines) to be traced and included in the same package. Alternatively, users can create a simple script that runs all the command lines, and pass that to reprozip trace. However, in this case, there will be no flexibility in choosing a single run to be reproduced, since the entire script will be re-executed.

Note that this flexibility has the caveat that users may reproduce the runs in a different order than the one originally used while tracing. If the order is important for the reproduction (e.g.: each run represents a step in a dataflow), please make sure to inform the correct reproduction order to whoever wants to replicate the experiment. This can also be obtained by running reprounzip graph; please refer to Creating a Provenance Graph for more information.

Packing GUI and Interactive Tools¶

ReproZip is able to pack GUI tools. Additionally, there is no restriction in packing interactive experiments (i.e., experiments that require input from users). Note, however, that if entering something different can make the experiment load additional dependencies, the experiment will probably fail when reproduced on a different machine.

Capturing Connections to Servers¶

When reproducing an experiment that communicates with a server, the experiment will try to connect to the same server, which may or may not fail depending on the status of the server at the moment of the reproduction. However, if the experiment uses a local server (e.g.: database) that the user has control over, this server can also be captured, together with the experiment, to ensure that the connection will succeed. Users should create a script to:

• start the server,
• execute the experiment, and
• stop the server,

and use reprozip to trace the script execution, rather than the experiment itself. In this way, ReproZip is able to capture the local server as well, which ensures that the server will be alive at the time of the reproduction. Alternatively, users can trace each run individually and include them all in the same package, but since one may reproduce individual runs, this does not ensure that the server will be alive when reproducing the experiment.

Excluding Sensitive and Third-Party Information¶

ReproZip automatically tries to identify log and temporary files, removing them from the package, but the configuration file should be edited to remove any sensitive information that the experiment uses, or any third-party file/software that should not be distributed. Note that the ReproZip team is not responsible for personal and non-authorized files that may get distributed in a package; users should double-check the configuration file and their package before sending it to others.

Identifying Output Files¶

The reprozip component tries to automatically identify the main output files generated by the experiment during the trace command to provide useful interfaces for users during the unpacking step. However, if the experiment creates unique names for its outputs every time it is executed (e.g.: names with current date and time), the reprounzip component will not be able to correctly detect these; it assumes that input and output files do not have their path names changed between different executions. In this case, handling output files will fail. It is recommended that users modify their experiment (or use a wrapper script) to generate a symbolic link (with a fixed name) that always points to the latest result, and use that as the output file’s path in the configuration file (under the inputs_outputs section).

Showing Package Information¶

Before unpacking an experiment, it is often useful to have further information with respect to its package. The reprounzip info command allows users to do so:

$ reprounzip info <package>

where <package> corresponds to the experiment package (i.e., the .rpz file).

The output of this command has three sections. The first section, Pack information, contains general information about the experiment package, including size and total number of files:

----- Pack information -----
Compressed size: <compressed-size>
Unpacked size: <unpacked-size>
Total packed paths: <number>

The next section, Metadata, contains information about dependencies (i.e., software packages), machine architecture from the packing environment, and experiment runs:

----- Metadata -----
Total software packages: <total-number-software-packages>
Packed software packages: <number-packed-software-packages>
Architecture: <original-architecture> (current: <current-architecture>)
Distribution: <original-operating-system> (current: <current-operating-system>)
Runs:
    <run-id>: <command-line>
    <run-id>: <command-line>
    ...

Note that, for Architecture and Distribution, the command shows information with respect to both the original environment (i.e., the environment where the experiment was packed) and the current one (i.e., the environment where the experiment is to be unpacked). This helps users understand the differences between the environments in order to provide a better guidance in choosing the most appropriate unpacker.

If the verbose mode is used, more detailed information on the runs is provided:

$ reprounzip -v info <package>
...
----- Metadata -----
...
Runs:
    <run-id>: <command-line>
        wd: <working-directory>
        exitcode: <exit-code>
    <run-id>: <command-line>
        wd: <working-directory>
        exitcode: <exit-code>
    ...

Last, the section Unpackers shows which of the installed reprounzip unpackers can be successfully used in the current environment:

----- Unpackers -----
Compatible:
    ...
Incompatible:
    ...
Unknown:
    ...

Compatible lists the unpackers that can be used in the current environment, while Incompatible lists the unpackers that are not supported in the current environment. When using the verbose mode, an additional Unknown list shows the installed unpackers that may not work. As an example, for an experiment originally packed on Ubuntu and a user reproducing it on Windows, the vagrant unpacker (available through the reprounzip-vagrant plugin) is compatible, but installpkgs is not; vagrant may also be listed under Unknown if vagrant is not in PATH (e.g.: if Vagrant is not installed).

Showing Input and Output Files¶

The reprounzip showfiles command can be used to list the input and output files defined for the experiment. These files are identified by an id, which is either chosen by ReproZip or set in the configuration file before creating the .rpz file:

$ reprounzip showfiles package.rpz
Input files:
    program_config
    ipython_config
    input_data
Output files:
    rendered_image
    logfile

Using the flag -v shows the complete path of each of these files in the experiment environment:

$ reprounzip -v showfiles package.rpz
Input files:
    program_config (/home/user/.progrc)
    ipython_config (/home/user/.ipython/profile_default/ipython_config.py)
    input_data (/home/user/experiment/input.bin)
Output files:
    rendered_image (/home/user/experiment/output.png)
    logfile (/home/user/experiment/log.txt)

This command is particularly useful if you want to replace an input file with your own, or to get and save an output file for further examination. Please refer to Managing Input and Output Files for more information.

Creating a Provenance Graph¶

ReproZip also allows users to generate a provenance graph related to the experiment execution. This graph shows the relationships between files, library dependencies, and binaries during the execution. To generate such a graph, the reprounzip graph command should be used:

$ reprounzip graph graph-file.dot package.rpz
$ dot -Tpng graph-file.dot -o image.png

where graph-file.dot corresponds to the graph, outputted in the DOT language. You can use Graphviz to load and visualize the graph.

The directory Unpacker: Unpacking as a Plain Directory¶

The directory unpacker (reprounzip directory) allows users to unpack the entire experiment (including library dependencies) in a single directory, and to reproduce the experiment directly from that directory. It does so by automatically setting up environment variables (e.g.: PATH, HOME, and LD_LIBRARY_PATH) that point the experiment execution to the created directory, which has the same structure as in the packing environment.

Please note that, although this unpacker is easy to use and does not require any privilege on the reproducing machine, it is unreliable since the directory is not isolated in any way from the remainder of the system. In particular, should the experiment use absolute paths, they will hit the host system instead. However, if the system has all the required packages (see The installpkgs Unpacker: Installing Software Packages), and the experiment’s files are addressed with relative paths, the use of this unpacker should not cause any problems.

The chroot Unpacker: Providing Isolation with the chroot Mechanism¶

In the chroot unpacker (reprounzip chroot), similar to reprounzip directory, a directory is created from the experiment package; however, a full system environment is also built, which can then be run with chroot(2), a Linux mechanism that changes the root directory / for the experiment to the experiment directory. Therefore, this unpacker addresses the limitation of the directory unpacker and does not fail in the presence of hardcoded absolute paths. Note as well that it does not interfere with the current environment since the experiment is isolated in that single directory.

The installpkgs Unpacker: Installing Software Packages¶

By default, ReproZip identifies if the current environment already has the required software packages for the experiment, then using the installed ones for reproduction. For the non-installed software packages, it uses the dependencies packed in the original environment and extracted under the experiment directory.

Users may also let ReproZip try and install all the dependencies of the experiment on their machine by using the installpkgs unpacker (reprounzip installpkgs). This unpacker currently works for Debian and Debian-based operating systems only (e.g.: Ubuntu), and uses the dpkg package manager to automatically install all the required software packages directly on the current machine, thus interfering with your environment.

To install the required dependencies, the following command should be used:

$ reprounzip installpkgs <package>

Users may use flag y or assume-yes to automatically confirm all the questions from the package manager; flag missing to install only the software packages that were not originally included in the experiment package (i.e.: software packages excluded in the configuration file); and flag summary to simply provide a summary of which software packages are installed or not in the current environment without installing any dependency.

The vagrant Unpacker: Building a Virtual Machine¶

The vagrant unpacker (reprounzip vagrant) allows an experiment to be unpacked and reproduced using a virtual machine created through Vagrant. Therefore, the experiment can be reproduced in any environment supported by this tool, i.e., Linux, Mac OS X, and Windows. Note that the plugin assumes that Vagrant is installed in the current environment.

In addition to the commands listed in Using an Unpacker, you can use suspend to save the virtual machine state to disk, and setup/start to restart a previously-created machine:

$ reprounzip vagrant suspend <path>
$ reprounzip vagrant setup/start <path>

The docker Unpacker: Building a Docker Container¶

ReproZip can also extract and reproduce experiments as Docker containers. The docker unpacker (reprounzip docker) is responsible for such integration and it assumes that Docker is already installed in the current environment.

Setting Up an Experiment Directory¶

To create the directory where the execution will take place, the setup command should be used:

$ reprounzip vagrant setup <package> <path>

where <path> is the directory where the experiment will be unpacked, i.e., the experiment directory.

Note that, once this is done, you should only remove <path> with the destroy command described below: deleting this directory manually might leave files behind, or even damage your system through bound filesystems.

The other unpacker commands take the <path> argument; they do not need the original package for the reproduction.

Reproducing the Experiment¶

After creating the directory, the experiment can be reproduced by issuing the run command:

$ reprounzip vagrant run <path>

which will execute the experiment inside the experiment directory. Users may also change the command line of the experiment by using --cmdline:

$ reprounzip vagrant run <path> --cmdline <new-command-line>

where <new-command-line> is the modified command line. This is particularly useful to reproduce and test the experiment under different input parameter values. Using --cmdline without an argument only prints the original command line.

If the package contains multiple runs (separate commands that were packed together), you need to provide the id of the run to be reproduced:

$ reprounzip vagrant run <path> <run-id>
$ reprounzip vagrant run <path> <run-id> --cmdline <new-command-line>

If the experiment involves running a GUI tool, the graphical interface can be enable by using --enable-x11:

$ reprounzip vagrant run <path> --enable-x11

which will forward the X connection from the experiment to the X server running on your machine. In this case, make sure you have a running X server.

Removing the Experiment Directory¶

The destroy command will unmount mounted paths, destroy virtual machines, free container images, and delete the experiment directory:

$ reprounzip vagrant destroy <path>

Make sure you always use this command instead of simply deleting the directory manually.

Managing Input and Output Files¶

When tracing an experiment, ReproZip tries to identify which are the input and output files of the experiment. This can also be adjusted in the configuration file before packing. If the unpacked experiment has such files, ReproZip provides some commands to manipulate them.

First, you can list these files using the showfiles command:

$ reprounzip showfiles <path>
Input files:
    program_config
    ipython_config
    input_data
Output files:
    rendered_image
    logfile

To replace an input file with your own, reprounzip, you can use the upload command:

$ reprounzip vagrant upload <path> <input-path>:<input-id>

where <input-path> is the new file’s path and <input-id> is the input file to be replaced (from showfiles). This command overwrites the original path in the environment with the file you provided from your system. To restore the original input file, the same command, but in the following format, should be used:

$ reprounzip vagrant upload <path> :<input-id>

Running the showfiles command shows what the input files are currently set to:

$ reprounzip showfiles <path>
Input files:
    program_config
        (original)
    ipython_config
        C:\Users\Remi\Documents\ipython-config
...

In this example, the input program_config has not been changed (the one bundled in the .rpz file will be used), while the input ipython_config has been replaced.

After running the experiment, all the generated output files will be located under the experiment directory. To copy an output file from this directory to another desired location, use the download command:

$ reprounzip vagrant download <path> <output-id>:<output-path>

where <output-id> is the output file to be copied (from showfiles) and <output-path> is the desired destination of the file. If no destination is specified, the file will be printed to stdout:

$ reprounzip vagrant download <path> <output-id>:

Note that the upload command takes the file id on the right side of the colon (meaning that the path is the origin, and the id is the destination), while the download command takes it on the left side (meaning that the id is the origin, and the path is the destination).

Reproducing Multiple Execution Paths¶

The reprozip component can only guarantee that reprounzip will successfully reproduce the same execution path that the original experiment followed. There is no guarantee that the experiment won’t need a different set of files if you use a different configuration; if some of these files were not packed into the .rpz package, the reproduction may fail.

ReproZip Pack Format (.rpz)¶

An .rpz file is a tar.gz archive that contains two directories: METADATA, which contains meta-information from reprozip, and DATA, which contains the actual files that were packed and that will be unpacked to the target directory for reproducing the experiment.

The METADATA/config.yml file is in the same format as the configuration file generated by reprozip, but without the additional_patterns section (at this point, it has already been expanded to the actual list of files while packing).

The METADATA/trace.sqlite3 file is the original trace generated by the C tracer and maintained in a SQLite database; it contains all the information about the experiment, in case the configuration file is insufficient in some aspect. This file is used, for instance, by the graph unpacker, so that it can recover the exact hierarchy of processes, together with the executable images they execute and the files they access (with the time and mode of these accesses).

Structure¶

An unpacker is a Python module. It can be distributed separately or be part of a bigger distribution, given that it is declared in that distribution’s setup.py as an entry_point to be registered with pkg_resources (see setuptools’ dynamic discovery of services and plugins section). You should declare a function as entry_point reprounzip.unpackers. The name of the entry_point (before =) will be the reprounzip subcommand, and the value is a callable that will get called with the argparse.ArgumentParser object for that subcommand.

The package reprounzip.unpackers is a namespace package, so you should be able to add your own unpackers there if you want to. Please remember to put the correct code in the __init__.py file (which you can copy from here) so that namespace packages work correctly.

The modules reprounzip.common, reprounzip.utils, and reprounzip.unpackers.common contain utilities that you might want to use (make sure to list reprounzip as a requirement in your setup.py).

Example of setup.py:

setup(name='reprounzip-vagrant',
      namespace_packages=['reprounzip', 'reprounzip.unpackers'],
      install_requires=['reprounzip>=0.4'],
      entry_points={
          'reprounzip.unpackers': [
              'vagrant = reprounzip.unpackers.vagrant:setup'
              # The setup() function sets up the parser for reprounzip vagrant
          ]
      }
      # ...
)

Usual Commands¶

If possible, you should try to follow the same command names that the official unpackers use, which are:

• setup: to create the experiment directory and set everything for execution;
• run: to reproduce the experiment;
• destroy: to bring down all that setup and to prepare and delete the experiment directory safely;
• upload and download: to replace input files in the experiment, and to get the output files for further examination, respectively.

If these commands can be broken down into different steps that you want to expose to the user, or if you provide completely different actions from these defaults, you are free to add them to the parser as well. For instance, the vagrant unpacker exposes setup/start, which starts or resumes the virtual machine, and destroy/vm, which stops and deallocates the virtual machine but leaves the template for possible reuse.

A Note on File Paths¶

ReproZip supports Python 2 and 3, is portable to different operating systems, and is meant to accept a wide variety of configurations so that it is compatible with most experiments out there. Even trickier, reprounzip-vagrant needs to manipulate POSIX filenames on Windows, e.g.: in the unpacker. Therefore, the rpaths library is used everywhere internally. You should make sure to use the correct type of path (either PosixPath or Path) and to cast these to the type that Python functions expect, keeping in mind 2/3 differences (most certainly either filename.path or str(filename)).

Experiment Directory Format¶

Unpackers usually create a directory with everything necessary to later run the experiment. This directory is created by the setup operation, cleaned up by destroy, and is the argument to every command. For example, with reprounzip-vagrant:

$ reprounzip vagrant setup someexperiment.rpz mydirectory
$ reprounzip vagrant upload mydirectory /tmp/replace.txt:input_text

Unpackers unpack the config.yml file to the root of that directory, and keep status information in a .reprounzip file, which is a dict in pickle format. Following the same structure will allow the showfiles command, as well as FileUploader and FileDownloader classes, to work correctly. Please try to follow this structure.

Signals¶

Since version 0.4.1, reprounzip has signals that can be used to hook in plugins, although no such plugin has been released at this time. To ensure that these work correctly when using your unpacker, you should emit them when appropriate. The complete list of signals is available in signal.py.