Welcome to the HashDist documentation¶

Features¶

• Re-downloading all the sources on each build gets old quickly...
• Native support for multiple retrieval mechanisms. This is important as one wants to use tarballs for slowly-changing stable code, but VCS for quickly-changing code.
• Isolates dealing with various source code retrieval mechanisms from upper layers, who can simply pass along two strings regardless of method.
• Safety: Hashes are re-checked on the fly while unpacking, to protect against corruption or tainting of the source cache.
• Should be safe for multiple users to share a source cache directory on a shared file-system as long as all have write access, though this may need some work with permissions.

Source keys¶

The keys for a given source item can be determined a priori. The rules are as follows:

Tarballs/archives:

SHA-256, encoded in base64 using format_digest(). The prefix is currently either tar.gz or tar.bz2.

Git commits:

Identified by their (SHA-1) commits prefixed with git:.

Individual files or directories (“hit-pack”):

A tarball hash is not deterministic from the file contents alone (there’s metadata, compression, etc.). In order to hash build scripts etc. with hashes based on the contents alone, we use a custom “archive format” as the basis of the hash stream. The format starts with the 8-byte magic string “HDSTPCK1”, followed by each file sorted by their filename (potentially containing “/”). Each file is stored as

little-endian uint32_t	length of filename
little-endian uint32_t	length of contents
—	filename (no terminating null)
—	contents

This stream is then encoded like archives (SHA-256 in base-64), and prefixed with files: to get the key.

Module reference¶

class hashdist.core.source_cache.ProgressSpinner¶

Replacement for ProgressBar when we don’t know the file length.

Methods

class hashdist.core.source_cache.SourceCache(cache_path, logger, mirrors=(), create_dirs=False)¶

Methods

static create_from_config(config, logger, create_dirs=False)¶

Creates a SourceCache from the settings in the configuration

fetch(*args, **kwargs)¶

Fetch sources whose key is known.

This is the method to use in automated settings. If the sources globally identified by key are already present in the cache, the method returns immediately, otherwise it attempts to download the sources from url. How to interpret the URL is determined by the prefix of key.

Parameters:

url : str or None Location to download sources from. Exact meaning depends on prefix of key. If None is passed, an exception is raised if the source object is not present. key : str Globally unique key for the source object. repo_name : str or None A unique ID for the source code repo; required for git and ignored otherwise. This must be present because a git “project” is distributed and cannot be deduced from URL (and pulling everything into the same repo was way too slow). Hopefully this can be mended in the future.

fetch_archive(url, type=None)¶

Fetches a tarball without knowing the key up-front.

In automated settings, fetch() should be used instead.

Parameters:	url : str Where to download archive from. Local files can be specified by prepending "file:" to the path. type : str (optional) Type of archive, such as "tar.gz", "tar.gz2". For use when this cannot be determined from the suffix of the url.

fetch_git(repository, rev, repo_name)¶

Fetches source code from git repository

With this method one does not need to know a specific commit, but can use a generic git rev such as master or revs/heads/master. In automated settings or if the commit hash is known exactly, fetch() should be used instead.

Parameters:	repository : str The repository URL (forwarded to git) rev : str The rev to download (forwarded to git) repo_name : str A unique name to use for the repository, e.g., numpy. This is currently required because git doesn’t seem to allow getting a unique ID for a remote repo; and cloning all repos into the same git repo has scalability issues.
Returns:	key : str The globally unique key; this is the git commit SHA-1 hash prepended by git:.

put(files)¶

Put in-memory contents into the source cache.

Parameters:	files : dict or list of (filename, contents) The contents of the archive. filename may contain forward slashes / as path separators. contents is a pure bytes objects which will be dumped directly to stream.
Returns:	key : str The resulting key, it has the files: prefix.

unpack(key, target_path)¶

Unpacks the sources identified by key to target_path

The sources are verified against their secure hash to guard against corruption/security problems. CorruptSourceCacheError will be raised in this case. In normal circumstances this should never happen.

The archive will be loaded into memory, checked against the hash, and then extracted from the memory copy, so that attacks through tampering with on-disk archives should not be possible.

Parameters:	key : str The source item key/secure hash target_path : str Path to extract in

class hashdist.core.source_cache.TarSubprocessHandler(logger)¶

Call external tar

This handler should only be used as fallback, it lacks some features and/or depends on the vagueries of the host tar.

Methods

hashdist.core.source_cache.hit_pack(files, stream=None)¶

Packs the given files in the “hit-pack” format documented above, and returns the resulting key. This is useful to hash a set of files solely by their contents, not metadata, except the filename.

Parameters:	files : list of (filename, contents) The contents of the archive. filename may contain forward slashes / as path separators. contents is a pure bytes objects which will be dumped directly to stream. stream : file-like (optional) Result of the packing, or None if one only wishes to know the hash.
Returns:	The key of the resulting pack : (e.g., ``files:cmRX4RyxU63D9Ciq8ZAfxWGjdMMOXn2mdCwHQqM4Zjw``). :

hashdist.core.source_cache.hit_unpack(stream, key)¶

Unpacks the files in the “hit-pack” format documented above, verifies that it matches the given key, and returns the contents (in memory).

Parameters:	stream : file-like Stream to read the pack from key : str Result from hit_pack().
Returns:	list of (filename, contents) :

hashdist.core.source_cache.scatter_files(files, target_dir)¶

Given a list of filenames and their contents, write them to the file system.

Will not overwrite files (raises an OSError(errno.EEXIST)).

This is typically used together with hit_unpack().

Parameters:	files : list of (filename, contents) target_dir : str Filesystem location to emit the files to

Principles¶

The build store is the very core of HashDist: Producing build artifacts identified by hash-IDs. It’s important to have a clear picture of just what the build store is responsible for and not.

Nix takes a pure approach where an artifact hash is guaranteed to identify the resulting binaries (up to anything inherently random in the build process, like garbage left by compilers). In contrast, HashDist takes a much more lenient approach where the strictness is configurable. The primary goal of HashDist is to make life simpler by reliably triggering rebuilds when software components are updated, not total control of the environment (in which case Nix is likely the better option).

The only concern of the build store is managing the result of a build. So the declared dependencies in the build-spec are not the same as “package dependencies” found in higher-level distributions; for instance, if a pure Python package has a NumPy dependency, this should not be declared in the build-spec because NumPy is not needed during the build; indeed, the installation can happen in parallel.

Artifact IDs¶

A HashDist artifact ID has the form name/hash, e.g., zlib/4niostz3iktlg67najtxuwwgss5vl6k4.

For the artifact paths on disk, a shortened form (4-char hash) is used to make things more friendly to the human user. If there is a collision, the length is simply increased for the one that comes later. Thus, the example above could be stored on disk as ~/.hit/opt/zlib/4nio, or ~/.hit/opt/zlib/1.2.7/4nios in the (rather unlikely) case of a collision. There is a symlink from the full ID to the shortened form. See also Discussion below.

Build specifications and inferring artifact IDs¶

The fundamental object of the build store is the JSON build specification. If you know the build spec, you know the artifact ID, since the latter is the hash of the former. The key is that both dependencies and sources are specified in terms of their hashes.

An example build spec:

{
    "name" : "<name of piece of software>",
    "description": "<what makes this build special>",
    "build": {
        "import" : [
             {"ref": "bash", "id": "virtual:bash"},
             {"ref": "make", "id": "virtual:gnu-make/3+"},
             {"ref": "zlib", "id": "zlib/1.2.7/fXHu+8dcqmREfXaz+ixMkh2LQbvIKlHf+rtl5HEfgmU"},
             {"ref": "unix", "id": "virtual:unix"},
             {"ref": "gcc", "id": "gcc/host-4.6.3/q0VSL7JmzH1P17meqITYc4kMbnIjIexrWPdlAlqPn3s", "before": ["virtual:unix"]},
         ],
         "commands" : [
             {"cmd": ["bash", "build.sh"]}
         ],
     },
     "sources" : [
         {"key": "git:c5ccca92c5f136833ad85614feb2aa4f5bd8b7c3"},
         {"key": "tar.bz2:RB1JbykVljxdvL07mN60y9V9BVCruWRky2FpK2QCCow", "target": "sources"},
         {"key": "files:5fcANXHsmjPpukSffBZF913JEnMwzcCoysn-RZEX7cM"}
     ],
}

name:

Should match [a-zA-Z0-9-_+]+.

version:

Should match [a-zA-Z0-9-_+]*.

build:

A job to run to perform the build. See hashdist.core.run_job for the documentation of this sub-document.

sources:

Sources are unpacked; documentation for now in ‘hit unpack-sources’

The build environment¶

See hashdist.core.execute_job for information about how the build job is executed. In addition, the following environment variables are set:

BUILD:

Set to the build directory. This is also the starting cwd of each build command. This directory may be removed after the build.

ARTIFACT:

The location of the final artifact. Usually this is the “install location” and should, e.g., be passed as the --prefix to ./configure-style scripts.

The build specification is available under $BUILD/build.json, and stdout and stderr are redirected to $BUILD/_hashdist/build.log. These two files will also be present in $ARTIFACT after the build.

Build artifact storage format¶

The presence of the ‘id’ file signals that the build is complete, and contains the full 256-bit hash.

More TODO.

Reference¶

class hashdist.core.build_store.BuildSpec(build_spec)¶

Wraps the document corresponding to a build.json

The document is wrapped in order to a) signal that is has been canonicalized, b) make the artifact id available under the artifact_id attribute.

class hashdist.core.build_store.BuildStore(temp_build_dir, artifact_root, gc_roots_dir, logger, create_dirs=False)¶

Manages the directory of build artifacts; this is usually the entry point for kicking off builds as well.

Parameters:

temp_build_dir : str Directory to use for temporary builds (these may be removed or linger depending on keep_build passed to ensure_present()). artifact_root : str Root of artifacts, this will be prepended to artifact_path_pattern with os.path.join. While this could be part of artifact_path_pattern, the meaning is that garbage collection will never remove contents outside of this directory. gc_roots_dir : str Directory of symlinks to symlinks to artifacts. Artifacts reached through these will not be collected in garbage collection. logger : Logger

Methods

static create_from_config(config, logger, **kw)¶

Creates a SourceCache from the settings in the configuration

create_symlink_to_artifact(artifact_id, symlink_target)¶

Creates a symlink to an artifact (usually a ‘profile’)

The symlink can be placed anywhere the users wants to access it. In addition to the symlink being created, it is listed in gc_roots.

The symlink will be created atomically, any target file/symlink will be overwritten.

delete(artifact_id)¶

Deletes an artifact ID from the store. This is simply an rmtree, i.e., it is (at least currently) possible to delete an aborted build, a build in progress etc., as long as it is present in the right path.

This is the backend of the hit purge command.

Returns the path that was removed, or None if no path was present.

ensure_present(build_spec, config, extra_env=None, virtuals=None, keep_build='never', debug=False)¶

Builds an artifact (if it is not already present).

extra_env: dict (optional)

Extra environment variables to pass to the build environment. These are NOT hashed!

gc()¶

Run garbage collection, removing any unneeded artifacts.

For now, this doesn’t care about virtual dependencies. They’re not used at the moment of writing this; it would have to be revisited in the future.

make_artifact_dir(build_spec)¶

Makes a directory to put the result of the artifact build in. This does not register the artifact in the db (which should be done after the artifact is complete).

make_build_dir(build_spec)¶

Creates a temporary build directory

Just to get a nicer name than mkdtemp would. The caller is responsible for removal.

resolve(artifact_id)¶

Given an artifact_id, resolve the short path for it, or return None if the artifact isn’t built.

hashdist.core.build_store.assert_safe_name(x)¶

Raises a ValueError if x does not match [a-zA-Z0-9-_+]+.

Returns x

hashdist.core.build_store.canonicalize_build_spec(spec)¶

Puts the build spec on a canonical form + basic validation

See module documentation for information on the build specification.

Parameters:	spec : json-like The build specification
Returns:	canonical_spec : json-like Canonicalized and verified build spec

hashdist.core.build_store.shorten_artifact_id(artifact_id, length=12)¶

Shortens the hash part of the artifact_id to the desired length

hashdist.core.build_store.strip_comments(spec)¶

Strips a build spec (which should be in canonical format) of comments that should not affect hash

hashdist.core.build_store.unpack_sources(logger, source_cache, doc, target_dir)¶

Executes source unpacking from ‘sources’ section in build.json

Job specification¶

The job spec is a document that contains what’s needed to set up a controlled environment and run the commands. The idea is to be able to reproduce a job run, and hash the job spec. Example:

{
    "import" : [
        {"ref": "BASH", "id": "virtual:bash"},
        {"ref": "MAKE", "id": "virtual:gnu-make/3+"},
        {"ref": "ZLIB", "id": "zlib/2d4kh7hw4uvml67q7npltyaau5xmn4pc"},
        {"ref": "UNIX", "id": "virtual:unix"},
        {"ref": "GCC", "id": "gcc/jonykztnjeqm7bxurpjuttsprphbooqt"}
     ],
     "commands" : [
         {"chdir": "src"},
         {"prepend_path": "FOOPATH", "value": "$ARTIFACT/bin"},
         {"set": "INCLUDE_FROB", "value": "0"},
         {"cmd": ["pkg-config", "--cflags", "foo"], "to_var": "CFLAGS"},
         {"cmd": ["./configure", "--prefix=$ARTIFACT", "--foo-setting=$FOO"]}
         {"cmd": ["bash", "$in0"],
          "inputs": [
              {"text": [
                  "["$RUN_FOO" != "" ] && ./foo"
                  "make",
                  "make install"
              ]}
         }
     ],
}

Job spec root node¶

The root node is also a command node, as described below, but has two extra allowed keys:

import:

The artifacts needed in the environment for the run. After the job has run they have no effect (i.e., they do not affect garbage collection or run-time dependencies of a build, for instance). The list is ordered and earlier entries are imported before latter ones.

• id: The artifact ID. If the value is prepended with "virtual:", the ID is a virtual ID, used so that the real one does not contribute to the hash. See section on virtual imports below.
• ref: A name to use to inject information of this dependency into the environment. Above, $ZLIB_DIR will be the absolute path to the zlib artifact, and $ZLIB_ID will be the full artifact ID. This can be set to None in order to not set any environment variables for the artifact.

When executing, the environment is set up as follows:

• Environment is cleared (os.environ has no effect)
• The initial environment provided by caller (e.g., BuildStore provides $ARTIFACT and $BUILD) is loaded
• The import section is processed
• Commands executed (which may modify env)

Command node¶

The command nodes is essentially a script language, but lacks any form of control flow. The purpose is to control the environment, and then quickly dispatch to a script in a real programming language.

Also, the overall flow of commands to set up the build environment are typically generated by a pipeline from a package definition, and generating a text script in a pipeline is no fun.

See example above for basic script structure. Rules:

• Every item in the job is either a cmd or a commands or a hit, i.e. those keys are mutually exclusive and defines the node type.

• commands: Push a new environment and current directory to stack, execute sub-commands, and pop the stack.

• cmd: The list is passed straight to subprocess.Popen() as is (after variable substitution). I.e., no quoting, no globbing.

• hit: executes the hit tool in-process. It acts like cmd otherwise, e.g., to_var works.

• chdir: Change current directory, relative to current one (same as modifying PWD environment variable)

• set, prepend/append_path, prepend/append_flag: Change environment variables, inserting the value specified by the value key, using variable substitution as explained below. set simply overwrites variable, while the others modify path/flag-style variables, using the os.path.patsep for prepend/append_path and a space for prepend/append_flag. NOTE: One can use nohash_value instead of value to avoid the value to enter the hash of a build specification.

• files specifies files that are dumped to temporary files and made available as $in0, $in1 and so on. Each file has the form {typestr: value}, where typestr means:

  • text: value should be a list of strings which are joined by newlines
  • string: value is dumped verbatim to file
  • json: value is any JSON document, which is serialized to the file

• stdout and stderr will be logged, except if to_var or append_to_file is present in which case the stdout is capture to an environment variable or redirected in append-mode to file, respectively. (In the former case, the resulting string undergoes strip(), and is then available for the following commands within the same scope.)

• Variable substitution is performed the following places: The cmd, value of set etc., chdir argument, stdout_to_file. The syntax is $CFLAGS and ${CFLAGS}. \$ is an escape for $, \ is an escape for \, other escapes not currently supported and \ will carry through unmodified.

For the hit tool, in addition to what is listed in hit --help, the following special command is available for interacting with the job runner:

• hit logpipe HEADING LEVEL: Creates a new Unix FIFO and prints its name to standard output (it will be removed once the job terminates). The job runner will poll the pipe and print anything written to it nicely formatted to the log with the given heading and log level (the latter is one of DEBUG, INFO, WARNING, ERROR).

Virtual imports¶

Some times it is not desirable for some imports to become part of the hash. For instance, if the cp tool is used in the job, one is normally ready to trust that the result wouldn’t have been different if a newer version of the cp tool was used instead.

Virtual imports, such as virtual:unix in the example above, are used so that the hash depends on a user-defined string rather than the artifact contents. If a bug in cp is indeed discovered, one can change the user-defined string (e.g, virtual:unix/r2) in order to change the hash of the job desc.

Reference¶

class hashdist.core.run_job.CommandTreeExecution(logger, temp_dir=None, debug=False, debug_shell='/bin/bash')¶

Class for maintaining state (in particular logging pipes) while executing script. Note that the environment is passed around as parameters instead.

Executing run() multiple times amounts to executing different variable scopes (but with same logging pipes set up).

Parameters:	logger : Logger rpc_dir : str A temporary directory on a local filesystem. Currently used for creating pipes with the “hit logpipe” command.

Methods

close()¶

Removes log FIFOs; should always be called when one is done

dump_inputs(inputs, node_pos)¶

Handles the ‘inputs’ attribute of a node by dumping to temporary files.

Returns:	A dict with environment variables that can be used to update `env`, : containing ``$in0``, ... :

logged_check_call(args, env, stdout_to)¶

Similar to subprocess.check_call, but multiplexes input from stderr, stdout and any number of log FIFO pipes available to the called process into a single Logger instance. Optionally captures stdout instead of logging it.

run_hit(args, env, stdout_to=None)¶

Run hit in the same process.

But do not emit INFO-messages from sub-command unless level is DEBUG.

run_node(node, env, node_pos)¶

Executes a script node and its children

Parameters:	node : dict A command node env : dict The environment (will be modified). The PWD variable tracks working directory and should always be set on input. node_pos : tuple Tuple of the “path” to this command node; e.g., (0, 1) for second command in first group.

hashdist.core.run_job.canonicalize_job_spec(job_spec)¶

Returns a copy of job_spec with default values filled in.

Also performs a tiny bit of validation.

hashdist.core.run_job.handle_imports(logger, build_store, artifact_dir, virtuals, job_spec)¶

Sets up environment variables for a job. This includes $MYIMPORT_DIR, $MYIMPORT_ID, $ARTIFACT, $HDIST_IMPORT, $HDIST_IMPORT_PATHS.

Returns:	env : dict Environment containing HDIST_IMPORT{,_PATHS} and variables for each import. script : list Instructions to execute; imports first and the job_spec commands afterwards.

hashdist.core.run_job.run_job(logger, build_store, job_spec, override_env, artifact_dir, virtuals, cwd, config, temp_dir=None, debug=False)¶

Runs a job in a controlled environment, according to rules documented above.

Parameters:

logger : Logger build_store : BuildStore BuildStore to find referenced artifacts in. job_spec : document See above override_env : dict Extra environment variables not present in job_spec, these will be added last and overwrite existing ones. artifact_dir : str The value $ARTIFACT should take after running the imports virtuals : dict Maps virtual artifact to real artifact IDs. cwd : str The starting working directory of the job. Currently this cannot be changed (though a cd command may be implemented in the future if necesarry) config : dict Configuration from hashdist.core.config. This will be serialied and put into the HDIST_CONFIG environment variable for use by hit. temp_dir : str (optional) A temporary directory for use by the job runner. Files will be left in the dir after execution. debug : bool Whether to run in debug mode.

Returns:

out_env: dict : The environment after the last command that was run (regardless of scoping/nesting). If the job spec is empty (no commands), this will be an empty dict.

hashdist.core.run_job.substitute(x, env)¶

Substitute environment variable into a string following the rules documented above.

Raises KeyError if an unreferenced variable is not present in env ($$ always raises KeyError)