BigchainDB Server Documentation¶

Introduction¶

This is the documentation for BigchainDB Server, the BigchainDB software that one runs on servers (but not on clients).

If you want to use BigchainDB Server, then you should first understand what BigchainDB is, plus some of the specialized BigchaindB terminology. You can read about that in the overall BigchainDB project documentation.

Note that there are a few kinds of nodes:

A dev/test node is a node created by a developer working on BigchainDB Server, e.g. for testing new or changed code. A dev/test node is typically run on the developer’s local machine.
A bare-bones node is a node deployed in the cloud, either as part of a testing network or as a starting point before upgrading the node to be production-ready.
A production node is a node that is part of a consortium’s BigchainDB network. A production node has the most components and requirements.

Setup Instructions for Various Cases¶

Can I Help?¶

Yes! BigchainDB is an open-source project; we welcome contributions of all kinds. If you want to request a feature, file a bug report, make a pull request, or help in some other way, please see the docs about Contributing to BigchainDB.

Quickstart¶

Try BigchainDB¶

Create a BigchainDB transaction and post it to a BigchainDB network in 20 seconds or less:

Try BigchainDB Now

Develop an App¶

To develop an app that talks to a BigchainDB network, you’ll want a test network to test it against. You have a few options:

The BigchainDB Test Network (or “Testnet”) is a free-to-use, publicly-available BigchainDB network that you can test against. You can find out more about the BigchainDB Testnet on the BigchainDB “Get Started” page.
You could also run a BigchainDB node on you local machine. One way is to use the Simple Deployment Template, with a one-node “network.” Another way is to use one of the deployment methods listed in the the docs about contributing to BigchainDB.

Regardless of which BigchainDB network you use, you’ll probably use one of the BigchainDB drivers or tools.

Help Develop BigchainDB Server¶

To help develop BigchainDB Server (the core software in each BigchainDB node), see the docs about contributing to BigchainDB.

Production Nodes¶

Production Node Requirements¶

This page is about the requirements of BigchainDB Server. You can find the requirements of MongoDB, Tendermint and other production node components in the documentation for that software.

OS Requirements¶

BigchainDB Server requires Python 3.5+ and Python 3.5+ will run on any modern OS, but we recommend using an LTS version of Ubuntu Server or a similarly server-grade Linux distribution.

Don’t use macOS (formerly OS X, formerly Mac OS X), because it’s not a server-grade operating system. Also, BigchaindB Server uses the Python multiprocessing package and some functionality in the multiprocessing package doesn’t work on Mac OS X.

General Considerations¶

BigchainDB Server runs many concurrent processes, so more RAM and more CPU cores is better.

As mentioned on the page about production node components, every machine running BigchainDB Server should be running an NTP daemon.

Production Node Assumptions¶

Be sure you know the key BigchainDB terminology:

We make some assumptions about production nodes:

Each production node is set up and managed by an experienced professional system administrator or a team of them.
Each production node in a network is managed by a different person or team.

Production Node Components¶

A production BigchainDB node must include:

BigchainDB Server
MongoDB Server 3.4+ (mongod)
Tendermint
Storage for MongoDB and Tendermint

It could also include several other components, including:

NGINX or similar, to provide authentication, rate limiting, etc.
An NTP daemon running on all machines running BigchainDB Server or mongod, and possibly other machines
Probably not MongoDB Automation Agent. It’s for automating the deployment of an entire MongoDB cluster.
MongoDB Monitoring Agent
MongoDB Backup Agent
Log aggregation software
Monitoring software
Maybe more

The relationship between the main components is illustrated below.

_images/Node-components.png Components of a production node

Production Node Security & Privacy¶

Here are some references about how to secure an Ubuntu 18.04 server:

Also, here are some recommendations a node operator can follow to enhance the privacy of the data coming to, stored on, and leaving their node:

Ensure that all data stored on a node is encrypted at rest, e.g. using full disk encryption. This can be provided as a service by the operating system, transparently to BigchainDB, MongoDB and Tendermint.
Ensure that all data is encrypted in transit, i.e. enforce using HTTPS for the HTTP API and the Websocket API. This can be done using NGINX or similar, as we do with the BigchainDB Testnet.

Using a Reverse Proxy¶

You may want to:

rate limit inbound HTTP requests,
authenticate/authorize inbound HTTP requests,
block requests with an HTTP request body that’s too large, or
enable HTTPS (TLS) between your users and your node.

While we could have built all that into BigchainDB Server, we didn’t, because you can do all that (and more) using a reverse proxy such as NGINX or HAProxy. (You would put it in front of your BigchainDB Server, so that all inbound HTTP requests would arrive at the reverse proxy before maybe being proxied onwards to your BigchainDB Server.) For detailed instructions, see the documentation for your reverse proxy.

Below, we note how a reverse proxy can be used to do some BigchainDB-specific things.

You may also be interested in our NGINX configuration file template (open source, on GitHub).

Enforcing a Max Transaction Size¶

The BigchainDB HTTP API has several endpoints, but only one of them, the POST /transactions endpoint, expects a non-empty HTTP request body: the transaction being submitted by the user.

If you want to enforce a maximum-allowed transaction size (discarding any that are larger), then you can do so by configuring a maximum request body size in your reverse proxy. For example, NGINX has the client_max_body_size configuration setting. You could set it to 15 kB with the following line in your NGINX config file:

client_max_body_size 15k;

For more information, see the NGINX docs about client_max_body_size.

Note: By enforcing a maximum transaction size, you indirectly enforce a maximum crypto-conditions complexity.

BigchainDB Networks¶

A BigchainDB network is a set of connected BigchainDB nodes, managed by a BigchainDB consortium (i.e. an organization). Those terms are defined in the BigchainDB Terminology page.

Consortium Structure & Governance¶

The consortium might be a company, a foundation, a cooperative, or some other form of organization. It must make many decisions, e.g. How will new members be added? Who can read the stored data? What kind of data will be stored? A governance process is required to make those decisions, and therefore one of the first steps for any new consortium is to specify its governance process (if one doesn’t already exist). This documentation doesn’t explain how to create a consortium, nor does it outline the possible governance processes.

It’s worth noting that the decentralization of a BigchainDB network depends, to some extent, on the decentralization of the associated consortium. See the pages about decentralization and node diversity.

DNS Records and SSL Certificates¶

We now describe how we set up the external (public-facing) DNS records for a BigchainDB network. Your consortium may opt to do it differently. There were several goals:

Allow external users/clients to connect directly to any BigchainDB node in the network (over the internet), if they want.
Each BigchainDB node operator should get an SSL certificate for their BigchainDB node, so that their BigchainDB node can serve the BigchainDB HTTP API via HTTPS. (The same certificate might also be used to serve the WebSocket API.)
There should be no sharing of SSL certificates among BigchainDB node operators.
Optional: Allow clients to connect to a “random” BigchainDB node in the network at one particular domain (or subdomain).

Node Operator Responsibilities¶

Register a domain (or use one that you already have) for your BigchainDB node. You can use a subdomain if you like. For example, you might opt to use abc-org73.net, api.dynabob8.io or figmentdb3.ninja.
Get an SSL certificate for your domain or subdomain, and properly install it in your node (e.g. in your NGINX instance).
Create a DNS A Record mapping your domain or subdomain to the public IP address of your node (i.e. the one that serves the BigchainDB HTTP API).

Consortium Responsibilities¶

Optional: The consortium managing the BigchainDB network could register a domain name and set up CNAME records mapping that domain name (or one of its subdomains) to each of the nodes in the network. For example, if the consortium registered bdbnetwork.io, they could set up CNAME records like the following:

CNAME record mapping api.bdbnetwork.io to abc-org73.net
CNAME record mapping api.bdbnetwork.io to api.dynabob8.io
CNAME record mapping api.bdbnetwork.io to figmentdb3.ninja

Simple Deployment Template¶

This section describes one way to deploy a BigchainDB network (i.e. a set of connected BigchainDB nodes). You can modify this simple deployment template as you see fit. It’s “simple” in the sense that each BigchainDB node is installed and run on a single virtual machine (or real machine). We also have a Kubernetes Deployment Template (not simple).

Note 1: These instructions will also work for a “network” with only one node. If you want your network to be able to handle the failure or misbehavior of one node, then your network must have at least four nodes. Nodes can be added or removed from a network after is it up and running.

Note 2: If you want to set up a node or network so that you can contribute to developing and testing the BigchainDB code, then see the docs about contributing to BigchainDB .

Deploy a Machine for Your BigchainDB Node¶

The first step is to deploy a machine for your BigchainDB node. It might be a virtual machine (VM) or a real machine. If you follow this simple deployment template, all your node’s software will run on that one machine.

We don’t make any assumptions about where you run the machine. It might be in Azure, AWS, your data center or a Raspberry Pi.

IP Addresses¶

The following instructions assume all the nodes in the network (including yours) have public IP addresses. (A BigchainDB network can be run inside a private network, using private IP addresses, but we don’t cover that here.)

Operating System¶

Use Ubuntu 18.04 or Ubuntu Server 18.04 as the operating system.

Similar instructions will work on other versions of Ubuntu, and other recent Debian-like Linux distros, but you may have to change the names of the packages, or install more packages.

Network Security Group¶

If your machine is in AWS or Azure, for example, and you want users to connect to BigchainDB via HTTPS, then you should configure its network security group to allow all incoming and outgoing traffic for:

TCP on port 22 (SSH)
TCP on port 80 (HTTP)
TCP on port 443 (HTTPS)
Any protocol on port 26656 (Tendermint P2P)

If you don’t care about HTTPS, then forget about port 443, and replace port 80 with port 9984 (the default BigchainDB HTTP port).

Update Your System¶

SSH into your machine and update all its OS-level packages:

sudo apt update
sudo apt full-upgrade

Node Security¶

If you’re going to use your node in production, then you should take additional steps to secure it. We don’t cover that here; there are many books and websites about securing Linux machines.

DNS Setup¶

Register a domain name for your BigchainDB node, such as example.com
Pick a subdomain of that domain for your BigchainDB node, such as bnode.example.com
Create a DNS “A Record” pointing your chosen subdomain (such as bnode.example.com) at your machine’s IP address.

Set Up NGINX¶

If you don’t want HTTPS (for communications between the external world and your node), then you can skip all the NGINX steps on this page.

Note: This simple deployment template uses NGINX for more than just HTTPS. For example, it also does basic rate limiting.

Install NGINX¶

SSH into your machine and install NGINX:

sudo apt update
sudo apt install nginx

Configure & Reload NGINX¶

Get an SSL certificate for your node’s subdomain (such as bnode.example.com).

Copy the SSL private key into /etc/nginx/ssl/cert.key
Create a “PEM file” (text file) by concatenating your SSL certificate with all intermediate certificates (in that order, with the intermediate certs last).
Copy that PEM file into /etc/nginx/ssl/cert.pem
In the bigchaindb/bigchaindb repository on GitHub, find the file nginx/nginx.conf and copy its contents to /etc/nginx/nginx.conf on your machine (i.e. replace the existing file there).
Edit that file (/etc/nginx/nginx.conf): replace the two instances of the string example.testnet2.com with your chosen subdomain (such as bnode.example.com).
Reload NGINX by doing:
```
sudo service nginx reload
```

Set Up BigchainDB, MongoDB and Tendermint¶

We now install and configure software that must run in every BigchainDB node: BigchainDB Server, MongoDB and Tendermint.

Install BigchainDB Server¶

BigchainDB Server requires Python 3.6+, so make sure your system has it.

Install the required OS-level packages:

# For Ubuntu 18.04:
sudo apt install -y python3-pip libssl-dev
# Ubuntu 16.04, and other Linux distros, may require other packages or more packages

BigchainDB Server requires gevent, and to install gevent, you must use pip 19 or later (as of 2019, because gevent now uses manylinux2010 wheels). Upgrade pip to the latest version:

sudo pip3 install -U pip

Now install the latest version of BigchainDB Server. You can find the latest version by going to the BigchainDB project release history page on PyPI. For example, to install version 2.0.0, you would do:

# Change 2.0.0 to the latest version as explained above:
sudo pip3 install bigchaindb==2.0.0

Check that you installed the correct version of BigchainDB Server using bigchaindb --version.

Configure BigchainDB Server¶

To configure BigchainDB Server, run:

bigchaindb configure

The first question is API Server bind? (default `localhost:9984`).

If you’re using NGINX (e.g. if you want HTTPS), then accept the default value (localhost:9984).
If you’re not using NGINX, then enter the value 0.0.0.0:9984

You can accept the default value for all other BigchainDB config settings.

If you’re using NGINX, then you should edit your BigchainDB config file (in $HOME/.bigchaindb by default) and set the following values under "wsserver":

"advertised_scheme": "wss",
"advertised_host": "bnode.example.com",
"advertised_port": 443

where bnode.example.com should be replaced by your node’s actual subdomain.

Install (and Start) MongoDB¶

Install a recent version of MongoDB. BigchainDB Server requires version 3.4 or newer.

sudo apt install mongodb

If you install MongoDB using the above command (which installs the mongodb package), it also configures MongoDB, starts MongoDB (in the background), and installs a MongoDB startup script (so that MongoDB will be started automatically when the machine is restarted).

Note: The mongodb package is not the official MongoDB package from MongoDB the company. If you want to install the official MongoDB package, please see the MongoDB documentation. Note that installing the official package doesn’t also start MongoDB.

Install Tendermint¶

The version of BigchainDB Server described in these docs only works well with Tendermint 0.31.5 (not a higher version number). Install that:

sudo apt install -y unzip
wget https://github.com/tendermint/tendermint/releases/download/v0.31.5/tendermint_v0.31.5_linux_amd64.zip
unzip tendermint_v0.31.5_linux_amd64.zip
rm tendermint_v0.31.5_linux_amd64.zip
sudo mv tendermint /usr/local/bin

Start Configuring Tendermint¶

You won’t be able to finish configuring Tendermint until you have some information from the other nodes in the network, but you can start by doing:

tendermint init

How to Set Up a BigchainDB Network¶

Until now, everything could be done by a node operator, by themselves. Now the node operators, also called Members, must share some information with each other, so they can form a network.

There is one special Member who helps coordinate everyone: the Coordinator.

Member: Connect to the Other Members¶

At this point the Member should have received the genesis.json file.

The Member must copy the genesis.json file into their local $HOME/.tendermint/config directory. Every Member now shares the same chain_id and genesis_time (used to identify the Network), and the same list of validators.

Each Member must edit their $HOME/.tendermint/config/config.toml file and make the following changes:

moniker = "Name of our node"
create_empty_blocks = false
log_level = "main:info,state:info,*:error"

persistent_peers = "<Member 1 node id>@<Member 1 hostname>:26656,\
<Member 2 node id>@<Member 2 hostname>:26656,\
<Member N node id>@<Member N hostname>:26656,"

send_rate = 102400000
recv_rate = 102400000

recheck = false

Note: The list of persistent_peers doesn’t have to include all nodes in the network.

Member: Start MongoDB¶

If you installed MongoDB using sudo apt install mongodb, then MongoDB should already be running in the background. You can check using systemctl status mongodb.

If MongoDB isn’t running, then you can start it using the command mongod, but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use mongod --fork --logpath /var/log/mongodb.log. (You might have to create the /var/log directory if it doesn’t already exist.)

If you installed MongoDB using sudo apt install mongodb, then a MongoDB startup script should already be installed (so MongoDB will start automatically when the machine is restarted). Otherwise, you should install a startup script for MongoDB.

Member: Start BigchainDB and Tendermint Using Monit¶

This section describes how to manage the BigchainDB and Tendermint processes using Monit, a small open-source utility for managing and monitoring Unix processes. BigchainDB and Tendermint are managed together, because if BigchainDB is stopped (or crashes) and is restarted, Tendermint won’t try reconnecting to it. (That’s not a bug. It’s just how Tendermint works.)

Install Monit:

sudo apt install monit

If you installed the bigchaindb Python package as above, you should have the bigchaindb-monit-config script in your PATH now. Run the script to build a configuration file for Monit:

bigchaindb-monit-config

Run Monit as a daemon, instructing it to wake up every second to check on processes:

monit -d 1

Monit will run the BigchainDB and Tendermint processes and restart them when they crash. If the root bigchaindb_ process crashes, Monit will also restart the Tendermint process.

You can check the status by running monit status or monit summary.

By default, it will collect program logs into the ~/.bigchaindb-monit/logs folder.

To learn more about Monit, use monit -h (help) or read the Monit documentation.

Check bigchaindb-monit-config -h if you want to arrange a different folder for logs or some of the Monit internal artifacts.

If you want to start and manage the BigchainDB and Tendermint processes yourself, then look inside the file bigchaindb/pkg/scripts/bigchaindb-monit-config to see how it starts BigchainDB and Tendermint.

How Others Can Access Your Node¶

If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL https://hostname or http://hostname:9984. That is, anyone can interact with your node using the BigchainDB HTTP API exposed at that address. The most common way to do that is to use one of the BigchainDB Drivers.

Tips¶

Tendermint Tips¶

Configure Tendermint to create no empty blocks.
Store the Tendermint data on a fast drive. You can do that by changing the location of TMHOME to be on the fast drive.

Refreshing Your Node¶

If you want to refresh your node back to a fresh empty state, then your best bet is to terminate it and deploy a new machine, but if that’s not an option, then you can:

drop the bigchain database in MongoDB using bigchaindb drop (but that only works if MongoDB is running)
reset Tendermint using tendermint unsafe_reset_all
delete the directory $HOME/.tendermint

Shutting Down BigchainDB¶

If you want to stop/kill BigchainDB, you can do so by sending SIGINT, SIGQUIT or SIGTERM to the running BigchainDB process(es). Depending on how you started BigchainDB i.e. foreground or background. e.g. you started BigchainDB in the background as mentioned above in the guide:

$ nohup bigchaindb start 2>&1 > bigchaindb.log &

$ # Check the PID of the main BigchainDB process
$ ps -ef | grep bigchaindb
<user>    *<pid> <ppid>   <C> <STIME> <tty>        <time> bigchaindb
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> gunicorn: master [bigchaindb_gunicorn]
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> bigchaindb_ws
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> bigchaindb_ws_to_tendermint
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> bigchaindb_exchange
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
...

$ # Send any of the above mentioned signals to the parent/root process(marked with `*` for clarity)
# Sending SIGINT
$ kill -2 <bigchaindb_parent_pid>

$ # OR

# Sending SIGTERM
$ kill -15 <bigchaindb_parent_pid>

$ # OR

# Sending SIGQUIT
$ kill -3 <bigchaindb_parent_pid>

# If you want to kill all the processes by name yourself
$ pgrep bigchaindb | xargs kill -9

If you started BigchainDB in the foreground, a Ctrl + C or Ctrl + Z would shut down BigchainDB.

Member: Dynamically Add or Remove Validators¶

One member can make a proposal to call an election to add a validator, remove a validator, or change the voting power of a validator. They then share the election/proposal ID with all the other members. Once more than 2/3 of the voting power votes yes, the proposed change comes into effect. The commands to create a new election/proposal, to approve an election/proposal, and to get the current status of an election/proposal can be found in the documentation about the bigchaindb election subcommands.

Logging and Log Rotation¶

See the page in the Appendices about logging and log rotation.

Troubleshooting¶

General Tips¶

Check the BigchainDB, Tendermint and MongoDB logs. For help with that, see the page about Logging and Log Rotation.
Try Googling the error message.

Resolving Tendermint Connectivity Problems¶

To check which nodes your node is connected to (via Tendermint protocols), do:

# if you don't have jq installed, then install it
sudo apt install jq
# then do
curl -s localhost:26657/net_info | jq ".result.peers[].node_info | {id, listen_addr, moniker}"

Note: Tendermint has other endpoints besides /net_info: see the Tendermint RPC docs.

If you’re running your network inside a private network, e.g. with IP addresses of the form 192.168.x.y, then you may have to change the following setting in config.toml:

addr_book_strict = false

Develop & Test BigchainDB Server¶

If you’d like to help develop and test BigchainDB Server, then see the section about that in the docs about contributing to BigchainDB.

Settings & CLI¶

Configuration Settings¶

Every BigchainDB Server configuration setting has two names: a config-file name and an environment variable name. For example, one of the settings has the config-file name database.host and the environment variable name BIGCHAINDB_DATABASE_HOST. Here are some more examples:

database.port ↔ BIGCHAINDB_DATABASE_PORT

database.keyfile_passphrase ↔ BIGCHAINDB_DATABASE_KEYFILE_PASSPHRASE

server.bind ↔ BIGCHAINDB_SERVER_BIND

The value of each setting is determined according to the following rules:

If it’s set by an environment variable, then use that value
Otherwise, if it’s set in a local config file, then use that value
Otherwise, use the default value

The local config file is $HOME/.bigchaindb by default (a file which might not even exist), but you can tell BigchainDB to use a different file by using the -c command-line option, e.g. bigchaindb -c path/to/config_file.json start or using the BIGCHAINDB_CONFIG_PATH environment variable, e.g. BIGHAINDB_CONFIG_PATH=.my_bigchaindb_config bigchaindb start. Note that the -c command line option will always take precedence if both the BIGCHAINDB_CONFIG_PATH and the -c command line option are used.

You can read the current default values in the file bigchaindb/__init__.py. (The link is to the latest version.)

Running bigchaindb -y configure localmongodb will generate a local config file in $HOME/.bigchaindb with all the default values.

database.*¶

The settings with names of the form database.* are for the backend database (currently only MongoDB). They are:

database.backend can only be localmongodb, currently.
database.host is the hostname (FQDN) of the backend database.
database.port is self-explanatory.
database.name is a user-chosen name for the database inside MongoDB, e.g. bigchain.
database.connection_timeout is the maximum number of milliseconds that BigchainDB will wait before giving up on one attempt to connect to the backend database.
database.max_tries is the maximum number of times that BigchainDB will try to establish a connection with the backend database. If 0, then it will try forever.
database.replicaset is the name of the MongoDB replica set. The default value is null because in BigchainDB 2.0+, each BigchainDB node has its own independent MongoDB database and no replica set is necessary. Replica set must already exist if this option is configured, BigchainDB will not create it.
database.ssl must be true or false. It tells BigchainDB Server whether it should connect to MongoDB using TLS/SSL or not. The default value is false.

There are three ways for BigchainDB Server to authenticate itself with MongoDB (or a specific MongoDB database): no authentication, username/password, and x.509 certificate authentication.

No Authentication

If you use all the default BigchainDB configuration settings, then no authentication will be used.

Username/Password Authentication

To use username/password authentication, a MongoDB instance must already be running somewhere (maybe in another machine), it must already have a database for use by BigchainDB (usually named bigchain, which is the default database.name), and that database must already have a “readWrite” user with associated username and password. To create such a user, login to your MongoDB instance as Admin and run the following commands:

use <database.name>
db.createUser({user: "<database.login>", pwd: "<database.password>", roles: [{role: "readWrite", db: "<database.name>"}]})

database.login is the user’s username.
database.password is the user’s password, given in plaintext.
database.ca_cert, database.certfile, database.keyfile, database.crlfile, and database.keyfile_passphrase are not used so they can have their default values.

x.509 Certificate Authentication

To use x.509 certificate authentication, a MongoDB instance must be running somewhere (maybe in another machine), it must already have a database for use by BigchainDB (usually named bigchain, which is the default database.name), and that database must be set up to use x.509 authentication. See the MongoDB docs about how to do that.

database.login is the user’s username.
database.password isn’t used so the default value (null) is fine.
database.ca_cert, database.certfile, database.keyfile and database.crlfile are the paths to the CA, signed certificate, private key and certificate revocation list files respectively.
database.keyfile_passphrase is the private key decryption passphrase, specified in plaintext.

Example using environment variables

export BIGCHAINDB_DATABASE_BACKEND=localmongodb
export BIGCHAINDB_DATABASE_HOST=localhost
export BIGCHAINDB_DATABASE_PORT=27017
export BIGCHAINDB_DATABASE_NAME=database8
export BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT=5000
export BIGCHAINDB_DATABASE_MAX_TRIES=3

Default values

If (no environment variables were set and there’s no local config file), or you used bigchaindb -y configure localmongodb to create a default local config file for a localmongodb backend, then the defaults will be:

"database": {
    "backend": "localmongodb",
    "host": "localhost",
    "port": 27017,
    "name": "bigchain",
    "connection_timeout": 5000,
    "max_tries": 3,
    "replicaset": null,
    "login": null,
    "password": null
    "ssl": false,
    "ca_cert": null,
    "certfile": null,
    "keyfile": null,
    "crlfile": null,
    "keyfile_passphrase": null,
}

server.*¶

server.bind, server.loglevel and server.workers are settings for the Gunicorn HTTP server, which is used to serve the HTTP client-server API.

server.bind is where to bind the Gunicorn HTTP server socket. It’s a string. It can be any valid value for Gunicorn’s bind setting. For example:

If you want to allow IPv4 connections from anyone, on port 9984, use 0.0.0.0:9984
If you want to allow IPv6 connections from anyone, on port 9984, use [::]:9984

In a production setting, we recommend you use Gunicorn behind a reverse proxy server such as NGINX. If Gunicorn and the reverse proxy are running on the same machine, then you can use localhost:9984 (the default value), meaning Gunicorn will talk to the reverse proxy on port 9984. The reverse proxy could then be bound to port 80 (for HTTP) or port 443 (for HTTPS), so that external clients would connect using that port. For example:

[External clients]—(port 443)—[NGINX]—(port 9984)—[Gunicorn / BigchainDB Server]

If Gunicorn and the reverse proxy are running on different machines, then server.bind should be hostname:9984, where hostname is the IP address or FQDN of the reverse proxy.

There’s more information about deploying behind a reverse proxy in the Gunicorn documentation. (They call it a proxy.)

server.loglevel sets the log level of Gunicorn’s Error log outputs. See Gunicorn’s documentation for more information.

server.workers is the number of worker processes for handling requests. If set to None, the value will be (2 × cpu_count + 1). Each worker process has a single thread. The HTTP server will be able to handle server.workers requests simultaneously.

Example using environment variables

export BIGCHAINDB_SERVER_BIND=0.0.0.0:9984
export BIGCHAINDB_SERVER_LOGLEVEL=debug
export BIGCHAINDB_SERVER_WORKERS=5

Example config file snippet

"server": {
    "bind": "0.0.0.0:9984",
    "loglevel": "debug",
    "workers": 5,
}

Default values (from a config file)

"server": {
    "bind": "localhost:9984",
    "loglevel": "info",
    "workers": null,
}

wsserver.*¶

wsserver.scheme, wsserver.host and wsserver.port¶

These settings are for the aiohttp server, which is used to serve the WebSocket Event Stream API. wsserver.scheme should be either "ws" or "wss" (but setting it to "wss" does not enable SSL/TLS). wsserver.host is where to bind the aiohttp server socket and wsserver.port is the corresponding port. If you want to allow connections from anyone, on port 9985, set wsserver.host to 0.0.0.0 and wsserver.port to 9985.

Example using environment variables

export BIGCHAINDB_WSSERVER_SCHEME=ws
export BIGCHAINDB_WSSERVER_HOST=0.0.0.0
export BIGCHAINDB_WSSERVER_PORT=9985

Example config file snippet

"wsserver": {
    "scheme": "wss",
    "host": "0.0.0.0",
    "port": 65000
}

Default values (from a config file)

"wsserver": {
    "scheme": "ws",
    "host": "localhost",
    "port": 9985
}

wsserver.advertised_scheme, wsserver.advertised_host and wsserver.advertised_port¶

These settings are for the advertising the Websocket URL to external clients in the root API endpoint. These configurations might be useful if your deployment is hosted behind a firewall, NAT, etc. where the exposed public IP or domain is different from where BigchainDB is running.

Example using environment variables

export BIGCHAINDB_WSSERVER_ADVERTISED_SCHEME=wss
export BIGCHAINDB_WSSERVER_ADVERTISED_HOST=mybigchaindb.com
export BIGCHAINDB_WSSERVER_ADVERTISED_PORT=443

Example config file snippet

"wsserver": {
    "advertised_scheme": "wss",
    "advertised_host": "mybigchaindb.com",
    "advertised_port": 443
}

Default values (from a config file)

"wsserver": {
    "advertised_scheme": "ws",
    "advertised_host": "localhost",
    "advertised_port": 9985
}

log.*¶

The log.* settings are to configure logging.

Example

{
    "log": {
        "file": "/var/log/bigchaindb.log",
        "error_file": "/var/log/bigchaindb-errors.log",
        "level_console": "info",
        "level_logfile": "info",
        "datefmt_console": "%Y-%m-%d %H:%M:%S",
        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",
        "fmt_console": "%(asctime)s [%(levelname)s] (%(name)s) %(message)s",
        "fmt_logfile": "%(asctime)s [%(levelname)s] (%(name)s) %(message)s",
        "granular_levels": {}
}

Default values

{
    "log": {
        "file": "~/bigchaindb.log",
        "error_file": "~/bigchaindb-errors.log",
        "level_console": "info",
        "level_logfile": "info",
        "datefmt_console": "%Y-%m-%d %H:%M:%S",
        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",
        "fmt_logfile": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "fmt_console": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "granular_levels": {}
}

log.file¶

The full path to the file where logs should be written. The user running bigchaindb must have write access to the specified path.

Log rotation: Log files have a size limit of about 200 MB and will be rotated up to five times. For example, if log.file is set to "~/bigchain.log", then logs would always be written to bigchain.log. Each time the file bigchain.log reaches 200 MB it will be closed and renamed bigchain.log.1. If bigchain.log.1 and bigchain.log.2 already exist they would be renamed bigchain.log.2 and bigchain.log.3. This pattern would be applied up to bigchain.log.5 after which bigchain.log.5 would be overwritten by bigchain.log.4, thus ending the rotation cycle of whatever logs were in bigchain.log.5.

log.error_file¶

Similar to log.file (see above), this is the full path to the file where error logs should be written.

log.level_console¶

The log level used to log to the console. Possible allowed values are the ones defined by Python, but case-insensitive for the sake of convenience:

"critical", "error", "warning", "info", "debug", "notset"

log.level_logfile¶

The log level used to log to the log file. Possible allowed values are the ones defined by Python, but case-insensitive for the sake of convenience:

"critical", "error", "warning", "info", "debug", "notset"

log.datefmt_console¶

The format string for the date/time portion of a message, when logged to the console.

For more information on how to construct the format string please consult the table under Python’s documentation of time.strftime(format[, t]).

log.datefmt_logfile¶

The format string for the date/time portion of a message, when logged to a log file.

For more information on how to construct the format string please consult the table under Python’s documentation of time.strftime(format[, t]).

log.fmt_console¶

A string used to format the log messages when logged to the console.

For more information on possible formatting options please consult Python’s documentation on LogRecord attributes.

log.fmt_logfile¶

A string used to format the log messages when logged to a log file.

For more information on possible formatting options please consult Python’s documentation on LogRecord attributes.

log.granular_levels¶

Log levels for BigchainDB’s modules. This can be useful to control the log level of specific parts of the application. As an example, if you wanted the logging of the core.py module to be more verbose, you would set the configuration shown in the example below.

Example

{
    "log": {
        "granular_levels": {
            "bichaindb.core": "debug"
        }
}

Default value

{}

tendermint.*¶

The settings with names of the form tendermint.* tell BigchainDB Server where it can connect to the node’s Tendermint instance.

tendermint.host is the hostname (FQDN)/IP address of the Tendermint instance.
tendermint.port is self-explanatory.

Example using environment variables

export BIGCHAINDB_TENDERMINT_HOST=tendermint
export BIGCHAINDB_TENDERMINT_PORT=26657

Default values

"tendermint": {
    "host": "localhost",
    "port": 26657
}

Command Line Interface (CLI)¶

The command-line command to interact with BigchainDB Server is bigchaindb.

bigchaindb --help¶

Show help for the bigchaindb command. bigchaindb -h does the same thing.

bigchaindb --version¶

Show the version number. bigchaindb -v does the same thing.

bigchaindb configure¶

Generate a local configuration file (which can be used to set some or all BigchainDB node configuration settings). It will ask you for the values of some configuration settings. If you press Enter for a value, it will use the default value.

At this point, only one database backend is supported: localmongodb.

If you use the -c command-line option, it will generate the file at the specified path:

bigchaindb -c path/to/new_config.json configure localmongodb

If you don’t use the -c command-line option, the file will be written to $HOME/.bigchaindb (the default location where BigchainDB looks for a config file, if one isn’t specified).

If you use the -y command-line option, then there won’t be any interactive prompts: it will use the default values for all the configuration settings.

bigchaindb -y configure localmongodb

bigchaindb show-config¶

Show the values of the BigchainDB node configuration settings.

bigchaindb init¶

Create a backend database (local MongoDB), all database tables/collections, various backend database indexes, and the genesis block.

bigchaindb drop¶

Drop (erase) the backend database (the local MongoDB database used by this node). You will be prompted to make sure. If you want to force-drop the database (i.e. skipping the yes/no prompt), then use bigchaindb -y drop

bigchaindb start¶

Start BigchainDB. It always begins by trying a bigchaindb init first. See the documentation for bigchaindb init. The database initialization step is optional and can be skipped by passing the --no-init flag, i.e. bigchaindb start --no-init.

Options¶

The log level for the console can be set via the option --log-level or its abbreviation -l. Example:

$ bigchaindb --log-level INFO start

The allowed levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL. For an explanation regarding these levels please consult the Logging Levels section of Python’s documentation.

For a more fine-grained control over the logging configuration you can use the configuration file as documented under Configuration Settings.

bigchaindb election¶

Manage elections to govern the BigchainDB network. The specifics of the election process are defined in BEP-18.

Election management is broken into several subcommands. Below is the command line syntax for each of them.

election new¶

Create a new election which proposes a change to the BigchainDB network.

If the command succeeds, it will post an election transaction and output election_id.

The election proposal consists of vote tokens allocated to every current validator proportional to his voting power. Validators spend their votes to approve the election using the election-approve command.

Every election has a type. Currently supported types are upsert-validator and chain-migration. Their transaction operations are VALIDATOR_ELECTION and CHAIN_MIGRATION accordingly. See below for how to create an election of a particular type.

Note that elections can only be proposed and approved by existing validators.

election new upsert-validator¶

Create an election to add, update, or remove a validator.

$ bigchaindb election new upsert-validator <public-key> <power> <node-id> --private-key <path-to-the-private-key>

<public-key> is the public key of the node to be added/updated/removed. The encoding and type of the key have to match those specified in genesis.json in the supported Tendermint version.
<power> is the new power for the validator. To remove the validator, set the power to 0.
<node-id> is the node identifier from Tendermint. A node operator can learn his node identifier by executing tendermint show_node_id.
<path-to-the-private-key> is the path to the private key of the validator who proposes the election. Tendermint places it at .tendermint/config/priv_validator.json.

Example:

$ bigchaindb election new upsert-validator HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY= 1 fb7140f03a4ffad899fabbbf655b97e0321add66 --private-key /home/user/.tendermint/config/priv_validator.json
[SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa

A successful execution of the above command does not imply the validator set has been updated but rather the proposal has been accepted by the network. Once election_id has been generated, the proposer should share it with other validators of the network (e.g. via email) and ask them to approve the proposal.

Note that election proposers do not automatically approve elections by proposing them.

For more details about how validator set changes work, refer to BEP-21.

election new chain-migration¶

Create an election to halt block production, to coordinate on making a Tendermint upgrade with a backwards-incompatible chain.

$ bigchaindb election new chain-migration --private-key <path-to-the-private-key>

<path-to-the-private-key> is the path to the private key of the validator who proposes the election. Tendermint places it at .tendermint/config/priv_validator.json.

Example:

$ bigchaindb election new migration --private-key /home/user/.tendermint/config/priv_validator.json
[SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa

Concluded chain migration elections halt block production at whichever block height they are approved. Afterwards, validators are supposed to upgrade Tendermint, set new chain_id, app_hash, and validators (to learn these values, use the election show command) in genesis.json, make and save a MongoDB dump, and restart the system.

For more details about how chain migrations work, refer to Type 3 scenarios in BEP-42.

election approve¶

Approve an election by voting for it. The command places a VOTE transaction, spending all of the validator’s vote tokens to the election address.

$ bigchaindb election approve <election-id> --private-key <path-to-the-private-key>

election-id is the election identifier the approval is given for.
<path-to-the-private-key> is the path to the private key of the validator who votes for the election. Tendermint places it at .tendermint/config/priv_validator.json.

Example:

$ bigchaindb election approve 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa --private-key /home/user/.tendermint/config/priv_validator.json
[SUCCESS] Your vote has been submitted

Once a proposal has been approved by the sufficient amount of validators (contributing more than 2/3 of the total voting power), the proposed change is applied to the network.

election show¶

Retrieves the information about elections.

$ bigchaindb election show <election-id>

status=<status>

status has three possible values:

ongoing, if the election can be concluded but has not yet collected enough votes,
concluded, if the election has been concluded,
inconclusive, if the validator set changed while the election was in process, rendering it undecidable.

After a chain migration is concluded, the show command also outputs chain_id, app_hash, and validators for genesis.json of the new chain.

bigchaindb tendermint-version¶

Show the Tendermint versions supported by BigchainDB server.

$ bigchaindb tendermint-version
{
    "description": "BigchainDB supports the following Tendermint version(s)",
    "tendermint": [
        "0.22.8"
    ]
}

The HTTP Client-Server API¶

This page assumes you already know an API Root URL for a BigchainDB node or reverse proxy. It should be something like https://example.com:9984 or https://12.34.56.78:9984.

If you set up a BigchainDB node or reverse proxy yourself, and you’re not sure what the API Root URL is, then see the last section of this page for help.

BigchainDB Root URL¶

If you send an HTTP GET request to the BigchainDB Root URL e.g. http://localhost:9984 or https://example.com:9984 (with no /api/v1/ on the end), then you should get an HTTP response with something like the following in the body:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "api": {
    "v1": {
      "assets": "/api/v1/assets/",
      "blocks": "/api/v1/blocks/",
      "docs": "https://docs.bigchaindb.com/projects/server/en/v2.2.1/http-client-server-api.html",
      "metadata": "/api/v1/metadata/",
      "outputs": "/api/v1/outputs/",
      "streams": "ws://localhost:9985/api/v1/streams/valid_transactions",
      "transactions": "/api/v1/transactions/",
      "validators": "/api/v1/validators"
    }
  },
  "docs": "https://docs.bigchaindb.com/projects/server/en/v2.2.1/",
  "software": "BigchainDB",
  "version": "2.2.1"
}

API Root Endpoint¶

If you send an HTTP GET request to the API Root Endpoint e.g. http://localhost:9984/api/v1/ or https://example.com:9984/api/v1/, then you should get an HTTP response that allows you to discover the BigchainDB API endpoints:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "assets": "/assets/",
  "blocks": "/blocks/",
  "docs": "https://docs.bigchaindb.com/projects/server/en/v2.2.1/http-client-server-api.html",
  "metadata": "/metadata/",
  "outputs": "/outputs/",
  "streams": "ws://localhost:9985/api/v1/streams/valid_transactions",
  "transactions": "/transactions/",
  "validators": "/validators"
}

Transactions¶

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/transactions/{transaction_id}¶

Get the transaction with the ID transaction_id.

If a transaction with ID transaction_id has been included in a committed block, then this endpoint returns that transaction, otherwise the response will be 404 Not Found.

Parameters:	transaction_id (hex string) – transaction ID

Example request:

GET /api/v1/transactions/4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "asset": {
    "data": {
      "msg": "Hello BigchainDB!"
    }
  },
  "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
      "fulfills": null,
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 0
  },
  "operation": "CREATE",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "version": "2.0"
}

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – A transaction with that ID was found. 404 Not Found – A transaction with that ID was not found.

GET /api/v1/transactions¶: Requests to the /api/v1/transactions endpoint without any query parameters will get a response status code 400 Bad Request.

GET /api/v1/transactions?asset_id={asset_id}&operation={CREATE|TRANSFER}&last_tx={true|false}¶

Get a list of transactions that use an asset with the ID asset_id.

If operation is CREATE, then the CREATE transaction which created the asset with ID asset_id will be returned.

If operation is TRANSFER, then every TRANSFER transaction involving the asset with ID asset_id will be returned. This allows users to query the entire history or provenance of an asset.

If operation is not included, then every transaction involving the asset with ID asset_id will be returned.

if last_tx is set to true, only the last transaction is returned instead of all transactions with the given asset_id.

This endpoint returns transactions only if they are in committed blocks.

Query Parameters:
	operation (string) – (Optional) `CREATE` or `TRANSFER`. asset_id (string) – asset ID. last_tx (string) – (Optional) `true` or `false`.

Example request:

GET /api/v1/transactions?operation=TRANSFER&asset_id=4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[{
  "asset": {
    "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317"
  },
  "id": "79ef6803210c941903d63d08b40fa17f0a5a04f11ac0ff04451553a187d97a30",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUAYRI8kzKaZcrW-_avQrAIk5q-7o_7U6biBvoHk1ioBLqHSBcE_PAdNEaeWesAAW_HeCqNUWKaJ5Lzo5Nfz7QgN",
      "fulfills": {
        "output_index": 0,
        "transaction_id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317"
      },
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 1
  },
  "operation": "TRANSFER",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "3yfQPHeWAa1MxTX9Zf9176QqcpcnWcanVZZbaHb8B3h9",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;lu6ov4AKkee6KWGnyjOVLBeyuP0bz4-O6_dPi15eYUc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "3yfQPHeWAa1MxTX9Zf9176QqcpcnWcanVZZbaHb8B3h9"
      ]
    }
  ],
  "version": "2.0"
},
{
  "asset": {
    "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317"
  },
  "id": "1fec726a3b426498147f1a1f19a92c187d551a7f66db4b88d666d7dcc10e86a4",
  "inputs": [
    {
      "fulfillment": "pGSAICw7Ul-c2lG6NFbHp3FbKRC7fivQcNGO7GS4wV3A-1QggUARCMty2JBK_OyPJntWEFxDG4-VbKMy853NtqwnPib5QUJIuwPQa1Y4aN2iIBuoqGE85Pmjcc1ScG9FCPSQHacK",
      "fulfills": {
        "output_index": 0,
        "transaction_id": "79ef6803210c941903d63d08b40fa17f0a5a04f11ac0ff04451553a187d97a30"
      },
      "owners_before": [
        "3yfQPHeWAa1MxTX9Zf9176QqcpcnWcanVZZbaHb8B3h9"
      ]
    }
  ],
  "metadata": {
    "sequence": 2
  },
  "operation": "TRANSFER",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "3Af3fhhjU6d9WecEM9Uw5hfom9kNEwE7YuDWdqAUssqm",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;Ll1r0LzgHUvWB87yIrNFYo731MMUEypqvrbPATTbuD4?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "3Af3fhhjU6d9WecEM9Uw5hfom9kNEwE7YuDWdqAUssqm"
      ]
    }
  ],
  "version": "2.0"
}]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – A list of transactions containing an asset with ID `asset_id` was found and returned. 400 Bad Request – The request wasn’t understood by the server, e.g. the `asset_id` querystring was not included in the request.

POST /api/v1/transactions?mode={mode}¶

This endpoint is used to send a transaction to a BigchainDB network. The transaction is put in the body of the request.

Query Parameters:
	mode (string) – (Optional) One of the three supported modes to send a transaction: `async`, `sync`, `commit`. The default is `async`.

Once the posted transaction arrives at a BigchainDB node, that node will check to see if the transaction is valid. If it’s invalid, the node will return an HTTP 400 (error). Otherwise, the node will send the transaction to Tendermint (in the same node) using the Tendermint broadcast API.

The meaning of the mode query parameter is inherited from the mode parameter in Tendermint’s broadcast API. mode=async means the HTTP response will come back immediately, before Tendermint asks BigchainDB Server to check the validity of the transaction (a second time). mode=sync means the HTTP response will come back after Tendermint gets a response from BigchainDB Server regarding the validity of the transaction. mode=commit means the HTTP response will come back once the transaction is in a committed block.

Note

In the async and sync modes, after a successful HTTP response is returned, the transaction may still be rejected later on. All the transactions are recorded internally by Tendermint in WAL (Write-Ahead Log) before the HTTP response is returned. Nevertheless, the following should be noted:

Transactions in WAL including the failed ones are not exposed in any of the BigchainDB or Tendermint APIs.
Transactions are never fetched from WAL. WAL is never replayed.
A critical failure (e.g. the system is out of disk space) may occur preventing transactions from being stored in WAL, even when the HTTP response indicates a success.
If a transaction fails the validation because it conflicts with the other transactions of the same block, Tendermint includes it into its block, but BigchainDB does not store these transactions and does not offer any information about them in the APIs.

Note

The posted transaction should be valid. The relevant BigchainDB Transactions Spec explains how to build a valid transaction and how to check if a transaction is valid. One would normally use a driver such as the BigchainDB Python Driver to build a valid transaction.

Note

A client can subscribe to the WebSocket Event Stream API to listen for committed transactions.

Example request:

POST /api/v1/transactions?mode=async HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "asset": {
    "data": {
      "msg": "Hello BigchainDB!"
    }
  },
  "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
      "fulfills": null,
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 0
  },
  "operation": "CREATE",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "version": "2.0"
}

Example response:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "asset": {
    "data": {
      "msg": "Hello BigchainDB!"
    }
  },
  "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
      "fulfills": null,
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 0
  },
  "operation": "CREATE",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "version": "2.0"
}

Response Headers:
	Content-Type – `application/json`
Status Codes:	202 Accepted – The meaning of this response depends on the value of the `mode` parameter. See above. 400 Bad Request – The posted transaction was invalid.

POST /api/v1/transactions¶: This endpoint (without any parameters) will push a new transaction. Since no mode parameter is included, the default mode is assumed: async.

Transaction Outputs¶

The /api/v1/outputs endpoint returns transactions outputs filtered by a given public key, and optionally filtered to only include either spent or unspent outputs.

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/outputs¶

Get transaction outputs by public key. The public_key parameter must be a base58 encoded ed25519 public key associated with transaction output ownership.

Returns a list of transaction outputs.

Parameters:

public_key – Base58 encoded public key associated with output ownership. This parameter is mandatory and without it the endpoint will return a 400 response code.
spent – (Optional) Boolean value (true or false) indicating if the result set should include only spent or only unspent outputs. If not specified, the result includes all the outputs (both spent and unspent) associated with the public_key.

GET /api/v1/outputs?public_key={public_key}¶

Return all outputs, both spent and unspent, for the public_key.

Example request:

GET /api/v1/outputs?public_key=1AAAbbb...ccc HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "output_index": 0,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  },
  {
    "output_index": 1,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  }
]

Status Codes:	200 OK – A list of outputs was found and returned in the body of the response. 400 Bad Request – The request wasn’t understood by the server, e.g. the `public_key` querystring was not included in the request.

GET /api/v1/outputs?public_key={public_key}&spent=true¶

Return all spent outputs for public_key.

Example request:

GET /api/v1/outputs?public_key=1AAAbbb...ccc&spent=true HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "output_index": 0,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  }
]

Status Codes:	200 OK – A list of outputs were found and returned in the body of the response. 400 Bad Request – The request wasn’t understood by the server, e.g. the `public_key` querystring was not included in the request.

GET /api/v1/outputs?public_key={public_key}&spent=false¶

Return all unspent outputs for public_key.

Example request:

GET /api/v1/outputs?public_key=1AAAbbb...ccc&spent=false HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "output_index": 1,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  }
]

Status Codes:	200 OK – A list of outputs were found and returned in the body of the response. 400 Bad Request – The request wasn’t understood by the server, e.g. the `public_key` querystring was not included in the request.

Assets¶

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/assets¶

Return all the assets that match a given text search.

Query Parameters:
	search (string) – Text search string to query. limit (int) – (Optional) Limit the number of returned assets. Defaults to `0` meaning return all matching assets.

GET /api/v1/assets/?search={search}¶

Return all assets that match a given text search.

Note

The id of the asset is the same id of the CREATE transaction that created the asset.

Note

You can use assets/?search or assets?search.

If no assets match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/assets/?search=bigchaindb HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "data": {"msg": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "data": {"msg": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
    {
        "data": {"msg": "Hello BigchainDB 3!"},
        "id": "fa6bcb6a8fdea3dc2a860fcdc0e0c63c9cf5b25da8b02a4db4fb6a2d36d27791"
    }
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

GET /api/v1/assets?search={search}&limit={n_documents}¶

Return at most n_documents assets that match a given text search.

If no assets match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/assets?search=bigchaindb&limit=2 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "data": {"msg": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "data": {"msg": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

Transaction Metadata¶

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/metadata¶

Return all the metadata objects that match a given text search.

Query Parameters:
	search (string) – Text search string to query. limit (int) – (Optional) Limit the number of returned metadata objects. Defaults to `0` meaning return all matching objects.

GET /api/v1/metadata/?search={search}¶

Return all metadata objects that match a given text search.

Note

The id of the metadata is the same id of the transaction where it was defined.

Note

You can use metadata/?search or metadata?search.

If no metadata objects match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/metadata/?search=bigchaindb HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "metadata": {"metakey1": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "metadata": {"metakey2": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
    {
        "metadata": {"metakey3": "Hello BigchainDB 3!"},
        "id": "fa6bcb6a8fdea3dc2a860fcdc0e0c63c9cf5b25da8b02a4db4fb6a2d36d27791"
    }
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

GET /api/v1/metadata/?search={search}&limit={n_documents}¶

Return at most n_documents metadata objects that match a given text search.

If no metadata objects match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/metadata?search=bigchaindb&limit=2 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "metadata": {"msg": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "metadata": {"msg": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

Validators¶

GET /api/v1/validators¶

Return the local validators set of a given node.

Example request:

GET /api/v1/validators HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "pub_key": {
               "data":"4E2685D9016126864733225BE00F005515200727FBAB1312FC78C8B76831255A",
               "type":"ed25519"
        },
        "power": 10
    },
    {
         "pub_key": {
               "data":"608D839D7100466D6BA6BE79C320F8B81DE93CFAA58CF9768CF921C6371F2553",
               "type":"ed25519"
         },
         "power": 5
    }
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully and validators set was returned.

Blocks¶

GET /api/v1/blocks/{block_height}¶

Get the block with the height block_height.

Parameters:	block_height (integer) – block height

Example request:

GET /api/v1/blocks/1 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "height": 1,
  "transactions": [
    {
      "asset": {
        "data": {
          "msg": "Hello BigchainDB!"
        }
      },
      "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
      "inputs": [
        {
          "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
          "fulfills": null,
          "owners_before": [
            "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
          ]
        }
      ],
      "metadata": {
        "sequence": 0
      },
      "operation": "CREATE",
      "outputs": [
        {
          "amount": "1",
          "condition": {
            "details": {
              "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
              "type": "ed25519-sha-256"
            },
            "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
          },
          "public_keys": [
            "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
          ]
        }
      ],
      "version": "2.0"
    }
  ]
}

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – A block with that block height was found. 400 Bad Request – The request wasn’t understood by the server, e.g. just requesting `/blocks` without the `block_height`. 404 Not Found – A block with that block height was not found.

GET /api/v1/blocks¶

The unfiltered /blocks endpoint without any query parameters returns a 400 Bad Request status code.

Example request:

GET /api/v1/blocks HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 400 Bad Request

Status Codes:	400 Bad Request – The request wasn’t understood by the server, e.g. just requesting `/blocks` without the `block_height`.

GET /api/v1/blocks?transaction_id={transaction_id}¶

Retrieve a list of block IDs (block heights), such that the blocks with those IDs contain a transaction with the ID transaction_id. A correct response may consist of an empty list or a list with one block ID.

Note

In case no block was found, an empty list and an HTTP status code 200 OK is returned, as the request was still successful.

Query Parameters:
	transaction_id (string) – (Required) transaction ID

Example request:

GET /api/v1/blocks?transaction_id=4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  1
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The request was properly formed and zero or more blocks were found containing the specified `transaction_id`. 400 Bad Request – The request wasn’t understood by the server, e.g. just requesting `/blocks`, without defining `transaction_id`.

Determining the API Root URL¶

When you start BigchainDB Server using bigchaindb start, an HTTP API is exposed at some address. The default is:

http://localhost:9984/api/v1/

It’s bound to localhost, so you can access it from the same machine, but it won’t be directly accessible from the outside world. (The outside world could connect via a SOCKS proxy or whatnot.)

The documentation about BigchainDB Server Configuration Settings has a section about how to set server.bind so as to make the HTTP API publicly accessible.

If the API endpoint is publicly accessible, then the public API Root URL is determined as follows:

The public IP address (like 12.34.56.78) is the public IP address of the machine exposing the HTTP API to the public internet (e.g. either the machine hosting Gunicorn or the machine running the reverse proxy such as NGINX). It’s determined by AWS, Azure, Rackspace, or whoever is hosting the machine.
The DNS hostname (like example.com) is determined by DNS records, such as an “A Record” associating example.com with 12.34.56.78
The port (like 9984) is determined by the server.bind setting if Gunicorn is exposed directly to the public Internet. If a reverse proxy (like NGINX) is exposed directly to the public Internet instead, then it could expose the HTTP API on whatever port it wants to. (It should expose the HTTP API on port 9984, but it’s not bound to do that by anything other than convention.)

The Events API¶

The WebSocket Event Stream API¶

Important

The WebSocket Event Stream runs on a different port than the Web API. The default port for the Web API is 9984, while the one for the Event Stream is 9985.

BigchainDB provides real-time event streams over the WebSocket protocol with the Event Stream API. Connecting to an event stream from your application enables a BigchainDB node to notify you as events occur, such as new valid transactions.

Demoing the API¶

You may be interested in demoing the Event Stream API with the WebSocket echo test to familiarize yourself before attempting an integration.

Determining Support for the Event Stream API¶

It’s a good idea to make sure that the node you’re connecting with has advertised support for the Event Stream API. To do so, send a HTTP GET request to the node’s API Root Endpoint (e.g. http://localhost:9984/api/v1/) and check that the response contains a streams property:

{
 ...,
 "streams": "ws://example.com:9985/api/v1/streams/valid_transactions",
 ...
}

Connection Keep-Alive¶

The Event Stream API supports Ping/Pong frames as descibed in RFC 6455.

Note

It might not be possible to send PING/PONG frames via web browsers because of non availability of Javascript API on different browsers to achieve the same.

Streams¶

Each stream is meant as a unidirectional communication channel, where the BigchainDB node is the only party sending messages. Any messages sent to the BigchainDB node will be ignored.

Streams will always be under the WebSocket protocol (so ws:// or wss://) and accessible as extensions to the /api/v<version>/streams/ API root URL (for example, valid transactions would be accessible under /api/v1/streams/valid_transactions). If you’re running your own BigchainDB instance and need help determining its root URL, then see the page titled Determining the API Root URL.

All messages sent in a stream are in the JSON format.

Note

For simplicity, BigchainDB initially only provides a stream for all committed transactions. In the future, we may provide streams for other information. We may also provide the ability to filter the stream for specific qualities, such as a specific output’s public_key.

If you have specific use cases that you think would fit as part of this API, consider creating a new BEP.

Valid Transactions¶

/valid_transactions

Streams an event for any newly valid transactions committed to a block. Message bodies contain the transaction’s ID, associated asset ID, and containing block’s height.

Example message:

{
    "transaction_id": "<sha3-256 hash>",
    "asset_id": "<sha3-256 hash>",
    "height": <int>
}

Note

Transactions in BigchainDB are committed in batches (“blocks”) and will, therefore, be streamed in batches.

Drivers & Tools¶

These drivers were originally created by the original BigchainDB team:

These drivers and tools were created by the BigchainDB community:

Warning

Some of these projects are a work in progress, but may still be useful. Others might not work with the latest version of BigchainDB.

ANSI C driver, should also work with C++ (working as of June 2019)
C# driver (working as of May 2019)
Haskell transaction builder
Go driver
Ruby driver
Ruby library for preparing/signing transactions and submitting them or querying a BigchainDB node (MIT licensed)

Data Models¶

The Transaction Model¶

See the BigchainDB Transactions Specs.

The Asset Model¶

Use the following link to find the BigchainDB Transactions Spec (or Specs) that are relevant to you:

BigchainDB Transactions Specs

Then see the section titled Transaction Components: Asset.

Inputs and Outputs¶

Use the following link to find the BigchainDB Transactions Spec (or Specs) that are relevant to you:

BigchainDB Transactions Specs

Then see the sections titled Transaction Components: Inputs and Transaction Components: Outputs.

Conditions¶

Use the following link to find the BigchainDB Transactions Spec (or Specs) that are relevant to you:

BigchainDB Transactions Specs

Then see the section titled Transaction Components: Conditions.

The Block Model¶

A block is a JSON object with a particular schema, as outlined in this page. A block must contain the following JSON keys (also called names or fields):

{
  "height": "<Height of the block>",
  "transactions": ["<List of transactions>"]
}

The JSON Keys in a Block¶

height

The block "height" (integer) denotes the height of the blockchain when the given block was committed. Since the blockchain height increases monotonically the height of block can be regarded as its id.

NOTE: The genesis block has height 0

transactions

A list of the transactions included in the block. (Each transaction is a JSON object.)

Kubernetes Deployment Template¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This section outlines a way to deploy a BigchainDB node (or BigchainDB network) on Microsoft Azure using Kubernetes. You may choose to use it as a template or reference for your own deployment, but we make no claim that it is suitable for your purposes. Feel free change things to suit your needs or preferences.

Overview¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page summarizes some steps to go through to set up a BigchainDB network. You can modify them to suit your needs.

Generate All Shared BigchainDB Setup Parameters¶

There are some shared BigchainDB setup paramters that every node operator in the consortium shares because they are properties of the Tendermint network. They look like this:

# Tendermint data
BDB_PERSISTENT_PEERS='bdb-instance-1,bdb-instance-2,bdb-instance-3,bdb-instance-4'
BDB_VALIDATORS='bdb-instance-1,bdb-instance-2,bdb-instance-3,bdb-instance-4'
BDB_VALIDATOR_POWERS='10,10,10,10'
BDB_GENESIS_TIME='0001-01-01T00:00:00Z'
BDB_CHAIN_ID='test-chain-rwcPML'

Those paramters only have to be generated once, by one member of the consortium. That person will then share the results (Tendermint setup parameters) with all the node operators.

The above example parameters are for a network of 4 initial (seed) nodes. Note how BDB_PERSISTENT_PEERS, BDB_VALIDATORS and BDB_VALIDATOR_POWERS are lists with 4 items each. If your consortium has a different number of initial nodes, then those lists should have that number or items. Use 10 for all the power values.

To generate a BDB_GENESIS_TIME and a BDB_CHAIN_ID, you can do this:

$ mkdir $(pwd)/tmdata
$ docker run --rm -v $(pwd)/tmdata:/tendermint/config tendermint/tendermint:v0.31.5 init
$ cat $(pwd)/tmdata/genesis.json

You should see something that looks like:

{"genesis_time": "0001-01-01T00:00:00Z",
 "chain_id": "test-chain-bGX7PM",
 "validators": [
     {"pub_key":
         {"type": "ed25519",
          "data": "4669C4B966EB8B99E45E40982B2716A9D3FA53B54C68088DAB2689935D7AF1A9"},
      "power": 10,
      "name": ""}
 ],
 "app_hash": ""
}

The value with "genesis_time" is BDB_GENESIS_TIME and the value with "chain_id" is BDB_CHAIN_ID.

Now you have all the BigchainDB setup parameters and can share them with all of the node operators. (They will put them in their vars file. We’ll say more about that file below.)

Things Each Node Operator Must Do¶

1. Make up an FQDN for your BigchainDB node (e.g. mynode.mycorp.com). This is where external users will access the BigchainDB HTTP API, for example. Make sure you’ve registered the associated domain name (e.g. mycorp.com).

Get an SSL certificate for your BigchainDB node’s FQDN. Also get the root CA certificate and all intermediate certificates. They should all be provided by your SSL certificate provider. Put all those certificates together in one certificate chain file in the following order:

Domain certificate (i.e. the one you ordered for your FQDN)
All intermediate certificates
Root CA certificate

DigiCert has a web page explaining certificate chains.

You will put the path to that certificate chain file in the vars file, when you configure your node later.

2a. If your BigchainDB node will use 3scale for API authentication, monitoring and billing, you will need all relevant 3scale settings and credentials.

2b. If your BigchainDB node will not use 3scale, then write authorization will be granted to all POST requests with a secret token in the HTTP headers. (All GET requests are allowed to pass.) You can make up that SECRET_TOKEN now. For example, superSECRET_token4-POST*requests. You will put it in the vars file later. Every BigchainDB node in a BigchainDB network can have a different secret token. To make an HTTP POST request to your BigchainDB node, you must include an HTTP header named X-Secret-Access-Token and set it equal to your secret token, e.g.

X-Secret-Access-Token: superSECRET_token4-POST*requests

3. Deploy a Kubernetes cluster for your BigchainDB node. We have some instructions for how to Deploy a Kubernetes cluster on Azure.

Warning

In theory, you can deploy your BigchainDB node to any Kubernetes cluster, but there can be differences between different Kubernetes clusters, especially if they are running different versions of Kubernetes. We tested this Kubernetes Deployment Template on Azure ACS in February 2018 and at that time ACS was deploying a Kubernetes 1.7.7 cluster. If you can force your cluster to have that version of Kubernetes, then you’ll increase the likelihood that everything will work.

4. Deploy your BigchainDB node inside your new Kubernetes cluster. You will fill up the vars file, then you will run a script which reads that file to generate some Kubernetes config files, you will send those config files to your Kubernetes cluster, and then you will deploy all the stuff that you need to have a BigchainDB node.

⟶ Proceed to deploy your BigchainDB node.

How to Set Up a Self-Signed Certificate Authority¶

This page enumerates the steps we use to set up a self-signed certificate authority (CA). This is something that only needs to be done once per BigchainDB network, by the organization managing the network, i.e. the CA is for the whole network. We use Easy-RSA.

Step 1: Install & Configure Easy-RSA¶

First create a directory for the CA and cd into it:

mkdir bdb-node-ca

cd bdb-node-ca

Then install and configure Easy-RSA in that directory.

Step 2: Create a Self-Signed CA¶

You can create a self-signed CA by going to the bdb-node-ca/easy-rsa-3.0.1/easyrsa3 directory and using:

./easyrsa init-pki

./easyrsa build-ca

You will also be asked to enter a PEM pass phrase (for encrypting the ca.key file). Make sure to securely store that PEM pass phrase. If you lose it, you won’t be able to add or remove entities from your PKI infrastructure in the future.

You will be prompted to enter the Distinguished Name (DN) information for this CA. For each field, you can accept the default value [in brackets] by pressing Enter.

Warning

Don’t accept the default value of OU (IT). Instead, enter the value ROOT-CA.

While Easy-RSA CA is a valid and acceptable Common Name, you should probably enter a name based on the name of the managing organization, e.g. Omega Ledger CA.

Tip: You can get help with the easyrsa command (and its subcommands) by using the subcommand ./easyrsa help

Step 3: Create an Intermediate CA¶

TODO

Step 4: Generate a Certificate Revocation List¶

You can generate a Certificate Revocation List (CRL) using:

./easyrsa gen-crl

You will need to run this command every time you revoke a certificate. The generated crl.pem needs to be uploaded to your infrastructure to prevent the revoked certificate from being used again.

Step 5: Secure the CA¶

The security of your infrastructure depends on the security of this CA.

Ensure that you restrict access to the CA and enable only legitimate and required people to sign certificates and generate CRLs.
Restrict access to the machine where the CA is hosted.
Many certificate providers keep the CA offline and use a rotating intermediate CA to sign and revoke certificates, to mitigate the risk of the CA getting compromised.
In case you want to destroy the machine where you created the CA (for example, if this was set up on a cloud provider instance), you can backup the entire easyrsa directory to secure storage. You can always restore it to a trusted instance again during the times when you want to sign or revoke certificates. Remember to backup the directory after every update.

How to Generate a Server Certificate for MongoDB¶

This page enumerates the steps we use to generate a server certificate for a MongoDB instance. A server certificate is also referred to as a “member certificate” in the MongoDB documentation. We use Easy-RSA.

Step 1: Install & Configure Easy–RSA¶

First create a directory for the server certificate (member cert) and cd into it:

mkdir member-cert

cd member-cert

Then install and configure Easy-RSA in that directory.

Step 2: Create the Server Private Key and CSR¶

You can create the server private key and certificate signing request (CSR) by going into the directory member-cert/easy-rsa-3.0.1/easyrsa3 and using something like:

Note

Please make sure you are fullfilling the requirements for MongoDB server/member certificates.

./easyrsa init-pki

./easyrsa --req-cn=mdb-instance-0 --subject-alt-name=DNS:localhost,DNS:mdb-instance-0 gen-req mdb-instance-0 nopass

You should replace the Common Name (mdb-instance-0 above) with the correct name for your MongoDB instance in the network, e.g. mdb-instance-5 or mdb-instance-12. (This name is decided by the organization managing the network.)

You will be prompted to enter the Distinguished Name (DN) information for this certificate. For each field, you can accept the default value [in brackets] by pressing Enter.

Warning

Don’t accept the default value of OU (IT). Instead, enter the value MongoDB-Instance.

Aside: You need to provide the DNS:localhost SAN during certificate generation for using the localhost exception in the MongoDB instance. All certificates can have this attribute without compromising security as the localhost exception works only the first time.

Step 3: Get the Server Certificate Signed¶

The CSR file created in the last step should be located in pki/reqs/mdb-instance-0.req (where the integer 0 may be different for you). You need to send it to the organization managing the BigchainDB network so that they can use their CA to sign the request. (The managing organization should already have a self-signed CA.)

If you are the admin of the managing organization’s self-signed CA, then you can import the CSR and use Easy-RSA to sign it. Go to your bdb-node-ca/easy-rsa-3.0.1/easyrsa3/ directory and do something like:

./easyrsa import-req /path/to/mdb-instance-0.req mdb-instance-0

./easyrsa --subject-alt-name=DNS:localhost,DNS:mdb-instance-0 sign-req server mdb-instance-0

Once you have signed it, you can send the signed certificate and the CA certificate back to the requestor. The files are pki/issued/mdb-instance-0.crt and pki/ca.crt.

Step 4: Generate the Consolidated Server PEM File¶

MongoDB requires a single, consolidated file containing both the public and private keys.

cat /path/to/mdb-instance-0.crt /path/to/mdb-instance-0.key > mdb-instance-0.pem

How to Generate a Client Certificate for MongoDB¶

This page enumerates the steps we use to generate a client certificate to be used by clients who want to connect to a TLS-secured MongoDB database. We use Easy-RSA.

Step 1: Install and Configure Easy-RSA¶

First create a directory for the client certificate and cd into it:

mkdir client-cert

cd client-cert

Then install and configure Easy-RSA in that directory.

Step 2: Create the Client Private Key and CSR¶

You can create the client private key and certificate signing request (CSR) by going into the directory client-cert/easy-rsa-3.0.1/easyrsa3 and using:

./easyrsa init-pki

./easyrsa gen-req bdb-instance-0 nopass

You should change the Common Name (e.g. bdb-instance-0) to a value that reflects what the client certificate is being used for, e.g. mdb-mon-instance-3 or mdb-bak-instance-4. (The final integer is specific to your BigchainDB node in the BigchainDB network.)

You will be prompted to enter the Distinguished Name (DN) information for this certificate. For each field, you can accept the default value [in brackets] by pressing Enter.

Warning

Don’t accept the default value of OU (IT). Instead, enter the value BigchainDB-Instance, MongoDB-Mon-Instance or MongoDB-Backup-Instance as appropriate.

Aside: The nopass option means “do not encrypt the private key (default is encrypted)”. You can get help with the easyrsa command (and its subcommands) by using the subcommand ./easyrsa help.

Note

For more information about requirements for MongoDB client certificates, please consult the official MongoDB documentation.

Step 3: Get the Client Certificate Signed¶

The CSR file created in the previous step should be located in pki/reqs/bdb-instance-0.req (or whatever Common Name you used in the gen-req command above). You need to send it to the organization managing the BigchainDB network so that they can use their CA to sign the request. (The managing organization should already have a self-signed CA.)

If you are the admin of the managing organization’s self-signed CA, then you can import the CSR and use Easy-RSA to sign it. Go to your bdb-node-ca/easy-rsa-3.0.1/easyrsa3/ directory and do something like:

./easyrsa import-req /path/to/bdb-instance-0.req bdb-instance-0

./easyrsa sign-req client bdb-instance-0

Once you have signed it, you can send the signed certificate and the CA certificate back to the requestor. The files are pki/issued/bdb-instance-0.crt and pki/ca.crt.

Step 4: Generate the Consolidated Client PEM File¶

Note

This step can be skipped for BigchainDB client certificate as BigchainDB uses the PyMongo driver, which accepts separate certificate and key files.

MongoDB, MongoDB Backup Agent and MongoDB Monitoring Agent require a single, consolidated file containing both the public and private keys.

cat /path/to/bdb-instance-0.crt /path/to/bdb-instance-0.key > bdb-instance-0.pem

 OR

cat /path/to/mdb-mon-instance-0.crt /path/to/mdb-mon-instance-0.key > mdb-mon-instance-0.pem

 OR

cat /path/to/mdb-bak-instance-0.crt /path/to/mdb-bak-instance-0.key > mdb-bak-instance-0.pem

How to Revoke an SSL/TLS Certificate¶

This page enumerates the steps we take to revoke a self-signed SSL/TLS certificate in a BigchainDB network. It can only be done by someone with access to the self-signed CA associated with the network’s managing organization.

Step 1: Revoke a Certificate¶

Since we used Easy-RSA version 3 to set up the CA, we use it to revoke certificates too.

Go to the following directory (associated with the self-signed CA): .../bdb-node-ca/easy-rsa-3.0.1/easyrsa3. You need to be aware of the file name used to import the certificate using the ./easyrsa import-req before. Run the following command to revoke a certificate:

./easyrsa revoke <filename>

This will update the CA database with the revocation details. The next step is to use the updated database to issue an up-to-date certificate revocation list (CRL).

Step 2: Generate a New CRL¶

Generate a new CRL for your infrastructure using:

./easyrsa gen-crl

The generated crl.pem file needs to be uploaded to your infrastructure to prevent the revoked certificate from being used again.

In particlar, the generated crl.pem file should be sent to all BigchainDB node operators in your BigchainDB network, so that they can update it in their MongoDB instance and their BigchainDB Server instance.

Template: Deploy a Kubernetes Cluster on Azure¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

A BigchainDB node can be run inside a Kubernetes cluster. This page describes one way to deploy a Kubernetes cluster on Azure.

Step 1: Get a Pay-As-You-Go Azure Subscription¶

Microsoft Azure has a Free Trial subscription (at the time of writing), but it’s too limited to run an advanced BigchainDB node. Sign up for a Pay-As-You-Go Azure subscription via the Azure website.

You may find that you have to sign up for a Free Trial subscription first. That’s okay: you can have many subscriptions.

Step 2: Create an SSH Key Pair¶

You’ll want an SSH key pair so you’ll be able to SSH to the virtual machines that you’ll deploy in the next step. (If you already have an SSH key pair, you could reuse it, but it’s probably a good idea to make a new SSH key pair for your Kubernetes VMs and nothing else.)

See the page about how to generate a key pair for SSH.

Step 3: Deploy an Azure Container Service (ACS)¶

It’s possible to deploy an Azure Container Service (ACS) from the Azure Portal (i.e. online in your web browser) but it’s actually easier to do it using the Azure Command-Line Interface (CLI).

Microsoft has instructions to install the Azure CLI 2.0 on most common operating systems. Do that.

If you already have the Azure CLI installed, you may want to update it.

Warning

az component update isn’t supported if you installed the CLI using some of Microsoft’s provided installation instructions. See the Microsoft docs for update instructions.

Next, login to your account using:

$ az login

It will tell you to open a web page and to copy a code to that page.

If the login is a success, you will see some information about all your subscriptions, including the one that is currently enabled ("state": "Enabled"). If the wrong one is enabled, you can switch to the right one using:

$ az account set --subscription <subscription name or ID>

Next, you will have to pick the Azure data center location where you’d like to deploy your cluster. You can get a list of all available locations using:

$ az account list-locations

Next, create an Azure “resource group” to contain all the resources (virtual machines, subnets, etc.) associated with your soon-to-be-deployed cluster. You can name it whatever you like but avoid fancy characters because they may confuse some software.

$ az group create --name <resource group name> --location <location name>

Example location names are koreacentral and westeurope.

Finally, you can deploy an ACS using something like:

$ az acs create --name <a made-up cluster name> \
--resource-group <name of resource group created earlier> \
--master-count 3 \
--agent-count 3 \
--admin-username ubuntu \
--agent-vm-size Standard_L4s \
--dns-prefix <make up a name> \
--ssh-key-value ~/.ssh/<name>.pub \
--orchestrator-type kubernetes \
--debug --output json

Note

The Azure documentation has a list of all az acs create options. You might prefer a smaller agent VM size, for example. You can also get a list of the options using:

$ az acs create --help

It takes a few minutes for all the resources to deploy. You can watch the progress in the Azure Portal: go to Resource groups (with the blue cube icon) and click on the one you created to see all the resources in it.

Trouble with the Service Principal? Then Read This!¶

If the az acs create command fails with an error message including the text, “The Service Principal in ServicePrincipalProfile could not be validated”, then we found you can prevent that by creating a Service Principal ahead of time and telling az acs create to use that one. (It’s supposed to create one, but sometimes that fails, I guess.)

Create a new resource group, even if you created one before. They’re free anyway:

$ az login
$ az group create --name <new resource group name> \
                  --location <Azure region like westeurope>

Note the id in the output. It looks like "/subscriptions/369284be-0104-421a-8488-1aeac0caecbb/resourceGroups/examplerg". It can be copied into the next command. Create a Service Principal using:

$ az ad sp create-for-rbac --role="Contributor" \
--scopes=<id value copied from above, including the double quotes on the ends>

Note the appId and password. Put those in a new az acs create command like above, with two new options added:

$ az acs create ... \
--service-principal <appId> \
--client-secret <password>

Optional: SSH to Your New Kubernetes Cluster Nodes¶

You can SSH to one of the just-deployed Kubernetes “master” nodes (virtual machines) using:

$ ssh -i ~/.ssh/<name> ubuntu@<master-ip-address-or-fqdn>

where you can get the IP address or FQDN of a master node from the Azure Portal. For example:

$ ssh -i ~/.ssh/mykey123 ubuntu@mydnsprefix.westeurope.cloudapp.azure.com

Note

All the master nodes are accessible behind the same public IP address and FQDN. You connect to one of the masters randomly based on the load balancing policy.

The “agent” nodes shouldn’t get public IP addresses or externally accessible FQDNs, so you can’t SSH to them directly, but you can first SSH to the master and then SSH to an agent from there using their hostname. To do that, you could copy your SSH key pair to the master (a bad idea), or use SSH agent forwarding (better). To do the latter, do the following on the machine you used to SSH to the master:

$ echo -e "Host <FQDN of the cluster from Azure Portal>\n  ForwardAgent yes" >> ~/.ssh/config

To verify that SSH agent forwarding works properly, SSH to the one of the master nodes and do:

$ echo "$SSH_AUTH_SOCK"

If you get an empty response, then SSH agent forwarding hasn’t been set up correctly. If you get a non-empty response, then SSH agent forwarding should work fine and you can SSH to one of the agent nodes (from a master) using:

$ ssh ubuntu@k8s-agent-4AC80E97-0

where k8s-agent-4AC80E97-0 is the name of a Kubernetes agent node in your Kubernetes cluster. You will have to replace it by the name of an agent node in your cluster.

Optional: Delete the Kubernetes Cluster¶

$ az acs delete \
--name <ACS cluster name> \
--resource-group <name of resource group containing the cluster>

Optional: Delete the Resource Group¶

CAUTION: You might end up deleting resources other than the ACS cluster.

$ az group delete \
--name <name of resource group containing the cluster>

Next, you can run a BigchainDB node/cluster(BFT) on your new Kubernetes cluster.

Kubernetes Template: Deploy a Single BigchainDB Node¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page describes how to deploy a BigchainDB node using Kubernetes. It assumes you already have a running Kubernetes cluster.

Below, we refer to many files by their directory and filename, such as configuration/config-map.yaml. Those files are files in the bigchaindb/bigchaindb repository on GitHub in the k8s/ directory. Make sure you’re getting those files from the appropriate Git branch on GitHub, i.e. the branch for the version of BigchainDB that your BigchainDB cluster is using.

Step 1: Install and Configure kubectl¶

kubectl is the Kubernetes CLI. If you don’t already have it installed, then see the Kubernetes docs to install it.

The default location of the kubectl configuration file is ~/.kube/config. If you don’t have that file, then you need to get it.

Azure. If you deployed your Kubernetes cluster on Azure using the Azure CLI 2.0 (as per our template), then you can get the ~/.kube/config file using:

$ az acs kubernetes get-credentials \
--resource-group <name of resource group containing the cluster> \
--name <ACS cluster name>

If it asks for a password (to unlock the SSH key) and you enter the correct password, but you get an error message, then try adding --ssh-key-file ~/.ssh/<name> to the above command (i.e. the path to the private key).

Note

About kubectl contexts. You might manage several Kubernetes clusters. To make it easy to switch from one to another, kubectl has a notion of “contexts,” e.g. the context for cluster 1 or the context for cluster 2. To find out the current context, do:

$ kubectl config view

and then look for the current-context in the output. The output also lists all clusters, contexts and users. (You might have only one of each.) You can switch to a different context using:

$ kubectl config use-context <new-context-name>

You can also switch to a different context for just one command by inserting --context <context-name> into any kubectl command. For example:

$ kubectl get pods

will get a list of the pods in the Kubernetes cluster associated with the context named k8s-bdb-test-cluster-0.

Step 2: Connect to Your Kubernetes Cluster’s Web UI (Optional)¶

You can connect to your cluster’s Kubernetes Dashboard (also called the Web UI) using:

$ kubectl proxy -p 8001

or

$ az acs kubernetes browse -g [Resource Group] -n [Container service instance name] --ssh-key-file /path/to/privateKey

or, if you prefer to be explicit about the context (explained above):

$ kubectl proxy -p 8001

The output should be something like Starting to serve on 127.0.0.1:8001. That means you can visit the dashboard in your web browser at http://127.0.0.1:8001/ui.

Note

Known Issue: If you are having accessing the UI i.e. accessing http://127.0.0.1:8001/ui in your browser returns a blank page and is redirected to http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy , you can access the UI by adding a / at the end of the redirected URL i.e. http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

Step 3: Configure Your BigchainDB Node¶

See the page titled How to Configure a BigchainDB Node.

Step 4: Start the NGINX Service¶

This will will give us a public IP for the cluster.

Once you complete this step, you might need to wait up to 10 mins for the public IP to be assigned.

You have the option to use vanilla NGINX without HTTPS support or an NGINX with HTTPS support.
Start the Kubernetes Service:
$ kubectl apply -f nginx-https/nginx-https-svc.yaml

OR

$ kubectl apply -f nginx-http/nginx-http-svc.yaml

Step 5: Assign DNS Name to the NGINX Public IP¶

This step is required only if you are planning to set up multiple BigchainDB nodes or are using HTTPS certificates tied to a domain.
The following command can help you find out if the NGINX service started above has been assigned a public IP or external IP address:
$ kubectl get svc -w
Once a public IP is assigned, you can map it to a DNS name. We usually assign bdb-test-node-0, bdb-test-node-1 and so on in our documentation. Let’s assume that we assign the unique name of bdb-test-node-0 here.

Set up DNS mapping in Azure. Select the current Azure resource group and look for the Public IP resource. You should see at least 2 entries there - one for the Kubernetes master and the other for the NGINX instance. You may have to Refresh the Azure web page listing the resources in a resource group for the latest changes to be reflected. Select the Public IP resource that is attached to your service (it should have the Azure DNS prefix name along with a long random string, without the master-ip string), select Configuration, add the DNS assigned above (for example, bdb-test-node-0), click Save, and wait for the changes to be applied.

To verify the DNS setting is operational, you can run nslookup <DNS name added in Azure configuration> from your local Linux shell.

This will ensure that when you scale to different geographical zones, other Tendermint nodes in the network can reach this instance.

Step 6: Start the MongoDB Kubernetes Service¶

Start the Kubernetes Service:
$ kubectl apply -f mongodb/mongo-svc.yaml

Step 7: Start the BigchainDB Kubernetes Service¶

Start the Kubernetes Service:
$ kubectl apply -f bigchaindb/bigchaindb-svc.yaml

Step 8(Optional): Start the OpenResty Kubernetes Service¶

Start the Kubernetes Service:
$ kubectl apply -f nginx-openresty/nginx-openresty-svc.yaml

Step 9: Start the NGINX Kubernetes Deployment¶

NGINX is used as a proxy to the BigchainDB, Tendermint and MongoDB instances in the node. It proxies HTTP/HTTPS requests on the node-frontend-port to the corresponding OpenResty(if 3scale enabled) or BigchainDB backend, TCP connections on mongodb-frontend-port, tm-p2p-port and tm-pub-key-access to MongoDB and Tendermint respectively.
This configuration is located in the file nginx-https/nginx-https-dep.yaml or nginx-http/nginx-http-dep.yaml.
Start the Kubernetes Deployment:
$ kubectl apply -f nginx-https/nginx-https-dep.yaml

OR

$ kubectl apaply -f nginx-http/nginx-http-dep.yaml

Step 10: Create Kubernetes Storage Classes for MongoDB¶

MongoDB needs somewhere to store its data persistently, outside the container where MongoDB is running. Our MongoDB Docker container (based on the official MongoDB Docker container) exports two volume mounts with correct permissions from inside the container:

The directory where the MongoDB instance stores its data: /data/db. There’s more explanation in the MongoDB docs about storage.dbpath.
The directory where the MongoDB instance stores the metadata for a sharded cluster: /data/configdb/. There’s more explanation in the MongoDB docs about sharding.configDB.

Explaining how Kubernetes handles persistent volumes, and the associated terminology, is beyond the scope of this documentation; see the Kubernetes docs about persistent volumes.

The first thing to do is create the Kubernetes storage classes.

Set up Storage Classes in Azure. First, you need an Azure storage account. If you deployed your Kubernetes cluster on Azure using the Azure CLI 2.0 (as per our template), then the az acs create command already created a storage account in the same location and resource group as your Kubernetes cluster. Both should have the same “storage account SKU”: Standard_LRS. Standard storage is lower-cost and lower-performance. It uses hard disk drives (HDD). LRS means locally-redundant storage: three replicas in the same data center. Premium storage is higher-cost and higher-performance. It uses solid state drives (SSD).

We recommend using Premium storage with our Kubernetes deployment template. Create a storage account for Premium storage and associate it with your Azure resource group. For future reference, the command to create a storage account is az storage account create.

Note

Please refer to Azure documentation for the list of VMs that are supported by Premium Storage.

The Kubernetes template for configuration of the MongoDB Storage Class is located in the file mongodb/mongo-sc.yaml.

You may have to update the parameters.location field in the file to specify the location you are using in Azure.

If you want to use a custom storage account with the Storage Class, you can also update parameters.storageAccount and provide the Azure storage account name.

Create the required storage classes using:

$ kubectl apply -f mongodb/mongo-sc.yaml

You can check if it worked using kubectl get storageclasses.

Step 11: Create Kubernetes Persistent Volume Claims for MongoDB¶

Next, you will create two PersistentVolumeClaim objects mongo-db-claim and mongo-configdb-claim.

This configuration is located in the file mongodb/mongo-pvc.yaml.

Note how there’s no explicit mention of Azure, AWS or whatever. ReadWriteOnce (RWO) means the volume can be mounted as read-write by a single Kubernetes node. (ReadWriteOnce is the only access mode supported by AzureDisk.) storage: 20Gi means the volume has a size of 20 gibibytes.

You may want to update the spec.resources.requests.storage field in both the files to specify a different disk size.

Create the required Persistent Volume Claims using:

$ kubectl apply -f mongodb/mongo-pvc.yaml

You can check its status using: kubectl get pvc -w

Initially, the status of persistent volume claims might be “Pending” but it should become “Bound” fairly quickly.

Note

The default Reclaim Policy for dynamically created persistent volumes is Delete which means the PV and its associated Azure storage resource will be automatically deleted on deletion of PVC or PV. In order to prevent this from happening do the following steps to change default reclaim policy of dyanmically created PVs from Delete to Retain

Run the following command to list existing PVs

$ kubectl get pv

Run the following command to update a PV’s reclaim policy to <Retain>

$ kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

For notes on recreating a private volume form a released Azure disk resource consult the page about cluster troubleshooting.

Step 12: Start a Kubernetes StatefulSet for MongoDB¶

Create the MongoDB StatefulSet using:
$ kubectl apply -f mongodb/mongo-ss.yaml
It might take up to 10 minutes for the disks, specified in the Persistent Volume Claims above, to be created and attached to the pod. The UI might show that the pod has errored with the message “timeout expired waiting for volumes to attach/mount”. Use the CLI below to check the status of the pod in this case, instead of the UI. This happens due to a bug in Azure ACS.
$ kubectl get pods -w

Step 13: Configure Users and Access Control for MongoDB¶

In this step, you will create a user on MongoDB with authorization to create more users and assign roles to it. We will also create MongoDB client users for BigchainDB and MongoDB Monitoring agent(Optional).
$ kubectl apply -f mongodb/configure_mdb.sh

Step 14: Create Kubernetes Storage Classes for BigchainDB¶

BigchainDB needs somewhere to store Tendermint data persistently, Tendermint uses LevelDB as the persistent storage layer.

The Kubernetes template for configuration of Storage Class is located in the file bigchaindb/bigchaindb-sc.yaml.

Details about how to create a Azure Storage account and how Kubernetes Storage Class works are already covered in this document: Step 10: Create Kubernetes Storage Classes for MongoDB.

Create the required storage classes using:

$ kubectl apply -f bigchaindb/bigchaindb-sc.yaml

You can check if it worked using kubectl get storageclasses.

Step 15: Create Kubernetes Persistent Volume Claims for BigchainDB¶

Next, you will create two PersistentVolumeClaim objects tendermint-db-claim and tendermint-config-db-claim.

This configuration is located in the file bigchaindb/bigchaindb-pvc.yaml.

Details about Kubernetes Persistent Volumes, Persistent Volume Claims and how they work with Azure are already covered in this document: Step 11: Create Kubernetes Persistent Volume Claims for MongoDB.

Create the required Persistent Volume Claims using:

$ kubectl apply -f bigchaindb/bigchaindb-pvc.yaml

You can check its status using:

kubectl get pvc -w

Step 16: Start a Kubernetes StatefulSet for BigchainDB¶

This configuration is located in the file bigchaindb/bigchaindb-ss.yaml.

Set the spec.serviceName to the value set in bdb-instance-name in the ConfigMap. For example, if the value set in the bdb-instance-name is bdb-instance-0, set the field to tm-instance-0.

Set metadata.name, spec.template.metadata.name and spec.template.metadata.labels.app to the value set in bdb-instance-name in the ConfigMap, followed by -ss. For example, if the value set in the bdb-instance-name is bdb-instance-0, set the fields to the value bdb-insance-0-ss.

As we gain more experience running Tendermint in testing and production, we will tweak the resources.limits.cpu and resources.limits.memory.
Create the BigchainDB StatefulSet using:
$ kubectl apply -f bigchaindb/bigchaindb-ss.yaml
$ kubectl get pods -w

Step 17(Optional): Start a Kubernetes Deployment for MongoDB Monitoring Agent¶

This configuration is located in the file mongodb-monitoring-agent/mongo-mon-dep.yaml.

Set metadata.name, spec.template.metadata.name and spec.template.metadata.labels.app to the value set in mdb-mon-instance-name in the ConfigMap, followed by -dep. For example, if the value set in the mdb-mon-instance-name is mdb-mon-instance-0, set the fields to the value mdb-mon-instance-0-dep.

The configuration uses the following values set in the Secret:

mdb-mon-certs

ca-auth

cloud-manager-credentials
Start the Kubernetes Deployment using:
$ kubectl apply -f mongodb-monitoring-agent/mongo-mon-dep.yaml

Step 18(Optional): Start a Kubernetes Deployment for OpenResty¶

This configuration is located in the file nginx-openresty/nginx-openresty-dep.yaml.

Set metadata.name and spec.template.metadata.labels.app to the value set in openresty-instance-name in the ConfigMap, followed by -dep. For example, if the value set in the openresty-instance-name is openresty-instance-0, set the fields to the value openresty-instance-0-dep.

Set the port to be exposed from the pod in the spec.containers[0].ports section. We currently expose the port at which OpenResty is listening for requests, openresty-backend-port in the above ConfigMap.

The configuration uses the following values set in the Secret:

threescale-credentials

The configuration uses the following values set in the ConfigMap:

node-dns-server-ip

openresty-backend-port

ngx-bdb-instance-name

bigchaindb-api-port
Create the OpenResty Deployment using:
$ kubectl apply -f nginx-openresty/nginx-openresty-dep.yaml
You can check its status using the command kubectl get deployments -w

Step 19(Optional): Configure the MongoDB Cloud Manager¶

Refer to the documentation for details on how to configure the MongoDB Cloud Manager to enable monitoring and backup.

Step 20(Optional): Only for multi site deployments(Geographically dispersed)¶

We need to make sure that clusters are able to talk to each other i.e. specifically the communication between the Tendermint peers. Set up networking between the clusters using Kubernetes Services.

Assuming we have a BigchainDB instance bdb-instance-1 residing in Azure data center location westeurope and we want to connect to bdb-instance-2, bdb-instance-3, and bdb-instance-4 located in Azure data centers eastus, centralus and westus, respectively. Unless you already have explicitly set up networking for bdb-instance-1 to communicate with bdb-instance-2/3/4 and vice versa, we will have to add a Kubernetes Service in each cluster to accomplish this goal in order to set up a Tendermint P2P network. It is similar to ensuring that there is a CNAME record in the DNS infrastructure to resolve bdb-instance-X to the host where it is actually available. We can do this in Kubernetes using a Kubernetes Service of type ExternalName.

This configuration is located in the file bigchaindb/bigchaindb-ext-conn-svc.yaml.
Set the name of the metadata.name to the host name of the BigchainDB instance you are trying to connect to. For instance if you are configuring this service on cluster with bdb-instance-1 then the metadata.name will be bdb-instance-2 and vice versa.
Set spec.ports.port[0] to the tm-p2p-port from the ConfigMap for the other cluster.
Set spec.ports.port[1] to the tm-rpc-port from the ConfigMap for the other cluster.
Set spec.externalName to the FQDN mapped to NGINX Public IP of the cluster you are trying to connect to. For more information about the FQDN please refer to: Assign DNS name to NGINX Public IP.

Note

This operation needs to be replicated n-1 times per node for a n node cluster, with the respective FQDNs we need to communicate with.

If you are not the system administrator of the cluster, you have to get in touch with the system administrator/s of the other n-1 clusters and share with them your instance name (tendermint-instance-name in the ConfigMap) and the FQDN of the NGINX instance acting as Gateway(set in: Assign DNS name to NGINX Public IP).

Step 21: Verify the BigchainDB Node Setup¶

Step 21.1: Testing Internally¶

To test the setup of your BigchainDB node, you could use a Docker container that provides utilities like nslookup, curl and dig. For example, you could use a container based on our bigchaindb/toolbox image. (The corresponding Dockerfile is in the bigchaindb/bigchaindb repository on GitHub.) You can use it as below to get started immediately:

$ kubectl   \
   run -it toolbox \
   --image bigchaindb/toolbox \
   --image-pull-policy=Always \
   --restart=Never --rm

It will drop you to the shell prompt.

To test the MongoDB instance:

$ nslookup mdb-instance-0

$ dig +noall +answer _mdb-port._tcp.mdb-instance-0.default.svc.cluster.local SRV

$ curl -X GET http://mdb-instance-0:27017

The nslookup command should output the configured IP address of the service (in the cluster). The dig command should return the configured port numbers. The curl command tests the availability of the service.

To test the BigchainDB instance:

$ nslookup bdb-instance-0

$ dig +noall +answer _bdb-api-port._tcp.bdb-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _bdb-ws-port._tcp.bdb-instance-0.default.svc.cluster.local SRV

$ curl -X GET http://bdb-instance-0:9984

$ curl -X GET http://bdb-instance-0:9986/pub_key.json

$ curl -X GET http://bdb-instance-0:26657/abci_info

$ wsc -er ws://bdb-instance-0:9985/api/v1/streams/valid_transactions

To test the OpenResty instance:

$ nslookup openresty-instance-0

$ dig +noall +answer _openresty-svc-port._tcp.openresty-instance-0.default.svc.cluster.local SRV

To verify if OpenResty instance forwards the requests properly, send a POST transaction to OpenResty at post 80 and check the response from the backend BigchainDB instance.

To test the vanilla NGINX instance:

$ nslookup ngx-http-instance-0

$ dig +noall +answer _public-node-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _public-health-check-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV

$ wsc -er ws://ngx-http-instance-0/api/v1/streams/valid_transactions

$ curl -X GET http://ngx-http-instance-0:27017

The above curl command should result in the response It looks like you are trying to access MongoDB over HTTP on the native driver port.

To test the NGINX instance with HTTPS and 3scale integration:

$ nslookup ngx-instance-0

$ dig +noall +answer _public-secure-node-port._tcp.ngx-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _public-mdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _public-insecure-node-port._tcp.ngx-instance-0.default.svc.cluster.local SRV

$ wsc -er wss://<node-fqdn>/api/v1/streams/valid_transactions

$ curl -X GET http://<node-fqdn>:27017

The above curl command should result in the response It looks like you are trying to access MongoDB over HTTP on the native driver port.

Step 21.2: Testing Externally¶

Check the MongoDB monitoring agent on the MongoDB Cloud Manager portal to verify they are working fine.

If you are using the NGINX with HTTP support, accessing the URL http://<DNS/IP of your exposed BigchainDB service endpoint>:node-frontend-port on your browser should result in a JSON response that shows the BigchainDB server version, among other things. If you are using the NGINX with HTTPS support, use https instead of http above.

Use the Python Driver to send some transactions to the BigchainDB node and verify that your node or cluster works as expected.

Next, you can set up log analytics and monitoring, by following our templates:

Log Analytics on Azure.

How to Configure a BigchainDB Node¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page outlines the steps to set a bunch of configuration settings in your BigchainDB node. They are pushed to the Kubernetes cluster in two files, named config-map.yaml (a set of ConfigMaps) and secret.yaml (a set of Secrets). They are stored in the Kubernetes cluster’s key-value store (etcd).

Make sure you did the first four operations listed in the section titled Things Each Node Operator Must Do.

Edit vars¶

This file is located at: k8s/scripts/vars and edit the configuration parameters. That file already contains many comments to help you understand each data value, but we make some additional remarks on some of the values below.

vars.NODE_FQDN¶

FQDN for your BigchainDB node. This is the domain name used to query and access your BigchainDB node. More information can be found in our Kubernetes template overview guide.

vars.SECRET_TOKEN¶

This parameter is specific to your BigchainDB node and is used for authentication and authorization of requests to your BigchainDB node. More information can be found in our Kubernetes template overview guide.

vars.HTTPS_CERT_KEY_FILE_NAME¶

Absolute path of the HTTPS certificate chain of your domain. More information can be found in our Kubernetes template overview guide.

vars.HTTPS_CERT_CHAIN_FILE_NAME¶

Absolute path of the HTTPS certificate key of your domain. More information can be found in our Kubernetes template overview guide.

vars.MDB_ADMIN_USER and vars.MDB_ADMIN_PASSWORD¶

MongoDB admin user credentials, username and password. This user is created on the admin database with the authorization to create other users.

vars.BDB_PERSISTENT_PEERS, BDB_VALIDATORS, BDB_VALIDATORS_POWERS, BDB_GENESIS_TIME and BDB_CHAIN_ID¶

These parameters are shared across the BigchainDB network. More information about the generation of these parameters can be found at Generate All Shared BigchainDB Setup Parameters.

vars.NODE_DNS_SERVER¶

IP of Kubernetes service(kube-dns), can be retrieved using using CLI(kubectl) or k8s dashboard. This parameter is used by the Nginx gateway instance to resolve the hostnames of all the services running in the Kubernetes cluster.

# retrieval via commandline.
$ kubectl get services --namespace=kube-system -l k8s-app=kube-dns

Generate configuration¶

After populating the k8s/scripts/vars file, we need to generate all the configuration required for the BigchainDB node, for that purpose we need to execute k8s/scripts/generate_configs.sh script.

$ bash generate_configs.sh

Note

During execution the script will prompt the user for some inputs.

After successful execution, this routine will generate config-map.yaml and secret.yaml under k8s/scripts.

Deploy Your config-map.yaml and secret.yaml¶

You can deploy your edited config-map.yaml and secret.yaml files to your Kubernetes cluster using the commands:

$ kubectl apply -f config-map.yaml

$ kubectl apply -f secret.yaml

Log Analytics on Azure¶

This page describes how we use Microsoft Operations Management Suite (OMS) to collect all logs from a Kubernetes cluster, to search those logs, and to set up email alerts based on log messages. The References section (below) contains links to more detailed documentation.

There are two steps:

Setup: Create a log analytics OMS workspace and a Containers solution under that workspace.
Deploy OMS agents to your Kubernetes cluster.

Step 1: Setup¶

Step 1 can be done the web browser way or the command-line way.

The Web Browser Way¶

To create a new log analytics OMS workspace:

Go to the Azure Portal in your web browser.
Click on More services > in the lower left corner of the Azure Portal.
Type “log analytics” or similar.
Select Log Analytics from the list of options.
Click on + Add to add a new log analytics OMS workspace.
Give answers to the questions. You can call the OMS workspace anything, but use the same resource group and location as your Kubernetes cluster. The free option will suffice, but of course you can also use a paid one.

To add a “Containers solution” to that new workspace:

In Azure Portal, in the Log Analytics section, click the name of the new workspace
Click OMS Workspace.
Click OMS Portal. It should launch the OMS Portal in a new tab.
Click the Solutions Gallery tile.
Click the Containers tile.
Click Add.

The Command-Line Way¶

We’ll assume your Kubernetes cluster has a resource group named:

resource_group

and the workspace we’ll create will be named:

work_space

If you feel creative you may replace these names by more interesting ones.

$ az group deployment create --debug \
    --resource-group resource_group \
    --name "Microsoft.LogAnalyticsOMS" \
    --template-file log_analytics_oms.json \
    --parameters @log_analytics_oms.parameters.json

An example of a simple template file (--template-file):

{
  "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "sku": {
      "type": "String"
    },
    "workspaceName": {
      "type": "String"
    },
    "solutionType": {
      "type": "String"
    },
    "resources": [
      {
        "apiVersion": "2015-03-20",
        "type": "Microsoft.OperationalInsights/workspaces",
        "name": "[parameters('workspaceName')]",
        "location": "[resourceGroup().location]",
        "properties": {
          "sku": {
            "name": "[parameters('sku')]"
          }
        },
        "resources": [
          {
            "apiVersion": "2015-11-01-preview",
            "location": "[resourceGroup().location]",
            "name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
            "type": "Microsoft.OperationsManagement/solutions",
            "id": "[Concat(resourceGroup().id, '/providers/Microsoft.OperationsManagement/solutions/', parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
            "dependsOn": [
              "[concat('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
            ],
            "properties": {
              "workspaceResourceId": "[resourceId('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
            },
            "plan": {
              "publisher": "Microsoft",
              "product": "[Concat('OMSGallery/', parameters('solutionType'))]",
              "name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
              "promotionCode": ""
            }
          }
        ]
      }
    ]
  }
}

An example of the associated parameter file (--parameters):

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "sku": {
      "value": "Free"
    },
    "workspaceName": {
      "value": "work_space"
    },
    "solutionType": {
      "value": "Containers"
    }
  }
}

Step 2: Deploy the OMS Agents¶

To deploy an OMS agent, two important pieces of information are needed:

workspace id
workspace key

You can obtain the workspace id using:

$ az resource show \
    --resource-group resource_group
    --resource-type Microsoft.OperationalInsights/workspaces
    --name work_space \
    | grep customerId
"customerId": "12345678-1234-1234-1234-123456789012",

Until we figure out a way to obtain the workspace key via the command line, you can get it via the OMS Portal. To get to the OMS Portal, go to the Azure Portal and click on:

Resource Groups > (Your Kubernetes cluster’s resource group) > Log analytics (OMS) > (Name of the only item listed) > OMS Workspace > OMS Portal

(Let us know if you find a faster way.) Then see Microsoft’s instructions to obtain your workspace ID and key (via the OMS Portal).

Once you have the workspace id and key, you can include them in the following YAML file (oms-daemonset.yaml):

# oms-daemonset.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: omsagent
spec:
  template:
    metadata:
      labels:
        app: omsagent
    spec:
      containers:
      - env:
        - name: WSID
          value: <workspace_id>
        - name: KEY
          value: <workspace_key>
        image: microsoft/oms
        name: omsagent
        ports:
        - containerPort: 25225
          protocol: TCP
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock
      volumes:
      - name: docker-sock
        hostPath:
          path: /var/run/docker.sock

To deploy the OMS agents (one per Kubernetes node, i.e. one per computer), simply run the following command:

$ kubectl create -f oms-daemonset.yaml

Search the OMS Logs¶

OMS should now be getting, storing and indexing all the logs from all the containers in your Kubernetes cluster. You can search the OMS logs from the Azure Portal or the OMS Portal, but at the time of writing, there was more functionality in the OMS Portal (e.g. the ability to create an Alert based on a search).

There are instructions to get to the OMS Portal above. Once you’re in the OMS Portal, click on Log Search and enter a query. Here are some example queries:

All logging messages containing the strings “critical” or “error” (not case-sensitive):

Type=ContainerLog (critical OR error)

Note

You can filter the results even more by clicking on things in the left sidebar. For OMS Log Search syntax help, see the Log Analytics search reference.

All logging messages containing the string “error” but not “404”:

Type=ContainerLog error NOT(404)

All logging messages containing the string “critical” but not “CriticalAddonsOnly”:

Type=ContainerLog critical NOT(CriticalAddonsOnly)

All logging messages from containers running the Docker image bigchaindb/nginx_3scale:1.3, containing the string “GET” but not the strings “Go-http-client” or “runscope” (where those exclusions filter out tests by Kubernetes and Runscope):

Type=ContainerLog Image="bigchaindb/nginx_3scale:1.3" GET NOT("Go-http-client") NOT(runscope)

Note

We wrote a small Python 3 script to analyze the logs found by the above NGINX search. It’s in k8s/logging-and-monitoring/analyze.py. The docsting at the top of the script explains how to use it.

Create an Email Alert¶

Once you’re satisfied with an OMS Log Search query string, click the 🔔 Alert icon in the top menu, fill in the form, and click Save when you’re done.

Some Useful Management Tasks¶

List workspaces:

$ az resource list \
    --resource-group resource_group \
    --resource-type Microsoft.OperationalInsights/workspaces

List solutions:

$ az resource list \
    --resource-group resource_group \
    --resource-type Microsoft.OperationsManagement/solutions

Delete the containers solution:

$ az group deployment delete --debug \
    --resource-group resource_group \
    --name Microsoft.ContainersOMS

$ az resource delete \
    --resource-group resource_group \
    --resource-type Microsoft.OperationsManagement/solutions \
    --name "Containers(work_space)"

Delete the workspace:

$ az group deployment delete --debug \
    --resource-group resource_group \
    --name Microsoft.LogAnalyticsOMS

$ az resource delete \
    --resource-group resource_group \
    --resource-type Microsoft.OperationalInsights/workspaces \
    --name work_space

References¶

Configure MongoDB Cloud Manager for Monitoring¶

This document details the steps required to configure MongoDB Cloud Manager to enable monitoring of data in a MongoDB Replica Set.

Configure MongoDB Cloud Manager for Monitoring¶

Once the Monitoring Agent is up and running, open MongoDB Cloud Manager.

Click Login under MongoDB Cloud Manager and log in to the Cloud Manager.

Select the group from the dropdown box on the page.

Go to Settings and add a Preferred Hostnames entry as a regexp based on the mdb-instance-name of the nodes in your cluster. It may take up to 5 mins till this setting takes effect. You may refresh the browser window and verify whether the changes have been saved or not.

For example, for the nodes in a cluster that are named mdb-instance-0, mdb-instance-1 and so on, a regex like ^mdb-instance-[0-9]{1,2}$ is recommended.

Next, click the Deployment tab, and then the Manage Existing button.

On the Import your deployment for monitoring page, enter the hostname to be the same as the one set for mdb-instance-name in the global ConfigMap for a node. For example, if the mdb-instance-name is set to mdb-instance-0, enter mdb-instance-0 as the value in this field.

Enter the port number as 27017, with no authentication.

If you have authentication enabled, select the option to enable authentication and specify the authentication mechanism as per your deployment. The default BigchainDB Kubernetes deployment template currently supports X.509 Client Certificate as the authentication mechanism.

If you have TLS enabled, select the option to enable TLS/SSL for MongoDB connections, and click Continue. This should already be selected for you in case you selected X.509 Client Certificate above.

Wait a minute or two for the deployment to be found and then click the Continue button again.

Verify that you see your process on the Cloud Manager UI. It should look something like this:

Click Continue.

Verify on the UI that data is being sent by the monitoring agent to the Cloud Manager. It may take upto 5 minutes for data to appear on the UI.

How to Install & Configure Easy-RSA¶

We use Easy-RSA version 3, a wrapper over complex openssl commands. Easy-RSA is available on GitHub and licensed under GPLv2.

Step 1: Install Easy-RSA Dependencies¶

The only dependency for Easy-RSA v3 is openssl, which is available from the openssl package on Ubuntu and other Debian-based operating systems, i.e. you can install it using:

sudo apt-get update

sudo apt-get install openssl

Step 2: Install Easy-RSA¶

Make sure you’re in the directory where you want Easy-RSA to live, then download it and extract it within that directory:

wget https://github.com/OpenVPN/easy-rsa/archive/3.0.1.tar.gz

tar xzvf 3.0.1.tar.gz

rm 3.0.1.tar.gz

There should now be a directory named easy-rsa-3.0.1 in your current directory.

Step 3: Customize the Easy-RSA Configuration¶

We now create a config file named vars by copying the existing vars.example file and then editing it. You should change the country, province, city, org and email to the correct values for your organisation. (Note: The country, province, city, org and email are part of the Distinguished Name (DN).) The comments in the file explain what each of the variables mean.

cd easy-rsa-3.0.1/easyrsa3

cp vars.example vars

echo 'set_var EASYRSA_DN "org"' >> vars
echo 'set_var EASYRSA_KEY_SIZE 4096' >> vars

echo 'set_var EASYRSA_REQ_COUNTRY "DE"' >> vars
echo 'set_var EASYRSA_REQ_PROVINCE "Berlin"' >> vars
echo 'set_var EASYRSA_REQ_CITY "Berlin"' >> vars
echo 'set_var EASYRSA_REQ_ORG "BigchainDB GmbH"' >> vars
echo 'set_var EASYRSA_REQ_OU "IT"' >> vars
echo 'set_var EASYRSA_REQ_EMAIL "devs@bigchaindb.com"' >> vars

Note: Later, when building a CA or generating a certificate signing request, you will be prompted to enter a value for the OU (or to accept the default). You should change the default OU from IT to one of the following, as appropriate: ROOT-CA, MongoDB-Instance, BigchainDB-Instance, MongoDB-Mon-Instance or MongoDB-Backup-Instance. To understand why, see the MongoDB Manual. There are reminders to do this in the relevant docs.

Step 4: Maybe Edit x509-types/server¶

Warning

Only do this step if you are setting up a self-signed CA.

Edit the file x509-types/server and change extendedKeyUsage = serverAuth to extendedKeyUsage = serverAuth,clientAuth. See the MongoDB documentation about x.509 authentication to understand why.

Kubernetes Template: Upgrade all Software in a BigchainDB Node¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page outlines how to upgrade all the software associated with a BigchainDB node running on Kubernetes, including host operating systems, Docker, Kubernetes, and BigchainDB-related software.

Upgrade Host OS, Docker and Kubernetes¶

Some Kubernetes installation & management systems can do full or partial upgrades of host OSes, Docker, or Kubernetes, e.g. Tectonic, Rancher, and Kubo. Consult the documentation for your system.

Azure Container Service (ACS). On Dec. 15, 2016, a Microsoft employee wrote: “In the coming months we [the Azure Kubernetes team] will be building managed updates in the ACS service.” At the time of writing, managed updates were not yet available, but you should check the latest ACS documentation to see what’s available now. Also at the time of writing, ACS only supported Ubuntu as the host (master and agent) operating system. You can upgrade Ubuntu and Docker on Azure by SSHing into each of the hosts, as documented on another page.

In general, you can SSH to each host in your Kubernetes Cluster to update the OS and Docker.

Note

Once you are in an SSH session with a host, the docker info command is a handy way to detemine the host OS (including version) and the Docker version.

When you want to upgrade the software on a Kubernetes node, you should “drain” the node first, i.e. tell Kubernetes to gracefully terminate all pods on the node and mark it as unscheduleable (so no new pods get put on the node during its downtime).

kubectl drain $NODENAME

There are more details in the Kubernetes docs, including instructions to make the node scheduleable again.

To manually upgrade the host OS, see the docs for that OS.

To manually upgrade Docker, see the Docker docs.

To manually upgrade all Kubernetes software in your Kubernetes cluster, see the Kubernetes docs.

Kubernetes Template: Deploying a BigchainDB network¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page describes how to deploy a static BigchainDB + Tendermint network.

If you want to deploy a stand-alone BigchainDB node in a BigchainDB cluster, or a stand-alone BigchainDB node, then see the page about that.

We can use this guide to deploy a BigchainDB network in the following scenarios:

Single Azure Kubernetes Site.
Multiple Azure Kubernetes Sites (Geographically dispersed).

Terminology Used¶

BigchainDB node is a set of Kubernetes components that join together to form a BigchainDB single node. Please refer to the architecture diagram for more details.

BigchainDB network will refer to a collection of nodes working together to form a network.

Below, we refer to multiple files by their directory and filename, such as bigchaindb/bigchaindb-ext-conn-svc.yaml. Those files are located in the bigchaindb/bigchaindb repository on GitHub in the k8s/ directory. Make sure you’re getting those files from the appropriate Git branch on GitHub, i.e. the branch for the version of BigchainDB that your BigchainDB cluster is using.

Note

This deployment strategy is currently used for testing purposes only, operated by a single stakeholder or tightly coupled stakeholders.

Note

Currently, we only support a static set of participants in the network. Once a BigchainDB network is started with a certain number of validators and a genesis file. Users cannot add new validator nodes dynamically. You can track the progress of this funtionality on our github repository.

Prerequisites¶

The deployment methodology is similar to one covered with Kubernetes Template: Deploy a Single BigchainDB Node, but we need to tweak some configurations depending on your choice of deployment.

The operator needs to follow some consistent naming convention for all the components covered here.

Lets assume we are deploying a 4 node cluster, your naming conventions could look like this:

{
  "MongoDB": [
    "mdb-instance-1",
    "mdb-instance-2",
    "mdb-instance-3",
    "mdb-instance-4"
  ],
  "BigchainDB": [
    "bdb-instance-1",
    "bdb-instance-2",
    "bdb-instance-3",
    "bdb-instance-4"
  ],
  "NGINX": [
    "ngx-instance-1",
    "ngx-instance-2",
    "ngx-instance-3",
    "ngx-instance-4"
  ],
  "OpenResty": [
    "openresty-instance-1",
    "openresty-instance-2",
    "openresty-instance-3",
    "openresty-instance-4"
  ],
  "MongoDB_Monitoring_Agent": [
    "mdb-mon-instance-1",
    "mdb-mon-instance-2",
    "mdb-mon-instance-3",
    "mdb-mon-instance-4"
  ]
}

Note

Blockchain Genesis ID and Time will be shared across all nodes.

Edit config.yaml and secret.yaml¶

Make N(number of nodes) copies of configuration/config-map.yaml and configuration/secret.yaml.

# For config-map.yaml
config-map-node-1.yaml
config-map-node-2.yaml
config-map-node-3.yaml
config-map-node-4.yaml

# For secret.yaml
secret-node-1.yaml
secret-node-2.yaml
secret-node-3.yaml
secret-node-4.yaml

Edit the data values as described in this document, based on the naming convention described above.

Only for single site deployments: Since all the configuration files use the same ConfigMap and Secret Keys i.e. metadata.name -> vars, bdb-config and tendermint-config and metadata.name -> cloud-manager-credentials, mdb-certs, mdb-mon-certs, bdb-certs, https-certs, three-scale-credentials, ca-auth respectively, each file will overwrite the configuration of the previously deployed one. We want each node to have its own unique configurations. One way to go about it is that, using the naming convention above we edit the ConfigMap and Secret keys.

# For config-map-node-1.yaml
metadata.name: vars -> vars-node-1
metadata.name: bdb-config -> bdb-config-node-1
metadata.name: tendermint-config -> tendermint-config-node-1

# For secret-node-1.yaml
metadata.name: cloud-manager-credentials -> cloud-manager-credentials-node-1
metadata.name: mdb-certs -> mdb-certs-node-1
metadata.name: mdb-mon-certs -> mdb-mon-certs-node-1
metadata.name: bdb-certs -> bdb-certs-node-1
metadata.name: https-certs -> https-certs-node-1
metadata.name: threescale-credentials -> threescale-credentials-node-1
metadata.name: ca-auth -> ca-auth-node-1

# Repeat for the remaining files.

Deploy all your configuration maps and secrets.

kubectl apply -f configuration/config-map-node-1.yaml
kubectl apply -f configuration/config-map-node-2.yaml
kubectl apply -f configuration/config-map-node-3.yaml
kubectl apply -f configuration/config-map-node-4.yaml
kubectl apply -f configuration/secret-node-1.yaml
kubectl apply -f configuration/secret-node-2.yaml
kubectl apply -f configuration/secret-node-3.yaml
kubectl apply -f configuration/secret-node-4.yaml

Note

Similar to what we did, with config-map.yaml and secret.yaml i.e. indexing them per node, we have to do the same for each Kubernetes component i.e. Services, StorageClasses, PersistentVolumeClaims, StatefulSets, Deployments etc.

# For Services
*-node-1-svc.yaml
*-node-2-svc.yaml
*-node-3-svc.yaml
*-node-4-svc.yaml

# For StorageClasses
*-node-1-sc.yaml
*-node-2-sc.yaml
*-node-3-sc.yaml
*-node-4-sc.yaml

# For PersistentVolumeClaims
*-node-1-pvc.yaml
*-node-2-pvc.yaml
*-node-3-pvc.yaml
*-node-4-pvc.yaml

# For StatefulSets
*-node-1-ss.yaml
*-node-2-ss.yaml
*-node-3-ss.yaml
*-node-4-ss.yaml

# For Deployments
*-node-1-dep.yaml
*-node-2-dep.yaml
*-node-3-dep.yaml
*-node-4-dep.yaml

Single Site: Single Azure Kubernetes Cluster¶

For the deployment of a BigchainDB network under a single cluster, we need to replicate the deployment steps for each node N number of times, N being the number of participants in the network.

In our Kubernetes deployment template for a single BigchainDB node, we covered the basic configurations settings here.

Since, we index the ConfigMap and Secret Keys for the single site deployment, we need to update all the Kubernetes components to reflect the corresponding changes i.e. For each Kubernetes Service, StatefulSet, PersistentVolumeClaim, Deployment, and StorageClass, we need to update the respective *.yaml file and update the ConfigMapKeyRef.name OR secret.secretName.

Example¶

Assuming we are deploying the MongoDB StatefulSet for Node 1. We need to update the mongo-node-1-ss.yaml and update the corresponding ConfigMapKeyRef.name or secret.secretNames.

########################################################################
# This YAML file desribes a StatefulSet with a service for running and #
# exposing a MongoDB instance.                                         #
# It depends on the configdb and db k8s pvc.                           #
########################################################################

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mdb-instance-0-ss
  namespace: default
spec:
  serviceName: mdb-instance-0
  replicas: 1
  template:
    metadata:
      name: mdb-instance-0-ss
      labels:
        app: mdb-instance-0-ss
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongodb
        image: bigchaindb/mongodb:3.2
        imagePullPolicy: IfNotPresent
        env:
        - name: MONGODB_FQDN
          valueFrom:
            configMapKeyRef:
             name: vars-1 # Changed from ``vars``
             key: mdb-instance-name
        - name: MONGODB_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MONGODB_PORT
          valueFrom:
            configMapKeyRef:
             name: vars-1 # Changed from ``vars``
             key: mongodb-backend-port
        - name: STORAGE_ENGINE_CACHE_SIZE
          valueFrom:
            configMapKeyRef:
             name: vars-1 # Changed from ``vars``
             key: storage-engine-cache-size
        args:
        - --mongodb-port
        - $(MONGODB_PORT)
        - --mongodb-key-file-path
        - /etc/mongod/ssl/mdb-instance.pem
        - --mongodb-ca-file-path
        - /etc/mongod/ca/ca.pem
        - --mongodb-crl-file-path
        - /etc/mongod/ca/crl.pem
        - --mongodb-fqdn
        - $(MONGODB_FQDN)
        - --mongodb-ip
        - $(MONGODB_POD_IP)
        - --storage-engine-cache-size
        - $(STORAGE_ENGINE_CACHE_SIZE)
        securityContext:
          capabilities:
            add:
            - FOWNER
        ports:
        - containerPort: "<mongodb-backend-port from ConfigMap>"
          protocol: TCP
          name: mdb-api-port
        volumeMounts:
        - name: mdb-db
          mountPath: /data/db
        - name: mdb-configdb
          mountPath: /data/configdb
        - name: mdb-certs
          mountPath: /etc/mongod/ssl/
          readOnly: true
        - name: ca-auth
          mountPath: /etc/mongod/ca/
          readOnly: true
        resources:
          limits:
            cpu: 200m
            memory: 5G
        livenessProbe:
          tcpSocket:
            port: mdb-api-port
          initialDelaySeconds: 15
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 15
          timeoutSeconds: 10
      restartPolicy: Always
      volumes:
      - name: mdb-db
        persistentVolumeClaim:
          claimName: mongo-db-claim-1 # Changed from ``mongo-db-claim``
      - name: mdb-configdb
        persistentVolumeClaim:
          claimName: mongo-configdb-claim-1 # Changed from ``mongo-configdb-claim``
      - name: mdb-certs
        secret:
          secretName: mdb-certs-1 # Changed from ``mdb-certs``
          defaultMode: 0400
      - name: ca-auth
        secret:
          secretName: ca-auth-1 # Changed from ``ca-auth``
          defaultMode: 0400

The above example is meant to be repeated for all the Kubernetes components of a BigchainDB node.

nginx-http/nginx-http-node-X-svc.yaml or nginx-https/nginx-https-node-X-svc.yaml
nginx-http/nginx-http-node-X-dep.yaml or nginx-https/nginx-https-node-X-dep.yaml
mongodb/mongodb-node-X-svc.yaml
mongodb/mongodb-node-X-sc.yaml
mongodb/mongodb-node-X-pvc.yaml
mongodb/mongodb-node-X-ss.yaml
bigchaindb/bigchaindb-node-X-svc.yaml
bigchaindb/bigchaindb-node-X-sc.yaml
bigchaindb/bigchaindb-node-X-pvc.yaml
bigchaindb/bigchaindb-node-X-ss.yaml
nginx-openresty/nginx-openresty-node-X-svc.yaml
nginx-openresty/nginx-openresty-node-X-dep.yaml

Multi Site: Multiple Azure Kubernetes Clusters¶

For the multi site deployment of a BigchainDB network with geographically dispersed nodes, we need to replicate the deployment steps for each node N number of times, N being the number of participants in the network.

The operator needs to follow a consistent naming convention which has already discussed in this document.

Note

Assuming we are using independent Kubernetes clusters, the ConfigMap and Secret Keys do not need to be updated unlike Single Site: Single Azure Kubernetes Cluster, and we also do not need to update corresponding ConfigMap/Secret imports in the Kubernetes components.

Deploy Kubernetes Services¶

Deploy the following services for each node by following the naming convention described above:

Only for multi site deployments¶

We need to make sure that clusters are able to talk to each other i.e. specifically the communication between the BigchainDB peers. Set up networking between the clusters using Kubernetes Services.

Assuming we have a BigchainDB instance bigchaindb-instance-1 residing in Azure data center location westeurope and we want to connect to bigchaindb-instance-2, bigchaindb-instance-3, and bigchaindb-instance-4 located in Azure data centers eastus, centralus and westus, respectively. Unless you already have explicitly set up networking for bigchaindb-instance-1 to communicate with bigchaindb-instance-2/3/4 and vice versa, we will have to add a Kubernetes Service in each cluster to accomplish this goal in order to set up a BigchainDB P2P network. It is similar to ensuring that there is a CNAME record in the DNS infrastructure to resolve bigchaindb-instance-X to the host where it is actually available. We can do this in Kubernetes using a Kubernetes Service of type ExternalName.

This configuration is located in the file bigchaindb/bigchaindb-ext-conn-svc.yaml.
Set the name of the metadata.name to the host name of the BigchainDB instance you are trying to connect to. For instance if you are configuring this service on cluster with bigchaindb-instance-1 then the metadata.name will be bigchaindb-instance-2 and vice versa.
Set spec.ports.port[0] to the tm-p2p-port from the ConfigMap for the other cluster.
Set spec.ports.port[1] to the tm-rpc-port from the ConfigMap for the other cluster.
Set spec.externalName to the FQDN mapped to NGINX Public IP of the cluster you are trying to connect to. For more information about the FQDN please refer to: Assign DNS name to NGINX Public IP.

Note

This operation needs to be replicated n-1 times per node for a n node cluster, with the respective FQDNs we need to communicate with.

If you are not the system administrator of the cluster, you have to get in touch with the system administrator/s of the other n-1 clusters and share with them your instance name (bigchaindb-instance-name in the ConfigMap) and the FQDN of the NGINX instance acting as Gateway(set in: Assign DNS name to NGINX Public IP).

Start NGINX Kubernetes deployments¶

Start the NGINX deployment that serves as a Gateway for each node by following the naming convention described above and referring to the following instructions:

Start the NGINX Kubernetes Deployment.

Deploy Kubernetes StorageClasses for MongoDB and BigchainDB¶

Deploy the following StorageClasses for each node by following the naming convention described above:

Deploy Kubernetes PersistentVolumeClaims for MongoDB and BigchainDB¶

Deploy the following services for each node by following the naming convention described above:

Deploy MongoDB Kubernetes StatefulSet¶

Deploy the MongoDB StatefulSet (standalone MongoDB) for each node by following the naming convention described above: and referring to the following section:

Start a Kubernetes StatefulSet for MongoDB.

Configure Users and Access Control for MongoDB¶

Configure users and access control for each MongoDB instance in the network by referring to the following section:

Configure Users and Access Control for MongoDB.

Start Kubernetes StatefulSet for BigchainDB¶

Start the BigchainDB Kubernetes StatefulSet for each node by following the naming convention described above and referring to the following instructions:

Start a Kubernetes Deployment for BigchainDB.

Start Kubernetes Deployment for MongoDB Monitoring Agent¶

Start the MongoDB monitoring agent Kubernetes deployment for each node by following the naming convention described above and referring to the following instructions:

Start a Kubernetes Deployment for MongoDB Monitoring Agent.

Start Kubernetes Deployment for OpenResty¶

Start the OpenResty Kubernetes deployment for each node by following the naming convention described above and referring to the following instructions:

Start a Kubernetes Deployment for OpenResty.

Verify and Test¶

Verify and test your setup by referring to the following instructions:

Verify the BigchainDB Node Setup.

Walkthrough: Deploy a Kubernetes Cluster on Azure using Tectonic by CoreOS¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

A BigchainDB node can be run inside a Kubernetes cluster. This page describes one way to deploy a Kubernetes cluster on Azure using Tectonic. Tectonic helps in easier cluster management of Kubernetes clusters.

If you would rather use Azure Container Service to manage Kubernetes Clusters, please read our guide for that.

Step 1: Prerequisites for Deploying Tectonic Cluster¶

Get an Azure account. Refer to this step in our docs.

Create an SSH Key pair for the new Tectonic cluster. Refer to this step in our docs.

Step 2: Get a Tectonic Subscription¶

CoreOS offers Tectonic for free for up to 10 nodes.

Sign up for an account here if you do not have one already and get a license for 10 nodes.

Login to your account, go to Overview > Your Account and save the CoreOS License and the Pull Secret to your local machine.

Step 3: Deploy the cluster on Azure¶

The latest instructions for deployment can be found here.

The following points suggests some customizations for a BigchainDB deployment when following the steps above:

Set the CLUSTER variable to the name of the cluster. Also note that the cluster will be deployed in a resource group named tectonic-cluster-CLUSTER.
Set the tectonic_base_domain to "" if you want to use Azure managed DNS. You will be assigned a cloudapp.azure.com sub-domain by default and you can skip the Configuring Azure DNS section from the Tectonic installation guide.
Set the tectonic_cl_channel to "stable" unless you want to experiment or test with the latest release.
Set the tectonic_cluster_name to the CLUSTER variable defined in the step above.
Set the tectonic_license_path and tectonic_pull_secret_path to the location where you have stored the tectonic-license.txt and the config.json files downloaded in the previous step.
Set the tectonic_etcd_count to "3", so that you have a multi-node etcd cluster that can tolerate a single node failure.
Set the tectonic_etcd_tls_enabled to "true" as this will enable TLS connectivity between the etcd nodes and their clients.
Set the tectonic_master_count to "3" so that you cane tolerate a single master failure.
Set the tectonic_worker_count to "2".
Set the tectonic_azure_location to "westeurope" if you want to host the cluster in Azure’s westeurope datacenter.
Set the tectonic_azure_ssh_key to the path of the public key created in the previous step.
We recommend setting up or using a CA(Certificate Authority) to generate Tectonic Console’s server certificate(s) and adding it to your trusted authorities on the client side, accessing the Tectonic Console i.e. Browser. If you already have a CA(self-signed or otherwise), Set the tectonic_ca_cert and tectonic_ca_key configurations with the content of PEM-encoded certificate and key files, respectively. For more information about, how to set up a self-signed CA, Please refer to How to Set up self-signed CA.
Note that the tectonic_azure_client_secret is the same as the ARM_CLIENT_SECRET.
Note that the URL for the Tectonic console using these settings will be the cluster name set in the configutation file, the datacenter name and cloudapp.azure.com. For example, if you named your cluster as test-cluster and specified the datacenter as westeurope, the Tectonic console will be available at test-cluster.westeurope.cloudapp.azure.com.
Note that, if you do not specify tectonic_ca_cert, a CA certificate will be generated automatically and you will encounter the untrusted certificate message on your client(Browser), when accessing the Tectonic Console.

Step 4: Configure kubectl¶

Refer to this tutorial for instructions on how to download the kubectl configuration files for your cluster.
Set the KUBECONFIG environment variable to make kubectl use the new config file along with the existing configuration.

$ export KUBECONFIG=$HOME/.kube/config:/path/to/config/kubectl-config

# OR to only use the new configuration, try

$ export KUBECONFIG=/path/to/config/kubectl-config

Next, you can follow one of our following deployment templates:

Kubernetes Template: Deploy a Single BigchainDB Node.

Tectonic References¶

Cluster Troubleshooting¶

This page describes some basic issues we have faced while deploying and operating the cluster.

1. MongoDB Restarts¶

We define the following in the mongo-ss.yaml file:

resources:
  limits:
    cpu: 200m
    memory: 5G

When the MongoDB cache occupies a memory greater than 5GB, it is terminated by the kubelet. This can usually be verified by logging in to the worker node running MongoDB container and looking at the syslog (the journalctl command should usually work).

This issue is resolved in PR #1757.

2. 502 Bad Gateway Error on Runscope Tests¶

It means that NGINX could not find the appropriate backed to forward the requests to. This typically happens when:

MongoDB goes down (as described above) and BigchainDB, after trying for BIGCHAINDB_DATABASE_MAXTRIES times, gives up. The Kubernetes BigchainDB Deployment then restarts the BigchainDB pod.
BigchainDB crashes for some reason. We have seen this happen when updating BigchainDB from one version to the next. This usually means the older connections to the service gets disconnected; retrying the request one more time, forwards the connection to the new instance and succeed.

3. Service Unreachable¶

Communication between Kubernetes Services and Deployments fail in v1.6.6 and before due to a trivial key lookup error for non-existent services in the kubelet. This error can be reproduced by restarting any public facing (that is, services using the cloud load balancer) Kubernetes services, and watching the kube-proxy failure in its logs. The solution to this problem is to restart kube-proxy on the affected worker/agent node. Login to the worker node and run:

docker stop `docker ps | grep k8s_kube-proxy | cut -d" " -f1`

docker logs -f `docker ps | grep k8s_kube-proxy | cut -d" " -f1`

This issue is fixed in Kubernetes v1.7.

4. Single Disk Attached to Multiple Mountpoints in a Container¶

This is currently the issue faced in one of the clusters and being debugged by the support team at Microsoft.

The issue was first seen on August 29, 2017 on the Test Network and has been logged in the Azure/acs-engine repo on GitHub.

This is apparently fixed in Kubernetes v1.7.2 which include a new disk driver, but is yet to tested by us.

5. MongoDB Monitoring Agent throws a dial error while connecting to MongoDB¶

You might see something similar to this in the MongoDB Monitoring Agent logs:

Failure dialing host without auth. Err: `no reachable servers`
    at monitoring-agent/components/dialing.go:278
    at monitoring-agent/components/dialing.go:116
    at monitoring-agent/components/dialing.go:213
    at src/runtime/asm_amd64.s:2086

The first thing to check is if the networking is set up correctly. You can use the (maybe using the toolbox container).

If everything looks fine, it might be a problem with the Preferred Hostnames setting in MongoDB Cloud Manager. If you do need to change the regular expression, ensure that it is correct and saved properly (maybe try refreshing the MongoDB Cloud Manager web page to see if the setting sticks).

Once you update the regular expression, you will need to remove the deployment and add it again for the Monitoring Agent to discover and connect to the MongoDB instance correctly.

More information about this configuration is provided in this document.

6. Create a Persistent Volume from existing Azure disk storage Resource¶

When deleting a k8s cluster, all dynamically-created PVs are deleted, along with the underlying Azure storage disks (so those can’t be used in a new cluster). resources are also deleted thus cannot be used in a new cluster. This workflow will preserve the Azure storage disks while deleting the k8s cluster and re-use the same disks on a new cluster for MongoDB persistent storage without losing any data.

The template to create two PVs for MongoDB Stateful Set (One for MongoDB data store and the other for MongoDB config store) is located at mongodb/mongo-pv.yaml.

You need to configure diskName and diskURI in mongodb/mongo-pv.yaml file. You can get these values by logging into your Azure portal and going to Resource Groups and click on your relevant resource group. From the list of resources click on the storage account resource and click the container (usually named as vhds) that contains storage disk blobs that are available for PVs. Click on the storage disk file that you wish to use for your PV and you will be able to see NAME and URL parameters which you can use for diskName and diskURI values in your template respectively and run the following command to create PVs:

$ kubectl --context <context-name> apply -f mongodb/mongo-pv.yaml

Note

Please make sure the storage disks you are using are not already being used by any other PVs. To check the existing PVs in your cluster, run the following command to get PVs and Storage disk file mapping.

$ kubectl --context <context-name> get pv --output yaml

Architecture of a BigchainDB Node Running in a Kubernetes Cluster¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Simple Deployment Template. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

If you deploy a BigchainDB node into a Kubernetes cluster as described in these docs, it will include:

NGINX, OpenResty, BigchainDB, MongoDB and Tendermint Kubernetes Services.
NGINX, OpenResty, BigchainDB and MongoDB Monitoring Agent Kubernetes Deployments.
MongoDB and Tendermint Kubernetes StatefulSets.
Third party services like 3scale, MongoDB Cloud Manager and the Azure Operations Management Suite.

BigchainDB Node¶

../_images/aafig-9a592ed24a98d20934bfe610d93b536495018d4e.svg

Note

The arrows in the diagram represent the client-server communication. For example, A–>B implies that A initiates the connection to B. It does not represent the flow of data; the communication channel is always fully duplex.

NGINX: Entrypoint and Gateway¶

We use an NGINX as HTTP proxy on port 443 (configurable) at the cloud entrypoint for:

Rate Limiting: We configure NGINX to allow only a certain number of requests (configurable) which prevents DoS attacks.
HTTPS Termination: The HTTPS connection does not carry through all the way to BigchainDB and terminates at NGINX for now.
Request Routing: For HTTPS connections on port 443 (or the configured BigchainDB public api port), the connection is proxied to:
1. OpenResty Service if it is a POST request.
2. BigchainDB Service if it is a GET request.

We use an NGINX TCP proxy on port 27017 (configurable) at the cloud entrypoint for:

Rate Limiting: We configure NGINX to allow only a certain number of requests (configurable) which prevents DoS attacks.
Request Routing: For connections on port 27017 (or the configured MongoDB public api port), the connection is proxied to the MongoDB Service.

OpenResty: API Management, Authentication and Authorization¶

We use OpenResty to perform authorization checks with 3scale using the app_id and app_key headers in the HTTP request.

OpenResty is NGINX plus a bunch of other components. We primarily depend on the LuaJIT compiler to execute the functions to authenticate the app_id and app_key with the 3scale backend.

MongoDB: Standalone¶

We use MongoDB as the backend database for BigchainDB.

We achieve security by avoiding DoS attacks at the NGINX proxy layer and by ensuring that MongoDB has TLS enabled for all its connections.

Tendermint: BFT consensus engine¶

We use Tendermint as the backend consensus engine for BFT replication of BigchainDB. In a multi-node deployment, Tendermint nodes/peers communicate with each other via the public ports exposed by the NGINX gateway.

We use port 9986 (configurable) to allow tendermint nodes to access the public keys of the peers and port 26656 (configurable) for the rest of the communications between the peers.

Release Notes¶

You can find a list of all BigchainDB Server releases and release notes on GitHub at:

https://github.com/bigchaindb/bigchaindb/releases

The CHANGELOG.md file contains much the same information, but it also has notes about what to expect in the next release.

We also have a roadmap document in ROADMAP.md.

Code Reference¶

This section contains auto-generated documentation of various functions, classes and methods in the BigchainDB Server code, based on Python docstrings in the code itself.

Warning

While we try to keep docstrings accurate, if you want to know for sure what the code does, then you have to read the code itself.

The BigchainDB Class¶

class bigchaindb.BigchainDB(connection=None)[source]¶

Bigchain API

Create, read, sign, write transactions to the database

post_transaction(transaction, mode)[source]¶: Submit a valid transaction to the mempool.

write_transaction(transaction, mode)[source]¶: Submit a valid transaction to the mempool.

update_utxoset(transaction)[source]¶

Update the UTXO set given transaction. That is, remove the outputs that the given transaction spends, and add the outputs that the given transaction creates.

Parameters:	transaction (`Transaction`) – A new transaction incoming into the system for which the UTXO set needs to be updated.

store_unspent_outputs(*unspent_outputs)[source]¶

Store the given unspent_outputs (utxos).

Parameters:	*unspent_outputs (`tuple` of `dict`) – Variable length tuple or list of unspent outputs.

get_utxoset_merkle_root()[source]¶

Returns the merkle root of the utxoset. This implies that the utxoset is first put into a merkle tree.

For now, the merkle tree and its root will be computed each time. This obviously is not efficient and a better approach that limits the repetition of the same computation when unnecesary should be sought. For instance, future optimizations could simply re-compute the branches of the tree that were affected by a change.

The transaction hash (id) and output index should be sufficient to uniquely identify a utxo, and consequently only that information from a utxo record is needed to compute the merkle root. Hence, each node of the merkle tree should contain the tuple (txid, output_index).

Important

The leaves of the tree will need to be sorted in some kind of lexicographical order.

Returns:	Merkle root in hexadecimal form.
Return type:	str

get_unspent_outputs()[source]¶

Get the utxoset.

Returns:	generator of unspent_outputs.

delete_unspent_outputs(*unspent_outputs)[source]¶

Deletes the given unspent_outputs (utxos).

Parameters:	*unspent_outputs (`tuple` of `dict`) – Variable length tuple or list of unspent outputs.

get_transactions_filtered(asset_id, operation=None, last_tx=None)[source]¶: Get a list of transactions filtered on some criteria

get_outputs_filtered(owner, spent=None)[source]¶

Get a list of output links filtered on some criteria

Parameters:	owner (str) – base58 encoded public_key. spent (bool) – If `True` return only the spent outputs. If `False` return only unspent outputs. If spent is not specified (`None`) return all outputs.
Returns:	list of `txid` s and `output` s pointing to another transaction’s condition
Return type:	`list` of TransactionLink

store_block(block)[source]¶: Create a new block.

get_latest_block()[source]¶: Get the block with largest height.

get_block(block_id)[source]¶

Get the block with the specified block_id.

Returns the block corresponding to block_id or None if no match is found.

Parameters:	block_id (int) – block id of the block to get.

get_block_containing_tx(txid)[source]¶

Retrieve the list of blocks (block ids) containing a: transaction with transaction id txid

Parameters:	txid (str) – transaction id of the transaction to query
Returns:	Block id list (list(int))

validate_transaction(tx, current_transactions=[])[source]¶: Validate a transaction against the current status of the database.

text_search(search, *, limit=0, table='assets')[source]¶

Return an iterator of assets that match the text search

Parameters:	search (str) – Text search string to query the text index limit (int, optional) – Limit the number of returned documents.
Returns:	An iterator of assets that match the text search.
Return type:	iter

get_assets(asset_ids)[source]¶

Return a list of assets that match the asset_ids

Parameters:	asset_ids (`list` of `str`) – A list of asset_ids to retrieve from the database.
Returns:	The list of assets returned from the database.
Return type:	list

get_metadata(txn_ids)[source]¶

Return a list of metadata that match the transaction ids (txn_ids)

Parameters:	txn_ids (`list` of `str`) – A list of txn_ids to retrieve from the database.
Returns:	The list of metadata returned from the database.
Return type:	list

store_validator_set(height, validators)[source]¶: Store validator set at a given height. NOTE: If the validator set already exists at that height then an exception will be raised.

migrate_abci_chain()[source]¶

Generate and record a new ABCI chain ID. New blocks are not accepted until we receive an InitChain ABCI request with the matching chain ID and validator set.

Chain ID is generated based on the current chain and height. chain-X => chain-X-migrated-at-height-5. chain-X-migrated-at-height-5 => chain-X-migrated-at-height-21.

If there is no known chain (we are at genesis), the function returns.

Database Backend Interfaces¶

Generic backend database interfaces expected by BigchainDB.

The interfaces in this module allow BigchainDB to be agnostic about its database backend. One can configure BigchainDB to use different databases as its data store by setting the database.backend property in the configuration or the BIGCHAINDB_DATABASE_BACKEND environment variable.

Generic Interfaces¶

`bigchaindb.backend.connection`¶

bigchaindb.backend.connection.connect(backend=None, host=None, port=None, name=None, max_tries=None, connection_timeout=None, replicaset=None, ssl=None, login=None, password=None, ca_cert=None, certfile=None, keyfile=None, keyfile_passphrase=None, crlfile=None)[source]¶

Create a new connection to the database backend.

All arguments default to the current configuration’s values if not given.

Parameters:	backend (str) – the name of the backend to use. host (str) – the host to connect to. port (int) – the port to connect to. name (str) – the name of the database to use. replicaset (str) – the name of the replica set (only relevant for MongoDB connections).
Returns:	An instance of `Connection` based on the given (or defaulted) `backend`.
Raises:	`ConnectionError` – If the connection to the database fails. `ConfigurationError` – If the given (or defaulted) `backend` is not supported or could not be loaded. `AuthenticationError` – If there is a OperationFailure due to Authentication failure after connecting to the database.

class bigchaindb.backend.connection.Connection(host=None, port=None, dbname=None, connection_timeout=None, max_tries=None, **kwargs)[source]¶

Connection class interface.

All backend implementations should provide a connection class that inherits from and implements this class.

__init__(host=None, port=None, dbname=None, connection_timeout=None, max_tries=None, **kwargs)[source]¶

Create a new Connection instance.

Parameters:

host (str) – the host to connect to.
port (int) – the port to connect to.
dbname (str) – the name of the database to use.
connection_timeout (int, optional) – the milliseconds to wait until timing out the database connection attempt. Defaults to 5000ms.
max_tries (int, optional) – how many tries before giving up, if 0 then try forever. Defaults to 3.
**kwargs – arbitrary keyword arguments provided by the configuration’s database settings

run(query)[source]¶

Run a query.

Parameters:	query – the query to run
Raises:	`DuplicateKeyError` – If the query fails because of a duplicate key constraint. `OperationFailure` – If the query fails for any other reason. `ConnectionError` – If the connection to the database fails.

connect()[source]¶

Try to connect to the database.

Raises:	`ConnectionError` – If the connection to the database fails.

`bigchaindb.backend.query`¶

Query interfaces for backends.

bigchaindb.backend.query.store_asset(connection, asset)[source]¶

Write an asset to the asset table.

Parameters:	asset (dict) – the asset.
Returns:	The result of the operation.

bigchaindb.backend.query.store_assets(connection, assets)[source]¶

Write a list of assets to the assets table.

Parameters:	assets (list) – a list of assets to write.
Returns:	The database response.

bigchaindb.backend.query.store_metadatas(connection, metadata)[source]¶

Write a list of metadata to metadata table.

Parameters:	metadata (list) – list of metadata.
Returns:	The result of the operation.

bigchaindb.backend.query.store_transactions(connection, signed_transactions)[source]¶: Store the list of transactions.

bigchaindb.backend.query.get_transaction(connection, transaction_id)[source]¶

Get a transaction from the transactions table.

Parameters:	transaction_id (str) – the id of the transaction.
Returns:	The result of the operation.

bigchaindb.backend.query.get_transactions(connection, transaction_ids)[source]¶

Get transactions from the transactions table.

Parameters:	transaction_ids (list) – list of transaction ids to fetch
Returns:	The result of the operation.

bigchaindb.backend.query.get_asset(connection, asset_id)[source]¶

Get a transaction from the transactions table.

Parameters:	asset_id (str) – the id of the asset
Returns:	The result of the operation.

bigchaindb.backend.query.get_spent(connection, transaction_id, condition_id)[source]¶

Check if a txid was already used as an input.

A transaction can be used as an input for another transaction. Bigchain needs to make sure that a given txid is only used once.

Parameters:	transaction_id (str) – The id of the transaction. condition_id (int) – The index of the condition in the respective transaction.
Returns:	The transaction that used the txid as an input else None

bigchaindb.backend.query.get_spending_transactions(connection, inputs)[source]¶

Return transactions which spend given inputs

Parameters:	inputs (list) – list of {txid, output}
Returns:	Iterator of (block_ids, transaction) for transactions that spend given inputs.

bigchaindb.backend.query.get_owned_ids(connection, owner)[source]¶

Retrieve a list of txids that can we used has inputs.

Parameters:	owner (str) – base58 encoded public key.
Returns:	Iterator of (block_id, transaction) for transactions that list given owner in conditions.

bigchaindb.backend.query.get_block(connection, block_id)[source]¶

Get a block from the bigchain table.

Parameters:	block_id (str) – block id of the block to get
Returns:	the block or None
Return type:	block (dict)

bigchaindb.backend.query.get_block_with_transaction(connection, txid)[source]¶

Get a block containing transaction id txid

Parameters:	txid (str) – id of transaction to be searched.
Returns:	the block id or None
Return type:	block_id (int)

bigchaindb.backend.query.get_metadata(connection, transaction_ids)[source]¶

Get a list of metadata from the metadata table.

Parameters:	transaction_ids (list) – a list of ids for the metadata to be retrieved from database. (the) –
Returns:	the list of returned metadata.
Return type:	metadata (list)

bigchaindb.backend.query.get_assets(connection, asset_ids)[source]¶

Get a list of assets from the assets table. :param asset_ids: a list of ids for the assets to be retrieved from :type asset_ids: list :param the database.:

Returns:	the list of returned assets.
Return type:	assets (list)

bigchaindb.backend.query.get_txids_filtered(connection, asset_id, operation=None)[source]¶

Return all transactions for a particular asset id and optional operation.

Parameters:	asset_id (str) – ID of transaction that defined the asset operation (str) (optional) – Operation to filter on

bigchaindb.backend.query.text_search(conn, search, *, language='english', case_sensitive=False, diacritic_sensitive=False, text_score=False, limit=0, table=None)[source]¶

Return all the assets that match the text search.

The results are sorted by text score. For more information about the behavior of text search on MongoDB see https://docs.mongodb.com/manual/reference/operator/query/text/#behavior

Parameters:	search (str) – Text search string to query the text index language (str, optional) – The language for the search and the rules for stemmer and tokenizer. If the language is `None` text search uses simple tokenization and no stemming. case_sensitive (bool, optional) – Enable or disable case sensitive search. diacritic_sensitive (bool, optional) – Enable or disable case sensitive diacritic search. text_score (bool, optional) – If `True` returns the text score with each document. limit (int, optional) – Limit the number of returned documents.
Returns:	a list of assets
Return type:	`list` of `dict`
Raises:	`OperationError` – If the backend does not support text search

bigchaindb.backend.query.get_latest_block(conn)[source]¶: Get the latest commited block i.e. block with largest height

bigchaindb.backend.query.store_block(conn, block)[source]¶

Write a new block to the blocks table

Parameters:	block (dict) – block with current height and block hash.
Returns:	The result of the operation.

bigchaindb.backend.query.store_unspent_outputs(connection, unspent_outputs)[source]¶: Store unspent outputs in utxo_set table.

bigchaindb.backend.query.delete_unspent_outputs(connection, unspent_outputs)[source]¶: Delete unspent outputs in utxo_set table.

bigchaindb.backend.query.delete_transactions(conn, txn_ids)[source]¶

Delete transactions from database

Parameters:	txn_ids (list) – list of transaction ids
Returns:	The result of the operation.

bigchaindb.backend.query.get_unspent_outputs(connection, *, query=None)[source]¶

Retrieves unspent outputs.

Parameters:	query (dict) – An optional parameter to filter the result set. Defaults to `None`, which means that all UTXO records will be returned.
Returns:	Generator yielding unspent outputs (UTXO set) according to the given query.

bigchaindb.backend.query.store_pre_commit_state(connection, state)[source]¶

Store pre-commit state.

Parameters:	state (dict) – pre-commit state.
Returns:	The result of the operation.

bigchaindb.backend.query.get_pre_commit_state(connection)[source]¶

Get pre-commit state.

Returns:	Document representing the pre-commit state.

bigchaindb.backend.query.store_validator_set(conn, validator_update)[source]¶: Store updated validator set

bigchaindb.backend.query.delete_validator_set(conn, height)[source]¶: Delete the validator set at the given height.

bigchaindb.backend.query.store_election(conn, election_id, height, is_concluded)[source]¶: Store election record

bigchaindb.backend.query.store_elections(conn, elections)[source]¶: Store election records in bulk

bigchaindb.backend.query.delete_elections(conn, height)[source]¶: Delete all election records at the given height

bigchaindb.backend.query.get_validator_set(conn, height)[source]¶: Get validator set for a given height, if height is not specified then return the latest validator set

bigchaindb.backend.query.get_election(conn, election_id)[source]¶: Return the election record

bigchaindb.backend.query.get_asset_tokens_for_public_key(connection, asset_id, public_key)[source]¶

Retrieve a list of tokens of type asset_id that are owned by the public_key. :param asset_id: Id of the token. :type asset_id: str :param public_key: base58 encoded public key :type public_key: str

Returns:	Iterator of transaction that list given owner in conditions.

bigchaindb.backend.query.store_abci_chain(conn, height, chain_id, is_synced=True)[source]¶

Create or update an ABCI chain at the given height. Usually invoked in the beginning of the ABCI communications (height=0) or when ABCI client (like Tendermint) is migrated (any height).

Parameters:	is_synced – True if the chain is known by both ABCI client and server

bigchaindb.backend.query.delete_abci_chain(conn, height)[source]¶: Delete the ABCI chain at the given height.

bigchaindb.backend.query.get_latest_abci_chain(conn)[source]¶: Returns the ABCI chain stored at the biggest height, if any, None otherwise.

`bigchaindb.backend.schema`¶

Database creation and schema-providing interfaces for backends.

bigchaindb.backend.schema.create_database(connection, dbname)[source]¶

Create database to be used by BigchainDB.

Parameters:	dbname (str) – the name of the database to create.

bigchaindb.backend.schema.create_tables(connection, dbname)[source]¶

Create the tables to be used by BigchainDB.

Parameters:	dbname (str) – the name of the database to create tables for.

bigchaindb.backend.schema.drop_database(connection, dbname)[source]¶

Drop the database used by BigchainDB.

Parameters:	dbname (str) – the name of the database to drop.
Raises:	`DatabaseDoesNotExist` – If the given `dbname` does not exist as a database.

bigchaindb.backend.schema.init_database(connection=None, dbname=None)[source]¶

Initialize the configured backend for use with BigchainDB.

Creates a database with dbname with any required tables and supporting indexes.

Parameters:	connection (`Connection`) – an existing connection to use to initialize the database. Creates one if not given. dbname (str) – the name of the database to create. Defaults to the database name given in the BigchainDB configuration.

bigchaindb.backend.schema.validate_language_key(obj, key)[source]¶

Validate all nested “language” key in obj.

Parameters:	obj (dict) – dictionary whose “language” key is to be validated.
Returns:	validation successful
Return type:	None Raises: ValidationError: will raise exception in case language is not valid.

bigchaindb.backend.schema.validate_language(value)[source]¶

Check if value is a valid language. https://docs.mongodb.com/manual/reference/text-search-languages/

Args:

value (str): language to validated

Returns:

None: validation successful

Raises:

ValidationError: will raise exception in case language is not valid.

`bigchaindb.backend.utils`¶

exception bigchaindb.backend.utils.ModuleDispatchRegistrationError[source]¶: Raised when there is a problem registering dispatched functions for a module

MongoDB Backend¶

MongoDB backend implementation.

Contains a MongoDB-specific implementation of the schema and query interfaces.

You can specify BigchainDB to use MongoDB as its database backend by either setting database.backend to 'localmongodb' in your configuration file, or setting the BIGCHAINDB_DATABASE_BACKEND environment variable to 'localmongodb'.

MongoDB is the default database backend for BigchainDB.

If configured to use MongoDB, BigchainDB will automatically return instances of LocalMongoDBConnection for connect() and dispatch calls of the generic backend interfaces to the implementations in this module.

`bigchaindb.backend.localmongodb.connection`¶

class bigchaindb.backend.localmongodb.connection.LocalMongoDBConnection(replicaset=None, ssl=None, login=None, password=None, ca_cert=None, certfile=None, keyfile=None, keyfile_passphrase=None, crlfile=None, **kwargs)[source]¶

collection(name)[source]¶

Return a lazy object that can be used to compose a query.

Parameters:	name (str) – the name of the collection to query.

run(query)[source]¶

Run a query.

Parameters:	query – the query to run
Raises:	`DuplicateKeyError` – If the query fails because of a duplicate key constraint. `OperationFailure` – If the query fails for any other reason. `ConnectionError` – If the connection to the database fails.

`bigchaindb.backend.localmongodb.query`¶

Query implementation for MongoDB

`bigchaindb.backend.localmongodb.schema`¶

Utils to initialize and drop the database.

Command Line Interface¶

`bigchaindb.commands.bigchaindb`¶

Implementation of the bigchaindb command, the command-line interface (CLI) for BigchainDB Server.

bigchaindb.commands.bigchaindb.run_show_config(args)[source]¶: Show the current configuration

bigchaindb.commands.bigchaindb.run_configure(args)[source]¶: Run a script to configure the current node.

bigchaindb.commands.bigchaindb.run_election(args)[source]¶: Initiate and manage elections

bigchaindb.commands.bigchaindb.run_election_new_upsert_validator(args, bigchain)[source]¶

Initiates an election to add/update/remove a validator to an existing BigchainDB network

Parameters:	args – dict args = { ‘public_key’: the public key of the proposed peer, (str) ‘power’: the proposed validator power for the new peer, (str) ‘node_id’: the node_id of the new peer (str) ‘sk’: the path to the private key of the node calling the election (str) } bigchain – an instance of BigchainDB
Returns:	election_id or False in case of failure

bigchaindb.commands.bigchaindb.run_election_new_chain_migration(args, bigchain)[source]¶

Initiates an election to halt block production

Parameters:	args – dict args = { ‘sk’: the path to the private key of the node calling the election (str) } bigchain – an instance of BigchainDB
Returns:	election_id or False in case of failure

bigchaindb.commands.bigchaindb.run_election_approve(args, bigchain)[source]¶

Approve an election

Parameters:	args – dict args = { ‘election_id’: the election_id of the election (str) ‘sk’: the path to the private key of the signer (str) } bigchain – an instance of BigchainDB
Returns:	success log message or False in case of error

bigchaindb.commands.bigchaindb.run_election_show(args, bigchain)[source]¶

Retrieves information about an election

Parameters:	args – dict args = { ‘election_id’: the transaction_id for an election (str) } bigchain – an instance of BigchainDB

bigchaindb.commands.bigchaindb.run_init(args)[source]¶: Initialize the database

bigchaindb.commands.bigchaindb.run_drop(args)[source]¶: Drop the database

bigchaindb.commands.bigchaindb.run_start(args)[source]¶: Start the processes to run the node

bigchaindb.commands.bigchaindb.run_tendermint_version(args)[source]¶: Show the supported Tendermint version(s)

`bigchaindb.commands.utils`¶

Utility functions and basic common arguments for argparse.ArgumentParser.

bigchaindb.commands.utils.configure_bigchaindb(command)[source]¶

Decorator to be used by command line functions, such that the configuration of bigchaindb is performed before the execution of the command.

Parameters:	command – The command to decorate.
Returns:	The command wrapper function.

bigchaindb.commands.utils.input_on_stderr(prompt='', default=None, convert=None)[source]¶

Output a string to stderr and wait for input.

Parameters:	prompt (str) – the message to display. default – the default value to return if the user leaves the field empty convert (callable) – a callable to be used to convert the value the user inserted. If None, the type of `default` will be used.

bigchaindb.commands.utils.start(parser, argv, scope)[source]¶

Utility function to execute a subcommand.

The function will look up in the scope if there is a function called run_<parser.args.command> and will run it using parser.args as first positional argument.

Parameters:	parser – an ArgumentParser instance. argv – the list of command line arguments without the script name. scope (dict) – map containing (eventually) the functions to be called.
Raises:	`NotImplementedError` – if `scope` doesn’t contain a function called `run_<parser.args.command>`.

Appendices¶

JSON Serialization¶

Use the following link to find the BigchainDB Transactions Spec (or Specs) that are relevant to you:

BigchainDB Transactions Specs

Then see the section titled JSON Serialization and Deserialization.

Cryptography¶

Use the following link to find the BigchainDB Transactions Spec (or Specs) that are relevant to you:

BigchainDB Transactions Specs

Then see the sections titled Cryptographic Hashes and Cryptographic Keys and Signatures.

Basic AWS Setup¶

Before you can deploy anything on AWS, you must do a few things.

Get an AWS Account¶

If you don’t already have an AWS account, you can sign up for one for free at aws.amazon.com.

Install the AWS Command-Line Interface¶

To install the AWS Command-Line Interface (CLI), just do:

pip install awscli

Create an AWS Access Key¶

The next thing you’ll need is AWS access keys (access key ID and secret access key). If you don’t have those, see the AWS documentation about access keys.

You should also pick a default AWS region name (e.g. eu-central-1). The AWS documentation has a list of them.

Once you’ve got your AWS access key, and you’ve picked a default AWS region name, go to a terminal session and enter:

aws configure

and answer the four questions. For example:

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-central-1
Default output format [None]: [Press Enter]

This writes two files: ~/.aws/credentials and ~/.aws/config. AWS tools and packages look for those files.

Generate an RSA Key Pair for SSH¶

Eventually, you’ll have one or more instances (virtual machines) running on AWS and you’ll want to SSH to them. To do that, you need a public/private key pair. The public key will be sent to AWS, and you can tell AWS to put it in any instances you provision there. You’ll keep the private key on your local workstation.

See the page about how to generate a key pair for SSH.

Send the Public Key to AWS¶

To send the public key to AWS, use the AWS Command-Line Interface:

aws ec2 import-key-pair \
--key-name "<key-name>" \
--public-key-material file://~/.ssh/<key-name>.pub

If you’re curious why there’s a file:// in front of the path to the public key, see issue aws/aws-cli#41 on GitHub.

If you want to verify that your key pair was imported by AWS, go to the Amazon EC2 console, select the region you gave above when you did aws configure (e.g. eu-central-1), click on Key Pairs in the left sidebar, and check that <key-name> is listed.

Generate a Key Pair for SSH¶

This page describes how to use ssh-keygen to generate a public/private RSA key pair that can be used with SSH. (Note: ssh-keygen is found on most Linux and Unix-like operating systems; if you’re using Windows, then you’ll have to use another tool, such as PuTTYgen.)

By convention, SSH key pairs get stored in the ~/.ssh/ directory. Check what keys you already have there:

ls -1 ~/.ssh/

Next, make up a new key pair name (called <name> below). Here are some ideas:

aws-bdb-2
tim-bdb-azure
chris-bcdb-key

Next, generate a public/private RSA key pair with that name:

ssh-keygen -t rsa -C "<name>" -f ~/.ssh/<name>

It will ask you for a passphrase. You can use whatever passphrase you like, but don’t lose it. Two keys (files) will be created in ~/.ssh/:

~/.ssh/<name>.pub is the public key
~/.ssh/<name> is the private key

Notes for Firewall Setup¶

This is a page of notes on the ports potentially used by BigchainDB nodes and the traffic they should expect, to help with firewall setup (or security group setup on cloud providers). This page is not a firewall tutorial or step-by-step guide.

Expected Unsolicited Inbound Traffic¶

The following ports should expect unsolicited inbound traffic:

Port 22 can expect inbound SSH (TCP) traffic from the node administrator (i.e. a small set of IP addresses).
Port 9984 can expect inbound HTTP (TCP) traffic from BigchainDB clients sending transactions to the BigchainDB HTTP API.
Port 9985 can expect inbound WebSocket traffic from BigchainDB clients.
Port 26656 can expect inbound Tendermint P2P traffic from other Tendermint peers.
Port 9986 can expect inbound HTTP (TCP) traffic from clients accessing the Public Key of a Tendermint instance.

All other ports should only get inbound traffic in response to specific requests from inside the node.

Port 22¶

Port 22 is the default SSH port (TCP) so you’ll at least want to make it possible to SSH in from your remote machine(s).

Port 53¶

Port 53 is the default DNS port (UDP). It may be used, for example, by some package managers when look up the IP address associated with certain package sources.

Port 80¶

Port 80 is the default HTTP port (TCP). It’s used by some package managers to get packages. It’s not the default port for the BigchainDB client-server HTTP API.

Port 123¶

Port 123 is the default NTP port (UDP). You should be running an NTP daemon on production BigchainDB nodes. NTP daemons must be able to send requests to external NTP servers and accept the respones.

Port 161¶

Port 161 is the default SNMP port (usually UDP, sometimes TCP). SNMP is used, for example, by some server monitoring systems.

Port 443¶

Port 443 is the default HTTPS port (TCP). Package managers might also get some packages using HTTPS.

Port 9984¶

Port 9984 is the default port for the BigchainDB client-server HTTP API (TCP), which is served by Gunicorn HTTP Server. It’s possible allow port 9984 to accept inbound traffic from anyone, but we recommend against doing that. Instead, set up a reverse proxy server (e.g. using Nginx) and only allow traffic from there. Information about how to do that can be found in the Gunicorn documentation. (They call it a proxy.)

If Gunicorn and the reverse proxy are running on the same server, then you’ll have to tell Gunicorn to listen on some port other than 9984 (so that the reverse proxy can listen on port 9984). You can do that by setting server.bind to ‘localhost:PORT’ in the BigchainDB Configuration Settings, where PORT is whatever port you chose (e.g. 9983).

You may want to have Gunicorn and the reverse proxy running on different servers, so that both can listen on port 9984. That would also help isolate the effects of a denial-of-service attack.

Port 9985¶

Port 9985 is the default port for the BigchainDB WebSocket Event Stream API.

Port 9986¶

Port 9986 is the default port to access the Public Key of a Tendermint instance, it is used by a NGINX instance that runs with Tendermint instance(Pod), and only hosts the Public Key.

Port 26656¶

Port 26656 is the default port used by Tendermint Core to communicate with other instances of Tendermint Core (peers).

Port 26657¶

Port 26657 is the default port used by Tendermint Core for RPC traffic. BigchainDB nodes use that internally; they don’t expect incoming traffic from the outside world on port 26657.

Port 26658¶

Port 26658 is the default port used by Tendermint Core for ABCI traffic. BigchainDB nodes use that internally; they don’t expect incoming traffic from the outside world on port 26658.

Other Ports¶

On Linux, you can use commands such as netstat -tunlp or lsof -i to get a sense of currently open/listening ports and connections, and the associated processes.

Notes on NTP Daemon Setup¶

There are several NTP daemons available, including:

The reference NTP daemon (ntpd) from ntp.org; see their support website
chrony
OpenNTPD
Maybe NTPsec, once it’s production-ready
Maybe Ntimed, once it’s production-ready
More

We suggest you run your NTP daemon in a mode which will tell your OS kernel to handle leap seconds in a particular way: the default NTP way, so that system clock adjustments are localized and not spread out across the minutes, hours, or days surrounding leap seconds (e.g. “slewing” or “smearing”). There’s a nice Red Hat Developer Blog post about the various options.

Use the default mode with ntpd and chronyd. For another NTP daemon, consult its documentation.

It’s tricky to make an NTP daemon setup secure. Always install the latest version and read the documentation about how to configure and run it securely. See the notes on firewall setup.

Amazon Linux Instances¶

If your BigchainDB node is running on an Amazon Linux instance (i.e. a Linux instance packaged by Amazon, not Canonical, Red Hat, or someone else), then an NTP daemon should already be installed and configured. See the EC2 documentation on Setting the Time for Your Linux Instance.

That said, you should check which NTP daemon is installed. Is it recent? Is it configured securely?

The Ubuntu ntp Packages¶

The Ubuntu ntp packages are based on the reference implementation of NTP.

The following commands will uninstall the ntp and ntpdate packages, install the latest ntp package (which might not be based on the latest ntpd code), and start the NTP daemon (a local NTP server). (ntpdate is not reinstalled because it’s deprecated and you shouldn’t use it.)

sudo apt-get --purge remove ntp ntpdate
sudo apt-get autoremove
sudo apt-get update
sudo apt-get install ntp
# That should start the NTP daemon too, but just to be sure:
sudo service ntp restart

You can check if ntpd is running using sudo ntpq -p.

You may want to use different NTP time servers. You can change them by editing the NTP config file /etc/ntp.conf.

Note: A server running an NTP daemon can be used by others for DRDoS amplification attacks. The above installation procedure should install a default NTP configuration file /etc/ntp.conf with the lines:

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

Those lines should prevent the NTP daemon from being used in an attack. (The first line is for IPv4, the second for IPv6.)

There are additional things you can do to make NTP more secure. See the NTP Support Website for more details.

Licenses¶

Information about how the BigchainDB Server code and documentation are licensed can be found in the LICENSES.md file of the bigchaindb/bigchaindb repository on GitHub.

Run BigchainDB with all-in-one Docker¶

For those who like using Docker and wish to experiment with BigchainDB in non-production environments, we currently maintain a BigchainDB all-in-one Docker image and a Dockerfile-all-in-one that can be used to build an image for bigchaindb.

This image contains all the services required for a BigchainDB node i.e.

BigchainDB Server
MongoDB
Tendermint

Note: NOT for Production Use: This is an single node opinionated image not well suited for a network deployment. This image is to help quick deployment for early adopters, for a more standard approach please refer to one of our deployment guides:

Prerequisite(s)¶

Docker

Pull and Run the Image from Docker Hub¶

With Docker installed, you can proceed as follows.

In a terminal shell, pull the latest version of the BigchainDB all-in-one Docker image using:

$ docker pull bigchaindb/bigchaindb:all-in-one

$ docker run \
  --detach \
  --name bigchaindb \
  --publish 9984:9984 \
  --publish 9985:9985 \
  --publish 27017:27017 \
  --publish 26657:26657 \
  --volume $HOME/bigchaindb_docker/mongodb/data/db:/data/db \
  --volume $HOME/bigchaindb_docker/mongodb/data/configdb:/data/configdb \
  --volume $HOME/bigchaindb_docker/tendermint:/tendermint \
  bigchaindb/bigchaindb:all-in-one

Let’s analyze that command:

docker run tells Docker to run some image
--detach run the container in the background
publish 9984:9984 map the host port 9984 to the container port 9984 (the BigchainDB API server)
- 9985 BigchainDB Websocket server
- 27017 Default port for MongoDB
- 26657 Tendermint RPC server
--volume "$HOME/bigchaindb_docker/mongodb:/data" map the host directory $HOME/bigchaindb_docker/mongodb to the container directory /data; this allows us to have the data persisted on the host machine, you can read more in the official Docker documentation
- $HOME/bigchaindb_docker/tendermint:/tendermint to persist Tendermint data.
bigchaindb/bigchaindb:all-in-one the image to use. All the options after the container name are passed on to the entrypoint inside the container.

Verify¶

$ docker ps | grep bigchaindb

Send your first transaction using BigchainDB drivers.

Building Your Own Image¶

Assuming you have Docker installed, you would proceed as follows.

In a terminal shell:

git clone git@github.com:bigchaindb/bigchaindb.git
cd bigchaindb/

Build the Docker image:

docker build --file Dockerfile-all-in-one --tag <tag/name:latest> .

Now you can use your own image to run BigchainDB all-in-one container.

Logging and Log Rotation¶

Each BigchainDB node runs:

MongoDB
BigchainDB Server
Tendermint

When running a BigchainDB node for long periods of time, we need to consider doing log rotation, i.e. we do not want the logs taking up large amounts of storage and making the node unresponsive or getting it into a bad state.

MongoDB Logging and Log Rotation¶

See the MongoDB docs about logging and log rotation.

BigchainDB Server Logging and Log Rotation¶

BigchainDB Server writes its logs to two files: normal logs and error logs. The names of those files, and their locations, are set as part of the BigchainDB configuration settings. The default names and locations are:

~/bigchaindb.log
~/bigchaindb-errors.log

Log rotation is baked into BigchainDB Server using Python’s logging module. The logs for BigchainDB Server are rotated when any of the above mentioned files exceeds 209715200 bytes (i.e. approximately 209 MB).

For more information, see the docs about the BigchainDB Server configuration settings related to logging.

Tendermint Logging and Log Rotation¶

Tendermint writes its logs to the files:

tendermint.out.log
tendermint.err.log

If you started BigchainDB Server and Tendermint using Monit, as suggested by our guide on How to Set Up a BigchainDB Network, then the logs will be written to $HOME/.bigchaindb-monit/logs/.

Moreover, if you started BigchainDB Server and Tendermint using Monit, then Monit monitors the Tendermint log files. Tendermint logs are rotated if any of the above mentioned log files exceeds 200 MB.

BigchainDB Server Documentation¶

Introduction¶

Setup Instructions for Various Cases¶

Can I Help?¶

Quickstart¶

Try BigchainDB¶

Develop an App¶

Help Develop BigchainDB Server¶

Production Nodes¶

Production Node Requirements¶

OS Requirements¶

General Considerations¶

Production Node Assumptions¶

Production Node Components¶

Production Node Security & Privacy¶

Using a Reverse Proxy¶

Enforcing a Max Transaction Size¶

BigchainDB Networks¶

Consortium Structure & Governance¶

DNS Records and SSL Certificates¶

Node Operator Responsibilities¶

Consortium Responsibilities¶

Simple Deployment Template¶

Deploy a Machine for Your BigchainDB Node¶

IP Addresses¶

Operating System¶

Network Security Group¶

Update Your System¶

Node Security¶

DNS Setup¶

Set Up NGINX¶

Install NGINX¶

Configure & Reload NGINX¶

Set Up BigchainDB, MongoDB and Tendermint¶

Install BigchainDB Server¶

Configure BigchainDB Server¶

Install (and Start) MongoDB¶

Install Tendermint¶

Start Configuring Tendermint¶

How to Set Up a BigchainDB Network¶

Member: Share hostname, pub_key.value and node_id¶

Coordinator: Create & Share the genesis.json File¶

Member: Connect to the Other Members¶

Member: Start MongoDB¶

Member: Start BigchainDB and Tendermint Using Monit¶

How Others Can Access Your Node¶

Tips¶

Tendermint Tips¶

Refreshing Your Node¶

Shutting Down BigchainDB¶

Member: Dynamically Add or Remove Validators¶

Logging and Log Rotation¶

Troubleshooting¶

General Tips¶

Resolving Tendermint Connectivity Problems¶

Other Problems¶

Develop & Test BigchainDB Server¶

Settings & CLI¶

Configuration Settings¶

database.*¶

server.*¶

wsserver.*¶

wsserver.scheme, wsserver.host and wsserver.port¶

wsserver.advertised_scheme, wsserver.advertised_host and wsserver.advertised_port¶

log.*¶

log.file¶

log.error_file¶

log.level_console¶

log.level_logfile¶

log.datefmt_console¶

log.datefmt_logfile¶

log.fmt_console¶

log.fmt_logfile¶

log.granular_levels¶

tendermint.*¶

Command Line Interface (CLI)¶

bigchaindb --help¶

bigchaindb --version¶

bigchaindb configure¶

bigchaindb show-config¶