BigchainDB Documentation¶

Meet BigchainDB. The blockchain database.

It has some database characteristics and some blockchain properties, including decentralization, immutability and native support for assets.

At a high level, one can communicate with a BigchainDB network (set of nodes) using the BigchainDB HTTP API, or a wrapper for that API, such as the BigchainDB Python Driver. Each BigchainDB node runs BigchainDB Server and various other software. The terminology page explains some of those terms in more detail.

More About BigchainDB¶

About BigchainDB¶

Basic Facts¶

One can store arbitrary data (including encrypted data) in a BigchainDB network, within limits: there’s a maximum transaction size. Every transaction has a metadata section which can store almost any Unicode string (up to some maximum length). Similarly, every CREATE transaction has an asset.data section which can store almost any Unicode string.
The data stored in certain BigchainDB transaction fields must not be encrypted, e.g. public keys and amounts. BigchainDB doesn’t offer private transactions akin to Zcoin.
Once data has been stored in a BigchainDB network, it’s best to assume it can’t be change or deleted.
Every node in a BigchainDB network has a full copy of all the stored data.
Every node in a BigchainDB network can read all the stored data.
Everyone with full access to a BigchainDB node (e.g. the sysadmin of a node) can read all the data stored on that node.
Everyone given access to a node via the BigchainDB HTTP API can find and read all the data stored by BigchainDB. The list of people with access might be quite short.
If the connection between an external user and a BigchainDB node isn’t encrypted (using HTTPS, for example), then a wiretapper can read all HTTP requests and responses in transit.
If someone gets access to plaintext (regardless of where they got it), then they can (in principle) share it with the whole world. One can make it difficult for them to do that, e.g. if it is a lot of data and they only get access inside a secure room where they are searched as they leave the room.

BigchainDB for Asset Registrations & Transfers¶

BigchainDB can store data of any kind, but it’s designed to be particularly good for storing asset registrations and transfers:

The fundamental thing that one sends to a BigchainDB network, to be checked and stored (if valid), is a transaction, and there are two kinds: CREATE transactions and TRANSFER transactions.
A CREATE transaction can be use to register any kind of asset (divisible or indivisible), along with arbitrary metadata.
An asset can have zero, one, or several owners.
The owners of an asset can specify (crypto-)conditions which must be satisfied by anyone wishing transfer the asset to new owners. For example, a condition might be that at least 3 of the 5 current owners must cryptographically sign a TRANSFER transaction.
BigchainDB verifies that the conditions have been satisfied as part of checking the validity of TRANSFER transactions. (Moreover, anyone can check that they were satisfied.)
BigchainDB prevents double-spending of an asset.
Validated transactions are immutable.

Note

We used the word “owners” somewhat loosely above. A more accurate word might be fulfillers, signers, controllers, or transfer-enablers. See the section titled A Note about Owners in the relevant BigchainDB Transactions Spec.

# Production-Ready?

Depending on your use case, BigchainDB may or may not be production-ready. You should ask your service provider. If you want to go live (into production) with BigchainDB, please consult with your service provider.

Note: BigchainDB has an open source license with a “no warranty” section that is typical of open source licenses. This is standard in the software industry. For example, the Linux kernel is used in production by billions of machines even though its license includes a “no warranty” section. Warranties are usually provided above the level of the software license, by service providers.

Storing Private Data Off-Chain¶

A system could store data off-chain, e.g. in a third-party database, document store, or content management system (CMS) and it could use BigchainDB to:

Keep track of who has read permissions (or other permissions) in a third-party system. An example of how this could be done is described below.
Keep a permanent record of all requests made to the third-party system.
Store hashes of documents-stored-elsewhere, so that a change in any document can be detected.
Record all handshake-establishing requests and responses between two off-chain parties (e.g. a Diffie-Hellman key exchange), so as to prove that they established an encrypted tunnel (without giving readers access to that tunnel). There are more details about this idea in the Privacy Protocols repository.

A simple way to record who has read permission on a particular document would be for the third-party system (“DocPile”) to store a CREATE transaction in a BigchainDB network for every document+user pair, to indicate that that user has read permissions for that document. The transaction could be signed by DocPile (or maybe by a document owner, as a variation). The asset data field would contain 1) the unique ID of the user and 2) the unique ID of the document. The one output on the CREATE transaction would only be transferable/spendable by DocPile (or, again, a document owner).

To revoke the read permission, DocPile could create a TRANSFER transaction, to spend the one output on the original CREATE transaction, with a metadata field to say that the user in question no longer has read permission on that document.

This can be carried on indefinitely, i.e. another TRANSFER transaction could be created by DocPile to indicate that the user now has read permissions again.

DocPile can figure out if a given user has read permissions on a given document by reading the last transaction in the CREATE → TRANSFER → TRANSFER → etc. chain for that user+document pair.

There are other ways to accomplish the same thing. The above is just one example.

You might have noticed that the above example didn’t treat the “read permission” as an asset owned (controlled) by a user because if the permission asset is given to (transferred to or created by) the user then it cannot be controlled any further (by DocPile) until the user transfers it back to DocPile. Moreover, the user could transfer the asset to someone else, which might be problematic.

Storing Private Data On-Chain, Encrypted¶

There are many ways to store private data on-chain, encrypted. Every use case has its own objectives and constraints, and the best solution depends on the use case. The IPDB consulting team can help you design the best solution for your use case.

Below we describe some example system setups, using various crypto primitives, to give a sense of what’s possible.

Please note:

Ed25519 keypairs are designed for signing and verifying cryptographic signatures, not for encrypting and decrypting messages. For encryption, you should use keypairs designed for encryption, such as X25519.
If someone (or some group) publishes how to decrypt some encrypted data on-chain, then anyone with access to that encrypted data will be able to get the plaintext. The data can’t be deleted.
Encrypted data can’t be indexed or searched by MongoDB. (It can index and search the ciphertext, but that’s not very useful.) One might use homomorphic encryption to index and search encrypted data, but MongoDB doesn’t have any plans to support that any time soon. If there is indexing or keyword search needed, then some fields of the asset.data or metadata objects can be left as plain text and the sensitive information can be stored in an encrypted child-object.

System Example 1¶

Encrypt the data with a symmetric key and store the ciphertext on-chain (in metadata or asset.data). To communicate the key to a third party, use their public key to encrypt the symmetric key and send them that. They can decrypt the symmetric key with their private key, and then use that symmetric key to decrypt the on-chain ciphertext.

The reason for using a symmetric key along with public/private keypairs is so the ciphertext only has to be stored once.

System Example 2¶

This example uses proxy re-encryption:

MegaCorp encrypts some data using its own public key, then stores that encrypted data (ciphertext 1) in a BigchainDB network.
MegaCorp wants to let others read that encrypted data, but without ever sharing their private key and without having to re-encrypt themselves for every new recipient. Instead, they find a “proxy” named Moxie, to provide proxy re-encryption services.
Zorban contacts MegaCorp and asks for permission to read the data.
MegaCorp asks Zorban for his public key.
MegaCorp generates a “re-encryption key” and sends it to their proxy, Moxie.
Moxie (the proxy) uses the re-encryption key to encrypt ciphertext 1, creating ciphertext 2.
Moxie sends ciphertext 2 to Zorban (or to MegaCorp who forwards it to Zorban).
Zorban uses his private key to decrypt ciphertext 2, getting the original un-encrypted data.

Note:

The proxy only ever sees ciphertext. They never see any un-encrypted data.
Zorban never got the ability to decrypt ciphertext 1, i.e. the on-chain data.
There are variations on the above flow.

System Example 3¶

This example uses erasure coding:

Erasure-code the data into n pieces.
Encrypt each of the n pieces with a different encryption key.
Store the n encrypted pieces on-chain, e.g. in n separate transactions.
Share each of the the n decryption keys with a different party.

If k < N of the key-holders gets and decrypts k of the pieces, they can reconstruct the original plaintext. Less than k would not be enough.

System Example 4¶

This setup could be used in an enterprise blockchain scenario where a special node should be able to see parts of the data, but the others should not.

The special node generates an X25519 keypair (or similar asymmetric encryption keypair).
A BigchainDB end user finds out the X25519 public key (encryption key) of the special node.
The end user creates a valid BigchainDB transaction, with either the asset.data or the metadata (or both) encrypted using the above-mentioned public key.
This is only done for transactions where the contents of asset.data or metadata don’t matter for validation, so all node operators can validate the transaction.
The special node is able to decrypt the encrypted data, but the other node operators can’t, and nor can any other end user.

Terminology¶

There is some specialized terminology associated with BigchainDB. To get started, you should at least know the following:

BigchainDB Node¶

A BigchainDB node is a machine (or logical machine) running BigchainDB Server and related software. Each node is controlled by one person or organization.

BigchainDB Network¶

A set of BigchainDB nodes can connect to each other to form a BigchainDB network. Each node in the network runs the same software. A BigchainDB network may have additional machines to do things such as monitoring.

BigchainDB Consortium¶

The people and organizations that run the nodes in a BigchainDB network belong to a BigchainDB consortium (i.e. another organization). A consortium must have some sort of governance structure to make decisions. If a BigchainDB network is run by a single company, then the “consortium” is just that company.

What’s the Difference Between a BigchainDB Network and a Consortium?

A BigchaindB network is just a bunch of connected nodes. A consortium is an organization which has a BigchainDB network, and where each node in that network has a different operator.

Transactions¶

Are described in detail in BigchainDB Transactions Spec <https://github.com/bigchaindb/BEPs/tree/master/tx-specs/>_ .

Permissions in BigchainDB¶

BigchainDB lets users control what other users can do, to some extent. That ability resembles “permissions” in the *nix world, “privileges” in the SQL world, and “access control” in the security world.

Permission to Spend/Transfer an Output¶

In BigchainDB, every output has an associated condition (crypto-condition).

To spend/transfer an unspent output, a user (or group of users) must fulfill the condition. Another way to say that is that only certain users have permission to spend the output. The simplest condition is of the form, “Only someone with the private key corresponding to this public key can spend this output.” Much more elaborate conditions are possible, e.g. “To spend this output, …”

“…anyone in the Accounting Group can sign.”
“…three of these four people must sign.”
“…either Bob must sign, or both Tom and Sylvia must sign.”

Once an output has been spent, it can’t be spent again: nobody has permission to do that. That is, BigchainDB doesn’t permit anyone to “double spend” an output.

Write Permissions¶

When someone builds a TRANSFER transaction, they can put an arbitrary JSON object in the metadata field (within reason; real BigchainDB networks put a limit on the size of transactions). That is, they can write just about anything they want in a TRANSFER transaction.

Does that mean there are no “write permissions” in BigchainDB? Not at all!

A TRANSFER transaction will only be valid (allowed) if its inputs fulfill some previous outputs. The conditions on those outputs will control who can build valid TRANSFER transactions. In other words, one can interpret the condition on an output as giving “write permissions” to certain users to write something into the history of the associated asset.

As a concrete example, you could use BigchainDB to write a public journal where only you have write permissions. Here’s how: First you’d build a CREATE transaction with the asset.data being something like {"title": "The Journal of John Doe"}, with one output. That output would have an amount 1 and a condition that only you (who has your private key) can spend that output. Each time you want to append something to your journal, you’d build a new TRANSFER transaction with your latest entry in the metadata field, e.g.

.. code-block:: json

{”timestamp”: “1508319582”, “entry”: “I visited Marmot Lake with Jane.”}

The TRANSFER transaction would have one output. That output would have an amount 1 and a condition that only you (who has your private key) can spend that output. And so on. Only you would be able to append to the history of that asset (your journal).

The same technique could be used for scientific notebooks, supply-chain records, government meeting minutes, and so on.

You could do more elaborate things too. As one example, each time someone writes a TRANSFER transaction, they give someone else permission to spend it, setting up a sort of writers-relay or chain letter.

.. note::

Anyone can write any JSON (again, within reason) in the asset.data field of a CREATE transaction. They don’t need permission.

Role-Based Access Control (RBAC)¶

In September 2017, we published a blog post about how one can define an RBAC sub-system on top of BigchainDB. At the time of writing (January 2018), doing so required the use of a plugin, so it’s not possible using standard BigchainDB (which is what’s available on the IPDB Testnet. That may change in the future. If you’re interested, contact IPDB <contact@ipdb.global>_.

Properties of BigchainDB¶

Decentralization¶

Decentralization means that no one owns or controls everything, and there is no single point of failure.

Ideally, each node in a BigchainDB network is owned and controlled by a different person or organization. Even if the network lives within one organization, it’s still preferable to have each node controlled by a different person or subdivision.

We use the phrase “BigchainDB consortium” (or just “consortium”) to refer to the set of people and/or organizations who run the nodes of a BigchainDB network. A consortium requires some form of governance to make decisions such as membership and policies. The exact details of the governance process are determined by each consortium, but it can be very decentralized.

A consortium can increase its decentralization (and its resilience) by increasing its jurisdictional diversity, geographic diversity, and other kinds of diversity.

There’s no node that has a long-term special position in the BigchainDB network. All nodes run the same software and perform the same duties.

If someone has (or gets) admin access to a node, they can mess with that node (e.g. change or delete data stored on that node), but those changes should remain isolated to that node. The BigchainDB network can only be compromised if more than one third of the nodes get compromised. See the Tendermint documentation for more details.

It’s worth noting that not even the admin or superuser of a node can transfer assets. The only way to create a valid transfer transaction is to fulfill the current crypto-conditions on the asset, and the admin/superuser can’t do that because the admin user doesn’t have the necessary information (e.g. private keys).

Byzantine Fault Tolerance¶

Tendermint is used for consensus and transaction replication, and Tendermint is Byzantine Fault Tolerant (BFT).

Node Diversity¶

Steps should be taken to make it difficult for any one actor or event to control or damage “enough” of the nodes. (Because BigchainDB Server uses Tendermint, “enough” is ⅓.) There are many kinds of diversity to consider, listed below. It may be quite difficult to have high diversity of all kinds.

Jurisdictional diversity. The nodes should be controlled by entities within multiple legal jurisdictions, so that it becomes difficult to use legal means to compel enough of them to do something.
Geographic diversity. The servers should be physically located at multiple geographic locations, so that it becomes difficult for a natural disaster (such as a flood or earthquake) to damage enough of them to cause problems.
Hosting diversity. The servers should be hosted by multiple hosting providers (e.g. Amazon Web Services, Microsoft Azure, Digital Ocean, Rackspace), so that it becomes difficult for one hosting provider to influence enough of the nodes.
Diversity in general. In general, membership diversity (of all kinds) confers many advantages on a consortium. For example, it provides the consortium with a source of various ideas for addressing challenges.

Note: If all the nodes are running the same code, i.e. the same implementation of BigchainDB, then a bug in that code could be used to compromise all of the nodes. Ideally, there would be several different, well-maintained implementations of BigchainDB Server (e.g. one in Python, one in Go, etc.), so that a consortium could also have a diversity of server implementations. Similar remarks can be made about the operating system.

Immutability¶

The blockchain community often describes blockchains as “immutable.” If we interpret that word literally, it means that blockchain data is unchangeable or permanent, which is absurd. The data can be changed. For example, a plague might drive humanity extinct; the data would then get corrupted over time due to water damage, thermal noise, and the general increase of entropy.

It’s true that blockchain data is more difficult to change (or delete) than usual. It’s more than just “tamper-resistant” (which implies intent), blockchain data also resists random changes that can happen without any intent, such as data corruption on a hard drive. Therefore, in the context of blockchains, we interpret the word “immutable” to mean practically immutable, for all intents and purposes. (Linguists would say that the word “immutable” is a term of art in the blockchain community.)

Blockchain data can be made immutable in several ways:

No APIs for changing or deleting data. Blockchain software usually doesn’t expose any APIs for changing or deleting the data stored in the blockchain. BigchainDB has no such APIs. This doesn’t prevent changes or deletions from happening in other ways; it’s just one line of defense.
Replication. All data is replicated (copied) to several different places. The higher the replication factor, the more difficult it becomes to change or delete all replicas.
Internal watchdogs. All nodes monitor all changes and if some unallowed change happens, then appropriate action can be taken.
External watchdogs. A consortium may opt to have trusted third-parties to monitor and audit their data, looking for irregularities. For a consortium with publicly-readable data, the public can act as an auditor.
Economic incentives. Some blockchain systems make it very expensive to change old stored data. Examples include proof-of-work and proof-of-stake systems. BigchainDB doesn’t use explicit incentives like those.
Data can be stored using fancy techniques, such as error-correction codes, to make some kinds of changes easier to undo.
Cryptographic signatures are often used as a way to check if messages (e.g. transactions) have been tampered with enroute, and as a way to verify who signed the messages. In BigchainDB, each transaction must be signed by one or more parties.
Full or partial backups may be recorded from time to time, possibly on magnetic tape storage, other blockchains, printouts, etc.
Strong security. Node owners can adopt and enforce strong security policies.

Basic usage¶

Transactions in BigchainDB¶

In BigchainDB, transactions are used to register, issue, create or transfer things (e.g. assets).

Transactions are the most basic kind of record stored by BigchainDB. There are two kinds: CREATE transactions and TRANSFER transactions.

You can view the transaction specifications in Github, which describe transaction components and the conditions they have to fulfill in order to be valid.

BigchainDB Transactions Specs

CREATE Transactions¶

A CREATE transaction can be used to register, issue, create or otherwise initiate the history of a single thing (or asset) in BigchainDB. For example, one might register an identity or a creative work. The things are often called “assets” but they might not be literal assets.

BigchainDB supports divisible assets as of BigchainDB Server v0.8.0. That means you can create/register an asset with an initial number of “shares.” For example, A CREATE transaction could register a truckload of 50 oak trees. Each share of a divisible asset must be interchangeable with each other share; the shares must be fungible.

A CREATE transaction can have one or more outputs. Each output has an associated amount: the number of shares tied to that output. For example, if the asset consists of 50 oak trees, one output might have 35 oak trees for one set of owners, and the other output might have 15 oak trees for another set of owners.

Each output also has an associated condition: the condition that must be met (by a TRANSFER transaction) to transfer/spend the output. BigchainDB supports a variety of conditions. For details, see the section titled Transaction Components: Conditions in the relevant BigchainDB Transactions Spec.

_images/CREATE_example.png Example BigchainDB CREATE transaction

Above we see a diagram of an example BigchainDB CREATE transaction. It has one output: Pam owns/controls three shares of the asset and there are no other shares (because there are no other outputs).

Each output also has a list of all the public keys associated with the conditions on that output. Loosely speaking, that list might be interpreted as the list of “owners.” A more accurate word might be fulfillers, signers, controllers, or transfer-enablers. See the section titled A Note about Owners in the relevant BigchainDB Transactions Spec.

A CREATE transaction must be signed by all the owners. (If you’re looking for that signature, it’s in the one “fulfillment” of the one input, albeit encoded.)

TRANSFER Transactions¶

A TRANSFER transaction can transfer/spend one or more outputs on other transactions (CREATE transactions or other TRANSFER transactions). Those outputs must all be associated with the same asset; a TRANSFER transaction can only transfer shares of one asset at a time.

Each input on a TRANSFER transaction connects to one output on another transaction. Each input must satisfy the condition on the output it’s trying to transfer/spend.

A TRANSFER transaction can have one or more outputs, just like a CREATE transaction (described above). The total number of shares coming in on the inputs must equal the total number of shares going out on the outputs.

_images/CREATE_and_TRANSFER_example.png Example BigchainDB transactions

Above we see a diagram of two example BigchainDB transactions, a CREATE transaction and a TRANSFER transaction. The CREATE transaction is the same as in the earlier diagram. The TRANSFER transaction spends Pam’s output, so the input on that TRANSFER transaction must contain a valid signature from Pam (i.e. a valid fulfillment). The TRANSFER transaction has two outputs: Jim gets one share, and Pam gets the remaining two shares.

Terminology: The “Pam, 3” output is called a “spent transaction output” and the “Jim, 1” and “Pam, 2” outputs are called “unspent transaction outputs” (UTXOs).

Example 1: Suppose a red car is owned and controlled by Joe. Suppose the current transfer condition on the car says that any valid transfer must be signed by Joe. Joe could build a TRANSFER transaction containing an input with Joe’s signature (to fulfill the current output condition) plus a new output condition saying that any valid transfer must be signed by Rae.

Example 2: Someone might construct a TRANSFER transaction that fulfills the output conditions on four previously-untransferred assets of the same asset type e.g. paperclips. The amounts might be 20, 10, 45 and 25, say, for a total of 100 paperclips. The TRANSFER transaction would also set up new transfer conditions. For example, maybe a set of 60 paperclips can only be transferred if Gertrude signs, and a separate set of 40 paperclips can only be transferred if both Jack and Kelly sign. Note how the sum of the incoming paperclips must equal the sum of the outgoing paperclips (100).

Transaction Validity¶

When a node is asked to check if a transaction is valid, it checks several things. We documented those things in a post on The BigchainDB Blog: “What is a Valid Transaction in BigchainDB?” (Note: That post was about BigchainDB Server v1.0.0.)

Example Transactions¶

There are example BigchainDB transactions in the HTTP API documentation and the Python Driver documentation.

Installation¶

You can install a single node to test out BigchainDB, connect it to a network or setup a network of nodes.

Introduction¶

This is the documentation for BigchainDB Server, or in other words, node - the BigchainDB software that is on servers (but not on clients).

Setup Instructions for Various Cases¶

Quickstart link below
Set up a local BigchainDB node for development, experimenting and testing
Set up and run a BigchainDB network

Develop an App Test¶

To develop an app that talks to a BigchainDB network, you’ll want a test network to test it against. You have a few options:

The IPDB Test Network (or “Testnet”) is a free-to-use, publicly-available test network that you can test against. It is available at IPDB testnet.
You could also run a BigchainDB node on you local machine. One way is to use this node setup guide with a one-node “network” by using the all-in-one docker solution, or manual installation and configuration of the components. Another way is to use one of the deployment methods listed in the network setup guide or in the the docs about contributing to BigchainDB.

(WIP) Quickstart¶

Try BigchainDB¶

Create a transaction and post it to the test network:

Try BigchainDB Now

Node setup¶

You can use the all-in-one docker solution, or install Tendermint, MongoDB, and BigchainDB step by step. For more advanced users and for development, the second option is recommended.

Deploy a Machine for Your BigchainDB Node¶

The first step is to deploy a machine for your BigchainDB node. It might be a virtual machine (VM) or a real machine, for example, an EC2 on AWS or a droplet on Digital Ocean. If you follow this simple deployment template, all your node’s software will run on that one machine.

We don’t make any assumptions about where you run the machine. It might be in Azure, AWS, your data center or a Raspberry Pi.

IP Addresses¶

The following instructions assume all the nodes in the network (including yours) have public IP addresses. (A BigchainDB network can be run inside a private network, using private IP addresses, but we don’t cover that here.)

Operating System¶

Use Ubuntu 18.04 or Ubuntu Server 18.04 as the operating system.

Similar instructions will work on other versions of Ubuntu, and other recent Debian-like Linux distros, but you may have to change the names of the packages, or install more packages.

Network Security Group¶

If your machine is in AWS or Azure, for example, and you want users to connect to BigchainDB via HTTPS, then you should configure its network security group to allow all incoming and outgoing traffic for:

TCP on port 22 (SSH)
TCP on port 80 (HTTP)
TCP on port 443 (HTTPS)
Any protocol on port 26656 (Tendermint P2P)

If you don’t care about HTTPS, then forget about port 443, and replace port 80 with port 9984 (the default BigchainDB HTTP port).

Update Your System¶

SSH into your machine and update all its OS-level packages:

sudo apt update
sudo apt full-upgrade

DNS Setup¶

Register a domain name for your BigchainDB node, such as example.com
Pick a subdomain of that domain for your BigchainDB node, such as bnode.example.com
Create a DNS “A Record” pointing your chosen subdomain (such as bnode.example.com) at your machine’s IP address.

Basic AWS Setup¶

Before you can deploy anything on AWS, you must do a few things.

Get an AWS Account¶

If you don’t already have an AWS account, you can sign up for one for free at aws.amazon.com.

Install the AWS Command-Line Interface¶

To install the AWS Command-Line Interface (CLI), just do:

pip install awscli

Create an AWS Access Key¶

The next thing you’ll need is AWS access keys (access key ID and secret access key). If you don’t have those, see the AWS documentation about access keys.

You should also pick a default AWS region name (e.g. eu-central-1). The AWS documentation has a list of them.

Once you’ve got your AWS access key, and you’ve picked a default AWS region name, go to a terminal session and enter:

aws configure

and answer the four questions. For example:

AWS Access Key ID [None]: AKIAIOSFODNN7EXAMPLE
AWS Secret Access Key [None]: wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
Default region name [None]: eu-central-1
Default output format [None]: [Press Enter]

This writes two files: ~/.aws/credentials and ~/.aws/config. AWS tools and packages look for those files.

Generate an RSA Key Pair for SSH¶

Eventually, you’ll have one or more instances (virtual machines) running on AWS and you’ll want to SSH to them. To do that, you need a public/private key pair. The public key will be sent to AWS, and you can tell AWS to put it in any instances you provision there. You’ll keep the private key on your local workstation.

See the appendix page about how to generate a key pair for SSH.

Send the Public Key to AWS¶

To send the public key to AWS, use the AWS Command-Line Interface:

aws ec2 import-key-pair \
--key-name "<key-name>" \
--public-key-material file://~/.ssh/<key-name>.pub

If you’re curious why there’s a file:// in front of the path to the public key, see issue aws/aws-cli#41 on GitHub.

If you want to verify that your key pair was imported by AWS, go to the Amazon EC2 console, select the region you gave above when you did aws configure (e.g. eu-central-1), click on Key Pairs in the left sidebar, and check that <key-name> is listed.

Run BigchainDB with all-in-one Docker¶

For those who like using Docker and wish to experiment with BigchainDB in non-production environments, we currently maintain a BigchainDB all-in-one Docker image and a Dockerfile-all-in-one that can be used to build an image for bigchaindb.

This image contains all the services required for a BigchainDB node i.e.

BigchainDB Server
MongoDB
Tendermint

Note: NOT for Production Use: This is an single node opinionated image not well suited for a network deployment. This image is to help quick deployment for early adopters, for a more standard approach please refer to one of our deployment guides:

Prerequisite(s)¶

Docker

Pull and Run the Image from Docker Hub¶

With Docker installed, you can proceed as follows.

In a terminal shell, pull the latest version of the BigchainDB all-in-one Docker image using:

$ docker pull bigchaindb/bigchaindb:all-in-one

$ docker run \
  --detach \
  --name bigchaindb \
  --publish 9984:9984 \
  --publish 9985:9985 \
  --publish 27017:27017 \
  --publish 26657:26657 \
  --volume $HOME/bigchaindb_docker/mongodb/data/db:/data/db \
  --volume $HOME/bigchaindb_docker/mongodb/data/configdb:/data/configdb \
  --volume $HOME/bigchaindb_docker/tendermint:/tendermint \
  bigchaindb/bigchaindb:all-in-one

Let’s analyze that command:

docker run tells Docker to run some image
--detach run the container in the background
publish 9984:9984 map the host port 9984 to the container port 9984 (the BigchainDB API server)
- 9985 BigchainDB Websocket server
- 27017 Default port for MongoDB
- 26657 Tendermint RPC server
--volume "$HOME/bigchaindb_docker/mongodb:/data" map the host directory $HOME/bigchaindb_docker/mongodb to the container directory /data; this allows us to have the data persisted on the host machine, you can read more in the official Docker documentation
- $HOME/bigchaindb_docker/tendermint:/tendermint to persist Tendermint data.
bigchaindb/bigchaindb:all-in-one the image to use. All the options after the container name are passed on to the entrypoint inside the container.

Verify¶

$ docker ps | grep bigchaindb

Send your first transaction using BigchainDB drivers.

Building Your Own Image¶

Assuming you have Docker installed, you would proceed as follows.

In a terminal shell:

git clone git@github.com:bigchaindb/bigchaindb.git
cd bigchaindb/

Build the Docker image:

docker build --file Dockerfile-all-in-one --tag <tag/name:latest> .

Now you can use your own image to run BigchainDB all-in-one container.

Setting up a network of nodes with the Ansible script¶

You can find one of the installation methods with Ansible on GitHub at:

Ansible script

It allows to install BigchainDB, MongoDB, Tendermint, and python, and then connect nodes into a network. Current tested machine is Ubuntu 18.04.

Set Up BigchainDB, MongoDB and Tendermint¶

We now install and configure software that must run in every BigchainDB node: BigchainDB Server, MongoDB and Tendermint.

Install BigchainDB Server¶

BigchainDB Server requires Python 3.6+, so make sure your system has it.

Install the required OS-level packages:

# For Ubuntu 18.04:
sudo apt install -y python3-pip libssl-dev
# Ubuntu 16.04, and other Linux distros, may require other packages or more packages

BigchainDB Server requires gevent, and to install gevent, you must use pip 19 or later (as of 2019, because gevent now uses manylinux2010 wheels). Upgrade pip to the latest version:

sudo pip3 install -U pip

Now install the latest version of BigchainDB Server. You can find the latest version by going to the BigchainDB project release history page on PyPI. For example, to install version 2.2.2, you would do:

# Change 2.0.0 to the latest version as explained above:
sudo pip3 install bigchaindb==2.2.2

Check that you installed the correct version of BigchainDB Server using bigchaindb --version.

Configure BigchainDB Server¶

To configure BigchainDB Server, run:

bigchaindb configure

The first question is API Server bind? (default `localhost:9984`).

If you’re using NGINX (e.g. if you want HTTPS), then accept the default value (localhost:9984).
If you’re not using NGINX, then enter the value 0.0.0.0:9984

You can accept the default value for all other BigchainDB config settings.

If you’re using NGINX, then you should edit your BigchainDB config file (in $HOME/.bigchaindb by default) and set the following values under "wsserver":

"advertised_scheme": "wss",
"advertised_host": "bnode.example.com",
"advertised_port": 443

where bnode.example.com should be replaced by your node’s actual subdomain.

Install (and Start) MongoDB¶

Install a recent version of MongoDB. BigchainDB Server requires version 3.4 or newer.

sudo apt install mongodb

If you install MongoDB using the above command (which installs the mongodb package), it also configures MongoDB, starts MongoDB (in the background), and installs a MongoDB startup script (so that MongoDB will be started automatically when the machine is restarted).

Note: The mongodb package is not the official MongoDB package from MongoDB the company. If you want to install the official MongoDB package, please see the MongoDB documentation. Note that installing the official package doesn’t also start MongoDB.

Install Tendermint¶

The version of BigchainDB Server described in these docs only works well with Tendermint 0.31.5 (not a higher version number). Install that:

sudo apt install -y unzip
wget https://github.com/tendermint/tendermint/releases/download/v0.31.5/tendermint_v0.31.5_linux_amd64.zip
unzip tendermint_v0.31.5_linux_amd64.zip
rm tendermint_v0.31.5_linux_amd64.zip
sudo mv tendermint /usr/local/bin

Start Configuring Tendermint¶

You won’t be able to finish configuring Tendermint until you have some information from the other nodes in the network, but you can start by doing:

tendermint init

Set Up NGINX¶

If you don’t want HTTPS (for communications between the external world and your node), then you can skip all the NGINX steps on this page.

Note: This simple deployment template uses NGINX for more than just HTTPS. For example, it also does basic rate limiting.

Install NGINX¶

SSH into your machine and install NGINX:

sudo apt update
sudo apt install nginx

Configure & Reload NGINX¶

Get an SSL certificate for your node’s subdomain (such as bnode.example.com).

Copy the SSL private key into /etc/nginx/ssl/cert.key
Create a “PEM file” (text file) by concatenating your SSL certificate with all intermediate certificates (in that order, with the intermediate certs last).
Copy that PEM file into /etc/nginx/ssl/cert.pem
In the bigchaindb/bigchaindb repository on GitHub, find the file nginx/nginx.conf and copy its contents to /etc/nginx/nginx.conf on your machine (i.e. replace the existing file there).
Edit that file (/etc/nginx/nginx.conf): replace the two instances of the string example.testnet2.com with your chosen subdomain (such as bnode.example.com).
Reload NGINX by doing:
```
sudo service nginx reload
```

Configuration Settings¶

Every BigchainDB Server configuration setting has two names: a config-file name and an environment variable name. For example, one of the settings has the config-file name database.host and the environment variable name BIGCHAINDB_DATABASE_HOST. Here are some more examples:

database.port ↔ BIGCHAINDB_DATABASE_PORT

database.keyfile_passphrase ↔ BIGCHAINDB_DATABASE_KEYFILE_PASSPHRASE

server.bind ↔ BIGCHAINDB_SERVER_BIND

The value of each setting is determined according to the following rules:

If it’s set by an environment variable, then use that value
Otherwise, if it’s set in a local config file, then use that value
Otherwise, use the default value

The local config file is $HOME/.bigchaindb by default (a file which might not even exist), but you can tell BigchainDB to use a different file by using the -c command-line option, e.g. bigchaindb -c path/to/config_file.json start or using the BIGCHAINDB_CONFIG_PATH environment variable, e.g. BIGHAINDB_CONFIG_PATH=.my_bigchaindb_config bigchaindb start. Note that the -c command line option will always take precedence if both the BIGCHAINDB_CONFIG_PATH and the -c command line option are used.

You can read the current default values in the file bigchaindb/__init__.py. (The link is to the latest version.)

Running bigchaindb -y configure localmongodb will generate a local config file in $HOME/.bigchaindb with all the default values.

database.*¶

The settings with names of the form database.* are for the backend database (currently only MongoDB). They are:

database.backend can only be localmongodb, currently.
database.host is the hostname (FQDN) of the backend database.
database.port is self-explanatory.
database.name is a user-chosen name for the database inside MongoDB, e.g. bigchain.
database.connection_timeout is the maximum number of milliseconds that BigchainDB will wait before giving up on one attempt to connect to the backend database.
database.max_tries is the maximum number of times that BigchainDB will try to establish a connection with the backend database. If 0, then it will try forever.
database.replicaset is the name of the MongoDB replica set. The default value is null because in BigchainDB 2.0+, each BigchainDB node has its own independent MongoDB database and no replica set is necessary. Replica set must already exist if this option is configured, BigchainDB will not create it.
database.ssl must be true or false. It tells BigchainDB Server whether it should connect to MongoDB using TLS/SSL or not. The default value is false.

There are three ways for BigchainDB Server to authenticate itself with MongoDB (or a specific MongoDB database): no authentication, username/password, and x.509 certificate authentication.

No Authentication

If you use all the default BigchainDB configuration settings, then no authentication will be used.

Username/Password Authentication

To use username/password authentication, a MongoDB instance must already be running somewhere (maybe in another machine), it must already have a database for use by BigchainDB (usually named bigchain, which is the default database.name), and that database must already have a “readWrite” user with associated username and password. To create such a user, login to your MongoDB instance as Admin and run the following commands:

use <database.name>
db.createUser({user: "<database.login>", pwd: "<database.password>", roles: [{role: "readWrite", db: "<database.name>"}]})

database.login is the user’s username.
database.password is the user’s password, given in plaintext.
database.ca_cert, database.certfile, database.keyfile, database.crlfile, and database.keyfile_passphrase are not used so they can have their default values.

x.509 Certificate Authentication

To use x.509 certificate authentication, a MongoDB instance must be running somewhere (maybe in another machine), it must already have a database for use by BigchainDB (usually named bigchain, which is the default database.name), and that database must be set up to use x.509 authentication. See the MongoDB docs about how to do that.

database.login is the user’s username.
database.password isn’t used so the default value (null) is fine.
database.ca_cert, database.certfile, database.keyfile and database.crlfile are the paths to the CA, signed certificate, private key and certificate revocation list files respectively.
database.keyfile_passphrase is the private key decryption passphrase, specified in plaintext.

Example using environment variables

export BIGCHAINDB_DATABASE_BACKEND=localmongodb
export BIGCHAINDB_DATABASE_HOST=localhost
export BIGCHAINDB_DATABASE_PORT=27017
export BIGCHAINDB_DATABASE_NAME=database8
export BIGCHAINDB_DATABASE_CONNECTION_TIMEOUT=5000
export BIGCHAINDB_DATABASE_MAX_TRIES=3

Default values

If (no environment variables were set and there’s no local config file), or you used bigchaindb -y configure localmongodb to create a default local config file for a localmongodb backend, then the defaults will be:

"database": {
    "backend": "localmongodb",
    "host": "localhost",
    "port": 27017,
    "name": "bigchain",
    "connection_timeout": 5000,
    "max_tries": 3,
    "replicaset": null,
    "login": null,
    "password": null
    "ssl": false,
    "ca_cert": null,
    "certfile": null,
    "keyfile": null,
    "crlfile": null,
    "keyfile_passphrase": null,
}

server.*¶

server.bind, server.loglevel and server.workers are settings for the Gunicorn HTTP server, which is used to serve the HTTP client-server API.

server.bind is where to bind the Gunicorn HTTP server socket. It’s a string. It can be any valid value for Gunicorn’s bind setting. For example:

If you want to allow IPv4 connections from anyone, on port 9984, use 0.0.0.0:9984
If you want to allow IPv6 connections from anyone, on port 9984, use [::]:9984

In a production setting, we recommend you use Gunicorn behind a reverse proxy server such as NGINX. If Gunicorn and the reverse proxy are running on the same machine, then you can use localhost:9984 (the default value), meaning Gunicorn will talk to the reverse proxy on port 9984. The reverse proxy could then be bound to port 80 (for HTTP) or port 443 (for HTTPS), so that external clients would connect using that port. For example:

[External clients]—(port 443)—[NGINX]—(port 9984)—[Gunicorn / BigchainDB Server]

If Gunicorn and the reverse proxy are running on different machines, then server.bind should be hostname:9984, where hostname is the IP address or FQDN of the reverse proxy.

There’s more information about deploying behind a reverse proxy in the Gunicorn documentation. (They call it a proxy.)

server.loglevel sets the log level of Gunicorn’s Error log outputs. See Gunicorn’s documentation for more information.

server.workers is the number of worker processes for handling requests. If set to None, the value will be (2 × cpu_count + 1). Each worker process has a single thread. The HTTP server will be able to handle server.workers requests simultaneously.

Example using environment variables

export BIGCHAINDB_SERVER_BIND=0.0.0.0:9984
export BIGCHAINDB_SERVER_LOGLEVEL=debug
export BIGCHAINDB_SERVER_WORKERS=5

Example config file snippet

"server": {
    "bind": "0.0.0.0:9984",
    "loglevel": "debug",
    "workers": 5,
}

Default values (from a config file)

"server": {
    "bind": "localhost:9984",
    "loglevel": "info",
    "workers": null,
}

wsserver.*¶

wsserver.scheme, wsserver.host and wsserver.port¶

These settings are for the aiohttp server, which is used to serve the WebSocket Event Stream API. wsserver.scheme should be either "ws" or "wss" (but setting it to "wss" does not enable SSL/TLS). wsserver.host is where to bind the aiohttp server socket and wsserver.port is the corresponding port. If you want to allow connections from anyone, on port 9985, set wsserver.host to 0.0.0.0 and wsserver.port to 9985.

Example using environment variables

export BIGCHAINDB_WSSERVER_SCHEME=ws
export BIGCHAINDB_WSSERVER_HOST=0.0.0.0
export BIGCHAINDB_WSSERVER_PORT=9985

Example config file snippet

"wsserver": {
    "scheme": "wss",
    "host": "0.0.0.0",
    "port": 65000
}

Default values (from a config file)

"wsserver": {
    "scheme": "ws",
    "host": "localhost",
    "port": 9985
}

wsserver.advertised_scheme, wsserver.advertised_host and wsserver.advertised_port¶

These settings are for the advertising the Websocket URL to external clients in the root API endpoint. These configurations might be useful if your deployment is hosted behind a firewall, NAT, etc. where the exposed public IP or domain is different from where BigchainDB is running.

Example using environment variables

export BIGCHAINDB_WSSERVER_ADVERTISED_SCHEME=wss
export BIGCHAINDB_WSSERVER_ADVERTISED_HOST=mybigchaindb.com
export BIGCHAINDB_WSSERVER_ADVERTISED_PORT=443

Example config file snippet

"wsserver": {
    "advertised_scheme": "wss",
    "advertised_host": "mybigchaindb.com",
    "advertised_port": 443
}

Default values (from a config file)

"wsserver": {
    "advertised_scheme": "ws",
    "advertised_host": "localhost",
    "advertised_port": 9985
}

log.*¶

The log.* settings are to configure logging.

Example

{
    "log": {
        "file": "/var/log/bigchaindb.log",
        "error_file": "/var/log/bigchaindb-errors.log",
        "level_console": "info",
        "level_logfile": "info",
        "datefmt_console": "%Y-%m-%d %H:%M:%S",
        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",
        "fmt_console": "%(asctime)s [%(levelname)s] (%(name)s) %(message)s",
        "fmt_logfile": "%(asctime)s [%(levelname)s] (%(name)s) %(message)s",
        "granular_levels": {}
}

Default values

{
    "log": {
        "file": "~/bigchaindb.log",
        "error_file": "~/bigchaindb-errors.log",
        "level_console": "info",
        "level_logfile": "info",
        "datefmt_console": "%Y-%m-%d %H:%M:%S",
        "datefmt_logfile": "%Y-%m-%d %H:%M:%S",
        "fmt_logfile": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "fmt_console": "[%(asctime)s] [%(levelname)s] (%(name)s) %(message)s (%(processName)-10s - pid: %(process)d)",
        "granular_levels": {}
}

log.file¶

The full path to the file where logs should be written. The user running bigchaindb must have write access to the specified path.

Log rotation: Log files have a size limit of about 200 MB and will be rotated up to five times. For example, if log.file is set to "~/bigchain.log", then logs would always be written to bigchain.log. Each time the file bigchain.log reaches 200 MB it will be closed and renamed bigchain.log.1. If bigchain.log.1 and bigchain.log.2 already exist they would be renamed bigchain.log.2 and bigchain.log.3. This pattern would be applied up to bigchain.log.5 after which bigchain.log.5 would be overwritten by bigchain.log.4, thus ending the rotation cycle of whatever logs were in bigchain.log.5.

log.error_file¶

Similar to log.file (see above), this is the full path to the file where error logs should be written.

log.level_console¶

The log level used to log to the console. Possible allowed values are the ones defined by Python, but case-insensitive for the sake of convenience:

"critical", "error", "warning", "info", "debug", "notset"

log.level_logfile¶

The log level used to log to the log file. Possible allowed values are the ones defined by Python, but case-insensitive for the sake of convenience:

"critical", "error", "warning", "info", "debug", "notset"

log.datefmt_console¶

The format string for the date/time portion of a message, when logged to the console.

For more information on how to construct the format string please consult the table under Python’s documentation of time.strftime(format[, t]).

log.datefmt_logfile¶

The format string for the date/time portion of a message, when logged to a log file.

For more information on how to construct the format string please consult the table under Python’s documentation of time.strftime(format[, t]).

log.fmt_console¶

A string used to format the log messages when logged to the console.

For more information on possible formatting options please consult Python’s documentation on LogRecord attributes.

log.fmt_logfile¶

A string used to format the log messages when logged to a log file.

For more information on possible formatting options please consult Python’s documentation on LogRecord attributes.

log.granular_levels¶

Log levels for BigchainDB’s modules. This can be useful to control the log level of specific parts of the application. As an example, if you wanted the logging of the core.py module to be more verbose, you would set the configuration shown in the example below.

Example

{
    "log": {
        "granular_levels": {
            "bichaindb.core": "debug"
        }
}

Default value

{}

tendermint.*¶

The settings with names of the form tendermint.* tell BigchainDB Server where it can connect to the node’s Tendermint instance.

tendermint.host is the hostname (FQDN)/IP address of the Tendermint instance.
tendermint.port is self-explanatory.

Example using environment variables

export BIGCHAINDB_TENDERMINT_HOST=tendermint
export BIGCHAINDB_TENDERMINT_PORT=26657

Default values

"tendermint": {
    "host": "localhost",
    "port": 26657
}

Command Line Interface (CLI)¶

The command-line command to interact with BigchainDB Server is bigchaindb.

bigchaindb --help¶

Show help for the bigchaindb command. bigchaindb -h does the same thing.

bigchaindb --version¶

Show the version number. bigchaindb -v does the same thing.

bigchaindb configure¶

Generate a local configuration file (which can be used to set some or all BigchainDB node configuration settings). It will ask you for the values of some configuration settings. If you press Enter for a value, it will use the default value.

At this point, only one database backend is supported: localmongodb.

If you use the -c command-line option, it will generate the file at the specified path:

bigchaindb -c path/to/new_config.json configure localmongodb

If you don’t use the -c command-line option, the file will be written to $HOME/.bigchaindb (the default location where BigchainDB looks for a config file, if one isn’t specified).

If you use the -y command-line option, then there won’t be any interactive prompts: it will use the default values for all the configuration settings.

bigchaindb -y configure localmongodb

bigchaindb show-config¶

Show the values of the BigchainDB node configuration settings.

bigchaindb init¶

Create a backend database (local MongoDB), all database tables/collections, various backend database indexes, and the genesis block.

bigchaindb drop¶

Drop (erase) the backend database (the local MongoDB database used by this node). You will be prompted to make sure. If you want to force-drop the database (i.e. skipping the yes/no prompt), then use bigchaindb -y drop

bigchaindb start¶

Start BigchainDB. It always begins by trying a bigchaindb init first. See the documentation for bigchaindb init. The database initialization step is optional and can be skipped by passing the --no-init flag, i.e. bigchaindb start --no-init.

Options¶

The log level for the console can be set via the option --log-level or its abbreviation -l. Example:

$ bigchaindb --log-level INFO start

The allowed levels are DEBUG, INFO, WARNING, ERROR, and CRITICAL. For an explanation regarding these levels please consult the Logging Levels section of Python’s documentation.

For a more fine-grained control over the logging configuration you can use the configuration file as documented under Configuration Settings.

bigchaindb election¶

Manage elections to govern the BigchainDB network. The specifics of the election process are defined in BEP-18.

Election management is broken into several subcommands. Below is the command line syntax for each of them.

election new¶

Create a new election which proposes a change to the BigchainDB network.

If the command succeeds, it will post an election transaction and output election_id.

The election proposal consists of vote tokens allocated to every current validator proportional to his voting power. Validators spend their votes to approve the election using the election-approve command.

Every election has a type. Currently supported types are upsert-validator and chain-migration. Their transaction operations are VALIDATOR_ELECTION and CHAIN_MIGRATION accordingly. See below for how to create an election of a particular type.

Note that elections can only be proposed and approved by existing validators.

election new upsert-validator¶

Create an election to add, update, or remove a validator.

$ bigchaindb election new upsert-validator <public-key> <power> <node-id> --private-key <path-to-the-private-key>

<public-key> is the public key of the node to be added/updated/removed. The encoding and type of the key have to match those specified in genesis.json in the supported Tendermint version.
<power> is the new power for the validator. To remove the validator, set the power to 0.
<node-id> is the node identifier from Tendermint. A node operator can learn his node identifier by executing tendermint show_node_id.
<path-to-the-private-key> is the path to the private key of the validator who proposes the election. Tendermint places it at .tendermint/config/priv_validator.json.

Example:

$ bigchaindb election new upsert-validator HHG0IQRybpT6nJMIWWFWhMczCLHt6xcm7eP52GnGuPY= 1 fb7140f03a4ffad899fabbbf655b97e0321add66 --private-key /home/user/.tendermint/config/priv_validator.json
[SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa

A successful execution of the above command does not imply the validator set has been updated but rather the proposal has been accepted by the network. Once election_id has been generated, the proposer should share it with other validators of the network (e.g. via email) and ask them to approve the proposal.

Note that election proposers do not automatically approve elections by proposing them.

For more details about how validator set changes work, refer to BEP-21.

election new chain-migration¶

Create an election to halt block production, to coordinate on making a Tendermint upgrade with a backwards-incompatible chain.

$ bigchaindb election new chain-migration --private-key <path-to-the-private-key>

<path-to-the-private-key> is the path to the private key of the validator who proposes the election. Tendermint places it at .tendermint/config/priv_validator.json.

Example:

$ bigchaindb election new migration --private-key /home/user/.tendermint/config/priv_validator.json
[SUCCESS] Submitted proposal with id: 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa

Concluded chain migration elections halt block production at whichever block height they are approved. Afterwards, validators are supposed to upgrade Tendermint, set new chain_id, app_hash, and validators (to learn these values, use the election show command) in genesis.json, make and save a MongoDB dump, and restart the system.

For more details about how chain migrations work, refer to Type 3 scenarios in BEP-42.

election approve¶

Approve an election by voting for it. The command places a VOTE transaction, spending all of the validator’s vote tokens to the election address.

$ bigchaindb election approve <election-id> --private-key <path-to-the-private-key>

election-id is the election identifier the approval is given for.
<path-to-the-private-key> is the path to the private key of the validator who votes for the election. Tendermint places it at .tendermint/config/priv_validator.json.

Example:

$ bigchaindb election approve 04a067582cf03eba2b53b82e4adb5ece424474cbd4f7183780855a93ac5e3caa --private-key /home/user/.tendermint/config/priv_validator.json
[SUCCESS] Your vote has been submitted

Once a proposal has been approved by the sufficient amount of validators (contributing more than 2/3 of the total voting power), the proposed change is applied to the network.

election show¶

Retrieves the information about elections.

$ bigchaindb election show <election-id>

status=<status>

status has three possible values:

ongoing, if the election can be concluded but has not yet collected enough votes,
concluded, if the election has been concluded,
inconclusive, if the validator set changed while the election was in process, rendering it undecidable.

After a chain migration is concluded, the show command also outputs chain_id, app_hash, and validators for genesis.json of the new chain.

bigchaindb tendermint-version¶

Show the Tendermint versions supported by BigchainDB server.

$ bigchaindb tendermint-version
{
    "description": "BigchainDB supports the following Tendermint version(s)",
    "tendermint": [
        "0.22.8"
    ]
}

Troubleshooting¶

General Tips¶

Check the BigchainDB, Tendermint and MongoDB logs. For help with that, see the page about Logging and Log Rotation.
Try Googling the error message.

Tendermint Tips¶

Configure Tendermint to create no empty blocks.
Store the Tendermint data on a fast drive. You can do that by changing the location of TMHOME to be on the fast drive.

See the Tendermint tips in the vrde/notes repository.

Resolving Tendermint Connectivity Problems¶

To check which nodes your node is connected to (via Tendermint protocols), do:

# if you don't have jq installed, then install it
sudo apt install jq
# then do
curl -s localhost:26657/net_info | jq ".result.peers[].node_info | {id, listen_addr, moniker}"

Note: Tendermint has other endpoints besides /net_info: see the Tendermint RPC docs.

If you’re running your network inside a private network, e.g. with IP addresses of the form 192.168.x.y, then you may have to change the following setting in config.toml:

addr_book_strict = false

Refreshing Your Node¶

If you want to refresh your node back to a fresh empty state, then your best bet is to terminate it and deploy a new machine, but if that’s not an option, then you can:

drop the bigchain database in MongoDB using bigchaindb drop (but that only works if MongoDB is running)
reset Tendermint using tendermint unsafe_reset_all
delete the directory $HOME/.tendermint

Shutting Down BigchainDB¶

If you want to stop/kill BigchainDB, you can do so by sending SIGINT, SIGQUIT or SIGTERM to the running BigchainDB process(es). Depending on how you started BigchainDB i.e. foreground or background. e.g. you started BigchainDB in the background as mentioned above in the guide:

$ nohup bigchaindb start 2>&1 > bigchaindb.log &

$ # Check the PID of the main BigchainDB process
$ ps -ef | grep bigchaindb
<user>    *<pid> <ppid>   <C> <STIME> <tty>        <time> bigchaindb
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> gunicorn: master [bigchaindb_gunicorn]
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> bigchaindb_ws
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> bigchaindb_ws_to_tendermint
<user>     <pid> <ppid>*  <C> <STIME> <tty>        <time> bigchaindb_exchange
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
<user>     <pid> <ppid>   <C> <STIME> <tty>        <time> gunicorn: worker [bigchaindb_gunicorn]
...

$ # Send any of the above mentioned signals to the parent/root process(marked with `*` for clarity)
# Sending SIGINT
$ kill -2 <bigchaindb_parent_pid>

$ # OR

# Sending SIGTERM
$ kill -15 <bigchaindb_parent_pid>

$ # OR

# Sending SIGQUIT
$ kill -3 <bigchaindb_parent_pid>

# If you want to kill all the processes by name yourself
$ pgrep bigchaindb | xargs kill -9

If you started BigchainDB in the foreground, a Ctrl + C or Ctrl + Z would shut down BigchainDB.

Member: Dynamically Add or Remove Validators¶

One member can make a proposal to call an election to add a validator, remove a validator, or change the voting power of a validator. They then share the election/proposal ID with all the other members. Once more than 2/3 of the voting power votes yes, the proposed change comes into effect. The commands to create a new election/proposal, to approve an election/proposal, and to get the current status of an election/proposal can be found in the documentation about the bigchaindb election subcommands.

Logging¶

See the page in the Appendices about logging and log rotation.

Production Nodes¶

Production Node Requirements¶

This page is about the requirements of BigchainDB Server. You can find the requirements of MongoDB, Tendermint and other production node components in the documentation for that software.

OS Requirements¶

BigchainDB Server requires Python 3.5+ and Python 3.5+ will run on any modern OS, but we recommend using an LTS version of Ubuntu Server or a similarly server-grade Linux distribution.

Don’t use macOS (formerly OS X, formerly Mac OS X), because it’s not a server-grade operating system. Also, BigchainDB Server uses the Python multiprocessing package and some functionality in the multiprocessing package doesn’t work on Mac OS X.

General Considerations¶

BigchainDB Server runs many concurrent processes, so more RAM and more CPU cores is better.

As mentioned on the page about production node components, every machine running BigchainDB Server should be running an NTP daemon.

Production Node Assumptions¶

Be sure you know the key BigchainDB terminology:

BigchainDB node, BigchainDB network and BigchainDB consortium

Note that there are a few kinds of nodes:

A dev/test node is a node created by a developer working on BigchainDB Server, e.g. for testing new or changed code. A dev/test node is typically run on the developer’s local machine.
A bare-bones node is a node deployed in the cloud, either as part of a testing network or as a starting point before upgrading the node to be production-ready.
A production node is a node that is part of a consortium’s BigchainDB network. A production node has the most components and requirements.

We make some assumptions about production nodes:

Each production node is set up and managed by an experienced professional system administrator or a team of them.
Each production node in a network is managed by a different person or team.

Production Node Components¶

A production BigchainDB node must include:

BigchainDB Server
MongoDB Server 3.4+ (mongod)
Tendermint
Storage for MongoDB and Tendermint

It could also include several other components, including:

NGINX or similar, to provide authentication, rate limiting, etc.
An NTP daemon running on all machines running BigchainDB Server or mongod, and possibly other machines
Probably not MongoDB Automation Agent. It’s for automating the deployment of an entire MongoDB cluster.
MongoDB Monitoring Agent
MongoDB Backup Agent
Log aggregation software
Monitoring software
Maybe more

The relationship between the main components is illustrated below.

_images/Node-components.png Components of a production node

Production Node Security & Privacy¶

Here are some references about how to secure an Ubuntu 18.04 server:

Also, here are some recommendations a node operator can follow to enhance the privacy of the data coming to, stored on, and leaving their node:

Ensure that all data stored on a node is encrypted at rest, e.g. using full disk encryption. This can be provided as a service by the operating system, transparently to BigchainDB, MongoDB and Tendermint.
Ensure that all data is encrypted in transit, i.e. enforce using HTTPS for the HTTP API and the Websocket API. This can be done using NGINX or similar, as we do with the IPDB Testnet.

Using a Reverse Proxy¶

You may want to:

rate limit inbound HTTP requests,
authenticate/authorize inbound HTTP requests,
block requests with an HTTP request body that’s too large, or
enable HTTPS (TLS) between your users and your node.

While we could have built all that into BigchainDB Server, we didn’t, because you can do all that (and more) using a reverse proxy such as NGINX or HAProxy. (You would put it in front of your BigchainDB Server, so that all inbound HTTP requests would arrive at the reverse proxy before maybe being proxied onwards to your BigchainDB Server.) For detailed instructions, see the documentation for your reverse proxy.

Below, we note how a reverse proxy can be used to do some BigchainDB-specific things.

You may also be interested in our NGINX configuration file template (open source, on GitHub).

Enforcing a Max Transaction Size¶

The BigchainDB HTTP API has several endpoints, but only one of them, the POST /transactions endpoint, expects a non-empty HTTP request body: the transaction being submitted by the user.

If you want to enforce a maximum-allowed transaction size (discarding any that are larger), then you can do so by configuring a maximum request body size in your reverse proxy. For example, NGINX has the client_max_body_size configuration setting. You could set it to 15 kB with the following line in your NGINX config file:

client_max_body_size 15k;

For more information, see the NGINX docs about client_max_body_size.

Note: By enforcing a maximum transaction size, you indirectly enforce a maximum crypto-conditions complexity.

Release Notes¶

You can find a list of all BigchainDB Server releases and release notes on GitHub at:

https://github.com/bigchaindb/bigchaindb/releases

The CHANGELOG.md file contains much the same information, but it also has notes about what to expect in the next release.

We also have a roadmap document in ROADMAP.md.

Network setup¶

There are several ways to setup a network. You can use the Kubernetes deployment template in this section, or use the Ansible solution in the Contributing section. Also, you can setup a single node on your machine and connect to an existing network.

BigchainDB Networks¶

A BigchainDB network is a set of connected BigchainDB nodes, managed by a BigchainDB consortium (i.e. an organization). Those terms are defined in the BigchainDB Terminology page.

Consortium Structure & Governance¶

The consortium might be a company, a foundation, a cooperative, or some other form of organization. It must make many decisions, e.g. How will new members be added? Who can read the stored data? What kind of data will be stored? A governance process is required to make those decisions, and therefore one of the first steps for any new consortium is to specify its governance process (if one doesn’t already exist). This documentation doesn’t explain how to create a consortium, nor does it outline the possible governance processes.

It’s worth noting that the decentralization of a BigchainDB network depends, to some extent, on the decentralization of the associated consortium. See the pages about decentralization and node diversity.

DNS Records and SSL Certificates¶

We now describe how we set up the external (public-facing) DNS records for a BigchainDB network. Your consortium may opt to do it differently. There were several goals:

Allow external users/clients to connect directly to any BigchainDB node in the network (over the internet), if they want.
Each BigchainDB node operator should get an SSL certificate for their BigchainDB node, so that their BigchainDB node can serve the BigchainDB HTTP API via HTTPS. (The same certificate might also be used to serve the WebSocket API.)
There should be no sharing of SSL certificates among BigchainDB node operators.
Optional: Allow clients to connect to a “random” BigchainDB node in the network at one particular domain (or subdomain).

Node Operator Responsibilities¶

Register a domain (or use one that you already have) for your BigchainDB node. You can use a subdomain if you like. For example, you might opt to use abc-org73.net, api.dynabob8.io or figmentdb3.ninja.
Get an SSL certificate for your domain or subdomain, and properly install it in your node (e.g. in your NGINX instance).
Create a DNS A Record mapping your domain or subdomain to the public IP address of your node (i.e. the one that serves the BigchainDB HTTP API).

Consortium Responsibilities¶

Optional: The consortium managing the BigchainDB network could register a domain name and set up CNAME records mapping that domain name (or one of its subdomains) to each of the nodes in the network. For example, if the consortium registered bdbnetwork.io, they could set up CNAME records like the following:

CNAME record mapping api.bdbnetwork.io to abc-org73.net
CNAME record mapping api.bdbnetwork.io to api.dynabob8.io
CNAME record mapping api.bdbnetwork.io to figmentdb3.ninja

How to Set Up a BigchainDB Network¶

You can setup or connect to a network once you have a single node running. Until now, everything could be done by a node operator, by themselves. Now the node operators, also called Members, must share some information with each other, so they can form a network.

There is one special Member who helps coordinate everyone: the Coordinator.

Member: Connect to the Other Members¶

At this point the Member should have received the genesis.json file.

The Member must copy the genesis.json file into their local $HOME/.tendermint/config directory. Every Member now shares the same chain_id and genesis_time (used to identify the Network), and the same list of validators.

Each Member must edit their $HOME/.tendermint/config/config.toml file and make the following changes:

moniker = "Name of our node"
create_empty_blocks = false
log_level = "main:info,state:info,*:error"

persistent_peers = "<Member 1 node id>@<Member 1 hostname>:26656,\
<Member 2 node id>@<Member 2 hostname>:26656,\
<Member N node id>@<Member N hostname>:26656,"

send_rate = 102400000
recv_rate = 102400000

recheck = false

Note: The list of persistent_peers doesn’t have to include all nodes in the network.

Member: Start MongoDB¶

If you installed MongoDB using sudo apt install mongodb, then MongoDB should already be running in the background. You can check using systemctl status mongodb.

If MongoDB isn’t running, then you can start it using the command mongod, but that will run it in the foreground. If you want to run it in the background (so it will continue running after you logout), you can use mongod --fork --logpath /var/log/mongodb.log. (You might have to create the /var/log directory if it doesn’t already exist.)

If you installed MongoDB using sudo apt install mongodb, then a MongoDB startup script should already be installed (so MongoDB will start automatically when the machine is restarted). Otherwise, you should install a startup script for MongoDB.

Member: Start BigchainDB and Tendermint Using Monit¶

This section describes how to manage the BigchainDB and Tendermint processes using Monit, a small open-source utility for managing and monitoring Unix processes. BigchainDB and Tendermint are managed together, because if BigchainDB is stopped (or crashes) and is restarted, Tendermint won’t try reconnecting to it. (That’s not a bug. It’s just how Tendermint works.)

Install Monit:

sudo apt install monit

If you installed the bigchaindb Python package as above, you should have the bigchaindb-monit-config script in your PATH now. Run the script to build a configuration file for Monit:

bigchaindb-monit-config

Run Monit as a daemon, instructing it to wake up every second to check on processes:

monit -d 1

Monit will run the BigchainDB and Tendermint processes and restart them when they crash. If the root bigchaindb_ process crashes, Monit will also restart the Tendermint process.

You can check the status by running monit status or monit summary.

By default, it will collect program logs into the ~/.bigchaindb-monit/logs folder.

To learn more about Monit, use monit -h (help) or read the Monit documentation.

Check bigchaindb-monit-config -h if you want to arrange a different folder for logs or some of the Monit internal artifacts.

If you want to start and manage the BigchainDB and Tendermint processes yourself, then look inside the file bigchaindb/pkg/scripts/bigchaindb-monit-config to see how it starts BigchainDB and Tendermint.

How Others Can Access Your Node¶

If you followed the above instructions, then your node should be publicly-accessible with BigchainDB Root URL https://hostname or http://hostname:9984. That is, anyone can interact with your node using the BigchainDB HTTP API exposed at that address. The most common way to do that is to use one of the BigchainDB Drivers.

Kubernetes Deployment Template¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This section outlines a way to deploy a BigchainDB node (or BigchainDB network) on Microsoft Azure using Kubernetes. You may choose to use it as a template or reference for your own deployment, but we make no claim that it is suitable for your purposes. Feel free change things to suit your needs or preferences.

Overview¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page summarizes some steps to go through to set up a BigchainDB network. You can modify them to suit your needs.

Generate All Shared BigchainDB Setup Parameters¶

There are some shared BigchainDB setup paramters that every node operator in the consortium shares because they are properties of the Tendermint network. They look like this:

# Tendermint data
BDB_PERSISTENT_PEERS='bdb-instance-1,bdb-instance-2,bdb-instance-3,bdb-instance-4'
BDB_VALIDATORS='bdb-instance-1,bdb-instance-2,bdb-instance-3,bdb-instance-4'
BDB_VALIDATOR_POWERS='10,10,10,10'
BDB_GENESIS_TIME='0001-01-01T00:00:00Z'
BDB_CHAIN_ID='test-chain-rwcPML'

Those paramters only have to be generated once, by one member of the consortium. That person will then share the results (Tendermint setup parameters) with all the node operators.

The above example parameters are for a network of 4 initial (seed) nodes. Note how BDB_PERSISTENT_PEERS, BDB_VALIDATORS and BDB_VALIDATOR_POWERS are lists with 4 items each. If your consortium has a different number of initial nodes, then those lists should have that number or items. Use 10 for all the power values.

To generate a BDB_GENESIS_TIME and a BDB_CHAIN_ID, you can do this:

$ mkdir $(pwd)/tmdata
$ docker run --rm -v $(pwd)/tmdata:/tendermint/config tendermint/tendermint:v0.31.5 init
$ cat $(pwd)/tmdata/genesis.json

You should see something that looks like:

{"genesis_time": "0001-01-01T00:00:00Z",
 "chain_id": "test-chain-bGX7PM",
 "validators": [
     {"pub_key":
         {"type": "ed25519",
          "data": "4669C4B966EB8B99E45E40982B2716A9D3FA53B54C68088DAB2689935D7AF1A9"},
      "power": 10,
      "name": ""}
 ],
 "app_hash": ""
}

The value with "genesis_time" is BDB_GENESIS_TIME and the value with "chain_id" is BDB_CHAIN_ID.

Now you have all the BigchainDB setup parameters and can share them with all of the node operators. (They will put them in their vars file. We’ll say more about that file below.)

Things Each Node Operator Must Do¶

1. Make up an FQDN for your BigchainDB node (e.g. mynode.mycorp.com). This is where external users will access the BigchainDB HTTP API, for example. Make sure you’ve registered the associated domain name (e.g. mycorp.com).

Get an SSL certificate for your BigchainDB node’s FQDN. Also get the root CA certificate and all intermediate certificates. They should all be provided by your SSL certificate provider. Put all those certificates together in one certificate chain file in the following order:

Domain certificate (i.e. the one you ordered for your FQDN)
All intermediate certificates
Root CA certificate

DigiCert has a web page explaining certificate chains.

You will put the path to that certificate chain file in the vars file, when you configure your node later.

2a. If your BigchainDB node will use 3scale for API authentication, monitoring and billing, you will need all relevant 3scale settings and credentials.

2b. If your BigchainDB node will not use 3scale, then write authorization will be granted to all POST requests with a secret token in the HTTP headers. (All GET requests are allowed to pass.) You can make up that SECRET_TOKEN now. For example, superSECRET_token4-POST*requests. You will put it in the vars file later. Every BigchainDB node in a BigchainDB network can have a different secret token. To make an HTTP POST request to your BigchainDB node, you must include an HTTP header named X-Secret-Access-Token and set it equal to your secret token, e.g.

X-Secret-Access-Token: superSECRET_token4-POST*requests

3. Deploy a Kubernetes cluster for your BigchainDB node. We have some instructions for how to Deploy a Kubernetes cluster on Azure.

Warning

In theory, you can deploy your BigchainDB node to any Kubernetes cluster, but there can be differences between different Kubernetes clusters, especially if they are running different versions of Kubernetes. We tested this Kubernetes Deployment Template on Azure ACS in February 2018 and at that time ACS was deploying a Kubernetes 1.7.7 cluster. If you can force your cluster to have that version of Kubernetes, then you’ll increase the likelihood that everything will work.

4. Deploy your BigchainDB node inside your new Kubernetes cluster. You will fill up the vars file, then you will run a script which reads that file to generate some Kubernetes config files, you will send those config files to your Kubernetes cluster, and then you will deploy all the stuff that you need to have a BigchainDB node.

⟶ Proceed to deploy your BigchainDB node.

How to Set Up a Self-Signed Certificate Authority¶

This page enumerates the steps we use to set up a self-signed certificate authority (CA). This is something that only needs to be done once per BigchainDB network, by the organization managing the network, i.e. the CA is for the whole network. We use Easy-RSA.

Step 1: Install & Configure Easy-RSA¶

First create a directory for the CA and cd into it:

mkdir bdb-node-ca

cd bdb-node-ca

Then install and configure Easy-RSA in that directory.

Step 2: Create a Self-Signed CA¶

You can create a self-signed CA by going to the bdb-node-ca/easy-rsa-3.0.1/easyrsa3 directory and using:

./easyrsa init-pki

./easyrsa build-ca

You will also be asked to enter a PEM pass phrase (for encrypting the ca.key file). Make sure to securely store that PEM pass phrase. If you lose it, you won’t be able to add or remove entities from your PKI infrastructure in the future.

You will be prompted to enter the Distinguished Name (DN) information for this CA. For each field, you can accept the default value [in brackets] by pressing Enter.

Warning

Don’t accept the default value of OU (IT). Instead, enter the value ROOT-CA.

While Easy-RSA CA is a valid and acceptable Common Name, you should probably enter a name based on the name of the managing organization, e.g. Omega Ledger CA.

Tip: You can get help with the easyrsa command (and its subcommands) by using the subcommand ./easyrsa help

Step 3: Create an Intermediate CA¶

TODO

Step 4: Generate a Certificate Revocation List¶

You can generate a Certificate Revocation List (CRL) using:

./easyrsa gen-crl

You will need to run this command every time you revoke a certificate. The generated crl.pem needs to be uploaded to your infrastructure to prevent the revoked certificate from being used again.

Step 5: Secure the CA¶

The security of your infrastructure depends on the security of this CA.

Ensure that you restrict access to the CA and enable only legitimate and required people to sign certificates and generate CRLs.
Restrict access to the machine where the CA is hosted.
Many certificate providers keep the CA offline and use a rotating intermediate CA to sign and revoke certificates, to mitigate the risk of the CA getting compromised.
In case you want to destroy the machine where you created the CA (for example, if this was set up on a cloud provider instance), you can backup the entire easyrsa directory to secure storage. You can always restore it to a trusted instance again during the times when you want to sign or revoke certificates. Remember to backup the directory after every update.

How to Generate a Server Certificate for MongoDB¶

This page enumerates the steps we use to generate a server certificate for a MongoDB instance. A server certificate is also referred to as a “member certificate” in the MongoDB documentation. We use Easy-RSA.

Step 1: Install & Configure Easy–RSA¶

First create a directory for the server certificate (member cert) and cd into it:

mkdir member-cert

cd member-cert

Then install and configure Easy-RSA in that directory.

Step 2: Create the Server Private Key and CSR¶

You can create the server private key and certificate signing request (CSR) by going into the directory member-cert/easy-rsa-3.0.1/easyrsa3 and using something like:

Note

Please make sure you are fullfilling the requirements for MongoDB server/member certificates.

./easyrsa init-pki

./easyrsa --req-cn=mdb-instance-0 --subject-alt-name=DNS:localhost,DNS:mdb-instance-0 gen-req mdb-instance-0 nopass

You should replace the Common Name (mdb-instance-0 above) with the correct name for your MongoDB instance in the network, e.g. mdb-instance-5 or mdb-instance-12. (This name is decided by the organization managing the network.)

You will be prompted to enter the Distinguished Name (DN) information for this certificate. For each field, you can accept the default value [in brackets] by pressing Enter.

Warning

Don’t accept the default value of OU (IT). Instead, enter the value MongoDB-Instance.

Aside: You need to provide the DNS:localhost SAN during certificate generation for using the localhost exception in the MongoDB instance. All certificates can have this attribute without compromising security as the localhost exception works only the first time.

Step 3: Get the Server Certificate Signed¶

The CSR file created in the last step should be located in pki/reqs/mdb-instance-0.req (where the integer 0 may be different for you). You need to send it to the organization managing the BigchainDB network so that they can use their CA to sign the request. (The managing organization should already have a self-signed CA.)

If you are the admin of the managing organization’s self-signed CA, then you can import the CSR and use Easy-RSA to sign it. Go to your bdb-node-ca/easy-rsa-3.0.1/easyrsa3/ directory and do something like:

./easyrsa import-req /path/to/mdb-instance-0.req mdb-instance-0

./easyrsa --subject-alt-name=DNS:localhost,DNS:mdb-instance-0 sign-req server mdb-instance-0

Once you have signed it, you can send the signed certificate and the CA certificate back to the requestor. The files are pki/issued/mdb-instance-0.crt and pki/ca.crt.

Step 4: Generate the Consolidated Server PEM File¶

MongoDB requires a single, consolidated file containing both the public and private keys.

cat /path/to/mdb-instance-0.crt /path/to/mdb-instance-0.key > mdb-instance-0.pem

How to Generate a Client Certificate for MongoDB¶

This page enumerates the steps we use to generate a client certificate to be used by clients who want to connect to a TLS-secured MongoDB database. We use Easy-RSA.

Step 1: Install and Configure Easy-RSA¶

First create a directory for the client certificate and cd into it:

mkdir client-cert

cd client-cert

Then install and configure Easy-RSA in that directory.

Step 2: Create the Client Private Key and CSR¶

You can create the client private key and certificate signing request (CSR) by going into the directory client-cert/easy-rsa-3.0.1/easyrsa3 and using:

./easyrsa init-pki

./easyrsa gen-req bdb-instance-0 nopass

You should change the Common Name (e.g. bdb-instance-0) to a value that reflects what the client certificate is being used for, e.g. mdb-mon-instance-3 or mdb-bak-instance-4. (The final integer is specific to your BigchainDB node in the BigchainDB network.)

You will be prompted to enter the Distinguished Name (DN) information for this certificate. For each field, you can accept the default value [in brackets] by pressing Enter.

Warning

Don’t accept the default value of OU (IT). Instead, enter the value BigchainDB-Instance, MongoDB-Mon-Instance or MongoDB-Backup-Instance as appropriate.

Aside: The nopass option means “do not encrypt the private key (default is encrypted)”. You can get help with the easyrsa command (and its subcommands) by using the subcommand ./easyrsa help.

Note

For more information about requirements for MongoDB client certificates, please consult the official MongoDB documentation.

Step 3: Get the Client Certificate Signed¶

The CSR file created in the previous step should be located in pki/reqs/bdb-instance-0.req (or whatever Common Name you used in the gen-req command above). You need to send it to the organization managing the BigchainDB network so that they can use their CA to sign the request. (The managing organization should already have a self-signed CA.)

If you are the admin of the managing organization’s self-signed CA, then you can import the CSR and use Easy-RSA to sign it. Go to your bdb-node-ca/easy-rsa-3.0.1/easyrsa3/ directory and do something like:

./easyrsa import-req /path/to/bdb-instance-0.req bdb-instance-0

./easyrsa sign-req client bdb-instance-0

Once you have signed it, you can send the signed certificate and the CA certificate back to the requestor. The files are pki/issued/bdb-instance-0.crt and pki/ca.crt.

Step 4: Generate the Consolidated Client PEM File¶

Note

This step can be skipped for BigchainDB client certificate as BigchainDB uses the PyMongo driver, which accepts separate certificate and key files.

MongoDB, MongoDB Backup Agent and MongoDB Monitoring Agent require a single, consolidated file containing both the public and private keys.

cat /path/to/bdb-instance-0.crt /path/to/bdb-instance-0.key > bdb-instance-0.pem

 OR

cat /path/to/mdb-mon-instance-0.crt /path/to/mdb-mon-instance-0.key > mdb-mon-instance-0.pem

 OR

cat /path/to/mdb-bak-instance-0.crt /path/to/mdb-bak-instance-0.key > mdb-bak-instance-0.pem

How to Revoke an SSL/TLS Certificate¶

This page enumerates the steps we take to revoke a self-signed SSL/TLS certificate in a BigchainDB network. It can only be done by someone with access to the self-signed CA associated with the network’s managing organization.

Step 1: Revoke a Certificate¶

Since we used Easy-RSA version 3 to set up the CA, we use it to revoke certificates too.

Go to the following directory (associated with the self-signed CA): .../bdb-node-ca/easy-rsa-3.0.1/easyrsa3. You need to be aware of the file name used to import the certificate using the ./easyrsa import-req before. Run the following command to revoke a certificate:

./easyrsa revoke <filename>

This will update the CA database with the revocation details. The next step is to use the updated database to issue an up-to-date certificate revocation list (CRL).

Step 2: Generate a New CRL¶

Generate a new CRL for your infrastructure using:

./easyrsa gen-crl

The generated crl.pem file needs to be uploaded to your infrastructure to prevent the revoked certificate from being used again.

In particlar, the generated crl.pem file should be sent to all BigchainDB node operators in your BigchainDB network, so that they can update it in their MongoDB instance and their BigchainDB Server instance.

Template: Deploy a Kubernetes Cluster on Azure¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

A BigchainDB node can be run inside a Kubernetes cluster. This page describes one way to deploy a Kubernetes cluster on Azure.

Step 1: Get a Pay-As-You-Go Azure Subscription¶

Microsoft Azure has a Free Trial subscription (at the time of writing), but it’s too limited to run an advanced BigchainDB node. Sign up for a Pay-As-You-Go Azure subscription via the Azure website.

You may find that you have to sign up for a Free Trial subscription first. That’s okay: you can have many subscriptions.

Step 2: Create an SSH Key Pair¶

You’ll want an SSH key pair so you’ll be able to SSH to the virtual machines that you’ll deploy in the next step. (If you already have an SSH key pair, you could reuse it, but it’s probably a good idea to make a new SSH key pair for your Kubernetes VMs and nothing else.)

See the page about how to generate a key pair for SSH.

Step 3: Deploy an Azure Container Service (ACS)¶

It’s possible to deploy an Azure Container Service (ACS) from the Azure Portal (i.e. online in your web browser) but it’s actually easier to do it using the Azure Command-Line Interface (CLI).

Microsoft has instructions to install the Azure CLI 2.0 on most common operating systems. Do that.

If you already have the Azure CLI installed, you may want to update it.

Warning

az component update isn’t supported if you installed the CLI using some of Microsoft’s provided installation instructions. See the Microsoft docs for update instructions.

Next, login to your account using:

$ az login

It will tell you to open a web page and to copy a code to that page.

If the login is a success, you will see some information about all your subscriptions, including the one that is currently enabled ("state": "Enabled"). If the wrong one is enabled, you can switch to the right one using:

$ az account set --subscription <subscription name or ID>

Next, you will have to pick the Azure data center location where you’d like to deploy your cluster. You can get a list of all available locations using:

$ az account list-locations

Next, create an Azure “resource group” to contain all the resources (virtual machines, subnets, etc.) associated with your soon-to-be-deployed cluster. You can name it whatever you like but avoid fancy characters because they may confuse some software.

$ az group create --name <resource group name> --location <location name>

Example location names are koreacentral and westeurope.

Finally, you can deploy an ACS using something like:

$ az acs create --name <a made-up cluster name> \
--resource-group <name of resource group created earlier> \
--master-count 3 \
--agent-count 3 \
--admin-username ubuntu \
--agent-vm-size Standard_L4s \
--dns-prefix <make up a name> \
--ssh-key-value ~/.ssh/<name>.pub \
--orchestrator-type kubernetes \
--debug --output json

Note

The Azure documentation has a list of all az acs create options. You might prefer a smaller agent VM size, for example. You can also get a list of the options using:

$ az acs create --help

It takes a few minutes for all the resources to deploy. You can watch the progress in the Azure Portal: go to Resource groups (with the blue cube icon) and click on the one you created to see all the resources in it.

Trouble with the Service Principal? Then Read This!¶

If the az acs create command fails with an error message including the text, “The Service Principal in ServicePrincipalProfile could not be validated”, then we found you can prevent that by creating a Service Principal ahead of time and telling az acs create to use that one. (It’s supposed to create one, but sometimes that fails, I guess.)

Create a new resource group, even if you created one before. They’re free anyway:

$ az login
$ az group create --name <new resource group name> \
                  --location <Azure region like westeurope>

Note the id in the output. It looks like "/subscriptions/369284be-0104-421a-8488-1aeac0caecbb/resourceGroups/examplerg". It can be copied into the next command. Create a Service Principal using:

$ az ad sp create-for-rbac --role="Contributor" \
--scopes=<id value copied from above, including the double quotes on the ends>

Note the appId and password. Put those in a new az acs create command like above, with two new options added:

$ az acs create ... \
--service-principal <appId> \
--client-secret <password>

Optional: SSH to Your New Kubernetes Cluster Nodes¶

You can SSH to one of the just-deployed Kubernetes “master” nodes (virtual machines) using:

$ ssh -i ~/.ssh/<name> ubuntu@<master-ip-address-or-fqdn>

where you can get the IP address or FQDN of a master node from the Azure Portal. For example:

$ ssh -i ~/.ssh/mykey123 ubuntu@mydnsprefix.westeurope.cloudapp.azure.com

Note

All the master nodes are accessible behind the same public IP address and FQDN. You connect to one of the masters randomly based on the load balancing policy.

The “agent” nodes shouldn’t get public IP addresses or externally accessible FQDNs, so you can’t SSH to them directly, but you can first SSH to the master and then SSH to an agent from there using their hostname. To do that, you could copy your SSH key pair to the master (a bad idea), or use SSH agent forwarding (better). To do the latter, do the following on the machine you used to SSH to the master:

$ echo -e "Host <FQDN of the cluster from Azure Portal>\n  ForwardAgent yes" >> ~/.ssh/config

To verify that SSH agent forwarding works properly, SSH to the one of the master nodes and do:

$ echo "$SSH_AUTH_SOCK"

If you get an empty response, then SSH agent forwarding hasn’t been set up correctly. If you get a non-empty response, then SSH agent forwarding should work fine and you can SSH to one of the agent nodes (from a master) using:

$ ssh ubuntu@k8s-agent-4AC80E97-0

where k8s-agent-4AC80E97-0 is the name of a Kubernetes agent node in your Kubernetes cluster. You will have to replace it by the name of an agent node in your cluster.

Optional: Delete the Kubernetes Cluster¶

$ az acs delete \
--name <ACS cluster name> \
--resource-group <name of resource group containing the cluster>

Optional: Delete the Resource Group¶

CAUTION: You might end up deleting resources other than the ACS cluster.

$ az group delete \
--name <name of resource group containing the cluster>

Next, you can run a BigchainDB node/cluster(BFT) on your new Kubernetes cluster.

Kubernetes Template: Deploy a Single BigchainDB Node¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page describes how to deploy a BigchainDB node using Kubernetes. It assumes you already have a running Kubernetes cluster.

Below, we refer to many files by their directory and filename, such as configuration/config-map.yaml. Those files are files in the bigchaindb/bigchaindb repository on GitHub in the k8s/ directory. Make sure you’re getting those files from the appropriate Git branch on GitHub, i.e. the branch for the version of BigchainDB that your BigchainDB cluster is using.

Step 1: Install and Configure kubectl¶

kubectl is the Kubernetes CLI. If you don’t already have it installed, then see the Kubernetes docs to install it.

The default location of the kubectl configuration file is ~/.kube/config. If you don’t have that file, then you need to get it.

Azure. If you deployed your Kubernetes cluster on Azure using the Azure CLI 2.0 (as per our template), then you can get the ~/.kube/config file using:

$ az acs kubernetes get-credentials \
--resource-group <name of resource group containing the cluster> \
--name <ACS cluster name>

If it asks for a password (to unlock the SSH key) and you enter the correct password, but you get an error message, then try adding --ssh-key-file ~/.ssh/<name> to the above command (i.e. the path to the private key).

Note

About kubectl contexts. You might manage several Kubernetes clusters. To make it easy to switch from one to another, kubectl has a notion of “contexts,” e.g. the context for cluster 1 or the context for cluster 2. To find out the current context, do:

$ kubectl config view

and then look for the current-context in the output. The output also lists all clusters, contexts and users. (You might have only one of each.) You can switch to a different context using:

$ kubectl config use-context <new-context-name>

You can also switch to a different context for just one command by inserting --context <context-name> into any kubectl command. For example:

$ kubectl get pods

will get a list of the pods in the Kubernetes cluster associated with the context named k8s-bdb-test-cluster-0.

Step 2: Connect to Your Kubernetes Cluster’s Web UI (Optional)¶

You can connect to your cluster’s Kubernetes Dashboard (also called the Web UI) using:

$ kubectl proxy -p 8001

or

$ az acs kubernetes browse -g [Resource Group] -n [Container service instance name] --ssh-key-file /path/to/privateKey

or, if you prefer to be explicit about the context (explained above):

$ kubectl proxy -p 8001

The output should be something like Starting to serve on 127.0.0.1:8001. That means you can visit the dashboard in your web browser at http://127.0.0.1:8001/ui.

Note

Known Issue: If you are having accessing the UI i.e. accessing http://127.0.0.1:8001/ui in your browser returns a blank page and is redirected to http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy , you can access the UI by adding a / at the end of the redirected URL i.e. http://127.0.0.1:8001/api/v1/namespaces/kube-system/services/kubernetes-dashboard/proxy/

Step 3: Configure Your BigchainDB Node¶

See the page titled How to Configure a BigchainDB Node.

Step 4: Start the NGINX Service¶

This will will give us a public IP for the cluster.

Once you complete this step, you might need to wait up to 10 mins for the public IP to be assigned.

You have the option to use vanilla NGINX without HTTPS support or an NGINX with HTTPS support.
Start the Kubernetes Service:
$ kubectl apply -f nginx-https/nginx-https-svc.yaml

OR

$ kubectl apply -f nginx-http/nginx-http-svc.yaml

Step 5: Assign DNS Name to the NGINX Public IP¶

This step is required only if you are planning to set up multiple BigchainDB nodes or are using HTTPS certificates tied to a domain.
The following command can help you find out if the NGINX service started above has been assigned a public IP or external IP address:
$ kubectl get svc -w
Once a public IP is assigned, you can map it to a DNS name. We usually assign bdb-test-node-0, bdb-test-node-1 and so on in our documentation. Let’s assume that we assign the unique name of bdb-test-node-0 here.

Set up DNS mapping in Azure. Select the current Azure resource group and look for the Public IP resource. You should see at least 2 entries there - one for the Kubernetes master and the other for the NGINX instance. You may have to Refresh the Azure web page listing the resources in a resource group for the latest changes to be reflected. Select the Public IP resource that is attached to your service (it should have the Azure DNS prefix name along with a long random string, without the master-ip string), select Configuration, add the DNS assigned above (for example, bdb-test-node-0), click Save, and wait for the changes to be applied.

To verify the DNS setting is operational, you can run nslookup <DNS name added in Azure configuration> from your local Linux shell.

This will ensure that when you scale to different geographical zones, other Tendermint nodes in the network can reach this instance.

Step 6: Start the MongoDB Kubernetes Service¶

Start the Kubernetes Service:
$ kubectl apply -f mongodb/mongo-svc.yaml

Step 7: Start the BigchainDB Kubernetes Service¶

Start the Kubernetes Service:
$ kubectl apply -f bigchaindb/bigchaindb-svc.yaml

Step 8(Optional): Start the OpenResty Kubernetes Service¶

Start the Kubernetes Service:
$ kubectl apply -f nginx-openresty/nginx-openresty-svc.yaml

Step 9: Start the NGINX Kubernetes Deployment¶

NGINX is used as a proxy to the BigchainDB, Tendermint and MongoDB instances in the node. It proxies HTTP/HTTPS requests on the node-frontend-port to the corresponding OpenResty(if 3scale enabled) or BigchainDB backend, TCP connections on mongodb-frontend-port, tm-p2p-port and tm-pub-key-access to MongoDB and Tendermint respectively.
This configuration is located in the file nginx-https/nginx-https-dep.yaml or nginx-http/nginx-http-dep.yaml.
Start the Kubernetes Deployment:
$ kubectl apply -f nginx-https/nginx-https-dep.yaml

OR

$ kubectl apaply -f nginx-http/nginx-http-dep.yaml

Step 10: Create Kubernetes Storage Classes for MongoDB¶

MongoDB needs somewhere to store its data persistently, outside the container where MongoDB is running. Our MongoDB Docker container (based on the official MongoDB Docker container) exports two volume mounts with correct permissions from inside the container:

The directory where the MongoDB instance stores its data: /data/db. There’s more explanation in the MongoDB docs about storage.dbpath.
The directory where the MongoDB instance stores the metadata for a sharded cluster: /data/configdb/. There’s more explanation in the MongoDB docs about sharding.configDB.

Explaining how Kubernetes handles persistent volumes, and the associated terminology, is beyond the scope of this documentation; see the Kubernetes docs about persistent volumes.

The first thing to do is create the Kubernetes storage classes.

Set up Storage Classes in Azure. First, you need an Azure storage account. If you deployed your Kubernetes cluster on Azure using the Azure CLI 2.0 (as per our template), then the az acs create command already created a storage account in the same location and resource group as your Kubernetes cluster. Both should have the same “storage account SKU”: Standard_LRS. Standard storage is lower-cost and lower-performance. It uses hard disk drives (HDD). LRS means locally-redundant storage: three replicas in the same data center. Premium storage is higher-cost and higher-performance. It uses solid state drives (SSD).

We recommend using Premium storage with our Kubernetes deployment template. Create a storage account for Premium storage and associate it with your Azure resource group. For future reference, the command to create a storage account is az storage account create.

Note

Please refer to Azure documentation for the list of VMs that are supported by Premium Storage.

The Kubernetes template for configuration of the MongoDB Storage Class is located in the file mongodb/mongo-sc.yaml.

You may have to update the parameters.location field in the file to specify the location you are using in Azure.

If you want to use a custom storage account with the Storage Class, you can also update parameters.storageAccount and provide the Azure storage account name.

Create the required storage classes using:

$ kubectl apply -f mongodb/mongo-sc.yaml

You can check if it worked using kubectl get storageclasses.

Step 11: Create Kubernetes Persistent Volume Claims for MongoDB¶

Next, you will create two PersistentVolumeClaim objects mongo-db-claim and mongo-configdb-claim.

This configuration is located in the file mongodb/mongo-pvc.yaml.

Note how there’s no explicit mention of Azure, AWS or whatever. ReadWriteOnce (RWO) means the volume can be mounted as read-write by a single Kubernetes node. (ReadWriteOnce is the only access mode supported by AzureDisk.) storage: 20Gi means the volume has a size of 20 gibibytes.

You may want to update the spec.resources.requests.storage field in both the files to specify a different disk size.

Create the required Persistent Volume Claims using:

$ kubectl apply -f mongodb/mongo-pvc.yaml

You can check its status using: kubectl get pvc -w

Initially, the status of persistent volume claims might be “Pending” but it should become “Bound” fairly quickly.

Note

The default Reclaim Policy for dynamically created persistent volumes is Delete which means the PV and its associated Azure storage resource will be automatically deleted on deletion of PVC or PV. In order to prevent this from happening do the following steps to change default reclaim policy of dyanmically created PVs from Delete to Retain

Run the following command to list existing PVs

$ kubectl get pv

Run the following command to update a PV’s reclaim policy to <Retain>

$ kubectl patch pv <pv-name> -p '{"spec":{"persistentVolumeReclaimPolicy":"Retain"}}'

For notes on recreating a private volume form a released Azure disk resource consult the page about cluster troubleshooting.

Step 12: Start a Kubernetes StatefulSet for MongoDB¶

Create the MongoDB StatefulSet using:
$ kubectl apply -f mongodb/mongo-ss.yaml
It might take up to 10 minutes for the disks, specified in the Persistent Volume Claims above, to be created and attached to the pod. The UI might show that the pod has errored with the message “timeout expired waiting for volumes to attach/mount”. Use the CLI below to check the status of the pod in this case, instead of the UI. This happens due to a bug in Azure ACS.
$ kubectl get pods -w

Step 13: Configure Users and Access Control for MongoDB¶

In this step, you will create a user on MongoDB with authorization to create more users and assign roles to it. We will also create MongoDB client users for BigchainDB and MongoDB Monitoring agent(Optional).
$ kubectl apply -f mongodb/configure_mdb.sh

Step 14: Create Kubernetes Storage Classes for BigchainDB¶

BigchainDB needs somewhere to store Tendermint data persistently, Tendermint uses LevelDB as the persistent storage layer.

The Kubernetes template for configuration of Storage Class is located in the file bigchaindb/bigchaindb-sc.yaml.

Details about how to create a Azure Storage account and how Kubernetes Storage Class works are already covered in this document: Step 10: Create Kubernetes Storage Classes for MongoDB.

Create the required storage classes using:

$ kubectl apply -f bigchaindb/bigchaindb-sc.yaml

You can check if it worked using kubectl get storageclasses.

Step 15: Create Kubernetes Persistent Volume Claims for BigchainDB¶

Next, you will create two PersistentVolumeClaim objects tendermint-db-claim and tendermint-config-db-claim.

This configuration is located in the file bigchaindb/bigchaindb-pvc.yaml.

Details about Kubernetes Persistent Volumes, Persistent Volume Claims and how they work with Azure are already covered in this document: Step 11: Create Kubernetes Persistent Volume Claims for MongoDB.

Create the required Persistent Volume Claims using:

$ kubectl apply -f bigchaindb/bigchaindb-pvc.yaml

You can check its status using:

kubectl get pvc -w

Step 16: Start a Kubernetes StatefulSet for BigchainDB¶

This configuration is located in the file bigchaindb/bigchaindb-ss.yaml.

Set the spec.serviceName to the value set in bdb-instance-name in the ConfigMap. For example, if the value set in the bdb-instance-name is bdb-instance-0, set the field to tm-instance-0.

Set metadata.name, spec.template.metadata.name and spec.template.metadata.labels.app to the value set in bdb-instance-name in the ConfigMap, followed by -ss. For example, if the value set in the bdb-instance-name is bdb-instance-0, set the fields to the value bdb-insance-0-ss.

As we gain more experience running Tendermint in testing and production, we will tweak the resources.limits.cpu and resources.limits.memory.
Create the BigchainDB StatefulSet using:
$ kubectl apply -f bigchaindb/bigchaindb-ss.yaml
$ kubectl get pods -w

Step 17(Optional): Start a Kubernetes Deployment for MongoDB Monitoring Agent¶

This configuration is located in the file mongodb-monitoring-agent/mongo-mon-dep.yaml.

Set metadata.name, spec.template.metadata.name and spec.template.metadata.labels.app to the value set in mdb-mon-instance-name in the ConfigMap, followed by -dep. For example, if the value set in the mdb-mon-instance-name is mdb-mon-instance-0, set the fields to the value mdb-mon-instance-0-dep.

The configuration uses the following values set in the Secret:

mdb-mon-certs

ca-auth

cloud-manager-credentials
Start the Kubernetes Deployment using:
$ kubectl apply -f mongodb-monitoring-agent/mongo-mon-dep.yaml

Step 18(Optional): Start a Kubernetes Deployment for OpenResty¶

This configuration is located in the file nginx-openresty/nginx-openresty-dep.yaml.

Set metadata.name and spec.template.metadata.labels.app to the value set in openresty-instance-name in the ConfigMap, followed by -dep. For example, if the value set in the openresty-instance-name is openresty-instance-0, set the fields to the value openresty-instance-0-dep.

Set the port to be exposed from the pod in the spec.containers[0].ports section. We currently expose the port at which OpenResty is listening for requests, openresty-backend-port in the above ConfigMap.

The configuration uses the following values set in the Secret:

threescale-credentials

The configuration uses the following values set in the ConfigMap:

node-dns-server-ip

openresty-backend-port

ngx-bdb-instance-name

bigchaindb-api-port
Create the OpenResty Deployment using:
$ kubectl apply -f nginx-openresty/nginx-openresty-dep.yaml
You can check its status using the command kubectl get deployments -w

Step 19(Optional): Configure the MongoDB Cloud Manager¶

Refer to the documentation for details on how to configure the MongoDB Cloud Manager to enable monitoring and backup.

Step 20(Optional): Only for multi site deployments(Geographically dispersed)¶

We need to make sure that clusters are able to talk to each other i.e. specifically the communication between the Tendermint peers. Set up networking between the clusters using Kubernetes Services.

Assuming we have a BigchainDB instance bdb-instance-1 residing in Azure data center location westeurope and we want to connect to bdb-instance-2, bdb-instance-3, and bdb-instance-4 located in Azure data centers eastus, centralus and westus, respectively. Unless you already have explicitly set up networking for bdb-instance-1 to communicate with bdb-instance-2/3/4 and vice versa, we will have to add a Kubernetes Service in each cluster to accomplish this goal in order to set up a Tendermint P2P network. It is similar to ensuring that there is a CNAME record in the DNS infrastructure to resolve bdb-instance-X to the host where it is actually available. We can do this in Kubernetes using a Kubernetes Service of type ExternalName.

This configuration is located in the file bigchaindb/bigchaindb-ext-conn-svc.yaml.
Set the name of the metadata.name to the host name of the BigchainDB instance you are trying to connect to. For instance if you are configuring this service on cluster with bdb-instance-1 then the metadata.name will be bdb-instance-2 and vice versa.
Set spec.ports.port[0] to the tm-p2p-port from the ConfigMap for the other cluster.
Set spec.ports.port[1] to the tm-rpc-port from the ConfigMap for the other cluster.
Set spec.externalName to the FQDN mapped to NGINX Public IP of the cluster you are trying to connect to. For more information about the FQDN please refer to: Assign DNS name to NGINX Public IP.

Note

This operation needs to be replicated n-1 times per node for a n node cluster, with the respective FQDNs we need to communicate with.

If you are not the system administrator of the cluster, you have to get in touch with the system administrator/s of the other n-1 clusters and share with them your instance name (tendermint-instance-name in the ConfigMap) and the FQDN of the NGINX instance acting as Gateway(set in: Assign DNS name to NGINX Public IP).

Step 21: Verify the BigchainDB Node Setup¶

Step 21.1: Testing Internally¶

To test the setup of your BigchainDB node, you could use a Docker container that provides utilities like nslookup, curl and dig. For example, you could use a container based on our bigchaindb/toolbox image. (The corresponding Dockerfile is in the bigchaindb/bigchaindb repository on GitHub.) You can use it as below to get started immediately:

$ kubectl   \
   run -it toolbox \
   --image bigchaindb/toolbox \
   --image-pull-policy=Always \
   --restart=Never --rm

It will drop you to the shell prompt.

To test the MongoDB instance:

$ nslookup mdb-instance-0

$ dig +noall +answer _mdb-port._tcp.mdb-instance-0.default.svc.cluster.local SRV

$ curl -X GET http://mdb-instance-0:27017

The nslookup command should output the configured IP address of the service (in the cluster). The dig command should return the configured port numbers. The curl command tests the availability of the service.

To test the BigchainDB instance:

$ nslookup bdb-instance-0

$ dig +noall +answer _bdb-api-port._tcp.bdb-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _bdb-ws-port._tcp.bdb-instance-0.default.svc.cluster.local SRV

$ curl -X GET http://bdb-instance-0:9984

$ curl -X GET http://bdb-instance-0:9986/pub_key.json

$ curl -X GET http://bdb-instance-0:26657/abci_info

$ wsc -er ws://bdb-instance-0:9985/api/v1/streams/valid_transactions

To test the OpenResty instance:

$ nslookup openresty-instance-0

$ dig +noall +answer _openresty-svc-port._tcp.openresty-instance-0.default.svc.cluster.local SRV

To verify if OpenResty instance forwards the requests properly, send a POST transaction to OpenResty at post 80 and check the response from the backend BigchainDB instance.

To test the vanilla NGINX instance:

$ nslookup ngx-http-instance-0

$ dig +noall +answer _public-node-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _public-health-check-port._tcp.ngx-http-instance-0.default.svc.cluster.local SRV

$ wsc -er ws://ngx-http-instance-0/api/v1/streams/valid_transactions

$ curl -X GET http://ngx-http-instance-0:27017

The above curl command should result in the response It looks like you are trying to access MongoDB over HTTP on the native driver port.

To test the NGINX instance with HTTPS and 3scale integration:

$ nslookup ngx-instance-0

$ dig +noall +answer _public-secure-node-port._tcp.ngx-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _public-mdb-port._tcp.ngx-instance-0.default.svc.cluster.local SRV

$ dig +noall +answer _public-insecure-node-port._tcp.ngx-instance-0.default.svc.cluster.local SRV

$ wsc -er wss://<node-fqdn>/api/v1/streams/valid_transactions

$ curl -X GET http://<node-fqdn>:27017

The above curl command should result in the response It looks like you are trying to access MongoDB over HTTP on the native driver port.

Step 21.2: Testing Externally¶

Check the MongoDB monitoring agent on the MongoDB Cloud Manager portal to verify they are working fine.

If you are using the NGINX with HTTP support, accessing the URL http://<DNS/IP of your exposed BigchainDB service endpoint>:node-frontend-port on your browser should result in a JSON response that shows the BigchainDB server version, among other things. If you are using the NGINX with HTTPS support, use https instead of http above.

Use the Python Driver to send some transactions to the BigchainDB node and verify that your node or cluster works as expected.

Next, you can set up log analytics and monitoring, by following our templates:

Log Analytics on Azure.

How to Configure a BigchainDB Node¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page outlines the steps to set a bunch of configuration settings in your BigchainDB node. They are pushed to the Kubernetes cluster in two files, named config-map.yaml (a set of ConfigMaps) and secret.yaml (a set of Secrets). They are stored in the Kubernetes cluster’s key-value store (etcd).

Make sure you did the first four operations listed in the section titled Things Each Node Operator Must Do.

Edit vars¶

This file is located at: k8s/scripts/vars and edit the configuration parameters. That file already contains many comments to help you understand each data value, but we make some additional remarks on some of the values below.

vars.NODE_FQDN¶

FQDN for your BigchainDB node. This is the domain name used to query and access your BigchainDB node. More information can be found in our Kubernetes template overview guide.

vars.SECRET_TOKEN¶

This parameter is specific to your BigchainDB node and is used for authentication and authorization of requests to your BigchainDB node. More information can be found in our Kubernetes template overview guide.

vars.HTTPS_CERT_KEY_FILE_NAME¶

Absolute path of the HTTPS certificate chain of your domain. More information can be found in our Kubernetes template overview guide.

vars.HTTPS_CERT_CHAIN_FILE_NAME¶

Absolute path of the HTTPS certificate key of your domain. More information can be found in our Kubernetes template overview guide.

vars.MDB_ADMIN_USER and vars.MDB_ADMIN_PASSWORD¶

MongoDB admin user credentials, username and password. This user is created on the admin database with the authorization to create other users.

vars.BDB_PERSISTENT_PEERS, BDB_VALIDATORS, BDB_VALIDATORS_POWERS, BDB_GENESIS_TIME and BDB_CHAIN_ID¶

These parameters are shared across the BigchainDB network. More information about the generation of these parameters can be found at Generate All Shared BigchainDB Setup Parameters.

vars.NODE_DNS_SERVER¶

IP of Kubernetes service(kube-dns), can be retrieved using using CLI(kubectl) or k8s dashboard. This parameter is used by the Nginx gateway instance to resolve the hostnames of all the services running in the Kubernetes cluster.

# retrieval via commandline.
$ kubectl get services --namespace=kube-system -l k8s-app=kube-dns

Generate configuration¶

After populating the k8s/scripts/vars file, we need to generate all the configuration required for the BigchainDB node, for that purpose we need to execute k8s/scripts/generate_configs.sh script.

$ bash generate_configs.sh

Note

During execution the script will prompt the user for some inputs.

After successful execution, this routine will generate config-map.yaml and secret.yaml under k8s/scripts.

Deploy Your config-map.yaml and secret.yaml¶

You can deploy your edited config-map.yaml and secret.yaml files to your Kubernetes cluster using the commands:

$ kubectl apply -f config-map.yaml

$ kubectl apply -f secret.yaml

Log Analytics on Azure¶

This page describes how we use Microsoft Operations Management Suite (OMS) to collect all logs from a Kubernetes cluster, to search those logs, and to set up email alerts based on log messages. The References section (below) contains links to more detailed documentation.

There are two steps:

Setup: Create a log analytics OMS workspace and a Containers solution under that workspace.
Deploy OMS agents to your Kubernetes cluster.

Step 1: Setup¶

Step 1 can be done the web browser way or the command-line way.

The Web Browser Way¶

To create a new log analytics OMS workspace:

Go to the Azure Portal in your web browser.
Click on More services > in the lower left corner of the Azure Portal.
Type “log analytics” or similar.
Select Log Analytics from the list of options.
Click on + Add to add a new log analytics OMS workspace.
Give answers to the questions. You can call the OMS workspace anything, but use the same resource group and location as your Kubernetes cluster. The free option will suffice, but of course you can also use a paid one.

To add a “Containers solution” to that new workspace:

In Azure Portal, in the Log Analytics section, click the name of the new workspace
Click OMS Workspace.
Click OMS Portal. It should launch the OMS Portal in a new tab.
Click the Solutions Gallery tile.
Click the Containers tile.
Click Add.

The Command-Line Way¶

We’ll assume your Kubernetes cluster has a resource group named:

resource_group

and the workspace we’ll create will be named:

work_space

If you feel creative you may replace these names by more interesting ones.

$ az group deployment create --debug \
    --resource-group resource_group \
    --name "Microsoft.LogAnalyticsOMS" \
    --template-file log_analytics_oms.json \
    --parameters @log_analytics_oms.parameters.json

An example of a simple template file (--template-file):

{
  "$schema": "http://schema.management.azure.com/schemas/2014-04-01-preview/deploymentTemplate.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "sku": {
      "type": "String"
    },
    "workspaceName": {
      "type": "String"
    },
    "solutionType": {
      "type": "String"
    },
    "resources": [
      {
        "apiVersion": "2015-03-20",
        "type": "Microsoft.OperationalInsights/workspaces",
        "name": "[parameters('workspaceName')]",
        "location": "[resourceGroup().location]",
        "properties": {
          "sku": {
            "name": "[parameters('sku')]"
          }
        },
        "resources": [
          {
            "apiVersion": "2015-11-01-preview",
            "location": "[resourceGroup().location]",
            "name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
            "type": "Microsoft.OperationsManagement/solutions",
            "id": "[Concat(resourceGroup().id, '/providers/Microsoft.OperationsManagement/solutions/', parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
            "dependsOn": [
              "[concat('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
            ],
            "properties": {
              "workspaceResourceId": "[resourceId('Microsoft.OperationalInsights/workspaces/', parameters('workspaceName'))]"
            },
            "plan": {
              "publisher": "Microsoft",
              "product": "[Concat('OMSGallery/', parameters('solutionType'))]",
              "name": "[Concat(parameters('solutionType'), '(', parameters('workspaceName'), ')')]",
              "promotionCode": ""
            }
          }
        ]
      }
    ]
  }
}

An example of the associated parameter file (--parameters):

{
  "$schema": "https://schema.management.azure.com/schemas/2015-01-01/deploymentParameters.json#",
  "contentVersion": "1.0.0.0",
  "parameters": {
    "sku": {
      "value": "Free"
    },
    "workspaceName": {
      "value": "work_space"
    },
    "solutionType": {
      "value": "Containers"
    }
  }
}

Step 2: Deploy the OMS Agents¶

To deploy an OMS agent, two important pieces of information are needed:

workspace id
workspace key

You can obtain the workspace id using:

$ az resource show \
    --resource-group resource_group
    --resource-type Microsoft.OperationalInsights/workspaces
    --name work_space \
    | grep customerId
"customerId": "12345678-1234-1234-1234-123456789012",

Until we figure out a way to obtain the workspace key via the command line, you can get it via the OMS Portal. To get to the OMS Portal, go to the Azure Portal and click on:

Resource Groups > (Your Kubernetes cluster’s resource group) > Log analytics (OMS) > (Name of the only item listed) > OMS Workspace > OMS Portal

(Let us know if you find a faster way.) Then see Microsoft’s instructions to obtain your workspace ID and key (via the OMS Portal).

Once you have the workspace id and key, you can include them in the following YAML file (oms-daemonset.yaml):

# oms-daemonset.yaml
apiVersion: extensions/v1beta1
kind: DaemonSet
metadata:
  name: omsagent
spec:
  template:
    metadata:
      labels:
        app: omsagent
    spec:
      containers:
      - env:
        - name: WSID
          value: <workspace_id>
        - name: KEY
          value: <workspace_key>
        image: microsoft/oms
        name: omsagent
        ports:
        - containerPort: 25225
          protocol: TCP
        securityContext:
          privileged: true
        volumeMounts:
        - mountPath: /var/run/docker.sock
          name: docker-sock
      volumes:
      - name: docker-sock
        hostPath:
          path: /var/run/docker.sock

To deploy the OMS agents (one per Kubernetes node, i.e. one per computer), simply run the following command:

$ kubectl create -f oms-daemonset.yaml

Search the OMS Logs¶

OMS should now be getting, storing and indexing all the logs from all the containers in your Kubernetes cluster. You can search the OMS logs from the Azure Portal or the OMS Portal, but at the time of writing, there was more functionality in the OMS Portal (e.g. the ability to create an Alert based on a search).

There are instructions to get to the OMS Portal above. Once you’re in the OMS Portal, click on Log Search and enter a query. Here are some example queries:

All logging messages containing the strings “critical” or “error” (not case-sensitive):

Type=ContainerLog (critical OR error)

Note

You can filter the results even more by clicking on things in the left sidebar. For OMS Log Search syntax help, see the Log Analytics search reference.

All logging messages containing the string “error” but not “404”:

Type=ContainerLog error NOT(404)

All logging messages containing the string “critical” but not “CriticalAddonsOnly”:

Type=ContainerLog critical NOT(CriticalAddonsOnly)

All logging messages from containers running the Docker image bigchaindb/nginx_3scale:1.3, containing the string “GET” but not the strings “Go-http-client” or “runscope” (where those exclusions filter out tests by Kubernetes and Runscope):

Type=ContainerLog Image="bigchaindb/nginx_3scale:1.3" GET NOT("Go-http-client") NOT(runscope)

Note

We wrote a small Python 3 script to analyze the logs found by the above NGINX search. It’s in k8s/logging-and-monitoring/analyze.py. The docsting at the top of the script explains how to use it.

Create an Email Alert¶

Once you’re satisfied with an OMS Log Search query string, click the 🔔 Alert icon in the top menu, fill in the form, and click Save when you’re done.

Some Useful Management Tasks¶

List workspaces:

$ az resource list \
    --resource-group resource_group \
    --resource-type Microsoft.OperationalInsights/workspaces

List solutions:

$ az resource list \
    --resource-group resource_group \
    --resource-type Microsoft.OperationsManagement/solutions

Delete the containers solution:

$ az group deployment delete --debug \
    --resource-group resource_group \
    --name Microsoft.ContainersOMS

$ az resource delete \
    --resource-group resource_group \
    --resource-type Microsoft.OperationsManagement/solutions \
    --name "Containers(work_space)"

Delete the workspace:

$ az group deployment delete --debug \
    --resource-group resource_group \
    --name Microsoft.LogAnalyticsOMS

$ az resource delete \
    --resource-group resource_group \
    --resource-type Microsoft.OperationalInsights/workspaces \
    --name work_space

References¶

Configure MongoDB Cloud Manager for Monitoring¶

This document details the steps required to configure MongoDB Cloud Manager to enable monitoring of data in a MongoDB Replica Set.

Configure MongoDB Cloud Manager for Monitoring Step by Step¶

Once the Monitoring Agent is up and running, open MongoDB Cloud Manager.

Click Login under MongoDB Cloud Manager and log in to the Cloud Manager.

Select the group from the dropdown box on the page.

Go to Settings and add a Preferred Hostnames entry as a regexp based on the mdb-instance-name of the nodes in your cluster. It may take up to 5 mins till this setting takes effect. You may refresh the browser window and verify whether the changes have been saved or not.

For example, for the nodes in a cluster that are named mdb-instance-0, mdb-instance-1 and so on, a regex like ^mdb-instance-[0-9]{1,2}$ is recommended.

Next, click the Deployment tab, and then the Manage Existing button.

On the Import your deployment for monitoring page, enter the hostname to be the same as the one set for mdb-instance-name in the global ConfigMap for a node. For example, if the mdb-instance-name is set to mdb-instance-0, enter mdb-instance-0 as the value in this field.

Enter the port number as 27017, with no authentication.

If you have authentication enabled, select the option to enable authentication and specify the authentication mechanism as per your deployment. The default BigchainDB Kubernetes deployment template currently supports X.509 Client Certificate as the authentication mechanism.

If you have TLS enabled, select the option to enable TLS/SSL for MongoDB connections, and click Continue. This should already be selected for you in case you selected X.509 Client Certificate above.

Wait a minute or two for the deployment to be found and then click the Continue button again.

Verify that you see your process on the Cloud Manager UI. It should look something like this:

Click Continue.

Verify on the UI that data is being sent by the monitoring agent to the Cloud Manager. It may take upto 5 minutes for data to appear on the UI.

How to Install & Configure Easy-RSA¶

We use Easy-RSA version 3, a wrapper over complex openssl commands. Easy-RSA is available on GitHub and licensed under GPLv2.

Step 1: Install Easy-RSA Dependencies¶

The only dependency for Easy-RSA v3 is openssl, which is available from the openssl package on Ubuntu and other Debian-based operating systems, i.e. you can install it using:

sudo apt-get update

sudo apt-get install openssl

Step 2: Install Easy-RSA¶

Make sure you’re in the directory where you want Easy-RSA to live, then download it and extract it within that directory:

wget https://github.com/OpenVPN/easy-rsa/archive/3.0.1.tar.gz

tar xzvf 3.0.1.tar.gz

rm 3.0.1.tar.gz

There should now be a directory named easy-rsa-3.0.1 in your current directory.

Step 3: Customize the Easy-RSA Configuration¶

We now create a config file named vars by copying the existing vars.example file and then editing it. You should change the country, province, city, org and email to the correct values for your organisation. (Note: The country, province, city, org and email are part of the Distinguished Name (DN).) The comments in the file explain what each of the variables mean.

cd easy-rsa-3.0.1/easyrsa3

cp vars.example vars

echo 'set_var EASYRSA_DN "org"' >> vars
echo 'set_var EASYRSA_KEY_SIZE 4096' >> vars

echo 'set_var EASYRSA_REQ_COUNTRY "DE"' >> vars
echo 'set_var EASYRSA_REQ_PROVINCE "Berlin"' >> vars
echo 'set_var EASYRSA_REQ_CITY "Berlin"' >> vars
echo 'set_var EASYRSA_REQ_ORG "BigchainDB GmbH"' >> vars
echo 'set_var EASYRSA_REQ_OU "IT"' >> vars
echo 'set_var EASYRSA_REQ_EMAIL "contact@ipdb.global"' >> vars

Note: Later, when building a CA or generating a certificate signing request, you will be prompted to enter a value for the OU (or to accept the default). You should change the default OU from IT to one of the following, as appropriate: ROOT-CA, MongoDB-Instance, BigchainDB-Instance, MongoDB-Mon-Instance or MongoDB-Backup-Instance. To understand why, see the MongoDB Manual. There are reminders to do this in the relevant docs.

Step 4: Maybe Edit x509-types/server¶

Warning

Only do this step if you are setting up a self-signed CA.

Edit the file x509-types/server and change extendedKeyUsage = serverAuth to extendedKeyUsage = serverAuth,clientAuth. See the MongoDB documentation about x.509 authentication to understand why.

Kubernetes Template: Upgrade all Software in a BigchainDB Node¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page outlines how to upgrade all the software associated with a BigchainDB node running on Kubernetes, including host operating systems, Docker, Kubernetes, and BigchainDB-related software.

Upgrade Host OS, Docker and Kubernetes¶

Some Kubernetes installation & management systems can do full or partial upgrades of host OSes, Docker, or Kubernetes, e.g. Tectonic, Rancher, and Kubo. Consult the documentation for your system.

Azure Container Service (ACS). On Dec. 15, 2016, a Microsoft employee wrote: “In the coming months we [the Azure Kubernetes team] will be building managed updates in the ACS service.” At the time of writing, managed updates were not yet available, but you should check the latest ACS documentation to see what’s available now. Also at the time of writing, ACS only supported Ubuntu as the host (master and agent) operating system. You can upgrade Ubuntu and Docker on Azure by SSHing into each of the hosts, as documented on another page.

In general, you can SSH to each host in your Kubernetes Cluster to update the OS and Docker.

Note

Once you are in an SSH session with a host, the docker info command is a handy way to detemine the host OS (including version) and the Docker version.

When you want to upgrade the software on a Kubernetes node, you should “drain” the node first, i.e. tell Kubernetes to gracefully terminate all pods on the node and mark it as unscheduleable (so no new pods get put on the node during its downtime).

kubectl drain $NODENAME

There are more details in the Kubernetes docs, including instructions to make the node scheduleable again.

To manually upgrade the host OS, see the docs for that OS.

To manually upgrade Docker, see the Docker docs.

To manually upgrade all Kubernetes software in your Kubernetes cluster, see the Kubernetes docs.

Kubernetes Template: Deploying a BigchainDB network¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

This page describes how to deploy a static BigchainDB + Tendermint network.

If you want to deploy a stand-alone BigchainDB node in a BigchainDB cluster, or a stand-alone BigchainDB node, then see the page about that.

We can use this guide to deploy a BigchainDB network in the following scenarios:

Single Azure Kubernetes Site.
Multiple Azure Kubernetes Sites (Geographically dispersed).

Terminology Used¶

BigchainDB node is a set of Kubernetes components that join together to form a BigchainDB single node. Please refer to the architecture diagram for more details.

BigchainDB network will refer to a collection of nodes working together to form a network.

Below, we refer to multiple files by their directory and filename, such as bigchaindb/bigchaindb-ext-conn-svc.yaml. Those files are located in the bigchaindb/bigchaindb repository on GitHub in the k8s/ directory. Make sure you’re getting those files from the appropriate Git branch on GitHub, i.e. the branch for the version of BigchainDB that your BigchainDB cluster is using.

Note

This deployment strategy is currently used for testing purposes only, operated by a single stakeholder or tightly coupled stakeholders.

Note

Currently, we only support a static set of participants in the network. Once a BigchainDB network is started with a certain number of validators and a genesis file. Users cannot add new validator nodes dynamically. You can track the progress of this funtionality on our github repository.

Prerequisites¶

The deployment methodology is similar to one covered with Kubernetes Template: Deploy a Single BigchainDB Node, but we need to tweak some configurations depending on your choice of deployment.

The operator needs to follow some consistent naming convention for all the components covered here.

Lets assume we are deploying a 4 node cluster, your naming conventions could look like this:

{
  "MongoDB": [
    "mdb-instance-1",
    "mdb-instance-2",
    "mdb-instance-3",
    "mdb-instance-4"
  ],
  "BigchainDB": [
    "bdb-instance-1",
    "bdb-instance-2",
    "bdb-instance-3",
    "bdb-instance-4"
  ],
  "NGINX": [
    "ngx-instance-1",
    "ngx-instance-2",
    "ngx-instance-3",
    "ngx-instance-4"
  ],
  "OpenResty": [
    "openresty-instance-1",
    "openresty-instance-2",
    "openresty-instance-3",
    "openresty-instance-4"
  ],
  "MongoDB_Monitoring_Agent": [
    "mdb-mon-instance-1",
    "mdb-mon-instance-2",
    "mdb-mon-instance-3",
    "mdb-mon-instance-4"
  ]
}

Note

Blockchain Genesis ID and Time will be shared across all nodes.

Edit config.yaml and secret.yaml¶

Make N(number of nodes) copies of configuration/config-map.yaml and configuration/secret.yaml.

# For config-map.yaml
config-map-node-1.yaml
config-map-node-2.yaml
config-map-node-3.yaml
config-map-node-4.yaml

# For secret.yaml
secret-node-1.yaml
secret-node-2.yaml
secret-node-3.yaml
secret-node-4.yaml

Edit the data values as described in this document, based on the naming convention described above.

Only for single site deployments: Since all the configuration files use the same ConfigMap and Secret Keys i.e. metadata.name -> vars, bdb-config and tendermint-config and metadata.name -> cloud-manager-credentials, mdb-certs, mdb-mon-certs, bdb-certs, https-certs, three-scale-credentials, ca-auth respectively, each file will overwrite the configuration of the previously deployed one. We want each node to have its own unique configurations. One way to go about it is that, using the naming convention above we edit the ConfigMap and Secret keys.

# For config-map-node-1.yaml
metadata.name: vars -> vars-node-1
metadata.name: bdb-config -> bdb-config-node-1
metadata.name: tendermint-config -> tendermint-config-node-1

# For secret-node-1.yaml
metadata.name: cloud-manager-credentials -> cloud-manager-credentials-node-1
metadata.name: mdb-certs -> mdb-certs-node-1
metadata.name: mdb-mon-certs -> mdb-mon-certs-node-1
metadata.name: bdb-certs -> bdb-certs-node-1
metadata.name: https-certs -> https-certs-node-1
metadata.name: threescale-credentials -> threescale-credentials-node-1
metadata.name: ca-auth -> ca-auth-node-1

# Repeat for the remaining files.

Deploy all your configuration maps and secrets.

kubectl apply -f configuration/config-map-node-1.yaml
kubectl apply -f configuration/config-map-node-2.yaml
kubectl apply -f configuration/config-map-node-3.yaml
kubectl apply -f configuration/config-map-node-4.yaml
kubectl apply -f configuration/secret-node-1.yaml
kubectl apply -f configuration/secret-node-2.yaml
kubectl apply -f configuration/secret-node-3.yaml
kubectl apply -f configuration/secret-node-4.yaml

Note

Similar to what we did, with config-map.yaml and secret.yaml i.e. indexing them per node, we have to do the same for each Kubernetes component i.e. Services, StorageClasses, PersistentVolumeClaims, StatefulSets, Deployments etc.

# For Services
*-node-1-svc.yaml
*-node-2-svc.yaml
*-node-3-svc.yaml
*-node-4-svc.yaml

# For StorageClasses
*-node-1-sc.yaml
*-node-2-sc.yaml
*-node-3-sc.yaml
*-node-4-sc.yaml

# For PersistentVolumeClaims
*-node-1-pvc.yaml
*-node-2-pvc.yaml
*-node-3-pvc.yaml
*-node-4-pvc.yaml

# For StatefulSets
*-node-1-ss.yaml
*-node-2-ss.yaml
*-node-3-ss.yaml
*-node-4-ss.yaml

# For Deployments
*-node-1-dep.yaml
*-node-2-dep.yaml
*-node-3-dep.yaml
*-node-4-dep.yaml

Single Site: Single Azure Kubernetes Cluster¶

For the deployment of a BigchainDB network under a single cluster, we need to replicate the deployment steps for each node N number of times, N being the number of participants in the network.

In our Kubernetes deployment template for a single BigchainDB node, we covered the basic configurations settings here.

Since, we index the ConfigMap and Secret Keys for the single site deployment, we need to update all the Kubernetes components to reflect the corresponding changes i.e. For each Kubernetes Service, StatefulSet, PersistentVolumeClaim, Deployment, and StorageClass, we need to update the respective *.yaml file and update the ConfigMapKeyRef.name OR secret.secretName.

Example¶

Assuming we are deploying the MongoDB StatefulSet for Node 1. We need to update the mongo-node-1-ss.yaml and update the corresponding ConfigMapKeyRef.name or secret.secretNames.

########################################################################
# This YAML file desribes a StatefulSet with a service for running and #
# exposing a MongoDB instance.                                         #
# It depends on the configdb and db k8s pvc.                           #
########################################################################

apiVersion: apps/v1beta1
kind: StatefulSet
metadata:
  name: mdb-instance-0-ss
  namespace: default
spec:
  serviceName: mdb-instance-0
  replicas: 1
  template:
    metadata:
      name: mdb-instance-0-ss
      labels:
        app: mdb-instance-0-ss
    spec:
      terminationGracePeriodSeconds: 10
      containers:
      - name: mongodb
        image: bigchaindb/mongodb:3.2
        imagePullPolicy: IfNotPresent
        env:
        - name: MONGODB_FQDN
          valueFrom:
            configMapKeyRef:
             name: vars-1 # Changed from ``vars``
             key: mdb-instance-name
        - name: MONGODB_POD_IP
          valueFrom:
            fieldRef:
              fieldPath: status.podIP
        - name: MONGODB_PORT
          valueFrom:
            configMapKeyRef:
             name: vars-1 # Changed from ``vars``
             key: mongodb-backend-port
        - name: STORAGE_ENGINE_CACHE_SIZE
          valueFrom:
            configMapKeyRef:
             name: vars-1 # Changed from ``vars``
             key: storage-engine-cache-size
        args:
        - --mongodb-port
        - $(MONGODB_PORT)
        - --mongodb-key-file-path
        - /etc/mongod/ssl/mdb-instance.pem
        - --mongodb-ca-file-path
        - /etc/mongod/ca/ca.pem
        - --mongodb-crl-file-path
        - /etc/mongod/ca/crl.pem
        - --mongodb-fqdn
        - $(MONGODB_FQDN)
        - --mongodb-ip
        - $(MONGODB_POD_IP)
        - --storage-engine-cache-size
        - $(STORAGE_ENGINE_CACHE_SIZE)
        securityContext:
          capabilities:
            add:
            - FOWNER
        ports:
        - containerPort: "<mongodb-backend-port from ConfigMap>"
          protocol: TCP
          name: mdb-api-port
        volumeMounts:
        - name: mdb-db
          mountPath: /data/db
        - name: mdb-configdb
          mountPath: /data/configdb
        - name: mdb-certs
          mountPath: /etc/mongod/ssl/
          readOnly: true
        - name: ca-auth
          mountPath: /etc/mongod/ca/
          readOnly: true
        resources:
          limits:
            cpu: 200m
            memory: 5G
        livenessProbe:
          tcpSocket:
            port: mdb-api-port
          initialDelaySeconds: 15
          successThreshold: 1
          failureThreshold: 3
          periodSeconds: 15
          timeoutSeconds: 10
      restartPolicy: Always
      volumes:
      - name: mdb-db
        persistentVolumeClaim:
          claimName: mongo-db-claim-1 # Changed from ``mongo-db-claim``
      - name: mdb-configdb
        persistentVolumeClaim:
          claimName: mongo-configdb-claim-1 # Changed from ``mongo-configdb-claim``
      - name: mdb-certs
        secret:
          secretName: mdb-certs-1 # Changed from ``mdb-certs``
          defaultMode: 0400
      - name: ca-auth
        secret:
          secretName: ca-auth-1 # Changed from ``ca-auth``
          defaultMode: 0400

The above example is meant to be repeated for all the Kubernetes components of a BigchainDB node.

nginx-http/nginx-http-node-X-svc.yaml or nginx-https/nginx-https-node-X-svc.yaml
nginx-http/nginx-http-node-X-dep.yaml or nginx-https/nginx-https-node-X-dep.yaml
mongodb/mongodb-node-X-svc.yaml
mongodb/mongodb-node-X-sc.yaml
mongodb/mongodb-node-X-pvc.yaml
mongodb/mongodb-node-X-ss.yaml
bigchaindb/bigchaindb-node-X-svc.yaml
bigchaindb/bigchaindb-node-X-sc.yaml
bigchaindb/bigchaindb-node-X-pvc.yaml
bigchaindb/bigchaindb-node-X-ss.yaml
nginx-openresty/nginx-openresty-node-X-svc.yaml
nginx-openresty/nginx-openresty-node-X-dep.yaml

Multi Site: Multiple Azure Kubernetes Clusters¶

For the multi site deployment of a BigchainDB network with geographically dispersed nodes, we need to replicate the deployment steps for each node N number of times, N being the number of participants in the network.

The operator needs to follow a consistent naming convention which has already discussed in this document.

Note

Assuming we are using independent Kubernetes clusters, the ConfigMap and Secret Keys do not need to be updated unlike Single Site: Single Azure Kubernetes Cluster, and we also do not need to update corresponding ConfigMap/Secret imports in the Kubernetes components.

Deploy Kubernetes Services¶

Deploy the following services for each node by following the naming convention described above:

Only for multi site deployments¶

We need to make sure that clusters are able to talk to each other i.e. specifically the communication between the BigchainDB peers. Set up networking between the clusters using Kubernetes Services.

Assuming we have a BigchainDB instance bigchaindb-instance-1 residing in Azure data center location westeurope and we want to connect to bigchaindb-instance-2, bigchaindb-instance-3, and bigchaindb-instance-4 located in Azure data centers eastus, centralus and westus, respectively. Unless you already have explicitly set up networking for bigchaindb-instance-1 to communicate with bigchaindb-instance-2/3/4 and vice versa, we will have to add a Kubernetes Service in each cluster to accomplish this goal in order to set up a BigchainDB P2P network. It is similar to ensuring that there is a CNAME record in the DNS infrastructure to resolve bigchaindb-instance-X to the host where it is actually available. We can do this in Kubernetes using a Kubernetes Service of type ExternalName.

This configuration is located in the file bigchaindb/bigchaindb-ext-conn-svc.yaml.
Set the name of the metadata.name to the host name of the BigchainDB instance you are trying to connect to. For instance if you are configuring this service on cluster with bigchaindb-instance-1 then the metadata.name will be bigchaindb-instance-2 and vice versa.
Set spec.ports.port[0] to the tm-p2p-port from the ConfigMap for the other cluster.
Set spec.ports.port[1] to the tm-rpc-port from the ConfigMap for the other cluster.
Set spec.externalName to the FQDN mapped to NGINX Public IP of the cluster you are trying to connect to. For more information about the FQDN please refer to: Assign DNS name to NGINX Public IP.

Note

This operation needs to be replicated n-1 times per node for a n node cluster, with the respective FQDNs we need to communicate with.

If you are not the system administrator of the cluster, you have to get in touch with the system administrator/s of the other n-1 clusters and share with them your instance name (bigchaindb-instance-name in the ConfigMap) and the FQDN of the NGINX instance acting as Gateway(set in: Assign DNS name to NGINX Public IP).

Start NGINX Kubernetes deployments¶

Start the NGINX deployment that serves as a Gateway for each node by following the naming convention described above and referring to the following instructions:

Start the NGINX Kubernetes Deployment.

Deploy Kubernetes StorageClasses for MongoDB and BigchainDB¶

Deploy the following StorageClasses for each node by following the naming convention described above:

Deploy Kubernetes PersistentVolumeClaims for MongoDB and BigchainDB¶

Deploy the following services for each node by following the naming convention described above:

Deploy MongoDB Kubernetes StatefulSet¶

Deploy the MongoDB StatefulSet (standalone MongoDB) for each node by following the naming convention described above: and referring to the following section:

Start a Kubernetes StatefulSet for MongoDB.

Configure Users and Access Control for MongoDB¶

Configure users and access control for each MongoDB instance in the network by referring to the following section:

Configure Users and Access Control for MongoDB.

Start Kubernetes StatefulSet for BigchainDB¶

Start the BigchainDB Kubernetes StatefulSet for each node by following the naming convention described above and referring to the following instructions:

Start a Kubernetes Deployment for BigchainDB.

Start Kubernetes Deployment for MongoDB Monitoring Agent¶

Start the MongoDB monitoring agent Kubernetes deployment for each node by following the naming convention described above and referring to the following instructions:

Start a Kubernetes Deployment for MongoDB Monitoring Agent.

Start Kubernetes Deployment for OpenResty¶

Start the OpenResty Kubernetes deployment for each node by following the naming convention described above and referring to the following instructions:

Start a Kubernetes Deployment for OpenResty.

Verify and Test¶

Verify and test your setup by referring to the following instructions:

Verify the BigchainDB Node Setup.

Walkthrough: Deploy a Kubernetes Cluster on Azure using Tectonic by CoreOS¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

A BigchainDB node can be run inside a Kubernetes cluster. This page describes one way to deploy a Kubernetes cluster on Azure using Tectonic. Tectonic helps in easier cluster management of Kubernetes clusters.

If you would rather use Azure Container Service to manage Kubernetes Clusters, please read our guide for that.

Step 1: Prerequisites for Deploying Tectonic Cluster¶

Get an Azure account. Refer to this step in our docs.

Create an SSH Key pair for the new Tectonic cluster. Refer to this step in our docs.

Step 2: Get a Tectonic Subscription¶

CoreOS offers Tectonic for free for up to 10 nodes.

Sign up for an account here if you do not have one already and get a license for 10 nodes.

Login to your account, go to Overview > Your Account and save the CoreOS License and the Pull Secret to your local machine.

Step 3: Deploy the cluster on Azure¶

The latest instructions for deployment can be found here.

The following points suggests some customizations for a BigchainDB deployment when following the steps above:

Set the CLUSTER variable to the name of the cluster. Also note that the cluster will be deployed in a resource group named tectonic-cluster-CLUSTER.
Set the tectonic_base_domain to "" if you want to use Azure managed DNS. You will be assigned a cloudapp.azure.com sub-domain by default and you can skip the Configuring Azure DNS section from the Tectonic installation guide.
Set the tectonic_cl_channel to "stable" unless you want to experiment or test with the latest release.
Set the tectonic_cluster_name to the CLUSTER variable defined in the step above.
Set the tectonic_license_path and tectonic_pull_secret_path to the location where you have stored the tectonic-license.txt and the config.json files downloaded in the previous step.
Set the tectonic_etcd_count to "3", so that you have a multi-node etcd cluster that can tolerate a single node failure.
Set the tectonic_etcd_tls_enabled to "true" as this will enable TLS connectivity between the etcd nodes and their clients.
Set the tectonic_master_count to "3" so that you cane tolerate a single master failure.
Set the tectonic_worker_count to "2".
Set the tectonic_azure_location to "westeurope" if you want to host the cluster in Azure’s westeurope datacenter.
Set the tectonic_azure_ssh_key to the path of the public key created in the previous step.
We recommend setting up or using a CA(Certificate Authority) to generate Tectonic Console’s server certificate(s) and adding it to your trusted authorities on the client side, accessing the Tectonic Console i.e. Browser. If you already have a CA(self-signed or otherwise), Set the tectonic_ca_cert and tectonic_ca_key configurations with the content of PEM-encoded certificate and key files, respectively. For more information about, how to set up a self-signed CA, Please refer to How to Set up self-signed CA.
Note that the tectonic_azure_client_secret is the same as the ARM_CLIENT_SECRET.
Note that the URL for the Tectonic console using these settings will be the cluster name set in the configutation file, the datacenter name and cloudapp.azure.com. For example, if you named your cluster as test-cluster and specified the datacenter as westeurope, the Tectonic console will be available at test-cluster.westeurope.cloudapp.azure.com.
Note that, if you do not specify tectonic_ca_cert, a CA certificate will be generated automatically and you will encounter the untrusted certificate message on your client(Browser), when accessing the Tectonic Console.

Step 4: Configure kubectl¶

Refer to this tutorial for instructions on how to download the kubectl configuration files for your cluster.
Set the KUBECONFIG environment variable to make kubectl use the new config file along with the existing configuration.

$ export KUBECONFIG=$HOME/.kube/config:/path/to/config/kubectl-config

# OR to only use the new configuration, try

$ export KUBECONFIG=/path/to/config/kubectl-config

Next, you can follow one of our following deployment templates:

Kubernetes Template: Deploy a Single BigchainDB Node.

Tectonic References¶

Cluster Troubleshooting¶

This page describes some basic issues we have faced while deploying and operating the cluster.

1. MongoDB Restarts¶

We define the following in the mongo-ss.yaml file:

resources:
  limits:
    cpu: 200m
    memory: 5G

When the MongoDB cache occupies a memory greater than 5GB, it is terminated by the kubelet. This can usually be verified by logging in to the worker node running MongoDB container and looking at the syslog (the journalctl command should usually work).

This issue is resolved in PR #1757.

2. 502 Bad Gateway Error on Runscope Tests¶

It means that NGINX could not find the appropriate backed to forward the requests to. This typically happens when:

MongoDB goes down (as described above) and BigchainDB, after trying for BIGCHAINDB_DATABASE_MAXTRIES times, gives up. The Kubernetes BigchainDB Deployment then restarts the BigchainDB pod.
BigchainDB crashes for some reason. We have seen this happen when updating BigchainDB from one version to the next. This usually means the older connections to the service gets disconnected; retrying the request one more time, forwards the connection to the new instance and succeed.

3. Service Unreachable¶

Communication between Kubernetes Services and Deployments fail in v1.6.6 and before due to a trivial key lookup error for non-existent services in the kubelet. This error can be reproduced by restarting any public facing (that is, services using the cloud load balancer) Kubernetes services, and watching the kube-proxy failure in its logs. The solution to this problem is to restart kube-proxy on the affected worker/agent node. Login to the worker node and run:

docker stop `docker ps | grep k8s_kube-proxy | cut -d" " -f1`

docker logs -f `docker ps | grep k8s_kube-proxy | cut -d" " -f1`

This issue is fixed in Kubernetes v1.7.

4. Single Disk Attached to Multiple Mountpoints in a Container¶

This is currently the issue faced in one of the clusters and being debugged by the support team at Microsoft.

The issue was first seen on August 29, 2017 on the Test Network and has been logged in the Azure/acs-engine repo on GitHub.

This is apparently fixed in Kubernetes v1.7.2 which include a new disk driver, but is yet to tested by us.

5. MongoDB Monitoring Agent throws a dial error while connecting to MongoDB¶

You might see something similar to this in the MongoDB Monitoring Agent logs:

Failure dialing host without auth. Err: `no reachable servers`
    at monitoring-agent/components/dialing.go:278
    at monitoring-agent/components/dialing.go:116
    at monitoring-agent/components/dialing.go:213
    at src/runtime/asm_amd64.s:2086

The first thing to check is if the networking is set up correctly. You can use the (maybe using the toolbox container).

If everything looks fine, it might be a problem with the Preferred Hostnames setting in MongoDB Cloud Manager. If you do need to change the regular expression, ensure that it is correct and saved properly (maybe try refreshing the MongoDB Cloud Manager web page to see if the setting sticks).

Once you update the regular expression, you will need to remove the deployment and add it again for the Monitoring Agent to discover and connect to the MongoDB instance correctly.

More information about this configuration is provided in this document.

6. Create a Persistent Volume from existing Azure disk storage Resource¶

When deleting a k8s cluster, all dynamically-created PVs are deleted, along with the underlying Azure storage disks (so those can’t be used in a new cluster). resources are also deleted thus cannot be used in a new cluster. This workflow will preserve the Azure storage disks while deleting the k8s cluster and re-use the same disks on a new cluster for MongoDB persistent storage without losing any data.

The template to create two PVs for MongoDB Stateful Set (One for MongoDB data store and the other for MongoDB config store) is located at mongodb/mongo-pv.yaml.

You need to configure diskName and diskURI in mongodb/mongo-pv.yaml file. You can get these values by logging into your Azure portal and going to Resource Groups and click on your relevant resource group. From the list of resources click on the storage account resource and click the container (usually named as vhds) that contains storage disk blobs that are available for PVs. Click on the storage disk file that you wish to use for your PV and you will be able to see NAME and URL parameters which you can use for diskName and diskURI values in your template respectively and run the following command to create PVs:

$ kubectl --context <context-name> apply -f mongodb/mongo-pv.yaml

Note

Please make sure the storage disks you are using are not already being used by any other PVs. To check the existing PVs in your cluster, run the following command to get PVs and Storage disk file mapping.

$ kubectl --context <context-name> get pv --output yaml

Architecture of a BigchainDB Node Running in a Kubernetes Cluster¶

Note

A highly-available Kubernetes cluster requires at least five virtual machines (three for the master and two for your app’s containers). Therefore we don’t recommend using Kubernetes to run a BigchainDB node if that’s the only thing the Kubernetes cluster will be running. Instead, see our Node Setup. If your organization already has a big Kubernetes cluster running many containers, and your organization has people who know Kubernetes, then this Kubernetes deployment template might be helpful.

If you deploy a BigchainDB node into a Kubernetes cluster as described in these docs, it will include:

NGINX, OpenResty, BigchainDB, MongoDB and Tendermint Kubernetes Services.
NGINX, OpenResty, BigchainDB and MongoDB Monitoring Agent Kubernetes Deployments.
MongoDB and Tendermint Kubernetes StatefulSets.
Third party services like 3scale, MongoDB Cloud Manager and the Azure Operations Management Suite.

BigchainDB Node Diagram¶

../../../_images/aafig-9a592ed24a98d20934bfe610d93b536495018d4e.svg

Note

The arrows in the diagram represent the client-server communication. For example, A–>B implies that A initiates the connection to B. It does not represent the flow of data; the communication channel is always fully duplex.

NGINX: Entrypoint and Gateway¶

We use an NGINX as HTTP proxy on port 443 (configurable) at the cloud entrypoint for:

Rate Limiting: We configure NGINX to allow only a certain number of requests (configurable) which prevents DoS attacks.
HTTPS Termination: The HTTPS connection does not carry through all the way to BigchainDB and terminates at NGINX for now.
Request Routing: For HTTPS connections on port 443 (or the configured BigchainDB public api port), the connection is proxied to:
1. OpenResty Service if it is a POST request.
2. BigchainDB Service if it is a GET request.

We use an NGINX TCP proxy on port 27017 (configurable) at the cloud entrypoint for:

Rate Limiting: We configure NGINX to allow only a certain number of requests (configurable) which prevents DoS attacks.
Request Routing: For connections on port 27017 (or the configured MongoDB public api port), the connection is proxied to the MongoDB Service.

OpenResty: API Management, Authentication and Authorization¶

We use OpenResty to perform authorization checks with 3scale using the app_id and app_key headers in the HTTP request.

OpenResty is NGINX plus a bunch of other components. We primarily depend on the LuaJIT compiler to execute the functions to authenticate the app_id and app_key with the 3scale backend.

MongoDB: Standalone¶

We use MongoDB as the backend database for BigchainDB.

We achieve security by avoiding DoS attacks at the NGINX proxy layer and by ensuring that MongoDB has TLS enabled for all its connections.

Tendermint: BFT consensus engine¶

We use Tendermint as the backend consensus engine for BFT replication of BigchainDB. In a multi-node deployment, Tendermint nodes/peers communicate with each other via the public ports exposed by the NGINX gateway.

We use port 9986 (configurable) to allow tendermint nodes to access the public keys of the peers and port 26656 (configurable) for the rest of the communications between the peers.

Network of nodes with the Ansible script¶

You can find one of the installation methods with Ansible on GitHub at:

Ansible script

It allows to install BigchainDB, MongoDB, Tendermint, and python, and then connect nodes into a network. Current tested machine is Ubuntu 18.04.

API¶

The HTTP Client-Server API¶

This page assumes you already know an API Root URL for a BigchainDB node or reverse proxy. It should be something like https://example.com:9984 or https://12.34.56.78:9984.

If you set up a BigchainDB node or reverse proxy yourself, and you’re not sure what the API Root URL is, then see the last section of this page for help.

BigchainDB Root URL¶

If you send an HTTP GET request to the BigchainDB Root URL e.g. http://localhost:9984 or https://example.com:9984 (with no /api/v1/ on the end), then you should get an HTTP response with something like the following in the body:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "api": {
    "v1": {
      "assets": "/api/v1/assets/",
      "blocks": "/api/v1/blocks/",
      "docs": "https://docs.bigchaindb.com/projects/server/en/v2.2.2/http-client-server-api.html",
      "metadata": "/api/v1/metadata/",
      "outputs": "/api/v1/outputs/",
      "streams": "ws://localhost:9985/api/v1/streams/valid_transactions",
      "transactions": "/api/v1/transactions/",
      "validators": "/api/v1/validators"
    }
  },
  "docs": "https://docs.bigchaindb.com/projects/server/en/v2.2.2/",
  "software": "BigchainDB",
  "version": "2.2.2"
}

API Root Endpoint¶

If you send an HTTP GET request to the API Root Endpoint e.g. http://localhost:9984/api/v1/ or https://example.com:9984/api/v1/, then you should get an HTTP response that allows you to discover the BigchainDB API endpoints:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "assets": "/assets/",
  "blocks": "/blocks/",
  "docs": "https://docs.bigchaindb.com/projects/server/en/v2.2.2/http-client-server-api.html",
  "metadata": "/metadata/",
  "outputs": "/outputs/",
  "streams": "ws://localhost:9985/api/v1/streams/valid_transactions",
  "transactions": "/transactions/",
  "validators": "/validators"
}

Transactions Endpoint¶

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/transactions/{transaction_id}¶

Get the transaction with the ID transaction_id.

If a transaction with ID transaction_id has been included in a committed block, then this endpoint returns that transaction, otherwise the response will be 404 Not Found.

Parameters:	transaction_id (hex string) – transaction ID

Example request:

GET /api/v1/transactions/4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "asset": {
    "data": {
      "msg": "Hello BigchainDB!"
    }
  },
  "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
      "fulfills": null,
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 0
  },
  "operation": "CREATE",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "version": "2.0"
}

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – A transaction with that ID was found. 404 Not Found – A transaction with that ID was not found.

GET /api/v1/transactions¶: Requests to the /api/v1/transactions endpoint without any query parameters will get a response status code 400 Bad Request.

GET /api/v1/transactions?asset_id={asset_id}&operation={CREATE|TRANSFER}&last_tx={true|false}¶

Get a list of transactions that use an asset with the ID asset_id.

If operation is CREATE, then the CREATE transaction which created the asset with ID asset_id will be returned.

If operation is TRANSFER, then every TRANSFER transaction involving the asset with ID asset_id will be returned. This allows users to query the entire history or provenance of an asset.

If operation is not included, then every transaction involving the asset with ID asset_id will be returned.

if last_tx is set to true, only the last transaction is returned instead of all transactions with the given asset_id.

This endpoint returns transactions only if they are in committed blocks.

Query Parameters:
	operation (string) – (Optional) `CREATE` or `TRANSFER`. asset_id (string) – asset ID. last_tx (string) – (Optional) `true` or `false`.

Example request:

GET /api/v1/transactions?operation=TRANSFER&asset_id=4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[{
  "asset": {
    "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317"
  },
  "id": "79ef6803210c941903d63d08b40fa17f0a5a04f11ac0ff04451553a187d97a30",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUAYRI8kzKaZcrW-_avQrAIk5q-7o_7U6biBvoHk1ioBLqHSBcE_PAdNEaeWesAAW_HeCqNUWKaJ5Lzo5Nfz7QgN",
      "fulfills": {
        "output_index": 0,
        "transaction_id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317"
      },
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 1
  },
  "operation": "TRANSFER",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "3yfQPHeWAa1MxTX9Zf9176QqcpcnWcanVZZbaHb8B3h9",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;lu6ov4AKkee6KWGnyjOVLBeyuP0bz4-O6_dPi15eYUc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "3yfQPHeWAa1MxTX9Zf9176QqcpcnWcanVZZbaHb8B3h9"
      ]
    }
  ],
  "version": "2.0"
},
{
  "asset": {
    "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317"
  },
  "id": "1fec726a3b426498147f1a1f19a92c187d551a7f66db4b88d666d7dcc10e86a4",
  "inputs": [
    {
      "fulfillment": "pGSAICw7Ul-c2lG6NFbHp3FbKRC7fivQcNGO7GS4wV3A-1QggUARCMty2JBK_OyPJntWEFxDG4-VbKMy853NtqwnPib5QUJIuwPQa1Y4aN2iIBuoqGE85Pmjcc1ScG9FCPSQHacK",
      "fulfills": {
        "output_index": 0,
        "transaction_id": "79ef6803210c941903d63d08b40fa17f0a5a04f11ac0ff04451553a187d97a30"
      },
      "owners_before": [
        "3yfQPHeWAa1MxTX9Zf9176QqcpcnWcanVZZbaHb8B3h9"
      ]
    }
  ],
  "metadata": {
    "sequence": 2
  },
  "operation": "TRANSFER",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "3Af3fhhjU6d9WecEM9Uw5hfom9kNEwE7YuDWdqAUssqm",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;Ll1r0LzgHUvWB87yIrNFYo731MMUEypqvrbPATTbuD4?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "3Af3fhhjU6d9WecEM9Uw5hfom9kNEwE7YuDWdqAUssqm"
      ]
    }
  ],
  "version": "2.0"
}]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – A list of transactions containing an asset with ID `asset_id` was found and returned. 400 Bad Request – The request wasn’t understood by the server, e.g. the `asset_id` querystring was not included in the request.

POST /api/v1/transactions?mode={mode}¶

This endpoint is used to send a transaction to a BigchainDB network. The transaction is put in the body of the request.

Query Parameters:
	mode (string) – (Optional) One of the three supported modes to send a transaction: `async`, `sync`, `commit`. The default is `async`.

Once the posted transaction arrives at a BigchainDB node, that node will check to see if the transaction is valid. If it’s invalid, the node will return an HTTP 400 (error). Otherwise, the node will send the transaction to Tendermint (in the same node) using the Tendermint broadcast API.

The meaning of the mode query parameter is inherited from the mode parameter in Tendermint’s broadcast API. mode=async means the HTTP response will come back immediately, before Tendermint asks BigchainDB Server to check the validity of the transaction (a second time). mode=sync means the HTTP response will come back after Tendermint gets a response from BigchainDB Server regarding the validity of the transaction. mode=commit means the HTTP response will come back once the transaction is in a committed block.

Note

In the async and sync modes, after a successful HTTP response is returned, the transaction may still be rejected later on. All the transactions are recorded internally by Tendermint in WAL (Write-Ahead Log) before the HTTP response is returned. Nevertheless, the following should be noted:

Transactions in WAL including the failed ones are not exposed in any of the BigchainDB or Tendermint APIs.
Transactions are never fetched from WAL. WAL is never replayed.
A critical failure (e.g. the system is out of disk space) may occur preventing transactions from being stored in WAL, even when the HTTP response indicates a success.
If a transaction fails the validation because it conflicts with the other transactions of the same block, Tendermint includes it into its block, but BigchainDB does not store these transactions and does not offer any information about them in the APIs.

Note

The posted transaction should be valid. The relevant BigchainDB Transactions Spec explains how to build a valid transaction and how to check if a transaction is valid. One would normally use a driver such as the BigchainDB Python Driver to build a valid transaction.

Note

A client can subscribe to the WebSocket Event Stream API to listen for committed transactions.

Example request:

POST /api/v1/transactions?mode=async HTTP/1.1
Host: example.com
Content-Type: application/json

{
  "asset": {
    "data": {
      "msg": "Hello BigchainDB!"
    }
  },
  "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
      "fulfills": null,
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 0
  },
  "operation": "CREATE",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "version": "2.0"
}

Example response:

HTTP/1.1 202 Accepted
Content-Type: application/json

{
  "asset": {
    "data": {
      "msg": "Hello BigchainDB!"
    }
  },
  "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
  "inputs": [
    {
      "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
      "fulfills": null,
      "owners_before": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "metadata": {
    "sequence": 0
  },
  "operation": "CREATE",
  "outputs": [
    {
      "amount": "1",
      "condition": {
        "details": {
          "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
          "type": "ed25519-sha-256"
        },
        "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
      },
      "public_keys": [
        "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
      ]
    }
  ],
  "version": "2.0"
}

Response Headers:
	Content-Type – `application/json`
Status Codes:	202 Accepted – The meaning of this response depends on the value of the `mode` parameter. See above. 400 Bad Request – The posted transaction was invalid.

POST /api/v1/transactions¶: This endpoint (without any parameters) will push a new transaction. Since no mode parameter is included, the default mode is assumed: async.

Transaction Outputs¶

The /api/v1/outputs endpoint returns transactions outputs filtered by a given public key, and optionally filtered to only include either spent or unspent outputs.

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/outputs¶

Get transaction outputs by public key. The public_key parameter must be a base58 encoded ed25519 public key associated with transaction output ownership.

Returns a list of transaction outputs.

Parameters:

public_key – Base58 encoded public key associated with output ownership. This parameter is mandatory and without it the endpoint will return a 400 response code.
spent – (Optional) Boolean value (true or false) indicating if the result set should include only spent or only unspent outputs. If not specified, the result includes all the outputs (both spent and unspent) associated with the public_key.

GET /api/v1/outputs?public_key={public_key}¶

Return all outputs, both spent and unspent, for the public_key.

Example request:

GET /api/v1/outputs?public_key=1AAAbbb...ccc HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "output_index": 0,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  },
  {
    "output_index": 1,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  }
]

Status Codes:	200 OK – A list of outputs was found and returned in the body of the response. 400 Bad Request – The request wasn’t understood by the server, e.g. the `public_key` querystring was not included in the request.

GET /api/v1/outputs?public_key={public_key}&spent=true¶

Return all spent outputs for public_key.

Example request:

GET /api/v1/outputs?public_key=1AAAbbb...ccc&spent=true HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "output_index": 0,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  }
]

Status Codes:	200 OK – A list of outputs were found and returned in the body of the response. 400 Bad Request – The request wasn’t understood by the server, e.g. the `public_key` querystring was not included in the request.

GET /api/v1/outputs?public_key={public_key}&spent=false¶

Return all unspent outputs for public_key.

Example request:

GET /api/v1/outputs?public_key=1AAAbbb...ccc&spent=false HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  {
    "output_index": 1,
    "transaction_id": "2d431073e1477f3073a4693ac7ff9be5634751de1b8abaa1f4e19548ef0b4b0e"
  }
]

Status Codes:	200 OK – A list of outputs were found and returned in the body of the response. 400 Bad Request – The request wasn’t understood by the server, e.g. the `public_key` querystring was not included in the request.

Assets¶

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/assets¶

Return all the assets that match a given text search.

Query Parameters:
	search (string) – Text search string to query. limit (int) – (Optional) Limit the number of returned assets. Defaults to `0` meaning return all matching assets.

GET /api/v1/assets/?search={search}¶

Return all assets that match a given text search.

Note

The id of the asset is the same id of the CREATE transaction that created the asset.

Note

You can use assets/?search or assets?search.

If no assets match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/assets/?search=bigchaindb HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "data": {"msg": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "data": {"msg": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
    {
        "data": {"msg": "Hello BigchainDB 3!"},
        "id": "fa6bcb6a8fdea3dc2a860fcdc0e0c63c9cf5b25da8b02a4db4fb6a2d36d27791"
    }
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

GET /api/v1/assets?search={search}&limit={n_documents}¶

Return at most n_documents assets that match a given text search.

If no assets match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/assets?search=bigchaindb&limit=2 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "data": {"msg": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "data": {"msg": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

Transaction Metadata¶

Note

If you want to do more sophisticated queries than those provided by the BigchainDB HTTP API, then one option is to connect to MongoDB directly (if possible) and do whatever queries MongoDB allows. For more about that option, see the page about querying BigchainDB.

GET /api/v1/metadata¶

Return all the metadata objects that match a given text search.

Query Parameters:
	search (string) – Text search string to query. limit (int) – (Optional) Limit the number of returned metadata objects. Defaults to `0` meaning return all matching objects.

GET /api/v1/metadata/?search={search}¶

Return all metadata objects that match a given text search.

Note

The id of the metadata is the same id of the transaction where it was defined.

Note

You can use metadata/?search or metadata?search.

If no metadata objects match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/metadata/?search=bigchaindb HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "metadata": {"metakey1": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "metadata": {"metakey2": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
    {
        "metadata": {"metakey3": "Hello BigchainDB 3!"},
        "id": "fa6bcb6a8fdea3dc2a860fcdc0e0c63c9cf5b25da8b02a4db4fb6a2d36d27791"
    }
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

GET /api/v1/metadata/?search={search}&limit={n_documents}¶

Return at most n_documents metadata objects that match a given text search.

If no metadata objects match the text search it returns an empty list.

If the text string is empty or the server does not support text search, a 400 Bad Request is returned.

The results are sorted by text score. For more information about the behavior of text search, see MongoDB text search behavior.

Example request:

GET /api/v1/metadata?search=bigchaindb&limit=2 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "metadata": {"msg": "Hello BigchainDB 1!"},
        "id": "51ce82a14ca274d43e4992bbce41f6fdeb755f846e48e710a3bbb3b0cf8e4204"
    },
    {
        "metadata": {"msg": "Hello BigchainDB 2!"},
        "id": "b4e9005fa494d20e503d916fa87b74fe61c079afccd6e084260674159795ee31"
    },
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully. 400 Bad Request – The query was not executed successfully. Returned if the text string is empty or the server does not support text search.

Validators¶

GET /api/v1/validators¶

Return the local validators set of a given node.

Example request:

GET /api/v1/validators HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-type: application/json

[
    {
        "pub_key": {
               "data":"4E2685D9016126864733225BE00F005515200727FBAB1312FC78C8B76831255A",
               "type":"ed25519"
        },
        "power": 10
    },
    {
         "pub_key": {
               "data":"608D839D7100466D6BA6BE79C320F8B81DE93CFAA58CF9768CF921C6371F2553",
               "type":"ed25519"
         },
         "power": 5
    }
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The query was executed successfully and validators set was returned.

Blocks¶

GET /api/v1/blocks/{block_height}¶

Get the block with the height block_height.

Parameters:	block_height (integer) – block height

Example request:

GET /api/v1/blocks/1 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

{
  "height": 1,
  "transactions": [
    {
      "asset": {
        "data": {
          "msg": "Hello BigchainDB!"
        }
      },
      "id": "4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317",
      "inputs": [
        {
          "fulfillment": "pGSAIDE5i63cn4X8T8N1sZ2mGkJD5lNRnBM4PZgI_zvzbr-cgUCy4BR6gKaYT-tdyAGPPpknIqI4JYQQ-p2nCg3_9BfOI-15vzldhyz-j_LZVpqAlRmbTzKS-Q5gs7ZIFaZCA_UD",
          "fulfills": null,
          "owners_before": [
            "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
          ]
        }
      ],
      "metadata": {
        "sequence": 0
      },
      "operation": "CREATE",
      "outputs": [
        {
          "amount": "1",
          "condition": {
            "details": {
              "public_key": "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD",
              "type": "ed25519-sha-256"
            },
            "uri": "ni:///sha-256;PNYwdxaRaNw60N6LDFzOWO97b8tJeragczakL8PrAPc?fpt=ed25519-sha-256&cost=131072"
          },
          "public_keys": [
            "4K9sWUMFwTgaDGPfdynrbxWqWS6sWmKbZoTjxLtVUibD"
          ]
        }
      ],
      "version": "2.0"
    }
  ]
}

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – A block with that block height was found. 400 Bad Request – The request wasn’t understood by the server, e.g. just requesting `/blocks` without the `block_height`. 404 Not Found – A block with that block height was not found.

GET /api/v1/blocks¶

The unfiltered /blocks endpoint without any query parameters returns a 400 Bad Request status code.

Example request:

GET /api/v1/blocks HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 400 Bad Request

Status Codes:	400 Bad Request – The request wasn’t understood by the server, e.g. just requesting `/blocks` without the `block_height`.

GET /api/v1/blocks?transaction_id={transaction_id}¶

Retrieve a list of block IDs (block heights), such that the blocks with those IDs contain a transaction with the ID transaction_id. A correct response may consist of an empty list or a list with one block ID.

Note

In case no block was found, an empty list and an HTTP status code 200 OK is returned, as the request was still successful.

Query Parameters:
	transaction_id (string) – (Required) transaction ID

Example request:

GET /api/v1/blocks?transaction_id=4957744b3ac54434b8270f2c854cc1040228c82ea4e72d66d2887a4d3e30b317 HTTP/1.1
Host: example.com

Example response:

HTTP/1.1 200 OK
Content-Type: application/json

[
  1
]

Response Headers:
	Content-Type – `application/json`
Status Codes:	200 OK – The request was properly formed and zero or more blocks were found containing the specified `transaction_id`. 400 Bad Request – The request wasn’t understood by the server, e.g. just requesting `/blocks`, without defining `transaction_id`.

Determining the API Root URL¶

When you start BigchainDB Server using bigchaindb start, an HTTP API is exposed at some address. The default is:

http://localhost:9984/api/v1/

It’s bound to localhost, so you can access it from the same machine, but it won’t be directly accessible from the outside world. (The outside world could connect via a SOCKS proxy or whatnot.)

The documentation about BigchainDB Server Configuration Settings has a section about how to set server.bind so as to make the HTTP API publicly accessible.

If the API endpoint is publicly accessible, then the public API Root URL is determined as follows:

The public IP address (like 12.34.56.78) is the public IP address of the machine exposing the HTTP API to the public internet (e.g. either the machine hosting Gunicorn or the machine running the reverse proxy such as NGINX). It’s determined by AWS, Azure, Rackspace, or whoever is hosting the machine.
The DNS hostname (like example.com) is determined by DNS records, such as an “A Record” associating example.com with 12.34.56.78
The port (like 9984) is determined by the server.bind setting if Gunicorn is exposed directly to the public Internet. If a reverse proxy (like NGINX) is exposed directly to the public Internet instead, then it could expose the HTTP API on whatever port it wants to. (It should expose the HTTP API on port 9984, but it’s not bound to do that by anything other than convention.)

The WebSocket Event Stream API¶

Important

The WebSocket Event Stream runs on a different port than the Web API. The default port for the Web API is 9984, while the one for the Event Stream is 9985.

BigchainDB provides real-time event streams over the WebSocket protocol with the Event Stream API. Connecting to an event stream from your application enables a BigchainDB node to notify you as events occur, such as new valid transactions.

Demoing the API¶

You may be interested in demoing the Event Stream API with the WebSocket echo test to familiarize yourself before attempting an integration.

Determining Support for the Event Stream API¶

It’s a good idea to make sure that the node you’re connecting with has advertised support for the Event Stream API. To do so, send a HTTP GET request to the node’s API Root Endpoint (e.g. http://localhost:9984/api/v1/) and check that the response contains a streams property:

{
 ...,
 "streams": "ws://example.com:9985/api/v1/streams/valid_transactions",
 ...
}

Connection Keep-Alive¶

The Event Stream API supports Ping/Pong frames as descibed in RFC 6455.

Note

It might not be possible to send PING/PONG frames via web browsers because of non availability of Javascript API on different browsers to achieve the same.

Streams¶

Each stream is meant as a unidirectional communication channel, where the BigchainDB node is the only party sending messages. Any messages sent to the BigchainDB node will be ignored.

Streams will always be under the WebSocket protocol (so ws:// or wss://) and accessible as extensions to the /api/v<version>/streams/ API root URL (for example, valid transactions would be accessible under /api/v1/streams/valid_transactions). If you’re running your own BigchainDB instance and need help determining its root URL, then see the page titled Determining the API Root URL.

All messages sent in a stream are in the JSON format.

Note

For simplicity, BigchainDB initially only provides a stream for all committed transactions. In the future, we may provide streams for other information. We may also provide the ability to filter the stream for specific qualities, such as a specific output’s public_key.

If you have specific use cases that you think would fit as part of this API, consider creating a new BEP.

Valid Transactions¶

/valid_transactions

Streams an event for any newly valid transactions committed to a block. Message bodies contain the transaction’s ID, associated asset ID, and containing block’s height.

Example message:

{
    "transaction_id": "<sha3-256 hash>",
    "asset_id": "<sha3-256 hash>",
    "height": <int>
}

Note

Transactions in BigchainDB are committed in batches (“blocks”) and will, therefore, be streamed in batches.

Commands And Backend¶

This section contains auto-generated documentation of various functions, classes and methods in the BigchainDB Server code, based on Python docstrings in the code itself.

Warning

While we try to keep docstrings accurate, if you want to know for sure what the code does, then you have to read the code itself.

Command Line Interface¶

`bigchaindb.commands.bigchaindb`¶

Implementation of the bigchaindb command, the command-line interface (CLI) for BigchainDB Server.

bigchaindb.commands.bigchaindb.run_show_config(args)[source]¶: Show the current configuration

bigchaindb.commands.bigchaindb.run_configure(args)[source]¶: Run a script to configure the current node.

bigchaindb.commands.bigchaindb.run_election(args)[source]¶: Initiate and manage elections

bigchaindb.commands.bigchaindb.run_election_new_upsert_validator(args, bigchain)[source]¶

Initiates an election to add/update/remove a validator to an existing BigchainDB network

Parameters:	args – dict args = { ‘public_key’: the public key of the proposed peer, (str) ‘power’: the proposed validator power for the new peer, (str) ‘node_id’: the node_id of the new peer (str) ‘sk’: the path to the private key of the node calling the election (str) } bigchain – an instance of BigchainDB
Returns:	election_id or False in case of failure

bigchaindb.commands.bigchaindb.run_election_new_chain_migration(args, bigchain)[source]¶

Initiates an election to halt block production

Parameters:	args – dict args = { ‘sk’: the path to the private key of the node calling the election (str) } bigchain – an instance of BigchainDB
Returns:	election_id or False in case of failure

bigchaindb.commands.bigchaindb.run_election_approve(args, bigchain)[source]¶

Approve an election

Parameters:	args – dict args = { ‘election_id’: the election_id of the election (str) ‘sk’: the path to the private key of the signer (str) } bigchain – an instance of BigchainDB
Returns:	success log message or False in case of error

bigchaindb.commands.bigchaindb.run_election_show(args, bigchain)[source]¶

Retrieves information about an election

Parameters:	args – dict args = { ‘election_id’: the transaction_id for an election (str) } bigchain – an instance of BigchainDB

bigchaindb.commands.bigchaindb.run_init(args)[source]¶: Initialize the database

bigchaindb.commands.bigchaindb.run_drop(args)[source]¶: Drop the database

bigchaindb.commands.bigchaindb.run_start(args)[source]¶: Start the processes to run the node

bigchaindb.commands.bigchaindb.run_tendermint_version(args)[source]¶: Show the supported Tendermint version(s)

`bigchaindb.commands.utils`¶

Utility functions and basic common arguments for argparse.ArgumentParser.

bigchaindb.commands.utils.configure_bigchaindb(command)[source]¶

Decorator to be used by command line functions, such that the configuration of bigchaindb is performed before the execution of the command.

Parameters:	command – The command to decorate.
Returns:	The command wrapper function.

bigchaindb.commands.utils.input_on_stderr(prompt='', default=None, convert=None)[source]¶

Output a string to stderr and wait for input.

Parameters:	prompt (str) – the message to display. default – the default value to return if the user leaves the field empty convert (callable) – a callable to be used to convert the value the user inserted. If None, the type of `default` will be used.

bigchaindb.commands.utils.start(parser, argv, scope)[source]¶

Utility function to execute a subcommand.

The function will look up in the scope if there is a function called run_<parser.args.command> and will run it using parser.args as first positional argument.

Parameters:	parser – an ArgumentParser instance. argv – the list of command line arguments without the script name. scope (dict) – map containing (eventually) the functions to be called.
Raises:	`NotImplementedError` – if `scope` doesn’t contain a function called `run_<parser.args.command>`.

The BigchainDB Class¶

class bigchaindb.BigchainDB(connection=None)[source]¶

Bigchain API

Create, read, sign, write transactions to the database

post_transaction(transaction, mode)[source]¶: Submit a valid transaction to the mempool.

write_transaction(transaction, mode)[source]¶: Submit a valid transaction to the mempool.

update_utxoset(transaction)[source]¶

Update the UTXO set given transaction. That is, remove the outputs that the given transaction spends, and add the outputs that the given transaction creates.

Parameters:	transaction (`Transaction`) – A new transaction incoming into the system for which the UTXO set needs to be updated.

store_unspent_outputs(*unspent_outputs)[source]¶

Store the given unspent_outputs (utxos).

Parameters:	*unspent_outputs (`tuple` of `dict`) – Variable length tuple or list of unspent outputs.

get_utxoset_merkle_root()[source]¶

Returns the merkle root of the utxoset. This implies that the utxoset is first put into a merkle tree.

For now, the merkle tree and its root will be computed each time. This obviously is not efficient and a better approach that limits the repetition of the same computation when unnecesary should be sought. For instance, future optimizations could simply re-compute the branches of the tree that were affected by a change.

The transaction hash (id) and output index should be sufficient to uniquely identify a utxo, and consequently only that information from a utxo record is needed to compute the merkle root. Hence, each node of the merkle tree should contain the tuple (txid, output_index).

Important

The leaves of the tree will need to be sorted in some kind of lexicographical order.

Returns:	Merkle root in hexadecimal form.
Return type:	str

get_unspent_outputs()[source]¶

Get the utxoset.

Returns:	generator of unspent_outputs.

delete_unspent_outputs(*unspent_outputs)[source]¶

Deletes the given unspent_outputs (utxos).

Parameters:	*unspent_outputs (`tuple` of `dict`) – Variable length tuple or list of unspent outputs.

get_transactions_filtered(asset_id, operation=None, last_tx=None)[source]¶: Get a list of transactions filtered on some criteria

get_outputs_filtered(owner, spent=None)[source]¶

Get a list of output links filtered on some criteria

Parameters:	owner (str) – base58 encoded public_key. spent (bool) – If `True` return only the spent outputs. If `False` return only unspent outputs. If spent is not specified (`None`) return all outputs.
Returns:	list of `txid` s and `output` s pointing to another transaction’s condition
Return type:	`list` of TransactionLink

store_block(block)[source]¶: Create a new block.

get_latest_block()[source]¶: Get the block with largest height.

get_block(block_id)[source]¶

Get the block with the specified block_id.

Returns the block corresponding to block_id or None if no match is found.

Parameters:	block_id (int) – block id of the block to get.

get_block_containing_tx(txid)[source]¶

Retrieve the list of blocks (block ids) containing a: transaction with transaction id txid

Parameters:	txid (str) – transaction id of the transaction to query
Returns:	Block id list (list(int))

validate_transaction(tx, current_transactions=[])[source]¶: Validate a transaction against the current status of the database.

text_search(search, *, limit=0, table='assets')[source]¶

Return an iterator of assets that match the text search

Parameters:	search (str) – Text search string to query the text index limit (int, optional) – Limit the number of returned documents.
Returns:	An iterator of assets that match the text search.
Return type:	iter

get_assets(asset_ids)[source]¶

Return a list of assets that match the asset_ids

Parameters:	asset_ids (`list` of `str`) – A list of asset_ids to retrieve from the database.
Returns:	The list of assets returned from the database.
Return type:	list

get_metadata(txn_ids)[source]¶

Return a list of metadata that match the transaction ids (txn_ids)

Parameters:	txn_ids (`list` of `str`) – A list of txn_ids to retrieve from the database.
Returns:	The list of metadata returned from the database.
Return type:	list

store_validator_set(height, validators)[source]¶: Store validator set at a given height. NOTE: If the validator set already exists at that height then an exception will be raised.

migrate_abci_chain()[source]¶

Generate and record a new ABCI chain ID. New blocks are not accepted until we receive an InitChain ABCI request with the matching chain ID and validator set.

Chain ID is generated based on the current chain and height. chain-X => chain-X-migrated-at-height-5. chain-X-migrated-at-height-5 => chain-X-migrated-at-height-21.

If there is no known chain (we are at genesis), the function returns.

Database Backend Interfaces¶

Generic backend database interfaces expected by BigchainDB.

The interfaces in this module allow BigchainDB to be agnostic about its database backend. One can configure BigchainDB to use different databases as its data store by setting the database.backend property in the configuration or the BIGCHAINDB_DATABASE_BACKEND environment variable.

Generic Interfaces¶

bigchaindb.backend.connection¶

bigchaindb.backend.connection.connect(backend=None, host=None, port=None, name=None, max_tries=None, connection_timeout=None, replicaset=None, ssl=None, login=None, password=None, ca_cert=None, certfile=None, keyfile=None, keyfile_passphrase=None, crlfile=None)[source]¶

Create a new connection to the database backend.

All arguments default to the current configuration’s values if not given.

Parameters:	backend (str) – the name of the backend to use. host (str) – the host to connect to. port (int) – the port to connect to. name (str) – the name of the database to use. replicaset (str) – the name of the replica set (only relevant for MongoDB connections).
Returns:	An instance of `Connection` based on the given (or defaulted) `backend`.
Raises:	`ConnectionError` – If the connection to the database fails. `ConfigurationError` – If the given (or defaulted) `backend` is not supported or could not be loaded. `AuthenticationError` – If there is a OperationFailure due to Authentication failure after connecting to the database.

class bigchaindb.backend.connection.Connection(host=None, port=None, dbname=None, connection_timeout=None, max_tries=None, **kwargs)[source]¶

Connection class interface.

All backend implementations should provide a connection class that inherits from and implements this class.

__init__(host=None, port=None, dbname=None, connection_timeout=None, max_tries=None, **kwargs)[source]¶

Create a new Connection instance.

Parameters:

host (str) – the host to connect to.
port (int) – the port to connect to.
dbname (str) – the name of the database to use.
connection_timeout (int, optional) – the milliseconds to wait until timing out the database connection attempt. Defaults to 5000ms.
max_tries (int, optional) – how many tries before giving up, if 0 then try forever. Defaults to 3.
**kwargs – arbitrary keyword arguments provided by the configuration’s database settings

run(query)[source]¶

Run a query.

Parameters:	query – the query to run
Raises:	`DuplicateKeyError` – If the query fails because of a duplicate key constraint. `OperationFailure` – If the query fails for any other reason. `ConnectionError` – If the connection to the database fails.

connect()[source]¶

Try to connect to the database.

Raises:	`ConnectionError` – If the connection to the database fails.

bigchaindb.backend.query¶

Query interfaces for backends.

bigchaindb.backend.query.store_asset(connection, asset)[source]¶

Write an asset to the asset table.

Parameters:	asset (dict) – the asset.
Returns:	The result of the operation.

bigchaindb.backend.query.store_assets(connection, assets)[source]¶

Write a list of assets to the assets table.

Parameters:	assets (list) – a list of assets to write.
Returns:	The database response.

bigchaindb.backend.query.store_metadatas(connection, metadata)[source]¶

Write a list of metadata to metadata table.

Parameters:	metadata (list) – list of metadata.
Returns:	The result of the operation.

bigchaindb.backend.query.store_transactions(connection, signed_transactions)[source]¶: Store the list of transactions.

bigchaindb.backend.query.get_transaction(connection, transaction_id)[source]¶

Get a transaction from the transactions table.

Parameters:	transaction_id (str) – the id of the transaction.
Returns:	The result of the operation.

bigchaindb.backend.query.get_transactions(connection, transaction_ids)[source]¶

Get transactions from the transactions table.

Parameters:	transaction_ids (list) – list of transaction ids to fetch
Returns:	The result of the operation.

bigchaindb.backend.query.get_asset(connection, asset_id)[source]¶

Get a transaction from the transactions table.

Parameters:	asset_id (str) – the id of the asset
Returns:	The result of the operation.

bigchaindb.backend.query.get_spent(connection, transaction_id, condition_id)[source]¶

Check if a txid was already used as an input.

A transaction can be used as an input for another transaction. Bigchain needs to make sure that a given txid is only used once.

Parameters:	transaction_id (str) – The id of the transaction. condition_id (int) – The index of the condition in the respective transaction.
Returns:	The transaction that used the txid as an input else None

bigchaindb.backend.query.get_spending_transactions(connection, inputs)[source]¶

Return transactions which spend given inputs

Parameters:	inputs (list) – list of {txid, output}
Returns:	Iterator of (block_ids, transaction) for transactions that spend given inputs.

bigchaindb.backend.query.get_owned_ids(connection, owner)[source]¶

Retrieve a list of txids that can we used has inputs.

Parameters:	owner (str) – base58 encoded public key.
Returns:	Iterator of (block_id, transaction) for transactions that list given owner in conditions.

bigchaindb.backend.query.get_block(connection, block_id)[source]¶

Get a block from the bigchain table.

Parameters:	block_id (str) – block id of the block to get
Returns:	the block or None
Return type:	block (dict)

bigchaindb.backend.query.get_block_with_transaction(connection, txid)[source]¶

Get a block containing transaction id txid

Parameters:	txid (str) – id of transaction to be searched.
Returns:	the block id or None
Return type:	block_id (int)

bigchaindb.backend.query.get_metadata(connection, transaction_ids)[source]¶

Get a list of metadata from the metadata table.

Parameters:	transaction_ids (list) – a list of ids for the metadata to be retrieved from database. (the) –
Returns:	the list of returned metadata.
Return type:	metadata (list)

bigchaindb.backend.query.get_assets(connection, asset_ids)[source]¶

Get a list of assets from the assets table. :param asset_ids: a list of ids for the assets to be retrieved from :type asset_ids: list :param the database.:

Returns:	the list of returned assets.
Return type:	assets (list)

bigchaindb.backend.query.get_txids_filtered(connection, asset_id, operation=None)[source]¶

Return all transactions for a particular asset id and optional operation.

Parameters:	asset_id (str) – ID of transaction that defined the asset operation (str) (optional) – Operation to filter on

bigchaindb.backend.query.text_search(conn, search, *, language='english', case_sensitive=False, diacritic_sensitive=False, text_score=False, limit=0, table=None)[source]¶

Return all the assets that match the text search.

The results are sorted by text score. For more information about the behavior of text search on MongoDB see https://docs.mongodb.com/manual/reference/operator/query/text/#behavior

Parameters:	search (str) – Text search string to query the text index language (str, optional) – The language for the search and the rules for stemmer and tokenizer. If the language is `None` text search uses simple tokenization and no stemming. case_sensitive (bool, optional) – Enable or disable case sensitive search. diacritic_sensitive (bool, optional) – Enable or disable case sensitive diacritic search. text_score (bool, optional) – If `True` returns the text score with each document. limit (int, optional) – Limit the number of returned documents.
Returns:	a list of assets
Return type:	`list` of `dict`
Raises:	`OperationError` – If the backend does not support text search

bigchaindb.backend.query.get_latest_block(conn)[source]¶: Get the latest commited block i.e. block with largest height

bigchaindb.backend.query.store_block(conn, block)[source]¶

Write a new block to the blocks table

Parameters:	block (dict) – block with current height and block hash.
Returns:	The result of the operation.

bigchaindb.backend.query.store_unspent_outputs(connection, unspent_outputs)[source]¶: Store unspent outputs in utxo_set table.

bigchaindb.backend.query.delete_unspent_outputs(connection, unspent_outputs)[source]¶: Delete unspent outputs in utxo_set table.

bigchaindb.backend.query.delete_transactions(conn, txn_ids)[source]¶

Delete transactions from database

Parameters:	txn_ids (list) – list of transaction ids
Returns:	The result of the operation.

bigchaindb.backend.query.get_unspent_outputs(connection, *, query=None)[source]¶

Retrieves unspent outputs.

Parameters:	query (dict) – An optional parameter to filter the result set. Defaults to `None`, which means that all UTXO records will be returned.
Returns:	Generator yielding unspent outputs (UTXO set) according to the given query.

bigchaindb.backend.query.store_pre_commit_state(connection, state)[source]¶

Store pre-commit state.

Parameters:	state (dict) – pre-commit state.
Returns:	The result of the operation.

bigchaindb.backend.query.get_pre_commit_state(connection)[source]¶

Get pre-commit state.

Returns:	Document representing the pre-commit state.

bigchaindb.backend.query.store_validator_set(conn, validator_update)[source]¶: Store updated validator set

bigchaindb.backend.query.delete_validator_set(conn, height)[source]¶: Delete the validator set at the given height.

bigchaindb.backend.query.store_election(conn, election_id, height, is_concluded)[source]¶: Store election record

bigchaindb.backend.query.store_elections(conn, elections)[source]¶: Store election records in bulk

bigchaindb.backend.query.delete_elections(conn, height)[source]¶: Delete all election records at the given height

bigchaindb.backend.query.get_validator_set(conn, height)[source]¶: Get validator set for a given height, if height is not specified then return the latest validator set

bigchaindb.backend.query.get_election(conn, election_id)[source]¶: Return the election record

bigchaindb.backend.query.get_asset_tokens_for_public_key(connection, asset_id, public_key)[source]¶

Retrieve a list of tokens of type asset_id that are owned by the public_key. :param asset_id: Id of the token. :type asset_id: str :param public_key: base58 encoded public key :type public_key: str

Returns:	Iterator of transaction that list given owner in conditions.

bigchaindb.backend.query.store_abci_chain(conn, height, chain_id, is_synced=True)[source]¶

Create or update an ABCI chain at the given height. Usually invoked in the beginning of the ABCI communications (height=0) or when ABCI client (like Tendermint) is migrated (any height).

Parameters:	is_synced – True if the chain is known by both ABCI client and server

bigchaindb.backend.query.delete_abci_chain(conn, height)[source]¶: Delete the ABCI chain at the given height.

bigchaindb.backend.query.get_latest_abci_chain(conn)[source]¶: Returns the ABCI chain stored at the biggest height, if any, None otherwise.

bigchaindb.backend.schema¶

Database creation and schema-providing interfaces for backends.

bigchaindb.backend.schema.create_database(connection, dbname)[source]¶

Create database to be used by BigchainDB.

Parameters:	dbname (str) – the name of the database to create.

bigchaindb.backend.schema.create_tables(connection, dbname)[source]¶

Create the tables to be used by BigchainDB.

Parameters:	dbname (str) – the name of the database to create tables for.

bigchaindb.backend.schema.drop_database(connection, dbname)[source]¶

Drop the database used by BigchainDB.

Parameters:	dbname (str) – the name of the database to drop.
Raises:	`DatabaseDoesNotExist` – If the given `dbname` does not exist as a database.

bigchaindb.backend.schema.init_database(connection=None, dbname=None)[source]¶

Initialize the configured backend for use with BigchainDB.

Creates a database with dbname with any required tables and supporting indexes.

Parameters:	connection (`Connection`) – an existing connection to use to initialize the database. Creates one if not given. dbname (str) – the name of the database to create. Defaults to the database name given in the BigchainDB configuration.

bigchaindb.backend.schema.validate_language_key(obj, key)[source]¶

Validate all nested “language” key in obj.

Parameters:	obj (dict) – dictionary whose “language” key is to be validated.
Returns:	validation successful
Return type:	None Raises: ValidationError: will raise exception in case language is not valid.

bigchaindb.backend.schema.validate_language(value)[source]¶

Check if value is a valid language. https://docs.mongodb.com/manual/reference/text-search-languages/

Args:

value (str): language to validated

Returns:

None: validation successful

Raises:

ValidationError: will raise exception in case language is not valid.

bigchaindb.backend.utils¶

exception bigchaindb.backend.utils.ModuleDispatchRegistrationError[source]¶: Raised when there is a problem registering dispatched functions for a module

MongoDB Backend¶

MongoDB backend implementation.

Contains a MongoDB-specific implementation of the schema and query interfaces.

You can specify BigchainDB to use MongoDB as its database backend by either setting database.backend to 'localmongodb' in your configuration file, or setting the BIGCHAINDB_DATABASE_BACKEND environment variable to 'localmongodb'.

MongoDB is the default database backend for BigchainDB.

If configured to use MongoDB, BigchainDB will automatically return instances of LocalMongoDBConnection for connect() and dispatch calls of the generic backend interfaces to the implementations in this module.

bigchaindb.backend.localmongodb.connection¶

class bigchaindb.backend.localmongodb.connection.LocalMongoDBConnection(replicaset=None, ssl=None, login=None, password=None, ca_cert=None, certfile=None, keyfile=None, keyfile_passphrase=None, crlfile=None, **kwargs)[source]¶

collection(name)[source]¶

Return a lazy object that can be used to compose a query.

Parameters:	name (str) – the name of the collection to query.

run(query)[source]¶

Run a query.

Parameters:	query – the query to run
Raises:	`DuplicateKeyError` – If the query fails because of a duplicate key constraint. `OperationFailure` – If the query fails for any other reason. `ConnectionError` – If the connection to the database fails.

bigchaindb.backend.localmongodb.query¶

Query implementation for MongoDB

bigchaindb.backend.localmongodb.schema¶

Utils to initialize and drop the database.

Appendices¶

Generate a Key Pair for SSH¶

This page describes how to use ssh-keygen to generate a public/private RSA key pair that can be used with SSH. (Note: ssh-keygen is found on most Linux and Unix-like operating systems; if you’re using Windows, then you’ll have to use another tool, such as PuTTYgen.)

By convention, SSH key pairs get stored in the ~/.ssh/ directory. Check what keys you already have there:

ls -1 ~/.ssh/

Next, make up a new key pair name (called <name> below). Here are some ideas:

aws-bdb-2
tim-bdb-azure
chris-bcdb-key

Next, generate a public/private RSA key pair with that name:

ssh-keygen -t rsa -C "<name>" -f ~/.ssh/<name>

It will ask you for a passphrase. You can use whatever passphrase you like, but don’t lose it. Two keys (files) will be created in ~/.ssh/:

~/.ssh/<name>.pub is the public key
~/.ssh/<name> is the private key

Cryptography¶

Use the following link to find the BigchainDB Transactions Spec (or Specs) that are relevant to you:

BigchainDB Transactions Specs

Then see the sections titled Cryptographic Hashes and Cryptographic Keys and Signatures.

Notes for Firewall Setup¶

This is a page of notes on the ports potentially used by BigchainDB nodes and the traffic they should expect, to help with firewall setup (or security group setup on cloud providers). This page is not a firewall tutorial or step-by-step guide.

Expected Unsolicited Inbound Traffic¶

The following ports should expect unsolicited inbound traffic:

Port 22 can expect inbound SSH (TCP) traffic from the node administrator (i.e. a small set of IP addresses).
Port 9984 can expect inbound HTTP (TCP) traffic from BigchainDB clients sending transactions to the BigchainDB HTTP API.
Port 9985 can expect inbound WebSocket traffic from BigchainDB clients.
Port 26656 can expect inbound Tendermint P2P traffic from other Tendermint peers.
Port 9986 can expect inbound HTTP (TCP) traffic from clients accessing the Public Key of a Tendermint instance.

All other ports should only get inbound traffic in response to specific requests from inside the node.

Port 22¶

Port 22 is the default SSH port (TCP) so you’ll at least want to make it possible to SSH in from your remote machine(s).

Port 53¶

Port 53 is the default DNS port (UDP). It may be used, for example, by some package managers when look up the IP address associated with certain package sources.

Port 80¶

Port 80 is the default HTTP port (TCP). It’s used by some package managers to get packages. It’s not the default port for the BigchainDB client-server HTTP API.

Port 123¶

Port 123 is the default NTP port (UDP). You should be running an NTP daemon on production BigchainDB nodes. NTP daemons must be able to send requests to external NTP servers and accept the respones.

Port 161¶

Port 161 is the default SNMP port (usually UDP, sometimes TCP). SNMP is used, for example, by some server monitoring systems.

Port 443¶

Port 443 is the default HTTPS port (TCP). Package managers might also get some packages using HTTPS.

Port 9984¶

Port 9984 is the default port for the BigchainDB client-server HTTP API (TCP), which is served by Gunicorn HTTP Server. It’s possible allow port 9984 to accept inbound traffic from anyone, but we recommend against doing that. Instead, set up a reverse proxy server (e.g. using Nginx) and only allow traffic from there. Information about how to do that can be found in the Gunicorn documentation. (They call it a proxy.)

If Gunicorn and the reverse proxy are running on the same server, then you’ll have to tell Gunicorn to listen on some port other than 9984 (so that the reverse proxy can listen on port 9984). You can do that by setting server.bind to ‘localhost:PORT’ in the BigchainDB Configuration Settings, where PORT is whatever port you chose (e.g. 9983).

You may want to have Gunicorn and the reverse proxy running on different servers, so that both can listen on port 9984. That would also help isolate the effects of a denial-of-service attack.

Port 9985¶

Port 9985 is the default port for the BigchainDB WebSocket Event Stream API.

Port 9986¶

Port 9986 is the default port to access the Public Key of a Tendermint instance, it is used by a NGINX instance that runs with Tendermint instance(Pod), and only hosts the Public Key.

Port 26656¶

Port 26656 is the default port used by Tendermint Core to communicate with other instances of Tendermint Core (peers).

Port 26657¶

Port 26657 is the default port used by Tendermint Core for RPC traffic. BigchainDB nodes use that internally; they don’t expect incoming traffic from the outside world on port 26657.

Port 26658¶

Port 26658 is the default port used by Tendermint Core for ABCI traffic. BigchainDB nodes use that internally; they don’t expect incoming traffic from the outside world on port 26658.

Other Ports¶

On Linux, you can use commands such as netstat -tunlp or lsof -i to get a sense of currently open/listening ports and connections, and the associated processes.

Notes on NTP Daemon Setup¶

There are several NTP daemons available, including:

The reference NTP daemon (ntpd) from ntp.org; see their support website
chrony
OpenNTPD
Maybe NTPsec, once it’s production-ready
Maybe Ntimed, once it’s production-ready
More

We suggest you run your NTP daemon in a mode which will tell your OS kernel to handle leap seconds in a particular way: the default NTP way, so that system clock adjustments are localized and not spread out across the minutes, hours, or days surrounding leap seconds (e.g. “slewing” or “smearing”). There’s a nice Red Hat Developer Blog post about the various options.

Use the default mode with ntpd and chronyd. For another NTP daemon, consult its documentation.

It’s tricky to make an NTP daemon setup secure. Always install the latest version and read the documentation about how to configure and run it securely. See the notes on firewall setup.

Amazon Linux Instances¶

If your BigchainDB node is running on an Amazon Linux instance (i.e. a Linux instance packaged by Amazon, not Canonical, Red Hat, or someone else), then an NTP daemon should already be installed and configured. See the EC2 documentation on Setting the Time for Your Linux Instance.

That said, you should check which NTP daemon is installed. Is it recent? Is it configured securely?

The Ubuntu ntp Packages¶

The Ubuntu ntp packages are based on the reference implementation of NTP.

The following commands will uninstall the ntp and ntpdate packages, install the latest ntp package (which might not be based on the latest ntpd code), and start the NTP daemon (a local NTP server). (ntpdate is not reinstalled because it’s deprecated and you shouldn’t use it.)

sudo apt-get --purge remove ntp ntpdate
sudo apt-get autoremove
sudo apt-get update
sudo apt-get install ntp
# That should start the NTP daemon too, but just to be sure:
sudo service ntp restart

You can check if ntpd is running using sudo ntpq -p.

You may want to use different NTP time servers. You can change them by editing the NTP config file /etc/ntp.conf.

Note: A server running an NTP daemon can be used by others for DRDoS amplification attacks. The above installation procedure should install a default NTP configuration file /etc/ntp.conf with the lines:

restrict -4 default kod notrap nomodify nopeer noquery
restrict -6 default kod notrap nomodify nopeer noquery

Those lines should prevent the NTP daemon from being used in an attack. (The first line is for IPv4, the second for IPv6.)

There are additional things you can do to make NTP more secure. See the NTP Support Website for more details.

Logging and Log Rotation¶

Each BigchainDB node runs:

MongoDB
BigchainDB Server
Tendermint

When running a BigchainDB node for long periods of time, we need to consider doing log rotation, i.e. we do not want the logs taking up large amounts of storage and making the node unresponsive or getting it into a bad state.

MongoDB Logging and Log Rotation¶

See the MongoDB docs about logging and log rotation.

BigchainDB Server Logging and Log Rotation¶

BigchainDB Server writes its logs to two files: normal logs and error logs. The names of those files, and their locations, are set as part of the BigchainDB configuration settings. The default names and locations are:

~/bigchaindb.log
~/bigchaindb-errors.log

Log rotation is baked into BigchainDB Server using Python’s logging module. The logs for BigchainDB Server are rotated when any of the above mentioned files exceeds 209715200 bytes (i.e. approximately 209 MB).

For more information, see the docs about the BigchainDB Server configuration settings related to logging.

Tendermint Logging and Log Rotation¶

Tendermint writes its logs to the files:

tendermint.out.log
tendermint.err.log

If you started BigchainDB Server and Tendermint using Monit, as suggested by our guide on How to Set Up a BigchainDB Network, then the logs will be written to $HOME/.bigchaindb-monit/logs/.

Moreover, if you started BigchainDB Server and Tendermint using Monit, then Monit monitors the Tendermint log files. Tendermint logs are rotated if any of the above mentioned log files exceeds 200 MB.

Licenses¶

Information about how the BigchainDB Server code and documentation are licensed can be found in the LICENSES.md file of the bigchaindb/bigchaindb repository on GitHub.

Drivers¶

Connectors to BigchainDB are referred to as drivers within the community. A driver is used to create valid transactions, to generate key pairs, to sign transactions and to post the transaction to the BigchainDB API.

These drivers were originally created by the original BigchainDB team:

These drivers and tools were created by the BigchainDB community:

Warning

Some of these projects are a work in progress, but may still be useful. Others might not work with the latest version of BigchainDB.

ANSI C driver, should also work with C++ (working as of June 2019)
C# driver (working as of May 2019)
Haskell transaction builder
Go driver
Ruby driver
Ruby library for preparing/signing transactions and submitting them or querying a BigchainDB node (MIT licensed)

Queries in BigchainDB¶

A node operator can use the full power of MongoDB’s query engine to search and query all stored data, including all transactions, assets and metadata. The node operator can decide for themselves how much of that query power they expose to external users.

Blog Post with Example Queries¶

We wrote a blog post in The BigchainDB Blog to show how to use some MongoDB tools to query a BigchainDB node’s MongoDB database. It includes some specific example queries for data about custom cars and their ownership histories. Check it out.

How to Connect to MongoDB¶

Before you can query a MongoDB database, you must connect to it, and to do that, you need to know its hostname and port.

If you’re running a BigchainDB node on your local machine (e.g. for dev and test), then the hostname should be localhost and the port should be 27017, unless you did something to change those values. If you’re running a BigchainDB node on a remote machine and you can SSH to that machine, then the same is true.

If you’re running a BigchainDB node on a remote machine and you configured its MongoDB to use auth and to be publicly-accessible (to people with authorization), then you can probably figure out its hostname and port.

How to Query¶

A BigchainDB node operator has full access to their local MongoDB instance, so they can use any of MongoDB’s APIs for running queries, including:

the Mongo Shell,
MongoDB Compass,
one of the MongoDB drivers, such as PyMongo, or
a third-party tool for doing MongoDB queries, such as RazorSQL, Studio 3T, Mongo Management Studio, NoSQLBooster for MongoDB, or Dr. Mongo.

Note

It’s possible to do query a MongoDB database using SQL. For example:

Studio 3T: “How to Query MongoDB with SQL”
NoSQLBooster for MongoDB: “How to Query MongoDB with SQL SELECT”

For example, if you’re on a machine that’s running a default BigchainDB node, then you can connect to it using the Mongo Shell (mongo) and look around like so:

$ mongo
MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.4
...
> show dbs
admin     0.000GB
bigchain  0.000GB
config    0.000GB
local     0.000GB
> use bigchain
switched to db bigchain
> show collections
abci_chains
assets
blocks
elections
metadata
pre_commit
transactions
utxos
validators

The above example illustrates several things:

When you don’t specify the hostname or port, the Mongo Shell assumes they are localhost and 27017, respectively. (localhost had IP address 127.0.0.1 on the machine in question, an Ubuntu machine.)
BigchainDB stores its data in a database named bigchain.
The bigchain database contains several collections.
Votes aren’t stored in any collection, currently. They are all handled and stored by Tendermint in its own (LevelDB) database.

Example Documents from Some Collections¶

The most interesting collections in the bigchain database are:

transactions
assets
metadata
blocks

You can explore those collections using MongoDB queries such as db.assets.findOne(). We now show some example documents from each of those collections.

Example Documents from transactions¶

A CREATE transaction from the transactions collection includes an extra "_id" field (added by MongoDB) and is missing its "asset" and "metadata" fields: that data was removed and stored in the assets and metadata collections.

{
    "_id":ObjectId("5b17b9fa6ce88300067b6804"),
    "inputs":[…],
    "outputs":[…],
    "operation":"CREATE",
    "version":"2.0",
    "id":"816c4dd7…851af1629"
}

A TRANSFER transaction from the transactions collection is similar, but it keeps its "asset" field.

{
    "_id":ObjectId("5b17b9fa6ce88300067b6807"),
    "inputs":[…],
    "outputs":[…],
    "operation":"TRANSFER",
    "asset":{
        "id":"816c4dd7ae…51af1629"
    },
    "version":"2.0",
    "id":"985ee697d…a3296b9"
}

Example Document from assets¶

A document from the assets collection has three top-level fields: an "_id" field added by MongoDB, the asset.data from a CREATE transaction, and the "id" of the CREATE transaction it came from.

{
    "_id":ObjectId("5b17b9fe6ce88300067b6823"),
    "data":{
        "type":"cow",
        "name":"Mildred"
    },
    "id":"96002ef8740…45869959d8"
}

Example Document from metadata¶

A document from the metadata collection has three top-level fields: an "_id" field added by MongoDB, the metadata from a transaction, and the "id" of the transaction it came from.

{
    "_id":ObjectId("5b17ba006ce88300067b683d"),
    "metadata":{
        "transfer_time":1058568256
    },
    "id":"53cba620e…ae9fdee0"
}

Example Document from blocks¶

{
    "_id":ObjectId("5b212c1ceaaa420006f41c57"),
    "app_hash":"2b0b75c2c2…7fb2652ce26c6",
    "height":17,
    "transactions":[
        "5f1f2d6b…ed98c1e"
    ]
}

What a Node Operator Can Expose to External Users¶

Each node operator can decide how they let external users get information from their local MongoDB database. They could expose:

their local MonogoDB database itself to queries from external users, maybe as a MongoDB user with a role that has limited privileges, e.g. read-only.
a limited HTTP API, allowing a restricted set of predefined queries, such as the HTTP API provided by BigchainDB Server, or a custom HTTP API implemented using Django, Express, Ruby on Rails, or ASP.NET.
some other API, such as a GraphQL API. They could do that using custom code or code from a third party.

Each node operator can expose a different level or type of access to their local MongoDB database. For example, one node operator might decide to specialize in offering optimized geospatial queries.

Security Considerations¶

In BigchainDB version 1.3.0 and earlier, there was one logical MongoDB database, so exposing that database to external users was very risky, and was not recommended. “Drop database” would delete that one shared MongoDB database.

In BigchainDB version 2.0.0 and later, each node has its own isolated local MongoDB database. Inter-node communications are done using Tendermint protocols, not MongoDB protocols, as illustrated in Figure 1 below. If a node’s local MongoDB database gets compromised, none of the other MongoDB databases (in the other nodes) will be affected.

Diagram of a four-node BigchainDB 2.0 network

Figure 1: A Four-Node BigchainDB 2.0 Network

Performance and Cost Considerations¶

Query processing can be quite resource-intensive, so it’s a good idea to have MongoDB running in a separate machine from those running BigchainDB Server and Tendermint Core.

A node operator might want to measure the resources used by a query, so they can charge whoever requested the query accordingly.

Some queries can take too long or use too many resources. A node operator should put upper bounds on the resources that a query can use, and halt (or prevent) any query that goes over.

To make MongoDB queries more efficient, one can create indexes. Those indexes might be created by the node operator or by some external users (if the node operator allows that). It’s worth noting that indexes aren’t free: whenever new data is appended to a collection, the corresponding indexes must be updated. The node operator might want to pass those costs on to whoever created the index. Moreover, in MongoDB, a single collection can have no more than 64 indexes.

One can create a follower node: a node with Tendermint voting power 0. It would still have a copy of all the data, so it could be used as read-only node. A follower node could offer specialized queries as a service without affecting the workload on the voting validators (which can also write). There could even be followers of followers.

JavaScript Query Code Examples¶

One can connect to a node’s MongoDB database using any of the MongoDB drivers, such as the MongoDB Node.js driver. Here are some links to example JavaScript code that queries a BigchainDB node’s MongoDB database:

Contributing to BigchainDB¶

There are many ways you can contribute to BigchainDB. It includes several sub-projects.

BigchainDB Server
BigchainDB Python Driver
BigchainDB JavaScript Driver
BigchainDB Java Driver
cryptoconditions (a Python package by us)
py-abci (a Python package we use)
BigchainDB Enhancement Proposals (BEPs)

Contents¶

Ways to Contribute¶

How to contribute?¶

Report a Bug¶

To report a bug, go to the relevant GitHub repository, click on the Issues tab, click on the New issue button, and read the instructions.

Write an Issue¶

To write an issue, go to the relevant GitHub repository, click on the Issues tab, click on the New issue button, and read the instructions.

Answer Questions¶

People ask questions about BigchainDB in the following places:

Feel free to hang out and answer some questions. People will be thankful.

Write a BigchainDB Enhancement Proposal (BEP)¶

If you have an idea for a new feature or enhancement, and you want some feedback before you write a full BigchainDB Enhancement Proposal (BEP), then feel free to:

ask in the bigchaindb/bigchaindb Gitter chat room or
open a new issue in the bigchaindb/BEPs repo and give it the label BEP idea.

If you want to discuss an existing BEP, then open a new issue in the bigchaindb/BEPs repo and give it the label discuss existing BEP.

Steps to Write a New BEP¶

Look at the structure of existing BEPs in the bigchaindb/BEPs repo. Note the section headings. BEP-2 (our variant of the consensus-oriented specification system [COSS]) says more about the expected structure and process.
Write a first draft of your BEP. It doesn’t have to be long or perfect.
Push your BEP draft to the bigchaindb/BEPs repo and make a pull request. BEP-1 (our variant of C4) outlines the process we use to handle all pull requests. In particular, we try to merge all pull requests quickly.
Your BEP can be revised by pushing more pull requests.

Write Docs¶

If you’re writing code, you should also update any related docs. However, you might want to write docs only, such as:

General explainers
Tutorials
Courses
Code explanations
How BigchainDB relates to other blockchain things
News from recent events

You can certainly do that!

The docs for BigchainDB Server live under bigchaindb/docs/ in the bigchaindb/bigchaindb repo.
There are docs for the Python driver under bigchaindb-driver/docs/ in the bigchaindb/bigchaindb-driver repo.
There are docs for the JavaScript driver under bigchaindb/js-bigchaindb-driver in the bigchaindb/js-bigchaindb-driver repo.
The source code for the BigchainDB website is in a private repo, but we can give you access if you ask.

The BigchainDB Transactions Specs (one for each spec version) are in the bigchaindb/BEPs repo.

You can write the docs using Markdown (MD) or RestructuredText (RST). Sphinx can understand both. RST is more powerful.

ReadTheDocs will automatically rebuild the docs whenever a commit happens on the master branch, or on one of the other branches that it is monitoring.

Developer Setup, Coding & Contribution Process¶

Write Code¶

Know What You Want to Write Code to Do¶

Do you want to write code to resolve an open issue (bug)? Which one?

Do you want to implement a BigchainDB Enhancement Proposal (BEP)? Which one?

You should know why you want to write code before you go any farther.

Refresh Yourself about the C4 Process¶

C4 is the Collective Code Construction Contract. It’s quite short: re-reading it will only take a few minutes.

Set Up Your Local Machine. Here’s How.¶

Make sure you have Git installed.
Get a text editor. Internally, we like:
- Vim
- PyCharm
- Visual Studio Code
- Atom
- GNU Emacs (Trent is crazy)
- GNU nano (Troy has lost his mind)
If you plan to do JavaScript coding, get the latest JavaScript stuff (npm etc.).
If you plan to do Python coding, get the latest Python, and get the latest pip.

Warning

Don’t use apt or apt-get to get the latest pip. It won’t work properly. Use get-pip.py from the pip website.

Use the latest pip to get the latest virtualenv:
```
$ pip install virtualenv
```
Create a Python Virttual Environment (virtualenv) for doing BigchainDB Server development. There are many ways to do that. Google around and pick one. An old-fashioned but fine way is:
```
$ virtualenv -p $(which python3.6) NEW_ENV_NAME
$ . NEW_ENV_NAME/bin/activate
```
Be sure to use Python 3.6.x as the Python version for your virtualenv. The virtualenv creation process will actually get the latest pip, wheel and setuptools and put them inside the new virtualenv.

Before You Start Writing Code¶

Read BEP-24 so you know what to do to ensure that your changes (i.e. your future pull request) can be merged. It’s easy and will save you some hassle later on.

Start Writing Code¶

Use the Git Fork and Pull Request Workflow. Tip: You could print that page for reference.

Your Python code should follow our Python Style Guide. Similarly for JavaScript.

Make sure pre-commit actually checks commits. Do:

$ pip install pre-commit  # might not do anything if it is already installed, which is okay
$ pre-commit install

That will load the pre-commit settings in the file .pre-commit-config.yaml. Now every time you do git commit <stuff>, pre-commit will run all those checks.

To install BigchainDB Server from the local code, and to keep it up to date with the latest code in there, use:

$ pip install -e .[dev]

The -e tells it to use the latest code. The . means use the current directory, which should be the one containing setup.py. The [dev] tells it to install some extra Python packages. Which ones? Open setup.py and look for dev in the extras_require section.

Remember to Write Tests¶

We like to test everything, if possible. Unit tests and also integration tests. We use the pytest framework to write Python tests. Read all about it.

Most tests are in the tests/ folder. Take a look around.

Running a Local Node/Network for Dev and Test¶

This is tricky and personal. Different people do it different ways. We documented some of those on separate pages:

Create the PR on GitHub¶

Git push your branch to GitHub so as to create a pull request against the branch where the code you want to change lives.

Travis and Codecov will run and might complain. Look into the complaints and fix them before merging. Travis gets its configuration and setup from the files:

Some environment variables, if they are used. See https://docs.travis-ci.com/user/environment-variables/
.travis.yml
tox.ini - What is tox? See tox.readthedocs.io
.ci/ (as in Travis CI = Continuous Integration)
- travis-after-success.sh
- travis-before-install.sh
- travis-before-script.sh
- travis-install.sh
- travis_script.sh

Read about the Travis CI build lifecycle to understand when those scripts run and what they do. You can have even more scripts!

Codecov gets its configuration from the file codeocov.yaml which is also documented at docs.codecov.io. Codecov might also use setup.cfg.

Merge!¶

Ideally, we like your PR and merge it right away. We don’t want to keep you waiting.

If we want to make changes, we’ll do them in a follow-up PR.

Notes on Running a Local Dev Node with Docker Compose¶

Setting up a single node development environment with docker-compose¶

Using the BigchainDB 2.0 developer toolbox¶

We grouped all useful commands under a simple Makefile.

Run a BigchainDB node in the foreground:

$ make run

There are also other commands you can execute:

make start: Run BigchainDB from source and daemonize it (stop it with make stop).
make stop: Stop BigchainDB.
make logs: Attach to the logs.
make test: Run all unit and acceptance tests.
make test-unit-watch: Run all tests and wait. Every time you change code, tests will be run again.
make cov: Check code coverage and open the result in the browser.
make doc: Generate HTML documentation and open it in the browser.
make clean: Remove all build, test, coverage and Python artifacts.
make reset: Stop and REMOVE all containers. WARNING: you will LOSE all data stored in BigchainDB.

Using docker-compose directly¶

The BigchainDB Makefile is a wrapper around some docker-compose commands we use frequently. If you need a finer granularity to manage the containers, you can still use docker-compose directly. This part of the documentation explains how to do that.

$ docker-compose build bigchaindb
$ docker-compose up -d bdb

The above command will launch all 3 main required services/processes:

mongodb
tendermint
bigchaindb

To follow the logs of the tendermint service:

$ docker-compose logs -f tendermint

To follow the logs of the bigchaindb service:

$ docker-compose logs -f bigchaindb

To follow the logs of the mongodb service:

$ docker-compose logs -f mdb

Simple health check:

$ docker-compose up curl-client

Post and retrieve a transaction – copy/paste a driver basic example of a CREATE transaction:

$ docker-compose -f docker-compose.yml run --rm bdb-driver ipython

TODO: A python script to post and retrieve a transaction(s).

Running Tests¶

Run all the tests using:

$ docker-compose run --rm --no-deps bigchaindb pytest -v

Run tests from a file:

$ docker-compose run --rm --no-deps bigchaindb pytest /path/to/file -v

Run specific tests:

$ docker-compose run --rm --no-deps bigchaindb pytest /path/to/file -k "<test_name>" -v

Building Docs¶

You can also develop and build the BigchainDB docs using docker-compose:

$ docker-compose build bdocs
$ docker-compose up -d bdocs

The docs will be hosted on port 33333, and can be accessed over localhost, 127.0.0.1 OR http:/HOST_IP:33333.

Notes on Running a Local Dev Node as Processes¶

The following doc describes how to run a local node for developing BigchainDB Tendermint version.

There are two crucial dependencies required to start a local node:

MongoDB
Tendermint

and of course you also need to install BigchainDB Sever from the local code you just developed.

Install and Run MongoDB¶

MongoDB can be easily installed, just refer to their installation documentation for your distro. We know MongoDB 3.4 and 3.6 work with BigchainDB. After the installation of MongoDB is complete, run MongoDB using sudo mongod

Install and Run Tendermint¶

Installing a Tendermint Executable¶

The version of BigchainDB Server described in these docs only works well with Tendermint 0.31.5 (not a higher version number). Install that:

$ sudo apt install -y unzip
$ wget https://github.com/tendermint/tendermint/releases/download/v0.31.5/tendermint_v0.31.5_linux_amd64.zip
$ unzip tendermint_v0.31.5_linux_amd64.zip
$ rm tendermint_v0.31.5_linux_amd64.zip
$ sudo mv tendermint /usr/local/bin

Installing Tendermint Using Docker¶

Tendermint can be run directly using the docker image. Refer here for more details.

Installing Tendermint from Source¶

Before we can begin installing Tendermint one should ensure that the Golang is installed on system and $GOPATH should be set in the .bashrc or .zshrc. An example setup is shown below

$ echo $GOPATH
/home/user/Documents/go
$ go -h
Go is a tool for managing Go source code.

Usage:

	go command [arguments]

The commands are:

	build       compile packages and dependencies
	clean       remove object files

...

We can drop GOPATH in PATH so that installed Golang packages are directly available in the shell. To do that add the following to your .bashrc

export PATH=${PATH}:${GOPATH}/bin

Follow the Tendermint docs to install Tendermint from source.

If the installation is successful then Tendermint is installed at $GOPATH/bin. To ensure Tendermint’s installed fine execute the following command,

$ tendermint -h
Tendermint Core (BFT Consensus) in Go

Usage:
  tendermint [command]

Available Commands:
  gen_validator               Generate new validator keypair
  help                        Help about any command
  init                        Initialize Tendermint
...

Running Tendermint¶

We can initialize and run tendermint as follows,

$ tendermint init
...

$ tendermint node --consensus.create_empty_blocks=false

The argument --consensus.create_empty_blocks=false specifies that Tendermint should not commit empty blocks.

To reset all the data stored in Tendermint execute the following command,

$ tendermint unsafe_reset_all

Install BigchainDB¶

To install BigchainDB from source (for dev), clone the repo and execute the following command, (it is better that you create a virtual env for this)

$ git clone https://github.com/bigchaindb/bigchaindb.git
$ cd bigchaindb
$ pip install -e .[dev]  #  or  pip install -e '.[dev]'  # for zsh

Running All Tests¶

To execute tests when developing a feature or fixing a bug one could use the following command,

$ pytest -v

NOTE: MongoDB and Tendermint should be running as discussed above.

One could mark a specific test and execute the same by appending -m my_mark to the above command.

Although the above should prove sufficient in most cases but in case tests are failing on Travis CI then the following command can be used to possibly replicate the failure locally,

$ docker-compose run --rm --no-deps bdb pytest -v --cov=bigchaindb

NOTE: before executing the above command the user must ensure that they reset the Tendermint container by executing tendermint usafe_reset_all command in the Tendermint container.

Run a BigchainDB network¶

NOT for Production Use

You can use the following instructions to deploy a single or multi node BigchainDB network for dev/test using the extensible stack script(s).

Currently, this workflow is only supported for the following Operating systems:

Ubuntu >= 16.04
CentOS >= 7
Fedora >= 24
MacOSX

Machine Minimum Requirements¶

Minimum resource requirements for a single node BigchainDB dev setup. The more the better:

Memory >= 512MB
VCPUs >= 1

Download the scripts¶

Note: If you’re working on BigchainDB Server code, on a branch based on recent code, then you already have local recent versions of stack.sh and unstack.sh in your bigchaindb/pkg/scripts/ directory. Otherwise you can get them using:

$ export GIT_BRANCH=master
$ curl -fOL https://raw.githubusercontent.com/bigchaindb/bigchaindb/${GIT_BRANCH}/pkg/scripts/stack.sh

# Optional
$ curl -fOL https://raw.githubusercontent.com/bigchaindb/bigchaindb/${GIT_BRANCH}/pkg/scripts/unstack.sh

Quick Start¶

If you run stack.sh out of the box i.e. without any configuration changes, you will be able to deploy a 4 node BigchainDB network with Docker containers, created from master branch of bigchaindb/bigchaindb repo and Tendermint version 0.22.8.

Note: Run stack.sh with either root or non-root user with sudo enabled.

$ bash stack.sh
...Logs..
.........
.........
Finished stacking!

Configure the BigchainDB network¶

The stack.sh script has multiple deployment methods and parameters and they can be explored using: bash stack.sh -h

$ bash stack.sh -h

    Usage: $ bash stack.sh [-h]

    Deploys the BigchainDB network.

    ENV[STACK_SIZE]
        Set STACK_SIZE environment variable to the size of the network you desire.
        Network mimics a production network environment with single or multiple BDB
        nodes. (default: 4).

    ENV[STACK_TYPE]
        Set STACK_TYPE environment variable to the type of deployment you desire.
        You can set it one of the following: ["docker", "local", "cloud"].
        (default: docker)

    ENV[STACK_TYPE_PROVIDER]
        Set only when STACK_TYPE="cloud". Only "azure" is supported.
        (default: )

    ENV[STACK_VM_MEMORY]
        (Optional) Set only when STACK_TYPE="local". This sets the memory
        of the instance(s) spawned. (default: 2048)

    ENV[STACK_VM_CPUS]
        (Optional) Set only when STACK_TYPE="local". This sets the number of VCPUs
        of the instance(s) spawned. (default: 2)

    ENV[STACK_BOX_NAME]
        (Optional) Set only when STACK_TYPE="local". This sets the box Vagrant box name
        of the instance(s) spawned. (default: ubuntu/xenial64)

    ENV[STACK_REPO]
        (Optional) To configure bigchaindb repo to use, set STACK_REPO environment
        variable. (default: bigchaindb/bigchaindb)

    ENV[STACK_BRANCH]
        (Optional) To configure bigchaindb repo branch to use set STACK_BRANCH environment
        variable. (default: master)

    ENV[TM_VERSION]
        (Optional) Tendermint version to use for the setup. (default: 0.22.8)

    ENV[MONGO_VERSION]
        (Optional) MongoDB version to use with the setup. (default: 3.6)

    ENV[AZURE_CLIENT_ID]
        Only required when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure". Steps to generate:
        https://github.com/Azure/vagrant-azure#create-an-azure-active-directory-aad-application

    ENV[AZURE_TENANT_ID]
        Only required when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure". Steps to generate:
        https://github.com/Azure/vagrant-azure#create-an-azure-active-directory-aad-application

    ENV[AZURE_SUBSCRIPTION_ID]
        Only required when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure". Steps to generate:
        https://github.com/Azure/vagrant-azure#create-an-azure-active-directory-aad-application

    ENV[AZURE_CLIENT_SECRET]
        Only required when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure". Steps to generate:
        https://github.com/Azure/vagrant-azure#create-an-azure-active-directory-aad-application

    ENV[AZURE_REGION]
        (Optional) Only applicable, when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure".
        Azure region for the BigchainDB instance. Get list of regions using Azure CLI.
        e.g. az account list-locations. (default: westeurope)

    ENV[AZURE_IMAGE_URN]
        (Optional) Only applicable, when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure".
        Azure image to use. Get list of available images using Azure CLI.
        e.g. az vm image list --output table. (default: Canonical:UbuntuServer:16.04-LTS:latest)

    ENV[AZURE_RESOURCE_GROUP]
        (Optional) Only applicable, when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure".
        Name of Azure resource group for the instance.
        (default: bdb-vagrant-rg-2018-05-30)

    ENV[AZURE_DNS_PREFIX]
        (Optional) Only applicable, when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure".
        DNS Prefix of the instance. (default: bdb-instance-2018-05-30)

    ENV[AZURE_ADMIN_USERNAME]
        (Optional) Only applicable, when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure".
        Admin username of the the instance. (default: vagrant)

    ENV[AZURE_VM_SIZE]
        (Optional) Only applicable, when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure".
        Azure VM size. (default: Standard_D2_v2)

    ENV[SSH_PRIVATE_KEY_PATH]
        Only required when STACK_TYPE="cloud" and STACK_TYPE_PROVIDER="azure". Absolute path of
        SSH keypair required to log into the Azure instance.

    -h
        Show this help and exit.

The parameter that differentiates between the deployment type is STACK_TYPE which currently, supports an opinionated deployment of BigchainDB on docker, local and cloud.

STACK_TYPE: docker¶

This configuration deploys a docker based BigchainDB network on the dev/test machine that you are running stack.sh on. This is also the default STACK_TYPE config for stack.sh.

Example: docker¶

Deploy a 4 node docker based BigchainDB network on your host.

#Optional, since 4 is the default size.
$ export STACK_SIZE=4

#Optional, since docker is the default type.
$ export STACK_TYPE=docker

#Optional, repo to use for the network deployment
# Default: bigchaindb/bigchaindb
$ export STACK_REPO=bigchaindb/bigchaindb

#Optional, codebase to use for the network deployment
# Default: master
$ export STACK_BRANCH=master

#Optional, since 0.22.8 is the default tendermint version.
$ export TM_VERSION=0.22.8

#Optional, since 3.6 is the default MongoDB version.
$ export MONGO_VERSION=3.6

$ bash stack.sh

Note: For MacOSX users, the script will not install Docker for Mac, it only detects if docker is installed on the system, if not make sure to install Docker for Mac. Also make sure Docker API Version > 1.25. To check Docker API Version:

docker version --format '{{.Server.APIVersion}}'

STACK_TYPE: local¶

This configuration deploys a VM based BigchainDB network on your host/dev. All the services are running as processes on the VMs. For local deployments the following dependencies must be installed i.e.

Vagrant
- Vagrant plugins.
  - vagrant-cachier
  - vagrant-vbguest
  - vagrant-hosts
  - vagrant-azure
    - vagrant plugin install vagrant-cachier vagrant-vbguest vagrant-hosts vagrant-azure
Virtualbox

Example: VM¶

Deploy a 4 node VM based BigchainDB network.

$ export STACK_TYPE=local

# Optional, since 4 is the default size.
$ export STACK_SIZE=4

# Optional, default is 2048
$ export STACK_VM_MEMORY=2048 

#Optional, default is 1
$ export STACK_VM_CPUS=1

#Optional, default is ubuntu/xenial64. Supported/tested images: bento/centos-7, fedora/25-cloud-base
$ export STACK_BOX_NAME=ubuntu/xenial64

#Optional, repo to use for the network deployment
# Default: bigchaindb/bigchaindb
$ export STACK_REPO=bigchaindb/bigchaindb

#Optional, codebase to use for the network deployment
# Default: master
$ export STACK_BRANCH=master

#Optional, since 0.22.8 is the default tendermint version
$ export TM_VERSION=0.22.8

#Optional, since 3.6 is the default MongoDB version.
$ export MONGO_VERSION=3.6

$ bash stack.sh

STACK_TYPE: cloud¶

This configuration deploys a docker based BigchainDB network on a cloud instance. Currently, only Azure is supported. For cloud deployments the following dependencies must be installed i.e.

Vagrant
- Vagrant plugins.
  - vagrant-cachier
  - vagrant-vbguest
  - vagrant-hosts
  - vagrant-azure
    - vagrant plugin install vagrant-cachier vagrant-vbguest vagrant-hosts vagrant-azure

Example: stack¶

Deploy a 4 node docker based BigchainDB network on an Azure instance.

Create an Azure Active Directory(AAD) Application
Generate or specify an SSH keypair to login to the Azure instance.
- Example:
```
$ ssh-keygen -t rsa -C "<name>" -f /path/to/key/<name>
```
Configure parameters for stack.sh

# After creating the AAD application with access to Azure Resource
# Group Manager for your subscription, it will return a JSON object

$ export AZURE_CLIENT_ID=<value from azure.appId>

$ export AZURE_TENANT_ID=<value from azure.tenant>

# Can be retrieved via
# az account list --query "[?isDefault].id" -o tsv
$ export AZURE_SUBSCRIPTION_ID=<your Azure subscription ID>

$ export AZURE_CLIENT_SECRET=<value from azure.password>

$ export STACK_TYPE=cloud

# Currently on azure is supported
$ export STACK_TYPE_PROVIDER=azure

$ export SSH_PRIVATE_KEY_PATH=</path/to/private/key>

# Optional, Azure region of the instance. Default: westeurope
$ export AZURE_REGION=westeurope

# Optional, Azure image urn of the instance. Default: Canonical:UbuntuServer:16.04-LTS:latest
$ export AZURE_IMAGE_URN=Canonical:UbuntuServer:16.04-LTS:latest

# Optional, Azure resource group. Default: bdb-vagrant-rg-yyyy-mm-dd(current date)
$ export AZURE_RESOURCE_GROUP=bdb-vagrant-rg-2018-01-01

# Optional, DNS prefix of the Azure instance. Default: bdb-instance-yyyy-mm-dd(current date)
$ export AZURE_DNS_PREFIX=bdb-instance-2018-01-01

# Optional, Admin username of the Azure instance. Default: vagrant
$ export AZURE_ADMIN_USERNAME=vagrant

# Optional, Azure instance size. Default: Standard_D2_v2
$ export AZURE_VM_SIZE=Standard_D2_v2

$ bash stack.sh

Delete/Unstack a BigchainDB network¶

Export all the variables exported for the corresponding stack.sh script and run unstack.sh to delete/remove/unstack the BigchainDB network/stack.

$ bash unstack.sh

OR

# -s implies soft unstack. i.e. Only applicable for local and cloud based
# networks. Only deletes/stops the docker(s)/process(es) and does not
# delete the instances created via Vagrant or on Cloud. Default: hard
$ bash unstack.sh -s

Run a BigchainDB network with Ansible¶

NOT for Production Use

You can use the following instructions to deploy a single or multi node BigchainDB network for dev/test using Ansible. Ansible will configure the BigchainDB node(s).

Currently, this workflow is only supported for the following distributions:

Ubuntu >= 16.04
CentOS >= 7
Fedora >= 24
MacOSX

Minimum Requirements¶

Minimum resource requirements for a single node BigchainDB dev setup. The more the better:

Memory >= 512MB
VCPUs >= 1

Clone the BigchainDB repository¶

$ git clone https://github.com/bigchaindb/bigchaindb.git

Install dependencies¶

Ansible

You can also install ansible and other dependencies, if any, using the boostrap.sh script inside the BigchainDB repository. Navigate to bigchaindb/pkg/scripts and run the bootstrap.sh script to install the dependencies for your OS. The script also checks if the OS you are running is compatible with the supported versions.

Note: bootstrap.sh only supports Ubuntu >= 16.04, CentOS >= 7 and Fedora >=24 and MacOSX.

$ cd bigchaindb/pkg/scripts/
$ bash bootstrap.sh --operation install

BigchainDB Setup Configuration(s)¶

Local Setup¶

You can run the Ansible playbook bigchaindb-start.yml on your local dev machine and set up the BigchainDB node where BigchainDB can be run as a process or inside a Docker container(s) depending on your configuration.

Before, running the playbook locally, you need to update the hosts and stack-config.yml configuration, which will notify Ansible that we need to run the play locally.

Update Hosts¶

Navigate to bigchaindb/pkg/configuration/hosts inside the BigchainDB repository.

$ cd bigchaindb/pkg/configuration/hosts

Edit all configuration file:

# Delete any existing configuration in this file and insert
# Hostname of dev machine
<HOSTNAME> ansible_connection=local

Update Configuration¶

Navigate to bigchaindb/pkg/configuration/vars inside the BigchainDB repository.

$ cd bigchaindb/pkg/configuration/vars/stack-config.yml

Edit bdb-config.yml configuration file as per your requirements, sample configuration file(s):

---
stack_type: "docker" 
stack_size: "4"


OR

---
stack_type: "local"
stack_type: "1"

BigchainDB Setup¶

Now, You can safely run the bigchaindb-start.yml playbook and everything will be taken care of by Ansible. To run the playbook please navigate to the bigchaindb/pkg/configuration directory inside the BigchainDB repository and run the bigchaindb-start.yml playbook.

$ cd bigchaindb/pkg/configuration/

$ ansible-playbook bigchaindb-start.yml -i hosts/all --extra-vars "operation=start home_path=$(pwd)"

After successful execution of the playbook, you can verify that BigchainDB docker(s)/process(es) is(are) running.

Verify BigchainDB process(es):

$ ps -ef | grep bigchaindb

OR

Verify BigchainDB Docker(s):

$ docker ps | grep bigchaindb

You can now send transactions and verify the functionality of your BigchainDB node. See the BigchainDB Python Driver documentation for details on how to use it.

Note: The bdb_root_url can be be one of the following:

# BigchainDB is running as a process
bdb_root_url = http://<HOST-IP>:9984

OR

# BigchainDB is running inside a docker container
bdb_root_url = http://<HOST-IP>:<DOCKER-PUBLISHED-PORT>

Note: BigchainDB has other drivers as well.

Experimental: Running Ansible a Remote Dev/Host¶

Remote Setup¶

You can also run the Ansible playbook bigchaindb-start.yml on remote machine(s) and set up the BigchainDB node where BigchainDB can run as a process or inside a Docker container(s) depending on your configuration.

Before, running the playbook on a remote host, you need to update the hosts and stack-config.yml configuration, which will notify Ansible that we need to run the play on a remote host.

Update Remote Hosts¶

Navigate to bigchaindb/pkg/configuration/hosts inside the BigchainDB repository.

$ cd bigchaindb/pkg/configuration/hosts

Edit all configuration file:

# Delete any existing configuration in this file and insert
<Remote_Host_IP/Hostname> ansible_ssh_user=<USERNAME> ansible_sudo_pass=<PASSWORD>

Note: You can add multiple hosts to the all configuration file. Non-root user with sudo enabled password is needed because ansible will run some tasks that require those permissions.

Note: You can also use other methods to get inside the remote machines instead of password based SSH. For other methods please consult Ansible Documentation.

Update Remote Configuration¶

Navigate to bigchaindb/pkg/configuration/vars inside the BigchainDB repository.

$ cd bigchaindb/pkg/configuration/vars/stack-config.yml

Edit stack-config.yml configuration file as per your requirements, sample configuration file(s):

---
stack_type: "docker" 
stack_size: "4"


OR

---
stack_type: "local"
stack_type: "1"

After, the configuration of remote hosts, run the Ansible playbook and verify your deployment.

Policies¶

Contributor Code of Conduct¶

As contributors and maintainers of this project, and in the interest of fostering an open and welcoming community, we pledge to respect all people who contribute to the project.

We are committed to making participation in this project a harassment-free experience for everyone, regardless of level of experience, gender, gender identity and expression, sexual orientation, disability, personal appearance, body size, race, ethnicity, age, religion, nationality, or species–no picking on Wrigley for being a buffalo!

Examples of unacceptable behavior by participants include:

The use of sexualized language or imagery
Personal attacks
Trolling or insulting/derogatory comments
Public or private harassment
Publishing other’s private information, such as physical or electronic addresses, without explicit permission
Deliberate intimidation
Other unethical or unprofessional conduct

Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.

By adopting this Code of Conduct, project maintainers commit themselves to fairly and consistently applying these principles to every aspect of managing this project. Project maintainers who do not follow or enforce the Code of Conduct may be permanently removed from the project team.

This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community.

Instances of abusive, harassing, or otherwise unacceptable behavior directed at yourself or another community member may be reported by contacting a project maintainer at contact@bigchaindb.com. All complaints will be reviewed and investigated and will result in a response that is appropriate to the circumstances. Maintainers are obligated to maintain confidentiality with regard to the reporter of an incident.

This Code of Conduct is adapted from the Contributor Covenant, version 1.3.0, available at http://contributor-covenant.org/version/1/3/0/

Python Style Guide¶

This guide starts out with our general Python coding style guidelines and ends with a section on how we write & run (Python) tests.

General Python Coding Style Guidelines¶

Our starting point is PEP8, the standard “Style Guide for Python Code.” Many Python IDEs will check your code against PEP8. (Note that PEP8 isn’t frozen; it actually changes over time, but slowly.)

BigchainDB uses Python 3.5+, so you can ignore all PEP8 guidelines specific to Python 2.

We use pre-commit to check some of the rules below before every commit but not everything is realized yet. The hooks we use can be found in the .pre-commit-config.yaml file.

Python Docstrings¶

PEP8 says some things about docstrings, but not what to put in them or how to structure them. PEP257 was one proposal for docstring conventions, but we prefer Google-style docstrings instead: they’re easier to read and the napoleon extension for Sphinx lets us turn them into nice-looking documentation. Here are some references on Google-style docstrings:

Google’s docs on Google-style docstrings
napoleon’s docs include an overview of Google-style docstrings
Example Google-style docstrings (from napoleon’s docs)

Maximum Line Length¶

PEP8 has some maximum line length guidelines, starting with “Limit all lines to a maximum of 79 characters” but “for flowing long blocks of text with fewer structural restrictions (docstrings or comments), the line length should be limited to 72 characters.”

We discussed this at length, and it seems that the consensus is: try to keep line lengths less than 79/72 characters, unless you have a special situation where longer lines would improve readability. (The basic reason is that 79/72 works for everyone, and BigchainDB is an open source project.) As a hard limit, keep all lines less than 119 characters (which is the width of GitHub code review).

Single or Double Quotes?¶

Python lets you use single or double quotes. PEP8 says you can use either, as long as you’re consistent. We try to stick to using single quotes, except in cases where using double quotes is more readable. For example:

print('This doesn\'t look so nice.')
print("Doesn't this look nicer?")

Breaking Strings Across Multiple Lines¶

Should we use parentheses or slashes (\) to break strings across multiple lines, i.e.

my_string = ('This is a very long string, so long that it will not fit into just one line '
             'so it must be split across multiple lines.')
# or
my_string = 'This is a very long string, so long that it will not fit into just one line ' \
            'so it must be split across multiple lines.'

It seems the preference is for slashes, but using parentheses is okay too. (There are good arguments either way. Arguing about it seems like a waste of time.)

How to Format Long import Statements¶

If you need to import lots of names from a module or package, and they won’t all fit in one line (without making the line too long), then use parentheses to spread the names across multiple lines, like so:

from Tkinter import (
    Tk, Frame, Button, Entry, Canvas, Text,
    LEFT, DISABLED, NORMAL, RIDGE, END,
)

# Or

from Tkinter import (Tk, Frame, Button, Entry, Canvas, Text,
                     LEFT, DISABLED, NORMAL, RIDGE, END)

For the rationale, see PEP 328.

Using the % operator or format() to Format Strings¶

Given the choice:

x = 'name: %s; score: %d' % (name, n)
# or
x = 'name: {}; score: {}'.format(name, n)

we use the format() version. The official Python documentation says, “This method of string formatting is the new standard in Python 3, and should be preferred to the % formatting described in String Formatting Operations in new code.”

Running the Flake8 Style Checker¶

We use Flake8 to check our Python code style. Once you have it installed, you can run it using:

flake8 --max-line-length 119 bigchaindb/

Writing and Running (Python) Tests¶

The content of this section was moved to bigchaindb/tests/README.md.

Note: We automatically run all tests on all pull requests (using Travis CI), so you should definitely run all tests locally before you submit a pull request. See the above-linked README file for instructions.

Our Release Process¶

Notes¶

BigchainDB follows the Python form of Semantic Versioning (i.e. MAJOR.MINOR.PATCH), which is almost identical to regular semantic versioning, but there’s no hyphen, e.g.

0.9.0 for a typical final release
4.5.2a1 not 4.5.2-a1 for the first Alpha release
3.4.5rc2 not 3.4.5-rc2 for Release Candidate 2

Note 1: For Git tags (which are used to identify releases on GitHub), we append a v in front. For example, the Git tag for version 2.0.0a1 was v2.0.0a1.

Note 2: For Docker image tags (e.g. on Docker Hub), we use longer version names for Alpha, Beta and Release Candidate releases. For example, the Docker image tag for version 2.0.0a2 was 2.0.0-alpha2.

We use 0.9 and 0.9.0 as example version and short-version values below. You should replace those with the correct values for your new version.

We follow BEP-1, which is our variant of C4, the Collective Code Construction Contract, so a release is just a tagged commit on the master branch, i.e. a label for a particular Git commit.

The following steps are what we do to release a new version of BigchainDB Server. The steps to release the Python Driver are similar but not the same.

Steps¶

Create a pull request where you make the following changes:
- Update CHANGELOG.md
- Update all Docker image tags in all Kubernetes YAML files (in the k8s/ directory). For example, in the files:
  - k8s/bigchaindb/bigchaindb-ss.yaml and
  - k8s/dev-setup/bigchaindb.yaml
  find the line of the form image: bigchaindb/bigchaindb:0.8.1 and change the version number to the new version number, e.g. 0.9.0. (This is the Docker image that Kubernetes should pull from Docker Hub.) Keep in mind that this is a Docker image tag so our naming convention is a bit different; see Note 2 in the Notes section above.
- In bigchaindb/version.py:
  - update __version__ to e.g. 0.9.0 (with no .dev on the end)
  - update __short_version__ to e.g. 0.9 (with no .dev on the end)
- In the docs about installing BigchainDB (and Tendermint), and in the associated scripts, recommend/install a version of Tendermint that actually works with the soon-to-be-released version of BigchainDB. You can find all such references by doing a search for the previously-recommended version number, such as 0.31.5.
- In setup.py, maybe update the development status item in the classifiers list. For example, one allowed value is "Development Status :: 5 - Production/Stable". The allowed values are listed at pypi.python.org.
Wait for all the tests to pass!
Merge the pull request into the master branch.
Go to the bigchaindb/bigchaindb Releases page on GitHub and click the “Draft a new release” button.
Fill in the details:
- Tag version: version number preceded by v, e.g. v0.9.1
- Target: the last commit that was just merged. In other words, that commit will get a Git tag with the value given for tag version above.
- Title: Same as tag version above, e.g v0.9.1
- Description: The body of the changelog entry (Added, Changed, etc.)
Click “Publish release” to publish the release on GitHub.
On your local computer, make sure you’re on the master branch and that it’s up-to-date with the master branch in the bigchaindb/bigchaindb repository (e.g. git pull upstream master). We’re going to use that to push a new bigchaindb package to PyPI.
Make sure you have a ~/.pypirc file containing credentials for PyPI.
Do make release to build and publish the new bigchaindb package on PyPI. For this step you need to have twine installed. If you get an error like Makefile:135: recipe for target 'clean-pyc' failed then try doing
```
sudo chown -R $(whoami):$(whoami) .
```
Log in to readthedocs.org and go to the BigchainDB Server project, then:

Click on “Builds”, select “latest” from the drop-down menu, then click the “Build Version:” button.
Wait for the build of “latest” to finish. This can take a few minutes.
Go to Admin –> Advanced Settings and make sure that “Default branch:” (i.e. what “latest” points to) is set to the new release’s tag, e.g. v0.9.1. (It won’t be an option if you didn’t wait for the build of “latest” to finish.) Then scroll to the bottom and click “Save”.
Go to Admin –> Versions and under Choose Active Versions, do these things:
1. Make sure that the new version’s tag is “Active” and “Public”
2. Make sure the stable branch is not active.
3. Scroll to the bottom of the page and click “Save”.

Go to Docker Hub and sign in, then:

Click on “Organizations”
Click on “bigchaindb”
Click on “bigchaindb/bigchaindb”
Click on “Build Settings”
Find the row where “Docker Tag Name” equals latest and change the value of “Name” to the name (Git tag) of the new release, e.g. v0.9.0.
If the release is an Alpha, Beta or Release Candidate release, then a new row must be added. You can do that by clicking the green “+” (plus) icon. The contents of the new row should be similar to the existing rows of previous releases like that.
Click on “Tags”
Delete the “latest” tag (so we can rebuild it)
Click on “Build Settings” again
Click on the “Trigger” button for the “latest” tag and make sure it worked by clicking on “Tags” again
If the release is an Alpha, Beta or Release Candidate release, then click on the “Trigger” button for that tag as well.

Congratulations, you have released a new version of BigchainDB Server!

BigchainDB 문서¶

블록체인 데이터베이스인 BigchainDB를 만나보세요.

분산형, 불변성 및 자산에 대한 네이티브 지원 을 포함한 일부 데이터베이스 특성들과 블록체인 특성을 가지고 있습니다.

높은 수준에서, 사용자는 BigchainDB HTTP API를 사용하는 BigchainDB 네트워크(노드 집합) 또는 BigchainDB 파이썬 드라이버와 같은 API용 래퍼로 통신할 수 있습니다. 각 BigchainDB 노드는 BigchainDB Server 및 다양한 다른 소프트웨어를 실행합니다. 더 자세한 사항은 용어 페이지에서 이러한 용어 중 일부를 설명합니다.

HTTP API 문서

서버 문서

BigchainDB에 대한 추가 정보¶

배포 - 준비¶

경우에 따라, BigchainDB는 배포-준비가 될 수도 있고 되지 않을 수도 있습니다. 서비스 공급자에게 문의해야 합니다. 만약 BigchainDB를 (배포로) 전환하고자 한다면, 서비스 공급자에게 문의하십시오.

참고 : BigchainDB는 “보증 없음” 섹션을 가지는 오픈소스 라이센스이며, 이는 전형적인 오픈소스 라이센스입니다. 이는 소프트웨어 산업의 표준입니다. 예를 들어, 리눅스 커널은 라이센스에 “보증 없음” 섹션을 가지고 있지만, 수십억 대의 시스템에 의해 배포되어 사용됩니다. 보증은 대개 서비스 공급자가 소프트웨어 라이센스 수준 이상으로 제공합니다.

용어¶

BigchainDB와 관련돈 몇 가지 전문화된 용어가 있습니다. 시작하기에 앞서, 최소한 다음과 같은 사항을 알아야합니다.

BigchainDB 노드¶

BigchainDB 노드는 BigchainDB 서버 및 관련된 소프트웨어를 실행하는 시스템(또는 논리적인 시스템)입니다. 각각의 노드는 한 개인이나 조직에 의해 제어될 수 있습니다.

BigchainDB 네트워크¶

BigchainDB 노드들의 집합은 서로 연결하여 BigchainDB 네트워크를 형성할 수 있습니다. 해당 네트워크에서 각각의 노드는 동일한 소프트웨어를 실행합니다. BigchainDB 네트워크는 모니터링 같은 것들을 하기 위한 추가적인 시스템이 있을 수 있습니다.

BigchainDB 컨소시엄¶

BigchainDB 네트워크에 노드들을 실행하는 사람과 조직은 BigchainDB 컨소시엄(즉, 다른 조직)에 속합니다. 컨소시엄은 결정을 하기 위해 일종의 거버넌스 구조를 가져야합니다. 만약 BigchainDB 네트워크가 단 하나의 회사에 의해서 운영된다면, “컨소시엄”은 단지 그 회사일 뿐입니다.

BigchainDB 네트워크와 컨소시엄의 차이는 무엇일까요?

BigchainDB 네트워크는 단지 연결된 노드들의 집합입니다. 컨소시엄은 하나의 BigchainDB 네트워크를 가지는 조직이며, 해당 네트워크에서 각각의 노드는 다른 운영자를 가집니다.

BigchainDB 분산 방식¶

분산이란 모든 것을 소유하거나 통제하는 사람이 없고, 단 하나의 실패 지점이 없다는 것을 의미합니다.

이상적으로, BigchainDB 네트워크에서 각각의 노드는 다른 개인 또는 조직에 의해 소유되고 관리됩니다. 네트워크가 한 조직 내에 상주하고 있더라도, 각 노드를 다른 사용자나 부서로 제어하는 것이 좋습니다.

우리는 “BigchainDB 컨소시엄” (또는 단지 “컨소시엄”)은 BigchainDB 네트워크의 노드를 구동하는 사람들 혹은 조직을 의미합니다. 컨소시엄은 회원제나 정책과 같은 결정을 내리기 위한 어떠한 형태의 거버넌스 요구합니다. 거버넌스 프로세스의 정확한 세부사항은 각 컨소시엄에 의해 결정되지만, 상당히 분산될 수 있습니다.

컨소시엄은 관할의 다양성과 지리적 다양성 및 기타 종류의 다양성을 증가시킴으로써 분산화(및 탄력성)를 증가시킬 수 있습니다. 이 아이디어는 노드 다양성 부문에서 확장됩니다.

BigchainDB 네트워크에는 오래된 특정한 위치를 가지는 노드가 없습니다. 모든 노드들은 동일한 소프트웨어를 실행하고 동일한 작업을 수행합니다.

만약 노드에 대한 관리자 접근 권한이 있는 경우, 해당 노드를 제거할 수 있지만(예: 해당 노드에 저장된 데이터 변경 또는 삭제), 이러한 변경 사항은 해당 노드에 분리된 상태로 유지되어야 합니다. BigchainDB 네트워크는 노드의 3분의 1 이상이 손상된 경우에만 손상될 수 있습니다. 자세한 내용은 Tendermint 문서을 참조하십시오.

노드의 관리자나 슈퍼 유저도 자산을 전송할 수 없다는 점에 유의하십시오. 유효한 전송 트랜잭션을 생성하는 유일한 방법은 자산에 대한 현재 암호화 조건을 충족하는 것이며 관리자/슈퍼사용자는 필요한 정보를 가지고 있지 않기 때문에 이 작업을 수행할 수 없습니다(예: 개인 키).

노드의 관리자나 슈퍼 사용자도 자산을 전송할 수는 없다는 점을 유의하십시오. 타당한 전송 트랜잭션을 만드는 유일한 방법은 자산에 대한 현재 암호화 조건을 충족시키는 것이며, 관리자 또는 슈퍼 사용자는 필요한 정보를 가지고 있지 않기 때문에 이 작업을 수행할 수 없습니다. (예: 개인 키)

노드 다양성의 종류¶

한 명의 유저나 이벤트가 노드의 “상당수” 부분을 제어하거나 손상시키는 것을 어렵게 만드는 절차가 수행되어야 합니다.(BigchainDB 서버는 Tendermint를 사용하기 때문에 여기서 “상당수”는 노드의 1/3을 말합니다.) 아래에 쓰여진 여러 가지의 다양성을 고려해야 합니다. 모든 종류에 있어서 높은 다양성을 갖는 것은 꽤 어려운 일입니다.

관할권 다양성. 노드는 여러 합법적 관할권 내의 참여자들이 제어합니다. 이는 어떤 일을 수행하기에 이 수단들을 사용하기 어렵게 할 수 있습니다.
지리적 다양성. 서버는 지리적으로 여러 곳에 물리적으로 위치합니다. 이는 자연 재해(홍수 또는 지진 등)가 문제를 야기할 만큼 손상되기 어렵게 합니다.
호스팅 다양성. 서버는 여러 호스팅 공급자(ex. Amazon Web Services, Microsoft Azure, Digital Oceure, Rackspace)가 호스팅해야 합니다. 이는 하나의 호스팅 공급자가 ‘상당 수’의 노드에 영향을 미치기가 어려워집니다.
일반적인 의미의 다양성. 일반적으로 멤버십 다양성은 컨소시엄에 많은 이점을 줍니다. 예를 들어, 그것은 문제 해결에 필요한 다양한 아이디어 소스를 컨소시엄에 제공합니다.

참고: 모든 노드가 동일한 코드(ex. BigchainDB의 동일한 구현)를 실행하고 있는 경우, 해당 코드의 버그를 사용하여 모든 노드를 손상시킬 수 있습니다. 이상적으로는 BigchainDB 서버(예: Python, Go 등)에서 운영되고 있는 다양한 구현이 있어, 컨소시엄은 다양한 서버 구현을 할 수 있을 것입니다. 운영 체제에 대해서도 이는 유사하게 적용됩니다.

어떻게 BigchainDB는 불변성을 갖는가¶

Imunable이라는 단어는 “시간 경과에 따른 불변성”을 의미합니다. 예를 들어, π의 10진수 값은 변경할 수 없습니다(3.14159…).

블록체인 커뮤니티는 종종 블록체인을 “불변하다”고 설명합니다. 우리가 그 단어를 문자 그대로 해석한다면, 그것은 블록체인 데이터가 변경할 수 없거나 영구적이라는 것을 의미하는데, 이것은 말이 안됩니다. 데이터는 변경 될 수 있습니다. 예를 들어, 전염병이 인류를 멸종 시킬 수도 있는 것처럼 데이터는 수분에 의한 손상, 온도에 의한 손상, 엔트로피의 일반적인 증가로 인해 시간이 지남에 따라 손상될 수 있습니다.

블록체인 데이터가 일반적인 경우보다 변경(혹은 삭제)하기가 더 어려운 것은 사실입니다. 블록체인 데이터는 단순히 (의도적인) “변형 방지”에 그치지 않고 하드 드라이브의 데이터 손상과 같은 비의도적으로 발생할 수 있는 무작위 변경에도 대응합니다. 따라서 블록체인에서 “불변한다”라는 단어를 우리는 어떤 모든 의도와 목적이 실제적으로 불변한 것으로 해석합니다. (언어학자들은 “불변한다”라는 단어가 블록체인 커뮤니티에서 쓰이는 기술적 용어라고 말할 것입니다.)

블록체인 데이터는 여러 가지 방법을 통해 불변성을 가질 수 있습니다:

데이터 변경 또는 삭제를 위한 API 없음. Blockchain 소프트웨어는 일반적으로 블록체인에 저장된 데이터를 변경하거나 삭제하기 위한 API를 제공하지 않습니다. BigchainDB 역시 관련한 API를 제공하지 않습니다 . 이것은 변경이나 삭제가 다른 방식으로 일어나는 것을 막지 못합니다. 이것은 단지 하나의 방어선일 뿐입니다.
복제. 모든 데이터는 여러 곳에 복제(복사)됩니다. 복제 팩터가 높을수록, 모든 복제본을 변경하거나 삭제하기가 더 어려워집니다.
내부 감시 장치. 모든 노드가 모든 변경 사항을 모니터링하고 허용되지 않은 변경 사항이 발생하면 적절한 조치를 취할 수 있습니다.
외부 감시 장치. 컨소시엄은 부정행위를 찾아 데이터를 모니터링하고 감사할 수 있는 검증된 제 3자를 선택할 수 있습니다. 공개적으로 읽을 수 있는 데이터를 가진 컨소시엄의 경우, 대중은 감사자 역할을 할 수 있습니다.
경제적 인센티브. 일부 블록체인 시스템은 기존의 저장된 데이터를 변경하는 데 많은 비용이 들게 합니다. 그 예로 작업 증명 및 지분 증명 시스템이 있습니다. BigchainDB의 경우에는 이런 인센티브를 사용하지 않습니다.
변화에 대한 손쉬운 실행 취소를 위해 오류 수정 코드와 같은 고급 기술을 사용해 데이터를 저장할 수 있습니다
암호화폐의 표식은 종종 메시지(예: 트랜잭션)가 도중에 손상되었는지 확인하고 메시지에 서명한 사용자를 확인하는 방법으로 사용됩니다. BigchainDB에서는 각 트랜잭션에 한 명 이상의 당사자가 서명해야 합니다
전체 또는 부분적 백업은 때때로 자기 테이프 저장소, 기타 블록체인, 인쇄물 등에 기록될 수 있습니다.
강력한 보안 노드 소유자는 강력한 보안 정책을 채택하고 적용할 수 있습니다.
노드 다양성. 다양성으로 인해서 한 가지 요소(예: 자연 재해 또는 운영 체제 버그)가 상당 수의 노드를 손상시킬 수 없도록 합니다. 노드 다양성의 종류에 대한 섹션을 참조하세요.

BigchainDB와 Byzantine Fault Tolerance¶

BigchainDB Server 는 블록체인 합의와 트랜잭션 복제에 Tendermint를 사용합니다.

그리고 Tendermint 는 Byzantine Fault Tolerant (BFT).

BigchainDB 쿼리¶

노드 operator는 MongoDB의 쿼리 엔진의 최대 성능을 사용하여 모든 트랜잭션, 자산 및 메타데이터를 포함하여 저장된 모든 데이터를 검색하고 쿼리할 수 있습니다. 노드 operator는 외부 사용자에게 얼마나 많은 쿼리 파워를 송출할지 스스로 결정할 수 있습니다.

예제 쿼리가 포함된 블로그 게시물¶

BigchainDB 블로그에 MongoDB 도구를 사용하여 BigchainDB 노드의 MongoDB 데이터베이스를 쿼리하는 방법에 대한 게시물을 올렸습니다. 데이터에 대한 일부 특정 예제 쿼리가 주요 내용입니다. 여기서 확인하세요

MongoDB에 연결하기¶

MongoDB 데이터베이스를 쿼리하려면 먼저 데이터베이스에 연결해야 합니다. 그러기 위해선 호스트 이름과 포트를 알아야 합니다.

개발 및 테스트를 위해 지역 컴퓨터에서 BigchainDB 노드를 실행 중인 경우 호스트 이름은 “로컬 호스트”여야 하며 이러한 값을 변경하지 않는 한 포트는 “27017”이어야 합니다. 원격 시스템에서 BigchainDB 노드를 실행 중이며 해당 시스템에 SSH할 수 있는 경우에도 마찬가지입니다.

원격 시스템에서 BigchainDB 노드를 실행하고 MongoDB를 auth를 사용하고 공개적으로 액세스할 수 있도록 구성한 경우(권한이 있는 사용자에게) 호스트 이름과 포트를 확인할 수 있습니다.

쿼리하기¶

BigchainDB 노드 운영자는 로컬 MongoDB 인스턴스에 대한 전체 액세스 권한을 가지므로 실행하는데 MongoDB의 다음의 API를 사용할 수 있습니다:

the Mongo Shell
MongoDB Compass
one of the MongoDB drivers, such as PyMongo, or
MongoDB 쿼리에 대한 서드파티툴, RazorSQL, Studio 3T, Mongo Management Studio, NoSQLBooster for MongoDB, or Dr. Mongo.

Note

SQL을 이용해 mongoDB 데이터베이스를 쿼리할 수 있습니다. 예를 들어:

Studio 3T: “How to Query MongoDB with SQL”
NoSQLBooster for MongoDB: “How to Query MongoDB with SQL SELECT”

예를 들어, 기본 BigchainDB 노드를 실행하는 시스템에 있는 경우 Mongo Shell (mongo)을 사용하여 연결하고 다음과 같이 볼 수 있습니다.

$ mongo
MongoDB shell version v3.6.5
connecting to: mongodb://127.0.0.1:27017
MongoDB server version: 3.6.4
...
> show dbs
admin     0.000GB
bigchain  0.000GB
config    0.000GB
local     0.000GB
> use bigchain
switched to db bigchain
> show collections
abci_chains
assets
blocks
elections
metadata
pre_commit
transactions
utxos
validators

위 예제는 몇 가지 상황을 보여줍니다:

호스트 이름이나 포트를 지정하지 않으면 Mongo Shell은 각각 localhost와 27017으로 가정합니다. (localhost는 우분투에 IP주소를 127.0.0.1로 설정했습니다.)

BigchainDB는 데이터를 bigchain이라는 데이터베이스에 저장합니다.
bigchain 데이터베이스에는 여러 collections가 포함되어 있습니다.
어떤 컬렉션에도 투표가 저장되지 않습니다. 이런 데이터는 모두 자체(LevelDB) 데이터베이스에 의해 처리되고 저장됩니다.

컬렉션에 대한 예시 문서¶

bigchain 데이터베이스의 가장 흥미로운 부분은 아래와 같습니다:

transactions
assets
metadata
blocks

db.assets.findOne() 은 MongoDB 쿼리를 사용하여 이러한 컬렉션들을 탐색할 수 있습니다.

트랜잭션에 대한 예시 문서¶

transaction 컬렉션에서 CREATE 트랜잭션에는 추가 "_id" 필드(MongoDB에 추가됨)가 포함되며 "asset"과 "metadata" 필드에는 데이터가 저장되어 있지 않습니다.

{  
    "_id":ObjectId("5b17b9fa6ce88300067b6804"),
    "inputs":[…],
    "outputs":[…],
    "operation":"CREATE",
    "version":"2.0",
    "id":"816c4dd7…851af1629"
}

A TRANSFER transaction from the transactions collection is similar, but it keeps its "asset" field.

{  
    "_id":ObjectId("5b17b9fa6ce88300067b6807"),
    "inputs":[…],
    "outputs":[…],
    "operation":"TRANSFER",
    "asset":{  
        "id":"816c4dd7ae…51af1629"
    },
    "version":"2.0",
    "id":"985ee697d…a3296b9"
}

assets에 대한 예시 문서¶

assets에 대한 기술에는 MongoDB가 추가한 "_id" 분야와 CREATE 거래에서 나온 asset.data 그리고 "id" 세 가지 최상위 분야로 구성되어 있습니다.

{  
    "_id":ObjectId("5b17b9fe6ce88300067b6823"),
    "data":{  
        "type":"cow",
        "name":"Mildred"
    },
    "id":"96002ef8740…45869959d8"
}

metadata에 대한 예시 문서¶

metadata 컬렉션의 문서는 MongoDB가 추가한 "_id"필드와 거래에서 나온 asset.data그리고 거래에서 나온 "id" 세 가지 최상위 분야로 구성되어 있습니다.

{  
    "_id":ObjectId("5b17ba006ce88300067b683d"),
    "metadata":{
        "transfer_time":1058568256
    },
    "id":"53cba620e…ae9fdee0"
}

blocks에 대한 예시 문서¶

{
    "_id":ObjectId("5b212c1ceaaa420006f41c57"),
    "app_hash":"2b0b75c2c2…7fb2652ce26c6",
    "height":17,
    "transactions":[
        "5f1f2d6b…ed98c1e"
    ]
}

노드 operator가 외부 유저에게 보낼 수 있는 것¶

각 노드 operator는 외부 사용자가 자신의 로컬 MongoDB 데이터베이스에서 정보를 얻는 방법을 결정할 수 있습니다. 그들은 다음과 같은 것들을 보낼 수 있습니다:

외부유저를 쿼리 처리하는 로컬 MongoDB 데이터베이스 한된 제한된 권한을 가진 역할을 가진 MongoDB 사용자 예) read-only
제한된 미리 정의된 쿼리 집합을 허용하는 제한된 HTTP API, BigchainDB 서버에서 제공하는 HTTP API, 혹은Django, Express, Ruby on Rails, or ASP.NET.를 이용해 구현된 커스텀 HTTP API
다른 API(예: GraphQL API) 제3자의 사용자 정의 코드 또는 코드를 사용하여 수행할 수 있습니다..

각 노드 operator는 로컬 MongoDB 데이터베이스에 대한 다른 레벨 또는 유형의 액세스를 노출할 수 있습니다. 예를 들어, 한 노드 operator가 최적화된 공간 쿼리를 전문으로 제공하기로 정할 수 있습니다.

보안 고려사항¶

BigchainDB 버전 1.3.0 이전 버전에서는 하나의 MongoDB 논리 데이터베이스가 있었기 때문에 외부 사용자에게 데이터베이스를 노출하는 것은 매우 위험했으며 권장되지 않습니다. “Drop database”는 공유된 MongoDB 데이터베이스를 삭제합니다.

BigchainDB 버전 2.0.0 이상에선 각 노드에 고유한 독립 로컬 MongoDB 데이터베이스가 존재합니다. 노드 간 통신은 아래 그림 1에서와 같이 MongoDB 프로토콜이 아닌 Tendermint 프로토콜을 사용하여 수행됩니다. 노드의 로컬 MongoDB 데이터베이스가 손상되어도 다른 노드는 영향을 받지 않습니다.

image

Figure 1: A Four-Node BigchainDB 2.0 Network

퍼포먼스 및 요금 고려사항¶

쿼리 프로세싱은 상당히 많은 리소스를 소모할 수 있으므로, BigchainDB 서버 및 Tendermint Core와 별도의 컴퓨터에서 MongoDB를 실행하는 것이 좋습니다.

노드 operator 는 조회에 사용되는 리소스를 측정하여 조회를 요청한 사람은 누구든지 요금을 청구할 수 있습니다.

일부 쿼리는 너무 오래 걸리거나 리소스를 너무 많이 사용할 수 있습니다. 노드 operator는 사용할 수 있는 리소스에 상한을 두고, 초과된다면 중지(또는 차단)해야 합니다.

MongoDB 쿼리를 더욱 효율적으로 만들기 위해 인덱스를 만들 수 있습니다. 이러한 인덱스는 노드 operator 또는 일부 외부 사용자가 생성할 수 있습니다(노드 운영자가 허용하는 경우). 인덱스는 비어 있지 않습니다. 새 데이터를 컬렉션에 추가할 때마다 해당 인덱스를 업데이트해야 합니다. 노드 운영자는 이러한 요금을 인덱스를 생성한 사람에게 전달하고자 할 수 있습니다. mongoDB에서는 단일 컬렉션은 64개 이하의 인덱스를 가질 수 있습니다.

Tendermint voting 파워가 0인 노드인 추종자 노드를 생성할 수 있다. 여전히 모든 데이터의 복사본이 있으므로 읽기 전용 노드로 사용할 수 있습니다. Follower 노드는 투표 검증자의 작업 부하에 영향을 미치지 않고 서비스로 전문화된 쿼리를 제공할 수 있습니다(쓰기도 가능). 팔로워의 팔로워들도 있을 수 있습니다.

자바스크립트 쿼리 코드 예시¶

MongoDB node.js 드라이버와 같은 MongoDB 드라이버를 사용하여 다음 중 하나를 사용하여 노드의 MongoDB 데이터베이스에 연결할 수 있습니다. 여기 자바스크립트 쿼리 코드에 대한 링크가 있습니다.

BigchainDB가 자산 등록 및 전송에 적합한 방법¶

BigchainDB는 모든 종류의 데이터를 저장할 수 있지만 자산 등록 및 전송을 저장하는 데 특히 유용합니다.:

BigchainDB 네트워크에 전송되어 체크되고 저장되는 (있는 경우) 트랜잭션은 기본적으로 CREATE 트랜잭션과 TRANSFER 트랜잭션의 두 가지가 있습니다.
CREATE 트랜잭션은 임의의 메타 데이터와 함께 모든 종류의 자산 (나눌 수 없거나 분할 할 수없는)을 등록하는 데 사용할 수 있습니다.
저작물에는 0 명, 1 명 또는 여러 명의 소유자가있을 수 있습니다.
자산 소유자는 자산을 신규 소유자에게 양도하려는 사람이 만족해야하는 조건을 지정할 수 있습니다. 예를 들어 5 명의 현재 소유자 중 최소 3 명이 TRANSFER 트랜잭션에 암호를 사용해야합니다.
BigchainDB는 TRANSFER 트랜잭션의 유효성을 검사하는 과정에서 조건이 충족되었는지 확인합니다. (또한 누구나 만족하는지 확인할 수 있습니다.)
BigchainDB는 자산의 이중 지출을 방지합니다.
유효성이 검증 된 트랜잭션은 변경불가능 입니다.

Note

우리는 “소유자”라는 단어를 다소 느슨하게 사용했습니다. 보다 정확한 단어는 이행자, 서명자, 조정자 또는 이전 가능 요소 일 수 있습니다. 관련 BigchainDB Transaction Spec의 Owners에 대한 참고 사항 절을 참조하십시오.

BigchainDB 및 스마트계약¶

BigchainDB에는 스마트 계약 (즉, 컴퓨터 프로그램)의 소스 코드를 저장할 수 있지만 BigchainDB는 임의의 스마트 계약을 실행하지 않습니다.

BigchainDB는 대체 가능한 자산과 대체 할 수없는 자산 모두를 전송할 수있는 권한을 가진 사람을 시행하는 데 사용할 수 있습니다. 이중 지출을 막을 것입니다. 즉, ERC-20 (대체 가능한 토큰) 또는 ERC-721 (대체 할 수없는 토큰) 스마트 계약 대신 BigchainDB 네트워크를 사용할 수 있습니다.

자산 이전 권한은 쓰기 권한으로 해석 될 수 있으므로 로그, 저널 또는 감사 내역에 기록 할 수있는 사람을 제어하는데 사용할 수 있습니다. BigchainDB의 사용 권한에 대한 자세한 내용은 페이지에서 확인하십시오.

BigchainDB 네트워크는 oracles 또는 체인 간 통신 프로토콜을 통해 다른 블록 체인 네트워크에 연결할 수 있습니다. 이는 BigchainDB를 다른 블록 체인을 사용하여 임의의 스마트 계약을 실행하는 솔루션의 일부로 사용할 수 있음을 의미합니다.

트랜잭션 개념¶

트랜잭션은 물건 (예 : 자산)을 등록, 발행, 생성 또는 전송하는 데 사용됩니다.

트랜잭션은 BigchainDB가 저장하는 가장 기본적인 종류의 레코드입니다. CREATE 트랜잭션과 TRANSFER 트랜잭션의 두 종류가 있습니다.

트랜잭션 생성¶

CREATE 트랜잭션은 BigchainDB에서 한 가지 (또는 자산)의 이력을 등록, 발행, 생성 또는 다른 방법으로 시작하는 데 사용될 수 있습니다. 예를 들어, 신원이나 창작물을 등록 할 수 있습니다. 이러한 것들을 종종 “자산”이라고 부르지만 literal 자산이 아닐 수도 있습니다.

BigchainDB는 BigchainDB Server v0.8.0부터 나눌 수있는 자산을 지원합니다. 이는 “공유”의 초기 숫자로 자산을 생성 / 등록 할 수 있음을 의미합니다. 예를 들어, CREATE 트랜잭션은 50 개의 오크 나무로 된 트럭로드를 등록 할 수 있습니다. 분할 가능한 자산의 각 주식은 서로 공유 할 수 있어야합니다. 주식은 대체 가능해야합니다.

CREATE 트랜잭션은 하나 이상의 출력을 가질 수 있습니다. 각 출력에는 관련 금액이 있습니다. 출력에 연결된 공유 수입니다. 예를 들어 자산이 50 개의 오크 나무로 구성되어있는 경우 한 출력에는 한 소유자 세트에 35 개의 오크 나무가 있고 다른 출력에는 다른 소유자 세트에는 15 개의 오크 나무가있을 수 있습니다.

또한 각 출력에는 연관된 조건이 있습니다. 출력을 전송 / 소비하기 위해 충족되어야하는 조건 (TRANSFER 트랜잭션에 의해). BigchainDB는 다양한 조건을 지원합니다. 자세한 내용은 관련 BigchainDB 트랜잭션 Spec과 관련된 트랜잭션 구성 요소 : 조건 섹션을 참조하십시오.

_images/CREATE_example1.png Example BigchainDB CREATE transaction

위의 예제에서는 BigchainDB CREATE 트랜잭션 다이어그램을 보여줍니다. Pam은 자산 3 주를 소유 / 통제하고 다른 주식은 없습니다 (다른 산출물이 없으므로).

각 출력에는 해당 출력의 조건과 연관된 모든 공개 키 목록이 있습니다. 다시 말하면, 그 목록은 “소유자”의 목록으로 해석 될 수 있습니다.보다 정확한 단어는 이행자, 서명자, 컨트롤러 또는 이전 가능 요소 일 수 있습니다. 관련 BigchainDB Transactions Spec 소유자에 관한 참고 사항 섹션을 참조하십시오.

CREATE 트랜잭션은 모든 소유자가 서명해야합니다. (만약 당신이 그 서명을 원한다면, 그것은 인코딩되었지만 하나의 입력의 “이행”에있다.)

트랜잭션 이전¶

트랜잭션 이전은 다른 트랜잭션 (CREATE 트랜잭션 또는 다른 TRANSFER 트랜잭션)에서 하나 이상의 출력을 전송 / 소비 할 수 있습니다. 이러한 출력물은 모두 동일한 자산과 연결되어야합니다. TRANSFER 트랜잭션은 한 번에 하나의 자산의 공유 만 전송할 수 있습니다.

트랜잭션 이전의 각 입력은 다른 트랜잭션의 한 출력에 연결됩니다. 각 입력은 전송 / 소비하려는 출력의 조건을 충족해야합니다.

트랜잭션 이전은 위에서 설명한 CREATE 트랜잭션과 마찬가지로 하나 이상의 출력을 가질 수 있습니다. 투입물에 들어오는 총 주식 수는 산출물에서 나가는 총 주식 수와 같아야합니다.

_images/CREATE_and_TRANSFER_example1.png Example BigchainDB transactions

위 그림은 두 개의 BigchainDB 트랜잭션, CREATE 트랜잭션 및 TRANSFER 트랜잭션의 다이어그램을 보여줍니다. CREATE 트랜잭션은 이전 다이어그램과 동일합니다. TRANSFER 트랜잭션은 Pam의 출력을 소비하므로 TRANSFER 트랜잭션의 입력에는 Pam의 유효한 서명 (즉, 유효한 이행)이 포함되어야합니다. TRANSFER 트랜잭션에는 두 개의 출력이 있습니다. Jim은 하나의 공유를 가져오고 Pam은 나머지 두 개의 공유를 가져옵니다.

용어 : “Pam, 3”출력을 “소비 된 트랜잭션 출력”이라고하며 “Jim, 1”및 “Pam, 2”출력을 “사용되지 않은 트랜잭션 출력”(UTXO)이라고합니다.

예제 1: 빨간 차가 Joe가 소유하고 관리한다고 가정합니다. 자동차의 현재 전송 조건에서 Joe가 유효한 전송을 서명해야한다고 가정합니다. Joe는 Joe의 서명 (현재 출력 조건을 충족시키기 위해)과 Rae가 유효한 전송을 서명해야한다는 새로운 출력 조건을 포함하는 입력을 포함하는 TRANSFER 트랜잭션을 작성할 수 있습니다.

예제 2: 예를 들어 동일한 자산 유형의 이전에 전송되지 않은 4 개의 자산에 대한 출력 조건을 충족하는 TRANSFER 트랜잭션을 생성 할 수 있습니다. 종이 클립. 총 금액은 20, 10, 45 및 25 일 수 있으며, 말하자면 총 100 개의 클립입니다. 또한 TRANSFER 트랜잭션은 새로운 전송 조건을 설정합니다. 예를 들어, Gertrude가 서명하는 경우에만 60 개의 클립 클립이 전송 될 수 있으며 Jack과 Kelly가 서명하는 경우에만 40 개의 클립 클립이 전송 될 수 있습니다. 들어오는 클립 클립의 합계가 나가는 클립 클립의 합계와 같아야합니다 (100).

트랜잭션 유효성¶

언제 트랜잭션이 유효한지 유효성을 검사하는 것에 관해 해당 블로그에 게시되어있습니다. The BigchainDB Blog: “What is a Valid Transaction in BigchainDB?” (Note: That post was about BigchainDB Server v1.0.0.)

Each BigchainDB Transactions Spec documents the conditions for a transaction (of that version) to be valid.

트랜잭션 예시¶

아래의 HTTP API 문서와 the Python 드라이버 문서에는 예제 BigchainDB 트랜잭션이 있습니다. .

BigchainDB에 파일을 저장하는 방법¶

BigchainDB 네트워크에 파일을 저장할 수는 있지만 그렇게하지 않는 것이 좋습니다. 파일이 아닌 구조화 된 데이터를 저장, 인덱싱 및 쿼리하는 데 가장 적합합니다.

분산 된 파일 저장소를 원하면 Storj, Sia, Swarm 또는 IPFS / Filecoin을 확인하십시오. 파일 URL, 해시 또는 기타 메타 데이터를 BigchainDB 네트워크에 저장할 수 있습니다.

BigchainDB 네트워크에 파일을 저장해야하는 경우,이를 수행하는 한 가지 방법은 긴 Base64 문자열로 변환 한 다음 해당 문자열을 하나 이상의 BigchainDB 트랜잭션 (CREATE 트랜잭션의 asset.data)에 저장하는 것입니다 , 또는 어떤 거래의 메타데이터 일 수도있다.

BigchainDB 사용 권한¶

BigchainDB를 사용하면 다른 사용자가 할 수 있는 것을 어느 정도 제어할 수 있습니다. 이 능력은 *nix환경에서의 “권한”, SQL에서의 “특권”, 보안 환경에서의 “액세스 제어”와 유사합니다.

출력 지출/이전 권한¶

BigchainDB에서, 모든 출력에는 연관된 조건(crypto-condition)이 있습니다.

사용되지 않은 출력을 쓰거나 전송하려면, 사용자(또는 사용자 그룹)이 조건을 충족시켜야 합니다. 특정 사용자만이 출력을 보낼 권한이 있다는 뜻입니다. 가장 단순한 조건은, “공용 키에 해당하는 개인 키를 가진 사람만이 출력을 보낼 수 있습니다.” 훨씬 더 정교한 조건들도 가능합니다, 예를 들어 “이 출력을 사용하려면,…”

“…회계 그룹의 모든 사람이 서명 할 수 있습니다.”
“…네 명 중 세 명이 서명해야 합니다.”
“…Bob이 반드시 서명해야 하거나 Tom과 Sylvia 둘 모두가 서명해야 합니다.”

자세한 내용은, BigchainDB Transactions Spec관련 트랜잭션 구성요소:조건 섹션을 참조하세요.

출력이 한번 소비되면 다시 사용할 수 없습니다: 아무도 그렇게 할 권한이 없습니다. 즉, BigchainDB는 누구나 출력을 “이중 소비” 하도록 허용 하지 않습니다.

쓰기 권한¶

누군가 TRANSFER 트랜잭션을 만들면, metadata 필드에 임의의 JSON 객체를 넣을 수 있다. (적정 범위 내에서; 실제 BigchainDB 네트워크는 트랜잭션의 크기에 제한을 둔다.) 즉, TRANSFER 트랜잭션에서 원하는 모든 것을 쓸 수 있다.

BigchainDB에서 “쓰기 권한”이 없다는 의미인가요? 아닙니다!!

TRANSFER 트랜잭션은 입력이 이전 출력을 충족시키는 경우에만 유효(허용)합니다. 이 출력들에 대한 조건은 누가 유효한 TRANSFER 트랜잭션을 할 수 있는지 조절 할 것입니다. 즉, 출력에 대한 조건은 특정 사용자에게 관련 자산 내역에 무엇인가 쓸 수 있는 “쓰기 권한”을 부여하는 것과 같습니다.

예를 들어, 당신은 BigchainDB를 사용하여 오직 당신만이 쓰기권한이 있는 공용 저널을 작성 할 수 있습니다. 방법은 다음과 같습니다: 먼저 하나의 출력으로 asset.data 을 통해 {"title": "The Journal of John Doe"} 와 같이 되도록 CREATE 트랜잭션을 생성합니다. 이 출력에는 금액 1과 사용자(개인 키를 가진)만이 출력을 보낼 수 있는 조건이 있습니다. 저널에 무엇인가를 추가하고 싶을 때마다, metadata 같은 필드에 최신 항목을 넣은 TRANSFER 트랜잭션을 새로 만들어야 합니다.

{"timestamp": "1508319582",
 "entry": "I visited Marmot Lake with Jane."}

TRANSFER 트랜잭션에는 하나의 출력이 있습니다. 이 출력에는 금액1과 사용자(개인키를 가진)만이 출력을 보낼 수 있는 조건이 있습니다. 기타 등등. 당신만이 자산 내역(당신의 저널)에 덧붙일 수 있습니다.

이와 같은 기술은 공학 노트북,공급망 기록,정부 회의록 등에도 사용 될 수 있습니다.

또한 더 정교한 것들도 할 수 있습니다. 예를 들어, 누군가가 TRANSFER 트랜잭션을 작성할 때마다, 다른 누군가에게 사용 권한을 부여하여 일종의 작성자-전달 혹은 연쇄 편지를 설정한다.

Note

누구나 CREATE 트랜잭션의 asset.data 필드에 있는 JSON(조건하에)을 쓸 수 있습니다. 허가가 필요하지 않습니다.

읽기 권한¶

다음 페이지를 참고하세요, :doc:BigchainDB, Privacy and Private Data.

역할 기반 액세스 제어(RBAC)¶

2017년 9월에, 우리는 BigchainDB RBAC 하부 시스템을 정의 할 수 있는 방법에 대한 블로그 게시물을 게재 했습니다. 글을 쓴 시점(2018년 1월)에는 플러그인을 사용해야 해서, 표준 BigchainDB(다음에서 사용가능한 BigchainDB Testnet 를 사용 할 수 없었습니다. 이는 미래에 바뀔 수 있습니다. 만약 관심이 있다면, BigchainDB로 연락하십시요.

BigchainDB, 개인정보 및 개인 데이터¶

기본 정보¶

한도 내에서 BigchainDB 네트워크에 임의의 데이터(암호화 된 데이터 포함)를 저장 할 수 있습니다. 모든 트랜잭션에는 거의 모든 유니코드 문자열(최대 길이까지)을 저장 할 수 있는 metadata 섹션이 있습니다. 마찬가지로, 모든 CREATE 트랜잭션에는 거의 모든 유니코드 문자열을 저장 할 수 있는 asset.data 섹션이 있습니다.
특정 BigchainDB 거래 필드에 저장된 데이터는 암호화 해서는 안됩니다, 예를 들어 공용키 및 자산과 같이. BigchainDB는 Zcoin과 비슷한 개인 거래를 제공하지 않습니다.
데이터가 BigchinDB 네트워크에 저장되면 변경 또는 삭제 될 수 없다고 가정하는 것이 좋습니다.
BigchainDB 네트워크의 모든 노드에는 저장된 모든 데이터의 전체 복사본이 있습니다.
BigchainDB 네트워크의 모든 노드는 저장된 모든 데이터를 읽을 수 있습니다.
BigchainDB 노드(예를 들어 노드이 sysadmin)에 대한 전체 액세스 권한을 가진 모든 사용자는해당 노드에 저장된 모든 데이터를 읽을 수 있습니다.
BigchainDB HTTP API를 통해 노드에 접근하는 모든 사용자는 BigchainDB에 저장된 모든 데이터를 찾고 읽을 수 있습니다. 액세스 권한이 있는 사람들의 목록은 매우 짧을 수 있습니다.
외부 사용자와 BigchainDB 노드 사이의 연결이(예를 들어 HTTPS를 사용하여) 암호화되 않으면도청자는 전송중인 모든 HTTP 요청 및 응답을 읽을 수 있습니다.
만약 누군가가 평문에 접근 할 수 있다면(어디에서 가져왔는지 관계없이), 원칙적으로 이것을 전 세계와 공유 할 수 있습니다. 그렇게 하는 것을 어렵게 만들 수 있습니다, 예를 들어 데이터가 많고 방을 나갈 떄 검색되는 안전한 방 안에만 들어 갈 수 있는 것과 같습니다.

오프 체인에서 개인 데이터 저장¶

시스템은 제3자 데이터베이스, 문서 저장소 또는 CMS(컨텐츠 관리 시스템)와 같은 오프 체인 데이터를 저장할 수 있으며, BigchinDB를 사용하여 다음 작업을 수행할 수 있습니다:

제3자 시스템에 읽기 권한 또는 기타 권한이 있는 사용자를 추적합니다. 이 작업을 수행하는 방법의 예는 아래에 있습니다.
제3자 시스템에 대한 모든 요청을 영구적으로 기록합니다.
모든 문서의 변경 사항을 감지 할 수 있도록, 다른 곳에 저장된 문서의 해시를 저장합니다.
암호화 된 터널을 설정했다는 것을 증명할 수 있도록 두 개의 오프 체인 파티(예:Diffie-Hellman 키 교환) 간의 모든 핸드셰이크 설정 요청 및 응답을 기록합니다(독자가 해당 터널에 액세스하지 않고). 이 아이디어에 대한 자세한 내용은 the BigchainDB Privacy Protocols 저장소에 있습니다.

특정 문서에 대한 읽기 권한을 가진 사람을 기록하는 간단한 방법은 제 3자 시스템(“Docpile“)이 모든 문서+사용자 쌍에 대해 BigchinDB 네트워크에 CREATE 트랜잭션을 저장하여 해당 사용자가 그 문서에 대한 읽기 권한을 가지고 있음을 나타낼 수 있습니다. 트랜잭션은 Docpile에 의해 서명 될 수 있습니다(또는 문서 소유자에 의해). 자산 데이터 필드는 1)사용자의 고유 ID 및 2)문서의 고유 ID를 포함합니다. CREATE 트랜잭션의 한 출력은 DocPile(또는 문서 소유자)에 의해서만 전송/소비 될 수 있습니다.

읽기 권한을 취소하기 위해, DocPile은 원래 사용자가 더 이상 해당 문서에 대한 읽기 권한을 가지고 있지 않다고 하는 메타 데이터 필드를 사용하여, 원래의 CREATE 트랜잭션에서 하나의 출력을 보내기 위한 TRANSFER 트랜잭션을 생성 할 수 있습니다.

이는 무한정으로 수행될 수 있습니다,즉.사용자가 다시 읽기 권한을 가지고 있음을 나타내기 위해 다른 TRANSFER 트랜잭션을 DocPile에서 작성할 수 있습니다.

DocPile은 CREATE → TRANSFER → TRANSFER → 사용자+문서 쌍에 대한 etc.chain 과정에서 사용자의 마지막 트랜잭션을 읽음으로써 주어진 문서에 대한 읽기 권한을 가지고 있는지 파악할 수 있습니다.

여기에 같은 일을 하는 다른 방법들이 있다. 위는 단지 하나의 예시이다.

위의 예시에서는 사용자가 소유한(통제 된)자산으로 “읽기 권한“을 취급하지 않았다는 것을 알 수 있습니다, 왜냐하면 사용 권한 자산이 사용자에게 주어 지면(사용자에 의해 양도되거나 사용자에 의해 생성된 경우) 사용자가 다시 Docpile로 전송 할 때까지 어떠한 것도 제어 할 수 없기 때문입니다(Docpile에 의해).

체인에서 암호화 된 개인 데이터 저장¶

체인상에서 개인 데이터를 암호화하여 저장하는 방법에는 여러 가지가 있습니다. 모든 유스 케이스에는 고유한 목표와 제약이 있으며, 최상의 해결책은 유스 케이스에 달려있다. BigchainDB 컨설팅 팀, 우리의 파트너와 함께, 당신의유스 케이스에 가장 적합한 솔루션을 설계하는 데 도움을 줄 수 있습니다.

아래에서는 다양한 암호화 기본 설정을 사용하여 가능한 시스템을 설정하는 예제를 설명합니다.

참고 사항:

Ed25519 키 쌍은 메시지 암호화 및 암호 해독이 아닌 암호화 서명 및 확인을 위해 설계되었습니다. 암호화의 경우, X25519와 같은 암호화를 위해 설계된 키 쌍을 사용해야 합니다.
누군가(또는 어떤 그룹)이 체인상의 암호화 된 데이터를 해독하는 방법을 발표하면 암호화 된 데이터에 액세스 할 수 있는 모든 사람이 평문을 가져올 수 있습니다. 데이터는 삭제할 수 없습니다.
암호화 된 데이터는 MongoDM에서 색인을 생성하거나 검색 할 수 없습니다.(암호문을 색인화하고 검색 할 수 있지만 유용하지는 않습니다.) 암호화 된 데이터를 색인화하고 검색하기 위해 준 유사 암호를 사용할 수 있지만, MongoDB는 이를 지원할 계획이 없습니다. 색인화 또는 키워드 검색이 필요한 경우 asset.data의 몇가지 필드 또는 metadata객체를 일반 텍스트로 남겨두고 민감한 정보를 암호화 된 하위 객체에 저장할 수 있습니다.

시스템 예시 1¶

대칭 키로 데이터를 암호화하고 체인에(metadata 또는 asset.data 에서) 암호문을 저장하십시오. 키를 제 3자에게 알리려면, 공용 키를 사용하여 대칭 키를 암호화하고 암호화 키를 보냅니다. 개인 키로 대칭 키의 암호를 해독한 다음 대칭 키를 사용하여 on-chain 암호문의 암호를 해독할 수 있습니다.

공용 키/ 개인 키 쌍과 함께 대칭 키를 사용하는 이유는 암호문을 한 번만 저장하면 되기 때문입니다.

시스템 예시 2¶

이 예시에서는 프록시 재-암호화 를 사용합니다:

MegaCorp는 자체 공용 키를 사용하여 일부 데이터를 암호화 한 후 암호화 된 데이터(암호문1)을 BigchainDB 네트워크에 저장합니다.
MegaCorp는 다른 사람들이 암호화 된 데이터를 읽을 수 있게 하고 싶지만, 공용 키를 공유하지 않고 모든 새로운 수신자에 대해 스스로를 다시 암호화 할 필요가 없습니다. 대신 프록시 재 암호화 서비스를 제공하기 위해 Moxie라는 “프록시“를 찾습니다.
Zorban은 MegaCorp에 연결하여 데이터 읽기 권한을 요청합니다.
MegaCorp는 Zorban에게 공용 키를 요청합니다.
MegaCorp “재 암호화 키“를 생성하여 프록시 Moxie로 전송합니다.
Moxie (프록시)는 재 암호화 키를 사용하여 암호문 1을 암호화하고 암호문 2를 만듭니다.
Moxie는 Zorban(또는 Zorban에게 전달하는 MegaCorp)에게 암호문 2를 보냅니다.
Zorban은 개인 키를 사용하여 암호문 2를 해독해서 원본 암호화되지 않은 데이터를 가져옵니다.

참고:

프록시는 암호문만 볼 수 있습니다. 암호화 되지 않은 데이터는 볼 수 없습니다.
Zorban은 암호문 1, 즉 체인 상의 데이터를 해독 할 수 있는 능력이 없습니다.
위의 흐름에는 다양한 변형이 있습니다.

시스템 예시 3¶

이 예시는 삭제 코딩을 사용합니다:

데이터를 n개의 조각으로 삭제하십시오.
서로 다른 암호화 키로 n개의 조각을 암호화 하십시오.
n 개의 암호화 된 부분을 체인에 저장합니다 (예: n개의 별도 트랜잭션).
n 개의 암호 해독 키 각각을 다른 당사자와 공유하십시오.

만약 k< N 인 키홀더가 k개의 조각들을 가져와서 해독한다면, 그것들은 원본 텍스트를 다시 만들 수 있습니다. k미만이면 충분하지 않습니다.

시스템 예시 4¶

이 설정은 특수 노드가 데이터의 일부를 볼 수 있어야 하지만, 다른 노드는 볼 수 없어야 하는 기업용 블록 체인 시나리오에서 사용할 수 있습니다.

특수 노드는 X25519 키 쌍 (또는 유사한 비대칭 암호화키 쌍)을 생성합니다 .
BigchainDB 최종 사용자는 특수 노드의 X25519 공용 키(암호화 키)를 찾습니다. -최종 사용자는 위에서 언급 한 공용 키를 사용하여, asset.data 또는 메타 데이터(또는 모두)를 사용하여 유효한 BigchainDB 트랜잭션을 생성합니다.
이는 asset.data 또는 메타 데이터의 내용이 유효성 검증에 중요하지 않은 트랜잭션에 대해서만 수행되므로, 모든 노드 운영자가 트랜잭션을 검증 할 수 있습니다.
특수 노드는 암호화 된 데이터를 해독 할 수 있지만, 다른 노드 운영자와 다른 최종 사용자는 할 수 없습니다.