Chainer – A flexible framework of neural networks¶
This is the Chainer documentation.
Install Guide¶
Before installing Chainer¶
We recommend these platforms.
Chainer is supported on Python 2.7.6+, 3.4.3+, 3.5.1+. Chainer uses C++ compiler such as g++. You need to install it before installing Chainer. This is typical installation method for each platform:
# Ubuntu 14.04
$ aptget install g++
# CentOS 7
$ yum install gccc++
If you use old setuptools
, upgrade it:
$ pip install U setuptools
Install Chainer¶
Chainer depends on these Python packages:
CUDA support
cuDNN support
 cuDNN v2, v3, v4, v5, v5.1
Caffe model support
 Protocol Buffers
 protobuf>=3.0.0 is required for Py3
All these libraries are automatically installed with pip
or setup.py
.
Image dataset is optional
HDF5 serialization is optional
 h5py 2.5.0
Install Chainer from source¶
You can use setup.py
to install Chainer from source:
$ tar zxf chainerx.x.x.tar.gz
$ cd chainerx.x.x
$ python setup.py install
When an error occurs...¶
Use vvvv
option with pip
command.
That shows all logs of installation. It may helps you:
$ pip install chainer vvvv
Install Chainer with CUDA¶
You need to install CUDA Toolkit before installing Chainer.
If you have CUDA in a default directory or set CUDA_PATH
correctly, Chainer installer finds CUDA automatically:
$ pip install chainer
Note
Chainer installer looks up CUDA_PATH
environment variable first.
If it is empty, the installer looks for nvcc
command from PATH
environment variable and use its parent directory as the root directory of CUDA installation.
If nvcc
command is also not found, the installer tries to use the default directory for Ubuntu /usr/local/cuda
.
If you installed CUDA into a nondefault directory, you need to specify the directory with CUDA_PATH
environment variable:
$ CUDA_PATH=/opt/nvidia/cuda pip install chainer
Warning
If you want to use sudo
to install Chainer, note that sudo
command initializes all environment variables.
Please specify CUDA_PATH
environment variable inside sudo
like this:
$ sudo CUDA_PATH=/opt/nvidia/cuda pip install chainer
Install Chainer with CUDA and cuDNN¶
cuDNN is a library for Deep Neural Networks that NVIDIA provides.
Chainer can use cuDNN.
If you want to enable cuDNN, install cuDNN and CUDA before installing Chainer.
We recommend you to install cuDNN to CUDA directory.
For example if you uses Ubuntu Linux, copy .h
files to include
directory and .so
files to lib64
directory:
$ cp /path/to/cudnn.h $CUDA_PATH/include
$ cp /path/to/libcudnn.so* $CUDA_PATH/lib64
The destination directories depend on your environment.
If you want to use cuDNN installed in other directory, please use CFLAGS
, LDFLAGS
and LD_LIBRARY_PATH
environment variables before installing Chainer:
export CFLAGS=I/path/to/cudnn/include
export LDFLAGS=L/path/to/cudnn/lib
export LD_LIBRARY_PATH=/path/to/cudnn/lib:$LD_LIBRARY_PATH
Install Chainer for developers¶
Chainer uses Cython (>=0.24).
Developers need to use Cython to regenerate C++ sources from pyx
files.
We recommend to use pip
with e
option for editable mode:
$ pip install U cython
$ cd /path/to/chainer/source
$ pip install e .
Users need not to install Cython as a distribution package of Chainer only contains generated sources.
Support image dataset¶
Install Pillow manually to activate image dataset. This feature is optional:
$ pip install pillow
Support HDF5 serialization¶
Install h5py manually to activate HDF5 serialization. This feature is optional:
$ pip install h5py
Before installing h5py, you need to install libhdf5. It depends on your environment:
# Ubuntu 14.04
$ aptget install libhdf5dev
# CentOS 7
$ yum y install epelrelease
$ yum install hdf5devel
Uninstall Chainer¶
Use pip to uninstall Chainer:
$ pip uninstall chainer
Note
When you upgrade Chainer, pip
sometimes installed various version of Chainer in sitepackages
.
Please uninstall it repeatedly until pip
returns an error.
Reinstall Chainer¶
If you want to reinstall Chainer, please uninstall Chainer and then install it.
We recommend to use nocachedir
option as pip
sometimes uses cache:
$ pip uninstall chainer
$ pip install chainer nocachedir
When you install Chainer without CUDA, and after that you want to use CUDA, please reinstall Chainer. You need to reinstall Chainer when you want to upgrade CUDA.
Run Chainer with Docker¶
We provide the official Docker image. Use nvidiadocker command to run Chainer image with GPU. You can login to the environment with bash, and run the Python interpreter:
$ nvidiadocker run it chainer/chainer /bin/bash
Or, run the interpreter directly:
$ nvidiadocker run it chainer/chainer /usr/bin/python
What “recommend” means?¶
We tests Chainer automatically with Jenkins. All supported environments are tested in this environment. We cannot guarantee that Chainer works on other environments.
FAQ¶
The installer says “hdf5.h is not found”¶
You don’t have libhdf5. Please install hdf5. See Before installing Chainer.
MemoryError happens¶
You maybe failed to install Cython. Please install it manually. See When an error occurs....
Examples says “cuDNN is not enabled”¶
You failed to build Chainer with cuDNN.
If you don’t need cuDNN, ignore this message.
Otherwise, retry to install Chainer with cuDNN.
vvvv
option helps you.
See Install Chainer with CUDA and cuDNN.
Chainer Tutorial¶
Introduction to Chainer¶
This is the first section of the Chainer Tutorial. In this section, you will learn about the following things:
 Pros and cons of existing frameworks and why we are developing Chainer
 Simple example of forward and backward computation
 Usage of links and their gradient computation
 Construction of chains (a.k.a. “model” in most frameworks)
 Parameter optimization
 Serialization of links and optimizers
After reading this section, you will be able to:
 Compute gradients of some arithmetics
 Write a multilayer perceptron with Chainer
Core Concept¶
As mentioned on the front page, Chainer is a flexible framework for neural networks. One major goal is flexibility, so it must enable us to write complex architectures simply and intuitively.
Most existing deep learning frameworks are based on the “DefineandRun” scheme. That is, first a network is defined and fixed, and then the user periodically feeds it with minibatches. Since the network is statically defined before any forward/backward computation, all the logic must be embedded into the network architecture as data. Consequently, defining a network architecture in such systems (e.g. Caffe) follows a declarative approach. Note that one can still produce such a static network definition using imperative languages (e.g. torch.nn, Theanobased frameworks, and TensorFlow).
In contrast, Chainer adopts a “DefinebyRun” scheme, i.e., the network is defined onthefly via the actual forward computation. More precisely, Chainer stores the history of computation instead of programming logic. This strategy enables us to fully leverage the power of programming logic in Python. For example, Chainer does not need any magic to introduce conditionals and loops into the network definitions. The DefinebyRun scheme is the core concept of Chainer. We will show in this tutorial how to define networks dynamically.
This strategy also makes it easy to write multiGPU parallelization, since logic comes closer to network manipulation. We will review such amenities in later sections of this tutorial.
Note
In the example code of this tutorial, we assume for simplicity that the following symbols are already imported:
import numpy as np
import chainer
from chainer import cuda, Function, gradient_check, report, training, utils, Variable
from chainer import datasets, iterators, optimizers, serializers
from chainer import Link, Chain, ChainList
import chainer.functions as F
import chainer.links as L
from chainer.training import extensions
These imports appear widely in Chainer code and examples. For simplicity, we omit these imports in this tutorial.
Forward/Backward Computation¶
As described above, Chainer uses the “DefinebyRun” scheme, so forward computation itself defines the network.
In order to start forward computation, we have to set the input array to a Variable
object.
Here we start with a simple ndarray
with only one element:
>>> x_data = np.array([5], dtype=np.float32)
>>> x = Variable(x_data)
A Variable object has basic arithmetic operators. In order to compute \(y = x^2  2x + 1\), just write:
>>> y = x**2  2 * x + 1
The resulting y
is also a Variable object, whose value can be extracted by accessing the data
attribute:
>>> y.data
array([ 16.], dtype=float32)
What y
holds is not only the result value.
It also holds the history of computation (or computational graph), which enables us to compute its differentiation.
This is done by calling its backward()
method:
>>> y.backward()
This runs error backpropagation (a.k.a. backprop or reversemode automatic differentiation).
Then, the gradient is computed and stored in the grad
attribute of the input variable x
:
>>> x.grad
array([ 8.], dtype=float32)
Also we can compute gradients of intermediate variables.
Note that Chainer, by default, releases the gradient arrays of intermediate variables for memory efficiency.
In order to preserve gradient information, pass the retain_grad
argument to the backward method:
>>> z = 2*x
>>> y = x**2  z + 1
>>> y.backward(retain_grad=True)
>>> z.grad
array([1.], dtype=float32)
All these computations are easily generalized to a multielement array input.
Note that if we want to start backward computation from a variable holding a multielement array, we must set the initial error manually.
This is done simply by setting the grad
attribute of the output variable:
>>> x = Variable(np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32))
>>> y = x**2  2*x + 1
>>> y.grad = np.ones((2, 3), dtype=np.float32)
>>> y.backward()
>>> x.grad
array([[ 0., 2., 4.],
[ 6., 8., 10.]], dtype=float32)
Links¶
In order to write neural networks, we have to combine functions with parameters and optimize the parameters. You can use links to do this. A link is an object that holds parameters (i.e. optimization targets).
The most fundamental ones are links that behave like regular functions while replacing some arguments by their parameters. We will introduce higher level links, but here think of links as simply functions with parameters.
One of the most frequently used links is the Linear
link (a.k.a. fullyconnected layer or affine transformation).
It represents a mathematical function \(f(x) = Wx + b\), where the matrix \(W\) and the vector \(b\) are parameters.
This link corresponds to its pure counterpart linear()
, which accepts \(x, W, b\) as arguments.
A linear link from threedimensional space to twodimensional space is defined by the following line:
>>> f = L.Linear(3, 2)
Note
Most functions and links only accept minibatch input, where the first dimension of the input array is considered as the batch dimension. In the above Linear link case, input must have shape of (N, 3), where N is the minibatch size.
The parameters of a link are stored as attributes.
Each parameter is an instance of Variable
.
In the case of the Linear link, two parameters, W
and b
, are stored.
By default, the matrix W
is initialized randomly, while the vector b
is initialized with zeros.
>>> f.W.data
array([[ 1.01847613, 0.23103087, 0.56507462],
[ 1.29378033, 1.07823515, 0.56423163]], dtype=float32)
>>> f.b.data
array([ 0., 0.], dtype=float32)
An instance of the Linear link acts like a usual function:
>>> x = Variable(np.array([[1, 2, 3], [4, 5, 6]], dtype=np.float32))
>>> y = f(x)
>>> y.data
array([[ 3.1757617 , 1.75755572],
[ 8.61950684, 7.18090773]], dtype=float32)
Gradients of parameters are computed by the backward()
method.
Note that gradients are accumulated by the method rather than overwritten.
So first you must clear gradients to renew the computation.
It can be done by calling the cleargrads()
method.
>>> f.cleargrads()
Note
cleargrads()
is introduced in v1.15 to replace zerograds()
for efficiency.
zerograds()
is left only for backward compatibility.
Now we can compute the gradients of parameters by simply calling the backward method.
>>> y.grad = np.ones((2, 2), dtype=np.float32)
>>> y.backward()
>>> f.W.grad
array([[ 5., 7., 9.],
[ 5., 7., 9.]], dtype=float32)
>>> f.b.grad
array([ 2., 2.], dtype=float32)
Write a model as a chain¶
Most neural network architectures contain multiple links. For example, a multilayer perceptron consists of multiple linear layers. We can write complex procedures with parameters by combining multiple links like this:
>>> l1 = L.Linear(4, 3)
>>> l2 = L.Linear(3, 2)
>>> def my_forward(x):
... h = l1(x)
... return l2(h)
Here the L
indicates the links
module.
A procedure with parameters defined in this way is hard to reuse.
More Pythonic way is combining the links and procedures into a class:
>>> class MyProc(object):
... def __init__(self):
... self.l1 = L.Linear(4, 3)
... self.l2 = L.Linear(3, 2)
...
... def forward(self, x):
... h = self.l1(x)
... return self.l2(h)
In order to make it more reusable, we want to support parameter management, CPU/GPU migration, robust and flexible save/load features, etc.
These features are all supported by the Chain
class in Chainer.
Then, what we have to do here is just define the above class as a subclass of Chain:
>>> class MyChain(Chain):
... def __init__(self):
... super(MyChain, self).__init__(
... l1=L.Linear(4, 3),
... l2=L.Linear(3, 2),
... )
...
... def __call__(self, x):
... h = self.l1(x)
... return self.l2(h)
Note
We often define a single forward method of a link by __call__
operator.
Such links and chains are callable and behave like regular functions of Variables.
It shows how a complex chain is constructed by simpler links.
Links like l1
and l2
are called child links of MyChain.
Note that Chain itself inherits Link.
It means we can define more complex chains that hold MyChain objects as their child links.
Another way to define a chain is using the ChainList
class, which behaves like a list of links:
>>> class MyChain2(ChainList):
... def __init__(self):
... super(MyChain2, self).__init__(
... L.Linear(4, 3),
... L.Linear(3, 2),
... )
...
... def __call__(self, x):
... h = self[0](x)
... return self[1](h)
ChainList can conveniently use an arbitrary number of links, however if the number of links is fixed like in the above case, the Chain class is recommended as a base class.
Optimizer¶
In order to get good values for parameters, we have to optimize them by the Optimizer
class.
It runs a numerical optimization algorithm on a given link.
Many algorithms are implemented in the optimizers
module.
Here we use the simplest one, called Stochastic Gradient Descent (SGD):
>>> model = MyChain()
>>> optimizer = optimizers.SGD()
>>> optimizer.use_cleargrads()
>>> optimizer.setup(model)
The method use_cleargrads()
is for efficiency. See use_cleargrads()
for detail.
The method setup()
prepares for the optimization given a link.
Some parameter/gradient manipulations, e.g. weight decay and gradient clipping, can be done by setting hook functions to the optimizer. Hook functions are called after the gradient computation and right before the actual update of parameters. For example, we can set weight decay regularization by running the next line beforehand:
>>> optimizer.add_hook(chainer.optimizer.WeightDecay(0.0005))
Of course, you can write your own hook functions. It should be a function or a callable object, taking the optimizer as the argument.
There are two ways to use the optimizer.
One is using it via Trainer
, which we will see in the following sections.
The other way is using it directly.
We here review the latter case.
If you are interested in getting able to use the optimizer in a simple way, skip this section and go to the next one.
There are two further ways to use the optimizer directly.
One is manually computing gradients and then calling the update()
method with no arguments.
Do not forget to clear the gradients beforehand!
>>> x = np.random.uniform(1, 1, (2, 4)).astype('f')
>>> model.cleargrads()
>>> # compute gradient here...
>>> loss = F.sum(model(chainer.Variable(x)))
>>> loss.backward()
>>> optimizer.update()
The other way is just passing a loss function to the update()
method.
In this case, cleargrads()
is automatically called by the update method, so the user does not have to call it manually.
>>> def lossfun(arg1, arg2):
... # calculate loss
... loss = F.sum(model(arg1  arg2))
... return loss
>>> arg1 = np.random.uniform(1, 1, (2, 4)).astype('f')
>>> arg2 = np.random.uniform(1, 1, (2, 4)).astype('f')
>>> optimizer.update(lossfun, chainer.Variable(arg1), chainer.Variable(arg2))
See Optimizer.update()
for the full specification.
Trainer¶
When we want to train neural networks, we have to run training loops that update the parameters many times. A typical training loop consists of the following procedures:
 Iterations over training datasets
 Preprocessing of extracted minibatches
 Forward/backward computations of the neural networks
 Parameter updates
 Evaluations of the current parameters on validation datasets
 Logging and printing of the intermediate results
Chainer provides a simple yet powerful way to make it easy to write such training processes. The training loop abstraction mainly consists of two components:
 Dataset abstraction.
It implements 1 and 2 in the above list.
The core components are defined in the
dataset
module. There are also many implementations of datasets and iterators indatasets
anditerators
modules, respectively.  Trainer.
It implements 3, 4, 5, and 6 in the above list.
The whole procedure is implemented by
Trainer
. The way to update parameters (3 and 4) is defined byUpdater
, which can be freely customized. 5 and 6 are implemented by instances ofExtension
, which appends an extra procedure to the training loop. Users can freely customize the training procedure by adding extensions. Users can also implement their own extensions.
We will see how to use Trainer in the example section below.
Serializer¶
Before proceeding to the first example, we introduce Serializer, which is the last core feature described in this page.
Serializer is a simple interface to serialize or deserialize an object.
Link
, Optimizer
, and Trainer
supports serialization.
Concrete serializers are defined in the serializers
module.
It supports NumPy NPZ and HDF5 formats.
For example, we can serialize a link object into NPZ file by the serializers.save_npz()
function:
>>> serializers.save_npz('my.model', model)
It saves the parameters of model
into the file 'my.model'
in NPZ format.
The saved model can be read by the serializers.load_npz()
function:
>>> serializers.load_npz('my.model', model)
Note
Note that only the parameters and the persistent values are serialized by this serialization code.
Other attributes are not saved automatically.
You can register arrays, scalars, or any serializable objects as persistent values by the Link.add_persistent()
method.
The registered values can be accessed by attributes of the name passed to the add_persistent method.
The state of an optimizer can also be saved by the same functions:
>>> serializers.save_npz('my.state', optimizer)
>>> serializers.load_npz('my.state', optimizer)
Note
Note that serialization of optimizer only saves its internal states including number of iterations, momentum vectors of MomentumSGD, etc. It does not save the parameters and persistent values of the target link. We have to explicitly save the target link with the optimizer to resume the optimization from saved states.
Support of the HDF5 format is enabled if the h5py package is installed.
Serialization and deserialization with the HDF5 format are almost identical to those with the NPZ format;
just replace save_npz()
and load_npz()
by save_hdf5()
and load_hdf5()
, respectively.
Example: Multilayer Perceptron on MNIST¶
Now you can solve a multiclass classification task using a multilayer perceptron (MLP).
We use a handwritten digits dataset called MNIST, which is one of the longstanding de facto “hello world” examples used in machine learning.
This MNIST example is also found in the examples/mnist directory of the official repository.
We show how to use Trainer
to construct and run the training loop in this section.
We first have to prepare the MNIST dataset.
The MNIST dataset consists of 70,000 greyscale images of size 28x28 (i.e. 784 pixels) and corresponding digit labels.
The dataset is divided into 60,000 training images and 10,000 test images by default.
We can obtain the vectorized version (i.e., a set of 784 dimensional vectors) by datasets.get_mnist()
.
>>> train, test = datasets.get_mnist()
...
This code automatically downloads the MNIST dataset and saves the NumPy arrays to the $(HOME)/.chainer
directory.
The returned train
and test
can be seen as lists of imagelabel pairs (strictly speaking, they are instances of TupleDataset
).
We also have to define how to iterate over these datasets.
We want to shuffle the training dataset for every epoch, i.e. at the beginning of every sweep over the dataset.
In this case, we can use iterators.SerialIterator
.
>>> train_iter = iterators.SerialIterator(train, batch_size=100, shuffle=True)
On the other hand, we do not have to shuffle the test dataset.
In this case, we can pass shuffle=False
argument to disable the shuffling.
It makes the iteration faster when the underlying dataset supports fast slicing.
>>> test_iter = iterators.SerialIterator(test, batch_size=100, repeat=False, shuffle=False)
We also pass repeat=False
, which means we stop iteration when all examples are visited.
This option is usually required for the test/validation datasets; without this option, the iteration enters an infinite loop.
Next, we define the architecture. We use a simple threelayer rectifier network with 100 units per layer as an example.
>>> class MLP(Chain):
... def __init__(self, n_units, n_out):
... super(MLP, self).__init__(
... # the size of the inputs to each layer will be inferred
... l1=L.Linear(None, n_units), # n_in > n_units
... l2=L.Linear(None, n_units), # n_units > n_units
... l3=L.Linear(None, n_out), # n_units > n_out
... )
...
... def __call__(self, x):
... h1 = F.relu(self.l1(x))
... h2 = F.relu(self.l2(h1))
... y = self.l3(h2)
... return y
This link uses relu()
as an activation function.
Note that the 'l3'
link is the final linear layer whose output corresponds to scores for the ten digits.
In order to compute loss values or evaluate the accuracy of the predictions, we define a classifier chain on top of the above MLP chain:
>>> class Classifier(Chain):
... def __init__(self, predictor):
... super(Classifier, self).__init__(predictor=predictor)
...
... def __call__(self, x, t):
... y = self.predictor(x)
... loss = F.softmax_cross_entropy(y, t)
... accuracy = F.accuracy(y, t)
... report({'loss': loss, 'accuracy': accuracy}, self)
... return loss
This Classifier class computes accuracy and loss, and returns the loss value.
The pair of arguments x
and t
corresponds to each example in the datasets (a tuple of an image and a label).
softmax_cross_entropy()
computes the loss value given prediction and ground truth labels.
accuracy()
computes the prediction accuracy.
We can set an arbitrary predictor link to an instance of the classifier.
The report()
function reports the loss and accuracy values to the trainer.
For the detailed mechanism of collecting training statistics, see Reporter.
You can also collect other types of observations like activation statistics in a similar ways.
Note that a class similar to the Classifier above is defined as chainer.links.Classifier
.
So instead of using the above example, we will use this predefined Classifier chain.
>>> model = L.Classifier(MLP(100, 10)) # the input size, 784, is inferred
>>> optimizer = optimizers.SGD()
>>> optimizer.setup(model)
Now we can build a trainer object.
>>> updater = training.StandardUpdater(train_iter, optimizer)
>>> trainer = training.Trainer(updater, (20, 'epoch'), out='result')
The second argument (20, 'epoch')
represents the duration of training.
We can use either epoch
or iteration
as the unit.
In this case, we train the multilayer perceptron by iterating over the training set 20 times.
In order to invoke the training loop, we just call the run()
method.
>>> trainer.run()
This method executes the whole training sequence.
The above code just optimizes the parameters.
In most cases, we want to see how the training proceeds, where we can use extensions inserted before calling the run
method.
>>> trainer.extend(extensions.Evaluator(test_iter, model))
>>> trainer.extend(extensions.LogReport())
>>> trainer.extend(extensions.PrintReport(['epoch', 'main/accuracy', 'validation/main/accuracy']))
>>> trainer.extend(extensions.ProgressBar())
>>> trainer.run()
These extensions perform the following tasks:
Evaluator
 Evaluates the current model on the test dataset at the end of every epoch.
LogReport
 Accumulates the reported values and emits them to the log file in the output directory.
PrintReport
 Prints the selected items in the LogReport.
ProgressBar
 Shows the progress bar.
There are many extensions implemented in the chainer.training.extensions
module.
The most important one that is not included above is snapshot()
, which saves the snapshot of the training procedure (i.e., the Trainer object) to a file in the output directory.
The example code in the examples/mnist directory additionally contains GPU support, though the essential part is the same as the code in this tutorial. We will review in later sections how to use GPU(s).
Recurrent Nets and their Computational Graph¶
In this section, you will learn how to write
 recurrent nets with full backprop,
 recurrent nets with truncated backprop,
 evaluation of networks with few memory.
After reading this section, you will be able to:
 Handle input sequences of variable length
 Truncate upper stream of the network during forward computation
 Use volatile variables to prevent network construction
Recurrent Nets¶
Recurrent nets are neural networks with loops. They are often used to learn from sequential input/output. Given an input stream \(x_1, x_2, \dots, x_t, \dots\) and the initial state \(h_0\), a recurrent net iteratively updates its state by \(h_t = f(x_t, h_{t1})\), and at some or every point in time \(t\), it outputs \(y_t = g(h_t)\). If we expand the procedure along the time axis, it looks like a regular feedforward network except that same parameters are repeatedly used within the network.
Here we learn how to write a simple onelayer recurrent net. The task is language modeling: given a finite sequence of words, we want to predict the next word at each position without peeking the successive words. Suppose there are 1,000 different word types, and that we use 100 dimensional real vectors to represent each word (a.k.a. word embedding).
Let’s start from defining the recurrent neural net language model (RNNLM) as a chain.
We can use the chainer.links.LSTM
link that implements a fullyconnected stateful LSTM layer.
This link looks like an ordinary fullyconnected layer.
On construction, you pass the input and output size to the constructor:
>>> l = L.LSTM(100, 50)
Then, call on this instance l(x)
executes one step of LSTM layer:
>>> l.reset_state()
>>> x = Variable(np.random.randn(10, 100).astype(np.float32))
>>> y = l(x)
Do not forget to reset the internal state of the LSTM layer before the forward computation! Every recurrent layer holds its internal state (i.e. the output of the previous call). At the first application of the recurrent layer, you must reset the internal state. Then, the next input can be directly fed to the LSTM instance:
>>> x2 = Variable(np.random.randn(10, 100).astype(np.float32))
>>> y2 = l(x2)
Based on this LSTM link, let’s write our recurrent network as a new chain:
class RNN(Chain):
def __init__(self):
super(RNN, self).__init__(
embed=L.EmbedID(1000, 100), # word embedding
mid=L.LSTM(100, 50), # the first LSTM layer
out=L.Linear(50, 1000), # the feedforward output layer
)
def reset_state(self):
self.mid.reset_state()
def __call__(self, cur_word):
# Given the current word ID, predict the next word.
x = self.embed(cur_word)
h = self.mid(x)
y = self.out(h)
return y
rnn = RNN()
model = L.Classifier(rnn)
optimizer = optimizers.SGD()
optimizer.setup(model)
Here EmbedID
is a link for word embedding.
It converts input integers into corresponding fixeddimensional embedding vectors.
The last linear link out
represents the feedforward output layer.
The RNN
chain implements a onestepforward computation.
It does not handle sequences by itself, but we can use it to process sequences by just feeding items in a sequence straight to the chain.
Suppose we have a list of word variables x_list
.
Then, we can compute loss values for the word sequence by simple for
loop.
def compute_loss(x_list):
loss = 0
for cur_word, next_word in zip(x_list, x_list[1:]):
loss += model(cur_word, next_word)
return loss
Of course, the accumulated loss is a Variable object with the full history of computation.
So we can just call its backward()
method to compute gradients of the total loss according to the model parameters:
# Suppose we have a list of word variables x_list.
rnn.reset_state()
model.cleargrads()
loss = compute_loss(x_list)
loss.backward()
optimizer.update()
Or equivalently we can use the compute_loss
as a loss function:
rnn.reset_state()
optimizer.update(compute_loss, x_list)
Truncate the Graph by Unchaining¶
Learning from very long sequences is also a typical use case of recurrent nets. Suppose the input and state sequence is too long to fit into memory. In such cases, we often truncate the backpropagation into a short time range. This technique is called truncated backprop. It is heuristic, and it makes the gradients biased. However, this technique works well in practice if the time range is long enough.
How to implement truncated backprop in Chainer?
Chainer has a smart mechanism to achieve truncation, called backward unchaining.
It is implemented in the Variable.unchain_backward()
method.
Backward unchaining starts from the Variable object, and it chops the computation history backwards from the variable.
The chopped variables are disposed automatically (if they are not referenced explicitly from any other user object).
As a result, they are no longer a part of computation history, and are not involved in backprop anymore.
Let’s write an example of truncated backprop. Here we use the same network as the one used in the previous subsection. Suppose we are given a very long sequence, and we want to run backprop truncated at every 30 time steps. We can write truncated backprop using the model defined above:
loss = 0
count = 0
seqlen = len(x_list[1:])
rnn.reset_state()
for cur_word, next_word in zip(x_list, x_list[1:]):
loss += model(cur_word, next_word)
count += 1
if count % 30 == 0 or count == seqlen:
model.cleargrads()
loss.backward()
loss.unchain_backward()
optimizer.update()
State is updated at model()
, and the losses are accumulated to loss
variable.
At each 30 steps, backprop takes place at the accumulated loss.
Then, the unchain_backward()
method is called, which deletes the computation history backward from the accumulated loss.
Note that the last state of model
is not lost, since the RNN instance holds a reference to it.
The implementation of truncated backprop is simple, and since there is no complicated trick on it, we can generalize this method to different situations. For example, we can easily extend the above code to use different schedules between backprop timing and truncation length.
Network Evaluation without Storing the Computation History¶
On evaluation of recurrent nets, there is typically no need to store the computation history. While unchaining enables us to walk through unlimited length of sequences with limited memory, it is a bit of a workaround.
As an alternative, Chainer provides an evaluation mode of forward computation which does not store the computation history.
This is enabled by just passing volatile
flag to all input variables.
Such variables are called volatile variables.
Volatile variable is created by passing volatile='on'
at the construction:
x_list = [Variable(..., volatile='on') for _ in range(100)] # list of 100 words
loss = compute_loss(x_list)
Note that we cannot call loss.backward()
to compute the gradient here, since the volatile variable does not remember the computation history.
Volatile variables are also useful to evaluate feedforward networks to reduce the memory footprint.
Variable’s volatility can be changed directly by setting the Variable.volatile
attribute.
This enables us to combine a fixed feature extractor network and a trainable predictor network.
For example, suppose we want to train a feedforward network predictor_func
, which is located on top of another fixed pretrained network fixed_func
.
We want to train predictor_func
without storing the computation history for fixed_func
.
This is simply done by following code snippets (suppose x_data
and y_data
indicate input data and label, respectively):
x = Variable(x_data, volatile='on')
feat = fixed_func(x)
feat.volatile = 'off'
y = predictor_func(feat)
y.backward()
At first, the input variable x
is volatile, so fixed_func
is executed in volatile mode, i.e. without memorizing the computation history.
Then the intermediate variable feat
is manually set to nonvolatile, so predictor_func
is executed in nonvolatile mode, i.e., with memorizing the history of computation.
Since the history of computation is only memorized between variables feat
and y
, the backward computation stops at the feat
variable.
Warning
It is not allowed to mix volatile and nonvolatile variables as arguments to same function.
If you want to create a variable that behaves like a nonvolatile variable while can be mixed with volatile ones, use 'auto'
flag instead of 'off'
flag.
Making it with Trainer¶
The above codes are written with plain Function/Variable APIs. When we write a training loop, it is better to use Trainer, since we can then easily add functionalities by extensions.
Before implementing it on Trainer, let’s clarify the training settings.
We here use Penn Tree Bank dataset as a set of sentences.
Each sentence is represented as a word sequence.
We concatenate all sentences into one long word sequence, in which each sentence is separated by a special word <eos>
, which stands for “End of Sequence”.
This dataset is easily obtained by chainer.datasets.get_ptb_words()
.
This function returns train, validation, and test dataset, each of which is represented as a long array of integers.
Each integer represents a word ID.
Our task is to learn a recurrent neural net language model from the long word sequence. We use words in different locations to form minibatches. It means we maintain \(B\) indices pointing to different locations in the sequence, read from these indices at each iteration, and increment all indices after the read. Of course, when one index reaches the end of the whole sequence, we turn the index back to 0.
In order to implement this training procedure, we have to customize the following components of Trainer:
 Iterator. Builtin iterators do not support reading from different locations and aggregating them into a minibatch.
 Update function. The default update function does not support truncated BPTT.
When we write a dataset iterator dedicated to the dataset, the dataset implementation can be arbitrary; even the interface is not fixed.
On the other hand, the iterator must support the Iterator
interface.
The important methods and attributes to implement are batch_size
, epoch
, epoch_detail
, is_new_epoch
, iteration
, __next__
, and serialize
.
Following is a code from the official example in the examples/ptb
directory.
from __future__ import division
class ParallelSequentialIterator(chainer.dataset.Iterator):
def __init__(self, dataset, batch_size, repeat=True):
self.dataset = dataset
self.batch_size = batch_size
self.epoch = 0
self.is_new_epoch = False
self.repeat = repeat
self.offsets = [i * len(dataset) // batch_size for i in range(batch_size)]
self.iteration = 0
def __next__(self):
length = len(self.dataset)
if not self.repeat and self.iteration * self.batch_size >= length:
raise StopIteration
cur_words = self.get_words()
self.iteration += 1
next_words = self.get_words()
epoch = self.iteration * self.batch_size // length
self.is_new_epoch = self.epoch < epoch
if self.is_new_epoch:
self.epoch = epoch
return list(zip(cur_words, next_words))
@property
def epoch_detail(self):
return self.iteration * self.batch_size / len(self.dataset)
def get_words(self):
return [self.dataset[(offset + self.iteration) % len(self.dataset)]
for offset in self.offsets]
def serialize(self, serializer):
self.iteration = serializer('iteration', self.iteration)
self.epoch = serializer('epoch', self.epoch)
train_iter = ParallelSequentialIterator(train, 20)
val_iter = ParallelSequentialIterator(val, 1, repeat=False)
Although the code is slightly long, the idea is simple.
First, this iterator creates offsets
pointing to positions equally spaced within the whole sequence.
The ith examples of minibatches refer the sequence with the ith offset.
The iterator returns a list of tuples of the current words and the next words.
Each minibatch is converted to a tuple of integer arrays by the concat_examples
function in the standard updater (see the previous tutorial).
Backprop Through Time is implemented as follows.
def update_bptt(updater):
loss = 0
for i in range(35):
batch = train_iter.__next__()
x, t = chainer.dataset.concat_examples(batch)
loss += model(chainer.Variable(x), chainer.Variable(t))
model.cleargrads()
loss.backward()
loss.unchain_backward() # truncate
optimizer.update()
updater = training.StandardUpdater(train_iter, optimizer, update_bptt)
In this case, we update the parameters on every 35 consecutive words.
The call of unchain_backward
cuts the history of computation accumulated to the LSTM links.
The rest of the code for setting up Trainer is almost same as one given in the previous tutorial.
In this section we have demonstrated how to write recurrent nets in Chainer and some fundamental techniques to manage the history of computation (a.k.a. computational graph).
The example in the examples/ptb
directory implements truncated backprop learning of a LSTM language model from the Penn Treebank corpus.
In the next section, we will review how to use GPU(s) in Chainer.
Using GPU(s) in Chainer¶
In this section, you will learn about the following things:
 Relationship between Chainer and CuPy
 Basics of CuPy
 SingleGPU usage of Chainer
 MultiGPU usage of modelparallel computing
 MultiGPU usage of dataparallel computing
After reading this section, you will be able to:
 Use Chainer on a CUDAenabled GPU
 Write modelparallel computing in Chainer
 Write dataparallel computing in Chainer
Relationship between Chainer and CuPy¶
Note
As of the release of v1.3.0, Chainer changes its GPU backend from PyCUDA to CuPy. CuPy covers all features of PyCUDA used by Chainer, though their interfaces are not compatible.
Chainer uses CuPy as its backend for GPU computation.
In particular, the cupy.ndarray
class is the GPU array implementation for Chainer.
CuPy supports a subset of features of NumPy with a compatible interface.
It enables us to write a common code for CPU and GPU.
It also supports PyCUDAlike userdefined kernel generation, which enables us to write fast implementations dedicated to GPU.
Note
The chainer.cuda
module imports many important symbols from CuPy.
For example, the cupy namespace is referred as cuda.cupy
in the Chainer code.
Note that the chainer.cuda
module can be imported even if CUDA is not installed.
Chainer uses a memory pool for GPU memory allocation.
As shown in the previous sections, Chainer constructs and destructs many arrays during learning and evaluating iterations.
It is not well suited for CUDA architecture, since memory allocation and release in CUDA (i.e. cudaMalloc
and cudaFree
functions) synchronize CPU and GPU computations, which hurts performance.
In order to avoid memory allocation and deallocation during the computation, Chainer uses CuPy’s memory pool as the standard memory allocator.
Chainer changes the default allocator of CuPy to the memory pool, so user can use functions of CuPy directly without dealing with the memory allocator.
Basics of cupy.ndarray
¶
Note
CuPy does not require explicit initialization, so cuda.init()
function is deprecated.
CuPy is a GPU array backend that implements a subset of NumPy interface.
The cupy.ndarray
class is in its core, which is a compatible GPU alternative of numpy.ndarray
.
CuPy implements many functions on cupy.ndarray
objects.
See the reference for the supported subset of NumPy API.
Understanding NumPy might help utilizing most features of CuPy.
See the NumPy documentation for learning it.
The main difference of cupy.ndarray
from numpy.ndarray
is that the content is allocated on the device memory.
The allocation takes place on the current device by default.
The current device can be changed by cupy.cuda.Device
object as follows:
with cupy.cuda.Device(1):
x_on_gpu1 = cupy.array([1, 2, 3, 4, 5])
Most operations of CuPy is done on the current device. Be careful that it causes an error to process an array on a noncurrent device.
Chainer provides some convenient functions to automatically switch and choose the device.
For example, the chainer.cuda.to_gpu()
function copies a numpy.ndarray
object to a specified device:
x_cpu = np.ones((5, 4, 3), dtype=np.float32)
x_gpu = cuda.to_gpu(x_cpu, device=1)
It is equivalent to the following code using CuPy:
x_cpu = np.ones((5, 4, 3), dtype=np.float32)
with cupy.cuda.Device(1):
x_gpu = cupy.array(x_cpu)
Moving a device array to the host can be done by chainer.cuda.to_cpu()
as follows:
x_cpu = cuda.to_cpu(x_gpu)
It is equivalent to the following code using CuPy:
with x_gpu.device:
x_cpu = x_gpu.get()
Note
The with statements in these codes are required to select the appropriate CUDA device.
If user uses only one device, these device switching is not needed.
chainer.cuda.to_cpu()
and chainer.cuda.to_gpu()
functions automatically switch the current device correctly.
Chainer also provides a convenient function chainer.cuda.get_device()
to select a device.
It accepts an integer, CuPy array, NumPy array, or None (indicating the current device), and returns an appropriate device object.
If the argument is a NumPy array, then a dummy device object is returned.
The dummy device object supports with statements like above which does nothing.
Here are some examples:
cuda.get_device(1).use()
x_gpu1 = cupy.empty((4, 3), dtype='f') # 'f' indicates float32
with cuda.get_device(1):
x_gpu1 = cuda.empty((4, 3), dtype='f')
with cuda.get_device(x_gpu1):
y_gpu1 = x_gpu + 1
Since it accepts NumPy arrays, we can write a function that accepts both NumPy and CuPy arrays with correct device switching:
def add1(x):
with cuda.get_device(x):
return x + 1
The compatibility of CuPy with NumPy enables us to write CPU/GPU generic code.
It can be made easy by the chainer.cuda.get_array_module()
function.
This function returns the numpy
or cupy
module based on arguments.
A CPU/GPU generic function is defined using it like follows:
# Stable implementation of log(1 + exp(x))
def softplus(x):
xp = cuda.get_array_module(x)
return xp.maximum(0, x) + xp.log1p(xp.exp(abs(x)))
Run Neural Networks on a Single GPU¶
SingleGPU usage is very simple.
What you have to do is transferring Link
and input arrays to the GPU beforehand.
In this subsection, the code is based on our first MNIST example in this tutorial.
A Link
object can be transferred to the specified GPU using the to_gpu()
method.
This time, we make the number of input, hidden, and output units configurable.
The to_gpu()
method also accepts a device ID like model.to_gpu(0)
.
In this case, the link object is transferred to the appropriate GPU device.
The current device is used by default.
If we use chainer.training.Trainer
, what we have to do is just let the updater know the device ID to send each minibatch.
updater = training.StandardUpdater(train_iter, optimizer, device=0)
trainer = training.Trainer(updater, (20, 'epoch'), out='result')
We also have to specify the device ID for an evaluator extension as well.
trainer.extend(extensions.Evaluator(test_iter, model, device=0))
When we write down the training loop by hand, we have to transfer each minibatch to the GPU manually:
model.to_gpu()
batchsize = 100
datasize = len(x_train)
for epoch in range(20):
print('epoch %d' % epoch)
indexes = np.random.permutation(datasize)
for i in range(0, datasize, batchsize):
x = Variable(cuda.to_gpu(x_train[indexes[i : i + batchsize]]))
t = Variable(cuda.to_gpu(y_train[indexes[i : i + batchsize]]))
optimizer.update(model, x, t)
Modelparallel Computation on Multiple GPUs¶
Parallelization of machine learning is roughly classified into two types called “modelparallel” and “dataparallel”. Modelparallel means parallelizations of the computations inside the model. In contrast, dataparallel means parallelizations using data sharding. In this subsection, we show how to use the modelparallel approach on multiple GPUs in Chainer.
Recall the MNIST example. Now suppose that we want to modify this example by expanding the network to 6 layers with 2000 units each using two GPUs. In order to make multiGPU computation efficient, we only make the two GPUs communicate at the third and sixth layer. The overall architecture looks like the following diagram:
(GPU0) input +> l1 > l2 > l3 +> l4 > l5 > l6 +> output
  
(GPU1) +> l1 > l2 > l3 +> l4 > l5 > l6 +
We can use the above MLP chain as following diagram:
(GPU0) input +> mlp1 +> mlp2 +> output
  
(GPU1) +> mlp1 +> mlp2 +
Let’s write a link for the whole network.
class ParallelMLP(Chain):
def __init__(self):
super(ParallelMLP, self).__init__(
# the input size, 784, is inferred
mlp1_gpu0=MLP(1000, 2000).to_gpu(0),
mlp1_gpu1=MLP(1000, 2000).to_gpu(1),
# the input size, 2000, is inferred
mlp2_gpu0=MLP(1000, 10).to_gpu(0),
mlp2_gpu1=MLP(1000, 10).to_gpu(1),
)
def __call__(self, x):
# assume x is on GPU 0
z0 = self.mlp1_gpu0(x)
z1 = self.mlp1_gpu1(F.copy(x, 1))
# sync
h0 = F.relu(z0 + F.copy(z1, 0))
h1 = F.relu(z1 + F.copy(z0, 1))
y0 = self.mlp2_gpu0(h0)
y1 = self.mlp2_gpu1(h1)
# sync
y = y0 + F.copy(y1, 0)
return y # output is on GPU0
Recall that the Link.to_gpu()
method returns the link itself.
The copy()
function copies an input variable to specified GPU device and returns a new variable on the device.
The copy supports backprop, which just reversely transfers an output gradient to the input device.
Note
Above code is not parallelized on CPU, but is parallelized on GPU. This is because all the functions in the above code run asynchronously to the host CPU.
An almost identical example code can be found at examples/mnist/train_mnist_model_parallel.py.
Dataparallel Computation on Multiple GPUs with Trainer¶
Dataparallel computation is another strategy to parallelize online processing. In the context of neural networks, it means that a different device does computation on a different subset of the input data. In this subsection, we review the way to achieve dataparallel learning on two GPUs.
Suppose again our task is the MNIST example. This time we want to directly parallelize the threelayer network. The most simple form of dataparallelization is parallelizing the gradient computation for a distinct set of data. First, define a model and optimizer instances:
model = L.Classifier(MLP(1000, 10)) # the input size, 784, is inferred
optimizer = optimizers.SGD()
optimizer.setup(model)
Recall that the MLP
link implements the multilayer perceptron, and the Classifier
link wraps it to provide a classifier interface.
We used StandardUpdater
in the previous example.
In order to enable dataparallel computation with multiple GPUs, we only have to replace it with ParallelUpdater
.
updater = training.ParallelUpdater(train_iter, optimizer,
devices={'main': 0, 'second': 1})
The devices
option specifies which devices to use in dataparallel learning.
The device with name 'main'
is used as the main device.
The original model is sent to this device, so the optimization runs on the main device.
In the above example, the model is also cloned and sent to GPU 1.
Half of each minibatch is fed to this cloned model.
After every backward computation, the gradient is accumulated into the main device, the parameter update runs on it, and then the updated parameters are sent to GPU 1 again.
See also the example code in examples/mnist/train_mnist_data_parallel.py.
Dataparallel Computation on Multiple GPUs without Trainer¶
We here introduce a way to write dataparallel computation without the help of Trainer
.
Most users can skip this section.
If you are interested in how to write a dataparallel computation by yourself, this section should be informative.
It is also helpful to, e.g., customize the ParallelUpdater
class.
We again start from the MNIST example.
At this time, we use a suffix like _0
and _1
to distinguish objects on each device.
First, we define a model.
model_0 = L.Classifier(MLP(1000, 10)) # the input size, 784, is inferred
We want to make two copies of this instance on different GPUs.
The Link.to_gpu()
method runs in place, so we cannot use it to make a copy.
In order to make a copy, we can use Link.copy()
method.
import copy
model_1 = copy.deepcopy(model_0)
model_0.to_gpu(0)
model_1.to_gpu(1)
The Link.copy()
method copies the link into another instance.
It just copies the link hierarchy, and does not copy the arrays it holds.
Then, set up an optimizer:
optimizer = optimizers.SGD()
optimizer.setup(model_0)
Here we use the first copy of the model as the master model.
Before its update, gradients of model_1
must be aggregated to those of model_0
.
Then, we can write a dataparallel learning loop as follows:
batchsize = 100
datasize = len(x_train)
for epoch in range(20):
print('epoch %d' % epoch)
indexes = np.random.permutation(datasize)
for i in range(0, datasize, batchsize):
x_batch = x_train[indexes[i : i + batchsize]]
y_batch = y_train[indexes[i : i + batchsize]]
x0 = Variable(cuda.to_gpu(x_batch[:batchsize//2], 0))
t0 = Variable(cuda.to_gpu(y_batch[:batchsize//2], 0))
x1 = Variable(cuda.to_gpu(x_batch[batchsize//2:], 1))
t1 = Variable(cuda.to_gpu(y_batch[batchsize//2:], 1))
loss_0 = model_0(x0, t0)
loss_1 = model_1(x1, t1)
model_0.cleargrads()
model_1.cleargrads()
loss_0.backward()
loss_1.backward()
model_0.addgrads(model_1)
optimizer.update()
model_1.copyparams(model_0)
Do not forget to clear the gradients of both model copies!
One half of the minibatch is forwarded to GPU 0, the other half to GPU 1.
Then the gradients are accumulated by the Link.addgrads()
method.
This method adds the gradients of a given link to those of the self.
After the gradients are prepared, we can update the optimizer in usual way.
Note that the update only modifies the parameters of model_0
.
So we must manually copy them to model_1
using Link.copyparams()
method.
Note
If the batch size used in one model remain the same, the scale of the gradient
is roughly proportional to the number of models, when we aggregate
gradients from all models by chainer.Link.addgrads()
. So you need to adjust the batch size
and/or learning rate of the optimizer accordingly.
Now you can use Chainer with GPUs.
All examples in the examples
directory support GPU computation, so please refer to them if you want to know more practices on using GPUs.
In the next section, we will show how to define a differentiable (i.e. backpropable) function on Variable objects.
We will also show there how to write a simple (elementwise) CUDA kernel using Chainer’s CUDA utilities.
Define your own function¶
In this section, you will learn about the following things:
 How to define a function on variables
 Useful tools to write a function using a GPU
 How to test the function definition
After reading this section, you will be able to:
 Write your own functions
 Define simple kernels in the function definition
Differentiable Functions¶
Chainer provides a collection of functions in the functions
module.
It covers typical use cases in deep learning, so many existing works can be implemented with them.
On the other hand, deep learning is evolving rapidly and we cannot cover all possible functions to define unseen architectures.
So it is important to learn how to define your own functions.
First, suppose we want to define an elementwise function \(f(x, y, z) = x * y + z\).
While it is possible to implement this equation using a combination of the *
and +
functions,
defining it as a single function may reduce memory consumption, so it is not only a toy example.
Here we call this function MulAdd.
Let’s start with defining MulAdd working on the CPU.
Any function must inherit the Function
class.
The skeleton of a function looks like:
class MulAdd(Function):
def forward_cpu(self, inputs):
# do forward computation on CPU
return some_tuple
def backward_cpu(self, inputs, grad_outputs):
# do backward computation on CPU
return some_tuple
We must implement forward_cpu()
and backward_cpu()
methods.
The nonself arguments of these functions are tuples of array(s), and these functions must return a tuple of array(s).
Warning
Be careful to return a tuple of arrays even if you have just one array to return.
MulAdd is simple and implemented as follows
class MulAdd(Function):
def forward_cpu(self, inputs):
x, y, z = inputs
w = x * y + z
return w,
def backward_cpu(self, inputs, grad_outputs):
x, y, z = inputs
gw, = grad_outputs
gx = y * gw
gy = x * gw
gz = gw
return gx, gy, gz
As per the warning above, the forward_cpu
method returns a tuple of single element.
Note that all arrays appearing in CPU functions are numpy.ndarray
.
The forward function is straightforward:
It unpacks the input tuple, computes the output, and packs it into a tuple.
The backward function is a bit more complicated.
Recall the rule of differentiation of multiplication.
This example just implements the rule.
Look at the return values, the function just packs the gradient of each input in same order and returns them.
By just defining the core computation of forward and backward, Function class provides a chaining logic on it (i.e. storing the history of computation, etc.).
Note
Assuming we implement a (forward) function \(y=f(x)\) which takes as input the
vector \(x \in \mathbb{R}^n\) and produces as output a vector
\(y \in \mathbb{R}^m\). Then the backward
method has to compute
where \(\gamma\) is the grad_outputs
. Note, that the
resulting vector \(\lambda\) must have the same shape as the arguments of the forward
method.
Now let’s define the corresponding GPU methods.
You can easily predict that the methods we have to write are named forward_gpu()
and backward_gpu()
:
class MulAdd(Function):
def forward_cpu(self, inputs):
...
def backward_cpu(self, inputs, grad_outputs):
...
def forward_gpu(self, inputs):
x, y, z = inputs
w = x * y + z
return w,
def backward_gpu(self, inputs, grad_outputs):
x, y, z = inputs
gw, = grad_outputs
gx = y * gw
gy = x * gw
gz = gw
return gx, gy, gz
In GPU methods, arrays are of type cupy.ndarray
.
We use arithmetic operators defined for this class.
These operators implement the basic elementwise arithmetics.
You may find that the definitions of GPU methods are exactly same as those of CPU methods.
In that case, we can reduce them to forward()
and backward()
methods
class MulAdd(Function):
def forward(self, inputs):
x, y, z = inputs
w = x * y + z
return w,
def backward(self, inputs, grad_outputs):
x, y, z = inputs
gw, = grad_outputs
gx = y * gw
gy = x * gw
gz = gw
return gx, gy, gz
Since the cupy.ndarray
class implements many methods of numpy.ndarray
, we can write these unified methods in most cases.
The MulAdd function is used as follows:
x = Variable(np.random.uniform(1, 1, (3, 2)).astype(np.float32))
y = Variable(np.random.uniform(1, 1, (3, 2)).astype(np.float32))
z = Variable(np.random.uniform(1, 1, (3, 2)).astype(np.float32))
w = MulAdd()(x, y, z)
It looks a bit ugly: we have to explicitly instantiate MulAdd before applying it to variables. We also have to be careful that one instance of MulAdd must not be used multiple times, since it acts as a node in the computational graph. In Chainer, we often define a thin wrapper Python function that hide the instantiation:
def muladd(x, y, z):
return MulAdd()(x, y, z)
w = muladd(x, y, z)
Unified forward/backward methods with NumPy/CuPy functions¶
CuPy also implements many functions that are compatible to those of NumPy. We can write unified forward/backward methods with them. Consider that we want to write a backpropable function \(f(x, y) = \exp(x) + \exp(y)\). We name it ExpAdd here. It can be written straightforward as follows
class ExpAdd(Function):
def forward_cpu(self, inputs):
x, y = inputs
z = np.exp(x) + np.exp(y)
return z,
def backward_cpu(self, inputs, grad_outputs):
x, y = inputs
gz, = grad_outputs
gx = gz * np.exp(x)
gy = gz * np.exp(y)
return gx, gy
def forward_gpu(self, inputs):
cupy = cuda.cupy
x, y = inputs
z = cupy.exp(x) + cupy.exp(y)
return z,
def backward_gpu(self, inputs, grad_outputs):
cupy = cuda.cupy
x, y = inputs
gz, = grad_outputs
gx = gz * cupy.exp(x)
gy = gz * cupy.exp(y)
return gx, gy
def expadd(x, y):
return ExpAdd()(x, y)
Note
Here we used cuda.cupy
instead of directly accessing cupy
.
This is because the cupy
module cannot be imported if the CUDA is not installed.
In order to keep the implementation valid in nonCUDA environment, we have to defer the access to the cupy
module.
Note that the chainer.cuda
module can be imported even if the CUDA is not installed.
Of course, the module in such environment is almost useless, but if the interpreter does not run through the code accessing CUDAdedicated functions, the code is still valid.
The CPU and GPU implementations are almost same, except that numpy
is replaced by cupy
in GPU methods.
We can unify these functions using the cuda.get_array_module()
function.
This function accepts arbitrary number of arrays, and returns an appropriate module for them.
See the following code
class ExpAdd(Function):
def forward(self, inputs):
xp = cuda.get_array_module(*inputs)
x, y = inputs
z = xp.exp(x) + xp.exp(y)
return z,
def backward(self, inputs, grad_outputs):
xp = cuda.get_array_module(*inputs)
x, y = inputs
gz, = grad_outputs
gx = gz * xp.exp(x)
gy = gz * xp.exp(y)
return gx, gy
def expadd(x, y):
return ExpAdd()(x, y)
Note that this code works correctly even if CUDA is not installed in the environment.
If CUDA is not found, get_array_module function always returns numpy
.
We often use the name xp
for the variadic module name, which is analogous to the abbreviation np
for NumPy and cp
for CuPy.
Write an Elementwise Kernel Function¶
Let’s turn back to the MulAdd example.
The GPU implementation of MulAdd as shown above is already fast and parallelized on GPU cores. However, it invokes two kernels during each of forward and backward computations. It might hurt performance, since the intermediate temporary arrays are read and written by possibly different GPU cores, which consumes much bandwidth. We can reduce the number of invocations by defining our own kernel. It also reduce the memory consumption.
Most functions only require elementwise operations like MulAdd.
CuPy provides a useful tool to define elementwise kernels, the cupy.elementwise.ElementwiseKernel
class, and Chainer wraps it by cuda.elementwise()
function.
Our MulAdd implementation can be improved as follows:
class MulAdd(Function):
def forward_cpu(self, inputs):
...
def backward_cpu(self, inputs, grad_outputs):
...
def forward_gpu(self, inputs):
cupy = cuda.cupy
x, y, z = inputs
w = cuda.elementwise(
'float32 x, float32 y, float32 z',
'float32 w',
'w = x * y + z',
'muladd_fwd')(x, y, z)
return w,
def backward_gpu(self, inputs, grad_outputs):
x, y, z = inputs
gw, = grad_outputs
gx, gy = cuda.elementwise(
'float32 x, float32 y, float32 gw',
'float32 gx, float32 gy',
'''
gx = y * gw;
gy = x * gw;
''',
'muladd_bwd')(x, y, gw)
gz = gw
return gx, gy, gz
cuda.elementwise()
function accepts the essential implementation of the kernel function, and returns a kernel invocation function (actually, it returns ElementwiseKernel
object, which is callable).
In typical usage, we pass four arguments to this function as follows:
 Input argument list. This is a commaseparated string each entry of which consists of a type specification and an argument name.
 Output argument list in the same format as the input argument list.
 Body of parallel loop. We can use the input/output argument names as an element of these arrays.
 Name of the kernel function, which is shown in debuggers and profilers.
Above code is not compiled on every forward/backward computation thanks to two caching mechanisms provided by cuda.elementwise()
.
The first one is binary caching:
cuda.elementwise()
function caches the compiled binary in the $(HOME)/.cupy/kernel_cache
directory with a hash value of the CUDA code, and reuses it if the given code matches the hash value.
This caching mechanism is actually implemented in CuPy.
The second one is upload caching:
Given a compiled binary code, we have to upload it to the current GPU in order to execute it.
cuda.elementwise()
function memoizes the arguments and the current device, and if it is called with the same arguments for the same device, it reuses the previously uploaded kernel code.
The above MulAdd code only works for float32 arrays.
The ElementwiseKernel
also supports the typevariadic kernel definition.
In order to define variadic kernel functions, you can use type placeholder by placing a single character as type specifier:
class MulAdd(Function):
def forward_cpu(self, inputs):
...
def backward_cpu(self, inputs, grad_outputs):
...
def forward_gpu(self, inputs):
cupy = cuda.cupy
x, y, z = inputs
w = cuda.elementwise(
'T x, T y, T z',
'T w',
'w = x * y + z',
'muladd_fwd')(x, y, z)
return w,
def backward_gpu(self, inputs, grad_outputs):
x, y, z = inputs
gw, = grad_outputs
gx, gy = cuda.elementwise(
'T x, T y, T gw',
'T gx, T gy',
'''
gx = y * gw;
gy = x * gw;
''',
'muladd_bwd')(x, y, gw)
gz = gw
return gx, gy, gz
The type placeholder T
indicates an arbitrary data type that CuPy supports.
There are more functionalities on userdefined kernels in CuPy. See the CuPy documentation on userdefined kernels for more details.
Links that wrap functions¶
Some functions are meant to be combined with parameters.
In such case, it is useful to write a small link that wraps the function.
We have already seen how to define a chain that wraps other links (by inheriting Chain
class).
Here we study how to define a link that does not hold any other links.
As the first example, suppose that we want to implement elementwise product function between the input array and the parameter array. It can be defined as follows:
class EltwiseParamProduct(Link):
def __init__(self, shape):
# By passing a shape of the parameter, the initializer allocates a
# parameter variable of the shape.
super(EltwiseParamProduct, self).__init__(W=shape)
self.W.data[...] = np.random.randn(*shape)
def __call__(self, x):
return self.W * x
We can also initialize the parameter after the initialization by the Link.add_param()
method.
class EltwiseParamProduct(Link):
def __init__(self, shape):
super(EltwiseParamProduct, self).__init__()
self.add_param('W', shape)
self.W.data[...] = np.random.randn(*shape)
def __call__(self, x):
return self.W * x
Note that the initializer and the add_param()
method does not initialize elements of the parameter array.
We have to manually initialize the elements by random values, zeros, etc.
For another example, assume we want to define a simple linear layer.
It is already defined as Linear
, so this is an educational example.
The linear layer is divided into two parts: a function and its wrapper link.
First, we have to define a function on variables:
class LinearFunction(Function):
def forward(self, inputs):
x, W, b = inputs
return x.dot(W.T) + b,
def backward(self, inputs, grad_outputs):
x, W, b = inputs
gy, = grad_outputs
gx = gy.dot(W)
gW = gy.T.dot(x)
gb = gy.sum(axis=0)
return gx, gW, gb
def linear(x, W, b):
return LinearFunction()(x, W, b)
This function takes three arguments: input, weight, and bias. It can be used as a part of model definition, though is inconvenient since the user have to manage the weight and bias parameters directly. In order to make a convenient module, let’s wrap it into a link:
class Linear(Link):
def __init__(self, in_size, out_size):
super(Linear, self).__init__(W=(out_size, in_size), b=out_size)
self.W.data[...] = np.random.randn(out_size, in_size) / math.sqrt(in_size)
self.b.data.fill(0)
def __call__(self, x):
return linear(x, self.W, self.b)
This link hides the parameters of the linear layer.
Note
An advanced tip to implement functions: if you want to preserve some information between forward and backward computations (e.g. to cache some arrays), you can store it as attributes. Be careful that it might increase the memory consumption during the whole forwardbackward computation. If you want to train very large networks on a GPU with limited memory, it is not recommended to cache arrays between forward and backward. There is one exception for this: caching the output arrays does not change the memory consumption, because they are also held by the output Variable objects.
Warning
You should not assume a onetoone match of calls of forward and backward. Some users may call backward more than once after one forward call.
Testing Function¶
In order to isolate the cause of learning failure from implementation bugs, it is important to test function implementations.
Chainer provides simple utilities to help writing unit tests.
They are defined in the gradient_check
module.
The most important test utility is the numerical_grad()
function.
This function computes the numerical gradient of given function using finite differences.
It can be used as follows
x = np.random.randn(4, 3).astype(np.float32)
gy = np.ones((4, 3), dtype=np.float32)
f = lambda: (x * x,)
gx = gradient_check.numerical_grad(f, (x,), (gy,))
f
is a closure that returns a tuple of array(s) computed from input arrays.
The second and third arguments of numerical_grad()
are tuples of input arrays and output gradient arrays, respectively.
The code above computes the numerical gradients of sum(f(x))
, where sum
indicates the summation over all elements.
The summation can be weighted by changing gy
.
numerical_grad()
function also accepts additional eps
argument, which indicates the quantization width of finite differences.
Note
numerical_grad()
function accepts both CPU and GPU arrays.
Note that we cannot mix CPU and GPU arrays.
Another utility is chainer.testing.assert_allclose()
function.
This is similar to numpy.testing.assert_allclose()
function.
The difference is that Chainer’s version accepts CPU and GPU arrays as inputs.
We can mix them in one invocation of chainer.testing.assert_allclose()
.
The default values of optional arguments are also different.
Here is a typical usage of gradient checking utilities.
This is a test example of functions.relu()
function
import unittest
from chainer import testing
class TestReLU(unittest.TestCase):
def test_backward_cpu(self):
x = Variable(np.random.randn(3, 2).astype(np.float32))
y = F.relu(x)
y.grad = np.random.randn(3, 2).astype(np.float32)
y.backward()
f = lambda: (F.relu(x).data,)
gx, = gradient_check.numerical_grad(f, (x.data,), (y.grad,))
testing.assert_allclose(gx, x.grad)
The first four lines of the test code are simple forward and backward computation of ReLU function. The next two lines compute numerical gradient using the same forward function without backward routine. And at last, we compare these two results elementwise. Note that the above test code can be easily modified to test GPU version just by replacing CPU arrays to GPU arrays.
You can find many examples of function tests under tests/chainer_tests/function_tests
directory.
Type check¶
In this section, you will learn about the following things:
 Basic usage of type check
 Detail of type information
 Internal mechanism of type check
 More complicated cases
 Call functions
 Typical type check example
After reading this section, you will be able to:
 Write a code to check types of input arguments of your own functions
Basic usage of type check¶
When you call a function with an invalid type of array, you sometimes receive no error, but get an unexpected result by broadcasting. When you use CUDA with an illegal type of array, it causes memory corruption, and you get a serious error. These bugs are hard to fix. Chainer can check preconditions of each function, and helps to prevent such problems. These conditions may help a user to understand specification of functions.
Each implementation of Function
has a method for type check, check_type_forward()
.
This function is called just before the forward()
method of the Function
class.
You can override this method to check the condition on types and shapes of arguments.
check_type_forward()
gets an argument in_types
:
def check_type_forward(self, in_types):
...
in_types
is an instance of TypeInfoTuple
, which is a subclass of tuple
.
To get type information about the first argument, use in_types[0]
.
If the function gets multiple arguments, we recommend to use new variables for readability:
x_type, y_type = in_types
In this case, x_type
represents the type of the first argument, and y_type
represents the second one.
We describe usage of in_types
with an example.
When you want to check if the number of dimension of x_type
equals to 2
, write this code:
utils.type_check.expect(x_type.ndim == 2)
When this condition is true, nothing happens. Otherwise this code throws an exception, and the user gets a message like this:
Traceback (most recent call last):
...
InvalidType: Expect: in_types[0].ndim == 2
Actual: 3 != 2
This error message means that “ndim
of the first argument expected to be 2
, but actually it is 3
”.
Detail of type information¶
You can access three information of x_type
.
.shape
is a tuple of ints. Each value is size of each dimension..ndim
isint
value representing the number of dimensions. Note thatndim == len(shape)
.dtype
isnumpy.dtype
representing data type of the value.
You can check all members. For example, the size of the first dimension must be positive, you can write like this:
utils.type_check.expect(x_type.shape[0] > 0)
You can also check data types with .dtype
:
utils.type_check.expect(x_type.dtype == np.float64)
And an error is like this:
Traceback (most recent call last):
...
InvalidType: Expect: in_types[0].dtype == <type 'numpy.float64'>
Actual: float32 != <type 'numpy.float64'>
You can also check kind
of dtype
.
This code checks if the type is floating point
utils.type_check.expect(x_type.dtype.kind == 'f')
You can compare between variables. For example, the following code checks if the first argument and the second argument have the same length:
utils.type_check.expect(x_type.shape[1] == y_type.shape[1])
Internal mechanism of type check¶
How does it show an error message like "in_types[0].ndim == 2"
?
If x_type
is an object containing ndim
member variable, we cannot show such an error message because this equation is evaluated as a boolean value by Python interpreter.
Actually x_type
is a Expr
objects, and doesn’t have a ndim
member variable itself.
Expr
represents a syntax tree.
x_type.ndim
makes a Expr
object representing (getattr, x_type, 'ndim')
.
x_type.ndim == 2
makes an object like (eq, (getattr, x_type, 'ndim'), 2)
.
type_check.expect()
gets a Expr
object and evaluates it.
When it is True
, it causes no error and shows nothing.
Otherwise, this method shows a readable error message.
If you want to evaluate a Expr
object, call eval()
method:
actual_type = x_type.eval()
actual_type
is an instance of TypeInfo
, while x_type
is an instance of Expr
.
In the same way, x_type.shape[0].eval()
returns an int value.
More powerful methods¶
Expr
class is more powerful.
It supports all mathematical operators such as +
and *
.
You can write a condition that the first dimension of x_type
is the first dimension of y_type
times four:
utils.type_check.expect(x_type.shape[0] == y_type.shape[0] * 4)
When x_type.shape[0] == 3
and y_type.shape[0] == 1
, users can get the error message below:
Traceback (most recent call last):
...
InvalidType: Expect: in_types[0].shape[0] == in_types[1].shape[0] * 4
Actual: 3 != 4
To compare a member variable of your function, wrap a value with Variable
to show readable error message:
x_type.shape[0] == utils.type_check.Variable(self.in_size, "in_size")
This code can check the equivalent condition below:
x_type.shape[0] == self.in_size
However, the latter condition doesn’t know the meaning of this value. When this condition is not satisfied, the latter code shows unreadable error message:
InvalidType: Expect: in_types[0].shape[0] == 4 # what does '4' mean?
Actual: 3 != 4
Note that the second argument of utils.type_check.Variable
is only for readability.
The former shows this message:
InvalidType: Expect: in_types[0].shape[0] == in_size # OK, `in_size` is a value that is given to the constructor
Actual: 3 != 4 # You can also check actual value here
Call functions¶
How to check summation of all values of shape?
Expr
also supports function call:
sum = utils.type_check.Variable(np.sum, 'sum')
utils.type_check.expect(sum(x_type.shape) == 10)
Why do we need to wrap the function numpy.sum
with utils.type_check.Variable
?
x_type.shape
is not a tuple but an object of Expr
as we have seen before.
Therefore, numpy.sum(x_type.shape)
fails.
We need to evaluate this function lazily.
The above example produces an error message like this:
Traceback (most recent call last):
...
InvalidType: Expect: sum(in_types[0].shape) == 10
Actual: 7 != 10
More complicated cases¶
How to write a more complicated condition that can’t be written with these operators?
You can evaluate Expr
and get its result value with eval()
method.
Then check the condition and show warning message by hand:
x_shape = x_type.shape.eval() # get actual shape (int tuple)
if not more_complicated_condition(x_shape):
expect_msg = 'Shape is expected to be ...'
actual_msg = 'Shape is ...'
raise utils.type_check.InvalidType(expect_msg, actual_msg)
Please write a readable error message. This code generates the following error message:
Traceback (most recent call last):
...
InvalidType: Expect: Shape is expected to be ...
Actual: Shape is ...
Typical type check example¶
We show a typical type check for a function.
First check the number of arguments:
utils.type_check.expect(in_types.size() == 2)
in_types.size()
returns a Expr
object representing the number of arguments.
You can check it in the same way.
And then, get each type:
x_type, y_type = in_types
Don’t get each value before checking in_types.size()
.
When the number of argument is illegal, type_check.expect
might output unuseful error messages.
For example, this code doesn’t work when the size of in_types
is 0:
utils.type_check.expect(
in_types.size() == 2,
in_types[0].ndim == 3,
)
After that, check each type:
utils.type_check.expect(
x_type.dtype == np.float32,
x_type.ndim == 3,
x_type.shape[1] == 2,
)
The above example works correctly even when x_type.ndim == 0
as all conditions are evaluated lazily.
Chainer Reference Manual¶
Core functionalities¶
Variable¶

class
chainer.
Variable
(data, volatile=OFF, name=None, grad=None)[source]¶ Array with a structure to keep track of computation.
Every variable holds a data array of type either
numpy.ndarray
orcupy.ndarray
.A Variable object may be constructed in two ways: by the user or by some function. When a variable is created by some function as one of its outputs, the variable holds a reference to that function. This reference is used in error backpropagation (a.k.a. backprop). It is also used in backward unchaining. A variable that does not hold a reference to its creator is called a root variable. A variable is root if it is created by the user, or if the reference is deleted by
unchain_backward()
.Users can disable this chaining behavior by setting the volatile flag for the initial variables. When a function gets volatile variables as its inputs, the output variables do not hold references to the function. This acts like unchaining on every function application.
Parameters: Variables:  data – Data array of type either
numpy.ndarray
orcupy.ndarray
.  grad – Gradient array.
 creator – The function who creates this variable. It is
None
if the variable is not created by any function.  volatile – Ternary
Flag
object. If'ON'
, the variable does not keep track of any function applications. SeeFlag
for the detail of ternary flags.

__getitem__
(x, slices)¶ Extract elements from array with specified shape, axes and offsets.
Parameters:  x (tuple of Variables) – Variable to be sliced.
 slices (int, slice, None or Ellipsis or tuple of them) – Basic slicing
to slice a variable. It supports
int
,slice
,newaxis
(equivalent toNone
) andEllipsis
.
Returns: Variable
object which contains sliced array ofx
.Return type: Note
See NumPy document for details of indexing.

__len__
()[source]¶ Returns the number of elements of the data array.
Returns: Number of elements of the data array. Return type: int

addgrad
(var)[source]¶ Accumulates the gradient array from given source variable.
This method just runs
self.grad += var.grad
, except that the accumulation is even done across the host and different devices.Parameters: var (Variable) – Source variable.

backward
(retain_grad=False)[source]¶ Runs error backpropagation (a.k.a. backprop) from this variable.
On backprop,
Function.backward()
is called on eachFunction
object appearing in the backward graph starting from this variable. The backward graph is represented by backward references from variables to their creators, and from functions to their inputs. The backprop stops at all root variables. Some functions setNone
as gradients of some inputs, where further backprop does not take place at such input variables.This method uses
grad
as the initial error array. User can manually set a gradient array before calling this method. Ifdata
contains only one element (i.e., it is scalar) andgrad
isNone
, then this method automatically complements 1.0 as the initial error. This is useful on starting backprop from some scalar loss value.Parameters: retain_grad (bool) – If
True
, the gradient arrays of all intermediate variables are kept. Otherwise,grad
of the intermediate variables are set toNone
on appropriate timing, which may reduce the maximum memory consumption.In most cases of training some models, the purpose of backprop is to compute gradients of parameters, not of variables, so it is recommended to set this flag
False
.

copydata
(var)[source]¶ Copies the data array from given source variable.
This method just copies the data attribute from given variable to this variable, except that the copy is even done across the host and different devices.
Parameters: var (Variable) – Source variable.

label
¶ Short text that represents the variable.

set_creator
(gen_func)[source]¶ Notifies the variable that the given function is its creator.
Parameters: gen_func (Function) – Function object that creates this variable as one of its outputs.

to_gpu
(device=None)[source]¶ Copies the data and gradient arrays to specified GPU.
Parameters: device – Target device specifier. If omitted, the current device is used.

unchain_backward
()[source]¶ Deletes references between variables and functions backward.
After this method completes, intermediate variables and functions that are not referenced from anywhere are deallocated by reference count GC. Also this variable itself deletes the reference to its creator function, i.e. this variable becomes root in the computation graph. It indicates that backprop after unchaining stops at this variable. This behavior is useful to implement truncated BPTT.

zerograd
()[source]¶ Initializes the gradient array by zeros.
Deprecated since version v1.15: Use
cleargrad()
instead.
 data – Data array of type either
Flag¶

class
chainer.
Flag
[source]¶ Ternary flag object for variables.
It takes three values: ON, OFF, and AUTO.
ON and OFF flag can be evaluated as a boolean value. These are converted to True and False, respectively. AUTO flag cannot be converted to boolean. In this case, ValueError is raised.
Parameters: name (str, bool, or None) – Name of the flag. Following values are allowed:
'on'
,'ON'
, orTrue
for ON value'off'
,'OFF'
, orFalse
for OFF value'auto'
,'AUTO'
, orNone
for AUTO value

chainer.
ON
= ON¶ Equivalent to Flag(‘on’).

chainer.
OFF
= OFF¶ Equivalent to Flag(‘off’).

chainer.
AUTO
= AUTO¶ Equivalent to Flag(‘auto’).

chainer.flag.
aggregate_flags
(flags)[source]¶ Returns an aggregated flag given a sequence of flags.
If both ON and OFF are found, this function raises an error. Otherwise, either of ON and OFF that appeared is returned. If all flags are AUTO, then it returns AUTO.
Parameters: flags (sequence of Flag) – Input flags. Returns: The result of aggregation. Return type: Flag
Function¶

class
chainer.
Function
[source]¶ Function on variables with backpropagation ability.
All function implementations defined in
chainer.functions
inherit this class.The main feature of this class is keeping track of function applications as a backward graph. When a function is applied to
Variable
objects, itsforward()
method is called ondata
fields of input variables, and at the same time it chains references from output variables to the function and from the function to its inputs.Note
As of v1.5, a function instance cannot be used twice in any computational graphs. In order to reuse a function object multiple times, use
copy.copy()
before the function applications to make a copy of the instance.This restriction also means that we cannot make a stateful function anymore. For example, it is now not allowed to let a function hold parameters. Define a function as a pure (stateless) procedure, and use
Link
to combine it with parameter variables.Example
Let
x
an instance ofVariable
andf
an instance ofFunction
taking only one argument. Then a line>>> import numpy, chainer, chainer.functions as F >>> x = chainer.Variable(numpy.zeros(10)) >>> f = F.Identity() >>> y = f(x)
computes a new variable
y
and creates backward references. Actually, backward references are set as per the following diagram:x < f < y
If an application of another function
g
occurs as>>> g = F.Identity() >>> z = g(x)
then the graph grows with a branch:
 f < y x <+  g < z
Note that the branching is correctly managed on backward computation, i.e. the gradients from
f
andg
are accumulated to the gradient ofx
.Every function implementation should provide
forward_cpu()
,forward_gpu()
,backward_cpu()
andbackward_gpu()
. Alternatively, one can provideforward()
andbackward()
instead of separate methods. Backward methods have default implementations that just returnNone
, which indicates that the function is non differentiable.Variables:  inputs – A tuple or list of input variables.
 outputs – A tuple or list of output variables.
 type_check_enable – When it is
True
, the function checks types of input arguments. SetCHAINER_TYPE_CHECK
environment variable0
to disable type check, or set the variable directly in your own program.

__call__
(*inputs)[source]¶ Applies forward propagation with chaining backward references.
Basic behavior is expressed in documentation of
Function
class.Note
If the
data
attribute of input variables exist on GPU device, then, before it callsforward()
method, the appropriate device is selected, so in most cases implementers do not need to take care of device selection.Parameters: inputs – Tuple of input Variable
,numpy.ndarray
orcupy.ndarray
objects. The volatile flags of all input variables must agree. If the input is annumpy.ndarray
or acupy.ndarray
, it is automatically wrapped withVariable
.Returns: One Variable
object or a tuple of multipleVariable
objects.

add_hook
(hook, name=None)[source]¶ Registers the function hook.
Parameters:  hook (FunctionHook) – Function hook to be registered.
 name (str) – Name of the function hook.
name must be unique among function hooks
registered to the function. If
None
, default name of the function hook is used.

backward
(inputs, grad_outputs)[source]¶ Applies backprop to output gradient arrays.
It delegates the procedure to
backward_cpu()
orbackward_gpu()
by default. Which it selects is determined by the type of input arrays and output gradient arrays. Implementations ofFunction
must implement either CPU/GPU methods or this method, if the function is intended to be backproped.Parameters:  inputs – Tuple of input arrays.
 grad_outputs – Tuple of output gradient arrays.
Returns: Tuple of input gradient arrays. Some or all of them can be
None
, if the function is not differentiable on inputs.Return type: Warning
Implementations of
Function
must take care that the return value must be a tuple even if it returns only one array.

backward_cpu
(inputs, grad_outputs)[source]¶ Applies backprop to output gradient arrays on CPU.
Parameters:  inputs – Tuple of input
numpy.ndarray
object(s).  grad_outputs – Tuple of output gradient
numpy.ndarray
object(s).
Returns: Tuple of input gradient
numpy.ndarray
object(s). Some or all of them can beNone
, if the function is not differentiable on corresponding inputs.Return type: Warning
Implementations of
Function
must take care that the return value must be a tuple even if it returns only one array. inputs – Tuple of input

backward_gpu
(inputs, grad_outputs)[source]¶ Applies backprop to output gradient arrays on GPU.
Parameters:  inputs – Tuple of input
cupy.ndarray
object(s).  grad_outputs – Tuple of output gradient
cupy.ndarray
object(s).
Returns: Tuple of input gradient
cupy.ndarray
object(s). Some or all of them can beNone
, if the function is not differentiable on corresponding inputs.Return type: Warning
Implementations of
Function
must take care that the return value must be a tuple even if it returns only one array. inputs – Tuple of input

check_type_forward
(in_types)[source]¶ Checks types of input data before forward propagation.
Before
forward()
is called, this function is called. You need to validate types of input data in this function using the type checking utilities.Parameters: in_types (TypeInfoTuple) – The type information of input data for forward()
.

delete_hook
(name)[source]¶ Unregisters the function hook.
Parameters: name (str) – the name of the function hook to be unregistered.

forward
(inputs)[source]¶ Applies forward propagation to input arrays.
It delegates the procedure to
forward_cpu()
orforward_gpu()
by default. Which it selects is determined by the type of input arrays. Implementations ofFunction
must implement either CPU/GPU methods or this method.Parameters: inputs – Tuple of input array(s). Returns: Tuple of output array(s). Warning
Implementations of
Function
must take care that the return value must be a tuple even if it returns only one array.

forward_cpu
(inputs)[source]¶ Applies forward propagation to input arrays on CPU.
Parameters: inputs – Tuple of numpy.ndarray
object(s).Returns: Tuple of numpy.ndarray
object(s).Return type: tuple Warning
Implementations of
Function
must take care that the return value must be a tuple even if it returns only one array.

forward_gpu
(inputs)[source]¶ Applies forward propagation to input arrays on GPU.
Parameters: inputs – Tuple of cupy.ndarray
object(s).Returns: Tuple of cupy.ndarray
object(s).Return type: tuple Warning
Implementations of
Function
must take care that the return value must be a tuple even if it returns only one array.

label
¶ Short text that represents the function.
The default implementation returns its type name. Each function should override it to give more information.

local_function_hooks
¶ Ordered Dictionary of registered function hooks.
Contrary to
chainer.thread_local.function_hooks
, which registers its elements to all functions, Function hooks in this property is specific to this function.

unchain
()[source]¶ Purges in/out variables and this function itself from the graph.
This method is called from
Variable.unchain_backward()
method.

chainer.
force_backprop_mode
(*args, **kwds)[source]¶ Enable backpropagation for Variable whose volatile is auto.
When you want to enable backpropagation in
no_backprop_mode()
, call this method. In this context,Variable
object whosevolatile
attribute is'auto'
behaves like a volatile variable. That means you can disableno_backprop_mode()
in this context.If you call this method outside of
no_backprop_mode()
context, it changes nothing.Variable
object withvolatile='auto'
behaves like a volatile variable by default.In this example, the volatility of
x
andy
is'auto'
. Inno_backprop_mode()
context,y
does not have a computational graph but inforce_backprop_mode()
it has a graph.>>> with chainer.no_backprop_mode(): ... # Variable with volatile='auto' behaves like volatile='on' ... with chainer.force_backprop_mode(): ... # Variable with volatile='auto' behaves like volatile='off' ... y = x + 1
See also
See
no_backprop_mode()
for details of backprop mode.

chainer.
no_backprop_mode
(*args, **kwds)[source]¶ Disable backpropagation for Variable whose volatile is auto.
In the default setting a
Variable
object whosevolatile
attribute is'auto'
behaves like a nonvolatile variable. That means such aVariable
object builds a computational graph, consumes memory to store the graph, and you can execute backpropagation for it. With this context such aVariable
object behaves like a volatile variable. So, you can easily switch training and evaluation.In this example, the volatility of
x
andy
is'auto'
. So,y
does not have a computational graph.>>> x = chainer.Variable(numpy.array([1,], 'f'), volatile='auto') >>> with chainer.no_backprop_mode(): ... y = x + 1
Link and Chain¶

class
chainer.
Link
(**params)[source]¶ Building block of model definitions.
Link is a building block of neural network models that support various features like handling parameters, defining network fragments, serialization, etc.
Link is the primitive structure for the model definitions. It supports management of parameter variables and persistent values that should be incorporated to serialization. Parameters are variables registered via the
add_param()
method, or given to the initializer method. Persistent values are arrays, scalars, or any other serializable values registered via theadd_persistent()
method.Note
Whereas arbitrary serializable objects can be registered as persistent values, it is strongly recommended to just register values that should be treated as results of learning. A typical example of persistent values is ones computed during training and required for testing, e.g. running statistics for batch normalization.
Parameters and persistent values are referred by their names. They can be accessed as attributes of the links. Link class itself manages the lists of names of parameters and persistent values to distinguish parameters and persistent values from other attributes.
Link can be composed into more complex models. This composition feature is supported by child classes like
Chain
andChainList
. One can create a chain by combining one or more links. See the documents for these classes for details.As noted above, Link supports the serialization protocol of the
Serializer
class. Note that only parameters and persistent values are saved and loaded. Other attributes are considered as a part of user program (i.e. a part of network definition). In order to construct a link from saved file, other attributes must be identically reconstructed by user codes.Example
This is a simple example of custom link definition. Chainer itself also provides many links defined under the
links
module. They might serve as examples, too.Consider we want to define a simple primitive link that implements a fullyconnected layer based on the
linear()
function. Note that this function takes input units, a weight variable, and a bias variable as arguments. Then, the fullyconnected layer can be defined as follows:import chainer import chainer.functions as F import numpy as np class LinearLayer(chainer.Link): def __init__(self, n_in, n_out): # Parameters are initialized as a numpy array of given shape. super(LinearLayer, self).__init__( W=(n_out, n_in), b=(n_out,), ) self.W.data[...] = np.random.randn(n_out, n_in) self.b.data.fill(0) def __call__(self, x): return F.linear(x, self.W, self.b)
This example shows that a user can define arbitrary parameters and use them in any methods. Links typically implement the
__call__
operator.Parameters: params – Names, shapes, and optional dtypes of initial parameters. The keywords are used as the parameter names and the corresponding values consist either of the shape or a tuple of shape and a dtype (shape, dtype). If only the shape is supplied, the default dtype will be used. Variables: name (str) – Name of this link, given by the parent chain (if exists). 
add_param
(name, shape, dtype=<type 'numpy.float32'>, initializer=None)[source]¶ Registers a parameter to the link.
The registered parameter is saved and loaded on serialization and deserialization, and involved in the optimization. The data and gradient of the variable are initialized by NaN arrays. If
initializer
is notNone
, the data is initialized byinitializer
.If the supplied
name
argument corresponds to an uninitialized parameter (that is, one that was added with theadd_uninitialized_param()
method),name
will be removed from the set of uninitialized parameters.The parameter is set to an attribute of the link with the given name.
Parameters:  name (str) – Name of the parameter. This name is also used as the attribute name. Any uninitialized parameters with the same name will be removed.
 shape (int or tuple of ints) – Shape of the parameter array.
 dtype – Data type of the parameter array.
 initializer (chainer.initializer.Initializer) – If it is not
None
, the data is initialized with the given initializer. Note that in this casedtype
argument is ignored.

add_persistent
(name, value)[source]¶ Registers a persistent value to the link.
The registered value is saved and loaded on serialization and deserialization. The value is set to an attribute of the link.
Parameters:  name (str) – Name of the persistent value. This name is also used for the attribute name.
 value – Value to be registered.

add_uninitialized_param
(name)[source]¶ Registers an uninitialized parameter to the link.
An uninitialized parameter is defined as a parameter that has a name but that does not yet have a shape. If the shape of a parameter depends on the shape of the inputs to the
__call__
operator, it can be useful to defer initialization (that is, setting the shape) until the first forward call of the link. Such parameters are intended to be defined as uninitialized parameters in the initializer and then initialized during the first forward call.An uninitialized parameter is intended to be registered to a link by calling this method in the initializer method. Then, during the first forward call, the shape of the parameter will be determined from the size of the inputs and the parameter must be initialized by calling the
add_param()
method.Parameters: name – (str): Name of the uninitialized parameter.

addgrads
(link)[source]¶ Accumulates gradient values from given link.
This method adds each gradient array of the given link to corresponding gradient array of this link. The accumulation is even done across host and different devices.
Parameters: link (Link) – Source link object.

children
()[source]¶ Returns a generator of all child links.
Returns: A generator object that generates all child links.

cleargrads
()[source]¶ Clears all gradient arrays.
This method should be called before the backward computation at every iteration of the optimization.

copy
()[source]¶ Copies the link hierarchy to new one.
The whole hierarchy rooted by this link is copied. The copy is basically shallow, except that the parameter variables are also shallowly copied. It means that the parameter variables of copied one are different from ones of original link, while they share the data and gradient arrays.
The name of the link is reset on the copy, since the copied instance does not belong to the original parent chain (even if exists).
Returns: Copied link object. Return type: Link

copyparams
(link)[source]¶ Copies all parameters from given link.
This method copies data arrays of all parameters in the hierarchy. The copy is even done across the host and devices. Note that this method does not copy the gradient arrays.
Parameters: link (Link) – Source link object.

has_uninitialized_params
¶ Check if the link has uninitialized parameters.
Returns: True
if the link has any uninitialized parameters. Otherwise returnsFalse
.Return type: bool

links
(skipself=False)[source]¶ Returns a generator of all links under the hierarchy.
Parameters: skipself (bool) – If True
, then the generator skips this link and starts with the first child link.Returns: A generator object that generates all links.

namedlinks
(skipself=False)[source]¶ Returns a generator of all (path, link) pairs under the hierarchy.
Parameters: skipself (bool) – If True
, then the generator skips this link and starts with the first child link.Returns: A generator object that generates all (path, link) pairs.

namedparams
()[source]¶ Returns a generator of all (path, param) pairs under the hierarchy.
Returns: A generator object that generates all (path, parameter) pairs. The paths are relative from this link.

params
()[source]¶ Returns a generator of all parameters under the link hierarchy.
Returns: A generator object that generates all parameters.

serialize
(serializer)[source]¶ Serializes the link object.
Parameters: serializer (AbstractSerializer) – Serializer object.

to_cpu
()[source]¶ Copies parameter variables and persistent values to CPU.
This method does not handle nonregistered attributes. If some of such attributes must be copied to CPU, the link implementation must override this method to do so.
Returns: self

to_gpu
(device=None)[source]¶ Copies parameter variables and persistent values to GPU.
This method does not handle nonregistered attributes. If some of such attributes must be copied to GPU, the link implementation must override this method to do so.
Parameters: device – Target device specifier. If omitted, the current device is used. Returns: self

xp
¶ Array module for this link.
Depending on which of CPU/GPU this link is on, this property returns
numpy
orcupy
.

zerograds
()[source]¶ Initializes all gradient arrays by zero.
This method can be used for the same purpose of cleargrads, but less efficient. This method is left for backward compatibility.
Deprecated since version v1.15: Use
cleargrads()
instead.


class
chainer.
Chain
(**links)[source]¶ Composable link with objectlike interface.
Composability is one of the most important features of neural nets. Neural net models consist of many reusable fragments, and each model itself might be embedded into a larger learnable system. Chain enables us to write a neural net based on composition, without bothering about routine works like collecting parameters, serialization, copying the structure with parameters shared, etc.
This class actually provides a way to compose one or more links into one structure. A chain can contain one or more child links. Child link is a link registered to the chain with its own name. The child link is stored to an attribute of the chain with the name. User can write a whole model or a fragment of neural nets as a child class of Chain.
Each chain itself is also a link. Therefore, one can combine chains into higherlevel chains. In this way, links and chains construct a link hierarchy. Link hierarchy forms a tree structure, where each node is identified by the path from the root. The path is represented by a string like a file path in UNIX, consisting of names of nodes on the path, joined by slashes
/
.Example
This is a simple example of custom chain definition. Chainer itself also provides some chains defined under the
links
module. They might serve as examples, too.Consider we want to define a multilayer perceptron consisting of two hidden layers with rectifiers as activation functions. We can use the
Linear
link as a building block:import chainer import chainer.functions as F import chainer.links as L class MultiLayerPerceptron(chainer.Chain): def __init__(self, n_in, n_hidden, n_out): # Create and register three layers for this MLP super(MultiLayerPerceptron, self).__init__( layer1=L.Linear(n_in, n_hidden), layer2=L.Linear(n_hidden, n_hidden), layer3=L.Linear(n_hidden, n_out), ) def __call__(self, x): # Forward propagation h1 = F.relu(self.layer1(x)) h2 = F.relu(self.layer2(h1)) return self.layer3(h2)
Child links are registered via the initializer method. They also can be registered by the
add_link()
method. The forward propagation is often implemented as The__call__
operator as the above example, though it is not mandatory.Parameters: links – Child links. The keywords are used as their names. The names are also set to the links. 
add_link
(name, link)[source]¶ Registers a child link to this chain.
The registered link is saved and loaded on serialization and deserialization, and involved in the optimization. The registered link is called a child. The child link is set to an attribute of the chain with the given name.
This method also sets the
name
attribute of the registered link. If the given link already has the name attribute set, then it raises an error.Parameters:


class
chainer.
ChainList
(*links)[source]¶ Composable link with listlike interface.
This is another example of compositional link. Unlike
Chain
, this class can be used like a list of child links. Each child link is indexed by a nonnegative integer, and it maintains the current number of registered child links. Theadd_link()
method inserts a new link at the end of the list. It is useful to write a chain with arbitrary number of child links, e.g. an arbitrarily deep multilayer perceptron.Note that this class does not implement all methods of
list
.Parameters: links – Initial child links. 
__getitem__
(index)[source]¶ Returns the child at given index.
Parameters: index (int) – Index of the child in the list. Returns: The index
th child link.Return type: Link

add_link
(link)[source]¶ Registers a child link to this chain.
The registered link is saved and loaded on serialization and deserialization, and involved in the optimization. The registered link is called a child. The child link is accessible via
children()
generator, which returns a generator running through the children in registered order.This method also sets the
name
attribute of the registered link. If the given link already has the name attribute set, then it raises an error.Parameters: link (Link) – The link object to be registered.

Optimizer¶

class
chainer.
Optimizer
[source]¶ Base class of all numerical optimizers.
This class provides basic features for all optimization methods. It optimizes parameters of a target link. The target link is registered via the
setup()
method, and then theupdate()
method updates its parameters based on a given loss function.Each optimizer implementation must be defined as a child class of Optimizer. It must override
update()
method. An optimizer can use internal states each of which is tied to one of the parameters. State is a dictionary of serializable values (typically arrays of size same as the corresponding parameters). In order to use state dictionaries, the optimizer must overrideinit_state()
method (or its CPU/GPU versions,init_state_cpu()
andinit_state_gpu()
).If the optimizer is based on single gradient computation (like most firstorder methods), then it should inherit
GradientMethod
, which adds some features dedicated for the first order methods.Optimizer instance also supports hook functions. Hook function is registered by the
add_hook()
method. Each hook function is called in registration order in advance of the actual parameter update.Variables:  target – Target link object. It is set by the
setup()
method.  t – Number of update steps. It must be incremented by the
update()
method.  epoch – Current epoch. It is incremented by the
new_epoch()
method.

accumulate_grads
(grads)[source]¶ Accumulates gradients from other source.
This method just adds given gradient arrays to gradients that this optimizer holds. It is typically used in dataparallel optimization, where gradients for different shards are computed in parallel and aggregated by this method. This method correctly treats multiple GPU devices.
Parameters: grads (Iterable) – Iterable of gradient arrays to be accumulated. Deprecated since version v1.5: Use the
chainer.Link.addgrads()
method of the target link instead.

add_hook
(hook, name=None)[source]¶ Registers a hook function.
Hook function is typically called right after the gradient computation, though the timing depends on the optimization method.
Parameters:

clip_grads
(maxnorm)[source]¶ Clips the norm of whole gradients up to the threshold.
Parameters: maxnorm (float) – Threshold of gradient L2 norm. Deprecated since version v1.5: Use the
GradientClipping
hook function instead.

compute_grads_norm
()[source]¶ Computes the norm of whole gradients.
Returns: L2 norm of whole gradients, i.e. square root of sum of square of all gradient elements. Return type: float Warning
This method returns a CPUcomputed value, which means that this method synchronizes between CPU and GPU if at least one of the gradients reside on the GPU.
Deprecated since version v1.5.

init_state
(param, state)[source]¶ Initializes the optimizer state corresponding to the parameter.
This method should add needed items to the
state
dictionary. Each optimizer implementation that uses its own states should override this method or CPU/GPU dedicated versions (init_state_cpu()
andinit_state_gpu()
).Parameters: See also

init_state_cpu
(param, state)[source]¶ Initializes the optimizer state on CPU.
This method is called from
init_state()
by default.Parameters:  param (Variable) – Parameter variable. Its data array is
of type
numpy.ndarray
.  state (dict) – State dictionary.
See also
 param (Variable) – Parameter variable. Its data array is
of type

init_state_gpu
(param, state)[source]¶ Initializes the optimizer state on GPU.
This method is called from
init_state()
by default.Parameters:  param (Variable) – Parameter variable. Its data array is
of type
cupy.ndarray
.  state (dict) – State dictionary.
See also
 param (Variable) – Parameter variable. Its data array is
of type

new_epoch
()[source]¶ Starts a new epoch.
This method increments the
epoch
count. Note that if the optimizer depends on the epoch count, then user should call this method appropriately at the beginning of each epoch.

prepare
()[source]¶ Prepares for an update.
This method initializes missing optimizer states (e.g. for newly added parameters after the set up), and copies arrays in each state dictionary to CPU or GPU according to the corresponding parameter array.

remove_hook
(name)[source]¶ Removes a hook function.
Parameters: name (str) – Registered name of the hook function to remove.

serialize
(serializer)[source]¶ Serializes or deserializes the optimizer.
It only saves or loads the following things:
 Optimizer states
 Global states (
t
andepoch
)
It does not saves nor loads the parameters of the target link. They should be separately saved or loaded.
Parameters: serializer (AbstractSerializer) – Serializer or deserializer object.

setup
(link)[source]¶ Sets a target link and initializes the optimizer states.
Given link is set to the
target
attribute. It also prepares the optimizer state dictionaries corresponding to all parameters in the link hierarchy. The existing states are discarded.Parameters: link (Link) – Target link object.

update
(lossfun=None, *args, **kwds)[source]¶ Updates the parameters and optimizer states.
This method updates the parameters of the target link and corresponding optimizer states. The behavior of this method is different for the cases either
lossfun
is given or not.If
lossfun
is given, then this method initializes the gradients by zeros, calls it with given extra arguments, and calls thebackward()
method of its output to compute the gradients. The implementation might calllossfun
more than once.If
lossfun
is not given, then this method assumes that the gradients of all parameters are already computed. An implementation that requires multiple gradient computations might raise an error on this case.In both cases, this method invokes the update procedure for all parameters.
Parameters:  lossfun (function) – Loss function. It accepts arbitrary arguments
and returns one
Variable
object that represents the loss (or objective) value. This argument can be omitted for single gradientbased methods. In this case, this method assumes gradient arrays computed.  kwds (args,) – Arguments for the loss function.
 lossfun (function) – Loss function. It accepts arbitrary arguments
and returns one

weight_decay
(decay)[source]¶ Applies weight decay to the parameter/gradient pairs.
Parameters: decay (float) – Coefficient of weight decay. Deprecated since version v1.5: Use the
WeightDecay
hook function instead.

zero_grads
()[source]¶ Fills all gradient arrays by zeros.
Deprecated since version v1.5: Use the
chainer.Link.cleargrads()
method for the target link instead.
 target – Target link object. It is set by the

class
chainer.
GradientMethod
[source]¶ Base class of all single gradientbased optimizers.
This is an extension of the
Optimizer
class. Typical gradient methods that just require the gradient at the current parameter vector on an update can be implemented as its child class.An implementation of a gradient method must override the following methods:
init_state()
or bothinit_state_cpu()
andinit_state_gpu()
update_one()
or bothupdate_one_cpu()
andupdate_one_gpu()
Note
It is recommended to call
use_cleargrads()
after creating aGradientMethod
object for efficiency.
update
(lossfun=None, *args, **kwds)[source]¶ Updates parameters based on a loss function or computed gradients.
This method runs in two ways.
 If
lossfun
is given, then use it as a loss function to compute gradients.  Otherwise, this method assumes that the gradients are already computed.
In both cases, the computed gradients are used to update parameters. The actual update routines are defined by the
update_one()
method (or its CPU/GPU versions,update_one_cpu()
andupdate_one_gpu()
). If

update_one
(param, state)[source]¶ Updates a parameter based on the corresponding gradient and state.
This method calls appropriate one from
update_param_cpu()
orupdate_param_gpu()
.Parameters:

use_cleargrads
(use=True)[source]¶ Enables or disables use of
cleargrads()
in update.Parameters: use (bool) – If True
, this function enables use of cleargrads. IfFalse
, disables use of cleargrads (zerograds is used).Note
Note that
update()
callszerograds()
by default for backward compatibility. It is recommended to call this method before first call of update because cleargrads is more efficient than zerograds.
Hook functions¶

class
chainer.optimizer.
WeightDecay
(rate)[source]¶ Optimizer hook function for weight decay regularization.
This hook function adds a scaled parameter to the corresponding gradient. It can be used as a regularization.
Parameters: rate (float) – Coefficient for the weight decay. Variables: rate (float) – Coefficient for the weight decay.

class
chainer.optimizer.
Lasso
(rate)[source]¶ Optimizer hook function for Lasso regularization.
This hook function adds a scaled parameter to the sign of each weight. It can be used as a regularization.
Parameters: rate (float) – Coefficient for the weight decay. Variables: rate (float) – Coefficient for the weight decay.

class
chainer.optimizer.
GradientClipping
(threshold)[source]¶ Optimizer hook function for gradient clipping.
This hook function scales all gradient arrays to fit to the defined L2 norm threshold.
Parameters: threshold (float) – L2 norm threshold. Variables: threshold (float) – L2 norm threshold of gradient norm.

class
chainer.optimizer.
GradientNoise
(eta, noise_func=<function exponential_decay_noise>)[source]¶ Optimizer hook function for adding gradient noise.
This hook function simply adds noise generated by the
noise_func
to the gradient. By default it adds timedependent annealed Gaussian noise to the gradient at every training step:\[g_t \leftarrow g_t + N(0, \sigma_t^2)\]where
\[\sigma_t^2 = \frac{\eta}{(1+t)^\gamma}\]with \(\eta\) selected from {0.01, 0.3, 1.0} and \(\gamma = 0.55\).
Parameters:  eta (float) – Parameter that defines the scale of the noise, which for the default noise function is recommended to be either 0.01, 0.3 or 1.0.
 noise_func (function) – Noise generating function which by default is given by Adding Gradient Noise Improves Learning for Very Deep Networks.
Serializer¶

class
chainer.
AbstractSerializer
[source]¶ Abstract base class of all serializers and deserializers.

__call__
(key, value)[source]¶ Serializes or deserializes a value by given name.
This operator saves or loads a value by given name.
If this is a serializer, then the value is simply saved at the key. Note that some type information might be missed depending on the implementation (and the target file format).
If this is a deserializer, then the value is loaded by the key. The deserialization differently works on scalars and arrays. For scalars, the
value
argument is used just for determining the type of restored value to be converted, and the converted value is returned. For arrays, the restored elements are directly copied into thevalue
argument. String values are treated like scalars. If thevalue
argument isNone
, the type of the restored value will typically be a numpy array but can depend on the particular subclass implementation.Parameters:  key (str) – Name of the serialization entry.
 value (scalar, array, None, or str) – Object to be (de)serialized.
None
is only supported by deserializers.
Returns: Serialized or deserialized value.

Dataset abstraction¶
Chainer has a support of common interface of training and validation datasets. The dataset support consists of three components: datasets, iterators, and batch conversion functions.
Dataset represents a set of examples. The interface is only determined by combination with iterators you want to use on it. The builtin iterators of Chainer requires the dataset to support __getitem__
and __len__
method. In particular, the __getitem__
method should support indexing by both an integer and a slice. We can easily support slice indexing by inheriting DatasetMixin
, in which case users only have to implement get_example()
method for indexing. Some iterators also restrict the type of each example. Basically, datasets are considered as stateless objects, so that we do not need to save the dataset as a checkpoint of the training procedure.
Iterator iterates over the dataset, and at each iteration, it yields a mini batch of examples as a list. Iterators should support the Iterator
interface, which includes the standard iterator protocol of Python. Iterators manage where to read next, which means they are stateful.
Batch conversion function converts the mini batch into arrays to feed to the neural nets. They are also responsible to send each array to an appropriate device. Chainer currently provides concat_examples()
as the only example of batch conversion functions.
These components are all customizable, and designed to have a minimum interface to restrict the types of datasets and ways to handle them. In most cases, though, implementations provided by Chainer itself are enough to cover the usages.
Chainer also has a light system to download, manage, and cache concrete examples of datasets. All datasets managed through the system are saved under the dataset root directory, which is determined by the CHAINER_DATASET_ROOT
environment variable, and can also be set by the set_dataset_root()
function.
Dataset representation¶
See Dataset examples for dataset implementations.

class
chainer.dataset.
DatasetMixin
[source]¶ Default implementation of dataset indexing.
DatasetMixin provides the
__getitem__()
operator. The default implementation usesget_example()
to extract each example, and combines the results into a list. This mixin makes it easy to implement a new dataset that does not support efficient slicing.Dataset implementation using DatasetMixin still has to provide the
__len__()
operator explicitly.
__getitem__
(index)[source]¶ Returns an example or a sequence of examples.
It implements the standard Python indexing. It uses the
get_example()
method by default, but it may be overridden by the implementation to, for example, improve the slicing performance.

get_example
(i)[source]¶ Returns the ith example.
Implementations should override it. It should raise
IndexError
if the index is invalid.Parameters: i (int) – The index of the example. Returns: The ith example.

Iterator interface¶
See Iterator examples for dataset iterator implementations.

class
chainer.dataset.
Iterator
[source]¶ Base class of all dataset iterators.
Iterator iterates over the dataset, yielding a minibatch at each iteration. Minibatch is a list of examples. Each implementation should implement an iterator protocol (e.g., the
__next__()
method).Note that, even if the iterator supports setting the batch size, it does not guarantee that each batch always contains the same number of examples. For example, if you let the iterator to stop at the end of the sweep, the last batch may contain a fewer number of examples.
The interface between the iterator and the underlying dataset is not fixed, and up to the implementation.
Each implementation should provide the following attributes (not needed to be writable).
batch_size
: Number of examples within each minibatch.epoch
: Number of completed sweeps over the dataset.epoch_detail
: Floating point number version of the epoch. For example, if the iterator is at the middle of the dataset at the third epoch, then this value is 2.5.is_new_epoch
:True
if the epoch count was incremented at the last update.
Each implementation should also support serialization to resume/suspend the iteration.

__next__
()[source]¶ Returns the next batch.
This is a part of the iterator protocol of Python. It may raise the
StopIteration
exception when it stops the iteration.

finalize
()[source]¶ Finalizes the iterator and possibly releases the resources.
This method does nothing by default. Implementation may override it to better handle the internal resources.

next
()[source]¶ Python2 alternative of
__next__
.It calls
__next__()
by default.
Batch conversion function¶

chainer.dataset.
concat_examples
(batch, device=None, padding=None)[source]¶ Concatenates a list of examples into array(s).
Dataset iterator yields a list of examples. If each example is an array, this function concatenates them along the newlyinserted first axis (called batch dimension) into one array. The basic behavior is same for examples consisting of multiple arrays, i.e., corresponding arrays of all examples are concatenated.
For instance, consider each example consists of two arrays
(x, y)
. Then, this function concatenatesx
‘s into one array, andy
‘s into another array, and returns a tuple of these two arrays. Another example: consider each example is a dictionary of two entries whose keys are'x'
and'y'
, respectively, and values are arrays. Then, this function concatenatesx
‘s into one array, andy
‘s into another array, and returns a dictionary with two entriesx
andy
whose values are the concatenated arrays.When the arrays to concatenate have different shapes, the behavior depends on the
padding
value. Ifpadding
isNone
(default), it raises an error. Otherwise, it builds an array of the minimum shape that the contents of all arrays can be substituted to. The padding value is then used to the extra elements of the resulting arrays.TODO(beam2d): Add an example.
Parameters:  batch (list) – A list of examples. This is typically given by a dataset iterator.
 device (int) – Device ID to which each array is sent. Negative value indicates the host memory (CPU). If it is omitted, all arrays are left in the original device.
 padding – Scalar value for extra elements. If this is None (default), an error is raised on shape mismatch. Otherwise, an array of minimum dimensionalities that can accommodate all arrays is created, and elements outside of the examples are padded by this value.
Returns: Array, a tuple of arrays, or a dictionary of arrays. The type depends on the type of each example in the batch.

chainer.dataset.
to_device
(device, x)[source]¶ Send an array to a given device.
This method send a given array to a given device. This method is used in
concat_examples()
. You can also use this method in a custom converter method used inUpdater
andExtension
such asStandardUpdater
andEvaluator
.Parameters:  device (int or None) – Device ID to which an array is sent. If it is
negative value, an array is sent to CPU. If it is positive, an
array is sent to GPU with the given ID. If it is
None
, an array is left in the original device.  x (numpy.ndarray or cupy.ndarray) – An array to send.
Returns: Converted array.
 device (int or None) – Device ID to which an array is sent. If it is
negative value, an array is sent to CPU. If it is positive, an
array is sent to GPU with the given ID. If it is
Dataset management¶

chainer.dataset.
get_dataset_root
()[source]¶ Gets the path to the root directory to download and cache datasets.
Returns: The path to the dataset root directory. Return type: str

chainer.dataset.
set_dataset_root
(path)[source]¶ Sets the root directory to download and cache datasets.
There are two ways to set the dataset root directory. One is by setting the environment variable
CHAINER_DATASET_ROOT
. The other is by using this function. If both are specified, one specified via this function is used. The default dataset root is$HOME/.chainer/dataset
.Parameters: path (str) – Path to the new dataset root directory.

chainer.dataset.
cached_download
(url)[source]¶ Downloads a file and caches it.
It downloads a file from the URL if there is no corresponding cache. After the download, this function stores a cache to the directory under the dataset root (see
set_dataset_root()
). If there is already a cache for the given URL, it just returns the path to the cache without downloading the same file.Parameters: url (str) – URL to download from. Returns: Path to the downloaded file. Return type: str

chainer.dataset.
cache_or_load_file
(path, creator, loader)[source]¶ Caches a file if it does not exist, or loads it otherwise.
This is a utility function used in dataset loading routines. The
creator
creates the file to given path, and returns the content. If the file already exists, theloader
is called instead, and it loads the file and returns the content.Note that the path passed to the creator is temporary one, and not same as the path given to this function. This function safely renames the file created by the creator to a given path, even if this function is called simultaneously by multiple threads or processes.
Parameters:  path (str) – Path to save the cached file.
 creator – Function to create the file and returns the content. It takes a path to temporary place as the argument. Before calling the creator, there is no file at the temporary path.
 loader – Function to load the cached file and returns the content.
Returns: It returns the returned values by the creator or the loader.
Training loop abstraction¶
Chainer provides a standard implementation of the training loops under the chainer.training
module. It is built on top of many other core features of Chainer, including Variable and Function, Link/Chain/ChainList, Optimizer, Dataset, and Reporter/Summary. Compared to the training loop abstraction of other machine learning tool kits, Chainer’s training framework aims at maximal flexibility, while keeps the simplicity for the typical usages. Most components are pluggable, and users can overwrite the definition.
The core of the training loop abstraction is Trainer
, which implements the training loop itself. The training loop consists of two parts: one is Updater
, which actually updates the parameters to train, and the other is Extension
for arbitrary functionalities other than the parameter update.
Updater and some extensions use dataset
and Iterator
to scan the datasets and load mini batches. The trainer also uses Reporter
to collect the observed values, and some extensions use DictSummary
to accumulate them and computes the statistics.
You can find many examples for the usage of this training utilities from the official examples. You can also search the extension implementations from Trainer extensions.
Trainer¶

class
chainer.training.
Trainer
(updater, stop_trigger=None, out='result')[source]¶ The standard training loop in Chainer.
Trainer is an implementation of a training loop. Users can invoke the training by calling the
run()
method.Each iteration of the training loop proceeds as follows.
 Update of the parameters. It includes the minibatch loading, forward and backward computations, and an execution of the update formula. These are all done by the update object held by the trainer.
 Invocation of trainer extensions in the descending order of their priorities. A trigger object is attached to each extension, and it decides at each iteration whether the extension should be executed. Trigger objects are callable objects that take the trainer object as the argument and return a boolean value indicating whether the extension should be called or not.
Extensions are callable objects that take the trainer object as the argument. There are three ways to define custom extensions: inheriting the
Extension
class, decorating functions bymake_extension()
, and defining any callable including lambda functions. SeeExtension
for more details on custom extensions and how to configure them.Users can register extensions to the trainer by calling the
extend()
method, where some configurations can be added. Trigger object, which is also explained above. In most cases,
IntervalTrigger
is used, in which case users can simply specify a tuple of the interval length and its unit, like(1000, 'iteration')
or(1, 'epoch')
.  The order of execution of extensions is determined by their priorities.
Extensions of higher priorities are invoked earlier. There are three
standard values for the priorities:
PRIORITY_WRITER
. This is the priority for extensions that write some records to theobservation
dictionary. It includes cases that the extension directly adds values to the observation dictionary, or the extension uses thechainer.report()
function to report values to the observation dictionary.PRIORITY_EDITOR
. This is the priority for extensions that edit theobservation
dictionary based on already reported values.PRIORITY_READER
. This is the priority for extensions that only read records from theobservation
dictionary. This is also suitable for extensions that do not use theobservation
dictionary at all.
 Extensions with
invoke_before_training
flag on are also invoked at the beginning of the training loop. Extensions that update the training status (e.g., changing learning rates) should have this flag to beTrue
to ensure that resume of the training loop correctly recovers the training status.
The current state of the trainer object and objects handled by the trainer can be serialized through the standard serialization protocol of Chainer. It enables us to easily suspend and resume the training loop.
Note
The serialization does not recover everything of the training loop. It only recovers the states which change over the training (e.g. parameters, optimizer states, the batch iterator state, extension states, etc.). You must initialize the objects correctly before deserializing the states.
On the other hand, it means that users can change the settings on deserialization. For example, the exit condition can be changed on the deserialization, so users can train the model for some iterations, suspend it, and then resume it with larger number of total iterations.
During the training, it also creates a
Reporter
object to store observed values on each update. For each iteration, it creates a fresh observation dictionary and stores it in theobservation
attribute.Links of the target model of each optimizer are registered to the reporter object as observers, where the name of each observer is constructed as the format
<optimizer name><link name>
. The link name is given by thechainer.Link.namedlink()
method, which represents the path to each link in the hierarchy. Other observers can be registered by accessing the reporter object via thereporter
attribute.The default trainer is plain, i.e., it does not contain any extensions.
Parameters:  updater (Updater) – Updater object. It defines how to update the models.
 stop_trigger – Trigger that determines when to stop the training loop.
If it is not callable, it is passed to
IntervalTrigger
.
Variables:  updater – The updater object for this trainer.
 stop_trigger – Trigger that determines when to stop the training loop.
The training loop stops at the iteration on which this trigger
returns
True
.  observation – Observation of values made at the last update. See the
Reporter
class for details.  out – Output directory.
 reporter – Reporter object to report observed values.

elapsed_time
¶ Total time used for the training.
The time is in seconds. If the training is resumed from snapshot, it includes the time of all the previous training to get the current state of the trainer.

extend
(extension, name=None, trigger=None, priority=None, invoke_before_training=None)[source]¶ Registers an extension to the trainer.
Extension
is a callable object which is called after each update unless the corresponding trigger object decides to skip the iteration. The order of execution is determined by priorities: extensions with higher priorities are called earlier in each iteration. Extensions with the same priority are invoked in the order of registrations.If two or more extensions with the same name are registered, suffixes are added to the names of the second to last extensions. The suffix is
_N
where N is the ordinal of the extensions.See
Extension
for the interface of extensions.Parameters:  extension – Extension to register.
 name (str) – Name of the extension. If it is omitted, the
default_name
attribute of the extension is used instead. Note that the name would be suffixed by an ordinal in case of duplicated names as explained above.  trigger (tuple or Trigger) – Trigger object that determines when to
invoke the extension. If it is
None
,extension.trigger
is used instead. If it isNone
and the extension does not have the trigger attribute, the extension is triggered at every iteration by default. If the trigger is not callable, it is passed toIntervalTrigger
to build an interval trigger.  priority (int) – Invocation priority of the extension. Extensions
are invoked in the descending order of priorities in each
iteration. If this is
None
,extension.priority
is used instead.  invoke_before_training (bool or None) – If
True
, the extension is also invoked just before entering the training loop. If this isNone
,extension.invoke_before_training
is used instead. This option is mainly used for extensions that alter the training configuration (e.g., learning rates); in such a case, resuming from snapshots require the call of extension to recover the configuration before any updates.
Updater¶

class
chainer.training.
Updater
[source]¶ Interface of updater objects for trainers.
TODO(beam2d): document it.

connect_trainer
(trainer)[source]¶ Connects the updater to the trainer that will call it.
The typical usage of this method is to register additional links to the reporter of the trainer. This method is called at the end of the initialization of
Trainer
. The default implementation does nothing.Parameters: trainer (Trainer) – Trainer object to which the updater is registered.

finalize
()[source]¶ Finalizes the updater object.
This method is called at the end of training loops. It should finalize each dataset iterator used in this updater.

get_all_optimizers
()[source]¶ Gets a dictionary of all optimizers for this updater.
Returns: Dictionary that maps names to optimizers. Return type: dict


class
chainer.training.
StandardUpdater
(iterator, optimizer, converter=<function concat_examples>, device=None, loss_func=None)[source]¶ Standard implementation of Updater.
This is the standard implementation of
Updater
. It accepts one or more training datasets and one or more optimizers. The default update routine assumes that there is only one training dataset and one optimizer. Users can override this update routine by inheriting this class and overriding theupdate_core()
method. Each batch is converted to input arrays byconcat_examples()
by default, which can also be manually set byconverter
argument.Parameters:  iterator – Dataset iterator for the training dataset. It can also be a
dictionary of iterators. If this is just an iterator, then the
iterator is registered by the name
'main'
.  optimizer – Optimizer to update parameters. It can also be a dictionary
of optimizers. If this is just an optimizer, then the optimizer is
registered by the name
'main'
.  converter – Converter function to build input arrays. Each batch
extracted by the main iterator and the
device
option are passed to this function.concat_examples()
is used by default.  device – Device to which the training data is sent. Negative value indicates the host memory (CPU).
 loss_func – Loss function. The target link of the main optimizer is used by default.
Variables:  converter – Converter function.
 loss_func – Loss function. If it is
None
, the target link of the main optimizer is used instead.  device – Device to which the training data is sent.
 iteration – Current number of completed updates.
 iterator – Dataset iterator for the training dataset. It can also be a
dictionary of iterators. If this is just an iterator, then the
iterator is registered by the name

class
chainer.training.
ParallelUpdater
(iterator, optimizer, converter=<function concat_examples>, models=None, devices=None, loss_func=None)[source]¶ Implementation of a parallel GPU Updater.
This is an implementation of
Updater
that uses multiple GPUs. It behaves similarly toStandardUpdater
. The update routine is modified to support dataparallel computation on multiple GPUs in one machine. It is based on synchronous parallel SGD: it parallelizes the gradient computation over a minibatch, and updates the parameters only in the main device.Parameters:  iterator – Dataset iterator for the training dataset. It can also be a
dictionary of iterators. If this is just an iterator, then the
iterator is registered by the name
'main'
.  optimizer – Optimizer to update parameters. It can also be a dictionary
of optimizers. If this is just an optimizer, then the optimizer is
registered by the name
'main'
.  converter – Converter function to build input arrays. Each batch
extracted by the main iterator is split equally between the
devices and then passed with corresponding
device
option to this function.concat_examples()
is used by default.  models – Dictionary of models. The main model should be the same model
attached to the
'main'
optimizer.  devices – Dictionary of devices to which the training data is sent. The
devices should be arranged in a dictionary with the same structure
as
models
.  loss_func – Loss function. The model is used as a loss function by default.
 iterator – Dataset iterator for the training dataset. It can also be a
dictionary of iterators. If this is just an iterator, then the
iterator is registered by the name
Extension¶

class
chainer.training.
Extension
[source]¶ Base class of trainer extensions.
Extension of
Trainer
is a callable object that takes the trainer object as the argument. It also provides some default configurations as its attributes, e.g. the default trigger and the default priority. This class provides a set of typical default values for these attributes.There are three ways to define users’ own extensions: inheriting this class, decorating closures by
make_extension()
, or using any callable including lambda functions as extensions. Decorator can slightly reduce the overhead and is much easier to use, while this class provides more flexibility (for example, it can have methods to configure the behavior). Using a lambda function allows oneline coding for simple purposes, but users have to specify the configurations as arguments toTrainer.extend()
. For a callable not inheriting this class, the default configurations of this class are used unless the user explicitly specifies them inTrainer.extend()
method.Variables:  trigger – Default value of trigger for this extension. It is set to
(1, 'iteration')
by default.  priority – Default priority of the extension. It is set to
PRIORITY_READER
by default.  invoke_before_training – Default flag to decide whether this extension
should be invoked before the training starts. The default value is
False
.

__call__
(trainer)[source]¶ Invokes the extension.
Implementations should override this operator. This method is called at iterations which the corresponding trigger accepts.
Parameters: trainer (Trainer) – Trainer object that calls this operator.

default_name
¶ Default name of the extension.
It is the name of the class by default. Implementation can override this property, or provide a class attribute to hide it.
 trigger – Default value of trigger for this extension. It is set to

chainer.training.
make_extension
(trigger=None, default_name=None, priority=None, invoke_before_training=False, finalizer=None)[source]¶ Decorator to make given functions into trainer extensions.
This decorator just adds some attributes to a given function. The value of the attributes are given by the arguments of this decorator.
See
Extension
for details of trainer extensions. Most of the default values of arguments also follow those for this class.Parameters:  trigger – Default trigger of the extension.
 default_name – Default name of the extension. The name of a given function is used by default.
 priority (int) – Default priority of the extension.
 invoke_before_training (bool) – Default flag to decide whether the extension should be invoked before any training.
 finalizer – Finalizer function of this extension. The finalizer is called at the end of the training loop.
Trigger¶
Trigger is a callable object to decide when to process some specific event within the training loop. It takes a Trainer object as the argument, and returns True if some event should be fired.
It is mainly used to determine when to call an extension. It is also used to determine when to quit the training loop.

chainer.training.
get_trigger
(trigger)[source]¶ Gets a trigger object.
Trigger object is a callable that accepts a
Trainer
object as an argument and returns a boolean value. When it returns True, various kinds of events can occur depending on the context in which the trigger is used. For example, if the trigger is passed to theTrainer
as the stop trigger, the training loop breaks when the trigger returns True. If the trigger is passed to theextend()
method of a trainer, then the registered extension is invoked only when the trigger returns True.This function returns a trigger object based on the argument. If
trigger
is already a callable, it just returns the trigger. Iftrigger
isNone
, it returns a trigger that never fires. Otherwise, it passes the value toIntervalTrigger
.Parameters: trigger – Trigger object. It can be either an already built trigger object (i.e., a callable object that accepts a trainer object and returns a bool value), or a tuple. In latter case, the tuple is passed to IntervalTrigger
.Returns: trigger
if it is a callable, otherwise aIntervalTrigger
object made fromtrigger
.
Debug mode¶
In debug mode, Chainer checks values of variables on runtime and shows more detailed error messages. It helps you to debug your programs. Instead it requires additional overhead time.
In debug mode, Chainer checks all results of forward and backward computation, and if it founds a NaN value, it raises RuntimeError
.
Some functions and links also check validity of input values.

chainer.
is_debug
()[source]¶ Get the debug mode.
Returns: Return True
if Chainer is in debug mode.Return type: bool

chainer.
set_debug
(debug)[source]¶ Set the debug mode.
Note
This method changes global state. When you use this method on multithreading environment, it may affects other threads.
Parameters: debug (bool) – New debug mode.

class
chainer.
DebugMode
(debug)[source]¶ Debug mode context.
This class provides a context manager for debug mode. When entering the context, it sets the debug mode to the value of debug parameter with memorizing its original value. When exiting the context, it sets the debug mode back to the original value.
Parameters: debug (bool) – Debug mode used in the context.
FunctionSet (deprecated)¶

class
chainer.
FunctionSet
(**links)[source]¶ Set of links (as “parameterized functions”).
FunctionSet is a subclass of
Chain
. Function registration is done just by adding an attribute toobject
.Deprecated since version v1.5: Use
Chain
instead.
__getitem__
(key)[source]¶ Returns an attribute by name.
Parameters: key (str) – Name of the attribute. Returns: Attribute. Example
>>> model = chainer.FunctionSet(l1=L.Linear(10, 10), ... l2=L.Linear(10, 10)) >>> l1 = model['l1'] # equivalent to l1 = model.l1

collect_parameters
()[source]¶ Returns a tuple of parameters and gradients.
Returns: Tuple (pair) of two tuples. The first element is a tuple of parameter arrays, and the second is a tuple of gradient arrays.

copy_parameters_from
(params)[source]¶ Copies parameters from another source without reallocation.
Parameters: params (Iterable) – Iterable of parameter arrays.

gradients
¶ Tuple of gradient arrays of all registered functions.
The order of gradients is consistent with
parameters()
property.

parameters
¶ Tuple of parameter arrays of all registered functions.
The order of parameters is consistent with
parameters()
property.

Utilities¶
CUDA utilities¶
Device, context and memory management on CuPy.
Chainer uses CuPy (with very thin wrapper) to exploit the speed of GPU
computation. Following modules and classes are imported to cuda
module for convenience (refer to this table when reading chainer’s source
codes).
imported name  original name 

chainer.cuda.cupy 
cupy 
chainer.cuda.ndarray 
cupy.ndarray 
chainer.cuda.cupy.cuda 
cupy.cuda 
chainer.cuda.Device 
cupy.cuda.Device 
chainer.cuda.Event 
cupy.cuda.Event 
chainer.cuda.Stream 
cupy.cuda.Stream 
Chainer replaces the default allocator of CuPy by its memory pool implementation. It enables us to reuse the device memory over multiple forward/backward computations, and temporary arrays for consecutive elementwise operations.
Devices¶

chainer.cuda.
get_device
(*args)[source]¶ Gets the device from a device object, an ID integer or an array object.
This is a convenient utility to select a correct device if the type of
arg
is unknown (i.e., one can use this function on arrays that may be on CPU or GPU). The returned device object supports the context management protocol of Python for the with statement.Parameters: args – Values to specify a GPU device. The first device object, integer or cupy.ndarray
object is used to select a device. If it is a device object, it is returned. If it is an integer, the corresponding device is returned. If it is a CuPy array, the device on which this array reside is returned. If any arguments are neither integers nor CuPy arrays, a dummy device object representing CPU is returned.Returns: Device object specified by given args
.See also
See
cupy.cuda.Device
for the device selection not by arrays.
CuPy array allocation and copy¶
Note
As of v1.3.0, the following array construction wrappers are marked as
deprecated. Use the corresponding functions of the cupy
module
instead. The main difference of them is that the default dtype is changed
from float32 to float64.
Deprecated functions  Recommended functions 

chainer.cuda.empty 
cupy.empty() 
chainer.cuda.empty_like 
cupy.empty_like() 
chainer.cuda.zeros 
cupy.zeros() 
chainer.cuda.zeros_like 
cupy.zeros_like() 
chainer.cuda.ones 
cupy.ones() 
chainer.cuda.ones_like 
cupy.ones_like() 
chainer.cuda.full 
cupy.full() 
chainer.cuda.full_like 
cupy.full_like() 

chainer.cuda.
copy
(array, out=None, out_device=None, stream=None)[source]¶ Copies a
cupy.ndarray
object using the default stream.This function can copy the device array to the destination array on another device.
Parameters:  array (cupy.ndarray) – Array to be copied.
 out (cupy.ndarray) – Destination array.
If it is not
None
, thenout_device
argument is ignored.  out_device – Destination device specifier. Actual device object is
obtained by passing this value to
get_device()
.  stream (cupy.cuda.Stream) – CUDA stream.
Returns: Copied array.
If
out
is not specified, then the array is allocated on the device specified byout_device
argument.Return type:

chainer.cuda.
to_cpu
(array, stream=None)[source]¶ Copies the given GPU array to host CPU.
Parameters:  array – Array to be sent to CPU.
 stream (cupy.cuda.Stream) – CUDA stream.
Returns: Array on CPU.
If given
array
is already on CPU, then this function just returnsarray
without performing any copy.Return type:

chainer.cuda.
to_gpu
(array, device=None, stream=None)[source]¶ Copies the given CPU array to specified device.
Parameters:  array – Array to be sent to GPU.
 device – Device specifier.
 stream (cupy.cuda.Stream) – CUDA stream. If not
None
, the copy runs asynchronously.
Returns: Array on GPU.
If
array
is already on GPU, then this function just returnsarray
without performing any copy. Note that this function does not copycupy.ndarray
into specified device.Return type:
Kernel definition utilities¶

chainer.cuda.
memoize
(for_each_device=False)[source]¶ Makes a function memoizing the result for each argument and device.
This is a similar version of
cupy.memoize()
. The difference is that this function can be used in the global scope even if CUDA is not available. In such case, this function does nothing.Note
This decorator acts as a dummy if CUDA is not available. It cannot be used for general purpose memoization even if
for_each_device
is set to False.

chainer.cuda.
clear_memo
()[source]¶ Clears the memoized results for all functions decorated by memoize.
This function works like
cupy.clear_memo()
as a counterpart forchainer.cuda.memoize()
. It can be used even if CUDA is not available. In such a case, this function does nothing.

chainer.cuda.
elementwise
()[source]¶ Creates an elementwise kernel function.
This function uses
memoize()
to cache the kernel object, i.e. the resulting kernel object is cached for each argument combination and CUDA device.The arguments are the same as those for
cupy.ElementwiseKernel
, except that thename
argument is mandatory.

chainer.cuda.
reduce
()[source]¶ Creates a global reduction kernel function.
This function uses
memoize()
to cache the resulting kernel object, i.e. the resulting kernel object is cached for each argument combination and CUDA device.The arguments are the same as those for
cupy.ReductionKernel
, except that thename
argument is mandatory.
CPU/GPU generic code support¶

chainer.cuda.
get_array_module
(*args)[source]¶ Gets an appropriate one from
numpy
orcupy
.This is almost equivalent to
cupy.get_array_module()
. The differences are that this function can be used even if CUDA is not available and that it will return their data arrays’ array module forVariable
arguments.Parameters: args – Values to determine whether NumPy or CuPy should be used. Returns: cupy
ornumpy
is returned based on the types of the arguments.Return type: module
Common algorithms¶

class
chainer.utils.
WalkerAlias
(probs)[source]¶ Implementation of Walker’s alias method.
This method generates a random sample from given probabilities \(p_1, \dots, p_n\) in \(O(1)\) time. It is more efficient than
choice()
. This class works on both CPU and GPU.Parameters: probs (float list) – Probabilities of entries. They are normalized with sum(probs). See: Wikipedia article

sample
(shape)[source]¶ Generates a random sample based on given probabilities.
Parameters: shape (tuple of int) – Shape of a return value. Returns: Returns a generated array with the given shape. If a sampler is in CPU mode the return value is a numpy.ndarray
object, and if it is in GPU mode the return value is acupy.ndarray
object.

Reporter¶
Reporter¶

class
chainer.
Reporter
[source]¶ Object to which observed values are reported.
Reporter is used to collect values that users want to watch. The reporter object holds a mapping from value names to the actually observed values. We call this mapping observations.
When a value is passed to the reporter, an object called observer can be optionally attached. In this case, the name of the observer is added as the prefix of the value name. The observer name should be registered beforehand.
See the following example:
>>> from chainer import Reporter, report, report_scope >>> >>> reporter = Reporter() >>> observer = object() # it can be an arbitrary (reference) object >>> reporter.add_observer('my_observer:', observer) >>> observation = {} >>> with reporter.scope(observation): ... reporter.report({'x': 1}, observer) ... >>> observation {'my_observer:x': 1}
There are also a global API to add values:
>>> observation = {} >>> with report_scope(observation): ... report({'x': 1}, observer) ... >>> observation {'my_observer:x': 1}
The most important application of Reporter is to report observed values from each link or chain in the training and validation procedures.
Trainer
and some extensions prepare their own Reporter object with the hierarchy of the target link registered as observers. We can usereport()
function inside any links and chains to report the observed values (e.g., training loss, accuracy, activation statistics, etc.).Variables: observation – Dictionary of observed values. 
__exit__
(exc_type, exc_value, traceback)[source]¶ Recovers the previous reporter object to the current.

add_observer
(name, observer)[source]¶ Registers an observer of values.
Observer defines a scope of names for observed values. Values observed with the observer are registered with names prefixed by the observer name.
Parameters:  name (str) – Name of the observer.
 observer – The observer object. Note that the reporter distinguishes
the observers by their object ids (i.e.,
id(owner)
), rather than the object equality.

add_observers
(prefix, observers)[source]¶ Registers multiple observers at once.
This is a convenient method to register multiple objects at once.
Parameters:  prefix (str) – Prefix of each name of observers.
 observers – Iterator of name and observer pairs.

report
(values, observer=None)[source]¶ Reports observed values.
The values are written with the key, prefixed by the name of the observer object if given.
Parameters:  values (dict) – Dictionary of observed values.
 observer – Observer object. Its object ID is used to retrieve the observer name, which is used as the prefix of the registration name of the observed value.

scope
(*args, **kwds)[source]¶ Creates a scope to report observed values to
observation
.This is a context manager to be passed to
with
statements. In this scope, the observation dictionary is changed to the given one.It also makes this reporter object current.
Parameters: observation (dict) – Observation dictionary. All observations reported inside of the with
statement are written to this dictionary.


chainer.
report
(values, observer=None)[source]¶ Reports observed values with the current reporter object.
Any reporter object can be set current by the
with
statement. This function calls theReport.report()
method of the current reporter. If no reporter object is current, this function does nothing.Example
The most typical example is a use within links and chains. Suppose that a link is registered to the current reporter as an observer (for example, the target link of the optimizer is automatically registered to the reporter of the
Trainer
). We can report some values from the link as follows:class MyRegressor(chainer.Chain): def __init__(self, predictor): super(MyRegressor, self).__init__(predictor=predictor) def __call__(self, x, y): # This chain just computes the mean absolute and squared # errors between the prediction and y. pred = self.predictor(x) abs_error = F.sum(F.abs(pred  y)) / len(x.data) loss = F.mean_squared_error(pred, y) # Report the mean absolute and squared errors. report({'abs_error': abs_error, 'squared_error': loss}, self) return loss
If the link is named
'main'
in the hierarchy (which is the default name of the target link in theStandardUpdater
), these reported values are named'main/abs_error'
and'main/squared_error'
. If these values are reported inside theEvaluator
extension,'validation/'
is added at the head of the link name, thus the item names are changed to'validation/main/abs_error'
and'validation/main/squared_error'
('validation'
is the default name of the Evaluator extension).Parameters:  values (dict) – Dictionary of observed values.
 observer – Observer object. Its object ID is used to retrieve the observer name, which is used as the prefix of the registration name of the observed value.
Summary and DictSummary¶

class
chainer.
Summary
[source]¶ Online summarization of a sequence of scalars.
Summary computes the statistics of given scalars online.

class
chainer.
DictSummary
[source]¶ Online summarization of a sequence of dictionaries.
DictSummary
computes the statistics of a given set of scalars online. It only computes the statistics for scalar values and variables of scalar values in the dictionaries.
add
(d)[source]¶ Adds a dictionary of scalars.
Parameters: d (dict) – Dictionary of scalars to accumulate. Only elements of scalars, zerodimensional arrays, and variables of zerodimensional arrays are accumulated.

compute_mean
()[source]¶ Creates a dictionary of mean values.
It returns a single dictionary that holds a mean value for each entry added to the summary.
Returns: Dictionary of mean values. Return type: dict

make_statistics
()[source]¶ Creates a dictionary of statistics.
It returns a single dictionary that holds mean and standard deviation values for every entry added to the summary. For an entry of name
'key'
, these values are added to the dictionary by names'key'
and'key.std'
, respectively.Returns: Dictionary of statistics of all entries. Return type: dict

Experimental feature annotation¶

chainer.utils.
experimental
(api_name)[source]¶ Declares that user is using an experimental feature.
The developer of an API can mark it as experimental by calling this function. When users call experimental APIs,
FutureWarning
is issued. The presentation ofFutureWarning
is disabled by settingchainer.disable_experimental_warning
toTrue
, which isFalse
by default.The basic usage is to call it in the function or method we want to mark as experimental along with the API name.
from chainer import utils def f(x): utils.experimental('chainer.foo.bar.f') # concrete implementation of f follows f(1)
... FutureWarning: chainer.foo.bar.f is experimental. The interface can change in the future. ...
We can also make a whole class experimental. In that case, we should call this function in its
__init__
method.class C(): def __init__(self): utils.experimental('chainer.foo.C') C()
... FutureWarning: chainer.foo.C is experimental. The interface can change in the future. ...
If we want to mark
__init__
method only, rather than class itself, it is recommended that we explicitly feed its API name.class D(): def __init__(self): utils.experimental('D.__init__') D()
... FutureWarning: D.__init__ is experimental. The interface can change in the future. ...
Currently, we do not have any sophisticated way to mark some usage of nonexperimental function as experimental. But we can support such usage by explicitly branching it.
def g(x, experimental_arg=None): if experimental_arg is not None: utils.experimental('experimental_arg of chainer.foo.g')
Parameters: api_name (str) – The name of an API marked as experimental.
Assertion and Testing¶
Chainer provides some facilities to make debugging easy.
Type checking utilities¶
Function
uses a systematic type checking of the chainer.utils.type_check
module.
It enables users to easily find bugs of forward and backward implementations.
You can find examples of type checking in some function implementations.

class
chainer.utils.type_check.
Expr
(priority)[source]¶ Abstract syntax tree of an expression.
It represents an abstract syntax tree, and isn’t a value. You can get its actual value with
eval()
function, and get syntax representation with the__str__()
method. Each comparison operator (e.g.==
) generates a newExpr
object which represents the result of comparison between two expressions.Example
Let
x
andy
be instances ofExpr
, then>>> x = Variable(1, 'x') >>> y = Variable(1, 'y') >>> c = (x == y)
is also an instance of
Expr
. To evaluate and get its value, calleval()
method:>>> c.eval() True
Call
str
function to get a representation of the original equation:>>> str(c) 'x == y'
You can actually compare an expression with a value:
>>> (x == 1).eval() True
Note that you can’t use boolean operators such as
and
, as they try to cast expressions to boolean values:>>> z = Variable(1, 'z') >>> x == y and y == z # raises an error Traceback (most recent call last): RuntimeError: Don't convert Expr to bool. Please call Expr.eval method to evaluate expression.

chainer.utils.type_check.
expect
(*bool_exprs)[source]¶ Evaluates and tests all given expressions.
This function evaluates given boolean expressions in order. When at least one expression is evaluated as
False
, that means the given condition is not satisfied. You can check conditions with this function.Parameters: bool_exprs (tuple of Bool expressions) – Bool expressions you want to evaluate.

class
chainer.utils.type_check.
TypeInfo
(shape, dtype)[source]¶ Type information of an input/gradient array.
It contains type information of an array, such as the shape of array and the number of dimensions. This information is independent of CPU or GPU array.
Gradient checking utilities¶
Most function implementations are numerically tested by gradient checking.
This method computes numerical gradients of forward routines and compares their results with the corresponding backward routines.
It enables us to make the source of issues clear when we hit an error of gradient computations.
The chainer.gradient_check
module makes it easy to implement the gradient checking.

chainer.gradient_check.
check_backward
(func, x_data, y_grad, params=(), eps=0.001, atol=1e05, rtol=0.0001, no_grads=None, dtype=None)[source]¶ Test backward procedure of a given function.
This function automatically check backwardprocess of given function. For example, when you have a
Function
classMyFunc
, that gets two arguments and returns one value, you can make its test like this:>> def test_my_func(self): >> func = MyFunc() >> x1_data = xp.array(...) >> x2_data = xp.array(...) >> gy_data = xp.array(...) >> check_backward(func, (x1_data, x2_data), gy_data)
This method creates
Variable
objects withx_data
and callsfunc
with theVariable
s to get its result asVariable
. Then, it setsy_grad
array tograd
attribute of the result and callsbackward
method to get gradients of the inputs. To check correctness of the gradients, the function callsnumerical_grad()
to calculate numerically the gradients and compares the types of gradients withchainer.testing.assert_allclose()
. If input objects (x1_data
or/andx2_data
in this example) represent integer variables, their gradients are ignored.You can simplify a test when
MyFunc
gets only one argument:>> check_backward(func, x1_data, gy_data)
If
MyFunc
is a loss function which returns a zerodimensional array, passNone
togy_data
. In this case, it sets1
tograd
attribute of the result:>> check_backward(my_loss_func, (x1_data, x2_data), None)
If
MyFunc
returns multiple outputs, pass all gradients for outputs as a tuple:>> gy1_data = xp.array(...) >> gy2_data = xp.array(...) >> check_backward(func, x1_data, (gy1_data, gy2_data))
You can also test a
Link
. To check gradients of parameters of the link, set a tuple of the parameters toparams
arguments:>> check_backward(my_link, (x1_data, x2_data), gy_data, >> (my_link.W, my_link.b))
Note that
params
are notndarray
s, butVariables
s.Function objects are acceptable as
func
argument:>> check_backward(lambda x1, x2: f(x1, x2), >> (x1_data, x2_data), gy_data)
Note
func
is called many times to get numerical gradients for all inputs. This function doesn’t work correctly whenfunc
behaves randomly as it gets different gradients.Parameters:  func (callable) – A function which gets
Variable
s and returnsVariable
s.func
must returns a tuple ofVariable
s or oneVariable
. You can useFunction
object,Link
object or a function satisfying the condition.  x_data (ndarray or tuple of ndarrays) – A set of
ndarray
s to be passed tofunc
. Ifx_data
is onendarray
object, it is treated as(x_data,)
.  y_grad (ndarray or tuple of ndarrays or None) – A set of
ndarray
s representing gradients of returnvalues offunc
. Ify_grad
is onendarray
object, it is treated as(y_grad,)
. Iffunc
is a lossfunction,y_grad
should be set toNone
.  params (Variable) – A set of
Variable
s whose gradients are checked. Whenfunc
is aLink
object, set its parameters asparams
. Ifparams
is oneVariable
object, it is treated as(params,)
.  eps (float) – Epsilon value to be passed to
numerical_grad()
.  atol (float) – Absolute tolerance to be passed to
chainer.testing.assert_allclose()
.  rtol (float) – Relative tolerance to be passed to
chainer.testing.assert_allclose()
.  no_grads (list of bool) – Flag to skip variable for gradient assertion.
It should be same length as
x_data
.  dtype (dtype) –
x_data
andy_grad
are casted to this dtype when calculating numerical gradients. Only float types andNone
are allowed.
 See:
numerical_grad()
 func (callable) – A function which gets

chainer.gradient_check.
numerical_grad
(f, inputs, grad_outputs, eps=0.001)[source]¶ Computes numerical gradient by finite differences.
This function is used to implement gradient check. For usage example, see unit tests of
chainer.functions
.Parameters:  f (function) – Python function with no arguments that runs forward computation and returns the result.
 inputs (tuple of arrays) – Tuple of arrays that should be treated as inputs. Each element of them is slightly modified to realize numerical gradient by finite differences.
 grad_outputs (tuple of arrays) – Tuple of arrays that are treated as output gradients.
 eps (float) – Epsilon value of finite differences.
Returns: Numerical gradient arrays corresponding to
inputs
.Return type:
Standard Assertions¶
The assertions have same names as NumPy’s ones.
The difference from NumPy is that they can accept both numpy.ndarray
and cupy.ndarray
.
Function testing utilities¶
Chainer provides some utilities for testing its functions.

chainer.testing.
unary_math_function_unittest
(func, func_expected=None, label_expected=None, make_data=None)[source]¶ Decorator for testing unary mathematical Chainer functions.
This decorator makes test classes test unary mathematical Chainer functions. Tested are forward and backward computations on CPU and GPU across parameterized
shape
anddtype
.Parameters:  func (Function) – Chainer function to be tested by the decorated test class.
 func_expected – Function used to provide expected values for
testing forward computation. If not given, a corresponsing numpy
function for
func
is implicitly picked up by its class name.  label_expected (string) – String used to test labels of Chainer
functions. If not given, the class name of
func
lowered is implicitly used.  make_data – Function to customize input and gradient data used
in the tests. It takes
shape
anddtype
as its arguments, and returns a tuple of input and gradient data. By default, uniform destribution ranged[1, 1]
is used for both.
The decorated test class tests forward and backward computations on CPU and GPU across the following
parameterize()
ed parameters: shape: rank of zero, and rank of more than zero
 dtype:
numpy.float16
,numpy.float32
andnumpy.float64
Additionally, it tests the label of the Chainer function.
Chainer functions tested by the test class decorated with the decorator should have the following properties:
 Unary, taking one parameter and returning one value
dtype
of input and output are the same Elementwise operation for the supplied ndarray
Example
The following code defines a test class that tests
sin()
Chainer function, which takes a parameter withdtype
of float and returns a value with the samedtype
.>>> import unittest >>> from chainer import testing >>> from chainer import functions as F >>> >>> @testing.unary_math_function_unittest(F.Sin()) ... class TestSin(unittest.TestCase): ... pass
Because the test methods are implicitly injected to
TestSin
class by the decorator, it is enough to placepass
in the class definition.Now the test is run with
nose
module.>>> import nose >>> nose.run( ... defaultTest=__name__, argv=['', 'a', '!gpu'], exit=False) True
To customize test data,
make_data
optional parameter can be used. The following is an example of testingsqrt
Chainer function, which is tested in positive value domain here instead of the default input.>>> import numpy >>> >>> def make_data(shape, dtype): ... x = numpy.random.uniform(0.1, 1, shape).astype(dtype) ... gy = numpy.random.uniform(1, 1, shape).astype(dtype) ... return x, gy ... >>> @testing.unary_math_function_unittest(F.Sqrt(), ... make_data=make_data) ... class TestSqrt(unittest.TestCase): ... pass ... >>> nose.run( ... defaultTest=__name__, argv=['', 'a', '!gpu'], exit=False) True
make_data
function which returns input and gradient data generated in proper value domains with givenshape
anddtype
parameters is defined, then passed to the decorator’smake_data
parameter.
Standard Function implementations¶
Chainer provides basic Function
implementations in the
chainer.functions
package. Most of them are wrapped by plain Python
functions, which users should use.
Note
As of v1.5, the concept of parameterized functions are gone, and they are
replaced by corresponding Link
implementations. They are
still put in the functions
namespace for backward
compatibility, though it is strongly recommended to use them via the
chainer.links
package.
Activation functions¶
clipped_relu¶

chainer.functions.
clipped_relu
(x, z=20.0)[source]¶ Clipped Rectifier Unit function.
For a clipping value \(z(>0)\), it computes
Parameters:  x (
Variable
ornumpy.ndarray
orcupy.ndarray
) – Input variable. A \((s_1, s_2, ..., s_n)\)shaped float array.  z (float) – Clipping value. (default = 20.0)
Returns: Output variable. A \((s_1, s_2, ..., s_n)\)shaped float array.
Return type: Example
>>> x = np.random.uniform(100, 100, (10, 20)).astype('f') >>> z = 10.0 >>> np.any(x < 0) True >>> np.any(x > z) True >>> y = F.clipped_relu(x, z=z) >>> np.any(y.data < 0) False >>> np.any(y.data > z) False
 x (
crelu¶

chainer.functions.
crelu
(x, axis=1)[source]¶ Concatenated Rectified Linear Unit function.
This function is expressed as \(f(x) = (\max(0, x), \max(0, x))\), where two output values are concatenated along an axis.
See: https://arxiv.org/abs/1603.05201
Parameters: Returns: Output variable.
Return type:
elu¶

chainer.functions.
elu
(x, alpha=1.0)[source]¶ Exponential Linear Unit function.
This function is expressed as
\[\begin{split}f(x) = \left \{ \begin{array}{ll} x & {\rm if}~ x \ge 0 \\ \alpha (\exp(x)  1) & {\rm if}~ x < 0, \end{array} \right.\end{split}\]where \(\alpha\) is a parameter. See: https://arxiv.org/abs/1511.07289
Parameters: Returns: Output variable.
Return type:
hard_sigmoid¶

chainer.functions.
hard_sigmoid
(x)[source]¶ Elementwise hardsigmoid function.
This function is defined as
\[\begin{split}f(x) = \left \{ \begin{array}{ll} 0 & {\rm if}~ x < 2.5 \\ 0.2 x + 0.5 & {\rm if}~ 2.5 < x < 2.5 \\ 1 & {\rm if}~ 2.5 < x. \end{array} \right.\end{split}\]Parameters: x (Variable) – Input variable. Returns: Output variable. Return type: Variable
leaky_relu¶
log_softmax¶

chainer.functions.
log_softmax
(x, use_cudnn=True)[source]¶ Channelwise logsoftmax function.
This function computes its logarithm of softmax along the second axis. Let \(x = (x_1, x_2, \dots, x_d)^{\top}\) be the d dimensional index array and \(f(x_1, \dots, x_d)\) be the corresponding input array. For each index \(x\) in the input array, it computes the logarithm of the probability \(\log p(x)\) defined as
\[p(x) = {\exp(f(x_1, x_2, \dots, x_d)) \over \sum_{x'_2} \exp(f(x_1, x'_2, \dots, x_d))}.\]This method is theoretically equivalent to
log(softmax(x))
but is more stable.Note
log(softmax(x))
may cause underflow whenx
is too small, becausesoftmax(x)
may returns0
.log_softmax
method is more stable.Parameters: Returns: Output variable.
Return type: See also
lstm¶

chainer.functions.
lstm
(c_prev, x)[source]¶ Long ShortTerm Memory units as an activation function.
This function implements LSTM units with forget gates. Let the previous cell state \(c_{\text{prev}}\) and the incoming signal \(x\).
First, the incoming signal \(x\) is split into four arrays \(a, i, f, o\) of the same shapes along the second axis. It means that \(x\) ‘s second axis must have 4 times the length of \(c_{\text{prev}}\).
The split input signals are corresponding to:
 \(a\) : sources of cell input
 \(i\) : sources of input gate
 \(f\) : sources of forget gate
 \(o\) : sources of output gate
Second, it computes outputs as:
\[\begin{split}c &= \tanh(a) \text{sigmoid}(i) + c_{\text{prev}} \text{sigmoid}(f), \\ h &= \tanh(c) \text{sigmoid}(o).\end{split}\]These are returned as a tuple of two variables.
This function supports variable length inputs. The minibatch size of the current input must be equal to or smaller than that of the previous one. When minibatch size of
x
is smaller than that ofc
, this function only updatesc[0:len(x)]
and doesn’t change the rest ofc
,c[len(x):]
. So, please sort input sequences in descending order of lengths before applying the function.Parameters: Returns: Two
Variable
objectsc
andh
.c
is the updated cell state.h
indicates the outgoing signal.Return type: See the original paper proposing LSTM with forget gates: Long ShortTerm Memory in Recurrent Neural Networks.
Example
Assuming
y
is the current input signal,c
is the previous cell state, andh
is the previous output signal from anlstm
function. Each ofy
,c
andh
hasn_units
channels. Most typical preparation ofx
is:>>> n_units = 100 >>> y = chainer.Variable(np.zeros((1, n_units), 'f')) >>> h = chainer.Variable(np.zeros((1, n_units), 'f')) >>> c = chainer.Variable(np.zeros((1, n_units), 'f')) >>> model = chainer.Chain(w=F.Linear(n_units, 4 * n_units), ... v=F.Linear(n_units, 4 * n_units),) >>> x = model.w(y) + model.v(h) >>> c, h = F.lstm(c, x)
It corresponds to calculate the input sources \(a, i, f, o\) from the current input
y
and the previous outputh
. Different parameters are used for different kind of input sources.
maxout¶

chainer.functions.
maxout
(x, pool_size, axis=1)[source]¶ Maxout activation function.
It accepts an input tensor
x
, reshapes theaxis
dimension (say the size beingM * pool_size
) into two dimensions(M, pool_size)
, and takes maximum along theaxis
dimension. The output of this function is same asx
except thataxis
dimension is transformed fromM * pool_size
toM
.Typically,
x
is the output of a linear layer or a convolution layer. The following is the example where we usemaxout()
in combination with a Linear link.>>> in_size, out_size, pool_size = 100, 100, 100 >>> l = L.Linear(in_size, out_size * pool_size) >>> x = chainer.Variable(np.zeros((1, in_size), 'f')) # prepare data >>> x = l(x) >>> y = F.maxout(x, pool_size)
Parameters: x (Variable) – Input variable. Its first dimension is assumed to be the minibatch dimension. The other dimensions are treated as one concatenated dimension. Returns: Output variable. Return type: Variable See also
prelu¶

chainer.functions.
prelu
(x, W)[source]¶ Parametric ReLU function.
It accepts two arguments: an input
x
and a weight arrayW
and computes the output as \(PReLU(x) = \max(x, W*x)\), where \(*\) is an elementwise multiplication for each sample in the batch.When the PReLU function is combined with twodimensional convolution, the elements of parameter \(a\) are typically shared across the same filter of different pixels. In order to support such usage, this function supports the shape of parameter array that indicates leading dimensions of input arrays except the batch dimension.
For example \(W\) has the shape of \((2, 3, 4)\), \(x\) must have the shape of \((B, 2, 3, 4, S1, ..., SN)\) where B is batch size and the number of trailing S’s is arbitrary nonnegative integer.
Parameters: Returns: Output variable
Return type: See also
relu¶

chainer.functions.
relu
(x, use_cudnn=True)[source]¶ Rectified Linear Unit function.
Parameters:  x (
Variable
ornumpy.ndarray
orcupy.ndarray
) – Input variable. A \((s_1, s_2, ..., s_n)\)shaped float array.  use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
Returns: Output variable. A \((s_1, s_2, ..., s_n)\)shaped float array.
Return type: Example
>>> x = np.random.uniform(1, 1, (3, 4, 5)).astype('f') >>> np.any(x < 0) True >>> y = F.relu(x) >>> np.any(y.data < 0) False >>> y.shape (3, 4, 5)
 x (
sigmoid¶

chainer.functions.
sigmoid
(x, use_cudnn=True)[source]¶ Elementwise sigmoid logistic function.
\[f(x)=(1 + \exp(x))^{1}.\]Parameters:  x (
Variable
ornumpy.ndarray
orcupy.ndarray
) – Input variable. A \((s_1, s_2, ..., s_n)\)shaped float array.  use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
Returns: Output variable. A \((s_1, s_2, ..., s_n)\)shaped float array.
Return type: Example
>>> x = np.random.uniform(0, 1, (3, 4)).astype('f') >>> y = F.sigmoid(x) >>> y.shape (3, 4)
 x (
slstm¶

chainer.functions.
slstm
(c_prev1, c_prev2, x1, x2)[source]¶ SLSTM units as an activation function.
This function implements SLSTM unit. It is an extension of LSTM unit applied to tree structures. The function is applied to binary trees. Each node has two child nodes. It gets four arguments, previous cell states \(c_1\) and \(c_2\), and incoming signals \(x_1\) and \(x_2\).
First both input signals \(x_1\) and \(x_2\) are split into eight arrays \(a_1, i_1, f_1, o_1\), and \(a_2, i_2, f_2, o_2\). They have the same shape along the second axis. It means that \(x_1\) and \(x_2\) ‘s second axis must have 4 times the length of \(c_{1 \text{prev}}\) and \(c_{2 \text{prev}}\).
The split input signals are corresponding to:
 \(a_i\) : sources of cell input
 \(i_i\) : sources of input gate
 \(f_i\) : sources of forget gate
 \(o_i\) : sources of output gate
It computes outputs as:
\[\begin{split}c &= \tanh(a_1 + a_2) \sigma(i_1 + i_2) + c_{1 \text{prev}} \sigma(f_1) + c_{2 \text{prev}} \sigma(f_2), \\ h &= \tanh(c) \sigma(o_1 + o_2),\end{split}\]where \(\sigma\) is the elementwise sigmoid function. The function returns \(c\) and \(h\) as a tuple.
Parameters:  c_prev1 (Variable) – Variable that holds the previous cell state of the first child node. The cell state should be a zero array or the output of the previous call of LSTM.
 c_prev2 (Variable) – Variable that holds the previous cell state of the second child node.
 x1 (Variable) – Variable that holds the incoming signal from the first child node. It must have the second dimension four times of that of the cell state,
 x2 (Variable) – Variable that holds the incoming signal from the second child node.
Returns: Two
Variable
objectsc
andh
.c
is the cell state.h
indicates the outgoing signal.Return type: See detail in paper: Long ShortTerm Memory Over Tree Structures.
softmax¶

chainer.functions.
softmax
(x, use_cudnn=True)[source]¶ Channelwise softmax function.
This function computes its softmax along the second axis. Let \(x = (x_1, x_2, \dots, x_d)^{\top}\) be the d dimensional index array and \(f(x)\) be the d dimensional input array. For each index \(x\) of the input array \(f(x)\), it computes the probability \(p(x)\) defined as \(p(x) = {\exp(f(x)) \over \sum_{x_2} \exp(f(x))}\).
Parameters: Returns: Output variable.
Return type:
Array manipulations¶
broadcast¶
broadcast_to¶
cast¶
concat¶
copy¶

chainer.functions.
copy
(x, dst)[source]¶ Copies the input variable onto the specified device.
This function copies the array of input variable onto the device specified by
dst
. Whendst == 1
, it copies the array onto the host memory. This function supports copies from host to device, from device to device and from device to host.Parameters:  x (Variable) – Variable to be copied.
 dst – Target device specifier.
Returns: Output variable.
Return type:
dstack¶
expand_dims¶
flatten¶
fliplr¶
flipud¶
get_item¶

chainer.functions.
get_item
(x, slices)[source]¶ Extract elements from array with specified shape, axes and offsets.
Parameters:  x (tuple of Variables) – Variable to be sliced.
 slices (int, slice, None or Ellipsis or tuple of them) – Basic slicing
to slice a variable. It supports
int
,slice
,newaxis
(equivalent toNone
) andEllipsis
.
Returns: Variable
object which contains sliced array ofx
.Return type: Note
See NumPy document for details of indexing.
hstack¶
permutate¶

chainer.functions.
permutate
(x, indices, axis=0, inv=False)[source]¶ Permutates a given variable along an axis.
This function permutate
x
with givenindices
. That meansy[i] = x[indices[i]]
for alli
. Note that this result is same asy = x.take(indices)
.indices
must be a permutation of[0, 1, ..., len(x)  1]
.When
inv
isTrue
,indices
is treated as its inverse. That meansy[indices[i]] = x[i]
.Parameters: Returns: Output variable.
Return type:
reshape¶
rollaxis¶
select_item¶
separate¶

chainer.functions.
separate
(x, axis=0)[source]¶ Separates an array along a given axis.
This function separates an array along a given axis. For example, shape of an array is
(2, 3, 4)
. When it separates the array withaxis=1
, it returns three(2, 4)
arrays.This function is an inverse of
chainer.functions.stack()
.Parameters:  x (chainer.Variable) – Variable to be separated.
 axis (int) – Axis along which variables are separated.
Returns: Output variables.
Return type: tuple of chainer.Variable
See also
split_axis¶

chainer.functions.
split_axis
(x, indices_or_sections, axis, force_tuple=False)[source]¶ Splits given variables along an axis.
Parameters:  x (tuple of Variables) – Variables to be split.
 indices_or_sections (int or 1D array) – If this argument is an integer, N, the array will be divided into N equal arrays along axis. If it is a 1D array of sorted integers, it indicates the positions where the array is split.
 axis (int) – Axis that the input array is split along.
 force_tuple (bool) – If
True
, this method returns a tuple even when the number of outputs is one.
Returns: Tuple of
Variable
objects if the number of outputs is more than 1 orVariable
otherwise. Whenforce_tuple
isTrue
, returned value is always a tuple regardless of the number of outputs.Return type: tuple or Variable
Note
This function raises
ValueError
if at least one of the outputs is split to zerosize (i.e.axis
th value of its shape is zero).
squeeze¶

chainer.functions.
squeeze
(x, axis=None)[source]¶ Remove demensions of size one from the shape of a ndarray.
Parameters:  x (chainer.Variable or :class:
numpy.ndarray
or cupy.ndarray) – Input data.  axis (None or int or tuple of ints) – A subset of the singledimensional
entries in the shape to remove. If
None
is supplied, all of them are removed. The dimension index starts at zero. If an axis with dimension greater than one is selected, an error is raised.
Returns: Variable whose dimensions of size 1 are removed.
Return type:  x (chainer.Variable or :class:
stack¶
swapaxes¶
tile¶

chainer.functions.
tile
(x, reps)[source]¶ Construct an array by tiling a given array.
Parameters:  x (chainer.Variable or
numpy.ndarray
or cupy.ndarray) – Input data.  reps (int or tuple of ints) – The number of times for each axis with which x is replicated.
Returns: Variable tiled the given array.
Return type:  x (chainer.Variable or
transpose¶

chainer.functions.
transpose
(x, axes=None)[source]¶ Permute the dimensions of an input variable without copy.
Parameters:  x (Variable) – Input variable.
 axes (tuple of ints) – By default, reverse the dimensions, otherwise permute the axes according to the values given.
Returns: Variable whose axes are permuted.
Return type:
transpose_sequence¶

chainer.functions.
transpose_sequence
(xs)[source]¶ Transpose a list of Variables.
This function transposes a list of
Variable
s and returns a list ofVariable
s. For example a user gives[(0, 1, 2, 3), (4, 5), (6)]
, the function returns[(0, 4, 6), (1, 5), (2), (3)]
. Note that a given list needs to be sorted by each length ofVariable
.Parameters: xs (list of ~chainer.Variable) – Variables to transpose. Returns: Transposed list. Return type: tuple or Variable
vstack¶
Neural network connections¶
bilinear¶

chainer.functions.
bilinear
(e1, e2, W, V1=None, V2=None, b=None)[source]¶ Applies a bilinear function based on given parameters.
This is a building block of Neural Tensor Network (see the reference paper below). It takes two input variables and one or four parameters, and outputs one variable.
To be precise, denote six input arrays mathematically by \(e^1\in \mathbb{R}^{I\cdot J}\), \(e^2\in \mathbb{R}^{I\cdot K}\), \(W\in \mathbb{R}^{J \cdot K \cdot L}\), \(V^1\in \mathbb{R}^{J \cdot L}\), \(V^2\in \mathbb{R}^{K \cdot L}\), and \(b\in \mathbb{R}^{L}\), where \(I\) is minibatch size. In this document, we call \(V^1\), \(V^2\), and \(b\) linear parameters.
The output of forward propagation is calculated as
\[y_{il} = \sum_{jk} e^1_{ij} e^2_{ik} W_{jkl} + \ \sum_{j} e^1_{ij} V^1_{jl} + \sum_{k} e^2_{ik} V^2_{kl} + b_{l}.\]Note that V1, V2, b are optional. If these are not given, then this function omits the last three terms in the above equation.
Note
This function accepts an input variable
e1
ore2
of a nonmatrix array. In this case, the leading dimension is treated as the batch dimension, and the other dimensions are reduced to one dimension.Note
In the original paper, \(J\) and \(K\) must be equal and the author denotes \([V^1 V^2]\) (concatenation of matrices) by \(V\).
Parameters: Returns: Output variable.
Return type:  See:
 Reasoning With Neural Tensor Networks for Knowledge Base Completion [Socher+, NIPS2013].
convolution_2d¶

chainer.functions.
convolution_2d
(x, W, b=None, stride=1, pad=0, use_cudnn=True, cover_all=False, deterministic=False)[source]¶ Twodimensional convolution function.
This is an implementation of twodimensional convolution in ConvNets. It takes three variables: the input image
x
, the filter weightW
, and the bias vectorb
.Notation: here is a notation for dimensionalities.
 \(n\) is the batch size.
 \(c_I\) and \(c_O\) are the number of the input and output channels, respectively.
 \(h\) and \(w\) are the height and width of the input image, respectively.
 \(k_H\) and \(k_W\) are the height and width of the filters, respectively.
Parameters:  x (Variable) – Input variable of shape \((n, c_I, h, w)\).
 W (Variable) – Weight variable of shape \((c_O, c_I, k_H, k_W)\).
 b (Variable) – Bias variable of length \(c_O\) (optional).
 stride (int or pair of ints) – Stride of filter applications.
stride=s
andstride=(s, s)
are equivalent.  pad (int or pair of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p)
are equivalent.  use_cudnn (bool) – If
True
, then this function uses cuDNN if available.  cover_all (bool) – If
True
, all spatial locations are convoluted into some output pixels. It may make the output size larger.  deterministic (bool) – The output of this function can be
nondeterministic when it uses cuDNN.
If this option is
True
, then it forces cuDNN to use a deterministic algorithm. This option is only available for cuDNN version >= v4.
Returns: Output variable.
Return type: The twodimensional convolution function is defined as follows. Then the
Convolution2D
function computes correlations between filters and patches of size \((k_H, k_W)\) inx
. Note that correlation here is equivalent to the inner product between expanded vectors. Patches are extracted at positions shifted by multiples ofstride
from the first positionpad
for each spatial axis. The rightmost (or bottommost) patches do not run over the padded spatial size.Let \((s_Y, s_X)\) be the stride of filter application, and \((p_H, p_W)\) the spatial padding size. Then, the output size \((h_O, w_O)\) is determined by the following equations:
\[\begin{split}h_O &= (h + 2p_H  k_H) / s_Y + 1,\\ w_O &= (w + 2p_W  k_W) / s_X + 1.\end{split}\]If the bias vector is given, then it is added to all spatial locations of the output of convolution.
See also
convolution_nd¶

chainer.functions.
convolution_nd
(x, W, b=None, stride=1, pad=0, use_cudnn=True, cover_all=False)[source]¶ Ndimensional convolution function.
This is an implementation of Ndimensional convolution which is generalized twodimensional convolution in ConvNets. It takes three variables: the input
x
, the filter weightW
and the bias vectorb
.Notation: here is a notation for dimensionalities.
 \(N\) is the number of spatial dimensions.
 \(n\) is the batch size.
 \(c_I\) and \(c_O\) are the number of the input and output channels, respectively.
 \(d_1, d_2, ..., d_N\) are the size of each axis of the input’s spatial dimensions, respectively.
 \(k_1, k_2, ..., k_N\) are the size of each axis of the filters, respectively.
Parameters:  x (Variable) – Input variable of shape \((n, c_I, d_1, d_2, ..., d_N)\).
 W (Variable) – Weight variable of shape \((c_O, c_I, k_1, k_2, ..., k_N)\).
 b (Variable) – Onedimensional bias variable with length \(c_O\) (optional).
 stride (int or tuple of ints) – Stride of filter applications
\((s_1, s_2, ..., s_N)\).
stride=s
is equivalent to(s, s, ..., s)
.  pad (int or tuple of ints) – Spatial padding width for input arrays
\((p_1, p_2, ..., p_N)\).
pad=p
is equivalent to(p, p, ..., p)
.  use_cudnn (bool) – If
True
, then this function uses cuDNN if available. See below for the excact conditions.  cover_all (bool) – If
True
, all spatial locations are convoluted into some output pixels. It may make the output size larger. cover_all needs to beFalse
if you want to use cuDNN.
Returns: Output variable.
Return type: This function uses cuDNN implementation for its forward and backward computation if ALL of the following conditions are satisfied:
cuda.cudnn_enabled
isTrue
use_cudnn
isTrue
 The number of spatial dimensions is more than one.
cover_all
isFalse
 The input’s
dtype
is equal to the filter weight’s.  The
dtype
is FP32, FP64 or FP16(cuDNN version is equal to or greater than v3)
See also
deconvolution_2d¶

chainer.functions.
deconvolution_2d
(x, W, b=None, stride=1, pad=0, outsize=None, use_cudnn=True, deterministic=False)[source]¶ Two dimensional deconvolution function.
This is an implementation of twodimensional deconvolution. It takes three variables: input image
x
, the filter weightW
, and the bias vectorb
.Parameters:  x (Variable) – Input variable of shape \((n, c_I, h, w)\).
 W (Variable) – Weight variable of shape \((c_I, c_O, k_H, k_W)\).
 b (Variable) – Bias variable of length \(c_O\) (optional).
 stride (int or pair of ints) – Stride of filter applications.
stride=s
andstride=(s, s)
are equivalent.  pad (int or pair of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p)
are equivalent.  outsize (tuple) – Expected output size of deconvolutional operation.
It should be pair of height and width \((out_H, out_W)\).
Default value is
None
and the outsize is estimated by input size, stride and pad.  use_cudnn (bool) – If
True
, then this function uses cuDNN if available.  deterministic (bool) – The output of this function can be
nondeterministic when it uses cuDNN.
If this option is
True
, then it forces cuDNN to use a deterministic algorithm. This option is only available for cuDNN version >= v4.
The filter weight has four dimensions \((c_I, c_O, k_H, k_W)\) which indicate the number of input channels, output channels, height and width of the kernels, respectively.
The bias vector is of size \(c_O\).
Let \(X\) be the input tensor of dimensions \((n, c_I, h, w)\), \((s_Y, s_X)\) the stride of filter application, and \((p_H, p_W)\) the spatial padding size. Then, the output size \((h_O, w_O)\) is determined by the following equations:
\[\begin{split}h_O &= s_Y (h  1) + k_H  2p_H,\\ w_O &= s_X (w  1) + k_W  2p_W.\end{split}\]
deconvolution_nd¶

chainer.functions.
deconvolution_nd
(x, W, b=None, stride=1, pad=0, outsize=None, use_cudnn=True)[source]¶ Ndimensional deconvolution function.
This is an implementation of Ndimensional deconvolution which generalizes twodimensional one. It takes three variables: input
x
, the filter weightW
, and the bias vectorb
.Parameters:  x (chainer.Variable or
numpy.ndarray
or cupy.ndarray) – Input data of shape \((n, c_I, d_1, d_2, ..., d_N)\).  W (chainer.Variable or
numpy.ndarray
or cupy.ndarray) – Weight data of shape \((c_I, c_O, k_1, k_2, ..., k_N)\).  b (chainer.Variable or
numpy.ndarray
or cupy.ndarray) – Bias vector of length \(c_O\) (optional).  stride (int or tuple of ints) – Stride of filter applications
\((s_1, s_2, ..., s_N)\).
stride=s
is equivalent to(s, s, ..., s)
.  pad (int or tuple of ints) – Spatial padding size for input arrays
\((p_1, p_2, ..., p_N)\).
pad=p
is equivalent to(p, p, ..., p)
.  outsize (tuple of ints) – Expected output size of deconvolutional
operation. It should be a tuple of ints
\((out_1, out_2, ..., out_N)\). Default value is
None
and the outsize is estimated by input size, stride and pad.  use_cudnn (bool) – If
True
, then this function uses cuDNN if available. Note that cuDNN supports more than onedimensional deconvolution operations only.
Returns: Output variable.
Return type: The filter weight has the following dimensions \((c_I, c_O, k_1, k_2, ..., k_N)\) which indicate the number of input channels, that of output channels and the filter’s spatial sizes, respectively.
The onedimensional bias vector is of size \(c_O\).
Let \(X\) be the input tensor of dimensions \((n, c_I, d_1, d_2, ..., d_N)\), \((s_1, s_2, ..., s_N)\) the stride of filter applications, and \((p_1, p_2, ..., p_N)\) the spacial padding size. Then the output size \((out_1, out_2, ..., out_N)\) is determined by the following equations:
\[\begin{split}out_1 &= s_1 (d_1  1) + k_1  2 p_1,\\ out_2 &= s_2 (d_2  1) + k_2  2 p_2,\\ ...,\\ out_N &= s_N (d_N  1) + k_N  2 p_N.\end{split}\]See also
links.DeconvolutionND
,deconvolution_2d()
 x (chainer.Variable or
dilated_convolution_2d¶

chainer.functions.
dilated_convolution_2d
(x, W, b=None, stride=1, pad=0, dilate=1, use_cudnn=True, cover_all=False)[source]¶ Twodimensional dilated convolution function.
This is an implementation of twodimensional dilated convolution in ConvNets. It takes three variables: the input image
x
, the filter weightW
, and the bias vectorb
.Notation: here is a notation for dimensionalities.
 \(n\) is the batch size.
 \(c_I\) and \(c_O\) are the number of the input and output, respectively.
 \(h\) and \(w\) are the height and width of the input image, respectively.
 \(k_H\) and \(k_W\) are the height and width of the filters, respectively.
Parameters:  x (Variable) – Input variable of shape \((n, c_I, h, w)\).
 W (Variable) – Weight variable of shape \((c_O, c_I, k_H, k_W)\).
 b (Variable) – Bias variable of length \(c_O\) (optional).
 stride (int or pair of ints) – Stride of filter applications.
stride=s
andstride=(s, s)
are equivalent.  pad (int or pair of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p)
are equivalent.  dilate (int or pair of ints) – Dilation factor of filter applications.
dilate=d
anddilate=(d, d)
are equivalent.  use_cudnn (bool) – If
True
, then this function uses cuDNN if available.  cover_all (bool) – If
True
, all spatial locations are convoluted into some output pixels. It may make the output size larger.
Returns: Output variable.
Return type: The twodimensional dilated convolution function is defined as follows. Then the
DilatedConvolution2D
function computes correlations between filters and patches of size \((k_H, k_W)\) inx
. Patches here are extracted at intervals of the dilation factor. Note that correlation here is equivalent to the inner product between expanded vectors. Patches are extracted at intervals of the dilation factor and at positions shifted by multiples ofstride
from the first positionpad
for each spatial axis. The rightmost (or bottommost) patches do not run over the padded spatial size.Let \((s_Y, s_X)\) be the stride of filter application, \((p_H, p_W)\) the spatial padding size, and \((d_Y, d_X)\) the dilation factor of filter application. Then, the output size \((h_O, w_O)\) is determined by the following equations:
\[\begin{split}h_O &= (h + 2p_H  k_H  (k_H  1) * (d_Y  1)) / s_Y + 1,\\ w_O &= (w + 2p_W  k_W  (k_W  1) * (d_X  1)) / s_X + 1.\end{split}\]If the bias vector is given, then it is added to all spatial locations of the output of convolution.
See also
DilatedConvolution2D
embed_id¶

chainer.functions.
embed_id
(x, W, ignore_label=None)[source]¶ Efficient linear function for onehot input.
This function implements so called word embedding. It takes two arguments: a set of IDs (words)
x
in \(B\) dimensional integer vector, and a set of all ID (word) embeddingsW
in \(V \times d\) float32 matrix. It outputs \(B \times d\) matrix whosei
th column is thex[i]
th column ofW
.This function is only differentiable on the input
W
.Parameters: Returns: Output variable.
Return type: See also
linear¶

chainer.functions.
linear
(x, W, b=None)[source]¶ Linear function, or affine transformation.
It accepts two or three arguments: an input minibatch
x
, a weight matrixW
, and optionally a bias vectorb
. It computes\[Y = xW^\top + b.\]Parameters:  x (
Variable
ornumpy.ndarray
orcupy.ndarray
) – Input variable, which is a \((s_B, s_1, s_2, ..., s_n)\)shaped float array. Its first dimension \((s_B)\) is assumed to be the minibatch dimension. The other dimensions are treated as concatenated one dimension whose size must be \((s_1 * ... * s_n = N)\).  W (
Variable
ornumpy.ndarray
orcupy.ndarray
) – Weight variable of shape \((M, N)\), where \((N = s_1 * ... * s_n)\).  b (
Variable
ornumpy.ndarray
orcupy.ndarray
) – Bias variable (optional) of shape \((M,)\).
Returns: Output variable. A float array with shape of \((s_B, M)\).
Return type: See also
Example
>>> x = np.random.uniform(0, 1, (3, 4)).astype('f') >>> W = np.random.uniform(0, 1, (5, 4)).astype('f') >>> b = np.random.uniform(0, 1, (5,)).astype('f') >>> y = F.linear(x, W, b) >>> y.shape (3, 5)
 x (
n_step_lstm¶

chainer.functions.
n_step_lstm
(n_layers, dropout_ratio, hx, cx, ws, bs, xs, train=True, use_cudnn=True)[source]¶ Stacked Long ShortTerm Memory function for sequence inputs.
This function calculates stacked LSTM with sequences. This function gets an initial hidden state \(h_0\), an initial cell state \(c_0\), an input sequence \(x\), weight matrices \(W\), and bias vectors \(b\). This function calculates hidden states \(h_t\) and \(c_t\) for each time \(t\) from input \(x_t\).
\[\begin{split}i_t &= \sigma(W_0 x_t + W_4 h_{t1} + b_0 + b_4) \\ f_t &= \sigma(W_1 x_t + W_5 h_{t1} + b_1 + b_5) \\ o_t &= \sigma(W_2 x_t + W_6 h_{t1} + b_2 + b_6) \\ a_t &= \tanh(W_3 x_t + W_7 h_{t1} + b_3 + b_7) \\ c_t &= f_t \dot c_{t1} + i_t \dot a_t \\ h_t &= o_t \dot \tanh(c_t)\end{split}\]As the function accepts a sequence, it calculates \(h_t\) for all \(t\) with one call. Eight weight matrices and eight bias vectors are required for each layers. So, when \(S\) layers exists, you need to prepare \(8S\) weigth matrices and \(8S\) bias vectors.
If the number of layers
n_layers
is greather than \(1\), input ofk
th layer is hidden stateh_t
ofk1
th layer. Note that all input variables except first layer may have different shape from the first layer.Parameters:  n_layers (int) – Number of layers.
 dropout_ratio (float) – Dropout ratio.
 hx (chainer.Variable) – Variable holding stacked hidden states.
Its shape is
(S, B, N)
whereS
is number of layers and is equal ton_layers
,B
is minibatch size, andN
is dimention of hidden units.  cx (chainer.Variable) – Variable holding stacked cell states.
It has the same shape as
hx
.  ws (list of list of chainer.Variable) – Weight matrices.
ws[i]
represents weights for ith layer. Eachws[i]
is a list containing eight matrices.ws[i][j]
is corresponding withW_j
in the equation. Onlyws[0][j]
where0 <= j < 4
is(I, N)
shape as they are multiplied with input variables. All other matrices has(N, N)
shape.  bs (list of list of chainer.Variable) – Bias vectors.
bs[i]
represnents biases for ith layer. Eachbs[i]
is a list containing eight vectors.bs[i][j]
is corresponding withb_j
in the equation. Shape of each matrix is(N,)
whereN
is dimention of hidden units.  xs (list of chainer.Variable) – A list of
Variable
holding input values. Each elementxs[t]
holds input value for timet
. Its shape is(B_t, I)
, whereB_t
is minibatch size for timet
, andI
is size of input units. Note that this functions supports variable length sequences. When sequneces has different lengths, sort sequences in descending order by length, and transpose the sorted sequence.transpose_sequence()
transpose a list ofVariable()
holding sequence. Soxs
needs to satisfyxs[t].shape[0] >= xs[t + 1].shape[0]
.  train (bool) – If
True
, this function executes dropout.  use_cudnn (bool) – If
True
, this function uses cuDNN if available.
Returns: This functions returns a tuple concaining three elements,
hy
,cy
andys
.hy
is an updated hidden states whose shape is same ashx
.cy
is an updated cell states whose shape is same ascx
.ys
is a list ofVariable
. Each elementys[t]
holds hidden states of the last layer corresponding to an inputxs[t]
. Its shape is(B_t, N)
whereB_t
is minibatch size for timet
, andN
is size of hidden units. Note thatB_t
is the same value asxs[t]
.
Return type: See also
Evaluation functions¶
accuracy¶
binary_accuracy¶
classification_summary¶

chainer.functions.
classification_summary
(y, t, label_num=None, beta=1.0, ignore_label=1)[source]¶ Calculates Precision, Recall, F beta Score, and support.
This function calculates the following quantities for each class.
 Precision: \(\frac{\mathrm{tp}}{\mathrm{tp} + \mathrm{fp}}\)
 Recall: \(\frac{\mathrm{tp}}{\mathrm{tp} + \mathrm{tn}}\)
 F beta Score: The weighted harmonic average of Precision and Recall.
 Support: The number of instances of each ground truth label.
Here,
tp
,fp
, andtn
stand for the number of true positives, false positives, and true negative, respectively.label_num
specifies the number of classes, that is, each value int
must be an integer in the range of[0, label_num)
. Iflabel_num
isNone
, this function regardslabel_num
as a maximum of int
plus one.ignore_label
determines which instances should be ignored. Specifically, instances with the given label are not taken into account for calculating the above quantities. By default, it is set to 1 so that all instances are taken into consideration, as labels are supposed to be nonnegative integers. Settingignore_label
to a nonnegative integer less thanlabel_num
is illegal and yields undefined behavior. In the current implementation, it arisesRuntimeWarning
andignore_label
th entries in output arrays do not contain correct quantities.Parameters:  y (Variable) – Variable holding a vector of scores.
 t (Variable) – Variable holding a vector of ground truth labels.
 label_num (int) – The number of classes.
 beta (float) – The parameter which determines the weight of precision in the Fbeta score.
 ignore_label (int) – Instances with this label are ignored.
Returns: 4tuple of ~chainer.Variable of size
(label_num,)
. Each element represents precision, recall, F beta score, and support of this minibatch.
r2_score¶

chainer.functions.
r2_score
(pred, true, sample_weight=None, multioutput='uniform_average')[source]¶ Computes R^2(coefficient of determination) regression score function.
Parameters:  pred (Variable) – Variable holding a vector, matrix or tensor of estimated target values.
 true (Variable) – Variable holding a vector, matrix or tensor of correct target values.
 sample_weight – This argument is for compatibility with scikitlearn’s implementation of r2_score. Current implementation admits None only.
 multioutput (string) – [‘uniform_average’, ‘raw_values’]. if ‘uniform_average’, this function returns an average of R^2 score of multiple output. If ‘raw_average’, this function return a set of R^2 score of multiple output.
Returns: A Variable holding a scalar array of the R^2 score if ‘multioutput’ is ‘uniform_average’ or a vector of R^2 scores if ‘multioutput’ is ‘raw_values’.
Return type: Note
This function is nondifferentiable.
Loss functions¶
bernoulli_nll¶

chainer.functions.
bernoulli_nll
(x, y)[source]¶ Computes the negative loglikelihood of a Bernoulli distribution.
This function calculates the negative loglikelihood of a Bernoulli distribution.
\[\log B(x; p) = \sum_i \{x_i \log(p_i) + (1  x_i)\log(1  p_i)\},\]where \(p = \sigma(y)\), \(\sigma(\cdot)\) is a sigmoid function, and \(B(x; p)\) is a Bernoulli distribution.
Note
As this function uses a sigmoid function, you can pass a result of fullyconnected layer (that means
Linear
) to this function directly.Parameters: Returns: A variable representing negative loglikelihood.
Return type:
black_out¶

chainer.functions.
black_out
(x, t, W, samples)[source]¶ BlackOut loss function.
BlackOut loss function is defined as
\[\log(p(t))  \sum_{s \in S} \log(1  p(s)),\]where \(t\) is the correct label, \(S\) is a set of negative examples and \(p(\cdot)\) is likelihood of a given label. And, \(p\) is defined as
\[p(y) = \frac{\exp(W_y^\top x)}{ \sum_{s \in samples} \exp(W_s^\top x)}.\]Parameters: Returns: Loss value.
Return type: See: BlackOut: Speeding up Recurrent Neural Network Language Models With Very Large Vocabularies
See also
connectionist_temporal_classification¶

chainer.functions.
connectionist_temporal_classification
(x, t, blank_symbol, input_length=None, label_length=None)[source]¶ Connectionist Temporal Classification loss function.
Connectionist Temporal Classification(CTC) [Graves2006] is a loss function of sequence labeling where the alignment between the inputs and target is unknown. See also [Graves2012]
Parameters:  x (sequence of Variable) – RNN output at each time.
x
must be a list ofVariable
s. Each element ofx
,x[i]
is aVariable
representing output of RNN at timei
.  t (Variable) – Expected label sequence.
 blank_symbol (int) – Index of blank_symbol. This value must be nonnegative.
 input_length (Variable) – Length of valid sequence for each of mini
batch
x
(optional). If input_length is skipped, It regards that all ofx
is valid input.  label_length (Variable) – Length of valid sequence for each of mini
batch
t
(optional). If label_length is skipped, It regards that all oft
is valid input.
Returns: A variable holding a scalar value of the CTC loss.
Return type: Note
You need to input
x
without applying to activation functions(e.g. softmax function), because this function applies softmax functions tox
before calculating CTC loss to avoid numerical limitations. You also need to apply softmax function to forwarded values before you decode it.Note
This function is differentiable only by
x
.Note
This function supports (batch, sequence, 1dimensional input)data.
[Graves2006] Alex Graves, Santiago Fernandez, Faustino Gomez, Jurgen Schmidhuber, Connectionist Temporal Classification: Labelling Unsegmented Sequence Data with Recurrent Neural Networks [Graves2012] Alex Graves, Supervised Sequence Labelling with Recurrent Neural Networks  x (sequence of Variable) – RNN output at each time.
contrastive¶

chainer.functions.
contrastive
(x0, x1, y, margin=1)[source]¶ Computes contrastive loss.
It takes a pair of variables and a label as inputs. The label is 1 when those two input variables are similar, or 0 when they are dissimilar. Let \(N\) and \(K\) denote minibatch size and the dimension of input variables, respectively. The shape of both input variables should be
(N, K)
.\[L = \frac{1}{2N} \left( \sum_{n=1}^N y_n d_n^2 + (1  y_n) \max ({\rm margin}  d_n, 0)^2 \right)\]where \(d_n = \ {\bf x_0}_n  {\bf x_1}_n \_2\). \(N\) denotes the minibatch size. Input variables, x0 and x1, have \(N\) vectors, and each vector is Kdimensional. Therefore, \({\bf x_0}_n\) and \({\bf x_1}_n\) are \(n\)th Kdimensional vectors of x0 and x1.
Parameters:  x0 (Variable) – The first input variable. The shape should be
(N, K), where N denotes the minibatch size, and K denotes the
dimension of
x0
.  x1 (Variable) – The second input variable. The shape should be
the same as
x0
.  y (Variable) – Labels. All values should be 0 or 1. The shape
should be
(N,)
, where N denotes the minibatch size.  margin (float) – A parameter for contrastive loss. It should be positive value.
Returns: A variable holding a scalar that is the loss value calculated by the above equation.
Return type: Note
This cost can be used to train siamese networks. See Learning a Similarity Metric Discriminatively, with Application to Face Verification for details.
 x0 (Variable) – The first input variable. The shape should be
(N, K), where N denotes the minibatch size, and K denotes the
dimension of
crf1d¶

chainer.functions.
crf1d
(cost, xs, ys)[source]¶ Calculates negative loglikelihood of linearchain CRF.
It takes a transition cost matrix, a sequence of costs, and a sequence of labels. Let \(c_{st}\) be a transition cost from a label \(s\) to a label \(t\), \(x_{it}\) be a cost of a label \(t\) at position \(i\), and \(y_i\) be an expected label at position \(i\). The negative loglikelihood of linearchain CRF is defined as
\[L = \left( \sum_{i=1}^l x_{iy_i} + \ \sum_{i=1}^{l1} c_{y_i y_{i+1}}  {\log(Z)} \right) ,\]where \(l\) is the length of the input sequence and \(Z\) is the normalizing constant called partition function.
Note
When you want to calculate the negative loglikelihood of sequences which have different lengths, sort the sequences in descending order of lengths and transpose the sequences. For example, you have three input seuqnces:
>>> a1 = a2 = a3 = a4 = np.random.uniform(1, 1, 3).astype('f') >>> b1 = b2 = b3 = np.random.uniform(1, 1, 3).astype('f') >>> c1 = c2 = np.random.uniform(1, 1, 3).astype('f')
>>> a = [a1, a2, a3, a4] >>> b = [b1, b2, b3] >>> c = [c1, c2]
where
a1
and all other variables are arrays with(K,)
shape. Make a transpose of the sequences:>>> x1 = np.stack([a1, b1, c1]) >>> x2 = np.stack([a2, b2, c2]) >>> x3 = np.stack([a3, b3]) >>> x4 = np.stack([a4])
and make a list of the arrays:
>>> xs = [x1, x2, x3, x4]
You need to make label sequences in the same fashion. And then, call the function:
>>> cost = chainer.Variable( ... np.random.uniform(1, 1, (3, 3)).astype('f')) >>> ys = [np.zeros(x.shape[0:1], dtype='i') for x in xs] >>> loss = F.crf1d(cost, xs, ys)
It calculates sum of the negative loglikelihood of the three sequences.
Parameters:  cost (Variable) – A \(K \times K\) matrix which holds transition cost between two labels, where \(K\) is the number of labels.
 xs (list of Variable) – Input vector for each label.
len(xs)
denotes the length of the sequence, and eachVariable
holds a \(B \times K\) matrix, where \(B\) is minibatch size, \(K\) is the number of labels. Note that \(B\) s in all the variables are not necessary the same, i.e., it accepts the input sequences with different lengths.  ys (list of Variable) – Expected output labels. It needs to have the
same length as
xs
. EachVariable
holds a \(B\) integer vector. Whenx
inxs
has the different \(B\), correspodingy
has the same \(B\). In other words,ys
must satisfyys[i].shape == xs[i].shape[0:1]
for alli
.
Returns: A variable holding the average negative loglikelihood of the input sequences.
Return type: Note
See detail in the original paper: Conditional Random Fields: Probabilistic Models for Segmenting and Labeling Sequence Data.

chainer.functions.
argmax_crf1d
(cost, xs)[source]¶ Computes a state that maximizes a joint probability of the given CRF.
Parameters:  cost (Variable) – A \(K \times K\) matrix which holds transition cost between two labels, where \(K\) is the number of labels.
 xs (list of Variable) – Input vector for each label.
len(xs)
denotes the length of the sequence, and eachVariable
holds a \(B \times K\) matrix, where \(B\) is minibatch size, \(K\) is the number of labels. Note that \(B\) s in all the variables are not necessary the same, i.e., it accepts the input sequences with different lengths.
Returns: A tuple of
Variable
objects
and alist
ps
. The shape ofs
is(B,)
, whereB
is the minibatch size. ith element ofs
,s[i]
, represents loglikelihood of ith data.ps
is a list ofnumpy.ndarray
orcupy.ndarray
, and denotes the state that maximizes the point probability.len(ps)
is equal tolen(xs)
, and shape of eachps[i]
is the minibatch size of the correspondingxs[i]
. That means,ps[i].shape == xs[i].shape[0:1]
.Return type:
cross_covariance¶

chainer.functions.
cross_covariance
(y, z)[source]¶ Computes the sumsquared crosscovariance penalty between
y
andz
Parameters: Returns: A variable holding a scalar of the cross covariance loss.
Return type: Note
This cost can be used to disentangle variables. See https://arxiv.org/abs/1412.6583v3 for details.
gaussian_kl_divergence¶

chainer.functions.
gaussian_kl_divergence
(mean, ln_var)[source]¶ Computes the KLdivergence of Gaussian variables from the standard one.
Given two variable
mean
representing \(\mu\) andln_var
representing \(\log(\sigma^2)\), this function returns a variable representing the KLdivergence between the given multidimensional Gaussian \(N(\mu, S)\) and the standard Gaussian \(N(0, I)\)\[D_{\mathbf{KL}}(N(\mu, S) \ N(0, I)),\]where \(S\) is a diagonal matrix such that \(S_{ii} = \sigma_i^2\) and \(I\) is an identity matrix.
Parameters: Returns: A variable representing KLdivergence between given gaussian distribution and the standard gaussian.
Return type:
gaussian_nll¶

chainer.functions.
gaussian_nll
(x, mean, ln_var)[source]¶ Computes the negative loglikelihood of a Gaussian distribution.
Given two variable
mean
representing \(\mu\) andln_var
representing \(\log(\sigma^2)\), this function returns the negative loglikelihood of \(x\) on a Gaussian distribution \(N(\mu, S)\),\[\log N(x; \mu, \sigma^2) = \log\left(\sqrt{(2\pi)^D S}\right) + \frac{1}{2}(x  \mu)^\top S^{1}(x  \mu),\]where \(D\) is a dimension of \(x\) and \(S\) is a diagonal matrix where \(S_{ii} = \sigma_i^2\).
Parameters: Returns: A variable representing the negative loglikelihood.
Return type:
hinge¶

chainer.functions.
hinge
(x, t, norm='L1')[source]¶ Computes the hinge loss for a oneofmany classification task.
\[L = \frac{1}{N} \sum_{n=1}^N \sum_{k=1}^K \left[ \max(0, 1  \delta\{l_n = k\} t_{nk}) \right]^p\]where \(N\) denotes the batch size, \(K\) is the number of classes of interest,
\[\begin{split}\delta \{ {\rm condition} \} = \left \{ \begin{array}{cc} 1 & {\rm if~condition} \\ 1 & {\rm otherwise,} \end{array} \right.\end{split}\]and
\[\begin{split}p = \left \{ \begin{array}{cc} 1 & {\rm if~norm} = {\rm 'L1'} \\ 2 & {\rm if~norm} = {\rm 'L2'.} \end{array} \right.\end{split}\]Parameters: Returns: A variable object holding a scalar array of the hinge loss \(L\).
Return type:
huber_loss¶

chainer.functions.
huber_loss
(x, t, delta)[source]¶ Loss function which is less sensitive to outliers in data than MSE.
\[a = x  t\]and
\[\begin{split}L_{\delta}(a) = \left \{ \begin{array}{cc} \frac{1}{2} a^2 & {\rm if~a \leq \delta} \\ \delta (a  \frac{1}{2} \delta) & {\rm otherwise,} \end{array} \right.\end{split}\]Parameters: Returns: A variable object holding a scalar array of the huber loss \(L_{\delta}\).
Return type:  See:
 Huber loss  Wikipedia.
mean_absolute_error¶
mean_squared_error¶
negative_sampling¶

chainer.functions.
negative_sampling
(x, t, W, sampler, sample_size)[source]¶ Negative sampling loss function.
In natural language processing, especially language modeling, the number of words in a vocabulary can be very large. Therefore, you need to spend a lot of time calculating the gradient of the embedding matrix.
By using the negative sampling trick you only need to calculate the gradient for a few sampled negative examples.
The objective function is below:
\[f(x, p) = \log \sigma(x^\top w_p) + \ k E_{i \sim P(i)}[\log \sigma( x^\top w_i)],\]where \(\sigma(\cdot)\) is a sigmoid function, \(w_i\) is the weight vector for the word \(i\), and \(p\) is a positive example. It is approximated with \(k\) examples \(N\) sampled from probability \(P(i)\), like this:
\[f(x, p) \approx \log \sigma(x^\top w_p) + \ \sum_{n \in N} \log \sigma(x^\top w_n).\]Each sample of \(N\) is drawn from the word distribution \(P(w)\). This is calculated as \(P(w) = \frac{1}{Z} c(w)^\alpha\), where \(c(w)\) is the unigram count of the word \(w\), \(\alpha\) is a hyperparameter, and \(Z\) is the normalization constant.
Parameters:  x (Variable) – Batch of input vectors.
 t (Variable) – Vector of ground truth labels.
 W (Variable) – Weight matrix.
 sampler (FunctionType) – Sampling function. It takes a shape and
returns an integer array of the shape. Each element of this array
is a sample from the word distribution.
A
WalkerAlias
object built with the power distribution of word frequency is recommended.  sample_size (int) – Number of samples.
See: Distributed Representations of Words and Phrases and their Compositionality
See also
sigmoid_cross_entropy¶

chainer.functions.
sigmoid_cross_entropy
(x, t, use_cudnn=True, normalize=True)[source]¶ Computes cross entropy loss for presigmoid activations.
Parameters:  x (Variable) – A variable object holding a matrix whose (i, j)th element indicates the unnormalized log probability of the jth unit at the ith example.
 t (Variable) – Variable holding an int32 vector of ground truth labels.
If
t[i] == 1
, correspondingx[i]
is ignored. Loss is zero if all ground truth labels are1
.  normalize (bool) – Variable holding a boolean value which determines the normalization constant. If true, this function normalizes the cross entropy loss across all instances. If else, it only normalizes along a batch size.
Returns: A variable object holding a scalar array of the cross entropy loss.
Return type: Note
This function is differentiable only by
x
.
softmax_cross_entropy¶

chainer.functions.
softmax_cross_entropy
(x, t, use_cudnn=True, normalize=True, cache_score=True, class_weight=None)[source]¶ Computes cross entropy loss for presoftmax activations.
Parameters:  x (Variable) – Variable holding a multidimensional array whose element indicates unnormalized log probability: the first axis of the variable represents the number of samples, and the second axis represents the number of classes. While this function computes a usual softmax cross entropy if the number of dimensions is equal to 2, it computes a cross entropy of the replicated softmax if the number of dimensions is greater than 2.
 t (Variable) – Variable holding an int32 vector of ground truth
labels. If
t[i] == 1
, correspondingx[i]
is ignored.  normalize (bool) – If
True
, this function normalizes the cross entropy loss across all instances. IfFalse
, it only normalizes along a batch size.  cache_score (bool) – When it is
True
, the function stores result of forward computation to use it on backward computation. It reduces computational cost though consumes more memory.  class_weight (ndarray) – An array
that contains constant weights that will be multiplied with the
loss values along with the second dimension. The shape of this
array should be
(x.shape[1],)
. If this is notNone
, each class weightclass_weight[i]
is actually multiplied toy[:, i]
that is the corresponding logsoftmax output ofx
and has the same shape asx
before calculating the actual loss value.
Returns: A variable holding a scalar array of the cross entropy loss.
Return type: Note
This function is differentiable only by
x
.
triplet¶

chainer.functions.
triplet
(anchor, positive, negative, margin=0.2)[source]¶ Computes triplet loss.
It takes a triplet of variables as inputs, \(a\), \(p\) and \(n\): anchor, positive example and negative example respectively. The triplet defines a relative similarity between samples. Let \(N\) and \(K\) denote minibatch size and the dimension of input variables, respectively. The shape of all input variables should be \((N, K)\).
\[L(a, p, n) = \frac{1}{N} \left( \sum_{i=1}^N \max \{d(a_i, p_i)  d(a_i, n_i) + {\rm margin}, 0\} \right)\]where \(d(x_i, y_i) = \ {\bf x}_i  {\bf y}_i \_2^2\).
Parameters:  anchor (Variable) – The anchor example variable. The shape should be \((N, K)\), where \(N\) denotes the minibatch size, and \(K\) denotes the dimension of the anchor.
 positive (Variable) – The positive example variable. The shape should be the same as anchor.
 negative (Variable) – The negative example variable. The shape should be the same as anchor.
 margin (float) – A parameter for triplet loss. It should be a positive value.
Returns: A variable holding a scalar that is the loss value calculated by the above equation.
Return type: Note
This cost can be used to train triplet networks. See Learning Finegrained Image Similarity with Deep Ranking for details.
Mathematical functions¶
arccos¶
arcsin¶
arctan¶
argmax¶

chainer.functions.
argmax
(x, axis=None)[source]¶ Returns index which holds maximum of array elements over a given axis.
Parameters:  x (Variable) – Array to find maximum elements.
 axis (None or int) – Axis over which a max is performed. The default (axis = None) is perform a max over all the dimensions of the input array.
Returns: Output variable.
Return type:
argmin¶

chainer.functions.
argmin
(x, axis=None)[source]¶ Returns index which holds minimum of array elements over a given axis.
Parameters:  x (Variable) – Array to find minimum elements.
 axis (None or int) – Axis over which a min is performed. The default (axis = None) is perform a min over all the dimensions of the input array.
Returns: Output variable.
Return type:
batch_inv¶

chainer.functions.
batch_inv
(a)[source]¶ Computes the inverse of a batch of square matrices.
Parameters: a (Variable) – Input array to compute the inverse for. Shape of the array should be (m, n, n)
wherem
is the number of matrices in the batch, andn
is the dimensionality of a square matrix.Returns: Inverse of every matrix in the batch of matrices. Return type: Variable
batch_l2_norm_squared¶

chainer.functions.
batch_l2_norm_squared
(x)[source]¶ L2 norm (a.k.a. Euclidean norm) squared.
This function implements the square of L2 norm on a vector. No reduction along batch axis is done.
Parameters: x (Variable) – Input variable. The first dimension is assumed to be the minibatch dimension. If x
has more than two dimensions all but the first dimension are flattened to one dimension.Returns: Two dimensional output variable. Return type: Variable
batch_matmul¶

chainer.functions.
batch_matmul
(a, b, transa=False, transb=False)[source]¶ Computes the batch matrix multiplications of two sets of arrays.
Parameters:  a (Variable) – The left operand of the batch matrix multiplications.
A 2D array of shape
(B, N)
is considered as B \(N \times 1\) matrices. A 3D array of shape(B, M, N)
is considered as B \(M \times N\) matrices.  b (Variable) – The right operand of the batch matrix multiplications.
Its array is treated as matrices in the same way as
a
‘s array.  transa (bool) – If
True
, transpose each matrix ina
.  transb (bool) – If
True
, transpose each matrix inb
.
Returns: The result of the batch matrix multiplications as a 3D array.
Return type:  a (Variable) – The left operand of the batch matrix multiplications.
A 2D array of shape
bias¶

chainer.functions.
bias
(x, y, axis=1)[source]¶ Elementwise summation with broadcasting.
Computes a elementwise summation of two input variables, with the shape of the latter variable broadcasted to match the shape of the former.
axis
is the first axis of the first variable along which the second variable is applied.The term “broadcasting” here comes from Caffe’s bias layer so the “broadcasting” with the following arguments:
x : 100 x 3 x 40 x 60 y : 3 x 40 axis : 1
is equivalent to the following numpy broadcasting:
x : 100 x 3 x 40 x 60 y : 1 x 3 x 40 x 1
Note that how the
axis
indicates to which axis ofx
we applyy
.Parameters: Returns: Output variable.
Return type:
ceil¶
clip¶
cosh¶
floor¶
inv¶
linear_interpolate¶
log10¶
log2¶
logsumexp¶

chainer.functions.
logsumexp
(x, axis=None)[source]¶ Logsumexp of array elements over a given axis.
This function calculates logarithm of sum of exponential of array elements.
\[y_i = \log\left(\sum_j \exp(x_{ij})\right)\]Parameters:  x (Variable) – Elements to logsumexp.
 axis (None, int, or tuple of int) – Axis which a sum is performed. The default (axis = None) is perform a sum over all the dimensions of the input array.
Returns: Output variable.
Return type:
matmul¶

chainer.functions.
matmul
(a, b, transa=False, transb=False)[source]¶ Computes the matrix multiplication of two arrays.
Parameters:  a (Variable) – The left operand of the matrix multiplication.
A 1D array of shape
(N,)
is considered as an \(N \times 1\) matrix. A 2D array of shape(M, N)
is considered as an \(M \times N\) matrix.  b (Variable) – The right operand of the matrix multiplication.
Its array is treated as a matrix in the same way as
a
‘s array.  transa (bool) – If
True
, transposea
.  transb (bool) – If
True
, transposeb
.
Returns: The result of the matrix multiplication as a 2D array.
Return type:  a (Variable) – The left operand of the matrix multiplication.
A 1D array of shape
max¶

chainer.functions.
max
(x, axis=None, keepdims=False)[source]¶ Maximum of array elements over a given axis.
Parameters:  x (Variable) – Array to be maximized.
 axis (None, int, or tuple of int) – Axis over which a max is performed. The default (axis = None) is perform a max over all the dimensions of the input array.
Returns: Output variable.
Return type:
maximum¶
min¶

chainer.functions.
min
(x, axis=None, keepdims=False)[source]¶ Minimum of array elements over a given axis.
Parameters:  x (Variable) – Array to be minimized.
 axis (None, int, or tuple of int) – Axis over which a min is performed. The default (axis = None) is perform a min over all the dimensions of the input array.
Returns: Output variable.
Return type:
minimum¶
rsqrt¶
scale¶

chainer.functions.
scale
(x, y, axis=1)[source]¶ Elementwise product with broadcasting.
Computes a elementwise product of two input variables, with the shape of the latter variable broadcasted to match the shape of the former.
axis
is the first axis of the first variable along which the second variable is applied.The term “broadcasting” here comes from Caffe’s scale layer so the “broadcasting” with the following arguments:
x : 100 x 3 x 40 x 60 y : 3 x 40 axis : 1
is equivalent to the following numpy broadcasting:
x : 100 x 3 x 40 x 60 y : 1 x 3 x 40 x 1
Note that how the
axis
indicates to which axis ofx
we applyy
.Parameters: Returns: Output variable.
Return type:
sinh¶
sqrt¶
square¶

chainer.functions.
square
(x)[source]¶ Elementwise square function.
\[y_i = x_i ^ 2.\]Parameters: x (chainer.Variable or numpy.ndarray
or cupy.ndarray) – Input variable.Returns: Output variable. Return type: Variable
squared_difference¶
sum¶

chainer.functions.
sum
(x, axis=None)[source]¶ Sum of array elements over a given axis.
Parameters:  x (Variable) – Elements to sum.
 axis (None, int, or tuple of int) – Axis which a sum is performed. The default (axis = None) is perform a sum over all the dimensions of the input array.
Returns: Output variable.
Return type:
Noise injections¶
dropout¶

chainer.functions.
dropout
(x, ratio=0.5, train=True)[source]¶ Drops elements of input variable randomly.
This function drops input elements randomly with probability
ratio
and scales the remaining elements by factor1 / (1  ratio)
. In testing mode, it does nothing and just returnsx
.Parameters: Returns: Output variable.
Return type: See the paper by G. Hinton: Improving neural networks by preventing coadaptation of feature detectors.
Normalization functions¶
batch_normalization¶

chainer.functions.
batch_normalization
(x, gamma, beta, eps=2e05, running_mean=None, running_var=None, decay=0.9, use_cudnn=True)[source]¶ Batch normalization function.
It takes the input variable
x
and two parameter variablesgamma
andbeta
. The input must have the batch size and the features (or channels) as the first two dimensions of its shape. The input can have more than two dimensions, where the remaining dimensions are considered as spatial dimensions, which are considered as a part of the batch size. That is, the total batch size will be considered to be the product of all dimensions except the second dimension.Note: If this function is called, it will not be possible to access the updated running mean and variance statistics, because they are members of the function object, which cannot be accessed by the caller. If it is desired to access the updated running statistics, it is necessary to get a new instance of the function object, call the object, and then access the running_mean and/or running_var attributes. See the corresponding Link class for an example of how to do this.
Parameters:  x (Variable) – Input variable.
 gamma (Variable) – Scaling parameter of normalized data.
 beta (Variable) – Shifting parameter of scaled normalized data.
 eps (float) – Epsilon value for numerical stability.
 running_mean (array) – Running average of the mean. This is a
running average of the mean over several minibatches using
the decay parameter. If
None
, the running average is not computed. If this isNone
, thenrunnng_var
must also beNone
.  running_var (array) – Running average of the variance. This is a
running average of the variance over several minibatches using
the decay parameter. If
None
, the running average is not computed. If this isNone
, thenrunning_mean
must also beNone
.  decay (float) – Decay rate of moving average. It is used during training.
 use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
See: Batch Normalization: Accelerating Deep Network Training by Reducing Internal Covariate Shift
See also
links.BatchNormalization
fixed_batch_normalization¶

chainer.functions.
fixed_batch_normalization
(x, gamma, beta, mean, var, eps=2e05, use_cudnn=True)[source]¶ Batch normalization function with fixed statistics.
This is a variant of batch normalization, where the mean and variance statistics are given by the caller as fixed variables. This is used on testing mode of the batch normalization layer, where batch statistics cannot be used for prediction consistency.
Parameters:  x (Variable) – Input variable.
 gamma (Variable) – Scaling parameter of normalized data.
 beta (Variable) – Shifting parameter of scaled normalized data.
 mean (Variable) – Shifting parameter of input.
 var (Variable) – Square of scaling parameter of input.
 eps (float) – Epsilon value for numerical stability.
 use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
See also
functions.batch_normalization()
,links.BatchNormalization
local_response_normalization¶

chainer.functions.
local_response_normalization
(x, n=5, k=2, alpha=0.0001, beta=0.75)[source]¶ Local response normalization across neighboring channels.
This function implements normalization across channels. Let \(x\) an input image with \(N\) channels. Then, this function computes an output image \(y\) by following formula:
\[y_i = {x_i \over \left( k + \ \alpha \sum_{j=\max{1, i  n/2}}^{\min{N, i + n/2}} \ x_j^2 \right)^\beta}.\]Parameters: Returns: Output variable.
Return type: See: Section 3.3 of ImageNet Classification with Deep Convolutional Neural Networks
normalize¶

chainer.functions.
normalize
(x, eps=1e05)[source]¶ L2 norm squared (a.k.a. Euclidean norm).
This function implements L2 normalization on a 1D vector. No reduction is done along batch axis. Let \(x\) be an input vector of dimension \((N, K)\), where \(N\) and \(K\) denote minibatch size and the dimension of the input variable. Then, this function computes an output vector \(y\) by the following equation:
\[y_i = {x_i \over \ x_i \_2}\]\(eps\) is used to avoid division by zero when \(x_i=0\)
Parameters: Returns: Two dimensional output variable, the same shape as \(x\).
Return type:
Spatial pooling¶
average_pooling_2d¶

chainer.functions.
average_pooling_2d
(x, ksize, stride=None, pad=0, use_cudnn=True)[source]¶ Spatial average pooling function.
This function acts similarly to
Convolution2D
, but it computes the average of input spatial patch for each channel without any parameter instead of computing the inner products.Parameters:  x (Variable) – Input variable.
 ksize (int or pair of ints) – Size of pooling window.
ksize=k
andksize=(k, k)
are equivalent.  stride (int or pair of ints or None) – Stride of pooling applications.
stride=s
andstride=(s, s)
are equivalent. IfNone
is specified, then it uses same stride as the pooling window size.  pad (int or pair of ints) – Spatial padding width for the input array.
pad=p
andpad=(p, p)
are equivalent.  use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
Returns: Output variable.
Return type: Note
This function currently does not support
cover_all
mode asmax_pooling_2d()
. Average pooling runs in noncoverall mode.
average_pooling_nd¶

chainer.functions.
average_pooling_nd
(x, ksize, stride=None, pad=0, use_cudnn=True)[source]¶ Ndimensionally spatial average pooling function.
This function provides a Ndimensionally generalized version of
average_pooling_2d()
. This acts similarly toConvolutionND
, but it computes the average of input spatial patch for each channel without any parameter instead of computing the inner products.Parameters:  x (Variable) – Input variable.
 ksize (int or tuple of ints) – Size of pooling window.
ksize=k
andksize=(k, k, ..., k)
are equivalent.  stride (int or tuple of ints or None) – Stride of pooling applications.
stride=s
andstride=(s, s, ..., s)
are equivalent. IfNone
is specified, then it uses same stride as the pooling window size.  pad (int or tuple of ints) – Spatial padding width for the input array.
pad=p
andpad=(p, p, ..., p)
are equivalent.  use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation. cuDNN supports more than onedimensional pooling.
Returns: Output variable.
Return type: Note
This function currently does not support
cover_all
mode asmax_pooling_nd()
. Average pooling runs in noncoverall mode.
max_pooling_2d¶

chainer.functions.
max_pooling_2d
(x, ksize, stride=None, pad=0, cover_all=True, use_cudnn=True)[source]¶ Spatial max pooling function.
This function acts similarly to
Convolution2D
, but it computes the maximum of input spatial patch for each channel without any parameter instead of computing the inner products.Parameters:  x (Variable) – Input variable.
 ksize (int or pair of ints) – Size of pooling window.
ksize=k
andksize=(k, k)
are equivalent.  stride (int or pair of ints or None) – Stride of pooling applications.
stride=s
andstride=(s, s)
are equivalent. IfNone
is specified, then it uses same stride as the pooling window size.  pad (int or pair of ints) – Spatial padding width for the input array.
pad=p
andpad=(p, p)
are equivalent.  cover_all (bool) – If
True
, all spatial locations are pooled into some output pixels. It may make the output size larger.  use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
Returns: Output variable.
Return type:
max_pooling_nd¶

chainer.functions.
max_pooling_nd
(x, ksize, stride=None, pad=0, cover_all=True, use_cudnn=True)[source]¶ Ndimensionally spatial max pooling function.
This function provides a Ndimensionally generalized version of
max_pooling_2d()
. This acts similarly toConvolutionND
, but it computes the maximum of input spatial patch for each channel without any parameter instead of computing the inner products.Parameters:  x (Variable) – Input variable.
 ksize (int or tuple of ints) – Size of pooling window.
ksize=k
andksize=(k, k, ..., k)
are equivalent.  stride (int or tuple of ints or None) – Stride of pooling applications.
stride=s
andstride=(s,s, ..., s)
are equivalent. IfNone
is specified, then it uses same stride as the pooling window size.  pad (int or tuple of ints) – Spatial padding width for the input array.
pad=p
andpad=(p, p, ..., p)
are equivalent.  cover_all (bool) – If
True
, all spatial locations are pooled into some output pixels. It may make the output size larger.  use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation. cuDNN supports more than onedimensional pooling.
Returns: Output variable.
Return type:
roi_pooling_2d¶

chainer.functions.
roi_pooling_2d
(x, rois, outh, outw, spatial_scale)[source]¶ Spatial Region of Interest (ROI) pooling function.
This function acts similarly to
MaxPooling2D
, but it computes the maximum of input spatial patch for each channel with the region of interest.Parameters:  x (Variable) – Input variable. The shape is expected to be 4 dimentional: (n: batch, c: channel, h, height, w: width).
 rois (Variable) – Input roi variable. The shape is expected to be (n: data size, 5), and each datum is set as below: (batch_index, x_min, y_min, x_max, y_max).
 outh (int) – Height of output image after pooled.
 outw (int) – Width of output image after pooled.
 spatial_scale (float) – Scale of the roi is resized.
Returns: Output variable.
Return type: See the original paper proposing ROIPooling: Fast RCNN.
spatial_pyramid_pooling_2d¶

chainer.functions.
spatial_pyramid_pooling_2d
(x, pyramid_height, pooling_class, use_cudnn=True)[source]¶ Spatial pyramid pooling function.
It outputs a fixedlength vector regardless of input feature map size.
It performs pooling operation to the input 4Darray
x
with different kernel sizes and padding sizes, and then flattens all dimensions except first dimension of all pooling results, and finally concatenates them along second dimension.At \(i\)th pyramid level, the kernel size \((k_h^{(i)}, k_w^{(i)})\) and padding size \((p_h^{(i)}, p_w^{(i)})\) of pooling operation are calculated as below:
\[\begin{split}k_h^{(i)} &= \lceil b_h / 2^i \rceil, \\ k_w^{(i)} &= \lceil b_w / 2^i \rceil, \\ p_h^{(i)} &= (2^i k_h^{(i)}  b_h) / 2, \\ p_w^{(i)} &= (2^i k_w^{(i)}  b_w) / 2,\end{split}\]where \(\lceil \cdot \rceil\) denotes the ceiling function, and \(b_h, b_w\) are height and width of input variable
x
, respectively. Note that index of pyramid level \(i\) is zerobased.See detail in paper: Spatial Pyramid Pooling in Deep Convolutional Networks for Visual Recognition.
Parameters:  x (Variable) – Input variable. The shape of
x
should be(batchsize, # of channels, height, width)
.  pyramid_height (int) – Number of pyramid levels
 pooling_class (MaxPooling2D or AveragePooling2D) – Only MaxPooling2D class can be available for now.
 use_cudnn (bool) – If
True
and cuDNN is enabled, then this function uses cuDNN as the core implementation.
Returns: Output variable. The shape of the output variable will be \((batchsize, c \sum_{h=0}^{H1} 2^{2h}, 1, 1)\), where \(c\) is the number of channels of input variable
x
and \(H\) is the number of pyramid levels.Return type: Note
This function uses some pooling classes as components to perform spatial pyramid pooling. Now it supports only
MaxPooling2D
as elemental pooling operator so far. x (Variable) – Input variable. The shape of
unpooling_2d¶

chainer.functions.
unpooling_2d
(x, ksize, stride=None, pad=0, outsize=None, cover_all=True)[source]¶ Inverse operation of pooling for 2d array.
This function acts similarly to
Deconvolution2D
, but it spreads input 2d array’s value without any parameter instead of computing the inner products.Parameters:  x (Variable) – Input variable.
 ksize (int or pair of ints) – Size of pooling window.
ksize=k
andksize=(k, k)
are equivalent.  stride (int, pair of ints or None) – Stride of pooling applications.
stride=s
andstride=(s, s)
are equivalent. IfNone
is specified, then it uses same stride as the pooling window size.  pad (int or pair of ints) – Spatial padding width for the input array.
pad=p
andpad=(p, p)
are equivalent.  outsize (None or pair of ints) – Expected output size (height, width)
of array after the operation. If
None
, the size (height or width) is estimated from the size of input array in first batch withget_deconv_outsize()
. If outsize is notNone
, the result of outsize applied toget_conv_outsize()
must be equal to the shape of the 2d array in the input batchx
.  cover_all (bool) – If
True
, the output size may be smaller than the size ifcover_all
isFalse
. This flag serves to align behavior to the pooling functions which can cover all input locations, seemax_pooling_2d()
andconvolution_2d()
.
Returns: Output variable.
Return type:
upsampling_2d¶

chainer.functions.
upsampling_2d
(x, indexes, ksize, stride=None, pad=0, outsize=None, cover_all=True)[source]¶ Upsampling using pooling indices.
This function produces an upsampled image using pooling indices.
Example
It should be noted that you need to specify
use_cudnn=False
when you create MaxPooling2D object because if cuDNN used for operating max pooling,indexes
is never created and stored in the MaxPooling2D object.>>> p = F.MaxPooling2D(2, 2, use_cudnn=False) >>> x = np.arange(1, 37).reshape(1, 1, 6, 6).astype('f') >>> x = chainer.Variable(x) >>> x.data array([[[[ 1., 2., 3., 4., 5., 6.], [ 7., 8., 9., 10., 11., 12.], [ 13., 14., 15., 16., 17., 18.], [ 19., 20., 21., 22., 23., 24.], [ 25., 26., 27., 28., 29., 30.], [ 31., 32., 33., 34., 35., 36.]]]], dtype=float32)
This is the original
x
before max pooling.>>> pooled_x = p(x) >>> pooled_x.data array([[[[ 8., 10., 12.], [ 20., 22., 24.], [ 32., 34., 36.]]]], dtype=float32)
This is the output of the max pooling operation.
upsampling_2d
needsindexes
array stored in the max pooling objectp
.>>> upsampled_x = F.upsampling_2d( ... pooled_x, p.indexes, p.kh, p.sy, p.ph, x.shape[2:]) >>> upsampled_x.shape (1, 1, 6, 6) >>> upsampled_x.data array([[[[ 0., 0., 0., 0., 0., 0.], [ 0., 8., 0., 10., 0., 12.], [ 0., 0., 0., 0., 0., 0.], [ 0., 20., 0., 22., 0., 24.], [ 0., 0., 0., 0., 0., 0.], [ 0., 32., 0., 34., 0., 36.]]]], dtype=float32)
Parameters:  x (Variable) – Input variable.
 indexes (ndarray) – Index array that was used to calculate x with MaxPooling2D.
 ksize (int or (int, int) – ksize attribute of MaxPooling2D object that is used to calculate x
 stride (int or (int, int) – stride attribute of MaxPooling2D object that is used to calculate x
 pad (int or (int, int) – pad attribute of MaxPooling2D object that is used to calculate x
 outsize ((int, int) – Expected output size (height, width).
 cover_all (bool) – Whether cover_all is used in the MaxPooling2D object or not.
Returns: Output variable.
Return type:
Utility functions¶
forget¶

chainer.functions.
forget
(func, *xs)[source]¶ Call a function without storing internal results.
On a forward propagation Chainer stores all internal results of
Function
on a computational graph as they are required on backwardpropagation. These results consume too much memory when the internal results are too large. This method forgets such internal results on forward propagation, and still supports backpropagation with recalculation.In a forward propagation, this method calls a given function with given variables without creating a computational graph. That means, no internal results are stored. In a backward propagation this method calls the given function again to create a computational graph to execute backpropagation.
This method reduces internal memory usage. Instead it requires more calculation time as it calls the function twice.
Example
Let
f
be a function defined as:>>> def f(a, b): ... return a + b + a
and,
x
andy
beVariable
:>>> x = chainer.Variable(np.random.uniform(1, 1, 5).astype('f')) >>> y = chainer.Variable(np.random.uniform(1, 1, 5).astype('f'))
When
z
is calculated asz = f(x, y)
, its internal resultx + y
is stored in memory. Instead if you callf
withforget()
:>>> z = F.forget(f, x, y)
internal
x + y
is forgotten.Note
The method does not support functions behaving randomly, such as
dropout()
andnegative_sampling()
. It is because first results of these function differ from the second one.Parameters: Returns: A variable
func
returns. If it returns a tuple, the method returns a tuple too.Return type:
Standard Link implementations¶
Chainer provides many Link
implementations in the
chainer.links
package.
Note
Some of the links are originally defined in the chainer.functions
namespace. They are still left in the namespace for backward compatibility,
though it is strongly recommended to use them via the chainer.links
package.
Learnable connections¶
Bias¶

class
chainer.links.
Bias
(axis=1, shape=None)[source]¶ Broadcasted elementwise summation with learnable parameters.
Computes a elementwise summation as
bias()
function does except that its second input is a learnable bias parameter \(b\) the link has.Parameters:  axis (int) – The first axis of the first input of
bias()
function along which its second input is applied.  shape (tuple of ints) – Shape of the learnable bias parameter. If
None
, this link does not have learnable parameters so an explicit bias needs to be given to its__call__
method’s second input.
See also
See
bias()
for details.Variables: b (Variable) – Bias parameter if shape
is given. Otherwise, no attributes. axis (int) – The first axis of the first input of
Bilinear¶

class
chainer.links.
Bilinear
(left_size, right_size, out_size, nobias=False, initialW=None, initial_bias=None)[source]¶ Bilinear layer that performs tensor multiplication.
Bilinear is a primitive link that wraps the
bilinear()
functions. It holds parametersW
,V1
,V2
, andb
corresponding to the arguments ofbilinear()
.Parameters:  left_size (int) – Dimension of input vector \(e^1\) (\(J\))
 right_size (int) – Dimension of input vector \(e^2\) (\(K\))
 out_size (int) – Dimension of output vector \(y\) (\(L\))
 nobias (bool) – If
True
, parametersV1
,V2
, andb
are omitted.  initialW (3D numpy array) – Initial value of \(W\).
Shape of this argument must be
(left_size, right_size, out_size)
. IfNone
, \(W\) is initialized by centered Gaussian distribution properly scaled according to the dimension of inputs and outputs. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  initial_bias (tuple) – Initial values of \(V^1\), \(V^2\)
and \(b\). The length this argument must be 3.
Each element of this tuple must have the shapes of
(left_size, output_size)
,(right_size, output_size)
, and(output_size,)
, respectively. IfNone
, \(V^1\) and \(V^2\) is initialized by scaled centered Gaussian distributions and \(b\) is set to \(0\). May also be a tuple of callables that takenumpy.ndarray
orcupy.ndarray
and edit its value.
See also
See
chainer.functions.bilinear()
for details.Variables:
Convolution2D¶

class
chainer.links.
Convolution2D
(in_channels, out_channels, ksize, stride=1, pad=0, wscale=1, bias=0, nobias=False, use_cudnn=True, initialW=None, initial_bias=None, deterministic=False)[source]¶ Twodimensional convolutional layer.
This link wraps the
convolution_2d()
function and holds the filter weight and bias vector as parameters.Parameters:  in_channels (int) – Number of channels of input arrays. If
None
, parameter initialization will be deferred until the first forward data pass at which time the size will be determined.  out_channels (int) – Number of channels of output arrays.
 ksize (int or pair of ints) – Size of filters (a.k.a. kernels).
ksize=k
andksize=(k, k)
are equivalent.  stride (int or pair of ints) – Stride of filter applications.
stride=s
andstride=(s, s)
are equivalent.  pad (int or pair of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p)
are equivalent.  wscale (float) – Scaling factor of the initial weight.
 bias (float) – Initial bias value.
 nobias (bool) – If
True
, then this link does not use the bias term.  use_cudnn (bool) – If
True
, then this link uses cuDNN if available.  initialW (4D array) – Initial weight value. If
None
, then this function uses to initializewscale
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  initial_bias (1D array) – Initial bias value. If
None
, then this function uses to initializebias
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  deterministic (bool) – The output of this link can be
nondeterministic when it uses cuDNN.
If this option is
True
, then it forces cuDNN to use a deterministic algorithm. This option is only available for cuDNN version >= v4.
See also
See
chainer.functions.convolution_2d()
for the definition of twodimensional convolution.Variables:  in_channels (int) – Number of channels of input arrays. If
ConvolutionND¶

class
chainer.links.
ConvolutionND
(ndim, in_channels, out_channels, ksize, stride=1, pad=0, initialW=None, initial_bias=None, use_cudnn=True, cover_all=False)[source]¶ Ndimensional convolution layer.
This link wraps the
convolution_nd()
function and holds the filter weight and bias vector as parameters.Parameters:  ndim (int) – Number of spatial dimensions.
 in_channels (int) – Number of channels of input arrays.
 out_channels (int) – Number of channels of output arrays.
 ksize (int or tuple of ints) – Size of filters (a.k.a. kernels).
ksize=k
andksize=(k, k, ..., k)
are equivalent.  stride (int or tuple of ints) – Stride of filter application.
stride=s
andstride=(s, s, ..., s)
are equivalent.  pad (int or tuple of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p, ..., p)
are equivalent.  initialW – Value used to initialize the filter weight. May be an
initializer instance or another value that
init_weight()
helper function can take.  initial_bias – Value used to initialize the bias vector. May be an
initializer instance or another value except
None
thatinit_weight()
helper function can take. IfNone
is given, this link does not use the bias vector.  use_cudnn (bool) – If
True
, then this link uses cuDNN if available. Seeconvolution_nd()
for exact conditions of cuDNN availability.  cover_all (bool) – If
True
, all spatial locations are convoluted into some output pixels. It may make the output size larger.cover_all
needs to beFalse
if you want to use cuDNN.
See also
See
convolution_nd()
for the definition of Ndimensional convolution. Seeconvolution_2d()
for the definition of twodimensional convolution.Variables:
Deconvolution2D¶

class
chainer.links.
Deconvolution2D
(in_channels, out_channels, ksize, stride=1, pad=0, wscale=1, bias=0, nobias=False, outsize=None, use_cudnn=True, initialW=None, initial_bias=None, deterministic=False)[source]¶ Two dimensional deconvolution function.
This link wraps the
deconvolution_2d()
function and holds the filter weight and bias vector as parameters.Parameters:  in_channels (int) – Number of channels of input arrays. If
None
, parameter initialization will be deferred until the first forward data pass at which time the size will be determined.  out_channels (int) – Number of channels of output arrays.
 ksize (int or pair of ints) – Size of filters (a.k.a. kernels).
ksize=k
andksize=(k, k)
are equivalent.  stride (int or pair of ints) – Stride of filter applications.
stride=s
andstride=(s, s)
are equivalent.  pad (int or pair of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p)
are equivalent.  wscale (float) – Scaling factor of the initial weight.
 bias (float) – Initial bias value.
 nobias (bool) – If
True
, then this function does not use the bias term.  outsize (tuple) – Expected output size of deconvolutional operation.
It should be pair of height and width \((out_H, out_W)\).
Default value is
None
and the outsize is estimated by input size, stride and pad.  use_cudnn (bool) – If
True
, then this function uses cuDNN if available.  initialW (4D array) – Initial weight value. If
None
, then this function uses to initializewscale
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  initial_bias (1D array) – Initial bias value. If
None
, then this function uses to initializebias
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  deterministic (bool) – The output of this link can be
nondeterministic when it uses cuDNN.
If this option is
True
, then it forces cuDNN to use a deterministic algorithm. This option is only available for cuDNN version >= v4.
The filter weight has four dimensions \((c_I, c_O, k_H, k_W)\) which indicate the number of input channels, output channels, height and width of the kernels, respectively. The filter weight is initialized with i.i.d. Gaussian random samples, each of which has zero mean and deviation \(\sqrt{1/(c_I k_H k_W)}\) by default. The deviation is scaled by
wscale
if specified.The bias vector is of size \(c_O\). Its elements are initialized by
bias
argument. Ifnobias
argument is set to True, then this function does not hold the bias parameter.See also
See
chainer.functions.deconvolution_2d()
for the definition of twodimensional convolution. in_channels (int) – Number of channels of input arrays. If
DeconvolutionND¶

class
chainer.links.
DeconvolutionND
(ndim, in_channels, out_channels, ksize, stride=1, pad=0, outsize=None, initialW=None, initial_bias=0, use_cudnn=True)[source]¶ Ndimensional deconvolution function.
This link wraps
deconvolution_nd()
function and holds the filter weight and bias vector as its parameters.Parameters:  ndim (int) – Number of spatial dimensions.
 in_channels (int) – Number of channels of input arrays.
 out_channels (int) – Number of channels of output arrays.
 ksize (int or tuple of ints) – Size of filters (a.k.a. kernels).
ksize=k
andksize=(k, k, ..., k)
are equivalent.  stride (int or tuple of ints) – Stride of filter application.
stride=s
andstride=(s, s, ..., s)
are equivalent.  pad (int or tuple of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p, ..., p)
are equivalent.  outsize (tuple of ints) – Expected output size of deconvolutional
operation. It should be a tuple of ints that represents the output
size of each dimension. Default value is
None
and the outsize is estimated with input size, stride and pad.  initialW – Value used to initialize the filter weight. May be an
initializer instance of another value the same with that
init_weight()
function can take.  initial_bias – Value used to initialize the bias vector. May be an
initializer instance or another value except
None
the same with thatinit_weight()
function can take. IfNone
is supplied, this link does not use the bias vector.  use_cudnn (bool) – If
True
, then this link uses cuDNN if available.
See also
Variables:
DilatedConvolution2D¶

class
chainer.links.
DilatedConvolution2D
(in_channels, out_channels, ksize, stride=1, pad=0, dilate=1, wscale=1, bias=0, nobias=False, use_cudnn=True, initialW=None, initial_bias=None)[source]¶ Twodimensional dilated convolutional layer.
This link wraps the
dilated_convolution_2d()
function and holds the filter weight and bias vector as parameters.Parameters:  in_channels (int) – Number of channels of input arrays. If
None
, parameter initialization will be deferred until the first forward data pass at which time the size will be determined.  out_channels (int) – Number of channels of output arrays.
 ksize (int or pair of ints) – Size of filters (a.k.a. kernels).
ksize=k
andksize=(k, k)
are equivalent.  stride (int or pair of ints) – Stride of filter applications.
stride=s
andstride=(s, s)
are equivalent.  pad (int or pair of ints) – Spatial padding width for input arrays.
pad=p
andpad=(p, p)
are equivalent.  dilate (int or pair of ints) – Dilation factor of filter applications.
dilate=d
anddilate=(d, d)
are equivalent.  wscale (float) – Scaling factor of the initial weight.
 bias (float) – Initial bias value.
 nobias (bool) – If
True
, then this link does not use the bias term.  use_cudnn (bool) – If
True
, then this link uses cuDNN if available.  initialW (4D array) – Initial weight value. If
None
, then this function uses to initializewscale
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  initial_bias (1D array) – Initial bias value. If
None
, then this function uses to initializebias
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.
See also
See
chainer.functions.dilated_convolution_2d()
for the definition of twodimensional dilated convolution.Variables:  in_channels (int) – Number of channels of input arrays. If
EmbedID¶

class
chainer.links.
EmbedID
(in_size, out_size, initialW=None, ignore_label=None)[source]¶ Efficient linear layer for onehot input.
This is a link that wraps the
embed_id()
function. This link holds the ID (word) embedding matrixW
as a parameter.Parameters:  in_size (int) – Number of different identifiers (a.k.a. vocabulary size).
 out_size (int) – Size of embedding vector.
 initialW (2D array) – Initial weight value. If
None
, then the matrix is initialized from the standard normal distribution. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  ignore_label (int or None) – If
ignore_label
is an int value,i
th column of return value is filled with0
.
See also
Variables: W (Variable) – Embedding parameter matrix.
GRU¶

class
chainer.links.
GRU
(n_units, n_inputs=None, init=None, inner_init=None, bias_init=0)[source]¶ Stateless Gated Recurrent Unit function (GRU).
GRU function has six parameters \(W_r\), \(W_z\), \(W\), \(U_r\), \(U_z\), and \(U\). All these parameters are \(n \times n\) matrices, where \(n\) is the dimension of hidden vectors.
Given two inputs a previous hidden vector \(h\) and an input vector \(x\), GRU returns the next hidden vector \(h'\) defined as
\[\begin{split}r &=& \sigma(W_r x + U_r h), \\ z &=& \sigma(W_z x + U_z h), \\ \bar{h} &=& \tanh(W x + U (r \odot h)), \\ h' &=& (1  z) \odot h + z \odot \bar{h},\end{split}\]where \(\sigma\) is the sigmoid function, and \(\odot\) is the elementwise product.
GRU
does not hold the value of hidden vector \(h\). So this is stateless. UseStatefulGRU
as a stateful GRU.Parameters:  See:
 On the Properties of Neural Machine Translation: EncoderDecoder Approaches [Cho+, SSST2014].
 Empirical Evaluation of Gated Recurrent Neural Networks on Sequence Modeling [Chung+NIPS2014 DLWorkshop].
See also
Highway¶

class
chainer.links.
Highway
(in_out_size, nobias=False, activate=<function relu>, init_Wh=None, init_Wt=None, init_bh=None, init_bt=1)[source]¶ Highway module.
In highway network, two gates are added to the ordinal nonlinear transformation (\(H(x) = activate(W_h x + b_h)\)). One gate is the transform gate \(T(x) = \sigma(W_t x + b_t)\), and the other is the carry gate \(C(x)\). For simplicity, the author defined \(C = 1  T\). Highway module returns \(y\) defined as
\[y = activate(W_h x + b_h) \odot \sigma(W_t x + b_t) + x \odot(1  \sigma(W_t x + b_t))\]The output array has the same spatial size as the input. In order to satisfy this, \(W_h\) and \(W_t\) must be square matrices.
Parameters:  in_out_size (int) – Dimension of input and output vectors.
 nobias (bool) – If
True
, then this function does not use the bias.  activate – Activation function of plain array. \(tanh\) is also available.
 init_Wh (2D array) – Initial weight value of plain array. If
None
, then this function uses it to initializewscale
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  init_bh (1D array) – Initial bias value of plain array. If
None
, then this function uses it to initialize zero vector. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  init_Wt (2D array) – Initial weight value of transform array.
If
None
, then this function uses it to initializewscale
. May also be a callable that takesnumpy.ndarray
orcupy.ndarray
and edits its value.  init_bt (1D array) – Initial bias value of transform array.
Default value is 1 vector.
May also be a callable that takes
numpy.ndarray
orcupy.ndarray
and edits its value. Negative value is recommended by the author of the paper. (e.g. 1, 3, ...).
 See:
 Highway Networks.
Inception¶

class
chainer.links.
Inception
(in_channels, out1, proj3, out3, proj5, out5, proj_pool, conv_init=None, bias_init=None)[source]¶ Inception module of GoogLeNet.
It applies four different functions to the input array and concatenates their outputs along the channel dimension. Three of them are 2D convolutions of sizes 1x1, 3x3 and 5x5. Convolution paths of 3x3 and 5x5 sizes have 1x1 convolutions (called projections) ahead of them. The other path consists of 1x1 convolution (projection) and 3x3 max pooling.
The output array has the same spatial size as the input. In order to satisfy this, Inception module uses appropriate padding for each convolution and pooling.
See: Going Deeper with Convolutions.
Parameters:  in_channels (int) – Number of channels of input arrays.
 out1 (int) – Output size of 1x1 convolution path.
 proj3 (int) – Projection size of 3x3 convolution path.
 out3 (int) – Output size of 3x3 convolution path.
 proj5 (int) – Projection size of 5x5 convolution path.
 out5 (int) – Output size of 5x5 convolution path.
 proj_pool (int) – Projection size of max pooling path.
 conv_init – A callable that takes
numpy.ndarray
orcupy.ndarray
and edits its value. It is used for initialization of the convolution matrix weights. Maybe beNone
to use default initialization.  bias_init – A callable that takes
numpy.ndarray