Welcome to the Graylog documentation¶

Log into the VM¶

We’re going to use rsyslog because we already have it from the Graylog server image. So, go to the image and login with ubuntu/ubuntu.

Modify rsyslog.conf¶

Go to the /etc directory, and use vi, vim (vim Cheat Sheet), or the editor of your choice to modify the /etc/rsyslog.conf file. There are excellent resources on the web for rsyslog configuration.

At the bottom of the file, add the following so messages will forward:

*.* @127.0.0.1:5140;RSYSLOG_SyslogProtocol23Format

In case you wanted to know, @ means UDP, 127.0.0.1 is localhost, and 5140 is the port.

You can find out more about ingesting syslog messages with Graylog in our Syslog configuration guide.

Restart rsyslog¶

Type:

$sudo service rsyslog status
$sudo service rsyslog restart

If you have modified the config file and it is somehow invalid, the service command will not bring rsyslog back up - so don’t worry, you can always delete the line!

Further information on how to send log messages are on a separate page in our documentation

Add a Dashboard¶

We’ll start by adding the message count data to a dashboard. Click Add count to dashboard, and it will say No Dashboards, create one? Yes! Click that.

Give your new dashboard a title and description.

Add a Dashboard Widget¶

Now it will let you create a widget. In this case, we are creating a widget from our search result of message count in the last 8 hours. I like to put a timeframe in the title, and trends are always a big bowl of sunshine.

When you hit create - wa la! Nothing happens. All you UX types, relax, we know. For now, click Dashboards and then the name of your dashboard.

Smile¶

And you’ll end up with the widget you created!

Extra Credit - One more¶

Let’s add a widget for root activity, because that sounds like it may actually be useful. We need to start with a search query for root. Click Search. Type root in the search and select your timeframe. Once the search results come in, click Add count to the dashboard.

Give your chart a title and hit Create.

The new widget is now on the screen. Good job - you’ve got this!

Go play around. If you want to know how to create more exciting charts and graphs, check out the section below.

Extra Credit - Graphs¶

Let’s start by searching for all messages within the last 1 hour. To do this, click Search, select Search in the last 1 hour, and leave the search bar blank. Once the search results populate, expand the messages field in the Search results sidebar and select Quick Values. Click Add to dashboard to add this entire pie chart and data table to your dashboard.

I like to track password changes, privilege assignments, root activity, system events, user logins, etc. Go knock yourself out and show your co-workers.

Once you have a few widgets in your dashboard, go into unlock / edit mode to quickly edit any widget, rearrange them on your dashboard, or delete. Make sure to click Lock to save!

You can find out why dashboards matter at our Dashboard documentation.

Create a Stream¶

In order to set up an alert, we need to first create a stream. Streams process incoming messages in real time based on conditions that you set. Click Streams.

Let’s create a stream for all incoming security/authentication error messages. Click Create Stream.

Type in a Title and Description.

Create a Stream Rule¶

Next, we are going to configure the stream to process our Syslog UDP input messages for any security alerts.

Hit the Edit rules button.

Pick the Syslog UDP Input, and click Add stream rule.

Then, type in the values shown below and hit save.

Then click I’m done!

We have just configured this stream to process in real time all the messages that come in from the security/authorization facility.

Now let’s create the alert.

Create the Alert¶

You can now either output your new stream to a 3rd party application or database, or trigger an alert to ping you in real time when a message that matches our stream rule comes in. Let’s create an alert that will email us when there are more than 2 messages in the last 2 minutes . Click Manage Alerts.

In the Add new alert condition section, let’s configure and add a new alert. Select message count condition, and configure the rest based on my screenshot (input 2’s in every field). Then click Add alert condition.

Send a Test Email¶

In the Callbacks section, select email alert callback, and input your email address in the Receivers section. After you’ve added a callback type and receiver, hit the blue ‘Send test alert’ button.

Going Further¶

If you want to configure an SMTP server, you can refer to The graylog-ctl script.

If you want to make this stream active, just go back to Streams and where you see the stream name, click the green Start stream button.

You are done - go grab a Creamsicle, take a deep breath, and chillax. Tomorrow you can configure all your own logs and alerts. To help, go and get some deep knowledge in the official documentation friendly people that will help and guide you can be found in our support community.

You can find more information on our Alerts page.

Pre-Considerations¶

This is a showcase of Graylog and its cluster mode. Please run this appliance always in a separated network that is isolated from the internet. Read also the notes about production readiness!

Download¶

Download the OVA image. If you are unsure what the latest version number is, take a look at our release page.

Run the image¶

You can run the OVA in many systems like VMware or Virtualbox. In this example we will guide you through running the OVA in the free Virtualbox on OSX.

In Virtualbox select File -> Import appliance:

Hit Continue and keep the suggested settings on the next page as they are. Make sure that you have enough RAM and CPUs on your local machine. You can lower the resources the virtual machine will get assigned but we recommend to not lower it to ensure a good Graylog experience. In fact you might have to raise it if you plan to scale out later and send more messages into Graylog.

Press Import to finish loading the OVA into Virtualbox:

You can now start the VM and should see a login shell like this when the boot completed:

Logging in¶

You can log into the shell of the operating system of the appliance with the user ubuntu and the password ubuntu. You should of course change those credentials.

The web interface is reachable on port 80 at the IP address of your virtual machine. The login prompt of the shell is showing you this IP address, too. (See screenshot above). DHCP should be enabled in your network otherwise take a look into the graylog-ctl command to apply a static IP address to the appliance.

The standard user for the web interface is admin with the password admin.

Basic configuration¶

We are shipping the graylog-ctl tool with the virtual machine appliances to get you started with a customised setup as quickly as possible. Run these (optional) commands to configure the most basic settings of Graylog in the appliance:

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

The graylog-ctl has much more functionality documented . We strongly recommend to learn more about it to ensure smooth operation of your virtual appliance.

VMWare tools¶

If you are using the appliance on a VMWare host, you might want to install the hypervisor tools:

sudo apt-get install -y open-vm-tools

Update OVA to latest Version¶

You can update your Appliance to the newest release without deploying a new template.

Production readiness¶

You can use the Graylog appliances (OVA, Docker, AWS, ...) for small production setups but please consider to harden the security of the box before.

• Set another password for the default ubuntu user
• Disable remote password logins in /etc/ssh/sshd_config and deploy proper ssh keys
• Seperate the box network-wise from the outside, otherwise Elasticsearch and MongoDB can be reached by anyone
• add additional RAM to the appliance and raise the java heap!
• add additional HDD to the appliance and extend disk space.
• add the appliance to your monitoring and metric systems.

If you want to create your own customised setup take a look at our other installation methods.

Prerequisites¶

Make sure to install and configure the following software before installing and starting any Graylog services:

• Java (>= 8)
• MongoDB (>= 2.4)
• Elasticsearch (>= 2.x)

DEB / APT¶

Download and install graylog-2.1-repository_latest.deb via dpkg(1) and also make sure that the apt-transport-https package is installed:

$ sudo apt-get install apt-transport-https
$ wget https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.deb
$ sudo dpkg -i graylog-2.1-repository_latest.deb
$ sudo apt-get update
$ sudo apt-get install graylog-server

After the installation completed successfully, Graylog can be started with the following commands. Make sure to use the correct command for your operating system.

The packages are configured to not start any Graylog services during boot. You can use the following commands to start Graylog when the operating system is booting.

$ wget https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.deb
$ sudo dpkg -i graylog-2.1-repository_latest.deb
$ sudo apt-get update
$ sudo apt-get install graylog-server

deb https://packages.graylog2.org/repo/debian/ stable 2.1

RPM / YUM / DNF¶

Download and install graylog-2.1-repository_latest.rpm via rpm(8):

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.rpm
$ sudo yum install graylog-server

After the installation completed successfully, Graylog can be started with the following commands. Make sure to use the correct command for your operating system.

The packages are configured to not start any Graylog services during boot. You can use the following commands to start Graylog when the operating system is booting.

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.rpm
$ sudo yum clean all
$ sudo yum install graylog-server

[graylog]
name=graylog
baseurl=https://packages.graylog2.org/repo/el/stable/2.1/$basearch/
gpgcheck=1
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-graylog

Step-by-step guides¶

$ sudo apt-get update && sudo apt-get upgrade
$ sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen

$ sudo apt-get install mongodb-server

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch

cluster.name: graylog

$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service

$ wget https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.deb
$ sudo dpkg -i graylog-2.1-repository_latest.deb
$ sudo apt-get update && sudo apt-get install graylog-server

echo -n yourpassword | sha256sum

$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service

$ sudo echo "deb http://ftp.debian.org/debian jessie-backports main" > /etc/apt/sources.list.d/backports.list
$ sudo apt-get update && sudo apt-get upgrade

$ sudo apt-get install apt-transport-https openjdk-8-jre-headless uuid-runtime pwgen

$ sudo apt-get install mongodb-server

$ wget -qO - https://packages.elastic.co/GPG-KEY-elasticsearch | sudo apt-key add -
$ echo "deb https://packages.elastic.co/elasticsearch/2.x/debian stable main" | sudo tee -a /etc/apt/sources.list.d/elasticsearch-2.x.list
$ sudo apt-get update && sudo apt-get install elasticsearch

cluster.name: graylog

$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service

$ wget https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.deb
$ sudo dpkg -i graylog-2.1-repository_latest.deb
$ sudo apt-get update && sudo apt-get install graylog-server

echo -n yourpassword | sha256sum

$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service

$ sudo yum install java-1.8.0-openjdk-headless.x86_64

[mongodb-org-3.2]
name=MongoDB Repository
baseurl=https://repo.mongodb.org/yum/redhat/$releasever/mongodb-org/3.2/x86_64/
gpgcheck=1
enabled=1
gpgkey=https://www.mongodb.org/static/pgp/server-3.2.asc

$ sudo chkconfig --add mongod
$ sudo systemctl daemon-reload
$ sudo systemctl enable mongod.service
$ sudo systemctl start mongod.service

[elasticsearch-2.x]
name=Elasticsearch repository for 2.x packages
baseurl=https://packages.elastic.co/elasticsearch/2.x/centos
gpgcheck=1
gpgkey=https://packages.elastic.co/GPG-KEY-elasticsearch
enabled=1

cluster.name: graylog

$ sudo chkconfig --add elasticsearch
$ sudo systemctl daemon-reload
$ sudo systemctl enable elasticsearch.service
$ sudo systemctl restart elasticsearch.service

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.1-repository_latest.rpm
$ sudo yum install graylog-server

echo -n yourpassword | sha256sum

$ sudo chkconfig --add graylog-server
$ sudo systemctl daemon-reload
$ sudo systemctl enable graylog-server.service
$ sudo systemctl start graylog-server.service

Feedback¶

Please file a bug report in the GitHub repository for the operating system packages if you run into any packaging related issues.

If you found this documentation confusing or have more questions, please open an issue in the Github repository for the documentation.

Requirements¶

You need a recent docker version installed.

This will create three containers with all Graylog services running:

$ docker run --name some-mongo -d mongo:3
$ docker run --name some-elasticsearch -d elasticsearch:2 elasticsearch -Des.cluster.name="graylog"
$ docker run --link some-mongo:mongo --link some-elasticsearch:elasticsearch -p 9000:9000 -e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api" -d graylog2/server

Testing a beta version¶

You can also run a pre-release (alpha, beta, or release candidate) version of Graylog using Docker. The pre-releases are included in the graylog2/server image. Follow this guide and pick an alpha/beta/rc tag like:

$ docker run --link some-mongo:mongo --link some-elasticsearch:elasticsearch -p 9000:9000 -e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api" -d graylog2/server:2.1.0-beta.4-1

We only recommend running pre-release versions if you are an experienced Graylog user and know what you are doing.

Settings¶

Graylog comes with a default configuration that works out of the box but you have to set a password for the admin user. Also the web interface needs to know how to connect from your browser to the Graylog API. Both can be done via environment variables:

-e GRAYLOG_PASSWORD_SECRET=somepasswordpepper
-e GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
-e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api"

In this case you can login to Graylog with the username and password admin. Generate your own password with this command:

$ echo -n yourpassword | shasum -a 256

This all can be put in a docker-compose.yml file, like:

version: '2'
services:
  mongo:
    image: "mongo:3"
  elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
  graylog:
    image: graylog2/server:2.1.2-1
    environment:
      GRAYLOG_PASSWORD_SECRET: somepasswordpepper
      GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - "9000:9000"

After starting the three containers with docker-compose up open your browser with the URL http://127.0.0.1:9000 and login with admin:admin

How to get log data in¶

You can create different kinds of inputs under System -> Inputs, however you can only use ports that have been properly mapped to your docker container, otherwise data will not go through.

E.g. to start a raw TCP input on port 5555, stop your container and recreate it, whilst appending -p 5555:5555 to your run argument. Similarly, the same can be done for UDP by appending -p 5555:5555/udp option. Then you can send raw text to Graylog like echo ‘first log message’ | nc localhost 5555

Persist log data¶

In order to make the log data and configuration of Graylog persistent, you can use external volumes to store all data. In case of a container restart simply re-use the existing data from the former instances. Create the configuration directory and copy the default files:

mkdir /graylog/config
cd /graylog/config
wget https://raw.githubusercontent.com/Graylog2/graylog2-images/2.1/docker/config/graylog.conf
wget https://raw.githubusercontent.com/Graylog2/graylog2-images/2.1/docker/config/log4j2.xml

The docker-compose.yml file looks like this:

version: '2'
services:
  mongo:
    image: "mongo:3"
    volumes:
      - /graylog/data/mongo:/data/db
  elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
    volumes:
      - /graylog/data/elasticsearch:/usr/share/elasticsearch/data
  graylog:
    image: graylog2/server:2.1.2-1
    volumes:
      - /graylog/data/journal:/usr/share/graylog/data/journal
      - /graylog/config:/usr/share/graylog/data/config
    environment:
      GRAYLOG_PASSWORD_SECRET: somepasswordpepper
      GRAYLOG_ROOT_PASSWORD_SHA2: 8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      GRAYLOG_WEB_ENDPOINT_URI: http://127.0.0.1:9000/api/
    depends_on:
      - mongo
      - elasticsearch
    ports:
      - "9000:9000"
      - "12201/udp:12201/udp"
      - "1514/udp:1514/udp"

Start all services with exposed data directories:

$ docker-compose up

Configuration¶

Every configuration option can be set via environment variables. Simply prefix the parameter name with GRAYLOG_ and put it all in upper case.

For example setting up a smtp configuration, the docker-compose.yml would look like this:

environment:
  GRAYLOG_TRANSPORT_EMAIL_ENABLED: "true"
  GRAYLOG_TRANSPORT_EMAIL_HOSTNAME: smtp
  GRAYLOG_TRANSPORT_EMAIL_PORT: 25
  GRAYLOG_TRANSPORT_EMAIL_USE_AUTH: "false"
  GRAYLOG_TRANSPORT_EMAIL_USE_TLS: "false"
  GRAYLOG_TRANSPORT_EMAIL_USE_SSL: "false"

Another option would be to store the configuration file outside of the container and edit it directly.

Plugins¶

In order to add plugins you can build a new image based on the existsing graylog2/server image with the needed plugin included. Simply create a new Dockerfile in an empty directory:

FROM graylog2/server:2.1.2-1
RUN wget -O /usr/share/graylog/plugin/graylog-plugin-beats-1.1.0.jar https://github.com/Graylog2/graylog-plugin-beats/releases/download/1.1.0/graylog-plugin-beats-1.1.0.jar

Build a new image from that:

$ docker build -t graylog-with-beats-plugin .

In this example we created a new image with the Beats plugin installed. From now on reference to that image instead of the graylog2/server e.g. in a docker-compose.yml file:

version: '2'
services:
  mongo:
    image: "mongo:3"
    volumes:
      - /graylog/data/mongo:/data/db
  elasticsearch:
    image: "elasticsearch:2"
    command: "elasticsearch -Des.cluster.name='graylog'"
    volumes:
      - /graylog/data/elasticsearch:/usr/share/elasticsearch/data
  graylog:
    image: graylog-with-beats-plugin
...

Problems¶

• In case you see warnings regarding open file limit, try to set ulimit from the outside of the container:

  $ docker run --ulimit nofile=64000:64000 ...
  

• The devicemapper storage driver can produce problems with Graylogs disk journal on some systems. In this case please pick another driver like aufs or overlay.

Build¶

To build the image from scratch run:

$ docker build --build-arg GRAYLOG_VERSION=${GRAYLOG_VERSION} -t graylog2/server .

Requirements¶

You need a recent vagrant version.

Installation¶

These steps will create a Vagrant virtual machine with all Graylog services running:

$ wget https://raw.githubusercontent.com/Graylog2/graylog2-images/2.1/vagrant/Vagrantfile
$ vagrant up

Usage¶

After starting the virtual machine, your Graylog instance is ready to use. You can reach the web interface by pointing your browser to : http://localhost:8080

The default login is Username: admin, Password: admin.

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

Installation¶

Download the Graylog image from the package site, uncompress it and import it into the Openstack image store:

$ wget https://packages.graylog2.org/releases/graylog-omnibus/qcow2/graylog-2.1.0-1.qcow2.gz
$ gunzip graylog-2.1.0-1.qcow2.gz
$ glance image-create --name='graylog' --is-public=true --container-format=bare --disk-format=qcow2 --file graylog-2.1.0-1.qcow2

You should now see an image called graylog in the Openstack web interface under Images

Usage¶

Launch a new instance of the image, make sure to reserve at least 4GB ram for the instance. After spinning up, login with the username ubuntu and your selected ssh key. Run the reconfigure program in order to setup Graylog and start all services:

$ ssh ubuntu@<vm IP>
$ sudo graylog-ctl reconfigure

Open http://<vm ip> in your browser to access the Graylog web interface. Default username and password is admin.

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

AMIs¶

Select your AMI and AWS Region.

Usage¶

• Click on Launch instance for your AWS region to start Graylog into.
• Choose an instance type with at least 4GB memory.
• Finish the wizard and spin up the VM.
• Login to the instance via SSH as user ubuntu.
• Run sudo graylog-ctl reconfigure.
• Open port 80 and 9000 in the applied security group to access the web interface.
• additionally open more ports for ingesting log data, like 514 for syslog or 12201 for the GELF protocol.

Open http://<private ip> in your browser to access the Graylog web interface. Default username and password is admin.

Networking¶

Your browser needs access to port 80 or 443 for reaching the web interface. The interface itself creates a connection back to the REST API of the Graylog server on port 9000. As long as you are in a private network like Amazon VPC for instance, this works out of the box. If you want to use the public IP address of your VM, this mechanism doesn’t work automatically anymore. You have to tell Graylog how to reach the API from the users browser perspective:

sudo graylog-ctl set-external-ip http://<public ip>:9000/api/
sudo graylog-ctl reconfigure

Also make sure that this port is open, even on the public IP.

HTTPS¶

In order to enable HTTPS for the web interface both ports need to be encrypted. Otherwise the web browser would show an error message. For this reason we created a proxy configuration on the appliance that can be enabled by running:

sudo graylog-ctl enforce-ssl
sudo graylog-ctl reconfigure

This command combines the Graylog web interface and the API on port 443. The API is accessable via the path /api. For this reason you have to set the external IP to an HTTPS address with the appended path /api:

sudo graylog-ctl set-external-ip https://<public ip>:443/api
sudo graylog-ctl reconfigure

Basic configuration¶

We are shipping the graylog-ctl tool with the virtual machine appliances to get you started with a customised setup as quickly as possible. Run these (optional) commands to configure the most basic settings of Graylog in the appliance:

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

The graylog-ctl has much more functionality documented. We strongly recommend to learn more about it to ensure smooth operation of your virtual appliance.

Graylog server on Linux¶

~$ tar xvfz graylog-VERSION.tgz
~$ cd graylog-VERSION

~# cp graylog.conf.example /etc/graylog/server/server.conf

~$ apt-get install openjdk-8-jre

~$ cd bin/
~$ ./graylogctl start

2013-10-01 12:13:22,382 DEBUG: org.elasticsearch.transport.netty - [graylog-server] connected to node [[Unuscione, Angelo][thN_gIBkQDm2ab7k-2Zaaw][inet[/10.37.160.227:9300]]]

-Dlog4j.configurationFile=file:///path/to/log4j2.xml

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
  <Appenders>
      <RollingFile name="RollingFile" fileName="/tmp/logs/graylog.log"
                   filePattern="/tmp/logs/graylog-%d{yyyy-MM-dd}.log.gz">
        <PatternLayout>
          <Pattern>%d %-5p: %c - %m%n</Pattern>
        </PatternLayout>
        <!-- Rotate logs every day or when the size exceeds 10 MB (whichever comes first) -->
        <Policies>
          <TimeBasedTriggeringPolicy modulate="true"/>
          <SizeBasedTriggeringPolicy size="10 MB"/>
        </Policies>
        <!-- Keep a maximum of 10 log files -->
        <DefaultRolloverStrategy max="10"/>
      </RollingFile>

      <Console name="STDOUT" target="SYSTEM_OUT">
          <PatternLayout pattern="%d %-5p: %c - %m%n"/>
      </Console>

      <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
      <Memory name="graylog-internal-logs" bufferSize="500"/>
  </Appenders>
  <Loggers>
      <Logger name="org.graylog2" level="info"/>
      <Logger name="com.github.joschi.jadconfig" level="warn"/>
      <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
      <Logger name="org.elasticsearch.script" level="warn"/>
      <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
      <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
      <Logger name="com.joestelmach.natty.Parser" level="warn"/>
      <Logger name="kafka.log.Log" level="warn"/>
      <Logger name="kafka.log.OffsetIndex" level="warn"/>
      <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
      <Root level="warn">
          <AppenderRef ref="STDOUT"/>
          <AppenderRef ref="RollingFile"/>
          <AppenderRef ref="graylog-internal-logs"/>
      </Root>
  </Loggers>
</Configuration>

~$ sudo -u graylog java -Djava.net.preferIPv4Stack=true -jar graylog.jar

Create a message input and send a first message¶

Log in to the web interface on port 9000 (e.g. http://127.0.0.1:9000) and navigate to System -> Nodes. Select your Graylog node there and click on Manage inputs.

Launch a new Raw/Plaintext UDP input, listening on port 9099 and listening on 127.0.0.1. No need to configure anything else for now. The list of running inputs on that node should show you your new input right away. Let’s send a message in:

echo "Hello Graylog, let's be friends." | nc -w 1 -u 127.0.0.1 9099

This has sent a short string to the raw UDP input you just opened. Now search for friends using the searchbar on the top and you should already see the message you just sent in. Click on it in the table and see it in detail:

You have just sent your first message to Graylog! Why not spawn a syslog input and point some of your servers to it? You could also create some user accounts for your colleagues.

Elasticsearch 2.x¶

The embedded Elasticsearch node being used by Graylog has been upgraded to Elasticsearch 2.x which includes some breaking changes. Graylog 2.x does not work with Elasticsearch 1.x anymore and cannot communicate with existing Elasticsearch 1.x clusters.

Please see Breaking changes in Elasticsearch 2.x for details.

The blog article Key points to be aware of when upgrading from Elasticsearch 1.x to 2.x also contains interesting information about the upgrade path from Elasticsearch 1.x to 2.x.

for i in `curl -s -XGET $elasticsearch:9200/_all/_mapping/index_range | jq -r "keys[]"`; do
    echo -n "Updating index $i: "
    echo -n "curl -XPUT $elasticsearch:9200/$i/_settings -d '{\"index.blocks.read_only\":false, \"index.blocks.write\":false}' : "
    curl -XPUT $elasticsearch:9200/$i/_settings -d '{"index.blocks.read_only":false, "index.blocks.write":false}'
    echo
done

for i in `curl -s -XGET $elasticsearch:9200/_all/_mapping/index_range | jq -r "keys[]"`; do
    echo -n "Updating index $i: "
    curl -XDELETE $elasticsearch:9200/$i/index_range
    echo
done

curl -X DELETE http://localhost:9200/_template/graylog-internal

MongoDB¶

Graylog 2.x requires MongoDB 2.4 or newer. We recommend using MongoDB 3.x and the WiredTiger storage engine.

When upgrading from MongoDB 2.0 or 2.2 to a supported version, make sure to read the Release Notes for the particular version.

Log4j 2 migration¶

Graylog switched its logging backend from Log4j 1.2 to Log4j 2.

Please refer to the Log4j Migration Guide for information on how to update your existing logging configuration.

Dead Letters feature removed¶

The Dead Letters feature, which stored messages that couldn’t be indexed into Elasticsearch for various reasons, has been removed.

This feature has been disabled by default. If you have enabled the feature the configuration file, please check the dead_letters_enabled collection in MongoDB and remove it afterwards.

Removed configuration settings¶

Changed configuration defaults¶

For better consistency, the defaults of some configuration settings have been changed after the project has been renamed from Graylog2 to Graylog.

Changed prefixes for configuration override¶

In the past it was possible to override configuration settings in Graylog using environment variables or Java system properties with a specific prefix.

For better consistency, these prefixes have been changed after the project has been renamed from Graylog2 to Graylog.

REST API Changes¶

The output ID key for the list of outputs in the /streams/* endpoints has been changed from _id to id.

 {
   "id": "564f47c41ec8fe7d920ef561",
   "creator_user_id": "admin",
   "outputs": [
     {
       "id": "56d6f2cce45e0e52d1e4b9cb", // ==> Changed from `_id` to `id`
       "title": "GELF Output",
       "type": "org.graylog2.outputs.GelfOutput",
       "creator_user_id": "admin",
       "created_at": "2016-03-02T14:03:56.686Z",
       "configuration": {
         "hostname": "127.0.0.1",
         "protocol": "TCP",
         "connect_timeout": 1000,
         "reconnect_delay": 500,
         "port": 12202,
         "tcp_no_delay": false,
         "tcp_keep_alive": false,
         "tls_trust_cert_chain": "",
         "tls_verification_enabled": false
       },
       "content_pack": null
     }
   ],
   "matching_type": "AND",
   "description": "All incoming messages",
   "created_at": "2015-11-20T16:18:12.416Z",
   "disabled": false,
   "rules": [],
   "alert_conditions": [],
   "title": "ALL",
   "content_pack": null
 }

Web Interface Config Changes¶

The web interface has been integrated into the Graylog server and was rewritten in React. Therefore configuring it has changed fundamentally since the last version(s). Please consult Web interface for details.

Please take note that the application.context configuration parameter present in Graylog 1.x (and earlier) is not existing anymore. The web interface can currently only be served without a path prefix.

HTTPS Setup¶

Previous versions of Graylog were automatically generating a private key/certificate pair for HTTPS if either the private key or the certificate (or both) for rest_tls_key_file, rest_tls_cert_file, web_tls_key_file, or web_tls_cert_file couldn’t be read. While this feature is very comfortable for inexperienced users, it has lots of serious drawbacks like very weak key sizes (only 1024 bits), being untrusted by all TLS libraries used by web browsers and other client software (because they are self-signed and not included in the system’s CA/trust store), and problems with inter-node communications with other Graylog nodes.

Due to those shortcomings, the feature has been removed completely. Users need to use proper certificates or generate their own self-signed certificates and configure them with the appropriate settings, see Using HTTPS for reference.

Web Interface Listener¶

Graylog 2.0.x has been using separate listeners for the REST API and the web interface by default. The Graylog REST API on http://127.0.0.1:12900, the Graylog web interface on http://127.0.0.1:9000. Beginning with Graylog 2.1.0 it is possible to run both the REST API and the web interface on the same host/port-combination and this is now the default. This means that the REST API is now running on http://127.0.0.1:9000/api/ by default and the web interface is now running on http://127.0.0.1:9000/. Furthermore, all requests going to http://127.0.0.1:9000/api/ requesting a content-type of text/html or application/xhtml+xml are redirected to the web interface, therefore making it even easier to set up Graylog and use it behind proxies, expose it externally etc.

Please take note that you can still run the REST API and the web interface on two separate listeners. If you are running a Graylog 2.0.x configuration specifying web_listen_uri explicitly and you want to keep that, you do not have to change anything.

Please also take note, that when you have configured rest_listen_uri and web_listen_uri to run on the same host/port-combination, the following configuration directives will have no effect:

• web_enable_tls, web_tls_cert_file, web_tls_key_file, web_tls_key_password (These will depend on the TLS configuration of the REST listener).
• web_enable_cors, web_enable_gzip, web_thread_pool_size, web_max_initial_line_length, web_max_header_size (Those will depend on the corresponding settings of the REST listener).

Internal Metrics to MongoDB¶

Previous versions of Graylog included a (long deprecated) metrics reporter for writing internal metrics into MongoDB in a fixed interval of 1 second.

This feature has been removed completely and can be optionally pulled in by using the Graylog Metrics Reporter Plugins.

Configuration file changes¶

sudo graylog-ctl set-listen-address --service rest --address http://0.0.0.0:12900
sudo graylog-ctl reconfigure

Graylog REST API¶

For Plugin Authors¶

Between Graylog 2.0.x and 2.1.0 we also made changes to the Plugin API. These include:

• Removing org.graylog2.plugin.streams.Stream#getAlertCondition, as it was faulty and not easily replaceable with a working version without breaking our separation of models and persistence services.

If you are maintaining a plugin that was originally written for Graylog 1.x or 2.0.x, you need to make sure that your plugin is still compiling and working under Graylog 2.1.x or adapt it if necessary.

Changed Elasticsearch Cluster Status Behavior¶

In previous versions Graylog stopped indexing into the current write index if the Elasticsearch cluster status turned RED. Since Graylog 2.1.0 only checks the status of the current write index when it tries to index messages.

If the current write index is GREEN or YELLOW, Graylog will continue to index messages even though the overall cluster status is RED. This avoids Graylog downtimes when doing Elasticsearch maintenance or when older indices have problems.

Changes in message field values trimming¶

Previous versions of Graylog were trimming message field values inconsistently, depending on the codec used. We have changed that behaviour in Graylog 2.1.0, so all message field values are trimmed by default. This means that leading or trailing whitespace of every field is removed during ingestion.

Important: This change will break your existing stream rules, extractors, and Drool rules if you are expecting leading or trailing white spaces in them. Please adapt them so they do not require those white spaces.

Configuration commands¶

The following commands are changing the configuration of Graylog:

After setting one or more of these options re-run:

$ sudo graylog-ctl reconfigure

You can also edit the full configuration files under /opt/graylog/conf manually. restart the related service afterwards:

$ sudo graylog-ctl restart graylog-server

Or to restart all services:

$ sudo graylog-ctl restart

Multi VM setup¶

At some point it make sense to not run all services on a single VM anymore. For performance reasons you might want to add more Elasticsearch nodes to the cluster or even add a second Graylog server. This can be achieved by changing IP addresses in the Graylog configuration files by hand or use our canned configurations which come with the graylog-ctl command.

The idea is to have one VM which is a central point for other VMs to fetch all needed configuration settings to join the cluster. Typically the first VM you spin up is used for this task. Automatically an instance of etcd is started and filled with the necessary settings for other hosts.

For example, to create a small cluster with a dedicated Graylog server node and another for Elasticsearch, spin up two VMs from the same Graylog image. On the first one start only Graylog and MongoDB:

vm1> sudo graylog-ctl set-admin-password sEcReT
vm1> sudo graylog-ctl reconfigure-as-server

On the second VM start only Elasticsearch. Before doing so set the IP of the first VM to fetch the configuration data from there:

vm2> sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm2> sudo graylog-ctl reconfigure-as-datanode

vm1> sudo graylog-ctl reconfigure-as-server

This results in a perfectly fine dual VM setup. However if you want to scale this setup out by adding an additional Elasticsearch node, you can proceed in the same way:

vm3> sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm3> sudo graylog-ctl reconfigure-as-datanode

vm1> sudo graylog-ctl reconfigure-as-server
vm2> sudo graylog-ctl reconfigure-as-datanode

Verify that all nodes are working as a cluster by going to the Kopf plugin on one of the Elasticsearch nodes open http://vm2:9200/_plugin/kopf/#!/nodes.

Important: In case you want to add a second Graylog server you have to set the same server secret on all machines. The secret is stored in the file /etc/graylog/graylog-secrets and can be applied to other hosts with the set-server-secret sub-command.

The following configuration modes do exist:

A server with only the web interface running is not supported as of Graylog 2.0. The web interface is now included in the server process. But you can create your own service combinations by editing the file /etc/graylog/graylog-services.json by hand and enable or disable single services. Just run graylog-ctl reconfigure afterwards.

Extend disk space¶

All data of the appliance setup is stored in /var/opt/graylog/data. In order to extend the disk space mount a second (virtual) hard drive into this directory.

Install Graylog plugins¶

The Graylog plugin directory is located in /opt/graylog/plugin/. Just drop a JAR file there and restart the server with sudo graylog-ctl restart graylog-server to load the plugin.

Install Elasticsearch plugins¶

Elasticsearch comes with a helper program to install additional plugins you can call it like this sudo JAVA_HOME=/opt/graylog/embedded/jre /opt/graylog/elasticsearch/bin/plugin

Install custom SSL certificates¶

During the first reconfigure run self signed SSL certificates are generated. You can replace this certificate with your own to prevent security warnings in your browser. Just drop the key and combined certificate file here: /opt/graylog/conf/nginx/ca/graylog.crt respectively /opt/graylog/conf/nginx/ca/graylog.key. Afterwards restart nginx with sudo graylog-ctl restart nginx.

Assign a static IP¶

Per default the appliance make use of DHCP to setup the network. If you want to access Graylog under a static IP please follow these instructions:

$ sudo ifdown eth0

Edit the file /etc/network/interfaces like this (just the important lines):

auto eth0
  iface eth0 inet static
  address <static IP address>
  netmask <netmask>
  gateway <default gateway>
  pre-up sleep 2

Activate the new IP and reconfigure Graylog to make use of it:

$ sudo ifup eth0
$ sudo graylog-ctl reconfigure

Wait some time until all services are restarted and running again. Afterwards you should be able to access Graylog with the new IP.

Upgrade Graylog¶

Always perform a full backup or snapshot of the appliance before proceeding. Only upgrade if the release notes say the next version is a drop-in replacement. Choose the Graylog version you want to install from the list of Omnibus packages . graylog_latest.deb always links to the newest version:

$ wget https://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_latest.deb
$ sudo graylog-ctl stop
$ sudo dpkg -G -i graylog_latest.deb
$ sudo graylog-ctl reconfigure
$ sudo reboot

Migrate manually from 1.x to 2.1.x¶

To update a 1.x appliance to 2.1.x the administrator has to purge the Graylog installation, migrate the stored log data and install the new version as Omnibus package. Before upgrading read the upgrade notes. This procedure can potentially delete log data or configuration settings. So it’s absolutely necessary to perform a backup or a snpashot before!

Stop all services but Elasticsearch:

$ sudo -s
$ graylog-ctl stop graylog-web
$ graylog-ctl stop graylog-server
$ graylog-ctl stop mongodb
$ graylog-ctl stop nginx
$ graylog-ctl stop etcd

Check for index range types. The output of this command should be {}, if not read these notes for how to fix this:

$ curl -XGET <appliance_IP>:9200/_all/_mapping/index_range; echo
{}

Delete the Graylog index template:

$ curl -X DELETE <appliance_IP>:9200/_template/graylog-internal

Migrate appliance configuration:

$ cd /etc
$ mv graylog graylog2.1
$ vi graylog2.1/graylog-secrets.json

# Remove the graylog_web section
},  << don't forget the comma!
"graylog_web": {
  "secret_token": "3552c87f3e3..."
}

$ vi graylog2.1/graylog-services.json

# Remove the graylog_web section
}, << don't forget the comma!
"graylog_web": {
  "enabled": true
}

$ vi graylog2.1/graylog-settings.json

# Remove "rotation_size", "rotation_time", "indices"
"enforce_ssl": false,
"rotation_size": 1073741824,
"rotation_time": 0,
"indices": 10,
"journal_size": 1,

Migrate appliance data:

$ cd /var/opt
$ mv graylog graylog2.1
$ mv graylog2.1/data/elasticsearch/graylog2 graylog2.1/data/elasticsearch/graylog

Delete old Graylog version and install new Omnibus package:

$ wget http://packages.graylog2.org/releases/graylog-omnibus/ubuntu/graylog_2.1.0-1_amd64.deb
$ apt-get purge graylog
$ dpkg -i graylog_2.1.0-1_amd64.deb

Move directories back:

$ cd /etc
$ mv graylog2.1 graylog
$ cd /var/opt/
$ mv graylog2.1 graylog

Reconfigure and Reboot:

$ graylog-ctl reconfigure
$ reboot

Graylog should now be updated and old data still available.

Advanced Settings¶

To change certain parameters used by graylog-ctl during a reconfigure run you can override all default parameters found in the attributes file.

If you want to change the username used by Graylog for example, edit the file /etc/graylog/graylog-settings.json like this:

"custom_attributes": {
  "user": {
    "username": "log-user"
  }
}

Afterwards run sudo graylog-ctl reconfigure and sudo graylog-ctl restart. The first command renders all changed configuration files and the later makes sure that all services restart to activate the change.

There are a couple of other use cases of this, e.g. change the default data directories used by Graylog to /data (make sure this is writeable by the graylog user):

"custom_attributes": {
    "elasticsearch": {
      "data_directory": "/data/elasticsearch"
    },
    "mongodb": {
      "data_directory": "/data/mongodb"
    },
    "etcd": {
      "data_directory": "/data/etcd"
    },
    "graylog-server": {
      "journal_directory": "/data/journal"
    }
  }

Or change the default memory settings used by Graylog or Elasticsearch:

"custom_attributes": {
     "graylog-server": {
       "memory": "1700m"
     },
     "elasticsearch": {
       "memory": "2200m"
     }
   }

Again, run reconfigure and restart afterwards to activate the changes.

Overview¶

The Graylog web interface was rewritten in JavaScript for 2.0 to be a client-side single-page browser application. This means its code is running solely in your browser, fetching all data via HTTP(S) from the REST API of your Graylog server.

Single or separate listeners for web interface and REST API?¶

Since Graylog 2.1 you have two options when it comes to exposing its web interface:

• Running both on the same port, using different paths (defaulting to http://localhost:9000/api/ for the REST API and http://localhost:9000/ for the web interface), this is the default since 2.1 and is assumed for most parts of the documentation.
• Running on two different ports (for example http://localhost:12900/ for the REST API and http://localhost:9000/ for the web interface)

Configuration Options¶

If our default settings do not work for you, there is a number of options in the Graylog server configuration file which you can change to influence its behavior:

How does the web interface connect to the Graylog server?¶

The web interface is fetching all information it is showing from the REST API of the Graylog server. Therefore it needs to connect to it using HTTP(S). There are several ways how you can define which way the web interface connects to the Graylog server. The URI used by the web interface is determined in this exact order:

• If the HTTP(S) client going to the web interface port sends a X-Graylog-Server-URL header, which contains a valid URL, then this is overriding everything else.
• If web_endpoint_uri is defined in the Graylog configuration file, this is used if the aforementioned header is not set.
• If both are not defined, rest_transport_uri is used.

Browser Compatibility¶

Writing the web interface as a single-page application is a challenging task. We want to provide the best possible experience to everyone, which often means using modern web technology only available in recent browsers, while keeping a reasonable compatibility with old and less-capable browsers. These browsers are officially supported in Graylog 2.0:

Please take into account that you need to enable JavaScript in order to use Graylog web interface.

Making the web interface work with load balancers/proxies¶

If you want to run a load balancer/reverse proxy in front of Graylog, you need to make sure that:

• The REST API port is accessible for clients
• The address for the Graylog server’s REST API is properly set (as explained in How does the web interface connect to the Graylog server?), so it is resolvable and accessible for any client of the web interface.
• You are either using only HTTP or only HTTPS (no mixed content) for both the web interface endpoint and the REST API endpoint.
• If you use SSL, your certificates must be valid and trusted by your clients.

server
{
  listen      80 default_server;
  listen      [::]:80 default_server ipv6only=on;
  server_name graylog.example.org;

  location /
    {
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Forwarded-Host $host;
        proxy_set_header    X-Forwarded-Server $host;
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    X-Graylog-Server-URL http://graylog.example.org/api;
        proxy_pass          http://127.0.0.1:9000;
    }
}

server
{
    listen      443 ssl spdy;
    server_name graylog.example.org;
    # <- your SSL Settings here!

    location /
    {
        proxy_set_header    X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header    Host $http_host;
        proxy_set_header    X-Graylog-Server-URL https://graylog.example.org/api;
        proxy_pass          http://127.0.0.1:9000;
    }
}

<VirtualHost *:80>
    ServerName graylog.example.org
    ProxyRequests Off
    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        RequestHeader set X-Graylog-Server-URL "http://graylog.example.org/api/"
        ProxyPass http://127.0.0.1:9000/
        ProxyPassReverse http://127.0.0.1:9000/
    </Location>

</VirtualHost>

<VirtualHost *:443>
    ServerName graylog.example.org
    ProxyRequests Off
    SSLEngine on
    # <- your SSL Settings here!

    <Proxy *>
        Order deny,allow
        Allow from all
    </Proxy>

    <Location />
        RequestHeader set X-Graylog-Server-URL "https://graylog.example.org/api/"
        ProxyPass http://127.0.0.1:9000/
        ProxyPassReverse http://127.0.0.1:9000/
    </Location>

</VirtualHost>

frontend http
    bind 0.0.0.0:80

    option forwardfor
    http-request add-header X-Forwarded-Host %[req.hdr(host)]
    http-request add-header X-Forwarded-Server %[req.hdr(host)]
    http-request add-header X-Forwarded-Port %[dst_port]
    acl is_graylog hdr_dom(host) -i -m str graylog.example.org
    use_backend     graylog if is_graylog

backend graylog
    description     The Graylog Web backend.
    http-request set-header X-Graylog-Server-URL http://graylog.example.org/api
    use-server graylog_1
    server graylog_1 127.0.0.1:9000 maxconn 20 check

frontend graylog_http
    bind *:80
    option forwardfor
    http-request add-header X-Forwarded-Host %[req.hdr(host)]
    http-request add-header X-Forwarded-Server %[req.hdr(host)]
    http-request add-header X-Forwarded-Port %[dst_port]
    acl is_graylog hdr_dom(host) -i -m str graylog.example.org
    use_backend     graylog

backend graylog
    description     The Graylog Web backend.
    balance roundrobin
    option httpchk HEAD /api/system/lbstatus
    http-request set-header X-Graylog-Server-URL http://graylog.example.org/api
    server graylog1 192.168.0.10:9000 maxconn 20 check
    server graylog2 192.168.0.11:9000 maxconn 20 check
    server graylog3 192.168.0.12:9000 maxconn 20 check

Load balancer state¶

However, load balancers usually need some way of determining whether a backend service is reachable and healthy or not. For this purpose Graylog exposes a load balancer state that is reachable via its REST API.

There are two ways the load balancer state can change:

• due to a lifecycle change (e.g. the server is starting to accept messages, or shutting down)
• due to manual intervention via the REST API

To query the current load balancer status of a Graylog instance, all you need to do is to issue a HTTP call to its REST API:

GET /api/system/lbstatus

The status knows two different states, ALIVE and DEAD, which is also the text/plain response of the resource. Additionally, the same information is reflected in the HTTP status codes: If the state is ALIVE the return code will be 200 OK, for DEAD it will be 503 Service unavailable. This is done to make it easier to configure a wide range of load balancer types and vendors to be able to react to the status.

The resource is accessible without authentication to make it easier for load balancers to access it.

To programmatically change the load balancer status, an additional endpoint is exposed:

PUT /api/system/lbstatus/override/alive
PUT /api/system/lbstatus/override/dead

Only authenticated and authorized users are able to change the status, in the currently released Graylog version this means only admin users can change it.

Graceful shutdown¶

Often, when running a service behind a load balancer, the goal is to be able to perform zero-downtime upgrades, by taking one of the servers offline, upgrading it, and then bringing it back online. During that time the remaining servers can take the load seamlessly.

By using the load balancer status API described above one can already perform such a task. However, it would still be guesswork when the Graylog server is done processing all the messages it already accepted.

For this purpose Graylog supports a graceful shutdown command, also accessible via the web interface and API. It will set the load balancer status to DEAD, stop all inputs, turn on messages processing (should it have been disabled manually previously), and flush all messages in memory to Elasticsearch. After all buffers and caches are processed, it will shut itself down safely.

Web Interface¶

It is possible to use the Graylog web interface behind a load balancer for high availability purposes.

As of version 2.0 you do not need a sticky session (previously required under 1.x). It is still important, however, that the API URL /api/system/deflector/cycle hits the configured Graylog master node, although this might change in a future release.

Please refer to your vendor’s documentation to learn how to route this URL to a specific host.

Certificate/Key file format¶

When you are configuring TLS, you need to make sure that your certificate/key files are in the right format, which is X.509 for certificates and PKCS#8 for the private keys. Both must to be stored in PEM format.

Creating a self-signed private key/certificate¶

Create a file named openssl-graylog.cnf with the following content (customized to your needs):

[req]
distinguished_name = req_distinguished_name
x509_extensions = v3_req
prompt = no

# Details about the issuer of the certificate
[req_distinguished_name]
C = US
ST = Some-State
L = Some-City
O = My Company
OU = My Division
CN = graylog.example.com

[v3_req]
keyUsage = keyEncipherment, dataEncipherment
extendedKeyUsage = serverAuth
subjectAltName = @alt_names

# IP addresses and DNS names the certificate should include
# Use IP.### for IP addresses and DNS.### for DNS names,
# with "###" being a consecutive number.
[alt_names]
IP.1 = 203.0.113.42
DNS.1 = graylog.example.com

Create PKCS#5 private key and X.509 certificate:

$ openssl version
OpenSSL 0.9.8zh 14 Jan 2016
$ openssl req -x509 -days 365 -nodes -newkey rsa:2048 -config openssl-graylog.cnf -keyout pkcs5-plain.pem -out cert.pem
Generating a 2048 bit RSA private key
............................+++
.+++
writing new private key to 'pkcs5-plain.pem'
-----

Convert PKCS#5 private key into a unencrypted PKCS#8 private key:

$ openssl pkcs8 -in pkcs5-plain.pem -topk8 -nocrypt -out pkcs8-plain.pem

Convert PKCS#5 private key into an encrypted PKCS#8 private key (using the passphrase secret):

$ openssl pkcs8 -in pkcs5-plain.pem -topk8 -out pkcs8-encrypted.pem -passout pass:secret

Converting a PKCS #12 (PFX) file to private key and certificate pair¶

PKCS #12 key stores (PFX files) are commonly used on Microsoft Windows.

In this example, the PKCS #12 (PFX) file is named keystore.pfx:

$ openssl pkcs12 -in keystore.pfx -nokeys -out graylog-certificate.pem
$ openssl pkcs12 -in keystore.pfx -nocerts -out graylog-pkcs5.pem
$ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem

The resulting graylog-certificate.pem and graylog-key.pem can be used in the Graylog configuration file.

Converting an existing Java Keystore to private key/certificate pair¶

This section describes how to export a private key and certificate from an existing Java KeyStore in JKS format.

The starting point is an existing Java KeyStore in JKS format which contains a private key and certificate which should be used in Graylog:

$ keytool -list -v -keystore keystore.jks -alias graylog.example.com
Enter keystore password:
Alias name: graylog.example.com
Creation date: May 10, 2016
Entry type: PrivateKeyEntry
Certificate chain length: 1
Certificate[1]:
Owner: CN=graylog.example.com, OU=Unknown, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Issuer: CN=graylog.example.com, OU=Unknown, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Serial number: 2b33832d
Valid from: Tue May 10 10:02:34 CEST 2016 until: Mon Aug 08 10:02:34 CEST 2016
Certificate fingerprints:
       MD5:  8A:3D:9F:ED:69:93:1B:6C:E3:29:66:EA:82:8D:42:BE
       SHA1: 5B:27:92:25:46:36:BC:F0:82:8F:9A:30:D8:50:D0:ED:32:4D:C6:A0
       SHA256: 11:11:77:F5:F6:6A:20:A8:E6:4A:5D:B5:20:21:4E:B8:FE:B6:38:1D:45:6B:ED:D0:7B:CE:B8:C8:BC:DD:B4:FB
       Signature algorithm name: SHA256withRSA
       Version: 3

Extensions:

#1: ObjectId: 2.5.29.14 Criticality=false
SubjectKeyIdentifier [
KeyIdentifier [
0000: AC 79 64 9F A1 60 14 B9   51 F4 F5 0B B3 B5 02 A5  .yd..`..Q.......
0010: B8 07 DC 7B                                        ....
]
]

The Java KeyStore in JKS format has to be converted to a PKCS#12 keystore, so that OpenSSL can work with it:

$ keytool -importkeystore -srckeystore keystore.jks -destkeystore keystore.p12 -deststoretype PKCS12
Enter destination keystore password:
Re-enter new password:
Enter source keystore password:
Entry for alias graylog.example.com successfully imported.
Import command completed:  1 entries successfully imported, 0 entries failed or cancelled

After the keystore has been successfully converted into PKCS#12 format, OpenSSL can export the X.509 certificate with PEM encoding:

$ openssl pkcs12 -in keystore.p12 -nokeys -out graylog-certificate.pem
Enter Import Password:
MAC verified OK

The private key can only be exported in PKCS#5 format with PEM encoding:

$ openssl pkcs12 -in keystore.p12 -nocerts -out graylog-pkcs5.pem
Enter Import Password:
MAC verified OK
Enter PEM pass phrase:
Verifying - Enter PEM pass phrase:

Graylog currently only supports PKCS#8 private keys with PEM encoding, so OpenSSL has to convert it into the correct format:

$ openssl pkcs8 -in graylog-pkcs5.pem -topk8 -out graylog-key.pem
Enter pass phrase for graylog-pkcs5.pem:
Enter Encryption Password:
Verifying - Enter Encryption Password:

The working directory should now contain the PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) to be used with Graylog:

$ head graylog-key.pem graylog-certificate.pem
==> graylog-key.pem <==
====-BEGIN ENCRYPTED PRIVATE KEY====-
MIIE6TAbBgkqhkiG9w0BBQMwDgQIwMhLa5bw9vgCAggABIIEyN42AeYJJNBEiqhI
mWqJDot4Jokw2vB4abcIJ5Do4+7tjtMrecVRCDSvBZzjkXjnbumBHEoxexe5f0/z
wgq6f/UDyTM3uKYQTG91fcqTyMDUlo3Wc8OqSqsNehOAQzA7hMCehqgNJHO0Zfny
EFvrXHurJWi4eA9vLRup86dbm4Wp3o8pmjOLduXieHfcgVtm5jfd7XfL5cRFS8kS
bSFH4v8xDxLNaJmKkKl9gPCACMRbO9nGk/Z9q9N8zkj+xG9lxlNRMX51SRzg20E0
nyyKTb39tJF35zjroB2HfiFWyrPQ1uF6yGoroGvu0L3eWosjBLjdRs0eBgjJCm5P
ic9zSVqMH6/4CPKJqvB97vP4QhpYcr9jlYJsbn6Zg4OIELpM00VLvp0yU9tqTuRR
TDPYAlNMLZ2RrV52CEsh3zO21WHM7r187x4WHgprDFnjkXf02DrFhgCsGwkEQnb3
vj86q13RHhqoXT4W0zugvcv2/NBLMv0HNQBAfEK3X1YBmtQpEJhwSxeszA1i7CpU

==> graylog-certificate.pem <==
Bag Attributes
    friendlyName: graylog.example.com
    localKeyID: 54 69 6D 65 20 31 34 36 32 38 36 37 38 32 33 30 39 32
subject=/C=DE/ST=Hamburg/L=Hamburg/O=Graylog, Inc./OU=Unknown/CN=graylog.example.com
issuer=/C=DE/ST=Hamburg/L=Hamburg/O=Graylog, Inc./OU=Unknown/CN=graylog.example.com
====-BEGIN CERTIFICATE====-
MIIDkTCCAnmgAwIBAgIEKzODLTANBgkqhkiG9w0BAQsFADB5MQswCQYDVQQGEwJE
RTEQMA4GA1UECBMHSGFtYnVyZzEQMA4GA1UEBxMHSGFtYnVyZzEWMBQGA1UEChMN
R3JheWxvZywgSW5jLjEQMA4GA1UECxMHVW5rbm93bjEcMBoGA1UEAxMTZ3JheWxv
Zy5leGFtcGxlLmNvbTAeFw0xNjA1MTAwODAyMzRaFw0xNjA4MDgwODAyMzRaMHkx

The resulting PKCS#8 private key (graylog-key.pem) and the X.509 certificate (graylog-certificate.pem) can now be used to enable encrypted connections with Graylog by enabling TLS for the Graylog REST API and the web interface in the Graylog configuration file:

# Enable HTTPS support for the REST API. This secures the communication with the REST API
# using TLS to prevent request forgery and eavesdropping.
rest_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the REST API.
rest_tls_cert_file = /path/to/graylog-certificate.pem

# The PKCS#8 private key file in PEM format to use for securing the REST API.
rest_tls_key_file = /path/to/graylog-key.pem

# The password to unlock the private key used for securing the REST API.
rest_tls_key_password = secret

# Enable HTTPS support for the web interface. This secures the communication the web interface
# using TLS to prevent request forgery and eavesdropping.
web_enable_tls = true

# The X.509 certificate chain file in PEM format to use for securing the web interface.
web_tls_cert_file = /path/to/graylog-certificate.pem

# The PKCS#8 private key file in PEM format to use for securing the web interface.
web_tls_key_file = /path/to/graylog-key.pem

# The password to unlock the private key used for securing the web interface.
web_tls_key_password = secret

Sample files¶

This section show the difference between following private key formats with samples.

PKCS#5 plain private key:

====-BEGIN RSA PRIVATE KEY====-
MIIBOwIBAAJBANxtmQ1Kccdp7HBNt8zgTai48Vv617bj4SnhkcMN99sCQ2Naj/sp
[...]
NiCYNLiCawBbpZnYw/ztPVACK4EwOpUy+u19cMB0JA==
====-END RSA PRIVATE KEY====-

PKCS#8 plain private key:

====-BEGIN PRIVATE KEY====-
MIIBVAIBADANBgkqhkiG9w0BAQEFAASCAT4wggE6AgEAAkEA6GZN0rQFKRIVaPOz
[...]
LaLGdd9G63kLg85eldSy55uIAXsvqQIgfSYaliVtSbAgyx1Yfs3hJ+CTpNKzTNv/
Fx80EltYV6k=
====-END PRIVATE KEY====-

PKCS#5 encrypted private key:

====-BEGIN RSA PRIVATE KEY====-
Proc-Type: 4,ENCRYPTED
DEK-Info: DES-EDE3-CBC,E83B4019057F55E9

iIPs59nQn4RSd7ppch9/vNE7PfRSHLoQFmaAjaF0DxjV9oucznUjJq2gphAB2E2H
[...]
y5IT1MZPgN3LNkVSsLPWKo08uFZQdfu0JTKcn7NPyRc=
====-END RSA PRIVATE KEY====-

PKCS#8 encrypted private key:

====-BEGIN ENCRYPTED PRIVATE KEY====-
MIIBpjBABgkqhkiG9w0BBQ0wMzAbBgkqhkiG9w0BBQwwDgQIU9Y9p2EfWucCAggA
[...]
IjsZNp6zmlqf/RXnETsJjGd0TXRWaEdu+XOOyVyPskX2177X9DUJoD31
====-END ENCRYPTED PRIVATE KEY====-

Adding a self-signed certificate to the JVM trust store¶

Graylog nodes inside a cluster need to communicate with each other using the Graylog REST API. When using HTTPS for the Graylog REST API, the X.509 certificate must be trusted by the JVM trust store (similar to the trusted CA bundle in an operating system), otherwise communication will fail.

The default trust store of an installed Java runtime environment can be found at $JAVA_HOME/jre/lib/security/cacerts. In order not to “pollute” the official trust store, we make a copy of it which we will use with Graylog instead:

$ cp -a "${JAVA_HOME}/jre/lib/security/cacerts" /path/to/cacerts.jks

After the original key store file has been copied, we can add the self-signed certificate (cert.pem, see Creating a self-signed private key/certificate) to the key store (the default password is changeit):

$ keytool -importcert -keystore /path/to/cacerts.jks -storepass changeit -alias graylog-self-signed -file cert.pem
Owner: CN=graylog.example.com, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Issuer: CN=graylog.example.com, O="Graylog, Inc.", L=Hamburg, ST=Hamburg, C=DE
Serial number: 8c80134cee556734
Valid from: Tue Jun 14 16:38:17 CEST 2016 until: Wed Jun 14 16:38:17 CEST 2017
Certificate fingerprints:
       MD5:  69:D1:B3:01:46:0D:E9:45:FB:C6:6C:69:EA:38:ED:3E
       SHA1: F0:64:D0:1B:3B:6B:C8:01:D5:4D:33:36:87:F0:FB:10:E1:36:21:9E
       SHA256: F7:F2:73:3D:86:DC:10:22:1D:14:B8:5D:66:B4:EB:48:FD:3D:74:89:EC:C4:DF:D0:D2:EC:F8:5D:78:49:E7:2F
       Signature algorithm name: SHA1withRSA
       Version: 3

Extensions:

[Other details about the certificate...]

Trust this certificate? [no]:  yes
Certificate was added to keystore

To verify that the self-signed certificate has indeed been added, it can be listed with the following command:

$ keytool -keystore /path/to/cacerts.jks -storepass changeit -list | grep graylog-self-signed -A1
graylog-self-signed, Jun 14, 2016, trustedCertEntry,
Certificate fingerprint (SHA1): F0:64:D0:1B:3B:6B:C8:01:D5:4D:33:36:87:F0:FB:10:E1:36:21:9E

The printed certificate fingerprint (SHA1) should match the one printed when importing the self-signed certificate.

In order for the JVM to pick up the new trust store, it has to be started with the JVM parameter -Djavax.net.ssl.trustStore=/path/to/cacerts.jks. If you’ve been using another password to encrypt the JVM trust store than the default changeit, you additionally have to set the JVM parameter -Djavax.net.ssl.trustStorePassword=secret.

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the javax.net.ssl.trustStore and (optionally) javax.net.ssl.trustStorePassword system properties.

Disabling specific TLS ciphers and algorithms¶

Since Java 7u76 it is possible to disable specific TLS algorithms and ciphers for secure connections.

In order to disable specific TLS algorithms and ciphers, you need to provide a properties file with a list of disabled algorithms and ciphers. Take a look at the example security.properties in the Graylog source repository.

For example, if you want to disable all algorithms except for TLS 1.2, the properties file has to contain the following line:

jdk.tls.disabledAlgorithms=SSLv2Hello, SSLv3, TLSv1, TLSv1.1

If additionally you want to disable DSA/RSA key sizes lower than 2048 bits and EC key sizes lower than 160 bits, the properties file has to contain the following line:

jdk.tls.disabledAlgorithms=SSLv2Hello, SSLv3, TLSv1, TLSv1.1, EC keySize < 160, RSA keySize < 2048, DSA keySize < 2048

To load the properties file into a JVM, you have to pass it to Java using the java.security.properties system property:

java -Djava.security.properties=/path/to/security.properties -jar /path/to/graylog.jar server

Most start and init scripts for Graylog provide a JAVA_OPTS variable which can be used to pass the java.security.properties system property.

Prerequisites¶

Every server which is part of this setup should have the software requirements installed to run the targeted software. All software requirements can be found in the installation manual.

We highly recommend that the system time on all systems is kept in sync via NTP or a similar mechanism. Needless to say that DNS resolution must be working, too. Because everything is a freaking DNS problem.

In order to simplify the installation process, the servers should have a working Internet connection.

MongoDB replica set¶

We recommend to deploy a MongoDB replica set.

MongoDB doesn’t have to run on dedicated servers for the workload generated by Graylog, but you should follow the recommendations given in the MongoDB documentation about architecture. Most important is that you have an odd number of MongoDB servers in the replica set.

In most setups, each Graylog server will also host an instance of MongoDB which is part of the same replica set and shares the data with all other nodes in the cluster.

The correct order of working steps should be as follows:

1. Create the replica set (rs01)
2. Create the database (graylog)
3. Create a user account for accessing the database, which has the roles readWrite and dbAdmin.

If your MongoDB needs to be reachable over network you should set the IP with bind_ip in the configuration.

Elasticsearch cluster¶

The Elasticsearch setup documentation should help you to install Elasticsearch with a robust base configuration.

It is important to name the Elasticsearch cluster not simply named elasticsearch to avoid accidental conflicts with Elasticsearch nodes using the default configuration. Just choose anything else (we recommend graylog), because this is the default name and any Elasticsearch instance that is started in the same network will try to connect to this cluster.

The Elasticsearch servers need one IP that can be reached over network set in network.host and some participants of the cluster in discovery.zen.ping.unicast.hosts. That is enough to have a minial cluster setup.

Graylog Multi-node¶

After the installation of Graylog, you should take care that only one Graylog node is configured to be master with the configuration setting is_master = true.

The URI configured in rest_listen_uri (or rest_transport_uri) must be accessable for all Graylog nodes of the cluster.

mongodb_uri = mongodb://USERNAME:PASSWORD@mongodb-node01:27017,mongodb-node02:27017,mongodb-node03:27017/graylog?replicaSet=rs01

Scaling¶

Each component in this multi-node setup can be scaled on the individual needs.

Depending on the amount of messages ingested and how long messages should be available for direct search, the Elasticsearch cluster will need most of the resources on your setup.

Keep an eye on your Elasticsearch cluster with plugins like Elastic HQ or Kopf. Those will help you to understand the Elasticsearch cluster health and behavior.

Graylog Metrics should be monitored with the Graylog Metrics Reporter plugins which are able to send the internal Graylog metrics to your favorite metrics collector (e. g. Graphite or Prometheus).

Up until today, we have almost never faced the issue that the MongoDB replica set needed special attention. But of course you should still monitor it and store its metrics - just to be sure.

Troubleshooting¶

• After every configuration change or service restart, watch the logfile of the applications you have worked on. Sometimes other log files can also give you hints about what went wrong. For example if you’re configuring Graylog and try to find out why the connection to the MongoDB isn’t working, the MongoDB logs can help to identify the problem.
• If HTTPS has been enabled for the Graylog REST API, it need to be setup for the Graylog web interface, too.

Elasticsearch versions¶

Graylog hosts an embedded Elasticsearch node which is joining the Elasticsearch cluster as a client node.

The following table provides an overview over the Elasticsearch version in Graylog:

Configuration¶

# List of Elasticsearch master nodes to connect to
elasticsearch_discovery_zen_ping_unicast_hosts = es-node-1.example.org:9300,es-node-2.example.org:9300

# Public IP address or host name of the Graylog node, accessible for the other Elasticsearch nodes
elasticsearch_network_host = 198.51.100.23

discovery.zen.ping.multicast.enabled: false
discovery.zen.ping.unicast.hosts: ["es-node-1.example.org:9300" , "es-node-2.example.org:9300"]

script.inline: false
script.indexed: false
script.file: false

now throttling indexing

indices.store.throttle.max_bytes_per_sec: 150mb

Avoiding split-brain and shard shuffling¶

# At least NODES/2+1 on clusters with NODES > 2, where NODES is the number of master nodes in the cluster
discovery.zen.minimum_master_nodes: 2

node.data: false
node.master: true

# Recover only after the given number of nodes have joined the cluster. Can be seen as "minimum number of nodes to attempt recovery at all".
gateway.recover_after_nodes: 8
# Time to wait for additional nodes after recover_after_nodes is met.
gateway.recover_after_time: 5m
# Inform ElasticSearch how many nodes form a full cluster. If this number is met, start up immediately.
gateway.expected_nodes: 10

Custom index mappings¶

Sometimes it’s useful to not rely on Elasticsearch’s dynamic mapping but to define a stricter schema for messages.

Graylog itself is using a default mapping which includes settings for the timestamp, message, full_message, and source fields of indexed messages:

$ curl -X GET 'http://localhost:9200/_template/graylog-internal?pretty'
{
  "graylog-internal" : {
    "order" : -2147483648,
    "template" : "graylog_*",
    "settings" : { },
    "mappings" : {
      "message" : {
        "_ttl" : {
          "enabled" : true
        },
        "_source" : {
          "enabled" : true
        },
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "full_message" : {
            "analyzer" : "standard",
            "index" : "analyzed",
            "type" : "string"
          },
          "streams" : {
            "index" : "not_analyzed",
            "type" : "string"
          },
          "source" : {
            "analyzer" : "analyzer_keyword",
            "index" : "analyzed",
            "type" : "string"
          },
          "message" : {
            "analyzer" : "standard",
            "index" : "analyzed",
            "type" : "string"
          },
          "timestamp" : {
            "format" : "yyyy-MM-dd HH:mm:ss.SSS",
            "type" : "date"
          }
        }
      }
    },
    "aliases" : { }
  }
}

In order to extend the default mapping of Elasticsearch and Graylog, you can create one or more custom index mappings and add them as index templates to Elasticsearch.

Let’s say we have a schema for our data like the following:

This would translate to the following additional index mapping in Elasticsearch:

"mappings" : {
  "message" : {
    "properties" : {
      "http_method" : {
        "type" : "string",
        "index" : "not_analyzed"
      },
      "http_response_code" : {
        "type" : "long"
      },
      "ingest_time" : {
        "type" : "date",
        "format": "strict_date_time"
      },
      "took_ms" : {
        "type" : "long"
      }
    }
  }
}

The format of the ingest_time field is described in the Elasticsearch documentation about the format mapping parameter. Also make sure to check the Elasticsearch documentation about Field datatypes.

In order to apply the additional index mapping when Graylog creates a new index in Elasticsearch, it has to be added to an index template. The Graylog default template (graylog-internal) has the lowest priority and will be merged with the custom index template by Elasticsearch.

{
  "template": "graylog_*",
  "mappings" : {
    "message" : {
      "properties" : {
        "http_method" : {
          "type" : "string",
          "index" : "not_analyzed"
        },
        "http_response_code" : {
          "type" : "long"
        },
        "ingest_time" : {
          "type" : "date",
          "format": "strict_date_time"
        },
        "took_ms" : {
          "type" : "long"
        }
      }
    }
  }
}

$ curl -X PUT -d @'graylog-custom-mapping.json' 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
{
  "acknowledged" : true
}

$ curl -X GET 'http://localhost:9200/graylog_deflector/_mapping?pretty'
{
  "graylog_2" : {
    "mappings" : {
      "message" : {
        "_ttl" : {
          "enabled" : true
        },
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "full_message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "http_method" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "http_response_code" : {
            "type" : "long"
          },
          "ingest_time" : {
            "type" : "date",
            "format" : "strict_date_time"
          },
          "message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "source" : {
            "type" : "string",
            "analyzer" : "analyzer_keyword"
          },
          "streams" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          },
          "took_ms" : {
            "type" : "long"
          }
        }
      }
    }
  }
}

$ curl -X DELETE 'http://localhost:9200/_template/graylog-custom-mapping?pretty'
{
  "acknowledged" : true
}

$ curl -X GET 'http://localhost:9200/graylog_deflector/_mapping?pretty'
{
  "graylog_3" : {
    "mappings" : {
      "message" : {
        "_ttl" : {
          "enabled" : true
        },
        "dynamic_templates" : [ {
          "internal_fields" : {
            "mapping" : {
              "index" : "not_analyzed",
              "type" : "string"
            },
            "match" : "gl2_*"
          }
        }, {
          "store_generic" : {
            "mapping" : {
              "index" : "not_analyzed"
            },
            "match" : "*"
          }
        } ],
        "properties" : {
          "full_message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "message" : {
            "type" : "string",
            "analyzer" : "standard"
          },
          "source" : {
            "type" : "string",
            "analyzer" : "analyzer_keyword"
          },
          "streams" : {
            "type" : "string",
            "index" : "not_analyzed"
          },
          "timestamp" : {
            "type" : "date",
            "format" : "yyyy-MM-dd HH:mm:ss.SSS"
          }
        }
      }
    }
  }
}

Cluster Status explained¶

Elasticsearch provides a classification for the cluster health.

The cluster status applies to different levels:

• Shard level - see status descriptions below
• Index level - inherits the status of the worst shard status
• Cluster level - inherits the status of the worst index status

That means that the Elasticsearch cluster status can turn red if a single index or shard has problems even though the rest of the indices/shards are okay.

Explanation of the different status levels:

DEB package¶

This paragraph covers Graylog installations on Ubuntu Linux, Debian Linux, and Debian derivates installed with the DEB package.

RPM package¶

This paragraph covers Graylog installations on Fedora Linux, Red Hat Enterprise Linux, CentOS Linux, and other Red Hat Linux derivates installed with the RPM package.

Omnibus package¶

This paragraph covers Graylog installations via OVA, on AWS (via AMI), and on Openstack using the Graylog Omnibus package.

Using the API Browser¶

After providing the credentials (username and password), you can browse all available HTTP resources of the Graylog REST API.

Interacting with the Graylog REST API¶

While having a graphical UI for the Graylog REST API is perfect for interactive usage and exploratory learning, the real power unfolds when using the Graylog REST API for automation or integrating Graylog into another system, such as monitoring or ticket systems.

Naturally, the same operations the API browser offers can be used on the command line or in scripts. A very common HTTP client being used for this kind of interaction is curl.

The following command displays Graylog cluster information as JSON, exactly the same information the web interface is displaying on the System / Nodes page:

curl -u GM:superpower -H 'Accept: application/json' -X GET 'http://192.168.178.26:9000/api/cluster?pretty=true'

The Graylog REST API will respond with the following information:

{
  "71ab6aaa-cb39-46be-9dac-4ba99fed3d66" : {
    "facility" : "graylog-server",
    "codename" : "Smuttynose",
    "node_id" : "71ab6aaa-cb39-46be-9dac-4ba99fed3d66",
    "cluster_id" : "3adaf799-1551-4239-84e5-6ed939b56f62",
    "version" : "2.1.1+01d50e5",
    "started_at" : "2016-09-23T10:39:00.179Z",
    "hostname" : "gm-01-c.fritz.box",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Europe/Berlin",
    "operating_system" : "Linux 3.10.0-327.28.3.el7.x86_64",
    "is_processing" : true
  },
  "ed0ad32d-8776-4d25-be2f-a8956ecebdcf" : {
    "facility" : "graylog-server",
    "codename" : "Smuttynose",
    "node_id" : "ed0ad32d-8776-4d25-be2f-a8956ecebdcf",
    "cluster_id" : "3adaf799-1551-4239-84e5-6ed939b56f62",
    "version" : "2.1.1+01d50e5",
    "started_at" : "2016-09-23T10:40:07.325Z",
    "hostname" : "gm-01-d.fritz.box",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Europe/Berlin",
    "operating_system" : "Linux 3.16.0-4-amd64",
    "is_processing" : true
  },
  "58c57924-808a-4fa7-be09-63ca551628cd" : {
    "facility" : "graylog-server",
    "codename" : "Smuttynose",
    "node_id" : "58c57924-808a-4fa7-be09-63ca551628cd",
    "cluster_id" : "3adaf799-1551-4239-84e5-6ed939b56f62",
    "version" : "2.1.1+01d50e5",
    "started_at" : "2016-09-30T13:31:39.051Z",
    "hostname" : "gm-01-u.fritz.box",
    "lifecycle" : "running",
    "lb_status" : "alive",
    "timezone" : "Europe/Berlin",
    "operating_system" : "Linux 4.4.0-36-generic",
    "is_processing" : true
  }

curl -u GM:superpower -H 'Accept: application/json' -X POST 'http://192.168.178.26:9000/api/users/GM/tokens/icinga?pretty=true'

{
   "name" : "icinga",
   "token" : "htgi84ut7jpivsrcldd6l4lmcigvfauldm99ofcb4hsfcvdgsru",
   "last_access" : "1970-01-01T00:00:00.000Z"
}

curl -u htgi84ut7jpivsrcldd6l4lmcigvfauldm99ofcb4hsfcvdgsru:token -H 'Accept: application/json' -X GET 'http://192.168.178.26:9000/api/cluster?pretty=true'

curl -u GM:superpower -H 'Accept: application/json' -X GET 'http://192.168.178.26:9000/api/users/GM/tokens/?pretty=true'

curl -u GM:superpower -H 'Accept: application/json' -X DELETE' http://192.168.178.26:9000/api/users/GM/tokens/ap84p4jehbf2jddva8rdmjr3k7m3kdnuqbai5s0h5a48e7069po?pretty=true'

curl -i -X POST -H 'Content-Type: application/json' -H 'Accept: application/json' 'http://192.168.178.26:9000/api/system/sessions' -d '{"username":"GM", "password":"superpower", "host":""}'

{
    "valid_until" : "2016-10-24T16:08:57.854+0000",
    "session_id" : "cf1df45c-53ea-446c-8ed7-e1df64861de7"
}

curl -u cf1df45c-53ea-446c-8ed7-e1df64861de7:session -H 'Accept: application/json' -X GET 'http://192.168.178.26:9000/api/cluster?pretty=true'