Welcome to the Graylog documentation¶

NOTE: There are multiple options for reading this documentation. See link to the lower left.

Contents:

Architectural considerations¶

There are a few rules of thumb when scaling resources for Graylog:

graylog-server nodes should have a focus on CPU power.
Elasticsearch nodes should have as much RAM as possible and the fastest disks you can get. Everything depends on I/O speed here.
MongoDB is only being used to store configuration and the dead letter messages, and can be sized fairly small.
graylog-web-interface nodes are mostly waiting for HTTP answers of the rest of the system and can also be rather small.
graylog-radio nodes act as workers. They don’t know each other and you can shut them down at any point in time without changing the cluster state at all.

Also keep in mind that messages are only stored in Elasticsearch. If you have data loss on Elasticsearch, the messages are gone - except if you have created backups of the indices.

MongoDB is only storing meta information and will be abstracted with a general database layer in future versions. This will allow you to use other databases like MySQL instead.

Minimum setup¶

This is a minimum Graylog setup that can be used for smaller, non-critical, or test setups. None of the components is redundant but it is easy and quick to setup.

Bigger production setup¶

This is a setup for bigger production environments. It has several graylog-server nodes behind a load balancer that share the processing load. The load balancer can ping the graylog-server nodes via REST/HTTP to check if they are alive and take dead nodes out of the cluster.

Highly available setup with Graylog Radio¶

Beginning with Graylog 1.0 on, we no longer recommend running Graylog Radio because we are now using a high-performant message journal (from the Apache Kafka project) in every graylog-server instance which is spooling all incoming messages to disk immediately and is able to buffer load spikes just at least as good as Graylog Radio was, but with less dependencies and maintenance overhead.

If you are running a setup with Graylog Radio we recommend to shut down the Graylog Radio architecture including AMQP or Kafka brokers completely and directly send messages to the graylog-server nodes. If you have been using Graylog Radio for load balancing, you should now put a classic load balancer in front of your graylog-server nodes.

This approach has been proven to work great in large high-throughput setups of several of our large scale customers and immensely reduced complexity of their setups.

The Kafka and AMQP inputs are still supported and can be used to build a custom setup using message brokers, if you want to keep using that. A reason for this might be that Graylog is not the only subscriber to the messages on the bus. However we would recommend to use Graylog forwarders to either write to a message bus after processing or write to other systems directly.

Getting Started¶

The fastest way to evaluate Graylog is to download the virtual appliance file and run it in VMWare or VirtualBox.

Install & Configure¶

Download Graylog¶

Go here and download the virtual image.

Import the VM¶

These screenshots are for VMWare (VirtualBox is nearly the same). Select File -> Import..., choose the graylog.ova file you downloaded, and follow the prompts.

Start VM¶

This is what you’ll see when you run the virtual machine. This is where all the Graylog server processes (more on that later) and the database will run. We can split them apart later for performance, but there’s no need to do that right now for a quick overview. Don’t close this window just yet, we’re going to need the IP for the next step.

Connect to the Web Console¶

Go to a web browser on your host machine and type the IP address you saw in the previous screen. You should get a Graylog Web Console login page. Enter admin/admin to login.

Logging in will get you to blank screen. Not very fun? No problem, let’s get some stuff in there!

Configure Graylog Input¶

Now you are now sending data to Graylog, so you need to configure an input. This will tell Graylog to accept the log messages.

Go back to the Graylog console open in your browser and click System -> Inputs. Then select Syslog UDP and click Launch new input. Fill out the circles with the values in the screen shown below.

Get Messages In¶

Log into the VM¶

We’re going to use rsyslog because we already have it from the Graylog server image. So, go to the image and login with ubuntu/ubuntu.

Modify rsyslog.conf¶

Go to the /etc directory, and use vi, vim (vim Cheat Sheet), or the editor of your choice to modify the /etc/rsyslog.conf file. There are excellent resources on the web for rsyslog configuration.

At the bottom of the file, add the following so messages will forward:

*.* @127.0.0.1:5140;RSYSLOG_SyslogProtocol23Format

In case you wanted to know, @ means UDP, 127.0.0.1 is localhost, and 5140 is the port.

Restart rsyslog¶

Type:

$sudo service rsyslog status
$sudo service rsyslog restart

If you have modified the config file and it is somehow invalid, the service command will not bring rsyslog back up - so don’t worry, you can always delete the line!

Check If You Have Messages¶

After that, you should see the Syslog UDP input appear on the screen.

Click Show received messages button on this screen, and you should have messages at the bottom. It may take a few minutes before you have messages coming in.

BOOM! Now that you have messages coming in, this is where the fun starts.

Skip the next section if you are all good.

If You Don’t Have Messages¶

Check to see that you made the proper entries in the rsyslog configuration file.
Check the syslog UDP configuration and make sure that is right - remember we changed the default port to 5140.

3. Check to see if rsyslog messages are being forwarded to the port. You can use the tcpdump command to do this:

$sudo tcpdump -i lo host 127.0.0.1 and udp port 5140

4. Check to see if the server is listening on the host:

$sudo netstat -peanut | grep ":5140"

Create Your Dashboard¶

You should be at a screen like the one below. If you dozed off or went to cook some meatballs, go to System -> Inputs, select the Syslog UDP input you created, and hit Show messages.

Now it’s go-time.

You’ve got data coming in, let’s add information to a dashboard to better visualize the data we want to see.

Add a Dashboard¶

We’ll start by adding the message count data to a dashboard. Click Add count to dashboard, and it will say No Dashboards, create one? Yes! Click that.

Give your new dashboard a title and description.

Smile¶

And you’ll end up with the widget you created!

Extra Credit - One more¶

Let’s add a widget for root activity, because that sounds like it may actually be useful. We need to start with a search query for root. Click Search. Type root in the search and select your timeframe. Once the search results come in, click Add count to the dashboard.

Give your chart a title and hit Create.

The new widget is now on the screen. Goob job - you’ve got this!

Go play around. If you want to know how to create more exciting charts and graphs, check out the section below.

Extra Credit - Graphs¶

Let’s start by searching for all messages within the last 1 hour. To do this, click Search, select Search in the last 1 hour, and leave the search bar blank. Once the search results populate, expand the messages field in the Search results sidebar and select Quick Values. Click Add to dashboard to add this entire pie chart and data table to your dashboard.

I like to track password changes, privilege assignments, root activity, system events, user logins, etc. Go knock yourself out and show your co-workers.

Once you have a few widgets in your dashboard, go into unlock / edit mode to quickly edit any widget, rearrange them on your dashboard, or delete. Make sure to click Lock to save!

Get Alerted¶

I know, we’re all lazy, and busy. Nobody wants to just stare at a dashboard all day like it’s the World Cup. That’s for management.

Let’s configure some proactive alerts to let us know when something needs our attention.

Create a Stream¶

In order to set up an alert, we need to first create a stream. Streams process incoming messages in real time based on conditions that you set. Click Streams.

Let’s create a stream for all incoming security/authentication error messages. Click Create Stream.

Type in a Title and Description.

Create a Stream Rule¶

Next, we are going to configure the stream to process our Syslog UDP input messages for any security alerts.

Hit the Edit rules button.

Pick the Syslog UDP Input, and click Add stream rule.

Then, type in the values shown below and hit save.

Then click I’m done!

We have just configured this stream to process in real time all the messages that come in from the security/authorization facility.

Now let’s create the alert.

Create the Alert¶

You can now either output your new stream to a 3rd party application or database, or trigger an alert to ping you in real time when a message that matches our stream rule comes in. Let’s create an alert that will email us when there are more than 2 messages in the last 2 minutes . Click Manage Alerts.

In the Add new alert condition section, let’s configure and add a new alert. Select message count condition, and configure the rest based on my screenshot (input 2’s in every field). Then click Add alert condition.

Send a Test Email¶

In the Callbacks section, select email alert callback, and input your email address in the Receivers section. After you’ve added a callback type and receiver, hit the blue ‘Send test alert’ button.

Going Further¶

If you want to configure an SMTP server, you can refer to the this documentation.

If you want to make this stream active, just go back to Streams and where you see the stream name, click the green Start stream button.

You are done - go grab a Creamsicle, take a deep breath, and chillax. Tomorrow you can configure all your own logs and alerts. To help, go and get some deep knowledge in the official documentation.

Installing Graylog¶

Modern server architectures and configurations are managed in many different ways. Some people still put new software somewhere in opt manually for each server while others have already jumped on the configuration management train and fully automated reproducible setups.

Graylog can be installed in many different ways so you can pick whatever works best for you. We recommend to start with the virtual machine appliances for the fastest way to get started and then pick one of the other, more flexible installation methods to build an easier to scale setup. (Note: The virtual machine appliances are suitable for production usage because they are also prepared to scale out to some level when required.)

System requirements

The Graylog server application has the following prerequisites:

Some modern Linux distribution (Debian Linux, Ubuntu Linux, or CentOS recommended)
Elasticsearch 1.7.1 or later (Elasticsearch 2.x is currently not supported)
MongoDB 2.0 or later (latest stable version is recommended)
Oracle Java SE 7 or later (Oracle Java SE 8 is supported, OpenJDK 7 and OpenJDK 8 also work; latest stable update is recommended)

The Graylog web interface has the following prerequisites:

Some modern Linux distribution (Debian Linux, Ubuntu Linux, or CentOS recommended)
Oracle Java SE 7 or later (Oracle Java SE 8 is supported, OpenJDK 7 and OpenJDK 8 also work; latest point release is recommended)

This chapter is explaining the many ways to install Graylog and aims to help choosing the one that fits your needs.

Choose an installation method:

Virtual Machine Appliances¶

Download¶

Download the OVA image from here and save it to your disk locally.

Run the image¶

You can run the OVA in many systems like VMware or Virtualbox. In this example we will guide you through running the OVA in the free Virtualbox on OSX.

In Virtualbox select File -> Import appliance:

Hit Continue and keep the suggested settings on the next page as they are. Make sure that you have enough RAM and CPUs on your local machine. You can lower the resources the virtual machine will get assigned but we recommend to not lower it to ensure a good Graylog experience. In fact you might have to raise it if you plan to scale out later and send more messages into Graylog.

Press Import to finish loading the OVA into Virtualbox:

You can now start the VM and should see a login shell like this when the boot completed:

Logging in¶

You can log into the shell of the operating system of the appliance with the user ubuntu and the password ubuntu. You should of course change those credentials if you plan to go into production with the appliance.

The web interface is reachable on port 80 at the IP address of your virtual machine. The login prompt of the shell is showing you this IP address, too. (See screenshot above)

The standard user for the web interface is admin with the password admin.

Basic configuration¶

We are shipping the graylog-ctl tool with the virtual machine appliances to get you started with a customised setup as quickly as possible. Run these (optional) commands to configure the most basic settings of Graylog in the appliance:

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

The graylog-ctl has much more functionality and is documented here. We strongly recommend to learn more about it to ensure smooth operation of your virtual appliance.

The graylog-ctl script¶

Some packages of Graylog (for example the virtual machine appliances) ship with a pre-installed graylog-ctl script to allow you easy configuration of certain settings.

Important: The manual setup, operating system packages, configuration management scripts etc are not shipping with this.

Configuration commands¶

The following commands are changing the configuration of Graylog:

Command	Description
sudo graylog-ctl set-admin-password <password>	Set a new admin password
sudo graylog-ctl set-admin-username <username>	Set a different username for the admin user
sudo graylog-ctl set-email-config <smtp server> [–port=<smtp port> –user=<username> –password=<password> –no-tls –no-ssl]	Configure SMTP settings to send alert mails
sudo graylog-ctl set-timezone <zone acronym>	Set Graylog’s timezone. Make sure system time is also set correctly with `sudo dpkg-reconfigure tzdata`
sudo graylog-ctl set-retention –size=<Gb> OR –time=<hours> –indices=<number> [–journal=<Gb>]	Configure message retention
sudo graylog-ctl enforce-ssl	Enforce HTTPs for the web interface

After setting one or more of these options re-run:

sudo graylog-ctl reconfigure

You can also edit the full configuration files under /opt/graylog/conf manually. restart the related service afterwards:

sudo graylog-ctl restart graylog-server

Or to restart all services:

sudo graylog-ctl restart

Multi VM setup¶

At some point it makes sense to not run all services in one VM anymore. For performance reasons you maybe want to add more Elasticsearch nodes or want to run the web interface separately from the server components. You can reach this by changing IP addresses in the Graylog configuration files or you can use our canned configurations which come with the graylog-ctl command.

The idea is to have one VM which is a central point for other VMs to fetch all needed configuration settings to join your cluster. Typically the first VM you spin up is used for this task. Automatically an instance of etcd is started and filled with the necessary settings for other hosts.

For example to split the web interface from the rest of the setup, spin up two VMs from the same graylog image. On the first only start graylog-server, elasticsearch and mongodb:

vm1> sudo graylog-ctl set-admin-password sEcReT
vm1> sudo graylog-ctl reconfigure-as-backend

On the second VM, start only the web interface but before set the IP of the first VM to fetch configuration data from:

vm2> sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm2> sudo graylog-ctl reconfigure-as-webinterface

This results in a perfectly fine dual VM setup. However if you want to scale this setup out by adding an additional Elasticsearch node, you can proceed in the same way:

vm3> sudo graylog-ctl set-cluster-master <ip-of-vm1>
vm3> sudo graylog-ctl reconfigure-as-datanode

The following configuration modes do exist:

Command	Services
sudo graylog-ctl reconfigure	Run all services on this box
sudo graylog-ctl reconfigure-as-backend	Run graylog-server, elasticsearch and mongodb
sudo graylog-ctl reconfigure-as-webinterface	Run only the web interface
sudo graylog-ctl reconfigure-as-datanode	Run only elasticsearch
sudo graylog-ctl reconfigure-as-server	Run graylog-server and mongodb (no elasticsearch)

Extend disk space¶

All data is stored in one directory /var/opt/graylog/data. In order to extend the disk space mount a second drive on this path. Make sure to move old data to the new drive before and give the graylog user permissions to read and write here.

Example procedure for an OVA appliance on VMWare:

Action	Explanation
shutdown the VM	Preparation for creating a consistend snapshot
take a snapshot through VMWare	Use the VMWare GUI to create a snapshot of the VM in case something goes wrong
attach an additional hard drive	Use the VMWare GUI to attach another harddrive suitable for the amount of logs you want to store
start the VM again and follow these steps:
sudo graylog-ctl stop	Stop all running services to prevent disk access
sudo lshw -class disk	Check for the logical name of the new hard drive. Usually this is /dev/sdb
sudo parted -a optimal /dev/sdb mklabel gpt (A reboot may be necessary at this point) sudo parted -a optimal – /dev/sdb unit \ compact mkpart primary ext3 “1” “-1” sudo mkfs.ext4 /dev/sdb1	Partition and format new disk
sudo mkdir /mnt/tmp sudo mount /dev/sdb1 /mnt/tmp	Mount disk to temporary mount point
cd /var/opt/graylog/data sudo cp -ax * /mnt/tmp/	Copy current data to new disk
sudo diff -qr –suppress-common-lines \ /var/opt/graylog/data /mnt/tmp	Compare both folders. Output should be: Only in /mnt/tmp: lost+found
sudo rm -rf /var/opt/graylog/data/*	Delete old data
sudo umount /mnt/tmp sudo mount /dev/sdb1 /var/opt/graylog/data	Mount new disk over data folder
echo “/dev/sdb1 /var/opt/graylog/data ext4 \ defaults 0 0” \| sudo tee -a /etc/fstab sudo shutdown -r now	Make change permanent

Install Graylog plugins¶

The Graylog plugin directory is located in /opt/graylog/plugin/. Just drop a JAR file there and restart the server with sudo graylog-ctl restart graylog-server to load the plugin.

Install Elasticsearch plugins¶

Elasticsearch comes with a helper program to install additional plugins you can call it like this sudo JAVA_HOME=/opt/graylog/embedded/jre /opt/graylog/elasticsearch/bin/plugin

Install custom SSL certificates¶

During the first reconfigure run self signed SSL certificates are generated. You can replace this certificate with your own to prevent security warnings in your browser. Just drop the key and combined certificate file here: /opt/graylog/conf/nginx/ca/graylog.crt respectively /opt/graylog/conf/nginx/ca/graylog.key. Afterwards restart nginx with sudo graylog-ctl restart nginx.

Configure Message Retention¶

Graylog is keeping a defined amount of messages. It is possible to decide whether you want to have a set storage size or a set time period of messages. Additionally Graylog writes a so called Journal. This is used to buffer messages in case of a unreachable Elasticsearch backend. To configure those settings use the set-retention command.

Retention by disk size:

sudo graylog-ctl set-retention --size=3 --indices=10
sudo graylog-ctl reconfigure

Indices would be rotated when they reach a size of 3Gb and Graylog would keep up to 10 indices, resulting in 30Gb maximum disk space.

Retention by time:

sudo graylog-ctl set-retention --time=24  --indices=30
sudo graylog-ctl reconfigure

Indices would be rotated after 24 hours and 30 indices would be kept, resulting in 30 days of stored logs.

Both commands can be extended with the –journal switch to set the maximum journal size in Gb:

sudo graylog-ctl set-retention --time=24  --indices=30 --journal=5
sudo graylog-ctl reconfigure

Assign a static IP¶

Per default the appliance make use of DHCP to setup the network. If you want to access Graylog under a static IP please follow these instructions:

$ sudo ifdown eth0

Edit the file /etc/network/interfaces like this (just the important lines):

auto eth0
  iface eth0 inet static
  address <static IP address>
  netmask <netmask>
  gateway <default gateway>
  pre-up sleep 2

Activate the new IP and reconfigure Graylog to make use of it:

$ sudo ifup eth0
$ sudo graylog-ctl reconfigure

Wait some time until all services are restarted and running again. Afterwards you should be able to access Graylog with the new IP.

Upgrade Graylog¶

Always perform a full backup or snapshot of the appliance before proceeding. Only upgrade if the release notes say the next version is a drop-in replacement. Look for the Graylog version you want to install here , graylog_latest always links to the newest version:

wget https://packages.graylog2.org/releases/graylog2-omnibus/ubuntu/graylog_latest.deb
sudo graylog-ctl stop
sudo dpkg -G -i graylog_latest.deb
sudo graylog-ctl reconfigure

Advanced Settings¶

To change certain parameters used by graylog-ctl during a reconfigure run you can override all default parameters found here. If you want to change the username used by Graylog for example, edit the file /etc/graylog/graylog-settings.json like this:

"custom_attributes": {
  "user": {
    "username": "log-user"
  }
}

Afterwards run sudo graylog-ctl reconfigure and sudo graylog-ctl restart. In this way you can change things like the path to the data directory or memory settings for Graylog and Elasticsearch

Production readiness¶

You can use the Graylog appliances (OVA, Docker, AWS, ...) for small production setups but please consider to harden the security of the box before.

Set another password for the default ubuntu user

Disable remote password logins in /etc/ssh/sshd_config and deploy proper ssh keys

Seperate the box network-wise from the outside, otherwise Elasticsearch can be reached by anyone

If you want to create your own customised setup take a look at our other installation methods.

Operating System Packages¶

Until configuration management systems made their way into broader markets and many datacenters, one of the most common ways to install software on Linux servers was to use operating system packages. Debian has DEB, Red Hat has RPM and many other distributions are based on those or come with own package formats. Online repositories of software packages and corresponding package managers make installing and configuring new software a matter of a single command and a few minutes of time.

Graylog offers official DEB and RPM package repositories for the following operating systems.

Ubuntu 12.04, 14.04
Debian 7, 8
RHEL/CentOS 6, 7

The repositories can be setup by installing a single package. Once that’s done the Graylog packages can be installed via apt-get or yum. The packages can also be downloaded with a web browser at https://packages.graylog2.org/ if needed.

Make sure to install and configure Java (>= 7), MongoDB and Elasticsearch 1.7 before starting the Graylog services.

Ubuntu 14.04¶

Download and install graylog-1.2-repository-ubuntu14.04_latest.deb via dpkg(1) and also make sure that the apt-transport-https package is installed:

$ sudo apt-get install apt-transport-https
$ wget https://packages.graylog2.org/repo/packages/graylog-1.2-repository-ubuntu14.04_latest.deb
$ sudo dpkg -i graylog-1.2-repository-ubuntu14.04_latest.deb
$ sudo apt-get update
$ sudo apt-get install graylog-server graylog-web