Welcome to the Graylog documentation¶

Minimalist¶

“Doing the needful”

The Minimalist Strategy proceeds from a “Default No” position when deciding which events to collect. What that means is you don’t collect any log unless it is required for an identified business use case. This strategy has some advantages, it keeps licensing and storage costs down, by reducing the volume of collected events. It also minimizes the “noise” produced by extraneous events, allowing analysts to focus on events that have maximum value. Finally, it improves system and query efficiency, improving performance overall.

Maximalist¶

“Collect it all, let Graylog sort it out.”

The Maximalist strategy is to collect all events that are produced by any source. The thinking goes, all log data is potentially valuable, especially for forensics. Collecting it all and keeping it forever guarantees you will have it if you need it. However, this strategy is often not practical, due to budgetary or other constraints. The cost of this strategy can be prohibitive, since many more technical and human resources must be devoted to collection, processing and storage of event data. There is a performance penalty associated with keeping extremely large data sets online that must be considered as well.

Virtual Appliance Caveats¶

Virtual appliances are not suitable for production deployment. They are created for lab or evaluation purposes. They do not have sufficient storage, nor do they offer capabilities like index replication that meet high availability requirements.

Also, because they are intended for internal testing and evaluation only, the virtual appliances are not hardened or otherwise secured. Use at your own risk and apply all security measures required by your organization.

General Properties¶

• is_master = true

  • If you are running more than one instances of Graylog server you must designate (only) one graylog-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t.

• password_secret = <secret>

  • You MUST set a secret that is used for password encryption and salting. The server will refuse to start if this value is not set. Use at least 64 characters. If you run multiple graylog-server nodes, make sure you use the same password_secret for all of them!

  Note

  Generate a secret with for example pwgen -N 1 -s 96.

• root_username = admin

  • The default root user is named admin.

• root_password_sha2 = <SHA2>

  • A SHA2 hash of the password you will use for your initial login. Insert a SHA2 hash generated with echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 and you will be able to log in to the web interface with username admin and password yourpassword.

  Caution

  You MUST specify a hash password for the root user (which you only need to initially set up the system and in case you lose connectivity to your authentication backend). This password cannot be changed using the API or via the web interface. If you need to change it, modify it in this file.

• rest_listen_uri = http://127.0.0.1:9000/api/

  • REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
  • Typically, this will be the “internal” IP address of the Graylog server.
  • When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.

• rest_transport_uri = http://192.168.1.1:9000/api/

  • REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
  • Typically, this will be the “external” or publicly accessible IP address of the Graylog server.
  • You will need to define this if your Graylog server is running behind a HTTP proxy that is rewriting the scheme, host name or URI.
  • If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
  • This must not contain a wildcard address (0.0.0.0).

Web Properties¶

• web_listen_uri = http://127.0.0.1:9000/

  • Web interface listen URI.
  • Typically, this will be the “internal” IP address of the Graylog server.
  • Configuring a path for the URI here effectively prefixes all URIs in the web interface. This is a replacement for the application.context configuration parameter in pre-2.0 versions of the Graylog web interface.

• web_endpoint_uri =

  • Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
  • It takes the value of rest_transport_uri by default. If the Graylog server is not behind a proxy or load balancer, changing the default setting is not necessary.

Elasticsearch Properties¶

• elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200

  • List of Elasticsearch hosts Graylog should connect to.
  • Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
  • If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that requires authentication.
  • Default: http://127.0.0.1:9200 You may retain the default setting only if Elasticsearch is installed on the same host as the Graylog server.

MongoDB¶

• mongodb_uri = mongdb://...

  • MongoDB connection string. Enter your MongoDB connection and authentication information here.

  • See https://docs.mongodb.com/manual/reference/connection-string/ for details.

  • Examples:

    • Simple: mongodb_uri = mongodb://localhost/graylog
    • Authenticate against the MongoDB server: mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog
    • Use a replica set instead of a single host: mongodb_uri = mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog

HTTP¶

• http_proxy_uri =

  • HTTP proxy for outgoing HTTP connections

• http_non_proxy_hosts =

  • A list of hosts that should be reached directly, bypassing the configured proxy server.
  • This is a list of patterns separated by ”,”. The patterns may start or end with a “*” for wildcards.
  • Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.

Overview¶

The Overview page displays information relating to the administration of the Graylog instance. It contains information on system notifications, system job status, ingestion rates, Elasticsearch cluster health, indexer failures, Time configuration and the system event messages.

Configuration¶

The Configuration page allows users to set options or variables related to searches, message processors and plugins.

Nodes¶

The Nodes page contains summary status information for each Graylog node. Detailed health information and metrics are available from buttons displayed on this page.

Inputs¶

Usually the first thing configured after initial system setup, Inputs are used to tell Graylog on which port to listen or how to go and retrieve event logs. The Inputs page allows users to create and configure new inputs, to manage extractors, to start and stop inputs, get metrics for each input and to add static fields to incoming messages.

Outputs¶

Outputs are used to define methods of forwarding data to remote systems, including port, protocol and any other required information. Out of the box, Graylog supports STDOUT and GELF outputs, but users may write their own and more are available in the Graylog Marketplace.

Authentication¶

The Authentication page is used to configure Graylog’s authentication providers and manage the active users of this Graylog cluster. Graylog supports LDAP or Active Directory for both authentication and authorization.

Content Packs¶

Content packs accelerate the set-up process for a specific data source. A content pack can include inputs/extractors, streams, dashboards, alerts, saved searches and pipeline processors.

Any program element created within Graylog may be exported as Content Packs for use on other systems. These may be kept private by the author, for use in quick deployment of new nodes internally, or may be shared with the community via the Graylog Marketplace. For example, users may create custom Inputs, Streams, Dashboards, and Alerts to support a security use case. These elements may be exported in a content pack and then imported on a newly installed Graylog instance to save configuration time and effort.

Users may download content packs created and shared by other users via the Graylog Marketplace. User created content packs are not supported by Graylog, but instead by their authors.

List of Elements Supported in Content Packs

• Inputs
• Grok Patterns
• Outputs
• Streams
• Dashboards
• Lookup Tables
• Lookup Caches
• Lookup Data Adapters

Indices¶

An Index is the basic unit of storage for data in Elasticsearch. Index sets provide configuration for retention, sharding, and replication of the stored data.

Values, like retention and rotation strategy, are set on a per index basis, so different data may be subjected to different handling rules.

For more details, please see Index model.

Collectors/Sidecars¶

Graylog created the Sidecar agent to manage fleets of log shippers like Beats or NXLog. These log shippers are used to collect OS logs from Windows servers, but also for OS logs from *nix systems. Log shippers are often the simplest way to read logs written locally to a flat file and send them to a centralized log management solution. Graylog supports management of any log shipper as a back-end, but includes Beats and NXLog binaries in the agent package.

For more details, please see Graylog Collector Sidecar.

Pipelines¶

Graylog’s Processing Pipelines are a powerful feature that enables user to run a rule, or a series of rules, against a specific type of event. Tied to streams, pipelines allow for routing, blacklisting, modifying and enriching messages as they flow through Graylog. Basically, if you want to parse, change, convert. add to, delete from or drop a message, Pipelines are the place to do it.

For more details, please see Processing Pipelines.

If You Don’t Have Messages¶

1. Check to see that you made the proper entries in the input configuration described above.
2. Check the configuration at the event source and make sure that it matches the ports and other options defined in the input. For example, if you changed the port for a Syslog UDP input to 5014, be sure the sending device has that same port defined.
3. Check to see if traffic is coming to the defined port. You can use the tcpdump command to do this:

$ sudo tcpdump -i lo host 127.0.0.1 and udp port 5014

4. Check to see if the server is listening on the host:

$ sudo netstat -peanut | grep ":5014"

If you still have issues, connect to our community support or get in touch with us via the professional support offering.

Update to latest version¶

If you’ve been using the repository package to install Graylog before, it has to be updated first. The new package will replace the repository URL, without which you will only be able to get bugfix releases of your previously installed version of Graylog.

The update basically works like a fresh installation:

$ wget https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.deb
$ sudo dpkg -i graylog-2.4-repository_latest.deb
$ sudo apt-get update
$ sudo apt-get install graylog-server

Manual Repository Installation¶

If you don’t like to install the repository DEB to get the repository configuration onto your system, you can do so manually (although we don’t recommend to do that).

First, add the Graylog GPG keyring which is being used to sign the packages to your system.

Now create a file /etc/apt/sources.list.d/graylog.list with the following content:

deb https://packages.graylog2.org/repo/debian/ stable 2.4

Update to latest version¶

If you’ve been using the repository package to install Graylog before, it has to be updated first. The new package will replace the repository URL, without which you will only be able to get bugfix releases of your previously installed version of Graylog.

The update basically works like a fresh installation:

$ sudo rpm -Uvh https://packages.graylog2.org/repo/packages/graylog-2.4-repository_latest.rpm
$ sudo yum clean all
$ sudo yum install graylog-server

Running yum clean all is required because YUM might use a stale cache and thus might be unable to find the latest version of the graylog-server package.

Manual Repository Installation¶

If you don’t like to install the repository RPM to get the repository configuration onto your system, you can do so manually (although we don’t recommend to do that).

First, add the Graylog GPG key which is being used to sign the packages to your system.

Now create a file named /etc/yum.repos.d/graylog.repo with the following content:

[graylog]
name=graylog
baseurl=https://packages.graylog2.org/repo/el/stable/2.4/$basearch/
gpgcheck=1
repo_gpgcheck=0
gpgkey=file:///etc/pki/rpm-gpg/RPM-GPG-KEY-graylog

How to get log data in¶

You can create different kinds of inputs under System / Inputs, however you can only use ports that have been properly mapped to your docker container, otherwise data will not go through.

For example, to start a Raw/Plaintext TCP input on port 5555, stop your container and recreate it, whilst appending -p 5555:5555 to your docker run command:

$ docker run --link mongo --link elasticsearch \
    -p 9000:9000 -p 12201:12201 -p 514:514 -p 5555:5555 \
    -e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api" \
    -d graylog/graylog:2.4

Similarly, the same can be done for UDP by appending -p 5555:5555/udp.

After that you can send a plaintext message to the Graylog Raw/Plaintext TCP input running on port 5555 using the following command:

$ echo 'First log message' | nc localhost 5555

Settings¶

Graylog comes with a default configuration that works out of the box but you have to set a password for the admin user and the web interface needs to know how to connect from your browser to the Graylog REST API.

Both settings can be configured via environment variables (also see Configuration):

-e GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
-e GRAYLOG_WEB_ENDPOINT_URI="http://127.0.0.1:9000/api"

In this case you can login to Graylog with the username and password admin.

Generate your own admin password with the following command and put the SHA-256 hash into the GRAYLOG_ROOT_PASSWORD_SHA2 environment variable:

echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1

All these settings and command line parameters can be put in a docker-compose.yml file, so that they don’t have to be executed one after the other.

Example:

version: '2'
services:
  # MongoDB: https://hub.docker.com/_/mongo/
  mongodb:
    image: mongo:3
  # Elasticsearch: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/docker.html
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.12
    environment:
      - http.host=0.0.0.0
      - transport.host=localhost
      - network.host=0.0.0.0
      # Disable X-Pack security: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/security-settings.html#general-security-settings
      - xpack.security.enabled=false
      - xpack.watcher.enabled=false
      - xpack.monitoring.enabled=false
      - xpack.security.audit.enabled=false
      - xpack.ml.enabled=false
      - xpack.graph.enabled=false
      - "ES_JAVA_OPTS=-Xms512m -Xmx512m"
    ulimits:
      memlock:
        soft: -1
        hard: -1
    mem_limit: 1g
  # Graylog: https://hub.docker.com/r/graylog/graylog/
  graylog:
    image: graylog/graylog:2.4
    environment:
      # CHANGE ME!
      - GRAYLOG_PASSWORD_SECRET=somepasswordpepper
      # Password: admin
      - GRAYLOG_ROOT_PASSWORD_SHA2=8c6976e5b5410415bde908bd4dee15dfb167a9c873fc4bb8a81f6f2ab448a918
      - GRAYLOG_WEB_ENDPOINT_URI=http://127.0.0.1:9000/api
    links:
      - mongodb:mongo
      - elasticsearch
    depends_on:
      - mongodb
      - elasticsearch
    ports:
      # Graylog web interface and REST API
      - 9000:9000
      # Syslog TCP
      - 514:514
      # Syslog UDP
      - 514:514/udp
      # GELF TCP
      - 12201:12201
      # GELF UDP
      - 12201:12201/udp

After starting all three Docker containers by running docker-compose up, you can open the URL http://127.0.0.1:9000 in a web browser and log in with username admin and password admin (make sure to change the password later).

Custom configuration files¶

Instead of using a long list of environment variables to configure Graylog (see Configuration), you can also overwrite the bundled Graylog configuration files.

The bundled configuration files are stored in /usr/share/graylog/data/config/ inside the Docker container.

Create the new configuration directory next to the docker-compose.yml file and copy the default files from GitHub:

$ mkdir -p ./graylog/config
$ cd ./graylog/config
$ wget https://raw.githubusercontent.com/Graylog2/graylog-docker/2.4/config/graylog.conf
$ wget https://raw.githubusercontent.com/Graylog2/graylog-docker/2.4/config/log4j2.xml

The newly created directory ./graylog/config/ with the custom configuration files now has to be mounted into the Graylog Docker container.

This can be done by adding an entry to the volumes section of the docker-compose.yml file:

version: '2'
services:
  mongodb:
    image: mongo:3
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.12
    # Other settings [...]
  graylog:
    image: graylog/graylog:2.4
    # Other settings [...]
    volumes:
      # Mount local configuration directory into Docker container
      - ./graylog/config:/usr/share/graylog/data/config

New Docker image¶

Simply create a new Dockerfile in an empty directory with the following contents:

FROM graylog/graylog:2.4
RUN wget -O /usr/share/graylog/plugin/graylog-plugin-auth-sso-2.4.0.jar https://github.com/Graylog2/graylog-plugin-auth-sso/releases/download/2.4.0/graylog-plugin-auth-sso-2.4.0.jar

Build a new image from the new Dockerfile (also see docker build):

$ docker build -t graylog-with-sso-plugin .

In this example, we created a new image with the SSO plugin installed. From now on reference to the newly built image instead of graylog/graylog.

The docker-compose.yml file has to reference the new Docker image:

version: '2'
services:
  mongo:
    image: "mongo:3"
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.12
    # Other settings [...]
  graylog:
    image: graylog-with-sso-plugin
    # Other settings [...]

Volume-mounted plugin¶

Instead of building a new docker image, you can also add additional plugins by mounting them directly and individually into the plugin folder of the original Docker image. This way, you don’t have to create a new docker image every time you want to add a new plugin (or remove an old one).

Simply create a plugin folder, download the plugin(s) you want to install into it and mount each file as an additional volume into the docker container:

$ mkdir -p ./graylog/plugin
$ wget -O ./graylog/plugin/graylog-plugin-auth-sso-2.4.0.jar https://github.com/Graylog2/graylog-plugin-auth-sso/releases/download/2.4.0/graylog-plugin-auth-sso-2.4.0.jar

The docker-compose.yml file has to reference the new Docker image:

version: '2'
services:
  mongo:
    image: "mongo:3"
    # Other settings [...]
  elasticsearch:
    image: docker.elastic.co/elasticsearch/elasticsearch:5.6.12
    # Other settings [...]
  graylog:
    image: graylog/graylog:2.4
    # Other settings [...]
    volumes:
      # Mount local plugin file into Docker container
      - ./graylog/plugin/graylog-plugin-auth-sso-2.4.0.jar:/usr/share/graylog/plugin/graylog-plugin-auth-sso-2.4.0.jar

You can add as many of these links as you wish in your docker-compose.yml file. Simply restart the container and docker will recreate the graylog container with the new volumes included:

$ docker-compose restart

Configuration¶

We are shipping the graylog-ctl tool with the virtual machine appliances to get you started with a customised setup as quickly as possible. Run these (optional) commands to configure the most basic settings of Graylog in the appliance:

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

The graylog-ctl has much more functionality documented. We strongly recommend to learn more about it to ensure smooth operation of your virtual appliance.

If you want to create your own customised setup take a look at our other installation methods.

Basic configuration¶

We are shipping the graylog-ctl tool with the virtual machine appliances to get you started with a customised setup as quickly as possible. Run these (optional) commands to configure the most basic settings of Graylog in the appliance:

sudo graylog-ctl set-email-config <smtp server> [--port=<smtp port> --user=<username> --password=<password>]
sudo graylog-ctl set-admin-password <password>
sudo graylog-ctl set-timezone <zone acronym>
sudo graylog-ctl reconfigure

The graylog-ctl has much more functionality documented. We strongly recommend to learn more about it to ensure smooth operation of your virtual appliance.

Production readiness¶

The Graylog appliance is not created to provide a production ready solution. It is build to offer a fast and easy way to try the software itself and not wasting time to install Graylog and it components to any kind of server.

If you want to create your own production ready setup take a look at our other installation methods.

Production readiness¶

The Graylog appliance is not created to provide a production ready solution. It is build to offer a fast and easy way to try the software itself and not wasting time to install Graylog and it components to any kind of server.

If you want to create your own production ready setup take a look at our other installation methods.

Prerequisites¶

Graylog depends on MongoDB and Elasticsearch to operate, please refer to the system requirements for details.

Downloading and extracting the server¶

Download the tar archive from the download pages and extract it on your system:

~$ tar xvfz graylog-VERSION.tgz
~$ cd graylog-VERSION

Configuration¶

Now copy the example configuration file:

~# cp graylog.conf.example /etc/graylog/server/server.conf

You can leave most variables as they are for a first start. All of them should be well documented.

Configure at least the following variables in /etc/graylog/server/server.conf:

• is_master = true

  • Set only one graylog-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t. Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.

• password_secret

  • You must set a secret that is used for password encryption and salting here. The server will refuse to start if it’s not set. Generate a secret with for example pwgen -N 1 -s 96. If you run multiple graylog-server nodes, make sure you use the same password_secret for all of them!

• root_password_sha2

  • A SHA2 hash of a password you will use for your initial login. Set this to a SHA2 hash generated with echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 and you will be able to log in to the web interface with username admin and password yourpassword.

• elasticsearch_shards = 4

  • The number of shards for your indices. A good setting here highly depends on the number of nodes in your Elasticsearch cluster. If you have one node, set it to 1.

• elasticsearch_replicas = 0

  • The number of replicas for your indices. A good setting here highly depends on the number of nodes in your Elasticsearch cluster. If you have one node, set it to 0.

• mongodb_uri

  • Enter your MongoDB connection and authentication information here.

Starting the server¶

You need to have Java installed. Running the OpenJDK is totally fine and should be available on all platforms. For example on Debian it is:

~$ apt-get install openjdk-8-jre

Start the server:

~$ cd bin/
~$ ./graylogctl start

The server will try to write a node_id to the graylog-server-node-id file. It won’t start if it can’t write there because of for example missing permissions.

See the startup parameters description below to learn more about available startup parameters. Note that you might have to be root to bind to the popular port 514 for syslog inputs.

You should see a line like this in the debug output of Graylog successfully connected to your Elasticsearch cluster:

2013-10-01 12:13:22,382 DEBUG: org.elasticsearch.transport.netty - [graylog-server] connected to node [[Unuscione, Angelo][thN_gIBkQDm2ab7k-2Zaaw][inet[/10.37.160.227:9300]]]

You can find the logs of Graylog in the directory logs/.

Important: All systems running Graylog must have synchronised system time. We strongly recommend to use NTP or similar mechanisms on all machines of your Graylog infrastructure.

Supplying external logging configuration¶

Graylog is using Apache Log4j 2 for its internal logging and ships with a default log configuration file which is embedded within the shipped JAR.

In case you need to modify Graylog’s logging configuration, you can supply a Java system property specifying the path to the configuration file in your start script (e. g. graylogctl).

Append this before the -jar parameter:

-Dlog4j.configurationFile=file:///path/to/log4j2.xml

Substitute the actual path to the file for the /path/to/log4j2.xml in the example.

In case you do not have a log rotation system already in place, you can also configure Graylog to rotate logs based on their size to prevent the log files to grow without bounds using the RollingFileAppender.

One such example log4j2.xml configuration is shown below:

<?xml version="1.0" encoding="UTF-8"?>
<Configuration packages="org.graylog2.log4j" shutdownHook="disable">
  <Appenders>
      <RollingFile name="RollingFile" fileName="/tmp/logs/graylog.log"
                   filePattern="/tmp/logs/graylog-%d{yyyy-MM-dd}.log.gz">
        <PatternLayout>
          <Pattern>%d %-5p: %c - %m%n</Pattern>
        </PatternLayout>
        <!-- Rotate logs every day or when the size exceeds 10 MB (whichever comes first) -->
        <Policies>
          <TimeBasedTriggeringPolicy modulate="true"/>
          <SizeBasedTriggeringPolicy size="10 MB"/>
        </Policies>
        <!-- Keep a maximum of 10 log files -->
        <DefaultRolloverStrategy max="10"/>
      </RollingFile>

      <Console name="STDOUT" target="SYSTEM_OUT">
          <PatternLayout pattern="%d %-5p: %c - %m%n"/>
      </Console>

      <!-- Internal Graylog log appender. Please do not disable. This makes internal log messages available via REST calls. -->
      <Memory name="graylog-internal-logs" bufferSize="500"/>
  </Appenders>
  <Loggers>
      <Logger name="org.graylog2" level="info"/>
      <Logger name="com.github.joschi.jadconfig" level="warn"/>
      <Logger name="org.apache.directory.api.ldap.model.message.BindRequestImpl" level="error"/>
      <Logger name="org.elasticsearch.script" level="warn"/>
      <Logger name="org.graylog2.periodical.VersionCheckThread" level="off"/>
      <Logger name="org.drools.compiler.kie.builder.impl.KieRepositoryImpl" level="warn"/>
      <Logger name="com.joestelmach.natty.Parser" level="warn"/>
      <Logger name="kafka.log.Log" level="warn"/>
      <Logger name="kafka.log.OffsetIndex" level="warn"/>
      <Logger name="org.apache.shiro.session.mgt.AbstractValidatingSessionManager" level="warn"/>
      <Root level="warn">
          <AppenderRef ref="STDOUT"/>
          <AppenderRef ref="RollingFile"/>
          <AppenderRef ref="graylog-internal-logs"/>
      </Root>
  </Loggers>
</Configuration>

Command line (CLI) parameters¶

There are a number of CLI parameters you can pass to the call in your graylogctl script:

• -h, --help: Show help message
• -f CONFIGFILE, --configfile CONFIGFILE: Use configuration file CONFIGFILE for Graylog; default: /etc/graylog/server/server.conf
• -d, --debug: Run in debug mode
• -l, --local: Run in local mode. Automatically invoked if in debug mode. Will not send system statistics, even if enabled and allowed. Only interesting for development and testing purposes.
• -p PIDFILE, --pidfile PIDFILE: Set the file containing the PID of graylog to PIDFILE; default: /tmp/graylog.pid
• -np, --no-pid-file: Do not write PID file (overrides -p/--pidfile)
• --version: Show version of Graylog and exit

Problems with IPv6 vs. IPv4?¶

If your Graylog node refuses to listen on IPv4 addresses and always chooses for example a rest_listen_address like :::9000 you can tell the JVM to prefer the IPv4 stack.

Add the java.net.preferIPv4Stack flag in your graylogctl script or from wherever you are calling the graylog.jar:

~$ sudo -u graylog java -Djava.net.preferIPv4Stack=true -jar graylog.jar

Create a message input and send a first message¶

Log in to the web interface on port 9000 (e.g. http://127.0.0.1:9000) and navigate to System -> Inputs.

Launch a new Raw/Plaintext UDP input, listening on 127.0.0.1 on port 9099. There’s no need to configure anything else for now.

The list of running inputs on that node should show you your new input right away.

Let’s send a message in:

echo "Hello Graylog, let's be friends." | nc -w 1 -u 127.0.0.1 9099

This has sent a short string to the raw UDP input you just opened. Now search for friends using the search bar on the top and you should already see the message you just sent in. Click on it in the table and see it in detail:

You have just sent your first message to Graylog! Why not spawn a syslog input and point some of your servers to it? You could also create some user accounts for your colleagues.

Multicast Discovery¶

Multicast discovery has been removed from Elasticsearch 2.x (although it is still provided as an Elasticsearch plugin for now).

To reflect this change, the elasticsearch_discovery_zen_ping_unicast_hosts now has to contain the address of at least one Elasticsearch node in the cluster which Graylog can connect to.

Default network host¶

The network interface which Elasticsearch binds to (elasticsearch_network_host) has been changed to localhost (i. e. 127.0.0.1 or ::1); see Network changes/Bind to localhost.

If Elasticsearch is not running on the same machine, elasticsearch_network_host must be set to a host name or an IP address which can be accessed by the other Elasticsearch nodes in the cluster.

Index range types¶

Some Graylog versions stored meta information about indices in elasticsearch, alongside the messages themselves. Since Elasticsearch 2.0 having multiple types with conflicting mappings is no longer possible, which means that the index_range type must be removed before upgrading to Elasticsearch 2.x.

Find out if your setup is affected by running (replace $elasticsearch with the address of one of your Elasticsearch nodes) curl -XGET $elasticsearch:9200/_all/_mapping/index_range; echo

If the output is {} you are not affected and can skip this step.

Otherwise, you need to delete the index_range type, Graylog does not use it anymore.

As Graylog sets older indices to read-only, first we need to remove the write block on those indices. Since we’ll be working with Elasticsearch’s JSON output, we recommend installing the jq utility which should be available on all popular package managers or directly at GitHub.

for i in `curl -s -XGET $elasticsearch:9200/_all/_mapping/index_range | jq -r "keys[]"`; do
    echo -n "Updating index $i: "
    echo -n "curl -XPUT $elasticsearch:9200/$i/_settings -d '{\"index.blocks.read_only\":false, \"index.blocks.write\":false}' : "
    curl -XPUT $elasticsearch:9200/$i/_settings -d '{"index.blocks.read_only":false, "index.blocks.write":false}'
    echo
done

The output for each of the curl commands should be {"acknowledged":true}. Next we have to delete the index_range mapping. We can perform this via the next command.

for i in `curl -s -XGET $elasticsearch:9200/_all/_mapping/index_range | jq -r "keys[]"`; do
    echo -n "Updating index $i: "
    curl -XDELETE $elasticsearch:9200/$i/index_range
    echo
done

It is not strictly necessary to set the indices back to read only, but if you prefer to do that, note the index names and commands during the first step and change the false into true.

Graylog Index Template¶

Graylog applies a custom index template to ensure that the indexed messages adhere to a specific schema.

Unfortunately the index template being used by Graylog 1.x is incompatible with Elasticsearch 2.x and has to be removed prior to upgrading.

In order to delete the index template the following curl command has to be issued against on of the Elasticsearch nodes:

curl -X DELETE http://localhost:9200/_template/graylog-internal

Graylog will automatically create the new index template on the next startup.

Dots in field names¶

One of the most important breaking changes in Elasticsearch 2.x is that field names may not contain dots anymore.

Using the Elasticsearch Migration Plugin might help to highlight some potential pitfalls if an existing Elasticsearch 1.x cluster should be upgraded to Elasticsearch 2.x.

Index Retention and Rotation Settings¶

In 2.0.0 the index rotation and retention settings have been moved from the Graylog server config file to the database and are now configurable via the web interface.

The old settings from the graylog.conf or /etc/graylog/server/server.conf will be migrated to the database.

Overview¶

Some settings, which have been deprecated in previous versions, have finally been removed from the Graylog configuration file.

Network settings¶

The network settings in the Graylog configuration file (rest_listen_uri, rest_transport_uri, and web_listen_uri) are now using the default ports for the HTTP (80) and HTTPS (443) if no custom port was given. Previously those settings were using the custom ports 12900 (Graylog REST API) and 9000 (Graylog web interface) if no explicit port was given.

Examples:

Collector Sidecar¶

The network changes are reflected in the Sidecar configuration as well and should be adopted. However it’s still possible to use the old API port by setting it explicitly. In case a mass deployment is too hard to change, just run the following to switch back to the old REST API port (OVA based installation):

sudo graylog-ctl set-listen-address --service rest --address http://0.0.0.0:12900
sudo graylog-ctl reconfigure

Removed resources¶

Removed index rotation/retention settings from “/system/configuration”¶

The index rotation and retention settings have been moved to MongoDB in Graylog 2.0.0 but the representation of the old configuration options was still present in the /system/configuration resource.

In order to stay in sync with the actual configuration file, the following values have been removed:

• rotation_strategy
• retention_strategy
• elasticsearch_max_docs_per_index
• elasticsearch_max_size_per_index
• elasticsearch_max_time_per_index
• elasticsearch_max_number_of_indices

The retention and rotation configuration settings can be retrieved using the following resources:

• /system/indices/rotation/config
• /system/indices/retention/config

UI Plugins¶

The new app prefix feature requires some changes in UI plugins to make them work with that.

• import webpackEntry from 'webpack-entry'; needs to be added at the very top of the src/web/index.jsx file
• The Routes.pluginRoute() function needs to be used instead of a literal string to build URLs for links and buttons

Please check the updated plugins documentation for details.

Streams API¶

Due to the introduction of index sets, the payload for creating, updating and cloning of streams now requires the index_set_id field. The value for this needs to be the ID of an existing index set.

Affected endpoints:

• POST /streams
• PUT  /streams/{streamId}
• POST /streams/{streamId}/clone

Special note for upgrading from an existing Graylog setup with a new Elasticsearch cluster¶

If you are upgrading the Elasticsearch cluster of an existing Graylog setup without migrating the indices, your Graylog setup contains stale index ranges causing nonexisting index errors upon search/alerting. To remediate this, you need to manually trigger an index range recalculation for all index sets once. This is possible using the web interface using the System->Indices functionality or by using the REST API using the /system/indices/ranges/<index set id>/rebuild endpoint.

Rotation and Retention strategies¶

The deprecated HTTP resources at /system/indices/rotation/config and /system/indices/retention/config, which didn’t work since Graylog 2.2.0, have been removed.

These settings are part of the index set configuration and can be configured under /system/indices/index_sets.

Stream List Response structure does not include in_grace field anymore¶

The response to GET /streams, GET /streams/<id> & PUT /streams/<id> does not contain the in_grace field for configured alert conditions anymore.

The value of this flag can be retrieved using the GET /alerts/conditions endpoint, or per stream using the GET /streams/<streamId>/alerts/conditions endpoint.

General¶

• is_master = true

  • If you are running more than one instances of Graylog server you have to select only one graylog-server node as the master. This node will perform periodical and maintenance actions that slave nodes won’t.
  • Every slave node will accept messages just as the master nodes. Nodes will fall back to slave mode if there already is a master in the cluster.

• node_id_file = /etc/graylog/server/<node-id>

  • The auto-generated node ID will be stored in this file and read after restarts. It is a good idea to use an absolute file path here if you are starting Graylog server from init scripts or similar.

• password_secret = <secret>

  • You MUST set a secret that is used for password encryption and salting. The server will refuse to start if it’s not set. Use at least 64 characters. If you run multiple graylog-server nodes, make sure you use the same password_secret for all of them!

  Note

  Generate a secret with for example pwgen -N 1 -s 96

• root_username = admin

  • The default root user is named admin.

• root_password_sha2 = <SHA2>

  • A SHA2 hash of a password you will use for your initial login. Set this to a SHA2 hash generated with echo -n "Enter Password: " && head -1 </dev/stdin | tr -d '\n' | sha256sum | cut -d" " -f1 and you will be able to log in to the web interface with username admin and password yourpassword.

  Caution

  You MUST specify a hash password for the root user (which you only need to initially set up the system and in case you lose connectivity to your authentication backend). This password cannot be changed using the API or via the web interface. If you need to change it, modify it in this file.

• root_email = ""

  • The email address of the root user. Default is empty.

• root_timezone = UTC

  • The time zone setting of the root user. See this list of valid time zones. Default is UTC.

• plugin_dir = plugin

  • Set plugin directory here (relative or absolute)

• rest_listen_uri = http://127.0.0.1:9000/api/

  • REST API listen URI. Must be reachable by other Graylog server nodes if you run a cluster.
  • When using Graylog Collectors, this URI will be used to receive heartbeat messages and must be accessible for all collectors.

• rest_transport_uri = http://192.168.1.1:9000/api/

  • REST API transport address. Defaults to the value of rest_listen_uri. Exception: If rest_listen_uri is set to a wildcard IP address (0.0.0.0) the first non-loopback IPv4 system address is used.
  • If set, this will be promoted in the cluster discovery APIs, so other nodes may try to connect on this address and it is used to generate URLs addressing entities in the REST API. (see rest_listen_uri)
  • You will need to define this, if your Graylog server is running behind a HTTP proxy that is rewriting the scheme, host name or URI.
  • This must not contain a wildcard address (0.0.0.0).

• rest_enable_cors = false

  • Enable CORS headers for REST API. This is necessary for JS-clients accessing the server directly.
  • If these are disabled, modern browsers will not be able to retrieve resources from the server. This is enabled by default.

• rest_enable_gzip = false

  • Enable GZIP support for REST API. This compresses API responses and therefore helps to reduce overall round trip times. This is enabled by default.

• rest_enable_tls = true

  • Enable HTTPS support for the REST API. This secures the communication with the REST API with TLS to prevent request forgery and eavesdropping. This is disabled by default.

• rest_tls_cert_file = /path/to/graylog.crt

  • The X.509 certificate chain file in PEM format to use for securing the REST API.

• rest_tls_key_file = /path/to/graylog.key

  • The PKCS#8 private key file in PEM format to use for securing the REST API.

• rest_tls_key_password = secret

  • The password to unlock the private key used for securing the REST API.

• rest_max_header_size = 8192

  • The maximum size of the HTTP request headers in bytes.

• rest_max_initial_line_length = 4096

  • The maximal length of the initial HTTP/1.1 line in bytes.

• rest_thread_pool_size = 16

  • The size of the thread pool used exclusively for serving the REST API.

• trusted_proxies = 127.0.0.1/32, 0:0:0:0:0:0:0:1/128

  • Comma separated list of trusted proxies that are allowed to set the client address with X-Forwarded-For header. May be subnets, or hosts.

Web¶

• web_enable = true

  • Enable the embedded Graylog web interface. Enabled by default.

• web_listen_uri = http://127.0.0.1:9000/

  • Web interface listen URI.
  • Configuring a path for the URI here effectively prefixes all URIs in the web interface. This is a replacement for the application.context configuration parameter in pre-2.0 versions of the Graylog web interface.

• web_endpoint_uri =

  • Web interface endpoint URI. This setting can be overriden on a per-request basis with the X-Graylog-Server-URL header.
  • It takes the value of rest_transport_uri by default.

• web_enable_cors = true

  • Enable CORS headers for the web interface. This is necessary for JS-clients accessing the server directly.
  • If these are disabled, modern browsers will not be able to retrieve resources from the server.

• web_enable_gzip = true

  • Enable/disable GZIP support for the web interface. This compresses HTTP responses and therefore helps to reduce overall round trip times. This is enabled by default.

• web_enable_tls = false

  • Enable HTTPS support for the web interface. This secures the communication of the web browser with the web interface using TLS to prevent request forgery and eavesdropping.
  • This is disabled by default. Set it to true to enable it and see the other related configuration settings.

• web_tls_cert_file = /path/to/graylog-web.crt

  • The X.509 certificate chain file in PEM format to use for securing the web interface.

• web_tls_key_file = /path/to/graylog-web.key

  • The PKCS#8 private key file in PEM format to use for securing the web interface.

• web_tls_key_password = secret

  • The password to unlock the private key used for securing the web interface.

• web_max_header_size = 8192

  • The maximum size of the HTTP request headers in bytes.

• web_max_initial_line_length = 4096

  • The maximal length of the initial HTTP/1.1 line in bytes.

• web_thread_pool_size = 16

  • The size of the thread pool used exclusively for serving the web interface.

Elasticsearch¶

• elasticsearch_hosts = http://node1:9200,http://user:password@node2:19200

  • List of Elasticsearch hosts Graylog should connect to.
  • Need to be specified as a comma-separated list of valid URIs for the http ports of your elasticsearch nodes.
  • If one or more of your elasticsearch hosts require authentication, include the credentials in each node URI that requires authentication.
  • Default: http://127.0.0.1:9200

• elasticsearch_connect_timeout = 10s

  • Maximum amount of time to wait for successfull connection to Elasticsearch HTTP port.
  • Default: 10 seconds

• elasticsearch_socket_timeout = 60s

  • Maximum amount of time to wait for reading back a response from an Elasticsearch server.
  • Default: 60 seconds

• elasticsearch_idle_timeout = -1s

  • Maximum idle time for an Elasticsearch connection. If this is exceeded, this connection will be tore down.
  • Default: infinity

• elasticsearch_max_total_connections = 20

  • Maximum number of total connections to Elasticsearch.
  • Default: 20

• elasticsearch_max_total_connections_per_route = 2

  • Maximum number of total connections per Elasticsearch route (normally this means per elasticsearch server).
  • Default: 2

• elasticsearch_max_retries = 2

  • Maximum number of times Graylog will retry failed requests to Elasticsearch.
  • Default: 2

• elasticsearch_discovery_enabled = false

  • Enable automatic Elasticsearch node discovery through Nodes Info, see Elasticsearch Reference » Cluster APIs » Nodes Info.
  • Default: false

  Warning

  Automatic node discovery does not work if Elasticsearch requires authentication, e. g. with Shield.

  Warning

  This setting must be false on AWS Elasticsearch Clusters (the hosted ones) and should be used carefully. In case of trouble with connections to ES this should be the first option to be disabled. See Automatic node discovery for more details.

• elasticsearch_discovery_filter = rack:42

  • Filter for including/excluding Elasticsearch nodes in discovery according to their custom attributes, see Elastic Search Reference » Cluster APIs » Node Specification.
  • Default: empty

• elasticsearch_discovery_frequency = 30s

  • Frequency of the Elasticsearch node discovery.
  • Default: 30 seconds

• elasticsearch_compression_enabled = false

  • Enable payload compression for Elasticsearch requests.
  • Default: false

Rotation¶

• rotation_strategy = count !

  • Graylog will use multiple indices to store documents in. You can configured the strategy it uses to determine when to rotate the currently active write index.
  • It supports multiple rotation strategies: - count of messages per index, use elasticsearch_max_docs_per_index - size per index, use elasticsearch_max_size_per_index
  • valid values are count, size and time, default is count.

• elasticsearch_max_docs_per_index = 20000000 !

  • (Approximate) maximum number of documents in an Elasticsearch index before a new index is being created, also see no_retention and elasticsearch_max_number_of_indices.
  • Configure this if you used rotation_strategy = count above.

• elasticsearch_max_size_per_index = 1073741824 !

  • (Approximate) maximum size in bytes per Elasticsearch index on disk before a new index is being created, also see no_retention and `elasticsearch_max_number_of_indices`. Default is 1GB.
  • Configure this if you used rotation_strategy = size above.

• elasticsearch_max_time_per_index = 1d !

  • (Approximate) maximum time before a new Elasticsearch index is being created, also see no_retention and elasticsearch_max_number_of_indices. Default is 1 day.

  • Configure this if you used rotation_strategy = time above.

  • Please note that this rotation period does not look at the time specified in the received messages, but is using the real clock value to decide when to rotate the index!

  • Specify the time using a duration and a suffix indicating which unit you want:

    • 1w = 1 week
    • 1d = 1 day
    • 12h = 12 hours

  • Permitted suffixes are: d for day, h for hour, m for minute, s for second.

• elasticsearch_max_number_of_indices = 20 !

  • How many indices do you want to keep?

• retention_strategy = delete !

  • Decide what happens with the oldest indices when the maximum number of indices is reached.

  • The following strategies are availble:

    • delete - Deletes the index completely (Default)
    • close - Closes the index and hides it from the system. Can be re-opened later.

• elasticsearch_disable_version_check = true

  • Disable checking the version of Elasticsearch for being compatible with this Graylog release.

  Warning

  Using Graylog with unsupported and untested versions of Elasticsearch may lead to data loss!

• no_retention = false

  • Disable message retention, i. e. disable Elasticsearch index rotation.

  Note

  If this is set to true Graylog will never rotate indices - whatever is selected in the Webinterface.

• elasticsearch_shards = 4 !!

  • The number of shards for your indices. A good setting here highly depends on the number of nodes in your Elasticsearch cluster. If you have one node, set it to 1.

• elasticsearch_replicas = 0 !!

  • The number of replicas for your indices. A good setting here highly depends on the number of nodes in your Elasticsearch cluster. If you have one node, set it to 0.

  Note

  elasticsearch_shards and elasticsearch_replicas only applies to newly created indices.

• elasticsearch_index_prefix = graylog !!

  • Prefix for all Elasticsearch indices and index aliases managed by Graylog.

• elasticsearch_template_name = graylog-internal !!

  • Name of the Elasticsearch index template used by Graylog to apply the mandatory index mapping.
  • Default: graylog-internal

• elasticsearch_analyzer = standard !!

  • Analyzer (tokenizer) to use for message and full_message field. The “standard” filter usually is a good idea.
  • All supported analyzers are: standard, simple, whitespace, stop, keyword, pattern, language, snowball, custom
  • Elasticsearch documentation: https://www.elastic.co/guide/en/elasticsearch/reference/5.6/analysis.html
  • Note that this setting only takes effect on newly created indices.

• disable_index_optimization = false !!

  • Disable the optimization of Elasticsearch indices after index cycling. This may take some load from Elasticsearch on heavily used systems with large indices, but it will decrease search performance. The default is to optimize cycled indices.

• index_optimization_max_num_segments = 1 !!

  • Optimize the index down to <= index_optimization_max_num_segments. A higher number may take some load from Elasticsearch on heavily used systems with large indices, but it will decrease search performance. The default is 1.

• allow_leading_wildcard_searches = false

  • Do you want to allow searches with leading wildcards? This can be extremely resource hungry and should only be enabled with care.
  • See also: Searching

• allow_highlighting = false

  • Do you want to allow searches to be highlighted? Depending on the size of your messages this can be memory hungry and should only be enabled after making sure your Elasticsearch cluster has enough memory.

• elasticsearch_request_timeout = 1m

  • Global request timeout for Elasticsearch requests (e. g. during search, index creation, or index time-range calculations) based on a best-effort to restrict the runtime of Elasticsearch operations.
  • Default: 1m

• elasticsearch_index_optimization_timeout = 1h

  • Global timeout for index optimization (force merge) requests.
  • Default: 1h

• elasticsearch_index_optimization_jobs = 20

  • Maximum number of concurrently running index optimization (force merge) jobs.
  • If you are using lots of different index sets, you might want to increase that number.
  • Default: 20

• index_ranges_cleanup_interval = 1h

  • Time interval for index range information cleanups. This setting defines how often stale index range information is being purged from the database.
  • Default: 1h

• output_batch_size = 500

  • Batch size for the Elasticsearch output. This is the maximum (!) number of messages the Elasticsearch output module will get at once and write to Elasticsearch in a batch call. If the configured batch size has not been reached within output_flush_interval seconds, everything that is available will be flushed at once. Remember that every output buffer processor manages its own batch and performs its own batch write calls. (outputbuffer_processors variable)

• output_flush_interval = 1

  • Flush interval (in seconds) for the Elasticsearch output. This is the maximum amount of time between two batches of messages written to Elasticsearch. It is only effective at all if your minimum number of messages for this time period is less than output_batch_size * outputbuffer_processors.

• output_fault_count_threshold = 5

• output_fault_penalty_seconds = 30

  • As stream outputs are loaded only on demand, an output which is failing to initialize will be tried over and over again. To prevent this, the following configuration options define after how many faults an output will not be tried again for an also configurable amount of seconds.

• processbuffer_processors = 5

• outputbuffer_processors = 3

  • The number of parallel running processors.
  • Raise this number if your buffers are filling up.

  Note

  The following settings (outputbuffer_processor_*) configure the thread pools backing each output buffer processor. See ThreadPoolExecutor for technical details.

• outputbuffer_processor_keep_alive_time = 5000

  • When the number of threads is greater than the core (see outputbuffer_processor_threads_core_pool_size), this is the maximum time in milliseconds that excess idle threads will wait for new tasks before terminating.

• outputbuffer_processor_threads_core_pool_size = 3

  • The number of threads to keep in the pool, even if they are idle

• outputbuffer_processor_threads_max_pool_size = 30

  • The maximum number of threads to allow in the pool

• udp_recvbuffer_sizes = 1048576

  • UDP receive buffer size for all message inputs (e. g. SyslogUDPInput).

• processor_wait_strategy = blocking

  • Wait strategy describing how buffer processors wait on a cursor sequence. (default: sleeping)

  • Possible types:

    • yielding - Compromise between performance and CPU usage.
    • sleeping - Compromise between performance and CPU usage. Latency spikes can occur after quiet periods.
    • blocking - High throughput, low latency, higher CPU usage.
    • busy_spinning - Avoids syscalls which could introduce latency jitter. Best when threads can be bound to specific CPU cores.

• ring_size = 65536

  • Size of internal ring buffers. Raise this if raising outputbuffer_processors does not help anymore.
  • For optimum performance your LogMessage objects in the ring buffer should fit in your CPU L3 cache.
  • Must be a power of 2. (512, 1024, 2048, ...)

• inputbuffer_ring_size = 65536

• inputbuffer_processors = 2

• inputbuffer_wait_strategy = blocking

• message_journal_enabled = true

  • Enable the disk based message journal.

• message_journal_dir = data/journal

  • The directory which will be used to store the message journal. The directory must me exclusively used by Graylog and must not contain any other files than the ones created by Graylog itself.

  Attention

  If you create a seperate partition for the journal files and use a file system creating directories like ‘lost+found’ in the root directory, you need to create a sub directory for your journal. Otherwise Graylog will log an error message that the journal is corrupt and Graylog will not start.

• message_journal_max_age = 12h

• message_journal_max_size = 5gb

  • Journal hold messages before they could be written to Elasticsearch.
  • For a maximum of 12 hours or 5 GB whichever happens first.
  • During normal operation the journal will be smaller.

• message_journal_flush_age = 1m

  • This setting allows specifying a time interval at which we will force an fsync of data written to the log. For example if this was set to 1000 we would fsync after 1000 ms had passed.

• message_journal_flush_interval = 1000000

  • This setting allows specifying an interval at which we will force an fsync of data written to the log. For example if this was set to 1 we would fsync after every message; if it were 5 we would fsync after every five messages.

• message_journal_segment_age = 1h

  • This configuration controls the period of time after which Graylog will force the log to roll even if the segment file isn’t full to ensure that retention can delete or compact old data.

• message_journal_segment_size = 100mb

• async_eventbus_processors = 2

  • Number of threads used exclusively for dispatching internal events. Default is 2.

• lb_recognition_period_seconds = 3

  • How many seconds to wait between marking node as DEAD for possible load balancers and starting the actual shutdown process. Set to 0 if you have no status checking load balancers in front.

• lb_throttle_threshold_percentage = 95

  • Journal usage percentage that triggers requesting throttling for this server node from load balancers. The feature is disabled if not set.

• stream_processing_timeout = 2000

• stream_processing_max_faults = 3

  • Every message is matched against the configured streams and it can happen that a stream contains rules which take an unusual amount of time to run, for example if its using regular expressions that perform excessive backtracking.
  • This will impact the processing of the entire server. To keep such misbehaving stream rules from impacting other streams, Graylog limits the execution time for each stream.
  • The default values are noted below, the timeout is in milliseconds.
  • If the stream matching for one stream took longer than the timeout value, and this happened more than “max_faults” times that stream is disabled and a notification is shown in the web interface.

• alert_check_interval = 60

  • Length of the interval in seconds in which the alert conditions for all streams should be checked and alarms are being sent.

• output_module_timeout = 10000

  • Time in milliseconds to wait for all message outputs to finish writing a single message.

• stale_master_timeout = 2000

  • Time in milliseconds after which a detected stale master node is being rechecked on startup.

• shutdown_timeout = 30000

  • Time in milliseconds which Graylog is waiting for all threads to stop on shutdown.

MongoDB¶

• mongodb_uri = mongodb://...

  • MongoDB connection string. Enter your MongoDB connection and authentication information here.

  • See https://docs.mongodb.com/manual/reference/connection-string/ for details.

  • Examples:

    • Simple: mongodb://localhost/graylog
    • Authenticate against the MongoDB server: mongodb_uri = mongodb://grayloguser:secret@localhost:27017/graylog
    • Use a replica set instead of a single host: mongodb://grayloguser:secret@localhost:27017,localhost:27018,localhost:27019/graylog
    • DNS Seedlist is set as mongodb+srv://server.example.org/graylog.

• mongodb_max_connections = 1000

  • Increase this value according to the maximum connections your MongoDB server can handle from a single client if you encounter MongoDB connection problems.

• mongodb_threads_allowed_to_block_multiplier = 5

  • Number of threads allowed to be blocked by MongoDB connections multiplier. Default: 5
  • If mongodb_max_connections is 100, and mongodb_threads_allowed_to_block_multiplier is 5, then 500 threads can block. More than that and an exception will be thrown.
  • http://api.mongodb.com/java/current/com/mongodb/MongoOptions.html#threadsAllowedToBlockForConnectionMultiplier

Email¶

• transport_email_enabled = false

• transport_email_hostname = mail.example.com

• transport_email_port = 587

• transport_email_use_auth = true

• transport_email_use_tls = true

  • Enable SMTP with STARTTLS for encrypted connections.

• transport_email_use_ssl = true

  • Enable SMTP over SSL (SMTPS) for encrypted connections.

• transport_email_auth_username = you@example.com

• transport_email_auth_password = secret

• transport_email_subject_prefix = [graylog]

• transport_email_from_email = graylog@example.com

• transport_email_web_interface_url = https://graylog.example.com

  • Specify this to include links to the stream in your stream alert mails.
  • This should define the fully qualified base url to your web interface exactly the same way as it is accessed by your users.

HTTP¶

• http_connect_timeout = 5s

  • The default connect timeout for outgoing HTTP connections.
  • Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
  • Default: 5s

• http_read_timeout = 10s

  • The default read timeout for outgoing HTTP connections.
  • Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
  • Default: 10s

• http_write_timeout = 10s

  • The default write timeout for outgoing HTTP connections.
  • Values must be a positive duration (and between 1 and 2147483647 when converted to milliseconds).
  • Default: 10s

• http_proxy_uri =

  • HTTP proxy for outgoing HTTP connections

• http_non_proxy_hosts =

  • A list of hosts that should be reached directly, bypassing the configured proxy server.
  • This is a list of patterns separated by ”,”. The patterns may start or end with a “*” for wildcards.
  • Any host matching one of these patterns will be reached through a direct connection instead of through a proxy.

Others¶

• rules_file = /etc/graylog/server/rules.drl

  • Drools Rule File (Use to rewrite incoming log messages)
  • See: http://docs.graylog.org/en/2.4/pages/drools.html

• gc_warning_threshold = 1s

  • The threshold of the garbage collection runs. If GC runs take longer than this threshold, a system notification will be generated to warn the administrator about possible problems with the system. Default is 1 second.

• ldap_connection_timeout = 2000

  • Connection timeout for a configured LDAP server (e. g. ActiveDirectory) in milliseconds.

• disable_sigar = false

  • Disable the use of SIGAR for collecting system stats.

• dashboard_widget_default_cache_time = 10s

  • The default cache time for dashboard widgets. (Default: 10 seconds, minimum: 1 second)

• content_packs_loader_enabled = true

  • Automatically load content packs in “content_packs_dir” on the first start of Graylog.

• content_packs_dir = data/contentpacks

  • The directory which contains content packs which should be loaded on the first start of Graylog.

• content_packs_auto_load = grok-patterns.json

  • A comma-separated list of content packs (files in “content_packs_dir”) which should be applied on the first start of Graylog.
  • Default: empty

• proxied_requests_thread_pool_size = 32

  • For some cluster-related REST requests, the node must query all other nodes in the cluster. This is the maximum number of threads available for this. Increase it, if /cluster/* requests take long to complete.
  • Should be rest_thread_pool_size * average_cluster_size if you have a high number of concurrent users.

Example procedure for the Graylog virtual appliance¶

• Shutdown the virtual machine as preparation for creating a consistent snapshot.

• Take a snapshot of the virtual machine in case something goes wrong.

  • Understanding VM snapshots in ESXi / ESX
  • VMware vSphere: Managing Snapshots
  • VirtualBox: Snapshots
  • Parallels: Save Snapshots of a Virtual Machine
  • Parallels: Working with snapshots

• Attach an additional hard drive to the virtual machine.

  • VMware Workstation: Adding a New Virtual Disk to a Virtual Machine
  • VMware vSphere: Virtual Disk Configuration
  • VirtualBox: Virtual storage
  • Parallels: Hard Disk

• Start the virtual machine again.

• Stop all services to prevent disk access:

  $ sudo graylog-ctl stop
  

• Check for the logical name of the new hard drive. Usually this is /dev/sdb:

  $ sudo lshw -class disk
  

• Partition and format new disk:

  $ sudo parted -a optimal /dev/sdb mklabel gpt
  # A reboot may be necessary at this point so that the updated GPT is being recognized by the operating system
  $ sudo parted -a optimal -- /dev/sdb unit compact mkpart primary ext3 "1" "-1"
  $ sudo mkfs.ext4 /dev/sdb1
  

• Mount disk into temporary directory /mnt/tmp:

  $ sudo mkdir /mnt/tmp
  $ sudo mount /dev/sdb1 /mnt/tmp
  

• Copy current data to new disk:

  $ sudo cp -ax /var/opt/graylog/data/* /mnt/tmp/
  

• Compare both folders:

  # Output should be: Only in /mnt/tmp: lost+found
  $ sudo diff -qr --suppress-common-lines /var/opt/graylog/data /mnt/tmp
  

• Delete old data:

  $ sudo rm -rf /var/opt/graylog/data/*
  

• Mount new disk into /var/opt/graylog/data directory:

  $ sudo umount /mnt/tmp
  $ sudo mount /dev/sdb1 /var/opt/graylog/data
  

• Make change permanent by adding an entry to /etc/fstab:

  $ echo '/dev/sdb1 /var/opt/graylog/data ext4 defaults 0 0' | sudo tee -a /etc/fstab
  

• Reboot virtual machine:

  $ sudo shutdown -r now