High level¶

• Provide a platform for use by defenders to rapidly discover and respond to security incidents.
• Automate interfaces to other systems like bunker, banhammer, mig
• Provide metrics for security events and incidents
• Facilitate real-time collaboration amongst incident handlers
• Facilitate repeatable, predictable processes for incident handling
• Go beyond traditional SIEM systems in automating incident handling, information sharing, workflow, metrics and response automation

Technical¶

• Replace a SIEM
• Scalable, should be able to handle thousands of events/s, provide fast searching, alerting and correlations and handle interactions between teams of incident handlers.

MozDef aims to provide traditional SIEM functionality including:

• Accepts events/logs from your systems
• Stores the events/logs
• Facilitate searches
• Facilitate alerting
• Facilitate log management (archiving,restoration)

It is non-traditional in that it:

• Accepts only JSON input
• Provides you open access to your data
• Integrates with a variety of log shippers including heka, logstash, beaver, nxlog and any shipper that can send JSON to either rabbit-mq or an HTTP endpoint.
• Provides easy python plugins to manipulate your data in transit
• Provides realtime access to teams of incident responders to allow each other to see their work simultaneously

Event Management¶

From an event management point of view MozDef relies on Elastic Search for:

• event storage
• event archiving
• event indexing
• event searching

This means if you use MozDef for your log management you can use the features of Elastic Search to store millions of events, archive them to Amazon if needed, index the fields of your events, and search them using highly capable interfaces like Kibana.

Mozdef differs from other log management solutions that use Elastic Search in that it does not allow your log shippers direct contact with Elastic Search itself. In order to provide advanced functionality like event correlation, aggregation and machine learning, MozDef inserts itself as a shim between your log shippers (rsyslog, syslog-ng, beaver, nxlog, heka, logstash) and Elastic Search. This means your log shippers interact with MozDef directly and MozDef handles translating their events as they make they’re way to Elastic Search.

Event Pipeline¶

The logical flow of events is:

                   +–––––––––––+              +––––––––––––––+
                   |    MozDef +––––––––––––––+              |
+––––––––––+       |    FrontEnd              | Elastic      |
| shipper  +–––––––+–––––––––––+              | Search       |
++++++++++++                                  | cluster      |
++++++++++++                                  |              |
| shipper  +–––––––+–––––––––––+              |              |
+––––––––––+       |    MozDef ++–––––––––––––+              |
                   |    FrontEnd+             |              |
                   +–––––––––––+              |              |
                                              +––––––––––––––+

Choose a shipper (logstash, nxlog, beaver, heka, rsyslog, etc) that can send JSON over http(s). MozDef uses nginx to provide http(s) endpoints that accept JSON posted over http. Each front end contains a Rabbit-MQ message queue server that accepts the event and sends it for further processing.

You can have as many front ends, shippers and cluster members as you with in any geographic organization that makes sense for your topology. Each front end runs a series of python workers hosted by uwsgi that perform:

• event normalization (i.e. translating between shippers to a common taxonomy of event data types and fields)
• event enrichment
• simple regex-based alerting
• machine learning on the real-time event stream

Event Enrichment¶

To facilitate event correlation, MozDef allows you to write plugins to populate your event data with consistent meta-data customized for your environment. Through simple python plug-ins this allows you to accomplish a variety of event-related tasks like:

• tag all events involving key staff
• tag all events involving previous attackers or hits on a watchlist
• correct fields not properly handled by log shippers
• tap into your event stream for ancilary systems
• geoIP tag your events
• maintain ‘last-seen’ lists for assets, employees, attackers

Event Correlation/Alerting¶

Correlation/Alerting is currently handled as a series of queries run periodically against the Elastic Search engine. This allows MozDef to make full use of the lucene query engine to group events together into summary alerts and to correlate across any data source accessible to python.

Incident Handling¶

From an incident handling point of view MozDef offers the realtime responsiveness of Meteor in a web interface. This allows teams of incident responders the ability to see each others actions in realtime, no matter their physical location.

Dockerfile¶

After installing docker, use this to build a new image:

cd docker && sudo make build

Running the container:

sudo make run

You’re done! Now go to:

• http://127.0.0.1:3000 < meteor (main web interface)
• http://127.0.0.1:9090 < kibana
• http://127.0.0.1:9200 < elasticsearch
• http://127.0.0.1:9200/_plugin/marvel < marvel (monitoring for elasticsearch)
• http://127.0.0.1:8080 < loginput
• http://127.0.0.1:8081 < rest api

Known issues¶

• Marvel doesn’t display node info: ` Oops! FacetPhaseExecutionException[Facet [fs.total.available_in_bytes]: failed to find mapping for fs.total.available_in_bytes]`

Marvel uses techniques to gather system info that are not compatible with docker. See https://groups.google.com/forum/#!topic/elasticsearch/dhpxaOuoZWI

Despite this issue, marvel runs fine.

• I don’t see any data or dashboards in Kibana

We need to create some sample data, it’s in our roadmap ;)

ElasticSearch¶

Installation instructions are available on Elasticsearch website. You should prefer packages over archives if one is available for your distribution.

Marvel plugin¶

Marvel is a monitoring plugin developed by Elasticsearch (the company).

WARNING: this plugin is NOT open source. At the time of writing, Marvel is free for development but you have to get a license for production.

To install Marvel, on each of your elasticsearch node, from the Elasticsearch home directory:

bin/plugin -i elasticsearch/marvel/latest
sudo service elasticsearch restart

You should now be able to access to Marvel at http://any-server-in-cluster:9200/_plugin/marvel

Python¶

We need to install a python2.7 virtualenv:

sudo yum install make zlib-devel bzip2-devel openssl-devel ncurses-devel sqlite-devel readline-devel tk-devel pcre-devel gcc gcc-c++
cd
wget http://python.org/ftp/python/2.7.6/Python-2.7.6.tgz
tar xvzf http://python.org/ftp/python/2.7.6/Python-2.7.6.tgz
./configure --prefix=/home/mozdef/python2.7 --enable-shared
make
make install

wget https://raw.github.com/pypa/pip/master/contrib/get-pip.py
export LD_LIBRARY_PATH=/home/netantho/python2.7/lib/
./python2.7/bin/python get-pip.py
./python2.7/bin/pip install virtualenv
mkdir ~/envs
cd ~/envs
~/python2.7/bin/virtualenv mozdef
source mozdef/bin/activate
pip install -r MozDef/requirements.txt

At this point when you launch python, It should tell you that you’re using Python 2.7.6.

Whenever you launch a python script from now on, you should have your mozdef virtualenv actived and your LD_LIBRARY_PATH env variable should include /home/mozdef/python2.7/lib/

RabbitMQ¶

RabbitMQ is used on workers to have queues of events waiting to be inserted into the Elasticsearch cluster (storage).

To install it, first make sure you enabled EPEL repos. Then you need to install an Erlang environment:

yum install erlang

You can then install the rabbitmq server:

rpm --import http://www.rabbitmq.com/rabbitmq-signing-key-public.asc
yum install rabbitmq-server-3.2.4-1.noarch.rpm

To start rabbitmq at startup:

chkconfig rabbitmq-server on

Meteor¶

Meteor is a javascript framework used for the realtime aspect of the web interface.

We first need to install Mongodb since it’s the DB used by Meteor. In /etc/yum.repo.d/mongo, add:

[mongodb]
name=MongoDB Repository
baseurl=http://downloads-distro.mongodb.org/repo/redhat/os/x86_64/
gpgcheck=0
enabled=1

Then you can install mongodb:

sudo yum install mongodb

For meteor, in a terminal:

curl https://install.meteor.com/ | sh

wget http://nodejs.org/dist/v0.10.26/node-v0.10.26.tar.gz
tar xvzf node-v0.10.26.tar.gz
cd node-v0.10.26
./configure
make
make install

Make sure you have meteorite/mrt:

npm install -g meteorite

Then from the meteor subdirectory of this git repository run:

mrt add iron-router
mrt add accounts-persona

You may want to edit the app/lib/settings.js file to properly point to your elastic search server:

elasticsearch={
  address:"http://servername:9200/",
  healthurl:"_cluster/health",
  docstatsurl:"_stats/docs"
}

Then start meteor with:

meteor

Nginx¶

We use nginx webserver.

You need to install nginx:

sudo yum install nginx

If you don’t have this package in your repos, before installing create /etc/yum.repos.d/nginx.repo with the following content:

[nginx]
name=nginx repo
baseurl=http://nginx.org/packages/centos/6/$basearch/
gpgcheck=0
enabled=1

UWSGI¶

We use uwsgi to interface python and nginx:

wget http://projects.unbit.it/downloads/uwsgi-2.0.2.tar.gz
~/python2.7/bin/python uwsgiconfig.py --build
~/python2.7/bin/python uwsgiconfig.py  --plugin plugins/python core
cp python_plugin.so ~/envs/mozdef/bin/
cp uwsgi ~/envs/mozdef/bin/

cd rest
# modify uwsgi.ini
vim uwsgi.ini
uwsgi --ini uwsgi.ini

cd ../loginput
# modify uwsgi.ini
vim uwsgi.ini
uwsgi --ini uwsgi.ini

sudo cp nginx.conf /etc/nginx
# modify /etc/nginx/nginx.conf
sudo vim /etc/nginx/nginx.conf
sudo service nginx restart

Kibana¶

Kibana is a webapp to visualize and search your Elasticsearch cluster data:

wget https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0milestone5.tar.gz
tar xvzf kibana-3.0.0milestone5.tar.gz
mv kibana-3.0.0milestone5 kibana
# configure /etc/nginx/nginx.conf to target this folder
sudo service nginx reload

Import dashboards from MozDef/kibana/dashboards into the kibana webUI

Events visualizations¶

Since the backend of MozDef is Elastic Search, you get all the goodness of Kibana with little configuration. The MozDef UI is focused on incident handling and adding security-specific visualizations of SIEM data to help you weed through the noise.

Alerts¶

Alerts are generally implemented as Elastic Search searches, or realtime examination of the incoming message queues. MozDef provides a plugin interface to allow open access to event data for enrichment, hooks into other systems, etc.

Incident handling¶

What should I log?¶

If your program doesn’t log anything it doesn’t exist. If it logs everything that happens it becomes like the proverbial boy who cried wolf. There is a fine line between logging too little and too much but here is some guidance on key events that should be logged and in what detail.

Background¶

Mozilla used CEF as a logging standard for compatibility with Arcsight and for standardization across systems. While CEF is an admirable standard, MozDef prefers JSON logging for the following reasons:

• Every development language can create a JSON structure
• JSON is easily parsed by computers/programs which are the primary consumer of logs
• CEF is primarily used by Arcsight and rarely seen outside that platform and doesn’t offer the extensibility of JSON
• A wide variety of log shippers (heka, logstash, fluentd, nxlog, beaver) are readily available to meet almost any need to transport logs as JSON.
• JSON is already the standard for cloud platforms like amazon’s cloudtrail logging

Description¶

As there is no common RFC-style standard for json logs, we prefer the following structure adapted from a combination of the graylog GELF and logstash specifications.

Note all fields are lowercase to avoid one program sending sourceIP, another sending sourceIp, another sending SourceIPAddress, etc. Since the backend for MozDef is elasticsearch and fields are case-sensitive this will allow for easy compatibility and reduce potential confusion for those attempting to use the data. MozDef will perform some translation of fields to a common schema but this is intended to allow the use of heka, nxlog, beaver and retain compatible logs.

Mandatory Fields¶

Details substructure (optional fields)¶

Examples¶

{
    "timestamp": "2014-02-14T11:48:19.035762739-05:00",
    "hostname": "fedbox",
    "processname": "/tmp/go-build278925522/command-line-arguments/_obj/exe/log_json",
    "processid": 3380,
    "severity": "INFO",
    "summary": "joe login failed",
    "category": "authentication",
    "source": "",
    "tags": [
        "MySystem",
        "Authentication"
    ],
    "details": {
        "user": "joe",
        "task": "access to admin page /admin_secret_radioactiv",
        "result": "10 authentication failures in a row"
    }
}

insert_simple.js¶

insert_simple.js sends indexing requests with 1 log/request.

Usage: node ./insert_simple.js <processes> <totalInserts> <host1> [host2] [host3] [...]

• processes: Number of processes to spawn
• totalInserts: Number of inserts to perform, please note after a certain number node will slow down. You want to have a lower number if you are in this case.
• host1, host2, host3, etc: Elasticsearch hosts to which you want to send the HTTP requests

insert_bulk.js¶

insert_bulk.js sends bulk indexing requests (several logs/request).

Usage: node ./insert_bulk.js <processes> <insertsPerQuery> <totalInserts> <host1> [host2] [host3] [...]

• processes: Number of processes to spawn
• insertsPerQuery: Number of logs per request
• totalInserts: Number of inserts to perform, please note after a certain number node will slow down. You want to have a lower number if you are in this case.
• host1, host2, host3, etc: Elasticsearch hosts to which you want to send the HTTP requests

search_all_fulltext.js¶

search_all_fulltext.js performs search on all indices, all fields in fulltext. It’s very stupid.

Usage: node ./search_all_fulltext.js <processes> <totalSearches> <host1> [host2] [host3] [...]

• processes: Number of processes to spawn
• totalSearches: Number of search requests to perform, please note after a certain number node will slow down. You want to have a lower number if you are in this case.
• host1, host2, host3, etc: Elasticsearch hosts to which you want to send the HTTP requests