DIMS Test Plan v 2.9.1¶

Identification¶

This Software System Test Plan (release 2.9.1) documents the objectives, approach, and scope/limitations of the testing and evaluation of the Distributed Incident Management System (DIMS). It describes the logistics for executing the plan, including test environment needs, high level schedule, and resource mapping. It also identifies technical and schedule risks and mitigation plans. This plan will be utilized by the project team to properly scope, manage, and comprehend test efforts. This test plan is specific to software, as defined within the statement of work.

This Test Plan describes the Formal Qualification Test (FQT) activities and the environment in which those activities will take place for the DIMS deployed system. DIMS is composed of the following Computer Software Configuration Items (CSCIs):

CSCI capability requirements¶

DIMS is funded by the Department of Homeland Security under contract HSHQDC- 13-C-B0013.

The DIMS system is divided into the following high-level CSCI sets, per the acquisition contract.

Reference DIMS System Requirements v 2.9.0 establishes the DIMS software requirements. Reference DIMS Operational Concept Description v 2.9.0 describes the operational concept (and open source design concept) for DIMS. Reference DIMS Architecture Design v 2.9.0 sets forward the architectural design. Unit testing and integration testing will be performed during development at the University of Washington. This document primarily addresses the FQT activities that take place when deploying the DIMS software.

System overview¶

The primary mission objectives for the DIMS system are operational in nature, focused on facilitating the exchange of operational intelligence and applying this intelligence to more efficiently respond and recover from cyber compromise. The secondary mission objectives are to create a framework in which tools to support the primary mission objectives can more quickly and easily be integrated and brought to bear against advancing techniques on the attacker side of the equation.

The DIMS project is intended to take this semi-automated sharing of structured threat information, building on the success of the Public Regional Information Security Event Monitoring (PRISEM) project and leveraging an existing community of operational security professionals known as Ops- Trust, and scale it to the next level. The intent of this software project is to allow for near real-time sharing of critical alerts and structured threat information that will allow each contributing party to receive information, alerts and data, analyze the data, and respond appropriately and in a timely manner through one user-friendly web application, DIMS.

Working with the use cases defined by MITRE and PRISEM users, building the features necessary to simplify structured information sharing, and operationalizing these within these existing communities, will allow DIMS to fill existing gaps in capabilities and support existing missions that are slowed down today by many complicated, manual processes.

The changes to existing systems consists of seamless integration of the three current systems into a single web application that enables each system to contribute to the data warehouse of information concerning threats, alerts, attacks and suspect or compromised user terminals within the infrastructure. Additionally, the integrated systems will be able to share and retrieve data, visually observe alerts through color coded visual indicators, while retaining the existing functionality of the current system.

Software System Test Strategy (General)¶

The software system test strategy we will employ is compromises two high-level sets of tests.

The first set of test will be developed as the system is being designed using the Agile coding methodology. Development and pre-release testing will be performed during “sprints” (cycles of development anticipated to be on 2 week time frames). These tests are centered on development and take place within the project team. The project team will extract user stories from the use cases and in turn create epics derived from the user stories. Tests will be written for the user stories and determine the necessary development. This is essentially a test-driven approach that pushes development from one sprint to the next.

The other set of tests will be performed with interaction from the stakeholders and will entail an environment in which the stakeholders will, at determinate intervals, access the system and execute one or more user stories. This environment, set up specifically for stakeholders, will enable them to interact with the system at their discretion and direct their feedback to the project team about the functionality delivered within their environment.

Delivery of software for testing will be conducted in the following manner:

• Software for the project will be developed using the Agile Methodology and will therefore be delivered incrementally, based on completed functionality for each Sprint.
• Each Sprint cycle is anticipated to be two to four (2-4) weeks.
• A final feature complete build will be delivered at the completion of development.
• Testing efforts will be structured to test the delivered functionality of each Sprint’s delivered software module(s).
• Feature complete builds will be tested as integrated modules.
• Iterations of testing will be conducted as builds are delivered to address bug fixes as well as delivered features or functionality.

Document overview¶

The purpose of this document is to establish the requirements for the testing of the Distributed Incident Management System (DIMS). The structure of this document and the strategy for testing has been adapted principally from MIL-STD-498 (see Section Referenced documents). It contains the following information:

• Section Referenced documents lists related documents.
• Section Software test environment specifies the test environment that will be used in testing DIMS CSCIs. It includes a description of the hardware, software and personnel resources needed for installation, testing and control.
• Section Test identification provides general information about test levels and test classes, general test conditions, and planned tests.
• Section Requirements traceability describes traceability of tests back to requirements.
• Section Notes provides an alphabetical listing of acronyms and abbreviations used in this document.

UW Tower server room¶

DIMS deployments are currently hosted in the University of Washington Tower (“UW Tower”) data center. Plans are in place to migrate hardware to another data center at Semaphore Corporation in Seattle.

Each deployed system will be separated from the others and operate independently (in terms of services and logical network topologies). Even if they are in the same rack and possibly sharing some fundamental resources (e.g., Network Attached Storage (NAS), or switched VLANs, they will each be deployed, configured, and will operate in such a manner as to be fully taken down without impacting the operation of the other environments.

This will allow development, test and evaluation, and user acceptance test

Software items¶

Hardware and firmware items¶

Proprietary nature, acquirer’s rights, and licensing¶

Some tests defined and anticipated under this plan involve use of a licensed Jira ticketing system using the Zephyr for Jira plug-in. These are primarily development-related tests that fall under the levels Unit, Integration, and/or Component Interface test levels, of type Expected Value and/or Desk check as defined in Section Test levels and Test classes, respectively.

The remainder of the tests will use open source products either acquired from their original source, or produced by the DIMS team and delivered with the final system. (See Section License for the license under which DIMS software is being released.)

Installation, testing, and control¶

Deployment tests will start with a bare-metal server with a network connection capable of routing to the internet. From there, the following general high-level steps will be performed:

1. Operating system installation to the bare-metal server will be performed according to steps outlined in documentation. (This may be done using DHCP dynamically assigned addresses so as to minimize the number of manual steps required to install the base operating system.)
2. Network level configuration of the operating system will be performed by manually entering the required attributes (e.g., the way Security Onion Setup Phase 1 is performed) into a software configuration database and/or configuration file.
3. The software configuration database and/or configuration file will be applied to the system, configuring all DIMS components for initial use.
4. Further manual steps will be necessary to provision initial user accounts in the portal and/or other DIMS system administration components.
5. The system will be put into a “test” mode to perform system tests to validate that all DIMS components are up and running and the system is functional. Initial data input tests may be performed at this point to validate that input and output functions are working.

Participating organizations¶

Table Participants Roles lists participants, their roles and responsibilities, related to testing.

Orientation plan¶

People involved with testing will be provided with guidance in the form of user documentation, a copy of this test plan, and any specifics of how to perform the specified tests. This may include providing them with access to software described in Software items, or some other form of checklist that will enable them to know the tests to be performed, acceptance criteria, and a means of reporting test results.

Tests to be performed¶

Specific tests of test classes expected value testing and desk check testing (see Section Test classes) that are manual in nature, and are expected to be performed by stakeholders for the purpose of acceptance testing, will be documented and provided as part of orientation prior to testing.

General information¶

Tests described in this section trace back to the DIMS System Requirements v 2.9.0 document, Section Requirements, as described in Section Requirements traceability.

General test conditions¶

Planned tests¶

Glossary of Terms¶

Agile

A programming methodology based on short cycles of feature-specific changes and rapid delivery, as opposed to the “Waterfall” model of system development with long requirements definition, specification, design, build, test, acceptance, delivery sequences of steps.

Botnets System

The name given to the re-implementation of Einstein 1 technology. See http://web.archive.org/web/20131115180654/http://www.botnets.org/

cron

A Unix/Linux service daemon that is responsible for running background tasks on a scheduled basis.

Git

A source code version management system in widespread use.

CIFglue

“Simple rails app to quickly add indicators to the Collective Intelligence Framework”

Cryptographic Hash
Cryptographic Hashing Algorithm

A mathematical method of uniquely representing a stream of bits with a fixed-length numeric value in a numeric space sufficiently large so as to be infeasible to predictably generate the same hash value for two different files. (Used as an integrity checking mechanism). Commonly used algorithms are MD5, SHA1, SHA224, SHA256, RIPEMD-128. (See also http://en.wikipedia.org/wiki/Cryptographic_hash_function).

Einstein 1

A network flow based behavioral and watchlist based detection system developed by University of Michigan and Merit Networks, Inc. for use by US-CERT. The re-implementation is known as the Botnets System.

Fusion Center

Entities created by DHS to integrate federal law enforcement and intelligence resources with state and local law enforcement for greater collaboration and information sharing across levels of SLTT governments.

GZIP

Gnu ZIP (file compression program)

MUTEX

Mutual Exclusion (object or lock, used to synchronize execution of independent threads or processes that must share a common resource in an exclusive manner, or to ensure only one copy of a program is running at a time)

NetFlow

Record format developed by Cisco for logging and storing Network Flow information (see also SiLKTools).

NoSQL

The term for database that does not use the typical table-based relational schema as Relational Database Management Systems (RDBMS)

Ops-Trust (ops-t)

Operational Security Trust organization (see http://ops-trust.net/)

Redis

A “NoSQL” database system used to store files in a key/value pair model via a RESTful HTTP/HTTPS interface.

SiLKTools

A network flow logging and archiving format and tool set developed by Carnegie Mellon’s Software Engineering Institute (in support of CERT/CC).

Team Cymru

(Pronounced “COME-ree”) – “Team Cymru Research NFP is a specialized Internet security research firm and 501(c)3 non-profit dedicated to making the Internet more secure. Team Cymru helps organizations identify and eradicate problems in their networks, providing insight that improves lives.”

Tupelo

A host-based forensic system (client and server) developed at the University of Washington, based on the Honeynet Project “Manuka” system.

List of Acronyms¶

AAA

Authentication, Authorization, and Accounting

AMQP

Advanced Message Queuing Protocol

AS

Autonomous System

ASN

Autonomous System Number

CI

Critical Infrastructure

CIDR

Classless Internet Domain Routing

CIF

Collective Intelligence Framework

CIP

Critical Infrastructure Protection

CISO

Chief Information and Security Officer

COA

Course of Action (steps to Respond and Recover)

CONOPS

Concept of Operations

CRADA

Cooperative Research and Development Agreement

CSC

Computer Software Component

CSCI

Computer Software Configuration Item

CSIRT

Computer Security Incident Response Team

CSV

Comma-separated Value (a semi-structured file format)

DIMS

Distributed Incident Management System

DNS

Domain Name System

DoS

Denial of Service

DDoS

Distributed Denial of Service

EO

Executive Order

FQT

Formal Qualification Test/Tests/Testing

HSPD

Homeland Security Presidential Directive

ICT

Information and Communication Technology

IOC

Indicators of Compromise

IP

Internet Protocol (TCP and UDP are examples of Internet Protocols)

IRC

Internet Relay Chat (an instant messaging system)

JSON

JavaScript Object Notation

MAPP

Microsoft Active Protections Program

MNS

Mission Needs Statement

NCFTA

National Cyber-Forensics & Training Alliance

NTP

Network Time Protocol (a service exploited to perform reflected/amplified DDoS attacks by spoofing the source address of requests, where the much larger responses flood the victim)

OODA

Observe, Orient, Decide, and Act (also known as the “Boyd Cycle”)

PPD

Presidential Policy Directive

PRISEM

Public Regional Information Security Event Management

RBAC

Role Based Access Control

RESTful

Representational State Transfer web service API

RPC

Remote Procedure Call

SCADA

Supervisory Control and Data Acquisition

SIEM

Security Information Event Management (sometimes referred to as Security Event Information Management, Security Event Monitoring, causing some to pronounce it as “sim-sem”.)

SLTT

State, Local, Territorial, and Tribal (classification of non-federal government entities)

SOC

Security Operations Center

SoD

Security on Demand (PRISEM project support vendor)

SSH

Secure Shell

STIX

Structure Threat Information Expression. A standard for information exchange developed by MITRE in support of DHS US-CERT.

TAXII

Trusted Automated Exchange of Indicator Information

TCP

Transmission Control Protocol (one of the Internet Protocols)

TLP

Traffic Light Protocol

TTP

Tools, Tactics, and Procedures

UC

Use Case

UDP

Unreliable Datagram Protocol (one of the Internet Protocols)

WCX

Western Cyber Exchange

Test Dashboard server VM¶

Tests performed using the Dashboard user interface, as well as some of the automated tests, require a Vagrant test server VM running on the tester’s workstation.

Whenever a test has a prerequisite of dashboard test server, it is referring to this VM, up and running on the tester’s workstation.

cd $GIT/dims-vagrant/ubuntu-14.04.2-amd64/dimstestserver
./run_playbook -g dashboard-test-servers dashboard-test-data-reset.yml -vv

Creating Tupelo server VM¶

Tupelo tests require a Tupelo server VM.

This server is created in a similar fashion to the dashboard test server:

1. Make sure you have the box file installed as shown above in step one of creating a dashboard test server.

2. Set up Vagrant directory for the VM, name it tupeloserver and give it an IP of 192.168.56.102.

   cd $GIT/dims-vagrant/ubuntu-14.04.2-amd64/
   make server NAME=tupeloserver
   cd tupeloserver
   ../nic2 192.168.56.102
   vagrant up
   

3. Run the Ansible playbook tupelo-test-server-provision.yml as shown below.

   ./run_playbook -g tupelo-servers tupelo-server-install.yml -vv
   

Installing Tupelo client¶

The tester needs the Tupelo client installed on their host machine to perform many of the Tupelo tests. The tester installs the tupelo client on his/her developer workstation via Ansible:

RUNHOST=localhost RUNGROUP=tupelo-clients ansible-playbook -i $GIT/ansible-playbooks/dyn_inv.py $GIT/ansible-playbooks/tupelo-shell-deploy.yml -e artifact_branch=develop --ask-sudo-pass -vv

Planning tests with JIRA¶

This section describes how to plan a test cycle and write tests using JIRA.

Test cycle¶

We use a test cycle to plan and execute our tests. To view test cycles, click Tests > Plan Test Cycle:

The list of cycles displays. We need a new cycle for the tests due on 11/15, so we’ll create one. Click the Create New Cycle button to bring up a dialog to create the cycle. Give it a name and description.

The new test cycle displays in the list. You can see that it doesn’t have any tests yet.

When you create a test in JIRA, you will add it to this test cycle.

Creating tests¶

To create a new test, select Tests > Create a Test.

The Create Issue screen displays, with the issue type already set to Test. Enter a summary for the test, and fill in choices in the testLevel, testClass, and qualificationMethod pick boxes. These are described in this Test Plan. Choose one item per pick box. You should also add the reference to the DIMS SR - these are referenced in Section 4 of this plan for each group of planned tests. typeOfData describes where the output data will be when you are done. You can add this later if you don’t know it at this time.

The following figure shows the first part of the Create Issue dialog being filled in:

Scrolling down, you describe the Environment and provide a Description of the test. The environment entry should be short. If the test needs a local Vagrant VM to run, then the Test should reference how that is created in the prerequisites.

We enter prerequisites in the first test step. When you initially create the test, you can just add a short description and add prerequisites by editing the test.

Save the test. You can further fill out fields by editing the test. For example, you can upload files needed to run the test. In this example, we are uploading a file with test data and a script which will run a number of automated tests using the test data file as input:

If files aren’t attached, the prerequisites should state where to get them.

Add the test to the desired test cycle. Select More Actions > Add to Test Cycle(s):

Select the test cycle. In this example, we choose the Sample Test Cycle. You would choose the 2015-100-15_test_report test cycle for actual tests.

The test will now show up in the list of tests for that test cycle. The E button on the right is the button to click when you are going to execute the test.

To create more tests, you can do so from scratch, or you can clone an existing test. Go to the existing test, and click Clone.

Enter a new summary for the new test. You can clone attachments if the same ones are used for the new test.

Here is an updated Sample test 2. Prerequisite info has been added to the description. The comment regarding “if test fails” isn’t needed - that was put in before we had the typeOfOutput field (will update this screenshot later);

Since this test is automated, we just have one step - to run the test script. The Expected Result is given as how many tests should pass.

Generating a Test Report with the report generation utility¶

A Test Report is produced using a reporting utility that generates a Sphinx document, based on source from the $GIT/dims-tr Git repository directory. It processes metadata descriptions of tests and their results, producing a Sphinx document.

Test cycles are named with a date, e.g., 2015-11-15. A simple directory structure is used that combines test results from both test managed within Jira using Zephyr for Jira, as well as non-Jira test results. Both of these sources are rooted at $GIT/dims-tr/test_cycles along with a copy of the Sphinx document skeleton found in the $GIT/dims-tr/docs directory. This separates the report and data from which the report was generated for each test cycle into its own directory tree. For example,

[dimscli] dittrich@27b:~/dims/git/dims-tr/test_cycles
(feature/dims-529*) $ tree
.
├── 2015-11-15
│   ├── docs
│   │   ├── Makefile
│   │   ├── build
│   │   └── source
│   ├── jira_data
│   │   ├── DIMS-553.json
│   │   ├── DIMS-553.pdf
│   │   ├── DIMS-554.json
│   │   ├── DIMS-554.pdf
│   │   ├── DIMS-565.json
│   │   ├── DIMS-565.pdf
│   │   ├── DIMS-566.json
│   │   ├── DIMS-569.json
│   │   ├── DIMS-570.json
│   │   ├── DIMS-570.pdf
│   │   ├── DIMS-571.json
│   │   ├── DIMS-571.pdf
│   │   ├── DIMS-574.json
│   │   └── DIMS-574.pdf
│   ├── jira_data_summary.json
│   └── nonjira_data
│       ├── test1.json
│       ├── test1.pdf
│       ├── test2.json
│       └── test3.json
└── 2016_00_00
    ├── docs
    │   ├── Makefile
    │   ├── build
    │   └── source
    ├── jira_data
    │   └── blahblah
    ├── jira_data_summary.json
    └── nonjira_data
        └── blahblah

20 directories, 91 files

A file $GIT/dims-tr/CURRENT_CYCLE contains the test cycle identifier for the current test cycle (and can be over-ridden with a command line option in the test utility.)

It can be used with inline command substitution in the BASH shell like this:

[dimscli] dittrich@27b:~/dims/git/dims-tr (feature/dims-529*) $ tree -L
1 test_cycles/$(cat CURRENT_CYCLE)
test_cycles/2015-11-15
├── docs
├── jira_data
├── jira_data_summary.json
└── nonjira_data

3 directories, 1 file


[dimscli] dittrich@27b:~/dims/git/dims-tr (feature/dims-529*) $ tree -L
1 test_cycles/$(cat CURRENT_CYCLE)/jira_data
test_cycles/2015-11-15/jira_data
├── DIMS-553.json
├── DIMS-553.pdf
├── DIMS-554.json
├── DIMS-554.pdf
├── DIMS-565.json
├── DIMS-565.pdf
├── DIMS-566.json
├── DIMS-569.json
├── DIMS-570.json
├── DIMS-570.pdf
├── DIMS-571.json
├── DIMS-571.pdf
├── DIMS-574.json
└── DIMS-574.pdf

0 directories, 14 files

There is a helper Makefile at the root of the repo to make it easier to generate a report.

dimscli] dittrich@27b:~/dims/git/dims-tr (feature/dims-529*) $ make help
/Users/dittrich/dims/git/dims-tr
[Using Makefile.dims.global v1.6.124 rev ]
---------------------------------------------------------------------------
Usage: make [something]

Where "something" is one of the targets listed in the sections below.

 ---------------------------------------------------------------------------
                 Targets from Makefile.dims.global

 help - Show this help information (usually the default rule)
 dimsdefaults - show default variables included from Makefile.dims.global
 version - show the Git revision for this repo
 envcheck - perform checks of requirements for DIMS development
 ---------------------------------------------------------------------------
                   Targets from Makefile

 all - defaults to 'report'
 showcurrent - show the current test cycle
 enter - enter a test description
 report - generate a 'results.rst' file in ../docs/source/
 autobuild - run dims.sphinxautobuild for this test cycle
 install - install Python script and pre-requisites
 clean - remove build files and generated .rst files.
 spotless - clean, then also get rid of dist/ directory
---------------------------------------------------------------------------

[dimscli] dittrich@27b:~/dims/git/dims-tr (feature/dims-529*) $ make
showcurrent
Current test cycle is 2015-11-15


[dimscli] dittrich@27b:~/dims/git/dims-tr (feature/dims-529*) $ make report
python scripts/get_test.py

[dimsenv] dittrich@27b:~/dims/git/dims-tr (feature/dims-529*) $ make
autobuild
tar -cf - docs | (cd "test_cycles/2015-11-15" && tar -xf -)
rm -rf build/*
[I 151119 21:35:18 server:271] Serving on http://127.0.0.1:48196
[I 151119 21:35:18 handlers:58] Start watching changes
[I 151119 21:35:18 handlers:60] Start detecting changes

+--------- source/test3.rst changed
---------------------------------------------
/Users/dittrich/dims/git/dims-tr/test_cycles/2015-11-15/docs/source/test3.rst::
WARNING: document isn't included in any toctree
+--------------------------------------------------------------------------------


+--------- source/index.rst changed
---------------------------------------------
+--------------------------------------------------------------------------------


+--------- source/results.rst changed
-------------------------------------------
+--------------------------------------------------------------------------------

[I 151119 21:35:24 handlers:131] Browser Connected: http://127.0.0.1:48196/

Using bats for Producing and Executing Tests¶

The DIMS project has adopted use of the Bats: Bash Automated Testing System (known as bats) to perform simple tests in a manner that produces parsable output following the Test Anything Protocol (TAP).

Bats is a TAP Producer, whose output can be processed by one of many TAP Consumers, including the Python program tap.py.

$ bats -h
Bats 0.4.0
Usage: bats [-c] [-p | -t] <test> [<test> ...]

  <test> is the path to a Bats test file, or the path to a directory
  containing Bats test files.

  -c, --count    Count the number of test cases without running any tests
  -h, --help     Display this help message
  -p, --pretty   Show results in pretty format (default for terminals)
  -t, --tap      Show results in TAP format
  -v, --version  Display the version number

  For more information, see https://github.com/sstephenson/bats

#!/usr/bin/env bats

@test "a" {
    [[ true ]]
}

$ tree tests
tests
├── a
│   └── a.bats
└── b
    ├── b.bats
    └── c
        └── c.bats

3 directories, 3 files

$ bats tests

0 tests, 0 failures

$ bats tests/a
 ✓ a

1 test, 0 failures

$ bats tests/b
 ✓ b

1 test, 0 failures

$ bats tests/b/c
 ✓ c

1 test, 0 failures

$ bats tests/a /tests/b tests/b/c
bats: /tmp/b does not exist
/usr/local/Cellar/bats/0.4.0/libexec/bats-exec-suite: line 20: let: count+=: syntax error: operand expected (error token is "+=")

$ bats tests/a/*.bats tests/b/*.bats tests/b/c/*.bats
 ✓ a
 ✓ b
 ✓ c

3 tests, 0 failures

$ find tests -name '*.bats' | xargs bats
1..3
ok 1 a
ok 2 b
ok 3 c

#!/usr/bin/env bats
#
# Ansible managed: /home/dittrich/dims/git/ansible-playbooks/v2/roles/pycharm/templates/../templates/tests/./system/pycharm.bats.j2 modified on 2016-09-15 20:14:38 by dittrich on dimsdemo1 [ansible-playbooks v1.3.33]
#
# vim: set ts=4 sw=4 tw=0 et :

load helpers

@test "[S][EV] Pycharm is not an installed apt package." {
    ! is_installed_package pycharm
}

@test "[S][EV] Pycharm Community edition is installed in /opt" {
    results=$(ls -d /opt/pycharm-community-* | wc -l)
    echo $results >&2
    [ $results -ne 0 ]
}

@test "[S][EV] \"pycharm\" is /opt/dims/bin/pycharm" {
    assert "pycharm is /opt/dims/bin/pycharm" type pycharm
}

@test "[S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm" {
    [ -L /opt/dims/bin/pycharm ]
}

@test "[S][EV] Pycharm Community installed version number is 2016.2.3" {
    assert "2016.2.3" bash -c "file $(which pycharm) | sed 's|\(.*/pycharm-community-\)\([^/]*\)\(/.*$\)|\2|'"
}

$ test.runner --level system --match pycharm
[+] Running test system/pycharm
 ✓ [S][EV] Pycharm is not an installed apt package.
 ✓ [S][EV] Pycharm Community edition is installed in /opt
 ✓ [S][EV] "pycharm" is /opt/dims/bin/pycharm
 ✓ [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
 ✓ [S][EV] Pycharm Community installed version number is 2016.2.3

5 tests, 0 failures

$ tree /opt/dims/tests.d
/opt/dims/tests.d

0 directories, 0 files

$ tree base/templates/tests
base/templates/tests
├── component
├── helpers.bash.j2
├── integration
├── README.txt
├── system
│   ├── deprecated.bats.j2
│   ├── dims-accounts.bats.j2
│   ├── dims-accounts-sudo.bats.j2
│   ├── dims-base.bats.j2
│   ├── dns.bats.j2
│   ├── proxy.bats.j2
│   ├── sudo
│   │   └── sudo-iptables.bats.j2
│   └── user
│       └── vpn.bats.j2
└── unit
    └── dims-filters.bats.j2

6 directories, 11 files

 $ tree /opt/dims/tests.d/
 /opt/dims/tests.d/
 ├── component
 │   └── helpers.bash -> /opt/dims/tests.d/helpers.bash
 ├── helpers.bash
 ├── integration
 │   └── helpers.bash -> /opt/dims/tests.d/helpers.bash
 ├── system
 │   ├── deprecated.bats
 │   ├── dims-accounts.bats
 │   ├── dims-accounts-sudo.bats
 │   ├── dims-base.bats
 │   ├── dims-ci-utils.bats
 │   ├── dns.bats
 │   ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
 │   ├── iptables-sudo.bats
 │   ├── proxy.bats
 │   └── user
 │       ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
 │       └── vpn.bats
 └── unit
     ├── bats-helpers.bats
     ├── dims-filters.bats
     ├── dims-functions.bats
     └── helpers.bash -> /opt/dims/tests.d/helpers.bash

 5 directories, 18 files

/docker/templates/tests
└── system
    ├── docker-consul.bats.j2
    ├── docker-core.bats.j2
    └── docker-network.bats.j2

1 directories, 3 files

$ dims.ansible-playbook --role docker

PLAY [Ansible (2.x / v2) Base Playbook] ****************************************

TASK [docker : include] ********************************************************
included: /home/dittrich/dims/git/ansible-playbooks/v2/tasks/pre_tasks.yml for dimsdemo1.devops.develop

 . . .

PLAY RECAP *********************************************************************
dimsdemo1.devops.develop   : ok=34   changed=20   unreachable=0    failed=0

 $ tree /opt/dims/tests.d
 /opt/dims/tests.d
 ├── component
 │   └── helpers.bash -> /opt/dims/tests.d/helpers.bash
 ├── helpers.bash
 ├── integration
 │   └── helpers.bash -> /opt/dims/tests.d/helpers.bash
 ├── system
 │   ├── deprecated.bats
 │   ├── dims-accounts.bats
 │   ├── dims-accounts-sudo.bats
 │   ├── dims-base.bats
 │   ├── dims-ci-utils.bats
 │   ├── dns.bats
 │   ├── docker-consul.bats
 │   ├── docker-core.bats
 │   ├── docker-network.bats
 │   ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
 │   ├── iptables-sudo.bats
 │   ├── proxy.bats
 │   └── user
 │       ├── helpers.bash -> /opt/dims/tests.d/helpers.bash
 │       └── vpn.bats
 └── unit
     ├── bats-helpers.bats
     ├── dims-filters.bats
     ├── dims-functions.bats
     └── helpers.bash -> /opt/dims/tests.d/helpers.bash

 5 directories, 21 files

$ base/templates/tests/test.runner --help
usage: test.runner [options] args
flags:
  -d,--[no]debug:  enable debug mode (default: false)
  -E,--exclude:  tests to exclude (default: '')
  -L,--level:  test level (default: 'system')
  -M,--match:  regex to match tests (default: '.*')
  -l,--[no]list-tests:  list available tests (default: false)
  -t,--[no]tap:  output tap format (default: false)
  -S,--[no]sudo-tests:  perform sudo tests (default: false)
  -T,--[no]terse:  print only failed tests (default: false)
  -D,--testdir:  test directory (default: '/opt/dims/tests.d/')
  -u,--[no]usage:  print usage information (default: false)
  -v,--[no]verbose:  be verbose (default: false)
  -h,--help:  show this help (default: false)

$ test.runner --list-tests
system/dims-base.bats
system/pycharm.bats
system/dns.bats
system/docker.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
system/deprecated.bats
system/coreos-prereqs.bats
system/user/vpn.bats
system/proxy.bats

$ test.runner --level "*" --list-tests
system/dims-base.bats
system/pycharm.bats
system/dns.bats
system/docker.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
system/deprecated.bats
system/coreos-prereqs.bats
system/user/vpn.bats
system/proxy.bats
unit/dims-filters.bats
unit/bats-helpers.bats

$ test.runner --list-tests --sudo-tests
system/dims-accounts-sudo.bats
system/iptables-sudo.bats

$ test.runner --level system --match dims --list-tests
system/dims-base.bats
system/dims-accounts.bats
system/dims-ci-utils.bats

$ test.runner --level system --match "dims|coreos" --list-tests
system/dims-base.bats
system/dims-accounts.bats
system/dims-ci-utils.bats
system/coreos-prereqs.bats

$ test.runner --level system --match "dims|coreos" --exclude "base|utils" --list-tests
system/dims-accounts.bats
system/coreos-prereqs.bats

Controlling the Amount and Type of Output¶

The default for the bats program is to use --pretty formatting when standard output is being sent to a terminal. This allows the use of colors and characters like ✓ and ✗ to be used for passed and failed tests (respectively).

$ bats --help

[No write since last change]
Bats 0.4.0
Usage: bats [-c] [-p | -t] <test> [<test> ...]

  <test> is the path to a Bats test file, or the path to a directory
  containing Bats test files.

  -c, --count    Count the number of test cases without running any tests
  -h, --help     Display this help message
  -p, --pretty   Show results in pretty format (default for terminals)
  -t, --tap      Show results in TAP format
  -v, --version  Display the version number

  For more information, see https://github.com/sstephenson/bats


Press ENTER or type command to continue

We will limit the tests in this example to just those for pycharm and coreos in their names. These are relatively small tests, so it is easier to see the effects of the options we will be examining.

$ test.runner --match "pycharm|coreos" --list-tests
system/pycharm.bats
system/coreos-prereqs.bats

The DIMS test.runner script follows this same default output style of bats, so just running the two tests above gives the following output:

$ test.runner --match "pycharm|coreos"
[+] Running test system/pycharm.bats
 ✓ [S][EV] Pycharm is not an installed apt package.
 ✓ [S][EV] Pycharm Community edition is installed in /opt
 ✓ [S][EV] "pycharm" is /opt/dims/bin/pycharm
 ✓ [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
 ✓ [S][EV] Pycharm Community installed version number is 2016.2.2

5 tests, 0 failures
[+] Running test system/coreos-prereqs.bats
 ✓ [S][EV] consul service is running
 ✓ [S][EV] consul is /opt/dims/bin/consul
 ✓ [S][EV] 10.142.29.116 is member of consul cluster
 ✓ [S][EV] 10.142.29.117 is member of consul cluster
 ✓ [S][EV] 10.142.29.120 is member of consul cluster
 ✓ [S][EV] docker overlay network "ingress" exists
 ✗ [S][EV] docker overlay network "app.develop" exists
   (from function `assert' in file system/helpers.bash, line 18,
    in test file system/coreos-prereqs.bats, line 41)
     `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
   expected: "app.develop"
   actual:   ""
 ✗ [S][EV] docker overlay network "data.develop" exists
   (from function `assert' in file system/helpers.bash, line 18,
    in test file system/coreos-prereqs.bats, line 45)
     `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
   expected: "data.develop"
   actual:   ""

8 tests, 2 failures

To get TAP compliant output, add the --tap (or -t) option:

$ test.runner --match "pycharm|coreos" --tap
[+] Running test system/pycharm.bats
1..5
ok 1 [S][EV] Pycharm is not an installed apt package.
ok 2 [S][EV] Pycharm Community edition is installed in /opt
ok 3 [S][EV] "pycharm" is /opt/dims/bin/pycharm
ok 4 [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
ok 5 [S][EV] Pycharm Community installed version number is 2016.2.2
[+] Running test system/coreos-prereqs.bats
1..8
ok 1 [S][EV] consul service is running
ok 2 [S][EV] consul is /opt/dims/bin/consul
ok 3 [S][EV] 10.142.29.116 is member of consul cluster
ok 4 [S][EV] 10.142.29.117 is member of consul cluster
ok 5 [S][EV] 10.142.29.120 is member of consul cluster
ok 6 [S][EV] docker overlay network "ingress" exists
not ok 7 [S][EV] docker overlay network "app.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
#  in test file system/coreos-prereqs.bats, line 41)
#   `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
# expected: "app.develop"
# actual:   ""
not ok 8 [S][EV] docker overlay network "data.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
#  in test file system/coreos-prereqs.bats, line 45)
#   `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
# expected: "data.develop"
# actual:   ""

When running a large suite of tests, the total number of individual tests can get very large (along with the resulting output). To increase the signal to noise ratio, you can use the --terse option to filter out all of the successful tests, just focusing on the remaining failed tests. This is handy for things like validation of code changes and regression testing of newly provisioned Vagrant virtual machines.

$ test.runner --match "pycharm|coreos" --terse
[+] Running test system/pycharm.bats

5 tests, 0 failures
[+] Running test system/coreos-prereqs.bats
 ✗ [S][EV] docker overlay network "app.develop" exists
   (from function `assert' in file system/helpers.bash, line 18,
    in test file system/coreos-prereqs.bats, line 41)
     `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
   expected: "app.develop"
   actual:   ""
 ✗ [S][EV] docker overlay network "data.develop" exists
   (from function `assert' in file system/helpers.bash, line 18,
    in test file system/coreos-prereqs.bats, line 45)
     `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
   expected: "data.develop"
   actual:   ""

8 tests, 2 failures

Here is the same examples as above, but this time using the TAP compliant output:

$ test.runner --match "pycharm|coreos" --tap
[+] Running test system/pycharm.bats
1..5
ok 1 [S][EV] Pycharm is not an installed apt package.
ok 2 [S][EV] Pycharm Community edition is installed in /opt
ok 3 [S][EV] "pycharm" is /opt/dims/bin/pycharm
ok 4 [S][EV] /opt/dims/bin/pycharm is a symbolic link to installed pycharm
ok 5 [S][EV] Pycharm Community installed version number is 2016.2.2
[+] Running test system/coreos-prereqs.bats
1..8
ok 1 [S][EV] consul service is running
ok 2 [S][EV] consul is /opt/dims/bin/consul
ok 3 [S][EV] 10.142.29.116 is member of consul cluster
ok 4 [S][EV] 10.142.29.117 is member of consul cluster
ok 5 [S][EV] 10.142.29.120 is member of consul cluster
ok 6 [S][EV] docker overlay network "ingress" exists
not ok 7 [S][EV] docker overlay network "app.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
#  in test file system/coreos-prereqs.bats, line 41)
#   `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
# expected: "app.develop"
# actual:   ""
not ok 8 [S][EV] docker overlay network "data.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
#  in test file system/coreos-prereqs.bats, line 45)
#   `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
# expected: "data.develop"
# actual:   ""

$ test.runner --match "pycharm|coreos" --tap --terse
[+] Running test system/pycharm.bats
1..5
[+] Running test system/coreos-prereqs.bats
1..8
not ok 7 [S][EV] docker overlay network "app.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
#  in test file system/coreos-prereqs.bats, line 41)
#   `assert 'app.develop' bash -c "docker network ls --filter driver=overlay | awk '/app.develop/ { print \$2; }'"' failed
# expected: "app.develop"
# actual:   ""
not ok 8 [S][EV] docker overlay network "data.develop" exists
# (from function `assert' in file system/helpers.bash, line 18,
#  in test file system/coreos-prereqs.bats, line 45)
#   `assert 'data.develop' bash -c "docker network ls --filter driver=overlay | awk '/data.develop/ { print \$2; }'"' failed
# expected: "data.develop"
# actual:   ""

Figure vagrantTestRunner shows the output of test.runner --level system --terse at the completion of provisioning of two Vagrants. The one on the left has passed all tests, while the Vagrant on the right has failed two tests. Note that the error result has been passed on to make, which reports the failure and passes it along to the shell (as seen by the red $ prompt on the right, indicating a non-zero return value).

Using DIMS Bash functions in Bats tests¶

The DIMS project Bash shells take advantage of a library of functions that are installed by the base role into $DIMS/bin/dims_functions.sh.

Bats has a pre- and post-test hooking feature that is very tersely documented (see setup and teardown: Pre- and post-test hooks):

You can define special setup and teardown functions, which run before and after each test case, respectively. Use these to load fixtures, set up your environment, and clean up when you’re done.

What this means is that if you define a setup() function, it will be run before every @test, and if you define a teardown() function, it will be run after every @test.

We can take advantage of this to source the common DIMS dims_functions.sh library, making any defined functions in that file available to be called directly in a @TEST the same way it would be called in a Bash script.

An example of how this works can be seen in the unit tests for the dims_functions.sh library itself.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64

 #!/usr/bin/env bats
 #
 # Ansible managed: /home/dittrich/dims/git/ansible-playbooks/v2/roles/base/templates/../templates/tests/./unit/dims-functions.bats.j2 modified on 2016-10-01 13:50:49 by dittrich on dimsdemo1 [ansible-playbooks v1.3.33]
 #
 # vim: set ts=4 sw=4 tw=0 et :

 load helpers

 function setup() {
     source $DIMS/bin/dims_functions.sh
 }

 @test "[U][EV] say() strips whitespace properly" {
     assert '[+] unce, tice, fee times a madie...' say '    unce,   tice,        fee times a madie...      '
 }

 # This test needs to directly source dims_functions in bash command string because of multi-command structure.
 @test "[U][EV] add_on_exit() saves and get_on_exit() returns content properly" {
     assert "'([0]=\"cat /dev/null\")'" bash -c ". $DIMS/bin/dims_functions.sh; touch /tmp/foo; add_on_exit cat /dev/null; get_on_exit"
 }

 @test "[U][EV] get_hostname() returns hostname" {
     assert "$(hostname)" get_hostname
 }

 @test "[U][EV] is_fqdn host.category.deployment returns success" {
     is_fqdn host.category.deployment
 }

 @test "[U][EV] is_fqdn host.subdomain.category.deployment returns success" {
     is_fqdn host.subdomain.category.deployment
 }

 @test "[U][EV] is_fqdn 12345 returns failure" {
     ! is_fqdn 12345
 }

 @test "[U][EV] parse_fqdn host.category.deployment returns 'host category deployment'" {
     assert "host category deployment" parse_fqdn host.category.deployment
 }

 @test "[U][EV] get_deployment_from_fqdn host.category.deployment returns 'deployment'" {
     assert "deployment" get_deployment_from_fqdn host.category.deployment
 }

 @test "[U][EV] get_category_from_fqdn host.category.deployment returns 'category'" {
     assert "category" get_category_from_fqdn host.category.deployment
 }

 @test "[U][EV] get_hostname_from_fqdn host.category.deployment returns 'host'" {
     assert "host" get_hostname_from_fqdn host.category.deployment
 }

 @test "[U][EV] plural_s returns 's' for 0" {
     assert "s" plural_s 0
 }

 @test "[U][EV] plural_s returns '' for 1" {
     assert "" plural_s 1
 }

 @test "[U][EV] plural_s returns 's' for 2" {
     assert "s" plural_s 2
 }