Release Engine¶

Scenario¶

We work in a software shop and we have been told to decrease our time to delivery. We took some measurements and realized that even with Jenkins and some home-brewed systems for deployment, we’re still spending ≥ 20% of our sprint time on just deploying to test environments. Let’s focus on getting back that 20%.

How do we approach this? What functionality must be present in any kind of system which can automate deployments? Also consider that we’re just one stop in a much larger shop. Given that constraint, it follows that whatever we build should be accessible to outsiders, extensible so other teams can build on it to fit their requirements, as well as have a clear language for describing steps in a release.

• Authentication and authorization
• Storage for deployment playbooks
• Loosey coupled components, so individual installations can scale to meet their owners requirements
• Something to manage all of the actual steps happening
• And, some sort of configurable notification system, so we can get updates in real time.

When used together, the Release Engine provides all these things.

Component Diagram¶

Interactions & Workflow¶

This section describes how the Release Engine components interact with each other and the supporting infrastructure. We’ll review these interaction by examining a typical workflow.

The Bus¶

Release Engine requires an AMQP service allowing messages to pass between components. The current, verified to work, AMQP service used with Release Engine is RabbitMQ, an Erlang-based open source messaging service. For more information on setting up a RabbitMQ server please read the project’s server documentation.

For security best practices, each component that transmits on the bus should have it’s own username and password combination. By enforcing component username/passwords access can be restricted to just what a component needs. This also allows quick deactivation of a component in the event something goes terribly wrong or a service is compromised.

The Datastore¶

Release engine utilizes MongoDB for storing playbooks and other persistent data. Authentication must be turned on and it’s highly recommended to create a username/password for every component that requires access to the data store.

RE-REST¶

RE-REST is the REST endpoint for interacting with the Release Engine. This is the only interaction point by design. RE-REST is a Flask based application and requires a few libraries before it will work properly.

RE-CORE¶

The core is essentially a finite state machine (FSM) hooked into a message bus and a database.

The core oversees the execution of all release steps for any given project. The core is separate from the actual execution of each release step. Execution is delegated to the worker components.

Typical Worker Flow¶

Now, let’s translate what this is saying into human readable words:

initialize worker

In this state a process running the code representing our worker has just been started.

listen to channel

Once the process has started our worker will open a channel to the message bus and begin consuming from a queue specifically dedicated to that kind of worker.

receive parameters

Once our worker is consuming from its queue, it will sit in a waiting state until a message is received from the FSM (re-core). This message is only sent when a release is started.

The message will contain the parameters, or more generally speaking, the configuration of this step in the release. See the re-core ↔ re-workers Deployment Message Format documentation for the specifics on what is contained in this information.

Note: The message consumed from the message queue is a serialized JSON datastructure. Most workers will deserialized this message using a JSON library feature, such as the Python json.loads() method.

The most important piece of information contained in this first message is the reply_to property. This property tells our worker the name of the temporary queue to continue all further communication with the FSM on. Messages sent to any other destination will be lost in the message exchange.

Additionally, this first message will contain a correlation ID. This information should be recorded by our worker because it is used for logging and communicating back to the FSM.

ACK parameters

Now that our worker has received its job parameters, the next step is to acknowledge receiving the parameters. Our worker does this using its AMQP librarie’s ack function or method. If our worker is using the Python Pika AMQP module, it will provide the delivery tag as the parameter to the ack function.

parse parameters

If our worker requires any unique information to do its job, then it must parse that information from the parameters provided by the FSM. This step typically involves verifying everything it needs to operate was provided and is valid.

This information often includes reading which sub-command (if applicable) to run, what (if any) hosts to operate on, parameters to provide to subcommands, and dynamic values passed in at deployment-time.

parameters verified

If the parameters are parsed and it is verified that all the required information is present, then our worker will reply back to the FSM indicating that it is going to start running the step now.

The body of the message sent back to the FSM is a JSON serialized datastructure. See the Response Message Format documentation in the re-core ↔ re-worker docs for more information.

Workers using the re-worker library typically respond by calling the worker.send() method. When responding they should provide the reply_to variable as the topic parameter and leave the exchange parameter as an empty string.

parameters invalid

Our worker must notify the FSM in the unfortunate event that the parameters provided were invalid. Similar to the previous step (valid parameters) our worker will use its send() method to send a job failed message.

Once the message has been sent our worker will abort all further execution. If the worker is designed such that it runs in a some kind of io-loop (such as in the re-worker library), this is as simple as returning False while still in the process() method.

do the needful

At this point our worker has been initialized, received operating parameters from the FSM, and communicated back that it is going to proceed with the release. The next step is for the worker to begin doing what it was instructed to do.

The specifics of what happens in this step are different from worker to worker. The BigIP worker, for example, will run one of three sub-commands at this point. The exact sub-command is dictated by the value of the subcommand parameter.

step complete

If the needful was a success, then our worker will reply back to the FSM one last time (again, using its send() method) with a JSON serialized datastructure. The message will include a status key set to completed.

After the message has been sent the worker will return True and continue its loop to begin the process all over again.

step failed

If the needful was not a success, then our worker will reply back to the FSM one last time (again, using its send() method) with a JSON serialized datastructure. The message will include a status key set to failed and possibly another key, data with various information about the exact nature of the failure.

After the message has been sent the worker will return False and continue its loop to begin the process all over again.

Exercise: Write a Worker from Scratch¶

In this section we will build a worker from scratch. The worker will be written in Python. Additionally, the worker will utilize the re-worker library.

To keep things simple, our new worker will pretend to frob (“manipulate or adjust, to tweak”) an arbitrary thing and then report the results. This worker will be called the megafrobber worker. The megafrobber worker will have one sub-command: frob.

The frob sub-command requires no arguments. When the sub-command is ran, it will take no actual actions. It will just randomly pass or fail.

This section is separated into several sub-sections. Each sub-section will incrementally build upon the work of the preceeding sections. At the end, we’ll have a deployable worker.

re-worker-megafrobber/     - Top level
└── replugin/              - Python package directory
    ├── megafrobberworker/ - Worker code directory
    │   └── __init__.py    - Worker code
    └── __init__.py        - Empty file, Python module requirement

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

$ WORKER=megafrobber
$ mkdir -p re-worker-${WORKER}/replugin/${WORKER}worker
$ touch re-worker-${WORKER}/replugin/__init__.py
$ touch re-worker-${WORKER}/replugin/${WORKER}worker/__init__.py
$ find .
.
./re-worker-megafrobber
./re-worker-megafrobber/replugin
./re-worker-megafrobber/replugin/__init__.py
./re-worker-megafrobber/replugin/megafrobberworker
./re-worker-megafrobber/replugin/megafrobberworker/__init__.py

1
2
3

#!/usr/bin/env python
import reworker.worker
import logging

1
2
3
4
5
6

class MegafrobberWorker(reworker.worker.Worker):
    """
    Plugin to frob the heck out of something
    """

    def process(self, channel, basic_deliver, properties, body, output):

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

def process(self, channel, basic_deliver, properties, body, output):
    # Output is a logger from the python logger library. This is
    # what we report progress through
    self.output = output

    # This is the ID given to the currently happening deployment. It
    # is a unique ID used to connect all passed messages together and
    # record the deployment state in the database.
    #
    # We use it when responding to the FSM.
    self.corr_id = str(properties.correlation_id)

    # If the FSM passed us any dynamic variables, they will be in
    # the 'dynamic' key of the body parameter
    dynamic = body.get('dynamic', {})

    # reply_to is the temporary message bus queue we respond to the
    # FSM through
    self.reply_to = properties.reply_to

1
2
3
4
5
6
7

def main():  # pragma: no cover
    from reworker.worker import runner
    runner(MegafrobberWorker)


if __name__ == '__main__':  # pragma: no cover
    main()

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50

# Begin parameter parsing
#
# It's usually a good idea to record all of your valid
# subcommands somewhere:
self._subcommands = ['frob']

# Grab the REQUESTED subcommand from the 'parameters' dictionary
_subcommand = body['parameters'].get('subcommand', None)

# Make sure it's recognized
if _subcommand in self._subcommands:
    # This is good, the requested subcommand is valid.
    #
    # ACK the message to make the message bus happy.
    self.ack(basic_deliver)
else:
    # This is bad, the playbook calls for an unknown subcommand
    #
    # Reject the message we received on the message bus
    self.reject(basic_deliver)

    # Output to the console that an error has occurred,
    # include the correlation ID so we can trace the error
    # back to this deployment
    self.app_logger.error(
        "%s - Rejecting message, invalid subcommand requested: %s" % (
            self.corr_id, _subcommand))

    # Use 'notify' to update the output worker of our
    # progress. This output is usually logged to a central
    # location.
    self.notify(
        'Frobbing Failed',
        "Frobbing failed. Invalid subcommand requested: %s" % _subcommand,
        'failed',
        self.corr_id)

    # Send a message to the FSM indicating that the release
    # has failed. This will cause the FSM to stop the
    # deployment.
    self.send(self.reply_to,
              self.corr_id,
              {'status': 'failed',
               "message": "invalid subcommand requested: %s" % _subcommand},
              exchange='')

    # Break out of this job and start over
    return False

# End parameter parsing

import random

1
2
3
4
5
6
7
8

def _frob(self):
    """
    Frob the random number generator.

    If the result is even then "frob successful". If the result is
    odd, then "frob failed"
    """
    return random.randint(0, 100)

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47

# Begin the actual job
#
# Let the FSM know we're starting the job now
self.send(
    self.reply_to, self.corr_id, {'status': 'started'}, exchange='')

self.app_logger.info('Beginning the frobbing')

_frob_result = self._frob()

# Process the results
if (_frob_result % 2) == 0:
    _msg = "The frobbing passed, even random number generated: %s" % _frob_result

    self.app_logger.info(_msg)
    self.notify(
        'Frobbing passed',
        _msg,
        'completed',
        self.corr_id)

    # When a job succeeds, let the FSM know by sending
    # a 'completed' message
    self.send(self.reply_to,
              self.corr_id,
              {'status': 'completed',
               "message": _msg},
              exchange='')
    return True
else:
    _msg = 'Frobbing failed, odd random number generated: %s' % _frob_result

    self.app_logger.error(_msg)
    self.notify(
        'Frobbing failed',
        _msg,
        'failed',
        self.corr_id)

    # When a job fails, let the FSM know by sending
    # a 'failed' message
    self.send(self.reply_to,
              self.corr_id,
              {'status': 'failed',
               "message": _msg},
              exchange='')
    return False

  1
  2
  3
  4
  5
  6
  7
  8
  9
 10
 11
 12
 13
 14
 15
 16
 17
 18
 19
 20
 21
 22
 23
 24
 25
 26
 27
 28
 29
 30
 31
 32
 33
 34
 35
 36
 37
 38
 39
 40
 41
 42
 43
 44
 45
 46
 47
 48
 49
 50
 51
 52
 53
 54
 55
 56
 57
 58
 59
 60
 61
 62
 63
 64
 65
 66
 67
 68
 69
 70
 71
 72
 73
 74
 75
 76
 77
 78
 79
 80
 81
 82
 83
 84
 85
 86
 87
 88
 89
 90
 91
 92
 93
 94
 95
 96
 97
 98
 99
100
101
102
103
104
105
106
107
108
109
110
111
112
113
114
115
116
117
118
119
120
121
122
123
124
125
126
127
128
129
130
131
132
133
134
135
136
137
138
139
140
141
142
143
144
145
146

#!/usr/bin/env python
import reworker.worker
import logging
import random

class MegafrobberWorker(reworker.worker.Worker):
    """
    Plugin to frob the heck out of something
    """

    def process(self, channel, basic_deliver, properties, body, output):
        # Output is a logger from the python logger library. This is
        # what we report progress through
        self.output = output

        # This is the ID given to the currently happening deployment. It
        # is a unique ID used to connect all passed messages together and
        # record the deployment state in the database.
        #
        # We use it when responding to the FSM.
        self.corr_id = str(properties.correlation_id)

        # If the FSM passed us any dynamic variables, they will be in
        # the 'dynamic' key of the body parameter
        dynamic = body.get('dynamic', {})

        # reply_to is the temporary message bus queue we respond to the
        # FSM through
        self.reply_to = properties.reply_to

        # Begin parameter parsing
        #
        # It's usually a good idea to record all of your valid
        # subcommands somewhere:
        self._subcommands = ['frob']

        # Grab the REQUESTED subcommand from the 'parameters' dictionary
        _subcommand = body['parameters'].get('subcommand', None)

        # Make sure it's recognized
        if _subcommand in self._subcommands:
            # This is good, the requested subcommand is valid.
            #
            # ACK the message to make the message bus happy.
            self.ack(basic_deliver)
        else:
            # This is bad, the playbook calls for an unknown subcommand
            #
            # Reject the message we received on the message bus
            self.reject(basic_deliver)

            # Output to the console that an error has occurred,
            # include the correlation ID so we can trace the error
            # back to this deployment
            self.app_logger.error(
                "%s - Rejecting message, invalid subcommand requested: %s" % (
                    self.corr_id, _subcommand))

            # Use 'notify' to update the output worker of our
            # progress. This output is usually logged to a central
            # location.
            self.notify(
                'Frobbing Failed',
                "Frobbing failed. Invalid subcommand requested: %s" % _subcommand,
                'failed',
                self.corr_id)

            # Send a message to the FSM indicating that the release
            # has failed. This will cause the FSM to stop the
            # deployment.
            self.send(self.reply_to,
                      self.corr_id,
                      {'status': 'failed',
                       "message": "invalid subcommand requested: %s" % _subcommand},
                      exchange='')

            # Break out of this job and start over
            return False

        # End parameter parsing

        # Begin the actual job
        #
        # Let the FSM know we're starting the job now
        self.send(
            self.reply_to, self.corr_id, {'status': 'started'}, exchange='')

        self.app_logger.info('Beginning the frobbing')

        _frob_result = self._frob()

        # Process the results
        if (_frob_result % 2) == 0:
            _msg = "The frobbing passed, even random number generated: %s" % _frob_result

            self.app_logger.info(_msg)
            self.notify(
                'Frobbing passed',
                _msg,
                'completed',
                self.corr_id)

            # When a job succeeds, let the FSM know by sending
            # a 'completed' message
            self.send(self.reply_to,
                      self.corr_id,
                      {'status': 'completed',
                       "message": _msg},
                      exchange='')
            return True
        else:
            _msg = 'Frobbing failed, odd random number generated: %s' % _frob_result

            self.app_logger.error(_msg)
            self.notify(
                'Frobbing failed',
                _msg,
                'failed',
                self.corr_id)

            # When a job fails, let the FSM know by sending
            # a 'failed' message
            self.send(self.reply_to,
                      self.corr_id,
                      {'status': 'failed',
                       "message": _msg},
                      exchange='')
            return False

    def _frob(self):
        """
        Frob the random number generator.

        If the result is even then "frob successful". If the result is
        odd, then "frob failed"
        """
        return random.randint(0, 100)


def main():  # pragma: no cover
    from reworker.worker import runner
    runner(MegafrobberWorker)


if __name__ == '__main__':  # pragma: no cover
    main()

Message Queue Bindings¶

This section will describe how to configure your message queue bindings so that messages are delivered to the right workers.

Other languages¶

Nothing is stopping you from writing a worker in any other language of your choice. If you decide to do so, keep a few things in mind:

• Try to follow the re-worker reference library as close as possible
• Make sure you ack receipt of the initial message
• The initial message is a dictionary serialized as a JSON string, you’ll need to deserialize it
• Talk to the FSM on the temporary queue provided in the reply_to property
• Make sure you notify the FSM upon initial failure or start, and final failure or completion

General Guidelines¶

Please conform to PEP 0008 for code formatting. This specification outlines the highlights that we require above and beyond. Your code must follow this (or note why it can’t) before patches will be accepted. There is one consistent exception to this rule:

E501

Line too long

The pep8 tests for the Release Engine include a --ignore option to specifically exclude E501 from the tests.

Write unittests for any new functionality, if you are up to the task. Not a requirement, but it does get you a lot of karma.

Write intelligent commit messages.

Issue Reporting¶

If you are reporting an issue with the Release Engine, please use the following template when describing your issue:

Description of the issue (include full error messages):

How to reproduce the issue:

How reproducable (every time? intermittently?):

Version of the product effected (git hashes are OK):

Your operating system release-version:

What you expected to happen:

What actually happened:

Components¶

Release Engine unit tests are integrated with/depend on the following items:

• Travis CI - Free online service providing continuous integration functionality for open source projects. Tests are ran automatically on every git commit.
• unittest - Python unit testing framework. All Release Engine tests are written using this framework.
• nose - Per the nose website: “extends unittest to make testing easier”. nose is used to run our unit tests.
• coverage - A tool for measuring code coverage of Python programs. For the Release Engine we require a minimum test coverage of 80%. This is invoked by nose automatically.
• mock - A library for testing in Python. It allows you to replace parts of your system under test with mock objects (such as fake REST endpoints) and make assertions about how they have been used.
• pep8 - A tool to check Python code against some of the style conventions in PEP 0008.
• pyflakes - A simple program which checks Python source files for errors.
• virtualenv - A tool to create isolated Python environments. Allows us to install additonal package dependencies without requiring access to the system site-packages directory.
• Makefiles - Utility scripts used for project building and testing. How Release Engine uses Makefiles is described later in this section.

Requirements¶

• python-nose
• python-coverage
• python-mock

Some components may have additional test requirements. For example, re-worker-func requires pyOpenSSL, which requires openssl-devel, openssl-libs, and libffi-devel. Additionally, re-rest requires openldap-devel to run its unit tests.

Targets¶

In the scope of this document and testing, we use the term target in the context of makefile targets. For the purpose of this documentation, we can think of these targets as pre-defined commands coded in a makefile. Release Engine testing targets include:

• ci - Run the tests exactly how they are ran in Travis-CI
• pep8 - Run PEP 0008 syntax checks
• pyflakes - Run pyflakes error checks
• clean - Remove temporary files and build artifacts from the checked-out repository.
• tests - A quicker version of ci. Different from ci in that tests uses libraries installed on the local development workstation. tests runs the unittests, pep8 tests, and pyflakes tests automatically.

To ensure the highest degree of confidence in test results you should always use the ci target.

When Travis-CI runs an integration test, it calls the ci target.

Running the Tests¶

The Release Engine test suite is invoked via the Makefile. The following is an example of how to run the ci test target manually on the re-core component.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61

[~/re-core]$ make ci

#############################################
# Creating a virtualenv
#############################################
virtualenv re-coreenv
New python executable in re-coreenv/bin/python
Installing Setuptools..............................................done.
Installing Pip.....................................................done.
. re-coreenv/bin/activate && pip install -r requirements.txt
Downloading/unpacking pika>=0.9.12 (from -r requirements.txt (line 1))

... snip ...

Successfully installed pep8 nose coverage mock
Cleaning up...
#############################################
# Listing all pip deps
#############################################
. re-coreenv/bin/activate && pip freeze
coverage==3.7.1
mock==1.0.1
nose==1.3.3
pep8==1.5.7
pika==0.9.13
pymongo==2.7.1
wsgiref==0.1.2
#############################################
# Running PEP8 Compliance Tests in virtualenv
#############################################
. re-coreenv/bin/activate && pep8 --ignore=E501,E121,E124 src/recore/
#############################################
# Running Unit Tests in virtualenv
#############################################
. re-coreenv/bin/activate && nosetests -v --with-cover --cover-min-percentage=80 --cover-package=recore test/
Verify using init_amqp provides us with a connection ... ok
Loggers are created with appropriate handlers associated ... ok

... snip ...

Verify create_json_str produces proper json ... ok
Verify load_json_str produces proper structures ... ok
Verify config parsing works as expected. ... ok

Name                Stmts   Miss  Cover   Missing
-------------------------------------------------
recore                 36      0   100%
recore.amqp            72      4    94%   79, 169-172
recore.constants        1      0   100%
recore.fsm            179     25    86%   97-103, 148-152, 199-249
recore.job              0      0   100%
recore.job.create      25      0   100%
recore.mongo           62      5    92%   92-100
recore.utils           13      0   100%
-------------------------------------------------
TOTAL                 388     34    91%
----------------------------------------------------------------------
Ran 35 tests in 0.047s

OK
:

On line 1 we see how to call a makefile target. In this case it’s quite straightforward: make ci. Other targets are called in the same way. For example, to run the clean target, you run the command make clean.

On line 29 we see a header printed, Running PEP8 Compliance Tests in virtualenv. By calling the ci target, make automatically knows what other targets must be called as well, such as ci-pep8 and ci-unittests (seen on line 33).

Troubleshooting¶

If you find yourself unable to run the unit tests:

1. Search for relevant error messages
2. Read the error message closely. The solution could be hidden in the error message output. The problem could be as simple as a missing dependency
3. If you are unable to figure out all the necessary dependencies to run the tests, file an issue on that specific projects GitHub issue tracker. Include the full error message.

Deployment Message Format¶

The Deployment Message format comes in two flavors, Simple and Dynamic.

{
   "group": "My Group",
   "playbook_id": "1234567890"
}

{
   "group": "My Group",
   "playbook_id": "1234567890",
   "dynamic": {
       "cart": "my cart",
       "environment": "QA"
   }
}

{
   "id": "9999999999999999"
}

{
   "id": null
}

Deployment Message Formats¶

This section describes the message formats emitted by re-core to workers.

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11

{
    "name": "docs test",
    "group": "test",
    "execution": [
        {
            "description": "sequence 0",
            "hosts": [ "host01.example.com" ],
            "steps": [ "bigip:OutOfRotation" ]
        }
    ]
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

{
    "parameters": {
        "hosts": [ "host01.example.com" ],
        "command": "bigip",
        "subcommand": "OutOfRotation"
    },
    "group": "test",
    "dynamic": {},
    "notify": {}
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
    "name": "docs test",
    "group": "test",
    "execution": [
        {
            "description": "sequence 0",
            "hosts": [ "host01.example.com" ],
            "steps": [
                {
                    "service:Restart": {
                        "service": "megafrobber"
                    }
                }
            ]
        }
    ]
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13

{
    "parameters": {
        "service": "megafrobber",
        "hosts": [
            "host01.example.com"
        ],
        "subcommand": "Restart",
        "command": "service"
    },
    "group": "test",
    "dynamic": {},
    "notify": {}
}

1
2
3
4
5
6
7
8

{
    "juicer:Promote": {
        "dynamic": [
            "cart",
            "environment"
        ]
    }
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19

{
    "parameters": {
        "command": "juicer",
        "dynamic": [
            "cart",
            "environment"
        ],
        "subcommand": "Promote",
        "hosts": [
            "host01.example.com"
        ]
    },
    "group": "test",
    "dynamic": {
        "cart": "bitmath",
        "environment": "re"
    },
    "notify": {}
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14

{
    "steps": [
        {
            "service:Restart": {
                "service": "megafrobber",
                "notify": {
                    "started": {
                        "irc": [ "PHB", "#teamchannel" ]
                    }
                }
            }
        }
    ]
}

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17

{
    "parameters": {
        "service": "megafrobber",
        "hosts": [
            "host01.example.com"
        ],
        "subcommand": "Restart",
        "command": "service"
    },
    "group": "test",
    "dynamic": {},
    "notify": {
        "started": {
            "irc": [ "PHB", "#teamchannel" ]
        }
    }
}

Response Message Formats¶

Complimenting the Deployment Message Formats are the Response Message Formats. There are three status messages which workers may reply to re-core with. This section describes the messages which workers send to the re-core component.

{
    "status": "started"
}

{
    "status": "completed"
}

1
2
3
4
5
6
7

{
    "status": "completed",
    "data": {
        "items_frobbed": 1337,
        "avg_time_to_frob_ms": 100
    }
}

{
    "status": "failed"
}

Required Items¶

This section describes the required items in execution sequences.

---
- hosts:
   - www01.web.ext.example.com
   - www02.web.ext.example.com
   - www03.web.ext.example.com

---
- hosts: [www01.web.ext.example.com, www02.web.ext.example.come, www03.web.ext.example.com]

1
2
3
4
5
6
7
8
9

---
group: inception
name: Simple playbook
execution:
  - description: Rotate all the megafrobber logs
    hosts:
      - foo.bar.example.com
    steps:
      - logrotate:Rotate

 1
 2
 3
 4
 5
 6
 7
 8
 9
10

---
group: inception
name: Simple playbook
execution:
  - description: Restart the megafrobber service
    hosts:
      - foo.bar.example.com
    steps:
      - service:Restart:
          service: megafrobber

1
2
3

---
# ...
      - service:Restart: {service: megafrobber}

1
2

---
service: megafrobber

1
2
3
4
5

---
- thingies:
    service: megafrobber
- stuffs:
    penguins: cute

1
2

---
[{thingies: {service: megafrobber}}, {stuffs: {penguins: cute}}]

1
2
3
4
5
6
7
8
9

---
# ...
execution:
  - description: Restart the megafrobber service
    hosts:
      - foo.bar.example.com
    steps:
      - service:Restart:
          service: megafrobber

1
2
3
4
5
6

---
# ...
      - service:Restart:
          service: megafrobber
          foo: bar
          noop: true