Botoform

Manage infrastructure running on AWS using YAML templates.

Botoform provides tools to manage the lifecycle of related AWS resources. We use a simple YAML schema to document resources as infrastructure. The YAML schema has self documenting qualities and works with version control.

In this example we use the bf create tool to build the infrastructure defined in helloworld.yaml:

_images/botoform-helloworld.gif

The bf tools use YAML architecture to create and manage environments. Botoform allows reproduction of any environment, no matter how complex.

Botoform abstracts and enriches the Boto3 and Botocore projects.

Quickstart

Quickstart

Installation

Virtualenv

Both the automatic and manual install assume that the virtualenv tool is installed. If you do not have virtualenv installed, you may do the following:

# only run this if you are missing the virtualenv tool.
sudo pip install virtualenv
Automatic install from source with botoform-bootstrap.sh

Note

You should always review scripts prior to piping them from the Internet into your shell.

This script automates the steps in the manual install from source section.

The following one-liner will install botoform (bf) into your home directory:

wget -O - https://raw.githubusercontent.com/russellballestrini/botoform/master/botoform-bootstrap.sh | sh

Next you should verify the botoform install.

Manual install from source

Clone botoform repo:

git clone https://github.com/russellballestrini/botoform.git $HOME/botoform
cd $HOME/botoform

Create and activate a Python virtualenv named env:

virtualenv env
. env/bin/activate

Install dependencies into virtualenv:

python setup.py develop

Next you should verify the botoform install.

Verify the botoform install

Whenever you want to use the bf tool, you need to activate the virtualenv:

You may verify installation by running:

bf --help

You should see usage information.

Next, edit your AWS configuration file with your access/secret keys.

Configuration

Setup your AWS CLI config file, for example -

~/.aws/config:

[profile development]
aws_access_key_id = AKIAIOSFODNN7EXAMPLE
aws_secret_access_key = wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY
region = us-west-2

You are now ready to begin using Botoform!

Using Botoform

Create VPC

Note

This section will create real resources on AWS.

bf --profile=development --region=ap-southeast-1 create dogtest01 -e 'vpc_cidr=192.168.1.0/24' tests/fixtures/webapp.yaml
Unlock VPC

Note

This command will unlock instances to allow them to be terminated.

Disable API Termination Protection on all instances in VPC.

bf --profile=development --region=ap-southeast-1 unlock dogtest01
Destroy VPC

Danger

This command will completely destroy the entire VPC and all related resources!

bf --profile=development --region=ap-southeast-1 destroy dogtest01

Example Output

bf -p development -r us-east-1 create -e 'vpc_cidr=10.20.20.0/24' stg example.yaml

creating vpc (stg, 10.20.20.0/24)
tagging vpc (Name:stg)
modifying vpc for dns support
modifying vpc for dns hostnames
creating internet_gateway (igw-stg)
tagging gateway (Name:igw-stg)
attaching igw to vpc (igw-stg)
creating DHCP Options Set for stg
associating DHCP Options dopt-30731454 with VPC stg
creating route_table (stg-public)
tagging route_table (Name:stg-public)
creating route_table (stg-private)
tagging route_table (Name:stg-private)
creating subnet 10.20.20.96/27 in us-east-1c
tagging subnet (Name:stg-private-2)
creating subnet 10.20.20.64/27 in us-east-1b
tagging subnet (Name:stg-private-1)
creating subnet 10.20.20.32/27 in us-east-1b
tagging subnet (Name:stg-public-1)
creating subnet 10.20.20.0/27 in us-east-1c
tagging subnet (Name:stg-public-2)
creating security_group stg-all
tagging security_group (Name:stg-all)
creating security_group stg-bastion
tagging security_group (Name:stg-bastion)
creating key pair default
associating rt private with sn private-2
associating rt private with sn private-1
associating rt public with sn public-1
associating rt public with sn public-2
creating role: bastion
1 instances of role bastion launching into stg-public-1 subnet
inbound rule: 'bastion' -> 'all' over ports 22 (TCP)
inbound rule: '98.110.147.178/32' -> 'bastion' over ports 22 (TCP)
inbound rule: '184.188.101.86/32' -> 'bastion' over ports 22 (TCP)
waiting for i-00b858ff5644d8634 to start
waiting for i-00b858ff5644d8634 to be in status OK
tagging instance i-00b858ff5644d8634 (Name:stg-bastion-delawareriver)
tagging volumes for instance stg-bastion-delawareriver (Name:stg-bastion-delawareriver)
allocating eip and associating with stg-bastion-delawareriver
allocated eip 54.80.219.185 and associated with stg-bastion-delawareriver
locking new normal (not autoscaled) instances to prevent termination
adding route ['0.0.0.0/0', 'internet_gateway'] to route_table (stg-public)
managing route53 private zone.
done! don't you look awesome. : )

Schema Reference

End user YAML Schema Reference for describing VPCs and related AWS resources.

Schema Reference

End user YAML Schema Reference for describing VPCs and related AWS resources.

These documents describe how to structure YAML in a Botoform compatible schema.

You may optionally use Jinja2 in your YAML config.

amis

Define AMIs using the following schema:

amis:

  human-readable-custom-name:

      region-name: ami-id

Example code snippet from amis.yaml:

amis:

  ubuntu-12.04-lts:

    us-east-1: ami-59a4a230
    us-west-1: ami-660c3023
    us-west-2: ami-fa9cf1ca
    sa-east-1: ami-1da90a00
    eu-west-1: ami-808675f7
    eu-central-1: ami-043d0b19
    ap-southeast-1: ami-3c39686e
    ap-southeast-2: ami-09f26b33
    ap-northeast-1: ami-f381f5f2

  ubuntu-12.04-lts-hvm:

    us-east-1: ami-5fa4a236
    us-west-1: ami-620c3027
    us-west-2 : ami-fc9cf1cc
    sa-east-1: ami-1fa90a02

route_tables

Define route_tables using the following schema:

route_tables:
  human-readable-custom-name:
      routes:
          - ['cidr', 'destination']

Example code snippet from route_tables.yaml:

# This is an example route_tables schema.
route_tables:

  private-1:
    routes:
      - ['0.0.0.0/0', 'nat']

  private-2:
    routes:
      - ['0.0.0.0/0', 'nat']

  private-3:
    routes:
      - ['0.0.0.0/0', 'nat']

  public:
    routes:
      - ['0.0.0.0/0', 'internet_gateway']

subnets

Define subnets using the following schema:

subnets:

  human-readable-custom-name:

      size: cidr_integer
      route_table: name-of-the-route-table
      description: human readable description (optional)
      availability_zone: char of az (optional)

Example code snippet from subnets.yaml:

# This is an example of a 4 subnet schema.
# Each subnet has a size of 27 which is a /27 CIDR (32 addresses).
# Two are attached to the private route_table, and two to the public.
# We round robin AZ letters if availability_zone is not defined.
# Instances will launch into subnet with public IPs, if public is True.
subnets:

  private-1:
    size: 27
    route_table: private

  private-2:
    size: 27
    route_table: private

  public-1:
    size: 27
    route_table: public
    availability_zone: a

  public-2:
    size: 27
    route_table: public
    availability_zone: b

tags

Optional tags to add to all taggable resources using the following schema:

tags:
  key: value

For example to create the env tag with the value dev add this to your schema:

tags:
  env: dev

vpc

Optional vpc schema settings:

vpc_cidr: the-cidr-block-to-allocate-to-your-vpc
vpc_tenancy: whether-or-not-your-vpc-should-use-default-or-dedicated

For example:

vpc_cidr: 10.10.10.0/24
vpc_tenancy: default

Hello World

Smallest botoform template, helloworld.yaml:

vpc_cidr: {{ vpc_cidr }}

amis:
  ubuntu-14.04-lts-hvm:
    us-east-1: ami-fce3c696

route_tables:
  public:
    routes:
      - ['0.0.0.0/0', 'internet_gateway']

subnets:
  public-1:
    size: 27
    route_table: public
    public: True

security_groups:
  bastion:
    inbound:
      - ['0.0.0.0/0', 'tcp',   22]

instance_roles:
  bastion:
    instance_type: t2.micro
    ami: 'ubuntu-14.04-lts-hvm'
    count: 1
    security_groups: ['bastion']
    subnets: ['public-1']
    eip: true
    block_devices:
      "/dev/sda1":
        size: 10

You may optionally use Jinja2 in your YAML config.

Tools

Tools

bf

Botoform tools are namespaced under the bf command.

For a list built-in subcommands and interactive help, run bf --help.

Currently Implemented subcommands:

{atmosphere,shell,cli,dump,list,lock,create,stop,start,unlock,repl,destroy}
list

List all existing VPC names.

For example:

bf list

You can also pass the particular AWS profile and/or region:

bf --profile developmemt --region us-west-2 list
create

Create a new VPC and related services, modeled after the given YAML template.

For example:

bf create dogtest01 -e 'vpc_cidr=192.168.1.0/24' tests/fixtures/webapp.yaml
destroy

Warning

this is destructive!

Destroy a VPC and all related services.

For example:

bf destroy dogtest01
refresh

Refresh VPC by adding resources defined but missing in given YAML template.

So far we have implemented the following subcommands:

  • instance_roles
  • security_groups
  • load_balacers
  • tags
  • private_zone
reflect

Note

not implemented yet.

Warning

this is destructive!

Make VPC reflect given YAML template by adding and removing resources.

  • instance_roles
  • security_groups
  • tags
  • private_zone
stop

Stop all instances in VPC including autoscaled instances.

Warning

Currently does _NOT_ skip “ephemeral” instances!

start

Start all instances in VPC including autoscaled instances.

lock

Enable API Termination Protection on all instances in VPC.

unlock

Disable API Termination Protection on all instances in VPC.

repl

Open an interactive REPL (read-eval-print-loop) with access to evpc object.

Once you have a repl, try running evpc.roles or evpc.instances.

usage: bf repl vpc_name  [-h]
Note:
Install bpython into your environment for more fun.
bf webapp01 repl

You now have access to the evpc object, for example: evpc.roles

>>> evpc.instances
[<botoform.enriched.instance.EnrichedInstance object at 0x10e194350>,
<botoform.enriched.instance.EnrichedInstance object at 0x10e1944d0>

>>> map(str, evpc.instances)
['webapp01-web01', 'webapp01-web02']
cli

An alias to repl so it works the same.

shell

An alias to repl so it works the same.

exec

Accept a Python program on STDIN and execute it.

Usage 1 (echo and pipe):

echo "print(set([i.image_id for i in evpc.instances]))" | bf --profile development exec dogtest01

Usage 2 (redirection):

bf --profile development exec dogtest01 < unique_active_amis.py

Where unique_active_amis.py has the following content:

print(set([i.image_id for i in evpc.instances]))

In both examples, the output would look something like this:

set(['ami-33333333', 'ami-55555555', 'ami-99999999', 'ami-77777777'])
dump

Output existing resources or services in a Botoform campatible format.

  • instances
  • security_groups
  • ansible_hosts
  • tags
atmosphere

For every AWS profile + region, dump every VPC to STDOUT.

This command takes a while to run, so you should likely redirect the output to a file.

Reason for this tool is we have many AWS accounts and we use many regions.

Using the output of this tool, we can easily grep for a vpc_name and find where it lives.

Developer Reference

Developer Python Library Reference for modifying or extending Botoform.

Developer Reference

Developer Reference for modifying or extending Botoform.

This document provides details for Botoform’s Python codebase.

builders.py

The builders.py file uses YAML config to build the environment.

class botoform.builders.EnvironmentBuilder(vpc_name, config=None, region_name=None, profile_name=None, log=None)
add_eip_to_instance(instance)
apply_all()

Build the environment specified in the config.

associate_route_tables_with_subnets(subnet_cfg)
attach_vpn_gateway(vpn_gateway_cfg)

Attach defined VPN gateway to VPC

autoscaling_instance_role(role_name, role_data, desired_count)
autoscaling_instance_roles(instance_role_cfg)

Create Autoscaling Groups and Launch Configurations.

build_internet_gateway()

Build and attach Internet Gateway to VPC.

build_vpc(cidrblock='172.31.0.0/16', tenancy='default')

Build VPC

create_dhcp_options(dhcp_configurations)

Creates and return a new dhcp_options set.

create_instance_profile(instance_profile_name)

Create instance_profile and role, return instance_profile.

db_instances(db_instance_cfg)

Build RDS DB Instances.

dhcp_options(dhcp_options_cfg)

Creates DHCP Options Set and associates with VPC

endpoints(route_tables)

Build VPC endpoints for given route_tables

finish_instance_roles(instance_role_cfg, instances=None)
get_instance_profile(instance_profile_name)

Return instance_profile or None.

instance_profiles(instance_role_cfg)
instance_role(role_name, role_data, desired_count)
instance_roles(instance_role_cfg)

Create instance roles defined in config.

key_pairs(key_pair_cfg)
load_balancers(load_balancer_cfg)

Build ELB load balancers.

route_table_rules(route_cfg)

Build route table rules defined in config

route_tables(route_cfg)

Build route_tables defined in config

security_group_inbound_rules(security_group_cfg)

Build inbound rule for Security Group defined in config.

security_group_outbound_revoke_default_rule(sg)
security_group_outbound_rules(security_group_cfg)

Build outbound rule for Security Group defined in config.

security_group_rule_to_permission(rule)

Return a permission dictionary from a rule tuple.

security_group_rules(security_group_cfg)

Build Security Group Rules defined in config.

security_group_rules_to_permissions(sg_name, rules, direction='inbound')
security_groups(security_group_cfg)

Build Security Groups defined in config.

subnets(subnet_cfg)

Build subnets defined in config.

tag_instance_name(**kw)

Accept a EnrichedInstance, objects create tags.

tag_instance_volumes(instance)

Accept an EnrichedInstance, tag all attached volumes.

tags(tag_cfg)
wait_for_instance_profile(**kw)
wait_for_instance_roles_to_exist(**kw)
botoform.builders.get_default_ec2_trust_policy(region_name)

config.py

The config.py turns YAML into Python dictionaries, supports Jinja2.

class botoform.config.ConfigLoader(template_dir=None, context_vars=None)

Loads Botoform Schema and returns a dictionary.

load(template_path=None, template_string=None)

Load a Botoform Schema config and render with Jinja2.

Parameters:
  • template_path – Path to the config to load and render.
  • template_string – String to load and render. Optional.
Returns:

Python dictionary representation of config.
render(template_file)

Return Jinja2 render of template with context_vars.

render_string(template_string)

Return Jinja2 render of template_str with context_vars.

template_dir

subnetallocator.py

botoform.subnetallocator.allocate(cidrs, sizes)

Accept network block CIDRs and list of subnetwork CIDR sizes. Return a list of IPNetwork objects allocated from the network block.

For example:

>>> allocate('10.10.10.0/24', [28,27,29])
[IPNetwork('10.10.10.0/27'), IPNetwork('10.10.10.32/28'), IPNetwork('10.10.10.48/29')]

util.py

The util.py file is sort of a junk drawer right now… but..

class botoform.util.BotoConnections(region_name=None, profile_name=None)

Central Management of boto3 client and resource connection objects.

azones

Return a list of available AZ names for active AWS profile/region.

profile_name
refresh_boto_connections()

Attach related Boto3 clients and resources.

region_name
setup_session_and_refresh_connections()
class botoform.util.BotoformDumper(stream, default_style=None, default_flow_style=None, canonical=None, indent=None, width=None, allow_unicode=None, line_break=None, encoding=None, explicit_start=None, explicit_end=None, version=None, tags=None)

A custom YAML dumper that is pretty.

increase_indent(flow=False, indentless=False)
yaml_representers = {None: <unbound method SafeRepresenter.represent_undefined>, <type 'module'>: <unbound method Representer.represent_module>, <type 'classobj'>: <unbound method Representer.represent_name>, <type 'function'>: <unbound method Representer.represent_name>, <type 'builtin_function_or_method'>: <unbound method Representer.represent_name>, <type 'int'>: <unbound method SafeRepresenter.represent_int>, <type 'long'>: <unbound method Representer.represent_long>, <type 'float'>: <unbound method SafeRepresenter.represent_float>, <type 'complex'>: <unbound method Representer.represent_complex>, <type 'unicode'>: <unbound method SafeRepresenter.represent_unicode>, <type 'set'>: <unbound method SafeRepresenter.represent_set>, <type 'NoneType'>: <unbound method SafeRepresenter.represent_none>, <type 'bool'>: <unbound method SafeRepresenter.represent_bool>, <type 'type'>: <unbound method Representer.represent_name>, <type 'tuple'>: <unbound method SafeRepresenter.represent_list>, <type 'dict'>: <unbound method SafeRepresenter.represent_dict>, <type 'str'>: <unbound method SafeRepresenter.represent_str>, <type 'list'>: <unbound method SafeRepresenter.represent_list>, <type 'datetime.datetime'>: <unbound method SafeRepresenter.represent_datetime>, <type 'datetime.date'>: <unbound method SafeRepresenter.represent_date>}
class botoform.util.Log(desired_level='debug', syslog=True, stdout=True, program='botoform')

Handles emitting logs to syslog and stdout.

emit(message, level='info')

Emit message if level meets requirement.

message:
Any object that has a __str__ method.
level:
The level or severity of this message (default info)
levels

Return a list of levels on and beyond desired_level.

botoform.util.collection_len(collection)
botoform.util.collection_to_list(collection)
botoform.util.dict_to_key_value(data, sep='=', pair_sep=', ')

Return a string representation of a dictionary.

by default this function will turn:

{'key1':'value1','key2':'value2'}

into:

key1=value1,key2=value2
Parameters:
  • data – The dictionary to convert into a string.
  • sep – Optional, string to separate keys and values (Default ‘=’)
  • pair_sep – Optional, string to separate key/value pairs (Default ‘,’)
Returns:

a string representation of the given dictionary.
botoform.util.generate_password(size=9, pool=None)

Return a system generated password.

Parameters:
  • size – The desired length of the password to generate. (Default 9)
  • pool – Pool of chars to choose from. (Default digits and letters [upper/lower])
Returns:

String (raw password)
botoform.util.get_block_device_map_from_role_config(role_cfg)

accept role config data and return a Boto3 friendly BlockDeviceMappings.

botoform.util.get_ids(objects)

Return a list of ids from a list of objects.

Parameters:objects – A list of objects all of whom have an id attribute.
Returns:A list of ids
botoform.util.get_port_range(raw_range, ip_protocol='tcp')

Returns a (from_port, to_port) tuple.

Examples:

>>> get_port_range(443)
(443, 443)

>>> get_port_range('all')
(1, 65535)

>>> get_port_range('5000-5009')
(5000, 5009)

>>> get_port_range(' 8080')
(8080, 8080)

>>> get_port_range('tacobell', ip_protocol='icmp')
(-1, -1)
Parameters:
  • raw_range – A string or integer.
  • ip_protocol – Optional, ‘tcp’, ‘udp’, ‘icpm’ (Default ‘tcp’)
Returns:

(from_port, to_port)
botoform.util.id_to_human(id_string)

Turn an id into a human readable hash digest.

Parameters:id_string – The subject string to generate a human hash of. 
>>> id_to_human('i-ceebb70c')
'friendisland'
botoform.util.key_value_to_dict(key_value_list, sep='=', pair_sep=', ')

Return a dictionary from a list of key/value strings.

turns key_value_list, like:

key_value_list = ['a=1,b=2', 'c=3, d=4', 'e=5']

into a dict, like:

{'a':'1', 'b':'2', 'c':'3', 'd':'4', 'e':'5'}
Parameters:
  • key_value_list – The list of key/value strings to convert into a dict.
  • sep – Optional, string which separates keys and values (Default ‘=’)
  • pair_sep – Optional, string which separates key/value pairs (Default ‘,’)
Returns:

A string representation of the given dictionary.
botoform.util.make_filter(key, values)

Return a filter document expected by Boto3.

Parameters:
  • key – The key name for this new filter document.
  • values – A value or a list of values to filter/match on.
Returns:

A filter document (list/dict) in the form that Boto3 expects.
botoform.util.make_tag_dict(ec2_object)

Return a dictionary of existing tags.

Parameters:ec2_object – A tagable Boto3 object with a tags attribute.
Returns:A dictionary where tag names are keys and tag values are values.
botoform.util.map_filter_false(function, items)

Works like map but automatically filters out untruthy items.

>>> map_filter_false(lambda i : i, [1,2,None,3,False,0,4])
[1, 2, 3, 4]
Parameters:
  • function – the function to map items through
  • items – a list of items to map through the function
Returns:

list
botoform.util.merge_pages(key, pages)

Merge boto3 paginated results into single list.

Parameters:
  • key – The document key to merge from all pages.
  • pages – An iterator of page documents.
Returns:

A single flat list containing results of all pages.
botoform.util.normalize_sg_port(sg_rule_tuple)

accept a security group rule tuple, return normalized port range.

botoform.util.normalize_sg_rules(sg_rules)

accept a list of security group rule tuples, return list with normalized port ranges.

botoform.util.output_formatter(data, output_format='newline')

Print data in the optional output_format.

botoform.util.reflect_attrs(child, parent, skip_attrs=None)

Composition Magic: reflect all missing parents attributes into child.

Parameters:
  • child – Object to receive attributes.
  • parent – Object to source attributes from.
  • skip_attrs – Optional list of attrs strings to not reflect.
Returns:

None
botoform.util.snake_to_camel_case(name, answers=None)

Accept a snake_case string and return a CamelCase string.

For example:

>>> snake_to_camel_case('cidr_block')
'CidrBlock'
botoform.util.tag_filter(tag_key, tag_values)

Return a tag filter document expected by boto3.

Parameters:
  • tag_key – A tag key or name.
  • tag_values – The tag value or a list of values to filter on.
Returns:

A filter document (list/dict) in the form that Boto3 expects.
botoform.util.update_tags(*args, **kw)

Add or update tags to reflect given keyword args

Parameters:
  • ec2_object – A tagable Boto3 object with a tags attribute.
  • **kwargs – key=value where key is tag name, value is tag value.
Returns:

None
botoform.util.write_private_key(key_pair)

Write private key to filesystem.

Parameters:key_pair – The Boto3 KeyPair object to write to filesystem.
Returns:None

enriched

Botoform abstracts and enriches the Boto3 and Botocore projects.

The references in this document describe how we extend each project.

We use composition to provide many helper attributes and methods.

autoscaling.py
class botoform.enriched.autoscaling.EnrichedAutoscaling(evpc)
get_all_autoscaling_group_descriptions()

Return a list of all autoscaling groups descriptions.

get_all_launch_config_descriptions()

Return a list of all launch configuration descriptions.

Return a list of related autoscaling groups descriptions.

Return a list of related launch configuration descriptions.

elasticache.py
class botoform.enriched.elasticache.EnrichedElastiCache(evpc)

delete all cache clusters and subnet groups related to this VPC.

Parameters:cluster_ids – optional list of cache_cluster_ids (names) to delete instead.
Returns:None
get_all_cluster_descriptions()
get_all_subnet_group_descriptions()

return a list of cache cluster descriptions related to VPC.

return a list of cache cluster dns endpoints related to this VPC

return a list of cache cluster ids related to this VPC

return a list of cache subnet group descriptions related to VPC.

wait for related dbs to transition to desired state.

elb.py
class botoform.enriched.elb.EnrichedElb(evpc)
format_instance_ids(instance_ids)

god boto3 is a pain sometimes.

format_listeners(listener_tuples)
get_all_elb_descriptions()

Return a list of all ELB (Load Balancer) descriptions.

Return a list of related ELB (Load Balancer) descriptions.

Return a list of related ELB (Load Balancer) names.

register_role_with_load_balancer(elb_name, role_name)
instance.py
class botoform.enriched.instance.EnrichedInstance(instance, evpc=None)

This class uses composition to enrich Boto3’s Instance resource class.

ec2.Instance(boto3.resources.base.ServiceResource)

Reference:

https://boto3.readthedocs.org/en/latest/reference/services/ec2.html#instance

allocate_and_associate_eip()

Allocate and Associate a new EIP with instance and return EIP.

Returns:New VpcAddress EIP object
allocate_eip()

Allocate a new EIP and return allocation_id

Returns:New allocation_id
associate_eip(**kw)

Associate EIP with instance and return EIP.

Returns:New VpcAddress EIP object
autoscale_group

Return autoscaling group name (AWS aws:autoscaling:groupName tag).

disassociate_eips(release=True)

Disassociate all EIPs associated with this instance.

Parameters:release – Also release allocations for EIPs. Default True.
Return type:None
eips

Return a list of VpcAddress objects associated to this instance. :rtype: list

hostname

Return hostname (AWS Name tag).

id_human

Return humanhash of id.

identifiers

Return a tuple of “unique” identifier strings for instance. :rtype: tuple

identity

Return hostname (AWS Name tag) or id.

is_autoscaled

Return True if this instance was autoscaled else False

is_spot

Return True is this instance is a spot instance else False

lock()

Lock this instance to prevent termination.

name

Return hostname (AWS Name tag).

reflect_attrs()

reflect all attributes of ec2.Instance into self.

reload()

run the reload method on the attached instance and reflect_attrs.

role

Return role from from ‘role’ or ‘Name’ tag: web, db, …

shortname

Return shortname from hostname: web-solarmaine, db-beerindia, …

source_dest_check_disable()

Disable source destination checking. Needed for NATs and Routers.

source_dest_check_enable()

Enable source destination checking. Default.

tag_dict

Return dictionary of tags.

unlock()

Unlock this instance to allow termination.

wait_until_status_ok()

Wait (block) until this instance state is ‘OK’.

key_pair.py
class botoform.enriched.key_pair.EnrichedKeyPair(evpc)
create_key_pair(short_key_pair_name)

Create a KeyPair object, update key_pairs AWS tag.

delete_key_pair(short_key_pair_name)

Delete a KeyPair object, update key_pairs AWS tag.

delete_key_pairs()

Delete ALL related KeyPair objects, update key_pairs AWS tag.

delete_key_pairs_tag()
get_key_name(short_key_pair_name)

Returns full key_name if key exists, else None

get_key_pair(short_key_pair_name)

Return a KeyPair object by key_name.

key_names

Return a list of ssh key pair names from AWS VPC tag.

key_pairs

Return a dictionary of ssh KeyPair objects.

rds.py
class botoform.enriched.rds.EnrichedRds(evpc)

delete all RDS DB instances and subnet groups related to this VPC.

rds_ids:
optional list of rds_ids (names) to delete instead.
get_all_db_descriptions()

return a list of all db description dictionaries.

return a dictionary of DB Connection data related to this VPC

return a list of db description dictionaries related to this VPC.

return a list of cache cluster dns endpoints related to this VPC

return a list of db instance identifiers related to this VPC.

reset_master_passwords(db_ids)

Iterate over list of db_ids (names) reset master password immediately. Return dictionary of changes in the form of {‘db_id’ : ‘new_pass’}.

wait for related dbs to transition to desired state.

route53.py
class botoform.enriched.route53.EnrichedRoute53(evpc)
change_private_zone(change_docs)
create_private_zone()
delete_private_zone()
empty_private_zone()
list_all_private_resource_record_sets()
list_all_resource_record_sets(hosted_zone_id, record_sets=None, marker=None)

Handles pagination unlike list_resource_record_sets.

private_zone_id

get the vpc with the private hosted zone id from vpc tag.

private_zone_name
refresh_private_zone()
vpc.py
class botoform.enriched.vpc.EnrichedVPC(vpc_name=None, region_name=None, profile_name=None, log=None)

This class uses composition to enrich Boto3’s VPC resource class. Here we relate AWS resources using various techniques like the vpc_name tag. We also provide methods for managing the lifecycle of related AWS resources.

associate_route_table_with_subnet(rt_name, sn_name)

Accept a route table name and subnet name, associate them.

attach_vpn_gateway(vgw_id)

Attach VPN gateway to the VPC

azones
connect(vpc_name)

connect to VPC and reflect all attributes into self.

delete_dhcp_options()

Delete DHCP Options Set

delete_instances(instances=None, wait=True)

Terminate all or a list of instances.

delete_internet_gateways()

Delete related internet gatways.

delete_route_tables()

Delete related route tables.

delete_security_group(**kw)
delete_security_groups()

Delete related security groups.

delete_subnets()

Delete related subnets.

detach_vpn_gateway()

Detach VPN gateway from VPC

enriched_security_groups

Format Security Groups (and permissions) in Botoform Schema.

Returns:security_groups in Botoform Schema.
ensure_vgw_state(**kw)

if vgw is not in expected state, throw exception.

exclude_instances(identifiers=None, roles=None)

Accept a list of identifiers and/or roles. Return a list of instances which do not match either qualifier list.

Note:
This method returns all instances if both identfiers and roles is None.
find_instance(identifier)

Return an instance or None which matches identifier.

Raises exception if multiple instances match identifier.

Parameters:identifier –
A list of identifiers to qualify instances by, for example:
  • custid-ui01
  • ui01
  • 192.168.1.9
  • i-01234567
Returns:EnrichedInstance or None
find_instances(identifiers=None, roles=None, exclude=False)

Accept a list of identifiers and/or roles. Return a list of instances which match either qualifier list.

Parameters:
  • identifiers –
    Optional, a list of identifiers to qualify instances by, for example:
    • custid-ui01
    • ui01
    • 192.168.1.9
    • i-01234567
  • roles –
    Optional, a list of roles to qualify instances by, for example:
    • ui
    • api
    • proxy
  • exclude – If True, qualifiers exclude instead of include! Defaults to False.
Danger:
This method will return no instances if all qualifiers are None. However, if exclude is True we could return all instances!
Returns:A list of EnrichedInstance objets or an empty list.
get_autoscaled_instances(instances=None)

return a list of instances which were created via autoscaling.

get_instances(instances=None)

Returns a possibly empty list of EnrichedInstance objects.

Parameters:instances – Optional, list or collection to convert to EnrichedInstance objects.
Returns:list of EnrichedInstance objects
get_main_route_table()

Return the main (default) route table for VPC.

get_normal_instances(instances=None)

return a list of instances which were _not_ created via autoscaling.

get_role(role_name, instances=None)

Return a possibly empty list of EnrichedInstance objects.

Parameters:
  • role_name – The name of the role whose instances to return.
  • instances – Optional, list or collection to search role from.
Returns:

A list of EnrichedInstance objects.
get_roles(instances=None)

Return a dict of lists where role is the key and a list of EnrichedInstance objects is the value.

get_route_table(name)

Accept route table name, return route_table object or None.

get_running_instances(instances=None)

Return list running EnrichedInstance object related to this VPC.

get_security_group(name)

Accept security group name, return security group object or None.

get_subnet(name)

Accept subnet name, return subnet object or None.

get_vgw(vgw_id)

Accept vgw_id and return vgw description.

get_vpc_by_name_tag(vpc_name)

lookup vpc by vpc_name tag. Raises exceptions on insanity.

get_vpn_gateways()

Gets all the VGWs attached to the VPC

identity
include_instances(identifiers=None, roles=None)

Accept a list of identifiers and/or roles. Return a list of instances which match either qualifier list.

Note:
This method returns no instances if both identfiers and roles is None.
instances
lock_instances(instances=None)

Lock all or a list of instances.

name
reflect_attrs()

reflect all attributes of boto3’s vpc resource object into self.

region_name
reload()

run the reload method on the attached instance and reflect_attrs.

revoke_inbound_rules_from_sg(sg)
revoke_outbound_rules_from_sg(sg)
revoke_security_group_rules(sg)
roles
start_instances(instances=None, wait=True)

Start all or a list of instances.

stop_instances(instances=None, wait=True)

Stop all or a list of instances.

tag_dict
taggable_resources

Return a list of taggable objects related to this VPC.

terminate()

Terminate all resources related to this VPC!

unlock_instances(instances=None)

Unlock all or a list of instances.

wait_until_instances(instances=None, state=None)
vpc_endpoint.py
class botoform.enriched.vpc_endpoint.EnrichedVpcEndpoint(evpc)
create_all(route_table_names)

Accept route table names, create all endpoints for route tables.

Delete all VPC endpoints related to this VPC.

Return VPC endpoint descriptions related to this VPC.

related_ids()

Return VPC endpoint ids related to this VPC.

services()

Return a list of available VPC endpoint services.

plugins

You may extend the bf tool by writing a plugin.

A bf plugin must take one of two forms:

function plugin

This example shows how to write a function plugin for destroying VPCs.

First we define an entry point and subcommand named destroy.

setup.py:

entry_points = {
  'botoform.plugins' : [
    'destroy = mybotoform.plugins.destroy:destroy',
  ]
}

The entry point named destroy shows the path to the destroy function.

bf function plugins must accept an args object and an evpc object.

For Example:

mybotoform/plugins/destroy.py:

def destroy(args, evpc):
    """
    Destroy a VPC and related resources and services.

    :param args: The parsed arguments and flags from the CLI.
    :param evpc: An instance of :meth:`botoform.enriched.vpc.EnrichedVPC`.

    :returns: None
    """
    evpc.terminate()
class plugin

This example shows how to write a class plugin to dump instances to STDOUT.

We show how to structure your code to define additional subparser arguments.

Note

Use a function plugin instead of a class plugin if your subcommand does not need to define additional flags or args.

setup.py:

entry_points = {
  'botoform.plugins' : [
    'dump-instances = mybotoform.plugins.dump:Instances',
  ]
}

mybotoform/plugins/dump.py:

from botoform.util import output_formatter

class Instances(object):
    """Output a list of instance names. (example botoform plugin)"""

    @staticmethod
    def setup_parser(parser):
        parser.add_argument('--output-format',
          choices=['csv', 'yaml', 'json', 'newline'], default='newline',
          help='the desired format of any possible output')

    @staticmethod
    def main(args, evpc):
        """Output a list of instance names. (example botoform plugin)"""
        instances = evpc.instances
        print(output_formatter(map(str, instances), args.output_format))

This is a special class that has two staticmethods.

setup_parser:
accepts subcommand parser and allows additional flags and args to be defined.
main:
main logic for this plugin.

Regardless of the form of plugin you choose, your plugin project’s setup.py must define an entry point in the botoform.plugins group. The name of the entry point will be the subcommand on the CLI.

All bf subcommands (core plugins) are implemented in this way.

See also

working examples plugins directory.

core plugins
core plugins

The Botoform bf tool ships with many core plugins and subcommands.

botoform.plugins.create
class botoform.plugins.create.Create

Create a new VPC and related services, modeled from YAML template.

This is a class plugin for the bf tool.

static main(args, evpc=None)

Creates a new VPC and related services, modeled from a YAML template.

Parameters:
Returns:

None
static setup_parser(parser)

Accepts a subparser and attaches additional arguments and flags.

Parameters:parser – An ArgumentParser sub parser. Reference: https://docs.python.org/3/library/argparse.html
Returns:None
botoform.plugins.destroy
botoform.plugins.destroy.destroy(args, evpc)

Destroy a VPC and related resources and services.

Parameters:
Returns:

None
botoform.plugins.lock
botoform.plugins.lock.lock(args, evpc)

Lock all instances in VPC to prevent termination.

Parameters:
Returns:

None
botoform.plugins.unlock
botoform.plugins.unlock.unlock(args, evpc)

Unock all instances in VPC to allow termination.

Parameters:
Returns:

None
botoform.plugins.dump
class botoform.plugins.dump.Dump

Dump AWS resourses as Botoform Schema.

This is a class plugin for the bf tool.

static main(args, evpc)
static setup_parser(parser)
botoform.plugins.dump.ansible_hosts(args, evpc)

Output an Ansible host inventory for the VPC.

Parameters:
Returns:

Ansible inventory to standard out.
botoform.plugins.dump.instances(args, evpc)

Output instance roles to standard out in Botoform Schema.

Parameters:
Returns:

instance_roles to standard out in Botoform Schema.
botoform.plugins.dump.security_groups(args, evpc)

Output Security Groups to standard out in Botoform Schema.

Parameters:
Returns:

security_groups to standard out in Botoform Schema.
botoform.plugins.repl
botoform.plugins.repl.REPL(args, evpc)

Open an interactive REPL (read-eval-print-loop) with access to evpc object

Parameters:
Subcommand aliases:
 

cli and shell
Returns:

Interactive shell with evpc botoform.enriched.vpc.EnrichedVPC().
terms
entry point
entry point:

An entry point is a Python object identified in a project’s setup.py file. The object is referenced by group and name to make it discoverable.

This means that another Python application can search for installed software. During the search, often the entry point group filters relevant objects.

Botoform uses this method to allow plugins to load at run time.