ZStack User Manual¶

The Setup of A Single Management Node¶

In the simplest setup, all ZStack software and third party dependencies are installed on a single Linux server. A typical setup includes five parts:

• RabbitMQ Message Server: The central message bus ZStack services use for communication.
• MySQL Database: The database ZStack stores metadata for resources in the cloud.
• Ansible: The configuration management tool ZStack uses to remotely deploy and upgrade agents.
• ZStack Management Node: The main process encompassing all ZStack orchestration services.
• ZStack UI Server: A web server providing user interface for end users.

Besides, several python agents, which need deploying to local or remote machines at runtime, are packaged in the WAR file of ZStack management node and are deployed using Ansible.

Because of ZStack’s asynchronous architecture, a single management node is normally enough to manage a big cloud that may have tens of thousands of physical servers, hundreds of thousands of virtual machines(virtual machine is referred as VM in future chapters), and tens of thousands of concurrent API requests. However, in case of high availability and scaling out for a super large cloud, a setup of multiple management nodes is still valuable.

A Deployment of Multiple Management Nodes¶

In this multiple nodes setup, the RabbitMQ server and MySQL database server are moved out to dedicated Linux machines; ZStack management nodes and Ansible are installed on every Linux server; multiple management nodes share the same RabbitMQ message server and MySQL database. ZStack UI servers, which also send API requests to management nodes through RabbitMQ, are deployed behind a load balancer which dispatches requests from users.

In terms of clustering RabbitMQ and MySQL, admin can setup two RabbitMQ servers and an additional slave MySQL database server.

ZStack’s World View of A Cloud¶

IaaS software usually use some terms such as ‘zone’, ‘cluster’ to describe how facilities in a data center make up a cloud, so does ZStack. To reduce the learning curve and to eliminate misunderstandings caused by self-created terms, ZStack tries to use terminologies that have been well known in existing IaaS software and datacenters as much as possible.

Below is a diagram that how ZStack maps facilities of datacenters into its own language.

A datacenter, in ZStack’s terms, is organized as follows:

• Zone:

  A zone is a logic group of resources, such as clusters, L2 networks, primary storage. ZStacks uses zones to define visibility boundary between resources. For example, a primary storage in zone A is not visible to a cluster in zone B. In practice, zones can also be used as isolated domains for fault tolerance, just as Amazon EC2 availability zones.

• Cluster:

  A cluster is a logic group of hosts. Hosts in the same cluster must have the same operating systems(hypervisor) and network configurations. Clusters are also known as host aggregations or host pools in other IaaS software.

• Host:

  A host is a physical server installed with an operating system(hypervisor) to run VMs.

• L2 Network:

  A L2 network is an abstraction of a layer 2 broadcast domain. Any technology, as long as providing a layer 2 broadcast domain, can be a type of L2 Network in ZStack. For example, VLAN, VxLan, or SDN technologies that create layer 2 overlay on layer 3 network.

• Primary Storage:

  A primary storage provides disk spaces to store VMs’ volumes which will be accessed by VMs’ operating system during running. Primary Storage can be either filesystem based like NFS or block storage based like ISCSI.

• Backup Storage:

  A backup Storage provides disk spaces to store images and volume snapshots both of which can be used to create volumes. Files on backup storage are not directly accessible to VMs; before being used, they need to be downloaded to primary storage. Backup Storage can either be filesystem based or object storage based.

ZStack uses a so-called ‘attaching strategy’ to describe relationships between resources, for example, a cluster can be attached with multiple primary storage and L2 networks, vice versa. See related chapters(e.g. primary storage, L2 network) for details.

A data center can have one or more zones. A diagram of multiple zones looks like:

Besides above terms describing datacenter facilities, there are some other terms such as VM, instance offering, disk offering, which describe virtual resources; check details in relevant chapters.

Create Resources¶

Every resource has own creational APIs. There is one parameter ‘resourceUuid’ common to all creational APIs. When ‘resourceUuid’ is not null, ZStack will use its value as the UUID for the resource being created; otherwise ZStack will automatically generate a UUID.

Here is an example of creating a cluster:

CreateCluster name=cluster1 description='awesome cluster' hypervisorType=KVM zoneUuid=061141410a0449b6919b50e90d68b7cd

or:

CreateCluster resourceUuid=f31e38309e2047beac588e111fa2051f name=cluster1 description='awesome cluster' hypervisorType=KVM zoneUuid=061141410a0449b6919b50e90d68b7cd

Read Resources¶

Every resource has own query API that returns a list of inventories for read. For details, see Query. Here is an example of querying VMs:

QueryVmInstance allVolumes.type=Data allVolumes.size>1099511627776

The example does: finding out all VMs which have one or more data volumes with size greater than 1099511627776 bytes(1T)

Update Resources¶

A resource can be updated by various APIs. Updating a resource is actually performing an action to the resource. For example, starting a VM, stopping a VM. Please refer to corresponding chapters for actions for resources. Here is an example of starting a VM:

StartVmInstance uuid=94d991c631674b16be65bfdf28b9e84a

Most update APIs will return a resource inventory.

Delete Resources¶

A resource can be deleted. ZStack’s philosophy for deleting is: every resource should be deletable; and deleting a resource should always be success unless user allows an expected failure; for example, a plugin may allow user to set a ‘none-deletable’ tag on a VM, and throw an error when the VM is being deleted.

Deleting a resource is not always easy in IaaS, especially for a resource that has many descendants; some software hard code to delete all descendant resources; some software simply throws an error when a resource being deleted still has descendant resources alive.

ZStack handles deleting in an elegant way. When a resource is being deleted, a so-called Cascade Framework will calculate relationships among this resource and rest resources in the cloud, and propagate proper actions to related resources if necessary. For example, when deleting a zone, a deleting action will be spread to all descendants of the zone, which means all descendant resources like VMs, hosts, clusters, L2 Networks in this zone will be deleted before the zone deleted; and backup storage attached to the zone will be detached. With the cascade framework, deleting resources in ZStack is easy and reliable.

Every resource has own deleting API. A parameter deleteMode which has options Permissive and Enforcing is common to all deleting APIs. When deleteMode is set to Permissive, ZStack will stop the deleting if an error happens, or the deleting is not permitted; in this case, an error code with detailed reason will be returned. When deleteMode is set to Enforcing, ZStack will ignore any errors and permissions but delete resources directly; in this case, a deleting will always be success.

Here is an example of deleting a VM:

DestroyVmInstance uuid=94d991c631674b16be65bfdf28b9e84a

Connect to ZStack management node¶

zstack-cli is installed by default after you install a ZStack management node. You can launch it by simply typing ‘zstack-cli’ in a shell console:

if no parameters are provided, zstack-cli will connect to 8080 port on localhost; to connect a remote ZStack management node, you can use options ‘-s’ and ‘-p’ to specify IP and port:

if you have a multi-node deployment, you can connect the zstack-cli to any management nodes.

Modes¶

zstack-cli can work in a command mode that receives parameters from shell, runs once, and prints results to the shell console, for example:

or an interactive shell mode that keeps a session for continuously executing, for example:

people usually prefer interactive mode for manual execution but command mode for script integration.

LogIn¶

In this ZStack version(0.6), the IAM(Identity and Access Management) system is not ready; only one account ‘admin’ with default password(‘password’) is available. Before executing any commands, you need to run the login command ‘LogInByAccount’ to get a session token which is automatically saved by zstack-cli to ~/.zstack/cli/session and you don’t need to keep it separately:

>>> LogInByAccount accountName=admin password=password

LogOut¶

Once you finish your work, you can use ‘LogOut’ to invalidate current session:

>>> LogOut

the LogOut command receives a parameter ‘sessionUuid’, but you don’t need to provide it as zstack-cli will retrieve it from where it’s kept.

Execute API Commands¶

Every API is a command with several parameters, you can execute them in either command mode or interactive mode:

>>> StartVmInstance uuid=11be8ac6adad44c68ae02493cba29846

[root@localhost ~]# zstack-cli StartVmInstance uuid=11be8ac6adad44c68ae02493cba29846

View Command History¶

You can use ‘more’ command to view your command history, for example:

>>> more

or:

[root@localhost ~]# zstack-cli more

the result format is the same to Linux more command, you can scroll up/down and search.

to view the details of a command, use ‘more’ command following a command number:

>>> more 6

or:

[root@localhost ~]# zstack-cli more 6

the result is like:

Export Command History¶

You can export command history by ‘save’ command, saving one history each time or multiple histories at once:

>>> save 1
Saved command: 1 result to file: /home/root/QueryZone-1.json

[root@localhost ~]# zstack-cli -s 192.168.0.212 save 1
Saved command: 1 result to file: /home/root/QueryZone-1.json

or:

>>>save 1,2,3
Saved command: 1 result to file: /home/root/QueryZone-1.json
Saved command: 2 result to file: /home/root/CreateZone-2.json
Saved command: 3 result to file: /home/root/LogInByAccount-3.json

[root@localhost ~]# zstack-cli -s 192.168.0.212 save 1,2,3
Saved command: 1 result to file: /home/root/QueryZone-1.json
Saved command: 2 result to file: /home/root/CreateZone-2.json
Saved command: 3 result to file: /home/root/LogInByAccount-3.json

by default results are saved to current working folder, you can specify a destination folder by supplying an extra parameter of folder path:

>>> save 1 /tmp
save history command 1 result to /tmp/COMMAND-1.json

Query API Parameters¶

Query Condition¶

Query APIs receive a list of query conditions which have properties as following:

a field name can be of a primitive field, or of a sub-field of a nested field, or of a sub-field of an expanded field(see Join); ‘op’ are comparison operators which are from SQL language.

The relation among conditions is logical AND, it’s the only relation supported in this ZStack version. For example:

QueryL3Network ipRanges.name=range1 name=L3Network1

is to find L3 networks whose names are ‘L3Network1’ AND which have one or more IP ranges with names ‘range1’.

CLI Query Conditions¶

There are two ways to write conditions in CLI, one is the original form of query API:

QueryHost conditions='[{"name":"name", "op":"=", "value":"KVM1"}]'

another is CLI form:

QueryHost name=KVM1

I am sure you will prefer the CLI form as it’s more intuitive and human readable. The CLI form always expresses query conditions in formula of:

condition_name(no_space)CLI_comparison_operator(no_space)condition_value

name=KVM1

name = KVM1

When typing in CLI, you can use Tab key for auto-completion and reminding you about queryable fields including primitive fields, nested fields, and expanded fields:

Join(Expanded Query)¶

Join is called expanded query in ZStack; it allows users to query a resource by fields that are neither primitive nor nested but other resources’ fields that have relation to this resource; those fields are called expanded fields in ZStack’s terms.

For example, to find the parent L3 network of a VM nic having an EIP with VIP 17.16.0.53:

QueryL3Network vmNic.eip.vipIp=17.16.0.53

here L3 network inventory has no field called ‘vmNic.eip.vipIp’; however, it has a relation to VM nic inventory that has a relation to EIP inventory; so we can construct an expanded query that spans to three inventories: L3 network inventory, VM nic inventory, and EIP inventory. Thanks for this nuclear weapon, ZStack has around four millions query conditions and countless combinations of conditions. Let’s see a more complex and artificial example:

QueryVolumeSnapshot volume.vmInstance.vmNics.l3Network.l2Network.attachedClusterUuids?=13238c8e0591444e9160df4d3636be82

This complex query is to find volume snapshots created from volumes of VMs that have nics on L3 networks whose parent L2 networks are attached to a cluster of uuid equal to 13238c8e0591444e9160df4d3636be82. Though users will barely do such a query, it shows the power of the query APIs.

Query List¶

When a field is a list, it can contain primitive types such as int, long, string or nested inventories. Querying list has nothing special; we have this section to remind you that don’t incorrectly think you can only use ‘in’(?=) and ‘not in’(!?=) when querying a list field; in fact, you can use all comparison operators; for example:

QueryL3Network dns~=72.72.72.%

is to find all L3 networks that have DNS like 72.72.72.*:

QueryL3Network ipRanges.startIp=192.168.0.10

is to find all L3 networks whose IP ranges starting with IP 192.168.0.10.

Query Tags¶

In section tags you will see every resource can have user tags and system tags both of which can be a part of query conditions. ZStack uses two special fields: __userTag__ and __systemTag__ for query; for example:

QueryVmInstance __userTag__?=web-tier-VMs

QueryHost __systemTag__?=os::distribution::Ubuntu managementIp=192.168.0.212

operators >, >=, <, <= only return resources that have tags matching specified conditions; ‘is not null’ returns resources that have tags; ‘is null’ returns resources that have no tags; !=, ‘not in’, ‘not like’ return resources that have tags not matching conditions as well as resources that have no tags.

QueryVmInstance __userTag__!=database  __userTag__!=null

Avoid Loop Query¶

Most ZStack resources have bi-direction expanded queries, for example, hosts have an expanded query to clusters and clusters also have an expanded query to hosts. This makes it’s possible to query a resource from any directions, which may also lead to loop queries. For example:

QueryHost vmInstance.vmNics.eip.vmNic.vmInstance.uuid=d40e459b97db5a63dedaffcd05cfe3c2

is a loop query, it does the thing equal to:

QueryHost vmInstance.uuid=d40e459b97db5a63dedaffcd05cfe3c2

Behaviors of loop queries is undefined; you may or may not get the correct results. Please avoid loop query in your practice.

Use Query Efficiently¶

Query APIs are powerful that you can do the same thing by different routes. For example, to find VMs that are running on the host of UUID e497e90ab1e64db099eea93f998d525b, you can either do:

QueryVmInstance hostUuid=e497e90ab1e64db099eea93f998d525b

or:

QueryVmInstance host.uuid=e497e90ab1e64db099eea93f998d525b

The first one is more efficient, because it queries a primitive field which only involves the VM table; the later one is an expanded query which joins both VM table and host table. When your query condition is a UUID, it’s always suggested querying the primitive field instead of the sub-field of an expanded field.

Normal Query¶

QueryL3Network name=L3-SYSTEM-PUBLIC

Query Count¶

QueryL3Network name=L3-SYSTEM-PUBLIC count=true

Normal Query With Count¶

QueryL3Network name=L3-SYSTEM-PUBLIC replyWithCount=true

Set Limit¶

QueryL3Network l2NetworkUuid=33107835aee84c449ac04c9622892dec limit=10

Set Start¶

QueryL3Network l2NetworkUuid=33107835aee84c449ac04c9622892dec start=10 limit=100

Select Fields¶

QueryL3Network fields=name,uuid l2NetworkUuid=33107835aee84c449ac04c9622892dec

Sort¶

QueryL3Network l2NetworkUuid=33107835aee84c449ac04c9622892dec sortBy=createDate sortDirection=desc

Example¶

{
    "category": "identity",
    "defaultValue": "500",
    "description": "Max number of sessions management server accepts. When this limit met, new session will be rejected",
    "name": "session.maxConcurrent",
    "value": "500"
}

Update Global Configurations¶

Admins can use UpdateGlobalConfig to update a global configuration. For example:

UpdateGlobalConfig category=host name=connection.autoReconnectOnError value=true

statistics.on¶

Whether enables statistics that count time consuming of each message through JMX.

node.heartbeatInterval¶

The interval that each management node writes heartbeat to database, in seconds.

node.joinDelay¶

If non zero, each management node will delay random seconds from 0 to ‘node.joinDelay’ before publishing join event on the message bus. This avoid storm of join event when a large number of management nodes start at the same time.

key.public¶

ZStack will inject this public SSH key to Linux servers that need to deploy agents; in this version, the Linux servers include KVM host, virtual router VMs, SFTP backup storage. After injecting, ZStack will use key.private when needing SSH login.

key.private¶

The private SSH key ZStack uses to SSH login remote Linux servers; see key.public.

Create Tags¶

There are two ways to create tags; for resources that have been created, users can use command CreateUserTag or CreateSystemTag to create a user tag or a system tag. For example:

CreateUserTag resourceType=DiskOfferingVO resourceUuid=50fcc61947f7494db69436ebbbefda34 tag=for-large-DB

CreateSystemTag resourceType=HostVO resourceUuid=50fcc61947f7494db69436ebbbefda34 tag=reservedMemory::1G

For a resource that is going to be created, as it’s not been created yet, there is no resource UUID that can be referred in the CreateUserTag and CreateSystemTag commands; in this case, users can use userTags and systemTags fields, both of which are of a list type that receives a list of tags, of every creational API command; for example:

CreateVmInstance name=testTag systemTags=hostname::web-server-1
userTags=in-super-data-center,has-public-IP,hot-fix-applied-2015-5-1
l3NetworkUuids=6572ce44c3f6422d8063b0fb262cbc62
instanceOfferingUuid=04b5419ca3134885be90a48e372d3895 imageUuid=f1205825ec405cd3f2d259730d47d1d8

Delete Tag¶

Users can use DeleteTag to delete a user tag or a non-inherent system tag. For example:

DeleteTag uuid=7813d03bb85840c489789f8df3a5915b

Query Tags¶

Users can use QueryUserTag to query user tags, for example:

QueryUserTag resourceUuid=0cd1ef8c9b9e0ba82e0cc9cc17226a26 tag~=web-server-%

or QuerySystemTag to query system tags, for example:

QuerySystemTag resourceUuid=50fcc61947f7494db69436ebbbefda34

QueryUserTag resourceUuid=0cd1ef8c9b9e0ba82e0cc9cc17226a26 resourceType=VmInstanceVO

Properties:¶

Example¶

{
  "uuid": "b729da71b1c7412781d5de22229d5e17",
  "name": "TestZone",
  "description": "Test",
  "state": "Enabled",
  "type": "zstack",
  "createDate": "Jun 1, 2015 6:04:52 PM",
  "lastOpDate": "Jun 1, 2015 6:04:52 PM"
}

State¶

Zones have two states: Enabled and Disabled. When changing a zone’s state, the operation will be cascaded to all clusters and hosts all of which belong to the zone. For example, disabling a zone will change states of all clusters and hosts in this zone to Disabled. Because no VM can be created or started on a disabled host, putting a zone into Disabled state can prevent any VM from being created or started in this zone.

Create Zone¶

Admins can use CreateZone command to create a new zone. For example:

CreateZone name='San Jose Zone' description='this is a zone in San Jose datacenter'

Delete Zone¶

Admins can use DeleteZone command to delete a zone. For example:

DeleteZone uuid=28e94936284b45f99842ababfc3f976d

Change State¶

Admins can use ChangeZoneState command to change the state of a zone. For example:

ChangeZoneState stateEvent=enable uuid=737896724f2645de9372f11b13a48223

Attach Backup Storage¶

see attach backup storage to zone.

Detach Backup Storage¶

see detach backup storage from zone.

Query Zone¶

Admins can use QueryZone to query zones. For example:

QueryZone name=zone1

QueryZone vmInstance.uuid=13238c8e0591444e9160df4d3636be82

System Tags¶

Properties¶

Example¶

{
  "inventory": {
    "uuid": "c1bd173d5cd84f0e9e7c47195ae27ec6",
    "name": "cluster1",
    "description": "test",
    "state": "Enabled",
    "zoneUuid": "1b830f5bd1cb469b821b4b77babfdd6f"
    "hypervisorType": "KVM",
    "lastOpDate": "Jun 1, 2015 5:54:09 PM",
    "createDate": "Jun 1, 2015 5:54:09 PM",
    "type": "zstack",
  }
}

Hypervisor Type¶

Hypervisor type indicates what hypervisor(operating system) installed on hosts in the cluster. In this ZStack version, the only supported hypervisor is KVM.

State¶

Cluster has two states: Enabled and Disabled, just like zone. When changing the state of a cluster, the operation will be spread to all hosts of the cluster; For example, disabling a cluster will disable all hosts in the cluster as well.

Create Cluster¶

Admins can use CreateCluster command to create a cluster. For example:

CreateCluster name=cluster1 hypervisorType=KVM zoneUuid=1b830f5bd1cb469b821b4b77babfdd6f

Delete Cluster¶

Admins can use DeleteCluster to delete a cluster. For example:

DeleteCluster uuid=c1bd173d5cd84f0e9e7c47195ae27ec6

Change State¶

Admins can use ChangeClusterState to change the state of a cluster. For example:

ChangeClusterState uuid=c1bd173d5cd84f0e9e7c47195ae27ec6 stateEvent=disable

Attach Primary Storage¶

Admins can use AttachPrimaryStorageToCluster command to attach a primary storage to a cluster. For example:

AttachPrimaryStorageToCluster clusterUuid=c1bd173d5cd84f0e9e7c47195ae27ec6 primaryStorageUuid=1b830f5bd1cb469b821b4b77babfdd6f

Detach Primary Storage¶

Admin cans use DetachPrimaryStorageFromCluster to detach a primary storage from a cluster. For example:

DetachPrimaryStorageFromCluster clusterUuid=c1bd173d5cd84f0e9e7c47195ae27ec6 primaryStorageUuid=1b830f5bd1cb469b821b4b77babfdd6f

Detaching primary storage is useful when admin wants to make a primary storage on longer accessible to a cluster. For example, in order to move VMs from a cluster equipped with aged hosts to a cluster with new, powerful hosts, admins can detach the primary storage on which root volumes of VMs locate from the old cluster and attach it to the new cluster, then start those stopped VMs; because the old cluster cannot access the primary storage anymore, ZStack will choose the new cluster to start VMs.

Attach L2 Network¶

Admin can use AttachL2NetworkToCluster command to attach a L2 network to a cluster. For example:

AttachL2NetworkToCluster clusterUuid=c1bd173d5cd84f0e9e7c47195ae27ec6 l2NetworkUuid=1b830f5bd1cb469b821b4b77babfdd6f

Detach L2 Network¶

Admins can use DetachL2NetworkFromCluster command to detach a L2 network from a cluster. For example:

DetachL2NetworkFromCluster clusterUuid=c1bd173d5cd84f0e9e7c47195ae27ec6 l2NetworkUuid=1b830f5bd1cb469b821b4b77babfdd6f

Detaching L2 networks can be useful when admins want to make network typology changes in datacenters. After hosts in a cluster no longer connect to a physical layer2 network, admin can detach the L2 network representing the physical layer2 network from the cluster.

Query Cluster¶

Admins can use QueryCluster to query clusters. For example:

QueryCluster hypervisorType=KVM

QueryCluster primaryStorage.availableCapacity>100000000

System Tags¶

Properties¶

Example¶

{
  "inventory": {
    "zoneUuid": "2893ce85c43d4a3a8d78f414da39966e",
    "name": "host1-192.168.0.203",
    "uuid": "43673938584447b2a29ab3d53f9d88d3",
    "clusterUuid": "8524072a4274403892bcc5b1972c2576",
    "description": "Test",
    "managementIp": "192.168.0.203",
    "hypervisorType": "KVM",
    "state": "Enabled",
    "status": "Connected",
    "createDate": "Jun 1, 2015 6:49:24 PM",
    "lastOpDate": "Jun 1, 2015 6:49:24 PM"
  }
}

Management IP¶

The management IP is used by ZStack management nodes to reach the operating systems(hypervisor) of hosts; depending on hypervisor types, it’s necessary or not. For example, in VMWare, the official way to reach an ESXi host is through the VCenter Server, then the management IP is not necessary; however, in KVM, ZStack will deploy an agent to the Linux operating system, then the management IP is necessary.

State¶

Hosts have four states:

• Enabled:

  the state that allows VMs to be created, started, or migrated to

• Disabled:

  the state that DOESN’T allow VMs to be created, started, or migrated to

• PreMaintenance:

  the intermediate state indicating host is entering Maintenance state. See maintenance mode.

• Maintenance:

  the state indicating host has been in maintenance mode.

A state transition diagram is like:

Status¶

A host’s status reflects the status of command channel between the host and a ZStack management node. Command channels are the ways that ZStack management nodes communicate with hosts to perform operations. For example, in KVM, command channels are the HTTP connections between ZStack management nodes and Python agents running on hosts; in VMWare, command channels are connections between the VCenter Server and ESXi hosts.

Hosts have three status:

• Connecting:

  A ZStack management node is trying to establish the command channel between itself and the host. No operations can be performed to the host.

• Connected

  The Command channel has been successfully established between a ZStack management node and the host. Operations can be performed to the host. This is the only status that a host can start or create VMs.

• Disconnected

  The Command channel has lost between a ZStack management node and the host. No operations can be performed to the host.

When booting, a ZStack management node will start the process of establishing the command channel to hosts it manages; in this stage, hosts’s status are Connecting; after command channels are established, hosts’ status change to Connected; if the management node fails to setup a command channel, or the command channel is detected as lost later on, the status of the host to which the command channel connect changes to Disconnected.

ZStack management nodes will periodically send ping commands to hosts to check health of command channels; once a host fails to respond, or a ping command times out, the host’s status changes to Disconnected.

A status transition diagram is like:

State and Status¶

There are no direct relations between states and status. States represent admin’s decisions to a host, while status represents communication condition of a host.

Add Host¶

The commands adding a host varies for different hypervisors.

AddKVMHost clusterUuid=8524072a4274403892bcc5b1972c2576 managementIp=192.168.10.10 name=kvm1 username=root password=password

Delete Host¶

Admins can use DeleteHost command to delete a host. For example:

DeleteHost uuid=2893ce85c43d4a3a8d78f414da39966e

Change Host State¶

Admins can use ChangeHostState command to change a host’s state. For example:

ChangeHostState stateEvent=preMaintain uuid=2893ce85c43d4a3a8d78f414da39966e

Reconnect Host¶

Admins can use ReconnectHost to re-establish the command channel between a ZStack management node and a host. For example:

ReconnectHost uuid=2893ce85c43d4a3a8d78f414da39966e

See status for details.

Query Host¶

Admins can use QueryHost to query hosts. For example:

QueryHost managementIp=192.168.0.100

QueryHost vmInstance.vmNics.ip=10.21.100.2

load.all¶

Whether to connect all hosts when management nodes boot. If set to true, management nodes will connect to all hosts simultaneously during booting time, which may exhaust resources of the machines running management nodes if there are a large number of hosts in the cloud; if set to false, accompanying with load.parallelismDegree, management nodes will connect a portion of hosts each time and repeat until all hosts are connected.

load.parallelismDegree¶

When load.all is set to false, this configuration defines the number of hosts that management nodes will connect simultaneously during booting time.

ping.timeout¶

The interval that management nodes periodically send ping commands to hosts in order to check connection status, in seconds.

ping.parallelismDegree¶

The parallel degree that management nodes send ping commands. If the amount of hosts are larger than this value, management nodes will repeat until all hosts are pinged. For example, ping first 100 hosts, then ping second 100 hosts ...

connection.autoReconnectOnError¶

Whether to reconnect hosts when their status change from Connected to Disconnected. If set to true, management nodes will reconnect hosts whose status change from Connected to Disconnected by ping commands, in order to catch up with operations missed during hosts in disconnected; if set to false, management nodes will not automatically reconnect them, admins may need to manually do it if necessary.

maintenanceMode.ignoreError¶

Whether to ignore errors happening during hosts enter maintenance mode. If set to true, errors are ignored and hosts always successfully enter maintenance mode; if set to false, hosts will fail to enter maintenance mode if any error happens, for example, failing to migrate a VM.

reservedCapacity.zoneLevel¶

Whether to enable host capacity reservation at zone level; see host capacity reservation.

reservedCapacity.clusterLevel¶

Whether to enable host capacity reservation at cluster level; see host capacity reservation.

reservedCapacity.hostLevel¶

Whether to enable host capacity reservation at host level; see host capacity reservation.

vm.migrationQuantity¶

The number that how many VMs can be migrated in parallel when KVM hosts enter maintenance mode.

reservedMemory¶

A string that memory capacity reserved on KVM hosts if reservedCapacity.hostLevel is set to true. The value is a number followed by a unit character that can be one of B/K/M/G/T; if no unit character followed, the number is treated as bytes.

dataVolume.maxNum¶

The max number of data volumes that can be attached to VMs of hypervisor type – KVM.

host.syncLevel¶

The max number of concurrent commands that can be simultaneously executed on KVM hosts.

System Tags¶

Properties¶

Example¶

{
  "inventory": {
    "uuid": "f4ac0a3119c94c6fae844c2298615d27",
    "zoneUuid": "f04caf351c014aa890126fc78193d063",
    "name": "nfs",
    "url": "192.168.0.220:/storage/nfs",
    "description": "Test Primary Storage",
    "totalCapacity": 10995116277768819,
    "availableCapacity": 10995162768,
    "type": "NFS",
    "state": "Enabled",
    "mountPath": "/opt/zstack/f4ac0a3119c94c6fae844c2298615d27",
    "createDate": "Jun 1, 2015 2:42:51 PM",
    "lastOpDate": "Jun 1, 2015 2:42:51 PM",
    "attachedClusterUuids": [
      "f23e402bc53b4b5abae87273b6004016",
      "4a1789235a86409a9a6db83f97bc582f",
      "fe755538d4e845d5b82073e4f80cb90b",
      "1f45d6d6c02b43bfb6196dcacb5b8a25"
    ]
  }
}

totalCapacity = NFS's total capacity
availableCapacity = totalCapacity - sum(volumes' virtual sizes)

ip-or-dns-name-of-nfs-server:/absolute-path-to-directory

192.168.0.220:/storage/nfs/

State¶

Primary storage has two states:

• Enabled:

  the state that allows volumes to be created

• Disabled:

  the state that DOESN’T allow volumes to be created

Status¶

Like host status, primary storage status reflect the status of command channels amid ZStack management nodes and primary storage. Command channels are the ways ZStack management nodes communicate with storage systems that primary storage represent; depending on primary storage types, for example, it can be HTTP connections among ZStack management nodes and primary storage agents or communication methods provided by storage SDKs.

There are three status:

• Connecting:

  A ZStack management node is trying to establish the command channel between itself and the primary storage. No operations can be performed to the primary storage.

• Connected

  The command channel has been successfully established between a ZStack management node and the primary storage. Operations can be performed to the primary storage.

• Disconnected

  The command channel has lost between a ZStack management node and the primary storage. No operations can be performed to the primary storage.

ZStack management nodes will try to establish command channels when booting and will periodically send ping commands to primary storage to check health of command channels during running; once a primary storage fails to respond, or a ping command times out, the command channel is considered as lost and the primary storage will be placed in Disconnected.

Here is the transition diagram:

State and Status¶

There is no direct relations between states and status. States represent admin’s decisions to primary storage, while status represent communication conditions of primary storage.

Attaching Cluster¶

Attaching clusters is to associate primary storage to sibling clusters, which provides a flexible way that manifests relations between hosts and storage systems in a real datacenter. Let’s see a concreted example; assuming you have a cluster (cluster A) attached to a NFS primary storage (NFS1), like below diagram:

Some time later, the cluster A is running out of memory but the primary storage still have plenty of disk spaces, so you decide to add another cluster (cluster B) which will also use NFS1; then you can create cluster B and attach NFS1 to it.

After running a while, the hardware of cluster A is getting outdated and you decide to retire them; you add a new powerful cluster (cluster C) attached to NFS1 and place all hosts in cluster A into maintenance mode, so all VMs running in cluster A are migrated to cluster B or cluster C; lastly, you detach NFS1 from cluster A and delete it. Now the datacenter looks like:

Finally, NFS1 starts running out of capacity, you add one more primary storage (NFS2), and attach it to both cluster B and cluster C.

Add Primary Storage¶

The commands adding a primary storage varies for different types of primary storage.

AddNfsPrimaryStorage name=nfs1 zoneUuid=1b830f5bd1cb469b821b4b77babfdd6f url=192.168.0.220:/storage/nfs

Delete Primary Storage¶

Admins can use DeletePrimaryStorage to delete a primary storage. For example:

DeletePrimaryStorage uuid=2c830f5bd1cb469b821b4b77babfdd6f

Change Primary Storage State¶

Admins can use ChangePrimaryStorageState to change the state of a primary storage. For example:

ChangePrimaryStorageState stateEvent=enable uuid=2c830f5bd1cb469b821b4b77babfdd6f

Attach Cluster¶

See Attach Primary Storage.

Detach Cluster¶

See Detach Primary Storage.

Query Primary Storage¶

Admins can use QueryPrimaryStorage to query primary storage. For example:

QueryPrimaryStorage totalCapacity<100000000000

QueryPrimaryStorage volumeSnapshot.uuid?=13238c8e0591444e9160df4d3636be82,33107835aee84c449ac04c9622892dec

mount.base¶

The mount point that NFS primary storage is mounted on the KVM hosts.

System Tags¶

Properties¶

Physical Interface¶

The physical interface is a string that contains information needed by a L2 network plugin for manipulating network system in a datacenter. The information encoded in physical interface is specific to L2 network types and hypervisor types of clusters that L2 networks may attach. This sounds a little complex. The complexity is originated from hypervisors using their own notations to describe L2 networks, and a L2 network can be attached to multiple clusters of different hypervisor types. A real world example may help to understand this.

Let’s say your datacenter has a L2 network (l2Network A) which spans to two clusters, one is a KVM cluster, another is a VMWare cluster. In KVM, the L2 network is realized by ethernet device in Linux operating system; in this example, let’s assume each eth0 of KVM hosts connects to the L2 network. In the VMWare cluster, the L2 network is realized by vswitch; in this example, let’s assume vswitch0 in the VMWare cluster connects to the L2 network; then the typology is like:

As mentioned in section host, lots of operations seemingly applied to clusters are actually delegated to hosts; Here, when attaching the L2 network A to the KVM cluster and the VMWare cluster, ZStack must understand what are notations describing the L2 network in those hypervisors of clusters; in this case, ZStack must know that on KVM hosts, eth0 is the representation of the L2 network, but on VMWare hosts, vswitch0 is the representation. Physical interface is the field that encodes those hypervisor specific information.

Attaching Cluster¶

Attaching cluster is to associate L2 networks to sibling clusters, which provides a flexible way that manifests relations between hosts and layer 2 networks in a real datacenter. Let’s see a concrete example.

Let’s assume the network typology in your datacenter is as above diagram. Eth0 of hosts in all clusters are on the same layer 2 network called L2 Network1; eth1 of cluster1 and cluster3 are on another layer 2 network called L2 network2. To describe this typology in ZStack, you can attach L2 network1 to all three clusters but attach L2 network2 to only cluster1 and cluster3.

A couple months later, the network typology needs changing because of business requirements, you unplug cables of eth1 of hosts in cluster3 from the rack switch, so cluster3 is not with L2 network2 anymore; you can detach the L2 network2 from cluster3 to notify ZStack about the network typology change.

L2NoVlanNetwork¶

L2NoVlanNetwork, whose properties are listed in properties is the base type of L2 Networks. The ‘NoVlan’ in the name DOESN’T mean the network cannot use VLAN technology, it only denotes that ZStack itself will not use VLAN to create a layer 2 broadcast domain in an active manner. To make it clear, take a look at below two diagrams:

In this setup, two switch ports 5 and 12 are untagged with VLAN 10(access port with VLAN 10 in Cisco term), and connect to eth0 on host1 and host2 respectively. This is a very valid setup matching to a L2NoVlanNetwork. Admin cans create a L2NoVlanNetwork with ‘physicalInterface’ = ‘eth0’ and attach it to the cluster.

In this setup, two switch ports 5 and 12 are tagged with VLAN 10(trunk port with VLAN 10 in Cisco term), and respectively connect to eth0.10 that is a pre-created VLAN device on host1 and host2. This is also a very valid setup matching to a L2NoVlanNetwork. Admins can create a L2NoVlanNetwork with ‘physicalInterface’ = ‘eth0.10’ and attach it to the cluster.

Now it should be understood that a L2NoVlanNetwork maps to a pre-created layer 2 broadcast domain; ZStack won’t create any new broadcast domain for L2NoVlanNetwork.

Assuming physicalInterface = eth0

brctl create br_eth0
brctl addif br_eth0 eth0

{
  "inventory": {
    "uuid": "f685ff94513542bbb8e814027f8deb13",
    "name": "l2-basic",
    "description": "Basic L2 Test",
    "zoneUuid": "45a2864b6ddf4d2fb9b4c3736a923dcb",
    "physicalInterface": "eth0",
    "type": "L2NoVlanNetwork",
    "createDate": "Jun 1, 2015 12:58:35 PM",
    "lastOpDate": "Jun 1, 2015 12:58:35 PM",
    "attachedClusterUuids": []
  }
}

L2VlanNetwork¶

A L2VlanNetwork is a L2 network that ZStack will actively use a VLAN to create a layer 2 broadcast domain. The ways that ZStack create layer 2 broadcast domains depend on hypervisor types of clusters, to which L2 networks are going to attach. In addition to properties, a L2VlanNetwork has one more property:

When attaching a L2VlanNetwork to a cluster, ZStack uses ‘vlan’ collaborating with ‘physicalInterface’ to create vlan devices on hosts in the cluster; in order to make this work, the switch ports to which ethernet devices identified by ‘physicalInterface’ connect must be tagged with ‘vlan’. For example:

In this setup, switch ports 5 and 12 have been tagged with VLAN 10, then admins can create a L2VlanNetwork with ‘physicalInterface’ = ‘eth0’ and ‘vlan’ = 10 and attach it to the cluster.

Assuming physicalInterface = eth0, vlan = 10

vconfig add eth0 10
brctl create br_eth0_10
brctl addif br_eth0_10 eth0.10

{
    "inventory": {
      "vlan": 10,
      "uuid": "14a01b0978684b2ea6e5a355c7c7fd73",
      "name": "TestL2VlanNetwork",
      "description": "Test",
      "zoneUuid": "c74f8ff8a4c5456b852713b82c034074",
      "physicalInterface": "eth0",
      "type": "L2VlanNetwork",
      "createDate": "Jun 1, 2015 4:31:47 PM",
      "lastOpDate": "Jun 1, 2015 4:31:47 PM",
      "attachedClusterUuids": []
    }
}

Create L2 Network¶

The commands creating L2 networks vary for different L2 network types.

Create L2NoVlanNetwork¶

Admins can use CreateL2NoVlanNetwork to create a L2NoVlanNetwork. For example:

CreateL2NoVlanNetwork name=management-network physicalInterface=eth0 zoneUuid=9a94e647a9f64bb392afcdc5396cc1e4

Delete L2 Network¶

Admins can use DeleteL2Network to delete a L2 network. For example:

DeleteL2Network uuid=a5535531eb7346ce89cfd7e643ad1ef8

Attach Cluster¶

See Attach L2 Network.

Detach Cluster¶

See Detach L2 Network.

Query L2 Network¶

Admins can use QueryL2Network to query L2 networks. For example:

QueryL2Network physicalInterface=eth0

QueryL2Network l3Network.ipRanges.startIp=192.168.0.2

Subnet¶

In a L3 network, the subnet can have a single consecutive IP range or multiple split IP ranges. The split IP ranges are typically useful when a portion of IP addresses needs to be reserved from the subnet. For example, let’s say you are going to create a L3 network as a management network that has a subnet 192.168.0.0/24; however, IP addresses of 192.168.0.50 ~ 192.168.0.100 have been occupied by some network devices and you don’t want ZStack to use them, then you can create two split IP ranges:

IP Range1

start IP: 192.168.0.2
end IP: 192.168.0.49
gateway: 192.168.0.1
netmask: 255.255.255.0

IP Range2

start IP: 192.168.0.101
end IP: 192.168.0.254
gateway: 192.168.0.1
netmask: 255.255.255.0

You can create split IP ranges as many as you want, as long as they all belong to the same CIDR.

Network Services¶

Network services implementing OSI layer 3 ~ layer 7 protocols aim to serve VMs on a L3 network. Network services are provided by network services providers that are associated to the parent L2 network of a L3 network. A type of network service can have multiple providers, and a provider can provide several types of different services. After a L3 network is created, users can attach network services to it and choose network services providers. In this ZStack version, a table of supported services/providers is shown as follows:

In the table, the column ‘Attachable L2 Network’ indicates what L2 networks providers can attach. If a provider cannot attach to a L2 network, it cannot provide services to child L3 networks of the L2 network.

Properties¶

Example¶

{
  "inventory": {
    "uuid": "f73926eb4f234f8195c61c33d8db419d",
    "name": "GuestNetwork",
    "description": "Test",
    "type": "L3BasicNetwork",
    "zoneUuid": "732fbb4383b24b019f60d862995976bf",
    "l2NetworkUuid": "f1a092c6914840c9895c564abbc55375",
    "state": "Enabled",
    "createDate": "Jun 1, 2015 11:07:24 PM",
    "lastOpDate": "Jun 1, 2015 11:07:24 PM",
    "dns": [],
    "ipRanges": [
      {
        "uuid": "78b43f4b0a9745fab49c967e1c35beb1",
        "l3NetworkUuid": "f73926eb4f234f8195c61c33d8db419d",
        "name": "TestIpRange",
        "description": "Test",
        "startIp": "10.10.2.100",
        "endIp": "10.20.2.200",
        "netmask": "255.0.0.0",
        "gateway": "10.10.2.1",
        "createDate": "Jun 1, 2015 11:07:24 PM",
        "lastOpDate": "Jun 1, 2015 11:07:24 PM"
      }
    ],
    "networkServices": [
      {
        "l3NetworkUuid": "f73926eb4f234f8195c61c33d8db419d",
        "networkServiceProviderUuid": "bbb525dc4cc8451295d379797e092dba",
        "networkServiceType": "DHCP"
      }
    ]
  }
}

State¶

L3 networks have two states:

• Enabled

  The state that allows new VMs to be created

• Disabled

  The state that DOESN’T allow new VMs to be created

  Note

  Existing VMs on disabled L3 networks can still be stopped, started, rebooted, and deleted.

DNS Domain¶

The DNS domain is used to expand hostnames of VMs on the L3 network to FQDNs(Full Qualified Domain Name); for example, if the hostname of a VM is ‘vm1’ and the DNS domain of the L3 network is ‘zstack.org’, the final hostname will be expanded to ‘vm1.zstack.org’.

IP Range¶

In this ZStack version, only IPv4 IP range is supported.

{
  "inventory": {
    "uuid": "b1cfcdeca4024d13ac82edbe8d959720",
    "l3NetworkUuid": "50e637dc68b7480291ba87cbb81d94ad",
    "name": "TestIpRange",
    "description": "Test",
    "startIp": "10.0.0.100",
    "endIp": "10.10.1.200",
    "netmask": "255.0.0.0",
    "gateway": "10.0.0.1",
    "createDate": "Jun 1, 2015 4:30:23 PM",
    "lastOpDate": "Jun 1, 2015 4:30:23 PM"
  }
}

DNS¶

A L3 network can have one or more DNS that take effect when the DNS network service is enabled.

L2 Networks and L3 Networks¶

As a layer2 broadcast domain can contain multiple subnets, nothing will stop you from creating multiple L3 networks on the same L2 network; however, those L3 networks are not isolated and network snooping can happen; please use on your own risks.

Network Service References¶

Network service references exhibit network services enabled on the L3 network and their providers.

{
  "l3NetworkUuid": "f73926eb4f234f8195c61c33d8db419d",
  "networkServiceProviderUuid": "bbb525dc4cc8451295d379797e092dba",
  "networkServiceType": "PortForwarding"
}

Create L3 Network¶

Users can use CreateL3Network to create a L3 network. For example:

CreateL3Network l2NetworkUuid=f1a092c6914840c9895c564abbc55375 name=GuestNetwork

Type¶

In this ZStack version, the only L3 network type is L3BasicNetwork. Users can leave field ‘type’ alone when calling CreateL3Network.

System L3 Network¶

A system L3 network is reserved for ZStack and cannot be used to create user VMs. System L3 networks are typically used for public networks and management networks. Usually, user VMs in a cloud should not have nics on a public network and a management network, but appliance VMs (e.g router VM) do need have nics on those networks; then the management network and the public network can be created as system L3 networks.

Delete L3 Network¶

Users can use DeleteL3Network to delete a L3 network. For example:

DeleteL3Network uuid=f73926eb4f234f8195c61c33d8db419d

Add IP Ranges¶

AddIpRange name=ipr1 startIp=192.168.0.2 endIp=192.168.0.100 netmask=255.255.255.0 gateway=192.168.0.1 resourceUuid=50e637dc68b7480291ba87cbb81d94ad

AddIpRangeByNetworkCidr name=ipr1 l3NetworkUuid=50e637dc68b7480291ba87cbb81d94ad networkCidr=10.0.1.0/24

network-number/prefix-length

Delete IP Range¶

Users can use DeleteIpRange to delete an IP range. For example:

DeleteIpRange uuid=b1cfcdeca4024d13ac82edbe8d959720

Add DNS¶

Users can use AddDnsToL3Network to add a DNS to a L3 network. For example:

AddDnsToL3Network l3NetworkUuid=50e637dc68b7480291ba87cbb81d94ad dns=8.8.8.8

Attach Network Service¶

After creating a L3 network and before creating any VMs on it, users can use AttachNetworkServiceToL3Network to attach network services to the L3 network. If a network service is attached to a L3 network that already has VMs running, the existing VMs can not use the network service until they are rebooted.

For example:

AttachNetworkServiceToL3Network l3NetworkUuid=50e637dc68b7480291ba87cbb81d94ad networkServices='{"1d1d5ff248b24906a39f96aa3c6411dd": ["DHCP", "DNS", "SNAT", "EIP"]}'

QueryNetworkServiceProvider fields=uuid name=VirtualRouter

QueryNetworkServiceProvider name=VirtualRouter

Query L3 Network¶

Users can use QueryL3Network to query L3 networks. For example:

QueryL3Network dnsDomain=zstack.org

QueryL3Network vmNic.ip=192.168.10.2

Properties¶

Example¶

{
    "backupStorageRefs": [
        {
            "backupStorageUuid": "8b99641a4d644820932e0ec5ada78eed",
            "createDate": "Jun 1, 2015 6:17:48 PM",
            "imageUuid": "b395386bdb4a4ff1b1850a457c949c5e",
            "installPath": "/export/backupStorage/sftp/templates/acct-36c27e8ff05c4780bf6d2fa65700f22e/b395386bdb4a4ff1b1850a457c949c5e/centos_400m_140925.template",
            "lastOpDate": "Jun 1, 2015 6:17:48 PM"
        }
    ],
    "createDate": "Jun 1, 2015 6:17:40 PM",
    "description": "Test Image Template for network test",
    "format": "qcow2",
    "guestOsType": "unknown",
    "lastOpDate": "Jun 1, 2015 6:17:40 PM",
    "md5Sum": "not calculated",
    "mediaType": "RootVolumeTemplate",
    "name": "image_for_sg_test",
    "platform": "Linux",
    "size": 419430400,
    "state": "Enabled",
    "status": "Ready",
    "system": false,
    "type": "zstack",
    "url": "http://172.16.0.220/templates/centos_400m_140925.img",
    "uuid": "b395386bdb4a4ff1b1850a457c949c5e"
},

State¶

Images have two states:

• Enabled:

  The state that allows VMs to be created from this image

• Disabled:

  The state that DOESN’T allow VMs to be created from this image

Status¶

Status indicates images’ lifecycle:

• Creating:

  The image is in process of creating from a volume or a volume snapshot; not ready to use.

• Downloading:

  The image is in process of downloading from a url; not ready to use.

• Ready:

  The image is on backup storage and ready to use.

URL¶

Depending on how an image was created on a backup storage, the url has different meanings; when an image was downloaded from a web server, the url is the HTTP/HTTPS link; when an image was created from a volume or a volume snapshot, the url is a string encoding UUID of the volume or the volume snapshot, like:

volume://b395386bdb4a4ff1b1850a457c949c5e
volumeSnapshot://b395386bdb4a4ff1b1850a457c949c5e

Media Type¶

A media type indicates the image’s usage.

• RootVolumeTemplate:

  The image is used to create root volumes.

• DataVolumeTemplate:

  The image is used to create data volumes.

• ISO:

  The image is used to install operating systems to blank root volumes.

Platform¶

Platform gives ZStack a hint that whether to use paravirtualization for VMs created from this image.

System Image¶

System images are images used only for appliance VMs but not for user VMs. This is normally used for virtual router image in this ZStack version.

Format¶

Format exhibits relationships between hypervisors and images. For example, images of format qcow2 can only be used for VMs of KVM. In this ZStack version, as KVM is the only supported hypervisor, the relationship table is like:

Volumes will inherit formats of images from which they are created; for example, root volumes created from images of format qcow2 will have format qcow2 too. Format ‘raw’ is an exception, volumes created from ‘raw’ images will have the format qcow2 because ZStack will thin-clone it using qcow2 format.

Backup Storage Reference¶

An image can be stored on more than one backup storage. For every backup storage, the image has a backup storage reference encompassing backup storage UUID and image’s installation path.

{
    "backupStorageUuid": "8b99641a4d644820932e0ec5ada78eed",
    "imageUuid": "b395386bdb4a4ff1b1850a457c949c5e",
    "installPath": "/export/backupStorage/sftp/templates/acct-36c27e8ff05c4780bf6d2fa65700f22e/b395386bdb4a4ff1b1850a457c949c5e/centos_400m_140925.template",
    "createDate": "Jun 1, 2015 6:17:48 PM",
    "lastOpDate": "Jun 1, 2015 6:17:48 PM"
}

Add Image¶

Admins can use AddImage to add an image. For example:

AddImage name=CentOS7 format=qcow2 backupStorageUuids=8b99641a4d644820932e0ec5ada78eed url=http://172.16.0.220/templates/centos7_400m_140925.img mediaType=RootVolumeTemplate platform=Linux