Python for System Administrators¶

virtualenv¶

The term “virtualenv” can refer to the command virtualenv, used to create a virtual environment, or to the virutal environment itself.

$ sudo apt-get install python-virtualenv

virtualenvwrapper¶

virtualenvwrapper is a set of extensions to the virtualenv tool. The extensions include wrappers for creating and deleting virtual environments and otherwise managing your development workflow, making it easier to work on more than one project at a time without introducing conflicts in their dependencies [1].

$ sudo apt-get install virtualenvwrapper
$ mkvirtualenv sysadmin
New python executable in sysadmin/bin/python
Installing distribute.............................................................................................................................................................................................done.
Installing pip...............done.
virtualenvwrapper.user_scripts creating /home/jason/.virtualenvs/sysadmin/bin/predeactivate
virtualenvwrapper.user_scripts creating /home/jason/.virtualenvs/sysadmin/bin/postdeactivate
virtualenvwrapper.user_scripts creating /home/jason/.virtualenvs/sysadmin/bin/preactivate
virtualenvwrapper.user_scripts creating /home/jason/.virtualenvs/sysadmin/bin/postactivate
virtualenvwrapper.user_scripts creating /home/jason/.virtualenvs/sysadmin/bin/get_env_details
(sysadmin)$ pip freeze # A few packages are installed by default
argparse==1.2.1
distribute==0.6.24
wsgiref==0.1.2

Note that when the virtualenv is active, its name (in this case “sysadmin”) is prepended to the shell prompt:

$ # Ordinary shell prompt
(sysadmin)$ # Virtualenv "sysadmin" is active

If later you have logged out, and want to activate this virtualenv, you can use the workon command:

$ workon sysadmin
(sysadmin)$

You can deactivate the virtualenv with the deactivate command:

(sysadmin)$ deactivate
$ # Back to normal shell prompt

Location of Virtualenvs¶

By default, virtualenvwrapper stores your virtualenvs in ~/.virtualenvs. However you can control this by setting the WORKON_HOME environment variable. This could potentially be used for shared virtualenvs, perhaps with group write permission.

export WORKON_HOME=/path/to/virtualenvs

Aptana / PyDev¶

The Python plugin for Eclipse, called PyDev, is now part of Aptana Studio. Aptana can be installed as a seperate download, or as an Eclipse plugin. For convenience we will download the whole application.

http://aptana.com/products/studio3/download

Installing Eclipse Plugins¶

Each Eclipse plugin has an Update Site URL, from which it can be installed.

To install a plugin in Eclipse, choose Install New Software... from the Help menu. Click the Add... button to add a new plugin repository. Put the plugin’s Update Site URL in the Location: field.

Once you have added the plugin repository, check the box of the plugin you want to install. Click Next >, then click thru until it is installed. Normally Eclipse will want to restart itself after a new plugin has been installed.

Vwrapper¶

Vrapper is an Eclipse plugin providing VI-keys support. Only install this plugin if you are certain you want it.

Update site:

http://vrapper.sourceforge.net/update-site/stable

Django Plugins¶

These packages extend/enhance the Django framework:

REST - Representational State Transfer¶

REpresentational State Transfer (REST) is a style of software architecture for distributed systems such as the World Wide Web. REST has emerged as a predominant Web service design model. [1]

The term representational state transfer was introduced and defined in 2000 by Roy Fielding in his doctoral dissertation. Fielding is one of the principal authors of the Hypertext Transfer Protocol (HTTP) specification versions 1.0 and 1.1.

Conforming to the REST constraints is generally referred to as being “RESTful”.

The REST Constraints¶

The REST architectural style describes the following six constraints applied to the architecture, while leaving the implementation of the individual components free to design: [2]

Client–server

A uniform interface separates clients from servers. This separation of concerns means that, for example, clients are not concerned with data storage, which remains internal to each server, so that the portability of client code is improved. Servers are not concerned with the user interface or user state, so that servers can be simpler and more scalable. Servers and clients may also be replaced and developed independently, as long as the interface between them is not altered.

Stateless

The client–server communication is further constrained by no client context being stored on the server between requests. Each request from any client contains all of the information necessary to service the request, and any session state is held in the client.

Cacheable

As on the World Wide Web, clients can cache responses. Responses must therefore, implicitly or explicitly, define themselves as cacheable, or not, to prevent clients reusing stale or inappropriate data in response to further requests. Well^managed caching partially or completely eliminates some client–server interactions, further improving scalability and performance.

Layered system

A client cannot ordinarily tell whether it is connected directly to the end server, or to an intermediary along the way. Intermediary servers may improve system scalability by enabling load^balancing and by providing shared caches. They may also enforce security policies.

Code on demand (optional)

Servers are able temporarily to extend or customize the functionality of a client by the transfer of executable code. Examples of this may include compiled components such as Java applets and client^side scripts such as JavaScript.

Uniform interface

The uniform interface between clients and servers, discussed below, simplifies and decouples the architecture, which enables each part to evolve independently. The four guiding principles of this interface are detailed below.

The only optional constraint of REST architecture is code on demand. If a service violates any other constraint, it cannot strictly be considered RESTful.

Complying with these constraints, and thus conforming to the REST architectural style enables any kind of distributed hypermedia system to have desirable emergent properties, such as performance, scalability, simplicity, modifiability, visibility, portability, and reliability.

JSON - Javascript Object Notation¶

JSON, or JavaScript Object Notation, is a text-based open standard designed for human-readable data interchange. It is derived from the JavaScript scripting language for representing simple data structures and associative arrays, called objects. Despite its relationship to JavaScript, it is language-independent, with parsers available for many languages. [3]

The JSON format was originally specified by Douglas Crockford, and is described in RFC 4627. The official Internet media type for JSON is application/json. The JSON filename extension is .json.

The JSON format is often used for serializing and transmitting structured data over a network connection. It is used primarily to transmit data between a server and web application, serving as an alternative to XML.

Basic Usage¶

Typical use involves creating a Python file named fabfile.py, containing one or more functions, then executing them via the fab command-line tool. Below is a small but complete fabfile.py containing a single task:

from fabric.api import run

def host_type():
    run('uname -s')

Once a task is defined, it may be run on one or more servers, like so:

(sysadmin)$ fab -H applebox,linuxbox host_type
[applebox] run: uname -s
[applebox] out: Darwin
[linuxbox] run: uname -s
[linuxbox] out: Linux

Done.
Disconnecting from localhost... done.
Disconnecting from linuxbox... done.

Library Usage¶

In addition to use via the fab tool, Fabric’s components may be imported into other Python code, providing a Pythonic interface to the SSH protocol suite at a higher level than that provided by e.g. the ssh library (which Fabric itself uses.) [3]

Consider the case where we want to collect average uptime from a list of hosts:

 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26

from fabric import tasks

env.hosts = ['localhost', 'sunflower.heliotropic.us']
pattern = re.compile(r'up (\d+) days')

# No need to decorate this function with @task
def uptime():
    res = run('uptime')
    match = pattern.search(res)
    if match:
        days = int(match.group(1))
        env['uts'].append(days)

def main():
    env['uts'] = []
    tasks.execute(uptime)
    uts_list = env['uts']
    if not uts_list:
        return # Perhaps we should print a notice here?
    avg = sum(uts_list) / float(len(uts_list))
    print '-' * 80
    print 'Average uptime: %s days' % avg
    print '-' * 80

if __name__ == '__main__':
    main()

Footnotes

The print statement¶

Sometimes the quickest/easiest solution is just to throw a few print statements into your code. Not a good idea for complex problems. Some very good programmers disdain this technique, calling it sloppy. However others really very good programmers, such as Rob Pike (early contributor to Unix, father of Plan 9, UTF-8, and Go) is said to approve of this method.

IPython¶

IPython is an interactive interpreter shell with more features than the default Python REPL shell.

While not used for debugging per se, an interactive session can be a good way to understand and explore small bits of code.

(venv)$ pip install ipython

Debugger¶

A debugger is a computer program that lets you run your program, line by line and examine the values of variables or look at values passed into functions and let you figure out why it isn’t running the way you expected it to. [1]

Logging¶

While running code in a debugger offers the maximum visibility into its operation, it may be difficult or impossible to correctly simulate a production environment (including number of connections, network topography, load, etc) while a program is running inside a debugger.

Logging statements offer an alternative way to gain insight into your code’s operation. Verbose logging can be especially helpful for understanding bugs in highly concurrent code, where it would be difficult to inspect each thread/process in an ordinary debugger.

Logging is an essential part of 12-Factor methodology for building modern applications.

Eclipse¶

Eclipse offers a graphical debugger, making it easy to explore your code and set breakpoints while running in the debugger.

Note: Eclipse’s PyDev debugger is powerful, but notoriously flakey. Expect a few hiccups!

pdb¶

Indentation¶

Usually results from mixture of tabs & spaces, causing the actual scope of some lines to be different than it appears on the programmer’s screen.

Wrong Number of Arguments¶

A function is called with the wrong number (too few or too many) arguments, causing an exception to be thrown.

Wrong Package¶

You type import foobar, and the module imported is foobar.py in the local folder, not the foobar module in the standard library.

Catchign Multiple Exceptions¶

Be careful catching multiple exception types: [2]

try:
    raise KeyError("hmm bug")
except KeyError, TypeError:
    print TypeError

This prints “hmm bug”, though it is not a bug; it looks like we are catching exceptions of both types, but instead we are catching KeyError only as variable TypeError.

PythonTutor can help us visualize what is happening.

The correct way to catch multiple exceptions is to put them in parentheses:

try:
    raise KeyError("hmm bug")
except (KeyError, TypeError):
    print TypeError

Visualize the correct program flow.

Unqualified except: block¶

Do you really want to catch all exceptions? Can your except block really recover from all the exceptions it catches?

Populating Arrays¶

When you need a population of arrays you might be tempted to type something like this: [3]

>>> a=[[1,2,3,4,5]]*4

And sure enough it will give you what you expect when you look at it

>>> from pprint import pprint
>>> pprint(a)

[[1, 2, 3, 4, 5],
 [1, 2, 3, 4, 5],
 [1, 2, 3, 4, 5],
 [1, 2, 3, 4, 5]]

But don’t expect the elements of your population to be seperate objects:

>>> a[0][0] = 2
>>> pprint(a)

[[2, 2, 3, 4, 5],
 [2, 2, 3, 4, 5],
 [2, 2, 3, 4, 5],
 [2, 2, 3, 4, 5]]

Unless this is what you need...

Visualize what is happening.

It is worth mentioning a workaround:

a = [[1,2,3,4,5] for i in range(4)]

Footnotes