Data Analysis in Python¶

Numbers¶

There are three numeric types:

• integers (int)
• floating point (float)
• complex (complex)

Hopefully, we will not need the last one. But if you see something like 3+5j or 6-7J, you know you are looking at complex type.

Note that if you want to define a float, you have to use the dot (.), otherwise the output is an integer. For example,

>>> type(1)
<class 'int'>
>>> type(1.)
<class 'float'>
>>> type(float(1))
<class 'float'>
>>> type(int(1.))
<class 'int'>
>>> type(0)
<class 'int'>
>>> type(0.)
<class 'float'>
>>> type(.0)
<class 'float'>
>>> type(0.0)
<class 'float'>

This was extremely important in Python 2 and was the source of many inadvertent errors (try dividing 1 by 2 - you’d be surprised). With Python 3 not anymore, but the general advice of being explicit in what you mean is still there.

Division (/) always returns a float. To do floor division and get an integer result (discarding any fractional result) you can use the // operator; to calculate the remainder you can use %:

>>> 17 / 3  # classic division returns a float
5.666666666666667
>>> 17 // 3  # floor division discards the fractional part
5
>>> 17 % 3  # the % operator returns the remainder of the division
2
>>> 5 * 3 + 2  # result * divisor + remainder
17

Notice one way of commenting your code: just use # after the code and before any text.

Calculating powers is done with ** operator.

>>> 2**2
4
>>> 3**3
27
>>> 4**.5
2.0

Booleans¶

bool type is essential for any programming logic. Normally, truth and falcity are defined as True and False:

>>> x = True
>>> print(x)
True
>>> type(x)
<class 'bool'>
>>> y = False
>>> print(y)
False
>>> type(y)
<class 'bool'>

Additionally, all non-empty and non-zero values are interpreted by bool() function as True, while all empty and zero values are False:

>>> print(bool(1), bool(1.), bool(-.1))
True True True
>>> print(bool(0), bool(.0), bool(None), bool(''), bool([]))
False False False False False

Strings¶

Strings can be difined using both single ('...') or double quotes ("..."). Backslash can be used to escape quotes.

>>> 'spam eggs'  # single quotes
'spam eggs'
>>> 'doesn\'t'  # use \' to escape the single quote...
"doesn't"
>>> "doesn't"  # ...or use double quotes instead
"doesn't"
>>> '"Yes," he said.'
'"Yes," he said.'
>>> "\"Yes,\" he said."
'"Yes," he said.'
>>> '"Isn\'t," she said.'
'"Isn\'t," she said.'

The print() function produces a more readable output, by omitting the enclosing quotes and by printing escaped and special characters:

>>> '"Isn\'t," she said.'
'"Isn\'t," she said.'
>>> print('"Isn\'t," she said.')
"Isn't," she said.
>>> s = 'First line.\nSecond line.'  # \n means newline
>>> s  # without print(), \n is included in the output
'First line.\nSecond line.'
>>> print(s)  # with print(), \n produces a new line
First line.
Second line.

If you don’t want characters prefaced by \ to be interpreted as special characters, you can use raw strings by adding an r before the first quote:

>>> print('C:\some\name')  # here \n means newline!
C:\some
ame
>>> print(r'C:\some\name')  # note the r before the quote
C:\some\name

Python is very sensitive to code aesthetics (see Style Guide). In particular, you shoud restrict yourself to 79 characters in one line! Use parenthesis to break long strings:

>>> text = ('Put several strings within parentheses '
            'to have them joined together.')
>>> text
'Put several strings within parentheses to have them joined together.'

Strings can be constructed using math operators and by converting numbers into strings via str() function:

>>> 2 * 'a' + '_' + 3 * 'b' + '_' + 4 * (str(.5) + '_')
'aa_bbb_0.5_0.5_0.5_0.5_'

Note that Python can not convert numbers into strings automatically. Unless you use print() function or convert explicitly.:

>>> 'a' + 1
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: Can't convert 'int' object to str implicitly
>>> 'a' + str(1)
'a1'
>>> print('a', 1)
a 1

Lists¶

Lists are very convenient and simplest data containers. Here is how we store a collection of numbers in a variable:

>>> a = [1, 3, 5]
>>> a
[1, 3, 5]
>>> type(a)
<class 'list'>

Lists are not restricted to be uniform in types of their elements:

>>> b = [5, 2.3, 'abc', [4, 'b'], a, print]
>>> b
[5, 2.3, 'abc', [4, 'b'], [1, 3, 5], <built-in function print>]

Lists can be modified:

>>> a[1] = 4
>>> a
[1, 4, 5]

Lists can be merged or repeated:

>>> a + a
[1, 4, 5, 1, 4, 5]
>>> 3 * a
[1, 4, 5, 1, 4, 5, 1, 4, 5]

You can add one item to the end of the list inplace:

>>> a.append(7)
>>> a
[1, 4, 5, 7]

or add a few items:

>>> a.extend([0, 2])
>>> a
[1, 4, 5, 7, 0, 2]

Note the difference:

>>> a = [1, 3, 5]
>>> b = [1, 3, 5]
>>> a.append([2, 4, 6])
>>> b.extend([2, 4, 6])
>>> a
[1, 3, 5, [2, 4, 6]]
>>> b
[1, 3, 5, 2, 4, 6]

If the end of the list is not what you want, insert the element after a specified position:

>>> a.insert(1, .5)
>>> a
[1, 0.5, 4, 5, 7, 0, 2]

There are at least two methods to remove elements from a list:

>>> x = ['a', 'b', 'c', 'b']
>>> x.remove('b')
>>> x
['a', 'c', 'b']
>>> x.remove('b')
>>> x
['a', 'c']
>>> x.remove('b')
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ValueError: list.remove(x): x not in list

or:

>>> y = ['a', 'b', 'c', 'b']
>>> y.pop()
'b'
>>> y
['a', 'b', 'c']
>>> y.pop(1)
'b'
>>> y
['a', 'c']

Here is how you sort a list without altering the original object, and inplace:

>>> x = ['a', 'b', 'c', 'b', 'a']
>>> sorted(x)
['a', 'a', 'b', 'b', 'c']
>>> x
['a', 'b', 'c', 'b', 'a']
>>> x.sort()
>>> x
['a', 'a', 'b', 'b', 'c']

Tuples¶

On the first glance tuples are very similar to lists. The difference in definition is the usage of parentheses () (or even without them) instead of square brackets []:

>>> t = 12345, 54321, 'hello!'
>>> t
(12345, 54321, 'hello!')
>>> type(t)
<class 'tuple'>
>>> t = (12345, 54321, 'hello!')
>>> t
(12345, 54321, 'hello!')

The main difference is that tuples are immutable (impossible to modify):

>>> t[0] = 10
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
TypeError: 'tuple' object does not support item assignment

Here are the reasons you want to use tuples:

• Tuples are faster than lists. If you’re defining a constant set of values and all you’re ever going to do with it is iterate through it, use a tuple instead of a list.
• It makes your code safer if you “write-protect” data that doesn’t need to be changed.
• Some tuples can be used as dictionary keys (specifically, tuples that contain immutable values like strings, numbers, and other tuples). Lists can never be used as dictionary keys, because lists are not immutable.

Dictionaries¶

A dictionary is an unordered set of key-value pairs. There are some restrictions on what can be a key. In general, keys can not be mutable objects. Keys must be unique. Below are a few example of dictionary initialization:

>>> empty_dict = dict()
>>> empty_dict
{}
>>> empty_dict = {}
>>> empty_dict
{}
>>> type(empty_dict)
<class 'dict'>
>>> grades = {'Ivan': 4, 'Olga': 5}
>>> grades
{'Ivan': 4, 'Olga': 5}
>>> grades['Petr'] = 'F'
>>> grades
{'Ivan': 4, 'Petr': 'F', 'Olga': 5}
>>> grades['Olga']
5

Keys and values can be accessed separately if needed:

>>> grades.keys()
dict_keys(['Ivan', 'Olga'])
>>> grades.values()
dict_values([4, 5])

Sets¶

A set is an unordered collection of unique values. A single set can contain values of any immutable datatype. Once you have two sets, you can do standard set operations like union, intersection, and set difference. Here is a brief demonstration:

>>> basket = {'apple', 'orange', 'apple', 'pear', 'orange', 'banana'}
>>> basket
{'orange', 'banana', 'pear', 'apple'}
>>> type(basket)
<class 'set'>
>>> 'orange' in basket
True
>>> 'crabgrass' in basket
False

Let’s create a second set and see what we can do with both:

>>> bag = {'banana', 'peach'}
>>> basket - bag
{'apple', 'orange', 'pear'}
>>> basket | bag
{'peach', 'orange', 'pear', 'banana', 'apple'}
>>> basket & bag
{'banana'}
>>> basket ^ bag
{'peach', 'apple', 'orange', 'pear'}

if/elif/else¶

Writing conditional statements in Python is very easy. Start from if, continue with elif, and finish with else. For example,

In [1]: if 2**2 == 4:
   ...:     print('Should be True')
   ...:
Should be True

Be careful to respect the indentation depth. The Ipython shell automatically increases the indentation depth after a column : sign; to decrease the indentation depth, go four spaces to the left with the Backspace key. Press the Enter key twice to leave the logical block.

In [1]: a = 10

In [2]: if a == 1:
   ...:     print(1)
   ...: elif a == 2:
   ...:     print(2)
   ...: elif a == 3:
   ...:     print(3)
   ...: else:
   ...:     print('A lot')
   ...:
A lot

Besides checking for equality as in the previous examples, you can check for other statements evaluating to bool. These are comparison operators: <, >, <=, =>. Testing for equality of two objects is done with is operator:

>>> a, b = 1, 1.
>>> a == b
True
>>> a is b
False

You can test whether an object belongs to a collection using in operator. Note that if a collection is of type dict, then the search is done over dictionaries:

>>> a = [1, 2, 4]
>>> 2 in a, 4 in a
(True, True)
>>> b = {'a': 3, 'c': 8}
>>> 'c' in b
True

Loops¶

If you do need to iterate over a sequence of numbers, the built-in function range() comes in handy. It generates arithmetic progressions:

In [7]: for i in range(4):
   ...:     print(i)
   ...:
0
1
2
3

The for statement in Python differs a bit from what you may be used to in other programming languages. Rather than always iterating over an arithmetic progression of numbers (like in Pascal), or giving the user the ability to define both the iteration step and halting condition (as in C), Python’s for statement iterates over the items of any sequence (a list or a string), in the order that they appear in the sequence.

In [6]: words = ['cat', 'window', 'bird']
   ...: for w in words:
   ...:     print(w, len(w))
   ...:
cat 3
window 6
bird 4

Here is another example.

In [1]: for letter in 'Python':
   ...:     print(letter)
   ...:
P
y
t
h
o
n

Coming back to range() function. It can have at most three arguments, range(first, last, step). Given this knowledge we can generate various sequences. Note that this function returns neither a list not a tuple. In fact, it is an object itself. In order to check what are the indices if we use range in a for loop, we can convert it to list using list() function. The reason behind this behavior is to save memory: range does not store the whole list, only its definition.

>>> list(range(2, 10))
[2, 3, 4, 5, 6, 7, 8, 9]
>>> list(range(2, 10, 3))
[2, 5, 8]
>>> list(range(-2, -10, -3))
[-2, -5, -8]

If you need to break out of the loop or skip an iteration, then you need to know two statements, break and continue, respectively.

In [3]: a = [1, 0, 2, 4]
   ...: for element in a:
   ...:     if element == 0:
   ...:         continue
   ...:     print(1. / element)
   ...:
1.0
0.5
0.25

or

In [4]: a = [1, 0, 2, 4]
   ...: for element in a:
   ...:     if element == 0:
   ...:         break
   ...:     print(1. / element)
   ...:
1.0

Common use case is to iterate over items while keeping track of current index. Quick and dirty way to do this is:

In [5]: words = ('cool', 'powerful', 'readable')
   ...: for i in range(0, len(words)):
   ...:     print(i, words[i])
   ...:
0 cool
1 powerful
2 readable

Yet, Python provides a much more elegant approach:

In [7]: for index, item in enumerate(words):
   ...:     print(index, item)
   ...:
0 cool
1 powerful
2 readable

Try iterating over dictionaries yourslef. You should find out that Python iterates over keys only. In order to have access to the whole pair, one should use items() method:

In [1]: grades = {'Ivan': 4, 'Olga': 5, 'Petr': 4.5}
   ...: for key, val in grades.items():
   ...:     print('%s has grade: %s' % (key, val))
   ...:
Ivan has grade: 4
Petr has grade: 4.5
Olga has grade: 5

Here is how you might compute Pi:

In [1]: pi = 2
   ...: for i in range(1, 1000):
   ...:     pi *= 4*i**2 / (4*i**2 - 1)
   ...: print(pi)
   ...:
3.1408069608284657

Or if you want to stop after certain precision was achieved (a common use case), you might want to use while loop:

In [3]: pi, error, i = 2, 1e10, 1
   ...: while error > 1e-3:
   ...:     pi *= 4*i**2 / (4*i**2 - 1)
   ...:     error = abs(pi - 3.141592653589793)
   ...:     i += 1
   ...: print(pi)
   ...:
3.1405927760475945

List comprehensions¶

List comprehensions provide a concise way to create lists. Common applications are to make new lists where each element is the result of some operations applied to each member of another sequence or iterable, or to create a subsequence of those elements that satisfy a certain condition.

For example, assume we want to create a list of squares, like:

In [1]: squares = []
   ...: for x in range(10):
   ...:     squares.append(x**2)
   ...: print(squares)
   ...:
[0, 1, 4, 9, 16, 25, 36, 49, 64, 81]

As always, Python has a more elegant solution with the same result:

>>> squares = [x**2 for x in range(10)]

List comprehensions can include more for statements and even if statements:

>>> [(x, y) for x in [1,2,3] for y in [3,1,4] if x != y]
[(1, 3), (1, 4), (2, 3), (2, 1), (2, 4), (3, 1), (3, 4)]

which creates a list of pairs with distinct elements. Equivalenly, one could write this over several lines:

In [1]: combs = []
   ...: for x in [1,2,3]:
   ...:     for y in [3,1,4]:
   ...:         if x != y:
   ...:             combs.append((x, y))
   ...: print(combs)

Below are a few more examples:

>>> vec = [-4, -2, 0, 2, 4]
>>> # create a new list with the values doubled
>>> [x*2 for x in vec]
[-8, -4, 0, 4, 8]
>>> # filter the list to exclude negative numbers
>>> [x for x in vec if x >= 0]
[0, 2, 4]
>>> # apply a function to all the elements
>>> [abs(x) for x in vec]
[4, 2, 0, 2, 4]
>>> # call a method on each element
>>> freshfruit = ['  banana', '  loganberry ', 'passion fruit  ']
>>> [weapon.strip() for weapon in freshfruit]
['banana', 'loganberry', 'passion fruit']
>>> # create a list of 2-tuples like (number, square)
>>> [(x, x**2) for x in range(6)]
[(0, 0), (1, 1), (2, 4), (3, 9), (4, 16), (5, 25)]
>>> # flatten a list using a listcomp with two 'for'
>>> vec = [[1,2,3], [4,5,6], [7,8,9]]
>>> [num for elem in vec for num in elem]
[1, 2, 3, 4, 5, 6, 7, 8, 9]

Finally, we can transpose a “matrix” represented as a list of lists in the following several ways.

In [1]: matrix = [
   ...:     [1, 2, 3, 4],
   ...:     [5, 6, 7, 8],
   ...:     [9, 10, 11, 12],
   ...: ]

First, the longest but clearest:

In [1]: transposed = []
   ...: for i in range(4):
   ...:     transposed_row = []
   ...:     for row in matrix:
   ...:         transposed_row.append(row[i])
   ...:     transposed.append(transposed_row)
   ...: print(transposed)
   ...:
[[1, 5, 9], [2, 6, 10], [3, 7, 11], [4, 8, 12]]

Next uses one list comprehension:

In [1]: transposed = []
   ...: for i in range(4):
   ...:     transposed.append([row[i] for row in matrix])

Or, one single nested list comprehension:

>>> [[row[i] for row in matrix] for i in range(4)]

And, finally, the most elegant (in the context of standard library) way:

>>> list(zip(*matrix))
[(1, 5, 9), (2, 6, 10), (3, 7, 11), (4, 8, 12)]

Function definition¶

The simplest function definition is illustrated in the following example:

def simplest_function():
    print('I\'m your function!')

which after calling produces the following output:

>>> simplest_function()
I'm your function!

The keyword def introduces a function definition. It must be followed by the function name and the parenthesized list of formal parameters. The statements that form the body of the function start at the next line, and must be indented.

A slightly more complicated example to compute Fibbonaci series:

def fib(n):
    """Print a Fibonacci series up to n."""
    a, b = 0, 1
    while a < n:
        print(a, end=' ')
        a, b = b, a+b

Let’s try and call this function to find out all Fibonacci numbers up to 2000:

>>> fib(2000)
0 1 1 2 3 5 8 13 21 34 55 89 144 233 377 610 987 1597

Notice the line below function name in tripled quotes. This is called docstring. We will come back to it in Documenenting your code.

The result of running function above is just a screen output. If we try to assign the result of this function to a new variable, we will only get None:

>>> out = fib(0)
>>> print(out)
None

What if you want to store the result? Then you have to use return statement and say explicitely what your function should produce in the end.

def fib(n):
    """Print a Fibonacci series up to n and return the result."""
    result = []
    a, b = 0, 1
    while a < n:
        result.append(a)
        a, b = b, a+b
    return result

Now let’s try this function instead:

>>> out = fib(2000)
>>> print(out)
[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, 233, 377, 610, 987, 1597]

Now variable out is non-empty. It holds the list of Fibonacci numbers.

Above examples have shown how to define functions without any arguments and with just one argument. In fact, function definition is much more flexible than that. Read on.

Positional arguments¶

Passing several arguments to a function is done with their order in mind.

def power(x, a):
    """Take a power"""
    return x**a

If you make a mistake in the order of arguments, the function has no way to see that:

>>> print(power(2, 3), power(3, 2))
8 9

Default argument values¶

Some arguments may have default values. This is used to simplify function calls especially if arguments are numerous. Default arguments always follow positional ones.

def power(x, a=2):
    """Take a power"""
    return x**a

Here is how you call it:

>>> print(power(2), power(3))
4 9

The default values are evaluated once at function definition.

i = 5

def fun(arg=i):
    print(arg)

i = 6

The call to this function prduces:

>>> fun()
5

The side effect is that the default value is shared between the calls:

def fun(a, L=[]):
    L.append(a)
    return L

print(fun(1))
print(fun(2))
print(fun(3))

This prints

[1]
[1, 2]
[1, 2, 3]

Here is one possible way to overcome this:

def f(a, L=None):
    if L is None:
        L = []
    L.append(a)
    return L

Keyword arguments¶

If keeping the order of the arguments becomes a problem, then keyword (or optional) arguments are here to help. These are the same arguments with default values but redefined in function calls.

def slicer(seq, start=None, stop=None, step=None):
        return seq[start:stop:step]

This function has three default values. They all follow the variable without default. Here are a few examples of using this function:

>>> print(rhyme)
['one', 'fish,', 'two', 'fish,', 'red', 'fish,', 'blue', 'fish']
>>> print(slicer(rhyme))
['one', 'fish,', 'two', 'fish,', 'red', 'fish,', 'blue', 'fish']
>>> print(slicer(rhyme, step=2))
['one', 'two', 'red', 'blue']
>>> print(slicer(rhyme, 1, step=2))
['fish,', 'fish,', 'fish,', 'fish']
>>> print(slicer(rhyme, stop=4, step=2, start=1))
['fish,', 'fish,']
>>> print(slicer(rhyme, 1, 4, 2))
['fish,', 'fish,']

The following are invalid calls:

>>> slicer()                     # required argument missing
>>> slicer(start=2, 'Python')    # non-keyword argument after a keyword argument
>>> slicer('Python', 2, start=3) # duplicate value for the same argument
>>> slicer(actor='John Cleese')  # unknown keyword argument

Arbitrary argument lists¶

If you do not know in advance how many arguments you will need to pass to a function, then you can use function definition as follows:

def fun(var, *args, **kwargs):
    print('First mandatory argument:', var)
    if len(args) > 0:
        print('\nOptional positional arguments:')
    for idx, arg in enumerate(args):
        print('Argument number "%s" is "%s"' % (idx, arg))
    if len(kwargs) > 0:
        print('\nOptional keyword arguments:')
    for key, value in kwargs.items():
        print('Argument called "%s" is "%s"' % (key, value))

Calling this function produces:

>>> fun(2, 'a', 'Python', method='OLS', limit=1e2)
First mandatory argument: 2

Optional positional arguments:
Argument number "0" is "a"
Argument number "1" is "Python"

Optional keyword arguments:
Argument called "method" is "OLS"
Argument called "limit" is "100.0"

At the same time, calling this function with the only mandatory argument results in a much simple output:

>>> fun(2)
First mandatory argument: 2

Placing a star in front of args makes interpreter to expect a tuple of arbitrary length which is then unpacked to separate arguments. Placing two stars in front of kwargs makes Python unpack it as a dictionary into key-value pairs. So, you can pass arguments as tuples and dictionaries which sometimes significantly improves readability of the code. The following lines produce the same output as in the first example of this subsection:

>>> args = ('a', 'Python')
>>> kwargs = {'method': 'OLS', 'limit': 1e2}
>>> fun(2, *args, **kwargs)

Lambda functions¶

Small anonymous functions can be created with the lambda keyword. Lambda functions can be used wherever function objects are required. They are restricted to be one-liners. Here is an example of a function that returns another function:

def make_power(n):
    return lambda x: x ** n

And the way to use it is as follows:

>>> power = make_power(3)
>>> print(power(0), power(2))
0 8

Another example shows how to pass a function as an argument without formally defining it:

>>> pairs = [(1, 'one'), (2, 'two'), (3, 'three'), (4, 'four')]
>>> pairs.sort(key=lambda pair: pair[1])
>>> pairs
[(4, 'four'), (1, 'one'), (3, 'three'), (2, 'two')]

Passing by value¶

In Python parameters to functions are references to objects, which are passed by value. When you pass a variable to a function, Python passes the reference to the object to which the variable refers (the value). Not the variable itself.

If the value passed in a function is immutable, the function does not modify the caller’s variable. If the value is mutable, the function may modify the caller’s variable in-place:

def try_to_modify(x, y, z):
    x = 23 # immutable object
    y.append(42)
    z = [99] # reference to new object
    print(x, y, z)

Here is what happens if we call this function:

>>> a = 77    # immutable variable
>>> b = [99]  # mutable variable
>>> c = [28]
>>> try_to_modify(a, b, c)
23 [99, 42] [99]
>>> print(a, b, c)
77 [99, 42] [28]

No Copy at All¶

Simple assignments make no copy of array objects or of their data.:

>>> a = arange(12)
>>> b = a  # no new object is created
>>> b is a  # a and b are two names for the same ndarray object
True
>>> b.shape = 3, 4  # changes the shape of a
>>> a.shape
(3, 4)

Python passes mutable objects as references, so function calls make no copy.

>>> def f(x):
...     # id is a unique identifier of an object
...     print id(x)
...
>>> id(a)
148293216
>>> f(a)
148293216

View or Shallow Copy¶

Different array objects can share the same data. The view method creates a new array object that looks at the same data.

>>> c = a.view()
>>> c is a
False
>>> c.base is a  # c is a view of the data owned by a
True
>>> c.flags.owndata
False
>>> c.shape = 2, 6  # a's shape doesn't change
>>> a.shape
(3, 4)
>>> c[0, 4] = 1234  # a's data changes
>>> a
array([[   0,    1,    2,    3],
       [1234,    5,    6,    7],
       [   8,    9,   10,   11]])

Slicing an array returns a view of it:

>>> s = a[:, 1:3]
>>> s[:] = 10  # s[:] is a view of s. Note the difference between s=10 and s[:]=10
>>> a
array([[   0,   10,   10,    3],
       [1234,   10,   10,    7],
       [   8,   10,   10,   11]])

Deep Copy¶

The copy method makes a complete copy of the array and its data.:

>>> d = a.copy()  # a new array object with new data is created
>>> d is a
False
>>> d.base is a  # d doesn't share anything with a
False
>>> d[0, 0] = 9999
>>> a
array([[   0,   10,   10,    3],
       [1234,   10,   10,    7],
       [   8,   10,   10,   11]])

Basic operations¶

With scalars:

>>> a = np.array([1, 2, 3, 4])
>>> a + 1
array([2, 3, 4, 5])
>>> 2**a
array([ 2,  4,  8, 16])

All arithmetic operates elementwise:

>>> b = np.ones(4) + 1
>>> a - b
array([-1.,  0.,  1.,  2.])
>>> a * b
array([ 2.,  4.,  6.,  8.])
>>> c = np.arange(5)
>>> 2**(c + 1) - c
array([ 2,  3,  6, 13, 28])

Array multiplication is not matrix multiplication:

>>> c = np.ones((3, 3))
>>> c * c
array([[ 1.,  1.,  1.],
       [ 1.,  1.,  1.],
       [ 1.,  1.,  1.]])
>>> c.dot(c)
array([[ 3.,  3.,  3.],
       [ 3.,  3.,  3.],
       [ 3.,  3.,  3.]])

Other operations¶

Comparisons:

>>> a = np.array([1, 2, 3, 4])
>>> b = np.array([4, 2, 2, 4])
>>> a == b
array([False,  True, False,  True], dtype=bool)
>>> a > b
array([False, False,  True, False], dtype=bool)

Array-wise comparisons:

>>> a = np.array([1, 2, 3, 4])
>>> b = np.array([4, 2, 2, 4])
>>> c = np.array([1, 2, 3, 4])
>>> np.array_equal(a, b)
False
>>> np.array_equal(a, c)
True
>>> np.allclose(a, a + a*1e-5)
True
>>> np.allclose(a, a + a*1e-4)
False

Logical operations:

>>> a = np.array([1, 1, 0, 0], dtype=bool)
>>> b = np.array([1, 0, 1, 0], dtype=bool)
>>> np.logical_or(a, b)
array([ True,  True,  True, False], dtype=bool)
>>> np.logical_and(a, b)
array([ True, False, False, False], dtype=bool)

The where function forms a new array from two arrays of equivalent size using a Boolean filter to choose between elements of the two. Its basic syntax is where(boolarray, truearray, falsearray):

>>> a = np.array([1, 3, 0], float)
>>> np.where(a != 0, 1 / a, a)
array([ 1. , 0.33333333, 0. ])

Broadcasting can also be used with the where function:

>>> np.where(a > 0, 3, 2)
array([3, 3, 2])

Broadcasting¶

Arrays that do not match in the number of dimensions will be broadcasted by Python to perform mathematical operations. This often means that the smaller array will be repeated as necessary to perform the operation indicated. Consider the following:

>>> a = np.array([[1, 2], [3, 4], [5, 6]], float)
>>> b = np.array([-1, 3], float)
>>> a
array([[ 1., 2.],
       [ 3., 4.],
       [ 5., 6.]])
>>> b
array([-1., 3.])
>>> a + b
array([[ 0., 5.],
       [ 2., 7.],
       [ 4., 9.]])

Here, the one-dimensional array b was broadcasted to a two-dimensional array that matched the size of a. In essence, b was repeated for each item in a, as if it were given by:

array([[-1., 3.],
       [-1., 3.],
       [-1., 3.]])

Python automatically broadcasts arrays in this manner. Sometimes, however, how we should broadcast is ambiguous. In these cases, we can use the newaxis constant to specify how we want to broadcast:

>>> a = np.zeros((2,2), float)
>>> b = np.array([-1., 3.], float)
>>> a
array([[ 0., 0.],
       [ 0., 0.]])
>>> b
array([-1., 3.])
>>> a + b
array([[-1., 3.],
       [-1., 3.]])
>>> a + b[np.newaxis, :]
array([[-1., 3.],
       [-1., 3.]])
>>> a + b[:, np.newaxis]
array([[-1., -1.],
       [ 3., 3.]])

Series¶

Series is a one-dimensional labeled array capable of holding any data type (integers, strings, floating point numbers, Python objects, etc.). The axis labels are collectively referred to as the index. The basic method to create a Series is to call:

>>> s = Series(data, index=index)

The first mandatory argument can be

• array-like
• dictionary
• scalar

In [4]: s = pd.Series(randn(5), index=['a', 'b', 'c', 'd', 'e'])

In [5]: s
Out[5]: 
a    0.982107
b    0.129563
c    0.226421
d    0.872921
e    0.711250
dtype: float64

In [6]: s.index
Out[6]: Index(['a', 'b', 'c', 'd', 'e'], dtype='object')

In [7]: pd.Series(randn(5))
Out[7]: 
0    0.786909
1    0.620836
2    0.250509
3    0.364346
4    0.141489
dtype: float64

In [8]: d = {'a' : 0., 'b' : 1., 'c' : 2.}

In [9]: pd.Series(d)
Out[9]: 
a    0.0
b    1.0
c    2.0
dtype: float64

In [10]: pd.Series(d, index=['b', 'c', 'd', 'a'])
Out[10]: 
b    1.0
c    2.0
d    NaN
a    0.0
dtype: float64

In [11]: pd.Series(5., index=['a', 'b', 'c', 'd', 'e'])
Out[11]: 
a    5.0
b    5.0
c    5.0
d    5.0
e    5.0
dtype: float64

In [12]: s[0]
Out[12]: 0.98210746077411037

In [13]: s[:3]
Out[13]: 
a    0.982107
b    0.129563
c    0.226421
dtype: float64

In [14]: s[s > s.median()]
Out[14]: 
a    0.982107
d    0.872921
dtype: float64

In [15]: s[[4, 3, 1]]
Out[15]: 
e    0.711250
d    0.872921
b    0.129563
dtype: float64

In [16]: s + s
Out[16]: 
a    1.964215
b    0.259127
c    0.452842
d    1.745842
e    1.422500
dtype: float64

In [17]: s * 2
Out[17]: 
a    1.964215
b    0.259127
c    0.452842
d    1.745842
e    1.422500
dtype: float64

In [18]: np.exp(s)
Out[18]: 
a    2.670077
b    1.138331
c    1.254103
d    2.393893
e    2.036536
dtype: float64

In [19]: s[1:] + s[:-1]
Out[19]: 
a         NaN
b    0.259127
c    0.452842
d    1.745842
e         NaN
dtype: float64

In [20]: s['a']
Out[20]: 0.98210746077411037

In [21]: s['e'] = 12.

In [22]: s
Out[22]: 
a     0.982107
b     0.129563
c     0.226421
d     0.872921
e    12.000000
dtype: float64

In [23]: 'e' in s
Out[23]: True

In [24]: 'f' in s
Out[24]: False

In [25]: s = pd.Series(np.random.randn(5), name='random series')

In [26]: s
Out[26]: 
0   -0.189764
1   -1.342810
2    0.231476
3   -0.173812
4   -0.957039
Name: random series, dtype: float64

In [27]: s.name
Out[27]: 'random series'

DataFrame¶

DataFrame is a 2-dimensional labeled data structure with columns of potentially different types. Like Series, DataFrame accepts many different kinds of input:

• Dict of 1D ndarrays, lists, dicts, or Series
• 2-D numpy.ndarray
• A Series
• Another DataFrame

Along with the data, you can optionally pass index (row labels) and columns (column labels) arguments. If you pass an index and / or columns, you are guaranteeing the index and / or columns of the resulting DataFrame. Thus, a dict of Series plus a specific index will discard all data not matching up to the passed index.

If axis labels are not passed, they will be constructed from the input data based on common sense rules.

In [28]: d = {'one' : pd.Series([1., 2., 3.], index=['a', 'b', 'c']),
   ....:      'two' : pd.Series([1., 2., 3., 4.], index=['a', 'b', 'c', 'd'])}
   ....: 

In [29]: df = pd.DataFrame(d)

In [30]: df
Out[30]: 
   one  two
a  1.0  1.0
b  2.0  2.0
c  3.0  3.0
d  NaN  4.0

In [31]: pd.DataFrame(d, index=['d', 'b', 'a'])
Out[31]: 
   one  two
d  NaN  4.0
b  2.0  2.0
a  1.0  1.0

In [32]: pd.DataFrame(d, index=['d', 'b', 'a'], columns=['two', 'three'])
Out[32]: 
   two three
d  4.0   NaN
b  2.0   NaN
a  1.0   NaN

In [33]: df.index
Out[33]: Index(['a', 'b', 'c', 'd'], dtype='object')

In [34]: df.columns
Out[34]: Index(['one', 'two'], dtype='object')

In [35]: d = {'one' : [1., 2., 3., 4.], 'two' : [4., 3., 2., 1.]}

In [36]: pd.DataFrame(d)
Out[36]: 
   one  two
0  1.0  4.0
1  2.0  3.0
2  3.0  2.0
3  4.0  1.0

In [37]: pd.DataFrame(d, index=['a', 'b', 'c', 'd'])
Out[37]: 
   one  two
a  1.0  4.0
b  2.0  3.0
c  3.0  2.0
d  4.0  1.0

In [38]: data2 = [{'a': 1, 'b': 2}, {'a': 5, 'b': 10, 'c': 20}]

In [39]: pd.DataFrame(data2)
Out[39]: 
   a   b     c
0  1   2   NaN
1  5  10  20.0

In [40]: pd.DataFrame(data2, index=['first', 'second'])
Out[40]: 
        a   b     c
first   1   2   NaN
second  5  10  20.0

In [41]: pd.DataFrame(data2, columns=['a', 'b'])
Out[41]: 
   a   b
0  1   2
1  5  10

In [42]: pd.DataFrame({('a', 'b'): {('A', 'B'): 1, ('A', 'C'): 2},
   ....:               ('a', 'a'): {('A', 'C'): 3, ('A', 'B'): 4},
   ....:               ('a', 'c'): {('A', 'B'): 5, ('A', 'C'): 6},
   ....:               ('b', 'a'): {('A', 'C'): 7, ('A', 'B'): 8},
   ....:               ('b', 'b'): {('A', 'D'): 9, ('A', 'B'): 10}})
   ....: 
Out[42]: 
       a              b      
       a    b    c    a     b
A B  4.0  1.0  5.0  8.0  10.0
  C  3.0  2.0  6.0  7.0   NaN
  D  NaN  NaN  NaN  NaN   9.0

Head and Tail¶

To view a small sample of a Series or DataFrame object, use the head() and tail() methods. The default number of elements to display is five, but you may pass a custom number.

In [46]: long_series = pd.Series(np.random.randn(1000))

In [47]: long_series.head()
Out[47]: 
0   -0.178060
1    0.867918
2    1.836369
3   -0.214974
4   -1.463309
dtype: float64

In [48]: long_series.tail(3)
Out[48]: 
997   -1.014987
998    1.227988
999    1.388823
dtype: float64

Attributes and the raw values¶

Pandas objects have a number of attributes enabling you to access the metadata

• shape: gives the axis dimensions of the object, consistent with ndarray

• Axis labels

  • Series: index (only axis)
  • DataFrame: index (rows) and columns

Note, these attributes can be safely assigned to!

In [49]: df[:2]
Out[49]: 
                   A         B         C
2000-01-01 -0.651168 -0.630427 -0.073821
2000-01-02  0.772458  0.481733 -0.669856

In [50]: df.columns = [x.lower() for x in df.columns]

In [51]: df
Out[51]: 
                   a         b         c
2000-01-01 -0.651168 -0.630427 -0.073821
2000-01-02  0.772458  0.481733 -0.669856
2000-01-03 -0.337150  0.667284 -0.107571
2000-01-04  2.009189 -1.049017 -0.516699
2000-01-05 -0.333968  0.713391  0.215411
2000-01-06  0.925424  0.486207 -0.918720
2000-01-07 -0.376341  0.194480  0.630616
2000-01-08 -1.390901  0.508826 -0.514008

To get the actual data inside a data structure, one need only access the values property:

In [52]: s.values
Out[52]: array([ 1.58298669, -2.33882916, -1.27643236, -0.91038164, -0.04600865])

In [53]: df.values
Out[53]: 
array([[-0.65116827, -0.63042718, -0.07382086],
       [ 0.77245824,  0.48173331, -0.66985579],
       [-0.33714961,  0.66728365, -0.10757056],
       [ 2.00918853, -1.04901662, -0.51669886],
       [-0.33396841,  0.71339139,  0.21541096],
       [ 0.92542418,  0.48620719, -0.91872048],
       [-0.3763413 ,  0.19447994,  0.6306162 ],
       [-1.39090104,  0.50882621, -0.51400773]])

Descriptive statistics¶

A large number of methods for computing descriptive statistics and other related operations on Series and DataFrame. Most of these are aggregations (hence producing a lower-dimensional result) like sum(), mean(), and quantile(), but some of them, like cumsum() and cumprod(), produce an object of the same size. Generally speaking, these methods take an axis argument, just like ndarray.{sum, std, ...}, but the axis can be specified by name or integer:

• Series: no axis argument needed
• DataFrame: “index” (axis=0, default), “columns” (axis=1)

In [54]: df = pd.DataFrame({'one' : pd.Series(np.random.randn(3), index=['a', 'b', 'c']),
   ....:                    'two' : pd.Series(np.random.randn(4), index=['a', 'b', 'c', 'd']),
   ....:                    'three' : pd.Series(np.random.randn(3), index=['b', 'c', 'd'])})
   ....: 

In [55]: df.mean(0)
Out[55]: 
one     -0.108969
three   -0.074926
two     -0.048194
dtype: float64

In [56]: df.mean(1)
Out[56]: 
a   -0.463770
b    0.795855
c   -0.521810
d   -0.319528
dtype: float64

All such methods have a skipna option signaling whether to exclude missing data (True by default):

In [57]: df.sum(0, skipna=False)
Out[57]: 
one           NaN
three         NaN
two     -0.192776
dtype: float64

In [58]: df.sum(axis=1, skipna=True)
Out[58]: 
a   -0.927540
b    2.387564
c   -1.565429
d   -0.639056
dtype: float64

Combined with the broadcasting / arithmetic behavior, one can describe various statistical procedures, like standardization (rendering data zero mean and standard deviation 1), very concisely:

In [59]: ts_stand = (df - df.mean()) / df.std()

In [60]: ts_stand.std()
Out[60]: 
one      1.0
three    1.0
two      1.0
dtype: float64

In [61]: xs_stand = df.sub(df.mean(1), axis=0).div(df.std(1), axis=0)

In [62]: xs_stand.std(1)
Out[62]: 
a    1.0
b    1.0
c    1.0
d    1.0
dtype: float64

Series also has a method nunique() which will return the number of unique non-null values:

In [63]: series = pd.Series(np.random.randn(500))

In [64]: series[20:500] = np.nan

In [65]: series[10:20] = 5

In [66]: series.nunique()
Out[66]: 11

Summarizing data: describe¶

There is a convenient describe() function which computes a variety of summary statistics about a Series or the columns of a DataFrame:

In [67]: series = pd.Series(np.random.randn(1000))

In [68]: series[::2] = np.nan

In [69]: series.describe()
Out[69]: 
count    500.000000
mean       0.019138
std        1.011299
min       -2.984094
25%       -0.628704
50%        0.074347
75%        0.648606
max        2.952020
dtype: float64

In [70]: frame = pd.DataFrame(np.random.randn(1000, 5),
   ....:                      columns=['a', 'b', 'c', 'd', 'e'])
   ....: 

In [71]: frame.ix[::2] = np.nan

In [72]: frame.describe()
Out[72]: 
                a           b           c           d           e
count  500.000000  500.000000  500.000000  500.000000  500.000000
mean    -0.004104    0.049340    0.025288    0.026288    0.021081
std      0.954792    1.027652    1.021062    1.033627    1.030866
min     -3.018188   -2.564202   -3.542306   -2.885479   -3.012862
25%     -0.707369   -0.694912   -0.599675   -0.685300   -0.657108
50%      0.009843    0.105978    0.026260    0.030318    0.030185
75%      0.646093    0.757424    0.673980    0.688099    0.741316
max      2.790769    2.892441    3.771537    3.262996    3.237205

You can select specific percentiles to include in the output:

In [73]: series.describe(percentiles=[.05, .25, .75, .95])
Out[73]: 
count    500.000000
mean       0.019138
std        1.011299
min       -2.984094
5%        -1.674625
25%       -0.628704
50%        0.074347
75%        0.648606
95%        1.669375
max        2.952020
dtype: float64

For a non-numerical Series object, describe() will give a simple summary of the number of unique values and most frequently occurring values:

In [74]: s = pd.Series(['a', 'a', 'b', 'b', 'a', 'a', np.nan, 'c', 'd', 'a'])

In [75]: s.describe()
Out[75]: 
count     9
unique    4
top       a
freq      5
dtype: object

Note that on a mixed-type DataFrame object, describe() will restrict the summary to include only numerical columns or, if none are, only categorical columns:

In [76]: frame = pd.DataFrame({'a': ['Yes', 'Yes', 'No', 'No'], 'b': range(4)})

In [77]: frame.describe()
Out[77]: 
              b
count  4.000000
mean   1.500000
std    1.290994
min    0.000000
25%    0.750000
50%    1.500000
75%    2.250000
max    3.000000

This behaviour can be controlled by providing a list of types as include/exclude arguments. The special value all can also be used:

In [78]: frame.describe(include=['object'])
Out[78]: 
          a
count     4
unique    2
top     Yes
freq      2

In [79]: frame.describe(include=['number'])
Out[79]: 
              b
count  4.000000
mean   1.500000
std    1.290994
min    0.000000
25%    0.750000
50%    1.500000
75%    2.250000
max    3.000000

In [80]: frame.describe(include='all')
Out[80]: 
          a         b
count     4  4.000000
unique    2       NaN
top     Yes       NaN
freq      2       NaN
mean    NaN  1.500000
std     NaN  1.290994
min     NaN  0.000000
25%     NaN  0.750000
50%     NaN  1.500000
75%     NaN  2.250000
max     NaN  3.000000

Index of Min/Max Values¶

The idxmin() and idxmax() functions on Series and DataFrame compute the index labels with the minimum and maximum corresponding values:

In [81]: s1 = pd.Series(np.random.randn(5))

In [82]: s1
Out[82]: 
0   -1.334603
1    0.772102
2   -1.098792
3    0.593070
4    0.936506
dtype: float64

In [83]: s1.idxmin(), s1.idxmax()
Out[83]: (0, 4)

In [84]: df1 = pd.DataFrame(np.random.randn(5,3), columns=['A','B','C'])

In [85]: df1
Out[85]: 
          A         B         C
0  0.385370 -1.067328 -0.577246
1 -0.008720 -0.126890  0.007873
2  0.619396  0.337610  1.238155
3  1.242305 -0.659185 -0.108661
4  0.772091  1.827445 -0.253363

In [86]: df1.idxmin(axis=0)
Out[86]: 
A    1
B    0
C    0
dtype: int64

In [87]: df1.idxmin(axis=1)
Out[87]: 
0    B
1    B
2    B
3    B
4    C
dtype: object

When there are multiple rows (or columns) matching the minimum or maximum value, idxmin() and idxmax() return the first matching index:

In [88]: df3 = pd.DataFrame([2, 1, 1, 3, np.nan], columns=['A'], index=list('edcba'))

In [89]: df3
Out[89]: 
     A
e  2.0
d  1.0
c  1.0
b  3.0
a  NaN

In [90]: df3['A'].idxmin()
Out[90]: 'd'

Value counts (histogramming) / Mode¶

The value_counts() Series method and top-level function computes a histogram of a 1D array of values.

In [91]: data = np.random.randint(0, 7, size=50)

In [92]: data
Out[92]: 
array([6, 6, 0, 4, 2, 3, 5, 3, 0, 0, 4, 5, 0, 6, 4, 2, 4, 2, 0, 2, 5, 4, 6,
       6, 5, 6, 6, 0, 5, 5, 3, 5, 0, 3, 4, 1, 1, 5, 3, 4, 0, 3, 6, 5, 3, 0,
       5, 6, 3, 6])

In [93]: s = pd.Series(data)

In [94]: s.value_counts()
Out[94]: 
6    10
5    10
0     9
3     8
4     7
2     4
1     2
dtype: int64

Similarly, you can get the most frequently occurring value(s) (the mode) of the values in a Series or DataFrame:

In [95]: s5 = pd.Series([1, 1, 3, 3, 3, 5, 5, 7, 7, 7])

In [96]: s5.mode()
Out[96]: 
0    3
1    7
dtype: int64

In [97]: df5 = pd.DataFrame({'A': np.random.randint(0, 7, size=50),
   ....:                     'B': np.random.randint(-10, 15, size=50)})
   ....: 

In [98]: df5.mode()
Out[98]: 
     A   B
0  2.0  -9
1  4.0  -1
2  NaN   0
3  NaN   2
4  NaN  10