pyfora¶

AWS Account¶

You’ll need an AWS account with an access key that has permission to launch EC2 instances. If you don’t yet have an access key, follow these instructions to create one.

boto¶

The pyfora_aws tool uses boto to communicate with AWS:

pip install boto

Credentials¶

pyfora_aws uses boto to interact with EC2 on your behalf. If you already have a Boto configuration file with your credentials then no additional configuration in needed. Otherwise, you can set your credentials using the environment variables AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY.

To set the envrionment variables, open a terminal window and type:

export AWS_ACCESS_KEY_ID=<your aws access key id>
export AWS_SECRET_ACCESS_KEY=<your aws secret key>

SSH Key-Pair¶

While not strictly required, it is strongly recommended that you register an SSH key-pair with EC2 and use it when launching instances. Otherwise, you will not be able to log in to the launched instances for diagnostics and troubleshooting. See Amazon EC2 Key Pairs for more information.

Start a Backend Instance¶

You are now ready to start some instances using pyfora_aws. The following command will launch and configure a single c3.8xlarge on-demand instance in the us-east-1 region. It takes about 5-6 minutes to complete:

    $ pyfora_aws start --ssh-keyname <name_of_your_SSH_keypair>

    Launching manager instance:
    Mon Mar  7 10:24:42 2016 -- i-4aef5b89: pending /
    Done

    Manager instance started:

        i-4aef5b89 | 184.169.200.155 | running | manager

    To tunnel the pyfora HTTP port (30000) over ssh, run the following command:
        ssh -i <ssh_key_file> -L 30000:localhost:30000 ubuntu@184.169.200.155

    Waiting for services:

    Mon Mar  7 10:26:20 2016 -- Instance:i-4aef5b89: installing dependencies -
    Mon Mar  7 10:29:10 2016 -- Instance:i-4aef5b89: installing docker 1.9 -
    Mon Mar  7 10:30:28 2016 -- Instance:i-4aef5b89: pulling docker image -
    Mon Mar  7 10:30:51 2016 -- Instance:i-4aef5b89: launching service -
    Mon Mar  7 10:30:52 2016 -- Instance:i-4aef5b89: ready
    Done

Where <name_of_your_SSH_keypair> is the name you gave your SSH key-pair in EC2.

SSH Tunnelling¶

By default, to keep things secure, pyfora_aws keeps all ports on launched instances inaccessible to incoming connections, with the exception of port 22 for SSH connections. The easiest secure way to connect to the launched instance from your machine is by tunnelling pyfora’s HTTP port - 30000 - over SSH. This means that all traffic between your machine and the instance is secured by SSH.

To establish a tunnel, open a new terminal window (it will need to stay open for the duration of your session) and run:

$ ssh -i <ssh_key_file> -L 30000:localhost:30000 ubuntu@<manager_ip_address>

Where <ssh_key_file> is the path to the private key file of the SSH key-pair you specified when launching the instance, and <manager_ip_address> is the public IP address of the manager machine (184.169.200.155 in the example above).

The -L option tells SSH to map port 30000 on your local machine to localhost:30000 on the remote.

OS¶

You’ll need an OS that can run docker. Currently this means:

• A reasonably recent version of a 64-bit Linux distribution such as: Ubuntu, Debian, RedHat, Fedora, Centos, Gentoo, Suse, Amazon, Oracle, etc.
• OS X 10.8 “Mountain Lion” or newer.
• Windows 7.1, 8/8.1 or newer.

Docker¶

Docker is available for Linux, OS X, or Windows. To install docker on your machine, visit http://www.docker.com/, click the Get Started link, and follow the instructions.

On Linux you can also, optionally, follow these instructions to enable running docker commands without sudo. Note, however, that the docker daemon still runs as root - it just saves you five keystrokes when running docker commands.

What does this do?¶

• docker run launches a new docker container.
• -d starts the container as daemon that runs in the background.
• --name pyfora names the new container pyfora for easy reference in subsequent commands.
• -p 30000:30000 maps port 30000 - pyfora’s default HTTP port - to the same port number in your host OS. This lets pyfora connect to the container using http://localhost:30000.
• -v /home/user/ufora:/var/ufora mounts the local directory ~/ufora into /var/ufora within the container. This is where Ufora writes all of its log files.
• ufora/service is the name of the Ufora service image to run. To use a version other than the latest, specify a version tag (e.g. ufora/service:0.2).

Environment Variables¶

There are several environment variable that can be set when launching a pyfora container to configure its behavior.

• UFORA_MANAGER_ADDRESS: the host-name or IP address of the manager. Setting this variable causes the container to only run the worker service. Without this variable, the container runs both manager and worker services.
• UFORA_WORKER_OWN_ADDRESS: the host-name or IP address that the worker uses to register itself with the manager. The manager and other workers try to connect to it using this address. This is useful in situations where you have multiple network interfaces (public and private, or a docker container running in bridge mode) and you want to tell the worker which address to register. The variable is required unless the worker container is started with the --net=host option, in which case the worker tries to figure out its own address using socket.gethostbyname(socket.getfqdn()) if the variable isn’t set.
• UFORA_WORKER_BASE_PORT: the first of two consecutive ports that the worker listens on. This is useful if you want to run multiple workers side-by-side.
• UFORA_NO_WORKER: Set this variable to 1 to prevent the manager container from also running a worker. This variable and UFORA_MANAGER_ADDRESS are mutually exclusive. At most one of them can be set.
• UFORA_WEB_HTTP_PORT: the port used by the manager’s HTTP server.

Ports¶

$ ssh user_name@54.144.209.248 -L 30000:localhost:30000

The Manager¶

Pick a machine to run the manager service and run the following command to start the manager and a worker on it:

$ sudo docker run -d --name pyfora_manager -p 30000:30000 -p 30002:30002 -v /home/user/ufora:/var/ufora ufora/service

To run the manager service without a worker run:

$ sudo docker run -d --name pyfora_manager -e UFORA_NO_WORKER=1 -p 30000:30000 -p 30002:30002 -v /home/user/ufora:/var/ufora ufora/service

Workers¶

If your manager is running, for example, at 192.168.1.15, and the worker is at 192.168.2.11, start a worker using:

$ sudo docker run -d --name pyfora_worker -e UFORA_MANAGER_ADDRESS=192.168.1.15 -e UFORA_WORKER_OWN_ADDRESS=192.168.2.11 -p 30009:30009 -p 30010:30010 -v /home/user/ufora:/var/ufora ufora/service

Repeat this on every machine you want to use as a worker in your cluster.

What just happened?¶

The code in the body of the with block was shipped to the cluster, along with any dependent objects and code (like isPrime() in this case) that it refers to, directly or indirectly.

The python code was translated into pyfora bitcode and executed on the cluster. The resulting objects (result in the example above) were downloaded from the cluster and copied back back into the local python environment because we used executor.remotely.downloadAll().

Depending on the code you are running, and the number of CPU cores available in your cluster, the runtime looks for opportunities to parallelize the execution of your code. In the example above, the runtime can see that individual invocations of isPrime() within the generator expression isPrime(x) for x in xrange(10 * 1000 * 1000) are independent of each other can therefore be run in parallel.

In fact, what the runtime does in this example is to split the xrange() iteration across all available cores in the cluster. If a particular subrange completes while others are still running, the runtime dynamically subdivides a range that is still computing to maximize the utilization of CPUs. This is something that is bound to happen in problems like this when time-complexity of a computation is not constant across the entire input space (determining whether a large number is prime is much harder than a small number).

Working with proxies¶

In the previous example, the result of the computation was the number of prime numbers in the specified range. That’s a single int that can be easily downloaded from the cluster and copied into the local python environment.

Now consider this variation of the code:

with executor.remotely.remoteAll():
    primes = [x for x in xrange(10 * 1000 * 1000) if isPrime(x)]

This time the result of the computation, the variable primes, is a list of all prime numbers in the range. But because we used executor.remotely.remoteAll(), the variable primes is a proxy to a list of primes that lives in-memory on the cluster (it is actually an instance of RemotePythonObject).

There are two things you can do with remote python objects:

1. Download them into the local python scope where they become regular python objects.
2. Use them in subsequent remote computations on the cluster.

Downloading a remote object is done using its toLocal() method, which returns a Future that resolves to the downloaded object. To do it all synchronously in one line you can write:

primes = primes.toLocal().result()

This call downloads all the data in the remote primes list from the cluster to the client machine where it is converted back into python. If the list is very large, or the connection to the cluster is slow, this can be a slow operation. Furthermore, the size of the list may be greater than the amount of memory available on the local machine, in which case it is impossible to download it this way.

As an alternative to downloading the entire result, we may choose to compute with it inside of another with executor.remotely block. For example:

with executor.remotely.downloadAll():
    lastFewPrimes = primes[-100:]

The backend recognizes that primes is a proxy to data that lives remotely in the cluster, and lets us refer to it in dependent computations, the result of which we then return as regular python objects.

For convenience, it also possible to write:

with executor.remotely.downloadSmall(bytecount=100*1024):
    ...

In this case, objects larger than bytecount are left in the cluster and returned as proxies, while smaller objects are downloaded and copied into the local scope.

Basic Behavior of the JIT¶

The pyfora JIT compiler optimizes the code your program spends the most time in. So, for instance, if you write:

def loopSum(x):
    result = 0.0
    while x > 0:
        result = result + x
        x = x - 1
    return result
print loopSum(1000000000)

The pyfora runtime will notice that your program is spending a huge amount of time in the while loop and produce a fast machine-code version of it in which x and result are held in registers (as 64 bit integer and floats, respectively). We generate machine code using the excellent and widely-used llvm project. In simple numerical programs, you’ll end up with the same code you’d get from a good C++ compiler. Today, these are table-stakes for all JIT compilers.

Higher-Order Functions, Classes, and other Language Constructs¶

Unlike most JIT compilers applied to dynamically typed languages, pyfora is designed to work well with higher-order functions and classes. In general, this is a thorny problem for any system attempting to speed up python because in regular python programs, it’s possible to modify class and instance methods during the execution of the program. This means that any generated code in tight loops has to repeatedly check to see whether some method call has changed[#mutating_code]_. Because this is disabled in pyfora code, pyfora can agressively optimize away these checks, perform agressive function inlining, and generally perform a lot of the optimizations you see in compilers optimizing statically typed languages like C++ or Java. This allows you to refactor your code into classes and objects without paying a performance penalty.

This flexibility comes at a cost: the compiler generates new code for every combination of types and functions that the it sees. For instance, if you write:

def loopSum(x, f):
    result = 0
    while x>0:
        result = result + f(x)
        x = x - 1
    return result

then the compiler will generate completely different code for loopSum(1000000, lambda x:x) and loopSum(1000000, lambda x:x*x) and both will be very fast. This extends to classes. For instance, if you write:

class Add:
    def __init__(self, arg):
        self.arg = arg

    def __call__(self, x):
        return self.arg + x

class Multiply:
    def __init__(self, arg):
        self.arg = arg

    def __call__(self, x):
        return self.arg * x

class Compose:
    def __init__(self, f, g):
        self.f = f
        self.g = g

    def __call__(self, x):
        return self.f(self.g(x))

you will get idential performance if you write loopSum(1000000, lambda x: x * 10.0 + 20.0) and loopSum(1000000, Compose(Multiply(10.0), Add(20.0))) - they boil down to the same mathematical operations, and because pyfora doesn’t allow class methods to be modified, it can reason about the code well enough to produce fast machine-code.

Keep the Total Number of Type Combinations Small¶

The pyfora compiler operates by tracking all the distinct combinations of types it sees for all the variables in a given stackframe, and generating code for each combination. This means that a function like:

def typeExplosion(v):
    a = None
    b = None
    c = None
    for x in v:
        if x < 1:
            a = x
        elif x < 2:
            b = x
        else:
            c = x
    return (a,b,c)

could generate a lot of code. For instance, if a, b, and c can all be None or an integer, you’ll end up with 8 copies of the loop. That by itself isn’t a problem, but if you keep adding variables, the total number of types grows exponentially - eventually, you’ll wind up waiting forever for the compiler to finish generating code.

Tuples as Structure¶

Speaking of “types”, pyfora considers function instances, class instances, and tuples to be “structural”. This means that the compiler will agressively track type information about the contents of these objects. So, for instance, lambda x: x + y is a different type if y is an integer or a float in the surrounding scope. Similarly, (x, y) tracks the type information of both x and y. This is one of the reasons why there is no penalty for putting values into objects or tuples - the compiler tracks that type information the whole way through, so that (x, y)[0] is semantically equivalent to x regardless of what y is.

This is great until you start using tuples to represent data with a huge variety of structure, which can overwhelm the compiler. For instance,

def buildTuple(ct):
    res = ()
    for ix in xrange(ct):
        res = res + (ix,)
    return res
print buildTuple(100)

will produce a lot of code, because it will produce separate copies of the loop for the types “empty tuple”, “tuple of one integer”, “tuple of two integers”, ..., “tuple of 99 integers”, etc.

Because of this, tuples should generally be used when their shape will be stable (i.e. producing a small number of types) over the course of the program and you want the compiler to be able to see it. [3]

Also note that this means that if you have tuples with heterogeneous types and you index into it with a non-constant index, you will generate slower code. This is because the compiler needs to generate a separate pathway for each possible resulting type. For instance, if you write:

aTuple = (0.0, 1, "a string", lambda x: x)
functionCount = floatCount = 0

for ix in range(100):
    # pull an element out of the tuple - the compiler can't tell what
    # kind of element it is ahead of time
    val = aTuple[ix % len(aTuple)]

    if isinstance(val, type(lambda: None)):
        functionCount = functionCount + 1
    elif isinstance(val, float):
        floatCount = floatCount + 1

then the compiler will need to generate branch code at the aTuple[...] instruction. This will work, but will be slower than it would be if the tuple index could be known ahead of time.

Lists¶

Lists are designed to hold data that varies in structure. The compiler doesn’t attempt to track the types of the individual objects inside of a list. Specifically, that means that [1, 2.0] and [2.0, 1] have the same type - they’re both ‘list of int and float’, whereas (1, 2.0) and (2.0, 1) are different types.

Lists are fastest when they’re homogenous (e.g. entirely containing elements of the same type). This is because the pyfora VM can pack data elements very tightly (since they all have the same layout) and can generate very efficient lookup code. Lists with heterogenous types are still fast, but the more types there are, the more code the compiler needs to generate in order to work with them, so try to keep the total number of types small.

In general, lists have more overhead in than in CPython [4] . This is because lists are the primary “big data” structure for pyfora - a list can be enormous (up to terabytes of data), and the data structure that represents them is rather large and complex. So, if possible, try to structure your program so that you create a few bigger lists, rather than a lot of little lists.

One exception to this rule: if v is a list, the operation: v + [element] will be fast and pyfora will optimize away the creation of the intermediate list and be careful not to duplicate v unless absolutely necessary. This is the fastest way to build a list.

Large lists are cheap to concatenate - they’re held as a tree structure, so you don’t have to worry that each time you concatenate you’re making a copy.

Finally, avoid nesting lists deeply - this places a huge strain on the “big data” component of pyfora’s internal infrastructure.

Dictionaries and Strings¶

Dictionaries are currently very slow [5] . Don’t use them inside of loops.

Strings are fast. The pyfora string structure is 32 bytes, allowing the VM to pack any string of 30 characters or less into a data structure that doesn’t hit the memory manager. Indexing into strings is also fast. Strings may be as large as you like (if necessary, they’ll be split across machines).

Note that for strings that are under 100000 characters, string concatenation makes a copy, so you can accidentally get O(N²) performance behavior if you write code where you are repeatedly concatenating a large string to a small string.

The Core Model of Parallelism in pyfora¶

pyfora exploits “dataflow” parallelism at the stack-frame level. It operates by executing your code on a single thread and then periodically interrupting it and walking its stack, looking at the flow of data within each stack frame to see whether there are upcoming calls to functions that it can schedule while the current call is executing.

For instance, if you write f(g(),h()), then while executing the call to g(), the runtime can see that you are going to execute h() next. If you have unsaturated cores, it will rewrite the stack frame to also call h() in parallel. When both calls return, it will resume and call f(). You can think of this as fork-join parallelism where the forks are placed automatically.

As an example, the simple divide-and-conquer implementation of a sum() function could be written as:

def sum(a,b,f):
    if a >= b:
        return 0
    if a + 1 >= b:
        return f(a)

    mid = (a+b)/2

    return sum(a,mid,f) + sum(mid,b,f)

We can then write sum(0, 1000000000000, lambda x: x**0.5) and get a parallel implementation. This works because each call to sum contains two recursive calls to sum, and pyfora can see that these are independent.

Note that pyfora assumes that exceptions are rare - in the case of f(g(),h()), pyfora assumes that by default, g() is not going to throw an exception and that it can start working on h(). In the case where g() routinely throws exceptions, pyfora will start working on h() only to find that the work is not relevant. Some python idioms use exceptions for flow control: for instance, accessing an attribute and then catching AttributeError as a way of determining if an object meets an interface. In this case, make sure that you don’t have an expensive operation in between the attribute check and the catch block.

Nested Parallelism¶

This model of parallelism naturally allows for nested parallelism. For instance, sum(0,1000,lambda x: sum(0,x,lambda y:x*y)) will be parallel in the outer sum() but also in the inner sum(). This is because pyfora doesn’t really distinguish between the two - it parallelizes stackframes, not algorithms.

Adaptive Parallelism¶

pyfora’s parallelism is adaptive and dynamic - it doesn’t know ahead of time how the workload is distributed across your functions. It operates by agressively splitting stackframes until cores are saturated, waiting for threads to finish, and then splitting additional threads.

This model is particularly effective when your functions have different runtimes depending on their input. For instance, consider:

def isPrime(p):
    if p < 2: return 0
    x = 2
    while x*x <= p:
        if p%x == 0:
            return 1
        x = x + 1
    return 1

sum(isPrime(x) for x in xrange(10000000))

Calls to isPrime() with large integers take a lot longer than calls to isPrime() with small integers, because we have to divide so many more numbers into the large ones. Naively allocating chunks of the 10000000 range to cores will end up with some cores working while others finish their tasks early. pyfora can handle this because it sees the fine structure of parallelism available to sum and can repeatedly subdivide the larger ranges, keeping all the cores busy.

This technique works best when your tasks subdivide to a very granular level. In the case where you have a few subtasks with long sections of naturally single-threaded code, pyfora may not schedule those sections until partway through the calculation. You’ll get better performance if you can find a way to get the calculation to break down as finely as possible.

It’s also important to note that the pyfora VM doesn’t penalize you for writing your code in a parallel way. pyfora machine-code is optimized for single-threaded execution - it’s only when there are unused cores and pyfora wants more tasks to work on that we split stackframes, in which case we pop the given stackframe out of native code and back into the interpreter.

The one caveat here is that function calls have stack-frame overhead. Code that’s optimized for maximum performance sometimes has conditions to switch it out of a recursive “parallelizable” form and into a loop. This is a tradeoff between single-threaded performance and parallelism granularity.

List Comprehensions and Sum are Parallel¶

By default, list comprehensions like [isPrime(x) for x in xrange(10000000)] are parallel if the generator in the righthand side supports the __pyfora_generator__ parallelism model, which both xrange(), and lists support out of the box.

Similarly, functions like sum() are parallel if their argument supports the __pyfora_generator__ interface. Note that this subtly changes the semantics of sum:() in standard python, sum(f(x) for x in xrange(4)) would be equivalent to:

(((f(0)+f(1))+f(2))+f(3))+f(4)

performing the addition operations linearly from left to right. In the parallel case, we have a tree structure:

(f(0)+f(1)) + (f(2)+f(3))

when addition is associative. Usually this produces the same results, but it’s not always true. For instance, roundoff errors in floating point arithmetic mean that floating point addition is not perfectly associative [6] . As this is a deviation from standard python, we plan to make it an optional feature in the future.

Loops are Sequential¶

Note that pyfora doesn’t try to parallelize across loops. The isPrime() example above runs sequentially. In the future, we plan to implement loop unrolling so that if you write something like:

res = None
for x in xrange(...):
    res = f(g(x), res)

if the calls to g() are sufficiently expensive, we’ll be able to schedule those calls in parallel and then execute the naturally sequential calls to f() as they complete. For the moment, however, assume that while and for loops are sequential (although functions inside them are all candidates for parallelism).

Lists Prefer Cache-Local Access¶

Lists are the basic building-block for “big data” in pyfora. A list that’s large enough will get split across multiple machines. pyfora organizes a list’s data into chunks of contiguous indices, where each chunk represents ~50-100 MB of data.

When one of your threads (in this context, a thread is just a collection of stackframes of python code that pyfora hasn’t decided to subdivide) indexes into a very large list and that data isn’t on the same machine as the thread, pyfora must decide what to do: (a) move the thread to the data, or (b) move the data to the thread? This is called a “cache miss.” Threads tend to be much smaller than 50MB, so usually it will move the thread to the remote machine.

One of the unique characteristics of the pyfora runtime: it will simulate the execution of code in advance of its execution to predict cache misses and move data and threads accordingly. For example, if your thread starts accessing two different blocks in a list, and those two blocks are on different machines, that thread may end up bouncing back and forth between the two machines in a slow oscillatory pattern. pyfora can predict these access patterns and optimize the layout of blocks and threads to prevent this in advance.

All of this infrastructure is useless if you index randomly into very big lists (here, we mean bigger than ~25% of a machine’s worth of data). This is because it’s now impossible for the scheduler to find an allocation of blocks to machines where a large fraction of your list accesses don’t require you to cross a machine boundary.

As a result, you’ll get the best performance if you can organize your program so that list accesses are “cache local”, meaning that when you access one part of a list you tend to access other parts of the list that are nearby in the index space [7] .

Footnote

class X:
    def f(self):
        return 0

x = X()

print x.f()

# modify all instances of 'X'
X.f = lambda self: return 1

print x.f()

# now modify x itself
x.f = lambda: return 2

print x.f()

Examples¶

$ pyfora_aws start --vpc-id vpc-0c73f14e --subnet-id subnet-7214f1a0 --ssh-keyname my_key -n 3

This will launch a cluster of three c3.8xlarge instances into the specified VPC and subnet in the default us-east-1 region, and use the EC2 ssh key-pair called my_key.

$ pyfora_aws start --instance-type g2.2xlarge --spot-price 0.3 --open-public-port

This will launch a single g2.2xlarge spot instance with a maximum bid price of $0.3 and open inbound traffic on port 30000.

Examples¶

$ pyfora_aws add -n 3 --ec2-region us-west-2 --security-group-id sg-2f28a1c0

This adds three instances to an existing cluster running in the us-west-2 region with security group sg-2f28a1c0.

Examples¶

 $ pyfora_aws list --ec2-region us-west-1
 3 instances:
     i-dc7acd1f | 50.18.72.241 | running | worker
     i-387ccbfb | 54.176.35.132 | running | worker
     i-ba7bcc79 | 54.177.18.215 | running | worker

Examples¶

 $ pyfora_aws stop --ec2-region us-west-1 --terminate
 Terminating 3 instances:
     i-dc7acd1f | 50.18.72.241 | running | worker
     i-387ccbfb | 54.176.35.132 | running | worker
     i-ba7bcc79 | 54.177.18.215 | running | worker

RemotePythonObject.DefinedRemotePythonObject¶

class pyfora.RemotePythonObject.DefinedRemotePythonObject(objectId, executor)¶

A proxy that represents a local object, which has been uploaded to a pyfora cluster.

Note

Only Executor is intended to create instances of DefinedRemotePythonObject. They are created by calling define().

Parameters:	objectId (int) – a value that uniquely identifies the remote object that this DefinedRemotePythonObject represents. executor – the Executor that created this DefinedRemotePythonObject.

RemotePythonObject.ComputedRemotePythonObject¶

class pyfora.RemotePythonObject.ComputedRemotePythonObject(computedValue, executor, isException)¶

A proxy that represents a remote object created on a pyfora cluster as a result of some computation.

Note

Only Executor is intended to create instances of ComputedRemotePythonObject. They are created by calling submit().

Parameters:	computedValue – an instance of a SubscribableWebObject computedValue representing the computation that produced this ComputedRemotePythonObject. executor – the Executor that created this DefinedRemotePythonObject.

Linear Regression¶

pyfora.algorithms.linearRegression(predictors, responses)¶

Compute the regression coefficients (with intercept) for a set of predictors against responses.

Parameters:	predictors (DataFrame) – a pandas.DataFrame with the predictor columns. responses (DataFrame) – a pandas.DataFrame whose first column is used as the regression’s target.
Returns:	A numpy.array with the regression coefficients. The last element in the array is the intercept.

Logistic Regression¶

class pyfora.algorithms.BinaryLogisticRegressionFitter(C, hasIntercept=True, method='newton-cg', interceptScale=1.0, tol=0.0001, maxIter=100000.0, splitLimit=1000000)¶

A logistic regression “fitter” ithat holds fitting parameters used to fit logit models.

Parameters:

C (float) – Inverse of regularization strength; must be a positive float. hasIntercept (bool) – If True, include an intercept (aka bias) term in the fitted models. method (string) – one of ‘newton-cg’ (default) or ‘majorization’ interceptScale (float) – When hasIntercept is True, feature vectors become [x, interceptScale], i.e. we add a “synthetic” feature with constant value interceptScale to all of the feature vectors. This synthetic feature is subject to regularization as all other features. To lessen the effect of regularization, users should increase this value. tol (float) – Tolerance for stopping criteria. Fitting stops when the l2-norm of the parameters to update do not change more than tol. maxIter (int) – A hard limit on the number of update cycles allowed.

fit(X, y)¶

fit a (regularized) logit model to the predictors X and responses y.

Parameters:	X – a dataframe of feature vectors. y – a dataframe (with one column) which contains the “target” values, corresponding to the feature vectors in X.
Returns:	A BinaryLogisticRegressionModel which represents the fit model.

Example:

# fit a logit model without intercept using regularizer 1.0

from pyfora.algorithms import BinaryLogisticRegressionFitter

fitter = BinaryLogisticRegressionFitter(1.0, False)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = fitter.fit(x, y)

class pyfora.algorithms.logistic.BinaryLogisticRegressionModel.BinaryLogisticRegressionModel(coefficients, classZeroLabel, classOneLabel, intercept, interceptScale, iters)¶

Represents a fit logit model.

coefficients¶

numpy.array – The regressions coefficients.

intercept¶

float – The fitted model’s intercept

Note

This class is not intended to be constructed directly. Instances of it are returned by fit().

predict(X)¶

Predict the class labels of X.

Parameters:	X (DataFrame, or numpy.array) – a set of feature vectors
Returns:	array containing the predicted class labels.
Return type:	numpy.array

predict_probability(X)¶

Estimate the conditional class-zero probability for the features in X.

Parameters:	X (DataFrame, or numpy.array) – a set of feature vectors
Returns:	array containing the predicted probabilities.
Return type:	numpy.array

score(X, y)¶

Returns the mean accuracy on the given test data and labels.

Parameters:	X (DataFrame) – Feature vectors y (DataFrame) – Target labels, corresponding to the vectors in X.
Returns:	The mean accuracy of predict() with respect to y.
Return type:	float

Regression Trees¶

class pyfora.algorithms.regressionTrees.RegressionTree.RegressionTreeBuilder(maxDepth, minSamplesSplit=2, numBuckets=10000, minSplitThresh=1000000)¶

Fits regression trees to data using specified tree parameters.

Parameters:	maxDepth (int) – The maximum depth of a fit tree minSamplesSplit (int) – The minimum number of samples required to split a node numBuckets (int) – The number of buckets used in the estimation of optimal column splits. minSplitThresh (int) – an “internal” argument, not generally of interest to casual users, giving the splitting rule in computeBucketedSampleSummaries.
Returns:	A RegressionTree instance.

Examples:

from pyfora.algorithms import RegressionTreeBuilder

builder = RegressionTreeBuilder(2)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})
regressionTree = builder.fit(x, y)

static buildTree(x, y, minSamplesSplit, maxDepth)¶

Fit a regression tree to predictors x and responses y using parameters minSamplesSplit and maxDepth.

Parameters:	x (pandas.DataFrame) – of the predictors. y (pandas.DataFrame) – giving the responses. maxDepth – The maximum depth of a fit tree minSamplesSplit – The minimum number of samples required to split a node

fit(x, y)¶

Using a RegressionTreeBuilder, fit a regression tree to predictors x and responses y.

Parameters:	x (pandas.DataFrame) – of the predictors. y (pandas.DataFrame) – giving the responses.
Returns:	a RegressionTree instance.

Examples:

builder = pyfora.algorithms.regressionTrees.RegressionTree.RegressionTreeBuilder(2)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})
regressionTree = builder.fit(x, y)

class pyfora.algorithms.regressionTrees.RegressionTree.RegressionTree(rules, numDimensions=None, columnNames=None)¶

A class representing a regression tree.

A regression tree is represented, essentially, as a list of “rules”, which are either SplitRule, giving “split” nodes, which divide the domain by a hyperplane, or RegressionLeafRule, which just hold a prediction value.

Note

This class is not generally instantiated directly by users. Instead, they are normally returned by RegressionTreeBuilder.

predict(x, depth=None)¶

Predicts the responses corresponding to pandas.DataFrame x.

Returns:	A pandas.Series giving the predictions of the rows of x.

Examples:

from pyfora.algorithms import RegressionTreeBuilder

builder = RegressionTreeBuilder(2)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})
regressionTree = builder.fit(x, y)

# predict `regressionTree` on `x` itself
regressionTree.predict(x)

score(x, yTrue)¶

Returns the coefficient of determination R² of the prediction.

The coefficient R² is defined as (1 - u / v), where u is the regression sum of squares ((yTrue - yPredicted) ** 2).sum() and v is the residual sum of squares ((yTrue - yTrue.mean()) ** 2).sum(). Best possible score is 1.0, lower values are worse.

Returns:	(float) the R2 value

Examples:

from pyfora.algorithms import RegressionTreeBuilder

builder = RegressionTreeBuilder(2)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})
regressionTree = builder.fit(x, y)

# predict `regressionTree` on `x` itself
regressionTree.score(x, y)

Gradient Boosting¶

class pyfora.algorithms.regressionTrees.GradientBoostedRegressorBuilder.GradientBoostedRegressorBuilder(maxDepth=3, nBoosts=100, learningRate=1.0, minSamplesSplit=2, numBuckets=10000, loss='l2')¶

A class which builds (or “fits”) gradient-boosted regression trees to data with specified parameters. These parameters are

Parameters:

maxDepth (int) – The max depth allowed of each constituent regression tree. nBoosts (int) – The number of “boosting iterations” used. learningRate (float) – The learning rate of the model, used for regularization. Each successive tree from boosting stages are added with multiplier learningRate. minSamplesSplit (int) – The minimum number of samples required to split a regression tree node. numBuckets (int) – The number of buckets used in the estimation of optimal column splits for building regression trees. loss – the loss used when forming gradients. Defaults to l2, for least-squares loss. The only other allowed value currently is lad, for “least absolute deviation” (aka l1-loss).

fit(X, y)¶

Fits predictors X to responses y.

Parameters:	X (pandas.DataFrame) – predictors. y (pandas.DataFrame) – responses.
Returns:	A RegressionModel instance.

Examples:

from pyfora.algorithms import GradientBoostedRegressorBuilder

builder = GradientBoostedRegressorBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

iterativeFitter(X, y)¶

Returns an IterativeFitter instance which can iteratively fit boosting models.

Parameters:	X (pandas.DataFrame) – predictors. y (pandas.DataFrame) – responses.

Examples:

from pyfora.algorithms import GradientBoostedRegressorBuilder

builder = GradientBoostedRegressorBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

fitter = builder.iterativeFitter(x, y)

# compute scores vs number of boosts
numBoosts = 5
scores = []
for ix in xrange(numBoosts):
    fitter = fitter.next()
    scores = scores + [fitter.model.score(x, y)]

class pyfora.algorithms.regressionTrees.RegressionModel.RegressionModel(additiveRegressionTree, X, XDimensions, yAsSeries, loss, regressionTreeBuilder, learningRate)¶

A class representing a gradient-boosted regression tree model fit to data.

Note

These classes are not normally instantiated directly. Instead, they are typically returned by GradientBoostedRegressorBuilder instances.

predict(df, nEstimators=None)¶

Predict on the pandas.DataFrame df.

Example:

from pyfora.algorithms import GradientBoostedRegressorBuilder

builder = GradientBoostedRegressorBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

# predict `x` using the model `model`:
model.score(x, y)

score(X, yTrue)¶

Return the coefficient of determination (R²) of the prediction.

The coefficient R² is defined as (1 - u / v), where u is the regression sum of squares ((yTrue - yPredicted) ** 2).sum() and v is the residual sum of squares ((yTrue - yTrue.mean()) ** 2).sum(). Best possible score is 1.0, lower values are worse.

Parameters:	X – the predictor DataFrame. yTrue – the (true) responses DataFrame.
Returns:	(float) the R2 value.

Example:

from pyfora.algorithms import GradientBoostedRegressorBuilder

builder = GradientBoostedRegressorBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

# compute the score of the fit model:
model.score(x, y)

class pyfora.algorithms.regressionTrees.GradientBoostedRegressorBuilder.IterativeFitter(model, predictions)¶

A sort of iterator class which is capable of fitting subsequent boosting models.

model¶

the current regression model.

predictions¶

the current predictions of the regression model (with respect to the training set implicit in model).

Note

This class is typically not instantiated directy. Instead these classes are returned from iterativeFitter().

next()¶

Fit one boosting stage, returning a new IterativeFitter object that holds the next regression model and predictions.

Examples:

from pyfora.algorithms import GradientBoostedRegressorBuilder

builder = GradientBoostedRegressorBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

fitter = builder.iterativeFitter(x, y)

# compute scores vs number of boosts
numBoosts = 5
scores = []
for ix in xrange(numBoosts):
    fitter = fitter.next()
    scores = scores + [fitter.model.score(x, y)]

class pyfora.algorithms.regressionTrees.GradientBoostedClassifierBuilder.GradientBoostedClassifierBuilder(maxDepth=3, nBoosts=100, learningRate=1.0, minSamplesSplit=2, numBuckets=10000)¶

A class which builds (or “fits”) gradient boosted (regression) trees to form classification models.

Parameters:

maxDepth (int) – The max depth allowed of each constituent regression tree. nBoosts (int) – The number of boosting iterations used. learningRate (float) – The learning rate of the model, used for regularization. Each successive tree from boosting stages are added with multiplier learningRate. minSamplesSplit (int) – The minimum number of samples required to split a regression tree node. numBuckets (int) – The number of buckets used in the estimation of optimal column splits for building regression trees. loss – the loss used when forming gradients. Defaults to l2, for least-squares loss. The only other allowed value currently is lad, for “least absolute deviation” (aka l1-loss).

Note

Only nClasses = 2 cases are currently supported.

fit(X, y)¶

Fit predictors X to responses y.

Parameters:	X (pandas.DataFrame) – predictors. y (pandas.DataFrame) – responses.
Returns:	a BinaryClassificationModel

Examples:

from pyfora.algorithms import GradientBoostedClassifierBuilder

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

iterativeFitter(X, y)¶

Create an IterativeFitter instance which can iteratively fit boosting models.

Parameters:	X (DataFrame) – predictors. y (DataFrame) – responses.

Examples:

from pyfora.algorithms import GradientBoostedClassifierBuilder

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

fitter = builder.iterativeFitter(x, y)

numBoosts = 5
for ix in xrange(numBoosts):
    fitter = fitter.next()

class pyfora.algorithms.regressionTrees.BinaryClassificationModel.BinaryClassificationModel(additiveRegressionTree, X, classes, XDimensions, yAsSeries, loss, baseModelBuilder, learningRate)¶

A class representing a gradient-boosted binary classification tree model fit to data.

Note

These classes are not normally instantiated directly. Instead, they are typically returned by GradientBoostedClassifierBuilder instances.

deviance(x, yTrue)¶

Compute the binomial deviance (average negative log-likihood) of the instances in predictors X with responses y.

Parameters:	x – the predictor DataFrame. yTrue – the (true) responses DataFrame.

Examples:

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

# compute the deviance:
model.deviance(x, y)

predict(df)¶

Predict the class labels of the rows of df.

Parameters:	df (pandas.DataFrame) – input DataFrame.
Returns:	A pandas.Series giving the row-wise predictions.

Examples:

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

# use the fit model to predict `x` itself:
model.predict(x)

predictProbability(df)¶

Return class-zero probability estimates of the rows of a DataFrame df.

Parameters:	df (pandas.DataFrame) – input DataFrame.
Returns:	A pandas.Series giving the row-wise estimated class-zero probability estimates

Examples:

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

# use the fit model to predict `x` itself:
model.predictProbability(x)

score(x, yTrue)¶

Compute the mean accuracy in predicting x with respect to yTrue.

Parameters:	x – the predictor DataFrame. yTrue – the (true) responses DataFrame.

Examples:

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

model = builder.fit(x, y)

# use the fit model to predict `x` itself:
model.score(x, y)

class pyfora.algorithms.regressionTrees.GradientBoostedClassifierBuilder.IterativeFitter(model, previousRegressionValues)¶

A sort of iterator class which is capable of fitting subsequent boosting models.

model

the current regression model.

predictions

the current predictions of the regression model (with respect to the training set implicit in model).

Note

This class is typically not instantiated directy: instead these classes are returned from iterativeFitter().

next()¶

Boost once and return a new IterativeFitter

Returns:	A IterativeFitter instance.

Example:

from pyfora.algorithms import GradientBoostedClassifierBuilder

builder = GradientBoostedClassifierBuilder(1, 1, 1.0)
x = pandas.DataFrame({'x0': [-1,0,1], 'x1': [0,1,1]})
y = pandas.DataFrame({'y': [0,1,1]})

fitter = builder.iterativeFitter(x, y)

# compute scores vs number of boosts
numBoosts = 5
scores = []
for ix in xrange(numBoosts):
    fitter = fitter.next()
    scores = scores + [fitter.model.score(x, y)]