BeanBag¶
BeanBag is a set of modules that provide some syntactic sugar to make interacting with REST APIs easy and pleasant.
A simple example:
>>> import beanbag # version 1 api
>>> github = beanbag.BeanBag("https://api.github.com")
>>> watchers = github.repos.ajtowns.beanbag.watchers()
>>> for w in watchers:
... print(w["login"])
>>> import beanbag.v2 as beanbag # version 2 api
>>> github = beanbag.BeanBag("https://api.github.com")
>>> watchers = GET(github.repos.ajtowns.beanbag.watchers)
>>> for w in watchers:
... print(w.login)
Contents¶
beanbag.v2 – REST API access¶
A quick example:
>>> from beanbag.v2 import BeanBag, GET
>>> gh = BeanBag("https://api.github.com/")
>>> watchers = GET(gh.repos.ajtowns.beanbag.watchers)
>>> for w in watchers:
... print(w.login)
Setup:
>>> import beanbag.v2 as beanbag
>>> from beanbag.v2 import GET, POST, PUT, PATCH, DELETE
>>> myapi = beanbag.BeanBag("http://hostname/api/")
To constuct URLs, you can use attribute-style access or dict-style access:
>>> print(myapi.foo)
http://hostname/api/foo
>>> print(myapi["bar"])
http://hostname/api/bar
You can chain paths as well:
>>> print(myapi.foo.bar["baz"][3].xyzzy)
http://hostname/api/foo/bar/baz/3/xyzzy
To do a request on a resource that requires a trailing slash:
>>> print(myapi.foo._)
http://hostname/api/foo/
>>> print(myapi.foo[""])
http://hostname/api/foo/
>>> print(myapi.foo["/"])
http://hostname/api/foo/
>>> print(myapi["foo/"])
http://hostname/api/foo/
>>> print(myapi.foo._.x == myapi.foo.x)
True
>>> print(myapi.foo["_"])
http://hostname/api/foo/_
You can add URL parameters using function calls:
>>> print(myapi.foo(a=1, b="foo"))
http://hostname/api/foo?a=1;b=foo
Finally, to actually do REST queries on these queries you can use the GET, POST, PUT, PATCH and DELETE functions. The first argument should be a BeanBag url, and the second argument (if provided) should be the request body, which will be json encoded before being sent. The return value is the request’s response (decoded from json).
>>> res = GET( foo.resource )
>>> res = POST( foo.resource, {"a": 12} )
>>> DELETE( foo.resource )
To access REST interfaces that require authentication, you need to specify a session object when instantiating the BeanBag initially. BeanBag supplies helpers to make Kerberos and OAuth 1.0a authentication easier.
BeanBag class¶
The BeanBag class does all the magic described above, using beanbag.namespace.
- class beanbag.v2.BeanBag(base_url, ext='', session=None, use_attrdict=True)¶
- __init__(base_url, ext='', session=None, use_attrdict=True)¶
Create a BeanBag referencing a base REST path.
Parameters: - base_url – the base URL prefix for all resources
- ext – extension to add to resource URLs, eg ”.json”
- session – requests.Session instance used for this API. Useful to set an auth procedure, or change verify parameter.
- use_attrdict – if true, decode() will wrap dicts and lists in a beanbag.attrdict.AttrDict for syntactic sugar.
- __call__(*args, **kwargs)¶
Set URL parameters
- __eq__(other)¶
self == other
- __getattr__(attr)¶
self.attr
- __getitem__(item)¶
self[attr]
- __invert__()¶
Provide access to the base/path via the namespace object
bb = BeanBag(...) base, path = ~bb.foo assert isinstance(base, BeanBagBase)
This is the little bit of glue needed so that it’s possible to call methods defined in BeanBagBase directly rather than just the operators BeanBag supports.
- __ne__(other)¶
self != other
- __repr__()¶
Human readable representation of object
- __str__()¶
Obtain the URL of a resource
This class can be subclassed. In particular, if you need to use something other than JSON for requests or responses, subclassing BeanBagBase and overriding the encode and decode methods is probably what you want to do. One caveat: due to the way beanbag.namespace works, if you wish to invoke the parent classes method, you’ll usually need the parent base class, accessed via ~BeanBag or super(~SubClass, self).
HTTP Verbs¶
Functions are provided for the standard set of HTTP verbs.
- beanbag.v2.GET(url, body=None)¶
GET verb function
- beanbag.v2.HEAD(url, body=None)¶
HEAD verb function
- beanbag.v2.POST(url, body=None)¶
POST verb function
- beanbag.v2.PUT(url, body=None)¶
PUT verb function
- beanbag.v2.PATCH(url, body=None)¶
PATCH verb function
- beanbag.v2.DELETE(url, body=None)¶
DELETE verb function
The verb function is used to create BeanBag compatible verbs. It is used as:
GET = verb("GET")
- beanbag.v2.verb(verbname)¶
Construct a BeanBag compatible verb function
Parameters: verbname – verb to use (GET, POST, etc)
Request¶
The Request class serves as a place holder for arguments to requests.Session.request. Normally this is constructed from a dict object passed to POST or PUT via json.dumps() however a Request object can also be created by hand and passed in as the body parameter in a POST or similar BeanBag request. For example to make a POST request with a body that isn’t valid JSON:
POST(bbexample.path.to.resource, Request(body="MAGIC STRING"))
This can be useful with GET requests as well, even though GET requests don’t have a body per se:
GET(bbexample.path.to.resource, Request(headers={"X-Magic": "String"}))
- class beanbag.v2.Request(**kwargs)¶
Bases: beanbag.attrdict.AttrDict
- __init__(**kwargs)¶
Create a Request object
Request objects act as placeholders for the arguments to the requests() function of the requests.Session being used. They are used as the interface between the encode() and make_request() functions, and may also be used by the API caller.
NB: A Request object is only suitable for one use, as it may be modified in-place during the request. For this reason, __init__ makes a (shallow) copy of all the keyword arguments supplied rather than using them directly.
BeanBagException¶
- exception beanbag.v2.BeanBagException(response, msg)¶
Exception thrown when a BeanBag request fails.
- Data members:
- msg – exception string, brief and human readable
- response – response object
You can get the original request via bbe.response.request.
- __init__(response, msg)¶
Create a BeanBagException
beanbag.v1 – Original-style REST API access¶
Setup:
>>> import beanbag.v1 as beanbag
>>> foo = beanbag.BeanBag("http://hostname/api/")
To do REST queries, then:
>>> r = foo.resource(p1=3.14, p2=2.718) # GET request
>>> r = foo.resource( {"a": 3, "b": 7} ) # POST request
>>> del foo.resource # DELETE request
>>> foo.resource = {"a" : 7, "b": 3} # PUT request
>>> foo.resource += {"a" : 7, "b": 3} # PATCH request
You can chain paths as well:
>>> print(foo.bar.baz[3]["xyzzy"].q)
http://hostname/api/foo/bar/baz/3/xyzzy/q
To do a request on a resource that requires a trailing slash:
>>> print(foo.bar._)
http://hostname/api/foo/bar/
>>> print(foo.bar[""])
http://hostname/api/foo/bar/
>>> print(foo.bar["/"])
http://hostname/api/foo/bar/
>>> print(foo["bar/"])
http://hostname/api/foo/bar/
>>> print(foo.bar._.x == foo.bar.x)
True
>>> print(foo.bar["_"])
http://hostname/api/foo/bar/_
To access REST interfaces that require authentication, you need to specify a session object. BeanBag supplies helpers to make Kerberos and OAuth 1.0a authentication easier.
To setup oauth using OAuth1 directly:
>>> import requests
>>> from requests_oauth import OAuth1
>>> session = requests.Session()
>>> session.auth = OAuth1( consumer creds, user creds )
>>> foo = beanbag.BeanBag("http://hostname/api/", session=session)
Using the OAuth10aDance helper is probably a good plan though.
BeanBag class¶
- class beanbag.v1.BeanBag(base_url, ext='', session=None, fmt='json')¶
- __init__(base_url, ext='', session=None, fmt='json')¶
Create a BeanBag referencing a base REST path.
Parameters: - base_url – the base URL prefix for all resources
- ext – extension to add to resource URLs, eg ”.json”
- session – requests.Session instance used for this API. Useful to set an auth procedure, or change verify parameter.
- fmt – either ‘json’ for json data, or a tuple specifying a content-type string, encode function (for encoding the request body) and a decode function (for decoding responses)
- __call__(*args, **kwargs)¶
Make a GET, POST or generic request to a resource.
Example: >>> x = BeanBag("http://host/api") >>> r = x() # GET request >>> r = x(p1='foo', p2=3) # GET request with parameters passed via query string >>> r = x( {'a': 1, 'b': 2} ) # POST request >>> r = x( "RANDOMIZE", {'a': 1, 'b': 2} ) # Custom HTTP verb with request body >>> r = x( "OPTIONS", None ) # Custom HTTP verb with empty request body
- __delattr__(attr)¶
del self.attr
- __delitem__(item)¶
del self[item]
- __getattr__(attr)¶
self.attr
- __getitem__(item)¶
self[attr]
- __iadd__(val)¶
Make a PATCH request to a resource.
Example: >>> x = BeanBag("http://host/api") >>> x += {"op": "replace", "path": "/a", "value": 3}
- __setattr__(attr, val)¶
self.attr = val
- __setitem__(item, val)¶
self[item] = val
- __str__()¶
Obtain the URL of a resource
BeanBagException¶
- exception beanbag.v1.BeanBagException(response, msg)¶
Exception thrown when a BeanBag request fails.
- Data members:
- msg – exception string, brief and human readable
- response – response object
You can get the original request via bbe.response.request.
- __init__(response, msg)¶
Create a BeanBagException
beanbag.auth – Authentication Helpers¶
Kerberos Helper¶
To setup kerberos auth:
>>> import requests
>>> session = requests.Session()
>>> session.auth = beanbag.KerbAuth()
>>> foo = beanbag.BeanBag("http://hostname/api/", session=session)
- class beanbag.auth.KerbAuth(timeout=180)¶
Helper class for basic Kerberos authentication using requests library. A single instance can be used for multiple sites. Each request to the same site will use the same authorization token for a period of 180 seconds.
Example: >>> session = requests.Session() >>> session.auth = KerbAuth()
- __init__(timeout=180)¶
OAuth 1.0a Helper¶
OAuth10aDance helps with determining the user creds, compared to using OAuth1 directly.
- class beanbag.auth.OAuth10aDance(req_token=None, acc_token=None, authorize=None, client_key=None, client_secret=None, user_key=None, user_secret=None)¶
- __init__(req_token=None, acc_token=None, authorize=None, client_key=None, client_secret=None, user_key=None, user_secret=None)¶
Create an OAuth10aDance object to negotiatie OAuth 1.0a credentials.
The first set of parameters are the URLs to the OAuth 1.0a service you wish to authenticate against.
Parameters: - req_token – Request token URL
- authorize – User authorization URL
- acc_token – Access token URL
These parameters (and the others) may also be provided by subclassing the OAuth10aDance class, eg:
Example: >>> class OAuthDanceTwitter(beanbag.OAuth10aDance): ... req_token = "https://api.twitter.com/oauth/request_token" ... authorize = "https://api.twitter.com/oauth/authorize" ... acc_token = "https://api.twitter.com/oauth/access_token"
The second set of parameters identify the client application to the server, and need to be obtained outside of the OAuth protocol.
Parameters: - client_key – client/consumer key
- client_secret – client/consumer secret
The final set of parameters identify the user to server. These may be left as None, and obtained using the OAuth 1.0a protocol via the obtain_creds() method or using the get_auth_url() and verify_user() methods.
Parameters: - user_key – user key
- user_secret – user secret
Assuming OAuthDanceTwitter is defined as above, and you have obtained the client key and secret (see https://apps.twitter.com/ for twitter) as k and s, then putting these together looks like:
Example: >>> oauthdance = OAuthDanceTwitter(client_key=k, client_secret=s) >>> oauthdance.obtain_creds() Please go to url: https://api.twitter.com/oauth/authorize?oauth_token=... Please input the verifier: 1111111 >>> session = requests.Session() >>> session.auth = oauthdance.oauth()
- have_creds()¶
Check whether all credentials are filled in
- get_auth_url()¶
URL for user to obtain verification code
- verify_user(verifier)¶
Set user key and secret based on verification code
- obtain_creds()¶
Fill in credentials by interacting with the user (input/print)
- oauth()¶
Create an OAuth1 authenticator using client and user credentials
beanbag.attrdict – Access dict members by attribute¶
AttrDict¶
This module provides the AttrDict class, which allows you to access dict members via attribute access, allowing similar syntax to javascript objects. For example:
d = {"foo": 1, "bar": {"sub": {"subsub": 2}}}
ad = AttrDict(d)
assert ad["foo"] == ad["foo"]
assert ad.foo == 1
assert ad.bar.sub.subsub == 2
Note that AttrDict simply provides a view on the native dict. That dict can be obtained using the plus operator like so:
ad = AttrDict(d)
assert +ad is d
This allows use of native dict methods such as d.update() or d.items(). Note that attribute access binds more tightly than plus, so brackets will usually need to be used, eg: (+ad.bar).items().
An AttrDict can also be directly used as an iterator (for key in attrdict: ...) and as a container (if key in attrdict: ...).
- class beanbag.attrdict.AttrDict(base=None)¶
- __delattr__(attr)¶
del self.attr
- __delitem__(item)¶
del self[item]
- __eq__(other)¶
self == other
- __getattr__(attr)¶
self.attr
- __getitem__(item)¶
self[attr]
- __init__(base=None)¶
Provide an AttrDict view of a dictionary.
Parameters: base – dictionary/list to be viewed
- __ne__(other)¶
self != other
- __pos__()¶
View underlying dict object
- __setattr__(attr, val)¶
self.attr = val
- __setitem__(item, val)¶
self[item] = val
beanbag.namespace¶
The beanbag.namespace module allows defining classes that provide arbitrary namespace behaviour. This is what allows the other beanbag modules to provide their clever syntactic sugar.
A entry in a namespace is identified by two components: a base and a path. The base is constructed once for a namespace and is common to all entries in the namespace, and each entry’s path is used to differentiate them. For example, with AttrDict, the base is the underlying dictionary (d), while the path is the sequence of references into that dictionary (eg, ("foo", "bar") corresponding to d["foo"]["bar"]). The reason for splitting these apart is mostly efficiency – the path element needs to be cheap and easy to construct and copy since that may need to happen for an attribute access.
To define a namespace you provide a class that inherits from beanbag.namespace.Namespace and defines the methods the base class should have. The NamespaceMeta metaclass then creates a new base class containing these methods, and builds the namespace class on top of that base class, mapping Python’s special method names to the corresponding base class methods, minus the underscores. For example, to define the behavour of the ~ operator (aka __invert__(self)), the Base class defines a method:
def invert(self, path):
...
The code can rely on the base value being self, and the path being path, then do whatever calculation is necessary to create a result. If that result should be a different entry in the same namespace, that can be created by invoking self.namespace(newpath).
In order to make inplace operations work more smoothly, returning None from those options will be automatically treated as returning the original namespace object (ie self.namespace(path), without the overhead of reconstructing the object). This is primarily to make it easier to avoid the “double setting” behaviour of python’s inplace operations, ie where a[i] += j is converted into:
tmp = a.__getitem__(i) # tmp = a[i]
res = tmp.__iadd__(j) # tmp += j
a.__setitem__(i, res) # a[i] = tmp
In particular, implementations of setitem and setattr can avoid poor behaviour here by testing whether the value being set (res) is already the existing value, and performing a no-op if so. The SettableHierarchialNS class implements this behaviour.
NamespaceMeta¶
The NamespaceMeta metaclass provides the magic for creating arbitrary namespaces from Base classes as discussed above. When set as the metaclass for a class, it will turn a base class into a namespace class directly, while constructing an appropriate base class for the namespace to use.
- class beanbag.namespace.NamespaceMeta¶
- __invert__()¶
Obtain base class for namespace
- __module__ = 'beanbag.namespace'¶
- static __new__(mcls, name, bases, nmspc)¶
- classmethod deferfn(mcls, cls, nsdict, basefnname, inum=False, attr=False)¶
- classmethod make_namespace(mcls, cls)¶
create a unique Namespace class based on provided class
- ops = ['repr', 'str', 'call', 'bool', 'getitem', 'setitem', 'delitem', 'len', 'iter', 'reversed', 'contains', 'enter', 'exit', 'pos', 'neg', 'invert', 'eq', 'ne', 'lt', 'le', 'gt', 'ge', 'add', 'sub', 'mul', 'pow', 'div', 'floordiv', 'lshift', 'rshift', 'and', 'or', 'xor', 'radd', 'rsub', 'rmul', 'rpow', 'rdiv', 'rfloordiv', 'rlshift', 'rrshift', 'rand', 'ror', 'rxor']¶
- ops_attr = ['getattr', 'setattr', 'delattr']¶
- ops_inum = ['iadd', 'isub', 'imul', 'ipow', 'idiv', 'ifloordiv', 'ilshift', 'irshift', 'iand', 'ior', 'ixor']¶
- static wrap_path_fn(basefn)¶
- static wrap_path_fn_attr(basefn)¶
- static wrap_path_fn_inum(basefn)¶
NamespaceBase¶
The generated base class will inherit from NamespaceBase (or the base class corresponding to any namespaces the namespace class inherits from), and will have a Namespace attribute referencing the namespace class. Further, the generated base class can be accessed by using the inverse opertor on the namespace class, ie MyNamespaceBase = ~MyNamespace.
Namespace¶
Namespace provides a trivial Base implementation. It’s primarily useful as a parent class for inheritance, so that you don’t have explicitly set NamespaceMeta as your metaclass.
- class beanbag.namespace.Namespace(*args, **kwargs)¶
HierarchialNS¶
HierarchialNS provides a simple basis for producing namespaces with freeform attribute and item hierarchies, eg, where you might have something like ns.foo.bar["baz"].
By default, this class specifies a path as a tuple of attributes, but this can be changed by overriding the path and _get methods. If some conversion is desired on either attribute or item access, the attr and item methods can be overridden respectively.
Otherwise, to get useful behaviour from this class, you probably want to provide some additional methods, such as __call__.
- class beanbag.namespace.HierarchialNS¶
Bases: beanbag.namespace.Namespace
- __eq__(other)¶
self == other
- __getattr__(attr)¶
self.attr
- __getitem__(item)¶
self[attr]
- __init__()¶
- __module__ = 'beanbag.namespace'¶
- __ne__(other)¶
self != other
- __repr__()¶
Human readable representation of object
- __str__()¶
Returns path joined by dots
SettableHierarchialNS¶
SettableHierarchialNS is intended to make life slightly easier if you want to be able to assign to your hierarchial namespace. It provides set and delete methods that you can implement, without having to go to the trouble of implementing both item and attribute variants of both functions.
This class implements the check for “setting to self” mentioned earlier in order to prevent inplace operations having two effects. It uses the eq method to test for equality.
- class beanbag.namespace.SettableHierarchialNS(*args, **kwargs)¶
Bases: beanbag.namespace.HierarchialNS
- __delattr__(attr)¶
del self.attr
- __delitem__(item)¶
del self[item]
- __eq__(other)¶
self == other
- __getattr__(attr)¶
self.attr
- __getitem__(item)¶
self[attr]
- __init__(*args, **kwargs)¶
- __module__ = 'beanbag.namespace'¶
- __ne__(other)¶
self != other
- __repr__()¶
Human readable representation of object
- __setattr__(attr, val)¶
self.attr = val
- __setitem__(item, val)¶
self[item] = val
- __str__()¶
Returns path joined by dots
sig_adapt¶
This is a helper function to make that generated methods in the namespace object provide more useful help.
- beanbag.namespace.sig_adapt(sigfn, dropargs=None, name=None)¶
Function decorator that changes the name and (optionally) signature of a function to match another function. This is useful for making the help of generic wrapper functions match the functions they’re wrapping. For example:
def foo(a, b, c, d=None): pass @sig_adapt(foo) def myfn(*args, **kwargs): pass
The optional “name” parameter allows renaming the function to something different to the original function’s name.
The optional “dropargs” parameter allows dropping arguments by position or name. (Note positions are 0 based, so to convert foo(self, a, b) to foo(a, b) specify dropargs=(“self”,) or dropargs=(0,))
Examples¶
What follows are some examples of using BeanBag for various services.
GitHub¶
GitHub’s REST API, using JSON for data and either HTTP Basic Auth or OAuth2 for authentication. Basic Auth is perfect for a command line app, since the user can just use their github account password directly.
The following example uses the github API to list everyone who’s starred one of your repos, and which repo it is that they’ve starred.
#!/usr/bin/env python
import beanbag.v1 as beanbag
import os
import requests
sess = requests.Session()
sess.auth = (os.environ["GITHUB_ACCT"], os.environ["GITHUB_PASS"])
github = beanbag.BeanBag("https://api.github.com/", session=sess)
myuser = github.user()
me = myuser["login"]
repos = github.users[me].repos()
repo = {}
who = {}
for r in repos:
rn = r["name"]
repo[rn] = github.repos[me][rn]()
stars = github.repos[me][rn].stargazers()
for s in stars:
sn = s["login"]
if sn not in who:
who[sn] = set()
who[sn].add(rn)
for w in sorted(who):
print("%s:" % (w,))
for rn in sorted(who[w]):
print(" %s -- %s" % (rn, repo[rn]["description"]))
Twitter¶
Twitter’s REST API is slightly more complicated. It still uses JSON, but requires OAuth 1.0a to be used for authentication. OAuth is designed primarily for webapps, where the application is controlled by a third party. In particular it is designed to allow an “application” to authenticate as “authorised by a particular user”, rather than allowing the application to directly authenticate itself as the user (eg, by using the user’s username and password directly, as we did above with github).
This in turn means that the application has to be able to identify itself. This is done by gaining “client credential”, in Twitter’s case via Twitter Apps.
The process of having an application to ask a user to provide a token that allows it to access Twitter on behalf of the user is encapsulated in the OAuth10aDance class. In the example below is subclassed in order to provide the Twitter-specific URLs that the user and application will need to visit in order to gain the right tokens to do the authentication. The obtain_creds() method is called, which will instruct the user to enter any necessary credentials, after which a Session object is created and setup to perform OAuth authentication using the provided credentials.
The final minor complication is that Twitter’s endpoints all end with ”.json”, which would be annoying to have to specify via beanbag (since ”.” is not a valid part of an attribute). The ext= keyword argument of the BeanBag constructor is used to supply this as the standard extension for all URLs in the Twitter API.
#!/usr/bin/env python
import beanbag
import requests
class OAuthDanceTwitter(beanbag.OAuth10aDance):
req_token = "https://api.twitter.com/oauth/request_token"
authorize = "https://api.twitter.com/oauth/authorize"
acc_token = "https://api.twitter.com/oauth/access_token"
client_key, client_secret = (None, None)
user_key, user_secret = (None, None)
oauthDance = OAuthDanceTwitter(
client_key=client_key, client_secret=client_secret,
user_key=user_key, user_secret=user_secret)
oauthDance.obtain_creds()
session = requests.Session()
session.auth = oauthDance.oauth()
twitter = beanbag.BeanBag("https://api.twitter.com/1.1/", ext=".json",
session=session)
myacct = twitter.account.settings()
me = myacct["screen_name"]
tweets = twitter.statuses.user_timeline(screen_name=me, count=7)
for tweet in tweets:
print(repr(tweet["text"]))
Credits¶
Code contributors:
- Anthony Towns <aj@erisian.com.au>
- Gary Martin <gjm@apache.org>
- Lubos Kocman <lkocman@redhat.com>
- Daniel Mach <dmach@redhat.com>
Documentation contributors:
- Anthony Towns <aj@erisian.com.au>
Test case contributors and bug reporters:
- Anthony Towns <aj@erisian.com.au>
- Russell Stuart <russell-github@stuart.id.au>
BeanBag is inspired by Kadir Pekel’s Hammock, though sadly only shares a license, and not any actual code. Hammock is available from https://github.com/kadirpekel/hammock.