Welcome to pymemcache’s documentation!

Contents:

Getting started!

A comprehensive, fast, pure-Python memcached client library.

Basic Usage

from pymemcache.client.base import Client

client = Client('localhost')
client.set('some_key', 'some_value')
result = client.get('some_key')

The server to connect to can be specified in a number of ways.

If using TCP connections over IPv4 or IPv6, the server parameter can be passed a host string, a host:port string, or a (host, port) 2-tuple. The host part may be a domain name, an IPv4 address, or an IPv6 address. The port may be omitted, in which case it will default to 11211.

ipv4_client = Client('127.0.0.1')
ipv4_client_with_port = Client('127.0.0.1:11211')
ipv4_client_using_tuple = Client(('127.0.0.1', 11211))

ipv6_client = Client('[::1]')
ipv6_client_with_port = Client('[::1]:11211')
ipv6_client_using_tuple = Client(('::1', 11211))

domain_client = Client('localhost')
domain_client_with_port = Client('localhost:11211')
domain_client_using_tuple = Client(('localhost', 11211))

Note that IPv6 may be used in preference to IPv4 when passing a domain name as the host if an IPv6 address can be resolved for that domain.

You can also connect to a local memcached server over a UNIX domain socket by passing the socket’s path to the client’s server parameter. An optional unix: prefix may be used for compatibility in code that uses other client libraries that require it.

client = Client('/run/memcached/memcached.sock')
client_with_prefix = Client('unix:/run/memcached/memcached.sock')

Using a client pool

pymemcache.client.base.PooledClient is a thread-safe client pool that provides the same API as pymemcache.client.base.Client. It’s useful in for cases when you want to maintain a pool of already-connected clients for improved performance.

from pymemcache.client.base import PooledClient

client = PooledClient('127.0.0.1', max_pool_size=4)

Using a memcached cluster

This will use a consistent hashing algorithm to choose which server to set/get the values from. It will also automatically rebalance depending on if a server goes down.

from pymemcache.client.hash import HashClient

client = HashClient([
    '127.0.0.1:11211',
    '127.0.0.1:11212',
])
client.set('some_key', 'some value')
result = client.get('some_key')

Key distribution is handled by the hasher argument in the constructor. The default is the built-in pymemcache.client.rendezvous.RendezvousHash hasher. It uses the built-in pymemcache.client.murmur3.murmur3_32 implementation to distribute keys on servers. Overriding these two parts can be used to change how keys are distributed. Changing the hashing algorithm can be done by setting the hash_function argument in the RendezvousHash constructor.

Rebalancing in the pymemcache.client.hash.HashClient functions as follows:

  1. A pymemcache.client.hash.HashClient is created with 3 nodes, node1, node2 and node3.

  2. A number of values are set in the client using set and set_many. Example:

    • key1 -> node2

    • key2 -> node3

    • key3 -> node3

    • key4 -> node1

    • key5 -> node2

  3. Subsequent get calls will hash to the correct server and requests are routed accordingly.

  4. node3 goes down.

  5. The hashclient tries to get("key2") but detects the node as down. This causes it to mark the node as down. Removing it from the hasher. The hasclient can attempt to retry the operation based on the retry_attempts and retry_timeout arguments. If ignore_exc is set, this is treated as a miss, if not, an exception will be raised.

  6. Any get/set for key2 and key3 will now hash differently, example:

    • key2 -> node2

    • key3 -> node1

  7. After the amount of time specified in the dead_timeout argument, node3 is added back into the hasher and will be retried for any future operations.

Using the built-in retrying mechanism

The library comes with retry mechanisms that can be used to wrap all kinds of pymemcache clients. The wrapper allows you to define the exceptions that you want to handle with retries, which exceptions to exclude, how many attempts to make and how long to wait between attempts.

The RetryingClient wraps around any of the other included clients and will have the same methods. For this example, we’re just using the base Client.

from pymemcache.client.base import Client
from pymemcache.client.retrying import RetryingClient
from pymemcache.exceptions import MemcacheUnexpectedCloseError

base_client = Client(("localhost", 11211))
client = RetryingClient(
    base_client,
    attempts=3,
    retry_delay=0.01,
    retry_for=[MemcacheUnexpectedCloseError]
)
client.set('some_key', 'some value')
result = client.get('some_key')

The above client will attempt each call three times with a wait of 10ms between each attempt, as long as the exception is a MemcacheUnexpectedCloseError.

Using TLS

Memcached supports authentication and encryption via TLS since version 1.5.13.

A Memcached server running with TLS enabled will only accept TLS connections.

To enable TLS in pymemcache, pass a valid TLS context to the client’s tls_context parameter:

import ssl
from pymemcache.client.base import Client

context = ssl.create_default_context(
    cafile="my-ca-root.crt",
)

client = Client('localhost', tls_context=context)
client.set('some_key', 'some_value')
result = client.get('some_key')

Serialization

import json
from pymemcache.client.base import Client

class JsonSerde(object):
    def serialize(self, key, value):
        if isinstance(value, str):
            return value, 1
        return json.dumps(value), 2

    def deserialize(self, key, value, flags):
       if flags == 1:
           return value
       if flags == 2:
           return json.loads(value)
       raise Exception("Unknown serialization format")

client = Client('localhost', serde=JsonSerde())
client.set('key', {'a':'b', 'c':'d'})
result = client.get('key')

pymemcache provides a default pickle-based serializer:

from pymemcache.client.base import Client
from pymemcache import serde

class Foo(object):
  pass

client = Client('localhost', serde=serde.pickle_serde)
client.set('key', Foo())
result = client.get('key')

The serializer uses the highest pickle protocol available. In order to make sure multiple versions of Python can read the protocol version, you can specify the version by explicitly instantiating pymemcache.serde.PickleSerde:

client = Client('localhost', serde=serde.PickleSerde(pickle_version=2))

Deserialization with Python 3

Values passed to the serde.deserialize() method will be bytestrings. It is therefore necessary to encode and decode them correctly. Here’s a version of the JsonSerde from above which is more careful with encodings:

class JsonSerde(object):
    def serialize(self, key, value):
        if isinstance(value, str):
            return value.encode('utf-8'), 1
        return json.dumps(value).encode('utf-8'), 2

    def deserialize(self, key, value, flags):
       if flags == 1:
           return value.decode('utf-8')
       if flags == 2:
           return json.loads(value.decode('utf-8'))
       raise Exception("Unknown serialization format")

Interacting with pymemcache

For testing purpose pymemcache can be used in an interactive mode by using the python interpreter or again ipython and tools like tox.

One main advantage of using tox to interact with pymemcache is that it comes with its own virtual environments. It will automatically install pymemcache and fetch all the needed requirements at run. See the example below:

$ podman run --publish 11211:11211 -it --rm --name memcached memcached
$ tox -e venv -- python
>>> from pymemcache.client.base import Client
>>> client = Client('127.0.0.1')
>>> client.set('some_key', 'some_value')
True
>>> client.get('some_key')
b'some_value'
>>> print(client.get.__doc__)
     The memcached "get" command, but only for one key, as a convenience.
     Args:
       key: str, see class docs for details.
       default: value that will be returned if the key was not found.
     Returns:
       The value for the key, or default if the key wasn't found.

You can instantiate all the classes and clients offered by pymemcache.

Your client will remain open until you decide to close it or until you decide to quit your interpreter. It can allow you to see what happens if your server is abruptly closed. Below is an example.

Starting your server:

$ podman run --publish 11211:11211 -it --name memcached memcached

Starting your client and set some keys:

$ tox -e venv -- python
>>> from pymemcache.client.base import Client
>>> client = Client('127.0.0.1')
>>> client.set('some_key', 'some_value')
True

Restarting the server:

$ podman restart memcached

The previous client is still open, now try to retrieve some keys:

>>> print(client.get('some_key'))
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/user/pymemcache/pymemcache/client/base.py", line 535, in get
    return self._fetch_cmd(b'get', [key], False).get(key, default)
  File "/home/user/pymemcache/pymemcache/client/base.py", line 910, in _fetch_cmd
    buf, line = _readline(self.sock, buf)
  File "/home/user/pymemcache/pymemcache/client/base.py", line 1305, in _readline
    raise MemcacheUnexpectedCloseError()
pymemcache.exceptions.MemcacheUnexpectedCloseError

We can see that the connection has been closed.

You can also pass a command directly from CLI parameters and get output directly:

$ tox -e venv -- python -c "from pymemcache.client.base import Client; client = Client('127.0.01'); print(client.get('some_key'))"
b'some_value'

This kind of usage is useful for debug sessions or to dig manually into your server.

Key Constraints

This client implements the ASCII protocol of memcached. This means keys should not contain any of the following illegal characters:

Keys cannot have spaces, new lines, carriage returns, or null characters. We suggest that if you have unicode characters, or long keys, you use an effective hashing mechanism before calling this client.

At Pinterest, we have found that murmur3 hash is a great candidate for this. Alternatively you can set allow_unicode_keys to support unicode keys, but beware of what unicode encoding you use to make sure multiple clients can find the same key.

Best Practices

  • Always set the connect_timeout and timeout arguments in the pymemcache.client.base.Client constructor to avoid blocking your process when memcached is slow. You might also want to enable the no_delay option, which sets the TCP_NODELAY flag on the connection’s socket.

  • Use the noreply flag for a significant performance boost. The noreply flag is enabled by default for “set”, “add”, “replace”, “append”, “prepend”, and “delete”. It is disabled by default for “cas”, “incr” and “decr”. It obviously doesn’t apply to any get calls.

  • Use pymemcache.client.base.Client.get_many() and pymemcache.client.base.Client.gets_many() whenever possible, as they result in fewer round trip times for fetching multiple keys.

  • Use the ignore_exc flag to treat memcache/network errors as cache misses on calls to the get* methods. This prevents failures in memcache, or network errors, from killing your web requests. Do not use this flag if you need to know about errors from memcache, and make sure you have some other way to detect memcache server failures.

  • Unless you have a known reason to do otherwise, use the provided serializer in pymemcache.serde.pickle_serde for any de/serialization of objects.

Warning

noreply will not read errors returned from the memcached server.

If a function with noreply=True causes an error on the server, it will still succeed and your next call which reads a response from memcached may fail unexpectedly.

pymemcached will try to catch and stop you from sending malformed inputs to memcached, but if you are having unexplained errors, setting noreply=False may help you troubleshoot the issue.

pymemcache

pymemcache package

Subpackages

pymemcache.client package
Submodules
pymemcache.client.base module
class pymemcache.client.base.Client(server: ~typing.Union[~typing.Tuple[str, int], str], serde=None, serializer=None, deserializer=None, connect_timeout: ~typing.Optional[float] = None, timeout: ~typing.Optional[float] = None, no_delay: bool = False, ignore_exc: bool = False, socket_module: module = <module 'socket' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/socket.py'>, socket_keepalive: ~typing.Optional[~pymemcache.client.base.KeepaliveOpts] = None, key_prefix: bytes = b'', default_noreply: bool = True, allow_unicode_keys: bool = False, encoding: str = 'ascii', tls_context: ~typing.Optional[~ssl.SSLContext] = None)

Bases: object

A client for a single memcached server.

Server Connection

The server parameter controls how the client connects to the memcached server. You can either use a (host, port) tuple for a TCP connection or a string containing the path to a UNIX domain socket.

The connect_timeout and timeout parameters can be used to set socket timeout values. By default, timeouts are disabled.

When the no_delay flag is set, the TCP_NODELAY socket option will also be set. This only applies to TCP-based connections.

Lastly, the socket_module allows you to specify an alternate socket implementation (such as gevent.socket).

Keys and Values

Keys must have a __str__() method which should return a str with no more than 250 ASCII characters and no whitespace or control characters. Unicode strings must be encoded (as UTF-8, for example) unless they consist only of ASCII characters that are neither whitespace nor control characters.

Values must have a __str__() method to convert themselves to a byte string. Unicode objects can be a problem since str() on a Unicode object will attempt to encode it as ASCII (which will fail if the value contains code points larger than U+127). You can fix this with a serializer or by just calling encode on the string (using UTF-8, for instance).

If you intend to use anything but str as a value, it is a good idea to use a serializer. The pymemcache.serde library has an already implemented serializer which pickles and unpickles data.

Serialization and Deserialization

The constructor takes an optional object, the “serializer/deserializer” (“serde”), which is responsible for both serialization and deserialization of objects. That object must satisfy the serializer interface by providing two methods: serialize and deserialize. serialize takes two arguments, a key and a value, and returns a tuple of two elements, the serialized value, and an integer in the range 0-65535 (the “flags”). deserialize takes three parameters, a key, value, and flags, and returns the deserialized value.

Here is an example using JSON for non-str values:

class JSONSerde(object):
    def serialize(self, key, value):
        if isinstance(value, str):
            return value, 1
        return json.dumps(value), 2

    def deserialize(self, key, value, flags):
        if flags == 1:
            return value

        if flags == 2:
            return json.loads(value)

        raise Exception("Unknown flags for value: {1}".format(flags))

Note

Most write operations allow the caller to provide a flags value to support advanced interaction with the server. This will override the “flags” value returned by the serializer and should therefore only be used when you have a complete understanding of how the value should be serialized, stored, and deserialized.

Error Handling

All of the methods in this class that talk to memcached can throw one of the following exceptions:

Instances of this class maintain a persistent connection to memcached which is terminated when any of these exceptions are raised. The next call to a method on the object will result in a new connection being made to memcached.

__init__(server: ~typing.Union[~typing.Tuple[str, int], str], serde=None, serializer=None, deserializer=None, connect_timeout: ~typing.Optional[float] = None, timeout: ~typing.Optional[float] = None, no_delay: bool = False, ignore_exc: bool = False, socket_module: module = <module 'socket' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/socket.py'>, socket_keepalive: ~typing.Optional[~pymemcache.client.base.KeepaliveOpts] = None, key_prefix: bytes = b'', default_noreply: bool = True, allow_unicode_keys: bool = False, encoding: str = 'ascii', tls_context: ~typing.Optional[~ssl.SSLContext] = None)

Constructor.

Parameters

  • server – tuple(hostname, port) or string containing a UNIX socket path.

  • serde – optional serializer object, see notes in the class docs.

  • serializer – deprecated serialization function

  • deserializer – deprecated deserialization function

  • connect_timeout – optional float, seconds to wait for a connection to the memcached server. Defaults to “forever” (uses the underlying default socket timeout, which can be very long).

  • timeout – optional float, seconds to wait for send or recv calls on the socket connected to memcached. Defaults to “forever” (uses the underlying default socket timeout, which can be very long).

  • no_delay – optional bool, set the TCP_NODELAY flag, which may help with performance in some cases. Defaults to False.

  • ignore_exc – optional bool, True to cause the “get”, “gets”, “get_many” and “gets_many” calls to treat any errors as cache misses. Defaults to False.

  • socket_module – socket module to use, e.g. gevent.socket. Defaults to the standard library’s socket module.

  • socket_keepalive – Activate the socket keepalive feature by passing a KeepaliveOpts structure in this parameter. Disabled by default (None). This feature is only supported on Linux platforms.

  • key_prefix – Prefix of key. You can use this as namespace. Defaults to b’’.

  • default_noreply – bool, the default value for ‘noreply’ as passed to store commands (except from cas, incr, and decr, which default to False).

  • allow_unicode_keys – bool, support unicode (utf8) keys

  • encoding – optional str, controls data encoding (defaults to ‘ascii’).

Notes

The constructor does not make a connection to memcached. The first call to a method on the object will do that.

add(key: Union[bytes, str], value: Any, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None) → bool

The memcached “add” command.

Parameters

  • key – str, see class docs for details.

  • value – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

If noreply is True (or if it is unset and self.default_noreply is True), the return value is always True. Otherwise the return value is True if the value was stored, and False if it was not (because the key already existed).

append(key: Union[bytes, str], value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None) → bool

The memcached “append” command.

Parameters

  • key – str, see class docs for details.

  • value – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

True.

cache_memlimit(memlimit) → bool

The memcached “cache_memlimit” command.

Parameters

memlimit – int, the number of megabytes to set as the new cache memory limit.

Returns

If no exception is raised, always returns True.

cas(key, value, cas, expire: int = 0, noreply=False, flags: Optional[int] = None) → Optional[bool]

The memcached “cas” command.

Parameters

  • key – str, see class docs for details.

  • value – str, see class docs for details.

  • cas – int or str that only contains the characters ‘0’-‘9’.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, False to wait for the reply (the default).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

If noreply is True (or if it is unset and self.default_noreply is True), the return value is always True. Otherwise returns None if the key didn’t exist, False if it existed but had a different cas value and True if it existed and was changed.

check_key(key: Union[bytes, str], key_prefix: bytes) → bytes

Checks key and add key_prefix.

close() → None

Close the connection to memcached, if it is open. The next call to a method that requires a connection will re-open it.

decr(key: Union[bytes, str], value: int, noreply: Optional[bool] = False) → Optional[int]

The memcached “decr” command.

Parameters

  • key – str, see class docs for details.

  • value – int, the amount by which to decrement the value.

  • noreply – optional bool, False to wait for the reply (the default).

Returns

If noreply is True, always returns None. Otherwise returns the new value of the key, or None if the key wasn’t found.

delete(key: Union[bytes, str], noreply: Optional[bool] = None) → bool

The memcached “delete” command.

Parameters

  • key – str, see class docs for details.

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

Returns

If noreply is True (or if it is unset and self.default_noreply is True), the return value is always True. Otherwise returns True if the key was deleted, and False if it wasn’t found.

delete_many(keys: Iterable[Union[bytes, str]], noreply: Optional[bool] = None) → bool

A convenience function to delete multiple keys.

Parameters

  • keys – list(str), the list of keys to delete.

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

Returns

True. If an exception is raised then all, some or none of the keys may have been deleted. Otherwise all the keys have been sent to memcache for deletion and if noreply is False, they have been acknowledged by memcache.

delete_multi(keys: Iterable[Union[bytes, str]], noreply: Optional[bool] = None) → bool

A convenience function to delete multiple keys.

Parameters

  • keys – list(str), the list of keys to delete.

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

Returns

True. If an exception is raised then all, some or none of the keys may have been deleted. Otherwise all the keys have been sent to memcache for deletion and if noreply is False, they have been acknowledged by memcache.

disconnect_all() → None

Close the connection to memcached, if it is open. The next call to a method that requires a connection will re-open it.

flush_all(delay: int = 0, noreply: Optional[bool] = None) → bool

The memcached “flush_all” command.

Parameters

  • delay – optional int, the number of seconds to wait before flushing, or zero to flush immediately (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

Returns

True.

gat(key: Union[bytes, str], expire: int = 0, default: Optional[Any] = None) → Any

The memcached “gat” command, but only for one key, as a convenience.

Parameters

  • key – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • default – value that will be returned if the key was not found.

Returns

The value for the key, or default if the key wasn’t found.

gats(key: Union[bytes, str], expire: int = 0, default: Optional[Any] = None, cas_default: Optional[Any] = None) → Tuple[Any, Any]

The memcached “gats” command, but only for one key, as a convenience.

Parameters

  • key – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • default – value that will be returned if the key was not found.

  • cas_default – same behaviour as default argument.

Returns

A tuple of (value, cas) or (default, cas_defaults) if the key was not found.

get(key: Union[bytes, str], default: Optional[Any] = None) → Any

The memcached “get” command, but only for one key, as a convenience.

Parameters

  • key – str, see class docs for details.

  • default – value that will be returned if the key was not found.

Returns

The value for the key, or default if the key wasn’t found.

get_many(keys: Iterable[Union[bytes, str]]) → Dict[Union[bytes, str], Any]

The memcached “get” command.

Parameters

keys – list(str), see class docs for details.

Returns

A dict in which the keys are elements of the “keys” argument list and the values are values from the cache. The dict may contain all, some or none of the given keys.

get_multi(keys: Iterable[Union[bytes, str]]) → Dict[Union[bytes, str], Any]

The memcached “get” command.

Parameters

keys – list(str), see class docs for details.

Returns

A dict in which the keys are elements of the “keys” argument list and the values are values from the cache. The dict may contain all, some or none of the given keys.

gets(key: Union[bytes, str], default: Optional[Any] = None, cas_default: Optional[Any] = None) → Tuple[Any, Any]

The memcached “gets” command for one key, as a convenience.

Parameters

  • key – str, see class docs for details.

  • default – value that will be returned if the key was not found.

  • cas_default – same behaviour as default argument.

Returns

A tuple of (value, cas) or (default, cas_defaults) if the key was not found.

gets_many(keys: Iterable[Union[bytes, str]]) → Dict[Union[bytes, str], Tuple[Any, Any]]

The memcached “gets” command.

Parameters

keys – list(str), see class docs for details.

Returns

A dict in which the keys are elements of the “keys” argument list and the values are tuples of (value, cas) from the cache. The dict may contain all, some or none of the given keys.

incr(key: Union[bytes, str], value: int, noreply: Optional[bool] = False) → Optional[int]

The memcached “incr” command.

Parameters

  • key – str, see class docs for details.

  • value – int, the amount by which to increment the value.

  • noreply – optional bool, False to wait for the reply (the default).

Returns

If noreply is True, always returns None. Otherwise returns the new value of the key, or None if the key wasn’t found.

prepend(key, value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)

The memcached “prepend” command.

Parameters

  • key – str, see class docs for details.

  • value – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

True.

quit() → None

The memcached “quit” command.

This will close the connection with memcached. Calling any other method on this object will re-open the connection, so this object can be re-used after quit.

raw_command(command: Union[str, bytes], end_tokens: Union[str, bytes] = '\r\n') → bytes

Sends an arbitrary command to the server and parses the response until a specified token is encountered.

Parameters

  • command – str|bytes: The command to send.

  • end_tokens – str|bytes: The token expected at the end of the response. If the end_token is not found, the client will wait until the timeout specified in the constructor.

Returns

The response from the server, with the end_token removed.

replace(key: Union[bytes, str], value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None) → bool

The memcached “replace” command.

Parameters

  • key – str, see class docs for details.

  • value – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

If noreply is True (or if it is unset and self.default_noreply is True), the return value is always True. Otherwise returns True if the value was stored and False if it wasn’t (because the key didn’t already exist).

set(key: Union[bytes, str], value: Any, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None) → Optional[bool]

The memcached “set” command.

Parameters

  • key – str, see class docs for details.

  • value – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

If no exception is raised, always returns True. If an exception is raised, the set may or may not have occurred. If noreply is True, then a successful return does not guarantee a successful set.

set_many(values: Dict[Union[bytes, str], Any], expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None) → List[Union[bytes, str]]

A convenience function for setting multiple values.

Parameters

  • values – dict(str, str), a dict of keys and values, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

Returns a list of keys that failed to be inserted. If noreply is True, always returns empty list.

set_multi(values: Dict[Union[bytes, str], Any], expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None) → List[Union[bytes, str]]

A convenience function for setting multiple values.

Parameters

  • values – dict(str, str), a dict of keys and values, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

  • flags – optional int, arbitrary bit field used for server-specific flags

Returns

Returns a list of keys that failed to be inserted. If noreply is True, always returns empty list.

shutdown(graceful: bool = False) → None

The memcached “shutdown” command.

This will request shutdown and eventual termination of the server, optionally preceded by a graceful stop of memcached’s internal state machine. Note that the server needs to have been started with the shutdown protocol command enabled with the –enable-shutdown flag.

Parameters

graceful – optional bool, True to request a graceful shutdown with SIGUSR1 (defaults to False, i.e. SIGINT shutdown).

stats(*args)

The memcached “stats” command.

The returned keys depend on what the “stats” command returns. A best effort is made to convert values to appropriate Python types, defaulting to strings when a conversion cannot be made.

Parameters

*arg – extra string arguments to the “stats” command. See the memcached protocol documentation for more information.

Returns

A dict of the returned stats.

touch(key: Union[bytes, str], expire: int = 0, noreply: Optional[bool] = None) → bool

The memcached “touch” command.

Parameters

  • key – str, see class docs for details.

  • expire – optional int, number of seconds until the item is expired from the cache, or zero for no expiry (the default).

  • noreply – optional bool, True to not wait for the reply (defaults to self.default_noreply).

Returns

True if the expiration time was updated, False if the key wasn’t found.

version() → bytes

The memcached “version” command.

Returns

A string of the memcached version.

class pymemcache.client.base.KeepaliveOpts(idle: int = 1, intvl: int = 1, cnt: int = 5)

Bases: object

A configuration structure to define the socket keepalive.

This structure must be passed to a client. The client will configure its socket keepalive by using the elements of the structure.

Parameters

  • idle – The time (in seconds) the connection needs to remain idle before TCP starts sending keepalive probes. Should be a positive integer most greater than zero.

  • intvl – The time (in seconds) between individual keepalive probes. Should be a positive integer most greater than zero.

  • cnt – The maximum number of keepalive probes TCP should send before dropping the connection. Should be a positive integer most greater than zero.

__init__(idle: int = 1, intvl: int = 1, cnt: int = 5) → None
cnt
idle
intvl
class pymemcache.client.base.PooledClient(server: ~typing.Union[~typing.Tuple[str, int], str], serde=None, serializer=None, deserializer=None, connect_timeout=None, timeout=None, no_delay=False, ignore_exc=False, socket_module=<module 'socket' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/socket.py'>, socket_keepalive=None, key_prefix=b'', max_pool_size=None, pool_idle_timeout=0, lock_generator=None, default_noreply: bool = True, allow_unicode_keys=False, encoding='ascii', tls_context=None)

Bases: object

A thread-safe pool of clients (with the same client api).

Parameters

  • max_pool_size – maximum pool size to use (going above this amount triggers a runtime error), by default this is 2147483648L when not provided (or none).

  • pool_idle_timeout – pooled connections are discarded if they have been unused for this many seconds. A value of 0 indicates that pooled connections are never discarded.

  • lock_generator – a callback/type that takes no arguments that will be called to create a lock or semaphore that can protect the pool from concurrent access (for example a eventlet lock or semaphore could be used instead)

Further arguments are interpreted as for Client constructor.

Note: if serde is given, the same object will be used for all clients in the pool. Your serde object must therefore be thread-safe.

__init__(server: ~typing.Union[~typing.Tuple[str, int], str], serde=None, serializer=None, deserializer=None, connect_timeout=None, timeout=None, no_delay=False, ignore_exc=False, socket_module=<module 'socket' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/socket.py'>, socket_keepalive=None, key_prefix=b'', max_pool_size=None, pool_idle_timeout=0, lock_generator=None, default_noreply: bool = True, allow_unicode_keys=False, encoding='ascii', tls_context=None)
add(key: Union[bytes, str], value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
append(key, value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
cas(key, value, cas, expire: int = 0, noreply=False, flags: Optional[int] = None)
check_key(key: Union[bytes, str]) → bytes

Checks key and add key_prefix.

client_class

Client class used to create new clients

alias of Client

close() → None
decr(key: Union[bytes, str], value, noreply=False)
delete(key: Union[bytes, str], noreply: Optional[bool] = None) → bool
delete_many(keys: Iterable[Union[bytes, str]], noreply: Optional[bool] = None) → bool
delete_multi(keys: Iterable[Union[bytes, str]], noreply: Optional[bool] = None) → bool
disconnect_all() → None
flush_all(delay=0, noreply=None) → bool
gat(key: Union[bytes, str], expire: int = 0, default: Optional[Any] = None) → Any
gats(key: Union[bytes, str], expire: int = 0, default: Optional[Any] = None) → Any
get(key: Union[bytes, str], default: Optional[Any] = None) → Any
get_many(keys: Iterable[Union[bytes, str]]) → Dict[Union[bytes, str], Any]
get_multi(keys: Iterable[Union[bytes, str]]) → Dict[Union[bytes, str], Any]
gets(key: Union[bytes, str]) → Tuple[Any, Any]
gets_many(keys: Iterable[Union[bytes, str]]) → Dict[Union[bytes, str], Tuple[Any, Any]]
incr(key: Union[bytes, str], value, noreply=False)
prepend(key, value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
quit() → None
raw_command(command, end_tokens=b'\r\n')
replace(key, value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
set(key, value, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
set_many(values, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
set_multi(values, expire: int = 0, noreply: Optional[bool] = None, flags: Optional[int] = None)
shutdown(graceful: bool = False) → None
stats(*args)
touch(key: Union[bytes, str], expire: int = 0, noreply=None)
version() → bytes
pymemcache.client.base.check_key_helper(key: Union[bytes, str], allow_unicode_keys: bool, key_prefix: bytes = b'') → bytes

Checks key and add key_prefix.

pymemcache.client.base.normalize_server_spec(server: Union[Tuple[str, int], str]) → Union[Tuple[str, int], str]
pymemcache.client.hash module
class pymemcache.client.hash.HashClient(servers, hasher=<class 'pymemcache.client.rendezvous.RendezvousHash'>, serde=None, serializer=None, deserializer=None, connect_timeout=None, timeout=None, no_delay=False, socket_module=<module 'socket' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/socket.py'>, socket_keepalive=None, key_prefix=b'', max_pool_size=None, pool_idle_timeout=0, lock_generator=None, retry_attempts=2, retry_timeout=1, dead_timeout=60, use_pooling=False, ignore_exc=False, allow_unicode_keys=False, default_noreply=True, encoding='ascii', tls_context=None)

Bases: object

A client for communicating with a cluster of memcached servers

__init__(servers, hasher=<class 'pymemcache.client.rendezvous.RendezvousHash'>, serde=None, serializer=None, deserializer=None, connect_timeout=None, timeout=None, no_delay=False, socket_module=<module 'socket' from '/home/docs/.pyenv/versions/3.7.9/lib/python3.7/socket.py'>, socket_keepalive=None, key_prefix=b'', max_pool_size=None, pool_idle_timeout=0, lock_generator=None, retry_attempts=2, retry_timeout=1, dead_timeout=60, use_pooling=False, ignore_exc=False, allow_unicode_keys=False, default_noreply=True, encoding='ascii', tls_context=None)

Constructor.

Parameters

  • servers – list() of tuple(hostname, port) or string containing a UNIX socket path.

  • hasher – optional class three functions get_node, add_node, and remove_node defaults to Rendezvous (HRW) hash.

  • use_pooling – use py:class:.PooledClient as the default underlying class. max_pool_size and lock_generator can be used with this. default: False

  • retry_attempts – Amount of times a client should be tried before it is marked dead and removed from the pool.

  • retry_timeout (float) – Time in seconds that should pass between retry attempts.

  • dead_timeout (float) – Time in seconds before attempting to add a node back in the pool.

  • encoding – optional str, controls data encoding (defaults to ‘ascii’).

Further arguments are interpreted as for Client constructor.

add(key, *args, **kwargs)
add_server(server, port=None) → None
append(key, *args, **kwargs)
cas(key, *args, **kwargs)
client_class

Client class used to create new clients

alias of Client

close()
decr(key, *args, **kwargs)
delete(key, *args, **kwargs)
delete_many(keys, *args, **kwargs) → bool
delete_multi(keys, *args, **kwargs) → bool
disconnect_all()
flush_all(*args, **kwargs) → None
gat(key, default=None, **kwargs)
gats(key, default=None, **kwargs)
get(key, default=None, **kwargs)
get_many(keys, gets=False, *args, **kwargs)
get_multi(keys, gets=False, *args, **kwargs)
gets(key, *args, **kwargs)
gets_many(keys, *args, **kwargs)
gets_multi(keys, *args, **kwargs)
incr(key, *args, **kwargs)
prepend(key, *args, **kwargs)
quit() → None
remove_server(server, port=None) → None
replace(key, *args, **kwargs)
set(key, *args, **kwargs)
set_many(values, *args, **kwargs)
set_multi(values, *args, **kwargs)
touch(key, *args, **kwargs)
pymemcache.client.murmur3 module
pymemcache.client.murmur3.murmur3_32(data, seed=0)

MurmurHash3 was written by Austin Appleby, and is placed in the public domain. The author hereby disclaims copyright to this source code.

pymemcache.client.rendezvous module
class pymemcache.client.rendezvous.RendezvousHash(nodes=None, seed=0, hash_function=<function murmur3_32>)

Bases: object

Implements the Highest Random Weight (HRW) hashing algorithm most commonly referred to as rendezvous hashing.

Originally developed as part of python-clandestined.

Copyright (c) 2014 Ernest W. Durbin III

__init__(nodes=None, seed=0, hash_function=<function murmur3_32>)

Constructor.

add_node(node)
get_node(key)
remove_node(node)
pymemcache.client.retrying module

Module containing the RetryingClient wrapper class.

class pymemcache.client.retrying.RetryingClient(client, attempts=2, retry_delay=0, retry_for=None, do_not_retry_for=None)

Bases: object

Client that allows retrying calls for the other clients.

__init__(client, attempts=2, retry_delay=0, retry_for=None, do_not_retry_for=None)

Constructor for RetryingClient.

Parameters

  • client – Client|PooledClient|HashClient, inner client to use for performing actual work.

  • attempts – optional int, how many times to attempt an action before failing. Must be 1 or above. Defaults to 2.

  • retry_delay – optional int|float, how many seconds to sleep between each attempt. Defaults to 0.

  • retry_for –

    optional None|tuple|set|list, what exceptions to allow retries for. Will allow retries for all exceptions if None. .. rubric:: Example

    (MemcacheClientError, MemcacheUnexpectedCloseError)

    Accepts any class that is a subclass of Exception. Defaults to None.

  • do_not_retry_for –

    optional None|tuple|set|list, what exceptions should be retried. Will not block retries for any Exception if None. .. rubric:: Example

    (IOError, MemcacheIllegalInputError)

    Accepts any class that is a subclass of Exception. Defaults to None.

Exceptions:

ValueError: If attempts is not 1 or above. ValueError: If retry_for or do_not_retry_for is not None, tuple or

Iterable.

ValueError: If any of the elements of retry_for or

do_not_retry_for is not a subclass of Exception.

ValueError: If there is any overlap between retry_for and

do_not_retry_for.

Module contents

Submodules

pymemcache.exceptions module
exception pymemcache.exceptions.MemcacheClientError

Bases: MemcacheError

Raised when memcached fails to parse the arguments to a request, likely due to a malformed key and/or value, a bug in this library, or a version mismatch with memcached.

exception pymemcache.exceptions.MemcacheError

Bases: Exception

Base exception class

exception pymemcache.exceptions.MemcacheIllegalInputError

Bases: MemcacheClientError

Raised when a key or value is not legal for Memcache (see the class docs for Client for more details).

exception pymemcache.exceptions.MemcacheServerError

Bases: MemcacheError

Raised when memcached reports a failure while processing a request, likely due to a bug or transient issue in memcached.

exception pymemcache.exceptions.MemcacheUnexpectedCloseError

Bases: MemcacheServerError

Raised when the connection with memcached closes unexpectedly.

exception pymemcache.exceptions.MemcacheUnknownCommandError

Bases: MemcacheClientError

Raised when memcached fails to parse a request, likely due to a bug in this library or a version mismatch with memcached.

exception pymemcache.exceptions.MemcacheUnknownError

Bases: MemcacheError

Raised when this library receives a response from memcached that it cannot parse, likely due to a bug in this library or a version mismatch with memcached.

pymemcache.fallback module

A client for falling back to older memcached servers when performing reads.

It is sometimes necessary to deploy memcached on new servers, or with a different configuration. In these cases, it is undesirable to start up an empty memcached server and point traffic to it, since the cache will be cold, and the backing store will have a large increase in traffic.

This class attempts to solve that problem by providing an interface identical to the Client interface, but which can fall back to older memcached servers when reads to the primary server fail. The approach for upgrading memcached servers or configuration then becomes:

  1. Deploy a new host (or fleet) with memcached, possibly with a new configuration.

  2. From your application servers, use FallbackClient to write and read from the new cluster, and to read from the old cluster when there is a miss in the new cluster.

  3. Wait until the new cache is warm enough to support the load.

  4. Switch from FallbackClient to a regular Client library for doing all reads and writes to the new cluster.

  5. Take down the old cluster.

Best Practices:

  • Make sure that the old client has “ignore_exc” set to True, so that it treats failures like cache misses. That will allow you to take down the old cluster before you switch away from FallbackClient.

class pymemcache.fallback.FallbackClient(caches)

Bases: object

__init__(caches)
add(key, value, expire=0, noreply=True)
append(key, value, expire=0, noreply=True)
cas(key, value, cas, expire=0, noreply=True)
close()

Close each of the memcached clients

decr(key, value, noreply=True)
delete(key, noreply=True)
flush_all(delay=0, noreply=True)
get(key)
get_many(keys)
gets(key)
gets_many(keys)
incr(key, value, noreply=True)
prepend(key, value, expire=0, noreply=True)
quit()
replace(key, value, expire=0, noreply=True)
set(key, value, expire=0, noreply=True)
stats()
touch(key, expire=0, noreply=True)
pymemcache.pool module
class pymemcache.pool.ObjectPool(obj_creator: Callable[[], T], after_remove: Optional[Callable] = None, max_size: Optional[int] = None, idle_timeout: int = 0, lock_generator: Optional[Callable] = None)

Bases: Generic[T]

A pool of objects that release/creates/destroys as needed.

__init__(obj_creator: Callable[[], T], after_remove: Optional[Callable] = None, max_size: Optional[int] = None, idle_timeout: int = 0, lock_generator: Optional[Callable] = None)
clear() → None
destroy(obj, silent=True) → None
property free
get()
get_and_release(destroy_on_fail=False) → Iterator[T]
release(obj, silent=True) → None
property used
pymemcache.serde module
class pymemcache.serde.CompressedSerde(compress=<built-in function compress>, decompress=<built-in function decompress>, serde=<pymemcache.serde.PickleSerde object>, min_compress_len=400)

Bases: object

An object which implements the serialization/deserialization protocol for pymemcache.client.base.Client and its descendants with configurable compression.

__init__(compress=<built-in function compress>, decompress=<built-in function decompress>, serde=<pymemcache.serde.PickleSerde object>, min_compress_len=400)
deserialize(key, value, flags)
serialize(key, value)
class pymemcache.serde.LegacyWrappingSerde(serializer_func, deserializer_func)

Bases: object

This class defines how to wrap legacy de/serialization functions into a ‘serde’ object which implements ‘.serialize’ and ‘.deserialize’ methods. It is used automatically by pymemcache.client.base.Client when the ‘serializer’ or ‘deserializer’ arguments are given.

The serializer_func and deserializer_func are expected to be None in the case that they are missing.

__init__(serializer_func, deserializer_func) → None
class pymemcache.serde.PickleSerde(pickle_version: int = 4)

Bases: object

An object which implements the serialization/deserialization protocol for pymemcache.client.base.Client and its descendants using the pickle module.

Serialization and deserialization are implemented as methods of this class. To implement a custom serialization/deserialization method for pymemcache, you should implement the same interface as the one provided by this object – pymemcache.serde.PickleSerde.serialize() and pymemcache.serde.PickleSerde.deserialize(). Then, pass your custom object to the pymemcache client object in place of PickleSerde.

For more details on the serialization protocol, see the class documentation for pymemcache.client.base.Client

__init__(pickle_version: int = 4) → None
deserialize(key, value, flags)
serialize(key, value)
pymemcache.serde.get_python_memcache_serializer(pickle_version: int = 4)

Return a serializer using a specific pickle version

pymemcache.serde.python_memcache_deserializer(key, value, flags)
pymemcache.serde.python_memcache_serializer(key, value, *, pickle_version=4)

Module contents

Changelog

New in version 4.0.0

  • Dropped Python 2 and 3.6 support #321 #363

  • Begin adding typing

  • Add pluggable compression serde #407

New in version 3.5.2

  • Handle blank STAT values.

New in version 3.5.1

  • Client.get returns the default when using ignore_exc and if memcached is unavailable

  • Added noreply support to HashClient.flush_all.

New in version 3.5.0

  • Sockets are now closed on MemcacheUnexpectedCloseError.

  • Added support for TCP keepalive for client sockets on Linux platforms.

  • Added retrying mechanisms by wrapping clients.

New in version 3.4.4

  • Idle connections will be removed from the pool after pool_idle_timeout.

New in version 3.4.3

  • Fix HashClient.{get,set}_many() with UNIX sockets.

New in version 3.4.2

  • Remove trailing space for commands that don’t take arguments, such as stats. This was a violation of the memcached protocol.

New in version 3.4.1

  • CAS operations will now raise MemcacheIllegalInputError when None is given as the cas value.

New in version 3.4.0

  • Added IPv6 support for TCP socket connections. Note that IPv6 may be used in preference to IPv4 when passing a domain name as the host if an IPv6 address can be resolved for that domain.

  • HashClient now supports UNIX sockets.

New in version 3.3.0

  • HashClient can now be imported from the top-level pymemcache package (e.g. pymemcache.HashClient).

  • HashClient.get_many() now longer stores False for missing keys from unavailable clients. Instead, the result won’t contain the key at all.

  • Added missing HashClient.close() and HashClient.quit().

New in version 3.2.0

  • PooledClient and HashClient now support custom Client classes

New in version 3.1.1

  • Improve MockMemcacheClient to behave even more like Client

New in version 3.1.0

  • Add TLS support for TCP sockets.

  • Fix corner case when dead hashed server comes back alive.

New in version 3.0.1

  • Make MockMemcacheClient more consistent with the real client.

  • Pass encoding from HashClient to its pooled clients when use_pooling is enabled.

New in version 3.0.0

  • The serialization API has been reworked. Instead of consuming a serializer and deserializer as separate arguments, client objects now expect an argument serde to be an object which implements serialize and deserialize as methods. (serialize and deserialize are still supported but considered deprecated.)

  • Validate integer inputs for expire, delay, incr, decr, and memlimit – non-integer values now raise MemcacheIllegalInputError

  • Validate inputs for cas – values which are not integers or strings of 0-9 now raise MemcacheIllegalInputError

  • Add prepend and append support to MockMemcacheClient.

  • Add the touch method to HashClient.

  • Added official support for Python 3.8.

New in version 2.2.2

  • Fix long_description string in Python packaging.

New in version 2.2.1

  • Fix flags when setting multiple differently-typed values at once.

New in version 2.2.0

  • Drop official support for Python 3.4.

  • Use setup.cfg metadata instead setup.py config to generate package.

  • Add default_noreply parameter to HashClient.

  • Add encoding parameter to Client constructors (defaults to ascii).

  • Add flags parameter to write operation methods.

  • Handle unicode key values in MockMemcacheClient correctly.

  • Improve ASCII encoding failure exception.

New in version 2.1.1

  • Fix setup.py dependency on six already being installed.

New in version 2.1.0

  • Public classes and exceptions can now be imported from the top-level pymemcache package (e.g. pymemcache.Client). #197

  • Add UNIX domain socket support and document server connection options. #206

  • Add support for the cache_memlimit command. #211

  • Commands key are now always sent in their original order. #209

New in version 2.0.0

  • Change set_many and set_multi api return value. #179

  • Fix support for newbytes from python-future. #187

  • Add support for Python 3.7, and drop support for Python 3.3

  • Properly batch Client.set_many() call. #182

  • Improve _check_key() and _store_cmd() performance. #183

  • Properly batch Client.delete_many() call. #184

  • Add option to explicitly set pickle version used by serde. #190

New in version 1.4.4

  • pypy3 to travis test matrix

  • full benchmarks in test

  • fix flake8 issues

  • Have mockmemcacheclient support non-ascii strings

  • Switch from using pickle format 0 to the highest available version. See #156

    Warning: different versions of python have different highest pickle versions: https://docs.python.org/3/library/pickle.html

New in version 1.4.3

  • Documentation improvements

  • Fixed cachedump stats command, see #103

  • Honor default_value in HashClient

New in version 1.4.2

  • Drop support for python 2.6, see #109

New in version 1.4.1

  • Python 3 serializations fixes #131

  • Drop support for pypy3

  • Comment cleanup

  • Add gets_many to hash_client

  • Better checking for illegal chars in key

New in version 1.4.0

  • Unicode keys support. It is now possible to pass the flag allow_unicode_keys when creating the clients, thanks @jogo!

  • Fixed a bug where PooledClient wasn’t following default_noreply arg set on init, thanks @kols!

  • Improved documentation

New in version 1.3.8

  • use cpickle instead of pickle when possible (python2)

New in version 1.3.7

  • default parameter on get(key, default=0)

  • fixed docs to autogenerate themselves with sphinx

  • fix linter to work with python3

  • improve error message on illegal Input for the key

  • refactor stat parsing

  • fix MockMemcacheClient

  • fix unicode char in middle of key bug

New in version 1.3.6

  • Fix flake8 and cleanup tox building

  • Fix security vulnerability by sanitizing key input

New in version 1.3.5

  • Bug fix for HashClient when retries is set to zero.

  • Adding the VERSION command to the clients.

New in version 1.3.4

  • Bug fix for the HashClient that corrects behavior when there are no working servers.

New in version 1.3.3

  • Adding caching to the Travis build.

  • A bug fix for pluggable hashing in HashClient.

  • Adding a default_noreply argument to the Client ctor.

New in version 1.3.2

  • Making the location of Memcache Exceptions backwards compatible.

New in version 1.3.0

  • Python 3 Support

  • Introduced HashClient that uses consistent hasing for allocating keys across many memcached nodes. It also can detect servers going down and rebalance keys across the available nodes.

  • Retry sock.recv() when it raises EINTR

New in version 1.2.9

  • Introduced PooledClient a thread-safe pool of clients

Indices and tables