Welcome to Pyzor’s documentation!¶

Pyzor Client¶

The Pyzor Client is a Python script deployed with the package. It provides a command line interface to the Pyzor Client API:

pyzor [options] command

You can also use the Python API directly to integrate Pyzor in your solution. For more information see pyzor.client.

$ pyzor check < spam.eml
public.pyzor.org:24441  (200, 'OK')     134504  4681

$ pyzor info < spam.eml
public.pyzor.org:24441  (200, 'OK')
        Count: 134538
        Entered: Sat Jan  4 10:01:34 2014
        Updated: Mon Mar 17 12:52:04 2014
WL-Count: 4681
        WL-Entered: Mon Jan  6 14:32:01 2014
        WL-Updated: Fri Mar 14 16:11:02 2014

$ pyzor report < spam.eml
public.pyzor.org:24441      (200, 'OK')

$ pyzor whitelist < spam.eml
public.pyzor.org:24441      (200, 'OK')

$ pyzor ping
public.pyzor.org:24441      (200, 'OK')

$ pyzor pong < ham.eml
public.pyzor.org:24441  (200, 'OK')     9223372036854775807     0

$ pyzor predigest < test.eml
Thisisatest.

$ pyzor digest < spam.eml
c3a8e8d987f07843792d2ab1823b04cc3cb87482

public.pyzor.org:24441

# This is comment
public.pyzor.org:24441
127.0.0.1:24441

$ pyzor ping
public.pyzor.org:24441  (200, 'OK')
127.0.0.1:24441 (200, 'OK')

Pyzor Server¶

The Pyzor Server will listen on the specified address and any serve request from Pyzor Clients.

$ pyzord --detach /dev/null --homedir=/home/user/.pyzor/

$ kill -TERM `cat /home/user/.pyzor/pyzord.pid`

$ kill -USR1 `cat /home/user/.pyzor/pyzord.pid`

[server]
Engine = gdbm
DigestDB = pyzord.db

[server]
Engine = mysql
DigestDB = localhost,user,password,pyzor,digests

[server]
Engine = redis
DigestDB = localhost,6379,,0

[server]
Engine = redis
DigestDB = localhost,6379,password,0

privilege ... : username ... : allow|deny

all : all anonymous : deny

check report ping pong info : anonymous : allow

Accounts¶

Pyzor Accounts can be used to grant or restrict access to the Pyzor Server, by ensuring the client are authenticated.

To get an account on a server requires coordination between the client user and server admin. Use the following steps:

1. User and admin should agree on a username for the user. Allowed characters for a username are alpha-numerics, the underscore, and dashes. The normative regular expression it must match is ^[-\.\w]+$. Let us assume they have agreed on bob.

2. User generates a key with pyzor genkey. Let us say that it generates the salt,key of:

   227bfb58efaba7c582d9dcb66ab2063d38df2923,8da9f54058c34e383e997f45d6eb74837139f83b
   

3. Assuming the server is at 127.0.0.1:9999, the user puts the following entry into ~/.pyzor/accounts:

   127.0.0.1 : 9999 : bob : 227bfb58efaba7c582d9dcb66ab2063d38df2923,8da9f54058c34e383e997f45d6eb74837139f83b
   

   This tells the Pyzor Client to use the bob account for server 127.0.0.1:9999. It will still use the anonymous user for all other servers.

4. The user then sends the key (the part to the right-hand side of the comma) to the admin.

5. The admin adds the key to their ~/.pyzor/pyzord.passwd:

   bob : 8da9f54058c34e383e997f45d6eb74837139f83b
   

6. Assuming the admin wants to give the privilege of whitelisting (in addition to the normal permissions), the admin then adds the appropriate permissions to ~/.pyzor/pyzord.access:

   check report ping pong info whitelist : bob : allow
   

   For more information see Access File.

7. To reload the account and access information send the USR1 signal to the daemon.

Procmail¶

To use Pyzor in a procmail system, consider using the following simple recipe:

 :0 Wc
| pyzor check :0 a
pyzor-caught

If you prefer, you can merely add a header to message marked with Pyzor, instead of immediately filtering them into a separate folder:

 :0 Wc
| pyzor check :0 Waf
| formail -A 'X-Pyzor: spam'

ReadyExec¶

ReadyExec is a system to eliminate the high startup-cost of executing scripts repeatedly. If you execute Pyzor a lot, you might be interested in installing ReadyExec and using it with Pyzor.

To use Pyzor with ReadyExec, the readyexecd.py server needs to be started as:

readyexecd.py socket_file pyzor.client.run

socket_file can be any (non-existing) filename you wish ReadyExec to use, such as /tmp/pyzor:

readyexecd.py /tmp/pyzor pyzor.client.run

Individual clients are then executed as:

readyexec socket_file options command cmd_options

For example:

readyexec /tmp/pyzor check
readyexec /tmp/pyzor report
readyexec /tmp/pyzor whitelist --style=mbox
readyexec /tmp/pyzor -d ping

pyzor.engines¶

Database backends for pyzord.

The database class must expose a dictionary-like interface, allowing access via __getitem__, __setitem__, and __delitem__. The key will be a forty character string, and the value should be an instance of the Record class.

If the database backend cannot store the Record objects natively, then it must transparently take care of translating to/from Record objects in __setitem__ and __getitem__.

The database class should take care of expiring old values at the appropriate interval.

pyzor.hacks¶

Various hack to make pyzor compatible with different Python versions.

pyzor.account¶

A collection of utilities that facilitate working with Pyzor accounts.

Note that accounts are not necessary (on the client or server), as an “anonymous” account always exists.

class pyzor.account.Account(username, salt, key)¶

Bases: object

pyzor.account.hash_key(key, user, hash_=<built-in function openssl_sha1>)¶

Returns the hash key for this username and password.

lower(H(U + ‘:’ + lower(K))) K is key (hex string) U is username H is the hash function (currently SHA1)

pyzor.account.key_from_hexstr(s)¶

pyzor.account.sign_msg(hashed_key, timestamp, msg, hash_=<built-in function openssl_sha1>)¶

Converts the key, timestamp (epoch seconds), and msg into a digest.

lower(H(H(M) + ‘:’ T + ‘:’ + K)) M is message T is integer epoch timestamp K is hashed_key H is the hash function (currently SHA1)

pyzor.account.verify_signature(msg, user_key)¶

Verify that the provided message is correctly signed.

The message must have “User”, “Time”, and “Sig” headers.

If the signature is valid, then the function returns normally. If the signature is not valid, then a pyzor.SignatureError() exception is raised.

pyzor.client¶

Networked spam-signature detection client.

>>> import pyzor
>>> import pyzor.client
>>> import pyzor.digest
>>> import pyzor.config

To load the accounts file:

>>> accounts = pyzor.config.load_accounts(filename)

To create a client (to then issue commands):

>>> client = pyzor.client.Client(accounts)

To create a client, using the anonymous user:

>>> client = pyzor.client.Client()

To get a digest (of an email.message.Message object, or similar):

>>> digest = pyzor.digest.get_digest(msg)

To query a server (where address is a (host, port) pair):

>>> client.ping(address)
>>> client.info(digest, address)
>>> client.report(digest, address)
>>> client.whitelist(digest, address)
>>> client.check(digest, address)

To query the default server (public.pyzor.org):

>>> client.ping()
>>> client.info(digest)
>>> client.report(digest)
>>> client.whitelist(digest)
>>> client.check(digest)

Response will contain, depending on the type of request, some of the following keys (e.g. client.ping()[‘Code’]):

All responses will have: - ‘Diag’ ‘OK’ or error message - ‘Code’ ‘200’ if OK - ‘PV’ Protocol Version - ‘Thread’

info and check responses will also contain: - ‘[WL-]Count’ Whitelist/Blacklist count

info responses will also have: - ‘[WL-]Entered’ timestamp when message was first whitelisted/blacklisted - ‘[WL-]Updated’ timestamp when message was last whitelisted/blacklisted

class pyzor.client.BatchClient(accounts=None, timeout=None, spec=None, batch_size=10)¶

Bases: pyzor.client.Client

Like the normal Client but with support for batching reports.

flush()¶

Deleting any saved digest reports.

force()¶

Force send any remaining reports.

report(digest, address=('public.pyzor.org', 24441))¶

whitelist(digest, address=('public.pyzor.org', 24441))¶

class pyzor.client.CheckClientRunner(routine, r_count=0, wl_count=0)¶

Bases: pyzor.client.ClientRunner

handle_response(response, message)¶

class pyzor.client.Client(accounts=None, timeout=None, spec=None)¶

Bases: object

check(digest, address=('public.pyzor.org', 24441))¶

info(digest, address=('public.pyzor.org', 24441))¶

max_packet_size = 8192¶

ping(address=('public.pyzor.org', 24441))¶

pong(digest, address=('public.pyzor.org', 24441))¶

read_response(sock, expected_id)¶

report(digest, address=('public.pyzor.org', 24441))¶

send(msg, address=('public.pyzor.org', 24441))¶

timeout = 5¶

whitelist(digest, address=('public.pyzor.org', 24441))¶

class pyzor.client.ClientRunner(routine)¶

Bases: object

handle_response(response, message)¶

mesaage is a string we’ve built up so far

run(server, args, kwargs=None)¶

class pyzor.client.InfoClientRunner(routine)¶

Bases: pyzor.client.ClientRunner

handle_response(response, message)¶

pyzor.config¶

Functions that handle parsing pyzor configuration files.

pyzor.config.expand_homefiles(homefiles, category, homedir, config)¶

Set the full file path for these configuration files.

pyzor.config.load_access_file(access_fn, accounts)¶

Load the ACL from the specified file, if it exists, and return an ACL dictionary, where each key is a username and each value is a set of allowed permissions (if the permission is not in the set, then it is not allowed).

‘accounts’ is a dictionary of accounts that exist on the server - only the keys are used, which must be the usernames (these are the users that are granted permission when the ‘all’ keyword is used, as described below).

Each line of the file should be in the following format:

operation : user : allow|deny

where ‘operation’ is a space-separated list of pyzor commands or the keyword ‘all’ (meaning all commands), ‘username’ is a space-separated list of usernames or the keyword ‘all’ (meaning all users) - the anonymous user is called “anonymous”, and “allow|deny” indicates whether or not the specified user(s) may execute the specified operations.

The file is processed from top to bottom, with the final match for user/operation being the value taken. Every file has the following implicit final rule:

all : all : deny

If the file does not exist, then the following default is used:

check report ping info : anonymous : allow

pyzor.config.load_accounts(filepath)¶

Layout of file is: host : port : username : salt,key

pyzor.config.load_local_whitelist(filepath)¶

Load the local digest skip file.

pyzor.config.load_passwd_file(passwd_fn)¶

Load the accounts from the specified file.

Each line of the file should be in the format:

username : key

If the file does not exist, then an empty dictionary is returned; otherwise, a dictionary of (username, key) items is returned.

pyzor.config.load_servers(filepath)¶

Load the servers file.

pyzor.config.setup_logging(log_name, filepath, debug, sentry_dsn=None, sentry_lvl='WARN')¶

Setup logging according to the specified options. Return the Logger object.

pyzor.digest¶

Handle digesting the messages.

class pyzor.digest.DataDigester(msg, spec=None)¶

Bases: object

The major workhouse class.

atomic_num_lines = 4¶

digest¶

classmethod digest_payloads(msg)¶

email_ptrn = <_sre.SRE_Pattern object>¶

handle_atomic(lines)¶

We digest everything.

handle_line(line)¶

handle_pieced(lines, spec)¶

Digest stuff according to the spec.

longstr_ptrn = <_sre.SRE_Pattern object>¶

min_line_length = 8¶

classmethod normalize(s)¶

static normalize_html_part(s)¶

classmethod should_handle_line(s)¶

unwanted_txt_repl = ''¶

url_ptrn = <_sre.SRE_Pattern object>¶

value¶

ws_ptrn = <_sre.SRE_Pattern object>¶

class pyzor.digest.HTMLStripper(collector)¶

Bases: HTMLParser.HTMLParser

Strip all tags from the HTML.

handle_data(data)¶

Keep track of the data.

handle_endtag(tag)¶

handle_starttag(tag, attrs)¶

class pyzor.digest.PrintingDataDigester(msg, spec=None)¶

Bases: pyzor.digest.DataDigester

Extends DataDigester: prints out what we’re digesting.

handle_line(line)¶

pyzor.forwarder¶

Manage the forwarder process.

class pyzor.forwarder.Forwarder(forwarding_client, remote_servers, max_queue_size=10000)¶

Bases: object

Forwards digest to remote pyzor servers

queue_forward_request(digest, whitelist=False)¶

If forwarding is enabled, insert a digest into the forwarding queue if whitelist is True, the digest will be forwarded as whitelist request if the queue is full, the digest is dropped

start_forwarding()¶

start the forwarding thread

stop_forwarding()¶

disable forwarding and tell the forwarding thread to end itself

pyzor.message¶

This modules contains the various messages used in the pyzor client server communication.

class pyzor.message.CheckRequest(digest=None)¶

Bases: pyzor.message.SimpleDigestBasedRequest

op = 'check'¶

class pyzor.message.ClientSideRequest¶

Bases: pyzor.message.Request

op = None¶

setup()¶

class pyzor.message.InfoRequest(digest=None)¶

Bases: pyzor.message.SimpleDigestBasedRequest

op = 'info'¶

class pyzor.message.Message¶

Bases: email.message.Message

ensure_complete()¶

init_for_sending()¶

setup()¶

class pyzor.message.PingRequest¶

Bases: pyzor.message.ClientSideRequest

op = 'ping'¶

class pyzor.message.PongRequest(digest=None)¶

Bases: pyzor.message.SimpleDigestBasedRequest

op = 'pong'¶

class pyzor.message.ReportRequest(digest=None, spec=None)¶

Bases: pyzor.message.SimpleDigestSpecBasedRequest

op = 'report'¶

class pyzor.message.Request¶

Bases: pyzor.message.ThreadedMessage

This is the class that should be used to read in Requests of any type. Subclasses are responsible for setting ‘Op’ if they are generating a message,

ensure_complete()¶

get_op()¶

class pyzor.message.Response¶

Bases: pyzor.message.ThreadedMessage

ensure_complete()¶

get_code()¶

get_diag()¶

head_tuple()¶

is_ok()¶

ok_code = 200¶

class pyzor.message.SimpleDigestBasedRequest(digest=None)¶

Bases: pyzor.message.ClientSideRequest

add_digest(digest)¶

class pyzor.message.SimpleDigestSpecBasedRequest(digest=None, spec=None)¶

Bases: pyzor.message.SimpleDigestBasedRequest

class pyzor.message.ThreadId¶

Bases: int

error_value = 0¶

full_range = (0, 65536)¶

classmethod generate()¶

in_ok_range()¶

ok_range = (1024, 65536)¶

class pyzor.message.ThreadedMessage¶

Bases: pyzor.message.Message

ensure_complete()¶

get_protocol_version()¶

get_thread()¶

init_for_sending()¶

set_thread(i)¶

class pyzor.message.WhitelistRequest(digest=None, spec=None)¶

Bases: pyzor.message.SimpleDigestSpecBasedRequest

op = 'whitelist'¶

pyzor.server¶

Networked spam-signature detection server.

The server receives the request in the form of a RFC5321 message, and responds with another RFC5321 message. Neither of these messages has a body - all of the data is encapsulated in the headers.

The response headers will always include a “Code” header, which is a HTTP-style response code, and a “Diag” header, which is a human-readable message explaining the response code (typically this will be “OK”).

Both the request and response headers always include a “PV” header, which indicates the protocol version that is being used (in a major.minor format). Both the requestion and response headers also always include a “Thread”, which uniquely identifies the request (this is a requirement of using UDP). Responses to requests may arrive in any order, but the “Thread” header of a response will always match the “Thread” header of the appropriate request.

Authenticated requests must also have “User”, “Time” (timestamp), and “Sig” (signature) headers.

class pyzor.server.BoundedThreadingServer(address, database, passwd_fn, access_fn, max_threads, forwarding_server=None)¶

Bases: pyzor.server.ThreadingServer

Same as ThreadingServer but this also accepts a limited number of concurrent threads.

process_request(request, client_address)¶

process_request_thread(request, client_address)¶

class pyzor.server.PreForkServer(address, database, passwd_fn, access_fn, prefork=4)¶

Bases: pyzor.server.Server

The same as Server, but prefork itself when starting the self, by forking a number of child-processes.

The parent process will then wait for all his child process to complete.

load_config()¶

If this is the parent process send the USR1 signal to all children, else call the super method.

serve_forever(poll_interval=0.5)¶

Fork the current process and wait for all children to finish.

shutdown()¶

If this is the parent process send the TERM signal to all children, else call the super method.

class pyzor.server.ProcessServer(address, database, passwd_fn, access_fn, max_children=40, forwarding_server=None)¶

Bases: SocketServer.ForkingMixIn, pyzor.server.Server

A multi-processing version of the pyzord server. Each connection is served in a new process. This may not be suitable for all database types.

class pyzor.server.RequestHandler(*args, **kwargs)¶

Bases: SocketServer.DatagramRequestHandler

Handle a single pyzord request.

dispatches = {'info': <function handle_info at 0x7fe78f9ae7d0>, 'whitelist': <function handle_whitelist at 0x7fe78f9ae758>, 'ping': None, 'report': <function handle_report at 0x7fe78f9ae6e0>, 'pong': <function handle_pong at 0x7fe78f9ae5f0>, 'check': <function handle_check at 0x7fe78f9ae668>}¶

handle()¶

Handle a pyzord operation, cleanly handling any errors.

handle_check(digests)¶

Handle the ‘check’ command.

This command returns the spam/ham counts for the specified digest.

handle_error(code, message)¶

Create an appropriate response for an error.

handle_info(digests)¶

Handle the ‘info’ command.

This command returns diagnostic data about a digest (timestamps for when the digest was first/last seen as spam/ham, and spam/ham counts).

handle_pong(digests)¶

Handle the ‘pong’ command.

This command returns maxint for report counts and 0 whitelist.

handle_report(digests)¶

Handle the ‘report’ command in a single step.

This command increases the spam count for the specified digests.

handle_whitelist(digests)¶

Handle the ‘whitelist’ command in a single step.

This command increases the ham count for the specified digests.

class pyzor.server.Server(address, database, passwd_fn, access_fn, forwarder=None)¶

Bases: SocketServer.UDPServer

The pyzord server. Handles incoming UDP connections in a single thread and single process.

handle_error(request, client_address)¶

load_config()¶

Reads the configuration files and loads the accounts and ACLs.

max_packet_size = 8192¶

reload_handler(*args, **kwargs)¶

Handler for the SIGUSR1 signal. This should be used to reload the configuration files.

shutdown_handler(*args, **kwargs)¶

Handler for the SIGTERM signal. This should be used to kill the daemon and ensure proper clean-up.

time_diff_allowance = 180¶

class pyzor.server.ThreadingServer(address, database, passwd_fn, access_fn, forwarder=None)¶

Bases: SocketServer.ThreadingMixIn, pyzor.server.Server

A threaded version of the pyzord server. Each connection is served in a new thread. This may not be suitable for all database types.