application Module¶

application.app Module¶

Application main interface.

class wpull.application.app.Application(pipeline_series: wpull.pipeline.pipeline.PipelineSeries)[source]¶

Bases: wpull.application.hook.HookableMixin

Default non-interactive application user interface.

This class manages process signals and displaying warnings.

ERROR_CODE_MAP = OrderedDict([(<class 'wpull.errors.AuthenticationError'>, 6), (<class 'wpull.errors.ServerError'>, 8), (<class 'wpull.errors.ProtocolError'>, 7), (<class 'wpull.errors.SSLVerificationError'>, 5), (<class 'wpull.errors.DNSNotFound'>, 4), (<class 'wpull.errors.ConnectionRefused'>, 4), (<class 'wpull.errors.NetworkError'>, 4), (<class 'OSError'>, 3)])¶

Mapping of error types to exit status.

EXPECTED_EXCEPTIONS = (<class 'wpull.errors.ServerError'>, <class 'wpull.errors.ProtocolError'>, <class 'wpull.errors.SSLVerificationError'>, <class 'wpull.errors.DNSNotFound'>, <class 'wpull.errors.ConnectionRefused'>, <class 'wpull.errors.NetworkError'>, <class 'OSError'>, <class 'OSError'>, <class 'wpull.application.hook.HookStop'>, <class 'StopIteration'>, <class 'SystemExit'>, <class 'KeyboardInterrupt'>)¶

Exception classes that are not crashes.

class Event[source]¶

Bases: enum.Enum

Application.exit_code¶

Application.run()[source]¶

Application.run_sync() → int[source]¶

Run the application.

This function is blocking.

Returns:	The exit status.
Return type:	int

Application.setup_signal_handlers()[source]¶

Setup Ctrl+C and SIGTERM handlers.

Application.stop()[source]¶

Application.update_exit_code(code: int)[source]¶

Set the exit code if it is serious than before.

Parameters:	code – The exit code.

class wpull.application.app.ApplicationState[source]¶

Bases: enum.Enum

application.builder Module¶

Application support.

class wpull.application.builder.Builder(args, unit_test=False)[source]¶

Bases: object

Application builder.

Parameters:	args – Options from argparse.ArgumentParser

build() → wpull.application.app.Application[source]¶

Put the application together.

build_and_run()[source]¶

Build and run the application.

Returns:	The exit status.
Return type:	int

factory¶

Return the Factory.

Returns:	An factory.Factory instance.
Return type:	Factory

get_stderr()[source]¶

Return stderr or something else if under unit testing.

application.factory Module¶

Instance creation and management.

class wpull.application.factory.Factory(class_map=None)[source]¶

Bases: collections.abc.Mapping, object

Allows selection of classes and keeps track of instances.

This class behaves like a mapping. Keys are names of classes and values are instances.

class_map¶

A mapping of names to class types.

instance_map¶

A mapping of names to instances.

is_all_initialized()[source]¶

Return whether all the instances have been initialized.

Returns:	bool

new(name, *args, **kwargs)[source]¶

Create an instance.

Parameters:	name (str) – The name of the class args – The arguments to pass to the class. kwargs – The keyword arguments to pass to the class.
Returns:	instance

set(name, class_)[source]¶

Set the callable or class to be used.

Parameters:	name (str) – The name of the class. class – The class or a callable factory function.

application.hook Module¶

Python and Lua scripting support.

See Plugin Scripting Hooks for an introduction.

class wpull.application.hook.Actions[source]¶

Bases: enum.Enum

Actions for handling responses and errors.

NORMAL¶

normal

Use Wpull’s original behavior.

RETRY¶

retry

Retry this item (as if an error has occurred).

FINISH¶

finish

Consider this item as done; don’t do any further processing on it.

STOP¶

stop

Raises HookStop to stop the Engine from running.

class wpull.application.hook.EventDispatcher[source]¶

Bases: collections.abc.Mapping

add_listener(name: str, callback)[source]¶

is_registered(name: str) → bool[source]¶

notify(name: str, *args, **kwargs)[source]¶

register(name: str)[source]¶

remove_listener(name: str, callback)[source]¶

unregister(name: str)[source]¶

exception wpull.application.hook.HookAlreadyConnectedError[source]¶

Bases: ValueError

A callback is already connected to the hook.

exception wpull.application.hook.HookDisconnected[source]¶

Bases: RuntimeError

No callback is connected.

class wpull.application.hook.HookDispatcher(event_dispatcher_transclusion: typing.Union=None)[source]¶

Bases: collections.abc.Mapping

Dynamic callback hook system.

call(name: str, *args, **kwargs)[source]¶

Invoke the callback.

call_async(name: str, *args, **kwargs)[source]¶

Invoke the callback.

connect(name, callback)[source]¶

Add callback to hook.

disconnect(name: str)[source]¶

Remove callback from hook.

is_connected(name: str) → bool[source]¶

Return whether the hook is connected.

is_registered(name: str) → bool[source]¶

register(name: str)[source]¶

Register hooks that can be connected.

unregister(name: str)[source]¶

Unregister hook.

exception wpull.application.hook.HookStop[source]¶

Bases: Exception

Stop the engine.

Raise this exception as a more graceful alternative to sys.exit().

class wpull.application.hook.HookableMixin[source]¶

Bases: object

connect_plugin(plugin: wpull.application.plugin.WpullPlugin)[source]¶

application.main Module¶

wpull.application.main.main(exit=True, install_tornado_bridge=True, use_signals=True)[source]¶

application.options Module¶

Program options.

class wpull.application.options.AppArgumentParser(*args, real_exit=True, **kwargs)[source]¶

Bases: argparse.ArgumentParser

An Argument Parser that builds up the application options.

classmethod comma_choice_list(string)[source]¶

Convert a comma separated string to CommaChoiceListArgs.

classmethod comma_list(string)[source]¶

Convert a comma separated string to list.

exit(status=0, message=None)[source]¶

classmethod get_argv_encoding(argv)[source]¶

classmethod int_0_inf(string)[source]¶

Convert string to int.

If inf is supplied, it returns 0.

classmethod int_bytes(string)[source]¶

Convert string describing size to int.

parse_args(args=None, namespace=None)[source]¶

class wpull.application.options.AppHelpFormatter(prog, indent_increment=2, max_help_position=24, width=None)[source]¶

Bases: argparse.HelpFormatter

class wpull.application.options.CommaChoiceListArgs[source]¶

Bases: frozenset

Specialized frozenset.

This class overrides the __contains__ function to allow the use of the in operator for ArgumentParser’s choices checking for comma separated lists. The function behaves differently only when the objects compared are CommaChoiceListArgs.

application.plugin Module¶

class wpull.application.plugin.InterfaceRegistry[source]¶

Bases: collections.abc.Mapping

register(name: typing.Any, interface: typing.Any, category: wpull.application.plugin.PluginFunctionCategory)[source]¶

wpull.application.plugin.PluginClientFunctionInfo¶

alias of _PluginClientFunctionInfo

class wpull.application.plugin.PluginFunctionCategory[source]¶

Bases: enum.Enum

class wpull.application.plugin.PluginFunctions[source]¶

Bases: enum.Enum

class wpull.application.plugin.WpullPlugin[source]¶

Bases: yapsy.IPlugin.IPlugin

get_plugin_functions() → typing.Iterator[source]¶

should_activate() → bool[source]¶

wpull.application.plugin.event(name: typing.Any)[source]¶

wpull.application.plugin.event_interface(name: typing.Any, interface_registry: wpull.application.plugin.InterfaceRegistry=<wpull.application.plugin.InterfaceRegistry object>)[source]¶

wpull.application.plugin.hook(name: typing.Any)[source]¶

wpull.application.plugin.hook_interface(name: typing.Any, interface_registry: wpull.application.plugin.InterfaceRegistry=<wpull.application.plugin.InterfaceRegistry object>)[source]¶

application.plugins Module¶

application.plugins.arg_warning.plugin Module¶

application.plugins.debug_console.plugin Module¶

application.plugins.download_progress.plugin Module¶

application.plugins.server_response.plugin Module¶

application.tasks Module¶

application.tasks.conversion Module¶

class wpull.application.tasks.conversion.LinkConversionSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.conversion.LinkConversionTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.application.tasks.conversion.QueuedFileSession)[source]¶

class wpull.application.tasks.conversion.QueuedFileSession(app_session: wpull.pipeline.app.AppSession, file_id: int, url_record: wpull.pipeline.item.URLRecord)[source]¶

Bases: object

class wpull.application.tasks.conversion.QueuedFileSource(app_session: wpull.pipeline.app.AppSession)[source]¶

Bases: wpull.pipeline.pipeline.ItemSource

get_item() → typing.Union[source]¶

application.tasks.database Module¶

class wpull.application.tasks.database.DatabaseSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.database.InputURLTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.download Module¶

class wpull.application.tasks.download.BackgroundAsyncTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.session.ItemSession)[source]¶

class wpull.application.tasks.download.CheckQuotaTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.session.ItemSession)[source]¶

class wpull.application.tasks.download.ClientSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.download.CoprocessorSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.download.ParserSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.download.ProcessTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.session.ItemSession)[source]¶

class wpull.application.tasks.download.ProcessorSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.download.ProxyServerSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

Build MITM proxy server.

application.tasks.log Module¶

class wpull.application.tasks.log.LoggingSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.log.LoggingShutdownTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.network Module¶

class wpull.application.tasks.network.NetworkSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.plugin Module¶

class wpull.application.tasks.plugin.PluginLocator(directories, paths)[source]¶

Bases: yapsy.IPluginLocator.IPluginLocator

gatherCorePluginInfo(directory, filename)[source]¶

locatePlugins()[source]¶

class wpull.application.tasks.plugin.PluginSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.resmon Module¶

class wpull.application.tasks.resmon.ResmonSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.resmon.ResmonSleepTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.session.ItemSession)[source]¶

application.tasks.rule Module¶

class wpull.application.tasks.rule.URLFiltersPostURLImportSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.rule.URLFiltersSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.shutdown Module¶

class wpull.application.tasks.shutdown.AppStopTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask, wpull.application.hook.HookableMixin

static plugin_exit_status(app_session: wpull.pipeline.app.AppSession, exit_code: int) → int[source]¶

Return the program exit status code.

Exit codes are values from errors.ExitStatus.

Parameters:	exit_code – The exit code Wpull wants to return.
Returns:	The exit code that Wpull will return.
Return type:	int

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.shutdown.BackgroundAsyncCleanupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.shutdown.CookieJarTeardownTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.sslcontext Module¶

class wpull.application.tasks.sslcontext.SSLContextTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.stats Module¶

class wpull.application.tasks.stats.StatsStartTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.stats.StatsStopTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask, wpull.application.hook.HookableMixin

static plugin_finishing_statistics(app_session: wpull.pipeline.app.AppSession, statistics: wpull.stats.Statistics)[source]¶

Callback containing final statistics.

Parameters:	start_time (float) – timestamp when the engine started end_time (float) – timestamp when the engine stopped num_urls (int) – number of URLs downloaded bytes_downloaded (int) – size of files downloaded in bytes

process(session: wpull.pipeline.app.AppSession)[source]¶

application.tasks.warc Module¶

class wpull.application.tasks.warc.WARCRecorderSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.warc.WARCRecorderTeardownTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

class wpull.application.tasks.warc.WARCVisitsTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

Populate the visits from the CDX into the URL table.

application.tasks.writer Module¶

class wpull.application.tasks.writer.FileWriterSetupTask[source]¶

Bases: wpull.pipeline.pipeline.ItemTask

process(session: wpull.pipeline.app.AppSession)[source]¶

body Module¶

Request and response payload.

class wpull.body.Body(file=None, directory=None, hint='lone_body')[source]¶

Bases: object

Represents the document/payload of a request or response.

This class is a wrapper around a file object. Methods are forwarded to the underlying file object.

file¶

file

The file object.

Parameters:	file (file, optional) – Use the given file as the file object. directory (str) – If file is not given, use directory for a new temporary file. hint (str) – If file is not given, use hint as a filename infix.

content()[source]¶

Return the content of the file.

If this function is invoked, the contents of the entire file is read and cached.

Returns:	The entire content of the file.
Return type:	bytes

size()[source]¶

Return the size of the file.

to_dict()[source]¶

Convert the body to a dict.

Returns:	The items are: filename (string, None): The path of the file. length (int, None): The size of the file.
Return type:	dict

wpull.body.is_seekable(file)[source]¶

wpull.body.new_temp_file(directory=None, hint='')[source]¶

Return a new temporary file.

cache Module¶

Caching.

class wpull.cache.BaseCache[source]¶

Bases: collections.abc.Mapping, object

clear()[source]¶

Remove all items in cache.

wpull.cache.Cache¶

alias of LRUCache

class wpull.cache.CacheItem(key, value, time_to_live=None, access_time=None)[source]¶

Bases: object

Info about an item in the cache.

Parameters:	key – The key value – The value time_to_live – The time in seconds of how long to keep the item access_time – The timestamp of the last use of the item

expire_time¶

When the item expires.

class wpull.cache.FIFOCache(max_items=None, time_to_live=None)[source]¶

Bases: wpull.cache.BaseCache

First in first out object cache.

Parameters:	max_items (int) – The maximum number of items to keep. time_to_live (float) – Discard items after time_to_live seconds.

Reusing a key to update a value will not affect the expire time of the item.

clear()[source]¶

trim()[source]¶

Remove items that are expired or exceed the max size.

class wpull.cache.LRUCache(max_items=None, time_to_live=None)[source]¶

Bases: wpull.cache.FIFOCache

Least recently used object cache.

Parameters:	max_items – The maximum number of items to keep time_to_live – The time in seconds of how long to keep the item

touch(key)[source]¶

wpull.cache.total_ordering(obj)¶

collections Module¶

Data structures.

class wpull.collections.FrozenDict(orig_dict)[source]¶

Bases: collections.abc.Mapping, collections.abc.Hashable

Immutable mapping wrapper.

hash_cache¶

orig_dict¶

class wpull.collections.LinkedList[source]¶

Bases: object

Doubly linked list.

map¶

dict

A mapping of values to nodes.

head¶

LinkedListNode

The first node.

tail¶

LinkedListNode

The last node.

append(value)[source]¶

appendleft(value)[source]¶

clear()[source]¶

index(value)[source]¶

pop()[source]¶

popleft()[source]¶

remove(value)[source]¶

remove_node(node)[source]¶

class wpull.collections.LinkedListNode(value, head=None, tail=None)[source]¶

Bases: object

A node in a LinkedList.

value¶

Any value.

head¶

LinkedListNode

The node in front.

tail¶

LinkedListNode

The node in back.

head

link_head(node)[source]¶

Add a node to the head.

link_tail(node)[source]¶

Add a node to the tail.

tail

unlink()[source]¶

Remove this node and link any head or tail.

value

class wpull.collections.OrderedDefaultDict(default_factory=None, *args, **kwargs)[source]¶

Bases: collections.OrderedDict

An ordered default dict.

http://stackoverflow.com/a/6190500/1524507

copy()[source]¶

converter Module¶

Document content post-processing.

class wpull.converter.BaseDocumentConverter[source]¶

Bases: object

Base class for classes that convert links within a document.

convert(input_filename, output_filename, base_url=None)[source]¶

class wpull.converter.BatchDocumentConverter(html_parser, element_walker, url_table, backup=False)[source]¶

Bases: object

Convert all documents in URL table.

Parameters:	url_table – An instance of database.URLTable. backup (bool) – Whether back up files are created.

convert_all()[source]¶

Convert all links in URL table.

convert_by_record(url_record)[source]¶

Convert using given URL Record.

class wpull.converter.CSSConverter(url_table)[source]¶

Bases: wpull.scraper.css.CSSScraper, wpull.converter.BaseDocumentConverter

CSS converter.

convert(input_filename, output_filename, base_url=None)[source]¶

convert_text(text, base_url=None)[source]¶

get_new_url(url, base_url=None)[source]¶

class wpull.converter.HTMLConverter(html_parser, element_walker, url_table)[source]¶

Bases: wpull.scraper.html.HTMLScraper, wpull.converter.BaseDocumentConverter

HTML converter.

convert(input_filename, output_filename, base_url=None)[source]¶

cookie Module¶

HTTP Cookies.

class wpull.cookie.BetterMozillaCookieJar(filename=None, delayload=False, policy=None)[source]¶

Bases: http.cookiejar.FileCookieJar

MozillaCookieJar that is compatible with Wget/Curl.

It ignores file header checks and supports session cookies.

header = '# Netscape HTTP Cookie File\n# http://curl.haxx.se/rfc/cookie_spec.html\n# This is a generated file! Do not edit.\n\n'¶

magic_re = re.compile('.')¶

save(filename=None, ignore_discard=False, ignore_expires=False)[source]¶

class wpull.cookie.DeFactoCookiePolicy(*args, **kwargs)[source]¶

Bases: http.cookiejar.DefaultCookiePolicy

Cookie policy that limits the content and length of the cookie.

Parameters:	cookie_jar – The CookieJar instance.

This policy class is not designed to be shared between CookieJar instances.

cookie_length(domain)[source]¶

Return approximate length of all cookie key-values for a domain.

count_cookies(domain)[source]¶

Return the number of cookies for the given domain.

set_ok(cookie, request)[source]¶

cookiewrapper Module¶

Wrappers that wrap instances to Python standard library.

class wpull.cookiewrapper.CookieJarWrapper(cookie_jar, save_filename=None, keep_session_cookies=False)[source]¶

Bases: object

Wraps a CookieJar.

Parameters:	cookie_jar – An instance of http.cookiejar.CookieJar. save_filename (str, optional) – A filename to save the cookies. keep_session_cookies (bool) – If True, session cookies are kept when saving to file.

add_cookie_header(request, referrer_host=None)[source]¶

Wrapped add_cookie_header.

Parameters:	request – An instance of http.request.Request. referrer_host (str) – An hostname or IP address of the referrer URL.

close()[source]¶

Save the cookie jar if needed.

cookie_jar¶

Return the wrapped Cookie Jar.

extract_cookies(response, request, referrer_host=None)[source]¶

Wrapped extract_cookies.

Parameters:	response – An instance of http.request.Response. request – An instance of http.request.Request. referrer_host (str) – An hostname or IP address of the referrer URL.

class wpull.cookiewrapper.HTTPResponseInfoWrapper(response)[source]¶

Bases: object

Wraps a HTTP Response.

Parameters:	response – An instance of http.request.Response

info()[source]¶

Return the header fields as a Message:

Returns:	An instance of email.message.Message. If Python 2, returns an instance of mimetools.Message.
Return type:	Message

wpull.cookiewrapper.convert_http_request(request, referrer_host=None)[source]¶

Convert a HTTP request.

Parameters:	request – An instance of http.request.Request. referrer_host (str) – The referrering hostname or IP address.
Returns:	An instance of urllib.request.Request
Return type:	Request

database Module¶

Storage for tracking URLs.

database.base Module¶

Base table class.

wpull.database.base.AddURLInfo¶

alias of _AddURLInfo

class wpull.database.base.BaseURLTable[source]¶

Bases: object

URL table.

add_many(new_urls: typing.Iterator) → typing.Iterator[source]¶

Add the URLs to the table.

Parameters:	new_urls – URLs to be added.
Returns:	The URLs added. Useful for tracking duplicates.

add_one(url: str, url_properties: typing.Union=None, url_data: typing.Union=None)[source]¶

Add a single URL to the table.

Parameters:	url – The URL to be added url_properties – Additional values to be saved url_data – Additional data to be saved

add_visits(visits)[source]¶

Add visited URLs from CDX file.

Parameters:	visits (iterable) – An iterable of items. Each item is a tuple containing a URL, the WARC ID, and the payload digest.

check_in(url: str, new_status: wpull.pipeline.item.Status, increment_try_count: bool=True, url_result: typing.Union=None)[source]¶

Update record for processed URL.

Parameters:	url – The URL. new_status – Update the item status to new_status. increment_try_count – Whether to increment the try counter for the URL. url_result – Additional values.

check_out(filter_status: wpull.pipeline.item.Status, filter_level: typing.Union=None) → wpull.pipeline.item.URLRecord[source]¶

Find a URL, mark it in progress, and return it.

Parameters:	filter_status – Gets first item with given status. filter_level – Gets item with filter_level or lower.
Raises:	NotFound

close()[source]¶

Run any clean-up actions and close the table.

contains(url: str)[source]¶

Return whether the URL is in the table.

convert_check_in(file_id: int, status: wpull.pipeline.item.Status)[source]¶

convert_check_out() -> (<class 'int'>, <class 'wpull.pipeline.item.URLRecord'>)[source]¶

count() → int[source]¶

Return the number of URLs in the table.

This call may be expensive.

get_all() → typing.Iterator[source]¶

Return all URLRecord.

get_hostnames()[source]¶

Return list of hostnames

get_one(url: str) → wpull.pipeline.item.URLRecord[source]¶

Return a URLRecord for the URL.

Raises:	NotFound

get_revisit_id(url, payload_digest)[source]¶

Return the WARC ID corresponding to the visit.

Returns:	str, None

get_root_url_todo_count() → int[source]¶

release()[source]¶

Mark any in_progress URLs to todo status.

remove_many(urls)[source]¶

Remove the URLs from the database.

remove_one(url)[source]¶

Remove a URL from the database.

update_one(url, **kwargs)[source]¶

Arbitrarily update values for a URL.

exception wpull.database.base.DatabaseError[source]¶

Bases: Exception

Any database error.

exception wpull.database.base.NotFound[source]¶

Bases: wpull.database.base.DatabaseError

Item not found in the table.

database.sqlmodel Module¶

Database SQLAlchemy model.

wpull.database.sqlmodel.DBBase¶

alias of Base

class wpull.database.sqlmodel.QueuedURL(**kwargs)[source]¶

Bases: sqlalchemy.ext.declarative.api.Base

filename¶

Local filename of the item.

id¶

inline_level¶

Depth of the page requisite object. 0 is the object, 1 is the object’s dependency, etc.

level¶

Recursive depth of the item. 0 is root, 1 is child of root, etc.

link_type¶

Expected content type of extracted link.

parent_url¶

A descriptor that presents a read/write view of an object attribute.

parent_url_string¶

parent_url_string_id¶

Optional referral URL

post_data¶

Additional percent-encoded data for POST.

priority¶

Priority of item.

root_url¶

A descriptor that presents a read/write view of an object attribute.

root_url_string¶

root_url_string_id¶

Optional root URL

status¶

Status of the completion of the item.

status_code¶

HTTP status code or FTP rely code.

to_plain() → wpull.pipeline.item.URLRecord[source]¶

try_count¶

Number of attempts made in order to process the item.

url¶

A descriptor that presents a read/write view of an object attribute.

url_string¶

url_string_id¶

Target URL to fetch

classmethod watch_urls_inserted(session)[source]¶

class wpull.database.sqlmodel.URLString(**kwargs)[source]¶

Bases: sqlalchemy.ext.declarative.api.Base

Table containing the URL strings.

The URL references this table.

classmethod add_urls(session, urls: typing.Iterable)[source]¶

id¶

url¶

class wpull.database.sqlmodel.WARCVisit(**kwargs)[source]¶

Bases: sqlalchemy.ext.declarative.api.Base

Standalone table for --cdx-dedup feature.

classmethod add_visits(session, visits)[source]¶

classmethod get_revisit_id(session, url, payload_digest)[source]¶

payload_digest¶

url¶

warc_id¶

class wpull.database.sqlmodel.Hostname(**kwargs)[source]¶

Bases: sqlalchemy.ext.declarative.api.Base

hostname¶

id¶

class wpull.database.sqlmodel.QueuedFile(**kwargs)[source]¶

Bases: sqlalchemy.ext.declarative.api.Base

id¶

queued_url¶

queued_url_id¶

status¶

database.sqltable Module¶

SQLAlchemy table implementations.

class wpull.database.sqltable.BaseSQLURLTable[source]¶

Bases: wpull.database.base.BaseURLTable

add_many(new_urls)[source]¶

add_visits(visits)[source]¶

check_in(url, new_status, increment_try_count=True, url_result=None)[source]¶

check_out(filter_status, level=None)[source]¶

convert_check_in(file_id, status)[source]¶

convert_check_out()[source]¶

count()[source]¶

get_all()[source]¶

get_hostnames()[source]¶

get_one(url)[source]¶

get_revisit_id(url, payload_digest)[source]¶

get_root_url_todo_count()[source]¶

release()[source]¶

remove_many(urls)[source]¶

update_one(url, **kwargs)[source]¶

class wpull.database.sqltable.SQLiteURLTable(path=':memory:')[source]¶

Bases: wpull.database.sqltable.BaseSQLURLTable

URL table with SQLite storage.

Parameters:	path – A SQLite filename

close()[source]¶

class wpull.database.sqltable.GenericSQLURLTable(url)[source]¶

Bases: wpull.database.sqltable.BaseSQLURLTable

URL table using SQLAlchemy without any customizations.

Parameters:	url – A SQLAlchemy database URL.

close()[source]¶

wpull.database.sqltable.URLTable¶

The default URL table implementation.

alias of SQLiteURLTable

wpull.database.sqltable.convert_dict_enum_values(dict_)[source]¶

database.wrap Module¶

URL table wrappers.

class wpull.database.wrap.URLTableHookWrapper(url_table)[source]¶

Bases: wpull.database.base.BaseURLTable, wpull.application.hook.HookableMixin

URL table wrapper with scripting hooks.

Parameters:	url_table – URL table.

url_table¶

URL table.

add_many(urls)[source]¶

add_visits(visits)[source]¶

check_in(url, new_status, increment_try_count=True, url_result=None)[source]¶

check_out(filter_status, filter_level=None)[source]¶

close()[source]¶

convert_check_in(file_id: int, status: wpull.pipeline.item.Status)[source]¶

convert_check_out()[source]¶

count()[source]¶

static dequeued_url(url_info: wpull.url.URLInfo, record_info: wpull.pipeline.item.URLRecord)[source]¶

Callback fired after an URL was retrieved from the queue.

get_all()[source]¶

get_hostnames()[source]¶

get_one(url)[source]¶

get_revisit_id(url, payload_digest)[source]¶

get_root_url_todo_count()[source]¶

queue_count()[source]¶

Return the number of URLs queued in this session.

static queued_url(url_info: wpull.url.URLInfo)[source]¶

Callback fired after an URL was put into the queue.

release()[source]¶

remove_many(urls)[source]¶

update_one(*args, **kwargs)[source]¶

debug Module¶

Debugging utilities.

class wpull.debug.DebugConsoleHandler(application, request, **kwargs)[source]¶

Bases: tornado.web.RequestHandler

TEMPLATE = '<html>\n <style>\n #commandbox {{\n width: 100%;\n }}\n </style>\n <body>\n <p>Welcome to DEBUG CONSOLE!</p>\n <p><tt>Builder()</tt> instance at <tt>wpull_builder</tt>.</p>\n <form method="post">\n <input id="commandbox" name="command" value="{command}">\n <input type="submit" value="Execute">\n </form>\n <pre>{output}</pre>\n </body>\n </html>\n '¶

get()[source]¶

post()[source]¶

decompression Module¶

Streaming decompressors.

class wpull.decompression.DeflateDecompressor[source]¶

Bases: wpull.decompression.SimpleGzipDecompressor

zlib decompressor with raw deflate detection.

This class doesn’t do any special. It only tries regular zlib and then tries raw deflate on the first decompress.

decompress(value)[source]¶

flush()[source]¶

class wpull.decompression.GzipDecompressor[source]¶

Bases: wpull.decompression.SimpleGzipDecompressor

gzip decompressor with gzip header detection.

This class checks if the stream starts with the 2 byte gzip magic number. If it is not present, it returns the bytes unchanged.

decompress(value)[source]¶

flush()[source]¶

class wpull.decompression.SimpleGzipDecompressor[source]¶

Bases: object

Streaming gzip decompressor.

The interface is like that of zlib.decompressobj (without some of the optional arguments, but it understands gzip headers and checksums.

decompress(value)[source]¶

Decompress a chunk, returning newly-available data.

Some data may be buffered for later processing; flush must be called when there is no more input data to ensure that all data was processed.

flush()[source]¶

Return any remaining buffered data not yet returned by decompress.

Also checks for errors such as truncated input. No other methods may be called on this object after flush.

wpull.decompression.gzip_uncompress(data, truncated=False)[source]¶

Uncompress gzip data.

Parameters:	data (bytes) – The gzip data. truncated (bool) – If True, the decompressor is not flushed.

This is a convenience function.

Returns:	The inflated data.
Return type:	bytes
Raises:	zlib.error

document Module¶

Document handling.

document.base Module¶

Document bases.

class wpull.document.base.BaseDocumentDetector[source]¶

Bases: object

Base class for classes that detect document types.

classmethod is_file(file)[source]¶

Return whether the reader is likely able to read the file.

Parameters:	file – A file object containing the document.
Returns:	bool

classmethod is_request(request)[source]¶

Return whether the request is likely supported.

Parameters:	request (http.request.Request) – An HTTP request.
Returns:	bool

classmethod is_response(response)[source]¶

Return whether the response is likely able to be read.

Parameters:	response (http.request.Response) – An HTTP response.
Returns:	bool

classmethod is_supported(file=None, request=None, response=None, url_info=None)[source]¶

Given the hints, return whether the document is supported.

Parameters:	file – A file object containing the document. request (http.request.Request) – An HTTP request. response (http.request.Response) – An HTTP response. url_info (url.URLInfo) – A URLInfo.
Returns:	If True, the reader should be able to read it.
Return type:	bool

classmethod is_url(url_info)[source]¶

Return whether the URL is likely to be supported.

Parameters:	url_info (url.URLInfo) – A URLInfo.
Returns:	bool

class wpull.document.base.BaseExtractiveReader[source]¶

Bases: object

Base class for document readers that can only extract links.

iter_links(file, encoding=None)[source]¶

Return links from file.

Returns:	Each item is a str which represents a link.
Return type:	iterator

class wpull.document.base.BaseHTMLReader[source]¶

Bases: object

Base class for document readers for handling SGML-like documents.

iter_elements(file, encoding=None)[source]¶

Return an iterator of elements found in the document.

Parameters:	file – A file object containing the document. encoding (str) – The encoding of the document.
Returns:	Each item is an element from document.htmlparse.element
Return type:	iterator

class wpull.document.base.BaseTextStreamReader[source]¶

Bases: object

Base class for document readers that filters link and non-link text.

iter_links(file, encoding=None, context=False)[source]¶

Return the links.

This function is a convenience function for calling iter_text() and returning only the links.

iter_text(file, encoding=None)[source]¶

Return the file text and links.

Parameters:	file – A file object containing the document. encoding (str) – The encoding of the document.
Returns:	Each item is a tuple: str: The text bool (or truthy value): Whether the text is a likely a link. If truthy value may be provided containing additional context of the link.
Return type:	iterator

The links returned are raw text and will require further processing.

wpull.document.base.VeryFalse = <wpull.document.base.VeryFalseType object>¶

Document is not definitely supported.

class wpull.document.base.VeryFalseType[source]¶

Bases: object

document.css Module¶

Stylesheet reader.

class wpull.document.css.CSSReader[source]¶

Bases: wpull.document.base.BaseDocumentDetector, wpull.document.base.BaseTextStreamReader

Cascading Stylesheet Document Reader.

BUFFER_SIZE = 1048576¶

IMPORT_URL_PATTERN = '@import\\s*(?:url\\()?[\'"]?([^\\s\'")]{1,500}).*?;'¶

STREAM_REWIND = 4096¶

URL_PATTERN = 'url\\(\\s*([\'"]?)(.{1,500}?)(?:\\1)\\s*\\)'¶

URL_REGEX = re.compile('url\\(\\s*([\'"]?)(.{1,500}?)(?:\\1)\\s*\\)|@import\\s*(?:url\\()?[\'"]?([^\\s\'")]{1,500}).*?;')¶

classmethod is_file(file)[source]¶

Return whether the file is likely CSS.

classmethod is_request(request)[source]¶

Return whether the document is likely to be CSS.

classmethod is_response(response)[source]¶

Return whether the document is likely to be CSS.

classmethod is_url(url_info)[source]¶

Return whether the document is likely to be CSS.

iter_text(file, encoding=None)[source]¶

document.html Module¶

HTML document readers.

wpull.document.html.COMMENT = <object object>¶

Comment element

class wpull.document.html.HTMLLightParserTarget(callback, text_elements=frozenset({'style', 'script', 'link', 'url', 'icon'}))[source]¶

Bases: object

An HTML parser target for partial elements.

Parameters:	callback – A callback function. The function should accept the :param 1. tag: The tag name of the element. :type 1. tag: str :param 2. attrib: The attributes of the element. :type 2. attrib: dict :param 3. text: The text of the element. type 3. text: str, None text_elements – A frozenset of element tag names that we should keep track of text.

close()[source]¶

data(data)[source]¶

end(tag)[source]¶

start(tag, attrib)[source]¶

class wpull.document.html.HTMLParserTarget(callback)[source]¶

Bases: object

An HTML parser target.

Parameters:	callback – A callback function. The function should accept the :param 1. tag: The tag name of the element. :type 1. tag: str :param 2. attrib: The attributes of the element. :type 2. attrib: dict :param 3. text: The text of the element. :type 3. text: str, None :param 4. tail: The text after the element. :type 4. tail: str, None :param 5. end: Whether the tag is and end tag. type 5. end: bool

close()[source]¶

comment(text)[source]¶

data(data)[source]¶

end(tag)[source]¶

start(tag, attrib)[source]¶

class wpull.document.html.HTMLReadElement(tag, attrib, text, tail, end)[source]¶

Bases: object

Results from HTMLReader.read_links().

tag¶

str

The element tag name.

attrib¶

dict

The element attributes.

text¶

str, None

The element text.

tail¶

str, None

The text after the element.

end¶

bool

Whether the tag is an end tag.

attrib

end

tag

tail

text

class wpull.document.html.HTMLReader(html_parser)[source]¶

Bases: wpull.document.base.BaseDocumentDetector, wpull.document.base.BaseHTMLReader

HTML document reader.

Parameters:	html_parser (document.htmlparse.BaseParser) – An HTML parser.

classmethod is_file(file)[source]¶

Return whether the file is likely to be HTML.

classmethod is_request(request)[source]¶

Return whether the Request is likely to be a HTML.

classmethod is_response(response)[source]¶

Return whether the Response is likely to be HTML.

classmethod is_url(url_info)[source]¶

Return whether the URLInfo is likely to be a HTML.

iter_elements(file, encoding=None)[source]¶

document.htmlparse Module¶

HTML parsing.

document.htmlparse.base Module¶

class wpull.document.htmlparse.base.BaseParser[source]¶

Bases: object

parse(file, encoding=None)[source]¶

Parse the document for elements.

Returns:	Each item is from :module:`.document.htmlparse.element`
Return type:	iterator

parser_error¶

Return the Exception class for parsing errors.

document.htmlparse.element Module¶

HTML tree things.

wpull.document.htmlparse.element.Comment¶

A comment.

wpull.document.htmlparse.element.text¶

str

The comment text.

alias of CommentType

wpull.document.htmlparse.element.Doctype¶

A Doctype.

wpull.document.htmlparse.element.text

str

The Doctype text.

alias of DoctypeType

wpull.document.htmlparse.element.Element¶

An HTML element.

Attributes

tag (str): The tag name of the element. attrib (dict): The attributes of the element. text (str, None): The text of the element. tail (str, None): The text after the element. end (bool): Whether the tag is and end tag.

alias of ElementType

document.htmlparse.html5lib_ Module¶

Parsing using html5lib python.

class wpull.document.htmlparse.html5lib_.HTMLParser[source]¶

Bases: wpull.document.htmlparse.base.BaseParser

parse(file, encoding=None)[source]¶

parser_error¶

document.htmlparse.lxml_ Module¶

Parsing using lxml and libxml2.

class wpull.document.htmlparse.lxml_.HTMLParser[source]¶

Bases: wpull.document.htmlparse.base.BaseParser

HTML document parser.

This reader uses lxml as the parser.

BUFFER_SIZE = 131072¶

classmethod detect_parser_type(file, encoding=None)[source]¶

Get the suitable parser type for the document.

Returns:	str

parse(file, encoding=None)[source]¶

classmethod parse_doctype(file, encoding=None)[source]¶

Get the doctype from the document.

Returns:	str, None

parse_lxml(file, encoding=None, target_class=<class 'wpull.document.htmlparse.lxml_.HTMLParserTarget'>, parser_type='html')[source]¶

Return an iterator of elements found in the document.

Parameters:	file – A file object containing the document. encoding (str) – The encoding of the document. target_class – A class to be used for target parsing. parser_type (str) – The type of parser to use. Accepted values: html, xhtml, xml.
Returns:	Each item is an element from document.htmlparse.element
Return type:	iterator

parser_error¶

class wpull.document.htmlparse.lxml_.HTMLParserTarget(callback)[source]¶

Bases: object

An HTML parser target.

Parameters:	callback – A callback function. The function should accept one argument from document.htmlparse.element.

close()[source]¶

comment(text)[source]¶

data(data)[source]¶

end(tag)[source]¶

start(tag, attrib)[source]¶

wpull.document.htmlparse.lxml_.to_lxml_encoding(encoding)[source]¶

Check if lxml supports the specified encoding.

Returns:	str, None

document.javascript Module¶

class wpull.document.javascript.JavaScriptReader[source]¶

Bases: wpull.document.base.BaseDocumentDetector, wpull.document.base.BaseTextStreamReader

JavaScript Document Reader.

BUFFER_SIZE = 1048576¶

STREAM_REWIND = 4096¶

URL_PATTERN = '(\\\\{0,8}[\'"])(https?://[^\'"]{1,500}|[^\\s\'"]{1,500})(?:\\1)'¶

URL_REGEX = re.compile('(\\\\{0,8}[\'"])(https?://[^\'"]{1,500}|[^\\s\'"]{1,500})(?:\\1)')¶

classmethod is_file(file)[source]¶

Return whether the file is likely JS.

classmethod is_request(request)[source]¶

Return whether the document is likely to be JS.

classmethod is_response(response)[source]¶

Return whether the document is likely to be JS.

classmethod is_url(url_info)[source]¶

Return whether the document is likely to be JS.

iter_text(file, encoding=None)[source]¶

read_links(file, encoding=None)[source]¶

Return an iterator of links found in the document.

Parameters:	file – A file object containing the document. encoding (str) – The encoding of the document.
Returns:	str
Return type:	iterable

document.sitemap Module¶

Sitemap.xml

class wpull.document.sitemap.SitemapReader(html_parser)[source]¶

Bases: wpull.document.base.BaseDocumentDetector, wpull.document.base.BaseExtractiveReader

Sitemap XML reader.

MAX_ROBOTS_FILE_SIZE = 4096¶

classmethod is_file(file)[source]¶

Return whether the file is likely a Sitemap.

classmethod is_request(request)[source]¶

Return whether the document is likely to be a Sitemap.

classmethod is_response(response)[source]¶

Return whether the document is likely to be a Sitemap.

classmethod is_url(url_info)[source]¶

Return whether the document is likely to be a Sitemap.

iter_links(file, encoding=None)[source]¶

document.util Module¶

Misc functions.

wpull.document.util.detect_response_encoding(response, is_html=False, peek=131072)[source]¶

Return the likely encoding of the response document.

Parameters:	response (Response) – An instance of http.Response. is_html (bool) – See util.detect_encoding(). peek (int) – The maximum number of bytes of the document to be analyzed.
Returns:	The codec name.
Return type:	str, None

wpull.document.util.get_heading_encoding(response)[source]¶

Return the document encoding from a HTTP header.

Parameters:	response (Response) – An instance of http.Response.
Returns:	The codec name.
Return type:	str, None

wpull.document.util.is_gzip(data)[source]¶

Return whether the data is likely to be gzip.