edeposit.amqp.aleph_link_export¶
Two-way communication system used to deliver update requests from E-deposit to Aleph used in Czech National library. Updates may contain new http links to other systems or identifiers, like URN:NBN, or UUID.
Links created in Aleph cannot be changed automatically by direct-manipulation, so this project defines AMQP to XML protocol bridge. XML then copied over SCP to Aleph, processed and resulting XML response is then translated back to AMQP messages received by E-deposit.
Package structure¶
File relations¶
API¶
link_export submodule¶
This module provides API wrapper over RequestDatabase
.
-
aleph_link_export.link_export.
get_rd
(fn)¶
-
aleph_link_export.link_export.
export
(request)¶ Save the export request to the database.
Parameters: request (obj) – Instance of LinkUpdateRequest
.
-
aleph_link_export.link_export.
collect_responses
()¶ Collect processed resposes.
Returns: List of LinkUpdateResponse
objects.Return type: list
request_database submodule¶
This module is used to keep incomming and outgoing structures and also for serialization/deserialization to/from XML.
-
class
aleph_link_export.request_database.
RequestDatabase
(log_fn='/home/aleph_export/link_export.log', db_fn='/home/aleph_export/request_datase.shelve', db_key='request_database', logging=True, req_fn='/home/aleph_export/edep2aleph/requests.xml', resp_fn='/home/aleph_export/aleph2edep/responses.xml', xsd_url='http://edeposit-aplikace.nkp.cz/link_export_notification.xsd')¶ Bases:
aleph_link_export.shelvedb.ShelveDatabase
Keep the track of requests and resposes and their serialization and deserialization from/to XMl.
-
req_fn
= None¶ Path to the request XML.
-
resp_fn
= None¶ Path to the response XML
-
add_request
(request)¶ Add new request object to database.
Parameters: request (obj) – Object with defined session_id
property andto_dict_xml()
method.
-
get_responses
()¶ Process response queue, remove finished requests from request queue, return list of response objects.
Returns: List of LinkUpdateResponse
objects.Return type: list
-
save
()¶ Read the response XML, process it, save the database and request XML.
-
static
load
(fn='/home/aleph_export/request_datase.shelve', db_key='request_database', creator=<function <lambda>>)¶ Load the database from the shelve fn.
Parameters: - fn (str) – Path to the database file. Default
DATABASE_FN
. - db_key (str) – What database key to use. Default
DATABASE_KEY
. - creator (fn reference) – Reference to the function, which will
create new
RequestDatabase
if the old is not found. Default lambda, which expects fn parameterlambda fn: ..
.
Returns: RequestDatabase
instance from the fn or newlycreated.
Return type: obj
- fn (str) – Path to the database file. Default
-
shelvedb submodule¶
Module used to load/store object from/to shelve database.
-
aleph_link_export.shelvedb.
shelver
(*args, **kwds)¶ In python 2.7, there is no context manager for shelve. So this is it.
-
class
aleph_link_export.shelvedb.
ShelveDatabase
(log_fn, db_fn, db_key, logging)¶ Bases:
object
Class that can save and load itself to shelve.
-
db_fn
= None¶ Path to the database file.
-
log_fn
= None¶ Path to the log file.
-
logging
= None¶ Is the logging enabled?
-
save
()¶ Save this object to shelve.
-
static
load
(fn, db_key, creator)¶ Load the database from the shelve fn.
Parameters: - fn (str) – Path to the database file.
- db_key (str) – What database key to use. Default
DATABASE_KEY
. - creator (reference) – Reference to the function, which will
create new
RequestDatabase
if the old is not found. Default lambda, which expects fn parameterlambda fn: ..
.
Returns: RequestDatabase
instance from the fn or newlycreated.
Return type: obj
-
settings submodule¶
Module is containing all necessary global variables for the package.
Module also has the ability to read user-defined data from two paths:
$HOME/_SETTINGS_PATH
/etc/_SETTINGS_PATH
See _SETTINGS_PATH
for details.
Note
If the first path is found, other is ignored.
Example of the configuration file ($HOME/edeposit/aleph_export.json
):
{
"REQUEST_FN": "/home/whatever/req.xml"
}
Attributes¶
-
aleph_link_export.settings.
REQUEST_FN
= '/home/aleph_export/edep2aleph/requests.xml'¶ Path to the XML file, where the requests will be stored.
-
aleph_link_export.settings.
RESPONSE_FN
= '/home/aleph_export/aleph2edep/responses.xml'¶ Path to the file, where the Aleph will put the XML responses.
-
aleph_link_export.settings.
DATABASE_FN
= '/home/aleph_export/request_datase.shelve'¶
-
aleph_link_export.settings.
LOG_FN
= '/home/aleph_export/link_export.log'¶ Path to the file, where the logs will be stored.
-
aleph_link_export.settings.
DATABASE_KEY
= 'request_database'¶ Don’t change this! Key for the database.
-
aleph_link_export.settings.
EXPORT_XSD_LINK
= 'http://edeposit-aplikace.nkp.cz/link_export_notification.xsd'¶ Link to the export XSD
-
aleph_link_export.settings.
LOGGING_ENABLED
= True¶ Logging enabled or not?
Request structures¶
-
class
aleph_link_export.structures.requests.
LinkDescription
¶ Bases:
aleph_link_export.structures.requests.LinkDescription
Optional structure, which can be used instead of string to describe the format of the url.
-
url
¶ str – URL of the document.
-
format
¶ str – Format of the document.
Create new instance of LinkDescription(url, format)
-
to_dict_xml
()¶ Serialize the object to dictionary, which may be later used for conversion to XML.
- Retruns:
- OrderedDict: Itself as ordered dict.
-
-
class
aleph_link_export.structures.requests.
LinkUpdateRequest
¶ Bases:
aleph_link_export.structures.requests.LinkUpdateRequest
Request to update metadata in Aleph.
-
uuid
¶ str – UUID for the doc_number you wish to update.
-
doc_number
¶ str – Document number of the document you wish to update. If there is a summary aleph record for this document at Aleph please send a doc_number of this summary aleph record.
-
document_urls
¶ list – Newly added public URL to the storage / whatever subsystem. List of strings or
LinkDescription
.
-
kramerius_url
¶ str, default None – Newly added URL to the Kramerius subsystem.
-
to_dict_xml
()¶ Convert the structure to nested ordered-dicts, which are later used for construction of the XML.
Returns: Itself as ordered dicts. Return type: OrderedDict
-
-
class
aleph_link_export.structures.requests.
StatusRequest
¶ Bases:
aleph_link_export.structures.requests.StatusRequest
This structure is used to wake the daemon to go and check whether the files on the disc changed or not.
Create new instance of StatusRequest()
Response structures¶
-
class
aleph_link_export.structures.responses.
LinkUpdateResponse
¶ Bases:
aleph_link_export.structures.responses.LinkUpdateResponse
Response to the
LinkUpdateRequest
request.This object is returned only when the Aleph signals, that the record was really updated.
-
status
¶ str – Status of the update request. Either
OK
, orERROR
.
-
session_id
¶ str – Corresponding session id.
-
reason
¶ str, default None – Optional reason why the request was rejected.
Create new instance of LinkUpdateResponse(status, session_id, reason)
-
AMQP protocol¶
Here is the list of Request -> Response
pairs describing responses to AMQP communication:
LinkUpdateRequest ---> 0-N × LinkUpdateResponse
StatusRequest -------> 0-N × LinkUpdateResponse
Protocol is really simple - you can send the LinkUpdateRequest
and
you will get back all waiting LinkUpdateResponse
responses. If there is none, you won’t get any.
You can also trigger the lookup for waiting responses by sending periodic StatusRequest
messages.
Installation¶
Module is hosted at PYPI, and can be easily installed using PIP:
sudo pip install edeposit.amqp.aleph_link_export
Don’t forget to add proper paths into your configuration file (see settings
for details) in /home/edeposit/aleph_export.json
or /etc/edeposit/aleph_export.json
.
Example:
{
"REQUEST_FN": "/home/aleph_export/edep2aleph.xml",
"RESPONSE_FN": "/home/aleph_export/aleph2edep.xml",
"DATABASE_FN": "/home/aleph_export/database.shelve",
"LOG_FN": "/home/aleph_export/log.txt"
}
- Warning:
- The directories have to be created before you try to run the project!
Source code¶
Project is released under the MIT license. Source code can be found at GitHub:
Unittests¶
Almost every feature of the project is tested by unittests. You can run those
tests using provided run_tests.sh
script, which can be found in the root
of the project.
If you have any trouble, just add --pdb
switch at the end of your run_tests.sh
command like this: ./run_tests.sh --pdb
. This will drop you to PDB shell.
Requirements¶
This script expects that package pytest is installed. In case you don’t have it yet, it can be easily installed using following command:
pip install --user pytest
or for all users:
sudo pip install pytest
Example¶
$ ./run_tests.sh
============================= test session starts ==============================
platform linux2 -- Python 2.7.6 -- py-1.4.30 -- pytest-2.7.2
rootdir: /home/bystrousak/Plocha/Dropbox/c0d3z/prace/edeposit.amqp.aleph_link_export, inifile:
plugins: cov
collected 17 items
tests/test_RequestDatabase.py .......
tests/test_amqp_chain.py ....
tests/structures/test_requests.py .....
tests/structures/test_responses.py .
========================== 17 passed in 0.25 seconds ===========================