AcousticBrainz Server Documentation¶
Contents:
Web API¶
The AcousticBrainz server provides a Web API for getting and submitting data.
Base URL: https://acousticbrainz.org
Endpoint Reference¶
Core data¶
-
GET
/api/v1/high-level
¶ Get high-level data for many recordings at once.
Example response:
{"mbid1": {"offset1": {document}, "offset2": {document}}, "mbid2": {"offset1": {document}}, "mbid_mapping": {"MBID1": "mbid1"} }
MBIDs and offset keys are returned as strings (as per JSON encoding rules). If an offset is not specified in the request for an mbid or is not a valid integer >=0, the offset will be 0.
MBID keys are always returned in a normalised form (all lower-case, separated in groups of 8-4-4-4-12 characters), even if the provided recording MBIDs are not given in this form. In the case that a requested MBID is not given in this normalised form, the value mbid_mapping in the response will be a dictionary mapping user-provided MBIDs to this form.
If the list of MBIDs in the query string has a recording which is not present in the database, then it is silently ignored and will not appear in the returned data.
Query Parameters: - recording_ids –
Required. A list of recording MBIDs to retrieve
Takes the form mbid[:offset];mbid[:offset]. Offsets are optional, and should be >= 0
You can specify up to
MAX_ITEMS_PER_BULK_REQUEST
MBIDs in a request. - map_classes – Optional. If set to ‘true’, map class names to human-readable values
Response Headers: - Content-Type – application/json
- recording_ids –
-
GET
/api/v1/low-level
¶ Get low-level data for many recordings at once.
Example response:
{"mbid1": {"offset1": {document}, "offset2": {document}}, "mbid2": {"offset1": {document}}, "mbid_mapping": {"MBID1": "mbid1"} }
MBIDs and offset keys are returned as strings (as per JSON encoding rules). If an offset is not specified in the request for an MBID or is not a valid integer >=0, the offset will be 0.
MBID keys are always returned in a normalised form, even if the provided recording MBIDs are not given in this form. In the case that a requested MBID is not given in this normalised form, the value mbid_mapping in the response will be a dictionary that maps user-provided MBIDs to this form.
If the list of MBIDs in the query string has a recording which is not present in the database, then it is silently ignored and will not appear in the returned data.
If you only need a specific feature and don’t want to download the whole low-level document, you can request specific features using the
features
parameter. The shape of the returned document will be the same as a full low-level document, and will contain only the specified features as well as themetadata.audio_properties
andmetadata.version
objects.Query Parameters: - recording_ids –
Required. A list of recording MBIDs to retrieve
Takes the form mbid[:offset];mbid[:offset]. Offsets are optional, and should be >= 0
You can specify up to
MAX_ITEMS_PER_BULK_REQUEST
MBIDs in a request. - features –
Optional. A list of features to be returned for each mbid.
Takes the form feature1;feature2.
You can specify the following features in a request:
LOWLEVEL_INDIVIDUAL_FEATURES
.
Response Headers: - Content-Type – application/json
- recording_ids –
-
GET
/api/v1/count
¶ Get low-level count for many recordings at once. MBIDs not found in the database are omitted in the response.
Example response:
{"mbid1": {"count": 3}, "mbid2": {"count": 1}, "mbid_mapping": {"MBID1": "mbid1"} }
MBID keys are always returned in a normalised form (all lower-case, separated in groups of 8-4-4-4-12 characters), even if the provided recording MBIDs are not given in this form. In the case that a requested MBID is not given in this normalised form, the value mbid_mapping in the response will be a dictionary mapping user-provided MBIDs to this form.
Query Parameters: - recording_ids –
Required. A list of recording MBIDs to retrieve
Takes the form mbid;mbid.
You can specify up to
MAX_ITEMS_PER_BULK_REQUEST
MBIDs in a request.Response Headers: - Content-Type – application/json
- recording_ids –
-
GET
/api/v1/
(uuid: mbid)/high-level
¶ Get high-level data for recording with a given MBID.
This endpoint returns one document at a time. If there are many submissions for an MBID, you can browse through them by specifying an offset parameter
n
. Documents are sorted by the submission time of their associated low-level documents.You can get the total number of low-level submissions using
/<mbid>/count
endpoint.Query Parameters: - n – Optional. Integer specifying an offset for a document.
- map_classes – Optional. If set to ‘true’, map class names to human-readable values
TODO: provide a link to what these mappings are
Response Headers: - Content-Type – application/json
-
GET
/api/v1/
(uuid: mbid)/low-level
¶ Get low-level data for a recording with a given MBID.
This endpoint returns one document at a time. If there are many submissions for an MBID, you can browse through them by specifying an offset parameter
n
. Documents are sorted by their submission time.You can the get total number of low-level submissions using the
/<mbid>/count
endpoint.Query Parameters: - n – Optional. Integer specifying an offset for a document.
Response Headers: - Content-Type – application/json
-
POST
/api/v1/
(uuid: mbid)/low-level
¶ Submit low-level data to AcousticBrainz.
Request Headers: - Content-Type – application/json
Response Headers: - Content-Type – application/json
-
GET
/api/v1/
(uuid: mbid)/count
¶ Get the number of low-level data submissions for a recording with a given MBID.
Example response:
{ "mbid": mbid, "count": n }
MBID values are always lower-case, even if the provided recording MBID is upper-case or mixed case.
Response Headers: - Content-Type – application/json
Datasets¶
-
POST
/api/v1/datasets/
¶ Create a new dataset.
Example request:
{ "name": "Mood", "description": "Dataset for mood classification.", "public": true, "classes": [ { "name": "Happy", "description": "Recordings that represent happiness.", "recordings": ["770cc467-8dde-4d22-bc4c-a42f91e"] }, { "name": "Sad" } ] }
Request Headers: - Content-Type – application/json
Request JSON Object: - name (string) – Required. Name of the dataset.
- description (string) – Optional. Description of the dataset.
- public (boolean) – Optional.
true
to make dataset public,false
to make it private. New datasets are public by default. - classes (array) –
Optional. Array of objects containing information about classes to add into new dataset. For example:
{ "name": "Happy", "description": "Recordings that represent happiness.", "recordings": ["770cc467-8dde-4d22-bc4c-a42f91e"] }
Response Headers: - Content-Type – application/json
Response JSON Object: - success (boolean) –
True
on successful creation. - dataset_id (string) – ID (UUID) of newly created dataset.
-
PUT
/api/v1/datasets/
(uuid: dataset_id)/recordings
¶ Add recordings to a class in a dataset.
Example request:
{ "class_name": "Happy", "recordings": ["770cc467-8dde-4d22-bc4c-a42f91e"] }
Request Headers: - Content-Type – application/json
Request JSON Object: - class_name (string) – Required. Name of the class.
- recordings (array) – Required. Array of recoding MBIDs (
string
) to add into that class.
Response Headers: - Content-Type – application/json
-
DELETE
/api/v1/datasets/
(uuid: dataset_id)/recordings
¶ Delete recordings from a class in a dataset.
Example request:
{ "class_name": "Happy", "recordings": ["770cc467-8dde-4d22-bc4c-a42f91e"] }
Request Headers: - Content-Type – application/json
Request JSON Object: - class_name (string) – Required. Name of the class.
- recordings (array) – Required. Array of recoding MBIDs (
string
) that need be deleted from a class.
Response Headers: - Content-Type – application/json
-
POST
/api/v1/datasets/
(uuid: dataset_id)/classes
¶ Add a class to a dataset.
The data can include an optional list of recording ids. If these are included, the recordings are also added to the list. Duplicate recording ids are ignored.
If a class with the given name already exists, the recordings (if provided) will be added to the existing class.
Example request:
{ "name": "Not Mood", "description": "Dataset for mood misclassification.", "recordings": ["770cc467-8dde-4d22-bc4c-a42f91e"] }
Request Headers: - Content-Type – application/json
Request JSON Object: - name (string) – Required. Name of the class. Must be unique within a dataset.
- description (string) – Optional. Description of the class.
- recordings (array) – Optional. Array of recording MBIDs (
string
) to add into that class. For example:["770cc467-8dde-4d22-bc4c-a42f91e"]
.
Response Headers: - Content-Type – application/json
-
PUT
/api/v1/datasets/
(uuid: dataset_id)/classes
¶ Update class in a dataset.
If one of the fields is not specified, it will not be updated.
Example request:
{ "name": "Very happy", "new_name": "Recordings that represent ultimate happiness." }
Request Headers: - Content-Type – application/json
Request JSON Object: - name (string) – Required. Current name of the class.
- new_name (string) – Optional. New name of the class. Must be unique within a dataset.
- description (string) – Optional. Description of the class.
Response Headers: - Content-Type – application/json
-
DELETE
/api/v1/datasets/
(uuid: dataset_id)/classes
¶ Delete class and all of its recordings from a dataset.
Example request:
{ "name": "Sad" }
Request Headers: - Content-Type – application/json
Request JSON Object: - name (string) – Required. Name of the class.
Response Headers: - Content-Type – application/json
-
GET
/api/v1/datasets/
(uuid: dataset_id)¶ Retrieve a dataset.
Response Headers: - Content-Type – application/json
-
DELETE
/api/v1/datasets/
(uuid: dataset_id)¶ Delete a dataset.
-
PUT
/api/v1/datasets/
(uuid: dataset_id)¶ Update dataset details.
If one of the fields is not specified, it will not be updated.
Example request:
{ "name": "Not Mood", "description": "Dataset for mood misclassification.", "public": true }
Request Headers: - Content-Type – application/json
Request JSON Object: - name (string) – Optional. Name of the dataset.
- description (string) – Optional. Description of the dataset.
- public (boolean) – Optional.
true
to make dataset public,false
to make it private.
Response Headers: - Content-Type – application/json
Similarity¶
-
GET
/api/v1/similarity/
(metric)/between/
¶ Get the distance measure for similarity between two MBIDs. The distance measure will correspond to the index of the specified metric.
The metrics available are shown here
BASE_METRIC_NAMES
.Example response:
{"metric": [<distance_vector>]}
NOTE If an (MBID, offset) combination specified does not exist, or is not present in the index, an empty dictionary will be returned.
Query Parameters: - recording_ids –
Required. A list of the two recordings for which similarity should be found between.
Takes the form mbid[:offset]:mbid[:offset]. Offsets are optional, and should be integers >= 0
Response Headers: - Content-Type – application/json
- recording_ids –
-
GET
/api/v1/similarity/
(metric)/
¶ Get the most similar submissions to multiple (MBID, offset) combinations.
Example response:
{"mbid1": {"offset1": [(most_similar_mbid, offset), ..., (least_similar_mbid, offset)], "offset2": [(most_similar_mbid, offset), ..., (least_similar_mbid, offset)] }, ..., "mbidN": {"offset1": [(most_similar_mbid, offset), ..., (least_similar_mbid, offset)] } }
MBIDs and offset keys are returned as strings (as per JSON encoding rules). If an offset is not specified in the request for an mbid or is not a valid integer >=0, the offset will be 0.
If the list of MBIDs in the query string has a recording which is not present in the database, then it is silently ignored and will not appear in the returned data.
Query Parameters: - recording_ids – Required. A list of recording MBIDs to retrieve Takes the form mbid[:offset];mbid[:offset]. Offsets are optional, and should be >= 0
- n_neighbours – Optional. The number of similar recordings that
should be returned for each item in
recording_ids
(1-1000). Default is 200 recordings. - metric – Required. String specifying the metric name to be
used when finding the most similar recordings.
The metrics available are shown here
BASE_METRIC_NAMES
. - threshold – Optional. Only return items whose distance from the query recording is less than this (0-1). This may result in the number of returned items being less than n_neighbours if it is set.
- remove_dups – Optional. If
samescore
, remove duplicate submissions that have the same score. Ifall
, remove all duplicate MBIDs even if they have different scores, returning only the submission with the lowest score. This may result in the number of returned items being less than n_neighbours if it is set.
Response Headers: - Content-Type – application/json
Rate limiting¶
The AcousticBrainz API is rate limited via the use of rate limiting headers that are sent as part of the HTTP response headers. Each call will include the following headers:
- X-RateLimit-Limit: Number of requests allowed in given time window
- X-RateLimit-Remaining: Number of requests remaining in current time window
- X-RateLimit-Reset-In: Number of seconds when current time window expires (recommended: this header is resilient against clients with incorrect clocks)
- X-RateLimit-Reset: UNIX epoch number of seconds (without timezone) when current time window expires [1]
We typically set the limit to 10 queries every 10 seconds per IP address, but these values may change. Make sure you check the response headers if you want to know the specific values.
Rate limiting is automatic and the client must use these headers to determine
the rate to make API calls. If the client exceeds the number of requests
allowed, the server will respond with error code 429: Too Many Requests
.
Requests that provide the Authorization header with a valid user token may
receive higher rate limits than those without valid user tokens.
[1] | Provided for compatibility with other APIs, but we still recommend using
X-RateLimit-Reset-In wherever possible |
Constants¶
Constants that are relevant to using the API:
-
webserver.views.api.v1.core.
MAX_ITEMS_PER_BULK_REQUEST
= 25¶ The maximum number of items that you can pass as a recording_ids parameter to bulk lookup endpoints
-
webserver.views.api.v1.core.
LOWLEVEL_INDIVIDUAL_FEATURES
= ['lowlevel.average_loudness', 'lowlevel.dynamic_complexity', 'metadata.audio_properties.replay_gain', 'metadata.tags', 'rhythm.beats_count', 'rhythm.beats_loudness.mean', 'rhythm.bpm', 'rhythm.bpm_histogram_first_peak_bpm.mean', 'rhythm.bpm_histogram_second_peak_bpm.mean', 'rhythm.danceability', 'rhythm.onset_rate', 'tonal.chords_changes_rate', 'tonal.chords_key', 'tonal.chords_scale', 'tonal.key_key', 'tonal.key_scale', 'tonal.key_strength', 'tonal.tuning_equal_tempered_deviation', 'tonal.tuning_frequency']¶ Features that can be selected individually from the bulk low-level endpoint
-
similarity.metrics.
BASE_METRIC_NAMES
= ['mfccs', 'mfccsw', 'gfccs', 'gfccsw', 'key', 'bpm', 'onsetrate', 'moods', 'instruments', 'dortmund', 'rosamerica', 'tzanetakis']¶ Available similarity metrics
Recording Similarity¶
An emerging feature in AcousticBrainz is recording similarity.
We compute similarity between recordings based on a selection of high and low-level features [metrics].
Similarity metrics for each recording should be computed on submission. When the metrics for a recording have been added to static index files, we can query for track-track similarity based on our choice of feature.
This dataset is made available through a set of API endpoints, and distribution of the static index files for more flexibility.
Similarity Metrics¶
Similarity metrics are methods of applying a vectorized measurement to one, or many features of high and low-level data.
Thus far, the following metrics can be used to assess similarity:
Metric | Hybrid | Category |
---|---|---|
MFCCs | False | Timbre |
MFCCs (Weighted) | False | Timbre |
GFCCs | False | Timbre |
GFCCs (Weighted) | False | Timbre |
Key | False | Rhythm (Key/Scale) |
BPM | False | Rhythm |
OnsetRate | False | Rhythm |
Moods | False | High-Level |
Instruments | False | High-Level |
Dortmund | False | High-Level (Genre) |
Rosamerica | False | High-Level (Genre) |
Tzanetakis | False | High-Level (Genre) |
Note: Hybrid metrics are combinations of multiple metrics. In the future, we hope to integrate more metrics that combine low-level features for a more holistic approach to similarity.
Similarity Statistics¶
Some of our metrics are normalized using the mean and standard deviation of their features from the lowlevel table. We must compute the statistics for such metrics prior to computing the metrics themselves. To do so, we collect a sample of the lowlevel table and use it to approximate the mean and standard deviation. Currently, the metrics that require statistics are the following:
- MFCCs
- Weighted MFCCs
- GFCCs
- Weighted GFCCs
Indexing¶
Metric vectors for each recording are added to static indices, created with the Annoy library. An individual index exists for each of the metrics available. An index uses a nearest neighbours approach for queries.
Note that computing an index takes time, and thus it cannot happen each time a recording is submitted. Indices are recomputed, including new submissions, on a time interval.
More can be read about indices and contributing to similarity work in the developer reference.
Evaluation¶
The similarity engine is an ongoing project, and its results are still largely experimental. While we tune different index parameters (described in the developer reference), we’d like to gather feedback from the community.
As such, we’ve made an evaluation available to the public. When viewing the summary data for a recording, you may access similar recordings organized by metric. Recordings are organized from most similar to least similar. Alongside the list of the most similar recordings, you may provide input:
- Rate whether the recording should be higher or lower on the list of similarity, or whether this result seems accurate.
- Provide additional suggestions related to a specific similar recording, or in general.
Feel free to provide as much or as little feedback as you wish when browsing. We appreciate your help in improving similarity at AcousticBrainz!
Deployment Practices¶
We run two instances of AcousticBrainz on the same data.
- AcousticBrainz (https://acousticbrainz.org)
- AcousticBrainz beta (https://beta.acousticbrainz.org)
The beta instance is used to test new features and bugfixes on the actual dataset before being officially deployed into production.
We follow the general MetaBrainz guidelines to deploy, test and release code into beta and production.
Developing for Similarity¶
Setting up the environment¶
NOTE: If there is no submitted data in the lowlevel table, similarity cannot be initialized.
Once your database and server are built, there are a few steps required to initialize the similarity engine. These steps can be run completed altogether using the following cli command:
./develop.sh manage similarity init
Environment subcommands¶
The init
similarity command runs each of these sub-commands in order. They can be run
individually to perform a specific step.
Compute required statistics over the lowlevel table:
./develop.sh manage similarity compute-stats
Extract data values for all recordings in the lowlevel table:
./develop.sh manage similarity add-metrics
This command can be run again at any time to extract values for recordings that have been added since the last time that this command was run.
Finally, build the annoy indices:
./develop.sh manage similarity add-indices
This command will rebuild an index from scratch, reading from the metrics that were extracted from
the add-metrics
command. This should be run each time after add-metrics
to rebuild the
similarity indexes.
It is possible to alter the index parameters using additional arguments.
Similarity implementation details¶
Metrics list¶
We store a list of metrics in the similarity.similarity_metrics database table.
A definition of these each of these metrics is stored in the similarity.metrics
python module.
These definitions describe which database field to load the metric from, and any preprocessing
that has to be performed before adding it to the index.
Statistics¶
Some features (Weighted MFCCs, Weighted GFCCs) require a mean and standard deviation value to
perform normalisation.
We can’t compute the mean of all items in the database as this would take too long, so
instead we take a random sample (by default similarity.manage.NORMALIZATION_SAMPLE_SIZE
, 10000) of items
and compute the mean and SD on those items. In our experiments this gave a good tradeoff of accuracy vs
computation speed.
Speed of database access and indexing¶
Reading all items directly from the database (lowlevel_json table) is too slow to do every time
the annoy index needs to be updated. Because of this, we split the import process into two steps.
The add-metrics
subcommand extracts data from the lowlevel table into a new summary table
(similarity.similarity) that includes only the specific numerical feature necessary for the similarity index.
The add-indices
subcommand reads from the similarity.similarity table and builds an Annoy index. We build
this index from scratch each time the command is run due to technical requirements in Annoy, and the fact that
this operation doesn’t take too long to complete.
Adding a new feature¶
(todo: fill in with more detail)
- Add new metric to
admin/sql/populate_metrics_table.sql
- Add new metric class to
similarity.metrics
- Add this class to
similarity.metrics.BASE_METRICS
- Extract new data if necessary in
db.similarity.get_batch_data
- alter similarity.similarity table to include new column for this feature
- todo: should we be able to fill in just this column, or do we recreate similarity.similarity?
Index parameters¶
Indices take a number of parameters when being built. We are currently evaluating them ourselves and have developed a set of base indices, but you may want to experiment with different parameters.
Queries to an index become more precise when the index is built with a higher number of trees, but they will also take longer to build. Indices can also vary in the method used to calculate distance between recordings. Limitations of parameter selection can be found in the Annoy documentation and our codebase