Django Concurrency

Overview

Test status Coverage

django-concurrency is an optimistic locking library for Django Models

It prevents users from doing concurrent editing in Django both from UI and from a django command.

How it works

Overview

django-concurrency works adding a concurrency.fields.VersionField to each model, each time a record is saved the version number changes (the algorithm used depends on the implementation of concurrency.fields.VersionField used (see Fields).

Each update is converted in the following SQL clause like:

UPDATE mymodel SET version=NEW_VERSION, ... WHERE id = PK AND version = VERSION_NUMBER

Table Of Contents

Install

Using pip:

pip install django-concurrency

Go to https://github.com/saxix/django-concurrency if you need to download a package or clone the repo.

django-concurrency does not need to be added into INSTALLED_APPS unless you want to run the tests or use the templatetags and/or admin integration

Test suite

django-concurrency comes with a set of tests that can simulate different scenarios

  • basic versioned model

  • inherited model

  • inherited model from abstract model

  • inherited model from external project model

  • django User model

  • models with custom save

  • proxy models

  • admin actions

How to run the tests
$ pip install tox
$ tox

Fields

VersionField

class concurrency.fields.VersionField(*args, **kwargs)[source]

Base class

IntegerVersionField

class concurrency.fields.IntegerVersionField(*args, **kwargs)[source]

Version Field that returns a “unique” version number for the record.

The version number is produced using time.time() * 1000000, to get the benefits of microsecond if the system clock provides them.

AutoIncVersionField

class concurrency.fields.AutoIncVersionField(*args, **kwargs)[source]

Version Field increment the revision number each commit

TriggerVersionField

class concurrency.fields.TriggerVersionField

This field use a database trigger to update the version field. Using this you can control external updates (ie using tools like phpMyAdmin, pgAdmin, SQLDeveloper). The trigger is automatically created during syncdb() or you can use the triggers management command.

Changed in version 1.0.

Warning

Before django-concurrency 1.0 two triggers per field were created, if you are upgrading you must manually remove old triggers and recreate them using triggers management command

trigger_name

New in version 1.0.

TriggerVersionField.trigger_name

Starting from 1.0 you can customize the name of the trigger created. Otherwise for each TriggerVersionField will be created two triggers named:

'concurrency_[DBTABLENAME]_[FIELDNAME]'

Warning

Any name will be automatically prefixed with concurrency_

triggers management command

Helper command to work with triggers:

  • list : list existing triggers for each database

  • drop : drop exisitng triggers

  • create : create required triggers

example

sax@: (concurrency) django-concurrency [feature/triggers*] $ ./demo/manage.py triggers create
DATABASE             TRIGGERS
default              concurrency_concurrency_triggerconcurrentmodel_u

ConditionalVersionField

New in version 1.1.

This field allow to configure which fields trigger the version increment so to limit the scope of the concurrency checks.

class User(models.Model):
    version = ConditionalVersionField()
    username = models.CharField(...)
    password = models.PasswordField(...)

    class ConcurrencyMeta:
        check_fields = ('username',)

ConcurrencyMiddleware

You can globally intercept RecordModifiedError adding ConcurrencyMiddleware to your MIDDLEWARE_CLASSES. Each time a RecordModifiedError is raised it goes up to the ConcurrencyMiddleware and the handler defined in CONCURRENCY_HANDLER409 is invoked.

Example

settings.py

MIDDLEWARE_CLASSES=('django.middleware.common.CommonMiddleware',
                    'concurrency.middleware.ConcurrencyMiddleware',
                    'django.contrib.sessions.middleware.SessionMiddleware',
                    'django.middleware.csrf.CsrfViewMiddleware',
                    'django.contrib.auth.middleware.AuthenticationMiddleware',
                    'django.contrib.messages.middleware.MessageMiddleware')

CONCURRENCY_HANDLER409 = 'demoproject.demoapp.views.conflict'
CONCURRENCY_POLICY = 2  # CONCURRENCY_LIST_EDITABLE_POLICY_ABORT_ALL

views.py

from diff_match_patch import diff_match_patch
from concurrency.views import ConflictResponse
from django.template import loader
from django.utils.safestring import mark_safe
from django.template.context import RequestContext

def get_diff(current, stored):
    data = []
    dmp = diff_match_patch()
    fields = current._meta.fields
    for field in fields:
        v1 = getattr(current, field.name, "")
        v2 = getattr(stored, field.name, "")
        diff = dmp.diff_main(unicode(v1), unicode(v2))
        dmp.diff_cleanupSemantic(diff)
        html = dmp.diff_prettyHtml(diff)
        html = mark_safe(html)
        data.append((field, v1, v2, html))
    return data

def conflict(request, target=None, template_name='409.html'):
    template = loader.get_template(template_name)
    try:
        saved = target.__class__._default_manager.get(pk=target.pk)
        diff = get_diff(target, saved)
    except target.__class__.DoesNotExists:
        saved = None
        diff = None

    ctx = RequestContext(request, {'target': target,
                                   'diff': diff,
                                   'saved': saved,
                                   'request_path': request.path})
    return ConflictResponse(template.render(ctx))

409.html

{% load concurrency %}
<table>
    <tr>
        <th>
            Field
        </th>
        <th>
            Current
        </th>
        <th>
            Stored
        </th>
        <th>
            Diff
        </th>

    </tr>
    <tr>
        {% for field, current, stored, entry in diff %}
            {% if not field.primary_key and not field|is_version %}
                <tr>
                    <td>
                        {{ field.verbose_name }}
                    </td>
                    <td>
                        {{ current }}
                    </td>
                    <td>
                        {{ stored }}
                    </td>
                    <td>
                        {{ entry }}
                    </td>
                </tr>
            {% endif %}
        {% endfor %}
    </tr>
</table>

If you want to use ConcurrencyMiddleware in the admin and you are using concurrency.admin.ConcurrentModelAdmin remember to set your ModelAdmin to NOT use concurrency.forms.ConcurrentForm

from django import forms

class MyModelAdmin(ConcurrentModelAdmin):
    form = forms.ModelForm  # overrides default ConcurrentForm

Admin Integration

Handle list_editable

New in version 0.6.

django-concurrency is able to handle conflicts in the admin’s changelist view when ModelAdmin.list_editable is enabled. To enable this feature simply extend your ModelAdmin from ConcurrentModelAdmin or use ConcurrencyListEditableMixin

See also

POLICY

Check admin’s action execution for concurrency

New in version 0.6.

Extend your ModelAdmin with ConcurrencyActionMixin or use ConcurrentModelAdmin

Update existing actions templates to be managed by concurrency

New in version 0.6.

You ca use the identity filter to pass both pk and version to your ModelAdmin. Each time you use {{ obj.pk }} simply change to {{ obj|identity }}. So in the admin/delete_selected_confirmation.html will have:

{% for obj in queryset %}
<input type="hidden" name="{{ action_checkbox_name }}" value="{{ obj|identity }}" />
{% endfor %}

API

Forms

ConcurrentForm
class concurrency.forms.ConcurrentForm(data=None, files=None, auto_id='id_%s', prefix=None, initial=None, error_class=<class 'django.forms.utils.ErrorList'>, label_suffix=None, empty_permitted=False, instance=None, use_required_attribute=None, renderer=None)[source]

Simple wrapper to ModelForm that try to mitigate some concurrency error. Note that is always possible have a RecordModifiedError in model.save(). Statistically form.clean() should catch most of the concurrent editing, but is good to catch RecordModifiedError in the view too.

VersionWidget
class concurrency.forms.VersionWidget(attrs=None)[source]

Widget that show the revision number using <div>

Usually VersionField use HiddenInput as Widget to minimize the impact on the forms, in the Admin this produce a side effect to have the label Version without any value, you should use this widget to display the current revision number

Exceptions

VersionChangedError
class concurrency.exceptions.VersionChangedError(message, code=None, params=None)[source]
class concurrency.exceptions.RecordModifiedError
RecordModifiedError
class concurrency.exceptions.RecordModifiedError(*args, **kwargs)[source]
InconsistencyError

Changed in version 0.7.

Warning

removed in 0.7

class concurrency.exceptions.InconsistencyError
VersionError
class concurrency.exceptions.VersionError(message=None, code=None, params=None, *args, **kwargs)[source]

Admin

ConcurrentModelAdmin
class concurrency.admin.ConcurrentModelAdmin(model, admin_site)[source]

Warning

If you customize fields or fieldsets remember to add version field to the list. (See issue issue #81)

ConcurrencyActionMixin
class concurrency.admin.ConcurrencyActionMixin[source]
ConcurrencyListEditableMixin
class concurrency.admin.ConcurrencyListEditableMixin[source]

Middleware

class concurrency.middleware.ConcurrencyMiddleware
ConcurrencyMiddleware

See also

ConcurrencyMiddleware

class concurrency.middleware.ConcurrencyMiddleware(get_response=None)[source]

Intercept RecordModifiedError and invoke a callable defined in CONCURRECY_HANDLER409 passing the request and the object.

concurrency.views.conflict
concurrency.views.conflict(request, target=None, template_name='409.html')[source]

409 error handler.

Parameters

  • request – Request

  • template_name – 409.html

  • target – The model to save

Helpers

apply_concurrency_check()

New in version 0.4.

Changed in version 0.8.

Add concurrency check to existing classes.

Note

With Django 1.7 and the new migrations management, this utility does not work anymore. To add concurrency management to a external Model, you need to use a migration to add a VersionField to the desired Model.

Note

See demo.auth_migrations for a example how to add IntegerVersionField to auth.Group )

operations = [
    # add version to django.contrib.auth.Group
    migrations.AddField(
        model_name='Group',
        name='version',
        field=IntegerVersionField(help_text=b'Version', default=1),
    ),
]

and put in your settings.py

MIGRATION_MODULES = {
    ...
    ...
    'auth': '<new.migration.package>',
}
disable_concurrency()

New in version 0.6.

Context manager to temporary disable concurrency checking.

Changed in version 0.9.

Starting from version 0.9, disable_concurrency can disable both at Model level or instance level, depending on the passed object. Passing Model is useful in django commands, load data or fixtures, where instance should be used by default

Changed in version 1.0.

Is now possible use disable_concurrency without any argument to disable concurrency on any Model. This features has been developed to be used in django commands

Changed in version 1.1.

Does not raise an exception if a model not under concurrency management is passed as argument.

examples
@disable_concurrency()
def recover_view(self, request, version_id, extra_context=None):
    return super().recover_view(request,
                                                        version_id,
                                                        extra_context)

def test_recover():
    deleted_list = revisions.get_deleted(ReversionConcurrentModel)
    delete_version = deleted_list.get(id=5)

    with disable_concurrency(ReversionConcurrentModel):
        deleted_version.revert()
concurrency_disable_increment()

New in version 1.1.

Context manager to temporary disable version increment. Concurrent save is still checked but no version increment is triggered, this creates ‘shadow saves’,

It accepts both a Model or an instance as target.

Templatetags

identity
concurrency.templatetags.concurrency.identity(obj)[source]

returns a string representing “<pk>,<version>” of the passed object

version
concurrency.templatetags.concurrency.version(obj)[source]

returns the value of the VersionField of the passed object

is_version
concurrency.templatetags.concurrency.is_version(field)[source]

returns True if passed argument is a VersionField instance

Triggers

TriggerFactory

New in version 2.3.

class concurrency.triggers.TriggerFactory(connection)[source]

Abstract Factory class to create triggers. Implemementations need to set the following attributes

update_clause, drop_clause and list_clause

Those will be formatted using standard python format() as:

self.update_clause.format(trigger_name=field.trigger_name,
                                   opts=field.model._meta,
                                   field=field)

So as example:

update_clause =  """CREATE TRIGGER {trigger_name}
            AFTER UPDATE ON {opts.db_table}
            BEGIN UPDATE {opts.db_table}
            SET {field.column} = {field.column}+1
            WHERE {opts.pk.column} = NEW.{opts.pk.column};
            END;
            """

See also

TRIGGERS_FACTORY

Test Utilties

ConcurrencyTestMixin
class concurrency.utils.ConcurrencyTestMixin[source]

Mixin class to test Models that use VersionField

this class offer a simple test scenario. Its purpose is to discover some conflict in the save() inheritance:

from concurrency.utils import ConcurrencyTestMixin
from myproject.models import MyModel

class MyModelTest(ConcurrencyTestMixin, TestCase):
    concurrency_model = TestModel0
    concurrency_kwargs = {'username': 'test'}

Signining

New in version 0.5.

VersionField is ‘displayed’ in the Form using an django.forms.HiddenInput widget, anyway to be sure that the version is not tampered with, its value is signed. The default VersionSigner is concurrency.forms.VersionFieldSigner that simply extends django.core.signing.Signer. If you want change your Signer you can set CONCURRENCY_FIELD_SIGNER in your settings

mysigner.py

class DummySigner():
    """ Dummy signer that simply returns the raw version value. (Simply do not sign it) """
    def sign(self, value):
        return smart_str(value)

    def unsign(self, signed_value):
        return smart_str(signed_value)

settings.py

CONCURRENCY_FIELD_SIGNER = "myapp.mysigner.DummySigner"

Settings

Here’s a full list of all available settings, in alphabetical order, and their default values.

Note

Each entry MUST have the prefix CONCURRENCY_ when used in your settings.py

AUTO_CREATE_TRIGGERS

New in version 2.3.

Default: True

If true automatically create triggers. To manually create triggers set CONCURRENCY_AUTO_CREATE_TRIGGERS=False and use triggers management command management command or create them manually using your DB client.

Note:: This flag deprecates MANUAL_TRIGGERS

ENABLED

New in version 0.10.

Default: True

enable/disable concurrency

CALLBACK

Changed in version 0.7.

Default: concurrency.views.callback

Handler invoked when a conflict is raised. The default implementation simply raise RecordModifiedError

Can be used to display the two version of the record and let the user to force the update or merge the values.

FIELD_SIGNER

New in version 0.5.

Default: concurrency.forms.VersionFieldSigner

Class used to sign the version numbers.

See also

Signining

HANDLER409

New in version 0.6.

Default: concurrency.views.conflict

Handler to intercept RecordModifiedError into ConcurrencyMiddleware. The default implementation (concurrency.views.conflict) renders 409.html while passing into the context the object that is going to be saved (target)

See also

ConcurrencyMiddleware

IGNORE_DEFAULT

New in version 1.2.

Changed in version 1.5.

Default: True

See also

VERSION_FIELD_REQUIRED

VERSION_FIELD_REQUIRED

New in version 1.5.

Default: True

Determines whether version number is mandatory in any save operation. Setting this flag to False can cause omitted version numbers to pass concurrency checks.

MANUAL_TRIGGERS

New in version 1.0.

Deprecated since version 2.3.

Default: False

If false do not automatically create triggers, you can create them using triggers management command

management command or manually using your DB client.

POLICY

Changed in version 0.7.

Default: CONCURRENCY_LIST_EDITABLE_POLICY_SILENT

CONCURRENCY_LIST_EDITABLE_POLICY_SILENT

Used by admin’s integrations to handle list_editable conflicts. Do not save conflicting records, continue and save all non-conflicting records, show a message to the user

CONCURRENCY_LIST_EDITABLE_POLICY_ABORT_ALL

Used by admin’s integations to handle list_editable. Stop at the first conflict and raise RecordModifiedError. Note that if you want to use ConcurrencyMiddleware based conflict management you must set this flag.

See also

Handle list_editable, ConcurrencyMiddleware

TRIGGERS_FACTORY

New in version 2.3.

Default:

{'postgresql': "concurrency.triggers.PostgreSQL",
 'mysql': "concurrency.triggers.MySQL",
 'sqlite3': "concurrency.triggers.Sqlite3",
 'sqlite': "concurrency.triggers.Sqlite3",
 }

dict to customise TriggerFactory. Use this to customise the SQL clause to create triggers.

Cookbook

Unable to import data ?

Sometimes you need to temporary disable concurrency (ie during data imports)

Temporary disable per Model

from concurrency.api import disable_concurrency

with disable_concurrency(instance):
    Model.object

Add version management to new models

models.py

from concurrency.fields import IntegerVersionField

class ConcurrentModel( models.Model ):
    version = IntegerVersionField( )

tests.py

a = ConcurrentModel.objects.get(pk=1)
b = ConcurrentModel.objects.get(pk=1)
a.save()
b.save() # this will raise ``RecordModifiedError``

Add version management to Django and/or plugged in applications models

Changed in version 0.8.

Concurrency can work even with existing models, anyway if you are adding concurrency management to an existing database remember to edit the database’s tables:

your_app.models.py

from django.contrib.auth import User
from concurrency.api import apply_concurrency_check

apply_concurrency_check(User, 'version', IntegerVersionField)

If used with Django>=1.7 remember to create a custom migration.

Test Utilities

ConcurrencyTestMixin offer a very simple test function for your existing models

from concurrency.utils import ConcurrencyTestMixin
from myproject.models import MyModel

class MyModelTest(ConcurrencyTestMixin, TestCase):
    concurrency_model = TestModel0
    concurrency_kwargs = {'username': 'test'}

Recover deleted record with django-reversion

Recovering deleted records with diango-reversion produces a RecordModifiedError, because both pk and version are present in the object, and django-concurrency tries to load the record (that does not exist), which raises RecordModifiedError then.

To avoid this simply disable concurrency, by using a mixin:

class ConcurrencyVersionAdmin(reversion.admin.VersionAdmin):

    @disable_concurrency()
    def revision_view(self, request, object_id, version_id, extra_context=None):
        return super().revision_view(request, object_id, version_id, extra_context=None)

    @disable_concurrency()
    def recover_view(self, request, version_id, extra_context=None):
        return super().recover_view(request, version_id, extra_context)

Changelog

This section lists the biggest changes done on each release.

Release 2.4

  • add support Django 4

  • add support Python 3.10

Release 2.3

  • Removes code producing DeprecationError

  • add AUTO_CREATE_TRIGGERS and deprecate MANUAL_TRIGGERS

  • add support for Postgres 13

  • add ability to customise SQL to create triggers TRIGGERS_FACTORY

Release 2.2

  • drop suppot django<3.0

  • drop suppot Python<3.6

Release 2.1.1

  • fixes packaging

Release 2.1 ( not released on pypi)

  • drop support Python < 3.5

  • add support Django 2.2 / 3.0

  • drop support Django < 1.11

Release 2.0

  • drop official support to Django < 1.10

  • add support Django 2.1

  • removed deprecated api concurrency_check

  • BACKWARD INCOMPATIBLE: version field is now mandatory in any save operation. Use VERSION_FIELD_REQUIRED=False to have the old behaviour.

  • disable_concurrency now has start(), finish() to be called as command

Release 1.4 (13 Sep 2016)

  • closes issue #81. Add docs and check.

  • fixes issue #80. (thanks Naddiseo for the useful support)

  • Django 1.11 compatibility

  • some minor support for Django 2.0

Release 1.3.2 (10 Sep 2016)

  • fixes bug in ConditionalVersionField that produced ‘maximum recursion error’ when a model had a ManyToManyField with a field to same model (self-relation)

Release 1.3.1 (15 Jul 2016)

  • just packaging

Release 1.3 (15 Jul 2016)

  • drop support for Python < 3.3

  • add support Django>=1.10

  • change license

  • fixes issue #36. (thanks claytondaley)

  • new IGNORE_DEFAULT to ignore default version number

Release 1.2 (05 Apr 2016)

  • better support for django 1.9 ( TemplateDoesNotExist is now in django.template.exceptions

  • improved eror message in ConcurrencyListEditableMixin issue #63 issue #64

  • fixes issue #61. Error in ConditionalVersionField (thanks ticosax)

  • fixes skipif test in pypy

Release 1.1 (13 Feb 2016)

  • drop support for django<1.7

  • add support for pypy

  • new concurrency.fields.ConditionalVersionField

  • new decorator concurrency.api.concurrency_disable_increment

  • concurrency.api.disable_concurrency is now a noop if applied to a model not under concurrency management

Release 1.0.1

  • fixes issue #56 “Can’t upgrade django-concurrency to 1.0” (thanks oppianmatt).

Release 1.0

  • BACKWARD INCOMPATIBLE:: dropped support for Django prior 1.6

  • code clean

  • fixes issue #54 “Incorrect default for IntegerVersionField” (thanks vmspike).

  • fixes issue #53. updates Documentation

  • disable_concurrency() can now disable concurrency in any model

  • disable_concurrency() is now also a decorator

  • BACKWARD INCOMPATIBLE:: removed custom backends. TriggerVerionField can be used with standard Django

  • new way to create triggers (thanks Naddiseo)

  • new trigger code

  • new TriggerVersionField.check.

  • new TriggerVersionField.trigger_name.

  • new CONCURRECY_ENABLED to fully disable concurrency

  • new CONCURRECY_MANUAL_TRIGGERS to disable triggers auto creation fixes issue #41 (thanks Naddiseo)

Release 0.9

  • Django 1.8 compatibility

  • python 3.4 compatibility

  • BACKWARD INCOMPATIBLE disable_concurrency() works differently if used with classes or instances

  • better support for external Models (models that are part of plugged-in applications)

  • fixes issue with TriggerVersionField and Proxy Models (thanx Richard Eames)

Release 0.8.1

  • avoid to use concurrency when selecting all items (select_across)

Release 0.8

  • Django 1.7 compatibility

  • fixes typo in delete_selected_confirmation.html template

  • python 3.2/3.3 compatibility

Release 0.7.1

  • backward compatibility updates. Do not check for concurrency if 0 is passed as version value

    (ie. no value provided by the form)

Release 0.7

  • new concurrency.fields.TriggerVersionField

  • start using pytest

  • moved tests outside main package

  • new protocol see:ref:protocols

  • it’s now possible disable concurrency in Models that extends concurrency enabled models

  • fixed issue #23 (thanks matklad)

  • new USE_SELECT_FOR_UPDATE

Release 0.6.0

Release 0.5.0

Release 0.4.0

  • start deprecation of concurrency.core.VersionChangedError, concurrency.core.RecordModifiedError,

    concurrency.core.InconsistencyError,moved in concurrency.exceptions

  • start deprecation of concurrency.core.apply_concurrency_check, concurrency.core.concurrency_check moved in concurrency.api

  • added CONCURRECY_SANITY_CHECK settings entry

  • signing of version number to avoid tampering (ConcurrentForm)

  • added ConcurrencyTestMixin to help test on concurrency managed models

  • changed way to add concurrency to existing models (apply_concurrency_check())

  • fixed issue #4 (thanks FrankBie)

  • removed RandomVersionField

  • new concurrency_check

  • added ConcurrentForm to mitigate some concurrency conflict

  • select_for_update now executed with nowait=True

  • removed some internal methods, to avoid unlikely but possible name clashes

Release 0.3.2

FAQ

I use Django-Rest-Framework and django-concurrency seems do not work

Use CONCURRENCY_IGNORE_DEFAULT accordingly or be sure that serializer does not set 0 as initial value

Just added django-concurrency to existing project and it does not work

Check that your records do not have 0 as version number and use CONCURRENCY_IGNORE_DEFAULT accordingly

South support ?

South support has been removed after version 1.0 when Django <1.6 support has been removed as well.

If needed add these lines to your models.py:

from south.modelsinspector import add_introspection_rules
add_introspection_rules([], ["^concurrency\.fields\.IntegerVersionField"])

How is managed update_fields

It is possible to use save(update_fields=…) parameter without interfree with the concurrency check algorithm

x1 = MyModel.objects.create(name='abc')
x2 = MyModel.objects.get(pk=x1.pk)

x1.save()
x2.save(update_fields=['username'])  # raise RecordModifiedError

anyway this will NOT raise any error

x1 = MyModel.objects.create(name='abc')
x2 = MyModel.objects.get(pk=x1.pk)

x1.save(update_fields=['username'])  # skip update version number
x2.save()  # saved

Note

TriggerVersionField will be always updated