runtime caches & perf considerations

[jerch]

While the resolver already holds most dependency graph aspects in flat data structures for fast access, we still have a few methods, that have to calculate their final values based on the given update_fields argument at runtime:

• get_model_updates (already cached by a previous PR)
• get_local_mro
• get_select_related
• get_prefetch_related
• get_querysize

Every update_dependent call has to touch these methods once, so lets check their overhead:

from time import time
from exampleapp.models import SelfRef
from computedfields.models import active_resolver

def test():
    start = time()
    uf = set(['name', 'xy'])
    # simulate 1M calls
    for _ in range(1000000):
        active_resolver.get_model_updates(SelfRef, uf)
        fs = active_resolver.get_local_mro(SelfRef, uf)
        active_resolver.get_select_related(SelfRef, fs)
        active_resolver.get_prefetch_related(SelfRef, fs)
        active_resolver.get_querysize(SelfRef, fs)
    end = time()
    return end - start

test()

~15s to get those values 1M times. 😱

Since the final values of those methods dont change anymore for a given update_fields argument, we should use runtime caches:

~1.7s now, much better. 😺

[jerch]

There is another not so nice overhead when calling update_dependent in a loop with a single model instance. While this is clearly an anti-pattern, ppl might use this to get cfs back in sync:

from time import time
from django.db.transaction import atomic
from computedfields.models import update_dependent, not_computed
from exampleapp.models import SelfRef

def test():
    start = time()
    uf = set(['name', 'xy'])
    with atomic(): # and not_computed(recover=True):
        for obj in SelfRef.objects.all():
          update_dependent(obj, update_fields=uf)
          # vs.
          # obj.save(update_fields=uf)
    end = time()
    return end - start

test()

For 100k SelfRef instances it takes:

• baseline obj.save ~36s (no matter if changed or not)
• update_dependent (read-only) ~48s, overhead 12s
• update_dependent (read-write) ~68s, overhead 32s
• with not_computed ~3.5s (no matter if changed or not)

Good to know, that not_computed is able to remove the toxic runtime effect. But whats wrong with update_dependent?

Taking the 36s of save as baseline, I expect not to get any better than that for looped update_dependent calls doing the work on a single instance.

Doing some profiling the culprit is quickly found:

django-computedfields/computedfields/resolver.py

Lines 503 to 505 in 1259b59

	queryset = instance if isinstance(instance, QuerySet) \
	else _model._base_manager.filter(pk__in=[instance.pk])
	self.bulk_updater(queryset, _update_fields, local_only=True, querysize=querysize)

So the differences to direct save are:

• re-query database to get a queryset (bulk_updater only supports querysets)
• re-query uses a suboptimal __in filter (should be pk= filter)
• bulk_updater does alot of queryset stuff before updating things (prolly not needed for a single instance)
• bulk_updater uses either fast_update or bulk_update for writes, both are more costly for single instances wrong: fast_update is actually much faster (~20s), bulk_update is really worse (~220s)

Solution:
Provide a method single_updater as a special case for a single model instance. This will effectively deactivate any select_related or prefetch_related optimizations (penalizes related values), but we can directly use instance-local values at no costs. To preserve queryset behavior with related optimizations the user can always feed a queryset like ModelXY.objects.filter(pk=instance.pk) to update_dependent.

Results are overwhelming (CFs on SelfRef pull only instance-local values) with fast_update:

• update_dependent (read-only) ~2s
• update_dependent (read-write) ~20s

but much worse with bulk_update:

• update_dependent (read-only) ~2s
• update_dependent (read-write) ~220s

So we still need to find a workaround, if fast_update is not activated.

[jerch]

Eww, bulk_update is really really bad. Seriously, why is it even in django core? I knew it is bad, thats the reason why I created fast_update. BUT - for computed fields it is always worse than using update directly:

# bad:
#queryset.model._base_manager.bulk_update(objs, fields)
# replaced by this:
manager = queryset.model._base_manager
fields = list(fields)
get = operator.attrgetter(*fields)
if len(fields) == 1:
    for o in objs:
        manager.filter(pk=o.pk).update(**{fields[0]: get(o)})
else:
    for o in objs:
        manager.filter(pk=o.pk).update(**dict(zip(fields, get(o))))

This is several times faster under atomic() than bulk_update with sqlite and postgres. With ALL models/field constellations I have tested.

Next steps:

• test for proper MT field handling
• discourage bulk_update in favor of fast_update (8-15x faster)
• for anyone not willing to use fast_update, create an update shim like above

Edit:
With the shim above the update_dependent loop finishes in ~31s (read-write). Better than save (36s), worse than fast_update (20s).

[jerch]

Ok the bad perf of bulk_update needs more attention. We prolly have to do just the next beta release with a totally revamped update handling. Idea:

• COMPUTEDFIELDS_UPDATE_BACKEND This will be a new settings.py variable, that MUST be configured to highlight the differences of the backends:
  • 'FAST'
    • uses fast_update
    • recommended as fastest backend
    • no support for duplicates or f-expressions
    • recommended batchsize: 10000+
  • 'UPDATE'
    • uses internal update implementation with value merging
    • second fastest backend
    • no supportfor duplicates (f-expression and MT support still unclear / in development)
    • recommended batchsize: 1000 - 10000
  • 'SAVE'
    • most vanilla backend
    • uses django's save method with updated_fields applied with the help of not_computed context
    • duplicates and f-expressions supported
    • recommended if instance signals are important
    • third fastest one
    • recommended batchsize: 1000 - 10000
  • 'BULK'
    • uses bulk_update
    • not recommended anymore (slowest backend)
    • supports duplicates (depending on DBMS) and f-expressions
    • recommended batchsize: 100 - 1000
• COMPUTEDFIELDS_BATCHSIZE Replaces other batchsize settings now denoting the batch size for whatever backend is configured. Default should be some safe default (1000?).
• remove COMPUTEDFIELDS_FASTUPDATE as it is now implicit from the backend setting