feat: cache find_existing_object results#155
Merged
mfiedorowicz merged 11 commits intodevelopfrom Mar 26, 2026
Merged
Conversation
Add cross-request caching for object lookups in find_existing_object. Entities with only simple field values (no unresolved references) are cached in Django's cache backend (Redis) with a configurable TTL (DIODE_FIND_OBJ_CACHE_TTL, default 5s). Both found (stores PK) and not-found results are cached, reducing DB queries from ~20 to ~5 per generate-diff request for typical dcim.interface payloads.
Replace DIODE_FIND_OBJ_CACHE_TTL env var with plugin config setting find_obj_cache_ttl, configurable via PLUGINS_CONFIG. Defaults to 5s.
Skip lists and dicts instead of bailing out, and encode unresolved references deterministically. This allows caching for entities like Site (which have tags lists) and Device (which have unresolved ref fields), increasing cache hit rate from ~7/13 to ~10-11/13.
mfiedorowicz
requested review from
MicahParks,
jajeffries,
leoparente and
ltucker
as code owners
mfiedorowicz
changed the title
Signed-off-by: Michal Fiedorowicz <mfiedorowicz@netboxlabs.com>
Cover cache key generation (scalar fields, underscore filtering, unresolved references, dicts/lists skipping, determinism) and caching behavior (miss→populate, hit→found, hit→not-found, stale→fallthrough, disabled when TTL=0).
Add generation counter to cache keys and increment it after apply_changeset. This prevents stale not-found results from being served after objects are created. Add tests for invalidation behavior.
Only cache found objects (not not-found). Store a reverse index (object_type:PK → lookup cache key) alongside each cached entry. On update, use the PK to look up and delete the exact cache entry. Creates don't need invalidation since not-found is not cached. Move invalidation from views into the applier where creates/updates happen, using (object_type, object_id) instead of data hashes.
NetBox 4.5.1 (PR #21080) fixed the Module.clean() recursion check, which now correctly rejects circular module bay/module structures. Update test data to remove circular references and split the module bay with module test into two steps (create bay, then install module).
NetBox 4.4.x removed is_staff from the User model. The tests already use explicit permission grants via add_permissions().
Update default NetBox version from v4.5.0 to v4.5.5 in Makefile and CI matrix. Bump netbox-branching to 0.8.3 to fix queue_name migration issue. Fix CI coverage comment to run for default NetBox version using a workflow-level env constant.
Move server-specific volume mounts out of shared docker-compose.yaml into version-specific overrides: v4.5.x mounts launch-netbox.sh and granian.py, v4.4.x mounts nginx-unit.json. Update Makefile to auto- include the override file via COMPOSE_FILES variable. Fix v4.4.x Dockerfile to reference its own requirements file instead of v4.5.x.
☂️ Python Coverage
Overall Coverage
New Files
Modified Files
|
MicahParks
approved these changes
Mar 26, 2026
jajeffries
approved these changes
Mar 26, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This pull request introduces a Redis-backed caching layer for the
find_existing_objectfunction in the Diode NetBox Plugin, improving performance by avoiding repeated database lookups for object matching. The cache is configurable, can be invalidated on object updates, and comes with comprehensive tests. Additionally, it updates several version pins and configuration defaults.Caching and object matching improvements:
find_existing_object, keyed deterministically on scalar fields, with configurable TTL via the newfind_obj_cache_ttlplugin config option (default 5 seconds, 0 disables caching). Includes logic to skip caching for objects with only unresolved references or complex fields. [1] [2] [3] [4] [5]invalidate_find_obj_entry) to ensure cache consistency when objects are updated or deleted. Integrated invalidation calls into update paths in the applier logic. [1] [2] [3] [4]test_matcher_cache.py) to verify cache key generation, cache hit/miss logic, invalidation, and edge cases for the matcher cache.Configuration and dependency updates:
v4.5.5in theMakefile, GitHub Actions workflow, and related environment variables, ensuring consistency across builds and coverage reporting. [1] [2] [3] [4]netboxlabs-netbox-branchingdependency from0.8.0to0.8.3in plugin requirements.Other configuration changes:
DIODE_PROFILEsetting in the NetBox plugin environment fromtruetofalse.