Skip to content

Commit 46c7541

Browse files
committed
fix(settings): make DATA_UPLOAD_MAX_MEMORY_SIZE env-configurable
Previously hardcoded at 100 MB in base.py. In practice ADC workers post ML result payloads for a full batch (detection coordinates + classifications for tens of images) in a single POST, and those have been observed in the 139–321 MB range on staging for the global_moths_2024 pipeline — well above the 100 MB ceiling. Raising the limit in code would be both premature (proper fix is worker-side incremental posting, tracked in #1223) and environment- specific (staging may need to tolerate today's payloads; production may want a tighter ceiling to catch regressions). Making it an env override lets each deployment tune without a code change and without maintaining a hot-patch on the server. Reads from ``DJANGO_DATA_UPLOAD_MAX_MEMORY_MB`` (integer, in MB). Default stays at 100 MB so existing deployments see no change unless they opt in. Nginx's ``client_max_body_size`` still needs to be raised in lockstep on the fronting proxy — that is independently configurable and lives outside this repo.
1 parent 8f8b177 commit 46c7541

1 file changed

Lines changed: 9 additions & 2 deletions

File tree

config/settings/base.py

Lines changed: 9 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -426,8 +426,15 @@ def _celery_result_backend_url(redis_url):
426426
CELERY_BROKER_CONNECTION_MAX_RETRIES = None # Retry forever
427427

428428

429-
# Allow large request bodies from ML workers posting classification results
430-
DATA_UPLOAD_MAX_MEMORY_SIZE = 100 * 1024 * 1024 # 100MB (default 2.5MB)
429+
# Allow large request bodies from ML workers posting classification results.
430+
# ML detection+classification payloads for a single batch can easily exceed
431+
# the Django default (2.5 MB) and even the previous hardcoded 100 MB ceiling.
432+
# Configurable via env (MB) so staging and production can tune without a
433+
# code change. See RolnickLab/antenna#1223 for the longer-term fix (worker-
434+
# side incremental result posting).
435+
DATA_UPLOAD_MAX_MEMORY_SIZE = (
436+
env.int("DJANGO_DATA_UPLOAD_MAX_MEMORY_MB", default=100) * 1024 * 1024 # type: ignore[no-untyped-call]
437+
)
431438

432439
# django-rest-framework
433440
# -------------------------------------------------------------------------------

0 commit comments

Comments
 (0)