system_test_assert_conditions/test/min_count flaky: Elasticsearch returns `minDoc must be >= 0` 400 error

[teresaromero]

Summary

The system_test_assert_conditions/test/min_count system test case fails intermittently with a 400 Bad Request from Elasticsearch. The failure is non-deterministic (different negative minDoc values each time) and unrelated to code changes — it has been observed on main builds and unrelated PRs.

Error

FAIL: system_test_assert_conditions/test/min_count (system)
    error: failed to search docs for data stream logs-system_test_assert_conditions.test-<id>:
    [400 Bad Request] {"error":{"root_cause":[{"type":"illegal_argument_exception",
    "reason":"minDoc must be >= 0 but got minDoc=-500"}], ...}}

Affected builds

Build	Branch	Date	minDoc value
#8591	main	2026-08-25	-423
#8597	main	2026-08-27	-261
#8603	fix/terraform-deployer-gcloud-cli	2026-08-27	-500

Root cause

getDocs in internal/testrunner/runners/system/tester.go always queries with WithSize(500) + WithSort("@timestamp:asc"). When Elasticsearch is still indexing and a shard is in a transitional state with fewer than 500 documents in a segment, the sort collector's internal lastDoc - numDocs calculation goes negative, producing the 400.

The 400 is treated as a fatal error and kills the test immediately — it is never retried — even though it is a transient condition.

A contributing factor: the test config data_stream/test/_dev/test/system/test-min_count-config.yml sets assert.min_count: 500, which is exactly elasticsearchQuerySize = 500 (tester.go:125). The test intentionally probes this boundary (a comment in the config notes this), but doing so is what pushes the search into the range where the ES behaviour triggers.

Proposed fix

Two complementary changes:

1. internal/testrunner/runners/system/tester.go — handle the transient 400 gracefully

Mirror the existing pattern for no_shard_available_action_exception (tester.go:853): detect the minDoc must be >= 0 400 and return (&hits{}, nil) instead of a fatal error, allowing the UntilTrue retry loop to wait for the shard to stabilise.

if resp.StatusCode == http.StatusBadRequest &&
    strings.Contains(resp.String(), "minDoc must be >= 0") {
    return &hits{}, nil
}

2. test-min_count-config.yml — reduce min_count away from the query size boundary

min_count: 500 == elasticsearchQuerySize is the exact boundary that triggers the issue. Reducing to e.g. 50 (still exercises the min_count assertion feature) removes the boundary condition entirely.

[mrodm]

This issue requires further investigation.

I did some tests adding the exception to retry those errors, but even with that it is not recovered.
Moreover, running the tests for one specific test package, it always fails with the same error minDoc must be >=

Commands used for testing locally:

elastic-package stack up -v -d --version 9.5.3
elastic-package -C test/packages/other/system_test_assert_conditions/ test system -v
elastic-package stack down -v