[Bug]: OR-to-IN optimizer rewrite with mixed-type JSON predicates causes assertion failure and incorrect results

[Nqn7m11]

Is there an existing issue for this?

• I have searched the existing issues

Environment

- Milvus version: v2.6.20 (also reproduced on v2.6.19)
- Deployment mode(standalone or cluster): standalone
- MQ type(rocksmq, pulsar or kafka): rocksmq
- SDK version(e.g. pymilvus v2.0.0rc2): pymilvus latest
- OS(Ubuntu or CentOS): macOS Darwin 25.3.0 (Apple Silicon) / Docker
- CPU/Memory: Apple M-series / 16GB
- GPU: N/A
- Others: Docker Compose deployment, tested with FLAT and HNSW index

Current Behavior

The query optimizer merges multiple field == value branches of an or expression on the same JSON field into an internal in list for efficiency. When the branch values have mixed types, the merged list inherits inconsistent types, causing either a segcore assertion crash (code=2000) or silent data loss (wrong results, no error).

Each individual or branch is a valid, single-type expression that works correctly in isolation. The bug is introduced solely by the optimizer's branch-merging logic — hand-written in [1.0, 2, 3, 4] returns correct results; the optimizer-merged equivalent does not.

Three sub-patterns share this root cause:

Sub-pattern	Type combination	Merge threshold	Behavior
A. float + int	1.0 mixed with 2, 3, ...	4 branches	Crash (even count) or silent data loss (odd count)
B. str/bool/... + int	"1" or true mixed with 2, 3	3 branches	Always crash
C. float + int (large)	1 float + N ints, N >= 4	5+ branches	Silent data loss scaling to 99%+

---

Sub-pattern A: float + int $or (4+ branches) — crash and silent data loss

Crash (4 branches, float first):

col.query(expr='(meta["p"] == 1.0) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4)')
# -> MilvusException code=2000, segcore assertion failure at Utils.h:228

Silent data loss (5 branches, float first):

col.query(expr='(meta["p"] == 1.0) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4) or (meta["p"] == 5)')
# Expected: 5 documents (p=1,2,3,4,5)
# Actual:   3 documents (p=1,2,5) -- p=3,4 silently dropped!

Position dependence — (N - pos) parity rule:

Whether the bug manifests as crash or data loss depends on the float value's position, not just whether it's first:

Float position	4br	5br	6br	7br	8br
pos=0	Crash	drop 2	Crash	drop 4	Crash
pos=1	OK	Crash	drop 3	Crash	drop 5
pos=2	Crash	drop 2	Crash	drop 4	Crash
pos=3	OK(0!)	Crash	drop 3	Crash	drop 5
pos=last	OK	OK	OK	OK	OK

Rule: (N - pos) even -> crash or data loss; (N - pos) odd -> OK; float at last position -> always safe.

Multiple float values — arrangement matrix (4 branches):

Arrangement	Result
IIII (0 float)	4 results (correct)
FIII (1 float)	Crash
FFII (2 float)	3 results (drops 1)
FFFI (3 float)	Crash
FFFF (4 float)	4 results (correct)
FIFI	Crash
IFIF	0 results (silent total loss!)
IIFF	Crash
FIIF	Crash
IFFI	0 results (silent total loss!)

Rule: odd number of floats -> crash; even number of floats -> data loss (amount depends on arrangement). Some arrangements (IFIF, IFFI) silently return 0 results — complete data loss with no error.

The dropped values are NOT simply "all ints" — float values can also be dropped:

FFII: survives p=1(F), p=3(I), p=4(I); lost p=2(F) -- the 2nd FLOAT is dropped
FFFII (5br): survives p=1(F), p=4(I), p=5(I); lost p=2(F), p=3(F)
FFIIII (6br): survives p=1(F), p=3(I), p=6(I); lost p=2(F), p=4(I), p=5(I)

This is an index offset error in the merged in list, not a simple type filter. Values at certain positions are silently skipped regardless of their type.

---

Sub-pattern B: str/bool/... + int $or (3+ branches) — crash

All non-numeric cross-type combinations crash at only 3 branches (lower threshold than float+int):

# str + int -- CRASH
col.query(expr='(meta["p"] == "1") or (meta["p"] == 2) or (meta["p"] == 3)')

# bool + int -- CRASH
col.query(expr='(meta["p"] == true) or (meta["p"] == 2) or (meta["p"] == 3)')

# str + float -- CRASH
col.query(expr='(meta["p"] == "1") or (meta["p"] == 2.0) or (meta["p"] == 3.0)')

# bool + float -- CRASH
col.query(expr='(meta["p"] == true) or (meta["p"] == 2.0) or (meta["p"] == 3.0)')

# bool + str -- CRASH
col.query(expr='(meta["p"] == true) or (meta["p"] == "2") or (meta["p"] == "3")')

2 branches of any combination work fine (optimizer does not merge <= 2 branches).

Complete crash threshold matrix:

Type combination	2 branches	3 branches	4+ branches
str + int	OK	Crash	Crash
bool + int	OK	Crash	Crash
str + float	OK	Crash	Crash
bool + float	OK	Crash	Crash
bool + str	OK	Crash	Crash
float + int	OK	OK	Crash (4br)
Same type	OK	OK	OK

---

Sub-pattern C: float + int $or (large branch count) — silent data loss at scale

With 1 float value and N-1 int values (odd total to avoid crash), the data loss scales with query size:

# Branches	Pure int	1 float + rest int	All float
3	3	3 (below threshold)	3
5	5	1 result (80% loss)	5
7	7	1 result (86% loss)	7
10	10	1 result (90% loss)	10
20	20	1 result (95% loss)	20
100	100	1 result (99% loss)	100
500	500	1 result (99.8% loss)	500

Contrast with direct $in: meta["p"] in [1.0, 2, 3, 4, 5] correctly returns 5 documents. The same values via $or return only 1. This confirms the optimizer's merge is the sole source of the bug.

Expected Behavior

All or expressions should either:

1. Return correct results (since each branch is individually valid), or
2. Return a clear parse error (code=1100) at the planning stage if mixed types are not supported

It should never reach the segcore C++ execution layer and trigger an assertion failure, and should never silently return incomplete results.

Steps To Reproduce

from pymilvus import (
    Collection, CollectionSchema, DataType, FieldSchema,
    connections, utility,
)
import time

connections.connect("default", host="localhost", port="19530")

# =============================================
# Setup
# =============================================

if utility.has_collection("repro_or_optimizer"):
    Collection("repro_or_optimizer").drop()

fields = [
    FieldSchema("id", DataType.VARCHAR, is_primary=True, max_length=64),
    FieldSchema("vec", DataType.FLOAT_VECTOR, dim=8),
    FieldSchema("meta", DataType.JSON),
]
col = Collection("repro_or_optimizer", CollectionSchema(fields))
col.create_index("vec", {"index_type": "FLAT", "metric_type": "L2", "params": {}})

col.insert([
    [f"d{i}" for i in range(20)],
    [[(i * 7 + j) % 100 / 100.0 for j in range(8)] for i in range(20)],
    [{"p": i + 1} for i in range(20)],
])
col.flush()
col.load()
time.sleep(1)

# =============================================
# Control: pure int $or -- correct
# =============================================

r = col.query(expr='(meta["p"] == 1) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4)')
print("Pure int 4br:", len(r), "results")
# -> 4 results (correct)

# =============================================
# Sub-pattern A: float+int 4br -- CRASH
# =============================================

try:
    col.query(expr='(meta["p"] == 1.0) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4)')
except Exception as e:
    print(f"float+int 4br: CRASH! {str(e)[:100]}")
    # -> segcore assertion failure at Utils.h:228

# =============================================
# Sub-pattern A: float+int 5br -- SILENT DATA LOSS
# =============================================

r = col.query(expr='(meta["p"] == 1.0) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4) or (meta["p"] == 5)')
print("float+int 5br:", len(r), "results (expected 5)")
# -> 3 results -- p=3,4 silently dropped!

# =============================================
# Sub-pattern B: str+int 3br -- CRASH
# =============================================

try:
    col.query(expr='(meta["p"] == "1") or (meta["p"] == 2) or (meta["p"] == 3)')
except Exception as e:
    print(f"str+int 3br: CRASH! {str(e)[:100]}")

# =============================================
# Sub-pattern B: bool+int 3br -- CRASH
# =============================================

try:
    col.query(expr='(meta["p"] == true) or (meta["p"] == 2) or (meta["p"] == 3)')
except Exception as e:
    print(f"bool+int 3br: CRASH! {str(e)[:100]}")

# =============================================
# Sub-pattern C: 1 float + 19 ints -- 95% data loss
# =============================================

parts = ['(meta["p"] == 1.0)'] + [f'(meta["p"] == {i+1})' for i in range(1, 20)]
r = col.query(expr=" or ".join(parts))
print("1 float + 19 ints:", len(r), "results (expected 20)")
# -> 1 result -- 19 documents silently missing!

# =============================================
# Contrast: direct $in handles mixed int/float correctly
# =============================================

r = col.query(expr='meta["p"] in [1.0, 2, 3, 4, 5]')
print("Direct $in [1.0, 2, 3, 4, 5]:", len(r), "results")
# -> 5 results (correct!)

# =============================================
# Empty collection immunity
# =============================================

if utility.has_collection("repro_or_empty"):
    Collection("repro_or_empty").drop()

col2 = Collection("repro_or_empty", CollectionSchema(fields))
col2.create_index("vec", {"index_type": "FLAT", "metric_type": "L2", "params": {}})
col2.load()
time.sleep(1)

r = col2.query(expr='(meta["p"] == 1.0) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4)')
print("Empty collection 4br:", len(r), "results (no crash)")
# -> 0 results, no crash -- bug only triggers when data exists

col2.insert([["d0"], [[0.1]*8], [{"p": 1}]])
col2.flush()
col2.load()
time.sleep(1)

try:
    col2.query(expr='(meta["p"] == 1.0) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4)')
except Exception as e:
    print(f"1-doc collection: CRASH! {str(e)[:80]}")
    # -> CRASH -- inserting just 1 doc triggers it

col.drop()
col2.drop()


**Verified stable across:**

- 10/10 repeated trials per test case (100% reproducible)
- Data sizes: 1, 5, 50, 500 documents
- Index types: FLAT, HNSW, IVF_FLAT
- APIs: both `query()` and `search()`
- After `release()` + `load()` cycle
- Milvus v2.6.19 and v2.6.20

Milvus Log

float + int crash:

Assert "(value_proto.val_case() == milvus::proto::plan::GenericValue::kInt64Val)"
at ../../../internal/core/src/exec/expression/Utils.h:228

str + int crash:

Assert "(value_proto.val_case() == milvus::proto::plan::GenericValue::kInt64Val)"
at ../../../internal/core/src/exec/expression/Utils.h:228

bool + int crash:

Assert "(value_proto.val_case() == milvus::proto::plan::GenericValue::kBoolVal)"
at ../../../internal/core/src/exec/expression/Utils.h:224

Anything else?

Root cause analysis:

The query optimizer detects that all branches of an or expression are equality conditions on the same JSON field, and merges them into an equivalent internal in list:

(p == 1.0) or (p == 2) or (p == 3) or (p == 4)
       -> optimizer merges to ->
p in [1.0, 2, 3, 4]   (internal representation)

This merge does NOT validate or normalize type consistency:

1. For float+int: int and float are partially compatible in Milvus (so the merge passes at 3 branches), but at 4+ branches the merged in list reaches segcore with inconsistent types. The C++ executor determines the expected type from the first element and either asserts (crash) or silently skips mismatched values (data loss).

2. For str+int, bool+int, etc.: these types are fully incompatible, so the merged in list crashes at the segcore assertion with just 3 branches.

3. The silent data loss is an index offset error in the merged list, not a simple type filter. Values at certain positions are skipped regardless of whether they are int or float, as demonstrated by the arrangement matrix (e.g., FFII drops the 2nd float, not an int).

4. Empty collection immunity: All crash/data-loss expressions succeed on empty collections. Inserting just 1 document triggers the crash. This confirms the bug is in segcore's C++ data scanning layer, not in the Go expression planning layer.

Impact:

• Crash (Sub-patterns A & B): Any application that builds or expressions dynamically with user-controlled values can be crashed (DoS vector). A malicious user only needs 3 values of different types.
• Silent data loss (Sub-patterns A & C): Applications mixing score == 1.0 with score == 2 in dynamic filters will get silently incomplete results that scale with query complexity — up to 99%+ of matching documents missing with no error or warning.
• The $in vs $or inconsistency is the key diagnostic: meta["p"] in [1.0, 2, 3, 4, 5] correctly returns 5 documents, but (p==1.0) or (p==2) or (p==3) or (p==4) or (p==5) returns only 1. These should be logically equivalent.

Workaround:

# Option 1: use consistent types in all $or branches
'(meta["p"] == 1) or (meta["p"] == 2) or (meta["p"] == 3) or (meta["p"] == 4)'

# Option 2: use direct $in (correctly handles mixed int/float)
'meta["p"] in [1, 2, 3, 4]'

Suggested fix:

1. Optimizer (Go): When merging or equality branches into an in list, either:

   • Validate type consistency and skip the merge if types differ (keeping original or structure), or
   • Normalize compatible numeric types (promote all to float or all to int) during merge
   • Reject incompatible types (str+int, bool+int) with a clear error (code=1100) at the planning stage

2. Segcore fallback (C++): Replace assert with graceful error return at Utils.h:224,228,243,277 so type mismatches produce a query error instead of crashing the QueryNode.

Versions verified:

Version	Crash	Silent data loss
v2.6.19	Reproduced	Reproduced
v2.6.20 (latest)	Reproduced	Reproduced