Environment
- Milvus version: current master (
e8c233b69d9f0a177baf5aa239329f8d523b29db)
- Component: Proxy search highlighter pipeline
- Highlighter: SemanticHighlighter
Current Behavior
When SemanticHighlighter highlights a schema VARCHAR field that contains nullable rows, the proxy passes the compact string payload directly to SemanticHighlight.Process.
For a result set with three logical hits and ValidData = [true, false, true], the string payload may contain only the two non-null values:
StringData.Data = []string{"text 1", "text 3"}The semantic highlighter currently receives:
[]string{"text 1", "text 3"}This loses the null row and shifts later text values to the wrong result row, so highlight output can be attached to the wrong hit or have the wrong cardinality.
Expected Behavior
Semantic highlighter input should be aligned to logical search result rows. Nullable rows should be represented by an empty string placeholder, matching the existing lexical highlighter row-alignment behavior.
For the same result set, SemanticHighlight.Process should receive:
[]string{"text 1", "", "text 3"}Reproduction Evidence
A focused Go test against the old code fails before the fix:
FAIL: TestSearchPipeline/TestSemanticHighlightOpNullableStringAlignsRows
expected: []string{"text 1", "", "text 3"}
actual : []string{"text 1", "text 3"}
Command used:
GOWORK=off bash -lc 'source scripts/setenv.sh && go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/proxy -run "TestSearchPipeline/TestSemanticHighlightOpNullableStringAlignsRows"'
Suspected Cause
semanticHighlightOperator.run uses fieldDatas.GetScalars().GetStringData().GetData() for schema fields, while the lexical highlighter path already uses rowAlignedStringFieldData(...) to expand nullable compact payloads by ValidData.
Impact
High correctness issue: valid semantic highlight results can be associated with the wrong search hit whenever a nullable highlighted VARCHAR field has null rows before non-null rows in the returned result set.
Environment
e8c233b69d9f0a177baf5aa239329f8d523b29db)Current Behavior
When
SemanticHighlighterhighlights a schema VARCHAR field that contains nullable rows, the proxy passes the compact string payload directly toSemanticHighlight.Process.For a result set with three logical hits and
ValidData = [true, false, true], the string payload may contain only the two non-null values:The semantic highlighter currently receives:
This loses the null row and shifts later text values to the wrong result row, so highlight output can be attached to the wrong hit or have the wrong cardinality.
Expected Behavior
Semantic highlighter input should be aligned to logical search result rows. Nullable rows should be represented by an empty string placeholder, matching the existing lexical highlighter row-alignment behavior.
For the same result set,
SemanticHighlight.Processshould receive:Reproduction Evidence
A focused Go test against the old code fails before the fix:
Command used:
GOWORK=off bash -lc 'source scripts/setenv.sh && go test -tags dynamic,test -gcflags="all=-N -l" -count=1 ./internal/proxy -run "TestSearchPipeline/TestSemanticHighlightOpNullableStringAlignsRows"'Suspected Cause
semanticHighlightOperator.runusesfieldDatas.GetScalars().GetStringData().GetData()for schema fields, while the lexical highlighter path already usesrowAlignedStringFieldData(...)to expand nullable compact payloads byValidData.Impact
High correctness issue: valid semantic highlight results can be associated with the wrong search hit whenever a nullable highlighted VARCHAR field has null rows before non-null rows in the returned result set.