feat: configurable multi-label classification for individual chunks

[mits87]

What is the proposed feature?

Add optional user-defined, multi-label classification for every chunk produced by Xberg.

This would be the chunk-level equivalent of PageClassificationConfig, but designed for potentially large domain taxonomies where every label has its own semantic definition rather than only a name.

For example:

{
  "chunking": {
    "max_characters": 1500,
    "overlap": 200
  },
  "chunk_classification": {
    "definitions": [
      {
        "label": "director_appointment",
        "description": "Apply when the text states that a person has been appointed, elected, or designated as a director."
      },
      {
        "label": "director_resignation",
        "description": "Apply when the text states that a director resigned, retired, was removed, or otherwise ceased to hold office."
      },
      {
        "label": "registered_office_change",
        "description": "Apply when the registered office or official legal address of an entity changes."
      }
    ],
    "multi_label": true,
    "llm": {
      "provider": "...",
      "model": "..."
    }
  }
}

An optional global prompt_template could receive variables such as:

• {{ chunk_text }}
• {{ heading_context }}
• {{ definitions }}
• {{ multi_label }}

The result could be attached directly to each chunk:

{
  "content": "...",
  "chunk_type": "NarrativeText",
  "labels": [
    {
      "label": "director_appointment",
      "confidence": 0.93
    },
    {
      "label": "registered_office_change",
      "confidence": 0.81
    }
  ],
  "metadata": {
    "chunk_index": 12,
    "total_chunks": 48
  }
}

Returning no labels should be valid. A chunk may match zero, one, or multiple definitions.

Scalability requirements

A naive implementation would send one request per chunk containing all configured definitions. Large documents and taxonomies of 100+ labels would cause excessive concurrency, repeated tokens, cost, latency, and provider rate-limit failures.

The processor should therefore use bounded, configurable execution. Possible strategies include:

1. Batch several numbered chunks in one LLM request and return a structured chunk-index-to-labels mapping.
2. Enforce bounded concurrency and backpressure, with retries and exponential backoff for transient rate-limit failures.
3. Optionally use local embeddings, zero-shot classification, or another lightweight local model to select the most relevant label definitions per chunk before invoking the LLM.
4. Allow a fully local ONNX or plugin-based classifier for high-volume or privacy-sensitive workloads.
5. Keep definitions in a stable prompt prefix to benefit from provider prompt caching.

A useful initial implementation could support batching and bounded concurrency, leaving local candidate selection as a later optimization.

Possible optional controls:

{
  "chunk_classification": {
    "definitions": [],
    "multi_label": true,
    "batch_size": 8,
    "max_concurrency": 4,
    "candidate_limit": 20,
    "min_confidence": 0.5,
    "include_heading_context": true
  }
}

Defaults should be conservative and avoid flooding external providers.

Expected behavior

• Classification runs only when chunking and chunk classification are configured.
• Each chunk receives zero or more labels selected according to their definitions.
• Returned labels always belong to the configured definition set.
• Results preserve a deterministic mapping to chunk_index.
• Processing uses bounded concurrency.
• Transient provider failures are retried or surfaced as processing warnings.
• Token and cost usage is included in the existing LLM usage reporting.
• The feature is exposed consistently across supported language bindings.

Why would this be a good addition?

PageClassificationConfig already provides caller-defined, single-label or multi-label classification at page level (#1019). Page-level classification is often too coarse for long or heterogeneous documents: one page may contain several independently meaningful chunks, while a relevant concept may occupy only a small part of a page.

Today, Chunk includes a heuristic chunk_type, but callers cannot classify chunks using their own domain ontology. Real document-intelligence workflows may define 100 or more concepts such as director appointment, beneficial ownership change, share transfer, loan agreement, repayment event, or identity information. A short label name is insufficient; each concept needs a definition explaining when it applies.

Chunk labels are extraction metadata, similar to embeddings, page classifications, entities, and chunk page ranges. Producing them in Xberg would let consumers:

• route chunks to specialized extraction pipelines
• filter or boost chunks during retrieval
• build metadata-rich vector indexes
• avoid independently implementing chunk iteration, concurrency, retries, and result mapping
• share one configuration across Xberg language bindings

This would make Xberg a stronger first stage for domain-specific document intelligence and RAG pipelines while handling high-volume LLM usage safely.