integrations/langchain.py: EvaluatorChain._acall discards run_manager, breaks async callback propagation

[Harsh23Kashyap]

Problem

src/ragas/integrations/langchain.py:98-129 (EvaluatorChain._acall) accepts a LangChain AsyncCallbackManagerForChainRun via the run_manager parameter but discards it and passes callbacks=[] to the inner metric call:

async def _acall(
    self,
    inputs: t.Union[t.Dict[str, t.Any], SingleTurnSample],
    run_manager: t.Optional[AsyncCallbackManagerForChainRun] = None,
) -> t.Dict[str, t.Any]:
    ...
    self._validate(inputs)
    _run_manager = run_manager or AsyncCallbackManagerForChainRun.get_noop_manager()
    # TODO: currently AsyncCallbacks are not supported in ragas
    _run_manager.get_child()
    assert isinstance(self.metric, SingleTurnMetric), (
        "Metric must be SingleTurnMetric"
    )
    score = await self.metric.single_turn_ascore(
        inputs,
        callbacks=[],   # <-- empty list, run_manager's handlers discarded
    )
    return {self.metric.name: score}

The sync _call method (lines 68-96) does the right thing — it captures _run_manager.get_child() into a callbacks variable and passes it to metric.single_turn_score. The async path is the only one that drops callbacks.

The self-acknowledged TODO at line 118 ("currently AsyncCallbacks are not supported in ragas") flags the gap.

Why it matters

When a user integrates a ragas metric into a LangChain chain running under acall, the metric's internal LLM calls (e.g. faithfulness's claim-decomposition prompt, answer relevancy's question-generation prompt) are NOT tracked by the parent chain's callback handlers. A user wiring up LangSmith / LangFuse / OpenInference on the parent chain sees the chain-level events but not the ragas-internal ones. The sync invoke path already works; the async ainvoke path is silently broken.

Reproduction

import asyncio
from langchain_core.callbacks import BaseCallbackHandler
from ragas.integrations.langchain import EvaluatorChain
from ragas.metrics._faithfulness import Faithfulness
from ragas.llms import LangchainLLMWrapper
from langchain_openai import ChatOpenAI

class RecordingHandler(BaseCallbackHandler):
    def __init__(self):
        self.events = []
    def on_llm_start(self, *a, **kw):
        self.events.append("llm_start")

handler = RecordingHandler()
llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o-mini"))
metric = Faithfulness(llm=llm)
chain = EvaluatorChain(metric=metric)

async def go():
    result = await chain.ainvoke(
        {"question": "...", "answer": "...", "contexts": ["..."]},
        config={"callbacks": [handler]},
    )
    print("events:", handler.events)  # [] — ragas-internal LLM calls not recorded
asyncio.run(go())

Expected

The run_manager passed to _acall should drive callback propagation into the metric, mirroring _call. The handlers on the run_manager's child should reach metric.single_turn_ascore so the metric's internal LLM invocations are visible to the parent chain's observability stack.

Proposed solution

Mirror the sync pattern in _acall:

_run_manager = run_manager or AsyncCallbackManagerForChainRun.get_noop_manager()
callbacks = _run_manager.get_child()    # capture
...
score = await self.metric.single_turn_ascore(
    inputs,
    callbacks=callbacks,    # propagate
)

Remove the now-stale TODO comment.

Acceptance criteria

• _acall passes the run_manager's child callbacks into metric.single_turn_ascore when run_manager is provided.
• _acall continues to work (no NameError, no regression) when run_manager=None.
• New unit test asserts the metric receives the run_manager's handlers.
• All existing tests still pass.
• ruff check, ruff format --check, pyright clean on the touched file.
• Backward compatible: no public API change, no signature change.

Out of scope

• Wiring on_chain_start / on_chain_end / on_chain_error on the run_manager inside _acall (the existing sync path does not do this either; can be a follow-up).
• Broader LangChain async-callback propagation for multi-turn metrics (MultiTurnMetric has the same shape and same TODO-able gap; address in a separate PR if desired).

Environment

• ragas main (post 0.4.3, commit 298b682)
• Python 3.12
• langchain 1.3.11, langchain-core 0.3.x

[chandekar-aditi]

Hi! I worked on this issue and have submitted a PR with a proposed fix.

PR: #2892

The fix forwards the async callback manager to single_turn_ascore() and includes a regression test to verify callback propagation.
I'd appreciate any feedback or suggestions. Thanks!