cookbook: add integration_ejentum_cognitive_harness notebook#2994
cookbook: add integration_ejentum_cognitive_harness notebook#2994ejentum wants to merge 2 commits into
Conversation
Adds a cookbook notebook that traces an Ejentum cognitive harness agent loop in Langfuse. Each turn produces a Langfuse trace with two child observations: ejentum.call (manual @observe-decorated REST wrapper with mode and scaffold length on metadata) and openai.chat.completions.create (from the langfuse.openai drop-in). The pattern generalises to any third-party REST tool an agent calls in-loop; Ejentum is the worked example because the agent calls a single endpoint with a mode arg.
![claude[bot]](https://avatars.githubusercontent.com/in/1236702?s=60&v=4){kind=small}
|
Check out this pull request on See visual diffs & provide feedback on Jupyter Notebooks. Powered by ReviewNB |
|
@ejentum is attempting to deploy a commit to the langfuse Team on Vercel. A member of the Team first needs to authorize it. |
|
|
dosubot
Bot
added
size:M
![greptile-apps[bot]](https://avatars.githubusercontent.com/in/867647?s=60&v=4){kind=small}
| "\n", | ||
| "Each turn does the same shape:\n", | ||
| "\n", | ||
| "1. `call_ejentum(...)` (Langfuse observation with mode + scaffold length).\n", |
There was a problem hiding this comment.
payload[0] is accessed without guarding against an empty list. If the Ejentum endpoint returns [] (e.g. on a quota or validation error that still yields HTTP 200), this line throws IndexError at runtime and the @observe span is never closed cleanly. The isinstance(payload, list) check only protects against a non-list response, not an empty one.
Prompt To Fix With AI
This is a comment left during a code review.
Path: cookbook/integration_ejentum_cognitive_harness.ipynb
Line: 127
Comment:
**IndexError on empty API response list**
`payload[0]` is accessed without guarding against an empty list. If the Ejentum endpoint returns `[]` (e.g. on a quota or validation error that still yields HTTP 200), this line throws `IndexError` at runtime and the `@observe` span is never closed cleanly. The `isinstance(payload, list)` check only protects against a non-list response, not an empty one.
How can I resolve this? If you propose a fix, please make it concise.| "2. `openai.chat.completions.create(...)` from the auto-instrumented client.\n", | ||
| "\n", | ||
| "Both observations sit inside a single trace per turn, so you can drill from the trace summary into the harness retrieval, the system message it produced, and the model call that consumed it." | ||
| ] | ||
| }, | ||
| { | ||
| "cell_type": "code", | ||
| "execution_count": null, | ||
| "metadata": {}, | ||
| "outputs": [], |
There was a problem hiding this comment.
The notebook instructs readers to filter the Traces view by metadata.mode and metadata.scaffold_length, but langfuse_context.update_current_observation(metadata=...) inside call_ejentum attaches those keys to the child span (ejentum.call), not to the root trace created by harness_augmented_turn. The Traces view in Langfuse filters on trace-level metadata, so both suggested filter expressions will silently return no results. To surface these values at the trace level, langfuse_context.update_current_trace(metadata={"mode": mode, "scaffold_length": ...}) should be called (from within call_ejentum or harness_augmented_turn) in addition to the existing observation-level update.
Prompt To Fix With AI
This is a comment left during a code review.
Path: cookbook/integration_ejentum_cognitive_harness.ipynb
Line: 128-137
Comment:
**Trace-level metadata missing — described UI filters will not work**
The notebook instructs readers to filter the Traces view by `metadata.mode` and `metadata.scaffold_length`, but `langfuse_context.update_current_observation(metadata=...)` inside `call_ejentum` attaches those keys to the child span (`ejentum.call`), not to the root trace created by `harness_augmented_turn`. The Traces view in Langfuse filters on trace-level metadata, so both suggested filter expressions will silently return no results. To surface these values at the trace level, `langfuse_context.update_current_trace(metadata={"mode": mode, "scaffold_length": ...})` should be called (from within `call_ejentum` or `harness_augmented_turn`) in addition to the existing observation-level update.
How can I resolve this? If you propose a fix, please make it concise.2 participants
Summary
Adds a new cookbook notebook under
cookbook/integration_ejentum_cognitive_harness.ipynbthat traces an Ejentum cognitive harness agent loop in Langfuse. Each turn produces one trace with two child observations:ejentum.call: an@observe-decorated REST wrapper aroundhttps://ejentum-main-ab125c3.zuplo.app/logicv1/, withmode,query_length, andscaffold_lengthattached to the observation metadata.openai.chat.completions.create: from thelangfuse.openaidrop-in. Prompt, response, model, and token usage are captured automatically.The pattern generalises beyond Ejentum: any in-loop REST call to a third-party tool can be wrapped the same way. Ejentum is the worked example because the agent calls a single endpoint with a
modeargument, which keeps the cookbook short.File
cookbook/integration_ejentum_cognitive_harness.ipynb(new)Naming and structure match existing cookbook integration entries (
integration_agno_agents.ipynb,example_langgraph_agents.ipynb, etc.): one self-contained Jupyter notebook with install + setup + demo + "what to look at" sections.What the notebook walks through
langfuse,openai,requests).call_ejentumas an@observe(name=\"ejentum.call\")function withupdate_current_observation(metadata=...)for the mode and scaffold length.from langfuse.openai import openaidrop-in for auto-instrumented OpenAI calls.harness_reasoningthenharness_anti_deception, splicing the returned scaffold into the system message.metadata.modeandmetadata.scaffold_lengthin the Langfuse UI.Affiliation
I maintain the Ejentum harness API. Submitting this as a tracing cookbook because the four-step pattern (drop-in OpenAI client +
@observeREST wrapper + metadata on the observation + scaffold-aware system message) is generally useful for any third-party REST tool, and Ejentum is a clean worked example. The notebook explicitly says this generalises. Ejentum has free and paid tiers; the notebook links to the dashboard for keys, not to a checkout.Test plan
%pip install).Greptile Summary
Adds a new Jupyter notebook cookbook demonstrating how to trace an Ejentum cognitive-harness REST API call alongside an OpenAI completion in Langfuse, using the
@observedecorator andlangfuse.openaidrop-in.call_ejentum— an@observe-decorated wrapper that posts to the Ejentum endpoint, parses the scaffold from the JSON response, and attachesmode/query_length/scaffold_lengthas observation metadata.harness_augmented_turnfunction that nests the Ejentum call and an auto-instrumentedopenai.chat.completions.createinside a single parent trace, then flushes at the end.Confidence Score: 3/5
The notebook has two functional defects: an unguarded list index that can raise at runtime, and metadata placed on the wrong Langfuse context level so the documented UI filters don't work.
The scaffold-extraction line
payload[0].get(mode, "")raisesIndexErroron an empty list response rather than returning an empty string, which can happen on quota or validation failures. The cookbook's "Useful filters" instructions point readers tometadata.modeandmetadata.scaffold_lengthin the Traces view, but those keys are attached to the child observation viaupdate_current_observation, not to the root trace — the filter expressions will silently return nothing. Both issues affect the primary demo code and the tutorial steps a reader would follow.cookbook/integration_ejentum_cognitive_harness.ipynb — the
call_ejentumresponse-parsing block and the metadata-propagation strategy both need changes before this notebook produces the experience it describes.Sequence Diagram
sequenceDiagram participant User participant harness_augmented_turn participant call_ejentum participant EjentumAPI participant OpenAI participant Langfuse User->>harness_augmented_turn: call(task, mode) Note over harness_augmented_turn,Langfuse: @observe creates root trace harness_augmented_turn->>call_ejentum: call_ejentum(task, mode) Note over call_ejentum,Langfuse: @observe creates child span "ejentum.call" call_ejentum->>EjentumAPI: "POST /logicv1/ {query, mode}" EjentumAPI-->>call_ejentum: "[{mode: scaffold_text}]" call_ejentum->>Langfuse: update_current_observation(metadata) call_ejentum-->>harness_augmented_turn: scaffold string harness_augmented_turn->>OpenAI: chat.completions.create(gpt-4o-mini) Note over harness_augmented_turn,Langfuse: langfuse.openai auto-instruments call OpenAI-->>harness_augmented_turn: completion harness_augmented_turn-->>User: answer text User->>Langfuse: langfuse.flush()Prompt To Fix All With AI
Reviews (1): Last reviewed commit: "cookbook: add integration_ejentum_cognit..." | Re-trigger Greptile