feat(evalhub): add RAGAS provider configuration and update images

[ruivieira]

• Introduced a new ConfigMap for the RAGAS provider, defining its metrics and benchmarks.
• Updated kustomization files to include the new RAGAS provider resources.
• Modified image references in params.env files for consistency across environments.

Summary by CodeRabbit

• New Features

  • Added RagAS evaluator provider with two benchmark suites: ragas_suite and ragas_qa.
  • Tenant-aware reconciliation: automatic provisioning and cleanup of tenant-scoped resources (service accounts, role bindings, CA config) and cross-namespace MLFlow bindings.

• Chores

  • Updated provider image references to newer registries/tags.
  • Added configuration entries and a configmap for the new provider across overlays.

• Other

  • Improved routing for job status/events API.

[coderabbitai]

Warning

Rate limit exceeded

@ruivieira has exceeded the limit for the number of commits that can be reviewed per hour. Please wait 2 minutes and 46 seconds before requesting another review.

⌛ How to resolve this issue?

After the wait time has elapsed, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

We recommend that you space out your commits to avoid hitting the rate limit.

🚦 How do rate limits work?

CodeRabbit enforces hourly rate limits for each developer per organization.

Our paid plans have higher rate limits than the trial, open-source and free plans. In all cases, we re-allow further reviews after a brief timeout.

Please see our FAQ for further information.

ℹ️ Review info

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro

Run ID: 6920cded-aa49-4069-891f-3b451c7b0ac2

📥 Commits

Reviewing files that changed from the base of the PR and between b12c0c7 and 677385d.

📒 Files selected for processing (6)

• config/base/kustomization.yaml
• config/base/params.env
• config/configmaps/evalhub/kustomization.yaml
• config/configmaps/evalhub/provider-ragas.yaml
• config/overlays/odh/params.env
• config/overlays/rhoai/params.env

📝 Walkthrough

Walkthrough

Adds a RagAS EvalHub provider ConfigMap and image variables, updates provider image references, grants namespace list/watch RBAC, and implements tenant-namespace reconciliation (create/cleanup of tenant-scoped SAs, RoleBindings, and service-CA ConfigMaps) plus related controller wiring and tests.

Changes

Cohort / File(s)	Summary
Kustomize base config/base/kustomization.yaml, config/base/params.env	Add evalhub-provider-ragas-image var to kustomization and update multiple provider image values in params.env.
EvalHub provider ConfigMaps config/configmaps/evalhub/kustomization.yaml, config/configmaps/evalhub/provider-ragas.yaml	Add new provider-ragas.yaml ConfigMap defining the RagAS provider, runtime image templating, and benchmark entries.
Overlay params config/overlays/odh/params.env, config/overlays/rhoai/params.env	Add overlay env entries for evalhub-provider-ragas-image and update lm-evaluation-harness image line in overlays.
RBAC config/rbac/role.yaml, RBAC marker updates in controllers/evalhub/evalhub_controller.go	Grant list,watch on namespaces (core API group) to support namespace event handling.
Controller: image & auth config controllers/evalhub/configmap.go	Switch EvalHub image retrieval to read from ConfigMap (error-propagating), adjust generateAuthConfigData to add /events endpoint mapping and remove duplicate mapping.
Controller: reconciliation wiring controllers/evalhub/evalhub_controller.go	Add Namespace→EvalHub mapper, wire Namespace watch into SetupWithManager, and call tenant namespace reconciliation from Reconcile; add mapper method.
Controller: tenant resources controllers/evalhub/tenant_namespaces.go	New tenant reconciliation implementation: create tenant SAs, cross-namespace RoleBindings, tenant CA-injection ConfigMaps, and cleanup of stale resources.
Controller: service accounts / RB bindings controllers/evalhub/service_accounts.go	Change createJobRoleBinding signature to accept optional subjectNamespace; create cross-namespace MLFlow RoleBinding for EvalHub service account in tenant namespaces.
Tests: controller unit & suite controllers/evalhub/suite_test.go, controllers/evalhub/unit_test.go, controllers/evalhub/suite_test.go	Add/test operator ConfigMap setup, extensive tests for tenant namespace reconciliation, SA/RoleBinding creation, service-CA ConfigMap handling, and cleanup semantics.

Sequence Diagram(s)

sequenceDiagram
  participant Namespace
  participant EvalHubController as EvalHub Controller
  participant K8sAPI as Kubernetes API
  participant ClusterRole
  participant ConfigMap

  Namespace->>EvalHubController: Namespace event (labelled tenant)
  EvalHubController->>EvalHubController: mapNamespaceToEvalHubs -> enqueue EvalHub reconcile
  EvalHubController->>K8sAPI: List EvalHub instances
  EvalHubController->>K8sAPI: create ServiceAccount (tenant ns)
  EvalHubController->>K8sAPI: create RoleBinding -> ClusterRole (cross-namespace subjectNamespace)
  EvalHubController->>K8sAPI: ensure service-CA ConfigMap with CA inject annotation
  EvalHubController->>ConfigMap: read operator image ConfigMap (get image)
  K8sAPI-->>EvalHubController: resource status / errors

Loading

Estimated code review effort

🎯 4 (Complex) | ⏱️ ~45 minutes

Possibly related PRs

• feat(operator): provision tenant namespace RBAC for multi-tenant job creation #663: Adds namespace mapping, tenant reconciliation, createJobRoleBinding changes, and RBAC namespace rule (strong overlap).
• feat(evalhub): add authorization configuration to ConfigMap #658: Modifies generateAuthConfigData and configmap-related config generation (overlaps with auth mapping and image retrieval changes).
• feat(evalhub): Add garak kfp adapter configmap #662: Adds EvalHub provider ConfigMaps and updates kustomize includes (overlaps with provider-ragas addition).

Suggested labels

feature, project/evalhub, ok-to-test, lgtm

Suggested reviewers

• tarilabs
• julpayne

Poem

🐰 A curious hop through namespaces wide,
I stitched ConfigMaps and RBAC with pride,
Tenants now find SAs and CA snug,
RagAS joins the stack with a happy hug,
Hooray — another provider at my side! 🥕

🚥 Pre-merge checks | ✅ 2 | ❌ 1

❌ Failed checks (1 warning)

Check name	Status	Explanation	Resolution
Docstring Coverage	⚠️ Warning	Docstring coverage is 75.00% which is insufficient. The required threshold is 80.00%.	Write docstrings for the functions missing them to satisfy the coverage threshold.

✅ Passed checks (2 passed)

Check name	Status	Explanation
Description Check	✅ Passed	Check skipped - CodeRabbit’s high-level summary is enabled.
Title check	✅ Passed	The pull request title accurately summarizes the main changes: adding RAGAS provider configuration and updating image references across multiple files.

_{✏️ Tip: You can configure your own custom pre-merge checks in the settings.}

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests
• Post copyable unit tests in a comment

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands and usage tips.}

[saichandrapandraju]

can we for now give quay.io/trustyai/trustyai-garak-lls-provider-dsp:latest for this?

[ruivieira]

Fixed in 677385d

[saichandrapandraju]

can we for now give quay.io/trustyai/trustyai-garak-lls-provider-dsp:latest for this?

[ruivieira]

Fixed in 677385d

[openshift-ci]

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: saichandrapandraju

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

• OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

[openshift-ci]

@ruivieira: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/trustyai-service-operator-e2e	677385d	link	true	/test trustyai-service-operator-e2e

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

[openshift-ci]

PR needs rebase.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.