Add release-tracker-workflow pipeline for Shaman-based RC testing

release-tracker-workflow/README.rst

@@ -0,0 +1,44 @@
+Release Tracker Workflow
+========================
+
+A Jenkins pipeline that automates RC (release candidate) testing for Ceph release trackers:
+
+- Resolve or accept a Ceph SHA1 (optional **CEPH_SHA1**; must exist on Shaman).
+- Wait for the SHA1 on Shaman, then schedule teuthology suites (all triggered at once, then wait for all).
+- Aggregate pass/fail results and post to Redmine (tracker.ceph.com) and/or send email.
+
+No build step: the pipeline relies on Shaman only (SHA1 must already be built and available).
+
+Requirements
+------------
+
+- Jenkins agent with label **teuthology-agent** (or set **AGENT_LABEL**); teuthology installed, ``~/.teuthology.yaml``, Shaman/Paddles reachable.
+- Credential **redmine-api-key** (or set **REDMINE_CREDENTIAL_ID**) in Jenkins for posting to tracker.ceph.com when **SKIP_TRACKER_UPDATE** is false.
+
+Parameters
+----------
+
+- **CEPH_BRANCH** / **CEPH_SHA1**: Branch to resolve SHA1 from, or a specific SHA1 to use (must exist on Shaman).
+- **SUITE_LIST_SOURCE**: Path relative to workspace (e.g. ``release-tracker-workflow/config/suites.yaml``) or URL for suite list; empty = use **SUITE_NAME**.
+- **SKIP_TRACKER_UPDATE** (default true): Do not post to Redmine.
+- **TRACKER_ISSUE_ID**: Redmine issue ID when posting (required when SKIP_TRACKER_UPDATE is false).
+
+Configuration (no hardcodings)
+------------------------------
+
+Paths and URLs are parameterized so the same job works across environments:
+
+- **AGENT_LABEL**: Jenkins agent label (default ``teuthology-agent``).
+- **TEUTHOLOGY_SCRIPT_DIR** / **TEUTHOLOGY_VIRTUALENV_PATH** / **TEUTHOLOGY_OVERRIDE_YAML**: Teuthology install path, virtualenv, and optional override YAML for teuthology-suite.
+- **PULPITO_BASE**: Base URL for Pulpito run links (default ``https://pulpito.ceph.com``).
+- **PADDLES_URL**: Paddles base URL for aggregation.
+- **REDMINE_CREDENTIAL_ID**: Jenkins credential ID for Redmine API key.
+- **SUITE_MACHINE_TYPE**, **SUITE_LIMIT**: Teuthology suite machine type and --limit.
+- **SHAMAN_WAIT_TIMEOUT**, **SHAMAN_WAIT_INTERVAL**, **SUITE_WAIT_SLEEP**: Timeouts and sleep for Shaman wait and suite scheduling.
+
+Suite lists
+-----------
+
+- ``release-tracker-workflow/config/suites.yaml``: list of teuthology suites to run.
+
+Set **SUITE_LIST_SOURCE** to this path (or a URL) to run multiple suites; all are triggered in parallel, then the pipeline waits for all and aggregates results.

release-tracker-workflow/build/scripts/redmine_post_note.sh

@@ -0,0 +1,13 @@
+#!/usr/bin/env bash
+# Post a note to a Redmine issue. Usage: redmine_post_note.sh <issue_id> <note_file>
+# Requires: REDMINE_API_KEY in environment.
+set -euo pipefail
+ISSUE_ID="${1:?}"; NOTE_FILE="${2:?}"; REDMINE_URL="${REDMINE_URL:-https://tracker.ceph.com}"
+[[ -z "${REDMINE_API_KEY:-}" ]] && { echo "REDMINE_API_KEY not set."; exit 0; }
+[[ ! -f "$NOTE_FILE" ]] && { echo "Note file not found: $NOTE_FILE"; exit 1; }
+NOTE_JSON=$(python3 -c "import json,sys; f=open(sys.argv[1]); print(json.dumps({'issue':{'notes':f.read()}}))" "$NOTE_FILE")
+HTTP_CODE=$(curl -s -w "%{http_code}" -o /tmp/redmine_resp.json -X PUT \
+  -H "X-Redmine-API-Key: $REDMINE_API_KEY" -H "Content-Type: application/json" \
+  -d "$NOTE_JSON" "${REDMINE_URL}/issues/${ISSUE_ID}.json")
+[[ "$HTTP_CODE" != "200" && "$HTTP_CODE" != "204" ]] && { echo "HTTP $HTTP_CODE"; cat /tmp/redmine_resp.json; exit 1; }
+echo "Posted note to ${REDMINE_URL}/issues/${ISSUE_ID}"

release-tracker-workflow/config/definitions/release-tracker-workflow.yml

@@ -0,0 +1,87 @@
+# Release tracker workflow: RC testing automation
+# SHA1 -> wait Shaman -> schedule suites -> aggregate -> report to Redmine (Shaman only, no build)
+# Requires: teuthology-agent label, redmine-api-key credential
+- job:
+    name: release-tracker-workflow
+    description: |
+      Resolve or use CEPH_SHA1, wait for Shaman, schedule teuthology suites (all at once),
+      aggregate results, post to Redmine and/or email. Relies on Shaman only (no build step).
+      Use CEPH_SHA1 to run on a given SHA1 if present on Shaman.
+    project-type: pipeline
+    quiet-period: 1
+    concurrent: true
+    pipeline-scm:
+      scm:
+        - git:
+            url: https://github.com/ceph/ceph-build
+            branches:
+              - ${{CEPH_BUILD_BRANCH}}
+            shallow-clone: true
+            submodule:
+              disable: true
+            wipe-workspace: true
+      script-path: release-tracker-workflow/build/Jenkinsfile
+      lightweight-checkout: true
+      do-not-fetch-tags: true
+
+    parameters:
+      - string:
+          name: CEPH_REPO
+          description: "Ceph repository URL"
+          default: "https://github.com/ceph/ceph.git"
+      - string:
+          name: CEPH_BRANCH
+          description: "Ceph branch (e.g. main, reef, tentacle)"
+          default: main
+      - string:
+          name: CEPH_SHA1
+          description: "Optional: Ceph commit SHA1. If set, run on this SHA1 (must exist on Shaman). Empty = resolve from branch tip."
+          default: ""
+      - string:
+          name: RELEASE_VERSION
+          description: "Release version (e.g. 20.1.0)"
+          default: "20.1.0"
+      - string:
+          name: TRACKER_ISSUE_ID
+          description: "Redmine tracker issue ID. Required when posting to tracker."
+          default: ""
+      - string:
+          name: SUITE_NAME
+          description: "Single suite when SUITE_LIST_SOURCE is empty"
+          default: smoke
+      - string:
+          name: SUITE_LIST_SOURCE
+          description: "File path (e.g. release-tracker-workflow/config/suites.yaml) or URL for suite list. Empty = use SUITE_NAME."
+          default: ""
+      - string:
+          name: SUITE_REPO
+          description: "Suite repo URL for teuthology-suite (--suite-repo, e.g. https://github.com/ceph/ceph-ci.git)"
+          default: "https://github.com/ceph/ceph-ci.git"
+      - string:
+          name: PADDLES_URL
+          description: "Paddles base URL for aggregation"
+          default: "http://paddles.front.sepia.ceph.com/"
+      - string:
+          name: EMAIL_RECIPIENTS
+          description: "Comma-separated emails to notify when run finishes."
+          default: ""
+      - bool:
+          name: USE_WORKSPACE_TEUTHOLOGY
+          description: "If true, clone teuthology into workspace. If false, use TEUTHOLOGY_SCRIPT_DIR."
+          default: true
+      - string:
+          name: TEUTHOLOGY_SCRIPT_DIR
+          description: "Path to existing teuthology clone (used when USE_WORKSPACE_TEUTHOLOGY is false)"
+          default: ""
+      - string:
+          name: TEUTHOLOGY_VIRTUALENV_PATH
+          description: "Path to teuthology virtualenv (used when USE_WORKSPACE_TEUTHOLOGY is false)"
+          default: ""
+      - bool:
+          name: SKIP_TRACKER_UPDATE
+          description: "If true, do not post to Redmine."
+          default: true
+      - string:
+          name: CEPH_BUILD_BRANCH
+          description: "Use the Jenkinsfile from this ceph-build branch"
+          default: main

release-tracker-workflow/config/suites.yaml

@@ -0,0 +1,12 @@
+# Suite list for release-tracker-workflow
+suites:
+  - teuthology:nop
+  - orch
+  - rbd
+  - fs
+  - rgw
+  - krbd
+  - upgrade
+  - crimson-rados
+  - crimson-rados/basic
+  - rados

release-tracker-workflow/scripts/aggregate_suite_results.py

@@ -0,0 +1,94 @@
+#!/usr/bin/env python3
+"""Fetch job results from Paddles for one or more runs and output aggregated pass/fail table.
+
+Usage:
+  aggregate_suite_results.py --run RUN [--run RUN ...] [--paddles-url URL] [--out FILE]
+"""
+import argparse
+import sys
+try:
+    import requests
+except ImportError:
+    print("pip install requests", file=sys.stderr)
+    sys.exit(2)
+
+DEFAULT_PADDLES = "http://paddles.front.sepia.ceph.com/"
+
+
+def get_jobs(paddles_url, run_name, fields=None):
+    fields = fields or ["job_id", "status", "description"]
+    if "job_id" not in fields:
+        fields = list(fields) + ["job_id"]
+    uri = f"{paddles_url.rstrip('/')}/runs/{run_name}/jobs/?fields={','.join(fields)}"
+    try:
+        r = requests.get(uri, timeout=60)
+        r.raise_for_status()
+        return r.json()
+    except Exception as e:
+        print(f"Failed to get jobs: {e}", file=sys.stderr)
+        return None
+
+
+def suite_from_desc(desc):
+    return (desc or "unknown").split("/")[0] if "/" in (desc or "") else (desc or "unknown").split()[0] if desc else "unknown"
+
+
+def aggregate(jobs):
+    by_suite = {}
+    for j in jobs:
+        desc = j.get("description") or ""
+        status = (j.get("status") or "unknown").lower()
+        suite = suite_from_desc(desc)
+        if suite not in by_suite:
+            by_suite[suite] = {"pass": 0, "fail": 0}
+        if status == "pass":
+            by_suite[suite]["pass"] += 1
+        else:
+            by_suite[suite]["fail"] += 1
+    return {s: "PASS" if c["fail"] == 0 and c["pass"] > 0 else "FAIL" for s, c in by_suite.items()}
+
+
+def merged_table(paddles_url, run_names):
+    combined = {}
+    any_jobs = False
+    for run_name in run_names:
+        jobs = get_jobs(paddles_url, run_name)
+        if jobs is None:
+            return None
+        if not jobs:
+            continue
+        any_jobs = True
+        combined.update(aggregate(jobs))
+    if not any_jobs:
+        return "No jobs found"
+    return "\n".join(
+        ["Suite | Status", "------|------"]
+        + [f"{k} | {v}" for k, v in sorted(combined.items())]
+    )
+
+
+def main():
+    ap = argparse.ArgumentParser()
+    ap.add_argument(
+        "--run",
+        action="append",
+        dest="runs",
+        required=True,
+        metavar="RUN",
+        help="Teuthology run name (repeat for multiple runs; merged into one table).",
+    )
+    ap.add_argument("--paddles-url", default=DEFAULT_PADDLES)
+    ap.add_argument("--out", default=None)
+    args = ap.parse_args()
+    table = merged_table(args.paddles_url, args.runs)
+    if table is None:
+        sys.exit(1)
+    print(table)
+    if args.out:
+        with open(args.out, "w") as f:
+            f.write(table)
+    return 0
+
+
+if __name__ == "__main__":
+    sys.exit(main())

release-tracker-workflow/scripts/run_teuthology_suite.sh

@@ -0,0 +1,99 @@
+#!/usr/bin/env bash
+# Run teuthology-suite with fixed argv, intended for Jenkins.
+# Required env -  VIRTUALENV_PATH, SUITE_NAME, CEPH_BRANCH, CEPH_SHA1
+# Optional - OVERRIDE_YAML, MACHINE_TYPE, CEPH_REPO, SUITE_LIMIT, SUITE_JOB_THRESHOLD, SUITE_SUBSET,
+# SUITE_REPO (--suite-repo), SUITE_SHA (--suite-sha1).
+# Job params map via the Jenkinsfile.
+set -euo pipefail
+
+: "${VIRTUALENV_PATH:?VIRTUALENV_PATH must be set}"
+: "${SUITE_NAME:?SUITE_NAME must be set}"
+: "${CEPH_BRANCH:?CEPH_BRANCH must be set}"
+: "${CEPH_SHA1:?CEPH_SHA1 must be set}"
+
+if [[ ! "${SUITE_NAME}" =~ ^[a-zA-Z0-9_/:-]+$ ]]; then
+  echo "run_teuthology_suite.sh: invalid SUITE_NAME: ${SUITE_NAME}" >&2
+  exit 1
+fi
+if [[ ! "${CEPH_BRANCH}" =~ ^[a-zA-Z0-9/._-]+$ ]]; then
+  echo "run_teuthology_suite.sh: invalid CEPH_BRANCH: ${CEPH_BRANCH}" >&2
+  exit 1
+fi
+if [[ "${CEPH_SHA1}" == "unknown" ]]; then
+  :
+elif [[ ! "${CEPH_SHA1}" =~ ^[a-fA-F0-9]+$ ]] || [[ ${#CEPH_SHA1} -lt 7 ]]; then
+  echo "run_teuthology_suite.sh: invalid CEPH_SHA1: ${CEPH_SHA1}" >&2
+  exit 1
+fi
+
+TEUTHOLOGY_SUITE="${VIRTUALENV_PATH}/bin/teuthology-suite"
+if [[ ! -f "${TEUTHOLOGY_SUITE}" ]]; then
+  echo "run_teuthology_suite.sh: teuthology-suite not found: ${TEUTHOLOGY_SUITE}" >&2
+  exit 1
+fi
+
+MACHINE_TYPE="${MACHINE_TYPE:-trial}"
+CEPH_REPO="${CEPH_REPO:-https://github.com/ceph/ceph.git}"
+SUITE_LIMIT="${SUITE_LIMIT:-1}"
+
+if [[ ! "${MACHINE_TYPE}" =~ ^[a-zA-Z0-9_.-]+$ ]]; then
+  echo "run_teuthology_suite.sh: invalid MACHINE_TYPE: ${MACHINE_TYPE}" >&2
+  exit 1
+fi
+if [[ ! "${SUITE_LIMIT}" =~ ^[0-9]+$ ]] || [[ "${SUITE_LIMIT}" == "0" ]]; then
+  echo "run_teuthology_suite.sh: invalid SUITE_LIMIT (positive integer): ${SUITE_LIMIT}" >&2
+  exit 1
+fi
+
+if [[ -n "${SUITE_REPO:-}" ]] && [[ ! "${SUITE_REPO}" =~ ^[a-zA-Z0-9@.:/_-]+$ ]]; then
+  echo "run_teuthology_suite.sh: invalid SUITE_REPO: ${SUITE_REPO}" >&2
+  exit 1
+fi
+if [[ -n "${SUITE_SHA:-}" ]]; then
+  if [[ ! "${SUITE_SHA}" =~ ^[a-fA-F0-9]+$ ]] || [[ ${#SUITE_SHA} -lt 7 ]]; then
+    echo "run_teuthology_suite.sh: invalid SUITE_SHA: ${SUITE_SHA}" >&2
+    exit 1
+  fi
+fi
+
+if [[ -n "${SUITE_JOB_THRESHOLD:-}" ]]; then
+  if [[ ! "${SUITE_JOB_THRESHOLD}" =~ ^[0-9]+$ ]]; then
+    echo "run_teuthology_suite.sh: invalid SUITE_JOB_THRESHOLD (non-negative integer): ${SUITE_JOB_THRESHOLD}" >&2
+    exit 1
+  fi
+fi
+if [[ -n "${SUITE_SUBSET:-}" ]]; then
+  if [[ ! "${SUITE_SUBSET}" =~ ^[0-9]+/[0-9]+$ ]]; then
+    echo "run_teuthology_suite.sh: invalid SUITE_SUBSET (expected N/M, e.g. 1/10000): ${SUITE_SUBSET}" >&2
+    exit 1
+  fi
+fi
+
+set -- \
+  "${TEUTHOLOGY_SUITE}" \
+  --suite "${SUITE_NAME}" \
+  --machine-type "${MACHINE_TYPE}" \
+  --ceph "${CEPH_BRANCH}" \
+  --ceph-repo "${CEPH_REPO}" \
+  --limit "${SUITE_LIMIT}"
+
+if [[ -n "${SUITE_JOB_THRESHOLD:-}" ]]; then
+  set -- "$@" --job-threshold "${SUITE_JOB_THRESHOLD}"
+fi
+if [[ -n "${SUITE_SUBSET:-}" ]]; then
+  set -- "$@" --subset "${SUITE_SUBSET}"
+fi
+
+set -- "$@" --sha1 "${CEPH_SHA1}"
+
+if [[ -n "${SUITE_REPO:-}" ]]; then
+  set -- "$@" --suite-repo "${SUITE_REPO}"
+fi
+if [[ -n "${SUITE_SHA:-}" ]]; then
+  set -- "$@" --suite-sha1 "${SUITE_SHA}"
+fi
+if [[ -n "${OVERRIDE_YAML:-}" ]]; then
+  set -- "$@" "${OVERRIDE_YAML}"
+fi
+
+exec "$@"