Insights on Premise aims to provide recommendations based on Insights archives in environments that cannot reach console.redhat.com. Specifically it is designed to be deployed in ACM clusters and for providing rule-based recommendations for managed clusters.
- Insights on Premise
Before going forward with deployment steps, check that:
- The hub is running on ACM 2.17.1+ and all clusters in the fleet are running OpenShift version >= 4.20.
- MultiClusterHub is created in
open-cluster-managementnamespace (it can take several minutes before all components are started). - Hub cluster self-management is enabled (default ACM behavior). The hub must be imported into ACM as a managed cluster (with the
local-cluster: "true"label) so that Policies can target it for certificate management. - Pull secret for quay.io/ccxdev/insights-on-premise-poc repository is saved as
deploy/02-pull-secret.ymlin the following format:
apiVersion: v1
kind: Secret
metadata:
name: ccxdev-insights-on-prem-pull-secret
namespace: insights-on-prem
data:
.dockerconfigjson: <INSERT YOUR BASE64-ENCODED PULL SECRET HERE>
type: kubernetes.io/dockerconfigjson- (optional) Multicluster Observability Operator is deployed according to these instructions. This step is required for enabling update risk predictions.
After confirming that prerequisites are met, you can install the addon with:
oc apply -f deploy/This applies all manifests in deploy/ to the cluster. It can take a while until all resources are properly deployed.
Note: Installation of Insights on Prem redirects Insights endpoints in ACM console and client deployments from console.redhat.com to Insights on Prem deployed on the hub cluster.
The postgres password is stored in the secret insights-postgres in the insights-on-prem namespace, defined in deploy/03-postgres.yml. Note that this is not the best practice, so please use the preferred method on your cluster to define the secret. We kept it there to make it easier to deploy the application without human intervention.
The application is configured via environment variables set on the insights-on-prem deployment in the insights-on-prem namespace. The following variables can be tuned after deployment:
| Variable | Default | Description |
|---|---|---|
DB_RETENTION_HOURS |
24 |
How long to keep processed records in the database before automatic cleanup |
DB_CLEANUP_INTERVAL_MINUTES |
60 |
How often the background cleanup task runs (in minutes) |
MAX_FILE_SIZE |
104857600 (100 MB) |
Maximum uploaded archive file size in bytes |
THANOS_URL |
https://rbac-query-proxy.open-cluster-management-observability.svc.cluster.local:8443 |
Thanos query endpoint URL (only relevant if MCO is deployed) |
THANOS_QUERY_TIMEOUT_SECONDS |
10 |
Timeout for Thanos queries |
THANOS_QUERY_LOOKBACK_MINUTES |
60 |
How far back to look when querying Thanos metrics |
MTLS_ENABLED |
true (in-cluster) |
Whether the server uses mTLS on port 8443 instead of plain HTTP on 8000 |
Database connection is configured through the insights-postgres secret (see Secrets above). The variables POSTGRES_HOST, POSTGRES_PORT, POSTGRES_USER, POSTGRES_PASSWORD, and POSTGRES_DB are all read from that secret in the default deployment manifests.
To change a setting, patch the deployment:
oc set env deployment/insights-on-prem -n insights-on-prem DB_RETENTION_HOURS=48After manifests are applied, you can check that everything was properly deployed by running the following:
# Check pod status
oc get pods -n insights-on-prem
# Check policy compliance (all should be Compliant)
oc get policy -n insights-on-prem
# Verify insights-client and console env overrides via MCH
oc get mch multiclusterhub -n open-cluster-management -o json | jq '.spec.overrides.components'
# Verify console URP URL
oc get configmap console-config -n open-cluster-management -o jsonpath='{.data.UPGRADE_RISKS_PREDICTION_URL}'
# Check logs
oc logs -f deployment/insights-on-prem -n insights-on-premInsights recommendations are visible in the cluster console under Fleet Management -> Home -> Overview, or go directly to this URL:
https://<CLUSTER_CONSOLE_URL>/multicloud/home/overview
The Insights section of that page has four panels:
| Panel | Depends on Insights on Prem | Depends on MCO |
|---|---|---|
| Cluster recommendations | Yes | No |
| Update risk predictions | Yes | Yes |
| Alerts | No | Yes |
| Failing operators | No | Yes |
Cluster recommendations are based on PolicyReport custom resources created by insights-client in each managed cluster's namespace. Update risk predictions are served by Insights on Prem, but rely on metrics collected by MCO into Thanos. Alerts and Failing operators are read directly from Thanos by the ACM console and do not involve Insights on Prem at all.
On-demand data gathering allows triggering Insights data collection outside the regular periodic schedule. Instead of waiting for the next periodic upload (default 2h, set to 1m by deploy/14-hub-config.yml), you can request an immediate gather-and-upload cycle and get results for that specific request.
Note: Conditional data gathering is not supported at this moment. Disable the
conditionalgatherer in theDataGatherCR to avoid unnecessary calls toconsole.redhat.comfor gathering rules (as shown in the following section).
Create a DataGather custom resource:
oc apply -f - <<'EOF'
apiVersion: insights.openshift.io/v1
kind: DataGather
metadata:
name: on-demand-test
spec:
gatherers:
mode: Custom
custom:
configs:
- name: conditional
state: Disabled
storage:
type: Ephemeral
EOFThe insights-operator detects the new CR, creates a Job in openshift-insights, and the Job:
- Runs all gatherers and writes an archive
- Uploads the archive to the on-prem service
- Polls the processing status endpoint until the archive is processed
- Logs success — the operator then fetches the report for the specific request ID
Watch the Job and its logs:
# Check job status
oc get jobs -n openshift-insights | grep -v periodic
# Follow the job pod logs
oc logs -n openshift-insights -l job-name=on-demand-test -f
# Check the DataGather CR status
oc get datagather on-demand-test -o yamlThe DataGather CR status conditions show the lifecycle:
DataRecorded— archive written to diskDataUploaded— archive uploaded to the on-prem serviceDataProcessed— archive processed and results available
Jobs and DataGather CRs older than 24 hours are automatically pruned by the Insights Operator. To delete manually:
oc delete datagather on-demand-test
oc delete job on-demand-test -n openshift-insightsAfter deploying Insights on Prem on a healthy cluster, the Fleet Overview panels will likely be empty:
In case you want to quickly trigger some results, you can run test_ui.sh with the hub cluster kubeconfig. The script will execute changes on cluster (uploading metrics, creatings resources) in order to trigger both upgrade risk predictions and cluster recommendations for the hub cluster:
./test_ui.shIf you want to revert the changes, see the script comments for cleanup instructions.
You can also trigger each section manually as described below.
The upgrade prediction service flags a cluster as at-risk when it detects two or more critical alerts. Create a PrometheusRule that fires two always-on critical alerts:
oc apply -f tests/ui/critical-alerts.yamlThe alerts need to reach Thanos before results appear (typically 2-5 minutes). By default the on-prem service queries Thanos at now - 60 minutes, so freshly fired alerts won't be visible. To query at the current timestamp instead:
oc set env deployment/insights-on-prem -n insights-on-prem THANOS_QUERY_LOOKBACK_MINUTES=0After the alerts propagate, you should see one cluster not recommended for update in the ACM console.
To clean up the changes, run:
oc delete prometheusrule insights-test-alerts -n openshift-monitoring
oc set env deployment/insights-on-prem -n insights-on-prem THANOS_QUERY_LOOKBACK_MINUTES-To trigger a sample Insights recommendation, at least one insights-core rule condition has to be met. The easiest way is to create a ValidatingWebhookConfiguration with a timeout larger than the default, which triggers the webhook_timeout_is_larger_than_default rule:
oc apply -f tests/ui/webhook-trigger.yamlDepending on the frequency of archive uploads from Insights Operator (set to 1 minute by deploy/14-hub-config.yml, but default value is 2 hours), the recommendation and the PolicyReport should be created. You can verify either via ACM console, or with:
oc get policyreport --all-namespacesAfter that you should be able to see at least one policyreport for the local-cluster (that is, for the ACM hub):
NAMESPACE NAME PASS FAIL WARN ERROR SKIP AGE
local-cluster local-cluster-policyreport 0 1 0 0 0 4m
To clean up the changes, run:
oc delete validatingwebhookconfiguration insights-test-webhook
The system consists of hub-side components that process Insights data and per-cluster HAProxy instances that route traffic from managed clusters to the hub.
On each managed cluster, the Insights Operator (in openshift-insights) collects diagnostic archives and sends them to a local HAProxy instance (in insights-on-prem). HAProxy terminates the local TLS connection and forwards the archive to the hub's Insights on Prem service through the passthrough Route, authenticating with a client certificate signed by the hub's client CA.
On the hub, Insights on Prem validates the client certificate, processes the archive using insights-core rules, and stores results in PostgreSQL. The Insights Client (in open-cluster-management) then polls Insights on Prem for processed results and creates PolicyReport custom resources, which surface as cluster recommendations in the ACM console.
For upgrade risk predictions, the ACM console queries Insights on Prem, which evaluates alerts and operator conditions retrieved from Thanos (in open-cluster-management-observability, deployed by the Multicluster Observability Operator).
HAProxy is deployed as an ACM managed cluster addon on every managed cluster, including the hub itself (which is self-managed). On the hub, HAProxy also serves as the local endpoint for the ACM console and Insights Client.
Note: The current deployment includes temporary workarounds that deviate from the diagram above:
- A CONNECT proxy in
open-cluster-managementworks around an ACM console limitation with service CA trust (to be removed by #209 once the console fix is backported).- A cluster-wide Proxy patch distributes the service CA to the Insights Operator until it natively supports a CA certificate field in its ConfigMap (to be removed by #204).
- HAProxy is currently deployed in the
openshift-insightsnamespace on managed clusters; it will be moved toinsights-on-premsince it is not used exclusively by the Insights Operator.
The deployment installs the cert-manager Operator and creates cert-manager issuers that manage the server-side TLS certificates. These certificates are distributed via ACM Policies. Client certificates are issued separately through ACM's CustomSigner addon registration:
| Certificate | Issued by | Scope | Purpose |
|---|---|---|---|
| Server CA | Self-signed bootstrap ClusterIssuer |
Hub | Signs the server leaf certificate for the Insights on Prem service |
| Client CA | Self-signed bootstrap ClusterIssuer |
Hub | Referenced by ACM's CustomSigner to sign client certificates for managed clusters |
| Server leaf cert | Namespaced Issuer (backed by server CA) |
Hub | Used by Insights on Prem for mTLS; includes the Route hostname as a SAN |
| Service-serving cert | OpenShift service serving certificate (via service.beta.openshift.io/serving-cert-secret-name annotation) |
Each managed cluster | Used by HAProxy to accept local connections from the Insights Operator |
| Client cert | ACM CustomSigner (backed by client CA) | Each managed cluster | Used by HAProxy to authenticate to the hub |
cert-manager automatically renews the server-side certificates before expiry. When the server leaf certificate is renewed, a ConfigurationPolicy watches the certificate's hash and patches the Deployment annotation to trigger a rolling restart. On managed clusters, HAProxy detects certificate changes via a liveness probe that compares certificate checksums, causing the pod to restart and load the new certificates.
Insights on Prem authenticates clients via mTLS client certificate verification. Since HAProxy is the only component that obtains a client certificate, it is the sole authorized client of the service. HAProxy itself is not exposed outside the cluster, but NetworkPolicies prevent unintended in-cluster traffic from reaching it.
Two NetworkPolicies restrict ingress to the HAProxy proxy pod (app: insights-operator-proxy):
| Policy | Deployed to | Allowed callers |
|---|---|---|
insights-operator-proxy (13-spoke-policy.yml) |
All managed clusters (including hub) | Insights Operator (app: insights-operator, same namespace) |
insights-operator-proxy-hub (14-hub-config.yml) |
Hub only | Insights Client and ACM console, both from open-cluster-management |
On managed clusters only the first policy applies, so only the Insights Operator can reach HAProxy. On the hub both policies apply and Kubernetes unions their ingress rules, additionally allowing the Insights Client and the ACM console.
The application deploys its own PostgreSQL database. Data older than 24 hours is cleaned up automatically by default (configurable via DB_RETENTION_HOURS environment variable).
Connect to database:
# Locally
docker compose exec postgres psql -U insights -d insights
# In cluster
oc exec -it deployment/insights-postgres -n insights-on-prem -- psql -U insights -d insightsPOST /api/ingress/v1/upload
Upload an Insights archive for processing.
Example:
curl -X POST http://localhost:8000/api/ingress/v1/upload -F "file=@/path/to/archive.tar.gz"GET /api/v2/cluster/{cluster_id}/reports
Retrieve processed report for a cluster.
POST /api/insights-results-aggregator/v2/upgrade-risks-prediction
Returns upgrade risk predictions for a list of clusters. These predictions are based on alerts and operator conditions that are retrieved from Thanos instance.
GET /api/v2/cluster/{cluster_id}/request/{request_id}/status
Check whether an on-demand data gathering request has been processed. Returns 404 while processing is in progress (the operator retries), 200 once ready.
GET /api/v2/cluster/{cluster_id}/request/{request_id}/report
Retrieve the simplified report for a specific on-demand data gathering request ID.
GET /health
When running Insights on Prem locally, you can access documentation via these endpoints:
- Swagger UI: http://localhost:8000/docs
- ReDoc: http://localhost:8000/redoc
For purposes of running the addon locally without need for the cluster, we maintain docker-compose.yml, so docker-compose is required. Alternatively, you can also use podman-compose. The commands are the same.
-
Start services:
docker compose up -d
-
Run database migrations:
docker compose exec app alembic -c migrations/alembic.ini upgrade head -
Verify:
curl http://localhost:8000/health
-
View logs:
docker compose logs -f app
-
Stop services:
docker compose down
The Dockerfile is built hermetically by Konflux (network access disabled during the build), using Hermeto to prefetch pip and RPM dependencies beforehand. See requirements-in.txt (source for pip-compile), requirements.txt/requirements-build.txt (pinned lockfiles), and rpms.in.yaml/rpms.lock.yaml (RPM lockfile, regenerated via scripts/update_rpm_lockfile.sh).
To add or upgrade a Python dependency, edit requirements-in.txt, then regenerate the pinned files. Always do this inside a linux/amd64 Python 3.12 container (matching the base image and Konflux's build platform) — running pip-compile on macOS/arm64 silently drops dependencies whose markers only match x86_64/aarch64 (e.g. SQLAlchemy's greenlet), since pip-compile resolves environment markers against the machine it runs on, not the target platform:
# Prefer the helper script (also used by ccx-rules-releaser):
podman run --rm --platform linux/amd64 -v "$(pwd):/work:Z" -w /work python:3.12-slim \
bash scripts/update_requirements.sh
# Equivalent manual commands:
podman run --rm --platform linux/amd64 -v "$(pwd):/work:Z" -w /work python:3.12-slim bash -c '
set -euo pipefail
pip install -q pip-tools pybuild-deps
pip-compile --output-file=requirements.txt requirements-in.txt
pybuild-deps compile --generate-hashes --output-file=requirements-build.txt requirements.txt
'requirements.txt is the fully-pinned runtime lockfile; requirements-build.txt lists the build-backend sdists (e.g. setuptools, cython, maturin) Hermeto needs to prefetch so packages without prebuilt wheels can be built from source in the hermetic build.
To regenerate rpms.lock.yaml, run the helper inside a linux/amd64 entitlement-capable RHEL9/UBI9 container (registry.access.redhat.com/ubi9/ubi, which includes subscription-manager for entitled RHEL CDN repos). Do not run it on the host. Requires RH_ORG_ID, RH_ACTIVATION_KEY, and scripts/.dockerconfig.json (for registry.redhat.io):
export RH_ORG_ID=...
export RH_ACTIVATION_KEY=...
podman run --rm --platform linux/amd64 \
-v "$(pwd):/work:Z" -w /work \
-e RH_ORG_ID -e RH_ACTIVATION_KEY \
registry.access.redhat.com/ubi9/ubi \
bash scripts/update_rpm_lockfile.shSome RPMs (e.g. postgresql-devel) are only available on the entitled RHEL CDN, not the public UBI repos. Konflux's prefetch-dependencies task needs an activation-key secret in the obsint-processing-tenant namespace to authenticate to that CDN — without it, prefetching fails with a misleading SSLCertVerificationError: self-signed certificate in certificate chain. This secret is namespace-scoped (shared with rules-containers), see the value in Bitwarden, so it only needs to be created once per tenant:
oc create -f <path-to>/activation-key-secret.yamlYou may need to follow https://konflux.pages.redhat.com/docs/users/building/activation-keys-subscription.html#Create-custom-activation-key-secret to troubleshoot any issues.
rpms.in.yaml's context.image must be kept in sync with the Dockerfile's FROM tag. rpm-lockfile-prototype resolves dependencies against the RPMs already installed in that image, so pointing it at a different image (or a stale tag) can lock in package versions (e.g. glibc-devel) that don't match what's actually baked into the real base image, causing microdnf install to fail in the hermetic build with nothing provides glibc = <locked-version>. Whenever you bump the base image tag in the Dockerfile, update context.image to match and rerun scripts/update_rpm_lockfile.sh.
In case you need to manually build and push a multiarch (amd64, arm64) image to Quay, run these commands (this step is necessary because cluster nodes may run on different architecture than the development environment):
# Login to Quay
docker login quay.io
# Build and push multiarch image
docker buildx build --platform linux/amd64,linux/arm64 \
-t quay.io/NAMESPACE/IMAGE:TAG \
--push .This project is licensed under the AGPL v3 - see the LICENSE file for details.