ci: auto-retry infra-caused script failures via exit code 75 sentinel (#3812)

* ci: auto-retry infra-caused script failures via exit code 75 sentinel

Service startup timeouts (Kafka, Zookeeper, MySQL, etc.) exit the job
with code 1, which GitLab classifies as script_failure — not covered by
the existing runner-level retry rules, and the team doesn't want to
enable blanket script_failure retry (hides real flakiness).

GitLab 14.9+ supports `retry: exit_codes:` which fires the
script_failure retry rule only when the exit code matches. We use
EX_TEMPFAIL (75) as the infra-sentinel: wait-for-service-ready.sh now
exits 75 on service timeout instead of 1, and the global default retry
block adds `script_failure` gated on exit code 75.

Effect: Kafka/Zookeeper/other service startup races are retried up to 2
times automatically. Real test failures (exit 1) are never retried.

* ci: remove inline comment on exit 75

* ci: fix retry config — exit_codes is OR not a filter on script_failure

Remove script_failure from the global when: list. Having both
script_failure and exit_codes: [75] retries on any script failure OR
exit code 75 — not exit code 75 only as intended.

exit_codes: [75] alone correctly retries only jobs that exit with
code 75 (the EX_TEMPFAIL sentinel from wait-for-service-ready.sh),
leaving all other script failures (exit 1, real test failures)
unretried.

* ci: add temporary test jobs to validate exit_codes retry behaviour

test-retry-exit-75: exits 75, no job-level retry override → inherits
global default → should be retried twice (3 total attempts)

test-no-retry-exit-1: exits 1, same config → should run exactly once

Both jobs are branch-scoped (leiyks/infra-failure-retry only) and
allow_failure: true. Remove after verification.

* ci: remove temporary retry validation test jobs

Author: Leiyks

.gitlab/generate-common.php

@@ -111,6 +111,8 @@ function windows_git_setup_with_packages() {
       - api_failure
       - stuck_or_timeout_failure
       - job_execution_timeout
+    exit_codes:
+      - 75
 
 .all_targets: &all_minor_major_targets
 <?php

.gitlab/wait-for-service-ready.sh

@@ -115,7 +115,7 @@ if [ -n "${WAIT_FOR:-}" ]; then
     service_type="$(detect_service_type "${host}")"
 
     if ! wait_for_single_service "${host}" "${port}" "${service_type}" 30 5; then
-      exit 1
+      exit 75
     fi
   done
 fi