Testing 4 GPU relvals in parallel

[smuzaffar]

By default we run N jobs in parallel for GPU relvals (where N is number of GPU available mostly 1). This PR s to test if we can run multiple relvals in parallel

[smuzaffar]

enable gpu

[smuzaffar]

please test

[cmsbuild]

A new Pull Request was created by @smuzaffar for branch master.

@akritkbehera, @iarspider, @raoatifshad, @smuzaffar can you please review it and eventually sign? Thanks.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.
cms-bot commands are listed here

[cmsbuild]

cms-bot internal usage

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5835b/53234/summary.html
COMMIT: a556f63
CMSSW: CMSSW_17_0_X_2026-05-12-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2747/53234/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 5 lines from the logs
• Reco comparison results: 16 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4187168
• DQMHistoTests: Total failures: 42
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4187106
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 197 edm output root files, 53 DQM output files
• TriggerResults: no differences found

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.404 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• You potentially removed 98 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 322 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 34886
• DQMHistoTests: Total nulls: 38
• DQMHistoTests: Total successes: 181335
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 48 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

AMD_W7900 Comparison Summary

Summary:

• You potentially removed 28 lines from the logs
• Reco comparison results: 363 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 31439
• DQMHistoTests: Total nulls: 31
• DQMHistoTests: Total successes: 184789
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

NVIDIA_H100 Comparison Summary

Summary:

• You potentially removed 10 lines from the logs
• Reco comparison results: 321 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 45249
• DQMHistoTests: Total nulls: 34
• DQMHistoTests: Total successes: 170976
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

• You potentially removed 9 lines from the logs
• Reco comparison results: 360 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 31135
• DQMHistoTests: Total nulls: 32
• DQMHistoTests: Total successes: 185092
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

[smuzaffar]

please test

[cmsbuild]

Pull request #2747 was updated.

[cmsbuild]

-1

Failed Tests: RelVals-AMD_W7900
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5835b/53244/summary.html
COMMIT: e23c8f2
CMSSW: CMSSW_17_0_X_2026-05-13-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2747/53244/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals-AMD_W7900

• 34634.75134634.751_TTbar_14TeV+Run4D121PU_HLT75e33TimingAlpaka/step2_TTbar_14TeV+Run4D121PU_HLT75e33TimingAlpaka.log

Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 4 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4187168
• DQMHistoTests: Total failures: 32
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4187116
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 197 edm output root files, 53 DQM output files
• TriggerResults: no differences found

AMD_MI300X Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.402 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

• You potentially removed 130 lines from the logs
• ROOTFileChecks: Some differences in event products or their sizes found
• Reco comparison results: 321 differences found in the comparisons
• DQMHistoTests: Total files compared: 12
• DQMHistoTests: Total histograms compared: 200550
• DQMHistoTests: Total failures: 23086
• DQMHistoTests: Total nulls: 34
• DQMHistoTests: Total successes: 177430
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
• Checked 47 log files, 48 edm output root files, 12 DQM output files
• TriggerResults: found differences in 1 / 11 workflows

NVIDIA_H100 Comparison Summary

Summary:

• You potentially removed 9 lines from the logs
• Reco comparison results: 369 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 31315
• DQMHistoTests: Total nulls: 34
• DQMHistoTests: Total successes: 184910
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 367 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 27881
• DQMHistoTests: Total nulls: 34
• DQMHistoTests: Total successes: 188344
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 2 / 12 workflows

[smuzaffar]

please test

[cmsbuild]

+1

Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-b5835b/53250/summary.html
COMMIT: e23c8f2
CMSSW: CMSSW_17_0_X_2026-05-13-2300/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week1/cms-sw/cms-bot/2747/53250/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

• You potentially removed 2 lines from the logs
• Reco comparison results: 8 differences found in the comparisons
• DQMHistoTests: Total files compared: 53
• DQMHistoTests: Total histograms compared: 4187168
• DQMHistoTests: Total failures: 37
• DQMHistoTests: Total nulls: 0
• DQMHistoTests: Total successes: 4187111
• DQMHistoTests: Total skipped: 20
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 52 files compared)
• Checked 227 log files, 197 edm output root files, 53 DQM output files
• TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

• You potentially added 4 lines to the logs
• Reco comparison results: 330 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 32848
• DQMHistoTests: Total nulls: 34
• DQMHistoTests: Total successes: 183377
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 2 / 12 workflows

AMD_W7900 Comparison Summary

Summary:

• You potentially added 3 lines to the logs
• Reco comparison results: 365 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 30644
• DQMHistoTests: Total nulls: 33
• DQMHistoTests: Total successes: 185582
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 2 / 12 workflows

NVIDIA_H100 Comparison Summary

Summary:

• You potentially removed 1 lines from the logs
• Reco comparison results: 381 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 28113
• DQMHistoTests: Total nulls: 34
• DQMHistoTests: Total successes: 188112
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

• You potentially added 9 lines to the logs
• Reco comparison results: 350 differences found in the comparisons
• DQMHistoTests: Total files compared: 13
• DQMHistoTests: Total histograms compared: 216259
• DQMHistoTests: Total failures: 28833
• DQMHistoTests: Total nulls: 33
• DQMHistoTests: Total successes: 187393
• DQMHistoTests: Total skipped: 0
• DQMHistoTests: Total Missing objects: 0
• DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
• Checked 49 log files, 50 edm output root files, 13 DQM output files
• TriggerResults: found differences in 1 / 12 workflows

[smuzaffar]

+externals

[cmsbuild]

This pull request is fully signed and it will be integrated in one of the next master IBs (tests are also fine). This pull request will now be reviewed by the release team before it's merged. @sextonkennedy, @ftenchini, @mandrenguyen (and backports should be raised in the release meeting by the corresponding L2)