[NGT] Fix content of `SiPixelClusterSoA` for empty Pixel Detector by JanGerritSchulz · Pull Request #50900 · cms-sw/cmssw

JanGerritSchulz · 2026-05-07T11:59:11Z

PR description:

This PR is introducing the zero-initialization of the SiPixelClusterSoA for events with zero digis in the pixel detector. It fixes an issue that is arising later in the CA pixel tracking when the (previously wrong) values of the SoA are used, leading to crashes. This is described in detail in #50881 .

PR validation:

It is a minor change that only affects empty events in the pixel tracker. I tested on the previously crashing workflow 34090.203, which runs after these changes. For any further validation the bot-tests should be sufficient.

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Not a backport.

JanGerritSchulz · 2026-05-07T11:59:25Z

type ngt

JanGerritSchulz · 2026-05-07T11:59:35Z

type bug-fix

cmsbuild · 2026-05-07T11:59:35Z

cms-bot internal usage

cmsbuild · 2026-05-07T12:01:06Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50900/49258

There are other open Pull requests which might conflict with changes you have proposed:
- File RecoLocalTracker/SiPixelClusterizer/plugins/alpaka/SiPixelRawToClusterKernel.dev.cc modified in PR(s): Embedding hep mc filter fix for photos #50894

cmsbuild · 2026-05-07T12:01:51Z

A new Pull Request was created by @JanGerritSchulz for master.

It involves the following packages:

RecoLocalTracker/SiPixelClusterizer (reconstruction)

@Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @srimanob can you please review it and eventually sign? Thanks.
@GiacomoSguazzoni, @VinInn, @VourMa, @dkotlins, @elusian, @felicepantaleo, @ferencek, @gpetruc, @mmasciov, @mmusich, @mroguljic, @mtosi, @rovere, @threus, @tsusa this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

mmusich · 2026-05-07T12:18:19Z

enable gpu

mmusich · 2026-05-07T12:18:38Z

test parameters:

relval_opts = -w upgrade
workflows = ph2_hlt, 34090.203

mmusich · 2026-05-07T12:18:44Z

@cmsbuild, please test

cmsbuild · 2026-05-08T08:28:30Z

+1

Size: This PR adds an extra 28KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4b7182/53122/summary.html
COMMIT: e9c2bb7
CMSSW: CMSSW_17_0_X_2026-05-07-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50900/53122/install.sh to create a dev area with all the needed externals and cmssw changes.

Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 6 differences found in the comparisons
DQMHistoTests: Total files compared: 66
DQMHistoTests: Total histograms compared: 4569622
DQMHistoTests: Total failures: 6
DQMHistoTests: Total nulls: 0
DQMHistoTests: Total successes: 4569596
DQMHistoTests: Total skipped: 20
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 65 files compared)
Checked 276 log files, 236 edm output root files, 66 DQM output files
TriggerResults: no differences found

AMD_MI300X Comparison Summary

Summary:

You potentially added 3 lines to the logs
Reco comparison results: 331 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 39771
DQMHistoTests: Total nulls: 33
DQMHistoTests: Total successes: 176455
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 1 / 12 workflows

AMD_W7900 Comparison Summary

There are some workflows for which there are errors in the baseline:
34634.751 step 2
The results for the comparisons for these workflows could be incomplete
This means most likely that the IB is having errors in the relvals.The error does NOT come from this pull request

Summary:

You potentially added 69 lines to the logs
ROOTFileChecks: Some differences in event products or their sizes found
Reco comparison results: 359 differences found in the comparisons
DQMHistoTests: Total files compared: 12
DQMHistoTests: Total histograms compared: 184046
DQMHistoTests: Total failures: 23865
DQMHistoTests: Total nulls: 16
DQMHistoTests: Total successes: 160165
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 11 files compared)
Checked 48 log files, 50 edm output root files, 12 DQM output files
TriggerResults: found differences in 1 / 11 workflows

NVIDIA_H100 Comparison Summary

Summary:

You potentially removed 12 lines from the logs
Reco comparison results: 372 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 28844
DQMHistoTests: Total nulls: 30
DQMHistoTests: Total successes: 187385
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 1 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

You potentially removed 5 lines from the logs
Reco comparison results: 357 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 31118
DQMHistoTests: Total nulls: 30
DQMHistoTests: Total successes: 185111
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: no differences found

Max Memory Comparisons exceeding threshold

@cms-sw/core-l2 , I found 2 workflow step(s) with memory usage exceeding the error threshold:

Expand to see workflows ...

Error: Workflow 34496.0_CloseByPGun_CE_E_Front_120um+Run4D121 step2 max memory diff 272.4 exceeds +/- 90.0 MiB
Error: Workflow 34500.0_CloseByPGun_CE_H_Coarse_Scint+Run4D121 step2 max memory diff 272.4 exceeds +/- 90.0 MiB

jfernan2 · 2026-05-08T09:07:38Z

+1

cmsbuild · 2026-05-08T13:40:21Z

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50900/49266

There are other open Pull requests which might conflict with changes you have proposed:
- File RecoLocalTracker/SiPixelClusterizer/plugins/alpaka/SiPixelRawToClusterKernel.dev.cc modified in PR(s): Embedding hep mc filter fix for photos #50894

cmsbuild · 2026-05-08T13:40:46Z

Pull request #50900 was updated. @Moanwar, @cmsbuild, @jfernan2, @mandrenguyen, @srimanob can you please check and sign again.

mmusich · 2026-05-08T13:40:55Z

@cmsbuild, please test

JanGerritSchulz · 2026-05-08T13:46:36Z

So, for future reference:
I just added a resetting of the host-side numbers of the module-start-first-element, the module-start-last-element and the bpix2-cluster-offset. All of these overwrite the zero-initialized nCluster and co values in the PortableDeviceCollection when getClusters() and getDigis() are called. This leads to some other issues down-stream (as noticed by @bfonta in the TrackFromSoAConverter). Again that occurs as only in even more special cases, then the original issue.

cmsbuild · 2026-05-08T21:36:20Z

-1

Failed Tests: RelVals
Size: This PR adds an extra 24KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-4b7182/53146/summary.html
COMMIT: 76fdaaf
CMSSW: CMSSW_17_0_X_2026-05-08-1100/el8_amd64_gcc13
Additional Tests: GPU,AMD_MI300X,AMD_W7900,NVIDIA_H100,NVIDIA_L40S
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50900/53146/install.sh to create a dev area with all the needed externals and cmssw changes.

Failed RelVals

34090.203

----- Begin Fatal Exception 08-May-2026 16:46:10 CEST-----------------------
An exception of category 'TiclOutOfBound' occurred while
   [0] Processing  Event run: 1 lumi: 1 event: 1 stream: 0
   [1] Running path 'dqmofflineOnPAT_1_step'
   [2] Prefetching for module SingleTopTChannelLeptonDQM_miniAOD/'singleTopElectronMediumDQM_miniAOD'
   [3] Prefetching for module PATMuonSlimmer/'slimmedMuons'
   [4] Prefetching for module PATMuonSelector/'selectedPatMuons'
   [5] Prefetching for module PATMuonProducer/'patMuons'
   [6] Prefetching for module MuonProducer/'muons'
   [7] Prefetching for module PFCandidateListMerger/'particleFlowTmp'
   [8] Prefetching for module PFProducer/'particleFlowTmpBarrel'
   [9] Prefetching for module PFBlockProducer/'particleFlowBlock'
   [10] Prefetching for module PFElecTkProducer/'pfTrackElec'
   [11] Prefetching for module GsfTrackProducer/'electronGsfTracks'
   [12] Prefetching for module CkfTrackCandidateMaker/'electronCkfTrackCandidates'
   [13] Prefetching for module ElectronSeedMerger/'electronMergedSeeds'
   [14] Prefetching for module ElectronSeedProducer/'ecalDrivenElectronSeeds'
   [15] Prefetching for module EGammaSuperclusterProducer/'particleFlowSuperClusterHGCal'
   [16] Prefetching for module TracksterLinksProducer/'ticlTracksterLinksSuperclusteringDNN'
   [17] Prefetching for module TrackstersProducer/'ticlTrackstersCLUE3DHigh'
   [18] Calling method for module TICLLayerTileProducer/'ticlLayerTileProducer'
Exception Message:
bin index -2147479490 is out of bounds (container size is 4284). It was generated from eta: -23123.967 phi: -nan
----- End Fatal Exception -------------------------------------------------

AMD_MI300X Comparison Summary

Summary:

You potentially added 19 lines to the logs
Reco comparison results: 357 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 39609
DQMHistoTests: Total nulls: 32
DQMHistoTests: Total successes: 176618
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: no differences found

AMD_W7900 Comparison Summary

Summary:

You potentially added 11 lines to the logs
Reco comparison results: 363 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 29149
DQMHistoTests: Total nulls: 36
DQMHistoTests: Total successes: 187074
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 2 / 12 workflows

NVIDIA_H100 Comparison Summary

Summary:

You potentially removed 11 lines from the logs
Reco comparison results: 354 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 40685
DQMHistoTests: Total nulls: 34
DQMHistoTests: Total successes: 175540
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 2 / 12 workflows

NVIDIA_L40S Comparison Summary

Summary:

You potentially added 1 lines to the logs
Reco comparison results: 336 differences found in the comparisons
DQMHistoTests: Total files compared: 13
DQMHistoTests: Total histograms compared: 216259
DQMHistoTests: Total failures: 42317
DQMHistoTests: Total nulls: 33
DQMHistoTests: Total successes: 173909
DQMHistoTests: Total skipped: 0
DQMHistoTests: Total Missing objects: 0
DQMHistoSizes: Histogram memory added: 0.0 KiB( 12 files compared)
Checked 49 log files, 50 edm output root files, 13 DQM output files
TriggerResults: found differences in 1 / 12 workflows

mmusich · 2026-05-08T21:44:08Z

-1

the failure is related to #42470 (and happens in step3, step3-like the step2 --previously failing-- now runs correctly)

mmusich · 2026-05-08T21:45:22Z

ignore tests-rejected with ib-failure

jfernan2 · 2026-05-11T07:55:10Z

+1

cmsbuild · 2026-05-11T07:55:38Z

This pull request is fully signed and it will be integrated in one of the next master IBs (test failures were overridden). This pull request will now be reviewed by the release team before it's merged. @mandrenguyen, @ftenchini, @sextonkennedy (and backports should be raised in the release meeting by the corresponding L2)

mandrenguyen · 2026-05-12T05:19:38Z

+1

Fix filling of SiPixelClusterSoA in event with empty pixel tracker

e9c2bb7

cmsbuild added this to the CMSSW_17_0_X milestone May 7, 2026

cmsbuild added reconstruction-pending pending-signatures tests-pending orp-pending code-checks-pending trk ngt labels May 7, 2026

cmsbuild added the bug-fix label May 7, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels May 7, 2026

cmsbuild added tests-started and removed tests-pending labels May 7, 2026

cmsbuild added tests-approved and removed tests-started labels May 8, 2026

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels May 8, 2026

cmsbuild added code-checks-approved and removed code-checks-pending labels May 8, 2026

cmsbuild added tests-started and removed tests-pending labels May 8, 2026

cmsbuild added tests-rejected and removed tests-started labels May 8, 2026

cmsbuild added tests-approved tests-ib-failure and removed tests-rejected labels May 8, 2026

cmsbuild added reconstruction-approved fully-signed and removed reconstruction-pending pending-signatures labels May 11, 2026

cmsbuild added orp-approved and removed orp-pending labels May 12, 2026

cmsbuild merged commit bc149ed into cms-sw:master May 12, 2026
22 of 23 checks passed

This was referenced May 12, 2026

BuildRules: Make sure to build local rocm/cuda dlink archives cms-sw/cmsdist#10546

Merged

[G4ADEPT] update AdePT specs to v0.3.3 cms-sw/cmsdist#10476

Merged

[EVO] fix merge conflict 2026/05/13 #50933

Merged

[EVOLUTION_X] Remove unnecessary file #50951

Merged

Conversation

JanGerritSchulz commented May 7, 2026

PR description:

PR validation:

If this PR is a backport please specify the original PR and why you need to backport that PR. If this PR will be backported please specify to which release cycle the backport is meant for:

Uh oh!

JanGerritSchulz commented May 7, 2026

Uh oh!

JanGerritSchulz commented May 7, 2026

Uh oh!

cmsbuild commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

cmsbuild commented May 7, 2026

Uh oh!

cmsbuild commented May 7, 2026

Uh oh!

mmusich commented May 7, 2026

Uh oh!

mmusich commented May 7, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

mmusich commented May 7, 2026

Uh oh!

cmsbuild commented May 8, 2026

Comparison Summary

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

Max Memory Comparisons exceeding threshold

Uh oh!

jfernan2 commented May 8, 2026

Uh oh!

cmsbuild commented May 8, 2026

Uh oh!

cmsbuild commented May 8, 2026

Uh oh!

mmusich commented May 8, 2026

Uh oh!

JanGerritSchulz commented May 8, 2026

Uh oh!

cmsbuild commented May 8, 2026

Failed RelVals

AMD_MI300X Comparison Summary

AMD_W7900 Comparison Summary

NVIDIA_H100 Comparison Summary

NVIDIA_L40S Comparison Summary

Uh oh!

mmusich commented May 8, 2026

Uh oh!

mmusich commented May 8, 2026

Uh oh!

jfernan2 commented May 11, 2026

Uh oh!

cmsbuild commented May 11, 2026

Uh oh!

mandrenguyen commented May 12, 2026

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

5 participants

cmsbuild commented May 7, 2026 •

edited

Loading

mmusich commented May 7, 2026 •

edited

Loading