Skip to content

[20_0_X] Move stored types in io_v1 namespace, reset class versions to 3#50968

Open
makortel wants to merge 1 commit into
cms-sw:masterfrom
makortel:evoIntegrationDataFormats
Open

[20_0_X] Move stored types in io_v1 namespace, reset class versions to 3#50968
makortel wants to merge 1 commit into
cms-sw:masterfrom
makortel:evoIntegrationDataFormats

Conversation

@makortel
Copy link
Copy Markdown
Contributor

PR description:

This PR integrates the data format changes prototyped and tested in the EVOLUTION_X branch to the master branch for 20_0_X. Following the strategy agreed in ORP 2026-05-05, it is opened already now to allow it to be reviewed such that it could be merged quickly when the master branch moves to 20_0_X.

This PR includes

  • move data types stored in files in workflow 34434.0 (plus those in PREMIX data tier) in io_v1 namespace to allow schema evolution for complex cases that ROOT's schema evolution can not support (e.g. pointed-to container type of edm::Ref being changed)
  • reset all class versions to 3 and all input-related read rules are removed (except of data types stored in RAW)
  • two earlier data type evolution attempt that were rejected at the time because they would have broken backwards compatibility

For more information please see the ORP 2026-05-05 presentation and the earlier talks linked there.

This PR is unusual in several ways:

Resolves cms-sw/framework-team#2227

PR validation:

None beyond the tests in the EVOLUTION_X IBs. Checked on CMSSW_17_0_X_2026-05-18-1100 that cherry-picking #50953, #50954, and this PR shows no difference wrt. CMSSW_17_0_EVOLUTION_X branch.

@makortel
Copy link
Copy Markdown
Contributor Author

hold

Until 20_0_X opens

@cmsbuild
Copy link
Copy Markdown
Contributor

cmsbuild commented May 18, 2026

cms-bot internal usage

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50968/49372

@cmsbuild
Copy link
Copy Markdown
Contributor

Pull request has been put on hold by @makortel
They need to issue an unhold command to remove the hold state or L1 can unhold it for all

@cmsbuild
Copy link
Copy Markdown
Contributor

A new Pull Request was created by @makortel for master.

It involves the following packages:

  • Alignment/CommonAlignmentProducer (alca)
  • Alignment/MillePedeAlignmentAlgorithm (alca)
  • AnalysisDataFormats/SUSYBSMObjects (analysis)
  • AnalysisDataFormats/TopObjects (analysis)
  • AnalysisDataFormats/TrackInfo (analysis)
  • DPGAnalysis/MuonTools (xpog)
  • DQM/DTMonitorModule (dqm)
  • DQM/GEM (dqm)
  • DQM/SiStripCommissioningSources (dqm)
  • DataFormats/Alignment (alca)
  • DataFormats/BTauReco (reconstruction)
  • DataFormats/BeamSpot (reconstruction)
  • DataFormats/CLHEP (core)
  • DataFormats/CSCDigi (simulation)
  • DataFormats/CSCRecHit (reconstruction)
  • DataFormats/CTPPSDigi (reconstruction, simulation)
  • DataFormats/CTPPSReco (reconstruction)
  • DataFormats/CaloRecHit (reconstruction)
  • DataFormats/CaloTowers (reconstruction)
  • DataFormats/Candidate (reconstruction)
  • DataFormats/CastorReco (reconstruction)
  • DataFormats/Common (core)
  • DataFormats/DTDigi (simulation)
  • DataFormats/DTRecHit (reconstruction)
  • DataFormats/EcalDetId (simulation)
  • DataFormats/EcalDigi (simulation)
  • DataFormats/EcalRawData (simulation)
  • DataFormats/EcalRecHit (reconstruction)
  • DataFormats/EgammaCandidates (reconstruction)
  • DataFormats/EgammaReco (reconstruction)
  • DataFormats/FEDRawData (daq)
  • DataFormats/FTLDigi (simulation)
  • DataFormats/FTLRecHit (reconstruction)
  • DataFormats/FWLite (core)
  • DataFormats/ForwardDetId (simulation)
  • DataFormats/GEMDigi (simulation)
  • DataFormats/GEMRecHit (reconstruction)
  • DataFormats/GeometryCommonDetAlgo (simulation)
  • DataFormats/GeometrySurface (simulation)
  • DataFormats/GeometryVector (simulation)
  • DataFormats/GsfTrackReco (reconstruction)
  • DataFormats/HGCDigi (simulation)
  • DataFormats/HGCRecHit (reconstruction)
  • DataFormats/HGCalDigi (simulation)
  • DataFormats/HGCalReco (reconstruction)
  • DataFormats/HLTReco (hlt)
  • DataFormats/HcalCalibObjects (alca)
  • DataFormats/HcalDetId (simulation)
  • DataFormats/HcalDigi (simulation)
  • DataFormats/HcalIsolatedTrack (reconstruction)
  • DataFormats/HcalRecHit (reconstruction)
  • DataFormats/HeavyIonEvent (reconstruction)
  • DataFormats/HepMCCandidate (generators)
  • DataFormats/JetMatching (simulation)
  • DataFormats/JetReco (reconstruction)
  • DataFormats/L1CSCTrackFinder (l1)
  • DataFormats/L1CaloTrigger (l1)
  • DataFormats/L1DTTrackFinder (l1)
  • DataFormats/L1GlobalCaloTrigger (l1)
  • DataFormats/L1GlobalMuonTrigger (l1)
  • DataFormats/L1GlobalTrigger (l1)
  • DataFormats/L1TCalorimeter (l1)
  • DataFormats/L1TCalorimeterPhase2 (l1)
  • DataFormats/L1TCorrelator (l1)
  • DataFormats/L1TGlobal (l1)
  • DataFormats/L1THGCal (l1)
  • DataFormats/L1TMuon (l1)
  • DataFormats/L1TMuonPhase2 (l1)
  • DataFormats/L1TParticleFlow (l1)
  • DataFormats/L1TrackTrigger (l1)
  • DataFormats/L1Trigger (l1)
  • DataFormats/LTCDigi (l1)
  • DataFormats/Luminosity (reconstruction)
  • DataFormats/METReco (reconstruction)
  • DataFormats/MuonDetId (simulation)
  • DataFormats/MuonReco (reconstruction)
  • DataFormats/MuonSeed (reconstruction)
  • DataFormats/NanoAOD (xpog)
  • DataFormats/OnlineMetaData (daq, reconstruction)
  • DataFormats/ParticleFlowCandidate (reconstruction)
  • DataFormats/ParticleFlowReco (reconstruction)
  • DataFormats/PatCandidates (reconstruction, xpog)
  • DataFormats/Phase2TrackerCluster (reconstruction)
  • DataFormats/Phase2TrackerDigi (reconstruction)
  • DataFormats/ProtonReco (reconstruction)
  • DataFormats/RPCDigi (simulation)
  • DataFormats/RPCRecHit (reconstruction)
  • DataFormats/RecoCandidate (reconstruction)
  • DataFormats/Scalers (daq, l1, reconstruction)
  • DataFormats/SiPixelCluster (reconstruction)
  • DataFormats/SiPixelDetId (simulation)
  • DataFormats/SiPixelDigi (simulation)
  • DataFormats/SiPixelRawData (simulation)
  • DataFormats/SiStripCluster (reconstruction)
  • DataFormats/SiStripCommon (reconstruction)
  • DataFormats/SiStripDetId (simulation)
  • DataFormats/SiStripDigi (simulation)
  • DataFormats/Streamer (core)
  • DataFormats/TCDS (daq, reconstruction)
  • DataFormats/TauReco (reconstruction)
  • DataFormats/TestObjects (core)
  • DataFormats/TotemReco (reconstruction)
  • DataFormats/TrackCandidate (reconstruction)
  • DataFormats/TrackReco (reconstruction)
  • DataFormats/TrackerCommon (reconstruction)
  • DataFormats/TrackerRecHit2D (reconstruction)
  • DataFormats/TrackingRecHit (reconstruction)
  • DataFormats/TrajectorySeed (reconstruction)
  • DataFormats/TrajectoryState (reconstruction)
  • DataFormats/V0Candidate (reconstruction)
  • DataFormats/VertexReco (reconstruction)
  • FWCore/Utilities (core)
  • FastSimDataFormats/External (fastsim)
  • FastSimDataFormats/NuclearInteractions (fastsim)
  • FastSimDataFormats/PileUpEvents (fastsim)
  • Fireworks/Calo (visualization)
  • Fireworks/Tracks (visualization)
  • HLTrigger/btau (hlt)
  • JetMETCorrections/Type1MET (reconstruction)
  • L1Trigger/GlobalTrigger (l1)
  • L1Trigger/GlobalTriggerAnalyzer (l1)
  • L1Trigger/L1THGCalUtilities (l1)
  • L1Trigger/L1TMuonBarrel (l1)
  • L1Trigger/Phase2L1ParticleFlow (l1)
  • PhysicsTools/NanoAOD (xpog)
  • PhysicsTools/SelectorUtils (reconstruction)
  • RecoEgamma/EgammaElectronProducers (reconstruction)
  • RecoLocalMuon/GEMCSCSegment (reconstruction)
  • RecoLocalMuon/RPCRecHit (reconstruction)
  • RecoMuon/MuonIdentification (reconstruction)
  • RecoMuon/MuonIsolation (reconstruction)
  • RecoMuon/TrackerSeedGenerator (reconstruction)
  • RecoTauTag/RecoTau (reconstruction)
  • RecoTracker/FinalTrackSelectors (reconstruction)
  • RecoTracker/TkSeedGenerator (reconstruction)
  • RecoTracker/TrackProducer (reconstruction)
  • SimDataFormats/Associations (simulation)
  • SimDataFormats/CaloAnalysis (simulation)
  • SimDataFormats/CaloHit (simulation)
  • SimDataFormats/CaloTest (simulation)
  • SimDataFormats/CrossingFrame (simulation)
  • SimDataFormats/DigiSimLinks (simulation)
  • SimDataFormats/EcalTestBeam (simulation)
  • SimDataFormats/EncodedEventId (simulation)
  • SimDataFormats/Forward (simulation)
  • SimDataFormats/GEMDigiSimLink (simulation)
  • SimDataFormats/GeneratorProducts (generators)
  • SimDataFormats/HcalTestBeam (simulation)
  • SimDataFormats/HiGenData (simulation)
  • SimDataFormats/PileupSummaryInfo (simulation)
  • SimDataFormats/RPCDigiSimLink (simulation)
  • SimDataFormats/RandomEngine (simulation)
  • SimDataFormats/Track (simulation)
  • SimDataFormats/TrackerDigiSimLink (simulation)
  • SimDataFormats/TrackingAnalysis (simulation)
  • SimDataFormats/TrackingHit (simulation)
  • SimDataFormats/ValidationFormats (simulation)
  • SimDataFormats/Vertex (simulation)
  • TBDataFormats/EcalTBObjects (analysis)
  • TBDataFormats/HcalTBObjects (analysis)
  • TauAnalysis/MCEmbeddingTools (simulation)
  • TrackingTools/GsfTracking (reconstruction)
  • Validation/RecoMuon (dqm)

@Alejandro1400, @BenjaminRS, @Dr15Jones, @JanChyczynski, @Martin-Grunewald, @Moanwar, @alja, @arunhep, @atpathak, @battibass, @civanch, @cmsbuild, @ctarricone, @emeschi, @ftorrresd, @gabrielmscampos, @jfernan2, @kpedro88, @lviliani, @makortel, @mandrenguyen, @mdhildreth, @mkirsano, @mmusich, @perrotta, @quinnanm, @rseidita, @sensrcn, @smorovic, @smuzaffar, @srimanob, @ssekmen, @theofil, @tvami can you please review it and eventually sign? Thanks.
@24LopezR, @AlexDeMoor, @CeliaFernandez, @Fedespring, @GiacomoSguazzoni, @HuguesBrun, @IzaakWN, @Martin-Grunewald, @Ming-Yan, @Prasant1993, @ReyerBand, @Sam-Harper, @Senphy, @VinInn, @VourMa, @a-kapoor, @abbiendi, @abdoulline, @afiqaize, @ahinzmann, @alesaggio, @alja, @amagitte, @amarini, @andrzejnovak, @apsallid, @argiro, @azotz, @battibass, @bellan, @bsunanda, @calderona, @castaned, @cericeci, @clelange, @cseez, @denizsun, @dgulhan, @dkotlins, @echabert, @elusian, @erikbutz, @eyigitba, @fabferro, @fabiocos, @felicepantaleo, @ferencek, @fsimone91, @giovanni-mocellin, @gkasieczka, @gouskos, @gpetruc, @grzanka, @hatakeyamak, @jainshilpi, @jbsauvan, @jdamgov, @jdolen, @jhgoh, @jlidrych, @jshlee, @lgray, @llunerti, @mandrenguyen, @mariadalfonso, @martinamalberti, @mbluj, @missirol, @mmarionncern, @mmasciov, @mmusich, @mroguljic, @mtosi, @nhanvtran, @pakhotin, @pavlo-kashko, @pfs, @philippgadow, @ptcox, @ram1123, @rappoccio, @rchatter, @robervalwalsh, @rociovilar, @rovere, @rsreds, @salimcerci, @sameasy, @sbein, @schoef, @seemasharmafnal, @sethzenz, @sobhatta, @sviret, @thomreis, @threus, @tlampen, @tocheng, @trocino, @tsusa, @valsdav, @vandreev11, @varuns23, @wang0jin, @watson-ij, @wddgit, @yetkinyilmaz, @yslai, @yuanchao this is something you requested to watch as well.
@ftenchini, @mandrenguyen, @sextonkennedy you are the release manager for this.

cms-bot commands are listed here

@makortel
Copy link
Copy Markdown
Contributor Author

@tvami
Copy link
Copy Markdown
Contributor

tvami commented May 19, 2026

+1

Comment thread DataFormats/EgammaCandidates/interface/GsfElectronCore.h Outdated
</class>
<class name="HFNoseDetId" ClassVersion="0">
<version ClassVersion="0" checksum="1469396081"/>
<class name="HFNoseDetId" ClassVersion="3">
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Isn't version 0 basically a do not store?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The version 0 was added in #23756. Given that all earlier *DetId types had version >= 3, and this one had the checksum, I assumed it was intended to be stored normally, and hence version 3.

@bsunanda Could you clarify the intent?

<class name="HFNoseTriggerDetId" ClassVersion="-1">
<version ClassVersion="-1" checksum="1686145543"/>
<version ClassVersion="0" checksum="1469396081"/>
<class name="HFNoseTriggerDetId" ClassVersion="3">
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Any idea what -1 was meant to be?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The versions 0 and -1 were added in #27858. Given that the earlier *DetId types had version >= 3, and this one had checksums, I assumed it was intended to be stored, and hence version 3.

@bsunanda Could you clarify the intent?

Comment thread DataFormats/L1TMuon/src/RegionalMuonShower.cc Outdated
Comment thread DataFormats/L1Trigger/src/EtSum.cc Outdated
Comment thread DataFormats/L1Trigger/src/L1Candidate.cc Outdated
Comment thread DataFormats/L1Trigger/src/Muon.cc Outdated
Comment thread DataFormats/L1Trigger/src/MuonShower.cc Outdated
Comment thread DataFormats/PatCandidates/src/PackedCandidate.cc Outdated
Comment thread DataFormats/PatCandidates/src/PackedGenParticle.cc Outdated
@smuzaffar
Copy link
Copy Markdown
Contributor

but this file (or package) is not touched in this PR, so I'm wondering why it is being reported here but e.g. not in the CLANG IB. I'll make a separate PR for it anyway (see #50971).

Bot reports warnings for files directly touched by PR or files which use (directly or indirectly) a file changed in the PR. I think in this case, one of changed header is included in Alignment/OfflineValidation/plugins/MuonAlignmentAnalyzer.cc

@makortel
Copy link
Copy Markdown
Contributor Author

@Dr15Jones' comments about changing the accented E and not all member functions in a source being inside io_v1 are addressed. Rebase to address the merge conflict in DataFormats/PatCandidates/interface/IsolatedTrack.h will follow.

The io_v1 namespace allows schema evolution of complex cases (e.g.
pointed-to container type of edm::Ref being changed) that ROOT's
schema evolution can not support.

Moving classes to io_v1 breaks backwards compatibility for these types
that was agreed to be done in CMSSW_20_0_0 that serves a start of
Phase 2 (only) targeting software.

Taking advantage of the backwards compatiblity break, the class
versions are reset to 3.

In addition, include two complex schema evolution cases that were too
difficult earlier
- Make PFCandidate to no longer inherit from CompositeCandidate
- Reduce use of RangeMap in muon code
@makortel makortel force-pushed the evoIntegrationDataFormats branch from 7b40d16 to 88e2876 Compare May 20, 2026 21:52
@makortel
Copy link
Copy Markdown
Contributor Author

Rebased to fix the conflict

@cmsbuild
Copy link
Copy Markdown
Contributor

+code-checks

Logs: https://cmssdt.cern.ch/SDT/code-checks/cms-sw-PR-50968/49426

@cmsbuild
Copy link
Copy Markdown
Contributor

@makortel
Copy link
Copy Markdown
Contributor Author

@makortel
Copy link
Copy Markdown
Contributor Author

@cmsbuild, please test

@cmsbuild
Copy link
Copy Markdown
Contributor

-1

Failed Tests: Build ClangBuild
Size: This PR adds an extra 40KB to repository
Summary: https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53399/summary.html
COMMIT: 88e2876
CMSSW: CMSSW_17_0_X_2026-05-20-1100/el8_amd64_gcc13
User test area: For local testing, you can use /cvmfs/cms-ci.cern.ch/week0/cms-sw/cmssw/50968/53399/install.sh to create a dev area with all the needed externals and cmssw changes.

The following merge commits were also included on top of IB + this PR after doing git cms-merge-topic:

You can see more details here:
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53399/git-recent-commits.json
https://cmssdt.cern.ch/SDT/jenkins-artifacts/pull-request-integration/PR-3887be/53399/git-merge-result

Failed Build

I found compilation error when building:

++ SCRAM_NOEDM_CHECKS=yes
++ /usr/bin/time -v scram build -v -k -j 16
***WARNING: Multiple usage of "DataFormatsL1DTTrackFinder". Please cleanup "lib" in "export" section of "src/DataFormats/L1DTTrackFinder/BuildFile.xml"
>> Local Products Rules ..... started
>> Local Products Rules ..... done
gmake: Circular tmp/el8_amd64_gcc13/cache/cuda_dlink/RecoTrackerLSTCoreCudaAsync <- tmp/el8_amd64_gcc13/cache/cuda_dlink/RecoTrackerLSTCoreCudaAsync dependency dropped.
gmake: Circular tmp/el8_amd64_gcc13/cache/rocm_dlink/RecoTrackerLSTCoreROCmAsync <- tmp/el8_amd64_gcc13/cache/rocm_dlink/RecoTrackerLSTCoreROCmAsync dependency dropped.
>> Building CMSSW version CMSSW_17_0_X_2026-05-20-1100 ----
>> Creating project symlinks
>> Entering Package Alignment/CommonAlignmentProducer
>> Entering Package CondFormats/Alignment


Failed Clang Build

I found compilation error while trying to compile with clang. Command used:

USER_CUDA_FLAGS='--expt-relaxed-constexpr' USER_CXXFLAGS='-Wno-register -fsyntax-only' /usr/bin/time -v scram build -k -j 32 COMPILER='llvm compile'

>> Entering Package TopQuarkAnalysis/TopSkimming
>> Entering Package TopQuarkAnalysis/TopTools
>> Entering Package TrackPropagation/Geant4e
>> Entering Package Utilities/ReleaseScripts
>> Compile sequence completed for CMSSW CMSSW_17_0_X_2026-05-20-1100
gmake: *** [There are compilation/build errors. Please see the detail log above.] Error 1
Command exited with non-zero status 1
	Command being timed: "scram build -k -j 32 COMPILER=llvm compile BUILD_LOG=yes"
	User time (seconds): 10984.20
	System time (seconds): 1170.33
	Percent of CPU this job got: 1360%


Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment