Skip to content

[Woptim] update local packages#793

Merged
davidbeckingsale merged 60 commits into
developfrom
woptim/update-local-packages
Jan 10, 2023
Merged

[Woptim] update local packages#793
davidbeckingsale merged 60 commits into
developfrom
woptim/update-local-packages

Conversation

@adrienbernede

@adrienbernede adrienbernede commented Nov 10, 2022 •
edited
Loading

Copy link
Copy Markdown
Member

It looks like the local and upstream Umpire Spack packages diverged about 3 years ago.

For the RAJA package update, my strategy was to take the upstream package, and get it to pass the CI.
For the Umpire one, the differences are greater. I am taking the upstream Package, adding the many variants for the local one, and some changes from the recent RAJA packages release changes (newer BLT, CMake requirements) and then I will make sure the CI passes.

List of changes:

  • Most recent versions
  • Turn off submodules for recent versions
  • Add patches from upstream Spack packages
  • Adding new variants from local package
  • Changing some variant defaults: shared, tests, device_alloc -> had to update the CI specs accordingly
  • Adding CMake and BLT requirements similar to RAJA -> guessed that requirements were the same
  • Merge compiler and toolchain options (gcc, hip, cuda)
  • Unified use of “ instead of ‘.
  • Verify that the CMake options prefix is correctly set regarding Umpire versions.
  • Pass list of arch to CMAKE_HIP_ARCHITECTURES
  • Default to ~tools, turn it on in CI except on corona. Add conflict +tools when +rocm.

updates:

  • spack@0.18.1 -> spack@0.19.0.
  • radiuss-spack-configs updated accordingly to support new Spack syntax.
  • radiuss-shared-ci@2022.09.0 -> radiuss-shared-ci@2022.12.0rc.

newer radiuss-shared-ci update brings:

  • support for spack@0.19.1
  • improved performance (not more project clone in status jobs)
  • improved logs (prints a complete reproducer of the current job)
  • jobs will not be triggered on a machine that is not available
  • possibility to prevent CI to run on Draft PRs.

Notes:

  • Attempt to set hip as CXX compiler failed: it’s causing a CMake error which is not visible in RAJA... under investigation

Relates to:
llnl/RAJA#1338
llnl/CHAI#218

@adrienbernede
Remove unnecessary and inaccurate cores per task, simplify
246f9a8
@adrienbernede
Remove SLURM_OVERLAP option to attempt to use --overlap instead
b3bbf34
@adrienbernede
Merge branch 'develop' into woptim/ci/allocations
2f37258
@adrienbernede
Fix sharing corona job
405a65b
@adrienbernede
Update radiuss-spack-configs and CI to run with hip@5.1.1 on corona
fb5cb4a
@adrienbernede
Merge branch 'woptim/hip-511-ci' into woptim/ci/allocations
3f2e9d5
@adrienbernede
Adapt to branch reorg in radiuss-shared-ci
438b220
@adrienbernede
Merge branch 'develop' into woptim/ci/allocations
dcbcf6c
@adrienbernede
Merge branch 'develop' into woptim/ci/allocations
708b9ea
@adrienbernede
Merge branch 'develop' into woptim/ci/allocations
0d59f17
@adrienbernede
Use new tag in radiuss-shared-ci
49a42b7
@adrienbernede
Changelog
47a39a3
@adrienbernede
Update documentation regarding Gitlab CI
da25a90
@adrienbernede
Try spack@0.18.1 to sync with RAJA and Co.
d6adfff
@adrienbernede
Update radiuss-shared-ci to only use xl@16.1.1.11
3a4bfe2
@adrienbernede
Fix wrong radiuss-spack-configs version
04373c9
@adrienbernede
Merge branch 'woptim/update-to-xl_16_1_1_11' into woptim/raja-umpire-…
84651d1 
…sync
@adrienbernede
Merge branch 'woptim/use-spack-0.18.1' into woptim/raja-umpire-sync
f2e74c4
@adrienbernede
Changelog
2010420
@adrienbernede
Naming covention + update to xl@16.1.1.12 + restore ~fortran as default
f1a7c58
@adrienbernede
Fix typo
1ee34dc
@adrienbernede
progressively reverting to initial config of failing spec
a45b65d
@adrienbernede
Adapt CI to new layout fixing variable overriding issue
d7b130e
@adrienbernede
Fix file name
2c66c6f
@adrienbernede
Extra quotes and consistent comment
e4635a4
@adrienbernede
Point to official radiuss-shared-ci release
5262412
@adrienbernede
Fix variable name in CI, remove unnecessary code in build-and-test, u…
c73e078 
…se shebang to find appropriate python for uberenv
@adrienbernede
Sync camp package with RAJA copy
c4b9642
@adrienbernede
Fix job names + fix custom vars and deps for lassen
4db8a79
@adrienbernede
Sort jobs to highlight the overridden specs
5b1ec64
This was referenced Nov 14, 2022
Merged
Merged
@adrienbernede

adrienbernede commented Nov 14, 2022

Copy link
Copy Markdown
Member Author

@davidbeckingsale your feedback is welcome on this.
This seems ready, but I’m currently updating the CHAI package and will update the Umpire one if there are inconsistencies.
Also, it’s a good time for change requests.

@adrienbernede adrienbernede mentioned this pull request Nov 29, 2022
Closed
@adrienbernede adrienbernede force-pushed the woptim/update-local-packages branch from 0d68e95 to 1ccda57 Compare November 29, 2022 14:03
@adrienbernede
Woptim/update shared ci (#797)
e56b322 
* Update to spack@v0.19.0

* Move to new Spack syntax to propagate flags

* Point to new shared-ci branch to test

* Update .gitlab-ci.yml using shared CI template

* Update gitlab config using shared one

* Move to release candidate branch in radiuss-shared-ci repo

* Move local packages to radiuss-spack-configs

* Test sharing the config with umpire, using camp package

* Update with fix

* Update with fix

* Update with fix

* Update with fix

* Update with fix

* Update with improved CHAI package

* Update with latest RAJA package

* Allow failure for xl@default +openmp_target

* Update radiuss-spack-configs

* Point to radiuss-shared-ci latest release tag (no code change)

* Activate Tioga CI

* Add missing tioga extra jobs file

* Add missing tioga variables
@adrienbernede

adrienbernede commented Dec 15, 2022

Copy link
Copy Markdown
Member Author

@davidbeckingsale sorry for the confusion, you’ll have to approve that one (I merged #797 in it).

@adrienbernede

adrienbernede commented Jan 2, 2023

Copy link
Copy Markdown
Member Author

@davidbeckingsale I’m confident the CI should pass this time.

@adrienbernede adrienbernede reopened this Jan 2, 2023
@mcfadden8

mcfadden8 commented Jan 10, 2023

Copy link
Copy Markdown
Collaborator

Hi @adrienbernede,

The notes mention a hip/CXX problem that is still under investigation. Did this get resolved? Is it required for this ticket?

@adrienbernede

adrienbernede commented Jan 10, 2023

Copy link
Copy Markdown
Member Author

@mcfadden8 Good point, and thank you for reading through the notes 👍🏻 .

In facts, this should not prevent the merge: The issue is that I cannot mimic exactly the way RAJA is handling the hip compiler in the RAJA package. But I feel like there is something to understand.

This is mostly a remark for you and @davidbeckingsale who will likely understand the reasons of this better than me.

mcfadden8
mcfadden8 approved these changes Jan 10, 2023
View reviewed changes

@mcfadden8 mcfadden8 left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Nice work @adrienbernede - I had one question, but if we decide to do anything about it, I think it should probably be done as a separate activity.

@davidbeckingsale davidbeckingsale merged commit 7fbcf58 into develop Jan 10, 2023
@davidbeckingsale davidbeckingsale deleted the woptim/update-local-packages branch January 10, 2023 19:14
@adrienbernede

adrienbernede commented Jan 10, 2023

Copy link
Copy Markdown
Member Author

Nice work @adrienbernede - I had one question, but if we decide to do anything about it, I think it should probably be done as a separate activity.

Please open an issue and mention me then ;).

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@mcfadden8 mcfadden8 mcfadden8 approved these changes
@davidbeckingsale davidbeckingsale davidbeckingsale left review comments

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@adrienbernede @mcfadden8 @davidbeckingsale