Merge pull request #19 from oer-particle-physics/gh-pages

module updates

Author: clelange

README.md

@@ -1,9 +1,13 @@
-# FIXME Lesson title
-
+# GitLab CI/CD in CMS(SW)
 [![Create a Slack Account with us](https://img.shields.io/badge/Create_Slack_Account-The_Carpentries-071159.svg)](https://swc-slack-invite.herokuapp.com/)
 
 This repository generates the corresponding lesson website from [The Carpentries](https://carpentries.org/) repertoire of lessons. 
 
+> **Note**
+> Click [here](https://oer-particle-physics.github.io/gitlab-cms) for the training website!
+
+The aim of this module is to learn how to run CMS software workflows (CMSSW) in a reproducible and secure way using the GitLab installation at CERN.
+
 ## Contributing
 
 We welcome all contributions to improve the lesson! Maintainers will do their best to help you if you have any
@@ -19,15 +23,11 @@ nicely explained in the chapter [Contributing to a Project](http://git-scm.com/b
 by Scott Chacon.
 Look for the tag ![good_first_issue](https://img.shields.io/badge/-good%20first%20issue-gold.svg). This indicates that the maintainers will welcome a pull request fixing this issue.  
 
-
 ## Maintainer(s)
 
 Current maintainers of this lesson are 
 
-* FIXME
-* FIXME
-* FIXME
-
+* [Clemens Lange](https://clange.ch/)
 
 ## Authors
 
@@ -37,4 +37,9 @@ A list of contributors to the lesson can be found in [AUTHORS](AUTHORS)
 
 To cite this lesson, please consult with [CITATION](CITATION)
 
+## Open Educational Resources (OER) on Zenodo
+
+This lesson is included in the ETH Domain Open Educational Resources for Research Data Management (RDM) community on Zenodo. These resources support researchers, students, and RDM staff in implementing best practices across the research data lifecycle. The materials developed here are published under open licenses (CC BY 4.0) and can be freely reused and adapted for teaching and training initiatives worldwide.
+
+
 [lesson-example]: https://carpentries.github.io/lesson-example

_config.yml

@@ -10,7 +10,7 @@
 carpentry: "swc"
 
 # Overall title for pages.
-title: "GitLab CI for CMS"
+title: "GitLab CI/CD in CMS(SW)"
 
 # Life cycle stage of the lesson
 # possible values: "pre-alpha", "alpha", "beta", "stable"

_episodes/01-introduction.md

@@ -1,13 +1,13 @@
 ---
-title: "Setting up an environment using CVMFS in GitLab CI (e.g. to run CMSSW)"
+title: "Setting up CMSSW in GitLab CI with CernVM File System (CVMFS)"
 teaching: 10
 exercises: 10
 questions:
 - "Which GitLab runners are needed?"
 - "What's different w.r.t. LXPLUS?"
 objectives:
-- "Know how to source the CMSSW environment"
-- "Understand the different commands that need to be used"
+- "Know how to source the CMSSW environment."
+- "Understand the different commands that need to be used."
 keypoints:
 - "Special GitLab CVMFS runners are required to run CI jobs that need CVMFS, e.g. to run CMSSW."
 - "If the setup script tries to access unset variables, then that can cause the CI to fail when using strict shell scripting checks."
@@ -18,19 +18,19 @@ CI/CD and also CERN-specific information:
 - [GitLab CI/CD documentation][gitlab-ci]
 - [CERN Knowledge Base Articles for the Git Service][snow-git]
 
-These pages serve as a good entrypoint in case of problems and questions.
+These pages serve as a good entry point in case of problems and questions.
 
 > ## Create a new GitLab project to follow along
 > Please [create a new GitLab project][gitlab-newproject] now to follow along.
 > To do so, select "Create blank project" and provide a project name.
-> You can for instance call it `awesome-gitlab-cms`. 
+> You can, for instance, call it `awesome-gitlab-cms`. 
 > You need to specify the project visibility level. Private, the default, is fine for this tutorial.
-> In the following, we will assume that all your work is in a directory called `awesome-workshop` in your
-> home directory and we will now clone the newly created project in a local repository therein:
+> In the following, we will assume that all your work is into a directory called `awesome-workshop` in your
+> home directory, and we will now clone the newly created project in a local repository therein:
 > `~/awesome-workshop/awesome-gitlab-cms`
 {: .callout}
 
-The commands would look like this (replace `${USER}` by your CERN
+The commands would look like this (replace `${USER}` with your CERN
 username in case it isn't the same as on your laptop):
 
 ~~~
@@ -43,9 +43,9 @@ cd awesome-gitlab-cms
 
 ## Choosing the correct GitLab runner
 
-We are now going to setup a GitLab CI. For that, we need to create a `.gitlab-ci.yml` file.
+We are now going to set up a GitLab CI. For that, we need to create a `.gitlab-ci.yml` file.
 Standard [GitLab CI runners at CERN](https://gitlab.docs.cern.ch/docs/Build%20your%20application/CI-CD/Runners/)
-do not mount CVMFS, which is required in many cases, for example for setting up CMSSW, to create a grid proxy, or to access LCG software stacks in `/cvmfs/sft.cern.ch/`.
+do not mount CVMFS, which is required in many cases, for example, for setting up CMSSW, to create a grid proxy, or to access LCG software stacks in `/cvmfs/sft.cern.ch/`.
 In order to get a runner that mounts CVMFS, you need
 to add a `tag` to your `gitlab-ci.yml` file:
 
@@ -69,7 +69,7 @@ cmssw_setup:
 The `cmssw_setup` line defines the name of the job, and all the job does is
 list `/cvmfs/cms.cern.ch/`, which would fail if CVMFS isn't mounted. 
 
-To trigger the pipeline we need to commit the file and push it to GitLab:
+To trigger the pipeline, we need to commit the file and push it to GitLab:
 
 ~~~
 git add .gitlab-ci.yml
@@ -78,12 +78,12 @@ git push
 ~~~
 {: .language-bash}
 
-If you now navigate the GitLab UI with your browser you will see the CI running and eventually finishing.
+If you now navigate the GitLab UI with your browser, you will see the CI running and eventually finishing.
 You can check the output, and also the `cvmfs` label:
 
 ![A job with a GitLab CVMFS Runner showing the cvmfs label](../fig/cvmfs_tag.png)
 
-In the following you'll will learn how to setup a GitLab CI job that runs CMSSW.
+In the following, you'll learn how to set up a GitLab CI job that runs CMSSW.
 
 This should be regarded as an example for any CI job requiring access to CVMFS and accessing CMS-restricted files.
 
@@ -108,7 +108,7 @@ cmsenv
 ~~~
 {: .language-bash}
 
-The first command is needed because CMSSW_10_6_30 is pretty old (we chose an old one on purpose!) and it does not have any build for the recent alma9 LXPLUS. 
+The first command is needed because CMSSW_10_6_30 is pretty old (we chose an old one on purpose!), and it does not have any build for the recent alma9 LXPLUS. 
 So we need to start a CentOS7 container first, which we do with the `cmssw-el7` command, as described in the [CMS singularity guide](https://cms-sw.github.io/singularity.html).
 
 Depending on the software version chosen, the third command may print out a warning such as
@@ -150,14 +150,14 @@ of this command is that a few helper functions are defined, such as `cmsrel` and
 
 A common pitfall when setting up CMSSW in GitLab is that the execution
 fails because the setup script doesn't follow best practices for shell
-scripts such as returning non-zero return values even if the setup is OK or
-using unset variables. Even if the script exits without visible error message,
+scripts, such as returning non-zero return values even if the setup is OK or
+using unset variables. Even if the script exits without a visible error message,
 there could be something wrong. It is therefore often a good idea to
 circumvent issues like that by disabling strict checks (issuing `set +u`) before running the
 setup command and enabling these checks afterwards again (issuing `set -u`).
 
 > ## Exercise: Set up CMSSW in GitLab
-> Knowing all this, can you write the `.gitlab-ci.yml` file to set up CMSSW in GitLab starting from the fragment above and check if this is all working by executing `cmsRun --help` at the end?
+> Knowing all this, can you write the `.gitlab-ci.yml` file to set up CMSSW in GitLab, starting from the fragment above, and check if this is all working by executing `cmsRun --help` at the end?
 {: .challenge}
 
 > ## Solution: Set up CMSSW in GitLab
@@ -181,8 +181,8 @@ setup command and enabling these checks afterwards again (issuing `set -u`).
 > ~~~
 > {: .language-yaml}
 >
-> The `image` directive tells the gitlab runner that it should run in a CentOS7 container, just like you would manually do on LXPLUS issuing `cmssw-el7`.
-> The `set +u` command turns off errors for referencing unset variables. It isn't really needed here, since `-u` (i.e. not allowing to use unset variables) isn't set by default, but the script would fail if one used `set -u` somewhere else, so it's safer to catch this here.
+> The `image` directive tells the GitLab runner that it should run in a CentOS7 container, just like you would manually do on LXPLUS issuing `cmssw-el7`.
+> The `set +u` command turns off errors for referencing unset variables. It isn't really needed here, since `-u` (i.e., not allowing to use unset variables) isn't set by default, but the script would fail if one used `set -u` somewhere else, so it's safer to catch this here.
 {: .solution}
 
 The reason why in the example above the variable `${CMS_PATH}` is used and not simply

_episodes/02-compiling.md

@@ -6,7 +6,7 @@ questions:
 - "How can I compile my CMSSW package using GitLab CI?"
 - "How do I add other CMSSW packages?"
 objectives:
-- "Successfully compile CMSSW example analysis code in GitLab CI"
+- "Successfully compile CMSSW example analysis code in GitLab CI."
 keypoints:
 - "For code to be compiled in CMSSW, it needs to reside within the work area's `src` directory."
 - "The analysis code from the Git repository is copied to the CMSSW work area in the CI script."
@@ -17,20 +17,20 @@ Now that you know how to get a CMSSW environment, it is time to do something use
 ## Compiling code within the repository
 
 For your analysis to be compiled with CMSSW, it needs to reside in the
-workarea's `src` directory, and in there follow the directory structure of
-two subdirectories (e.g. `AnalysisCode/MyAnalysis`) within which there can be
+workarea's `src` directory, and, within it, follow the directory structure of
+two subdirectories (e.g., `AnalysisCode/MyAnalysis`) in which there can be
 `src`, `interface`, `plugin` and further directories. Your analysis code
 (under version control in GitLab/GitHub) will usually not contain the
 CMSSW workarea. The git repository will either
 contain the analysis code at the lowest level or have a subdirectory 
-for the analysis code, to disentangle it from your configuration files such as the
+for the analysis code, to disentangle it from your configuration files, such as the
 `.gitlab-ci.yml` file.
 
 We will use an example analysis, which selects pairs of electrons and muons.
 [Download the zip file containing the analysis](../files/ZPeakAnalysis.zip)
 and extract it now. The analysis code is
 in a directory called `ZPeakAnalysis` within which `plugins` (the C++ code)
-and `test` (the python config) directories reside.
+and `test` (the Python config) directories reside.
 Add this directory to your repository:
 
 ~~~
@@ -43,7 +43,7 @@ git commit -m "Add ZPeakAnalysis"
 {: .language-bash}
 
 When compiling the code in a GitLab pipeline, the `ZPeakAnalysis` needs
-to be copied into the CMSSW workarea, and it's advisable to use environment
+to be copied into the CMSSW work area, and it's advisable to use environment
 variables for this purpose. This would be achieved like this:
 
 ~~~
@@ -52,7 +52,7 @@ cp -r "${CI_PROJECT_DIR}/ZPeakAnalysis" "${CMSSW_BASE}/src/AnalysisCode/"
 ~~~
 {: .language-bash}
 
-With these two commands we will now be able to extend the `.gitlab-ci.yml`
+With these two commands, we will now be able to extend the `.gitlab-ci.yml`
 file such that we can compile our analysis code in GitLab. To improve the
 readability of the file, the `CMSSW_RELEASE` is defined as a variable:
 
@@ -87,15 +87,15 @@ cmssw_compile:
 
 > ## Always add CMSSW packages before compiling analysis code!
 >
-> Adding CMSSW packages has to happen *before* compiling analysis code in the
+> Adding CMSSW packages has to happen *before* compiling the analysis code in the
 > repository, since `git cms-addpkg` will call `git cms-init` for the
 > `$CMSSW_BASE/src` directory, and `git init` doesn't work if the directory
 > already contains files.
 {: .callout}
 
 When developing CMSSW code, you will sometimes find yourself in the situation where you need
 to rebuild one of the CMSSW packages. 
-This need can arise either from the fact you are modifying this package, 
+This need can arise either from the fact that you are modifying this package, 
 or from the fact that another CMSSW package depends on the one you are developing
 and thus needs to be rebuilt.
 
@@ -131,7 +131,7 @@ There are a couple of options to make things work:
 For simplicity, and since we do not need to commit anything back to CMSSW from
 GitLab, we will use the latter approach.
 A complete `yaml` fragment that checks out a CMSSW package after having set up
-CMSSW and then compiles the code looks as follows:
+CMSSW and then compiles the code, which looks as follows:
 
 ~~~
 cmssw_addpkg:
@@ -164,17 +164,17 @@ The additional two variables that are exported here, `CMSSW_MIRROR` and
 `CMSSW_GIT_REFERENCE` can speed up interaction with git, in particular
 faster package checkouts. The `CMSSW_MIRROR` points to a GitHub mirror within the CERN network,
 accessible only from machines within the same network, which is the case for the GitLab runners.
-Settings these variables is *not* mandatory.
+Setting these variables is *not* mandatory.
 
-> ## Bonus: have you tried putting both `cmssw_compile` and `cmssw_addpkg` in the same `.gitlab-ci.yml` file?
+> ## Bonus: Have you tried putting both `cmssw_compile` and `cmssw_addpkg` in the same `.gitlab-ci.yml` file?
 > If you do that, the two jobs will run in parallel.
-> GitLab CI also allows running jobs in stages. Stages define group of jobs running together.
+> GitLab CI also allows running jobs in stages. Stages define a group of jobs running together.
 > To assign a job to a stage, just add `stage: some_name` in its definition.
 > Stages run in sequence according to the order typically defined at the beginning of your `.gitlab-ci.yml`.
 > If any job in a stage fails, the next stage will not run.
 {: .testimonial}
 
-> ## Exercise: can you put `cmssw_compile` and `cmssw_addpkg` in two stages?
+> ## Exercise: Can you put `cmssw_compile` and `cmssw_addpkg` in two stages?
 > 
 {: .challenge}
 
@@ -226,7 +226,7 @@ Settings these variables is *not* mandatory.
 >     - scram b
 > ~~~
 > {: language-yaml}
-> In the solution above you will also notice that we have moved the definition of some variables
+> In the solution above, you will also notice that we have moved the definition of some variables
 > outside of the job definition, because they are used by both jobs.
 {: .solution}
 

_episodes/03-vomsproxy.md

@@ -5,12 +5,12 @@ exercises: 15
 questions:
 - "How can I obtain a grid proxy in GitLab?"
 objectives:
-- "Securely add grid proxy certificates and passwords to GitLab"
-- "Successfully obtain a grid proxy for the CMS VO"
+- "Securely add grid proxy certificates and passwords to GitLab."
+- "Successfully obtain a grid proxy for the CMS VO."
 keypoints:
-- "Special care is needed when adding secrets in GitLab"
-- "Passwords and certificates should always be set to `Protected` state"
-- "Certificates need to be `base64`-encoded for use as secrets"
+- "Special care is needed when adding secrets in GitLab."
+- "Passwords and certificates should always be set to `Protected` state."
+- "Certificates need to be `base64`-encoded for use as secrets."
 ---
 
 ## Securely adding passwords and files to GitLab
@@ -22,24 +22,24 @@ files, which by default will reside in the `~/.globus` directory, will need
 to be stored in GitLab.
 
 > ## Keep your secrets secret!
-> Please be extra careful when it comes to your account and grid passwords as
+> Please be extra careful when it comes to your account and grid passwords, as
 > well as your certificates! They should never be put in any public place.
 > Putting them under version control is risky, since even if you delete them
 > from the `HEAD` of your `master` branch, they will still be in the commit
-> history. Furthermore, putting them in a public, or even a private but shared
-> repository, is a violation of grid policy, and could lead to access being
+> history. Furthermore, putting them in a public or even a private but shared
+> repository is a violation of grid policy and could lead to access being
 > revoked for the offending user. Should you accidentally have put sensitive
 > data to a repository, please see the guides by [GitHub][removing-sensitive-github]
 > and [GitLab][removing-sensitive-gitlab] to
 > remove them (though the data should still be considered compromised).
 {: .callout}
 
-For more information see the section on
+For more information, see the section on
 [private information/access control][lesson-gitlab-secrets]
 from the
 [Continuous Integration / Continuous Development (CI/CD)][lesson-gitlab]
-on how to add variables in GitLab CI/CD in general. From that lesson you will
-know how to add e.g. your grid proxy password. The grid certificate itself,
+on how to add variables in GitLab CI/CD in general. From that lesson, you will
+know how to add e.g., your grid proxy password. The grid certificate itself,
 however, consists of two files that look like this:
 
 ~~~
@@ -64,7 +64,7 @@ TH1s1SNT4R34lGr1DC3rt1f1C4t3But1Th4s4l3NgtH0F64CH4r4ct3rSP3rL1N3
 ### We need more base: `base64`
 
 Simply pasting them into GitLab does not work since the line breaks will not
-be reflected correctly. There is a trick we can play though: we can encode the
+be reflected correctly. There is a trick we can play, though: we can encode the
 files including line breaks so that they are simply a string, which we can
 decode to yield the same result as the input. The tool of our choice is
 `base64`. Let's give this a go.
@@ -73,7 +73,7 @@ decode to yield the same result as the input. The tool of our choice is
 >
 > Copy the output of the `cat ~/.globus/usercert.pem` output above into a
 > text file called `testcert.txt`, and pipe the content of this file to the
-> `base64` command or use it as input file directly (hint: `base64 --help`).
+> `base64` command or use it as an input file directly (hint: `base64 --help`).
 >
 {: .challenge}
 
@@ -139,12 +139,12 @@ There are a couple of important things to keep in mind when adding passwords
 and certificates as variables to GitLab:
 
 - Variables should always be set to `Protected` state.
-- As an additional safety measure, set them as `Masked` as well if possible (this will not work for the certificates but should for your grid password).
+- As an additional safety measure, set them as `Masked` as well if possible (this will not work for the certificates, but should for your grid password).
 
 For more details, see the
 [GitLab CI/CD variables][gitlab-variables-advanced]
 of the GitLab documentation. Setting variables to `Protected` means that
-they are only available in protected branches, e.g. your `master` branch.
+they are only available in protected branches, e.g., your `master` branch.
 This is important when collaborating with others, since anyone with access
 could just `echo` the variables when making a merge request if you run
 automated tests on merge requests.
@@ -181,7 +181,7 @@ base64 -i ~/.globus/userkey.pem -w 0
 and copy the output into GitLab.
 
 > ## Every equal sign counts!
-> Make sure to copy the full string including the trailing equal signs.
+> Make sure to copy the full string, including the trailing equal signs.
 {: .callout}
 
 The `Settings` --> `CI / CD` --> `Variables` section should look like this:
@@ -242,7 +242,7 @@ voms_proxy_test:
 ~~~
 {: .language-yaml}
 
-You could take this further by e.g. performing a DAS query to keep your input files
+You could take this further, e.g., by performing a DAS query to keep your input files
 up-to-date.
 
 Confirm that this works for you before moving on to the next section!