Skip to content

vm-setup.rst

Latest commit

 

History

History
427 lines (319 loc) · 16.5 KB

vm-setup.rst

File metadata and controls

427 lines (319 loc) · 16.5 KB
Raw

Virtual machine setup

CERN VMM virtual machine

  1. Request a VM from VMM

    Go to vmm.cern.ch, click "Request a Virtual Machine", and fill in the fields as follows. Submit the request and wait for an e-mail telling you the machine is ready. If you need to share the VM with other users, create an e-group and set it as the owner, and all the accounts in that e-group will be able to ssh and sudo on the VM.

    • Owner and main user: you
    • Name and description: (something pertinent)
    • Operating system: SLC5 - x86_64
    • Memory 2 GB (not less!), CPUs: 1

  2. Basic system install

    SSH to your VM and run the following commands to do basic system setup. It requests a host certificate using your ~/.globus/user{cert,key}.pem, so make sure those files are up to date. Running Deploy will ask for your user certificate password. If your certificate was issued by an authority other than CERN CA, associate your CERN account to the certificate before attempting to run this step. While doing the association, do NOT delete Kerberosservice if it exists. For the certificate generation to succeed you must also be the owner or the responsible of the VM in VMM.:

    sudo yum install git.x86_64
mkdir -p /tmp/foo
cd /tmp/foo
(git clone git://github.com/dmwm/deployment.git cfg && cd cfg && git reset --hard HG1401a)
sudo -l  # this is so the following won't prompt
cfg/Deploy -t dummy -s post $PWD system/devvm
# OPTIONAL: review what happened: less /tmp/foo/.deploy/*
rm -fr /tmp/foo

    If the above fails when running careq-cern-ch, then you can acquire host certificate manually. Here are the steps you need to do:

    1. Generate private key/certificate with the following command:

      openssl req -new -subj "/CN=`hostname -f`" -out newcsr.csr -nodes -sha512 -newkey rsa:2048

    2. Send the request to CERN CA: go to New Host Certificate on https://gridca.cern.ch/gridca/ web site and select host for which you request a certificate. Copy/Paste in the text box the content of file newcsr.csr generated by OpenSSL and click Submit. Click Download Certificate Base 64 Encoded and save the file in the same folder as newcsr.csr. Then run the following set of commands:

      sudo mv host.cert /etc/grid-security/hostcert.pem
sudo mv privkey.pem /etc/grid-security/hostkey.pem
sudo chmod 400 /etc/grid-security/hostkey.pem
sudo chown root:root /etc/grid-security/host{cert,key}.pem
rm newcsr.csr

    3. Then run again the Basic system install.

  3. Request proxy renewal rights for your VM

    This step is optional.

    If your server needs a proxy certificate to work, request the host to be included in myproxy.cern.ch. Write a message something like the one below, where HOSTNAME is the full host nameas shown by hostname -f. Run openssl x509 -in /etc/grid-security/hostcert.pem -noout -subject and verify the host certificate is correct in the message body! The request usually takes one to several days to fulfill. In the mean time you can continue to use the machine, but you have to renew the proxy manually as per step 8.

    From: Your Name <you@cern.ch>
To: px.support@cern.ch
Cc: cms-service-webtools@cern.ch
Subject: myproxy registration request for HOSTNAME

Could you please add the following host certificate to myproxy.cern.ch
trusted_retrievers, authorized_retrievers, authorized_renewers policy?
This is a development server for CMS web services and requires use of
grid proxy certificates.

/DC=ch/DC=cern/OU=computers/CN=HOSTNAME

Regards,
Your Name

  4. Log out

    Log out. You may optionally reboot the server, but it isn't necessary.

    If you use SSH persistent connections, be sure to terminate the master connection before logging in again. The commands in step 2 modify users and groups, and it's important they apply when you execute the next steps. This requires completely new log-in, and using persistent SSH connections may cause you to use cached credentials.

  5. Log in

    SSH into your VM again. Run the following to double check you got the multiple service accounts.:

    id # should print out large number local _foo groups now

  6. Set up authentication

    This step is optional.

    If you develop a server with secrets, grab them now. You only need the secrets for the servers you manage, such as /data/auth/t0wmadatasvc/t0auth.py. However you do not need this at all - you can just run step 7, which will then create dummy auth info, which you can then overwrite with real data. If you do create the auth directory, it must be adequately protected and _sw group readable, so do run all the chmod/chgrp commands shown below.

    mkdir -p /data/auth
mkdir -p /data/auth/myservice
vi /data/auth/myservice/mysecret # e.g. /data/auth/t0wmadatasvc/t0auth.py
# or grab them all:
# rsync -avu cmsweb@vocmsNNN.cern.ch:private/conf/ /data/auth/
chgrp -R _sw /data/auth
chmod ug=r,o-rwx $(find /data/auth -type f)
chmod u=rwx,g=rx,o-rwx $(find /data/auth -type d)

    Note that you normally create the info by hand, instead of copying from the cmsweb account as only the admins can do the latter. The important thing is you set up directory structure /data/auth with the secrets info that you need, for only those services you plan to install. Also note that you need to create a blank /data/auth/wmcore directory if you are deploying any wmcore based service or deploying the frontend. This directory holds the key used by both frontends and backend services to hmac verify the http headers. A blank directory there tells the deploy to generate a new key.

  7. Software installation

    The following installs standard multi-account setup using the deployment/configuration from the HG1401a release, and the corresponding RPMs from the comp.pre repository. You may pick up any release tag you need. The list of comp release tags can be found here.

    You could overwrite specific service versions using @theversion following each service name. I.e. das@2.3.1-comp2.

    Note you will be asked for the privkey passphrase in case your service requires a proxy certificate to work.

    If you did not do step 6, drop the ``-a $PWD/auth`` option.

    # Get the configuration
(cd /data; git clone git://github.com/dmwm/deployment.git cfg && cd cfg && git reset --hard HG1401a)

# Use "-R cmsweb@$VER" instead of "-R comp@$VER" below for HG13* or older tags
(VER=HG1401a REPO="-r comp=comp.pre" A=/data/cfg/admin;
 cd /data;
 $A/InstallDev -R comp@$VER -s image -v $VER -a $PWD/auth $REPO -p "admin frontend das mongodb")

(A=/data/cfg/admin; cd /data; $A/InstallDev -s start)

    To install the full set of services use the -p argument with: admin frontend couchdb das dbs dqmgui filemover mongodb phedex overview sitedb/legacy stagemanager reqmgr workqueue reqmon alertscollector crabserver crabcache dmwmmon asyncstageout t0wmadatasvc dbsmigration t0_reqmon acdcserver reqmgr2 gitweb

  8. Proxy renewal

    This step is optional.

    On deployments of services that require a proxy certificate to work, the procedure in the previous step uploads the user credentials to myproxy that are valid for one month (aka long term proxy). The VM, in turn, will keep renewing a short term proxy until the long term one expires.

    You will get mail notifications days before it expires, though. Upon reception of such messages, run the procedure below to renew the long term proxy. If your request in step 3 has not been handled yet, you'll need to run it every 36 hours until the machine gets included into myproxy.cern.ch. You can also run it at any time even when the proxy is not yet about to expire.

    cd /data
$PWD/cfg/admin/ProxySeed -t dev -d $PWD/HG1401a/auth/proxy

  9. Manage servers

    Check server status, start servers using these commands:

    cd /data
$PWD/cfg/admin/InstallDev -s status
$PWD/cfg/admin/InstallDev -s start

  10. Clean up

To clean up state, create VM snapshots and roll back to suitable point in time, scrap the VM and recreate it, or use the following commands to roll things back to where they were until step 7. WARNING: The rm command will wipe out almost everything on /data - MAKE SURE you run it in right place, and want to run it!

([ "$(hostname)" = "mydevvmname" ] || exit;
 echo "Deleting...";
 cd /data;
 $PWD/cfg/admin/InstallDev -s stop;
 crontab -r;
 killall python;
 sudo rm -fr [^aceu]* .??* current enabled)
  1. Develop server
Repeat steps 7 to 10 for any new software versions. You can use private RPM repository such as comp.pre.yourlogin to exercise builds which haven't been synced back to comp.pre yet. See Developing Against RPMS for details on how to upload to private repositories.

Local virtual machine

These instructions create Scientific Linux 5.7 virtual machine under VirtualBox 4.1.x. You can another hypervisor if you prefer; the translation should be very straightforward. VirtualBox is easy to use and free for uses such as this, so an attractive choice if you don't have another hypervisor. The instructions are somewhat geared towards CERN-like environment. You can adjust them to your local site conventions as far as groups, time servers, and such.

First create a new virtual machine labelled SL5.7, Linux/RedHat (64-bit), at least 2048 MB RAM. Create a new start-up disk: VDI, dynamically allocated, 40 GB in size. Download the install boot image and save it as boot_sl7_x86_64.iso in your downloads folder. Attach it on IDE into your VM. Set networking to bridged mode, and give the VM a pre-allocated fixed MAC address.

There are three main reasons for bridged networking and a preallocated MAC address. The first is that it's a great deal easier to SSH into and use the web server in the VM when your VM appears as any other server on your LAN with bridged networking. The second reason is that in order to get a host certificate for your VM, your site will likely require you to register the hostname and the MAC address. Specifically many sites, including CERN, will not grant a host certificate for a laptop. Third is that in order to get myproxy renewal rights, your host needs a stable name, and to get one you typically need a pre-registered MAC address.

Install minimal SL5.7 server into the VM using the boot image:

  • Language: English; Keyboard: us; Method: HTTP, DHCP no IPv6, linuxsoft.cern.ch, /cern/slc5X/x86_64/
  • OK to initialise partition table
  • Remove all partitions and create default layout: sda1:/boot 101 MB, sda2:LVM VG00 [LV01 swap 4000 MB, LV00 / ext3 rest]
  • Install grub loader [default]
  • Network: eth0, IPv4 DHCP, IPv6 Disabled, hostname via DHCP [default]
  • Region: Europe/Zurich, system clock uses UTC [default]
  • Set root password
  • Installation: server, customize now
    • Clear everything in: Desktop environments, Servers, Cluster Storage, Clustering, SLC Customizations
    • Applications: Text-based Internet (only)
    • Development: Development Libraries, Tools (only)
    • Base System: Administrative Tools, Base, Java (only)
  • After install remove CD, reboot into first boot:
    • Authentication: MD5 + shadow (no kerberos);
    • Firewall: enabled, SELinux: enforcing; Customize: ssh, Other ports: empty (remove afs3-callback:udp)
    • Keyboard: U.S. English
    • Network: DNS: Hostname: (give a name) (all other defaults)
    • System services: (defaults) + ntpd
    • Timezone: Europe/Zurich, system clock uses UTC; use ntp, servers: ip-time-{0,1,2}.cern.ch
    • Sound card: defaults (Intel 82801AA-ICH)

Now login as root and run the following, possibly adjusted for your site:

vi /etc/ntp.conf   # server ip-time-{0,1,2}.cern.ch
service ntpd restart
yum -y update
yum -y install zsh
yum -y clean packages
vi /etc/sudoers    # uncomment "%wheel ALL=(ALL) NOPASSWD: ALL"

ME=<your_afs_login>
echo your.account@cern.ch > /root/.forward
groupadd -g 1399 zh
useradd -M -g zh -G wheel -s /bin/zsh -u 12345 -c "Your Name" -d /home/$ME $ME
passwd $ME
mkdir -p /home/$ME /data
chown -R $ME:zh /home/$ME /data

# install guest additions
mount /dev/cdrom /media && cd /media
sh ./VBoxLinuxAdditions-amd64.run
cd /; umount /dev/cdrom

# upgrade zsh (optional)
cp -p /bin/zsh{,.old}
cd /tmp
wget http://downloads.sourceforge.net/zsh/zsh-4.3.12.tar.bz2
tar jxf zsh-*.tar.bz2
cd zsh-*/
./configure --prefix=/usr --libdir=/usr/lib64 zsh_cv_sys_link=no
make -j 2
make install # DESTDIR=/tmp/foobar for test
rm -f /bin/zsh; ln /usr/bin/zsh /bin/zsh
rm -fr /tmp/zsh*

# turn off
shutdown -h 0

Create VM snapshot for installed state. Restart. Run post-install, e.g. copy your shell environment:

scp ~/.z{log{in,out},sh{env,rc}} your-vm-host:
scp -rp ~/.globus your-vm-host:

Your VM is ready for use. SSH into it and deploy servers normally as per dev-vm instructions:

# one-time preparation
mkdir -p /tmp/foo
cd /tmp/foo
svn co svn+ssh://svn.cern.ch/reps/CMSDMWM/Infrastructure/trunk/Deployment cfg
sudo -l
cfg/Deploy -t dummy -s post $PWD system/devvm
rm -fr /tmp/foo

sudo yum -y install voms-clients myproxy
B=/afs/cern.ch/project/gd/LCG-share/3.2.8-0
sudo scp -rp you@lxplus.cern.ch:$B/glite/etc/vomses /etc/vomses
sudo scp -rp you@lxplus.cern.ch:$B/external/etc/grid-security/vomsdir /etc/grid-security

# server installation, using admin tools as shortcuts
cd /data
rsync -avu cmsweb@lxplus.cern.ch:private/conf/ /data/auth/
chgrp -R _sw /data/auth
chmod ug=r,o-rwx $(find /data/auth -type f)
chmod u=rwx,g=rx,o-rwx $(find /data/auth -type d)

A=/data/cfg/admin REPO="-r comp=comp.pre" VER=1111d
PKGS="admin frontend base couchdb das dbs dbsweb dqmgui filemover mongodb phedex"
PKGS="$PKGS overview sitedb/legacy stagemanager t0datasvc t0mon reqmgr workqueue"
$A/InstallDev -s image -v hg$VER -a $PWD/auth ${=REPO} -p "$PKGS"
$A/InstallDev -s start
$A/InstallDev -s status

# cleanup
cd /data
$A/InstallDev -s stop
crontab -r
killall python
sudo rm -fr [^aceu]* .??* current enabled

Environment on a Mac OS X system

This is really not a virtual machine environment, but there is experimental support for settings this up on an OS X laptop. This has only been tested with Snow Leopard:

# Fake enough of grid environment
sudo mkdir -p /etc/grid-security
B=/afs/cern.ch/project/gd/LCG-share/3.2.8-0
GS=/etc/grid-security BGS=$B/external/etc/grid-security
sudo rsync -av --delete you@lxplus.cern.ch:$B/../certificates $GS/certificates/
sudo rsync -av --delete you@lxplus.cern.ch:$B/glite/etc/vomses/ /etc/vomses/
sudo rsync -av --delete you@lxplus.cern.ch:$B/glite/etc/vomses/ /etc/vomses/
sudo rsync -av --delete you@lxplus.cern.ch:$BGS/vomsdir/ $GS/vomsdir/
sudo chown -R root:$(id -gn root) /etc/grid-security /etc/vomses

# Create accounts and all the rest; this installs into /users/cmssw/test
# instead of using /data. You may need to iterate and copy a host cert
# from somewhere into machine if the default rule doesn't work.
mkdir -p /tmp/foo
cd /tmp/foo
svn co svn+ssh://svn.cern.ch/reps/CMSDMWM/Infrastructure/trunk/Deployment cfg
sudo -l
CMS_DEV_ROOT=/users/cmssw/test cfg/Deploy -t dummy -s post $PWD system/devmac
cd; rm -fr /tmp/foo

# Install software using roughly standard dev-vm instructions.
cd /users/cmssw/test
rsync -avu cmsweb@lxplus.cern.ch:private/conf/ $PWD/auth/
chgrp -R _sw $PWD/auth
chmod ug=r,o-rwx $(find $PWD/auth -type f)
chmod u=rwx,g=rx,o-rwx $(find $PWD/auth -type d)

cd /users/cmssw/test
A=$PWD/cfg/admin REPO="-r comp=comp.pre" VER=1111a
PKGS="admin frontend base couchdb das dbs dbsweb dqmgui filemover mongodb phedex"
PKGS="$PKGS overview sitedb/legacy stagemanager t0datasvc t0mon reqmgr workqueue"
$A/InstallDev -s image -v hg$VER -a $PWD/auth ${=REPO} -p "$PKGS"
$A/InstallDev -s start