How to Create a New OSG Singularity Container

  1. create a repository on GitHub
    • must contain a Dockerfile in the top level
    • sufficient if not necessary: clone another such repository, e. g. JeffersonLab/gluex_docker_prod
    • call it my_account_github/my_container_github
  2. create a repository on DockerHub
    • from Dashboard (e. g. for account my_account_docker):
      • Use Create pull-down
      • Click on Create Automated Build
      • Click on Create Auto-build/Github (GitHub account must have been linked in advance)
      • Click on the GitHub repository created above (my_account_github/my_container_github)
      • Fill in the Create Automated Build form
        • choose Public for Visibility
        • give the repository a name for Docker
        • E. g., my_container_docker
    • new repository should show up on the Dashboard (my_container_docker)
      • Click on new repository, should see Source Repository box filled in with my_account_github/my_containter_github
  3. to trigger build, change Dockerfile on GitHub on the master branch
    • must be something other than comments
  4. Ask Richard to submit a pull request to list the new Docker repository on the OSG list
    • He needs to add the Docker repository (e. g., my_account_docker/my_container_docker) as a single line to the file docker_images.txt in the top level of the opensciencegrid/cvmfs-singularity-sync repository.
    • A corresponding singularity container (my_container_docker) will appear in the OSG cvmfs share (/cvmfs/singularity.opensciencegrid.org/my_account_docker/my_container_docker:latest)
    • Successful builds of the Docker container will refresh the Singularity container.
Advertisements

Build_scripts on OSG

  • Timeline
    • Mon.: Richard puts in pull request for markito3/gluex_docker_devel
    • Tue.: Get hd_root help message on OSG
    • Wed: Successful run of MCwrapper rho simulation
    • Thu: Resources and fresh CCDB and RCDB uploaded
  • Features
    • environment setting, standard syntax works:
      bs=/group/halld/Software/build_scripts
      dist=/group/halld/www/halldweb/html/dist
      source $bs/gluex_env_jlab.sh $dist/version_2.26.xml
    • resources in standard location:
      export JANA_RESOURCE_DIR=/group/halld/www/halldweb/html/resources
    • sqlite databases in standard location:
      export JANA_CALIB_URL=sqlite:////group/halld/www/halldweb/html/dist/ccdb.sqlite
    • reduces load on halldweb.jlab.org
  • Modifications
    • Environment setting was removed from container
    • Oasis bound to container, connection to /group done with soft link:
      $ ls -l /group
      lrwxrwxrwx 1 981 668 44 Mar 2 08:48 /group -> /cvmfs/oasis.opensciencegrid.org/gluex/group
  • Container dance
    • go to Docker Hub, create a repository (think historical collection of Docker containers)
    • link to a “personal” GitHub repository that has a Dockerfile
    • submit a pull request on OSG GitHub repository to add Docker repository to list of Docker repositories
    • wait…Singularity container will appear on OSG’s Singularity CVMFS:
      > ls -l /cvmfs/singularity.opensciencegrid.org/markito3/
      total 1
      lrwxrwxrwx 1 cvmfs cvmfs 112 Mar 2 08:48 gluex_docker_devel:latest -> /cvmfs/singularity.opensciencegrid.org/.images/71/71051c12b2d682bad4d96b8b2f1861486842b29d45a8be9baf3f8d38c32537
    • Changes to the container:
      • Push new Dockerfile to personal GitHub repository.
      • Docker Hub automatically builds a new instance of the Docker container.
      • OSG automatically creates a new Singularity container and updates CVMFS.
      • All OSG nodes automatically see new version of container.
  • Minor Issues
    • Standard CentOS7 build at JLab will not work in “naive” container.
      • Non-standard container or non-standard build?
    • /group binding mechanism does not work for non-CVMFS /group directory.
      • Different container or better CVMFS binding scheme?
    • Finalize rsync scheme for code/databases/resources

Getting a build_scripts build to work on the OSG

With the following changes, was able to get the hd_root usage message on the grid!

Changes to osg-container.sh

The original version assumed a /environment file that would be sourced. If that did not exist, then it tried to run singularity directly. In the former case it is assumed that the script wakes up in a container, in the latter that the container has to be fired up.

The new container did not have a /environment file and singularity was not in the path on the remote OSG node, so the script would drop down to “container not found,” a red herring.

The new version of the script simply assumes that we are inside the container and does not set-up an environment.

Original:

#!/bin/sh
#
# osg-container.sh - gluex job wrapper script for osg jobs
#
# This script contains GENERIC (non-job-specific) steps for getting
# set up inside the gluex container on an osg worker node and launching
# the job script. It should not normally be modified by the user.
#
# Usage: osg-container.sh  [job script arguments]
#
# Author: Richard.T.Jones at uconn.edu
# Version: June 8, 2017

container="/cvmfs/singularity.opensciencegrid.org/rjones30/gluex:latest"
#oasismount="/cvmfs/oasis.opensciencegrid.org"
oasismount="/cvmfs"
dockerimage="docker://rjones30/gluex:latest"
userproxy=x509up_u$UID

# define the container context for running on osg workers

if [[ -f /environment ]]; then
    echo "Job running on" `hostname`
    [ -r .$userproxy ] && mv .$userproxy /tmp/$userproxy
    source /environment
    unset CCDB_CONNECTION
    unset RCDB_CONNECTION
    eval $*; retcode=$?
    echo "Job finished with exit code" $retcode
    rm -rf *.sqlite
    exit $retcode

elif [[ -f $container/environment ]]; then
    echo "Starting up container on" `hostname`
    [ -r /tmp/$userproxy ] && cp /tmp/$userproxy .$userproxy
    exec singularity exec --containall --bind ${oasismount} --home `pwd`:/srv --pwd /srv --scratch /tmp,/var/tmp ${container} \
    bash $0 $*

else
    echo "Job container not found on" `hostname`
fi

New:

#!/bin/sh
#
# osg-container.sh - gluex job wrapper script for osg jobs
#<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
# This script contains GENERIC (non-job-specific) steps for getting
# set up inside the gluex container on an osg worker node and launching
# the job script. It should not normally be modified by the user.
#
# Usage: osg-container.sh  [job script arguments]
#
# Author: Richard.T.Jones at uconn.edu
# Version: June 8, 2017

container="/cvmfs/singularity.opensciencegrid.org/rjones30/gluex:latest"
#oasismount="/cvmfs/oasis.opensciencegrid.org"
oasismount="/cvmfs"
dockerimage="docker://rjones30/gluex:latest"
userproxy=x509up_u$UID

# define the container context for running on osg workers

echo "Job running on" `hostname`
[ -r .$userproxy ] &amp;&amp; mv .$userproxy /tmp/$userproxy
eval $*; retcode=$?
echo "Job finished with exit code" $retcode
exit $retcode

Changes to MCOSG.submit

The only substantive changes were to use the new container and to use the new osg-container.sh as the executable. (Arguments to MakeMC.csh were truncated, a simpler script was substituted for testing.)

Original:

universe = vanilla
Executable = /home/marki/hd_utilities/MCwrapper/osg-container.sh
Arguments  = ./MakeMC.csh 1 /srv/setup.csh /srv/gen_2pi_amp.cfg ./ 30966 0 1000 mc 2017-11-27-19-05-00 1 1 1 1 1 1 1 0 OSG 4 gen_2pi_amp 3 None 0 rcdb rcdb 3 11.5 I_dont_have_one None 10000 ./ /srv/ccdb.sqlite /srv/rcdb.sqlite 0 rcdb rcdb
Requirements = (HAS_SINGULARITY == TRUE) && (HAS_CVMFS_oasis_opensciencegrid_org == True)
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/rjones30/gluex:latest"
+SingularityBindCVMFS = True
+SingularityAutoLoad = True
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
concurrency_limits = GluexProduction
on_exit_remove = true
on_exit_hold = false
Error      = /osgpool/halld/marki/TEST//log/error_recon_2017_01_v2_take2_rho_30966_30966_0.log
output      = /osgpool/halld/marki/TEST//log/out_recon_2017_01_v2_take2_rho_30966_30966_0.log
log = /osgpool/halld/marki/TEST//log/OSG_recon_2017_01_v2_take2_rho_30966_30966_0.log
initialdir = ./
transfer_input_files = /home/marki/hd_utilities/MCwrapper/MakeMC.csh, /osgpool/halld/jrsteven/setup.csh, /osgpool/halld/marki/gen_2pi_amp.cfg, /group/halld/www/halldweb/html/dist/ccdb.sqlite, /group/halld/www/halldweb/html/dist/rcdb.sqlite
transfer_output_files = 30966_0
transfer_output_remaps = "30966_0=/osgpool/halld/marki/TEST/"
queue

New:

universe = vanilla
Executable = /osgpool/halld/marki/my-osg-container.sh
Arguments  = ./MakeMC.csh
Requirements = (HAS_SINGULARITY == TRUE) && (HAS_CVMFS_oasis_opensciencegrid_org == True)
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/markito3/gluex_docker_devel:latest"
+SingularityBindCVMFS = True
+SingularityAutoLoad = True
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
concurrency_limits = GluexProduction
on_exit_remove = true
on_exit_hold = false
Error      = /osgpool/halld/marki/TEST//log/error_recon_2017_01_v2_take2_rho_30966_30966_0.log
output      = /osgpool/halld/marki/TEST//log/out_recon_2017_01_v2_take2_rho_30966_30966_0.log
log = /osgpool/halld/marki/TEST//log/OSG_recon_2017_01_v2_take2_rho_30966_30966_0.log
initialdir = ./
transfer_input_files = /home/marki/hd_utilities/MCwrapper/MakeMC.csh, /osgpool/halld/jrsteven/setup.csh, /osgpool/halld/marki/gen_2pi_amp.cfg
transfer_output_files = 30966_0
transfer_output_remaps = "30966_0=/osgpool/halld/marki/TEST/"
queue

Changes to the gluex_docker_devel container

/group is no longer a directory to which the oasis directory is bound. Rather it is a soft link to the appropriate point in the already bound oasis tree. From the Docker build:

Step 7/8 : RUN ln -sv /cvmfs/oasis.opensciencegrid.org/gluex/group /group
---> Running in ef6ea6dd5ed3
'/group' -> '/cvmfs/oasis.opensciencegrid.org/gluex/group'