Singularity Command on CUE, non-Oasis

singularity shell --bind /group/halld $WEB/dist/gluex_centos7.img

Creating an Ubuntu Singularity Container

Create a sandbox container in a local directory ubuntu-latest using the recipe file /local/containers/singularity/Singularity.ubuntu-latest:

> cd /local/containers/builds
> sudo /usr/local/bin/singularity build --sandbox ubuntu-latest /local/containers/singularity/Singularity.ubuntu-latest


> cat /local/containers/singularity/Singularity.ubuntu-latest
Bootstrap: docker
From: ubuntu:latest

Inside the container, as root, make directories to (a) bind the host directory for build products and to (b) mount the home directory with the install scripts:

> sudo /usr/local/bin/singularity shell --writable ubuntu-latest
> mkdir /scratch
> mkdir /home/marki

As root, install the system supplied software

> sudo /usr/local/bin/singularity shell --bind /home/marki --bind /local/scratch:/scratch --writable ubuntu-latest
> apt-get update
> cd /home/marki/gluex_install
> ./

Build the gluex software stack as a non-privileged user on the local disk.

> cd /local/scratch
> mkdir ub
> cd /local/containers/builds
> /usr/local/bin/singularity shell --bind /local/scratch:/scratch ubuntu-latest
> cd /scratch/ub
> /home/marki/gluex_install/

How to Create a New OSG Singularity Container

  1. create a repository on GitHub
    • must contain a Dockerfile in the top level
    • sufficient if not necessary: clone another such repository, e. g. JeffersonLab/gluex_docker_prod
    • call it my_account_github/my_container_github
  2. create a repository on DockerHub
    • from Dashboard (e. g. for account my_account_docker):
      • Use Create pull-down
      • Click on Create Automated Build
      • Click on Create Auto-build/Github (GitHub account must have been linked in advance)
      • Click on the GitHub repository created above (my_account_github/my_container_github)
      • Fill in the Create Automated Build form
        • choose Public for Visibility
        • give the repository a name for Docker
        • E. g., my_container_docker
    • new repository should show up on the Dashboard (my_container_docker)
      • Click on new repository, should see Source Repository box filled in with my_account_github/my_containter_github
  3. to trigger build, change Dockerfile on GitHub on the master branch
    • must be something other than comments
  4. Ask Richard to submit a pull request to list the new Docker repository on the OSG list
    • He needs to add the Docker repository (e. g., my_account_docker/my_container_docker) as a single line to the file docker_images.txt in the top level of the opensciencegrid/cvmfs-singularity-sync repository.
    • A corresponding singularity container (my_container_docker) will appear in the OSG cvmfs share (/cvmfs/
    • Successful builds of the Docker container will refresh the Singularity container.

Creating a Singularity Container for Building

April 10, 2018

To build a sandbox container from latest CentOS7:

sudo /usr/local/bin/singularity build --sandbox centos7.builder Singularity.centos7.builder

To enter the container and make modifications to it (e. g., install RPMs via gluex_install):

sudo /usr/local/bin/singularity shell --bind /home/marki/gluex_install:/root/gluex_install --writable centos7.builder

GlueX Bi-Weekly Meeting Report

March 13, 2018

  • New sim-recon release, version 2.24.0.
  • MCwrapper 1.13 has been released.
  • Open Science Grid open for business (OSG-OFB) for GlueX simulations for all collaborators
    • software and database update being done automatically
      • 3 new builds out there
      • calibration constants (CCDB) and run constants (RCDB) updated nightly
    • mechanism for random trigger distribution in place
    • access via MCwrapper transparently
    • N. B: you need to get a personal certificate from the OSG. See the preparation section of “Using the Grid” on the wiki.
  • Other container related work continues
    • Running at NERSC.
    • Container standardization/distribution.
    • Using GlueX software on your desktop via containers.
    • We are still meeting weekly on container-related topics.


Build_scripts on OSG

  • Timeline
    • Mon.: Richard puts in pull request for markito3/gluex_docker_devel
    • Tue.: Get hd_root help message on OSG
    • Wed: Successful run of MCwrapper rho simulation
    • Thu: Resources and fresh CCDB and RCDB uploaded
  • Features
    • environment setting, standard syntax works:
      source $bs/ $dist/version_2.26.xml
    • resources in standard location:
      export JANA_RESOURCE_DIR=/group/halld/www/halldweb/html/resources
    • sqlite databases in standard location:
      export JANA_CALIB_URL=sqlite:////group/halld/www/halldweb/html/dist/ccdb.sqlite
    • reduces load on
  • Modifications
    • Environment setting was removed from container
    • Oasis bound to container, connection to /group done with soft link:
      $ ls -l /group
      lrwxrwxrwx 1 981 668 44 Mar 2 08:48 /group -> /cvmfs/
  • Container dance
    • go to Docker Hub, create a repository (think historical collection of Docker containers)
    • link to a “personal” GitHub repository that has a Dockerfile
    • submit a pull request on OSG GitHub repository to add Docker repository to list of Docker repositories
    • wait…Singularity container will appear on OSG’s Singularity CVMFS:
      > ls -l /cvmfs/
      total 1
      lrwxrwxrwx 1 cvmfs cvmfs 112 Mar 2 08:48 gluex_docker_devel:latest -> /cvmfs/
    • Changes to the container:
      • Push new Dockerfile to personal GitHub repository.
      • Docker Hub automatically builds a new instance of the Docker container.
      • OSG automatically creates a new Singularity container and updates CVMFS.
      • All OSG nodes automatically see new version of container.
  • Minor Issues
    • Standard CentOS7 build at JLab will not work in “naive” container.
      • Non-standard container or non-standard build?
    • /group binding mechanism does not work for non-CVMFS /group directory.
      • Different container or better CVMFS binding scheme?
    • Finalize rsync scheme for code/databases/resources