singularity – Page 2 – Mark's Notebook

Singularity Command on CUE, non-Oasis

February 4, 2019markito3Leave a comment

singularity shell --bind /group/halld $WEB/dist/gluex_centos7.img

Notes from Doing GlueX Install on Trusty

January 4, 2019markito3Leave a comment

ubuntu on windows cannot run singularity, i. e., no container within a container
singularity will not run natively on windows without a vm (i. e. vmware)
make -jN works for geant4
there is no FORCE in Makefile_halld_sim

There is a build error on Ubuntu Trusty for CCDB that probably does not affect the build since the CCDB environment is set up by build_scripts. If that changes however…

make -C/usr/local/gluex/gluex_top/ccdb -f/usr/local/gluex/gluex_top/build_scripts/Makefile_ccdb
make[1]: Entering directory `/usr/local/gluex/gluex_top/ccdb'
source ccdb_1.06.06/environment.bash ; \
	cd ccdb_1.06.06 ; \
	python scripts/scons.py
/bin/sh: 1: source: not found
scons: Reading SConscript files ...
To compile with -m32 forced use 'with-m32=true' flag
Building CCDB using MySQL dependencies

Developer-Friendly Oasis Distribution

December 27, 2018March 22, 2019markito3Leave a comment

Questions

how does singularity know which kind of container to build
is recipe a docker term or a singularity term
what does a singularity recipe look like
how do you create a singularity container from a docker container on dockerhub

with Friday only, 114 GB on oasis now

On scosg16, the following is sufficient to run the container:

 singularity shell --home ${PWD}:/srv --pwd /srv --bind /cvmfs --contain --ipc --pid /cvmfs/singularity.opensciencegrid.org/markito3/gluex_docker_devel:latest

 Derived from a hint on Docker and Singularity Containers (https://support.opensciencegrid.org/support/solutions/articles/12000024676-docker-and-singularity-containers)

Creating an Ubuntu Singularity Container

October 4, 2018October 5, 2018markito3Leave a comment

Create a sandbox container in a local directory ubuntu-latest using the recipe file /local/containers/singularity/Singularity.ubuntu-latest:

> cd /local/containers/builds
> sudo /usr/local/bin/singularity build --sandbox ubuntu-latest /local/containers/singularity/Singularity.ubuntu-latest

where

> cat /local/containers/singularity/Singularity.ubuntu-latest
Bootstrap: docker
From: ubuntu:latest

Inside the container, as root, make directories to (a) bind the host directory for build products and to (b) mount the home directory with the install scripts:

> sudo /usr/local/bin/singularity shell --writable ubuntu-latest
> mkdir /scratch
> mkdir /home/marki

As root, install the system supplied software

> sudo /usr/local/bin/singularity shell --bind /home/marki --bind /local/scratch:/scratch --writable ubuntu-latest
> apt-get update
> cd /home/marki/gluex_install
> ./gluex_prereqs_ubuntu_x86-64.sh

Build the gluex software stack as a non-privileged user on the local disk.

> cd /local/scratch
> mkdir ub
> cd /local/containers/builds
> /usr/local/bin/singularity shell --bind /local/scratch:/scratch ubuntu-latest
> cd /scratch/ub
> /home/marki/gluex_install/gluex_install.sh

How to Create a New OSG Singularity Container

May 30, 2018May 30, 2018markito3Leave a comment

create a repository on GitHub
- must contain a Dockerfile in the top level
- sufficient if not necessary: clone another such repository, e. g. JeffersonLab/gluex_docker_prod
- call it my_account_github/my_container_github
create a repository on DockerHub
- from Dashboard (e. g. for account my_account_docker):
  - Use Create pull-down
  - Click on Create Automated Build
  - Click on Create Auto-build/Github (GitHub account must have been linked in advance)
  - Click on the GitHub repository created above (my_account_github/my_container_github)
  - Fill in the Create Automated Build form
    - choose Public for Visibility
    - give the repository a name for Docker
    - E. g., my_container_docker
- new repository should show up on the Dashboard (my_container_docker)
  - Click on new repository, should see Source Repository box filled in with my_account_github/my_containter_github
to trigger build, change Dockerfile on GitHub on the master branch
- must be something other than comments
Ask Richard to submit a pull request to list the new Docker repository on the OSG list
- He needs to add the Docker repository (e. g., my_account_docker/my_container_docker) as a single line to the file docker_images.txt in the top level of the opensciencegrid/cvmfs-singularity-sync repository.
- A corresponding singularity container (my_container_docker) will appear in the OSG cvmfs share (/cvmfs/singularity.opensciencegrid.org/my_account_docker/my_container_docker:latest)
- Successful builds of the Docker container will refresh the Singularity container.

Creating a Singularity Container for Building

April 11, 2018April 11, 2018markito3Leave a comment

April 10, 2018

To build a sandbox container from latest CentOS7:

sudo /usr/local/bin/singularity build --sandbox centos7.builder Singularity.centos7.builder

To enter the container and make modifications to it (e. g., install RPMs via gluex_install):

sudo /usr/local/bin/singularity shell --bind /home/marki/gluex_install:/root/gluex_install --writable centos7.builder

Build_scripts on OSG

March 2, 2018March 2, 2018markito3Leave a comment

Timeline
- Mon.: Richard puts in pull request for markito3/gluex_docker_devel
- Tue.: Get hd_root help message on OSG
- Wed: Successful run of MCwrapper rho simulation
- Thu: Resources and fresh CCDB and RCDB uploaded

Features

environment setting, standard syntax works:

bs=/group/halld/Software/build_scripts
dist=/group/halld/www/halldweb/html/dist
source $bs/gluex_env_jlab.sh $dist/version_2.26.xml

resources in standard location:

export JANA_RESOURCE_DIR=/group/halld/www/halldweb/html/resources

sqlite databases in standard location:

export JANA_CALIB_URL=sqlite:////group/halld/www/halldweb/html/dist/ccdb.sqlite

reduces load on halldweb.jlab.org

Modifications
- Environment setting was removed from container
- Oasis bound to container, connection to /group done with soft link:
```
$ ls -l /group
lrwxrwxrwx 1 981 668 44 Mar 2 08:48 /group -> /cvmfs/oasis.opensciencegrid.org/gluex/group
```
Container dance
- go to Docker Hub, create a repository (think historical collection of Docker containers)
- link to a “personal” GitHub repository that has a Dockerfile
- submit a pull request on OSG GitHub repository to add Docker repository to list of Docker repositories
- wait…Singularity container will appear on OSG’s Singularity CVMFS:
```
> ls -l /cvmfs/singularity.opensciencegrid.org/markito3/
total 1
lrwxrwxrwx 1 cvmfs cvmfs 112 Mar 2 08:48 gluex_docker_devel:latest -> /cvmfs/singularity.opensciencegrid.org/.images/71/71051c12b2d682bad4d96b8b2f1861486842b29d45a8be9baf3f8d38c32537
```
- Changes to the container:
  - Push new Dockerfile to personal GitHub repository.
  - Docker Hub automatically builds a new instance of the Docker container.
  - OSG automatically creates a new Singularity container and updates CVMFS.
  - All OSG nodes automatically see new version of container.
Minor Issues
- Standard CentOS7 build at JLab will not work in “naive” container.
  - Non-standard container or non-standard build?
- /group binding mechanism does not work for non-CVMFS /group directory.
  - Different container or better CVMFS binding scheme?
- Finalize rsync scheme for code/databases/resources

CentOS 7 build at JLab No Go in CentOS 7 Container

February 28, 2018February 28, 2018markito31 Comment

This was a suspicion. Problem is the mysqlclient library coming from a non-standard rpm at JLab:

sh-4.2$ hd_root
hd_root: error while loading shared libraries: libmysqlclient.so.20: cannot open shared object file: No such file or directory

Getting a build_scripts build to work on the OSG

February 27, 2018February 27, 2018markito3Leave a comment

With the following changes, was able to get the hd_root usage message on the grid!

Changes to osg-container.sh

The original version assumed a /environment file that would be sourced. If that did not exist, then it tried to run singularity directly. In the former case it is assumed that the script wakes up in a container, in the latter that the container has to be fired up.

The new container did not have a /environment file and singularity was not in the path on the remote OSG node, so the script would drop down to “container not found,” a red herring.

The new version of the script simply assumes that we are inside the container and does not set-up an environment.

Original:

#!/bin/sh
#
# osg-container.sh - gluex job wrapper script for osg jobs
#
# This script contains GENERIC (non-job-specific) steps for getting
# set up inside the gluex container on an osg worker node and launching
# the job script. It should not normally be modified by the user.
#
# Usage: osg-container.sh  [job script arguments]
#
# Author: Richard.T.Jones at uconn.edu
# Version: June 8, 2017

container="/cvmfs/singularity.opensciencegrid.org/rjones30/gluex:latest"
#oasismount="/cvmfs/oasis.opensciencegrid.org"
oasismount="/cvmfs"
dockerimage="docker://rjones30/gluex:latest"
userproxy=x509up_u$UID

# define the container context for running on osg workers

if [[ -f /environment ]]; then
    echo "Job running on" `hostname`
    [ -r .$userproxy ] && mv .$userproxy /tmp/$userproxy
    source /environment
    unset CCDB_CONNECTION
    unset RCDB_CONNECTION
    eval $*; retcode=$?
    echo "Job finished with exit code" $retcode
    rm -rf *.sqlite
    exit $retcode

elif [[ -f $container/environment ]]; then
    echo "Starting up container on" `hostname`
    [ -r /tmp/$userproxy ] && cp /tmp/$userproxy .$userproxy
    exec singularity exec --containall --bind ${oasismount} --home `pwd`:/srv --pwd /srv --scratch /tmp,/var/tmp ${container} \
    bash $0 $*

else
    echo "Job container not found on" `hostname`
fi

New:

#!/bin/sh
#
# osg-container.sh - gluex job wrapper script for osg jobs
#<span id="mce_SELREST_start" style="overflow:hidden;line-height:0;"></span>
# This script contains GENERIC (non-job-specific) steps for getting
# set up inside the gluex container on an osg worker node and launching
# the job script. It should not normally be modified by the user.
#
# Usage: osg-container.sh  [job script arguments]
#
# Author: Richard.T.Jones at uconn.edu
# Version: June 8, 2017

container="/cvmfs/singularity.opensciencegrid.org/rjones30/gluex:latest"
#oasismount="/cvmfs/oasis.opensciencegrid.org"
oasismount="/cvmfs"
dockerimage="docker://rjones30/gluex:latest"
userproxy=x509up_u$UID

# define the container context for running on osg workers

echo "Job running on" `hostname`
[ -r .$userproxy ] &amp;&amp; mv .$userproxy /tmp/$userproxy
eval $*; retcode=$?
echo "Job finished with exit code" $retcode
exit $retcode

Changes to MCOSG.submit

The only substantive changes were to use the new container and to use the new osg-container.sh as the executable. (Arguments to MakeMC.csh were truncated, a simpler script was substituted for testing.)

Original:

universe = vanilla
Executable = /home/marki/hd_utilities/MCwrapper/osg-container.sh
Arguments  = ./MakeMC.csh 1 /srv/setup.csh /srv/gen_2pi_amp.cfg ./ 30966 0 1000 mc 2017-11-27-19-05-00 1 1 1 1 1 1 1 0 OSG 4 gen_2pi_amp 3 None 0 rcdb rcdb 3 11.5 I_dont_have_one None 10000 ./ /srv/ccdb.sqlite /srv/rcdb.sqlite 0 rcdb rcdb
Requirements = (HAS_SINGULARITY == TRUE) && (HAS_CVMFS_oasis_opensciencegrid_org == True)
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/rjones30/gluex:latest"
+SingularityBindCVMFS = True
+SingularityAutoLoad = True
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
concurrency_limits = GluexProduction
on_exit_remove = true
on_exit_hold = false
Error      = /osgpool/halld/marki/TEST//log/error_recon_2017_01_v2_take2_rho_30966_30966_0.log
output      = /osgpool/halld/marki/TEST//log/out_recon_2017_01_v2_take2_rho_30966_30966_0.log
log = /osgpool/halld/marki/TEST//log/OSG_recon_2017_01_v2_take2_rho_30966_30966_0.log
initialdir = ./
transfer_input_files = /home/marki/hd_utilities/MCwrapper/MakeMC.csh, /osgpool/halld/jrsteven/setup.csh, /osgpool/halld/marki/gen_2pi_amp.cfg, /group/halld/www/halldweb/html/dist/ccdb.sqlite, /group/halld/www/halldweb/html/dist/rcdb.sqlite
transfer_output_files = 30966_0
transfer_output_remaps = "30966_0=/osgpool/halld/marki/TEST/"
queue

New:

universe = vanilla
Executable = /osgpool/halld/marki/my-osg-container.sh
Arguments  = ./MakeMC.csh
Requirements = (HAS_SINGULARITY == TRUE) && (HAS_CVMFS_oasis_opensciencegrid_org == True)
+SingularityImage = "/cvmfs/singularity.opensciencegrid.org/markito3/gluex_docker_devel:latest"
+SingularityBindCVMFS = True
+SingularityAutoLoad = True
should_transfer_files = YES
when_to_transfer_output = ON_EXIT
concurrency_limits = GluexProduction
on_exit_remove = true
on_exit_hold = false
Error      = /osgpool/halld/marki/TEST//log/error_recon_2017_01_v2_take2_rho_30966_30966_0.log
output      = /osgpool/halld/marki/TEST//log/out_recon_2017_01_v2_take2_rho_30966_30966_0.log
log = /osgpool/halld/marki/TEST//log/OSG_recon_2017_01_v2_take2_rho_30966_30966_0.log
initialdir = ./
transfer_input_files = /home/marki/hd_utilities/MCwrapper/MakeMC.csh, /osgpool/halld/jrsteven/setup.csh, /osgpool/halld/marki/gen_2pi_amp.cfg
transfer_output_files = 30966_0
transfer_output_remaps = "30966_0=/osgpool/halld/marki/TEST/"
queue

Changes to the gluex_docker_devel container

/group is no longer a directory to which the oasis directory is bound. Rather it is a soft link to the appropriate point in the already bound oasis tree. From the Docker build:

Step 7/8 : RUN ln -sv /cvmfs/oasis.opensciencegrid.org/gluex/group /group
---> Running in ef6ea6dd5ed3
'/group' -> '/cvmfs/oasis.opensciencegrid.org/gluex/group'

GlueX Singularity Container Notes

February 16, 2018February 16, 2018markito3Leave a comment

OSG experience

Ran with Richard’s osg-container.sh
Invoked using Thomas’s MCwrapper
Jobs fail for lack of RCDB access

Singularity work

singularity ext3 gluex image
- start from Docker, centos:latest
- initial build into “sandbox” (standard directory tree)
- do package additions via build_scripts
- convert from sandbox to ext3
- container size: 1.1 GB
complete build of gluex stack using container, but external to container
- built with version_2.26.xml (most recent is 2.27)
- starts at 28 GB
- after trimming: 8.4 GB with everything (below, sizes in kB)
  - 4914204 sim-recon
  - 1613104 geant4
  - 1146088 root
  - 383184 hdgeant4
  - 189660 jana
  - 152736 cernlib
  - 105552 lapack
  - 65732 ccdb
  - 58660 xerces-c
  - 58376 rcdb
  - 10268 sqlitecpp
  - 6988 hdds
  - 4344 evio
  - 3460 amptools
  - 2924 gluex_root_analysis
  - 1728 hd_utilities
  - 428 build_scripts-latest
  - 48 latest.tar.gz
  - 4 version.xml
  - 4 setup.sh
  - 4 setup.csh
  - 0 build_scripts
questions:
- how to put on oasis?
  - proposal: use build_scripts directory structure
- will it run in existing container?
  - likely yes
- what to do about CCDB, RCDB, resources?
  - proposal: reproduce /group/halld directory structure
    - can update in an rsync-like manner