Running CCDB 2 Against a CCDB 2 Databases

Against mysql the error in hdgeant4 is:

src/JANA/JApplication.cc:1285
src/JANA/JApplication.cc:1286 Unable to create JCalibration object!
src/JANA/JApplication.cc:1287     URL: mysql://ccdb_user@localhost/ccdb2
src/JANA/JApplication.cc:1288 context: default
src/JANA/JApplication.cc:1289     run: 11366

Against sqlite, the error in hdgeant4 is

JANA ERROR>> Failed to get XML for /GEOMETRY/main_HDDS.xml! Geometry info not available!!
HDGeant4 - error parsing HDDS document, cannot continue

Testing CCDB 2

Using this script to test the four cases, CCDB 1 vs. 2, mysql vs. sqlite:

#!/bin/bash
ccdb[1]=mysql://ccdb_user@localhost/ccdb
ccdb[2]=mysql://ccdb_user@localhost/ccdb2
ccdb[3]=sqlite:////data/scratch/ccdb_convert_scratch/ccdb1.sqlite
ccdb[4]=sqlite:////data/scratch/ccdb_convert_scratch/ccdb2.sqlite
rundir[1]=mysql1
rundir[2]=mysql2
rundir[3]=sqlite1
rundir[4]=sqlite2

source /data/gluex/gluex_top/gluex_env_boot.sh

vs[1]=$HALLD_VERSIONS/version_5.2.2.xml
vs[2]=$HALLD_VERSIONS/ccdb2_test.xml
vs[3]=$HALLD_VERSIONS/version_5.2.2.xml
vs[4]=$HALLD_VERSIONS/ccdb2_test.xml

cd $SCR
time=`date +%j:%H:%M:%S`
dir=matrix_$time
mkdir -v $dir
cd $dir

for i in {1..4}
do
    gxenv ${vs[$i]}
    export JANA_CALIB_URL=${ccdb[$i]}
    bp=$HD_UTILITIES_HOME/b1pi_test
    mkdir -v ${rundir[$i]}
    pushd ${rundir[$i]} 
    $bp/my_b1pi_test.sh -d$bp -4 -n1000 -s123 1> out.log 2> err.log &
    popd
done

Differences between CCDB 1 and 2 Schemas

1c1
< == ccdb 1: assignments ==
---
> == ccdb 2: assignments ==
15c15
< == ccdb 1: columns ==
---
> == ccdb 2: columns ==
28c28
< == ccdb 1: constantSets ==
---
> == ccdb 2: constantSets ==
38,50c38,52
< == ccdb 1: directories ==
< +----------+--------------+------+-----+---------------------+----------------+
< | Field    | Type         | Null | Key | Default             | Extra          |
< +----------+--------------+------+-----+---------------------+----------------+
< | id       | int(11)      | NO   | PRI | NULL                | auto_increment |
< | created  | timestamp    | NO   |     | current_timestamp() |                |
< | modified | timestamp    | NO   |     | 2007-01-01 00:00:00 |                |
< | name     | varchar(255) | NO   |     |                     |                |
< | parentId | int(11)      | NO   | MUL | 0                   |                |
< | authorId | int(11)      | NO   |     | 1                   |                |
< | comment  | text         | YES  |     | NULL                |                |
< +----------+--------------+------+-----+---------------------+----------------+
< == ccdb 1: eventRanges ==
---
> == ccdb 2: directories ==
> +----------------+--------------+------+-----+---------------------+----------------+
> | Field          | Type         | Null | Key | Default             | Extra          |
> +----------------+--------------+------+-----+---------------------+----------------+
> | id             | int(11)      | NO   | PRI | NULL                | auto_increment |
> | created        | timestamp    | NO   |     | current_timestamp() |                |
> | modified       | timestamp    | NO   |     | 2007-01-01 00:00:00 |                |
> | name           | varchar(255) | NO   |     |                     |                |
> | parentId       | int(11)      | NO   | MUL | 0                   |                |
> | authorId       | int(11)      | NO   |     | 1                   |                |
> | comment        | text         | YES  |     | NULL                |                |
> | isDeprecated   | tinyint(1)   | NO   |     | 0                   |                |
> | deprecatedById | int(11)      | NO   |     | -1                  |                |
> +----------------+--------------+------+-----+---------------------+----------------+
> == ccdb 2: eventRanges ==
62c64
< == ccdb 1: logs ==
---
> == ccdb 2: logs ==
74c76
< == ccdb 1: runRanges ==
---
> == ccdb 2: runRanges ==
86c88
< == ccdb 1: schemaVersions ==
---
> == ccdb 2: schemaVersions ==
93c95
< == ccdb 1: tags ==
---
> == ccdb 2: tags ==
100,115c102,122
< == ccdb 1: typeTables ==
< +--------------+--------------+------+-----+---------------------+----------------+
< | Field        | Type         | Null | Key | Default             | Extra          |
< +--------------+--------------+------+-----+---------------------+----------------+
< | id           | int(11)      | NO   | PRI | NULL                | auto_increment |
< | created      | timestamp    | NO   |     | current_timestamp() |                |
< | modified     | timestamp    | NO   |     | 2007-01-01 00:00:00 |                |
< | directoryId  | int(11)      | NO   | MUL | NULL                |                |
< | name         | varchar(255) | NO   |     | NULL                |                |
< | nRows        | int(11)      | NO   |     | 1                   |                |
< | nColumns     | int(11)      | NO   |     | NULL                |                |
< | nAssignments | int(11)      | NO   |     | 0                   |                |
< | authorId     | int(11)      | NO   |     | 1                   |                |
< | comment      | text         | YES  |     | NULL                |                |
< +--------------+--------------+------+-----+---------------------+----------------+
< == ccdb 1: users ==
---
> == ccdb 2: typeTables ==
> +----------------+--------------+------+-----+---------------------+----------------+
> | Field          | Type         | Null | Key | Default             | Extra          |
> +----------------+--------------+------+-----+---------------------+----------------+
> | id             | int(11)      | NO   | PRI | NULL                | auto_increment |
> | created        | timestamp    | NO   |     | current_timestamp() |                |
> | modified       | timestamp    | NO   |     | 2007-01-01 00:00:00 |                |
> | directoryId    | int(11)      | NO   | MUL | NULL                |                |
> | name           | varchar(255) | NO   |     | NULL                |                |
> | nRows          | int(11)      | NO   |     | 1                   |                |
> | nColumns       | int(11)      | NO   |     | NULL                |                |
> | nAssignments   | int(11)      | NO   |     | 0                   |                |
> | authorId       | int(11)      | NO   |     | 1                   |                |
> | comment        | text         | YES  |     | NULL                |                |
> | isDeprecated   | tinyint(1)   | NO   |     | 0                   |                |
> | deprecatedById | int(11)      | NO   |     | -1                  |                |
> | isLocked       | tinyint(1)   | NO   |     | 0                   |                |
> | lockAuthorId   | int(11)      | YES  |     | NULL                |                |
> | lockTime       | timestamp    | YES  |     | NULL                |                |
> +----------------+--------------+------+-----+---------------------+----------------+
> == ccdb 2: users ==
128,141c135,153
< == ccdb 1: variations ==
< +-------------+--------------+------+-----+---------------------+----------------+
< | Field       | Type         | Null | Key | Default             | Extra          |
< +-------------+--------------+------+-----+---------------------+----------------+
< | id          | int(11)      | NO   | PRI | NULL                | auto_increment |
< | created     | timestamp    | NO   |     | 2007-01-01 00:00:00 |                |
< | modified    | timestamp    | NO   |     | current_timestamp() |                |
< | name        | varchar(100) | NO   | MUL | default             |                |
< | description | varchar(255) | YES  |     | NULL                |                |
< | authorId    | int(11)      | NO   |     | 1                   |                |
< | comment     | text         | YES  |     | NULL                |                |
< | parentId    | int(11)      | NO   | MUL | 0                   |                |
< +-------------+--------------+------+-----+---------------------+----------------+
< == ccdb 1: variations_has_tags ==
---
> == ccdb 2: variations ==
> +----------------+--------------+------+-----+---------------------+----------------+
> | Field          | Type         | Null | Key | Default             | Extra          |
> +----------------+--------------+------+-----+---------------------+----------------+
> | id             | int(11)      | NO   | PRI | NULL                | auto_increment |
> | created        | timestamp    | NO   |     | 2007-01-01 00:00:00 |                |
> | modified       | timestamp    | NO   |     | current_timestamp() |                |
> | name           | varchar(100) | NO   | MUL | default             |                |
> | description    | varchar(255) | YES  |     | NULL                |                |
> | authorId       | int(11)      | NO   |     | 1                   |                |
> | comment        | text         | YES  |     | NULL                |                |
> | parentId       | int(11)      | NO   | MUL | 0                   |                |
> | isLocked       | tinyint(1)   | NO   |     | 0                   |                |
> | lockTime       | timestamp    | YES  |     | NULL                |                |
> | lockAuthorId   | int(11)      | YES  |     | NULL                |                |
> | goBackBehavior | int(11)      | NO   |     | 0                   |                |
> | goBackTime     | timestamp    | YES  |     | NULL                |                |
> +----------------+--------------+------+-----+---------------------+----------------+
> == ccdb 2: variations_has_tags ==

The comparison was generated by this script:

#!/bin/bash
mysql -B -N -uccdb_user ccdb -e 'show tables' | awk '{print "echo == ccdb 1: "$1" ==; mysql -t -uccdb_user ccdb -e \"explain "$1"\""}' | bash >| ccdb_1_tables.txt
mysql -B -N -uccdb_user ccdb -e 'show tables' | awk '{print "echo == ccdb 2: "$1" ==; mysql -t -uccdb_user ccdb2 -e \"explain "$1"\""}' | bash >| ccdb_2_tables.txt
diff ccdb_1_tables.txt ccdb_2_tables.txt

Problem Building JANA on CentOS 7 with CCDB 2

Build complains about missing sqlite library and missing fmt library.

Linking    [.Linux_CentOS7-x86_64-gcc4.8.5-cntr/unit_tests/associated_objects/associated_objects]
/data/scratch/gluex_top_c7/ccdb/ccdb/lib/libccdb.a(SQLiteDataProvider.cc.o): In function `ccdb::SQLiteDataProvider::Connect(std::string const&)':
SQLiteDataProvider.cc:(.text+0x31e): undefined reference to `sqlite3_open_v2'
SQLiteDataProvider.cc:(.text+0x34d): undefined reference to `sqlite3_errmsg'
SQLiteDataProvider.cc:(.text+0x44b): undefined reference to `sqlite3_exec'
.
.
.
/data/scratch/gluex_top_c7/ccdb/ccdb/lib/libccdb.a(SQLiteDataProvider.cc.o): In function `std::make_unsigned<long>::type fmt::v6::internal::to_unsigned<long>(long)':
SQLiteDataProvider.cc:(.text._ZN3fmt2v68internal11to_unsignedIlEENSt13make_unsignedIT_E4typeES4_[_ZN3fmt2v68internal11to_unsignedIlEENSt13make_unsignedIT_E4typeES4_]+0x27): undefined reference to `fmt::v6::internal::assert_fail(char const*, int, char const*)'

sqlite and fmt comes in in a patch and that patch is only applied if python3 is being used. That is not the case for CentOS 7.

  • sqlite and fmt added into sbms.py.patch in v2.31 of build scripts
    • sqlite is needed since it is not longer a part of ccdb[?]
  • test for python3 in Makefile_jana introduced in transition from v 1.59 to 2.0. patches/jana/sbms.py.patch does not exist before v2.0
  • sbms.py.patch is unchanged from v2.0 to v2.30
  • go to a scheme where there are two patches to sbms.py, one if python3 is being used, and a second if ccdb 2 is being used. See this pull request.

CCDB 2 Not Linked with MySQL Support

  • Using debugger, see that the CCDB_MYSQL is not defined. Do not see anywhere in the CMakeLists where this switch can be thrown. In ccdb/cpp/src/CCDB/CalibrationGenerator.cc, MySQL database cannot be “openable” without this switch:
    bool CalibrationGenerator::CheckOpenable( const std::string & str)
    {
        //Check through known connections
        #ifdef CCDB_MYSQL
        if(str.find("mysql://")== 0) return true;
        #endif

        if(str.find("sqlite://")== 0) return true;
        return false;
    }

  • In ccdb/cpp/src/CCDB/CMakeLists.txt see the MySQL data provider commented out of the list of sources:
# Source files
set(SOURCE_FILES

        #user api
        Calibration.cc
        CalibrationGenerator.cc
        SQLiteCalibration.cc
        # MySQLCalibration.cc

        #helper classes
        Helpers/StringUtils.cc
        Helpers/PathUtils.cc
        Helpers/TimeProvider.cc
        Helpers/SQLite.h

        Model/Assignment.cc
        Model/ConstantsTypeColumn.cc
        Model/ConstantsTypeTable.cc
        Model/Directory.cc
        Model/RunRange.cc

        Providers/DataProvider.cc
        Providers/SQLiteDataProvider.cc

        #model and provider
        # Providers/MySQLDataProvider.cc
        )

Add -DCCDB_MYSQL compiler flag by adding

add_compile_definitions(CCDB_MYSQL)

to ccdb/cpp/src/CCDB/CMakeLists.txt on branch “add_-DCCDB_MYSQL_to_cmake” of markito3/ccdb repo. The next error is:

[  7%] Building CXX object src/CCDB/CMakeFiles/ccdb.dir/CalibrationGenerator.cc.o
cd /home/marki/git/ccdb/cpp/build/src/CCDB && /usr/bin/c++ -DCCDB_MYSQL -I/data/gluex/gluex_top/sqlite/sqlite-3.36.0/include -I/home/marki/git/ccdb/cpp/src -I/usr/include/mysql -fPIC -std=gnu++11 -MD -MT src/CCDB/CMakeFiles/ccdb.dir/CalibrationGenerator.cc.o -MF CMakeFiles/ccdb.dir/CalibrationGenerator.cc.o.d -o CMakeFiles/ccdb.dir/CalibrationGenerator.cc.o -c /home/marki/git/ccdb/cpp/src/CCDB/CalibrationGenerator.cc
In file included from /home/marki/git/ccdb/cpp/src/CCDB/CalibrationGenerator.cc:10:
/home/marki/git/ccdb/cpp/src/CCDB/Providers/MySQLDataProvider.h:93:14: error: conflicting return type specified for ‘virtual bool ccdb::MySQLDataProvider::LoadDirectories()’
   93 |         bool LoadDirectories() override;
      |              ^~~~~~~~~~~~~~~
In file included from /home/marki/git/ccdb/cpp/src/CCDB/Calibration.h:12,
                 from /home/marki/git/ccdb/cpp/src/CCDB/CalibrationGenerator.h:9,
                 from /home/marki/git/ccdb/cpp/src/CCDB/CalibrationGenerator.cc:4:
/home/marki/git/ccdb/cpp/src/CCDB/Providers/DataProvider.h:96:22: note: overridden function is ‘virtual void ccdb::DataProvider::LoadDirectories()’
   96 |         virtual void LoadDirectories() = 0;
      |                      ^~~~~~~~~~~~~~~
make[2]: *** [src/CCDB/CMakeFiles/ccdb.dir/build.make:90: src/CCDB/CMakeFiles/ccdb.dir/CalibrationGenerator.cc.o] Error 1
make[2]: Leaving directory '/home/marki/git/ccdb/cpp/build'
make[1]: *** [CMakeFiles/Makefile2:116: src/CCDB/CMakeFiles/ccdb.dir/all] Error 2
make[1]: Leaving directory '/home/marki/git/ccdb/cpp/build'
make: *** [Makefile:136: all] Error 2

This is because the paradigm for checking connection has changed in v2. SQLiteDataProvider.cc looks like:

void ccdb::SQLiteDataProvider::LoadDirectories()
{

    string thisFunc("SQliteDataProvider::LoadDirectories");
    //
    if (!IsConnected()) {
        throw std::runtime_error(thisFunc + " => Not connected to SQLite database ");
    }
    // prepare the SQL statement from the command line

    SQLiteStatement query(mDatabase, "SELECT `id`, `name`, `parentId`, `comment` FROM `directories`");
...

and MySQLDataProvider.cc has the old signature:

bool ccdb::MySQLDataProvider::LoadDirectories()
{
	//
	if(IsConnected())
	{
		if(!QuerySelect("SELECT `id`, `parentId`, `name`, `comment` FROM `directories`"))
		{
			//TODO: report error
			return false;
		}
...

making it inconsistent with the new, v2 base class in DataProvider.h:

        /** @brief Reads all directories from DB
         *
         * Explicitly forces to load directories from DB and build directory structure
         * (!) At this implementation all existing directories references will be deleted,
         * thus  references to them will become broken
         * @return   bool
         */
        virtual void LoadDirectories() = 0;

So a not insignificant change. Might leave this one alone for now.

More CCDB 2 Work

Continuing on from More Further Notes on CCDB 2.

  • Needed to revise patch to jana’s sbms_config.py to get showbuild to work on Fedora 35. See Pull Request #232.

Now that SHOWBUILD is fixed, here is the jana error:

make -C/home/marki/gluex_top_ccdb2/jana -f/home/marki/gluex_top_ccdb2/build_scripts/Makefile_jana
make[1]: Entering directory '/home/marki/gluex_top_ccdb2/jana'
Makefile_jana info: SCONS = scons
cd /home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2; scons SHOWBUILD=1
scons: Reading SConscript files ...
Test compiling pthread:
['c++', '-o', '.Linux_Fedora35-x86_64-gcc11.2.1_pthread', '.Linux_Fedora35-x86_64-gcc11.2.1_pthread.cc']
---Succeeded
looking for sbms_b'Linux'.py
No module named "sbms_b'Linux'"
looking for sbms_Linux_Fedora35-x86_64-gcc11-2-1.py
No module named 'sbms_Linux_Fedora35-x86_64-gcc11-2-1'
sbms : Making jana_config.h in .Linux_Fedora35-x86_64-gcc11.2.1/JANA
Test compiling xerces-c:
['c++', '-o', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c.cc', '-I/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/include', '-L/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/lib', '-lxerces-c']
---Succeeded
Test compiling xerces-c:
['c++', '-o', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c.cc', '-I/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/include', '-L/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/lib', '-lxerces-c']
---Succeeded
--- Configuration results ----
    HAVE_ROOT = 1
  HAVE_XERCES = 1
      XERCES3 = 1
    HAVE_CCDB = 1
------------------------------
sbms : Making jana-config in Linux_Fedora35-x86_64-gcc11.2.1/bin
scons: done reading SConscript files.
scons: Building targets ...
c++ -o .Linux_Fedora35-x86_64-gcc11.2.1/unit_tests/associated_objects/associated_objects .Linux_Fedora35-x86_64-gcc11.2.1/unit_tests/associated_objects/AssociatedObject_test.o -LLinux_Fedora35-x86_64-gcc11.2.1/lib -L/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/lib -L/usr/lib64 -L/home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib -lmariadb -lJANA -lccdb -lxerces-c -ldl
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteDataProvider::Connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
SQLiteDataProvider.cc:(.text+0x345): undefined reference to `sqlite3_open_v2'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x374): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x47e): undefined reference to `sqlite3_exec'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteDataProvider::Disconnect()':
SQLiteDataProvider.cc:(.text+0x604): undefined reference to `sqlite3_close'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `unsigned long ccdb::SQLiteStatement::Execute<ccdb::SQLiteDataProvider::LoadDirectories()::{lambda(unsigned long)#1}>(ccdb::SQLiteDataProvider::LoadDirectories()::{lambda(unsigned long)#1})':
SQLiteDataProvider.cc:(.text+0x2543): undefined reference to `sqlite3_column_count'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2563): undefined reference to `sqlite3_step'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x25af): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2644): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `unsigned long ccdb::SQLiteStatement::Execute<ccdb::SQLiteDataProvider::GetConstantsTypeTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ccdb::Directory*, bool)::{lambda(unsigned long)#1}>(ccdb::SQLiteDataProvider::GetConstantsTypeTable(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, ccdb::Directory*, bool)::{lambda(unsigned long)#1})':
SQLiteDataProvider.cc:(.text+0x26f7): undefined reference to `sqlite3_column_count'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2717): undefined reference to `sqlite3_step'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2763): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x27f8): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `unsigned long ccdb::SQLiteStatement::Execute<ccdb::SQLiteDataProvider::GetAllConstantsTypeTables(bool)::{lambda(unsigned long)#1}>(ccdb::SQLiteDataProvider::GetAllConstantsTypeTables(bool)::{lambda(unsigned long)#1})':
SQLiteDataProvider.cc:(.text+0x28ab): undefined reference to `sqlite3_column_count'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x28cb): undefined reference to `sqlite3_step'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2917): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x29ac): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `unsigned long ccdb::SQLiteStatement::Execute<ccdb::SQLiteDataProvider::LoadColumns(ccdb::ConstantsTypeTable*)::{lambda(unsigned long)#1}>(ccdb::SQLiteDataProvider::LoadColumns(ccdb::ConstantsTypeTable*)::{lambda(unsigned long)#1})':
SQLiteDataProvider.cc:(.text+0x2a5f): undefined reference to `sqlite3_column_count'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2a7f): undefined reference to `sqlite3_step'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2acb): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2b60): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `unsigned long ccdb::SQLiteStatement::Execute<ccdb::SQLiteDataProvider::SelectVariation(ccdb::SQLiteStatement&)::{lambda(unsigned long)#1}>(ccdb::SQLiteDataProvider::SelectVariation(ccdb::SQLiteStatement&)::{lambda(unsigned long)#1})':
SQLiteDataProvider.cc:(.text+0x2c13): undefined reference to `sqlite3_column_count'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2c33): undefined reference to `sqlite3_step'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2c7f): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2d14): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `unsigned long ccdb::SQLiteStatement::Execute<ccdb::SQLiteDataProvider::GetAssignmentShort(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda(unsigned long)#1}>(ccdb::SQLiteDataProvider::GetAssignmentShort(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, long, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&, bool)::{lambda(unsigned long)#1})':
SQLiteDataProvider.cc:(.text+0x2dc7): undefined reference to `sqlite3_column_count'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2de7): undefined reference to `sqlite3_step'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2e33): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x2ec8): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::~SQLiteStatement()':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatementD2Ev[_ZN4ccdb15SQLiteStatementD5Ev]+0x17): undefined reference to `sqlite3_finalize'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::Prepare(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement7PrepareERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4ccdb15SQLiteStatement7PrepareERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x4f): undefined reference to `sqlite3_prepare_v2'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement7PrepareERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4ccdb15SQLiteStatement7PrepareERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x6f): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement7PrepareERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4ccdb15SQLiteStatement7PrepareERKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0xfa): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::ValidateColumnIndex(int)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement19ValidateColumnIndexEi[_ZN4ccdb15SQLiteStatement19ValidateColumnIndexEi]+0xe2): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::BindInt32(int, int)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9BindInt32Eii[_ZN4ccdb15SQLiteStatement9BindInt32Eii]+0x3d): undefined reference to `sqlite3_bind_int'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9BindInt32Eii[_ZN4ccdb15SQLiteStatement9BindInt32Eii]+0x68): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9BindInt32Eii[_ZN4ccdb15SQLiteStatement9BindInt32Eii]+0xfd): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::BindInt64(int, long)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9BindInt64Eil[_ZN4ccdb15SQLiteStatement9BindInt64Eil]+0x3f): undefined reference to `sqlite3_bind_int64'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9BindInt64Eil[_ZN4ccdb15SQLiteStatement9BindInt64Eil]+0x6a): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9BindInt64Eil[_ZN4ccdb15SQLiteStatement9BindInt64Eil]+0xff): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::BindString(int, std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement10BindStringEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4ccdb15SQLiteStatement10BindStringEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x54): undefined reference to `sqlite3_bind_text'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement10BindStringEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4ccdb15SQLiteStatement10BindStringEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x7f): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement10BindStringEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE[_ZN4ccdb15SQLiteStatement10BindStringEiRKNSt7__cxx1112basic_stringIcSt11char_traitsIcESaIcEEE]+0x114): undefined reference to `fmt::v8::vformat[abi:cxx11](fmt::v8::basic_string_view<char>, fmt::v8::basic_format_args<fmt::v8::basic_format_context<fmt::v8::appender, char> >)'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::ReadInt32(int)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement9ReadInt32Ei[_ZN4ccdb15SQLiteStatement9ReadInt32Ei]+0x30): undefined reference to `sqlite3_column_int'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::ReadUInt32(int)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement10ReadUInt32Ei[_ZN4ccdb15SQLiteStatement10ReadUInt32Ei]+0x30): undefined reference to `sqlite3_column_int'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::ReadUInt64(int)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement10ReadUInt64Ei[_ZN4ccdb15SQLiteStatement10ReadUInt64Ei]+0x30): undefined reference to `sqlite3_column_int64'
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteStatement::ReadString[abi:cxx11](int)':
SQLiteDataProvider.cc:(.text._ZN4ccdb15SQLiteStatement10ReadStringB5cxx11Ei[_ZN4ccdb15SQLiteStatement10ReadStringB5cxx11Ei]+0x35): undefined reference to `sqlite3_column_text'
collect2: error: ld returned 1 exit status
scons: *** [.Linux_Fedora35-x86_64-gcc11.2.1/unit_tests/associated_objects/associated_objects] Error 1
scons: building terminated because of errors.
make[1]: *** [/home/marki/gluex_top_ccdb2/build_scripts/Makefile_jana:47: /home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2/.sconstruct_done] Error 2
make[1]: Leaving directory '/home/marki/gluex_top_ccdb2/jana'
make: *** [build_scripts/Makefile_all:99: jana_make] Error 2

sqlite3_open_v2 is an unresolved reference in both ccdb 1 and ccdb 2 versions of libccdb.a. When linking associated_objects in ccdb 1 and ccdb 2, there is no search of the sqlite library. Not sure why it is needed in ccdb 2 and not 1.

Solution: added search of sqlite and fmt libraries to jana, via the patch to sbms.py. That seems to work with both ccdb 1 and ccdb 2 builds as far as jana is concerned. This is pull request #233.

Successful Complete Build

  • Modified halld_recon, halld_sim, and hdgeant4 to search fmt libraries.
  • These versions checked in on branches called “add_search_of_fmt_lib” on JeffersonLab repos.
  • Updated version set file is ccdb2_test.xml.
  • Successful builds on both Fedora 35 and CentOS 7

More Further Notes on CCDB 2

  • Can do a successful build using branch v2-work of markito3/ccdb repo. Using build_scripts 2.28.
  • Try to do a complete build on Fedora 35. No go.

The version set file is ccdb2_test.xml from the halld_versions master branch.

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="https://halldweb.jlab.org/halld_versions/version6.xsl"?>
<gversions file="ccdb2_test.xml" date="2022-01-09">
<description>Based on version 5.2.2.</description>
<package name="amptools" version="0.12.2" dirtag="root62404"/>
<package name="ccdb" version="2.0-test" url="https://github.com/markito3/ccdb"/>
<package name="cernlib" version="2005" word_length="64-bit"/>
<package name="diracxx" version="2.0.2"/>
<package name="evio" version="4.4.6"/>
<package name="evtgen" version="01.07.00"/>
<package name="geant4" version="10.04.p02"/>
<package name="gluex_MCwrapper" version="v2.6.0"/>
<package name="gluex_root_analysis" version="1.20.0"  dirtag="ccdb2"/>
<package name="halld_recon" version="4.31.1" dirtag="ccdb2"/>
<package name="halld_sim" version="4.35.0" dirtag="ccdb2"/>
<package name="hdds" version="4.14.0" dirtag="root62404"/>
<package name="hdgeant4" version="2.31.0" dirtag="ccdb2"/>
<package name="hd_utilities" version="1.37"/>
<package name="hepmc" version="2.06.10"/>
<package name="jana" version="0.8.2" dirtag="ccdb2"/>
<package name="lapack" version="3.9.0"/>
<package name="photos" version="3.61"/>
<package name="rcdb" version="0.06.00"/>
<package name="root" version="6.24.04"/>
<package name="sqlitecpp" version="3.1.1"/>
<package name="sqlite" version="3.36.0" year="2021"/>
<package name="xerces-c" version="3.2.3"/>
</gversions>

The error:

[marki@markdesk5 jana]$ make -f $BUILD_SCRIPTS/Makefile_jana
Makefile_jana info: SCONS = scons
cd /home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2; scons 
scons: Reading SConscript files ...
sbms : Making jana_config.h in .Linux_Fedora35-x86_64-gcc11.2.1/JANA
sbms : Making jana-config in Linux_Fedora35-x86_64-gcc11.2.1/bin
scons: done reading SConscript files.
scons: Building targets ...
Linking    [.Linux_Fedora35-x86_64-gcc11.2.1/unit_tests/associated_objects/associated_objects]
/usr/bin/ld: /home/marki/gluex_top_ccdb2/ccdb/ccdb_2.0-test/lib/libccdb.a(SQLiteDataProvider.cc.o): in function `ccdb::SQLiteDataProvider::Connect(std::__cxx11::basic_string<char, std::char_traits<char>, std::allocator<char> > const&)':
SQLiteDataProvider.cc:(.text+0x345): undefined reference to `sqlite3_open_v2'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x374): undefined reference to `sqlite3_errmsg'
/usr/bin/ld: SQLiteDataProvider.cc:(.text+0x47e): undefined reference to `sqlite3_exec'

Try the jana build with SHOWBUILD turned on to see what the link command looks like. That does not work:

[marki@markdesk5 jana]$ make -f $BUILD_SCRIPTS/Makefile_jana JANA_SCONS_OPTIONS="SHOWBUILD=1"
Makefile_jana info: SCONS = scons
cd /home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2; scons SHOWBUILD=1
scons: Reading SConscript files ...
Test compiling pthread:
['c++', '-o', '.Linux_Fedora35-x86_64-gcc11.2.1_pthread', '.Linux_Fedora35-x86_64-gcc11.2.1_pthread.cc']
---Succeeded
looking for sbms_b'Linux'.py
No module named "sbms_b'Linux'"
looking for sbms_Linux_Fedora35-x86_64-gcc11-2-1.py
No module named 'sbms_Linux_Fedora35-x86_64-gcc11-2-1'
sbms : Making jana_config.h in .Linux_Fedora35-x86_64-gcc11.2.1/JANA
Test compiling xerces-c:
['c++', '-o', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c.cc', '-I/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/include', '-L/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/lib', '-lxerces-c']
---Succeeded
Test compiling xerces-c:
['c++', '-o', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c', '.Linux_Fedora35-x86_64-gcc11.2.1_xerces-c.cc', '-I/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/include', '-L/home/marki/gluex_top_ccdb2/xerces-c/xerces-c-3.2.3/lib', '-lxerces-c']
---Succeeded
TypeError: '>' not supported between instances of 'str' and 'int':
  File "/home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2/SConstruct", line 131:
    sbms_config.mk_jana_config_h(env)
  File "/home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2/SBMS/sbms_config.py", line 94:
    if(env['SHOWBUILD']>0):
make: *** [/home/marki/gluex_top_ccdb2/build_scripts/Makefile_jana:47: /home/marki/gluex_top_ccdb2/jana/jana_0.8.2^ccdb2/.sconstruct_done] Error 2

Further Notes on CCDB 2

  • on markdesk5, Fedora 35
  • using build scripts branch jeffersonlab/ccdb-2.00
  • using ccdb branch jeffersonlab/v2-main
  • running make -f Makefile_ccdb gives the following error:
CMake Error at /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:230 (message):
  Could NOT find SQLite3 (missing: SQLite3_INCLUDE_DIR SQLite3_LIBRARY)
Call Stack (most recent call first):
  /usr/share/cmake/Modules/FindPackageHandleStandardArgs.cmake:594 (_FPHSA_FAILURE_MESSAGE)
  /usr/share/cmake/Modules/FindSQLite3.cmake:54 (find_package_handle_standard_args)
  src/CCDB/CMakeLists.txt:16 (find_package)

  • If you look at /usr/share/cmake/Modules/FindSQLite3.cmake you see that it keys off of the cmake variables SQLite3_INCLUDE_DIR and SQLite3_LIBRARY.
  • Invoking cmake by hand with the following syntax:
cmake -DSQLite3_INCLUDE_DIR=/data/gluex/gluex_top/sqlite/sqlite-3.36.0/include \
    -DSQLite3_LIBRARY=/data/gluex/gluex_top/sqlite/sqlite-3.36.0/lib ..
  • …gets past the error. The error now is:
CMake Error at src/fmt/CMakeLists.txt:194 (add_library):
  Cannot find source file:

    include/fmt/os.h

  Tried extensions .c .C .c++ .cc .cpp .cxx .cu .mpp .m .M .mm .ixx .cppm .h
  .hh .h++ .hm .hpp .hxx .in .txx .f .F .for .f77 .f90 .f95 .f03 .hip .ispc


CMake Error at src/fmt/CMakeLists.txt:194 (add_library):
  No SOURCES given to target: fmt

  • Installing fmt-devel does not fix it. Installing it was referenced in a previous post.
  • Fixed by by-passing fmt as a target library in cmake system. Then system drops back to using installed rpm fmt-devel.
  • Does not compile on Fedora 35. The error is
[ 60%] Building CXX object src/tests/CMakeFiles/CCDB_tests.dir/tests.cc.o
cd /home/marki/ccdb2/ccdb/cpp/build/src/tests && /usr/bin/c++  -I/home/marki/ccdb2/ccdb/cpp/src -I/data/gluex/gluex_top/sqlite/sqlite-3.36.0/include -std=c++11 -MD -MT src/tests/CMakeFiles/CCDB_tests.dir/tests.cc.o -MF CMakeFiles/CCDB_tests.dir/tests.cc.o.d -o CMakeFiles/CCDB_tests.dir/tests.cc.o -c /home/marki/ccdb2/ccdb/cpp/src/tests/tests.cc
In file included from /usr/include/signal.h:328,
                 from /home/marki/ccdb2/ccdb/cpp/src/tests/catch.hpp:6117,
                 from /home/marki/ccdb2/ccdb/cpp/src/tests/tests.cc:13:
/home/marki/ccdb2/ccdb/cpp/src/tests/catch.hpp:8743:54: error: call to non-‘constexpr’ function ‘long int sysconf(int)’
 8743 | constexpr static std::size_t sigStackSize = 32768 >= MINSIGSTKSZ ? 32768 : MINSIGSTKSZ;
      |                                                      ^~~~~~~~~~~
In file included from /usr/include/bits/sigstksz.h:24,
                 from /usr/include/signal.h:328,
                 from /home/marki/ccdb2/ccdb/cpp/src/tests/catch.hpp:6117,
                 from /home/marki/ccdb2/ccdb/cpp/src/tests/tests.cc:13:
/usr/include/unistd.h:640:17: note: ‘long int sysconf(int)’ declared here
  640 | extern long int sysconf (int __name) __THROW;
      |                 ^~~~~~~
In file included from /home/marki/ccdb2/ccdb/cpp/src/tests/tests.cc:13:
/home/marki/ccdb2/ccdb/cpp/src/tests/catch.hpp:8802:41: error: size of array ‘altStackMem’ is not an integral constant-expression
 8802 | char FatalConditionHandler::altStackMem[sigStackSize] = {};
      |                                         ^~~~~~~~~~~~
make[3]: *** [src/tests/CMakeFiles/CCDB_tests.dir/build.make:76: src/tests/CMakeFiles/CCDB_tests.dir/tests.cc.o] Error 1
make[3]: Leaving directory '/home/marki/ccdb2/ccdb/cpp/build'
make[2]: *** [CMakeFiles/Makefile2:142: src/tests/CMakeFiles/CCDB_tests.dir/all] Error 2
make[2]: Leaving directory '/home/marki/ccdb2/ccdb/cpp/build'
make[1]: *** [Makefile:136: all] Error 2
make[1]: Leaving directory '/home/marki/ccdb2/ccdb/cpp/build'
make: *** [build_scripts/Makefile_ccdb:81: ccdb/.make_done] Error 2

  • Got a working version in CentOS 7 container. Needed to add fmt-devel to the prerequisite script. The branch is markito3/v2-build-fixes. Submitted a pull prequest to JLab repo.
  • Saved working version in google drive: documents_work/saved_work/save_ccdb2_2021-12-22.tar.gz
  • build scripts working version on markito3/build_v2_on_centos7, pull request created on jlab repo, merged, branch deleted. upshot jlab master should work
  • Upgrading catch.hpp allows building on Fedora 35. See Pull Request #86.