Annual DataBase Update Procedures

This page describes annual release procedures for updating database with new set of THE SPECIALIST LEXICON data in lexAccess tools. This is the only procedure needed after release 2013+.

1. LA PreProc
   • Generate Db table files (UTF-8)

     shell> cd ${LA_PRE_PROC}/PreDatabase/bin
     shell> 1.GenerateDbFiles ${YEAR}
     --------------------------------------
     Which Program ?
     --------------------------------------
     1) Copy LEXICON
     2) Copy inflVars.data
     3) Generate lexicon.data
     4) Generate eui.data
     ----------
     5) Copy above 4 files to LexAccessDbData2013+
     => Go to LexAccessDbData2013+ to load files to DB tables
     ----------
     6) Copy Db tables from LexAccessDbData2013+ to laData
     7) Back Db tables from laData to Backup/Releases/LA/laData
     ----------
     10) All (default: Step 1- 5)
     ----------
     

     
     Steps 1-5: 4 files are copied and generated:

     Option	File name	Action
     1	LEXICON	copy ${LEX_DIR}/data/${YEAR}/tables.frozen/LEXICON to ${LA_PRE_DB_DIR}/data/tables/LEXICON
     2	inflVars.data	copy ${LEX_DIR}/data/${YEAR}/tables.frozen/inflVars.data to ${LA_PRE_DB_DIR}/data/tables/inflVars.data
     3	lexicon.data	Generate ${LA_DIR}/data/tables/lexicon.data (from LEXICON) shell> ${LA}/PreDatabase/bin/ModifyLexicon -f:d -i:LEXICON -o:lexicon.data
     4	eui.data	Generate eui.data. The total number of EUI should be equals to total lexical records in LEXICON because this file include deleted records if using lexicon.data from LB.
     5	Above 4 files	copy to ${DEV}/LA/lexAccessDbData2013+/data.${YEAR}

   • Check Max. length of fields in Lexicon.data with database tables

     shell> cd ${LA_PRE_PROC}/PreDatabase/bin
     shell> 2.AnalyzeDbFiles ${YEAR}
     

2. LoadDb Program (After Step-5)
   • Update db.cfg
     • update ${lexAccessDbData2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
       • DB_NAME=lexAccess${YEAR}
       • DB_DATA_DIR=data.${YEAR}/HSqlDb

   • Setup directory

     shell> cd ${lexAccessDbData2013+}/loadDb/bin
     shell> 1.Setup ${YEAR}
     

     • mkdir ${LA_DB_DATA_DIR}/data.${YEAR}
     • ln -s ${LA_DB_DATA_DIR}/data.${YEAR} ${LA_DB_DATA_DIR}/data => Use this script to create Db data directory. The database name and username are defined in the cfg.data file.

   • Load data into HSqlDb (skip MySql)

     Manully link tables

     shell> cd ${lexAccessDbData2013+}/data.${YEAR}
     shell> ln -sf ./tables.utf8 tables
     

     shell> cd ${lexAccessDbData2013+}/loadDb/bin
     shell> 2.LoadDb
     1
     3
     

     • Update DB:
       add "readonly=true" in *.properties after tables are created/loaded

   • Check loaded data in Db

     shell> cd ${lexAccessDbData2013+}/loadDb/bin
     shell> 3.TestDb
     1
     1
     
     1
     2
     

     => The total number of lexicon does not includes deleted records.

     => Manully move HSqlDb to HSqlDb.utf8 (${lexAccessDbData2013+}/data.${YEAR}/

3. laData and backup

   shell> cd ${LA_PRE_PROC}/PreDatabase/bin
   shell> 1.GenerateDbFiles ${YEAR}
   --------------------------------------
   Which Program ?
   --------------------------------------
   ...
   ----------
   6) Copy Db tables from LexAccessDbData2013+ to laData
   7) Back Db tables from laData to Backup/Releases/LA/laData
   ----------
   

   • laData

     shell> cd ${lexAccess2013+}/data.${YEAR}
     shell> mv HSqlDb to HSqlDb.utf8
     shell> 1.GenerateDbFiles ${YEAR}
     6
     

     => Copy -rp ${lexAccess2013+}/data.${YEAR}/HSqlDb.utf8 to ${DEV}/LA/laData/HSqlDb.${YEAR}

   • Backup

     shell> 1.GenerateDbFiles ${YEAR}
     7
     

     => Copy -rp ${LA_DATA}/HSqlDb.${YEAR} ${BACKUP}Releases/LA/laData/.
     => tar -czvf HSqlDb.${YEAR}.tgz HSqlDb.${YEAR}

     => Manully mv HSqlDb.${YEAR}.tgz to ${BACKUP}/Releases/LA/laData/HSqlDb.2.3.2/.
     => Manully rm -rf HSqlDb.${YEAR}

4. Generate pure ASCII Database (for MetaMap Project)
   • generate and copy 4 ASCII files to ${LA_DIR}/data/tables.ascii

     shell> cd ${LA_PRE_DIR}/bin
     shell> 3.GenerateAsciiDbFiles ${YEAR}
     1
     2
     3
     4
     5
     

   • LoadDb Program (silimar to Step in UTF-8):
     • Update ${lexAccessDbData2013+}/loadDb/sources/gov/nih/nlm/nls/lexAccess/loadDb/HSqlDb/db.cfg
       • DB_NAME=lexAccess${YEAR}ASCII

     • shell> cd ${lexAccessDbData2013+}/data.${YEAR}
       shell> ln -s ./tables.ascii tables
       shell> cd ${lexAccessDbData2013+}/loadDb/bin
       shell> 2.LoadDb
       	1
       	3
       

     • change "readonly=true" in *.properties after tables are loaded
     • Rename HSqlDb to HSqbDb.ascii in ${LA_DIR}/data/
   • laData (silimar to Step in UTF-8):
     shell> 3.GenerateAsciiDbFiles ${YEAR}
6
   • Backup (silimar to Step in UTF-8):
     shell> 3.GenerateAsciiDbFiles ${YEAR}
7

     => Manully mv HSqlDb.${YEAR}ASCII.tgz to ${BACKUP}Releases/LA/laData/HSqlDb.2.3.2/.
     => Manully rm HSqlDb.${YEAR}ASCII

5. Add new Database to ${PROJECT}/lexAccess
   • LexAccess2023 & lagt:
     • ${LA_DIR} is ${PROJECTS}/LA/lexAccess2023
     • Update config file: ${LA_DIR}/data/config/lexAccess.properties
       • Update LATEST_VERSION=${YEAR} for the lastest version
       • Update DB_NAME=lexAccess${YEAR} for default Db table
     • Copy HSqlDb files to ${PROJECTS}/LA/laData/HSqlDb
       => link to /nfsvol/lex/Lu/Development/LA/laData/HSqlDb
       • lexAccess${YEAR}.data,
       • lexAccess${YEAR}.properties
       • lexAccess${YEAR}.script

       • lexAccess${YEAR}ASCII.data,
       • lexAccess${YEAR}ASCII.properties
       • lexAccess${YEAR}ASCII.script

6. go through unit test