Howto install and Configure a Regional MonaLisa Collector

Installing and Configuring a Regional MonaLisa Collector (MLrepository)

Useful links: MonaLisa Cluster Monitoring by Costin Grigoras [1]

1) Check your host definition file (linux: /etc/hosts file). Make sure your machine is properly listed

127.0.0.1 YourMachineName localhost.localdomain localhost
::1       localhost6.localdomain6 localhost6

You might need other entries.
2) execute the command hostsname -f and make sure it returns your Fully Qualified Domain Name (FWDN).
3) Check that you have the correct java version. java -version It needs to be versin 1.6 or greater.
4) From a browser, download the Mona Lisa Repository tar ball from
http://monalisa.cern.ch/monalisa_Download_MonALISA_Service.html
and then click on the tab labeled "Downloads". From there get the latest version of the "MonALISA Repository Download".
5) The installer will want to make an account, or you can create one yourself (typically MonaLisa or some such thing). It could be good to extract the tar ball you got in step 4) there.
6) NOT AS ROOT! extract the tar ball tar -zxvf MLrepository.tar.bz2. This will create a MLrepository directory. The INSTALL file in MLrepository will give you the most up to date information.
7) check your firewalls to see that ports 8080 and 5432 are open (8080 and 5432 at the very least. Check files MLrepository/conf/ostgresql.conf and storage_configuration.properties for the correct port numbers.). On many Linux systems under the "System" tab, choose "Administration" and the application "Security Level and Firewall". Here you can see which ports are "open" and for which protocols. If needed add 8080 as a tcp protocol port. This will require root privilege. nmap localhost will show you which ports are open and being used (8080 should not be listed there until you start the repository).
8) Enter into MLrepository and run the install.sh script. If you created an account enter that account name. Do not start the repository yet (no big deal if you did).
9) To display the google map on the web page, you will need to get a key from google. Note this google map v2 is obsolete. It is being changed to google map v3.

i) go to the google web page
https://developers.google.com/maps/documentation/javascript/v2/introduction#Obtaining_Key
You will need a google account. If you don't have one you will be asked to create one. Click on the link to create the key. You will be shown a long list of applications. Choose "google maps API v2" and switch it on. If you don't see this list, make sure the panel on the top left has "Services" selected. Then select "API Access" and listed under "Simple API Access" will be some information including the key.

ii) Open the file MLrepository/tomcat/webapps/ROOT/map.jsp at about line 48 is listed

 String sKey = "abunchoflettersandnumbers";

and replace the string with the key from the google page (copy and past so that you don't mistake an O for 0.

10) now from MLrepository execute the start.sh script.
11) In a browser enter the browser location "http://localhost:8080/" and this should display a MonaLisa page with a google map.
12) Now also try this with your machines tcp/ip name (for ALICE-USA I am using http://pdsf1.nersc.gov:8080) {only from inside NERSC}, except that it is now accessible from http://portal.nersc.gov/ALICE). Also note, that to make this address work with ALICE, I have had to add a link from .../MLrepository/tomcat/webapp/ROOT to .../MLrepository/tomcat/webapp/ALICE and I have had to change many of the files below to point/use the ALICE directory (ROOT is the default and not shown in these files).
If you get a massage that you need a google maps key, you did not do part 9 correctly.
13) Now stop the repository so that we can finish configuring it.
i) edit MLrepository/JStoreClient/conf/App.properties
Line containing "lia.Monitor.group =" should be equal to your group (for our case that is alice).
Line containing "lia.Monitor.Jiniclient.Store.predicates=" should list all of the sites/cluster/data to be collected and stored in the repository (for ALICE-USA I have, at the moment,

 LBL/ABPing/,CERN/ALICE_Sites_Jobs_Summary/%/RUNNING_jobs,\
 LBL/ALICE::LBL::SE*/*/*/,LBL/FDT_Transfers/*/*/,LBL/Master/*/*/,\
 LBL/MonaLisa*/*/*/,LBL/SE_Traffic_Summary/*/*,\
 LBL/LBL_JobAgent_Summary/ce/,LBL/LBL_Nodes*/,LBL/Site_*/*/,\
 LBL/LBL_CE_Alice::LBL::CONDOR/*/,LBL/XrdServers*/,\
 LLNL/ABPing/,LLNL/ALICE::LLNL::SE*/*/*/,LLNL/FDT_Transfers/*/*/,\
 LLNL/Master/*/*/,\
 LLNL/MonaLisa*/*/*/,LLNL/SE_Traffic_Summary/*/*,\
 LLNL/LLNL_JobAgent_Summary/ce/,LLNL/LLNL_Nodes*/,LLNL/Site_*/*/,\
 LLNL/LLNL_CE_Alice::LLNL::CREAM/*/,LLNL/XrdServers*/,\
 OSC/ABPing/,OSC/ALICE::OSC::SE*/*/*/,OSC/FDT_Transfers/*/*/,\
 OSC/Master/*/*/,\
 OSC/MonaLisa*/*/*/,OSC/SE_Traffic_Summary/*/*,\
 OSC/OSC_JobAgent_Summary/ce/,OSC/OSC_Nodes*/,OSC/Site_*/*/,\
 OSC/OSC_Alice::OSC::PBS/*/,OSC/XrdServers*/,\
 OSC/ABPing/,NERSC/FDT_Transfers/*/*/, \
 NERSC/Master/*/*/,\
 NERSC/MonaLisa*/*/*/,\
 NERSC/Site_*/*/,\
 NERSC/XrdService*/

where "\" is a continuation line.

NERSC Notes

The MonALISA starts a cron job. At NERSC, this cron job needs to be modified, because they use a virtual OS system the OS needs to be chosen and properly initilized. To do this modify the MLrepository/start.sh script. There the line

CRON=`crontab -l | grep -v -E -e "(scripts/verify.sh)|(^#.+(DO NOT)|(/tmp/crontab)|Cron version)"`

(echo "$CRON"; echo "* * * * * `pwd`/scripts/verify.sh >> `pwd`/scripts/verify.log 2>&1") | crontab -

should read

CRON=`crontab -l | grep -v -E -e "(scripts/verify.sh)|(^#.+(DO NOT)|(/tmp/crontab)|Cron version)"`

(echo "$CRON"; echo "* * * * * CHOS=sl64 /usr/bin/chos `pwd`/scripts/verify.sh >> `pwd`/scripts/verify.log 2>&1") | crontab -

The CHOS=sl64 chooses Scientific Linux 6.4 and the /usr/bin/chos make this work.

Adding additional information to the MonaLisa Collector

To add additional information which can be monitored by MonaLisa first that information must be collected by the site's collector. The file which configures what data will be collected can be found in the sites MonaLisa directory tree (~/MonaLisa/Service/myFarm). The file is called ml.properties. This file configures many features of the local collector including what data will be collected. To collect information that is not supported by an existing package (class) an External Script can be used. The command to be added to the lm.properties files is
*ExternalScripts{monStatusCmd, localhost, "/path/to/the/command.sh"}%60
ExternalScripts is the existing class where this collected data is kept in.
monStatusCmd is the "Command/filter" which decodes the <tab> separated file of information.
localhost is the host name of the host where the data is located only experts should use something other than localhost.
/path/to/the/command.sh is the shell script to be executed to collect and configure the data. Its output should be in the form Property_Class<tab>Property_name_0<tab>Value_0 [...<tab>property_name_1<tab>Value_1]

Notes on Moving repository and/or MonALISA tomcat

Moving MonALISA and its repository to a new location, file, disk, or machine, is not difficult. To start with tar up the existing installation (starting from MLrepository and down) and untar it into its new location. Now there are a few files with will need to be modified with the new location.
.../MLrepository/conf/env.JAVA_HOME this file should contain the path to the java jdk, output from

which java

. It is probably best it this file does not end with a new line (blank line). One way to do this would be to

echo -n "/usr/common/usg/software/java/jdk1.7.0_60" > env.JAVA_HOME

.
.../MLrepository/conf/env.ACCOUNT, if you changed user, this file will need to be modifite as well (don't forget to make sure that the file owner have been properly changed as well).
.../MLrepository/conf/storage_configuration.properties the line lia.Repository.selfMonitoring.database_location= should point to the location of the data repository. Probably best to resolve links where since this is access a lot. Note: the line like lia.Monitor.UserName = mon_user should not be changed since this is the database userName and not the MonALISA account name. The line starting with lia.Monitor.JiniClient.Store.predicates= lists the data which this repository will keep and should be changed accordingly.
.../MLrepository/tomcat/webapps/ROOT/WEB_INF/classes/alimonitor/Page.java, and any other .java file there, and make sure its path is correct. You are likely going to need to recompile this using .../MLrepository/tomcat/webapps/ROOT/WEB-INF/classes/compile.sh and then stop/start MonALISA.

Log files to look into for problems

There are a number of different log files to search to find problems, these include:
.../MLrepository/JStoreClient/log.out which contains messages from the Java Virtual Machine (JVM) (started by the njGlobal shell scripit in that same directory. This is also where System.println(...) statements will output to.
.../MLrepository/logs/lazyj/error and .../MLrepository/logs/lazyj/waring is another place to look.
.../MLrepository/pgsql_store/init.log will have info on the repository db.
.../MLrepository//pgsql_store/postmaster.log Shows problems with the repository. Another place to look for problems is using your browser's developers tools (firefox Tools->Web Developer->Toggle Tools will open up a pannel and the Console tab (and others) there will have many massages, some relevant.

Files of Interest

There are a number of files that are interesting if you are wanting to make modifications to this MonALISA Monitoring page.
.../MLrepository/tomcat/ALICE/js/map/map_functions.js ---- This file setup the google map and is where the default and other view are defined.
.../MLrepository/tomcat/ALICE/js/map/osg_functions.js ---- Same as above (I don't know which is used).
.../MLrepository/tomcat/conf/tomcat-users.xml --- Is where MonALISA users are defined (admin to be specific).