Howto install and Configure a Regional MonaLisa Collector
Contents |
Installing and Configuring a Regional MonaLisa Collector (MLrepository)
Useful links: MonaLisa Cluster Monitoring by Costin Grigoras [1]
1) Check your host definition file (linux: /etc/hosts file). Make sure your machine is properly listed
127.0.0.1 YourMachineName localhost.localdomain localhost ::1 localhost6.localdomain6 localhost6
You might need other entries.
2) execute the command hostsname -f and make sure it returns your Fully Qualified Domain Name (FWDN).
3) Check that you have the correct java version. java -version It needs to be versin 1.6 or greater.
4) From a browser, download the Mona Lisa Repository tar ball from
http://monalisa.cern.ch/monalisa_Download_MonALISA_Service.html
and then click on the tab labeled "Downloads". From there get the latest version of the "MonALISA Repository Download".
5) The installer will want to make an account, or you can create one yourself (typically MonaLisa or some such thing).
It could be good to extract the tar ball you got in step 4) there.
6) NOT AS ROOT! extract the tar ball tar -zxvf MLrepository.tar.bz2. This will create a MLrepository directory. The INSTALL
file in MLrepository will give you the most up to date information.
7) check your firewalls to see that ports 8080 and 5432 are open (8080 and 5432 at the very least. Check files MLrepository/conf/ostgresql.conf
and storage_configuration.properties for the correct port numbers.). On many Linux systems under
the "System" tab, choose "Administration" and the application "Security Level and Firewall". Here you can see which ports
are "open" and for which protocols. If needed add 8080 as a tcp protocol port. This will require root privilege. nmap localhost
will show you which ports are open and being used (8080 should not be listed there until you start the repository).
8) Enter into MLrepository and run the install.sh script. If you created an account enter that account name. Do not start
the repository yet (no big deal if you did).
9) To display the google map on the web page, you will need to get a key from google. Note this google map v2 is
obsolete. It is being changed to google map v3.
i) go to the google web page
https://developers.google.com/maps/documentation/javascript/v2/introduction#Obtaining_Key
You will need a google account. If you don't have one you will be asked to create one. Click on the link to create the key. You will be shown a long list of applications. Choose "google maps API v2" and switch it on. If you don't see this list, make sure the panel on the top left has "Services" selected. Then select "API Access" and listed under "Simple API Access" will be some information including the key.
ii) Open the file MLrepository/tomcat/webapps/ROOT/map.jsp at about line 48 is listed
String sKey = "abunchoflettersandnumbers";
and replace the string with the key from the google page (copy and past so that you don't mistake an O for 0.
10) now from MLrepository execute the start.sh script.
11) In a browser enter the browser location "http://localhost:8080/" and this should display a MonaLisa page with a google map.
12) Now also try this with your machines tcp/ip name (for ALICE-USA I am using http://pdsf1.nersc.gov:8080) {only from inside NERSC},
except that it is now accessible from http://portal.nersc.gov/ALICE). Also note, that to make this address work with ALICE, I have
had to add a link from .../MLrepository/tomcat/webapp/ROOT to .../MLrepository/tomcat/webapp/ALICE and I have had to change many
of the files below to point/use the ALICE directory (ROOT is the default and not shown in these files).
If you get a massage that you need a google maps key, you did not do part 9 correctly.
13) Now stop the repository so that we can finish configuring it.
i) edit MLrepository/JStoreClient/conf/App.properties
Line containing "lia.Monitor.group =" should be equal to your group (for our case that is alice).
Line containing "lia.Monitor.Jiniclient.Store.predicates=" should list all of the sites/cluster/data to be collected and stored
in the repository (for ALICE-USA I have, at the moment,
LBL/ABPing/,CERN/ALICE_Sites_Jobs_Summary/%/RUNNING_jobs,\ LBL/ALICE::LBL::SE*/*/*/,LBL/FDT_Transfers/*/*/,LBL/Master/*/*/,\ LBL/MonaLisa*/*/*/,LBL/SE_Traffic_Summary/*/*,\ LBL/LBL_JobAgent_Summary/ce/,LBL/LBL_Nodes*/,LBL/Site_*/*/,\ LBL/LBL_CE_Alice::LBL::CONDOR/*/,LBL/XrdServers*/,\ LLNL/ABPing/,LLNL/ALICE::LLNL::SE*/*/*/,LLNL/FDT_Transfers/*/*/,\ LLNL/Master/*/*/,\ LLNL/MonaLisa*/*/*/,LLNL/SE_Traffic_Summary/*/*,\ LLNL/LLNL_JobAgent_Summary/ce/,LLNL/LLNL_Nodes*/,LLNL/Site_*/*/,\ LLNL/LLNL_CE_Alice::LLNL::CREAM/*/,LLNL/XrdServers*/,\ OSC/ABPing/,OSC/ALICE::OSC::SE*/*/*/,OSC/FDT_Transfers/*/*/,\ OSC/Master/*/*/,\ OSC/MonaLisa*/*/*/,OSC/SE_Traffic_Summary/*/*,\ OSC/OSC_JobAgent_Summary/ce/,OSC/OSC_Nodes*/,OSC/Site_*/*/,\ OSC/OSC_Alice::OSC::PBS/*/,OSC/XrdServers*/,\ OSC/ABPing/,NERSC/FDT_Transfers/*/*/, \ NERSC/Master/*/*/,\ NERSC/MonaLisa*/*/*/,\ NERSC/Site_*/*/,\ NERSC/XrdService*/
where "\" is a continuation line.
NERSC Notes
The MonALISA starts a cron job. At NERSC, this cron job needs to be modified, because
they use a virtual OS system the OS needs to be chosen and properly initilized. To do
this modify the MLrepository/start.sh script. There the line
CRON=`crontab -l | grep -v -E -e "(scripts/verify.sh)|(^#.+(DO NOT)|(/tmp/crontab)|Cron version)"`
(echo "$CRON"; echo "* * * * * `pwd`/scripts/verify.sh >> `pwd`/scripts/verify.log 2>&1") | crontab -
should read
CRON=`crontab -l | grep -v -E -e "(scripts/verify.sh)|(^#.+(DO NOT)|(/tmp/crontab)|Cron version)"`
(echo "$CRON"; echo "* * * * * CHOS=sl64 /usr/bin/chos `pwd`/scripts/verify.sh >> `pwd`/scripts/verify.log 2>&1") | crontab -
The CHOS=sl64 chooses Scientific Linux 6.4 and the /usr/bin/chos make this work.
Adding additional information to the MonaLisa Collector
To add additional information which can be monitored by MonaLisa first that information must be collected
by the site's collector. The file which configures what data will be collected can be found in the
sites MonaLisa directory tree (~/MonaLisa/Service/myFarm). The file is called ml.properties. This file
configures many features of the local collector including what data will be collected. To collect information
that is not supported by an existing package (class) an External Script can be used. The command to be added
to the lm.properties files is
*ExternalScripts{monStatusCmd, localhost, "/path/to/the/command.sh"}%60
ExternalScripts is the existing class where this collected data is kept in.
monStatusCmd is the "Command/filter" which decodes the <tab> separated file of information.
localhost is the host name of the host where the data is located only experts should use
something other than localhost.
/path/to/the/command.sh is the shell script to be executed to collect and configure the data.
Its output should be in the form Property_Class<tab>Property_name_0<tab>Value_0 [...<tab>property_name_1<tab>Value_1]
Notes on Moving repository and/or MonALISA tomcat
Moving MonALISA and its repository to a new location, file, disk, or machine, is not difficult. To start with tar up the existing
installation (starting from MLrepository and down) and untar it into its new location. Now there are a few files with will need
to be modified with the new location.
.../MLrepository/conf/env.JAVA_HOME this file should contain the path to the java jdk, output from
which java
. It is probably best it this file does not end with a new line (blank line). One way to do this would be to
echo -n "/usr/common/usg/software/java/jdk1.7.0_60" > env.JAVA_HOME
.
.../MLrepository/conf/env.ACCOUNT, if you changed user, this file will need to be modifite as well (don't forget to
make sure that the file owner have been properly changed as well).
.../MLrepository/conf/storage_configuration.properties the line lia.Repository.selfMonitoring.database_location= should
point to the location of the data repository. Probably best to resolve links where since this is access a lot. Note: the line like
lia.Monitor.UserName = mon_user should not be changed since this is the database userName and not the MonALISA account name. The
line starting with lia.Monitor.JiniClient.Store.predicates= lists the data which this repository will keep and should be changed
accordingly.
.../MLrepository/tomcat/webapps/ROOT/WEB_INF/classes/alimonitor/Page.java, and any other .java file there, and make sure its
path is correct. You are likely going to need to recompile this using .../MLrepository/tomcat/webapps/ROOT/WEB-INF/classes/compile.sh
and then stop/start MonALISA.
Log files to look into for problems
There are a number of different log files to search to find problems, these include:
.../MLrepository/JStoreClient/log.out which contains messages from the Java Virtual Machine (JVM) (started by the njGlobal shell
scripit in that same directory. This is also where System.println(...) statements will output to.
.../MLrepository/logs/lazyj/error and .../MLrepository/logs/lazyj/waring is another place to look.
.../MLrepository/pgsql_store/init.log will have info on the repository db.
.../MLrepository//pgsql_store/postmaster.log Shows problems with the repository.
Another place to look for problems is using your browser's developers tools (firefox Tools->Web Developer->Toggle Tools will open up a pannel
and the Console tab (and others) there will have many massages, some relevant.
Files of Interest
There are a number of files that are interesting if you are wanting to make modifications to this MonALISA
Monitoring page.
.../MLrepository/tomcat/ALICE/js/map/map_functions.js ---- This file setup the google map and is where the default and other view are defined.
.../MLrepository/tomcat/ALICE/js/map/osg_functions.js ---- Same as above (I don't know which is used).
.../MLrepository/tomcat/conf/tomcat-users.xml --- Is where MonALISA users are defined (admin to be specific).