Building a Standalone IJP in a VM

This document describes a stand-alone installation of the ICAT Job Portal on a single machine. It describes specific configuration details for this particular installation, on a linux (CentOS) VM, covering details that are to some extent abstract in the installation instructions. It is an example of one way in which this can be done; it is by no means mandatory; and details and processes on other systems are likely to be different.

By a "standalone" instance we mean a single system that acts as both a central IJP server and as a batch server. (Indeed, in our case, we also installed ICAT and IDS instances, though this is not obligatory.) This means that we installed these components:

ijp.server
The central IJP server, which provides the portal.
ijp.unixbatch
"Leaf" batch server that uses the Unix batch system to submit jobs
ijp.demojobs
A set of job types and executable scripts that can be used to demonstrate and test basic usage of the IJP

We will concentrate on installation details that are deliberately abstract in the installation notes of each component, particularly as regards prerequisites. Some key configuration decisions / steps are:

  • creation of appropriate users and user groups
  • creation of folders to hold job output etc.
  • modifications to /etc/sudoers

Basic Prerequisites: mysql, python, glassfish

The system in this case was a VM running CentOS linux, called sig-23. We created an initial user, br54; this account became the glassfish owner and was used for the bulk of the ICAT component installations.

MySQL

Outline

  • install mysql
  • run mysql_secure_installation
  • create icat user and database

Detail

Mysql was already installed on the VM, but the root password was unknown, so we removed and reinstalled it; running as root:

  • yum remove mysql
  • found the existing mysql DBs (in /var/lib/mysql or thereabouts), and deleted them (rm -rf *)
  • yum install mysqld
  • Start mysqld: service mysqld start
  • Run mysql_secure_installation, changing the (mysql) root password and disabling the anonymous account.

For the local ICAT installation, we created an icat user and database in mysql: 

sig-23$ mysql -u root -p
Password: *****
mysql> create user 'icat' identified by 'secret';
...
mysql> grant all on icat.* to icat;
...

and: 

sig-23$ mysql -u icat -p
Password: *****
mysql> create database icat;
...

Later, we created separate databases for IDS and IJP.

Python

  • Confirmed that the python version was 2.6.6 (which matches the ICAT requirement).
  • Confirmed that python-suds is installed.

Glassfish

We installed glassfish 4.0, following the Glassfish installation instructions, including extraction and import of the unsigned security certificate (into /usr/lib/java/jre/lib/security).

(Note: in the first attempt, we installed glassfish 4.1; this created problems that were not revealed until we tried testicat after the ICAT installation. Replacement with glassfish 4.0 worked.)

Downloaded a JDBC connector jar for mysql, and added to /home/br54/glassfish4/glassfish/domains/domain1/lib/.

ICAT and IDS

It is not necessary to install local instances of ICAT and IDS, so we will not go into detail here, except to mention that they were installed locally (as was ICE). A prerequisite is that at least one authenticator should be installed first. We installed authn_simple and authn_db. icat_setup was used to create a user in the powerusers group, which was given the rule ALL CRUD.

The initial configuration of an IJP installation uses (absolute) URLs for the ICAT and IDS instances on this VM. Any other installation will have to change these.

ICE was used to create some trivial dummy entities (TestFacility, TestDataset, etc.); the initial configuration of the demo jobs scripts defines properties that match this dummy setup. These should be changed to align with your chosen ICAT / IDS instances.

Later, we modified ICE's configuration by adding Dataset to the beans.list property; this enabled us to create initial datasets (TestDataset, TestDataset2) that could be found in IJP searches, and to which we could add datafiles from test jobs.

IJP Server

In practice, it would be reasonable to install the unixbatch batch server at this point, but we did it in the opposite order.

We downloaded the IJP snapshot distribution into /home/br54/Downloads/, and unzipped it there to obtain ijp.server/.

Created a database in mysql for ijp (still using the icat user): 

mysql -u root -p 
create database ijp; 
grant all on ijp.* to 'icat'@'localhost';

The installation was straightforward. (Well, eventually: the setup process was honed over several iterations. It should be straightforward now!). As with other ICAT component installations, we ran ./setup configure -v and followed instructions, modifying configuration files to match the specific system configuration at each stage.

Modified ijp-setup.properties: 

# Driver and connection properties for MySQL
driver=com.mysql.jdbc.jdbc2.optional.MysqlDataSource
dbProperties=user=icat:password=secret:databaseName=ijp

# Glassfish home, must contain "glassfish/domains"
glassfish=/home/br54/glassfish4

# Glassfish admin port (normally 4848)
port=4848

Modified ijp.properties: 

icat.url = https://sig-23.esc.rl.ac.uk:8181
ids.url = https://sig-23.esc.rl.ac.uk:8181

reader = simple username br54 password secret

# unixbatch on sig-23:
batchserverUrls = https://sig-23.esc.rl.ac.uk:8181

# authn methods
authn.list simple db

authn.simple.list username password
authn.simple.password.visible false

authn.db.list username password
authn.db.password.visible false

authn.anon.list

The other configuration files (families.xml, dataset_search_items.xml, datafile_search_items.xml and the ijp subfolder hierarchy) were left unchanged; but note that the family names families.xml must match configuration details in unixbatch.

Once configuration was complete, we ran ./setup install -v.

Unixbatch

Create batch user accounts

As root, we created a unixbatch group and users: 

[root@sig-23 ijp]# groupadd unixbatch
[root@sig-23 ijp]# useradd -g unixbatch batch01
[root@sig-23 ijp]# useradd -g unixbatch batch02
[root@sig-23 ijp]# useradd -g unixbatch batch03
[root@sig-23 ijp]# useradd -g unixbatch ingest01
[root@sig-23 ijp]# useradd -g unixbatch ingest02

These match the default configuration in unixbatch.properties.

Modifications to the sudoers file

As root, we used visudo to modify /etc/sudoers as described in the unixbatch installation notes:

  • Commented-out the line: 
    # Defaults requiretty
  • Added an entry to allow glassfish to sudo to the batch users to run the unix batch system commands: 
    br54        ALL=(batch01,batch02,batch03,ingest01,ingest02) NOPASSWD: /usr/bin/batch, /usr/bin/atq, /usr/bin/atrm, /usr/bin/kill

(We will modify /etc/sudoers again later, to add the job scripts folder to the security policy.)

Create mysql database

mysql -u root -p 
create database unixbatch; 
grant all on unixbatch.* to 'icat'@'localhost';

Installation and configuration

We gave the glassfish user (br54) access to the home directory of each of the batch users: 

setfacl -m user:br54:rx /home/batch01

etc.

Created /home/br54/unixbatch/jobOutputDir/. The installation instructions say that this folder has to be readable by the batch users; this means that /home/br54/ and /home/br54/unixbatch/ must be readable too. The simplest way to do this is (as the glassfish user br54): 

chmod +rx /home/br54
chmod +rx /home/br54/unixbatch
chmod +rx /home/br54/unixbatch/jobOutputDir

A more precise solution might be to add the glassfish user to the unixbatch group, then do chmod g+rx etc.

unixbatch-setup.properties was modified: 

# Driver and connection properties for MySQL
driver=com.mysql.jdbc.jdbc2.optional.MysqlDataSource
dbProperties=user=icat:password=icat:databaseName=unixbatch

# Glassfish home, must contain "glassfish/domains"
glassfish=/home/br54/glassfish4

# Glassfish admin port (normally 4848)
port=4848

unixbatch.properties was modified: 

icat.url = https://sig-23.esc.rl.ac.uk:8181

families.list = batch ingest
families.batch = batch01 batch02 batch03
families.ingest = ingest01 ingest02

jobOutputDir = /home/br54/unixbatch/jobOutputDir

Note that families.list matches the family names in families.xml in the IJP server configuration.

At this stage, it was possible to run the simple "date" job type. Further job types were added by hand; but we will try to describe the process in terms of the ijp.demojobs distribution.

Demo Jobs

Prerequisites

As root, we created a dedicated folder to hold batch job scripts, /opt/ijp/bin. The job scripts have to be visible to the batch users, when batch jobs are run from glassfish via sudo. This means that we have to add the above folder to the sudoers security policy. We did this by modifying (as root) /etc/sudoers using visudo: 

# Add /opt/ijp/bin to secure_path, for ijp/unixbatch
Defaults    secure_path = /sbin:/bin:/usr/sbin:/usr/bin:/opt/ijp/bin

(We also added /opt/ijp/bin to each batch user's PATH, by editing their profile; this may not have been necessary, as the path should be set by sudo.)

When the IJP server and the batch server are on the same machine, the demojobs installer can be used to install the job types in the IJP server, and the job scripts to a scripts folder. To do this, the installer must have write access to both locations. This suggests that the scripts folder should belong to the glassfish user. However, our root-owned scripts folder adds an extra complication: when the demojobs installer is run by br54, it has to be configured to extract the scripts to a folder owned by br54, and then the root user has to copy them to /opt/ijp/bin. In retrospect, it would have been simpler to add the br54-owned scripts folder to the sudoers secure path.

Installation

Configured demojobs.properties as follows: 

# Configuration for IJP sample job scripts

# Glassfish connection details (if current system is an IJP server)
# Glassfish home, must contain "glassfish/domains"
glassfish=/home/br54/glassfish4

# Glassfish admin port (normally 4848)
port=4848

# Location on batch server for job scripts (if current system is an IJP batch server)
# ijp.scripts.dir=/opt/ijp/bin
ijp.scripts.dir=/home/br54/Downloads/scripts-test/scripts

# DatasetType 1 - used in most job types; must exist in ICAT
dataset.type.1=TestDatasetType

# DatasetType 2 - used in some job types; must exist in ICAT
dataset.type.2=TestDatasetType2

# Facility name (used by copy/create datafile jobs): must exist in ICAT
facility.name=TestFacility

# Data format and version (used by create_datafile): must exist in ICAT
data.format.name=TestDataFormat
data.format.version=0.1

Note:

  • we used a br54-owned folder as the scripts target
  • the dataset, facility and data format properties are chosen to match entities in the ICAT instance

Post-installation

We ran ./setup install -v to install the job types into the IJP server, and the job scripts into /home/br54/Downloads/scripts-test/scripts/. As root, then we copied the scripts from there into /opt/ijp/bin/.

It was now possible to use the IJP web interface to submit jobs to the unix batch system, etc.