Setting Up

The web service is accessed via a proxy (conventionally known as a port). The proxy (here given a variable name of icat) may be obtained by the following:

URL hostUrl = new URL("https://<hostname>:8181")
URL icatUrl = new URL(hostUrl, "ICATService/ICAT?wsdl");
QName qName = new QName("http://icatproject.org", "ICATService");
ICATService service = new ICATService(icatUrl, qName);
ICAT icat = service.getICATPort();

where <hostname> should be the full name of the ICAT server. For a secure installation, just specifying localhost will not work, the name must match what is on the host certificate.

Session management

When you login to ICAT you will be given back a string, the sessionId, which must be used as the first argument of almost all ICAT calls. The only exceptions being the login call itself, getEntityInfo and getApiVersion - none of which require authentication.

String login(String plugin, Credentials credentials)

where the plugin is the mnemonic defined in the ICAT installation for the authentication plugin you wish to use and credentials is essentially a map. The names of the keys and their meaning is defined by the plugin.

This sessionId returned will be valid for a period determined by the ICAT server.

The example below shows how it works for the authn_db plugin at the time of writing, where the plugin has been given the mnemonic "db".

Credentials credentials = new Credentials();
List<Entry> entries = credentials.getEntry();
Entry e;

e = new Entry();
e.setKey("username");
e.setValue("root");
entries.add(e);
e = new Entry();
e.setKey("password");
e.setValue("secret");
entries.add(e);

String sessionId = icat.login("db", credentials);

double getRemainingMinutes(String sessionId)

This returns the number of minutes left in the session. A session may not be extended but a user may have more than one session at once.

void logout(String sessionId)

This invalidates the sessionId.

Exceptions

There is only one exception thrown by ICAT. This is the IcatException_Exception which is a wrapper around the real exception which in turn includes an enumerated code to identify the kind of exception and the usual message. The codes and their meanings are:

BAD_PARAMETER

generally indicates a problem with the arguments made to a call.

INTERNAL

may be caused by network problems, database problems, glassfish problems or bugs in ICAT.

INSUFFICIENT_PRIVILEGES

indicates that the authorization rules have not matched your request.

NO_SUCH_OBJECT_FOUND

is thrown when something is not found.

OBJECT_ALREADY_EXISTS

is thrown when type to create something but there is already one with the same values of the constraint fields.

SESSION

is used when the sessionId you have passed into a call is not valid or if you are unable to authenticate.

VALIDATION

marks an exception which was thrown instead of placing the database in an invalid state.

For example to print what has happened you might use the following:

String sessionId;
try {
   sessionId = icat.login("db", credentials);
} catch (IcatException_Exception e) {
   IcatException ue = e.getFaultInfo();
   System.out.println("IcatException " + ue.getType() + " " + ue.getMessage()
    + (ue.getOffset() >= 0 ? " at offset " + ue.getOffset() : ""));
}

Operations which work on a list of objects, such as createMany, may fail because of failure to process one of the objects. In this case the state of the database will be rolled back and the offset within the list of the entry causing the error will be stored in the IcatException. For other calls the offset will be negative, as it is with certain internal exceptions which are not associated with any specific object in a list.

The schema

To understand exactly how the data manipulation calls work requires an understanding of the schema. Please take a look now to make sense of the following explanation.

Each table in the database, representing a set of entities, is mapped onto a class in the API so terminology mixes OO and database concepts. Each class has uniqueness constraints, relationships and other fields. Each object is identified by a field "id" which is managed by ICAT and is returned when you create an object. This is common to all objects and is not described in the schema. The "id" field is used as the primary key in the database. There will normally be some combinations of fields, some of which may be relationships, which must be unique across all entries in the table. This is marked as "Uniqueness constraint". For Dataset this is sample, investigation, name, type which, apart from name, are all relationships. No more than one one Dataset may exist with those four fields having the same value. These constraints are enforced by ICAT.

The relationship table is shown next. The first column shows the minimum and maximum cardinality of the relationships. A Dataset may be related to any number of OutputDatasets, to at most one investigation and to exactly one DatasetType. The next column shows the name of the related class and this is followed by the name of the field which is used to represent the relationship. The basic field name is normally the name of the related class unless it is ambiguous or unnecessarily long. The field name is in the plural for "to many" relationships. The next column, "cascaded", is marked yes to show that create and delete operations are cascaded. If a Dataset is deleted then all its OutputDatasets, DatasetParameters, Datafiles and InputDatasets are deleted at the same time by one call to ICAT. In a similar manner a tree, created in memory with a Dataset having a a set of Datafiles and Datasetparameters, can be persisted to ICAT in a single call. This will be explained more later.

Note that currently all "one to many" relationships are cascaded but no "many to one" relationships. Do not assume that this will always be true.

Note also that all relationships are navigable in both directions.

Creating an Object

long create(String sessionId, EntityBaseBean bean)

To create an object in ICAT, first instantiate the object of interest, for example a Dataset, and then call the setters to set its attributes and finally make a call to create the object in ICAT.

So typical code in Java might look like:

Dataset ds = new Dataset();
ds.setName("Name of dataset");
ds.set ...
Long dsid = icat.create(sessionId, ds);

You will see that no convenient constructors are generated, rather each field of the object must be set individually. Most fields are optional and may be left with null values, however some are compulsory and the call to create will fail if they are not set. Each object has a primary key that identifies it in the database - this is a value of type "long" that is generated by ICAT and is used to represent relationships in a regular manner.

Some fields represent attributes of the object but others are used to represent relationships. The relationships are represented in the class definitions by a variable which either holds a reference to a single object or a list of objects. In the case of a list it may be "cascaded". Consider creating an Dataset with a set of Datafiles. Because the relationship from Dataset to Datafile is cascaded they may be created in one call as outlined below:

Dataset ds = new Dataset();
ds.setName(dsName);
ds.setType(type);
Datafile datafile = new Datafile();
datafile.setDatafileFormat(format);
datafile.setName(dfName);
ds.getDatafiles().add(datafile); // Add the datafile to the dataset
icat.create(sessionId, ds);

The call to create returns the key of the created object. If you choose to write:

ds.setId(icat.create(sessionId, ds));

then the client copy of the Dataset will be updated to have the correct key value - however the keys in any other objects "within" the Dataset will still be null on the client side. In this case datafile.getId() will remain null.

When creating multiple objects in one call, the value of the cascaded flag must be noted. The line ds.getDatafiles().add(datafile) requires that the datafile is not already known to ICAT because the cascade flag is set. If the cascaded flag is set then objects to be included in the "create" operation must not exist. However if the cascaded flag is not set then objects which are being referenced must already exist in ICAT.

We now have an example of adding a datafile to an existing dataset, ds

Datafile datafile = new Datafile();
datafile.setDatafileFormat(format);
datafile.setName(name);
datafile.setDataset(ds); // Relate the datafile to an existing dataset
datafile.setId(icat.create(sessionId, datafile)); // Create datafile and store id on client side

List <Long> createMany(String sessionId, List <EntityBaseBean> beans)

This call, as its name suggests, creates many objects. It takes the list of objects to create and returns a list of ids. If any of the individual operations fail the whole call fails and the database will be unchanged. The objects to be created need not be of the same type. For an example (where they are of the same type) consider adding many Datafiles to a existing Dataset, ds:

List <Datafile> dfs = new ArrayList<Datafile>();
for (int i = 0; i < n; i++) {
   final Datafile datafile = new Datafile();
   datafile.setDatafileFormat(dfmt);
   datafile.setName("bill" + i);
   datafile.setDataset(ds);
   dfs.add(datafile);
}
icat.createMany(sesionId, dfs); // many datafiles are stored in one call

Retrieving an object when you know its id

EntityBaseBean get(String sessionId, String query, long id)

If dsid is the id of a Dataset then it may be retrieved by the call:

Dataset ds = (Dataset) icat.get(sessionId, "Dataset", dsid);

The second parameter is a string holding the name of the type of object to retrieve and some other optional information. By default only the requested object is returned and no related objects. If you want the Dataset along with its related Datafiles, DatasetParameters and DatafileParameters then replace: "Dataset" with "Dataset INCLUDE Datafile,DatasetParameter,DatafileParameter"

The related types must be all be related to the original type or to some other type in the list. This means that you could not have "Dataset INCLUDE DatafileParameter" . Not all relationships can be followed, the rule appears complex but is there to support the authorization scheme which is explained later. Starting from the first type (before the INCLUDE keyword) cascaded "one to many" relationships may be followed repeatedly. In addition "many to one" relationships can be followed. The idea behind this is that cascaded "one to many" relationships represent composition/ownership - i.e. a dataset is made up of datafiles and the "many to one" is allowed as these are typically shared objects. This means that starting from a Dataset, one could follow the cascaded "one to many" relationships to get to Datafile, DatasetParameter and DatafileParameter. From this set of types (Dataset, Datafile, DatasetParameter and DatafileParameter) one can follow the "many to one" relationships to reach at most one of each of Facility, Investigation, InvestigationType, DatasetType, Instrument, FacilityCycle, ParameterType and DatafileFormat. Finally there must be only one route from the original type to each of the included types.

Updating an Object

void update(String sessionId, EntityBaseBean bean)

To update an object simply update the fields you want to change and call update. For example:

Dataset ds = (Dataset) icat.get(sessionId, "Dataset INCLUDE 1", dsid);
ds.setInvestigation(anotherInvestigation);
icat.update(sessionId, ds);

As suggested by the example above "many to one" relationships, such as the investigation relationship to the dataset, will be updated as will any simple field values. Consequently it is essential to get the existing values for any "many to one" relationships. This is most reliably achieved by the notation INCLUDE 1 as shown here. The effect of the "1" is to include all "many to one" related types. "One to many" relationships are ignored by the update mechanism so you need to start at the correct end of the relationship to have the desired effect.

Deleting an Object

void delete(String sessionId, EntityBaseBean bean)

The following code will get a dataset and delete it.

Dataset ds = (Dataset) icat.get(sessionId, "Dataset", dsid);
icat.delete(sessionId, ds);

All cascaded "one to many" related objects will also be deleted. In the extreme case, if you delete a facility, you lose everything associated with that facility. This privilege should not be given to many - see the authorization section later.

Searching for an Object

List<Object> search(String sessionId, String query)

A rather powerful search mechanism is provided. It will be introduced by means of examples:

List<Object> results = icat.search(sessionId, "Dataset");

will return all Datasets. If the query is:

"Dataset.name"

this will return all Dataset names. Multiple datasets with the same name are permitted and this call will include duplicates. Instead

"DISTINCT Dataset.name"

will avoid duplicates. To get related objects returned, then the same INCLUDE syntax that was described for the get call may be used with exactly the same restrictions and semantics:

"Dataset INCLUDE Datafile,DatasetParameter,DatafileParameter"

You can specify an order (which may precede or follow an INCLUDE clause):

"Dataset.id ORDER BY id"

Restrictions can be placed on the data returned. For example:

"Dataset.id [type.name IN ('GS', 'GQ')]"

which could also be written:

"Dataset.id [type.name = 'GS' OR type.name = 'GQ']"

The restriction in the square brackets can be as complex as required - but must only refer to attributes of the object being restricted - in this case the Dataset. Expressions may use parentheses, AND, OR, <, <=, >, >=, =, <>, !=, NOT, IN, LIKE and BETWEEN. Currently the BETWEEN operator does not work on strings. This appears to be a JPA bug.

Functions: MAX, MIN, COUNT, AVG and SUM may also be used such as:

"MAX (Dataset.id)"

Selection may involve more than one related object. To show the relationship a "<->" token is used. For example:

"Dataset.id <-> DatasetParameter[type.name = 'TIMESTAMP']"

Note also here the use of the JPQL style path: type.name . This expressions means ids of Datasets which have a DatasetParameter which has a type with a name of TIMESTAMP. Multiple " <->" may appear but all the objects involved, including the first one, must be connectable in only one way.

It is also possible to restrict the number of results returned by specifying a pair of numbers at the beginning of the query string. This construct would normally be used with an ORDER BY clause. The first number is the offset from within the full list of available results from which to start returning data and the second is the maximum number of results to return. These numbers if specified must be positive. If the offset is greater than or equal to the number of internal results then no data will be returned. The default values are 0 and "infinity". The numbers must be separated by a comma though either may be omitted. The following are all valid. The last example is rather pointless and does the same as the first. A number without a comma is illegal.

"    Dataset.id ORDER BY id"
"3,5 Dataset.id ORDER BY id"
"3,  Dataset.id ORDER BY id"
" ,5 Dataset.id ORDER BY id"
" ,  Dataset.id ORDER BY id"

Rules

By default access is denied to all objects, rules allow access. It is only necessary to be permitted by one rule where that rule is only applied to the object referenced directly in the API call. The Rule table has two exposed fields: crudFlags and what . The field crudFlags contains letters from the set "CRUD" to indicate which types of operation are being allowed (Create, Read, Update and/or Delete). The other field, what , is the rule itself. There is also a "many to one" relationship to Group which may be absent.

Consider:

Rule rule = new Rule();
rule.setGroup(userOffice);
rule.setCrudFlags("CRUD");
rule.setWhat("Investigation");
icat.create(sessionId, rule);

allows members of the userOffice group full access to all Investigations. Note that the id field of the rule on the client side is not set on the assumption that the client side copy of the rule will not be needed further.

Rule rule = new Rule();
rule.setGroup(null); // Not necessary as it will be null on a newly created rule
rule.setCrudFlags("R");
rule.setWhat("ParameterType");
icat.create(sessionId, rule);

allows any authenticated user (with a sessionId) to read Parameters. Consider a group of users fredReaders To allow fredReaders to read a datafile with a name of "fred" we could have:

Rule rule = new Rule();
rule.setGroup(fredReaders;
rule.setCrudFlags("R");
rule.setWhat("Datafile [name='fred']");
icat.create(sessionId, rule);

More complex restrictions can be added using other related objects. For example to allow read access to Datasets belonging to an Investigation which includes an InvestigationUser which has a user with a name matching the currently authenticated user (from the sessionId) we can have:

Rule rule = new Rule();
rule.setGroup(null);
rule.setCrudFlags("R");
rule.setWhat("Dataset <-> Investigation <-> InvestigationUser [user.name = :user]");
icat.create(sessionId, rule);

where the :user denotes the currently authenticated user (derived from the sessionId). You will note that the syntax is very similar to that used by the search.