Information for Users
- Java Client User Manual (SOAP)
- Java Client User Manual (REST)
- Java Client Javadoc
- Issue Tracking
- Release Notes
Information for Installers
Project Documentation
The ICAT4 API is a layer on top of a relational DBMS. The database is wrapped as a web service so that the tables are not exposed directly. Each table in the database is mapped onto a data structure exposed by the web service. When the web service interface definition (WSDL) is processed for Java then each data structure results in a class definition.
Installation and accessing from maven is explained in Java Client Installation.
The web service is accessed via a proxy (conventionally known as a port). The proxy (here given a variable name of icat) may be obtained by the following:
URL hostUrl = new URL("https://<hostname>:8181")where <hostname> should be the full name of the ICAT server. For a secure installation, just specifying localhost will not work, the name must match what is on the host certificate.
When you login to ICAT you will be given back a string, the sessionId, which must be used as the first argument of almost all ICAT calls. The only exceptions being the login call itself, getEntityInfo and getApiVersion - none of which require authentication.
String login(String plugin, Credentials credentials)
where the plugin is the mnemonic defined in the ICAT installation for the authentication plugin you wish to use and credentials is essentially a map. The names of the keys and their meaning is defined by the plugin.
This sessionId returned will be valid for a period determined by the ICAT server.
The example below shows how it works for the authn_db plugin at the time of writing, where the plugin has been given the mnemonic "db".
Credentials credentials = new Credentials();double getRemainingMinutes(String sessionId)
This returns the number of minutes left in the session. A user may have more than one session at once.
String getUserName(String sessionId)
This returns the string identifying the user of the session as provided by the authentication plugin.
void refresh(String sessionId)
This resets the time-to-live of the session as it was when the session was first obtained.
void logout(String sessionId)
This invalidates the sessionId.
There is only one exception thrown by ICAT. This is the IcatException_Exception which is a wrapper around the real exception which in turn includes an enumerated code to identify the kind of exception and the usual message. The codes and their meanings are:
For example to print what has happened you might use the following:
String sessionId;Operations which work on a list of objects, such as createMany, may fail because of failure to process one of the objects. In this case the state of the database will be rolled back and the offset within the list of the entry causing the error will be stored in the IcatException. For other calls the offset will be negative, as it is with certain internal exceptions which are not associated with any specific object in a list.
To understand exactly how the data manipulation calls work requires an understanding of the schema. Please take a look now to make sense of the following explanation.
Each table in the database, representing a set of entities, is mapped onto a class in the API so terminology mixes OO and database concepts. Each class has uniqueness constraints, relationships and other fields. Each object is identified by a field "id" which is managed by ICAT and is returned when you create an object. This is common to all objects and is not described in the schema. The "id" field is used as the primary key in the database. There will normally be some combinations of fields, some of which may be relationships, which must be unique across all entries in the table. This is marked as "Uniqueness constraint". For Dataset this is investigation, name where investigation represents a relationship. No more than one one Dataset may exist with those two fields having the same value. These constraints are enforced by ICAT.
The relationship table is shown next. The first column shows the minimum and maximum cardinality of the relationships. A Dataset may be related to any number of DataCollectionDatasets, to at most one Sample and to exactly one DatasetType. The next column shows the name of the related class and this is followed by the name of the field which is used to represent the relationship. The basic field name is normally the name of the related class unless it is ambiguous or unnecessarily long. The field name is in the plural for "to many" relationships. The next column, "cascaded", is marked yes to show that create and delete operations are cascaded. If a Dataset is deleted then all its DataCollectionDatasets, DatasetParameters and Datafiles are deleted at the same time by one call to ICAT. In a similar manner a tree, created in memory with a Dataset having a a set of Datafiles and Datasetparameters, can be persisted to ICAT in a single call. This will be explained more later.
Note that:
In addition each entity has four special attributes: createId, createTime, modId and modTime which are maintained by ICAT though they may be queried. For each each entity they record who created it and when, and who modified it and when.
long create(String sessionId, EntityBaseBean bean)
To create an object in ICAT, first instantiate the object of interest, for example a Dataset, and then call the setters to set its attributes and finally make a call to create the object in ICAT.
So typical code in Java might look like:
Dataset ds = new Dataset();You will see that no convenient constructors are generated, rather each field of the object must be set individually. Most fields are optional and may be left with null values, however some are compulsory and the call to create will fail if they are not set. Each object has a primary key that identifies it in the database - this is a value of type "long" that is generated by ICAT and is used to represent relationships in a regular manner.
Some fields represent attributes of the object but others are used to represent relationships. The relationships are represented in the class definitions by a variable which either holds a reference to a single object or a list of objects. In the case of a list it may be "cascaded". Consider creating a dataset with a set of datafiles. Because the relationship from dataset to datafile is cascaded they may be created in one call as outlined below:
Dataset ds = new Dataset();The call to create returns the key of the created object. If you choose to write:
ds.setId(icat.create(sessionId, ds));then the client copy of the Dataset will be updated to have the correct key value - however the keys in any other objects "within" the Dataset will still be null on the client side. In this case datafile.getId() will remain null.
When creating multiple objects in one call, the value of the cascaded flag must be noted. The line ds.getDatafiles().add(datafile) requires that the datafile is not already in the ICAT database because the cascade flag is set. If the cascaded flag is set then objects to be included in the "create" operation must not exist. However if the cascaded flag is not set then objects which are being referenced must already exist in the ICAT database.
It might help to understand what is happening when you call create. The client side object you pass in, and everything it refers to , is encoded as XML and sent to the server. There it is unpacked into the same set of objects and persisted in the database. The structure passed in must be a tree structure (or more correctly a DAG ) - if, for example, you modify the code above to create a dataset with one datafile and add datafile.setDataset(dataset) attempting to put in a reverse link which physically will appear in the ICAT database the call will be rejected by the client because you have a loop in the graph as you can then go backwards and forwards between datafile and dataset. If you passed in a proper DAG the database will create one row in the dataset table and one in the datafile table where the datafile entry includes the field datafile.dataset_id holding the dataset.id of the dataset entry just created. Relationships are represented in the database by holding the id value of the related object.
We now have an example of adding a datafile to an existing dataset, ds
Datafile datafile = new Datafile();This is the only way to create a datafile - you cannot do it by any operation upon the dataset. After the call is made a new entry will have been been added to the datafile table in the database where again the field datafile.dataset_id in the database will have been set to the dataset.id of the related dataset which in this case may be more intuitive as it corresponds to the line of user code datafile.setDataset(ds) .
It is worth noting that the dataset object you have in memory is not affected by the call to create a new datafile. More datafiles can be created referencing the same dataset without retrieving an updated copy of the dataset..
List <Long> createMany(String sessionId, List <EntityBaseBean> beans)
This call, as its name suggests, creates many objects. It takes the list of objects to create and returns a list of ids. If any of the individual operations fail the whole call fails and the database will be unchanged. The objects to be created need not be of the same type. For an example (where they are of the same type) consider adding many Datafiles to a existing Dataset, ds:
List <Datafile> dfs = new ArrayList<Datafile>();The only reason that this call exists is to minimize calls to ICAT as it is faster to call createMany than to make many calls to create .
There are alternative syntaxes for the query, the JPQL style syntax and the concise syntax. There are a number of reasons to prefer the JPQL style syntax.
On the other hand the concise syntax is obviously shorter and perhaps more readable.
If the query is getting complex then use the JPQL style syntax which will be explained first each time it occurs.
EntityBaseBean get(String sessionId, String query, long id)
The first parameter is the sessionId, the second is a query string in either the JPQL style or the concise syntax and the last parameter is the id of the object. You may know the id of an object because you have created it or it was returned from a search call. For example:
get(sessionId, "Dataset", 75L)will return the Dataset with an id of 75 or throw an exception if it is not found. The dataset is returned on its own; the set of related datafiles will be empty and the type will be null. Often however you will want to return more than just one object but will want to include related objects as well.
To get a Dataset with all its Datafiles you can have the query:
Dataset ds INCLUDE ds.datafilesThis uses the variable "ds" to identify the selected dataset. The field "datafiles" will be followed to include all those "Datafiles" related to the selected dataset. Those datafiles which the user is not allowed to read are silently ignored.
To get a Dataset with all its Datafiles, DatasetParameters and DatafileParameters you can have the query:
Dataset ds INCLUDE ds.datafiles df, df.parameters, ds.parameterswhich has introduced another variable df to represent the set of (readable) datafiles and which are related to the initial dataset ds. The variable could be preceded by the keyword "AS" if you feel this makes it easier to read as in:
Dataset ds INCLUDE ds.datafiles AS df, df.parameters, ds.parametersTo save typing you could write the equivalent:
Dataset ds INCLUDE ds.datafiles.parameters, ds.parametersThis slightly shortened form may make it less obvious that the datafiles are also included in the set of returned objects. In fact they must be present as the returned objects always form a DAG again and the DatafileParameters need the Datafiles to connect in to the structure.
It is permissible to visit an entity type more than once in an INCLUDE - for example following a provenance chain or including the datasets with the same type as a particular dataset which would be:
Dataset ds INCLUDE ds.type t, t.datasetsIf you want the Dataset along with its related Datafiles, DatasetParameters and DatafileParameters then in the concise syntax instead of giving the field to use for navigation from one object to another you simply list the set of types. as:
Dataset INCLUDE Datafile,DatasetParameter,DatafileParameterThe related types must be all be related to the original type or to some other type in the list. This means that you could not have "Dataset INCLUDE DatafileParameter" as there must be only one route from the original type to each of the included types - i.e. you can only construct one DAG from the starting object. There is evidently no way to express the query "dataset including the datasets with the same type as a particular dataset" as dataset appears twice. So you would need to use the JPQL style syntax.
void update(String sessionId, EntityBaseBean bean)
To update an object simply update the fields you want to change and call update. For example:
Dataset ds = (Dataset) icat.get(sessionId, "Dataset INCLUDE 1", dsid);As suggested by the example above "many to one" relationships, such as the investigation relationship to the dataset, will be updated as will any simple field values. Consequently it is essential to get the existing values for any "many to one" relationships. This is most reliably achieved by the notation INCLUDE 1 as shown here. The effect of the "1" is to include all "many to one" related types. "One to many" relationships are ignored by the update mechanism. Simple attributes may also be modified as:
Dataset ds = (Dataset) icat.get(sessionId, "Dataset INCLUDE 1", dsid);Note that once again the INCLUDE 1 is present to avoid losing "many to one" related objects. The name of the dataset is changed to "Fred". This is permissible even though name is one of the "uniqeness constraint" fields. If your attempted change violates a uniqeness constraint an exception will be thrown.
void delete(String sessionId, EntityBaseBean bean)
The following code will get a dataset and delete it.
Dataset ds = (Dataset) icat.get(sessionId, "Dataset", dsid);All cascaded "one to many" related objects will also be deleted. In the extreme case, if you delete a facility, you lose everything associated with that facility. This privilege should not be given to many - see the authorization section later. When you get a local copy of the object to delete there is no need to use INCLUDE 1 to populate the "one to many" related objects as all cascades will be followed. In fact the only part of the object that is used by the delete call is the id. So the following code will have the same effect and avoids one ICAT call:
Dataset ds = new Dataset()The search call supports JPQL syntax as well as the concise syntax.
The syntax is a standard JPQL "SELECT" statement with with two extensions a LIMIT clause and an INCLUDE clause which may come in either order after the standard JPQL. The only restriction is that returned item must be a set of entities, the result of an aggregate function (such as COUNT or MAX) or a set of values of one field of an entity type. If you use nested selects please define new variable names for use within such a construct. Some keywords, such as FETCH, are not relevant and are ignored. The language is too large to explain here but the Oracle JPQL documentation is good.
There are also two extensions to JPQL a LIMIT clause and an INCLUDE clause which may come in either order after the standard JPQL. The LIMIT clase follows MySQL syntax and takes the form: LIMIT 10, 100 which will skip 10 results and return the next 100. A LIMIT clause will normally be used with an ORDER BY clause. The INCLUDE clause is just as as has been explained for the get call with JPQL syntax . A few examples are shown below:
SELECT ds FROM Dataset ds INCLUDE ds.datafiles.parameters, ds.parametersThis uses the variable "ds" defined in the FROM clause. It means that the "Dataset" field "datafiles" will be followed to include all those "Datafiles" and that for each "Datafile" the "parameters" field will be followed to get the "DatafileParameters". In addition the "DatasetParameters" will be included. A more interesting example is:
SELECT ds.name FROM Dataset ds WHERE ds.type.name = 'GS' ORDER BY ds.name LIMIT 0, 10"This will take the first 10 datasets (as ordered by name) which have a type with a name of 'GS'. The next example (which cannot be expressed with the concise syntax) is looking for the ids of datasets which have both a parameter A1 > 50 and a parameter A2 > 20
SELECT ds.id FROM Dataset ds, ds.parameters dp1, ds.parameters dp2 WHERENote in the example above how the parameters dp1 and dp2 were introduced. The form :user may be used to denote the currently authenticated user (derived from the sessionId). For example to see the investigations to which you are associated:
SELECT i FROM Investigation i, i.investigationUsers ius WHERE ius.user.name = :userTime literals (which are implementation dependent in JPQL) should be expressed as shown in the next example:
SELECT i FROM Investigation i WHERE i.createTime > {ts 2011-01-15 00:00:00}The timestamp format must be exactly as shown. Literal boolean values are TRUE and FALSE as in:
SELECT ds FROM Dataset ds WHERE ds.complete = TRUEand finally enums are expressed as shown below:
SELECT pt FROM ParameterType pt WHERE pt.valueType = org.icatproject.ParameterValueType.DATE_AND_TIMEwhich is selecting those ParameterTypes which have a valueType of ParameterValueType.DATE_AND_TIME. Note that the full class name org.icatproject.ParameterValueType must be specified.
List<Object> search(String sessionId, String query)
If the query is simply Dataset this will return all Datasets. If the query is Dataset.name this will return all Dataset names. To get related objects returned, then the same INCLUDE syntax that was described for the get call with concise syntax may be used with exactly the same restrictions and semantics:
Dataset INCLUDE Datafile,DatasetParameter,DatafileParameterYou can specify an order (which may precede or follow an INCLUDE clause):
Dataset.id ORDER BY idRestrictions can be placed on the data returned. For example:
Dataset.id [type.name IN ('GS', 'GQ')]which could also be written:
Dataset.id [type.name = 'GS' OR type.name = 'GQ']The restriction in the square brackets can be as complex as required - but must only refer to attributes of the object being restricted - in this case the Dataset. Expressions may use parentheses, AND, OR, <, <=, >, >=, =, <>, !=, NOT, IN, LIKE and BETWEEN. Currently the BETWEEN operator does not work on strings. This appears to be a JPA bug.
Functions: MAX, MIN, COUNT, AVG and SUM may also be used such as:
MAX (Dataset.id)Selection may involve more than one related object. To show the relationship a "<->" token is used. For example:
Dataset.id <-> DatasetParameter[type.name = 'TIMESTAMP']Note also here the use of the JPQL style path: type.name . This expressions means ids of Datasets which have a DatasetParameter which has a type with a name of TIMESTAMP. Multiple " <->" may appear but all the objects involved, including the first one, must be connectable in only one way . The next example show how restrictions may be applied to both the Dataset and the DatasetParameter:
Dataset.id [name like 'fred%'] <-> DatasetParameter[type.name = 'TIMESTAMP']It is also possible to restrict the number of results returned by specifying a pair of numbers at the beginning of the query string. This construct would normally be used with an ORDER BY clause. The first number is the offset from within the full list of available results from which to start returning data and the second is the maximum number of results to return. These numbers if specified must be positive. If the offset is greater than or equal to the number of internal results then no data will be returned. The default values are 0 and "infinity". The numbers must be separated by a comma though either may be omitted. The following are all valid. The last example is rather pointless and does the same as the first. A number without a comma is illegal.
Dataset.id ORDER BY idTo see the investigations to which you are associated you can make use of the special value :user as in:
Investigation <-> InvestigationUser [name = :user]Time literals (which are implementation dependent in JPQL) should be expressed as shown in the next example:
Investigation [createTime > {ts 2011-01-15 00:00:00}]The timestamp format must be exactly as shown. Literal boolean values are TRUE and FALSE as in:
Dataset [complete = TRUE]and finally enums are expressed as shown below:
ParameterType [valueType = DATE_AND_TIME]which is selecting those ParameterTypes which have a valueType of DATE_AND_TIME from ParameterTypeValue.
List<Object> searchText (String sessionId, String query, int maxCount, String entityName)
This treats each ICAT entry as a document. The contents of that document is formed by concatenating all the non-blank text fields (with a space bewteen them). These documents are then indexed by Lucene. Each create, update or delete call updates the set of available "documents". The indices are updated periodically - so new entries will not be immediately visible. The freshness of the data is determined by the ICAT configuration and may be adjusted. As with the other search call you will only see the data you are allowed to see by the authorization rules. The syntax of the search query is defined by the Lucene library (currently version 4.3) where the option to allow leading wildcards has been enabled. Please see the following examples.
List<Object> results = icat.searchText(sessionId, "king", 50, null);This obtains the 50 "best" documents with the work "king" in them.
List<Object> results = icat.searchText(sessionId, "king queen", 50, null);returns documents with "king" or "queen" in them.
List<Object> results = icat.searchText(sessionId, "king AND (queen OR harp", 50, null);returns documents with "king" and either "queen" or "harp".
Case is ignored and there is no need to put in words with different endings but the same stem as common suffices are removed both when storing the document indices and when looking them up. Wild cards may be used - the ? for a single character and * for zero or more. The last argument my be the simple name of an entity as shown below.
List<Object> results = icat.searchText(sessionId, "king", 50, "Investigation");This restricts the search to the "Investigation" set of entities.
The mechanism is rule based. Rules allow groupings of users to do things. There are four things that can be done: Create, Read, Update and Delete. It makes use of five entity types: Rule, User, Grouping, UserGroup and PublicStep. The name "Grouping" has been introduced as "Group" is a reserved word in JPQL. The authentication mechanism authenticates a person with a certain name and this name identifies the User in the ICAT User table. Groupings have names and the UserGroup performs the function of a "many to many" relationship between Users and Groupings. Rules are applied to Groupings. There are special "root users" with full access to all entities. The set of "root users" is a configuration parameter of the ICAT installation. Only a root user can set up the initial set of authorization rules though these rules can then allow others to manipulate rules.
By default access is denied to all objects, rules allow access. It is only necessary to be permitted by one rule where that rule is only applied to the object referenced directly in the API call. The Rule table has two exposed fields: crudFlags and what . The field crudFlags contains letters from the set "CRUD" to indicate which types of operation are being allowed (Create, Read, Update and/or Delete). The other field, what , is the rule itself. There is also a "many to one" relationship to Group which may be absent.
Consider:
Rule rule = new Rule();allows members of the userOffice group full access to all Investigations.
Rule rule = new Rule();allows any authenticated user (with a sessionId) to read Parameters. Consider a group of users: fredReaders. To allow fredReaders to read a datafile with a name of "fred" we could have:
Rule rule = new Rule();The what field may contain almost anything which may appear as a search query. The query, if evaluated, would return the set of objects which can be read, updated or deleted in the case of R, U and D crudFlags. For the C crudFlag which controls create operations, the call is allowed if after creation of the object it would be in the set defined by the what field. The search query in the what field must return a set of objects rather than a set of fields or an aggregate function. The query must not contain INCLUDE, LIMIT nor ORDER BY clauses. There is currently an important restriction to avoid a problem which has occured in testing: with the JPQL syntax only one dot may appear for terms in the WHERE clause and for the concise syntax no dots are allowed in the condition in square brackets. You will get an error message if you forget. In the example above the JPQL syntax is used. If preferred the concise syntax may also be used as shown below:
Rule rule = new Rule();More complex restrictions can be added using other related objects. For example to allow read access to Datasets belonging to an Investigation which includes an InvestigationUser which has a user with a name matching the currently authenticated user (from the sessionId) we can have:
Rule rule = new Rule();which in the concise syntax becomes
Rule rule = new Rule();Rules which allow everyone to read a table are cached in memory and are good for performance. For example:
Rule rule = new Rule();This table has two columns (origin and field). An entry in this table affects the way in which INCLUDE authorizationis carried out. Each entry permits all users to make a step from the origin entity by the specifed relationship field without any further checking. This information is held in memory for speed. For those INCLUDEs that are not mentioned in the PublicStep table a full read authorization check must be made before including an object to be returned - which can be expensive.
boolean isAccessAllowed(String sessionId, EntityBaseBean bean, AccessType accessType)
This call returns true if the access to the bean specified by the accessType is permitted. For example:
Dataset ds = new Dataset();This code sets up a Dataset and then prints whether or not it would be allowed to create it.
This call is expected to be made from GUIs so that they can avoid offering operations that will fail. As such, though READ acess may be queried it is unlikely to be useful as the GUI user will not have found out about the object to be checked. If READ, DELETE or UPDATE access is queried for an object that does not exist it will return false.
In the case of CREATE, the entity is created within a database transaction, the check is made and the transaction is rolled back. Note that if a create operation would result in a duplicate this will cause an exception to be thrown.
Logging to a table with the entity name Log (and/or a file in the logs directory) may be enabled in the icat.properties file of the ICAT server. Records of the type requested in the icat.properites file are added to this table for each eligible call. The information in the table may be regarded as sensitive so appropriate authorization rules should be created.
ICAT is able to send JMS messages for create, update and delete of sslected entities. This is controlled by the icat.properties file of the ICAT server which can specify a list of the entity types to consider, and for each type which action to generate a message for. The JMS message is always PubSub rather than point to point. This means that there can be multiple listeners for a message.
Any receiver must be set up to receive messages with a topic of "jms/ICAT/Topic". The messages all have properties of "entity" which is the type of the entity such as Dataset and "operation" which is one of the letters: C, U and D (for create, update and delete respectively). The body of the message is an object which holds the entity id as a Long. For the "xxxMany" calls multiple notifications will be generated. A receiver typically implemented as an MDB (Message Driven Bean) should filter the messages it processes by using the properties of the message.
It should be noted that by the time you can react to a deletion notification the entity id you are sent will refer to an object which no longer exists.
This mechanism does not leak information becuase all the user receives is an entity id. To read the entity with that id the user must have read access to that entity instance.
There is an example MDB available which should make this easier to understand.
String getApiVersion()
returns the version of the server
List<String> getEntityNames()
Returns an alphabetic list of all the entity names known to ICAT. This is of most value for tools.
EntityInfo getEntityInfo(String beanName)
returns full information about a table given its name. For example:
EntityInfo ei = icat.getEntityInfo("Investigation");Prints out some information about the Investigation table. For a list of all available fields in EntityInfo and the objects it references please consult the javadoc for EntityInfo .
To be authorized to use these administration calls you must be authenticated with a name listed in the rootUserNames in the icat.properties file.
List<String> getProperties(String sessionId)
lists the active contents of the icat.properties file. It does this by examining the properties after they have been read in so any superfluous definitions in the original properties file will not be seen. The current physical file is not re-examined
void lucenePopulate(String sessionId, String entityName)
instructs lucene to populate indices for the specified entityName. This is useful if the database has been modified directly rather than by using the ICAT API. This call is asynchronous and simply places the request in a set of entity types to be populated. When the request is processed all lucene entries of the specified entity type are first cleared then the corresponding icat entries are scanned to re-populate lucene. To find the prcoessing state use the luceneGetPopulating() call desribed below. Note that because of caching, ICAT should ideally be reloaded after any direct database modifications are made.
List<String> luceneGetPopulating(String sessionId)
returns a list of entity types to be processed for populating lucene following calls to lucenePopulate(). Normally the first item returned will be being processed currently. If nothing is returned then processing has completed.
void luceneCommit(String sessionId)
instructs lucene to update indices. Normally this is not needed as it is will be done periodically according to the value of lucene.commitSeconds in the icat.properties file.
void luceneClear(String sessionId)
clears all the lucene indices. It does not commit itself; you may simply wait for the periodic commit depending upon the value of lucene.commitSeconds in the icat.properties file.
List<String> luceneSearch(String sessionId, String query, int maxCount, String entityName)
searches lucene indices and returns a list of entity_name:entity_id values. "query" is a lucene query. Queries can contain AND and OR in upper case as well as parentheses. The default operator is OR. Wildcards of * and ? are also supported. Other features are described for the searchText call. The maxCount argument specifies the maximum number of values to return and the entityName, if not null, restricts results to entities with that name.