ICAT Server Release Notes

4.8.0

Support update operations with control on individual attributes and bug fixes

  • Protect the SOAP interface from bad plugin mnemonics on login calls
  • Support update operations with control on individual attributes
  • The testicat program has been recoded to use only REST calls.
  • A new clone operation has been added to the REST interface
  • A new property "key" may be specified in the icat.properties file. If there is an IDS server in use and it has a key for digest protection of Datafile.location then this key value must be identical.

4.7.0

Enhanced Rest interface and changes to the authorization code

  • A restful select can mention multiple quantities to be returned
  • A major re-write of the authorization code should make it more efficient in most cases as all the work is carried out by the database unless the INCLUDE mechanism is used. This has required that the Rule table has a slightly different structure from 4.6.0 so the correct upgrade procedure from earlier versions must be followed.
  • COUNT queries will not fail because they are counting more than maxEntities values
  • A couple of bugs in eclipse link result in incorrect information being returned. One of these bugs was trapped previously but with the generality of the queries now supported it has become impractical to trap.
    • "SELECT f from Facility WHERE f.id = 1 LIMIT 1,10" which should always return an empty list now ignores the LIMIT clause.
    • "SELECT ds.complete FROM Dataset ds" only returns the false values but that "SELECT ds.id, ds.complete FROM Dataset ds" works properly.
  • A restful delete call has been added
  • The delete(Many) call, whether restful or not, checks the authz rules for all objects that will be deleted by the cascade mechanism.
  • A restful write call to do both create(Many) and update(Many) has been written. Notifications are sent for all objects created and/or updated. Authz is also applied to all new or updated entries.
  • Notifications are no longer generated for deletes as the information could never be used. This means that only C and U are valid notification letters in the icat.properties file.
  • DataCollection now has a doi attribute
  • Call logging has been changed to generate only jms messages. It is very similar to the call logging in the IDS. It is controlled by a log.list in the icat.properties which may contain one or more of READ, WRITE, SESSION and INFO.
  • Updates whether from the SOAP call or the restful write call are both checked for UPDATE permission before the call and if a relationship field is changed which is used to identify that object then DELETE permission is required before the change and CREATE permission afterwards.
  • Added attributes email and orcidId to the User table
  • testicat and icatadmin have certificate checking disabled. This used to be the normal behaviour but newer version of python, by default, expect a certificate accessible by openssl.
  • Produce an error message rather than a null pointer exception if an invalid authenticator mnemonic is used in a login call.
  • Added "isLogedIn" REST call
  • Fixed bug when trying to read or write ParameterValueType values (an enum) through REST call. These are now represented as json string values.
  • Fixed bug trying to read back an individual date value through a REST call.
  • Remove db.vendor and add db.target and db.logging to icat-setup.properties.
  • The jdbc driver jar if present in the domain's lib directory must be moved to lib/ext.

4.6.1

Bug fix release

  • Previously the update call failed if a Datafile was passed in with all datafileCreateTime, datafileModTime and modTime all set to null. If the Datafile was fetched, modified in memory and then update was called then modTime would be set and the problem would not reveal itself.
  • It should also be noted that from release 4.6.0 Java 8 is required to run the code.

4.6.0

Small change to icat.properties

  • notification.list and log.list may now be omitted but must not be present and empty in the icat.properties file
  • Provide a "getProperties()" restful call to find out what can be revealed about a server without a sessionId. This includes maxEntities and information about the available authenticators.
  • Support the ESCAPE keyword in JPQL queries.
  • New calls added to the restful interface for lucene queries.
  • Documentation for the restful interface is now generated by MireDot.
  • Changes to properties files to work with container independent setup script.
  • Change from log4j to slf4j with logback
  • Corrected the "Study" entity to avoid NullPointer exceptions
  • Fixed a resource leak in lucene populate
  • Lucene populate is now much faster. This is mainly because it uses multiple threads
  • It is recommended to avoid this version of ICAT if you are using multiple icat servers as testing of such configurations has not been carried out.

4.5.1

Fix bug where ICAT won't start up with an empty lucene index

4.5.0

Remove lucene calls and apply bug fixes.

  • Permit lucene not to be used by omitting references in icat.setup.
  • All lucene calls have been commented out - they will be reinstated in 4.6 but they will be different.
  • Allow queries of the form "SELECT a.b.c FROM ..." This was not documented but used to work in ICAT 4.3 and is standard JPQL.
  • Provide better protection against queries that are getting too large a quantity of data back from the database. Previously the "maxEntities" check was only made at the time when the XML response to the user was generated by which time it could have been too late. A side effect of this is that in some cases when an aggregate function is part of a query which would of course only return one record this may now fail if the data being aggregated are very large. Even if such queries are handled by the database, SUM and AVG are potentially very slow as they cannot benefit from any indexing.
  • Fixed the ordering of entities appearing in the output file when using exportData in the restful API so that the file can be re-imported.
  • Instead of doubled up '"' characters use '\"' and also introduce \b \t \n \f \r \\ and the not strictly necessary \' for the import export format.
  • Avoid the use of a sequence table for allocating IDs. It should be noted that a side effect of this is that for MySQL there is no longer a global ID sequence but the ID sequences are specific to a table.
  • LIMIT clause in the query now works properly when searching by id. This requires a manual schema upgrade if you already have an ICAT installed.
  • Ensure that createTime and endTime are represented by ISO 8601 style json strings like other TIMESTAMP fields using the RESTful search call.
  • Provide protection against an EclipseLink bug with Oracle which resulted in BigDecimal objects being returned instead of Long or Double values when MIN or MAX of a value of that type was requested.
  • The documentation for the rootUserNames parameter has been updated to match a change introduced in release 4.4 whereby users listed in rootUserNames have full access to all tables.
  • json returned by restful calls now encodes numbers and booleans correctly.
  • The restful calls now produce CORS headers.

4.4.0

This introduces a number of "RESTful" calls and has some other small improvements and bug fixes.

  • A bug in the authorization computation has been fixed which was previously prohibiting operations that should have been allowed in some cases where multiple authorization rules were relevant.
  • A partial prototype of a "RESTful" interface has been added.
  • There is a new parameter maxEntities which must be specified in icat.properties to restrict the total number of entities returned in calls to get, search and searchText.
  • There is a new parameter maxIdsInQuery which must be specified in icat.properties to restrict the size of ICAT generated queries. This must not exceed 1000 if Oracle is being used.
  • There are new parameters importCacheSize and exportCacheSize which must be specified in icat.properties to restrict the size of caches used during import and export. The cache is local to a call to import or export to ensure that authorization rules are enforced. As a by-product modifications to ICAT are less likely to result in stale information being returned from the cache.
  • ID values are obtained from the database SEQUENCE table in groups of 500 rather than 50 to improve performance. This is to circumvent a problem seen occasionally in Oracle.
  • The DISTINCT keyword now works as in regular JPQL.
  • Queries are now truncated to 4000 characters before storing them in the table or file log.
  • The installation script has been made more robust and will also ensure that JDBC connectors are set up in a resilient manner for Oracle RAC servers.
  • It is now treated as an error to have redundant paths in an INCLUDE clause.
  • The application is now built and deployed as a .war files rather than a .ear file which makes it easier to deploy on Wildfly.
  • The InvestigationGroup has been introduced. It is expected that this will be generally used in Rule based authorization in preference to InvestigationUser except for those cases where the role of the individual is significant.
  • The role field in the InvestigationUser is now part of the key and can now not be null. This allows a User to have multiple roles relative to an Investigation
  • The Rule.what column has been increased in length from the default of 255 to 1024.

4.3.3

This is a bug fix release with support for a group of ICATs working together. It is not a cluster in the Glassfish sense.

  • The isAccessAllowed call, when used to check CREATE access, is now protected from invalid input.
  • All standard JPQL functions are now accepted in search() calls.
  • Queries returning results of aggregate functions should always return an array of length 1. However, due to the behaviour of JAXB a list containing just a null is not transmitted properly and instead an empty list is returned. This is a problem with the functions MIN, MAX, SUM and AVG all of which return null when they have no data to process. The COUNT function was also returning an empty list when it had no data but now returns a list containing 0L.
  • More than one ICAT may be used together to share the load and provide more resilience.
  • The setup script does not require calling with the configure option.
  • The icat-setup.properties file now uses the keyword dbProperties rather than icatProperties.
  • Fixed bug with handling of unary minus.

4.3.2

This is a bug fix release.

  • A get call using the old (non-JPQL) syntax will now accept the INCLUDE keyword in lower or mixed case.

4.3.1

This is a bug fix release.

  • The methods logout and refresh when called with an expired or otherwise invalid sessionId now throw an exception of type SESSION rather than INTERNAL.
  • A small improvement to testicat has been made to show who you are logged in as.
  • The relationship from DataCollection to DataCollectionParameters is now called parameters and can be accessed by getParameters and setParameters.

4.3.0

This includes schema changes and a number of speculative features.

  • The "ICATCompat" service for backwards compatibility with ICAT 3.x has now been removed.
  • A method "void refresh(String sid)" has been added to refresh an ICAT session.
  • The notification mechanism has been completely changed.
  • It is not legal in queries with an INCLUDE to use a "1" in conjunction with and entity name. This will now reliably throw an exception.
  • A new call "boolean isAccessAllowed" has been created to test whether a specific access type is allowed.
  • A call has been provided to return all the available entity names. This is of more use to tools than regular users.
  • Column names in the Database table for UserGroup have been changed as they were inconsistent.
  • The Rule entities are now store in a database table called RULE_. for compatibility with Microsoft SQLServer.
  • A file indices.sql is being distributed with the .ear file with indices which may improve performance. You should assess whether or not each index will be useful to your installation for while an index can give a huge improvement to a query there is also a cost in maintaining an index.
  • JPA mappings for floating point numbers now use precision 38 and scale 19. This is still not good on Oracle as mentioned in the installation instructions.
  • testicat.py (which replaces test.py) expects https://hostname:port rather than a pair of parameters
  • Call logging has been added either to file or to a log table or both.
  • Lucene free text search has been included via the new searchText call.
  • Changed uniqueness constraints to:
    Datafile
    dataset, name (-location)
    Dataset
    investigation, name (-sample -type)
    Application
    facility, name, version (+facility)
    Sample
    investigation, name (-type)
    Investigation
    facility, name, visitid (-facilityCycle -instrument)
    SampleType
    facility, name, molecularFormula (+molecularFormula)
  • Made all constraint fields not nullable. This is checked by ICAT to ensure that future changes do not break this rule.
  • Add a string attribute: "arguments" to the job to store the program arguments that were used.
  • Relationship between FacilityCycle and Investigation removed.
  • Table between Investigation and Instrument added to represent many-to-many relationships between them.
  • Added DataCollection and changed Job to make use if it.
  • Group has been changed to Grouping as Group is a JPQL keyword.
  • Add attribute 'url' to Instrument.
  • Added new alternate query syntax that may be used in queries and authz rules. It is distinguished by starting with the keyword "SELECT".
  • It is no longer considered an error to have no authz rules matching a search call - you just get no data back.
  • Packaging now conforms to http://code.google.com/p/icatproject/wiki/ComponentGuidelines
  • A command icatadmin has been introduced to perform operations for which the rule based authorization is not appropriate.
  • Expired sessions are cleaned out of the database every hour.
  • The generated client jar contains two packages: org.icatproject and org.icatproject_4_3_0. Normal users should use the org.icatproject one. The other exists primarily for TopCAT so that multiple ICAT clients can co-exist.
  • The installation mechanism now has a single setup Python script which works both on Linux and Windows.

4.2.5

This is a bug fix release. If you already have 4.2.0, 4.2.1, 4.2.2, 4.2.3 or 4.2.4 installed, then simply remove the old ear file and deploy the new one. If you have changed log4j.properties from the one supplied then add an entry to your icat.properties: log4j.properties = log4j.properties which tells the ICAT to use the existing log4j.properties file in the config directory rather than the built in one. If you have not changed it then it is preferable to delete the log4j.properties from the config directory to avoid any possible confusion. The log4j.properties file is no longer reread, if you have configured your own copy and wish to change it then you must at least reload ICAT for the properties file to be read again.

  • A significant leak has been fixed which was causing an unlimited number of threads to be created. This has been addressed by changing the way in which the log4j.properties file is read. Only if you have a line in the icat.properties identifying a specific log4j.properties file will that file be used, otherwise a default file packaged within the .ear file will be used.

4.2.4

This is a bug fix release. If you already have 4.2.0, 4.2.1, 4.2.2 or 4.2.3 installed, then simply remove the old ear file and deploy the new one.

  • Version number updated to 4.2.4 and test updated to ensure that the version corresponds to the maven project version.
  • LICENSE.txt file added to top directory of the source tree.
  • Creating an Investigation which refers to a new InvestigationUser now works properly.
  • Using INCLUDE with User and Group now works properly. Previously it could throw a null pointer exception.
  • Database requirements have been made more precise in the deployment instructions.

4.2.3

This is a bug fix release. If you already have 4.2.0, 4.2.1 or 4.2.2 installed, then simply remove the old ear file and deploy the new one.

  • Two different error conditions were not being handled correctly in calls to createMany when a user attempted to violate a uniqueness constraint. In the case of a conflict with a pre-existing entry in ICAT the offset returned was -1. In the case that there were duplicates in the list passed in the call to createMany - rather than conflicts with existing database entries - an IcatException of type INTERNAL was being thrown, also with an offset of -1. These bugs are now fixed.

4.2.2

This is a bug fix release. If you already have 4.2.0 or 4.2.1 installed, then simply remove the old ear file and deploy the new one.

  • A bug causing read access to be denied in certain circumstances, when it should have been permitted, has been fixed.
  • When attempting to format an error message about an invalid list of INCLUDES a null pointer exception is no longer thrown.

4.2.1

This is a bug fix release. If you already have 4.2.0 installed, then simply remove the old ear file and deploy the new one.

  • Authorization rules without explicit relationships to groups (of the form "Investigation <-> InvestigationUser <-> User [name = :user]") now work properly.
  • It is now possible to use boolean values in queries. Attributes can be compared against literal values of TRUE and FALSE (not case sensitive) as in "ParameterType [enforced = TRUE]"
  • Null pointer exceptions were being throw with some authorization rules when using an Oracle database. This has now been fixed.

4.2.0

This introduces a clean mechanism for handling authentication plugins and provides significant speed enhancements.

  • ICAT now contains no authentication plugins but it does provide an Authenticator interface for plugin writers. If you plan to use the new authn_db database authentication plugin you will need to manually move data from the old table to the new passwd table.
  • The login call now accepts two parameters, the mnemonic for the authentication plugin and a map of credentials. This change necessitated the increase in the version number to 4.2.
  • There was a serious performance bug in release 4.1 and to a lesser extent in 4.0. In both case this affected read operations - search and get. This bug in the marshalling of the XML has now been addressed.
  • Previously JMS messages had the property "query" set to the primary key and the actual query value was not available. They are now assigned correctly to the properties "query" and "entityId".
  • You will need an up-to-date version of JAX-WS such as the one packaged with Glassfish 3.1.2
  • The main database schema is almost unchanged. The only difference is that some text fields have been made larger.
  • A bug has been fixed in the handling of enum types in expressions.

4.1.1

This is a bug fix release.

  • Aggregate functions (MIN, MAX, AVG, COUNT etc) no longer generate an internal exception.
  • Minor improvements to the installation documentation.

4.1.0

This is the first release of ICAT4 which is meant to be approaching production quality.

  • More changes to the API - too numerous to list here providing
    • support for pluggable authn,
    • support for rule based authz,
    • support for notifications and
    • a powerful search mechanism
  • Database independent
  • More regular schema with simple provenance mechanism

4.0.0

Technology preview release - not for production use

  • Many changes to the API

3.3.4

  • Changed the JPA provider from TopLink to EclipseLink
  • Removed Dataportal from the installation

3.3.3

Public release