DGW Download API requires Java 11+, icat.server 4.6+ and ids.server 1.5.0.+
TopCAT 2.4.3 reflects schema changes added in icat.server 4.10. If your configuration requires any of these schema changes then you should install 2.4.3 or later.
If you are using an older version of Topcat you may need to apply some migrations to your database. These can be found in the “migrations/” directory within the Topcat release.
The CACHE table now requires an extra column, CREATION_TIME. The simplest upgrade is to drop the existing table; this is done by the provided migration scripts.
Topcat is configured in two main places:
The backend configuration consists of two files:
The frontend configuration consists of three files:
Note: ’#’s are comments.
# Derby Database !db.target = derby !db.driver = org.apache.derby.jdbc.ClientDataSource !db.url = jdbc:derby:topcat;create\\=true !db.username = APP !db.password = APP # MySQL Database db.target = mysql db.url = jdbc:mysql://localhost:3306/topcat db.driver = com.mysql.jdbc.jdbc2.optional.MysqlDataSource db.username = icat db.password = icat secure = true container = Glassfish home = /home/fisher/pf/glassfish4 port = 4848 # Email setup see http://docs.oracle.com/cd/E26576_01/doc.312/e24938/create-javamail-resource.htm#GSRFM00035 mail.host=smtp.example.com mail.user=user@example.com mail.from=no-reply@example.com #see https://javamail.java.net/nonav/docs/api/ for list of properties mail.property=mail.smtp.port="25":mail.smtp.from="no-reply@example.com"
The mail.xxx properties can be omitted if email support is not required in Topcat (but see IMPORTANT below). In this case, mail.enable in topcat.properties MUST NOT be set to true. This is NOT checked during setup.
IMPORTANT The mail.xxx properties must NOT be omitted if Topcat is configured to download from two-tier IDS instances. This is true even if mail is not required; in that case, dummy values such as the above can be used.
The ids.timeout property can now have an optional s or m suffix to specify seconds or minutes. The default remains milliseconds (so existing configurations do not need to be changed).
New properties have been added to allow finer control over caching of Investigation sizes. Investigations can change size over time, so setting a lifetime on cached values may be useful. It may be useful not to cache zero-sized Investigations at all. Note that size requests are also cached in the browser, so check settings in topcat.json too. However, the browser cache is less persistent, and can be cleared by reloading the page, so browser cache consistency is less of an issue.
These properties are optional, and the defaults preserve the original behaviour, so they need not be added when upgrading.
# Lifetime in seconds for cached Investigation sizes (default is 0, which means "immortal") investigationSizeCacheLifetimeSeconds=600 # Whether to cache zero-sized Investigations (default is false) neverCacheZeroSizedInvestigations=true
If you are upgrading from 2.3.6 or earlier to 2.4.0, the following properties should be added to your topcat.properties: facility.list, facility.[facilityName].icatUrl, facility.[facilityName].idsUrl (for each facilityName in facility.list).
Where a facility offers multiple download transfer types (as specified in topcat.json), a specific URL (prefix) for a particular facility and download type can be specified by defining facility.[facilityName].downloadType.[transport] - see below for examples. If the property is not defined for a particular facility and download type, the facility’s idsUrl will be used instead.
Note: ’#’s are comments.
# List of Facility names
# These names will be used by Topcat over the REST API;
# each facility name in topcat.json must contain a match in this list,
# and each name must be mapped to ICAT / IDS urls.
# Edit these values to match your installation.
facility.list = LILS YFH
facility.LILS.icatUrl = http://localhost:8080
facility.LILS.idsUrl = http://localhost:8080
facility.YFH.icatUrl = https://your.facility.icatUrl.here
facility.YFH.idsUrl = https://your.facility.idsUrl.here
# Download transport URLs
# topcat.json can specify one or more download transport types for each facility, egs "https", "globus";
# each may have a distinct download URL (prefix). To specify the download URL for a specific facility
# and transport type, set an appropriate property as below.
# Note that the transport type will be set in requests from the Topcat Javascript application,
# but the URL to specify here should be the IDS url that will be used in the Java clients;
# so it may be that the Javascript transport type https should be mapped here to an
# internal http URL.
# If the property is not defined, Topcat will use the facility's idsUrl by default.
facility.LILS.downloadType.http = http://localhost:8080
facility.YFH.downloadType.https = https://your.facility.downloadIdsUrl.here
facility.YFH.downloadType.globus = https://your.facility.globusUrl.here
# enable send email
mail.enable=true
# The email subject. Tokens available are:
# ${userName} - user username
# ${email} - user email
# ${facilityName} - the facility key (defined in frontend config)
# ${preparedId} - the prepared Id of the download request
# ${fileName} - the download name
# ${size} - the download size
# ${downloadUrl} - the download url
mail.subject=TopCAT Download ${fileName} Ready
# The email body message for https downloads. All subject tokens as above are available.
mail.body.https=Hi ${userName},\n\nYour ${size} download ${fileName} has been prepared and is ready for download at ${downloadUrl}.\n\nThank you for using TopCAT.
# The email body message for https downloads. All subject tokens as above are available.
mail.body.globus=Hi ${userName}, \n\nYour ${size} Globus download ${fileName} is ready. Please see https:/example.com/#/globus-faq for more information on how to download using Globus.\n\nThank you for using TopCAT
# The email body message for smartclient downloads. All subject tokens as above are available.
mail.body.smartclient=Hi ${userName}, \n\nYour ${size} SmartClient download ${fileName} is ready. Please check your smartclient home directory for your files.\n\nThank you for using TopCAT
# The email body message for SCARF downloads. All subject tokens as above are available.
mail.body.scarf=Hi ${userName}, \n\nYour ${size} SCARF download ${fileName} is ready. Please see https:/example.com/#/scarf-faq for more information on how to download using SCARF.\n\nThank you for using TopCAT
# The maximum number of datafiles for a getStatus call to the IDS for two level storage
ids.getStatus.max=100
# The delay in seconds before polling starts. This delay is to gives the ids a chance to do its thing before we query it
poll.delay=600
# The wait time in seconds between each poll to the IDS.
poll.interval.wait=600
# A list of usernames that can use the admin REST API and Topcat admin user interface
adminUserNames=simple/root, uows/elz087, asd345, ldap/fgh123
# The maximum number objects that can be cached before pruning will take place
maxCacheSize=100000
# The following properties allow finer control over caching of Investigation sizes.
# Investigations can change size over time, so setting a lifetime on cached values may be useful.
# It may be useful not to cache zero-sized Investigations at all.
# IMPORTANT: size requests are also cached in the browser, so check settings in topcat.json too.
# Lifetime in seconds for cached Investigation sizes (default is 0, which means "immortal")
# investigationSizeCacheLifetimeSeconds=600
# Whether to cache zero-sized Investigations (default is false)
# neverCacheZeroSizedInvestigations=false
# Timeout for IDS connections.
# Optional (default is no timeout). Can be in seconds (suffix 's'), minutes ('m') or milliseconds (no suffix)
# ids.timeout=180000
# ids.timeout=180s
# ids.timeout=3m
A high level overview of the topcat.json file is as follows:
{
"site": {...},
"facilities": {...}
}
from the above example there are two attributes defined:
If we expand the attributes in high level overview we can get a medium overview as follows:
{
"site": {
"topcatUrl": "https://example.com",
"home" : "my-data",
"enableEuCookieLaw" : true,
"investigationSizeCacheLifetimeSeconds" : 0,
"dontCacheZeroSizedInvestigations": false,
"paging" : {...},
"breadcrumb": {...},
"serviceStatus": {...},
"maintenanceMode": {...},
"search": {...},
"browse":{...},
"cart":{...},
"myDownloads":{...},
"pages" : [...]
},
"facilities": [
{
"name": "DLS",
"title": "Diamond Light Source",
"icatUrl": "https://example.com",
"idsUrl": "https://example.com",
"idsUploadDatafileFormat": "example_datafile_format_name",
"idsUploadDatasetType": "example_dataset_type_name",
"idsUploadMaxTotalFileSize": 10000000000,
"hierarchy": [...],
"authenticationTypes": [...],
"downloadTransportTypes": [...],
"admin":{...},
"myData": {...},
"browse":{...}
}
]
}
from the above following attributes defined:
Specifies the paging type configuration i.e. for either paged or infinite scroll.
An example configuration for infinite scrolling:
{
"site": {
"paging" : {
"pagingType": "scroll",
"scrollPageSize": 50,
"scrollRowFromEnd": 10
}
}
}
An example configuration for standard page based pagination:
{
"site": {
"paging" : {
"pagingType": "page",
"paginationNumberOfRows": 10,
"paginationPageSizes": [
10,
25,
50,
100,
1000
]
}
}
}
Specifies global breadcrumb options.
An example breadcrumb configuration:
{
"site": {
"breadcrumb": {
"maxTitleLength": 30
}
}
}
Specifies the structure of the search results and search fields for searching across facilities.
An example (partial) configuration:
{
"site": {
"search": {
"disabled": false,
"enableTextBox": true,
"enableDateRange": true,
"enableParameters": true,
"enableSamples": true,
"enableInvestigation": true,
"enableDataset": true,
"enableDatafile": true,
"gridOptions": {
"investigation": {...},
"dataset": {...},
"datafile": {...}
}
}
}
}
Specifies the structure of the root of “browse” section of the site. This only applies to a Topcat with multiple facilities.
An example (partial) configuration:
{
"site": {
"browse":{
"gridOptions": {...},
"metaTabs": [...]
}
}
}
Specifies the structure of the cart.
An example (partial) configuration:
{
"site": {
"cart":{
"enableLimits": true,
"maxDatafileCount": 1000000,
"maxTotalSize": 1000000000000,
"gridOptions": {...}
}
}
}
Specifies the structure of ‘downloads’ dialog box.
An example (partial) configuration:
{
"site": {
"myDownloads":{
"gridOptions": {...}
}
}
}
Allows you to define information pages e.g. “About Us” or “Contact” etc…
An example configuration:
{
"site": {
"pages" : [
{
"name" : "about",
"addToNavBar": {
"label" : "About",
"align" : "left"
}
}
]
}
}
The entity heierachy of the browse section.
An example configuration:
{
"facilities": [
{
"hierarchy": [
"facility",
"proposal",
"investigation",
"dataset",
"datafile"
]
}
]
}
The above configuration will make the user browse via “facility” > “proposal” > “investigation” > “dataset” > “datafile”.
Specifies the authentication plugins to be used e.g. LDAP
An example configuration:
{
"facilities": [
{
"authenticationTypes": [
{
"title": "Simple",
"plugin": "simple"
},
{
"title": "DB",
"plugin": "db"
}
]
}
]
}
Optional. Specifies extra buttons to be added to the login page, as links to (possibly external) resources. For example, to add a button to register an account with the ISIS User Office:
{
"facilities": [
{
"extraLoginButtons": [
{
"title": "Register",
"url": "https://users.facilities.rl.ac.uk/auth/CreateAccount.aspx"
}
],
...
}
]
}
Optional. Specifies the authentication method and credentials to be used (if requested) by DOI redirections. Defaults to the “anon” authenticator (in which case it must be available and supported by the ICAT instance). Note that the configuration can be inspected by users in the browser console, so setting this is NOT RECOMMENDED for production systems. (Its main use is for development, where the LILS facility does not support anonymous authentication.)
{
"facilities": [
{
"doiAutoLoginAuth": {
"plugin": "simple",
"credentials": {
"username": "root",
"password": "vegetable"
}
},
...
}
]
}
Specifies the download delivery methods e.g. ‘https’ (via a browser) or ‘globus’ (a type of glorified ftp to deal with large files).
An example configuration:
{
"facilities": [
{
"downloadTransportTypes": [
{
"type" : "https",
"idsUrl": "https://fdsgos11.fds.rl.ac.uk"
},
{
"type" : "globus",
"idsUrl": "https://fdsgos11.fds.rl.ac.uk",
"displayName": "Globus access",
"description": "Use Globus to download your data"
},
{
"type" : "scarf",
"idsUrl": "https://dls-ids01.scarf.rl.ac.uk",
"displayName": "Download to Scarf"
}
]
]
}
}
Defines the structure of admin interface. This interface is only available to admin users (specified in topcat.properties).
An example (partial) configuration:
{
"facilities": [
{
"admin":{
"gridOptions": {...}
}
}
]
}
Specifies the structure of the “My Data” section.
An example (partial) configuration:
{
"facilities": [
{
"myData": {
"entityType" : "investigation",
"gridOptions": {...}
}
}
]
}
Specifies the structure of the hierachical browse section.
An example (partial) configuration:
{
"facilities": [
{
"browse": {
"investigation": {
"gridOptions": {...},
"metaTabs": [...]
},
"proposal": {...},
"dataset": {...},
"datafile": {...}
}
}
]
}
General principles configuring the gridOptions.
An example configuration:
"gridOptions": {
"enableSelection": true,
"enableUpload": true,
"columnDefs": [
{
"field": "visitId",
"link": true,
"breadcrumb": true
},
{
"field": "size"
},
{
"field": "datasetCount"
},
{
"field": "datafileCount"
},
{
"field": "investigationInstrument.fullName"
},
{
"field": "startDate",
"excludeFuture": true,
"sort": {
"direction": "desc",
"priority": 1
}
},
{
"field": "endDate"
}
]
}
General principles configuring the metaTabs.
An example configuration:
"metaTabs": [
{
"title": "METATABS.DATASET.TABTITLE",
"items": [
{
"field": "name"
},
{
"field": "description"
},
{
"label": "METATABS.DATASET.START_DATE",
"field": "startDate",
"template": "{{content.value | date:'yyyy-MM-dd'}}"
},
{
"label": "METATABS.DATASET.END_DATE",
"field": "endDate",
"template": "{{content.value | date:'yyyy-MM-dd'}}"
}
]
},
{
"title": "METATABS.DATASET_TYPE.TABTITLE",
"items": [
{
"field": "datasetType.name"
},
{
"field": "datasetType.description"
}
]
}
]
Topcat defines the following jpql variables - which if used from any icat entity type will automatically work out any joins required to create a jpql statement for fetching the data from icat:
To get a better understanding of what these jpql variables refer to please view the icat schema documentation.
After unzipping a Topcat release you will be presented with a directory called “content.example”, which you need to make a copy of called “content” i.e.
cp -R content.example content
This newly created directory allows you to either replace or add files within the war file (which is actually the same format as a zip file), to view the existing contents of this war (i.e. zip) file by running the following:
unzip -l topcat-2.4.0.war
after running:
./setup install
the contents of the “content” directory will be injected into the war file just before deploying.
You can use this feature to create a “pages” directory, which is where the html content specified within topcat.json’s “pages” section is defined, you can also this feature to replace or add any other content e.g. the favicon.
Allows you to replace text/phrases within Topcat’s interface with your own prefered alternatives e.g. rather than term “Investigation” you might prefer “Visit”.
Within lang.json is a json object that represents a hierarchy of translation strings, with a bit of experimentation you can modify virtually any text in Topcat’s front end e.g. if you wanted to change the “My Data” tab text simply edit “MAIN_NAVIGATION” -> “MAIN_TAB” -> “MY_DATA”.
(Added in 2.4.2)
In the browser console, typing:
tc.setMonitoring(true)
will start monitoring behaviour of the (browser-side) cache and the low-priority queue. At regular intervals, Topcat will report to the console recent and all-time counts of cache hits/misses, the maximum length of the low-priority queue (which is used to process getSize requests), and the maximum wait time (in milliseconds) of entries in the queue. Here, “recent” means “in the last 10 minutes”.
This information may be useful when experimenting with cache control configuration for investigation sizes.
To turn off monitoring again, type tc.setMonitoring(false) into the console.
The topcat_admin script was added to Topcat in October 2017, but was not included in the distribution until version 2.4.4. It uses the new REST API for disabling/enabling download types (which cannot be done from the browser Admin interface at present.) The setup script will copy this script to the user’s bin directory if it exists. (It does not insist that it exists, for backwards compatibility with installation processes that work with previous versions.)
topcat_admin requires the Python requests module to be installed; setup neither checks nor ensures that this is the case. The script can be run from any suitable system; it need not be run on the Topcat server.