Huygens Remote Manager documentation

This is the user and system administrator documentation for the Huygens Remote Manager (HRM).

Main sections

Installation instructions

These are the detailed installation and administration instructions for the Huygens Remote Manager.

Conventions

This page lists some conventions and notations used in this documentation.

Using sudo

Throughout this document, we’re assuming that commands which require root permissions can be issued by using sudo. If your system is not configured for this, you can either login directly as root (considered as “bad practice”) or use the su - command to switch to the root user.

We highly recommend the usage of sudo though!

Operating systems

The recommended linux distributions are:

Icon	Distribution name
	Debian
	Ubuntu
	Rocky Linux
	Alma Linux

We will use tabs to group the code and instructions for respective operating system families

Tab name	Corresponding operating system family
Debian	Debian, Ubuntu
RHEL	RedHat, Rocky Linux, Alma Linux

Warning

Please notice that with release 3.1, we dropped support for Mac OS X. HRM 3.0 is still known to work on Mac OS X from 10.5 (Leopard) onward, but no effort will be made to make future versions of the HRM compatible with Mac OS X. Also notice that the HRM was never tested on Mavericks.

Variables

Variable	Description	(Example) value
$WWW_ROOT	Web server document root	/var/www/html
$HRM_HOME	HRM root (home) folder	$WWW_ROOT/hrm
$HRM_CONFIG	HRM configuration folder	$HRM_HOME/config
$HRM_SAMPLES	HRM configuration samples folder	$HRM_HOME/config/samples
$HRM_RESRC	HRM resources folder	$HRM_HOME/resources
$HRM_SETUP	HRM setup folder	$HRM_HOME/setup
$HRM_BIN	HRM executables folder	$HRM_HOME/bin
$HRM_USER	HRM customization folder	$HRM_HOME/user
$HRM_LOG	HRM logging folder	/var/log/hrm
$HRM_DATA	HRM data folder	/data/hrm_data

If you want to use those variables in your interactive shell later on, just copy-paste the following lines into your session and adjust the values accordingly:

WWW_ROOT=/var/www/html                 # Web server document root
HRM_HOME=$WWW_ROOT/hrm                 # HRM root (home) folder
HRM_CONFIG=$HRM_HOME/config            # HRM configuration folder
HRM_SAMPLES=$HRM_HOME/config/samples   # HRM configuration samples folder
HRM_RESRC=$HRM_HOME/resources          # HRM resources folder
HRM_SETUP=$HRM_HOME/setup              # HRM setup folder
HRM_BIN=$HRM_HOME/bin                  # HRM executables folder
HRM_USER=$HRM_HOME/user                # HRM customization folder
HRM_LOG=/var/log/hrm                   # HRM logging folder
HRM_DATA=/data/hrm_data                # HRM data folder

Manual installation (recommended)

Install the prerequisites

The HRM requires a few prerequisites for its functions.

Operating system

If you have not yet read the conventions, please consult the Operating systems.

Note

For those who plan on using the (Optional) OMERO support it is strongly recommended to set up the HRM and OMERO using the same Linux distribution and version, simply to avoid Ice version incompatibilities, which will prevent the HRM-OMERO connector from working. Details can be found in the OMERO system administrator documentation.

PLEASE MIND: this does NOT mean you should install HRM and OMERO on the same machine! You should just run the same Linux installation on them.

Huygens Core

The HRM is an interface to Scientific Volume Imaging’s Huygens Core. Huygens Core is is a fully scriptable compute engine intended to run image processing and deconvolution jobs on large 64 bit multiprocessor servers in headless mode, i.e. without a specific graphical interface. The HRM provides such an interface for multi-user, batch access to Huygens Core.

Note

If the web and the processing server are not on the same machine, you will need an additional Huygens Core for the web server with a reader license (free of charge).

Apache2 web server

DebianRHEL

sudo apt install apache2

Web pages can be installed globally or per-user.

The Apache2 global document root is /var/www/html.

If you plan to install the HRM in a specific user directory, use /home/<hrm_user>/public_html.

Apache2 access handling

HRM uses .htaccess files to prevent access to configuration files. Make sure to configure Apache2 to use them by setting the AllowOverride and Require directives in configuration files.

DebianRHEL

/etc/apache2/sites-available/000-default.conf

If you are installing HRM in your user directory also put the directives in /etc/apache2/mods-available/userdir.conf Make sure to enable the userdir mod first by running the follwing line in the shell:

sudo a2enmod userdir

See also Enabling use of Apache htaccess files.

Please notice that the Require directive is required from Apache version 2.4. Here is the code snippet for the config files

<Directory /var/www/html/hrm>
    Options Indexes FollowSymLinks
    AllowOverride All
    Require all granted
</Directory>

PHP

Note

Minimum required PHP version is 8.1.

The HRM is made of two parts, a web interface and a queue manager, both written in PHP but with different requirements. The web interface requires the PHP module for Apache2, the queue manager requires the PHP command line interpreter.

DebianRHEL

sudo apt install \
    libapache2-mod-php \
    php \
    php-cli \
    php-common \
    php-json \
    php-xml \
    php-ldap \
    php-mbstring

Note

When installing a PHP version that is not the default of your system it might be necessary to specify for which PHP version the packages are needed. Change php to for example php8.1 in the packages to get the packages for PHP 8.1. It might also be necessary to change the PHP version Apache2 uses with the a2dismod and a2enmod commands.

Production php.ini settings

Please configure the HRM machine for production. Edit the php.ini configuration file

DebianRHEL

/etc/php/8.1/apache2/php.ini

In there set at least the values below (more information can be found in the php.ini file itself).

display_errors = Off
display_startup_errors = Off
error_reporting = E_ALL & ~E_DEPRECATED & ~E_STRICT

A relational database

The HRM officially supports two relational databases: MySQL and PostgreSQL.

MySQL

DebianRHEL

sudo apt install php-mysql mysql-server

Note

It is recommended to install a database management tool like phpmyadmin.

PostgreSQL

DebianRHEL

sudo apt install php-pgsql postgresql

You will need to manually enable PostgreSQL:

sudo systemctl enable postgresql

Note

It is recommended to install a database manageent tool like phppgadmin.

Some additional information:

• This is a good tutorial from the Ubuntu Community on how to set up PostgreSQL to use with the HRM.

Disable SElinux

HRM needs access to several operations that are blocked by SElinux.

DebianRHEL

By default SElinux is disabled. In case SElinux has been enabled edit file /etc/selinux/config and update the following variable.

SELINUX=permissive

Note

Restart the machine after changing the value of SElinux.

Sendmail (postfix)

HRM uses the PHP mail() function to notify the users:

“For the Mail functions to be available, PHP must have access to the sendmail binary on your system during compile time. If you use another mail program, such as qmail or postfix, be sure to use the appropriate sendmail wrappers that come with them.” More…

DebianRHEL

sudo apt install postfix

Note

If your mail server is set up correctly and still PHP cannot send e-mails, SELinux might be blocking it. Query the status of httpd_can_sendmail as follows:

$ /usr/sbin/getsebool httpd_can_sendmail

If httpd_can_sendmail is off, you can enable it with:

$ /usr/sbin/setsebool -P httpd_can_sendmail on

PHP date() and default timezone

Please make sure to set the default timezone in php.ini as follows:

[Date]
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = "Europe/Zurich"

Otherwise you will get the following warning every time the PHP function date() is called within the HRM:

PHP Warning: date(): It is not safe to rely on the system's timezone settings. You are
required to use the date.timezone setting or the date_default_timezone_set() function. (...)

Click here for the full list of supported time zones.

SSH

If the queue manager and the image processing server are not on the same machine (see installation instructions), HRM transfers files via ssh between the two using sudo. To allow HRM to login and run commands as sudo via remote, it is necessary to comment out the line 'Defaults requiretty' in the /etc/sudoers file.

Compressors

The HRM compresses files to be downloaded (such as deconvolution results). Several options are possible (and more can be added in the configuration files), but by default the HRM uses zip.

DebianRHEL

.. code-block:: sh

    sudo apt install zip

(Optional) OMERO support

For using the OMERO connector, you will need to install the corresponding Python package. Please refer to the HRM-OMERO Connector project for installation details.

Get the HRM

Current version of the HRM is 3.8.x.

From archive

Download or checkout the HRM from Sourceforge and unzip it into the $WWW_ROOT.

Bellow is an command example:

sudo unzip hrm_3.9.0.zip -d /var/www/html

Using Git

The advantage of using git is that later upgrades are easy to perform, and all the modifications are documented this way. You need to install git on the machine.

DebianRHEL

sudo apt install git

To get the latest stable release directly from our git repository, change to the $WWW_ROOT directory and run the following in a shell:

cd $WWW_ROOT
git clone -b master --single-branch https://github.com/aarpon/hrm
cd hrm
./setup/setup_release.sh

To use an older release, versions can be listed with:

git tag -l

The tags are the version numbers, that can be used to work with a specific release by checking it out:

git checkout <tag>

Once the release of choice is checked out, you might want to create a local branch before starting to configure the HRM.

git checkout -b deployed

Per default the $HRM_HOME/config is in the .gitignore file, so if you want to put your configurations under version control you have to modify the .gitignore and commit your changes to the “deployed” branch.

HRM development

If you want to follow the HRM development, you can checkout the read-only HRM repository and and configure your working copy for development:

$ git clone https://github.com/aarpon/hrm
$ cd hrm
$ git checkout devel
$ ./setup/setup_devel.sh

Find more detailed instructions in the installation pages.

Warning

Please keep in mind that the latest development snapshot is unstable software and should not be deployed on production machines!

Configure the HRM

User and group permissions

In the following instructions, HRM_DATA points to the data folder that contains all user subfolders (for example: /data/hrm_data) and HRM_LOG is the log folder for the HRM (for example: /var/log/hrm). Please see also the conventions. It is important that those folder have the correct ownership and permissions set for HRM to work properly. Please follow these steps.

Create a Unix group hrm and user hrm on the web server machine.

$ sudo groupadd --system hrm
$ sudo useradd hrm --system --gid hrm --create-home

Make sure the log and data directories exist:

sudo mkdir -pv ${HRM_DATA}
sudo mkdir -pv ${HRM_LOG}

Make sure hrm owns (and has full read-write access) to HRM_DATA and HRM_LOG. This is done by setting the group ownership of HRM_DATA and HRM_LOG to hrm:

sudo chown -R hrm:hrm ${HRM_DATA}
sudo chmod -R u+s,g+ws ${HRM_DATA}
sudo chown -R hrm:hrm ${HRM_LOG}
sudo chmod -R u+s,g+ws ${HRM_LOG}

Add the Apache user to the hrm group:

DebianRHEL

sudo usermod www-data --append --groups hrm

Finally restart your web server for the group changes to be activated.

DebianRHEL

sudo systemctl restart apache2.service

Server

The hrm.conf file is used by both the webserver and the queue-manager (back-end).

Copy the sample file

The sample configuration file $HRM_SAMPLES/etc/hrm.conf.sample must be copied to /etc/hrm.conf and then edited as explained in the following sections.

sudo cp $HRM_SAMPLES/etc/hrm.conf.sample /etc/hrm.conf

Edit the configuration file

This is the content of the sample configuration file:

#!/bin/bash
#
# HRM configuration file
#
# This file is part of the Huygens Remote Manager
# Copyright and license notice: see license.txt

# HRM install directory
HRM_HOME="/path/to/hrm/home"

# HRM image share
HRM_DATA="/path/to/hrm/data"

# Source and destination folder names
HRM_SOURCE="src"
HRM_DEST="dst"

# User to run the HRM daemon with rights to
# execute /bin/mkdir, /bin/chown, /bin/chmod and
# /bin/rm on HRM_DATA (see above) for user management.
SUSER="hrm"

# HRM log directory
HRM_LOG="/var/log/hrm"

# Interaction with OMERO (if switched on in hrm/config).
OMERO_HOSTNAME="full.qualified.name.of.omero.server"
OMERO_PORT="4064"

# Set a custom PHP CLI binary if necessary:
# PHP_CLI="/usr/local/php/bin/php"

Explanation

• HRM_HOME points to $WWW_ROOT/hrm (for example: /var/www/html/hrm).

• HRM_DATA points to the data folder that contains all user subfolders (for example: /data/hrm_data).

• HRM_SOURCE and HRM_DEST are the source and destination subfolders inside the user directory (for example: src and dst). The source folder for an hypothetical user ‘john’ would then be /data/hrm_data/john/src.

• SUSER is the Unix user that runs the Queue Manager (for example: hrm).

• HRM_LOG is the log folder for the HRM (for example: /var/log/hrm)

• (optional) OMERO_HOSTNAME and OMERO_PORT: see OMERO connector for details.

• (optional) PHP_CLI is the path tho the php CLI executable if it is not in the path or another one should be used (for example: /usr/local/php/bin/php).

Webapp (PHP)

These configurations concern PHP, hence the web-application (front end) itself.

Copy the sample file

Copy $HRM_SAMPLES/hrm_config.inc.sample to $HRM_CONFIG/hrm_config.inc:

sudo cp -v $HRM_SAMPLES/hrm_config.inc.sample $HRM_CONFIG/hrm_config.inc

Edit the configuration file

Here we will assume a one-machine installation:

<?php
// This file is part of the Huygens Remote Manager
// Copyright and license notice: see license.txt

// This configuration file is used by the QUEUE MANAGER.

//==============================================================================
// Database settings
//==============================================================================

// The database type ('mysqli' or 'postgres'; mysql is deprecated).
// Please notice that it is highly recommended to use the mysqli driver for
// PHP 5.5 and above. In PHP >= 5.6, the mysql driver is deprecated and will
// overflow your log files when used!
$db_type = "mysqli";

?>

Note

Although the database abstraction library (ADOdb) we use in the HRM can interface with a large number of databases, we currently only support MySQL and PostgreSQL. Other DBs may work, but we haven’t tested them.

Note

Also, please use the ‘mysqli’ driver instead of ‘mysql’ as in previous versions of the HRM. From PHP 5.6, the ‘mysql’ driver is deprecated!

<?php
...
// The name of the database host machine (this may be localhost if it is the
// machine on which the web server runs)
$db_host = "localhost";

// The name of the database used by HRM
$db_name = "hrm";

// The name of the user with which the database is accessed
$db_user = "hrm";

// The password of the account with which the database is accessed
$db_password = "dbpasswd";

//==============================================================================
// Huygens settings
//==============================================================================

// Huygens server default user
$huygens_user = "hrm";
$huygens_group = "hrm";
?>

The $huygens_user is used in case the processing machine (the one where Huygens Core is installed) does not have direct access to the file server (i.e. the files to be deconvolved must be copied from the file server to the processing machine via ssh). For this to work, you will need to set up password-less ssh connection between file and processing server for the $huygens_user (see for example here) and also set the variable $copy_images_to_huygens_server to true (see below).

<?php
...
// Path to a local Huygens Core executable, to handle image information and
// thumbnails. This installatation of hucore doesn't require a full license
// unless this is also the computation server. For image information and
// thumbnails mostly freeware functions will be used, but support for certain
// propietary file formats requires a B flag in the license string. (See
// http://support.svi.nl/wiki/FileFormats).
//
// In combination with Huygens 3.5.1 or higher, HRM can also provide an
// estimate of the Signal-To-Noise ratio (SNR) of an image, interactively. At
// least that same simple license is required for this to work.
$local_huygens_core = "/usr/local/bin/hucore";
?>

The web interface uses hucore for some operations on the input and result files (for example to generate image previews or estimate the SNR interactively). An additional copy of hucore needs therefore to be installed on the web server. However, since all these functions used are accessible in reader mode, only a reader license is needed (free of charge).

<?php
//==============================================================================
// File server settings
//==============================================================================

// File server host name
$image_host = "localhost";

// File server default user
$image_user = "hrm";
$image_group = "hrm";

// File server base folder
$image_folder = "/data/hrm_data";

// File server image source folder
$image_source = "src";

// File server image destination folder
$image_destination = "dst";

// File server base folder as seen from the Huygens server machines
$huygens_server_image_folder = "/data/hrm_data";
?>

This is information needed for the web interface and the queue manager to login to the file server. If the file server is not on the same machine, its host name must be given.

<?php
...
// Allow HTTP transfer of the restored (download) images
$allowHttpTransfer = true;

// Allow HTTP transfer of original (upload) images.
// Limitations in upload size must be configured in the below parameters
// $max_upload_limit AND $max_post_limit. If these are set to 0, they are
// instead read from the PHP settings. If set to higher values, make sure
// that the PHP settings allow for such transfer limits. These PHP settings
// are normally located in php.ini, with the options upload_max filesize AND
// post_max size.
// Unzipping large archives may also require high max_execution_time and
// memory_limit variables.
$allowHttpUpload = true;
$max_upload_limit = 0; // Maximum file size for uploads in Mb
$max_post_limit = 0;   // Maximum post size for uploads in Mb

// Archiver for downloading the results via the web browser:
// $compressBin is the archiver, including options.  %DEST% will be replaced
// with the user's destination path; can be used as an option or part of the
// command to change directories with 'cd'. Separate commands with "\n".
// $packExcludePath controls whether the archiving command excludes the whole
// file path per file when this is taken into account in the $compressBin
// command itself by using %DEST% properly.

$compressExt = ".zip";

switch ($compressExt) {
    case ".tgz":
    case ".tar.gz":
        $compressBin = "cd %DEST% \n /bin/tar chzf ";
        $packExcludePath = true;
        $dlMimeType = "application/x-force-download";
        break;
    case ".tar":
        $compressBin = "cd %DEST% \n /bin/tar cfh ";
        $packExcludePath = true;
        $dlMimeType = "application/x-force-download";
        break;
    case ".zip":
        $compressBin = "cd %DEST% \n /usr/bin/zip ";
        $packExcludePath =true;
        $dlMimeType = "application/x-force-download";
        break;
    default:
        $compressExt = ".zip";
        $compressBin = "cd %DEST% \n/usr/bin/zip -D";
        $packExcludePath = true;
        $dlMimeType = "application/x-force-download";
}

// Tools to decompress uploaded archives. This is an array, each key linked to
// an archive extension.  %DEST% will be replaced with the user's destination
// path; can be used as an option or part of the command to change directories
// with 'cd'.
// More formats are possible, just add more commands to the array.

$decompressBin['zip'] = "cd %DEST% \n /usr/bin/unzip -o ";
$decompressBin['tgz'] = "cd %DEST% \n /usr/bin/tar xzf ";
$decompressBin['tar'] = "cd %DEST% \n /usr//bin/tar xf ";
$decompressBin['tar.gz'] = "cd %DEST% \n /usr/bin/tar xzf ";
?>

If $allowHttpTransfer is true, the results of deconvolution can be downloaded through the web interface. If it is false, then the results can only be accessed by other means (e.g. via network shares). Downloaded files are compressed to reduce bandwidth load: $compressExt defines the type of compression (default is zip).

Note

For the compressed file to be served by the web server, it has to be re-read in. The HRM uses chunked reading, that should make sure that even large files can be read without problems. Should the users still encounter problems downloading files from the HRM, then try increasing the value of memory_limit in php.ini.

Example:

memory_limit = 128M

Warning

Although HRM takes care of cleaning all temporary files after each download, it may happen that data is left behind if the process fails for any reason. Is is recommended that an HRM admin checks and if necessary cleans the ${HRM_DATA}/.hrm_downloads folder from time to time.

If $allowHttpUpload is true, an HTTP uploader will be in place to allow for the uploading of the files to be deconvolved through the web interface. Multi-file upload (with directory structures) is indirectly possible by first compressing the files into an archive that the HRM will automatically extract at the end upload process. Default supported formats are zip, tgz, tar and tar.gz (as long as the corresponding executables are installed on the system) but more can be added by extending the $decompressBin array.

Note

The behavior of the HTTP uploader depends on the php.ini directives upload_max_filesize and post_max_size, but can also optionally be tweaked by the HRM variables $max_upload_limit and $max_post_limit.

The HTTP uploader uses the php.ini directive upload_max_filesize to define the maximum size of each selected file and the post_max_size directive to define the total upload size allowed in one upload session.

The HRM configuration variables $max_upload_limit and $max_post_limit can be used to set more restrictive values in case the web server runs other services besides HRM. By default, both variables are set to 0 so that the values from php.ini are used.

Note

The HRM administrator can check the values in use in the System summary page, under Configuration.

Example:

post_max_size = 2048M
upload_max_filesize = 1024M

Note

Notice that despite some script executions are relatively fast (like file extraction from zip archives, image previews generation, or SNR estimation), they may take a long time if the images are very large or if deconvolution jobs are already running in the web server. You may need to increase also PHP’s max_execution_time directive in php.ini.

Example:

max_execution_time = 120

Warning

Although HRM takes care of cleaning all temporary files after each upload, it may happen that data is left behind if the process fails for any reason. Is is recommended that an HRM admin checks and if necessary cleans the ${HRM_DATA}/.hrm_chunks and ${HRM_DATA}/.hrm_files folders from time to time.

<?php
...
//==============================================================================
// HRM configuration parameters
//==============================================================================

// URL
$hrm_url = "http://localhost/hrm";

// Install dir
$hrm_path = "/var/www/html/hrm";

// Logging
// Please create a directory for the logging (default and recommended is /var/log/hrm)
// and make sure to grant read/write access to the web server user.
$log_verbosity = 1;         // 0=quiet, 1=normal, 2=debug
$logdir  = "/var/log/hrm";
$logfile = "hrm.log";
$logfile_max_size = 100;    // maximum size of the logfile in MB
?>

The $logdir variable points to the directory where the log files created by the web server user and the queue manager should reside. Default (and recommended) directory for the logs is /var/log/hrm. Please create this directory and make sure to grant the relevant users read/write access to it!

<?php
...
// Email
$send_mail = true;
$email_sender = "hrm@localhost";
$email_admin = "hrm@localhost";
// Comma (',') is the standard separator for lists of email addresses.
// Please see: http://tools.ietf.org/html/rfc2822#section-3.6.3
// Unfortunately, Microsoft Outlook uses semicolon (';') by default and
// ',' only optionally. Please see: http://support.microsoft.com/kb/820868
// on how to configure Outlook to support comma-separated lists.
// If your users are mostly using Outlook, you can tell the HRM to use ';'
// instead of the standard ',' for distribution lists.
$email_list_separator = ",";    // Valid options are ',' and ';'.

Enable and configure the e-mail service.

<?php
...
// Supported authentication types. Possible values are 'integrated', 'active_dir', 'ldap'.
// These values replace the previous ones (but the old ones are still recognized):
// 'integrated' replaces 'MYSQL';
// 'active_dir' replaces 'ACTIVE_DIR';
// 'ldap' replaces 'LDAP'.
// You can use more than one authentication mechanism by listing them all in the
// $authenticateAgainst array below.
//
// e.g. $authenticateAgainst = array("active_dir", integrated");
//
// The first entry in the array is the default authentication mode that will be used
// for new users. Administrators can then change the authentication mode for individual
// users in the user management pages.
//
// Make sure to properly configure all authentication modes
// (see sample files in config/samples).
$authenticateAgainst = array("integrated");
?>

Setting the authentication mode to integrated enables the embedded user management system (independent of the actual database you are using, e.g. postgresql). Use this if you don’t have any other user management system in place. Alternatively, simple authentication against Microsoft’s Active Directory and OpenLDAP is possible.

In case one or more of the alternative authentication types are selected (i.e. active_dir or ldap), some additional configuration will be required (see later).

Note

The super administrator ‘admin’ always has authentication mode ‘integrated’!

<?php
...
// If true, use DES password encryption; if false, use MD5.
// This variable is DEPRECATED and will be removed in the next HRM version!
$useDESEncryption = false;

// If true, the queue manager and the image processing server run on the same
// machine
$imageProcessingIsOnQueueManager = true;

// If true the images are copied to the huygens server machine
$copy_images_to_huygens_server = false;
?>

Set $imageProcessingIsOnQueueManager to false if the queue manager is not on the same machine as the Huygens Core that runs deconvolution.

Set $copy_images_to_huygens_server to true if the files have to be copied to the machine where the Huygens Core is installed via ssh (using password-less ssh connection!). If Huygens Core has direct access to the file area (e.g. through NFS mount), set this to false.

Note

The use of the variable $useDESEncryption is deprecated and will be removed in the next HRM version!

<?php
...
// Thumbnails require Huygens Core 3.3.2 or higher.

// Notice that despite some script executions are relatively fast (like
// thumbnail generation and SNR estimation), they still may take a long time if
// the images are very large, or if deconvolution jobs are already running in
// the local server. You may need to increase PHP's max_execution_time
// directive in php.ini, and restart the web server.

// If true, allows generates thumbnails and previews for the originals and the
// results during the deconvolution (i.e. in the computation servers). These
// are maximum intensity projections (MIP) of the stacks along the main axes,
// so they provide xy, xz and zy views.
$useThumbnails = true;

// If true, allows on-demand generation of a preview of the original images in
// this web server prior to the deconvolution. For this to work the variable
// $local_huygens_core must be set correctly above. Notice that not all file
// formats are readable in a freeware Huygens Core and for full functionality
// you may need a license string. Otherwise just let the original thumbnails
// be generated during the deconvolution itself with the previous option.
// If this is true and the correct version of Huygens is installed, the SNR
// estimator tools is also enabled (see $enable_code_for_huygens above).
// This requires $useThumbnails = true;
$genThumbnails = true;

// Moreover, if $genThumbnails is true and Huygens 3.5.1 or higher is
// installed (see $enable_code_for_huygens above), the Signal To Noise ratio
// can be estimated visually from raw images, in the restoration parameters
// editor.

// If larger than zero, generates an AVI movie for 3D stacks that allows
// browsing along the XY slices, and for time series to browse along MIPs in
// time. The movie will have this maximum number of pixels in the largest
// dimension.
// This requires $useThumbnails = true;
$movieMaxSize = 300;

// If true, (and $useThumbnails is also true), it generates not only the MIP
// previews but also Simulated Fluorescence Process top-view renderings of the
// original and the restored images (http://support.svi.nl/wiki/SFP).
$saveSfpPreviews = true;

// If non-zero, save stack and time series previews of the original and restored
// data side by side, to allow comparisons. (Requires $useThumbnails and
// Huygens Core 3.5.2). This is the max size in pixels for each of the images's
// slicer.
$maxComparisonSize = 300;

These were several options for preview generations.

<?php
...
// A shell command that executes ping four times and stops afterwards
$ping_command = 'ping -w 4';    // use this on linux systems
// $ping_command = 'ping -t 4'; // use this on macosx systems
// $ping_command = 'ping';  // use this on cygwin

// The parameter for the ping command after the hostname
$ping_parameter = '';           // use this on linux systems
//$ping_parameter = '56 4'; // use this on cygwin
?>

This is used by the queue manager to test whether the processing machine is reachable.

Note

In HRM it is possible to use a machine for each of the components, i.e. web server, database server, one or more processing machines, file server and queue manager. Configuration files are needed only on the web server and the machine running the queue manager. Make sure to fill the various path and user fields from the perspective of the machine that hosts each of the configuration files!

Authentication

The HRM curently supports three user authentication mechanisms:

• Simple, internal user management module (default - no additional configuration needed)

• Generic LDAP server

• Microsoft’s Active Directory - (AD) server

The HRM can be configured to use LDAP/AD to test user credentials and obtain limited user information (e-mail address angd group).

You can enable any combination of authentication mechanisms by editing $HRM_CONFIG/hrm_config.inc. The first one is the default for new users. Valid examples are:

<?php
...
// Only integrated authentication (default)
$authenticateAgainst = array("integrated");

// Authenticate against Active Directory by default;
// some users may be checked against integrated authentication
$authenticateAgainst = array("active_dir", "integrated");

// Integrated authentication by default;
// some users may be authenticated against ldap
$authenticateAgainst = array("integrated", "ldap");

// This is also possible, however unlikely in reality:
$authenticateAgainst = array("active_dir", "ldap", "integrated");
...

The following pages explain the required configuration steps for the support authentication mechanisms:

Active Directory authentication

To enable Active Directory authentication, please add “active_dir” to the $authenticateAgainst array in $HRM_CONFIG/hrm_config.inc

<?php
...
$authenticateAgainst = array("active_dir");
...

Then, copy $HRM_SAMPLES/active_directory.config.inc.sample to $HRM_CONFIG/active_directory.config.inc and edit it.

<?php
// This file is part of the Huygens Remote Manager
// Copyright and license notice: see license.txt

// See http://adldap.sourceforge.net/wiki/doku.php?id=api_configuration for
// detailed help on configuring adLdap.

// The account suffix for your domain
$ACCOUNT_SUFFIX = "@mydomain.local";

// The base dn for your domain
$BASE_DN = "DC=mydomain,DC=local";

// Array of domain controllers. Specifiy multiple controllers if you would
// like the class to balance the LDAP queries amongst multiple servers
$DOMAIN_CONTROLLERS = array ("dc01.mydomain.local");

// Domain controller port
$AD_PORT = 389;

// User name suffix
$AD_USERNAME_SUFFIX = "";

// User name suffix matching string
$AD_USERNAME_SUFFIX_REPLACE_MATCH = "";

// User name suffix replacing string
$AD_USERNAME_SUFFIX_REPLACE_STRING = "";

// Optional account with higher privileges for searching (otherwise
// leave it to NULL). This should be set to a domain admin account (only
// read operations are performed!)
$AD_USERNAME = NULL;
$AD_PASSWORD = NULL;

// Tweak to get the real primary group from Active Directory. It works if
// the user's primary group is domain users.
// http://support.microsoft.com/?kbid=321360
$REAL_PRIMARY_GROUP = true;

// Use SSL (LDAPS)
$USE_SSL = false;

// Use TLS: if you wish to use TLS you should ensure that $USE_SSL is
// set to false and vice-versa
$USE_TLS = false;

// When querying group memberships, do it recursively
$RECURSIVE_GROUPS = true;

// Group filtering and authorization.
//
// Users in Active Directory usually belong to several groups. Many of those groups
// will not be relevant for the administrator of the HRM (groups like 'Domain Users',
// 'Building 12', 'Staff') and the important ones, like the research group, are hidden
// somewhere in this list.
//
// The $VALID_GROUPS array can be used to specify only those groups that are interesting
// for the HRM and that will be used as the user group in the HRM and will show up in the
// usage statistics. For example, only the research groups at the institution might be
// included in this list.
//
// Additionally, not all research groups might be allowed to use the HRM. The
// $AUTHORIZED_GROUPS array can be used to specify those groups that are allowed to use
// the HRM.
//
// Please notice that $AUTHORIZED_GROUPS must not strictly be a subset of $VALID_GROUPS.
//
// Example 1
// =========
//
// $AUTHORIZED_GROUPS = array('group_einstein', 'group_bohr');
// $VALID_GROUPS = array('group_einstein', 'group_bohr');
//
// User 'john' has groups {'Domain Users', 'Building 12', 'Staff', 'group_einstein'}
//
// 'john' is allowed to log in and the group used in HRM is 'group_einstein'.
//
// User 'stephanie' has groups {'Domain Users', 'Building 12', 'Staff', 'group_bohr'}
//
// 'stephanie' is allowed to log in and the group used in HRM is 'group_bohr'.
//
// User 'jeff' has groups {'Domain Users', 'Building 1', 'Admin Staff', 'group_admin'}
//
// 'jeff' is not allowed to log in ('group_admin' is not in $AUTHORIZED_GROUPS)'.
//
// Example 2
// =========
//
// $AUTHORIZED_GROUPS = array('hrm');
// $VALID_GROUPS = array('group_einstein', 'group_bohr');
//
// User 'john' has groups {'Domain Users', 'Building 12', 'Staff', 'group_einstein', 'hrm'}
//
// 'john' is allowed to log in and the group used in HRM is 'group_einstein'.
//
// User 'stephanie' has groups {'Domain Users', 'Building 12', 'Staff', 'group_bohr'}
//
// 'stephanie' is not allowed to log in (she does not belong to group 'hrm').
//
// User 'jeff' has groups {'Domain Users', 'Building 1', ''Admin Staff', 'group_admin'}
//
// 'jeff' is not allowed to log in (she does not belong to group 'hrm').
//
// Example 3
// =========
//
// $AUTHORIZED_GROUPS = array();
// $VALID_GROUPS = array('group_einstein', 'group_bohr');
//
// No restriction on which groups can log in to HRM. Group filtering works as in Examples 1
// and 2.
//
// Example 4
// =========
//
// $AUTHORIZED_GROUPS = array();
// $VALID_GROUPS = array();
//
// If neither valid nor authorized groups are defined, all groups are allowed to log in and
// the first returned group will be used in HRM.
//
// User 'john' has groups {'Domain Users', 'Building 12', 'Staff', 'group_einstein', 'hrm'}
//
// john's group in HRM will be 'Domain Users'.
//
// Example 5
// =========
//
// $AUTHORIZED_GROUPS = array(... any number of entries ...);
// $VALID_GROUPS = array();
//
// If $VALID_GROUPS is empty and $AUTHORIZED_GROUPS contains one or more entries,
// $VALID_GROUPS will be reset to be the same as $AUTHORIZED_GROUPS (the end behavior
// will be as in Example 1).

// Groups to use in HRM (filter)
$VALID_GROUPS = array();

// Groups authorized to log in to HRM.
$AUTHORIZED_GROUPS = array();

Special case: Active Directory forests

Warning

For most installations, you can skip the entire rest of this section and proceed with the next one by simply leaving the variables $AD_USERNAME_SUFFIX, $AD_USERNAME_SUFFIX_REPLACE_MATCH and $AD_USERNAME_SUFFIX_REPLACE_STRING with their default values (empty, i.e. "").

This part of the config is meant for the special case of an Active Directory with multiple domains (commonly referred to as forest) which usually consist of a top-level domain (the forest root domain) and several sub-domains. If the HRM users belong to domains in all levels of the forest it would be required to enter the full domain name for logging in, which can become quite tedious (and therefore error-prone).

Consider the following domain setup, a root domain called SATURN with two sub-domains RHEA and TETHYS. Those “plain” domain names are usually referred to as NetBIOS Domain Names. A hierarchical representation could look like this:

SATURN
  |
  +-- RHEA
  |
  +-- TETHYS

Now, assuming a top-level suffix of .private, in Microsoft’s Active Directory naming scheme this is translated into fully qualified domain names (FQDN’s) as follows. The root domain (SATURN) gets prefixed by ads. and followed by the top-level .private suffix, whereas the two subdomains just get postfixed by the FQDN of the root domain. The graphical representation from above would therefore translate into this:

ads.saturn.private
  |
  +-- rhea.ads.saturn.private
  |
  +-- tethys.ads.saturn.private

To make authentication work in such a scenario, it is required to authenticate against the GLOBAL CATALOG of the Active Directory. This is an LDAP service provided only on the domain controllers of the root domain on the special port 3268. The usernames used to authenticate have to be of the form username@fully.qualified.domain.name to provide the global catalog with the information to which sub-domain a certain account belongs as it is possible to have the same username for different accounts in different domains of a single domain forest.

Now given the HRM configuration option $ACCOUNT_SUFFIX would be set to .saturn.private, a user foo from the domain SATURN could log in using something like foo@ads whereas a user bar from RHEA would have to type bar@rhea.ads. This is obviously screaming for problems as normal users don’t understand where and why to put those weird suffixes, especially as they’ve never heard of something like ads before.

Using the sample config from below, users can simply log on by using their regular account name followed by an @ sign and the (short) NetBIOS domain name, e.g.

foo@saturn

or

bar@rhea

which is way easier to explain and remember for them, especially as the short domain names are usually visible for them when logging into their workstation.

<?php
...
$ACCOUNT_SUFFIX = "";
$BASE_DN = "DC=ads, DC=saturn, DC=private";

// set this to a domain controller of the *ROOT* domain:
$DOMAIN_CONTROLLERS = array("dc01.ads.saturn.private");

// the port is required to connect to the global catalog:
$AD_PORT = 3268;

// give the FQDN of the root domain with a leading dot here:
$AD_USERNAME_SUFFIX = ".ads.saturn.private";

// account names from the root domain will end up wrong, looking like this
// (note the leading "@" and the double occurence of the root domain):
$AD_USERNAME_SUFFIX_REPLACE_MATCH = "@saturn.ads.saturn.private";

// they have to be rewritten using the correct suffix instead (removing the
// first occurence of the root domain name):
$AD_USERNAME_SUFFIX_REPLACE_STRING = "@ads.saturn.private";
?>

Please feel free to contact the HRM developers in case you need help setting up authentication in a multi-domain forest scenario!

TLS Configurations

To secure the connection between the HRM server and the LDAP server, it is recommended to enable the transport layer security (TLS). Once the LDAP authentication described in the previous section works, We need to do the modifications bellow to encrypt the communication between our servers.

First we need to edit the /etc/ldap/ldap.conf and add the following line in the file:

TLS_REQCERT never

Then in $HRM_CONFIG/active_directory_config.inc change the following lines:

$AD_PORT = 636;
$USE_SSL = true;

Testing and Debugging

To test if the base_DN and search_DN work as well as for testing the TLS configurations, there is a helper script in $HRM_HOME/resources/debugActiveDirectory.php. Before testing this script needs editing to assign to the $query_u variable a meaningful user name. Then the $HRM_HOME/resources directory it can by directly executed with the following command.

php debugActiveDirectory.php

This will generate some useful output to help troubleshooting connection and query problems.

LDAP authentication

To enable LDAP authentication, please add “ldap” to the $authenticateAgainst array in $HRM_CONFIG/hrm_config.inc

<?php
...
$authenticateAgainst = array("ldap");
...

Then, copy $HRM_SAMPLES/ldap.config.inc.sample to $HRM_CONFIG/ldap.config.inc and edit it.

<?php
// This file is part of the Huygens Remote Manager
// Copyright and license notice: see license.txt

// The machine on which the ldap server is running
$ldap_host = "localhost";

// The port for the ldap connection
$ldap_port = "389";

// Use SSL (LDAPS)
$ldap_use_ssl = false;

// Use TLS: if you wish to use TLS you should ensure that $ldap_use_ssl is
// set to false and vice-versa
$ldap_use_tls = false;

// The ldap root
$ldap_root = "dc=root,dc=country";

// The base for the manager DN (either cn or uid)
$ldap_manager_base_DN = "cn";

// The ldap manager (username only!)
$ldap_manager = "manager";

// The ldap manager password
$ldap_password = "secret";

// The ldap manager OU: use this in case the ldap_manager is in some special OU
// that distinguishes it from the other users. $ldap_manager_ou will be
// prepended to $ldap_user_search_DN.
// Set it to "" if $ldap_user_search_DN can be used for binding
$ldap_manager_ou = "ou=special_user";

// The user search DN (without ldap root)
$ldap_user_search_DN = "cn=users";

// Group filtering and authorization.
//
// Users in LDAP usually belong to several groups. Many of those groups
// will not be relevant for the administrator of the HRM (groups like 'Domain Users',
// 'Building 12', 'Staff') and the important ones, like the research group, are hidden
// somewhere in this list.
//
// The $ldap_valid_groups array can be used to specify only those groups that are
// interesting for the HRM and that will be used as the user group in the HRM and will
// show up in the usage statistics. For example, only the research groups at the institution
// might be included in this list.
//
// Additionally, not all research groups might be allowed to use the HRM. The
// $ldap_authorized_groups array can be used to specify those groups that are allowed
// to use the HRM.
//
// Please notice that $ldap_authorized_groups must not strictly be a subset of
// $ldap_valid_groups.
//
// Example 1
// =========
//
// $ldap_authorized_groups = array('group_einstein', 'group_bohr');
// $ldap_valid_groups = array('group_einstein', 'group_bohr');
//
// User 'john' has groups {'Domain Users', 'Building 12', 'Staff', 'group_einstein'}
//
// 'john' is allowed to log in and the group used in HRM is 'group_einstein'.
//
// User 'stephanie' has groups {'Domain Users', 'Building 12', 'Staff', 'group_bohr'}
//
// 'stephanie' is allowed to log in and the group used in HRM is 'group_bohr'.
//
// User 'jeff' has groups {'Domain Users', 'Building 1', 'Admin Staff', 'group_admin'}
//
// 'jeff' is not allowed to log in ('group_admin' is not in $ldap_authorized_groups)'.
//
// Example 2
// =========
//
// $ldap_authorized_groups = array('hrm');
// $ldap_valid_groups = array('group_einstein', 'group_bohr');
//
// User 'john' has groups {'Domain Users', 'Building 12', 'Staff', 'group_einstein', 'hrm'}
//
// 'john' is allowed to log in and the group used in HRM is 'group_einstein'.
//
// User 'stephanie' has groups {'Domain Users', 'Building 12', 'Staff', 'group_bohr'}
//
// 'stephanie' is not allowed to log in (she does not belong to group 'hrm').
//
// User 'jeff' has groups {'Domain Users', 'Building 1', ''Admin Staff', 'group_admin'}
//
// 'jeff' is not allowed to log in (she does not belong to group 'hrm').
//
// Example 3
// =========
//
// $ldap_authorized_groups = array();
// $ldap_valid_groups = array('group_einstein', 'group_bohr');
//
// No restriction on which groups can log in to HRM. Group filtering works as in Examples 1
// and 2.
//
// Example 4
// =========
//
// $ldap_authorized_groups = array();
// $ldap_valid_groups = array();
//
// If neither valid nor authorized groups are defined, all groups are allowed to log in and
// the first returned group will be used in HRM.
//
// User 'john' has groups {'Domain Users', 'Building 12', 'Staff', 'group_einstein', 'hrm'}
//
// john's group in HRM will be 'Domain Users'.
//
// Example 5
// =========
//
// $ldap_authorized_groups = array(... any number of entries ...);
// $ldap_valid_groups = array();
//
// If $ldap_valid_groups is empty and $ldap_authorized_groups contains one or more entries,
// $ldap_valid_groups will be reset to be the same as $ldap_authorized_groups (the end
// behavior will be as in Example 1).

// Groups to use in HRM (filter)
$ldap_valid_groups = array();

// Groups authorized to log in to HRM.
$ldap_authorized_groups = array();

Database setup

Preparing the database for the HRM consists of two steps, first the corresponding user needs to be created and configured and then the database itself has to be created and prepared for the HRM.

Create the HRM database user

Unless the database user for the HRM is already existing, you need to create it and assign the corresponding permissions.

MySQL

Note

On Ubuntu 22.04 LTS and newer you might need to follow these instructions to set the mysql root password.

The following command will create a new MySQL user hrm and grant the permissions to a database hrm@localhost using the mysql command line tool. Of course, you can also use a database management tool like phpmyadmin to perform this task if you prefer. Make sure to adjust the password for the new database user so it matches the one from your configuration files.

# start the mysql command line client and connect as root:
mysql -u root -p
# Create the database if it does not exist yet:
CREATE DATABASE hrm CHARACTER SET utf8;
# now from within the client, create the database user:
CREATE USER 'hrm'@'localhost' IDENTIFIED BY 'dbpasswd';
# grant permissions to the user:
GRANT ALL ON hrm.* to 'hrm'@'localhost';
exit;

PostgreSQL

su postgres -c "createuser -e -P -d -A -S -R hrm"
su postgres -c "createdb hrm"

RHEL

MD5 host authentication has to be enabled explicitly on RHEL systems. This can be done using the following commands:

sudo -s
echo -e "host all hrm 127.0.0.1/32 md5\n" >> /var/lib/pgsql/data/pg_hba.conf
echo -e "host all hrm ::1/127 md5\n" >> /var/lib/pgsql/data/pg_hba.conf
service postgresql restart

On Ubuntu MD5 host authentication is enabled by default.

Create or update the database

Run the php script $HRM_SETUP/dbupdate.php from the shell to create and populate the HRM database. The same command can be used to perform an upgrade of the database when the HRM version is upgraded:

cd $HRM_SETUP
php dbupdate.php

If the database does not exist, it will be created using the information stored in the $HRM_CONFIG/hrm_config.inc and filled with content for the latest revision. If it exists, it will be updated from whichever revision it currently has.

Note

MySQL 8+ might return the following error: ‘The server requested authentication method unknown to the client’. Which can be fixed by

# Start the mysql command line client and connect as root:
mysql -u root -p
# Notify MySQL that the HRM user logs in with a password.
ALTER USER '$HRMUSER'@'localhost' IDENTIFIED WITH mysql_native_password BY
'$HRMUSERPASSWORD';
FLUSH PRIVILEGES;
exit;

Processing server (HuCore)

In the table server of the HRM database, two default values are set for the fields name (i.e. the name of the server running Huygens Core) and huscript_path (i.e. the path pointing to the hucore executable).

name has default value localhost, but can also be something like remote.server.com. huscript_path has default value /usr/local/bin/hucore.

If you need to change the default values, you can use your favorite SQL client and execute a query similar to this (for MySQL):

UPDATE server SET name = 'remote.server.com', huscript_path = '/usr/bin/hucore';

Note

The HRM requires the hucore binary to work, not huscript. Before hucore was introduced HRM was indeed using huscript; the huscript_path entry in the database was kept not to break existing HRM installations.

Warning

The server table contains two other entries: status and job. These are used by the interface at runtime, please do not set them manually!

Daemon setup

The HRM daemon consists of two parts, the actual queue manager (a PHP process) and a shell-based wrapper in $HRM_BIN/hrm_queuemanager that prepares the environment, checks the config, launches the queue manager and cleans up after the process has terminated.

The configuration file for the HRM daemon is /etc/hrm.conf (see Server)

Warning

You should never start the PHP process directly as unpredicted effects will happen, in worst case including damage to the HRM database.

Depending on whether your Linux installation is using a “classical” System-V like init system (including upstart) or the more modern systemd, you will need to follow different instructions. Please note that systemd is the default in recent versions of Debian and RedHat-based systems.

HRM daemon with System-V-like init systems

Warning

Please make sure to only follow EITHER the instructions given in this chapter here for installing the daemon on System-V or upstart based systems, OR the ones for systemd, but NEVER BOTH.

If unsure, use the instructions here!

Install the init script

To launch the HRM daemon at boot, we are providing an init script in $HRM_RESRC/sysv-init-lsb/hrmd which can be placed in /etc/init.d/. The script makes use of the LSB init functions and therefore requires /lib/lsb/init-functions to be available in your distribution (which is true for Ubuntu and RHEL systems, as well as most other major up-to-date Linux distributions).

For installing the init script, type the following commands in a shell:

sudo cp -v $HRM_RESRC/sysv-init-lsb/hrmd /etc/init.d/hrmd
sudo chmod +x /etc/init.d/hrmd

Start / stop the daemon at boot / shutdown

Ubuntu: To set up the appropriate run-level links for starting and stopping the daemon during boot and shutdown, use this command:

sudo update-rc.d hrmd defaults

This will configure your system to run the init script in the default run-levels. Please note that issuing this command does NOT actually start the daemon until the next reboot. To start it right now follow the steps in the section below.

RHEL

Recent versions of RHEL-based sysmtems do not install the Linux Standard Base (LSB) specification by default, as they are using systemd init. Therefore we recommend using the systemd unit file, described in HRM daemon with systemd.

Note

If you want to use the classical init script approach nevertheless, please make sure the package redhat-lsb.x86_64 is installed, i.e. by running:

sudo dnf install redhat-lsb.x86_64

Start manually

Make sure to set the proper value of SUSER in /etc/hrm.conf (see Server).

Start a shell and type:

sudo /etc/init.d/hrmd start

If the queue manager started correctly, you should see:

Forking background process...
Reporting stdout to '/var/log/hrm/log.txt' and stderr to '/var/log/hrm/error_log.txt'.

To check if the queue manager daemon is running, use the status argument. In case the service is operational, it will show a message like this:

[ ok ] HRM is running (/var/www/hrm/bin/hrm_queuemanager), PID: 1234.

The queue manager can be started, stopped and restarted by using:

sudo /etc/init.d/hrmd start
sudo /etc/init.d/hrmd stop
sudo /etc/init.d/hrmd restart

HRM daemon with systemd

If your Linux distribution is using the systemd init system, we recommend to not using the “classical” init script (described in HRM daemon with System-V-like init systems). Instead, we are providing a unit file that can be used with systemd directly.

Warning

Please make sure to only follow EITHER the instructions given in this chapter here for installing the daemon on systemd based systems, OR the ones for System-V/upstart, but NEVER BOTH.

If unsure, or in case of problems with systemd, use the instructions for System-V!

Change the default configuration in the unit file

By default, the unit file is configured to run as the hrm user and to depend on mysql.service. If needed, you can change those values before you proceed with the installation.

[Unit]
Description=HRM (Huygens Remote Manager) Queue Manager Service
# For both 'Requires=' and 'After=', please set one of mysql.service,
# mariadb.service (fork of mysql), postgresql.service.
Requires=mysql.service
Wants=network-online.target
After=mysql.service network.target network-online.target

[Service]
# If needed, change 'User=' and 'Group=' to point to the correct values.
User=hrm
Group=hrm
ExecStart=/var/www/html/hrm/bin/hrm_queuemanager --detach
Type=forking
PIDFile=/var/log/hrm/hrmd.pid
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target

Please change the values of Requires=, After= to point to the correct database you are using for HRM (one of mariadb.service, mysql.service, postgresql.service) and User= and Group= if you are planning to run the Queue Manager as a different user (please see also set-up-the-hrm-user-and-group).

Install the unit file

To launch the HRM daemon as a systemd service, the unit file has to be copied into /etc/systemd/system/ and systemd has to be notified of the new unit. This is done with these commands:

sudo cp -v $HRM_RESRC/systemd/hrmd.service /etc/systemd/system/
sudo systemctl daemon-reload

Start / stop the daemon at boot / shutdown

To tell systemd to start the service at boot and stop it during shutdown, use this command:

sudo systemctl enable hrmd.service

Start / stop / check the HRM daemon manually

To start the daemon manually, type in a shell:

sudo systemctl start hrmd.service

Note that in case of a successful start, there will be no output. If the daemon did not start correctly, systemd will produce a number of output lines telling you about how to debug the problem.

To check if the queue manager daemon is running, use the status command.

sudo systemctl status hrmd.service

In case the service is operational, it will show a message like this:

* hrmd.service - HRM (Huygens Remote Manager) Queue Manager Service
   Loaded: loaded (/etc/systemd/system/hrmd.service; enabled; vendor preset: disabled)
   Active: active (running) since Thu 2016-02-11 15:05:22 CET; 18ms ago
  Process: 7782 ExecStart=/var/www/html/hrm/bin/hrm_queuemanager --detach (code=exited, status=0/SUCCESS)
 Main PID: 7785 (hrm_queuemanage)
   CGroup: /system.slice/hrmd.service
           |-7785 /bin/bash /var/www/html/hrm/bin/hrm_queuemanager
           |-7791 php -q /var/www/html/hrm/run/runHuygensRemoteManager.php
           `-7792 sleep 1

The queue manager can be started, stopped and restarted by using:

sudo systemctl start hrmd.service
sudo systemctl stop hrmd.service
sudo systemctl restart hrmd.service

Change the admin password!

The admin user can access the administration backend of the HRM with the username and default password specified in the $HRM_RESRC/passwd.README file.

Please, immediately go to the account page and change the admin password!

Optional configurations

Login page customization

You can customize the login page by renaming the $HRM_USER/login_user.inc.sample file into $HRM_USER/login_user.inc and adding information that you would like to appear on the HRM login page, such as contact information, quick instructions, etc.

Warning

Please use valid HTML code: no validation is performed on your $HRM_USER/login_user.inc HTML code, and you are responsible for any vulnerability you introduce into the HRM!

Example

<h2>Contact information</h2>
<ul>
    <li><a href="mailto:admin@facility.edu?Subject=Help!">HRM admin</a></li>
</ul>

<h2>HRM custom instructions</h2>
<ul>
    <li><a href="http://www.facility.edu/hrm_instr.pdf">Using the HRM at our facility</a></li>
</ul>

With a bit of HTML and CSS skills you can make your login page shine!

Export data share

The HRM offers data upload and download through the broswer. For large datasets, however, it is recommended to set up more performant ways to get data in and out of the HRM file server.

One approach consists of exporting the $HRM_DATA folder as a samba share. See for instance here on how to setup samba.

What follows is a sample configuration for exporting the $HRM_DATA folder (add it to the /etc/samba/smb.conf file):

[hrm_data]
    comment = HRM data folder
    path = /data/hrm_data
    force user = hrm
    force group = hrm
    writeable = Yes
    create mask = 0666
    directory mask = 0777
    guest ok = No
    strict locking = no

OMERO connector

For setting up the HRM-OMERO Connector please first follow the installation instructions from the project’s documentation.

Then enable the OMERO connector by setting the value of $omero_transfers to true in $HRM_CONFIG/hrm_config.inc.

<?php
...
//======================================================================
// HRM + OMERO
//======================================================================

// Switch on/off (true/false) data transfers between HRM and OMERO.
$omero_transfers = true;

?>

In addition, the following values need to be set accordingly in /etc/hrm.conf. By default, those settings are commented out, so make sure to remove the comment sign # in front of those lines!

Assuming your OMERO instance is running on host omero.mynetwork.xy and is using the default port 4064, this should look as shown below.

# Interaction with OMERO (if switched on in hrm/config).
OMERO_HOSTNAME="omero.mynetwork.xy"
OMERO_PORT="4064"

GPUs setup

Graphics processors significantly reduce computing time (see SVI page on GPU’s).

Configuration

Log on to HRM as admin and go to Servers and GPUs . The form in Servers and GPUs allows you to register GPUs (as well as other processing servers) in HRM for further parallelization of the HRM processing queue.

To add a new entry, enter the machine name and the absolute path of HuCore on that machine.

To register a machine with no GPU cards enter a negative integer for the GPU ID. Use, for example, GPU ID = -1.

To register a machine with one or more GPU cards simply add different entries, one per GPU. All these entries should have the same Server Name and HuCore Path, but different GPU IDs. As follows,

HRM will consider each GPU card as a separate server. The HRM queue will process as many images in parallel as the total number of registered GPU cards.

The GPU IDs can be retrieved via HuCore. For example, the following machine reports 2 GPUs with ID 0 (Titan X) and ID 1 (Quadro P5000):

> hucore
% huOpt gpu -query names
    {0: TITAN X (Pascal)} {1: Quadro P5000}

Performance considerations

Every time Huygens is launched NVIDIA checks to compile some Huygens code for each of the GPUs in the system. This can easily take about 20 or more seconds. Whenever possible NVIDIA caches the compilation results to avoid repeating this time-consuming process. Thus, in practice, if the cache works properly, new compilations only occur once, when a new NVIDIA driver is installed. That one time Huygens shows the message ‘Slow GPU initialization …’ while launching.

As it turns out the NVIDIA cache is user-dependent and is located in the ‘.nv’ folder of the user running the process. Therefore, it is important that the home folders of the 2 HRM system users (‘hrm’ and ‘apache’) have the proper ownership, so that this cache can be created. Otherwise, every single HRM job will incur in a significant performance penalty.

Thus, simply make sure the 2 following folders and ownerships are set properly:

• Frontend machine:

# www-data in Ubuntu, apache in |RHEL|
sudo chown -R www-data:www-data /var/www

Add 'export HOME=/var/www' after 'unset HOME' to '/etc/apache2/envvars'.

Restart Apache.

• Processing machine(s):

sudo mkdir -p /home/hrm
sudo chown -R hrm:hrm /home/hrm

As a reminder, also notice that each HRM job has a corresponding Huygens log in the HRM ‘Detailed view’. If the system is properly configured then the message ‘Slow GPU initialization’ should not be logged.

Distributed computing

This section describes the specific steps required to configure the HRM to run on multiple machines. In particular, we address the following setup:

• Machine 1: runs the web server (and database) and the queue manager

• Machine 2: runs Huygens Core

• The data folder is not directly shared (via NFS or Samba) between the two machines.

Note

There could be more than on processing machine running Huygens Core in your institution: set up all additional machines like Machine 2.

Public-key authentication

Image data for deconvolution is transferred from Machine 1 to Machine 2 via secure copy (scp). For this, a specified user on Machine 2 must be created/chosen, and remote access from Machine 1 via password-less, public key authentication must be set up for this user.

Note

For the sake of the example, we will call this specific user hrm. You can pick another user, just make sure to replace hrm with your user name in what follows.

On Machine 2, create a user with name hrm and group hrm that will be used for remote authentication and data transfer. Configure password-less, public key authentication as explained on this page. The public key is uploaded to Machine 2, whereas the private key is stored on Machine 1 for the selected user (e.g. in /home/hrm/.ssh/).

HRM configuration files

On Machine 1, configure the HRM as usual (as in the single-server configuration) but set the following variables in $HRM_CONFIG/hrm_config.inc as shown:

<?php
...
$copy_images_to_huygens_server = true;
$imageProcessingIsOnQueueManager = false;
$huygens_user = "hrm"   // This is the user that will log in to Machine 2
$huygens_group = "hrm"  // And this is its group

Replace hrm with your own user name and group if different.

Register processing machines

Log on to HRM as admin and go to Servers and GPUs . Register the processing servers in HRM by adding an entry for every processing machine, specifiying the absolute Huygens Core path on that machine, as well as the machine name. For details on the GPU ID field see LDAP authentication.

Data folder

On Machine 2, create the data folder pointed at by $huygens_server_image_folder. This folder should be owned by $huygens_user (e.g. “hrm”) and have group ownership as set by $huygens_group (e.g. “hrm”).

Automated installation (beta)

Note

This feature is still considered experimental but greatly simplifies the installation of an HRM website compared to the Manual installation (recommended) instructions. After a final cycle of testing, this method will become the new recommended way to install HRM.

Get the installation script

To get the latest release directly from our git repository, run the following in a shell:

$ git clone https://github.com/aarpon/hrm_installer

Basic operation

The most basic way to run the script to install HRM is to simply launch setup.sh as a priviledge user. In a shell, type the following:

$ sudo ./setup.sh

There are a few prerequisites to take care of before the script is run. In case these are not met, the script will display helpful error messages and abort. You will need:

• A supported OS. For this release, this means Ubuntu 20.04, Debian 11, Rocky Linux 8.5 or AlmaLinux 8.4 only.

• The script must be run as a priviledged user, either as root or using sudo.

• You will need an active Internet connection to clone the HRM repository.

• Hucore must be installed and a license available.

Further documentation

For now, the documentation is available on https://github.com/aarpon/hrm_installer and better instructions will be added here later.

Getting help

If you are interested in this installation method and have any issues you want to discuss, please enquire on the image.sc forum over at https://forum.image.sc/tag/svi-huygens

Note

Version 3.9 requires PHP 8.1 to run!

Note

These instructions always explain the steps required to upgrade from last stable release to current stable release. However, each section also provides links to instructions for older versions.

Version upgrade

If you have upgraded the HRM in the past, you will know that some steps must be performed in addition to replacing the old HRM code with the new one: some entries might have been added or changed in the configuration files (hrm_config.inc), and the database structure might have been changed.

Stop the Queue Manager

Significant parts of the configuration as well as the database are usually changed during an upgrade, so the Queue Manager needs to be stopped first.

Shut down the Queue Manager with:

sudo /etc/init.d/hrmd stop

if you are using System-V or upstart, or with:

sudo systemctl stop hrmd.service

if you are using systemd.

In some rare situations, the Queue Manager might get stuck. To ensure the stop command did work properly, run the following check.

On System-V or upstart:

sudo /etc/init.d/hrmd status

On systemd:

sudo systemctl status hrmd.service

Alternatively, you can use:

ps aux | grep -i [r]unHuygens

that should return empty (nothing).

Get the new HRM release

To install the new HRM version you need to download the .zip file from the website or github as explained in downloading the standard archive.

Warning

Please do not extract the new archive on top of the previous HRM installation! See details below.

Clean up previous installations

Because of changes in tree structure of the code and of the external dependences (starting from version 3.4), we highly recommend not to extract the new archive on top of the old one.

Please rename the old hrm folder, extract the code into a fresh ${HRM_ROOT} and move the configuration files from the old config subfolder into the new ${HRM_ROOT}/config.

You might also want to reinstall the hrmd or hrmd.service scripts.

Note

Please follow these instructions first if you are upgrading from older versions.

Check the configuration file

An easy way to check for modifications is by running the $HRM_HOME/resources/checkConfig.php script. From the shell, run:

cd $HRM_HOME
php resources/checkConfig.php config/hrm_config.inc

There were no configuration changes between versions 3.8 and 3.9 of HRM. The output of the checkConfig.php script should be:

Checking against HRM v3.9.
Check completed successfully! Your configuration file is valid!

Please make sure to fix all problems you might have! The sample files and the Manual installation (recommended) instructions will help you set the correct parameters.

Note

Please follow these instructions first if you are upgrading from older versions.

Update the database

Newer versions of the HRM might use slightly different/updated versions of the database back-end than previous ones.

HRM version	Database version
3.9	20
3.8	19
3.7	18
3.6	17
3.5	16
3.4	15
3.3	14
3.2	13
3.1	12
3.0.3	11
3.0	10
2.1	9
2.0	8
1.2.3	7

For this reason, the first time you run the HRM after an update you will be told that the database must be updated and that you are not allowed to continue until this has been done!

Note

Database updates are supported across HRM versions, i.e. it is possible to upgrade the database from revision 7 to 20 in one step.

The following describes two possible ways to update the database.

Note

Although we test this procedure quite carefully, it is highly recommended to backup the database before updating!

Updating from the web interface

Login to the HRM as the admin user: you will be brought directly to the Database update page. Click on the update button. If everything works properly (as it should…), the following message should be displayed.

Needed database revision for HRM v3.9 is number 20.
Current database revision is number 19.
Updating...

Database successfully updated to revision 20.

The database is now at the latest revision.

Updating from the console

Alternatively, the database can be updated from the console (see create or update database). Please pay attention to what the update process will report! The output should be the same as the one listed in the previous section, but if the update fails, you might want to report it.

Check your HRM-OMERO connector

In case you are using the HRM-OMERO connector, you will need to update that one as well as it has been split into a separate project. Please refer to the instructions on the HRM-OMERO project page for details.

Re-start the Queue Manager

After processing the described upgrade steps, the Queue Manager needs to be started again, with:

sudo /etc/init.d/hrmd start

if you are using System-V or upstart, or with:

sudo systemctl start hrmd.service

if you are using systemd.

Upgrade from previous releases

The following pages are linked to from the relevant sections above, but are listed here again for convenience.

Clean up previous installations

Warning

These instructions refer to older versions of HRM!

3.4 and newer

It is highly recommended not to extract the new archive on top of the old one.

Please rename the old hrm folder, extract the code into a fresh ${HRM_ROOT} and move the configuration files from the old config subfolder into the new ${HRM_ROOT}/config.

You might also want to reinstall the hrmd or hrmd.service scripts.

3.3 to 3.4

In version 3.4, we started a major reorganization of the code structure of HRM and it is therefore highly recommended not to extract the new archive on top of the old one.

Please rename the old hrm folder, extract the code into a fresh ${HRM_ROOT} and move the configuration files from the old config subfolder into the new ${HRM_ROOT}/config.

You might also want to reinstall the hrmd or hrmd.service scripts.

3.2 to 3.3

HRM 3.3 uses new init scripts. Please delete the old files $HRM_BIN/hrm_user_manager, $HRM_BIN/hrm_user_manager_old, $HRM_BIN/hrmd and $HRM_BIN/ome_hrm and then follow the instructions in Upgrade the init script.

Upgrade the init script

Note

If your Linux installation is using the systemd init system, please have a look at the instructions about how to set up the HRM daemon with systemd. Please make sure to remove the old init script at /etc/init.d/hrmd in this case!

Otherwise, proceed as described here for the init script.

This step is basically identical to the initial installation of the init script as described in installing the daemon. You need to copy the new script from $HRM_RESRC/sysv-init-lsb/hrmd to /etc/init.d/ and make sure it is executable. This can be done using the following commands:

sudo cp -v $HRM_RESRC/sysv-init-lsb/hrmd /etc/init.d/hrmd
sudo chmod +x /etc/init.d/hrmd

Update the data folder permissions

As of HRM 3.2.0, the system users running the Queue Manager and the web server are expected to have full read-write access to $HRM_DATA. The supported way of doing this is explained in setting up the HRM user and group. Briefly:

• a user hrm and its corresponding group hrm are created

• the web server user ( www-data, RHEL apache) is added to the hrm group

• the variable SUSER is set to hrm in /etc/hrm.conf

• the $HRM_DATA and $HRM_LOG group ownership is set to hrm with the setgid bit set

For detailed instructions, please see setting up the HRM user and group.

Warning

With HRM 3.2 it was possible to preserve the old behavior by setting a configuration variable $change_ownership. This is not supported with HRM 3.3 any more!

3.1 to 3.2

rm -v inc/ActiveDirectory.inc.php inc/Ldap.inc.php

Update configuration files

Warning

These instructions refer to older versions of HRM!

3.7 to 3.8

HRM 3.8 brings a few simplifications in the configuration files. Please run the following commands:

cd $HRM_HOME/config
cp -p hrm_server_config.inc hrm_config.inc

Note

The following files have become redundant and can be removed:

$HRM_HOME/config/hrm_server_config.inc

$HRM_HOME/config/hrm_client_config.inc

If you are upgrading from an older version of HRM and both files had the same content, you can simply rename one of them to hrm_config.inc.

3.6 to 3.7

There were no changes in the configuration files between versions 3.6 and 3.7 of HRM.

3.5 to 3.6

There were no changes in the configuration files between versions 3.5 and 3.6 of HRM.

3.4 to 3.5

There were no changes in the configuration files between versions 3.4 and 3.5 of HRM.

3.3 to 3.4

$authenticateAgainst

The support for various authentication mechanisms was extended in HRM 3.4. This comes with a change in configuration: the $authenticateAgainst variable is now an array and its values have also changed (although, temporarily, the old ones are still supported). Example:

<?php
...
$authenticateAgainst = array("active_dir", "integrated");
...

$useDESEncryption

The configuration variable $useDESEncryption (that was not used) must be removed from the configuration files!

Note

Please follow these instructions first if you are upgrading from older versions.

3.2 to 3.3

The $change_ownership option is not supported in version 3.3 any more. If the variable is present in the configuration files hrm_{server|client}_config.inc it will be silently ignored. For consistency reasons it is therefore recommend to remove this setting from the config file(s).

Group authentication

As of HRM 3.3, external authentication via Active Directory or generic LDAP now supports group authorization. An additional array $AUTHORIZED_GROUPS can be set to define the set of Active Directory or generic LDAP groups that are granted access to HRM.

3.1 to 3.2

As of version 3.2 of HRM, the system users running the Queue Manager and the web server are expected to have direct read-write access to the data folders. If this is not the case for your setup and you rely on adding the web server user to /etc/sudoers, please notice that this behavior is deprecated in 3.2 but can still be enabled by adding:

$change_ownership=true;

in hrm_{server|client}_config.inc. The variable $change_ownership defaults to false if not explicitly set to true in the configuration, in compliance to the new behavior.

Note

As of HRM 3.3, this variable will be ignored and the new behavior will be enforced!

3.0.x to 3.1

No changes in the configuration files.

2.1.x to 3.0

From HRM 2.1.x to HRM 3.0, one variable was added in the configuration files.

$omero_transfers={true|false}

1.2.x to 2.0

From HRM 1.2.x to HRM 2.0, three variables were added in the configuration files:

[+] max_upload_limit
[+] max_post_limit
[+] email_list_separator

Moreover, three variables were removed:

[-] resultImagesOwnedByUser
[-] resultImagesRenamed
[-] enable_code_for_huygens

Note

If you are upgrading straight from HRM 1.2.x, please notice that as of HRM 2.0 configuration and sample files were moved as per following table.

Config files (new)	Sample files (new)	Config files (1.x)	Sample files (1.x)
$HRM_HOME/config	$HRM_HOME/config/samples	$HRM_HOME/inc	$HRM_HOME/resources

Check configuration files

Warning

These instructions refer to older versions of HRM!

An easy way to check for modifications is by running the $HRM_HOME/resources/checkConfig.php script. From the shell, run:

cd $HRM_HOME
php resources/checkConfig.php config/hrm_config.inc

3.7 to 3.8

Checking the 3.7 files with the 3.8.x checkConfig.php script will result in the following output:

Checking against HRM v3.8.
Check completed successfully! Your configuration file is valid!

3.6 to 3.7

Checking the 3.6 files with the 3.7.x checkConfig.php script will result in the following output:

Checking against HRM v3.7.
Check completed successfully! Your configuration file is valid!

3.5.x to 3.6

Checking the 3.5 files with the 3.6.x checkConfig.php script will result in the following output:

Checking against HRM v3.6.x.

* * * Error: variable default_output_format not set or empty.
* * * Error: variable min_free_mem_launch_requirement not set or empty.
Check completed with errors! Please fix your configuration!

3.4.x to 3.5

Checking the 3.4 files with the 3.5.x checkConfig.php script will result in the following output:

Check against HRM v3.5.x.
Check completed successfully! Your configuration file is valid!

3.3.x to 3.4

Checking the 3.3 files with the 3.4.x checkConfig.php script will result in the following output:

Check against HRM v3.4.x.
* * * Error: variable useDESEncryption must be removed from the configuration files!
* * * Error: variable 'authenticateAgainst' must be an array!
* * * Moreover, please change 'MYSQL' into 'integrated'.
Check completed with errors! Please fix your configuration!

3.2.x to 3.3

Checking the 3.2.x files with the 3.3 checkConfig.php script will result in the following output:

Check against HRM v3.3.x.
* * * Error: variable change_ownership must be removed from the configuration files!
Check completed with errors! Please fix your configuration!

3.1.x to 3.2.x

Checking the 3.1.x files with the 3.2.x checkConfig.php script will result in the following output:

Check against HRM v3.2.x.
Check completed successfully! Your configuration file is valid!

3.0.x to 3.1.x

Checking the 3.0.x files with the 3.1.x checkConfig.php script will result in the following output:

Check against HRM v3.1.x.
Check completed successfully! Your configuration file is valid!

2.1.x to 3.0.x

Checking the 2.1.x files with the 3.0 checkConfig.php script will result in the following output:

Check against HRM v3.0.x.
* * * Error: variable omero_transfers not set or empty.
Check completed with errors! Please fix your configuration!

1.2.x to 2.1.x

Checking the 1.2.x files with the 2.1.x checkConfig.php script will result in the following output:

Check against HRM v2.1.x.
* * * Error: variable max_upload_limit not set or empty.
* * * Error: variable max_post_limit not set or empty.
* * * Error: variable email_list_separator not set or empty.
* * * Error: variable resultImagesOwnedByUser must be removed from the configuration files!
* * * Error: variable resultImagesRenamed must be removed from the configuration files!
* * * Error: variable enable_code_for_huygens must be removed from the configuration files!
Check completed with errors! Please fix your configuration!

Please make sure to fix all problems. The sample files and the Manual installation (recommended) instructions will help you set the correct parameters.

User manual

These are the detailed instructions for the Huygens Remote Manager User Manual.

Introduction

The HRM is a collaborative open-source interface to Huygens Core for multi-user, web-based, scheduled, batch deconvolution.

The Huygens Remote Manager (HRM) enables large groups of users to work with Huygens on a central system through an easy-to-use interface. HRM is web-based and therefore easily accessible from your workstation, home PC, tablet or smart phone.

An HRM server can distribute work among several processing machines smartly, which allows for intelligent use of hardware and fast processing times. It features an intelligent queue that switches among the jobs of all users.

Who makes HRM

HRM is an open source project, this means that the code is freely accessible and it can be adapted and modified to any needs by any user.

HRM is a joined collaboration of Huygens users from:

• Montpellier RIO Imaging,

• Facility for Advanced Imaging and Microscopy at the Friedrich Miescher Institute (FMI, Basel),

• The BioImaging and Optics Platform at the Ecole Polytechnique Fédérale de Lausanne,

• The Department of Biosystems Science and Engineering at the ETH Zürich,

• The Leibniz Institute for Neurobiology (Magdeburg),

• The Combinatorial Neuroimaging facility (Magdeburg),

• The Imaging Core Facility at the Biozentrum (University of Basel).

• The University of Fribourg.

• The Microscopy and Image Analysis Platform of the University of Freiburg.

• The University of Manchester.

Scientific Volume Imaging participates in this project by contributing its experience in deconvolution and software engineering.

Where to find HRM

HRM is a free and open source project, and can be found on http://huygens-rm.org. The source code is hosted on https://github.com/aarpon/hrm.

More information about HRM and links to other HRM resources can be found in this online article of the SVI Wiki. Instructions for online testing, downloading and installing HRM are also linked on that page.

Installing HRM on a regular web server is not very difficult. Please consult the Installation instructions and the SVI Wiki.

Getting started

Here is a quick guide to get you started with the HRM/HuCore deconvolution. In order to optimize results, please refer to section Deconvolution with the HRM.

Select UI theme in HRM

HRM offers two UI themes (dark, light) that can be selected from the login page.

The theme selected at the moment of logging in is preserved during the entire session. In this manual screenshots from both themes are displayed interchangeably.

Create an account in HRM

• If your imaging facility already offers deconvolution via HRM, proceed to the facility’s HRM login page.

• If HRM is not available at your imaging facility you can still familiarize with HRM by using the HRM demo server offered by SVI.

Follow the registration link on the HRM login page and complete the short form. Names are case sensitive. Upon registration the new HRM account needs to be validated by the administrator.

Upload a raw image

Upon successful logon the HRM Home panel will be displayed:

To upload images click on the Raw images icon . The following, intuitive upload tool allows for multiple uploads from your local device:

Deconvolve

To submit a deconvolution job proceed to Launch jobs from the home panel. The process of submitting the job is split into the following 5 steps:

• Step 1/5 - Select images: click on the “Image File Format” drop down widget to select the file type of the images to deconvolve. All uploaded files of that format will belisted. Use the icons to add/remove images to/from the selection. When selecting a file, a preview can be generated by using the link on the right hand side panel. The image preview also contains information about the dimensions, sampling and number of channels.

• Step 2/5 - Image parameters: A parameter template describing the imaging conditions of acquisition system. The template can be created from scratch or derived from an existing one. Templates shared by other HRM users (including the administrator) can also be used. Choosing the correct parameters is absolutely crucial for obtaining high-quality deconvolution results.

  Use an existing parameter template (easy): When selecting an existing parameter template the contents are displayed in the panel on the right hand side. Make sure that the microscope type and the number of channels of the template match the images selected in Step 1/5.

  Create a new parameter template (elaborated): Click the New template button . Type a name for the new template and click ‘Create’. The template editor opens, select the correct number of channels and PSF type. Move on to the optical parameters and select the correct microscope type. For a detailed explanation on the remaining parameters please refer to section Deconvolution with the HRM.

• Step 3/5 - Restoration parameters: As in the previous step it’s possible to choose between an existing template or creating a new one.

  Use existing parameter template (easy): Select a parameter template and verify its contents on the preview shown on the right hand side panel. Particularly, pay attention to the value of the Signal to Noise ratio (SNR) (see Signal to Noise ratio for further help).

  Create a new parameter template (elaborated): Click the New template button . Type a name for the new template and click ‘Create’. Once in the template editor select a deconvolution algorithm (CMLE is a good choice) and an SNR. Follow the instructions of the SNR estimator as a guide for the SNR of the selected images. Set the background mode to automatic, the number of iterations to 40 and the quality change to 0.01. For more detailed fine-tuning follow the instructions at Deconvolution with the HRM.

• Step 4/5 - Analysis parameters: This step is only active when 2 or more channels have been specified in the previous steps. When active, the Analysis step allows the user to execute Colocalization analysis on the deconvolved images.

  Use existing parameter template (easy): Select the analysis parameter template of your choice and verify that the values displayed on the preview on the right hand side panel match your analysis needs.

  Create a new parameter template (elaborated): Click the New template button . Type a name for the new template and click ‘Create’. Select whether colocalization should be enabled. For a colocalization run select the channels to be analyzed as well as the colocalization coefficients of interest. Set the threshold to “Automatic estimation” and the Colocalization map to “Pearson”. In order to further fine-tune the Colocalization parameters please refer to Deconvolution with the HRM.

• Step 5/5 - Launch job: Select the output file format for the restored images. ICS and HDF5 are the recommended formats because they provide good dynamic range and the necessary meta data infrastructure to save all the image parameters. Plus, they allow for good compression levels that minimize the amount of disk space needed for storage. Next, review the parameter summaries as well as the image selection. If all the settings are correct then proceed to submit the job to the server by clicking .

Queue

After submitting the job HRM returns to the Home panel. Access the queue by clicking . A typical processing queue looks like the following:

The queue is handled in an intelligent way, allowing jobs from different users to be processed at different times.

Results

When a deconvolution and analysis job is finished a notification email is sent with a link to the result. Alternatively, go to the HRM home panel and click on the Results icon . The contents of the Results folder will be listed. Select the image that you would like to visualize and a preview will be displayed on the right hand side panel:

For a detailed view of both the raw image and the restored image click on .

A results page opens showing MIP, SFP and slicer comparisons of the raw image versus the deconvolved image. Summary tables show information about possible scaling factors, colocalization results, image parameters and deconvolution parameters. Also, movies along the Z direction and/or through time will be available for download.

Tips & Tricks

• When a deconvolution run gives optimal results download the processed data from the detailed view by clicking on .

• Click at steps 2, 3 or 4 to share any of the templates with one or more colleagues.

• To optimize the deconvolution results it is necessary to understand the different properties of the image and some of the mechanisms behind deconvolution. This introductory chapter skipped details that can make a difference on images with spherical aberration, extreme SNR values, undersampling, etc. For a detailed explanation please see Deconvolution with the HRM.

Deconvolution with the HRM

Home panel

Once logged in HRM, the home panel is displayed:

From this page the user can manage deconvolution jobs as well as other HRM administrative tasks. The following shortcuts are available:

• Launch jobs : Start new jobs for deconvolution, stabilization, chromatic aberration correction, colocalization, etc.

• Queue status : See all jobs. Manage owned jobs.

• Raw images: Upload new data for deconvolution.

• Results: View and download deconvolved data.

• Statistics: Summary of job statistics.

• Account: View and change login data.

Launch a job

To start a new job with deconvolution, chromatic aberration, stabilization, or colocalization analysis click on Launch jobs in the home panel. This page describes the general workflow which is split into the following five main steps. The detailed description of all the parameters included by the templates used in this workflow is documented on the following sub-pages:

Image template

An image parameter template consists of several microscopic pararameters relevant for deconvolution. To get a preview of the template contents select the template, the contents will be displayed on the right panel.

Note

When editing a template help links are displayed at key locations to point you to specific articles of the SVI-wiki.

Number of channels

Number of fluorescent channels in the image.

Note

Transmission channels can NOT be deconvolved with the algorithms currently available in HRM and should be skipped!

PSF Type

To perform image deconvolution a Point Spread Function (PSF) is needed.

The Huygens software can compute a theoretical PSF from the image parameters used during deconvolution or it can use a measured PSF.

Depending on the user choice HRM will, at a later stage, ask for:

• the path to the file containing the PSF if you selected Measured,

• the settings to correct for spherical aberration if you selected Theoretical and there is a refractive index mismatch.

In most cases a theoretical PSF works fine.

Microscope type

Choose among the supported microscope types: widefield, confocal, spinning disk, multiphoton, STED, STED 3D, SPIM, Rescan, and Array detector (Airyscan). Depending on the Huygens license some of these options might not show.

Note

HRM requires the microscope type as input from the user. HRM will NOT load the microscope type from the selected images automatically.

Import metadata

Some parameters are always present in the meta data of specific file formats and can be imported from the image at run time, while deconvolving the image, to save you time.

Note

When a parameter is missing from the meta data HRM needs input from the user. This is highlighted by a message like this:

Note

When a parameter is present in the meta data HRM can work without the input from the user. This is indicated with

At the end of the deconvolution job, HRM informs on the values of all used parameters as well as their origin (user input or meta data).

Numerical aperture

The Numerical Aperture (NA) describes the amount of light coming from the focus that the objective can collect.

The NA depends on the half angle of the maximum cone of light that can enter or exit the lens. It is directly linked to the resolution of the objective.

Usually, the NA can be found engraved on the objective next to the magnification.

Note

Depending on the selected file format this parameter may be skipped, as explained in Import metadata.

Wavelengths

Excitation and emission wavelengths of each channel. For the emission wavelength the central value of the emission spectrum of the fluorophore can be considered.

Note

Make sure to insert these values in the same order as they were acquired.

Objective Type

Whether a dry (air) or immersion objective (water, glycerol, oil) was used. For other immersion types it is possible to enter their refractive indexes directly.

Note

Several file formats report this value in the meta data. To leave empty please click on .

Sample medium

Whether glycerol, polyvinyl alcohol, vectashield or water was used to embed the sample.

For other embedding media it is also possible to specify their refractive indexes directly.

Note

Few file formats report this value in the meta data. To leave empty please click on .

Voxel Size

The voxel size or sampling size is a very important parameter in image deconvolution.

Note

According to the Nyquist criterion its value should not be larger than half the optical resolution of the imaging system.

HRM offers several utilities to help you set up this parameter correctly.

Ideally, the optimal voxel sizes are calculated before acquiring the images. By doing so, one can ensure that the images are compliant with the Nyquist criterion. Click on to determine how large the voxel sizes should be. By using these values, there will be no lost information during the image acquisition and the deconvolution will be able to restore the original signal better.

If the optimal voxel sizes have not been determined before the acquisition stage one can calculate the resulting sizes in widefield and spinning disk microscopy by using . Here the voxel sizes are calculated from the CCD camera element, the objective magnification, etc., although no optimal voxel size can be guaranteed.

Note

Make sure to set a voxel size consistent with the Nyquist criterion. Undersampled images will often show artifacts after deconvolution.

Lastly, for deconvolving time series set the time interval between subsequent frames.

Among other things, this will help HRM to decide when to stabilize the image.

Using a measured PSF

This option will only appear if ‘measured PSF’ was selected as PSF Type in a previous step.

Distilled PSFs need to be created beforehand using Huygens Professional and uploaded to the Raw images folder.

Note

One PSF image per channel. Multichannel PSFs are not supported.

When presented with the Distilled PSF file selection window,

browse for the distilled PSF appropriate for the channel(s) of the image.

Select the PSF file and click close. Repeat for all channels.

Afterwards, press save and return to the image parameter selection page.

Note

Files that can’t work as PSFs for the selected image(s) are crossed off and displayed in red.

Pinhole Radius

Pinholes are used in confocal, spinning disk, STED, STED 3D and Rescan microscopy to get rid of out-of-focus light. Since there’s no pinhole in widefield, two photon and SPIM microscopes HRM skips this parameter in those cases.

Note

HRM needs the value of the pinhole backprojected on the specimen.

Further help on how to calculate the backprojected pinhole radius can be found on .

Pinhole Spacing

Spinning disk microscopy uses more than one pinhole to speed up image acquisition. The backprojected distance between these pinholes also plays a role in image deconvolution. HRM skips this parameter for all other microscope types.

Click on for further help on the calculation of the backprojected value.

STED parameters

If the selected microscope type is STED or STED 3D then a few STED parameters need to be specified to describe the STED acquisition. When working with other microscope types HRM will skip these parameters.

First, select the STED depletion mode: Pulsed, Continous wave gated detection, Continous wave non gated detection or Off/Confocal.

Note

STED and confocal microscopy are often combined. That’s why even if the selected microscope type is STED it’s still possible to set specific channels as confocal at this point.

Set the STED saturation factor of each channel. Typical values are in the range of 10 to 50. The higher this factor, the more suppression of fluorescence away from the optical axis, the more resolution.

Next, set the STED depletion wavelength of each channel. Notice that Channel 1 is grayed out in these figures because it was set as confocal.

The STED immunity fraction tells Huygens the percentage of fluorescent molecules that are not susceptible to the STED laser. Typical values are in the range 0% to 10%.

Finally, depending on whether the microscope type is STED 3D or not, set the percentage of STED 3D per channel. This tells Huygens how much power was used in the Z depletion beam. The remaining power is used for the vortex path.

SPIM parameters

If the selected microscope type is SPIM then a few SPIM parameters need to be specified to describe the SPIM acquisition. When working with other microscope types HRM will skip these parameters.

First, select the SPIM excitation mode of each channel: Gaussian light sheet, Gaussian MultiView light sheet, High fill factor (cylinder), or High fill factor (scanning beam).

Set the width of the Gaussian light sheet for the channels acquired with one of the Gaussian modes. The field will be grayed out for the other channels.

The width is defined as the distance between the two (symmetric) points of the Gaussian profile where the intensity values drop to 2/e^2 of the peak value.

Set the SPIM Focus Offset of the light sheet. This is the distance along the optical axis of the detection lens between the focus of the illumination lens and the focus of the detection lens.

Similarly, set the SPIM Center Offset of the light sheet. This is the distance along the optical axis of the illumination lens between the focus of the illumination lens and the focus of the detection lens.

Set the SPIM NA for the channels aquired with the one of the High Fill Factor modes. This is the Numerical Aperture of the illumination lens.

Then, the fill factor of the illumination lens, for the channels acquired with one of the High Fill Factor modes. This is the ratio between the beam width and the diameter of the objective pupil in the illumination lens.

Lastly, the SPIM Illumination Direction. This can be left, right, bottom, top for any of the non-MuVi (Multi-View) excitation modes; and right+left or top+bottom for the Gaussian MuVi excitation mode.

Spherical Aberration correction

Spherical aberration is mainly caused by a mismatch between the refractive indexes of the objective immersion medium and the sample embedding medium.

HRM will automatically detect this mismatch and ask whether the aberration should be corrected for when using a theoretical PSF.

Note

Spherical aberration can be corrected for by creating different theoretical PSFs (with slightly changing shapes) at different depths: depth-dependent PSF.

When using a depth-dependent PSF one also needs to specify whether the first plane in the dataset is closest or farthest to/from the coverslip.

The Correction mode allows for a more detailed choice as to how the depth-dependent PSF should vary with depth.

• Automatic correction: will generate a depth-dependent PSF where the PSF is automatically selected at each depth, as explained above.

• Advanced correction: allows to specify further details about the depth-dependent PSF.

The Advanced correction will further enable a number of advanced options.

• Depth-dependent PSF on few bricks: a different PSF will be generated for each Z brick in which the original data is split.

• Depth-dependent PSF slice by slice: a different PSF will be generated for each slice.

• Depth-dependent PSF on few slabs: several PSFs will be generated for each Z brick. This option is memory-consuming but the best choice when the spherical aberration is very strong.

Note

Splitting the original data into bricks while deconvolving not only helps to correct for spherical aberration but it also minimizes the amount of RAM memory needed for deconvolution.

Restoration template

A restoration parameter template consists of a number of options which provide Huygens Core with information on the restoration procedure. To get a preview of the template contents select the template, the contents will be displayed on the right panel.

Note

When editing a template help links are displayed at key locations to point you to specific articles of the SVI-wiki.

Deconvolution algorithm

From HRM 3.7 onwards it is possible to select one deconvolution algorithm per image channel. In most cases this will not be necessary and the user will likely find it convenient to deconvolve all the image channels with the same deconvolution algorithm.

For each channel choose one of the deconvolution algorithms from the corresponding drop down widget.

The available choices are:

• Classic Maximum Likelihood Estimation (CMLE): CMLE is the method of choice under most circumstances. In case of doubt, CMLE should be used.

• Quick Maximum Likelihood Estimation (QMLE): QMLE is faster than CMLE, but gives slightly less precise solutions in some cases. One may consider using QMLE in compute-intensive situations, for example, when deconvolving widefield 3D-time series.

• Good’s roughness Maximum Likelihood Estimation (GMLE): GMLE is a good alternative for high-noise images, such as STED or confocal.

• Skip : for not deconvolving this channel and keeping the raw data instead.

Note

Skip all channels of an image to use HRM as a pure batch file converter, chromatic aberration corrector, time stabilizer, colocalization analyzer, etc.

Signal to Noise ratio

In the Huygens Software the Signal-to-Noise ratio (SNR) is treated as a regularization parameter, i.e., a means to control the sharpness of the deconvolution result.

Yet, please be aware of the effect of extreme SNR values on image deconvolution: too high SNR values can produce restoration artifacts; too low SNR values can lead to smooth structures.

Note

After a little practice it is possible to assess the SNR range for deconvolution visually. In general, the SNR should increase with the quality and amount of signal of the raw data.

There are different acquisition settings that have an impact on the quality of the recorded signal. Changing these in the acquisition system will most likely require a modification of the SNR value for deconvolution. These acquisition settings are:

• Gain / offset

• Time exposure / scanning velocity

• Summing / averaging

• Laser power

• Spectral detection range

Because these settings are typically different for each channel HRM will require an SNR value per channel, like this:

As a guide, these are typical SNR values for different microscopy tecniques:

• Confocal: between 20 and 40.

• STED: between 10 and 20.

• Widefield: between 40 and 60.

HRM offers an automatic SNR estimator to determine suitable SNR values for the raw images. Click on to use the SNR estimator.

In a new page you will be asked to select an SNR estimation method.

The Auto method gets good estimations on images without a baseline. When working with images that contain a baseline the Legacy method can be used instead.

Next, select an image representative of the batch selected in Select images (1/5). Press for the SNR estimation:

The output of the SNR estimator shows a region of the original data next to 4 different noise simulations of the same region. In bold letters the SNR of the simulation that most resembles the original image.

By pressing the estimated SNR values are copied into the restoration parameter template.

Acuity

As of HRM 3.8 the acuity parameter controls the sharpness of the result.

In older Huygens and HRM versions this could be controlled by setting lower or higher than expected Signal to Noise ratio values for the image.

If you would like to keep working like in the older versions simply set the acuity mode to legacy:

For integrating the acuity parameter in your workflows and to leave the legacy mode behind enable the acuity mode and set acuity values for each channel. The accepted values range between -100 and +100.

• Positive acuity: the more positive the acuity, the sharper the result and the less noise suppression (or potentially noise amplification).

• Negative acuity: the more negative the acuity, the more noise suppression and the less sharpness.

Crop image

HRM can intelligently crop your images without losing important data.

Select from:

• Apply conservative crop: to save computation time.

• Do not crop the image: if computation time is not an issue.

Note

The time needed to deconvolve an image increases more than proportionally with the image volume. Thus, cropping the image will speed up its restoration and the processing of the HRM queue.

Bleaching Correction

Huygens can automatically estimate the amount of bleaching in the image and correct for it. This type of correction is mostly necessary for 2 types of images:

• Images from widefield-like systems. Since the specimen is exposed to large amounts of light it can be easily bleached.

• Time series from any type of microscope. Depending on how long the time series is the specimen can also be exposed to large amounts of excitation light.

Note

2D/3D images froms confocal-like systems (no time series) don’t need to be corrected for bleaching.

When presented with the Bleaching correction option, it is recommended to select

for any time series and/or widefield-like images.

Select

for all of the other images.

Stabilization in Z

This option will only be shown when restoring STED images (including STED 3D). Due to the high lateral resolution in STED, image acquisition in more sensitive to drift in STED than in other microscopy modes.

Note

It is strongly recommended to select Z stabilization on STED images.

The stabilization is performed before deconvolution. This significantly improves the restoration results.

Z stabilization is also recommended on STED 3D images although Huygens can automatically skip it depending on the PSF shape of each particular case.

Array Detector Reduction Mode

This option will only be shown when restoring Array Detector images (Airyscan, SPAD, etc). Array Detector images contain one acquisition per detector. The user can tell Huygens how to combine the data from all the detectors for deconvolution.

The following options are possible:

• All: the images from all detectors are centered and combined, thus increasing the SNR. The resulting, combined image is then deconvolved.

• No: the independent detectors are not combined prior to deconvolution. They are used as separate inputs of a ‘parallel’ deconvolution.

• Core all: Same as All but taking into account the central detectors only.

• Core no: Same as No but taking into account the central detectors only.

• SuperY: Create an image with 4 times as many samples in Y. This is mostly useful when dealing with images recorded with the Zeiss Airyscan in fast mode.

• SuperXY: Create an image with double the samples in X and Y. Thus, producing an image with 4 times more samples. This is a good option when dealing with images that have been acquired well under the Nyquist sampling rate.

• Auto: Automatic selection of one of the above mentioned modes depending on the microscopic parameters and detector model of the image.

In all cases the SNR value specified in Signal to Noise ratio is for the central detector. Huygens will scale the SNR for the combined detectors automatically.

Background mode

The background intensity is a nearly constant value that is present everywhere in the image.

Three options are available for background correction. These options can return slightly different values, therefore this choice can affect the deconvolution result:

• Automatic background estimation: This estimation usually works well. The background is estimated within a region with a low mean value.

• In/near object: The background is estimated around intensity peaks. This option can be interesting, for example, when having bright little objects in a cell with a strong cytoplasmic background.

• Remove constant absolute value: To make sure that the same background level is removed from all the images in the batch, insert manually a measured mean background for each channel. This option is typically useful for those interested in doing fluorescence quantification or stitching.

Stopping criteria

The restoration algorithms offered in HRM, CMLE, QMLE, and GMLE, are iterative methods. This means that the algorithms compute sequential solutions which converge to a stable result.

Deconvolution will stop when either of the following two conditions is met.

• Number of iterations: sets the maximum number of iterations that Huygens will compute.

• Quality change: how much the results of two consecutive iterations differ. If two subsequent results differ less than this factor then convergence has been reached.

As of HRM 3.9 it is possible to set the stopping criteria seperately for each channel. Leaving later channels empty will use the last set value.

Chromatic Aberration Correction

This option will only be shown when restoring multi channel images. In order to let HRM know that an image batch contains multi channel images please set the number of channels correctly, as explained in Number of channels.

The restoration template editor contains a section for post-deconvolution operations. Options for these operations (chromatic aberration correction and stabilization of time series) only show up when applicable, i.e, multichannel images and time series.

Therefore, the following section will only be displayed when restoring multi channel images:

Choose a reference channel:

The chromatic aberration can be described by an x-y-z shift + rotation + scale (zoom) of each channel with respect to the reference channel. Therefore, by definition the reference channel has no x-y-z shift, rotation or scale. For the other channels, type in the shifts, rotations and scales.

The table will contain one entry per channel.

For an accurate estimation of the chromatic aberration one can use a local installation of Huygens Professional or Huygens Essential, see Chromatic Aberration Corrector.

Use the estimated x-y-z shifts, rotations and scale values reported by Huygens Professional or Essential to fill out the table. All images recorded with the same settings and the same microscope can be corrected with the same chromatic aberration values.

As of Huygens 21.10.1 the Chromatic Aberration correction has an option for higher order corrections (e.g.: barrel and pincushion distortions). The description of these deformations can be imported into HRM from the corresponding Huygens template. However, due to the complexity of the parameters they can’t be either edited or viewed in HRM.

HRM 3.8 and higher will let you choose between the higher-order correction and the 5 parameter correction (x,y,z, rotation, scale) when importing templates from Huygens. Please do keep in mind that the extra parameters will be handled under the hood and are not displayed in the HRM UI.

An option to edit the shifts, rotation and scale while discarding the higher-order corrections from a Huygens template will appear on importing, as follows:

Note

For skipping the chromatic aberration correction simply leave the table fields empty.

Stabilization of time series

This option will only be shown when restoring time series. In order to let HRM know that an image batch contains time series please set the time interval correctly, as explained in Voxel Size.

The restoration template editor contains a section for post-deconvolution operations. Options for these operations (chromatic aberration correction and stabilization of time series) only show up when applicable, i.e, multichannel images and time series.

Therefore, the following option will only be displayed when restoring a time series:

Set a stabilization method:

• Center of mass: works best if the image contains a single large object. No objects should cross the image borders.

• Correlation: for general x-y-z translations and axial rotations. Adjacent time frames will be compared. The software will try to find the best alignment by maximizing structural overlap.

• Model based: if the geometry of the imaged object did not change much during the acquisition.

Choose whether to detect and correct rotations:

Lastly, specify the cropping mode for the resulting, stabilized time series:

• Original: to crop the result to the size of the raw data.

• Full: to preserve the size of the stabilized data.

• Tight: to crop away large background regions.

Note

Depending on the cropping mode the dimensions of the result and the raw data will differ. This will become noticeable when comparing the data later. See Results - detailed view.

Hot Pixel Correction

Hot Pixel Correction masks need to be created beforehand using Huygens Professional or Essential and uploaded to the Raw images folder.

Note

Multichannel masks are supported.

When presented with the Hot Pixel Correction file selection window,

browse for the mask of the camera used for the image acquisition.

Select the Hot Pixel mask and click close.

Afterwards, press save and return to the image parameter selection page.

The correction will only be applied to those images of the batch that have the same XY dimensions as the mask.

Note

This correction is optional. The selection can be left empty. Use the reset button to unselect a previously selected mask.

Analysis Template

An analysis parameter template consists of a number of options that provide Huygens Core with information about the type of analysis to execute on the images. To get a preview of the template contents select the template, the contents will be displayed on the right panel.

Note

When editing a template help links are displayed at key locations to point you to specific articles of the SVI-wiki.

Colocalization

Colocalization is a type of analysis which measures the amount of overlap between the objects of two channels.

Here simply select whether or not colocalization analysis should be performed after deconvolution.

Note

Colocalization can only be performed on multichannel images.

Channels

Select the channels to be analyzed.

All the combinations of the selected channels will be studied. For example, the selection shown in the figure above will result in the analysis of the following channels:

• 0 and 1,

• 0 and 2,

• 1 and 2.

Colocalization coefficients

HRM can report all the colocalization coefficients that are most widely used in fluorescence microscopy.

Select here the colocalization coefficients for the analysis:

A colocalization report with all these findings will be ready when the job is finished.

For further information on specific colocalization coefficients please read this article from the SVI wiki.

Threshold

In order to discard background signals from the colocalization analysis it is possible to set a threshold level.

The automatic option typically works well for most purposes. Yet, it’s possible to specify a threshold per channel to discard particular features of each colour.

Colocalization map

While a colocalization coefficient quantifies the level of overlap between two whole channels, a colocalization map displays the value of a specific colocalization coefficient for each voxel of the two channels.

Therefore, a colocalization map is a 3D image of the level of overlap between the two channels, according to a specific coefficient.

It is possible to choose between the 4 most widely used colocalization maps. The maps will be displayed in the colocalization report after the job is finished.

Note

The colocalization maps help us to assess the colocalization regions visually.

As an example the below figure shows the deconvolution result of channels 0 and 3 next to the colocalization map (Pearson coefficient) of the same channels.

Select images (1/5)

In step 1 - Select Images - the user can add images for deconvolution. A batch of images with the same properties can be selected with Ctrl+click or Shift+click and added in one go.

Select the file format from the drop down widget and use to add images to the Selected images area.

Note

The images of a batch must share the same microscopic parameters in order to obtain good deconvolution results because they will be restored with the same settings. This is generally true if the images are recorded with the same setup.

When an image is selected a preview is displayed on the right panel whenever possible. If not yet available, a button shows up to generate the preview on the fly.

Note

Image previews also display the image dimensions, number of channels, and sampling sizes in the overlay.

By default no particular image format is preselected and HRM shows all images available in the user’s raw images folder. Select a file format to filter out images from other file extensions.

The Image Parameters (2/5)

In step 2 - Image Parameter - the microscope settings can be specified and saved. The image parameters are grouped in templates. A parameter template can be reused in future deconvolution jobs. This will be useful when there are more images recorded under the same conditions. Moreover, the parameter templates can be shared with one or more HRM users.

The following selections are possible at this stage:

• Admin image templates: parameter templates created by the HRM administrator to be used as references. These templates cannot be changed directly. Instead, they can be copied to Your image templates (see below) for further changes. For copying to Your image templates select the admin template and use .

• Your image templates: the image templates that the user can use directly. All templates in this area may be selected, edited, removed, etc.

New templates can be added to the list by creating them from scratch using the button, by using the image metadata , by importing a shared template , or by modifying an existing template . To modify a template simply click on the button.

Note

show tooltips describing their actions w.r.t templates. Namely, create, edit, clone, import from an image, import from a Huygens template, share, mark as favourite and delete.

The Restoration Parameters (3/5)

In step 3 - Restoration Parameter - the restoration settings can be specified and saved. As in the previous secion the restoration parameters are grouped in templates. A template can be reused in future deconvolution jobs. This will be useful when other images need to be restored with the same settings. Moreover, the templates can be shared among HRM users, fostering collaboration and re-usage of good restoration settings.

The following selections are possible at this stage:

• Admin restoration templates: parameter templates created by the HRM administrator to be used as references. They can be copied and edited by the HRM users. In order to use them, they first need to be copied to Your restoration templates by using .

• Your restoration templates: the restoration templates that the user can use directly. All templates in this area may be selected, edited, removed, etc.

Note

show tooltips describing their actions w.r.t templates. Namely, create, edit, clone, import from a Huygens template, share, mark as favourite and delete.

Analysis parameters (4/5)

In step 4 - Analysis Parameters - the analysis settings for your images can be specified and saved. As always, the analysis parameters are grouped in templates. A parameter template can be reused in future deconvolution and analysis jobs. This will be useful when the same type of analysis has to be performed on other images. Moreover, the parameter templates can be shared among HRM users, fostering collaboration and re-usage of good analysis settings.

The following selections are possible at this stage:

• Admin analysis templates: parameter templates created by the HRM administrator to be used as references. They can be copied and edited by the HRM users. In order to use them, they first need to be copied to Your analysis templates by using .

• Your analysis templates: the analysis templates that the user can use directly. All templates in this area may be selected, edited, removed, etc.

Note

show tooltips stating their action w.r.t templates. Namely, create, edit, clone, share, mark as favourite and delete.

Launch the job (5/5)

In this last step several summaries display all the chosen parameters.

Also, the output format for the result images can be selected at the top of the page:

• ICS, ICS2 and HDF5: support multi-channel images, 32 bit dynamic range (preserve all image details) and all the microscopic metadata parameters. Furthermore, they support high compression levels. Since deconvolved images often contain background regions with near 0 intensities the compression algorithms usualy manage to greatly reduce the size of the images. For all these reasons these are the formats recommended for further work with Huygens.

• TIFF: this format can be used for analysis such as counting or segmentation. For quantification one needs to scale the TIFF image back to its original dynamic range. The corresponding scaling factors are reported by HRM and can be found in Parameter summaries.

• IMS, OME-XML, R3D: formats offered for compatibility with other programs.

Next, review all the selected parameters before submitting the job to the server. It’s possible to go back to the image, restoration, and analysis template editors by clicking on the corresponding links.

Click on to change the image parameters.

Click on to change the restoration parameters.

Click on to change the analysis parameters.

To change the image selection click on .

Finally submit the job by pressing .

Queue status

After submitting a job HRM returns to the home panel. The Queue Status button shows how many queued jobs you own, as follows:

Click on the button to get a detailed view of the job queue.

HRM manages the deconvolution of multiple jobs owned by different users through a smart queue. The job being processed is marked in green.

The server field will show the name of the machine and, when applicable, the number of the GPU card processing the image. HRM can process multiple jobs at the same time.

Note

The total number of processing machines and GPU cards defines the jobs than HRM can process in parallel.

To delete any of your jobs, select them in the queue table and click .

Note

Depending on the HRM configuration settings, a notification email will be sent to the job owner when the job is finished.

If the job’s result seems wrong, try to verify if there is a mistake in the settings. Contact the HRM administrator otherwise.

Results - detailed view

When the job finishes, the files are placed in the Results folder, accessible via the home panel.

Click on an image result for further inspection as explained in Results .

The detailed view shows the following information:

• MIP of original vs deconvolved,

• SFP of original vs deconvolved,

• Z Slicer of original vs deconvolved,

• T Slicer of original vs deconvolved,

• Stack movie of deconvolved,

• T movie of deconvolved,

• T SFP movie of deconvolved,

• Colocalization results,

• Parameter summaries,

• Huygens log.

Note

The number of tools shown in the detailed view depends on the type of restoration, the image geometry and the HRM configuration.

Comparing MIPs

A comparison of the Maximum Intensity Projections of the raw image and the restored image shows the effect of deconvolution. This can be assessed in both XY and XZ projections.

Comparing SFPs

A comparison of the Simulated Fluoresence Process in both the raw image and the restored image also shows the effect of deconvolution and difference in noise.

Comparing Z Slices

The slicer allows for a comparision of the raw image and the restored image slice by slice along the Z direction at any depth.

Comparing T frames

The time slicer allows for a comparision of the raw image and the restored image frame by frame.

Z, T and T-SFP movies

Click on the corresponding links to download any of the movies to your local computer.

Note

These automatically made movies are quite useful for prepraring presentations.

Colocalization

The selected colocalization coefficients of each two channels are listed in a table with an entry per frame.

Also, 2D histograms show the correlations between any two channels.

Lastly, the colocalization maps show the regions where colocalization between any two channels is strongest.

Parameter summaries

HRM also gives feedback on all the parameter values used during deconvolution. This is particularly useful when the parameters are read from the image metadata.

Notice that green entries indicate parameters loaded from the image metadata, whereas violet entries indicate parameters defined by the user.

If the image had to be scaled to fit in the output file format (some formats have very limited dynamic ranges) HRM will show the scaling factors for further quantification:

Huygens log

The exact steps followed by Huygens to process the data can be found on the Huygens log, under the link .

Note

All these results can be donwloaded along with the deconvolved dataset by clicking at the top of the page.

Tips & Tricks

• File uploads: When using the browser uploader it is possible to upload compressed files containing more images. HRM will uncompress the file and set the contents in your raw folder.

• File uploads: Add more files to your selection by clicking . Use drag&drop from your desktop into the browser.

• Image previews: Select an image and click on Generate preview to visualize the image contents plus the dimensions and sampling sizes. The colours used in the preview depict the wavelengths provided by the image metadata.

• Voxel sizes: Whenever possible HRM highlights in yellow the ideal voxel sizes according to the Nyquist criterion.

• SNR estimation: Notice that it’s possible to zoom in the thumbnails displayed by the SNR estimator by moving the mouse pointer over the thumbnails. This can be useful to confirm the correctness of the SNR

• SNR estimation: The SNR can highly influence the deconvolution result. On the one hand, if the deconvolution result looks too smooth and details are missing, a higher SNR value can be used. On the other hand, if the result looks too grainy a lower SNR value may be better.

• Colocalization: Filter out unsignificant colocalization coefficients by using the Highlight button of the colocalization section in the detailed view.

• Select images (1/5): Tick the “When applicable, load file series automatically” checkbox if several independent files should be loaded into a single image.

For instructions on how to get an HRM account, got to this section.

Additional features

OMERO connector

HRM can import and export images from/to a local or remote OMERO server.

Note

This feature is not active by default. Ask the HRM administrator for more information about the HRM/OMERO connector.

If the connection between HRM and OMERO is enabled the OMERO logo will be is displayed in the Raw images and Results folders.

Click on . The following window will ask you to enter your OMERO login credentials.

Upon logging in successfully a file tree will be displayed listing all the images in the OMERO repository that you are allowed to visualize, like this:

Note

The OMERO credentials and file tree will be used throughout the HRM session. When the session is closed HRM will remove the OMERO credentials from the cache.

The OMERO tree is requested automatically only the first time in every HRM session. This saves waiting time in subsequent import/export operations with OMERO. To force a tree rebuild click .

The following actions are possible:

• Import data from OMERO: In the Raw images folder select an image from the OMERO tree and click . The image will be imported to HRM with extension .ome.tiff. Select more images with Ctrl + click for importing multiple images in one go.

• Export data to OMERO: In the Results folder select the HRM image to be exported. Select the target OMERO dataset. Click . The image will be attached to the OMERO project with an annotation describing all the parameters used for deconvolution. Select more images from the Results folder for exporting multiple images in one go.

Usage Statistics

The user statistics can be reached from the HRM home panel.

Click on Statistics to create reports about the deconvolution and analysis jobs within a specific time window.

The following reports are possible:

• Input file format.

• Output file format.

• Type of PSF used during deconvolution.

• Microscope type.

• Run time per user.

User account

Click on in the home panel to change any of the details linked to your HRM account.

Administration of the HRM

HRM consists of two main components: a web based interface and a queue manager.

The web interface allows:

• the management of users by the system administrator;

• the management of parameter sets (templates) that all users can copy or use directly;

• the creation of deconvolution jobs, including image selection, setting microscopic, restoration and analysis parameters;

• inspecting the job queue status, and allowing the users to delete their own jobs from it;

• previewing and analyzing deconvolved images, including a slicer, MIP and SFP renderers, movies and colocalization results;

• sharing templates among users;

• visualizing user and jobs statistics.

HRM is equipped with a simple http file uploader/downloader to send raw images from the user’s local machine to the HRM server, as well as to retrieve the deconvolution results from the server. The server administrator can set up a limit for these transactions.

The queue manager, running the background, dispatches:

• the jobs created via the web interface to any of the dedicated servers running Huygens Core.

• the emails to inform the users that the job is finished and that the restored datasets are available.

Installation and requirements

To install HRM the following pre-requisites (at least) must be fulfilled:

• Operating system: Any recent Linux distribution. Based on Debian or RHEL.

• Huygens Core: HRM is just an interface and needs Huygens Core to perform deconvolution on the raw images. Note that Huygens Core needs a license.

• Apache2 web server.

• PHP version ≥ 5.6: Both the HRM queue manager and the web interface are written in PHP and need PHP to operate.

• MySQL or PostgreSQL: A relational database management system is required to store deconvolution parameters, job descriptions and, optionally, user accounts.

• A file server: To temporarily store input and restored datasets.

The setup is highly configurable, since the file server, the processing servers and the queue manager can either be all hosted by the same machine or be distributed across two, three or more computers.

To follow the steps for a guided installation please see the installation page at the HRM project’s site.

Administrator’s options

Log in to HRM with the credentials of the administrator. Instead of the regular user home panel HRM will display the administrator home panel, like this:

There are a number of extra options that are available to the administrator only:

• Manage users: Add, remove, edit, enable and disable users. Alternatively, HRM allows for LDAP/Active Directory user management.

• Queue status: Manage jobs from all users.

• Global statistics: See statistics from all users and jobs, including microscope types, PSFs, research groups, dates, etc.

• Database update: Carry out a database update when a new HRM release requires it.

• System summary: Get an overview of all the versions and configuration options that play a role in HRM.

• Check for updates: Get automatic feedback on whether you are running the latest HRM version.

• Create global templates: The administrator can create global templates accessible to all users. This is meaningful for lowering the threshold for those beginning users who need assistance to start deconvolving their raw data. Global templates can be created at any of the states where templates are required: The Image Parameters (2/5), The Restoration Parameters (3/5) and The Restoration Parameters (3/5).

• Servers & GPUs: Add and remove processing machines and GPU cards. This has an effect on the total number of jobs that can run in parallel.

• HuCore licenses: Get an overview of the installed Huygens license.

Version upgrade

Note

Before upgrading HRM please proceed to backup the HRM database, images and source files.

To upgrade to a new version, download the latest version from here. Then follow the intructions on specific version upgrades.

Database update

Note

The HRM database should NOT be deleted and replaced by a newer version.

The HRM database contains the existing users, statistics, parameter templates, etc. To update these contents to the latest release click on Database Update from the admin home panel and follow the instructions.

In order for this update to work properly it is advised to upgrade the HRM source code before upgrading the database.

System configuration

HRM can be configured during the installation. The configuration settings are saved in a configuration file. Some settings, as the maximum upload limit, need to be adjusted both in HRM and in PHP.

For example, if the PHP configuration file states a max upload file size of 256MB, this is applicable to all programs using PHP. Suppose the HRM settings state that the max upload limit is 200MB, since 200MB is lower than 256MB, the HRM configuration does not conflict with the PHP configuration and 200MB is the maximum allowed file size.

Note

The PHP configuration file: /path/to/php/php.ini

The HRM configuration file: /path/to/hrm/config/hrm_config.inc

How HRM communicates with Huygens Core

For each deconvolution task in the job queue the HRM queue manager automatically generates a Huygens Batch template for Huygens Core that

• loads the raw image from a source directory,

• applies the microscopic parameters to it as defined by the user or reads the microscopic parameters from the image metadata,

• optionally loads another image containing the microscope Point Spread Function,

• deconvolves the image using the restoration parameters chosen by the user,

• optionally applies a chromatic aberration correction,

• optionally stabilizes the time series,

• optionally computes the colocalization coefficients of multi-channel images and generates colocalization maps,

• stores the resulting restored image in a destination directory,

• generates several visualizations of the raw and deconvolved images for the user to see the effect of the restoration,

• and finally writes a tag in the destination directory to inform the HRM queue manager that the job is finished.

When the job is finished the queue manager optionally sends the user an e-mail announcing the end of the job and its status. The administrator may configure this.

Multiple jobs can be processed in parallel depending on how HRM is configured, the multiprocessing capabilities of the server, the number of available computation servers and GPU cards.

Tips & Tricks

• Use the global statistics to get an idea on which microscopes are used most, which file formats, PSFs, etc.

• On the home panel, click on System summary to get an overview of your current HRM settings. Expand this information with the PHP settings and/or test the HRM e-mails by clicking on .

• When performing an HRM upgrade disable all users first. Upon successfully upgrading HRM enable the users back again.

• If you experience unexpected behaviour from the HRM queue manager, such as a deleted job which persists in the queue, restart the HRM daemon.

• Encourage your users to save deconvolved data in ICS or HDF5 formats since the dynamic range and metadata parameters are conserved (unlike in TIFF). Moreover both ICS and HDF5 allow for good compression rates.

Clean a Stalled Queue in HRM

It could happen that the queue breaks or freezes and the database is polluted with outdated information.

To fix this, one must currently go and clean up a few places in the hrm database as well as restart the queue manager. Here is a short checklist of what to do.

1. shut down the hrm daemon by using

sudo /etc/init.d/hrmd stop

or

sudo systemctl stop hrmd

2. go to the hrm database and do the following

   1. Remove broken or killed jobs from the job_* tables

   2. Make sure that the server table has a status of ‘free’ and job set to NULL

   Both measures can be achieved with the following SQL script:

DELETE FROM job_analysis_parameter WHERE setting IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_analysis_setting WHERE name IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_files WHERE job IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_parameter WHERE setting IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_parameter_setting WHERE name IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_task_parameter WHERE setting IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_task_setting WHERE name IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_queue  WHERE status="broken" OR status="kill";
UPDATE server SET status= 'free', job = NULL;

3. restart the hrm daemon

sudo /etc/init.d/hrmd start