Using sudo¶

Throughout this document, we’re assuming that commands which require root permissions can be issued by using sudo. If your system is not configured for this, you can either login directly as root (considered as “bad practice”) or use the su - command to switch to the root user.

We highly recommend the usage of sudo though!

Operating systems¶

We will use visual shortcuts to refer to the operating system families supported by the HRM, as in the following table:

Icon	Corresponding operating system family
	Fedora, CentOS, RHEL
	Ubuntu and derivatives

Variables¶

If you want to use those variables in your interactive shell later on, just copy-paste the following lines into your session and adjust the values accordingly:

WWW_ROOT=/var/www/html                 # Web server document root
HRM_HOME=$WWW_ROOT/hrm                 # HRM root (home) folder
HRM_CONFIG=$HRM_HOME/config            # HRM configuration folder
HRM_SAMPLES=$HRM_HOME/config/samples   # HRM configuration samples folder
HRM_RESRC=$HRM_HOME/resources          # HRM resources folder
HRM_SETUP=$HRM_HOME/setup              # HRM setup folder
HRM_BIN=$HRM_HOME/bin                  # HRM executables folder
HRM_USER=$HRM_HOME/user                # HRM customization folder
HRM_LOG=/var/log/hrm                   # HRM logging folder
HRM_DATA=/data/hrm_data                # HRM data folder

Current version¶

Current version of the HRM is 3.3.x.

Download installation script¶

The script downloads, installs and sets up the HRM and its requested dependences while allowing for a certain degree of configurability. Please follow the Automated installation instructions.

Download the installation script.

Download standard archive¶

The HRM archive is the standard installation approach: you have full control over the installation procedure, but you’ll have to do all the work yourself. Just unpack the code and install it following the Manual installation instructions.

Download the archive.

Clone git repository¶

If you want to follow the HRM development, you can also clone the read-only HRM repository from github.

Stop the Queue Manager¶

Significant parts of the configuration as well as the database are usually changed during an upgrade, so the Queue Manager needs to be stopped first.

sudo /etc/init.d/hrmd stop

In some rare situations, the Queue Manager might get stuck. To ensure the stop command did work properly, run the following check which should return empty:

ps aux | grep -i [r]unHuygens

Download and extract the new HRM release¶

To install the new HRM version you need to download the .tar.gz file from the website or github as explained in downloading the standard archive.

The downloaded package then needs to be extracted inside the HRM’s installation directory, overwriting updated files but not touching your configuration files etc. Assuming your downloaded version is 3.3.0 and you were placing the package in your home directory, this can be done like this:

cd $HRM_HOME
tar xzf $HOME/hrm-3.3.0.tar.gz --strip=1

Clean up previous installations¶

Sometimes files were part of a previous release of the HRM but they are not contained in the current one any more. With the above described method those files don’t get removed as only new files are extracted from the tar package. To avoid cluttering up the installation they should be removed according to the versions involved.

HRM 3.3 uses new init scripts. Please delete the old files $HRM_BIN/hrm_user_manager, $HRM_BIN/hrm_user_manager_old, $HRM_BIN/hrmd and $HRM_BIN/ome_hrm and then follow the instructions in Upgrade the init script.

Upgrade the init script¶

This step is basically identical to the initial installation of the init script as described in installing the daemon. You need to copy the new script from $HRM_RESRC/sysv-init-lsb/hrmd to /etc/init.d/ and make sure it is executable. This can be done using the following commands:

sudo cp -v $HRM_RESRC/sysv-init-lsb/hrmd /etc/init.d/hrmd
sudo chmod +x /etc/init.d/hrmd

Update the data folder permissions¶

As of HRM 3.2.0, the system users running the Queue Manager and the web server are expected to have full read-write access to $HRM_DATA. The supported way of doing this is explained in setting up the HRM user and group. Briefly:

• a user hrm and its corresponding group hrm are created
• the web server user ( www-data, apache) is added to the hrm group
• the variable SUSER is set to hrm in /etc/hrm.conf
• the $HRM_DATA and $HRM_LOG group ownership is set to hrm with the setgid bit set

For detailed instructions, please see setting up the HRM user and group.

Update the configuration files¶

As stated in the previous section, the $change_ownership option is not supported in version 3.3 any more. If the variable is present in the configuration files hrm_{server|client}_config.inc it will be silently ignored. For consistency reasons it is therefore recommend to remove this setting from the config file(s).

Check the configuration files¶

An easy way to check for modifications is by running the $HRM_HOME/resources/checkConfig.php script. From the shell, run:

cd $HRM_HOME
php resources/checkConfig.php config/hrm_server_config.inc

Checking the 3.2.x files with the 3.3.x checkConfig.php script will result in the following output:

Check against HRM v3.3.x.
* * * Error: variable change_ownership must be removed from the configuration files!
Check completed with errors! Please fix your configuration!

Please make sure to fix all problems. The sample files and the Manual installation instructions will help you set the correct parameters.

Update the database¶

Newer versions of the HRM might use slightly different/updated versions of the database back-end than previous ones.

For this reason, the first time you run the HRM after an update you will be told that the database must be updated and that you are not allowed to continue until this has been done!

The following describes two possible ways to update the database.

Needed database revision for HRM v3.3 is number 14.
Current database revision is number 13.
Updating...

Database successfully updated to revision 14.

Re-start the Queue Manager¶

After processing the described upgrade steps, the Queue Manager needs to be started again.

sudo /etc/init.d/hrmd start

Upgrade from previous releases¶

The following pages are linked to from the relevant sections above, but are listed here again for convenience.

rm -v inc/ActiveDirectory.inc.php inc/Ldap.inc.php

$change_ownership=true;

$omero_transfers={true|false}

[+] max_upload_limit
[+] max_post_limit
[+] email_list_separator

[-] resultImagesOwnedByUser
[-] resultImagesRenamed
[-] enable_code_for_huygens

cd $HRM_HOME
php resources/checkConfig.php config/hrm_server_config.inc

Check against HRM v3.2.x.
Check completed successfully! Your configuration file is valid!

Check against HRM v3.1.x.
Check completed successfully! Your configuration file is valid!

Check against HRM v3.0.x.
* * * Error: variable omero_transfers not set or empty.
Check completed with errors! Please fix your configuration!

Check against HRM v2.1.x.
* * * Error: variable max_upload_limit not set or empty.
* * * Error: variable max_post_limit not set or empty.
* * * Error: variable email_list_separator not set or empty.
* * * Error: variable resultImagesOwnedByUser must be removed from the configuration files!
* * * Error: variable resultImagesRenamed must be removed from the configuration files!
* * * Error: variable enable_code_for_huygens must be removed from the configuration files!
Check completed with errors! Please fix your configuration!

What is HRM¶

HRM is a collaborative open-source interface to Huygens Core for multi-user, web-based, scheduled, batch deconvolution.

The Huygens Remote Manager (HRM) enables large groups of users to work with Huygens on a central system through an easy-to-use interface. HRM is web-based and therefore easily accessible from your workstation, home PC, tablet or smart phone.

An HRM server can distribute work among several processing machines smartly, which allows for intelligent use of hardware and fast processing times. It features an intelligent queue that switches among the jobs of all users.

Who makes HRM¶

HRM is an open source project, this means that the code is freely accessible and it can be adapted and modified to any needs by any user.

HRM is a joined collaboration of Huygens users from:

• Montpellier RIO Imaging,
• Facility for Advanced Imaging and Microscopy at the Friedrich Miescher Institute (FMI, Basel),
• The BioImaging and Optics Platform at the Ecole Polytechnique Fédérale de Lausanne,
• The Department of Biosystems Science and Engineering at the ETH Zürich,
• The Leibniz Institute for Neurobiology (Magdeburg),
• The Combinatorial Neuroimaging facility (Magdeburg),
• The Imaging Core Facility at the Biozentrum (University of Basel).

Scientific Volume Imaging participates in this project by contributing its experience in deconvolution and software engineering.

Where to find HRM¶

HRM is a free and open source project, and can be found on http://huygens-rm.org. The source code is hosted on https://github.com/aarpon/hrm.

More information about HRM and links to other HRM resources can be found in this online article of the SVI Wiki. Instructions for online testing, downloading and installing HRM are also linked on that page.

Installing HRM on a regular web server is not very difficult. Please consult the Installation instructions and the SVI Wiki.

Create an account in HRM¶

• If your imaging facility already offers deconvolution via HRM, proceed to the facility’s HRM login page.
• If HRM is not available at your imaging facility you can still familiarize with HRM by using the HRM demo server offered by SVI.

Follow the registration link on the HRM login page and complete the short form. Notice that names are case sensitive. Upon registration the new HRM account needs to be validated by the administrator.

Upload a raw image¶

Upon successful logon the HRM Home panel will be displayed:

To upload images click on the Raw images icon . The following, intuitive upload tool allows for multiple uploads from your local device:

Deconvolve¶

To submit a deconvolution job proceed to Start a job from the home panel. The process of submitting the job is split into the following 5 steps:

• Step 1/5 - Select images: click on the “Image File Format” drop down widget to select the desired file type for the images that you wish to deconvolve. All files of that format in your “Raw Images” folder will be listed. Use the icons to discard/include images from/into the selection. When selecting a file, a preview can be requested by using the link on the right hand side panel. The image preview also provides information about the dimensions, sampling and number of channels.

• Step 2/5 - Image parameters: An image parameter template matching the imaging conditions of the microscope can be created at this stage. Alternatively, an existing template can be selected from a previous run. Also, other HRM users (including the administrator) may have shared templates with you. Notice that choosing the correct parameters is absolutely crucial for obtaining high-quality deconvolution results.

  Use existing parameter template (easy): When selecting an existing parameter template the parameter contents are displayed on the right hand side panel. Make sure that the microscope type and the number of channels of the selected template match those of the images to deconvolve.

  Create a new parameter template (elaborated): Type a name for the new parameter template in the New/clone image template name field, then click the create button . The template editor opens, select the correct number of channels and PSF type. Move on to the optical parameters and select the correct microscope type. For a detailed explanation on the remaining parameters please refer to section Advanced deconvolution in HRM.

• Step 3/5 - Restoration parameters: As in the previous step it’s possible to choose between an existing template or creating a new one.

  Use existing parameter template (easy): Select a parameter template and verify its contents on the preview shown on the right hand side panel. Particularly, pay attention to the value of the Signal to Noise ratio (SNR) (see Signal to Noise ratio for further help).

  Create a new parameter template (elaborated): Follow the instructions of the previoius step to create a new template. Once in the template editor select a deconvolution algorithm (CMLE is a good choice) and an SNR. Follow the instructions of the embedded SNR estimator for a hint on the SNR of your images. Set the background mode to automatic, the number of iterations to 40 and the quality change to 0.01. For more detailed fine-tuning follow the instructions at Advanced deconvolution in HRM.

• Step 4/5 - Analysis parameters: This step is active only when 2 or more channels have been specified in the previous steps. When active, the Analysis step allows the user to execute Colocalization analysis on the deconvolved images.

  Use existing parameter template (easy): Select the analysis parameter template of your choice and verify that the values displayed on the preview on the right hand side panel match your analysis needs.

  Create a new parameter template (elaborated): Create a new template by following the instructions given in Step 2. Select whether colocalization should be enabled. For a colocalization run select the channels that should be inspected and the colocalization coefficients to be reported. Set the threshold to “Automatic estimation” and the Colocalization map to “Pearson”. In order to further fine-tune the Colocalization parameters please refer to Advanced deconvolution in HRM.

• Step 5/5 - Launch job: Select the output file format for the restored images. ICS and HDF5 are the recommended formats because they provide good dynamic range and the necessary metadata infrastructure to save all the image parameters. Plus, they allow for good compression levels that minimize the amount of disk space needed for storage. Next, review the parameter summaries as well as the image selection. If all the settings are correct then proceed to submit the job to the server by clicking .

Queue¶

After submitting the job HRM returns to the Home panel from where you can access the queue by clicking . A typical processing queue looks like the following:

where you view how much work from you and other HRM users has been submitted to the processing server. This queue is handled in an intelligent way, allowing jobs from different users to be processed at different times.

Results¶

When a deconvolution and analysis job is finished you will receive a notification email about your results being available. In order to view the deconvolved images go to the HRM home panel and click on the Results icon . The contents of the Results folder will be listed. Select the image that you would like to visualize and a preview will be displayed on the right hand side panel, like this:

For a detailed view of both the raw image and the restored image click on .

A results page will open showing MIP, SFP and slicer comparisons of the raw image versus the deconvolved image. Summary tables will show information about possible scaling factors, colocalization results as well as deconvolution and image parameters. Also, movies along the Z direction and/or through time will be available for download.

Tips & Tricks¶

• When a deconvolution run gives excellent results download the processed data from the detailed view by clicking on .
• If you are satisfied with any of the templates that you have created do share them with your colleagues by clicking on in steps 2, 3 or 4.
• To optimize your deconvolution results it is necessary to understand the different properties of your image and some of the mechanisms behind deconvolution. If you’ve followed this guide, many properties have been assigned default values. We would like to encourage the users to read into and explore different settings, which will help you improve your deconvolution results. Some more in-depth knowledge is given in section section Advanced deconvolution in HRM.

Technical features¶

HRM consists of two main components: a web based interface and a queue manager.

The web interface allows:

• the management of users by the system administrator;
• the management of parameter sets that all users can copy or use directly;
• the creation of deconvolution jobs, including image selection, setting microscopic, restoration and analysis parameters;
• inspecting the job queue status, and allowing the users to delete their own jobs from it;
• previewing and analyzing deconvolved images, including a slicer, MIP and SFP renderers, movies and colocalization results;
• sharing templates among users;
• visualizing user and jobs statistics.

HRM is equipped with a simple http file uploader/downloader to send raw images from the user’s local machine to the HRM server, as well as to retrieve the deconvolution results from the server. The server administrator can set up a limit for these transactions.

The queue manager, running the background, dispatches:

• the jobs created via the web interface to any of the dedicated servers running Huygens Core.
• the emails to inform the users that the job is finished and that the restored datasets are available.

Installation and requirements¶

To install HRM the following pre-requisites (at least) must be fulfilled:

• Operating system: Any recent Linux distribution. Ubuntu and Fedora are the recommended distro’s for HRM.
• Huygens Core: HRM is just an interface and needs Huygens Core to perform deconvolution on the raw images. Note that Huygens Core needs a license.
• Apache2 web server.
• PHP version ≥ 5.3: Both the HRM queue manager and the web interface are written in PHP and need PHP to operate.
• MySQL or PostgreSQL: A relational database management system is required to store deconvolution parameters, job descriptions and, optionally, user accounts.
• A file server: To temporarily store input and restored datasets.

The setup is highly configurable, since the file server, the processing servers and the queue manager can either be all hosted by the same machine or be distributed across two, three or more computers.

To follow the steps for a guided installation please see the installation page at the HRM project’s site.

Administrator’s options¶

Log in to HRM with the credentials of the administrator. Instead of the regular user home panel HRM will display the administrator home panel, like this:

There are a number of extra options that are available to the administrator only:

• Manage users: Add, remove, edit, enable and disable users. Alternatively, HRM allows for LDAP/Active Directory user management.
• Queue status: Manage jobs from all users.
• Global statistics: See statistics from all users and jobs, including microscope types, PSFs, research groups, dates, etc.
• Database update: Carry out a database update when a new HRM release requires it.
• System summary: Get an overview of all the versions and configuration options that play a role in HRM.
• Check for updates: Get automatic feedback on whether you are running the latest HRM version.
• Create global templates: The administrator can create global templates accessible to all users. This is meaningful for lowering the threshold for those beginning users who need assistance to start deconvolving their raw data. Global templates can be created at any of the states where templates are required: The Image Parameters (2/5), The Restoration Parameters (3/5) and Analysis parameters (4/5).

Version upgrade¶

To upgrade to a new version, download the latest version from here. Next, replace the old source files with the new ones. For Linux you can use something like:

cp -r /path/to/newHRM /path/to/oldHRM*

Because the configuration file is not included in new versions, the existing configuration file will be conserved.

For intructions on specific version upgrades please refer to this page at the HRM project’s site.

Database update¶

The HRM database contains the existing users, statistics, parameter templates, etc. To update these contents to the latest release click on Database Update from the admin home panel and follow the instructions.

In order for this update to work properly it is advised to upgrade the HRM source code before upgrading the database.

System configuration¶

HRM can be configured during the installation. The configuration settings are saved in a configuration file. Some settings, as the maximum upload limit, need to be adjusted both in HRM and in PHP.

For example, if the PHP configuration file states a max upload file size of 256MB, this is applicable to all programs using PHP. Suppose the HRM settings state that the max upload limit is 200MB, since 200MB is lower than 256MB, the HRM configuration does not conflict with the PHP configuration and 200MB is the maximum allowed file size.

How HRM communicates with Huygens Core¶

For each deconvolution task in the job queue the HRM queue manager automatically generates a Huygens Batch template for Huygens Core that

• loads the raw image from a source directory,
• applies the microscopic parameters to it as defined by the user or reads the microscopic parameters from the image metadata,
• optionally loads another image containing the microscope Point Spread Function,
• deconvolves the image using the restoration parameters chosen by the user,
• stores the resulting restored image in a destination directory,
• generates several visualizations of the raw and deconvolved images for the user to see the effect of the restoration,
• and finally writes a tag in the destination directory to inform the HRM queue manager that the job is finished.

When the job is finished the queue manager optionally sends the user an e-mail announcing the end of the job and its status. The administrator may configure this.

Multiple jobs can be processed in parallel depending on how HRM is configured, the multiprocessing capabilities of the server and the number of available computation servers.

Tips & Tricks¶

• Use the global statistics to get an idea on which microscopes are used most, which file formats, PSFs, etc.
• On the home panel, click on System summary to get an overview of your current HRM settings. Expand this information with the PHP settings and/or test the HRM e-mails by clicking on .
• When performing an HRM upgrade disable all users first. Upon successfully upgrading HRM enable the users back again.
• If you experience unexpected behaviour from the HRM queue manager, such as a deleted job which persists in the queue, restart the HRM daemon.
• Encourage your users to save deconvolved data in ICS or HDF5 formats since the dynamic range and metadata parameters are conserved (unlike in TIFF). Moreover both ICS and HDF5 allow for good compression rates.

Clean a Stalled Queue in HRM¶

It could happen that the queue breaks or freezes and the database is polluted with outdated information.

To fix this, one must currently go and clean up a few places in the hrm database as well as restart the queue manager. Here is a short checklist of what to do.

1. shut down the hrm daemon

sudo /etc/init.d/hrmd stop

2. go to the hrm database and do the following
   1. Remove broken or killed jobs from the job_* tables
   2. Make sure that the server table has a status of ‘free’ and job set to NULL

DELETE FROM job_analysis_parameter WHERE setting IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_analysis_setting WHERE name IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_files WHERE job IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_parameter WHERE setting IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_parameter_setting WHERE name IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_task_parameter WHERE setting IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_task_setting WHERE name IN (SELECT id FROM job_queue WHERE status="broken" OR status="kill");
DELETE FROM job_queue  WHERE status="broken" OR status="kill";
UPDATE server SET status= 'free', job = NULL;

3. restart the hrm daemon

sudo /etc/init.d/hrmd start