Huygens Remote Manager 3.2.1 documentation¶
These are the detailed installation and administration instructions for the Huygens Remote Manager.
Contents¶
Conventions¶
This page lists some conventions and notations used in this documentation.
Using sudo¶
Throughout this document, we’re assuming that commands which require root permissions can be issued by using sudo. If your system is not configured for this, you can either login directly as root (considered as “bad practice”) or use the su - command to switch to the root user.
We highly recommend the usage of sudo though!
Operating systems¶
We will use visual shortcuts to refer to the operating system families supported by the HRM, as in the following table:
Icon | Corresponding operating system family |
---|---|
Fedora, CentOS, RHEL | |
Ubuntu and derivatives |
Warning
Support for Mac OS X was dropped in HRM version 3.1.
Variables¶
Variable | Description | (Example) value |
---|---|---|
$WWW_ROOT | Web server document root | /var/www/html |
$HRM_HOME | HRM root (home) folder | $WWW_ROOT/hrm |
$HRM_CONFIG | HRM configuration folder | $HRM_HOME/config |
$HRM_SAMPLES | HRM configuration samples folder | $HRM_HOME/config/samples |
$HRM_RESRC | HRM resources folder | $HRM_HOME/resources |
$HRM_SETUP | HRM setup folder | $HRM_HOME/setup |
$HRM_BIN | HRM executables folder | $HRM_HOME/bin |
$HRM_USER | HRM customization folder | $HRM_HOME/user |
$HRM_DATA | HRM data folder | /data/hrm_data |
If you want to use those variables in your interactive shell later on, just copy-paste the following lines into your session and adjust the values accordingly:
WWW_ROOT=/var/www/html # Web server document root
HRM_HOME=$WWW_ROOT/hrm # HRM root (home) folder
HRM_CONFIG=$HRM_HOME/config # HRM configuration folder
HRM_SAMPLES=$HRM_HOME/config/samples # HRM configuration samples folder
HRM_RESRC=$HRM_HOME/resources # HRM resources folder
HRM_SETUP=$HRM_HOME/setup # HRM setup folder
HRM_BIN=$HRM_HOME/bin # HRM executables folder
HRM_USER=$HRM_HOME/user # HRM customization folder
HRM_LOG=/var/log/hrm # HRM logging folder
HRM_DATA=/data/hrm_data # HRM data folder
Note
Please notice that $WWW_ROOT is /var/www/html in and in as of version 14.04 LTS. In earlier versions of , however, $WWW_ROOT used to be /var/www.
Get the HRM¶
Current version¶
Current version of the HRM is 3.2.1.
Download installation script¶
The script downloads, installs and sets up the HRM and its requested dependences while allowing for a certain degree of configurability. Please follow the Automated installation instructions.
Download standard archive¶
The HRM archive is the standard installation approach: you have full control over the installation procedure, but you’ll have to do all the work yourself. Just unpack the code and install it following the Manual installation instructions.
Note
This is the only way to install the HRM on platforms not supported by the installation script or to tweak the set up beyond the set of options provided by it.
Automated installation¶
We will release the installation scripts soon. The documentation will follow!
Manual installation¶
Install the prerequisites¶
The HRM requires a few prerequisites for its functions.
Operating system¶
The HRM should work on any recent Linux distribution, but we only support Ubuntu (with its derivatives) and Fedora (that will apply to RHEL and CentOS). Distribution-specific differences are marked in the following with the corresponding distribution logos.
Warning
Please notice that with release 3.1, we dropped support for Mac OS X. HRM 3.0 is still known to work on Mac OS X from 10.5 (Leopard) onward, but no effort will be made to make future versions of the HRM compatible with Mac OS X. Also notice that the HRM was never tested on Mavericks. For reference, see the old documentation.
Huygens Core¶
The HRM is an interface to Scientific Volume Imaging’s Huygens Core. Huygens Core is is a fully scriptable compute engine intended to run image processing and deconvolution jobs on large 64 bit multiprocessor servers in headless mode, i.e. without a specific graphical interface. The HRM provides such an interface for multi-user, batch access to Huygens Core.
Note
If the web and the processing server are not on the same machine, you will need an additional Huygens Core for the web server with a reader license (free of charge).
Apache2 web server¶
sudo apt-get install apache2
sudo yum install httpd
Web pages can be installed globally or per-user.
The Apache2 global document root is /var/www, or /var/www/html in more recent versions (14.04 LTS and newer).
The Apache2 global document root is /var/www/html.
If you plan to install the HRM in a specific user directory, use /home/<hrm_user>/public_html.
Apache2 access handling¶
HRM uses .htaccess files to prevent access to configuration files. Make sure to set the AllowOverride directive in /etc/apache2/sites-available/default resp. FIXME to enable .htaccess files in the HRM on the web server (AllowOverride All), or at least make sure to prevent access to the subdirectories inc, config, run, resources and setup.
If you are installing the HRM in your user dir, make sure to change AllowOverride to All in /etc/apache2/mods-available/userdir.conf resp. FIXME (make sure to enable the userdir mod first by running sudo a2enmod userdir in the shell).
See also Enabling use of Apache htaccess files.
PHP ≥ 5.3¶
The HRM is made of two parts, a web interface and a queue manager, both written in PHP but with different requirements. The web interface requires the PHP 5 module for Apache2, the queue manager requires the PHP 5 command line interpreter.
Note
Minimum required PHP version is 5.3.
sudo apt-get install libapache2-mod-php5 php5 php5-cli php5-common php5-json
Note
JSON support for PHP was moved into a separate package php5-json in Ubuntu 14.04LTS; in older versions, JSON support is part of the core php5 package.
sudo yum install php php-cli php-common php-process php-json
A relational database¶
The HRM officially supports two relational databases: MySQL and PostgreSQL.
MySQL¶
sudo apt-get install php5-mysql mysql-server
sudo yum install php-mysql php-pdo mysql-server
Note
It is recommended to install a database management tool like phpmyadmin.
PostgreSQL¶
sudo apt-get install php5-pgsql postgresql
sudo yum install php-pgsql postgresql-server postgresql-contrib
You will need to manually enable PostgreSQL:
sudo systemctl enable postgresql
Note
It is recommended to install a database manageent tool like phppgadmin.
Some additional information:
(Optional) LDAP support¶
If you plan to configure the HRM to use either Active Directory authentication or LDAP authentication, you will need to install the php-ldap package as well:
sudo apt-get install php5-ldap
sudo yum install php-ldap
Sendmail (postfix)¶
HRM uses the PHP mail() function to notify the users:
“For the Mail functions to be available, PHP must have access to the sendmail binary on your system during compile time. If you use another mail program, such as qmail or postfix, be sure to use the appropriate sendmail wrappers that come with them.” More...
sudo apt-get install postfix
sudo yum install postfix
PHP date() and default timezone¶
Please make sure to set the default timezone in php.ini as follows:
[Date]
; Defines the default timezone used by the date functions
; http://php.net/date.timezone
date.timezone = "Europe/Zurich"
Otherwise you will get the following warning every time the PHP function date() is called within the HRM:
PHP Warning: date(): It is not safe to rely on the system's timezone settings. You are
required to use the date.timezone setting or the date_default_timezone_set() function. (...)
Click here for the full list of supported time zones.
SSH¶
If the queue manager and the image processing server are not on the same machine (see installation instructions), HRM transfers files via ssh between the two using sudo. To allow HRM to login and run commands as sudo via remote, it is necessary to comment out the line 'Defaults requiretty' in the /etc/sudoers file.
Compressors¶
The HRM compresses files to be downloaded (such as deconvolution results). Several options are possible (and more can be added in the configuration files), but by default the HRM uses zip.
sudo apt-get install zip
sudo yum install zip
(Optional) OMERO support¶
If you plan to use the OMERO connector, you will need to download the “server” package from the OMERO website that matches your existing OMERO installation and the Ice version installed on your HRM system.
Note
It is NOT required to do any installation or configuration of the downloaded OMERO package! HRM just needs this package as a means to communicate with your existing installation of OMERO server. Thus, this package allows HRM to communicate with any OMERO server connected to the network.
As an example on how to download these packages, the commands used to fetch OMERO 5.0.3 and Ice 3.4 are shown below. For other combinations please have a look at the OMERO download site. We recommend placing the downloaded OMERO “server” package into a subdirectory of /opt/OMERO, as follows:
sudo apt-get install python-zeroc-ice libicessl34
wget http://downloads.openmicroscopy.org/omero/5.0.3/artifacts/OMERO.server-5.0.3-ice34-b41.zip -O /tmp/OMERO.server.zip
sudo mkdir -pv /opt/OMERO
cd /opt/OMERO
sudo unzip /tmp/OMERO.server.zip
rm /tmp/OMERO.server.zip
sudo yum install ice-python
wget http://downloads.openmicroscopy.org/omero/5.0.3/artifacts/OMERO.server-5.0.3-ice34-b41.zip -O /tmp/OMERO.server.zip
sudo mkdir -pv /opt/OMERO
cd /opt/OMERO
sudo unzip /tmp/OMERO.server.zip
rm /tmp/OMERO.server.zip
Note
Due to an issue in OMERO up to version 5.0.4 there’s an attempt to store and read OMERO session files in a subfolder of the user’s HOME directory who executes OMERO queries - in our case the same user that is running Apache. This will fail on most standard installations due to the default directory permissions in Apache’s document root, therefore it is necessary to manually create this session directory and adjust the permissions accordingly, as follows:
sudo mkdir /var/www/omero
sudo chown www-data /var/www/omero
sudo chmod u+w /var/www/omero
Please contact us in case you’re trying to set up the OMERO connector on a Fedora system and you’re running into trouble there!
Install the HRM¶
Download or checkout the HRM as explained here.
Unpack the downloaded archive to web server document root directory. This is the directory where Apache2 finds the html and php files to serve.
up to 13.10
sudo unzip hrm_x.y.z.zip -d /var/www
and 14.04 LTS and later
sudo unzip hrm_x.y.z.zip -d /var/www/html
where x.y.z is a placeholder for the HRM version.
Note
You can of course extract or clone the HRM somewhere else: just add the location to the Apache2 configuration (httpd.conf).
Configure the HRM¶
Mandatory configuration steps¶
Edit hrm.conf¶
The hrm.conf file is used by both the Web server and the Queue Manager.
Copy the sample file¶
The sample configuration file $HRM_SAMPLES/hrm.conf.sample must be copied to /etc/hrm.conf and then edited as explained in the following sections.
sudo cp $HRM_SAMPLES/hrm.conf.sample /etc/hrm.conf
Edit the configuration file¶
This is the content of the sample configuration file:
#!/bin/bash
#
# HRM configuration file
#
# This file is part of the Huygens Remote Manager
# Copyright and license notice: see license.txt
# HRM install directory
HRM_HOME="/path/to/hrm/home"
# HRM image share
HRM_DATA="/path/to/hrm/data"
# Source and destination folder names
HRM_SOURCE="huygens_src"
HRM_DEST="huygens_dst"
# User to run the HRM daemon with rights to
# execute /bin/mkdir, /bin/chown, /bin/chmod and
# /bin/rm on HRM_DATA (see above) for user management.
SUSER="hrm"
# HRM log directory
HRM_LOG="/var/log/hrm"
# Interaction with OMERO (if switched on in hrm/config).
OMERO_PKG="/opt/OMERO/OMERO.server"
OMERO_HOSTNAME="full.qualified.name.of.omero.server"
OMERO_PORT="4064"
# Set a custom PHP CLI binary if necessary:
# PHP_CLI="/usr/local/php/bin/php"
Explanation¶
- HRM_HOME points to $WWW_ROOT/hrm (for example: /var/www/html/hrm).
- HRM_DATA points to the data folder that contains all user subfolders (for example: /data/hrm_data).
- HRM_SOURCE and HRM_DEST are the source and destination subfolders inside the user directory (for example: src and dst). The source folder for an hypothetical user ‘john’ would then be /data/hrm_data/john/src.
- SUSER is the Unix user that runs the Queue Manager (for example: hrm).
- HRM_LOG is the log folder for the HRM (for example: /var/log/hrm)
- (optional) OMERO_PKG, OMERO_HOSTNAME and OMERO_PORT: see OMERO connector for details.
- (optional) PHP_CLI is the path tho the php CLI executable if it is not in the path or another one should be used (for example: /usr/local/php/bin/php).
Set up the HRM user and group¶
Create a Unix group hrm and user hrm on the web server machine.
$ sudo groupadd --system hrm
$ sudo useradd hrm --system --gid hrm
Create the log directory:
sudo mkdir ${HRM_LOG}
Make sure hrm owns (and has full read-write access) to HRM_DATA and HRM_LOG. This is done by setting the group ownership of HRM_DATA and HRM_LOG to hrm:
sudo chown -R hrm:hrm ${HRM_DATA}
sudo chmod -R u+s,g+ws ${HRM_DATA}
sudo chown -R hrm:hrm ${HRM_LOG}
sudo chmod -R u+s,g+ws ${HRM_LOG}
Add the Apache user ( www-data, apache) to the hrm group:
# www-data in Ubuntu, apache in Fedora
sudo usermod www-data --append --groups hrm
Note
You might have to restart your server for the group changes to be activated.
Edit hrm_{server|client}_config.inc¶
Copy the sample files¶
Copy $HRM_SAMPLES/hrm_server_config.inc.sample to $HRM_CONFIG/hrm_server_config.inc, this file is used by the queue manager:
sudo cp -v $HRM_SAMPLES/hrm_server_config.inc.sample $HRM_CONFIG/hrm_server_config.inc
In a single-machine HRM installation, the server and client configuration files are identical. It is therefore recommended to create a symlink for the client configuration file that points to the server config file. This prevents out-of-sync configurations:
sudo ln -s hrm_server_config.inc $HRM_CONFIG/hrm_client_config.inc
Edit the configuration files¶
Here we will assume a one-machine installation, and we will therefore show just one file ($HRM_CONFIG/hrm_server_config.inc).
<?php
// This file is part of the Huygens Remote Manager
// Copyright and license notice: see license.txt
// This configuration file is used by the QUEUE MANAGER.
//==============================================================================
// Database settings
//==============================================================================
// The database type (mysql or postgres)
$db_type = "mysql";
?>
Note
Although the database abstraction library (ADOdb) we use in the HRM can interface with a large number of databases, we currently only support MySQL and PostgreSQL. Other DBs may work, but we haven’t tested them.
<?php
...
// The name of the database host machine (this may be localhost if it is the
// machine on which the web server runs)
$db_host = "localhost";
// The name of the database used by HRM
$db_name = "hrm";
// The name of the user with which the database is accessed
$db_user = "dbuser";
// The password of the account with which the database is accessed
$db_password = "dbpasswd";
//==============================================================================
// Huygens settings
//==============================================================================
// Huygens server default user
$huygens_user = "huygens";
$huygens_group = "huygens";
?>
The $huygens_user is used in case the processing machine (the one where Huygens Core is installed) does not have direct access to the file server (i.e. the files to be deconvolved must be copied from the file server to the processing machine via ssh). For this to work, you will need to set up password-less ssh connection between file and processing server for the $huygens_user (see for example here) and also set the variable $copy_images_to_huygens_server to true (see below).
<?php
...
// Path to a local Huygens Core executable, to handle image information and
// thumbnails. This installatation of hucore doesn't require a full license
// unless this is also the computation server. For image information and
// thumbnails mostly freeware functions will be used, but support for certain
// propietary file formats requires a B flag in the license string. (See
// http://support.svi.nl/wiki/FileFormats).
//
// In combination with Huygens 3.5.1 or higher, HRM can also provide an
// estimate of the Signal-To-Noise ratio (SNR) of an image, interactively. At
// least that same simple license is required for this to work.
$local_huygens_core = "/usr/local/bin/hucore";
?>
The web interface uses hucore for some operations on the input and result files (for example to generate image previews or estimate the SNR interactively). An additional copy of hucore needs therefore to be installed on the web server. However, since all these functions used are accessible in reader mode, only a reader license is needed (free of charge).
<?php
//==============================================================================
// File server settings
//==============================================================================
// File server host name
$image_host = "localhost";
// File server default user
$image_user = "apache";
$image_group = "apache";
// File server base folder (without trailing /)
$image_folder = "/path/to/hrm_data";
// File server image source folder
$image_source = "huygens_src";
// File server image destination folder
$image_destination = "huygens_dst";
// File server base folder as seen from the Huygens server machines (with trailing /)
$huygens_server_image_folder = "/path/to/hrm_data/";
?>
This is information needed for the web interface and the queue manager to login to the file server. If the file server is not on the same machine, its host name must be given.
<?php
...
// Allow HTTP transfer of the restored (download) images
$allowHttpTransfer = true;
// Allow HTTP transfer of original (upload) images.
// Limitations in upload size must be configured in the below parameters
// $max_upload_limit AND $max_post_limit. If these are set to 0, they are
// instead read from the PHP settings. If set to higher values, make sure
// that the PHP settings allow for such transfer limits. These PHP settings
// are normally located in php.ini, with the options upload_max filesize AND
// post_max size.
// Unzipping large archives may also require high max_execution_time and
// memory_limit variables.
$allowHttpUpload = true;
$max_upload_limit = 0; // Maximum file size for uploads in Mb
$max_post_limit = 0; // Maximum post size for uploads in Mb
// Archiver for downloading the results via the web browser:
// $compressBin is the archiver, including options. %DEST% will be replaced
// with the user's destination path; can be used as an option or part of the
// command to change directories with 'cd'. Separate commands with "\n".
// $packExcludePath controls whether the archiving command excludes the whole
// file path per file when this is taken into account in the $compressBin
// command itself by using %DEST% properly.
$compressExt = ".zip";
switch ($compressExt) {
case ".tgz":
case ".tar.gz":
$compressBin = "cd %DEST% \n /bin/tar chzf ";
$packExcludePath = true;
$dlMimeType = "application/x-force-download";
break;
case ".tar":
$compressBin = "cd %DEST% \n /bin/tar cfh ";
$packExcludePath = true;
$dlMimeType = "application/x-force-download";
break;
case ".zip":
$compressBin = "cd %DEST% \n /usr/bin/zip ";
$packExcludePath =true;
$dlMimeType = "application/x-force-download";
break;
default:
$compressExt = ".zip";
$compressBin = "cd %DEST% \n/usr/bin/zip -D";
$packExcludePath = true;
$dlMimeType = "application/x-force-download";
}
// Tools to decompress uploaded archives. This is an array, each key linked to
// an archive extension. %DEST% will be replaced with the user's destination
// path; can be used as an option or part of the command to change directories
// with 'cd'.
// More formats are possible, just add more commands to the array.
$decompressBin['zip'] = "cd %DEST% \n /usr/bin/unzip -o ";
$decompressBin['tgz'] = "cd %DEST% \n /usr/bin/tar xzf ";
$decompressBin['tar'] = "cd %DEST% \n /usr//bin/tar xf ";
$decompressBin['tar.gz'] = "cd %DEST% \n /usr/bin/tar xzf ";
?>
If $allowHttpTransfer is true, the results of deconvolution can be downloaded through the web interface. If it is false, then the results can only be accessed by other means (e.g. via network shares). Downloaded files are compressed to reduce bandwidth load: $compressExt defines the type of compression (default is zip).
Note
For the compressed file to be served by the web server, it has to be re-read in. The HRM uses chunked reading, that should make sure that even large files can be read without problems. Should the users still encounter problems downloading files from the HRM, then try increasing the value of memory_limit in php.ini.
Example:
memory_limit = 128M
If $allowHttpUpload is true, an HTTP uploader will be in place to allow uploading of the files to be deconvolved through the web interface. Multi-file upload (with directory structures) is indirectly possible by first compressing the files into an archive that the HRM will automatically extract at the end upload process. Default supported formats are zip, tgz, tar and tar.gz (as long as the corresponding executables are installed on the system) but more can be added by extending the $decompressBin array.
Warning
Make sure to change the values of upload_max_filesize AND post_max_size in php.ini: with the default values, only extremely small files can be uploaded!
Example:
post_max_size = 1024M
upload_max_filesize = 1024M
Note
Notice that despite some script executions are relatively fast (like file extraction from zip archives, image previews generation, or SNR estimation), they may take a long time if the images are very large or if deconvolution jobs are already running in the web server. You may need to increase also PHP’s max_execution_time directive in php.ini.
Example:
max_execution_time = 120
<?php
...
//==============================================================================
// HRM configuration parameters
//==============================================================================
// URL
$hrm_url = "http://localhost/hrm";
// Install dir
$hrm_path = "/var/www/html/hrm";
// Logging
// Please create a directory for the logging (default and recommended is /var/log/hrm)
// and make sure to grant read/write access to the web server user.
$log_verbosity = 1; // 0=quiet, 1=normal, 2=debug
$logdir = "/var/log/hrm";
$logfile = "hrm.log";
$logfile_max_size = 100; // maximum size of the logfile in MB
?>
The $logdir variable points to the directory where the log files created by the web server user and the queue manager should reside. Default (and recommended) directory for the logs is /var/log/hrm. Please create this directory and make sure to grant the relevant users read/write access to it!
<?php
...
// Email
$send_mail = true;
$email_sender = "hrm@localhost";
$email_admin = "hrm@localhost";
// Comma (',') is the standard separator for lists of email addresses.
// Please see: http://tools.ietf.org/html/rfc2822#section-3.6.3
// Unfortunately, Microsoft Outlook uses semicolon (';') by default and
// ',' only optionally. Please see: http://support.microsoft.com/kb/820868
// on how to configure Outlook to support comma-separated lists.
// If your users are mostly using Outlook, you can tell the HRM to use ';'
// instead of the standard ',' for distribution lists.
$email_list_separator = ","; // Valid options are ',' and ';'.
// Authentication type: MYSQL, ACTIVE_DIR or LDAP
$authenticateAgainst = "MYSQL";
?>
Setting the authentication mode to MYSQL enables the embedded user management system (independent of the actual database you are using, e.g. postgresql). Use this if you don’t have any other user management system in place. Alternatively, simple authentication against Microsoft’s Active Directory and OpenLDAP is possible.
In case one of the alternative authentication types are selected (i.e. ACTIVE_DIR or LDAP), some additional configuration will be required (see later).
<?php
...
// If true, use DES password encryption; if false, use MD5
$useDESEncryption = false;
// If true, the queue manager and the image processing server run on the same
// machine
$imageProcessingIsOnQueueManager = true;
// If true the images are copied to the huygens server machine
$copy_images_to_huygens_server = false;
?>
Set $imageProcessingIsOnQueueManager to false if the queue manager is not on the same machine as the Huygens Core that runs deconvolution.
Set $copy_images_to_huygens_server to true if the files have to be copied to the machine where the Huygens Core is installed via ssh (using password-less ssh connection!). If Huygens Core has direct access to the file area (e.g. through NFS mount), set this to false.
<?php
...
// Thumbnails require Huygens Core 3.3.2 or higher.
// Notice that despite some script executions are relatively fast (like
// thumbnail generation and SNR estimation), they still may take a long time if
// the images are very large, or if deconvolution jobs are already running in
// the local server. You may need to increase PHP's max_execution_time
// directive in php.ini, and restart the web server.
// If true, allows generates thumbnails and previews for the originals and the
// results during the deconvolution (i.e. in the computation servers). These
// are maximum intensity projections (MIP) of the stacks along the main axes,
// so they provide xy, xz and zy views.
$useThumbnails = true;
// If true, allows on-demand generation of a preview of the original images in
// this web server prior to the deconvolution. For this to work the variable
// $local_huygens_core must be set correctly above. Notice that not all file
// formats are readable in a freeware Huygens Core and for full functionality
// you may need a license string. Otherwise just let the original thumbnails
// be generated during the deconvolution itself with the previous option.
// If this is true and the correct version of Huygens is installed, the SNR
// estimator tools is also enabled (see $enable_code_for_huygens above).
// This requires $useThumbnails = true;
$genThumbnails = true;
// Moreover, if $genThumbnails is true and Huygens 3.5.1 or higher is
// installed (see $enable_code_for_huygens above), the Signal To Noise ratio
// can be estimated visually from raw images, in the restoration parameters
// editor.
// If larger than zero, generates an AVI movie for 3D stacks that allows
// browsing along the XY slices, and for time series to browse along MIPs in
// time. The movie will have this maximum number of pixels in the largest
// dimension.
// This requires $useThumbnails = true;
$movieMaxSize = 300;
// If true, (and $useThumbnails is also true), it generates not only the MIP
// previews but also Simulated Fluorescence Process top-view renderings of the
// original and the restored images (http://support.svi.nl/wiki/SFP).
$saveSfpPreviews = true;
// If non-zero, save stack and time series previews of the original and restored
// data side by side, to allow comparisons. (Requires $useThumbnails and
// Huygens Core 3.5.2). This is the max size in pixels for each of the images's
// slicer.
$maxComparisonSize = 300;
These were several options for preview generations.
// A shell command that executes ping four times and stops afterwards
$ping_command = 'ping -w 4'; // use this on linux systems
// $ping_command = 'ping -t 4'; // use this on macosx systems
// $ping_command = 'ping'; // use this on cygwin
// The parameter for the ping command after the hostname
$ping_parameter = ''; // use this on linux systems
//$ping_parameter = '56 4'; // use this on cygwin
?>
This is used by the queue manager to test whether the processing machine is reachable.
Note
In HRM it is possible to use a machine for each of the components, i.e. web server, database server, one or more processing machines, file server and queue manager. Configuration files are needed only on the web server and the machine running the queue manager. Make sure to fill the various path and user fields from the perspective of the machine that hosts each of the configuration files!
Configure authentication¶
The HRM curently supports three user authentication mechanisms; if your institution relies on Microsoft’s Active Directory or a generic LDAP server, the HRM can be configured to use them to test user credentials and obtain limited user information (e-mail address angd group). Otherwise, the HRM implements a simple, internal user management module.
The following pages explain the required configuration steps for the support authntication mechanisms.
Internal authentication¶
To enable internal authentication, please set the $authenticateAgainst variable to “MYSQL” in $HRM_CONFIG/hrm_client_config.inc and in $HRM_CONFIG/hrm_server_config.inc
<?php
...
// Authentication type: MYSQL, ACTIVE_DIR or LDAP
$authenticateAgainst = "MYSQL";
...
This is the default value, so nothing needs to be changed in a fresh installation to use internal authentication.
Note
Please notice that you should use "MYSQL" even if you are using PostgreSQL as your database backend!
Active Directory authentication¶
To enable Active Directory authentication, please set the $authenticateAgainst variable to “ACTIVE_DIR” in $HRM_CONFIG/hrm_client_config.inc and in $HRM_CONFIG/hrm_server_config.inc
<?php
...
// Authentication type: MYSQL, ACTIVE_DIR or LDAP
$authenticateAgainst = "ACTIVE_DIR";
...
Then, copy $HRM_SAMPLES/active_directory.config.inc.sample to $HRM_CONFIG/active_directory.config.inc and edit it.
<?php
// This file is part of the Huygens Remote Manager
// Copyright and license notice: see license.txt
// See http://adldap.sourceforge.net/wiki/doku.php?id=api_configuration for
// detailed help on configuring adLdap.
// The account suffix for your domain
$ACCOUNT_SUFFIX = "@mydomain.local";
// The base dn for your domain
$BASE_DN = "DC=mydomain,DC=local";
// Array of domain controllers. Specifiy multiple controllers if you would
// like the class to balance the LDAP queries amongst multiple servers
$DOMAIN_CONTROLLERS = array ("dc01.mydomain.local");
// Domain controller port
$AD_PORT = 389;
// User name suffix
$AD_USERNAME_SUFFIX = "";
// User name suffix matching string
$AD_USERNAME_SUFFIX_REPLACE_MATCH = "";
// User name suffix replacing string
$AD_USERNAME_SUFFIX_REPLACE_STRING = "";
// Optional account with higher privileges for searching (otherwise
// leave it to NULL). This should be set to a domain admin account (only
// read operations are performed!)
$AD_USERNAME = NULL;
$AD_PASSWORD = NULL;
// Tweak to get the real primary group from Active Directory. It works if
// the user's primary group is domain users.
// http://support.microsoft.com/?kbid=321360
$REAL_PRIMARY_GROUP = true;
// Use SSL (LDAPS)
$USE_SSL = false;
// Use TLS: if you wish to use TLS you should ensure that $USE_SSL is
// set to false and vice-versa
$USE_TLS = false;
// When querying group memberships, do it recursively
$RECURSIVE_GROUPS = true;
// Users usually belong to several groups. GROUP_INDEX defines
// which level of the hierarchy to consider (starts by 0). This
// obviously assumes that the sequence of groups memberships is
// constant among users. Set $GROUP_INDEX to -1 to get an array
// with all groups for the user. Please notice that this can also
// be used in combination with the next parameter.
$GROUP_INDEX = 0;
// If there is no way to get the desired group by using $GROUP_INDEX,
// this is an additional little trick. For this to work, $GROUP_INDEX
// MUST be set to -1, and the following array must contain the list of
// valid groups to be considered. This way, for each user the complete
// list of groups will be returned from the Active Directory, and of
// those, the one that is contained in the following array will be picked.
// If more than one matching group is found, the first will be taken. If
// no match is found, all groups will be returned.
// Fill the array as follows: $VALID_GROUPS = array( "group1", "group2" )
// with as many groups as you need.
$VALID_GROUPS = array( );
Warning
For most installations, you can skip the entire rest of this section and proceed with the next one by simply leaving the variables $AD_USERNAME_SUFFIX, $AD_USERNAME_SUFFIX_REPLACE_MATCH and $AD_USERNAME_SUFFIX_REPLACE_STRING with their default values (empty, i.e. "").
This part of the config is meant for the special case of an Active Directory with multiple domains (commonly referred to as forest) which usually consist of a top-level domain (the forest root domain) and several sub-domains. If the HRM users belong to domains in all levels of the forest it would be required to enter the full domain name for logging in, which can become quite tedious (and therefore error-prone).
Consider the following domain setup, a root domain called SATURN with two sub-domains RHEA and TETHYS. Those “plain” domain names are usually referred to as NetBIOS Domain Names. A hierarchical representation could look like this:
SATURN
|
+-- RHEA
|
+-- TETHYS
Now, assuming a top-level suffix of .private, in Microsoft’s Active Directory naming scheme this is translated into fully qualified domain names (FQDN’s) as follows. The root domain (SATURN) gets prefixed by ads. and followed by the top-level .private suffix, whereas the two subdomains just get postfixed by the FQDN of the root domain. The graphical representation from above would therefore translate into this:
ads.saturn.private
|
+-- rhea.ads.saturn.private
|
+-- tethys.ads.saturn.private
To make authentication work in such a scenario, it is required to authenticate against the GLOBAL CATALOG of the Active Directory. This is an LDAP service provided only on the domain controllers of the root domain on the special port 3268. The usernames used to authenticate have to be of the form username@fully.qualified.domain.name to provide the global catalog with the information to which sub-domain a certain account belongs as it is possible to have the same username for different accounts in different domains of a single domain forest.
Now given the HRM configuration option $ACCOUNT_SUFFIX would be set to .saturn.private, a user foo from the domain SATURN could log in using something like foo@ads whereas a user bar from RHEA would have to type bar@rhea.ads. This is obviously screaming for problems as normal users don’t understand where and why to put those weird suffixes, especially as they’ve never heard of something like ads before.
Using the sample config from below, users can simply log on by using their regular account name followed by an @ sign and the (short) NetBIOS domain name, e.g.
foo@saturn
or
bar@rhea
which is way easier to explain and remember for them, especially as the short domain names are usually visible for them when logging into their workstation.
<?php
...
$ACCOUNT_SUFFIX = "";
$BASE_DN = "DC=ads, DC=saturn, DC=private";
// set this to a domain controller of the *ROOT* domain:
$DOMAIN_CONTROLLERS = array("dc01.ads.saturn.private");
// the port is required to connect to the global catalog:
$AD_PORT = 3268;
// give the FQDN of the root domain with a leading dot here:
$AD_USERNAME_SUFFIX = ".ads.saturn.private";
// account names from the root domain will end up wrong, looking like this
// (note the leading "@" and the double occurence of the root domain):
$AD_USERNAME_SUFFIX_REPLACE_MATCH = "@saturn.ads.saturn.private";
// they have to be rewritten using the correct suffix instead (removing the
// first occurence of the root domain name):
$AD_USERNAME_SUFFIX_REPLACE_STRING = "@ads.saturn.private";
?>
Please feel free to contact the HRM developers in case you need help setting up authentication in a multi-domain forest scenario!
LDAP authentication¶
To enable LDAP authentication, please set the $authenticateAgainst variable to “LDAP” in $HRM_CONFIG/hrm_client_config.inc and in $HRM_CONFIG/hrm_server_config.inc
<?php
...
// Authentication type: MYSQL, ACTIVE_DIR or LDAP
$authenticateAgainst = "LDAP";
...
Then, copy $HRM_SAMPLES/ldap.config.inc.sample to $HRM_CONFIG/ldap.config.inc and edit it.
<?php
// This file is part of the Huygens Remote Manager
// Copyright and license notice: see license.txt
// the machine on which the ldap server is running
$ldap_host = "localhost";
// the port for the ldap connection
$ldap_port="389";
// Use SSL (LDAPS)
$ldap_use_ssl = false;
// Use TLS: if you wish to use TLS you should ensure that $ldap_use_ssl is
// set to false and vice-versa
$ldap_use_tls = false;
// the ldap root
$ldap_root="dc=root,dc=country";
// the base for the manager DN (either cn or uid)
$ldap_manager_base_DN="cn";
// the ldap manager (username only!)
$ldap_manager="manager";
// the ldap manager password
$ldap_password = "secret";
// ldap manager OU: use this in case the ldap_manager is in some special OU
// that distinguishes it from the other users. $ldap_manager_ou will be
// prepended to $ldap_user_search_DN.
// Set it to "" if $ldap_user_search_DN can be used for binding
$ldap_manager_ou="ou=special_user";
// User search DN (without ldap root)
$ldap_user_search_DN="cn=users";
// Users often belong to more than one group. To filter for the desired one,
// fill the $ldap_valid_groups array with the ones to be filtered against.
// If more than one group fits, the first will be taken.
// Fill the array as follows: $ldap_valid_groups = array( "group1", "group2" )
// with as many groups as you need.
$ldap_valid_groups = array( );
Set up the HRM database¶
Preparing the database for the HRM consists of two steps, first the corresponding user needs to be created and configured and then the database itself has to be created and prepared for the HRM.
Create the HRM database user¶
Unless the database user for the HRM is already existing, you need to create it and assign the corresponding permissions.
The following command will create a new MySQL user hrm and grant the permissions to a database hrm@localhost using the mysql command line tool. Of course, you can also use a database management tool like phpmyadmin to perform this task if you prefer. Make sure to adjust the password for the new database user so it matches the one from your configuration files.
# start the mysql command line client and connect as root:
mysql -u root -p
# now from within the client, create the database user:
CREATE USER 'hrm'@'localhost' IDENTIFIED BY 'dbpasswd';
# grant permissions to the user:
GRANT ALL ON hrm.* to 'hrm'@'localhost';
exit;
su postgres -c "createuser -e -P -d -A -S -R -N hrm"
su postgres -c "createdb hrm"
MD5 host authentication has to be enabled explicitly on Fedora. This can be done using the following commands:
sudo -s
echo "host all hrm 127.0.0.1/32 md5" >> /var/lib/pgsql/data/pg_hba.conf
service postgresql restart
Create or update the database¶
Run the php script $HRM_SETUP/dbupdate.php from the shell to create and populate the HRM database. The same command can be used to perform an upgrade of the database when the HRM version is upgraded:
cd $HRM_SETUP
php dbupdate.php
If the database does not exist, it will be created using the information stored in the $HRM_CONFIG/hrm_client_config.inc and filled with content for the latest revision. If it exists, it will be updated from whichever revision it currently has.
Review processing server configuration¶
In the table server of the HRM database, two default values are set for the fields name (i.e. the name of the server running Huygens Core) and huscript_path (i.e. the path pointing to the hucore executable).
name has default value localhost, but can also be something like remote.server.com. huscript_path has default value /usr/local/bin/hucore.
If you need to change the default values, you can use your favorite SQL client and execute a query similar to this (for MySQL):
UPDATE server SET name = 'remote.server.com', huscript_path = '/usr/bin/hucore';
Note
The HRM requires the hucore binary to work, not huscript. Before hucore was introduced HRM was indeed using huscript; the huscript_path entry in the database was kept not to break existing HRM installations.
Warning
The server table contains two other entries: status and job. These are used by the interface at runtime, please do not set them manually!
Install the hrmd daemon¶
Use the $HRM_BIN/hrmd daemon in Ubuntu Linux to start the queue manager. The configuration file for the hrmd daemon is /etc/hrm.conf (see Edit hrm.conf)
Install the init script¶
To start the queue manager at boot, copy $HRM_BIN/hrmd to /etc/init.d/. Make sure it is executable by typing in a shell:
sudo cp -v $HRM_BIN/hrmd /etc/init.d/hrmd
sudo chmod +x /etc/init.d/hrmd
Start / stop the daemon at boot / shutdown¶
Ubuntu: To set up the appropriate run-level links for starting and stopping the daemon during boot and shutdown, use this command:
sudo update-rc.d hrmd defaults
This will configure your system to run the init script in the default run-levels. Please note that issuing this command does NOT actually start the daemon until the next reboot. To start it right now follow the steps in the section below.
Fedora: Coming soon.
Start manually¶
Make sure to set the proper value of SUSER in /etc/hrm.conf (see Edit hrm.conf).
Start a shell and type:
sudo /etc/init.d/hrmd start
If the queue manager started correctly, you should see:
Reporting to /var/log/hrm/log.txt /var/log/hrm/error_log.txt
Starting HRM daemon...................ok
The queue manager can be started, stopped and restarted by using:
sudo /etc/init.d/hrmd start
sudo /etc/init.d/hrmd stop
sudo /etc/init.d/hrmd restart
Change the admin password!¶
The admin user can access the administration backend of the HRM with the username and default password specified in the $HRM_RESRC/passwd.README file.
Please, immediately go to the account page and change the admin password!
Optional configuration steps¶
Customize the login page¶
You can customize the login page by renaming the $HRM_USER/login_user.inc.sample file into $HRM_USER/login_user.inc and adding information that you would like to appear on the HRM login page, such as contact information, quick instructions, etc.
Warning
Please use valid HTML code: no validation is performed on your $HRM_USER/login_user.inc HTML code, and you are responsible for any vulnerability you introduce into the HRM!
Example¶
<h2>Contact information</h2>
<ul>
<li><a href="mailto:admin@facility.edu?Subject=Help!">HRM admin</a></li>
</ul>
<h2>HRM custom instructions</h2>
<ul>
<li><a href="http://www.facility.edu/hrm_instr.pdf">Using the HRM at our facility</a></li>
</ul>
With a bit of HTML and CSS skills you can make your login page shine!
OMERO connector¶
Please make sure to have the OMERO prerequisites set up correctly. To enable the OMERO connector, then set the $omero_transfers variable to true in $HRM_CONFIG/hrm_client_config.inc and in $HRM_CONFIG/hrm_server_config.inc.
<?php
...
//======================================================================
// HRM + Omero
//======================================================================
// Switch on/off (true/false) data transfers between HRM and Omero.
$omero_transfers = true;
?>
In addition, the following values need to be set accordingly in /etc/hrm.conf. Assuming your OMERO server is running on a host called omero.mynetwork.xy and using the versions shown in the example in the OMERO prerequisites, this would look as below. Please make sure to adjust OMERO_PKG and OMERO_HOSTNAME so it is matching your current setup.
# Interaction with OMERO (if switched on in hrm/config).
OMERO_PKG="/opt/OMERO/OMERO.server-5.0.3-ice34-b41.linux"
OMERO_HOSTNAME="omero.mynetwork.xy"
OMERO_PORT="4064"
Configure the HRM (multi-server)¶
This section describes the specific steps required to configure the HRM to run on multiple machines. In particular, we address the following setup:
- Machine 1: runs the web server (and database) and the queue manager
- Machine 2: runs Huygens Core
- The data folder is not directly shared (via NFS or Samba) between the two machines.
Note
There could be more than on processing machine running Huygens Core in your institution: set up all additional machines like Machine 2.
Configuration steps¶
Public-key authentication¶
Image data for deconvolution is transferred from Machine 1 to Machine 2 via secure copy (scp). For this, a specified user on Machine 2 must be created/chosen, and remote access from Machine 1 via password-less, public key authentication must be set up for this user.
Note
For the sake of the example, we will call this specific user hrm. You can pick another user, just make sure to replace hrm with your user name in what follows.
On Machine 2, create a user with name hrm and group hrm that will be used for remote authentication and data transfer. Configure password-less, public key authentication as explained on this page. The public key is uploaded to Machine 2, whereas the private key is stored on Machine 1 for the selected user (e.g. in /home/hrm/.ssh/).
HRM configuration files¶
On Machine 1, configure the HRM as usual (as in the single-server configuration) but set the following variables in $HRM_CONFIG/hrm_client_config.inc and $HRM_CONFIG/hrm_server_config.inc as shown:
<?php
...
$copy_images_to_huygens_server = true;
$imageProcessingIsOnQueueManager = false;
$huygens_user = "hrm" // This is the user that will log in to Machine 2
$huygens_group = "hrm" // And this is its group
Replace hrm with your own user name and group if different.
Processing server¶
On Machine 1, add the name of Machine 2 and the absolute Huygens Core path on that machine to the server table of the HRM database (see Review processing server configuration).
Note
If you have more than one processing server, add each server to the table in a separate row.
Data folder¶
On Machine 2, create the data folder pointed at by $huygens_server_image_folder. This folder should be owned by $huygens_user (e.g. “hrm”) and have group ownership as set by $huygens_group (e.g. “hrm”).
Warning
As of version 3.2.0, the HRM expects specific file permissions on the file area! Please see Update the data folder permissions below.
Version upgrade¶
If you have upgraded the HRM in the past, you will know that some steps must be performed in addition to replacing the old HRM code with the new one: some entries might have been added or changed in the configuration files (hrm_{server|client}_config.inc), and the database structure might have been changed.
Stop the Queue Manager¶
Significant parts of the configuration as well as the database are usually changed during an upgrade, so the Queue Manager needs to be stopped first.
sudo /etc/init.d/hrmd stop
In some rare situations, the Queue Manager might get stuck. To ensure the stop command did work properly, run the following check which should return empty:
ps aux | grep -i [r]unHuygens
Download and extract the new HRM release¶
To install the new HRM version you need to download the .tar.gz file from the website or github as explained in downloading the standard archive.
The downloaded package then needs to be extracted inside the HRM’s installation directory, overwriting updated files but not touching your configuration files etc. Assuming your downloaded version is 3.2.0 and you were placing the package in your home directory, this can be done like this:
cd $HRM_HOME
tar xzf $HOME/hrm-3.2.0.tar.gz --strip=1
Cleaning up leftover files from previous installations¶
Sometimes files were part of a previous release of the HRM but they are not contained in the current one any more. With the above described method those files don’t get removed as only new files are extracted from the tar package. To avoid cluttering up the installation they should be removed according to the versions involved.
3.1 to 3.2¶
rm -v inc/ActiveDirectory.inc.php inc/Ldap.inc.php
Upgrade the init script¶
This step is more or less identical to the initial installation of the init script as described in installing the daemon. You need to copy the new script from $HRM_BIN/hrmd to /etc/init.d/ and make sure it is executable. This can be done using the following commands:
sudo cp -v $HRM_BIN/hrmd /etc/init.d/
sudo chmod +x /etc/init.d/hrmd
Note
There should be NO need to run update-rc.d to update the run-level links.
Update the data folder permissions¶
As of HRM 3.2.0, the system users running the Queue Manager and the web server are expected to have full read-write access to $HRM_DATA. The supported way of doing this is explained in setting up the HRM user and group. Briefly:
- a user hrm and its corresponding group hrm are created
- the web server user ( www-data, apache) is added to the hrm group
- the variable SUSER is set to hrm in /etc/hrm.conf
- the $HRM_DATA and $HRM_LOG group ownership is set to hrm with the setgid bit set
For detailed instructions, please see setting up the HRM user and group.
Note
Temporarily, the old behavior can still be preserved by setting a configuration variable $change_ownership as explained in the next section.
Update the configuration files¶
3.1 to 3.2¶
In version 3.2 of the HRM, the system users running the Queue Manager and the web server are expected to have direct read-write access to the data folders. If this is not the case for your setup and you rely on adding the web server user to /etc/sudoers, please notice that this behavior is obsolete in 3.2 but can still be enabled by adding:
$change_ownership=true;
in hrm_{server|client}_config.inc. The variable $change_ownership defaults to false if not explicitly set to true in the configuration, in compliance to the new behavior.
Note
As of HRM 3.3, this variable will be ignored and the new behavior will be enforced!
3.0.x to 3.1¶
No changes in the configuration files.
2.1.x to 3.0¶
From HRM 2.1.x to HRM 3.0, one variable was added in the configuration files.
$omero_transfers={true|false}
1.2.x to 2.0¶
From HRM 1.2.x to HRM 2.0, three variables were added in the configuration files:
[+] max_upload_limit
[+] max_post_limit
[+] email_list_separator
Moreover, three variables were removed:
[-] resultImagesOwnedByUser
[-] resultImagesRenamed
[-] enable_code_for_huygens
Note
If you are upgrading straight from HRM 1.2.x, please notice that as of HRM 2.0 configuration and sample files were moved as per following table.
Config files (new) | Sample files (new) | Config files (1.x) | Sample files (1.x) |
---|---|---|---|
$HRM_HOME/config | $HRM_HOME/config/samples | $HRM_HOME/inc | $HRM_HOME/resources |
Check the configuration files¶
An easy way to check for modifications is by running the $HRM_HOME/resources/checkConfig.php script. From the shell, run:
cd $HRM_HOME
php resources/checkConfig.php config/hrm_server_config.inc
Replace $HRM_HOME with the hrm root (e.g /var/www/hrm).
Checking the 3.0.x files with the 3.1 checkConfig.php script will result in the following output:
Check against HRM v3.1.x.
Check completed successfully! Your configuration file is valid!
Checking the 2.1.x files with the 3.0 checkConfig.php script will result in the following output:
Check against HRM v3.0.x.
* * * Error: variable omero_transfers not set or empty.
Check completed with errors! Please fix your configuration!
Checking the 1.2.x files with the 2.1.x checkConfig.php script will result in the following output:
Check against HRM v2.1.x.
* * * Error: variable max_upload_limit not set or empty.
* * * Error: variable max_post_limit not set or empty.
* * * Error: variable email_list_separator not set or empty.
* * * Error: variable resultImagesOwnedByUser must be removed from the configuration files!
* * * Error: variable resultImagesRenamed must be removed from the configuration files!
* * * Error: variable enable_code_for_huygens must be removed from the configuration files!
Check completed with errors! Please fix your configuration!
Please make sure to fix all problems. The sample files and the Manual installation instructions will help you set the correct parameters.
Update the database¶
Newer versions of the HRM might use slightly different/updated versions of the database back-end than previous ones.
HRM version | Database version |
---|---|
1.2.3 | 7 |
2.0 | 8 |
2.1 | 9 |
3.0 | 10 |
3.0.3 | 11 |
3.1 | 12 |
3.2 | 13 |
For this reason, the first time you run the HRM after an update you will be told that the database must be updated and that you are not allowed to continue until this has been done!
Note
Database updates are supported across HRM versions, i.e. it is possible to upgrade the database from revision 7 to 13 in one step.
The following describes two possible ways to update the database.
Note
Although we test this procedure quite carefully, it is highly recommended to backup the database before updating!
Updating from the web interface¶
Login to the HRM as the admin user: you will be brought directly to the Database update page. Click on the update button. If everything works properly (as it should...), the following message should be displayed.
Needed database revision for HRM v3.2 is number 13.
Current database revision is number 12.
Updating...
Database successfully updated to revision 13.
The database is now at the latest revision.
Updating from the console¶
Alternatively, the database can be updated from the console (see create or update database). Please pay attention to what the update process will report! The output should be the same as the one listed in the previous section, but if the update fails, you might want to report it.
Re-start the Queue Manager¶
After processing the described upgrade steps, the Queue Manager needs to be started again.
sudo /etc/init.d/hrmd start