VisiOmatic documentation¶

Data management¶

All the data for a given image data cube are stored in a single, huge BigTIFF file. The initial conversion from the FITS astronomical image format to BigTIFF is handled by the STIFF software [7]. On modern hardware, the current STIFF conversion rate for transcoding a FITS file to an IIPImage-ready tiled pyramidal BigTIFF ranges from about 5Mpixel/s to 25Mpixel/s (20-100MB/s) depending on the chosen TIFF compression scheme and system I/O performance. Hence in principle FITS frames with dimensions of up to 16k×16k can be converted in a matter of seconds, and just-in-time conversion could be a viable option for such images.

Vector data are stored in GeoJSON format. GeoJSON offers a compact, yet human-friendly representation of features such as markers, lines, polygons, and can easily be generated from e.g., CSV or ds9 region files.

Security¶

Data security is a major issue for online services. In IIPImage, three mechanisms allow the webmaster to control the dissemination of data:

• Paths where the data files are located are not accessible from the web; they are only accessible by the FCGI application on the server, and absolute or relative access through upper levels of the directory tree is forbidden.
• The same origin security policy in web browsers restricts data access to servers located in the same domain as the server hosting the client code, and prevents Javascript applications on websites from other domains sending AJAX requests directly to the IIPImage server. If however data sharing with other domains is a desired feature, IIPImage implements the CORS mechanism to allow such requests from selected or all external websites.
• As other tile-based image servers, IIPImage itself would not prevent a user to download a complete high resolution JPEG image through small pieces, using a dedicated script. If this is an issue, IIPImage offers a configurable on-the-fly, randomized digital watermarking feature.

Performance¶

IIPImage is known for being particularly efficient. Since 2013 a significant amount of work has gone into optimizing further the C++ server code, by streamlining the processing of vectorized floating-point data and improving I/O performance with Terabyte-sized datasets [1]. Current server code is able to serve from about 50 to 500 256×256 JPEG tiles per second per CPU core (2.5GHz). The output image stream from a single 16-core server under heavy load (tens of thousands of tile requests per second) is thus capable of saturating a 1 Gbit/s connection.

Provided that the client interface restricts the number of operations possible with the data, caching image tiles is an efficient way of increasing the throughput of the system. In addition to the caching of image tiles done client-side in the browser, IIPImage offers two explicit levels of tile caches server-side:

• In memory, integrated within the application
• Distributed, thanks to the MemCached caching system.

Channel Mixing ¶

The Channel Mixing panel has two modes, which can be selected using the radio buttons located at the top of the dialog: the Single Channel (monochromatic) mode, and the Multi-Channel (color composite) mixing mode.

In Single Channel mode, one can:

• Select an image channel for display
• Select a color map among a selection of four
• Set the minimum and maximum channel levels

In Multi-Channel mode, one can:

• Select an image channel to be included in the color mix
• Set the color this channel contributes to the mix
• Set the minimum and maximum channel levels
• Click on a channel name in the active channel list to edit a channel contributing to the current mix, or click on the trashcan button to remove it.

Image Preferences ¶

The Image Preference dialog gives access to global image display settings:

• Color map inversion (negative mode)
• Contrast (scaling factor)
• Color saturation (0 for black&white, >1 for exaggerating colors)
• Gamma correction (2.2 for linear output on a properly calibrated monitor, higher values for brightening dark regions, lower values for darkening)
• JPEG quality percentage. The lower the quality percentage, the more compressed the image and the more artifacts in the rendering.

Catalog Overlays ¶

The Catalog Overlay dialog allows the user to download and overlay catalogs in the current field of view. The available list of catalogs and the rendering of catalog sources (marker, cross, circle with magnitude-dependent radius, ellipse, etc.) depends on client settings. In the current default interface all catalogs are queried from the VizieR service at CDS [5].

To query a catalog, move the map and adjust the zoom level to the desired field of view; choose the catalog from the drop-down catalog selector and an overlay color from the color selector, then click the “GO” button. After a few seconds a new overlay with the chosen color should appear, as well as a new entry in the active catalog list below the drop down selector. Each overlay may be turned off or on by clicking on the check-mark in the corresponding entry of the active catalog list, or simply discarded by clicking on the trashcan button.

Depending on the implementation in the client, sources may be clickable and have pop-up information windows attached.

Region Overlays ¶

The Region Overlay dialog allows the user to overlay Points/Regions Of Interest, such as local catalogs, detector footprints, etc. These POIs/ROIs may be clickable and have information attached for display in pop-up windows.

The Region Overlay selection mechanism is exactly the same as that of Catalog Overlays.

Profile Overlays ¶

The Profile overlay dialog gives the user the possibility to extract pixel values directly from the scientific data. These data are unaffected by channel mixing, image scaling or compression.

The profile option extracts series of pixel values along lines of constant galactic longitude or latitude. The line color is pink by default and can be changed using the color picker. The line itself is positioned on the image by first dragging the map to the desired start coordinate, pressing the “START” button, and dragging the map to the desired end coordinate and pressing the “END” button. After some calculation, a window appears with a plot of the image profile along the selected line. In mono-channel mode, a single line is plotted that corresponds to the currently selected channel. In color mode, all active channels are plotted, with their channel mixing color. On devices equipped with a mouse one can zoom inside the plots by clicking with the left button and selecting the zoom region. Double-click to zoom out. The plot window can be closed by clicking/touching the small cross in the upper right corner, and reopened at any time by clicking on the line.

The spectrum option, as its name implied extracts pixel data along the channel axis at a given position. The position is selected by dragging the map to the desired coordinate and clicking on the “Go” button. A circle marker appears with the color that was selected using the color picker (purple by default). After some calculation, a window pops out with the “spectrum” of pixel values at the selected coordinate. Note that the ordering of pixel values follows that of the channels in the data cube.

Documentation panel ¶

Clicking on the symbol (which may be located at the bottom of the taskbar) brings up the online documentation panel. A navigation bar is located at the bottom of the panel to facilitate browsing through the provided documentation or website sections and come back to the main page. Finally, a download button (PDF Symbol ) located at the bottom right of the Manual Pane may be present to allow the user saving an entire manual as a PDF file for offline reading.

L.TileLayer.IIP¶

L.TileLayer.IIP manages tile layers with image data queried from an IIPimage tile server.

var map = L.map('map', {fullscreenControl: true});
var iip = '/fcgi-bin/iipsrv.fcgi?FIF=image.ptif';
var ima = L.tileLayer.iip(iip, {cmap: 'jet'}).addTo(map);

L.CRS.WCS¶

L.CRS.WCS is a new class that extends the original L.CRS object to support celestial projections described by the FITS WCS standard. Currently only the PIXEL (Cartesian), CAR, COE, TAN and ZEA projections are supported, in equatorial, ecliptic, galactic, and supergalactic coordinates.

var wcs = L.CRS.wcs(header);

L.Control.IIP.Channel¶

L.Control.IIP.Channel is a new control class for managing the channel mixing interface.

var control = L.control.iip.channel({cMap: 'jet'}).addTo(map);

L.Control.IIP.Image¶

L.Control.IIP.Image is a new control class for managing the image settings interface.

var control = L.control.iip.image().addTo(map);

L.Control.IIP.Catalog¶

L.Control.IIP.Catalog is a new control class for managing the interface that controls catalog queries and catalog overlays.

var control = L.control.iip.catalog([L.Catalog['2MASS'], L.Catalog['SDSS'],
              L.Catalog['Hudelot']]).addTo(map);

L.Control.IIP.Profile¶

L.Control.IIP.Profile is a new control class for managing the interface that controls profile extraction and profile overlays.

var control = L.control.iip.profile().addTo(map);

L.Control.IIP.Region¶

L.Control.IIP.Region is a new control class for managing the region interface.

var control = L.control.iip.region([
  {
    name: 'Region 1',
    description: 'A first region',
    url: 'region1.json',
    color: 'blue',
    load: false
  },{
    name: 'Region 2',
    description: 'A second region',
    url: 'region2.json',
    color: 'orange',
    load: false
    }
  ],
  { nativeCelsys: true }
).addTo(map);

{"type":"FeatureCollection",
  "features": [
    {
      "type":"Feature",
      "geometry": {
        "type":"Polygon",
        "coordinates": [[
          [266.46398,-27.94870],[267.43523,-28.46585],
          [266.47605,-29.85244],[265.49663,-29.32837],[266.46398,-27.94870]
        ]]
      },
      "properties": {
        "description":"Limits of region 1."
      }
    },{
      "type":"Feature",
      "geometry": {
        "type":"Point",
        "coordinates": [266.46042,-28.82444]
      },
      "properties": {
        "description":"<h2>A Point of Interest</h2><p>This object should be worth monitoring.</p>"
      }
    }
  ]
}

L.Control.IIP.Doc¶

L.Control.IIP.Doc is a new control class for displaying online documentation.

var control = L.control.iip.doc('mydoc/index.html', {pdflink: 'mydoc/mydoc.pdf'}).addTo(map);

L.Control.Sidebar¶

L.Control.Sidebar is a control class for adding sliding panes to the left or the right of the map. It is derived from the leaflet-sidebar plugin by Tobias Bieniek.

var sidebar = L.control.sidebar().addTo(map);

L.Control.WCS¶

L.Control.WCS is a new control class for managing an input/output coordinate widget.

var control =  L.control.wcs(
  {coordinates: [
    {label: 'RA,Dec', units: 'HMS'},
    {label: 'Gal l,b', units: 'deg', nativeCelsys: true}
  ]}
).addTo(map);

L.Control.Scale.WCS¶

L.Control.Scale.WCS is a new control class derived from L.Control.Scale that adds a scale to the map. It supports both angular and pixel units.

var control = L.control.scale.wcs({pixels: false}).addTo(map);

L.Control.Reticle¶

L.Control.Reticle is a new control class that adds a crosshair at the center of the map window.

var control = L.control.reticle().addTo(map);

L.Control.ExtraMap¶

L.Control.ExtraMap is a new control class for displaying an additional map synchronized with the main map, picture-in-picture style. It is an adaptation of Leaflet-MiniMap by Norkart.

var control = L.control.extraMap(
  L.tileLayer.iip('/fcgi-bin/iipsrv.fcgi?FIF=image.ptif'),
    {
      position: 'topright',
      width: 192,
      height: 128,
      zoomLevelOffset: -6,
    }
).addTo(map);

L.RGB¶

L.RGB is a new class for managing R,G,B color triplets that describe color pixels.

var rgb = L.rgb(0.2,0.4,0.0);
var rgb = L.rgb([0.2,0.4,0.0]);
var rgb = L.rgb('#336600');

L.Ellipse¶

L.EllipseMarker¶

L.Ellipse and L.EllipseMarker are new classes for drawing ellipse overlays on a map, with parameters defined in world and pixel coordinates, respectively.

Usage example

var path = L.ellipse(latLng, {
  majAxis: 0.0210, minAxis: 0.0045, posAngle: -30.0
}).addTo(map);

var path = L.ellipseMarker(latLng, {
  majAxis: 10.0, minAxis: 3.0, posAngle: 24.0
}).addTo(map);

L.IIPUtils.requestURL()¶

L.IIPUtils.requestURL() is used throughout VisiOmatic to send Ajax requests to a given URL.

L.IIPUtils.requestURL(
  'http://cdsweb.u-strasbg.fr/cgi-bin/nph-sesame/-oI/A?M31',
  'getting coordinates for M31',
  function (context, httpRequest) { console.log(context, httpRequest); },
  this,
  10
);

L.IIPUtils.parseURL()¶

L.IIPUtils.parseURL() parses a URL and returns a dictionary of query string keyword/value pairs.

args = L.IIPUtils.parseURL(
  'http://myviewer.org/?channel=2&mode=mono'
);
console.log(args['channel'], args['mode']);

L.IIPUtils.checkDomain()¶

L.IIPUtils.checkDomain() parses a URL and returns the domain name associated with it.

domain = L.IIPUtils.checkDomain('http://myviewer.org/test');

L.IIPUtils.isExternal()¶

L.IIPUtils.isExternal() parses a URL and returns true if it belongs to the current domain or false otherwise.

flag = L.IIPUtils.isExternal('http://myviewer.org/doc/');

L.IIPUtils.copyToClipboard()¶

L.IIPUtils.copyToClipboard() copies the given string to clipboard.

flag = L.IIPUtils.copyToClipboard('http://myurl');

L.IIPUtils.flashElement()¶

L.IIPUtils.flashElement() temporily highlights a DOM element using a short (<400ms) “flashing” animation.

flag = L.IIPUtils.flashElement(myInput);

Hardware¶

The IIPImage server (iipsrv) is meant to run on one or several server machines, with multiple CPU cores. The “astronomy-oriented” version of iipsrv is largely vectorized but it is single threaded; multiple CPU cores are taken advantage of through multiple instances spawned by the web server.

iipsrv operates on large data files (image data cubes), which can be as large as several Terabytes. The performance of the astro version is often I/O-limited, especially when it comes to latency. It is therefore important that the data files be located on a fast storage system, with low access times. Ideally this will be an enterprise-level array of SSDs. The lower performance of slower devices, such as spinning disks, can partially be compensated with random access memory by taking advantage of the operating system caching of file operations, as well as the built-in iipsrv caching mechanisms. In all cases, the more memory available for caching there is on the system, the more responsive the server will be under heavy load.

In what follows, we will assume that the image data reside in the /raid/array/ directory.

Operating System¶

This installation guide focuses on Linux. Nevertheless IIPImage has been designed to be cross-platform and has been successfully tested on Linux, Sun Solaris, Mac OS X and Windows.

Software¶

Before starting the installation, one should make sure that “development packages” (coming with header files) of the following libraries have been installed on the server:

• LibFCGI v2.4+
• LibJPEG Turbo v1.2+, or libJPEG
• LibTIFF v4.0+
• zLib v1.2+

Note that iipsrv relies on the BigTIFF format for managing image data files larger than 2GB. If BigTIFF support is not included in the LibTIFF packages available for your Linux distribution (e.g., because it is too old), a manual installation of LibTIFF v4.0+ may be necessary.

Assessing server performance¶

In [1], four HTTP server packages — Apache v2.4.1, lighttpd v1.4.35, NGINX v1.4.7 and OpenLiteSpeed v1.2.7 — were benchmarked with iipsrv. Fig. 1 shows how efficiently each of these servers responds to bursts of tile queries for various levels of concurrency (the number of queries in each burst). Tile queries were simulated using a modified version of the Apache Benchmarking tool benchmarking tool on a local machine, connected through a 10GbE link. The result is expressed in terms of throughput (the number of tiles per seconds) and latency (the average time it takes for tiles to be returned). Perfect server behaviour would consist of constant throughput, and latency that increases linearly with concurrency beyond some threshold, as queries are queued inside the server pipeline.

IIPImage-astro tile-serving throughput and latency as a function of concurrency for four different HTTP servers running on the same 12-core Linux system.

As can be seen, HTTP server packages behave very differently in this test. If we set the limit for acceptable latency to 1 second, we see that a 12-core system is able to manage bursts of up to 2,000 simultaneous queries with OpenLiteSpeed, which corresponds to about 100 users frantically browsing through the image. With Apache, these numbers are ten times lower, and both throughput and latency behave much more erratically.

All HTTP server packages were configured so as to maximize tile serving performance. Although some aspects may have not been fully optimized, OpenLiteSpeed comes out as the clear winner in this test. This is why we recommend it for running IIPImage. Nevertheless, in the sections below we also provide configuration guides for all four packages.

Apache¶

The mod_fastcgi module must be installed and enabled. A directory containing FastCGI programs should be created and readable by Apache processes. Make sure that your Apache configuration file (or the configuration file for the FCGI module) contains the following lines

LoadModule fastcgi_module /path/to/apachemodules/mod_fastcgi.so
# Create a directory for the iipsrv binary
ScriptAlias /fcgi-bin/ "/path/to/fcgi/directory/fcgi-bin/"
#
# Set the options on that directory
<Directory "/path/to/fcgi/directory/fcgi-bin/">
 AllowOverride None
 Options None
 # Syntax for access is different in Apache 2.4 - uncomment appropriate version
 # Apache 2.2
 #   Order allow,deny
 #   Allow from all
 #
 # Apache 2.4
 Require all granted
</Directory>

Finally, the iipsrv configuration file, iipsrv.conf must be copied in the Apache configuration directory (e.g., /etc/httpd/conf.d/). The following iipsrv.conf features typical settings for a 12-core machine:

# Set our environment variables for the IIP server
FcgidInitialEnv VERBOSITY "0"
FcgidInitialEnv LOGFILE "/tmp/iipsrv.log"
FcgidInitialEnv MAX_IMAGE_CACHE_SIZE "100"
FcgidInitialEnv JPEG_QUALITY "90"
FcgidInitialEnv MAX_CVT "3000"
FcgidInitialEnv MEMCACHED_SERVERS "localhost"
FcgidInitialEnv FILESYSTEM_PREFIX "/raid/iip/"
# Define the idle timeout as unlimited and the number of # processes we want
FcgidIdleTimeout -1
FcgidMaxProcessesPerClass 12

lighttpd¶

lighttpd comes with built-in FastCGI support. To configure iipsrv, add in your lighttpd directory an ASCII file iipsrv.conf containing

fastcgi.server = ( "/fcgi-bin/iipsrv.fcgi" =>
      (( "host" => "127.0.0.1",
              "port" => 9000,
              "check-local" => "disable",
              "min-procs" => 1,
              "max-procs" => 12,
              "bin-path" => server_root + "/fcgi-bin/iipsrv.fcgi",
              "bin-environment" => (
                      "LOGFILE" => log_root + "/iipsrv.log",
                      "VERBOSITY" => "3",
                      "MAX_IMAGE_CACHE_SIZE" => "100",
                      "FILENAME_PATTERN" => "_pyr_",
                      "JPEG_QUALITY" => "90",
                      "MAX_CVT" => "3000",
                      "MEMCACHED_SERVERS" => "localhost",
                      "FILESYSTEM_PREFIX" => "/raid/iip/"
              )
      ))
)

For best performances edit the lighttpd server settings in /etc/lighttpd/lighttpd.conf:

server.use-ipv6 = "disable"
server.document-root = server_root + "/html"
server.max-fds = 300000
server.stat-cache-engine = "fam"
server.max-connections = 100000
server.max-keep-alive-idle = 4
server.max-keep-alive-requests = 4

Then restart the lighttpd server.

NGINX¶

OpenLiteSpeed¶

The latest stable version of the OpenLiteSpeed web server (lsws) can be downloaded from http://open.litespeedtech.com/. On most machines, installing lsws from the source package is as simple as

$ ./configure
$ make
$ sudo make install

On RedHat-like systems (e.g., RedHat, Fedora, CentOS,...), the OpenLiteSpeed web server is started with

$ service lsws start

OpenLiteSpeed comes with a graphical web interface which is by default accessible on port 7080 of the server. The default administrator login and password are admin and 123456. Any change to the server configuration made in the interface requires applying a “Graceful Restart” (follow instructions on the web page).

Configuring OpenLiteSpeed for iipsrv starts by adding a new “virtual host”, or modifying the default one that comes with OpenLiteSpeed (Fig. 2).

A single “virtual host” is sufficient for operating iipsrv.

The virtual host menu has several tabs. For this FastCGI application we are mostly concerned with the Basic, External App and Context tabs.

In the Basic tab, one should change the Virtual Host Name field to, e.g., iipsrv-vhost, and set ExtApp Set UID Mode to DocRoot UID (see Fig. 3).

Example of a configuration for the Basic tab in the Virtual Host section.

The External App tab is where the FastCGI executable must be defined and where it can be fine-tuned. Fig. 4 shows an example of a configuration which is appropriate for a 12-core server (Max Connections = 12 and Instances = 12) with high throughput (Connection Keepalive Timout = 5).

Example of a configuration for the External App tab in the Virtual Host section.

Finally, an FCGI entry should be added in the Context tab (Fig. 5).

Example of a FastCGI entry in the Context tab of the Virtual Host section.