spec2nexus

Converts SPEC data files and scans into NeXus HDF5 files:

$ spec2nexus  path/to/file/specfile.dat

Writes path/to/file/specfile.hdf5

Provides

• spec2nexus : command-line tool: Convert SPEC data files to NeXus HDF5
• extractSpecScan : command-line tool: Save columns from SPEC data file scan(s) to TSV files
• spec : library: python binding to read SPEC data files
• eznx : library: (Easy NeXus) supports writing NeXus HDF5 files using h5py
• specplot : command-line tool: plot a SPEC scan to an image file
• specplot_gallery : command-line tool: call specplot for all scans in a list of files, makes a web gallery

Package Information

• author: Pete R. Jemian
• email: prjemian@gmail.com
• copyright: 2014-2020, Pete R. Jemian
• license: Creative Commons Attribution 4.0 International Public License (see LICENSE.txt file)
• URL: documentation: http://spec2nexus.readthedocs.io
• git: source: https://github.com/prjemian/spec2nexus
• PyPI: Distribution: https://pypi.python.org/pypi/spec2nexus/
• OpenHub: Compare open source software: https://www.openhub.net/p/spec2nexus
• version: 2021.1.8rc5
• release: 1.g291d6bf.dirty
• published: Nov 10, 2020

Contents

spec2nexus

Converts SPEC data files and scans into NeXus HDF5 files.

How to use spec2nexus

Convert all scans in a SPEC data file:

$ spec2nexus  path/to/file/specfile.dat

Writes path/to/file/specfile.hdf5 (Will not overwrite if the HDF5 exists, use the -f option to force overwrite).

show installed version

Verify the version of the installed spec2nexus:

$ spec2nexus  -v
2014.03.02

command-line options

  user@host ~$ spec2nexus.py -h
  usage: spec2nexus [-h] [-e HDF5_EXTENSION] [-f] [-v] [-s SCAN_LIST] [-t]
                    [--quiet | --verbose]
                    infile [infile ...]

  spec2nexus: Convert SPEC data file into a NeXus HDF5 file.

  positional arguments:
    infile                SPEC data file name(s)

  optional arguments:
    -h, --help            show this help message and exit
    -e HDF5_EXTENSION, --hdf5-extension HDF5_EXTENSION
                          NeXus HDF5 output file extension, default = .hdf5
    -f, --force-overwrite
                          overwrite output file if it exists
    -v, --version         show program's version number and exit
    -s SCAN_LIST, --scan SCAN_LIST
                          specify which scans to save, such as: -s all or -s 1
                          or -s 1,2,3-5 (no spaces!), default = all
    --quiet                     suppress all program output (except errors), do not
                          use with --verbose option
    --verbose                   print more program output, do not use with --quiet
                          option

Note

Where’s the source code to spec2nexus?

In the source code, the spec2nexus program is started from file nexus.py (in the spec2nexus.nexus.main() method, for those who look at the source code):

$ python nexus.py specfile.dat

You’re not really going to call that from the source directory, are you? It will work, if you have put that source directory on your PYTHONPATH.

---

source code documentation

Converts SPEC data files and scans into NeXus HDF5 files

spec2nexus.nexus.get_user_parameters()

configure user’s command line parameters from sys.argv

spec2nexus.nexus.main()

entry point for command-line interface

spec2nexus.nexus.parse_scan_list_spec(scan_list_spec)

parses the argument of the -s option, returns a scan number list

spec2nexus.nexus.pick_scans(all_scans, opt_scan_list)

edit opt_scan_list for the scans to be converted

To be converted, a scan number must be first specified in opt_scan_list and then all_scans is checked to make sure that scan exists. The final list is returned.

extractSpecScan

Command line tool to extract scan data from a SPEC data file.

How to use extractSpecScan

Extract one scan from a SPEC data file:

user@host ~$ extractSpecScan data/APS_spec_data.dat -s 1 -c mr USAXS_PD I0 seconds

the usage message:

user@host ~$ extractSpecScan
usage: extractSpecScan [-h] [-v] [--nolabels] -s SCAN [SCAN ...] -c COLUMN
                       [COLUMN ...] [-G] [-P] [-Q] [-V] [--quiet | --verbose]
                       spec_file

the version number:

user@host ~$ extractSpecScan -v
2017.0201.0

the help message:

user@host ~$ extractSpecScan -h
usage: extractSpecScan [-h] [-v] [--nolabels] -s SCAN [SCAN ...] -c COLUMN
                       [COLUMN ...] [-G] [-P] [-Q] [-V] [--quiet | --verbose]
                       spec_file

Save columns from SPEC data file scan(s) to TSV files URL:
http://spec2nexus.readthedocs.org/en/latest/extractSpecScan.html v2016.1025.0

positional arguments:
  spec_file             SPEC data file name(s)

optional arguments:
  -h, --help            show this help message and exit
  -v, --version         print version number and exit
  --nolabels            do not write column labels to output file (default:
                        write labels)
  -s SCAN [SCAN ...], --scan SCAN [SCAN ...]
                        scan number(s) to be extracted (must specify at least
                        one)
  -c COLUMN [COLUMN ...], --column COLUMN [COLUMN ...]
                        column label(s) to be extracted (must specify at least
                        one)
  -G                    report scan Geometry (#G) header information
  -P                    report scan Positioners (#O & #P) header information
  -Q                    report scan Q (#Q) header information
  -V                    report scan (UNICAT-style #H & #V) header information
  --quiet               suppress all program output (except errors), do not
                        use with --verbose option
  --verbose             print more program output, do not use with --quiet
                        option

Example

Extract four columns (mr, USAXS_PD, I0, seconds) from two scans (1, 6) in a SPEC data file:

$ extractSpecScan data/APS_spec_data.dat -s 1 6 -c mr USAXS_PD I0 seconds

program: /path/to/extractSpecScan.py
read: data/APS_spec_data.dat
wrote: data/APS_spec_data_1.dat
wrote: data/APS_spec_data_6.dat

Here’s the contents of data/APS_spec_data_6.dat:

# mr  USAXS_PD I0 seconds
15.61017 9.0   243.0 0.3
15.61 13.0  325.0 0.3
15.60984 19.0  460.0 0.3
15.60967 30.0  609.0 0.3
15.6095  54.0  883.0 0.3
15.60934 161.0 1780.0   0.3
15.60917 499.0 3649.0   0.3
15.609   1257.0   6588.0   0.3
15.60884 2832.0   10245.0  0.3
15.60867 7294.0   13118.0  0.3
15.6085  139191.0 16527.0  0.3
15.60834 299989.0 17893.0  0.3
15.60817 299989.0 18276.0  0.3
15.608   299989.0 18240.0  0.3
15.60784 299989.0 18266.0  0.3
15.60767 299989.0 18616.0  0.3
15.6075  299989.0 19033.0  0.3
15.60734 299989.0 19036.0  0.3
15.60717 299988.0 18587.0  0.3
15.607   299989.0 17471.0  0.3
15.60684 123003.0 14814.0  0.3
15.60667 11060.0  11861.0  0.3
15.6065  2217.0   8131.0   0.3
15.60634 637.0 4269.0   0.3
15.60617 254.0 2632.0   0.3
15.606   132.0 1927.0   0.3
15.60584 79.0  1406.0   0.3
15.60567 58.0  1075.0   0.3
15.6055  32.0  695.0 0.3
15.60534 17.0  374.0 0.3
15.60517 10.0  245.0 0.3

source code documentation

Save columns from SPEC data file scan(s) to TSV files

Note

TSV: tab-separated values

Usage:

extractSpecScan.py /tmp/CeCoIn5 -s 5 -c HerixE Ana5 ICO-C
extractSpecScan.py ./testdata/11_03_Vinod.dat   -s 2 12   -c USAXS.m2rp Monitor  I0

Note

sdpecified column names MUST appear in all chosen scans

Compatible with Python 2.7+

spec2nexus.extractSpecScan.get_user_parameters()

configure user’s command line parameters from sys.argv

spec2nexus.extractSpecScan.main()

read the data file, find each scan, find the columns, save the data

Parameters:cmdArgs ([str]) – Namespace from argparse, returned from get_user_parameters()

Note

Each column label must match exactly the name of a label in each chosen SPEC scan number or the program will skip that particular scan

If more than one column matches, the first match will be selected.

example output:

# mr    I0    USAXS_PD
1.9475    65024    276
1.9725    64845    352
1.9975    65449    478

spec2nexus.extractSpecScan.makeOutputFileName(specFile, scanNum)

return an output file name based on specFile and scanNum

Parameters:

• specFile (str) – name of existing SPEC data file to be read
• scanNum (str) – number of chosen SPEC scan

append scanNum to specFile to get output file name (before file extension if present)

Always add a file extension to the output file. If none is present, use “.dat”.

Examples:

specFile	scanNum	outFile
CeCoIn5	scan 5	CeCoIn5_5.dat
CeCoIn5.dat	scan 77	CeCoIn5_77.dat
CeCoIn5.dat	scan 5.1	CeCoIn5_5_1.dat

specplot

Read a SPEC data file and plot a thumbnail image.

This code can be called as a standalone program or it can be imported into another program and called as a subroutine, as shown in the specplot_gallery program.

The standard representation of a SPEC scan is a line plot of the last data column versus the first data column. Any SPEC macro which name ends with scan ([1]) will be plotted as a line plot.

A special case SPEC scan macro is the hklscan where one of the three reciprocal space axes is scanned while the other two remain constant. A special handler (SPEC’s hklscan macro) is provided to pick properly the scanned axis (not always the first column) for representation as a line plot.

Some SPEC macros scan two positioners over a grid to collect a 2-D image one pixel at a time. These scans are represented as color-mapped images where the first two columns are the vertical and horizontal axes and the image is color-mapped to intensity. Any SPEC macro which name ends with mesh will be plotted as an image plot.

[1]	scan: any scan where the last four letters converted to lower case match scan, such as ascan, a2scan, Escan, tscan, uascan, FlyScan, unusual_custom_user_scan, …

Different handling can be customized for scan macros, as described in How to write a custom scan handling for specplot.

How to use specplot

Plot a scan from one of the sample data files supplied with spec2nexus:

user@host ~$ specplot src/spec2nexus/data/APS_spec_data.dat 2 specplot.png

Plot of scan #2 from example data file APS_spec_data.dat.

Usage

user@host ~$ specplot
usage: specplot.py [-h] specFile scan_number plotFile

Help

user@host ~$ specplot -h
usage: specplot.py [-h] specFile scan_number plotFile

read a SPEC data file and plot scan n

positional arguments:
  specFile     SPEC data file name
  scan_number  scan number in SPEC file
  plotFile     output plot file name

optional arguments:
  -h, --help   show this help message and exit

source code documentation

Plot the data from scan N in a SPEC data file

Selector()	associate SPEC scan macro names with image makers
ImageMaker()	superclass to handle plotting of data from a SPEC scan
LinePlotter()	create a line plot
MeshPlotter()	create a mesh plot (2-D image)
openSpecFile(specFile)	convenience routine so that others do not have to import spec2nexus.spec

Exceptions:

NoDataToPlot	No data found.
NotPlottable	No plottable data for this scan.
ScanAborted	Scan aborted before all points acquired.
UnexpectedObjectTypeError	Incorrect Python object type: programmer error.

class spec2nexus.specplot.HKLScanPlotter

create a line plot from hklscan macros

retrieve_plot_data()

retrieve default data from spec data file

class spec2nexus.specplot.ImageMaker

superclass to handle plotting of data from a SPEC scan

Internal data model

Signal:name of the signal data (default data to be plotted) Data:values of various collected arrays {label: array} Axes:names of the axes of signal data

USAGE:

1. Create a subclass of ImageMaker

2. Override any of these methods:

   data_file_name()	the name of the file with the actual data
   make_image(plotFile)	make MatPlotLib chart image from the SPEC scan
   plottable()	can this data be plotted as expected?
   plot_options()	re-define any plot options in a subclass
   retrieve_plot_data()	retrieve default plottable data from spec data file and store locally

EXAMPLE

class LinePlotter(ImageMaker):
    '''create a line plot'''

    def make_image(self, plotFile):
        '''
        make MatPlotLib chart image from the SPEC scan

        :param obj plotData: object returned from :meth:`retrieve_plot_data`
        :param str plotFile: name of image file to write
        '''
        assert(self.signal in self.data)
        assert(len(self.axes) == 1)
        assert(self.axes[0] in self.data)

        y = self.data[self.signal]
        x = self.data[self.axes[0]]
        xy_plot(x, y,  plotFile,
               title = self.plot_title(),
               plot_subtitle = self.plot_subtitle(),
               xtitle = self.x_title(),
               ytitle = self.y_title(),
               xlog = self.x_log(),
               ylog = self.y_log(),
               timestamp_str = self.timestamp())

sfile = specplot.openSpecFile(specFile)
scan = sfile.getScan(scan_number)
plotter = LinePlotter()
plotter.plot_scan(scan, plotFile, y_log=True)

data_file_name()

the name of the file with the actual data

Usually, this is the SPEC data file but it could be something else

data_is_newer_than_plot(plotFile)

only proceed if mtime of SPEC data file is newer than plotFile

make_image(plotFile)

make MatPlotLib chart image from the SPEC scan

The data to be plotted are provided in:

• self.signal
• self.axes
• self.data

Parameters:plotFile (str) – name of image file to write

plot_options()

re-define any plot options in a subclass

plot_scan(scan, plotFile, maker=None)

make an image plot of the data in the scan

Parameters:

• scan (obj) – instance of SpecDataFileScan
• plotFile (str) – file name for plot output

plot_subtitle()

Return the plot_subtitle.

plot_title()

Return the plot title.

plottable()

can this data be plotted as expected?

retrieve_plot_data()

retrieve default plottable data from spec data file and store locally

This method must retrieve the data to be plotted, either from the SPEC data file scan or from a file which name is provided in the scan detalis.

These attributes must be set by this method:

Data:dictionary containing values of the various collected arrays {label: array} Signal:name of the ‘signal’ data (default data to be plotted) Axes:names of the axes of signal data

Example data

self.data = {
    'angle': [1, 2, 3, 4, 5],
    'counts': [0. 2. 55. 3. 0]}
self.signal = 'counts'
self.axes = ['angle']

Raise any of these exceptions as appropriate:

NoDataToPlot	No data found.
NotPlottable	No plottable data for this scan.
ScanAborted	Scan aborted before all points acquired.

set_plot_subtitle(text)

Set the plot_subtitle.

set_plot_title(text)

Set the plot title.

set_timestamp(text)

Set the plot time stamp.

set_x_log(choice)

Set the x axis logarithmic if True.

set_x_title(text)

Set the x axis title.

set_y_log(choice)

Set the y axis logarithmic if True.

set_y_title(text)

Set the y axis title.

set_z_log(choice)

Set the z axis (image) logarithmic if True.

timestamp()

Return the time of this scan as a string.

x_log()

Boolean: should the X axis be plotted on a log scale?

x_title()

Return the title for the X axis.

y_log()

Boolean: should the Y axis be plotted on a log scale?

y_title()

Return the title for the Y axis.

z_log()

Boolean: should the Z axis (image) be plotted on a log scale?

class spec2nexus.specplot.LinePlotter

create a line plot

make_image(plotFile)

make MatPlotLib chart image from the SPEC scan

Parameters:plotFile (str) – name of image file to write

plot_options()

define the settings for this, accepting any non-default values first

plottable()

can this data be plotted as expected?

retrieve_plot_data()

retrieve default data from spec data file

class spec2nexus.specplot.MeshPlotter

create a mesh plot (2-D image)

..rubric:: References:

Mesh 2-D parser:  

http://www.certif.com/spec_help/mesh.html

mesh motor1 start1 end1 intervals1 motor2 start2 end2 intervals2 time

Hklmesh 2-D parser:  

http://www.certif.com/spec_help/hklmesh.html

hklmesh Q1 start1 end1 intervals1 Q2 start2 end2 intervals2 time

make_image(plotFile)

make MatPlotLib chart image from the SPEC scan

Parameters:plotFile (str) – name of image file to write

plot_options()

define the settings for this, accepting any non-default values first

plottable()

can this data be plotted as expected?

retrieve_plot_data()

retrieve default data from spec data file

data parser for 2-D mesh and hklmesh

exception spec2nexus.specplot.NoDataToPlot

No data found.

exception spec2nexus.specplot.NotPlottable

No plottable data for this scan.

exception spec2nexus.specplot.ScanAborted

Scan aborted before all points acquired.

class spec2nexus.specplot.Selector

associate SPEC scan macro names with image makers

Image maker:subclass of ImageMaker

To include a custom image maker from outside this module, create the subclass and then add it to an instance of this class. Such as this plotter that defaults to a logarithmic scale for the X axis for all logxscan macros:

from spec2nexus import specplot

class LogX_Plotter(specplot.ImageMaker):

def x_log(self):

return True

# …

selector = specplot.Selector() selector.add(‘logxscan’, LogX_Plotter)

# …

image_maker = specplot.Selector().auto(scan) plotter = image_maker() plotter.plot_scan(scan, fullPlotFile)

This class is a singleton which means you will always get the same instance when you call this class many times in your program.

auto(scan)	automatically choose a scan image maker based on the SPEC scan macro
add(key, value[, default])	register a new value by key
update(key, value[, default])	replace an existing key with a new value
get(key)	return a value by key
exists(key)	is the key known?
default()	retrieve the value of the default key

add(key, value, default=False)

register a new value by key

Parameters:

key (str) – name of key, typically the macro name

Raises:

• KeyError – if key exists
• UnexpectedObjectTypeError – if value is not subclass of ImageMaker

auto(scan)

automatically choose a scan image maker based on the SPEC scan macro

Selection Rules:

• macro ends with “scan”: use LinePlotter
• macro ends with “mesh”: use MeshPlotter
• default: use default image maker (initially LinePlotter)

default()

retrieve the value of the default key

exists(key)

is the key known?

get(key)

return a value by key

Returns:subclass of ImageMaker or None if key not found

update(key, value, default=False)

replace an existing key with a new value

Parameters:

key (str) – name of key, typically the macro name

Raises:

• KeyError – if key does not exist
• UnexpectedObjectTypeError – if value is not subclass of ImageMaker

exception spec2nexus.specplot.UnexpectedObjectTypeError

Incorrect Python object type: programmer error.

spec2nexus.specplot.openSpecFile(specFile)

convenience routine so that others do not have to import spec2nexus.spec

specplot_gallery

Read a list of SPEC data files (or directory(s) containing SPEC data files) and plot images of all scans. specplot_gallery will store these images in subdirectories of the given base directory (default: current directory) based on this structure:

{base directory}
   /{year}
      /{month}
         /{spec file name}
             /index.html
              s00001.png
              s00002.png

The year and month are taken from the SPEC data file when the data were collected. The plot names include the scan numbers padded with leading zeroes to five places (so the file names sort numerically).

The results will be shown as a WWW page (index.html) of thumbnail images and a separate list of any scans that could not generate plots. A reason will accompany these scans, as shown in the example.

How to use specplot_gallery: command line

Here is an example:

user@host ~$ specplot_gallery -d ./__demo__ ../src/spec2nexus/data/33bm_spec.dat

Example of specplot_gallery showing scans from test file 33bm_spec.dat.

Note that one of the scans could not be plotted. Looking at the data file, it shows there is no data to plot (this particular scan was aborted before any data was collected):

#C Wed Jun 16 19:00:10 2010.  Scan aborted after 0 points.

The last scan shown is from a hklmesh (2-D) scan. It is mostly a constant background level, thus the large black area.

Each of the plots in the web page can be enlarged (by clicking on it).

How to use specplot_gallery: periodic background task (cron)

This script could be called from a Linux background task scheduler (cron) entry. To add the entry, type the crontab -e command which opens the task list in a screen editor and add lines such as these to the file:

# every five minutes (generates no output from outer script)
0-59/5 * * * *  /path/to/specplot_gallery.py -d /web/page/dir /spec/data/file/dirs

If the specplot_gallery script is called too frequently and the list of plots to be generated is large enough, it is possible for more than one process to be running. In one extreme case, many processes were found running due to problems with the data files. To identify and stop all processes of this program, use this on the command line:

kill -9 `ps -ef | grep python | awk '/specplot_gallery.py/ {print $2}' -`

source code documentation

read a list of SPEC data files (or directories) and plot images of all scans

DirectoryNotFoundError	Exception: The requested directory does not exist
PlotSpecFileScans(filelist[, plotDir, …])	read a SPEC data file and plot thumbnail images of all its scans
Cache_File_Mtime(base_dir)	Maintain a list of all known data file modification times.
datePath(date)	Convert the date into a path: yyyy/mm.
getSpecFileDate(specFile)	Return the #D date of the SPEC data file or None.
needToMakePlot(fullPlotFile, mtime_specFile)	Determine if a plot needs to be (re)made.
timestamp()	current time as yyyy-mm-dd hh:mm:ss
buildIndexHtml(specFile, plotted_scans, …)	Build index.html content.
logger(message)	Log a message or report from this module.

RESULT

The images are stored in files within a directory structure that is organized chronologically, such as: yyyy/mm/spec_file/s1.svg. The root of the directory is either specified by the command line -d option or defaults to the current working directory. The yyyy/mm (year and month) are taken from the #D line of the SPEC data file. The spec_file is the file name with file extension and directory name removed. The image file names are derived from the scan numbers.

Linux CRON task

This script could be called from a cron entry, such as:

# every five minutes (generates no output from outer script)
0-59/5 * * * *  /some/directory/specplot_gallery.py -d /web/page/dir /spec/data/file/dir

If this script is called too frequently and the list of plots to be generated is large enough, it is possible for more than one process to be running. In one extreme case, many processes were found running due to problems with the data files. To identify and stop all processes of this program:

kill -9 `ps -ef | grep python | awk '/specplot_gallery.py/ {print $2}' -`

class spec2nexus.specplot_gallery.Cache_File_Mtime(base_dir)

Maintain a list of all known data file modification times.

Parameters:base_dir (str) – name of base directory to store output image thumbnails

This list will allow the code to avoid unnecessary work reparsing and plotting of unchanged SPEC data files.

get(fname, default={'mtime': 0, 'size': 0})

Get the mtime cache entry for data file fname.

Parameters:fname (str) – file name, already known to exist Returns:time (float) cached value of when fname was last modified or None if not known

read()

Read the cache from storage.

was_file_updated(fname)

Compare the mtime between disk and cache.gy

Parameters:fname (str) – file name, already known to exist Return bool:True if file is newer than the cache (or new to the cache)

write()

Write the cache to storage.

exception spec2nexus.specplot_gallery.DirectoryNotFoundError

Exception: The requested directory does not exist

exception spec2nexus.specplot_gallery.PathIsNotDirectoryError

Exception: The path is not a directory

class spec2nexus.specplot_gallery.PlotSpecFileScans(filelist, plotDir=None, reverse_chronological=False)

read a SPEC data file and plot thumbnail images of all its scans

Parameters:

• filelist ([str]) – list of SPEC data files to be checked
• plotDir (str) – name of base directory to store output image thumbnails

specFileUpdated(specFile)	Report if specFile has been updated.
plot_all_scans(specFile)	Plot all the recognized scans from file specFile.
getPlotDir(specFile)	Return the plot directory based on the specFile.
getBaseDir(basename, date)	Find the path based on the date in the spec file.
href_format(basePlotFile, altText)

getBaseDir(basename, date)

Find the path based on the date in the spec file.

getPlotDir(specFile)

Return the plot directory based on the specFile.

Parameters:specFile (str) – name of SPEC data file (relative or absolute)

plot_all_scans(specFile)

Plot all the recognized scans from file specFile.

specFileUpdated(specFile)

Report if specFile has been updated.

Return mtime cache entry (or None if not updated)

spec2nexus.specplot_gallery.buildIndexHtml(specFile, plotted_scans, problem_scans)

Build index.html content.

Parameters:

• specFile (str) – name of SPEC data file (relative or absolute)
• plotList ([str]) – list of HTML <a> elements, one for each plot image

spec2nexus.specplot_gallery.datePath(date)

Convert the date into a path: yyyy/mm.

Parameters:date (str) – text date from SPEC file #D line: ‘Thu Jun 19 12:21:55 2014’

spec2nexus.specplot_gallery.developer()

Supply a file and a directory as command-line arguments to “paths”.

spec2nexus.specplot_gallery.getSpecFileDate(specFile)

Return the #D date of the SPEC data file or None.

Parameters:specFile (str) – name of SPEC data file (relative or absolute)

spec2nexus.specplot_gallery.logger(message)

Log a message or report from this module.

Parameters:message (str) – text to be logged

spec2nexus.specplot_gallery.needToMakePlot(fullPlotFile, mtime_specFile)

Determine if a plot needs to be (re)made. Use mtime as the basis.

Return bool:True if plot should be made again

spec2nexus.specplot_gallery.timestamp()

current time as yyyy-mm-dd hh:mm:ss

Return str:

spec2nexus.spec

Library of classes to read the contents of a SPEC data file.

How to use spec2nexus.spec

spec2nexus.spec provides Python support to read the scans in a SPEC data file. (It does not provide a command-line interface.) Here is a quick example how to use spec:

from spec2nexus.spec import SpecDataFile

specfile = SpecDataFile('data/33id_spec.dat')
print 'SPEC file name:', specfile.specFile
print 'SPEC file time:', specfile.headers[0].date
print 'number of scans:', len(specfile.scans)

for scanNum, scan in specfile.scans.items():
    print scanNum, scan.scanCmd

For one example data file provided with spec2nexus.spec, the output starts with:

How to read one scan

Here is an example how to read one scan:

from spec2nexus.spec import SpecDataFile

specfile = SpecDataFile('data/33id_spec.dat')
specscan = specfile.getScan(5)
print specscan.scanNum
print specscan.scanCmd

which has this output:

5
ascan  del 84.3269 84.9269  30 1

The data columns are provided in a dictionary. Using the example above, the dictionary is specscan.data where the keys are the column labels (from the #L line) and the values are from each row. It is possible to make a default plot of the last column vs. the first column. Here’s how to find that data:

x_label = specscan.L[0]          # first column from #L line
y_label = specscan.L[-1]         # last column from #L line
x_data = specscan.data[x_label]  # data for first column
y_data = specscan.data[y_label]  # data for last column

Get a list of the scans

The complete list of scan numbers from the data file is obtained (sorting is necessary since the list of dictionary keys is returned in a scrambled order):

all_scans = sorted(specfile.scans.keys())

---

SPEC data files

The SPEC data file format is described in the SPEC manual. [1] This manual is taken as a suggested starting point for most users. Data files with deviations from this standard are produced at some facilities.

[1]	SPEC manual: http://www.certif.com/spec_manual/user_1_4_1.html

Assumptions about data file structure

These assumptions are used to parse SPEC data files:

1. SPEC data files are text files organized by lines. The lines can be categorized as: control lines, data lines, and blank lines.

   line type	description
   control	contain a # character in the first column followed by a command word [2]
   data	generally contain a row of numbers (the scan data)
   special data	containing MCA data [3]

2. Lines in a SPEC data file start with a file name control line, then series of blocks. Each block may be either a file header block or a scan block. (Most SPEC files have only one header block. A new header block is created if the list of positioners is changed in SPEC without creating a new file. SPEC users are encouraged to always start a new data file after changing the list of positioners.) A block consists of a series of control, data, and blank lines.

   SPEC data files are composed of a sequence of a single file header block and zero or more scan blocks. [4]

3. A SPEC data file always begins with this control lines: #F, such as:

   #F samplecheck_7_17_03
   

4. A file header block begins with these control lines in order: #E #D #C, such as:

   #E 1058427452
   #D Thu Jul 17 02:37:32 2003
   #C psic  User = epix
   

5. A scan block begins with these command lines in order: #S #D, such as:

   #S 78  ascan  del 84.6484 84.8484  20 1
   #D Thu Jul 17 08:03:54 2003
   

[2]	See Example of Control Lines

[3]	See Example of MCA data lines

[4]	It is very unusual to have more than one file header block in a SPEC data file.

Control lines (keys) defined by SPEC

Here is a list [5] of keys (command words) from the comments in the file.mac (SPEC v6) macro source file:

command word	description
#C	comment line
#D date	current date and time in UNIX format
#E num	the UNIX epoch (seconds from 00:00 GMT 1/1/70)
#F name	name by which file was created
#G1 …	geometry parameters from G[] array (geo mode, sector, etc)
#G2 …	geometry parameters from U[] array (lattice constants, orientation reflections)
#G3 …	geometry parameters from UB[] array (orientation matrix)
#G4 …	geometry parameters from Q[] array (lambda, frozen angles, cut points, etc)
#I num	a normalizing factor to apply to the data
#j% …	mnemonics of counter (% = 0,1,2,… with eight counters per row)
#J% …	names of counters (each separated by two spaces)
#L s1 …	labels for the data columns
#M num	data was counted to this many monitor counts
#N num [num2]	number of columns of data [ num2 sets per row ]
#o% …	mnemonics of motors (% = 0,1,2,… with eight motors per row)
#O% …	names of motors (each separated by two spaces)
#P% …	positions of motors corresponding to above #O/#o
#Q	a reciprocal space position (H K L)
#R	user-defined results from a scan
#S num	scan number
#T num	data was counted for this many seconds
#U	user defined
#X	a temperature
#@MCA fmt	this scan contains MCA data (array_dump() format, as in "%16C")
#@CALIB a b c	coefficients for x[i] = a + b * i + c * i * i for MCA data
#@CHANN n f l r	MCA channel information (number_saved, first_saved, last_saved, reduction coef)
#@CTIME p l r	MCA count times (preset_time, elapsed_live_time, elapsed_real_time)
#@ROI n f l	MCA ROI channel information (ROI_name, first_chan, last_chan)

[5]	Compare with Supplied spec plugin modules

Example of Control Lines

The command word of a control line may have a number at the end, indicating it is part of a sequence, such as these control lines (see Control lines (keys) defined by SPEC for how to interpret):

Example of MCA data lines

Lines with MCA array data begin with the @A command word. (If such a data line ends with a continuation character \, the next line is read as part of this line.)

This is an example of a 91-channel MCA data array with trivial (zero) values:

@A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0

Several MCA spectra may be written to a scan. In this case, a number follows @A indicating which spectrum, such as in this example with four spectra:

     @A1 0 0 0 0 0 0 35 0 0 35
     @A2 0 0 0 0 0 0 0 35 0 35
     @A3 0 0 35 35 0 0 0 0 0 0
     @A4 0 0 0 0 0 35 35 0 35 0

Supported header keys (command words)

The SPEC data file keys recognized by spec are listed in Supplied spec plugin modules.

---

source code summary

classes

SpecDataFile	contents of a SPEC data file
SpecDataFileHeader	contents of a spec data file header (#E) section
SpecDataFileScan	contents of a spec data file scan (#S) section

methods

strip_first_word	return everything after the first space on the line from the spec data file
is_spec_file	test if a given file name is a SPEC data file

exceptions

SpecDataFileNotFound	data file was not found
SpecDataFileCouldNotOpen	data file could not be opened
SpecDataFileNotFound	data file was not found
DuplicateSpecScanNumber	multiple use of scan number in a single SPEC data file
UnknownSpecFilePart	unknown part in a single SPEC data file

dependencies

os	OS routines for NT or Posix depending on what system we’re on.
re	Support for regular expressions (RE).
sys	This module provides access to some objects used or maintained by the interpreter and to functions that interact strongly with the interpreter.

internal structure of spec2nexus.spec.SpecDataFileScan

The internal variables of a Python class are called attributes. It may be convenient, for some, to think of them as variables.

scan attributes

parent:obj - instance of spec2nexus.spec.SpecDataFile scanNum:int - SPEC scan number scanCmd:str - SPEC command line raw:str - text of scan, as reported in SPEC data file

scan attributes (variables) set after call to plugins

These attributes are only set after the scan’s interpret() method is called. This method is called automatically when trying to read any of the following scan attributes:

comments:[str] - list of all comments reported in this scan data:{label,[number]} - written by spec2nexus.plugins.spec_common_spec2nexus.data_lines_postprocessing() data_lines:[str] - raw data (and possibly MCA) lines with comment lines removed date:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Date G:{key,[number]} - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Geometry I:float - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_NormalizingFactor header:obj - instance of spec2nexus.spec.SpecDataFileHeader L:[str] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Labels M:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Monitor positioner:{key,number} - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Positioners.postprocess N:[int] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_NumColumns P:[str] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Positioners Q:[number] - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_HKL S:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_Scan T:str - written by spec2nexus.plugins.spec_common_spec2nexus.SPEC_CountTime V:{key,number|str} - written by spec2nexus.plugins.unicat_spec2nexus.UNICAT_MetadataValues column_first:str - label of first (ordinate) data column column_last:str - label of last (abscissa) data column

internal use only - do not modify

These scan attributes are for internal use only and are not part of the public interface. Do not modify them or write code that depends on them.

postprocessors:{key,obj} - dictionary of postprocessing methods h5writers:{key,obj} - dictionary of methods that write HDF5 structure __lazy_interpret__:  bool - Is lazy (on-demand) call to interpret() needed? __interpreted__:  bool - Has interpret() been called?

source code documentation

Provides a set of classes to read the contents of a SPEC data file.

author:Pete Jemian email:jemian@anl.gov

SpecDataFile() is the only class users will need to call. All other spec classes are called from this class. The read() method is called automatically.

The user should create a class instance for each spec data file, specifying the file reference (by path reference as needed) and the internal routines will take care of all that is necessary to read and interpret the information.

is_spec_file(filename)	test if a given file name is a SPEC data file
is_spec_file_with_header(filename)	test if a given file name is a SPEC data file
SpecDataFile(filename)	contents of a SPEC data file
SpecDataFileHeader(buf[, parent])	contents of a spec data file header (#E) section
SpecDataFileScan(header, buf[, parent])	contents of a spec data file scan (#S) section

Note that the SPEC geometry control lines (#G0 #G1 …) have meanings that are unique to specific diffractometer geometries including different numbers of values. Consult the geometry macro file for specifics.

Examples

Get the first and last scan numbers from the file:

>>> from spec2nexus import spec
>>> spec_data = spec.SpecDataFile('path/to/my/spec_data.dat')
>>> print(spec_data.fileName)
path/to/my/spec_data.dat
>>> print('first scan: ', spec_data.getFirstScanNumber())
1
>>> print('last scan: ', spec_data.getLastScanNumber())
22

Get plottable data from scan number 10:

>>> from spec2nexus import spec
>>> spec_data = spec.SpecDataFile('path/to/my/spec_data.dat')
>>> scan10 = spec_data.getScan(10)
>>> x_label = scan10.L[0]
>>> y_label = scan10.L[-1]
>>> x_data = scan10.data[x_label]
>>> y_data = scan10.data[y_label]

Try to read a file that does not exist:

>>> spec_data = spec.SpecDataFile('missing_file')
Traceback (most recent call last):
  ...
spec.SpecDataFileNotFound: file does not exist: missing_file

Classes and Methods

exception spec2nexus.spec.DuplicateSpecScanNumber

multiple use of scan number in a single SPEC data file

exception spec2nexus.spec.NotASpecDataFile

content of file is not SPEC data (first line must start with #F)

class spec2nexus.spec.SpecDataFile(filename)

contents of a SPEC data file

dissect_file()	divide (SPEC data file text) buffer into sections
getFirstScanNumber()	return the first scan
getLastScanNumber()	return the last scan
getMaxScanNumber()	return the highest numbered scan
getMinScanNumber()	return the lowest numbered scan
getScan([scan_number])	return the scan number indicated, None if not found
getScanCommands([scan_list])	return all the scan commands as a list, with scan number
getScanNumbers()	return a list of all scan numbers sorted by scan number
getScanNumbersChronological()	return a list of all scan numbers sorted by date
read()	Reads and parses a spec data file
refresh()	update (refresh) the content if the file is updated
update_available	Has the file been updated since the last time it was read?

dissect_file()

divide (SPEC data file text) buffer into sections

internal: A block starts with either #F | #E | #S

RETURNS

[block]

list of blocks where each block is one or more lines of text with one of the above control lines at its start

getFirstScanNumber()

return the first scan

getLastScanNumber()

return the last scan

getMaxScanNumber()

return the highest numbered scan

getMinScanNumber()

return the lowest numbered scan

getScan(scan_number=0)

return the scan number indicated, None if not found

getScanCommands(scan_list=None)

return all the scan commands as a list, with scan number

getScanNumbers()

return a list of all scan numbers sorted by scan number

getScanNumbersChronological()

return a list of all scan numbers sorted by date

read()

Reads and parses a spec data file

refresh()

update (refresh) the content if the file is updated

returns previous last_scan or None if file not updated

update_available

Has the file been updated since the last time it was read?

Reference file modification time is stored after file is read in read() method.

EXAMPLE USAGE

Open the SPEC data file (example):

sdf = spec.SpecDataFile(filename)

then, monitor (continuing example):

if sdf.update_available:

myLastScan = sdf.last_scan sdf.read() plot_scan_and_newer(myLastScan) # new method myLastScan = sdf.last_scan

exception spec2nexus.spec.SpecDataFileCouldNotOpen

data file could not be opened

class spec2nexus.spec.SpecDataFileHeader(buf, parent=None)

contents of a spec data file header (#E) section

interpret()	interpret the supplied buffer with the spec data file header
addPostProcessor(label, func)	add a function to be processed after interpreting all lines from a header
addH5writer(label, func)	add a function to be processed when writing the scan header
getLatestScan()

addH5writer(label, func)

add a function to be processed when writing the scan header

Parameters:

• label (str) – unique label by which this writer will be known
• func (obj) – function reference of writer

The writers will be called when the HDF5 file is to be written.

addPostProcessor(label, func)

add a function to be processed after interpreting all lines from a header

Parameters:

• label (str) – unique label by which this postprocessor will be known
• func (obj) – function reference of postprocessor

The postprocessors will be called at the end of header interpretation.

interpret()

interpret the supplied buffer with the spec data file header

exception spec2nexus.spec.SpecDataFileNotFound

data file was not found

class spec2nexus.spec.SpecDataFileScan(header, buf, parent=None)

contents of a spec data file scan (#S) section

get_macro_name()	name of the SPEC macro used for this scan
interpret()	interpret the supplied buffer with the spec scan data
add_interpreter_comment(comment)	allow the interpreter to communicate information to the caller
get_interpreter_comments()	return the list of comments
addPostProcessor(label, func)	add a function to be processed after interpreting all lines from a scan
addH5writer(label, func)	add a function to be processed when writing the scan data

addH5writer(label, func)

add a function to be processed when writing the scan data

Parameters:

• label (str) – unique label by which this writer will be known
• func (obj) – function reference of writer

The writers will be called when the HDF5 file is to be written.

addPostProcessor(label, func)

add a function to be processed after interpreting all lines from a scan

Parameters:

• label (str) – unique label by which this postprocessor will be known
• func (obj) – function reference of postprocessor

The postprocessors will be called at the end of scan data interpretation.

add_interpreter_comment(comment)

allow the interpreter to communicate information to the caller

see issue #66: https://github.com/prjemian/spec2nexus/issues/66

get_interpreter_comments()

return the list of comments

see issue #66: https://github.com/prjemian/spec2nexus/issues/66

get_macro_name()

name of the SPEC macro used for this scan

interpret()

interpret the supplied buffer with the spec scan data

exception spec2nexus.spec.UnknownSpecFilePart

unknown part in a single SPEC data file

spec2nexus.spec.is_spec_file(filename)

test if a given file name is a SPEC data file

Parameters:filename (str) – path/to/possible/spec/data.file

filename is a SPEC file if it contains at least one #S control line

spec2nexus.spec.is_spec_file_with_header(filename)

test if a given file name is a SPEC data file

Parameters:filename (str) – path/to/possible/spec/data.file

filename is a SPEC file only if the file starts [6] with these control lines in order:

• #F - original filename
• #E - the UNIX epoch (seconds from 00:00 GMT 1/1/70)
• #D - current date and time in UNIX format
• #C - comment line (the first one provides the filename again and the user name)

such as:

#F LNO_LAO
#E 1276730676
#D Wed Jun 16 18:24:36 2010
#C LNO_LAO  User = epix33bm

[6]	SPEC manual, Standard Data File Format, http://www.certif.com/spec_manual/user_1_4_1.html

spec2nexus.charts

source code documentation

charting for spec2nexus

make_png(image, image_file[, axes, title, …])	read the image from the named HDF5 file and make a PNG file
xy_plot(x, y, plot_file[, title, subtitle, …])	with MatPlotLib, generate a plot of a scan (as if data from a scan in a SPEC file)

spec2nexus.charts.make_png(image, image_file, axes=None, title='2-D data', subtitle='', log_image=False, hsize=9, vsize=5, cmap='cubehelix', xtitle=None, ytitle=None, timestamp_str=None)

read the image from the named HDF5 file and make a PNG file

Test that the HDF5 file exists and that the path to the data exists in that file. Read the data from the named dataset, mask off some bad values, convert to log(image) and use Matplotlib to make the PNG file.

Parameters:

• image (obj) – array of data to be rendered
• image_file (str) – name of image file to be written (path is optional)
• log_image (bool) – plot log(image)
• hsize (int) – horizontal size of the PNG image (default: 7)
• hsize – vertical size of the PNG image (default: 3)
• cmap (str) – colormap for the image (default: ‘cubehelix’), ‘jet’ is another good one

Return str:

image_file

The HDF5 file could be a NeXus file, or some other layout.

spec2nexus.charts.xy_plot(x, y, plot_file, title=None, subtitle=None, xtitle=None, ytitle=None, xlog=False, ylog=False, hsize=9, vsize=5, timestamp_str=None)

with MatPlotLib, generate a plot of a scan (as if data from a scan in a SPEC file)

Parameters:

• x ([float]) – horizontal axis data
• y ([float]) – vertical axis data
• plot_file (str) – file name to write plot image
• xtitle (str) – horizontal axis label (default: not shown)
• ytitle (str) – vertical axis label (default: not shown)
• title (str) – title for plot (default: date time)
• subtitle (str) – subtitle for plot (default: not shown)
• xlog (bool) – should X axis be log (default: False=linear)
• ylog (bool) – should Y axis be log (default: False=linear)
• timestamp_str (str) – date to use on plot (default: now)

Tip

when using this module as a background task …

MatPlotLib has several interfaces for plotting. Since this module runs as part of a background job generating lots of plots, MatPlotLib’s standard plt code is not the right model. It warns after 20 plots and will eventually run out of memory.

Here’s the fix used in this module: http://stackoverflow.com/questions/16334588/create-a-figure-that-is-reference-counted/16337909#16337909

How to write a custom scan handling for specplot

Sometimes, it will be obvious that a certain scan macro never generates any plot images, or that the default handling creates a plot that is a poor representation of the data, such as the hklscan where only one of the the axes hkl is scanned. To pick the scanned axis for plotting, it is necessary to prepare custom handling and replace the default handling.

Overview

It is possible to add in additional handling by writing a Python module. This module creates a subclass of the standard handling, such as LinePlotter, MeshPlotter, or their superclass ImageMaker. The support is added to the macro selection class Selector with code such as in the brief example described below: Change the plot title text in ascan macros:

selector = spec2nexus.specplot.Selector()
selector.add('ascan', Custom_Ascan)
spec2nexus.specplot_gallery.main()

Data Model

The data to be plotted is kept in an appropriate subclass of PlotDataStructure in attributes show in the next table. The data model is an adaptation of the NeXus NXdata base class. [1]

attribute	description
self.signal	name of the dependent data (y axis or image) to be plotted
self.axes	list of names of the independent axes [2]
self.data	dictionary with the data, indexed by name

[1]	NeXus NXdata base class: http://download.nexusformat.org/doc/html/classes/base_classes/NXdata.html

[2]	The number of names provided in self.axes is equal to the rank of the signal data (self.data[self.signal]). For 1-D data, self.axes has one name and the signal data is one-dimensional. For 2-D data, self.axes has two names and the signal data is two-dimensional.

Steps

In all cases, custom handling of a specific SPEC macro name is provided by creating a subclass of ImageMaker and defining one or more of its methods. In the simplest case, certain settings may be changed by calling spec2nexus.specplot.ImageMaker.configure() with the custom values. Examples of further customization are provided below, such as when the data to be plotted is stored outside of the SPEC data file. This is common for images from area detectors.

It may also be necessary to create a subclass of PlotDataStructure to gather the data to be plotted or override the default spec2nexus.specplot.ImageMaker.plottable() method. An example of this is shown with the MeshPlotter and associated MeshStructure classes.

Examples

A few exmaples of custom macro handling are provided, some simple, some complex. In each example, decisions have been made about where to provide the desired features.

Change the plot title text in ascan macros

The SPEC ascan macro is a workhorse and records the scan of a positioner and the measurement of data in a counter. Since this macro name ends with “scan”, the default selection in specplot images this data using the LinePlotter class. Here is a plot of the default handling of data from the ascan macro:

Standard plot of data from ascan macro

We will show how to change the plot title as a means to illustrate how to customize the handling for a scan macro.

We write Custom_Ascan which is a subclass of LinePlotter. The get_plot_data method is written (overrides the default method) to gain access to the place where we can introduce the change. The change is made by the call to the configure method (defined in the superclass). Here’s the code:

ascan.py example

#!/usr/bin/env python

#-----------------------------------------------------------------------------
# :author:    Pete R. Jemian
# :email:     prjemian@gmail.com
# :copyright: (c) 2014-2020, Pete R. Jemian
#
# Distributed under the terms of the Creative Commons Attribution 4.0 International Public License.
#
# The full license is in the file LICENSE.txt, distributed with this software.
#-----------------------------------------------------------------------------

'''
Plot all scans that used the SPEC `ascan` macro, showing only the scan number (not full scan command)

This is a simple example of how to customize the scan macro handling.
There are many more ways to add complexity.
'''

import spec2nexus.specplot
import spec2nexus.specplot_gallery


class Custom_Ascan(spec2nexus.specplot.LinePlotter):
    '''simple customization'''
    
    def retrieve_plot_data(self):
        '''substitute with the data&time the plot was created'''
        import datetime
        spec2nexus.specplot.LinePlotter.retrieve_plot_data(self)
        self.set_plot_subtitle(str(datetime.datetime.now()))


def main():
    selector = spec2nexus.specplot.Selector()
    selector.add('ascan', Custom_Ascan)
    spec2nexus.specplot_gallery.main()


if __name__ == '__main__':
    main()

See the changed title:

Customized plot of data from ascan macro

Make the y-axis log scale

A very simple customization can make the Y axis to be logarithmic scale. (This customization is planned for an added feature [3] in a future relase of the spec2nexus package.) We present two examples.

modify handling of a2scan

One user wants all the a2scan images to be plotted with a logarithmic scale on the Y axis. Here’s the code:

custom_a2scan_gallery.py example

#!/usr/bin/env python

'''
Customization for specplot_gallery: plot a2scan with log(y) axis

This program changes the plotting for all scans that used the *a2scan* SPEC macro.
The Y axis of these plots will be plotted as logarithmic if all the data values are 
greater than zero.  Otherwise, the Y axis scale will be linear.
'''

import spec2nexus.specplot
import spec2nexus.specplot_gallery

class Custom_a2scan_Plotter(spec2nexus.specplot.LinePlotter):
    '''plot `a2scan` y axis as log if possible'''
    
    def retrieve_plot_data(self):
        '''plot the vertical axis on log scale'''
        spec2nexus.specplot.LinePlotter.retrieve_plot_data(self)

        choose_log_scale = False

        if self.signal in self.data:    # log(y) if all data positive
            choose_log_scale = min(self.data[self.signal]) > 0

        self.set_y_log(choose_log_scale)


def main():
    selector = spec2nexus.specplot.Selector()
    selector.add('a2scan', Custom_a2scan_Plotter)
    spec2nexus.specplot_gallery.main()


if __name__ == '__main__':
    # debugging_setup()
    main()

'''
Instructions:

Save this file in a directory you can write and call it from your cron tasks.  

Note that in cron entries, you cannot rely on shell environment variables to 
be defined.  Best to spell things out completely.  For example, if your $HOME 
directory is `/home/user` and you have these directories:

* `/home/user/bin`: various custom executables you use
* `/home/user/www/specplots`: a directory you access with a web browser for your plots
* `/home/user/spec/data`: a directory with your SPEC data files

then save this file to `/home/user/bin/custom_a2scan_gallery.py` and make it executable
(using `chmod +x ./home/user/bin/custom_a2scan_gallery.py`).

Edit your list of cron tasks using `crontab -e` and add this (possibly 
replacing a call to `specplot_gallery` with this call `custom_a2scan_gallery.py`)::

    # every five minutes (generates no output from outer script)
    0-59/5 * * * *  /home/user/bin/custom_a2scan_gallery.py -d /home/user/www/specplots /home/user/spec/data 2>&1 >> /home/user/www/specplots/log_cron.txt

Any output from this periodic task will be recorded in the file
`/home/user/www/specplots/log_cron.txt`.  This file can be reviewed
for diagnostics or troubleshooting.
'''

custom uascan

The APS USAXS instrument uses a custom scan macro called uascan for routine step scans. Since this macro name ends with “scan”, the default selection in specplot images this data using the LinePlotter class. Here is a plot of the default handling of data from the uascan macro:

USAXS uascan, handled as LinePlotter

The can be changed by making the y axis log scale. To do this, a custom version of LinePlotter is created as Custom_Ascan. The get_plot_data method is written (overrides the default method) to make the y axis log-scale by calling the configure method (defined in the superclass). Here’s the code:

usaxs_uascan.py example

#!/usr/bin/env python

#-----------------------------------------------------------------------------
# :author:    Pete R. Jemian
# :email:     prjemian@gmail.com
# :copyright: (c) 2014-2017, Pete R. Jemian
#
# Distributed under the terms of the Creative Commons Attribution 4.0 International Public License.
#
# The full license is in the file LICENSE.txt, distributed with this software.
#-----------------------------------------------------------------------------

'''
Plot data from the USAXS uascan macro

.. autosummary::

    ~UAscan_Plotter

'''

import spec2nexus.specplot
import spec2nexus.specplot_gallery


class UAscan_Plotter(spec2nexus.specplot.LinePlotter):
    '''simple customize of `uascan` handling'''
    
    def retrieve_plot_data(self):
        '''plot the vertical axis on log scale'''
        spec2nexus.specplot.LinePlotter.retrieve_plot_data(self)

        if self.signal in self.data:
            if min(self.data[self.signal]) <= 0:
                # TODO: remove any data where Y <= 0 (can't plot on log scale)
                msg = 'cannot plot Y<0: ' + str(self.scan)
                raise spec2nexus.specplot.NotPlottable(msg)

        # in the uascan, a name for the sample is given in `self.scan.comments[0]`
        self.set_y_log(True)
        self.set_plot_subtitle(
            '#%s uascan: %s' % (str(self.scan.scanNum), self.scan.comments[0]))


def debugging_setup():
    import os, sys
    import shutil
    import ascan
    selector = spec2nexus.specplot.Selector()
    selector.add('ascan', ascan.Custom_Ascan)   # just for the demo
    path = '__usaxs__'
    shutil.rmtree(path, ignore_errors=True)
    os.mkdir(path)
    sys.argv.append('-d')
    sys.argv.append(path)
    sys.argv.append(os.path.join('..', 'src', 'spec2nexus', 'data', 'APS_spec_data.dat'))


def main():
    selector = spec2nexus.specplot.Selector()
    selector.add('uascan', UAscan_Plotter)
    spec2nexus.specplot_gallery.main()


if __name__ == '__main__':
    # debugging_setup()
    main()

Note that in the uascan, a name for the sample provided by the user is given in self.scan.comments[0]. The plot title is changed to include this and the scan number. The customized plot has a logarithmic y axis:

USAXS uascan, with logarithmic y axis

The most informative view of this data is when the raw data are reduced to \(I(Q)\) and viewed on a log-log plot, but that process is beyond this simple example. See the example Get xy data from HDF5 file below.

[3]	specplot: add option for default log(signal)

SPEC’s hklscan macro

The SPEC hklscan macro appears in a SPEC data file due to either a hscan, kscan, or lscan. In each of these one of the hkl vectors is scanned while the other two remain constant.

The normal handling of the ascan macro plots the last data column against the first. This works for data collected with the hscan. For kscan or lscan macros, the h axis is still plotted by default since it is in the first column.

SPEC hklscan (lscan, in this case), plotted against the (default) first axis H

To display the scanned axis, it is necessary to examine the data in a custom subclass of LinePlotter. The HKLScanPlotter subclass, provided with specplot, defines the get_plot_data() method determines the scanned axis, setting it by name:

plot.axes = [axis,]
self.scan.column_first = axis

Then, the standard plot handling used by LinePlotter uses this information to make the plot.

SPEC hklscan (lscan), plotted against L

Get xy data from HDF5 file

One example of complexity is when SPEC has been used to direct data collection but the data is not stored in the SPEC data file. The SPEC data file scan must provide some indication about where the collected scan data has been stored.

The USAXS instrument at APS has a FlyScan macro that commands the instrument to collect data continuously over the desired \(Q\) range. The data is written to a NeXus HDF5 data file. Later, a data reduction process converts the arrays of raw data to one-dimensional \(I(Q)\) profiles. The best representation of this reduced data is on a log-log plot to reveal the many decades of both \(I\) and \(Q\) covered by the measurement.

With the default handling by LinePlotter, no plot can be generated since the dfata is given in a separate HDF5 file. That file is read with the custom handling of the usaxs_flyscan.py demo:

usaxs_flyscan.py example

#!/usr/bin/env python

#-----------------------------------------------------------------------------
# :author:    Pete R. Jemian
# :email:     prjemian@gmail.com
# :copyright: (c) 2014-2017, Pete R. Jemian
#
# Distributed under the terms of the Creative Commons Attribution 4.0 International Public License.
#
# The full license is in the file LICENSE.txt, distributed with this software.
#-----------------------------------------------------------------------------

'''
Plot data from the USAXS FlyScan macro

.. autosummary::

    ~read_reduced_fly_scan_file
    ~retrieve_flyScanData
    ~USAXS_FlyScan_Structure
    ~USAXS_FlyScan_Plotter

'''

import h5py
import numpy
import os

import spec2nexus.specplot
import spec2nexus.specplot_gallery


# methods picked (& modified) from the USAXS livedata project
def read_reduced_fly_scan_file(hdf5_file_name):
    '''
    read any and all reduced data from the HDF5 file, return in a dictionary
    
    dictionary = {
      'full': dict(Q, R, R_max, ar, fwhm, centroid)
      '250':  dict(Q, R, dR)
      '5000': dict(Q, R, dR)
    }
    '''

    reduced = {}
    hdf = h5py.File(hdf5_file_name, 'r')
    entry = hdf['/entry']
    for key in entry.keys():
        if key.startswith('flyScan_reduced_'):
            nxdata = entry[key]
            d = {}
            for dsname in ['Q', 'R']:
                if dsname in nxdata:
                    value = nxdata[dsname]
                    if value.size == 1:
                        d[dsname] = float(value[0])
                    else:
                        d[dsname] = numpy.array(value)
            reduced[key[len('flyScan_reduced_'):]] = d
    hdf.close()
    return reduced


# $URL: https://subversion.xray.aps.anl.gov/small_angle/USAXS/livedata/specplot.py $
REDUCED_FLY_SCAN_BINS   = 250       # the default
def retrieve_flyScanData(scan):
    '''retrieve reduced, rebinned data from USAXS Fly Scans'''
    path = os.path.dirname(scan.header.parent.fileName)
    key_string = 'FlyScan file name = '
    comment = scan.comments[2]
    index = comment.find(key_string) + len(key_string)
    hdf_file_name = comment[index:-1]
    abs_file = os.path.abspath(os.path.join(path, hdf_file_name))

    plotData = {}
    if os.path.exists(abs_file):
        reduced = read_reduced_fly_scan_file(abs_file)
        s_num_bins = str(REDUCED_FLY_SCAN_BINS)

        choice = reduced.get(s_num_bins) or reduced.get('full')

        if choice is not None:
            plotData = {axis: choice[axis] for axis in 'Q R'.split()}

    return plotData


class USAXS_FlyScan_Plotter(spec2nexus.specplot.LinePlotter):
    '''
    customize `FlyScan` handling, plot :math:`log(I)` *vs.* :math:`log(Q)`
    
    The USAXS FlyScan data is stored in a NeXus HDF5 file in a subdirectory
    below the SPEC data file.  This code uses existing code from the
    USAXS instrument to read that file.
    '''
    
    def retrieve_plot_data(self):
        '''retrieve reduced data from the FlyScan's HDF5 file'''
        # get the data from the HDF5 file
        fly_data = retrieve_flyScanData(self.scan)
        
        if len(fly_data) != 2:
            raise spec2nexus.specplot.NoDataToPlot(str(self.scan))

        self.signal = 'R'
        self.axes = ['Q',]
        self.data = fly_data

        # customize the plot just a bit
        # sample name as given by the user?
        subtitle = '#' + str(self.scan.scanNum)
        subtitle += ' FlyScan: ' + self.scan.comments[0]
        self.set_plot_subtitle(subtitle)
        self.set_x_log(True)
        self.set_y_log(True)
        self.set_x_title(r'$|\vec{Q}|, 1/\AA$')
        self.set_y_title(r'USAXS $R(|\vec{Q}|)$, a.u.')
    
    def plottable(self):
        '''
        can this data be plotted as expected?
        '''
        if self.signal in self.data:
            signal = self.data[self.signal]
            if signal is not None and len(signal) > 0 and len(self.axes) == 1:
                if len(signal) == len(self.data[self.axes[0]]):
                    return True
        return False


def debugging_setup():
    import sys
    import shutil
    sys.path.insert(0, os.path.join('..', 'src'))
    path = '__usaxs__'
    shutil.rmtree(path, ignore_errors=True)
    os.mkdir(path)
    sys.argv.append('-d')
    sys.argv.append(path)
    sys.argv.append(os.path.join('..', 'src', 'spec2nexus', 'data', '02_03_setup.dat'))


def main():
    selector = spec2nexus.specplot.Selector()
    selector.add('FlyScan', USAXS_FlyScan_Plotter)
    spec2nexus.specplot_gallery.main()


if __name__ == '__main__':
    # debugging_setup()
    main()

The data is then rendered in a customized log-log plot of \(I(Q)\):

USAXS FlyScan, handled by USAXS_FlyScan_Plotter

Usage

When a custom scan macro handler is written and installed using code similar to the custom ascan handling above:

def main():
    selector = spec2nexus.specplot.Selector()
    selector.add('ascan', Custom_Ascan)
    spec2nexus.specplot_gallery.main()


if __name__ == '__main__':
    main()

then the command line arugment handling from spec2nexus.specplot_gallery.main() can be accessed from the command line for help and usage information.

Usage:

user@localhost ~/.../spec2nexus/demo $ ./ascan.py
usage: ascan.py [-h] [-r] [-d DIR] paths [paths ...]
ascan.py: error: too few arguments

Help:

user@localhost ~/.../spec2nexus/demo $ ./ascan.py -h
usage: ascan.py [-h] [-r] [-d DIR] paths [paths ...]

read a list of SPEC data files (or directories) and plot images of all scans

positional arguments:
  paths              SPEC data file name(s) or directory(s) with SPEC data
                     files

optional arguments:
  -h, --help         show this help message and exit
  -r                 sort images from each data file in reverse chronolgical
                     order
  -d DIR, --dir DIR  base directory for output (default:/home/prjemian/Documen
                     ts/eclipse/spec2nexus/demo)

spec2nexus.eznx

(Easy NeXus) support library for reading & writing NeXus HDF5 files using h5py

How to use spec2nexus.eznx

Here is a simple example to write a NeXus data file using eznx:

#!/usr/bin/env python
# -*- coding: utf-8 -*-

"""
Writes a simple NeXus HDF5 file using h5py with links.

This example is based on ``writer_2_1`` of the NeXus Manual:
http://download.nexusformat.org/doc/html/examples/h5py/index.html
"""

from spec2nexus import eznx


HDF5_FILE = "eznx_example.hdf5"

I_v_TTH_DATA = """
17.92608    1037
17.92558    2857
17.92508    23819
17.92458    49087
17.92408    66802
17.92358    66206
17.92308    64129
17.92258    56795
17.92208    29315
17.92158    6622
17.92108    1321
"""
# ---------------------------

tthData, countsData = zip(
    *[map(float, _.split()) for _ in I_v_TTH_DATA.strip().splitlines()]
)

f = eznx.makeFile(HDF5_FILE)  # create the HDF5 NeXus file
f.attrs["default"] = "entry"

nxentry = eznx.makeGroup(f, "entry", "NXentry", default="data")
nxinstrument = eznx.makeGroup(nxentry, "instrument", "NXinstrument")
nxdetector = eznx.makeGroup(nxinstrument, "detector", "NXdetector")

tth = eznx.makeDataset(nxdetector, "two_theta", tthData, units="degrees")
counts = eznx.makeDataset(nxdetector, "counts", countsData, units="counts")

nxdata = eznx.makeGroup(
    nxentry,
    "data",
    "NXdata",
    signal=1,
    axes="two_theta",
    two_theta_indices=0,
)
eznx.makeLink(nxdetector, tth, nxdata.name + "/two_theta")
eznx.makeLink(nxdetector, counts, nxdata.name + "/counts")

f.close()  # be CERTAIN to close the file

The output of this code is an HDF5 file (binary). It has this structure:

    eznx_example.hdf5:NeXus data file
      @default = entry
      entry:NXentry
        @NX_class = NXentry
        @default = data
        data:NXdata
          @NX_class = NXdata
          @signal = counts
          @axes = two_theta
          @two_theta_indices = 0
          counts --> /entry/instrument/detector/counts
          two_theta --> /entry/instrument/detector/two_theta
        instrument:NXinstrument
          @NX_class = NXinstrument
          detector:NXdetector
            @NX_class = NXdetector
            counts:NX_FLOAT64[11] = __array
              @units = counts
              @target = /entry/instrument/detector/counts
              __array = [1037.0, 2857.0, 23819.0, '...', 1321.0]
            two_theta:NX_FLOAT64[11] = __array
              @units = degrees
              @target = /entry/instrument/detector/two_theta
              __array = [17.926079999999999, 17.92558, 17.925080000000001, '...', 17.92108]

NeXus HDF5 File Structure

The output of this code is an HDF5 file (binary). It has this general structure (indentation shows HDF5 groups, @ signs describe attributes of the preceding item):

  hdf5_file:NeXus data file
     @default = S1
     S1:NXentry     (one NXentry for each scan)
        @default = data
        title = #S
        T or M: #T or #M
        comments: #C for entire scan
        date: #D
        scan_number: #S
        G:NXcollection
           @description = SPEC geometry arrays, meanings defined by SPEC diffractometer support
           G0:NX_FLOAT64[] #G0
           G1:NX_FLOAT64[] #G1
           ...
        data:NXdata
           @description = SPEC scan data (content from #L and data lines)
           @signal = I0
           @axes = mr
           @mr_indices = 0
           Epoch:NX_FLOAT64[]
           I0:NX_FLOAT64[]         (last data column)
             @spec_name = I0
           mr:NX_FLOAT64[]         (first data column)
           ...
        metadata:NXcollection
           @description = SPEC metadata (UNICAT-style #H & #V lines)
           ARenc_0:NX_FLOAT64 = 0.0
           ...
        positioners:NXcollection
           @description = SPEC positioners (#P & #O lines)
           mr:NX_FLOAT64
           ...

APIs provided:

spec2nexus.writer

This is an internal library of the spec2nexus software. It is not expected that users of this package will need to call the writer module directly.

source code documentation

(internal library) Parses SPEC data using spec2nexus.eznx API (only requires h5py)

class spec2nexus.writer.Writer(spec_data)

writes out scans from SPEC data file to NeXus HDF5 file

Parameters:spec_data (obj) – instance of SpecDataFile

mca_spectra(nxdata, scan, primary_axis_label)

internal: parse for optional 2-D MCA spectra

mesh(nxdata, scan)

internal: data parser for 2-D mesh and hklmesh

oneD(nxdata, scan)

internal: generic data parser for 1-D column data, returns signal and axis

root_attributes()

internal: returns the attributes to be written to the root element as a dict

save(hdf_file, scan_list=None)

save the information in this SPEC data file to a NeXus HDF5 file

Each scan in scan_list will be converted to a NXentry group. Scan data will be placed in a NXdata group where the attribute signal=1 is the last column and the corresponding attribute axes=<name of the first column>. There are variations on this for 2-D and higher dimensionality data, such as mesh scans.

In general, the tree structure of the NeXus HDF5 file is:

hdf5_file: NXroot
    @default="S1"
    definition="NXspecdata"
    # attributes
    S1:NXentry
        @default="data"
        # attributes and metadata fields
        data:NXdata
            @signal=<name of signal field>
            @axes=<name(s) of axes of signal>
            @<axis>_indices=<list of indices in "axis1">
            <signal_is_the_last_column>:NX_NUMBER[number of points] = ... data ...
                @signal=1
                @axes='<axis_is_name_of_first_column>'
                @<axis>_indices=<list of indices in "axis1" used as dimension scales of the "signal">
            <axis_is_name_of_first_column>:NX_NUMBER[number of points] = ... data ...
            # other columns from the scan

Parameters:

• hdf_file (str) – name of NeXus/HDF5 file to be written
• scanlist ([int]) – list of scan numbers to be read

save_data(nxdata, scan)

internal: store the scan data

save_dict(group, data)

internal: store a dictionary

save_scan(nxentry, scan)

internal: save the data from each SPEC scan to its own NXentry group

write_ds(group, label, data, **attr)

internal: writes a dataset to the HDF5 file, records the SPEC name as an attribute

---

source code methods

addAttributes	add attributes to an h5py data item
makeFile	create and open an empty NeXus HDF5 file using h5py
makeDataset	create and write data to a dataset in the HDF5 file hierarchy
makeExternalLink	create an external link from sourceFile, sourcePath to targetPath in hdf5FileObject
makeGroup	create a NeXus group
openGroup	open or create the NeXus/HDF5 group, return the object
makeLink	create an internal NeXus (hard) link in an HDF5 file
read_nexus_field	get a dataset from the HDF5 parent group
read_nexus_group_fields	return the fields in the NeXus group as a dict(name=dataset)
write_dataset	write to the NeXus/HDF5 dataset, create it if necessary, return the object

source code documentation

(Easy NeXus) support reading & writing NeXus HDF5 files using h5py

predecessor:NeXus h5py example code: my_lib.py [1]

[1]	http://download.nexusformat.org/doc/html/examples/h5py/index.html#mylib-support-module

Dependencies

• h5py: interface to HDF5 file format

Exceptions raised

• None

Example

root = eznx.makeFile('test.h5', creator='eznx', default='entry')
nxentry = eznx.makeGroup(root, 'entry', 'NXentry', default='data')
ds = eznx.write_dataset(nxentry, 'title', 'simple test data')
nxdata = eznx.makeGroup(nxentry, 'data', 'NXdata', signal='counts', axes='tth', tth_indices=0)
ds = eznx.write_dataset(nxdata, 'tth', [10.0, 10.1, 10.2, 10.3], units='degrees')
ds = eznx.write_dataset(nxdata, 'counts', [1, 50, 1000, 5], units='counts', axes="tth")
root.close()

The resulting (binary) data file has this structure:

test.h5:NeXus data file
  @creator = eznx
  @default = 'entry'
  entry:NXentry
    @NX_class = NXentry
    @default = 'data'
    title:NX_CHAR = simple test data
    data:NXdata
      @NX_class = NXdata
      @signal = 'counts'
      @axes = 'tth'
      @tth_indices = 0
      counts:NX_INT64[4] = [1, 50, 1000, 5]
        @units = counts
        @axes = tth
      tth:NX_FLOAT64[4] = [10.0, 10.1, 10.199999999999999, 10.300000000000001]
        @units = degrees

Classes and Methods

spec2nexus.eznx.addAttributes(parent, **attr)

add attributes to an h5py data item

Parameters:

• parent (obj) – h5py parent object
• attr (dict) – optional dictionary of attributes

spec2nexus.eznx.makeDataset(parent, name, data=None, **attr)

create and write data to a dataset in the HDF5 file hierarchy

Any named parameters in the call to this method will be saved as attributes of the dataset.

Parameters:

• parent (obj) – parent group
• name (str) – valid NeXus dataset name
• data (obj) – the information to be written
• attr (dict) – optional dictionary of attributes

Returns:

h5py dataset object

spec2nexus.eznx.makeExternalLink(hdf5FileObject, sourceFile, sourcePath, targetPath)

create an external link from sourceFile, sourcePath to targetPath in hdf5FileObject

Parameters:

• hdf5FileObject (obj) – open HDF5 file object
• sourceFile (str) – file containing existing HDF5 object at sourcePath
• sourcePath (str) – path to existing HDF5 object in sourceFile
• targetPath (str) – full node path to be created in current open HDF5 file, such as /entry/data/data

Note

Since the object retrieved is in a different file, its “.file” and “.parent” properties will refer to objects in that file, not the file in which the link resides.

See:http://www.h5py.org/docs-1.3/guide/group.html#external-links

This routine is provided as a reminder how to do this simple operation.

spec2nexus.eznx.makeFile(filename, **attr)

create and open an empty NeXus HDF5 file using h5py

Any named parameters in the call to this method will be saved as attributes of the root of the file. Note that **attr is a dictionary of named parameters.

Parameters:

• filename (str) – valid file name
• attr (dict) – optional dictionary of attributes

Returns:

h5py file object

spec2nexus.eznx.makeGroup(parent, name, nxclass, **attr)

create a NeXus group

Any named parameters in the call to this method will be saved as attributes of the group. Note that **attr is a dictionary of named parameters.

Parameters:

• parent (obj) – parent group
• name (str) – valid NeXus group name
• nxclass (str) – valid NeXus class name
• attr (dict) – optional dictionary of attributes

Returns:

h5py group object

spec2nexus.eznx.makeLink(parent, sourceObject, targetName)

create an internal NeXus (hard) link in an HDF5 file

Parameters:

• parent (obj) – parent group of source
• sourceObject (obj) – existing HDF5 object
• targetName (str) – HDF5 node path to be created, such as /entry/data/data

spec2nexus.eznx.openGroup(parent, name, nx_class, **attr)

open or create the NeXus/HDF5 group, return the object

Parameters:

• parent (obj) – h5py parent object
• name (str) – valid NeXus group name to open or create
• nxclass (str) – valid NeXus class name (base class or application definition)
• attr (dict) – optional dictionary of attributes

spec2nexus.eznx.read_nexus_field(parent, dataset_name, astype=None)

get a dataset from the HDF5 parent group

Parameters:

• parent (obj) – h5py parent object
• dataset_name (str) – name of the dataset (NeXus field) to be read
• astype (obj) – option to return as different data type

spec2nexus.eznx.read_nexus_group_fields(parent, name, fields)

return the fields in the NeXus group as a dict(name=dataset)

This routine provides a mass way to read a directed list of datasets (NeXus fields) in an HDF5 group.

Parameters:

• parent (obj) – h5py parent object
• name (str) – name of the group containing the fields
• fields ([name]) – list of field names to be read

Returns:

dictionary of {name:dataset}

Raises:

KeyError – if a field is not found

spec2nexus.eznx.write_dataset(parent, name, data, **attr)

write to the NeXus/HDF5 dataset, create it if necessary, return the object

Parameters:

• parent (obj) – h5py parent object
• name (str) – valid NeXus dataset name to write
• data (obj) – the information to be written
• attr (dict) – optional dictionary of attributes

spec2nexus.plugin

An extensible plug-in architecture is used to handle the different possible control line control lines (such as #F, #E, #S, …) in a SPEC data file.

A SPEC control line provides metadata about the SPEC scan or SPEC data file.

Plugins can be used to parse or ignore certain control lines in SPEC data files. Through this architecture, it is possible to support custom control lines, such as #U (SPEC standard control line for any user data). One example is support for the UNICAT-style of metadata provided in the scan header.

Plugins are now used to handle all control lines in spec2nexus.spec. Any control line encountered but not recognized will be placed as text in a NeXus NXnote group named unrecognized_NNN (where NNN is from 1 to the maximum number of unrecognized control lines).

Supplied spec plugin modules

These plugin modules are supplied:

spec_common	SPEC data file standard control lines
fallback	Fallback handling for any SPEC data file control lines not recognized by other handlers
apstools_specwriter	#MD : Bluesky metadata from apstools SpecWriterCallback.
unicat	#H & #V - Metadata in SPEC data files as defined by APS UNICAT
uim	#UIM : Image header information from EPICS areaDetector
uxml	#UXML: UXML structured metadata
XPCS	SPEC data file control lines unique to the APS XPCS instrument

XPCS plugin

SPEC data file control lines unique to the APS XPCS instrument

class spec2nexus.plugins.XPCS.XPCS_CCD

#CCD

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

optional: Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.XPCS.XPCS_VA

#VA

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to write VA data

class spec2nexus.plugins.XPCS.XPCS_VD

#VD

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to write VD data

class spec2nexus.plugins.XPCS.XPCS_VE

#VE

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to write VE data

class spec2nexus.plugins.XPCS.XPCS_XPCS

#XPCS

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

optional: Describe how to store this data in an HDF5 NeXus file

apstools SpecWriterCallback metadata plugin

Looks for #MD control line control lines. These lines contain metadata supplied to the bluesky RunEngine and recorded during the execution of a scan. The data are stored in a dictionary of each scan: scan.MD. If there are no #MD control lines, then scan.MD does not exist.

see https://apstools.readthedocs.io/en/latest/source/_filewriters.html#apstools.filewriters.SpecWriterCallback

---

#MD : Bluesky metadata from apstools SpecWriterCallback.

EXAMPLE:

#MD APSTOOLS_VERSION = 1.1.0
#MD BLUESKY_VERSION = 1.5.2
#MD EPICS_CA_MAX_ARRAY_BYTES = 1280000
#MD EPICS_HOST_ARCH = linux-x86_64
#MD OPHYD_VERSION = 1.3.2
#MD beamline_id = APS USAXS 9-ID-C
#MD datetime = 2019-04-19 10:04:44.400750
#MD login_id = usaxs@usaxscontrol.xray.aps.anl.gov
#MD pid = 27062
#MD proposal_id = testing Bluesky installation
#MD purpose = tuner
#MD tune_md = {'width': -0.004, 'initial_position': 8.824885, 'time_iso8601': '2019-04-19 10:04:44.402643'}
#MD tune_parameters = {'num': 31, 'width': -0.004, 'initial_position': 8.824885, 'peak_choice': 'com', 'x_axis': 'm_stage_r', 'y_axis': 'I0_USAXS'}

class spec2nexus.plugins.apstools_specwriter.MD_apstools

#MD – Bluesky metadata from apstools SpecWriterCallback

process(text, scan, *args, **kws)

read #MD lines from SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

Fallback plugin

Fallback handling for any SPEC data file control lines not recognized by other handlers

class spec2nexus.plugins.fallback.UnrecognizedControlLine

unrecognized control line

process(text, spec_obj, *args, **kws)

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

write the data in a NeXus group named unrecognized

SPEC standard plugin

SPEC data file standard control lines

see:SPEC manual, Standard Data File Format, http://www.certif.com/spec_manual/user_1_4_1.html

class spec2nexus.plugins.spec_common.SPEC_Comment

#C – any comment either in the scan header or somewhere in the scan

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): comments
• (SpecDataFileScan): comments

HDF5/NeXus REPRESENTATION

• file root-level attribute: SPEC_comments : string array of all comments from first header block
• dataset named comments under /NXentry group, such as /S1/comments : string array of all comments from this scan block

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_CountTime

#T – counting against this constant number of seconds (see #M)

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): T

HDF5/NeXus REPRESENTATION

• Dataset named T in the NXentry group, such as /S1/T
• Dataset named counting_basis in the NXentry group with value SPEC scan with constant counting time, such as /S1/counting_basis

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_CounterMnemonics

#j – mnemonics of counter (new with SPEC v6)

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): j : mnemonics

HDF5/NeXus REPRESENTATION

• NXnote group named counter_cross_reference in the NXentry group, such as /S1/counter_cross_reference
  • datasets with names supplied as SPEC counter mnemonics, string values supplied as SPEC counter names

postprocess(header, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, header, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, header, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_CounterNames

#J – names of counters (each separated by two spaces) (new with SPEC v6)

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): J : mnemonics

HDF5/NeXus REPRESENTATION

• NXnote group named counter_cross_reference in the NXentry group, such as /S1/counter_cross_reference
  • datasets with names supplied as SPEC counter mnemonics, string values supplied as SPEC counter names

postprocess(header, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, header, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, header, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_DataLine

(scan_data) – scan data line

Scan data could include interspersed MCA data or even describe 2-D or 3-D data. T his method reads the data lines and buffers them for post-processing in spec2nexus.plugins.spec_common_spec2nexus.data_lines_postprocessing().

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): data_lines : values
• (SpecDataFileScan): data : {labels: values}

HDF5/NeXus REPRESENTATION

• NXdata group named data in the NXentry group, such as /S1/data
• datasets with names supplied in L, array values collected in data_lines

match_key(text)

Easier to try conversion to number than construct complicated regexp

postprocess(scan, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_Date

#D – date/time stamp

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): date str, ISO8601 format

HDF5/NeXus REPRESENTATION

• file root-level attribute: SPEC_date str (value for 1st header block is used)

process(text, sdf_object, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, sdf_object, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_Epoch

#E – the UNIX epoch (seconds from 00:00 GMT 1/1/70)

In SPEC data files, the #E control line indicates the start of a header block.

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): epoch int

HDF5/NeXus REPRESENTATION

• file root-level attribute: SPEC_epoch int

process(buf, sdf_object, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_File

#F – original data file name (starts a file header block)

Module spec2nexus.spec is responsible for handling this control line.

IN-MEMORY REPRESENTATION

• (SpecDataFile): fileName
• (SpecDataFileHeader) : file

HDF5/NeXus REPRESENTATION

• file root-level attribute: SPEC_file

process(text, spec_file_obj, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_Geometry

#G – diffractometer geometry (numbered rows: #G0, #G1, …)

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): G

HDF5/NeXus REPRESENTATION

• NXnote group named G in the NXentry group, such as /S1/G
  • Datasets created from dictionary <scan>.G (indexed by number from the scan block, such as G0, G1, …). Meaning of contents for each index are defined by geometry-specific SPEC diffractometer support.
• NXinstrument & NXsample groups for interpreted information

postprocess(scan, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_HKL

#Q – \(Q\) (\(hkl\)) at start of scan

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): Q

HDF5/NeXus REPRESENTATION

• Dataset named Q in the NXentry group, such as /S1/M

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_Labels

#L – data column labels

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): L : labels
• (SpecDataFileScan): data : {labels: values}

HDF5/NeXus REPRESENTATION

• NXdata group named data in the NXentry group, such as /S1/data
• datasets with names supplied in L, array values collected in data_lines

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_MCA

#@MCA – MCA data formatting declaration (ignored for now)

declares this scan contains MCA data (SPEC’s array_dump() format, such as "#@MCA 16C")

From documentation provided by the ESRF BLISS group: (http://www.esrf.eu/blissdb/macros/getsource.py?macname=mca.mac)

#@MCA 16C Format string passed to data_dump() function. This format string is held by the global variable “MCA_FMT” and can then been adapted to particular needs. “%%16C” is the default. It dumps data on 1 line, cut every 16 points:

@A 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0\
 0 0 0 0 0 0 0 0 0 0 0 ...

“%%16” would do the same without any backslash “1” would dump 1 point per line, …

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_MCA_Array

@A – MCA Array data

MCA data. Each value is the content of one channel, or an integrated value over several channels if a reduction was applied.

Since the MCA Array data is interspersed with scan data, this method reads the data lines and buffers them for post-processing in spec2nexus.plugins.spec_common_spec2nexus.data_lines_postprocessing().

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): data_lines : values
• (SpecDataFileScan): data : {labels: values}

HDF5/NeXus REPRESENTATION

• NXdata group named data in the NXentry group, such as /S1/data
  • Dataset _mca_ : float MCA data reported on @A lines

  • Dataset _mca_channel_: provided as HDF5 dimension scale for _mca_ dataset

    • if CALIB data specified: float scaled MCA channels – \(x_k = a +bk + ck^2\)
    • if CALIB data not specified: int MCA channel numbers

postprocess(scan, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_MCA_Calibration

#@CALIB – coefficients to compute a scale based on the MCA channel number

\(x_k = a +bk + ck^2\) for MCA data, \(k\) is channel number

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): MCA[‘CALIB’] = dict(a, b, c)

HDF5/NeXus REPRESENTATION

• defines a dimension scale for MCA data
• NXnote group named MCA in the NXentry group, such as /S1/MCA
  • Dataset calib_a : float
  • Dataset calib_b : float
  • Dataset calib_c : float

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_MCA_ChannelInformation

#@CHANN – MCA channel information

number_saved, first_saved, last_saved, reduction_coef

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): MCA[‘CALIB’] = dict(number_saved, first_saved, last_saved, reduction_coef)

HDF5/NeXus REPRESENTATION

• NXnote group named MCA in the NXentry group, such as /S1/MCA
  • Dataset number_saved : int number of channels saved
  • Dataset first_saved : int first channel saved
  • Dataset first_saved : int last channel saved
  • Dataset reduction_coef : float reduction coefficient

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_MCA_CountTime

#@CTIME – MCA count times

preset_time, elapsed_live_time, elapsed_real_time

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): MCA[‘CALIB’] = dict(preset_time, elapsed_live_time, elapsed_real_time)

HDF5/NeXus REPRESENTATION

• NXnote group named MCA in the NXentry group, such as /S1/MCA
  • Dataset preset_time : float
  • Dataset elapsed_live_time : float
  • Dataset elapsed_real_time : float

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_MCA_RegionOfInterest

#@ROI – MCA ROI (Region Of Interest) channel information

ROI_name, first_chan, last_chan

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): MCA[‘ROI’] = {ROI_name:dict(first_chan, last_chan)}

HDF5/NeXus REPRESENTATION

• NXnote group ROI in in NXnote group named MCA in the NXentry group, such as /S1/MCA/ROI
  • Dataset {ROI_name} : int [first_chan, last_chan]

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_Monitor

#M – counting against this constant monitor count (see #T)

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): M

HDF5/NeXus REPRESENTATION

• Dataset named M in the NXentry group, such as /S1/M
• Dataset named counting_basis in the NXentry group with value SPEC scan with constant monitor count, such as /S1/counting_basis

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_NormalizingFactor

#I – intensity normalizing factor

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): I

HDF5/NeXus REPRESENTATION

• Dataset named intensity_factor in the NXentry group, such as /S1/intensity_factor

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_NumColumns

#N – number of columns of data [ num2 sets per row ]

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): N : [int]

HDF5/NeXus REPRESENTATION

• not written to file

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_PositionerMnemonics

#o – positioner mnemonics (new with SPEC v6)

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): o : mnemonics

HDF5/NeXus REPRESENTATION

• NXnote group named positioner_cross_reference in the NXentry group, such as /S1/positioner_cross_reference
  • datasets with names supplied as SPEC positioner mnemonics, string values supplied as SPEC positioner names

postprocess(header, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, header, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, header, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_PositionerNames

#O – positioner names (numbered rows: #O0, #O1, …)

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader) : O : label
• (SpecDataFileScan): positioner : {label: value}

HDF5/NeXus REPRESENTATION

• NXnote group named positioners in the NXentry group, such as /S1/positioners
  • datasets created from dictionary <scan>.positioner
• NXnote group named positioner_cross_reference in the NXentry group, such as /S1/positioner_cross_reference
  • datasets with names supplied as SPEC positioner mnemonics, string values supplied as SPEC positioner names

process(text, sdf_object, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_Positioners

#P – positioner values at start of scan (numbered rows: #P0, #P1, …)

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader) : O : label
• (SpecDataFileScan): positioner : {label: value}

HDF5/NeXus REPRESENTATION

• NXnote group named positioners in the NXentry group, such as /S1/positioners
  • datasets created from dictionary <scan>.positioner

postprocess(scan, *args, **kws)

interpret the motor positions from the scan header

Parameters:scan (SpecDataFileScan) – data from a single SPEC scan

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_Scan

#S – SPEC scan

In SPEC data files, the #S control line indicates the start of a scan block. Each scan will be written to a separate NXentry group in the HDF5 file.

NXentry:

“The top-level NeXus group which contains all the data and associated information that comprise a single measurement.”

– http://download.nexusformat.org/doc/html/classes/base_classes/NXentry.html

IN-MEMORY REPRESENTATION

• (SpecDataFile):
• (SpecDataFileHeader):

HDF5/NeXus REPRESENTATION

• /NXentry group named ‘S%d` scan_number at root level, such as /S1

process(part, sdf, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.spec_common.SPEC_TemperatureSetPoint

#X – Temperature Set Point (desired temperature)

The default declaration of the #X control line is written:

def Fheader '_cols++;printf("#X %gKohm (%gC)\n",TEMP_SP,DEGC_SP)'

The supplied macro alters this slightly (replacing %g with %f) and uses the spec2nexus.scanf.scanf() implementation with this format:

fmt = "#X %fKohm (%fC)"

Depending on the circumstances, this might be a good candidate to override with a custom ControlLineHandler that parses the data as written. If the conversion process fails for any reason in this implementation, the #X line is ignored.

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): TEMP_SP
• (SpecDataFileScan): DEGC_SP

HDF5/NeXus REPRESENTATION

• Dataset named TEMP_SP in the NXentry group, such as /S1/TEMP_SP
• Dataset named DEGC_SP in the NXentry group, such as /S1/DEGC_SP

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

class spec2nexus.plugins.spec_common.SPEC_UserReserved

#U – Reserved for user

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader): U, [str]
• (SpecDataFileScan): U, [str]

HDF5/NeXus REPRESENTATION

• Within a group named UserReserved in the NXentry group: dataset(s) named header_## (from the SPEC data file header section) or item_## (from the SPEC data file scan section), such as /S1/UserReserved/header_1 and /S1/UserReserved/item_5

process(text, sdf_object, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, sdf_object, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

spec2nexus.plugins.spec_common.combine_split_NM_lines(nm, data_lines)

combine split lines of data

#N N [M]

Indicates there are N columns of data. If M is present, it indicates there are M sets of data columns on each line.

spec2nexus.plugins.spec_common.data_lines_postprocessing(scan)

interpret the data lines from the body of the scan

Parameters:scan (SpecDataFileScan) – data from a single SPEC scan

spec2nexus.plugins.spec_common.data_lines_writer(h5parent, writer, scan, *args, **kws)

Describe how to store scan data in an HDF5 NeXus file

UIM plugin

#UIM : Image header information from EPICS areaDetector

deprecated:in favor of UXML

class spec2nexus.plugins.uim.UIM_generic

#UIM – various image header information

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

unicat plugin

#H & #V - Metadata in SPEC data files as defined by APS UNICAT

Handles the UNICAT control lines which write additional metadata in the scans using #H/#V pairs of labels/values.

class spec2nexus.plugins.unicat.UNICAT_MetadataMnemonics

#H – UNICAT metadata names (numbered rows: #H0, #H1, …)

Individual metadata names are expected to be single-word strings but may be multi-word strings as long as the words in the string are separated by only one space. The delimiter between metadata names is two consecutive spaces. A tab (\t) character is also acceptable but should be avoided.

IN-MEMORY REPRESENTATION

• (SpecDataFileHeader) : H : labels
• (SpecDataFileScan): metadata : {labels: values}

HDF5/NeXus REPRESENTATION

• NXnote group named metadata below the NXentry group, such as /entry/metadata
  • datasets created from dictionary <scan>.metadata

process(text, spec_obj, *args, **kws)

required: handle this line from a SPEC data file

class spec2nexus.plugins.unicat.UNICAT_MetadataValues

#V – UNICAT metadata values (numbered rows: #V0, #V1, …)

Individual metadata values are expected to be numbers but may be multi-word strings as long as the words in the string are separated by only one space. The delimiter between metadata values is two consecutive spaces. A tab ('\t') character is also acceptable but should be avoided.

All numerical values will be converted into floating point numbers. Only if that conversion fails, the text of the value will be reported verbatim.

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): V : values
• (SpecDataFileScan): metadata : {labels: values}

HDF5/NeXus REPRESENTATION

• NXnote group named metadata below the NXentry group, such as /entry/metadata
  • datasets created from dictionary <scan>.metadata

postprocess(scan, *args, **kws)

interpret the UNICAT metadata (mostly floating point) from the scan header

Parameters:scan (SpecDataFileScan) – data from a single SPEC scan (instance of SpecDataFileScan)

process(text, scan, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

#UXML: UXML metadata plugin

Looks for #UXML control line control lines. These lines contain metadata written as XML structures and formatted according to the supplied XML Schema uxml.xsd in the same directory as the uxml.py plugin. The lines which comprise the XML are written as a list in each scan: scan.UXML. If there are no #UXML control lines, then scan.UXML does not exist.

Once the scan has been fully read scan.UXML is converted into an XML document structure (using the lxml.etree package) which is stored in scan.UXML_root. The structure is validated against the XML Schema uxml.xsd. If invalid, the error message is reported by raising a UXML_Error python exception.

A fully-validated structure can be written using the Writer class. The UXML metadata is written to the scan’s NXentry group as subgroup named UXML with NeXus base class NXnote. The hierarchy within this UXML is defined from the content provided in the SPEC scan.

Please consult the XML Schema file for the rules governing the use of #UXML in a SPEC data file: * uxml.xsd

---

#UXML: UXML structured metadata

exception spec2nexus.plugins.uxml.UXML_Error

class spec2nexus.plugins.uxml.UXML_metadata

#UXML – XML metadata in scan header

IN-MEMORY REPRESENTATION

• (SpecDataFileScan): UXML : XML document root

HDF5/NeXus REPRESENTATION

• various items below the NXentry parent group, as indicated in the UXML

Public methods

process(text, scan, *args, **kws)

read #UXML lines from SPEC data file into scan.UXML

Internal methods

walk_xml_tree(h5parent, xml_node)	parse the XML node into HDF5 objects
make_NeXus_links()	create all the hardlinks as directed
prune_dict(d, keys)	remove keys from dictionary d
dataset(h5parent, xml_node)	HDF5/NeXus dataset specification
group(h5parent, xml_node)	HDF5/NeXus group specification
hardlink(h5parent, xml_node)	HDF5/NeXus hard link specification

dataset(h5parent, xml_node)

HDF5/NeXus dataset specification

group(h5parent, xml_node)

HDF5/NeXus group specification

hardlink(h5parent, xml_node)

HDF5/NeXus hard link specification

make_NeXus_links()

create all the hardlinks as directed

postprocess(scan, *args, **kws)

convert the UXML text into an XML object (scan.UXML_root)

Parameters:scan (SpecDataFileScan) – data from a single SPEC scan

process(text, scan, *args, **kws)

read #UXML lines from SPEC data file into scan.UXML

prune_dict(d, keys)

remove keys from dictionary d

walk_xml_tree(h5parent, xml_node)

parse the XML node into HDF5 objects

writer(nxentry, writer, scan, *args, **kws)

Describe how to store this data in an HDF5 NeXus file

Writing a custom plugin

While spec2nexus provides a comprehensive set of plugins to handle the common SPEC control line control lines, custom control lines are used at many facilities to write additional scan data and scan metadata into the SPEC data file. Custom plugins are written to process these additions.

How to write a custom plugin module

The code to write plugins has changed with release 2021.0.0.

The changes are summarized in the section below titled Changes in plugin format with release 2021.0.0.

Sections

• Load a plugin module
• Write a plugin module
• Full Example: #PV control line
• Example to ignore a #Y control line
• Postprocessing
• Example postprocessing
• Summary Example Custom Plugin with postprocessing
• Custom HDF5 writer
• Custom key match function
• Summary Requirements for custom plugin
• Changes in plugin format with release 2021.0.0
• Footnotes

A custom plugin module for spec2nexus.spec is provided in a python module (Python source code file). In this custom plugin module are subclasses for each new control line to be supported. An exception will be raised if a custom plugin module tries to provide support for an existing control line.

Load a plugin module

Control line handling plugins for spec2nexus will automatically register themselves when their module is imported. Be sure that you call get_plugin_manager() before you import your plugin code. This step sets up the plugin manager to automatically register your new plugin.

import spec2nexus.plugin
import spec2nexus.spec

# get the plugin manager BEFORE you import any custom plugins
manager = plugin.get_plugin_manager()

import MY_PLUGIN_MODULE
# ... more if needed ...

# read a SPEC data file, scan 5
spec_data_file = spec2nexus.spec.SpecDataFile("path/to/spec/datafile")
scan5 = spec_data_file.getScan(5)

Write a plugin module

Give the custom plugin module a name ending with .py. As with any Python module, the name must be unique within a directory. If the plugin is not in your working directory, there must be a __init__.py file in the same directory (even if that file is empty) so that your plugin module can be loaded with import <MODULE>.

Plugin module setup

The six package

The six package is used to make our plugins run with either Python 2.7 or Python 3.5+.

Please view the existing plugins in spec_common for examples. The custom plugin module should contain, at minimum one subclass of spec2nexus.plugin.ControlLineHandler which is decorated with @six.add_metaclass(spec2nexus.plugin.AutoRegister). The add_metaclass decorator allows our custom ControlLineHandlers to register themselves when their module is imported. A custom plugin module can contain many such handlers, as needs dictate.

Useful import

It is also useful to import the strip_first_word() utility method.

These imports are necessary to to write plugins for spec2nexus:

import six
from spec2nexus.plugin import AutoRegister
from spec2nexus.plugin import ControlLineHandler
from spec2nexus.utils import strip_first_word

regular expressions

There are several regular expression testers available on the web. Try this one, for example: http://regexpal.com/

Attribute: ``key`` (required)

Each subclass must define key key as a regular expression match for the control line key. It is possible to override any of the supplied plugins for scan control line control lines. Caution is advised to avoid introducing instability.

Attribute: ``scan_attributes_defined`` (optional)

If your plugin creates any attributes to the spec2nexus.spec.SpecDataScan object (such as the hypotetical scan.hdf5_path and scan.hdf5_file), you declare the new attributes in the scan_attributes_defined list. Such as this:

scan_attributes_defined = ['hdf5_path', 'hdf5_file']

Method: ``process()`` (required)

Each subclass must also define a process() method to process the control line. A NotImplementedError exception is raised if key is not defined.

Method: ``match_key()`` (optional)

For difficult regular expressions (or other situations), it is possible to replace the function that matches for a particular control line key. Override the handler’s match_key() method. For more details, see the section Custom key match function.

Method: ``postprocess()`` (optional)

For some types of control lines, processing can only be completed after all lines of the scan have been read. In such cases, add a line such as this to the process() method:

scan.addPostProcessor(self.key, self.postprocess)

(You could replace self.key here with some other text. If you do, make sure that text will be unique as it is used internally as a python dictionary key.) Then, define a postprocess() method in your handler:

def postprocess(self, scan, *args, **kws):
    # handle your custom info here

See section Postprocessing below for more details. See spec2nexus.plugins.spec_common for many examples.

Method: ``writer()`` (optional)

Writing a NeXus HDF5 data file is one of the main goals of the spec2nexus package. If you intend data from your custom control line handler to end up in the HDF5 data file, add a line such as this to either the process() or postprocess() method:

scan.addH5writer(self.key, self.writer)

Then, define a writer() method in your handler. Here’s an example:

def writer(self, h5parent, writer, scan, nxclass=None, *args, **kws):
    """Describe how to store this data in an HDF5 NeXus file"""
    desc='SPEC positioners (#P & #O lines)'
    group = makeGroup(h5parent, 'positioners', nxclass, description=desc)
    writer.save_dict(group, scan.positioner)

See section Custom HDF5 writer below for more details.

Full Example: #PV control line

Consider a SPEC data file (named pv_data.txt) with the contrived example of a #PV control line that associates a mnemonic with an EPICS process variable (PV). Suppose we take this control line content to be two words (text with no whitespace):

#F pv_data.txt
#E 1454539891
#D Wed Feb 03 16:51:31 2016
#C pv_data.txt  User = spec2nexus
#O0 USAXS.a2rp  USAXS.m2rp  USAXS.asrp  USAXS.msrp  mr  unused37  mst  ast
#O1 msr  asr  unused42  unused43  ar  ay  dy  un47

#S 1  ascan  mr 10.3467 10.3426  30 0.1
#D Wed Feb 03 16:52:03 2016
#T 0.1  (seconds)
#P0 3.5425 6.795 7.7025 5.005 10.34465 0 0 0
#P1 7.6 17.17188 -8.67896 -0.351 10.318091 0 18.475664 0
#C tuning USAXS motor mr
#PV mr ioc:m1
#PV ay ioc:m2
#PV dy ioc:m3
#N 18
#L mr    ay  dy  ar_enc  pd_range  pd_counts  pd_rate  pd_curent  I0_gain  I00_gain  Und_E  Epoch  seconds  I00  USAXS_PD  TR_diode  I0  I0
10.34665  0.000 18.476 10.318091 1 5 481662 0.000481658 1e+07 1e+09 18.172565 33.037 0.1 199 2 1 114 114
10.34652  0.000 18.476 10.318091 1 5 481662 0.000481658 1e+07 1e+09 18.172565 33.294 0.1 198 2 1 139 139
10.34638  0.000 18.476 10.318091 1 5 481662 0.000481658 1e+07 1e+09 18.172565 33.553 0.1 198 2 1 181 181
10.34625  0.000 18.476 10.318091 1 5 481662 0.000481658 1e+07 1e+09 18.172565 33.952 0.1 198 2 1 274 274
10.34278  0.000 18.476 10.318091 1 5 481662 0.000481658 1e+07 1e+09 18.172309 41.621 0.1 198 2 1 232 232
10.34265  0.000 18.476 10.318091 1 5 481662 0.000481658 1e+07 1e+09 18.172565 41.867 0.1 199 2 1 159 159
#C Wed Feb 03 16:52:14 2016.  removed many data rows for this example.

A plugin (named pv_plugin.py) to handle the #PV control lines could be written as:

from collections import OrderedDict
import six
from spec2nexus.plugin import AutoRegister
from spec2nexus.plugin import ControlLineHandler
from spec2nexus.utils import strip_first_word

@six.add_metaclass(AutoRegister)
class PV_ControlLine(ControlLineHandler):
    '''**#PV** -- EPICS PV associates mnemonic with PV'''
    
    key = '#PV'
    scan_attributes_defined = ['EPICS_PV']
    
    def process(self, text, spec_obj, *args, **kws):
        args = strip_first_word(text).split()
        mne = args[0]
        pv = args[1]
        if not hasattr(spec_obj, "EPICS_PV"):
            # use OrderedDict since it remembers the order we found these
            spec_obj.EPICS_PV = OrderedDict()
        spec_obj.EPICS_PV[mne] = pv

When the scan parser encounters the #PV lines in our SPEC data file, it will call this process() code with the full text of the line and the spec scan object where this data should be stored. We will choose to store this (following the pattern of other data names in SpecDataFileScan) as scan_obj.EPICS_PV using a dictionary.

It is up to the user what to do with the scan_obj.EPICS_PV data. We will not consider the write() method in this example. (We will not write this infromation to a NeXus HDF5 file.)

We can then write a python program (named pv_example.py) that will load the data file and interpret it using our custom plugin:

import spec2nexus.plugin
import spec2nexus.spec

# call get_plugin_manager() BEFORE you import any custom plugins
manager = spec2nexus.plugin.get_plugin_manager()

# show our plugin is not loaded
print("known: ", "#PV" in manager.registry) # expect False

import pv_plugin
# show that our plugin is registered
print("known: ", "#PV" in manager.registry) # expect True

# read a SPEC data file, scan 1
spec_data_file = spec2nexus.spec.SpecDataFile("pv_data.txt")
scan = spec_data_file.getScan(1)

# Do we have our PV data?
print(hasattr(scan, "EPICS_PV"))    # expect True
print(scan.EPICS_PV)

The output of our program:

known:  False
known:  True
False
True
OrderedDict([('mr', 'ioc:m1'), ('ay', 'ioc:m2'), ('dy', 'ioc:m3')])

Example to ignore a #Y control line

Suppose a control line in a SPEC data file must be ignored. For example, suppose a SPEC file contains this control line: #Y 1 2 3 4 5. Since there is no standard handler for this control line, we create one that ignores processing by doing nothing:

import six
from spec2nexus.plugin import AutoRegister
from spec2nexus.plugin import ControlLineHandler

@six.add_metaclass(AutoRegister)
class Ignore_Y_ControlLine(ControlLineHandler):
    '''
    **#Y** -- as in ``#Y 1 2 3 4 5``

    example: ignore any and all #Y control lines
    '''

    key = '#Y'

    def process(self, text, spec_obj, *args, **kws):
        pass # do nothing

Postprocessing

Sometimes, it is necessary to defer a step of processing until after the complete scan data has been read. One example is for 2-D or 3-D data that has been acquired as a vector rather than matrix. The matrix must be constructed only after all the scan data has been read. Such postprocessing is handled in a method in a plugin file. The postprocessing method is registered from the control line handler by calling the addPostProcessor() method of the spec_obj argument received by the handler’s process() method. A key name [1] is supplied when registering to avoid registering this same code more than once. The postprocessing function will be called with the instance of SpecDataFileScan as its only argument.

An important role of the postprocessing is to store the result in the scan object. It is important not to modify other data in the scan object. Pick an attribute named similarly to the plugin (e.g., MCA configuration uses the MCA attribute, UNICAT metadata uses the metadata attribute, …) This attribute will define where and how the data from the plugin is available. The writer() method (see below) is one example of a user of this attribute.

Example postprocessing

Consider the #U control line example above. For some contrived reason, we wish to store the sum of the numbers as a separate number, but only after all the scan data has been read. This can be done with the simple expression:

spec_obj.U_sum = sum(spec_obj.U)

To build a postprocessing method, we write:

def contrived_summation(scan):
    '''
    add up all the numbers in the #U line

    :param SpecDataFileScan scan: data from a single SPEC scan
    '''
    scan.U_sum = sum(scan.U)

To register this postprocessing method, place this line in the process() of the handler:

spec_obj.addPostProcessor('contrived_summation', contrived_summation)

Summary Example Custom Plugin with postprocessing

Gathering all parts of the examples above, the custom plugin module is:

import six
from spec2nexus.plugin import AutoRegister
from spec2nexus.plugin import ControlLineHandler
from spec2nexus.utils import strip_first_word

@six.add_metaclass(AutoRegister)
class User_ControlLine(ControlLineHandler):
    '''**#U** -- User data (#U user1 user2 user3)'''

    key = '#U'

    def process(self, text, spec_obj, *args, **kws):
        args = strip_first_word(text).split()
        user1 = float(args[0])
        user2 = float(args[1])
        user3 = float(args[2])
        spec_obj.U = [user1, user2, user3]
        spec_obj.addPostProcessor('contrived_summation', contrived_summation)


def contrived_summation(scan):
    '''
    add up all the numbers in the #U line

    :param SpecDataFileScan scan: data from a single SPEC scan
    '''
    scan.U_sum = sum(scan.U)


@six.add_metaclass(AutoRegister)
class Ignore_Y_ControlLine(ControlLineHandler):
    '''**#Y** -- as in ``#Y 1 2 3 4 5``'''

    key = '#Y'

    def process(self, text, spec_obj, *args, **kws):
        pass

Custom HDF5 writer

A custom HDF5 writer method defines how the data from the plugin will be written to the HDF5+NeXus data file. The writer will be called with several arguments:

h5parent: obj : the HDF5 group that will hold this plugin’s data

writer: obj : instance of spec2nexus.writer.Writer that manages the content of the HDF5 file

scan: obj : instance of spec2nexus.spec.SpecDataFileScan containing this scan’s data

nxclass: str : (optional) name of NeXus base class to be created

Since the file is being written according to the NeXus data standard [2], use the NeXus base classes [3] as references for how to structure the data written by the custom HDF5 writer.

One responsibility of a custom HDF5 writer method is to create unique names for every object written in the h5parent group. Usually, this will be a NXentry [4] group. You can determine the NeXus base class of this group using code such as this:

>>> print h5parent.attrs['NX_class']
<<< NXentry

If your custom HDF5 writer must create group and you are uncertain which base class to select, it is recommended to use a NXcollection [5] (an unvalidated catch-all base class) which can store any content. But, you are encouraged to find one of the other NeXus base classes that best fits your data. Look at the source code of the supplied plugins for examples.

The writer uses the spec2nexus.eznx module to create and write the various parts of the HDF5 file.

Here is an example writer() method from the spec2nexus.plugins.unicat module:

 def writer(self, h5parent, writer, scan, nxclass=None, *args, **kws):
     '''Describe how to store this data in an HDF5 NeXus file'''
     if hasattr(scan, 'metadata') and len(scan.metadata) > 0:
         desc='SPEC metadata (UNICAT-style #H & #V lines)'
         group = eznx.makeGroup(h5parent, 'metadata', nxclass, description=desc)
         writer.save_dict(group, scan.metadata)

Custom key match function

The default test that a given line matches a specific spec2nexus.plugin.ControlLineHandler subclass is to use a regular expression match.

 def match_key(self, text):
     '''default regular expression match, based on self.key'''
     t = re.match(self.key, text)
     if t is not None:
         if t.regs[0][1] != 0:
             return True
     return False

In some cases, that may prove tedious or difficult, such as when testing for a floating point number with optional preceding white space at the start of a line. This is typical for data lines in a scan or continued lines from an MCA spectrum. in such cases, the handler can override the match_key() method. Here is an example from SPEC_DataLine:

 def match_key(self, text):
     '''
     Easier to try conversion to number than construct complicated regexp
     '''
     try:
         float( text.strip().split()[0] )
         return True
     except ValueError:
         return False

Summary Requirements for custom plugin

• file can go in your working directory or any directory that has __init__.py file
• multiple control line handlers can go in a single file
• for each control line:
  • subclass spec2nexus.plugin.ControlLineHandler
  • add @six.add_metaclass(AutoRegister) decorator to auto-register the plugin
  • import the module you defined (FIXME: check this and revise)
  • identify the control line pattern
  • define key with a regular expression to match [6]
    • key is used to identify control line handlers
    • redefine existing supported control line control lines to replace supplied behavior (use caution!)
    • Note: key="scan data" is used to process the scan data: spec2nexus.plugins.spec_common.SPEC_DataLine()
  • define process() to handle the supplied text
  • define writer() to write the in-memory data structure from this plugin to HDF5+NeXus data file
  • (optional) define match_key() to override the default regular expression to match the key
• for each postprocessing function:
  • write the function
  • register the function with spec_obj.addPostProcessor(key_name, the_function) in the handler’s process()

---

Changes in plugin format with release 2021.0.0

With release 2021.0.0, the code to setup plugins has changed. The new code allows all plugins in a module to auto-register themselves as long as the module is imported. All custom plugins must be modified and import code revised to work with new system. See the spec2nexus.plugins.spec_common source code for many examples.

• SAME: The basics of writing the plugins remains the same.
• CHANGED: The method of registering the plugins has changed.
• CHANGED: The declaration of each plugin has changed.
• CHANGED: The name of each plugin file has been relaxed.
• CHANGED: Plugin files do not have to be in their own directory.
• REMOVED: The SPEC2NEXUS_PLUGIN_PATH environment variable has been eliminated.

---

Footnotes

[1]	The key name must be unique amongst all postprocessing functions. A good choice is the name of the postprocessing function itself.

[2]	http://nexusformat.org

[3]	http://download.nexusformat.org/doc/html/classes/base_classes/

[4]	http://download.nexusformat.org/doc/html/classes/base_classes/NXentry.html

[5]	http://download.nexusformat.org/doc/html/classes/base_classes/NXcollection.html

[6]	It is possible to override the default regular expression match in the subclass with a custom match function. See the match_key() method for an example.

Overview of the supplied spec plugins

Plugins for these control lines [1] are provided in spec2nexus:

SPEC_File	#F – original data file name (starts a file header block)
SPEC_Epoch	#E – the UNIX epoch (seconds from 00:00 GMT 1/1/70)
SPEC_Date	#D – date/time stamp
SPEC_Comment	#C – any comment either in the scan header or somewhere in the scan
SPEC_Geometry	#G – diffractometer geometry (numbered rows: #G0, #G1, …)
SPEC_NormalizingFactor	#I – intensity normalizing factor
SPEC_CounterNames	#J – names of counters (each separated by two spaces) (new with SPEC v6)
SPEC_CounterMnemonics	#j – mnemonics of counter (new with SPEC v6)
SPEC_Labels	#L – data column labels
SPEC_Monitor	#M – counting against this constant monitor count (see #T)
SPEC_NumColumns	#N – number of columns of data [ num2 sets per row ]
SPEC_PositionerNames	#O – positioner names (numbered rows: #O0, #O1, …)
SPEC_PositionerMnemonics	#o – positioner mnemonics (new with SPEC v6)
SPEC_Positioners	#P – positioner values at start of scan (numbered rows: #P0, #P1, …)
SPEC_HKL	#Q – \(Q\) (\(hkl\)) at start of scan
SPEC_Scan	#S – SPEC scan
SPEC_CountTime	#T – counting against this constant number of seconds (see #M)
SPEC_UserReserved	#U – Reserved for user
SPEC_TemperatureSetPoint	#X – Temperature Set Point (desired temperature)
SPEC_DataLine	(scan_data) – scan data line
SPEC_MCA	#@MCA – MCA data formatting declaration (ignored for now)
SPEC_MCA_Array	@A – MCA Array data
SPEC_MCA_Calibration	#@CALIB – coefficients to compute a scale based on the MCA channel number
SPEC_MCA_ChannelInformation	#@CHANN – MCA channel information
SPEC_MCA_CountTime	#@CTIME – MCA count times
SPEC_MCA_RegionOfInterest	#@ROI – MCA ROI (Region Of Interest) channel information
UnrecognizedControlLine	unrecognized control line
UNICAT_MetadataMnemonics	#H – UNICAT metadata names (numbered rows: #H0, #H1, …)
UNICAT_MetadataValues	#V – UNICAT metadata values (numbered rows: #V0, #V1, …)
UIM_generic	#UIM – various image header information
XPCS_VA	#VA
XPCS_VD	#VD
XPCS_VE	#VE

[1]	Compare this list with Control lines (keys) defined by SPEC

source code documentation

define the plug-in architecture

Use spec2nexus.plugin.ControlLineHandler as a metaclass to create a plugin handler class for each SPEC control line. In each such class, it is necessary to:

• define a string value for the key (class attribute)
• override the definition of process()

It is optional to:

• define postprocess()
• define writer()
• define match_key()

Classes

ControlLineHandler	base class for SPEC data file control line handler plugins
PluginManager()	Manage the set of SPEC data file control line plugins

Exceptions

DuplicateControlLineKey	This control line key regular expression has been used more than once.
DuplicateControlLinePlugin	This control line handler has been used more than once.
DuplicatePlugin	This plugin file name has been used more than once.
PluginBadKeyError	The plugin ‘key’ value is not acceptable.
PluginDuplicateKeyError	This plugin key has been used before.
PluginKeyNotDefined	Must define ‘key’ in class declaration.
PluginProcessMethodNotDefined	Must define ‘process()’ method in class declaration.

class spec2nexus.plugin.AutoRegister(*args)

plugin to handle a single control line in a SPEC data file

This class is a metaclass to auto-register plugins to handle various parts of a SPEC data file. See spec_common for many examples.

Parameters:key (str) – regular expression to match a control line key, up to the first space Returns:None

class spec2nexus.plugin.ControlLineHandler

base class for SPEC data file control line handler plugins

define one ControlLineHandler class for each different type of control line

Parameters:

• key (str) – regular expression to match a control line key, up to the first space
• scan_attributes_defined ([str]) – list of scan attributes defined in this class

Returns:

None

EXAMPLE of match_key method:

Declaration of the match_key method is optional in a subclass. This is used to test a given line from a SPEC data file against the key of each ControlLineHandler.

If this method is defined in the subclass, it will be called instead of match_key(). This is the example used by SPEC_DataLine:

def match_key(self, text):
    try:
        float( text.strip().split()[0] )
        return True
    except ValueError:
        return False

postprocess(header, *args, **kws)

optional: additional processing deferred until after data file has been read

process(text, spec_file_obj, *args, **kws)

required: handle this line from a SPEC data file

writer(h5parent, writer, scan, nxclass=None, *args, **kws)

optional: Describe how to store this data in an HDF5 NeXus file

exception spec2nexus.plugin.DuplicateControlLineKey

This control line key regular expression has been used more than once.

exception spec2nexus.plugin.DuplicateControlLinePlugin

This control line handler has been used more than once.

exception spec2nexus.plugin.DuplicatePlugin

This plugin file name has been used more than once.

exception spec2nexus.plugin.PluginBadKeyError

The plugin ‘key’ value is not acceptable.

exception spec2nexus.plugin.PluginDuplicateKeyError

This plugin key has been used before.

exception spec2nexus.plugin.PluginException

parent exception for this module

exception spec2nexus.plugin.PluginKeyNotDefined

Must define ‘key’ in class declaration.

class spec2nexus.plugin.PluginManager

Manage the set of SPEC data file control line plugins

Class Methods

get(key)	return the handler identified by key or None
getKey(spec_data_file_line)	Find the key that matches this line in a SPEC data file.
load_plugins()	load all spec2nexus plugin modules
match_key(text)	test if any handler’s key matches text
process(key, *args, **kw)	pick the control line handler by key and call its process() method
register_control_line_handler(handler)	auto-registry of all AutoRegister plugins

get(key)

return the handler identified by key or None

getKey(spec_data_file_line)

Find the key that matches this line in a SPEC data file. Return None if not found.

Parameters:spec_data_file_line (str) – one line from a SPEC data file

load_plugins()

load all spec2nexus plugin modules

called from spec2nexus.plugin.get_plugin_manager()

match_key(text)

test if any handler’s key matches text

Parameters:text (str) – first word on the line, up to but not including the first whitespace Returns:key or None

Applies a regular expression match using each handler’s key as the regular expression to match with text.

process(key, *args, **kw)

pick the control line handler by key and call its process() method

register_control_line_handler(handler)

auto-registry of all AutoRegister plugins

Called from AutoRegister.__init__

exception spec2nexus.plugin.PluginProcessMethodNotDefined

Must define ‘process()’ method in class declaration.

spec2nexus.plugin.get_plugin_manager()

get the instance of the plugin_manager (a singleton)

Create instance of PluginManager() if necessary. Also,

Common Methods: spec2nexus.utils

source code documentation

(internal library) common methods used in spec2nexus modules

clean_name(key)	create a name that is allowed by both HDF5 and NeXus rules
iso8601(date)	convert SPEC time (example: Wed Nov 03 13:39:34 2010) into ISO8601 string
strip_first_word(line)	return everything after the first space on the line from the spec data file
sanitize_name(group, key)	make name that is allowed by HDF5 and NeXus rules
reshape_data(scan_data, scan_shape)	Shape scan data from raw to different dimensionality

spec2nexus.utils.clean_name(key)

create a name that is allowed by both HDF5 and NeXus rules

Parameters:key (str) – identifying string from SPEC data file See:http://download.nexusformat.org/doc/html/datarules.html

The “sanitized” name fits this regexp:

[A-Za-z_][\w_]*

An easier expression might be: [\w_]* but this will not pass the rule that valid NeXus group or field names cannot start with a digit.

spec2nexus.utils.iso8601(date)

convert SPEC time (example: Wed Nov 03 13:39:34 2010) into ISO8601 string

Parameters:date (str) – time string from SPEC data file

Example

SPEC:Wed Nov 03 13:39:34 2010 ISO8601:2010-11-03T13:39:34 SPOCK:09/15/17 04:39:10 ISO8601:2017-09-15T04:39:10

spec2nexus.utils.reshape_data(scan_data, scan_shape)

Shape scan data from raw to different dimensionality

Some SPEC macros collect data in a mesh or grid yet report the data as a 1-D sequence of observations. For further processing (such as plotting), the scan data needs to be reshaped according to its intended dimensionality.

modified from nexpy.readers.readspec.reshape_data

spec2nexus.utils.sanitize_name(group, key)

make name that is allowed by HDF5 and NeXus rules

Note:

deprecated use clean_name() instead (group is never used)

Parameters:

• group (str) – unused
• key (str) – identifying string from SPEC data file

See:

http://download.nexusformat.org/doc/html/datarules.html

sanitized name fits this regexp:

[A-Za-z_][\w_]*

An easier expression might be: [\w_]* but this will not pass the rule that valid names cannot start with a digit.

spec2nexus.utils.split_column_labels(text)

SPEC labels may contain one space

spec2nexus.utils.strip_first_word(line)

return everything after the first space on the line from the spec data file

spec2nexus.scanf

Simple scanf-implementation. This module provides an easy way to parse simple formatted strings. It works similar to the version C programmers are used to.

source code documentation

Small scanf-implementation.

• Created by Henning Schroeder on Mon, 12 Feb 2007
• PSF license

Python has powerful regular expressions but sometimes they are totally overkill when you just want to parse a simple-formatted string. C programmers use the scanf-function for these tasks (see link below).

This implementation of scanf translates the simple scanf-format into regular expressions. Unlike C you can be sure that there are no buffer overflows possible.

source: http://code.activestate.com/recipes/502213-simple-scanf-implementation/

For more information see:

• http://www.python.org/doc/current/lib/node49.html
• http://en.wikipedia.org/wiki/Scanf

spec2nexus.scanf.scanf(fmt, s=None)

scanf supports the following formats:

format	description
%c	One character
%5c	5 characters
%d	int value
%7d	int value with length 7
%f	float value
%o	octal value
%X, %x	hex value
%s	string terminated by whitespace

Examples: >>> scanf(“%s - %d errors, %d warnings”, “/usr/sbin/sendmail - 0 errors, 4 warnings”) (‘/usr/sbin/sendmail’, 0, 4) >>> scanf(“%o %x %d”, “0123 0x123 123”) (66, 291, 123)

If the parameter s is a file-like object, s.readline is called. If s is not specified, stdin is assumed.

The function returns a tuple of found values or None if the format does not match.

spec2nexus.singletons

This is an internal library of the spec2nexus software. It is not expected that users of this package will need to call the singletons module directly.

source code documentation

singletons: Python 2 and 3 Compatible Version

see:http://stackoverflow.com/questions/6760685/creating-a-singleton-in-python

USAGE:

class Logger(Singleton):
    pass

class spec2nexus.singletons.Singleton

Public interface

Installation

Released versions of spec2nexus are available on PyPI.

If you have pip installed, then you can install:

$ pip install spec2nexus

If you are using Anaconda Python and have conda installed, then you can install with either of these:

$ conda install -c aps-anl-tag spec2nexus
$ conda install -c aps-anl-dev spec2nexus
$ conda install -c prjemian spec2nexus

Note that channel aps-anl-tag is for production versions while channel aps-anl-dev is for development/testing versions. The channel prjemian is an alternate with all versions available.

The latest development versions of spec2nexus can be downloaded from the GitHub repository listed above:

$ git clone http://github.com/prjemian/spec2nexus.git

To install in the standard Python location:

$ cd spec2nexus
$ python setup.py install

To install in user’s home directory:

$ python setup.py install --user

To install in an alternate location:

$ python setup.py install --prefix=/path/to/installation/dir

Required Libraries

These libraries are required to write NeXus data files. They are not required to read SPEC data files.

Library	URL
h5py	http://www.h5py.org
numpy	http://numpy.scipy.org/

Optional Libraries

These libraries are used by the specplot and specplot_gallery modules of the spec2nexus package but are not required just to read SPEC data files or write NeXus data files.

Library	URL
MatPlotLib	http://matplotlib.org/

Unit Testing

Since release 2017.0201.0, this project relies on the Python unittest [1] package to apply unit testing [2] to the source code. The test code is in the tests directory. Various tests have been developed starting with the 2017.0201.0 release to provide features or resolve problems reported. The tests are not yet exhaustive yet the reported code coverage [3] is well over 80%.

The unit tests are implemented in a standard manner such that independent review [4] can run the tests on this code based on the instructions provided in a .travis.yml configuration file in the project directory.

This command will run the unit tests locally:

python tests

Additional information may be learned with a Python package to run the tests:

coverage run -a tests && coverage report -m

The coverage command ([5]), will run the tests and then prepare a report of the percentage of the Python source code that has been executed during the unit tests.

Note

The number of lines reported by coverage may differ from that reported by travis-ci. The primary reason is that certain tests involving access to information from GitHub may succeed or not depending on the “Github API rate limit”. [6]

[1]	Python unittest package: https://docs.python.org/2/library/unittest.html

[2]	unit testing: https://en.wikipedia.org/wiki/Unit_testing

[3]	coveralls code coverage: https://coveralls.io/github/prjemian/spec2nexus

[4]	travis-ci continuous intregration: https://travis-ci.org/prjemian/spec2nexus

[5]	coverage: https://coverage.readthedocs.io

[6]	Github API rate limit: https://developer.github.com/v3/rate_limit/

Example data

About these example data files

These files are examples of various data files that may be read by spec2nexus. They are used to test various components of the interface.

file		type description
02_03_setup.dat	SPEC scans	1-D scans, some have no data lines (data are stored in HDF5 file)
03_06_JanTest.dat	SPEC scans	1-D scans, USAXS scans, Fly scans, #O+#o and #J+#j control lines
05_02_test.dat	SPEC scans	1-D scans, USAXS scans, Fly scans, multiple #F control lines, multiple #S 1 control lines
33bm_spec.dat	SPEC scans	1-D & 2-D scans (includes hklscan & hklmesh)
33id_spec.dat	SPEC scans	1-D & 2-D scans (includes mesh & Escan scans & MCA data)
APS_spec_data.dat	SPEC scans	1-D scans (ascan & uascan), includes lots of metadata and comments
CdOsO	SPEC scans	1-D scans (ascan), four #E (2, 3659, 3692, 3800) and two #S 1 (35, 3725)
CdSe	SPEC scans	1-D scans (ascan), problem with scan abort on lines 5918-9, in scan 92
compression.h5	NeXus HDF5	2-D compressed image, also demonstrates problem to be resolved in code
Data_Q.h5	NeXus HDF5	2-D image at /entry/data/{I,Q}, test file and variable-length strings
lmn40.spe	SPEC scans	1-D & 2-D scans (hklmesh), two #E lines, has two header sections
mca_spectra_example.dat	SPEC scans	1-D scans (cscan) with 4 MCA spectra in each scan (issue #55)
spec_from_spock.spc	SPEC scans	no header section, uses “nan”, from sardana
startup_1.spec	SPEC scans	1-D scans with SCA spectra & UXML headers for RSM code
user6idd.dat	SPEC scans	1-D scans, aborted scan, control lines: #R #UB #UE #UX #UX1 #UX2 #X, non-default format in #X lines
usaxs-bluesky-specwritercallback.dat	SPEC scans	1-D scans, #MD control lines
writer_1_3.h5	NeXus HDF5	1-D NeXus User Manual example
YSZ011_ALDITO_Fe2O3_planar_fired_1.spc	SPEC scans	1-D scans, text in #V metadata, also has #UIM control lines

Downloads

These downloads are also available online: https://github.com/prjemian/spec2nexus/tree/master/src/spec2nexus/data

• 33bm_spec.dat
• 33id_spec.dat
• APS_spec_data.dat
• CdSe
• compression.h5
• Data_Q.h5
• lmn40.spe
• mca_spectra_example.dat
• user6idd.dat
• writer_1_3.h5
• YSZ011_ALDITO_Fe2O3_planar_fired_1.spc

Change History

Note

Python 2 end of support

spec2nexus stopped development for Python 2 after release 2021.1.7, 2019-11-21. For more information, visit https://python3statement.org/.

Production

2021.1.8:

released 2020.11.10

• #221

  move CI from travis-ci to Github Actions, test with python 3.8

• #217

  raise ValueError when #L and #N lines do not agree

2021.1.7:

released 2019-11-21

Note: Last version with support for Python 2

• #213

  copy data file to gallery

• #208

  add more diagnostics to gallery web page comments

• #191

  write each positioner to NXpositioner group

• #188

  catenate continued lines before parsing data

• #186

  remove unused code

2021.1.6:

released 2019.11.01

• #210

  add -c prjemian conda channel

2021.1.5:

released 2019.11.01

• #209

  pyRestTable added to installation requirements

2021.1.4:

released 2019.10.18

• #206

  specplot_gallery: replot shows all existing plots

2021.1.3:

released 2019.08.19 - only update plots with new content

• #202

  specplot_gallery: switch to SVG (from PNG) for plots

• #201

  spec: subsequent calls to read() duplicate scans – FIXED

• #126

  spec: new update_available property

• #108

  specplot_gallery: only update plots with new content

2021.1.2:

released 2019.08.15, plugin enhancements

• #197

  plugins: handle empty empty #O0 or #P0 list

• #195

  drop CII badge: not useful to spec2nexus

• #190

  writer: link content into NXinstrument group

• #51

  plugins: interpret #Gn control lines

2021.1.1:

released 2019.07.22, refactor

• #181

  plugins: revised technique to load control line handlers

2021.1.0:

released 2019.07.15, new features

NEW

• support for #UXML metadata
• support for hklscan scans
• improved support for mesh and hklmesh scans

• #159

  handle #UXML metadata control lines

• #155

  module: writer - recognize hklscan

• #150

  module: writer - increase coverage of unit tests: mesh, hklmesh

• #148

  module: eznx - increase coverage of unit tests

2021.0.1:

released 2019.07.13, plugin loading and documentation

• #170

  describe how to write & load Control Line Handler plugins

• #169

  announce deprecation of python 2

• #165

  resolve conda build error

• #149

  unit tests: units module

2021.0.0:

released 2019.07.12, API change affecting plugins

API change: Changed how plugins are defined and registered. Custom plugins must be modified and import code revised to work with new system.

• #168

  plugins are now self-registering

• #166

  fix conda packaging

2020.0.2:

released 2019.07.09, bug fixes and code review suggestions

NOTE: conda package is broken (no plugins directory). Only use pip install spec2nexus with this release.

• #164

  post conda packages to aps-anl-tag channel

• #161

  read files with no #E control line

• #156

  LGTM code review

• #153

  LGTM code review

2020.0.0:

released 2019.05.16, major release

• #145

  unit tests for header content

• #144

  eznx makeDataset() now recognizes if data is ndarray

• #123

  Accept data files with no header control lines (#F #E #D #C sequence)

• #113

  unit tests for eznx

• #70

  remove h5toText, find this now in punx package

2019.0503.0:

released 2019.05.03, tag

• #142

  DuplicateSpecScanNumber with multiple #F sections

• #137

  (again) bug in #U control line handling

2019.0501.0:

released 2019.05.01, tag

• #137

  bug in #U control line handling

• #140

  change: #U data goes into <object>.U list (name changed from UserReserved)

2.1.0:

2019.04.26, release

• #135

  switch to semantic versioning

• #133

  support user control line “#U ” with plugin

• #131

  support #MD control lines from apstools.SpecWriterCallback

• #125

  fluorescence spectra in files for RSM3D

• #120

  do not mock six package in documentation

• #119

  delimiters in #H/#V lines with or without text values

• #116

  process data from spock

  see [release notes](https://github.com/prjemian/spec2nexus/wiki/releasenotes__2-1-0)

  It takes a couple steps to upgrade an existing conda installation from version 2017.nnnn to newer version 2.1.0

  • add a declaration of spec2nexus < 2000 in the conda-meta/pinned file in the conda environment
  • conda update -c prjemian spec2nexus (should change to 2.1.0)

  It may still be necessary to uninstall and reinstall spec2nexus to effect an update:

  conda uninstall -y spec2nexus conda install -c prjemian spec2nexus

2019.0422.0:

(tag only)

• tag as-is, for issue #131

2019.0321.0:

(tag only)

• tag as-is, post conda noarch package and post to pypi

2017.901.4:

• #62

  support Python3

• #112

  merge py3-62 branch

• #111

  Change raise statements to use parens around arguments. Affects issue #62

• #114

  travis-ci for python 3.5 & 3.6

• #107

  Problems accessing SpecDataFileScan.data

• #95

  document final release steps

2017.711.0:

• #110

  Ownership of info between #L/data & #S n

• #109 Spaces in data labels on #L and other lines

2017.522.1:

• #105 ignore extra content in #@CALIB control lines
• #104 use versioneer (again)

• #101

  documentation URL & date/time added to every gallery page

• #100 conda package installs properly on Windows now
• #99 BUG: specplot_gallery: plots of hklscan from file lmn40.spe
• #98 BUG: specplot_gallery: identify as directory not found
• #52 remove deprecated prjPySpec code

2017.317.0:

• minor update of the 2017.3.0 release

2017.3.0:

• #103 changed converters back to utils
• #97 PyPI project description now formatted properly
• #90 use versioneer (again)

2017-0202.0:

• #99 fix list index error in hklscan when hkl are all constant
• #96 combine steps when publishing to PyPI

2017-0201.0:

• milestone punch list
• #73 refactor mesh and MCA data parsing code
• #67 apply continuous integration via travis-ci
• #66 add verbosity option
• #65 apply unit testing
• #64 extractSpecScan: fixed list index out of range
• #63 extractSpecScan: command line option to select range of scans
• #56 specplot and specplot_gallery: add from USAXS instrument and generalize

2016.1025.0:

standardize the versioning kit with pyRestTable and pvWebMonitor

2016.1004.0:

• #61 release info from git (dropped versioneer package)

2016.0829.0:

• #60 Add new plugin test for XPCS plugin (thanks to John Hammonds)

2016.0615.1:

• #57 keep information from unrecognized control lines,
• #56 add specplot support,
• #55 accept arbitrary number of MCA spectra

2016.0601.0:

match complete keys, use unix EOL internally, do not fail if no metadata

2016.0216.0:

• #36 identify NIAC2014-compliant NeXus files

2016.0210.0:

bugfix: eznx.makeGroup() now correctly sets attributes on new group + documentation for NIAC2014 attributes

2016.0204.0:

• #45 handle case when no data points in scan ,
• #46 spec.getScan() ensures argument is used as str

2016.0201.0:

added spec.getScanNumbersChronological(), spec.getFirstScanNumber(), and spec.getLastScanNumber()

2016.0131.0:

• #43 support new NeXus method for default/signal/axes/_indices,

2016.0130.0:

fixed #44

2015.1221.1:

• #40 added versioneer support

2015.1221.0:

• #39 read scans with repeated scan numbers

2015.0822.0:

extractSpecScan: add option to report scan heading data, such as positioners and Q

2015.0214.0:

h5toText: handle HDF5 ‘O’ data type (variable length strings)

2015.0127.0:

spec: ignore bad data lines

2015.0125.0:

spec: change handling of #L & #X, refactor detection of scanNum and scanCmd

2015.0113.0:

dropped requirement of lxml package

2014.1228.1:

spec: build mne:name cross-references for counters and positioners

2014.1228.0:

show version in documentation

2014.1028.0:

spec: quietly ignore unrecognized scan content for now

2014.1027.1:

spec: major changes in SPEC file support: custom plugins

• spec based on plugins for each control line, users can add plugins
• declared prjPySpec module as legacy, code is frozen at 2014.0623.0 release
• added spec module to replace prjPySpec

2014.0623.0:

updated argparse settings

2014.0622.2:

added extractSpecScan.py to the suite from the USAXS project

2014.0410.0:

restore scan.fileName variable to keep interface the same for some legacy clients

2014.0404.1:

fix sdist utf8 problem, see: http://bugs.python.org/issue11638

2014.0404.0:

tree_api_parser moved back into NeXpy project

2014.0320.6:

handle multiple header sections in SPEC data file

2014.0320.5:

fix the new project URL

2014.0320.4:

Sphinx cannot build PDF with code-block in a footnote

2014.0320.3:

note the new home URL in the packaging, too, drop nexpy requirement, default docs theme

2014.0320.2:

tree_api_parse will go back into nexpy project, remove docs of it here

2014.0320.1:

allow readthedocs to build Sphinx without extra package requirements

2014.0320.0:

• new home page at http://spec2nexus.readthedocs.org, easier to publish there
• move common methods from __init__.py so docs will build at readthedocs.org
• new test case fails existing SPEC reader, ignore blank lines

2014.03.11:

documentation

2014.03.09:

h5toText: option to suppress printing of attributes, put URLs in command-line usage documentation, better test of is_spec_file()

2014.03.08:

fixed string writer and content display bug in eznx, added h5toText.py, prjPySpec docs improved again

2014.03.051:

prjPySpec now handles SPEC v6 data file header additions, add new getScanCommands() method

2014.03.04:

(2014_Mardi_Gras release) removed nexpy project requirement from setup, prjPySpec raises exceptions now

2014.03.02:

drops nexus tree API (and its dependencies) in favor of native h5py writer

Development: GitHub repository

2014.02.20:version number fits PEP440, LICENSE file included in sdist, more documentation and examples 2014-02-19:reference published documentation (re-posted) 2014-02-19:add documentation framework 2014-02-18:fork to GitHub to make generally available

Development: NeXpy branch

2014-01:

briefly, a branch in https://github.com/nexpy/nexpy

• spec2nexus added during this phase
• relies on nexpy.api.nexus for NeXus support

Production: USAXS livedata

2010-2014:

production use

• support livedata WWW page of APS USAXS instrument
  • (http://usaxs.xray.aps.anl.gov/livedata/),
• https://subversion.xray.aps.anl.gov/trac/small_angle/browser/USAXS/livedata/prjPySpec.py
• converted from Tcl

2000-2010:

Tcl code (readSpecData.tcl) in production use at APS sectors 32, 33, & 34

License

Creative Commons Attribution 4.0 International Public License

By exercising the Licensed Rights (defined below), You accept and agree to be bound by the terms and conditions of this Creative Commons Attribution 4.0 International Public License ("Public License"). To the extent this Public License may be interpreted as a contract, You are granted the Licensed Rights in consideration of Your acceptance of these terms and conditions, and the Licensor grants You such rights in consideration of benefits the Licensor receives from making the Licensed Material available under these terms and conditions.

Section 1 -- Definitions.

    Adapted Material means material subject to Copyright and Similar Rights that is derived from or based upon the Licensed Material and in which the Licensed Material is translated, altered, arranged, transformed, or otherwise modified in a manner requiring permission under the Copyright and Similar Rights held by the Licensor. For purposes of this Public License, where the Licensed Material is a musical work, performance, or sound recording, Adapted Material is always produced where the Licensed Material is synched in timed relation with a moving image.
    Adapter's License means the license You apply to Your Copyright and Similar Rights in Your contributions to Adapted Material in accordance with the terms and conditions of this Public License.
    Copyright and Similar Rights means copyright and/or similar rights closely related to copyright including, without limitation, performance, broadcast, sound recording, and Sui Generis Database Rights, without regard to how the rights are labeled or categorized. For purposes of this Public License, the rights specified in Section 2(b)(1)-(2) are not Copyright and Similar Rights.
    Effective Technological Measures means those measures that, in the absence of proper authority, may not be circumvented under laws fulfilling obligations under Article 11 of the WIPO Copyright Treaty adopted on December 20, 1996, and/or similar international agreements.
    Exceptions and Limitations means fair use, fair dealing, and/or any other exception or limitation to Copyright and Similar Rights that applies to Your use of the Licensed Material.
    Licensed Material means the artistic or literary work, database, or other material to which the Licensor applied this Public License.
    Licensed Rights means the rights granted to You subject to the terms and conditions of this Public License, which are limited to all Copyright and Similar Rights that apply to Your use of the Licensed Material and that the Licensor has authority to license.
    Licensor means the individual(s) or entity(ies) granting rights under this Public License.
    Share means to provide material to the public by any means or process that requires permission under the Licensed Rights, such as reproduction, public display, public performance, distribution, dissemination, communication, or importation, and to make material available to the public including in ways that members of the public may access the material from a place and at a time individually chosen by them.
    Sui Generis Database Rights means rights other than copyright resulting from Directive 96/9/EC of the European Parliament and of the Council of 11 March 1996 on the legal protection of databases, as amended and/or succeeded, as well as other essentially equivalent rights anywhere in the world.
    You means the individual or entity exercising the Licensed Rights under this Public License. Your has a corresponding meaning.

Section 2 -- Scope.

    License grant.
        Subject to the terms and conditions of this Public License, the Licensor hereby grants You a worldwide, royalty-free, non-sublicensable, non-exclusive, irrevocable license to exercise the Licensed Rights in the Licensed Material to:
            reproduce and Share the Licensed Material, in whole or in part; and
            produce, reproduce, and Share Adapted Material.
        Exceptions and Limitations. For the avoidance of doubt, where Exceptions and Limitations apply to Your use, this Public License does not apply, and You do not need to comply with its terms and conditions.
        Term. The term of this Public License is specified in Section 6(a).
        Media and formats; technical modifications allowed. The Licensor authorizes You to exercise the Licensed Rights in all media and formats whether now known or hereafter created, and to make technical modifications necessary to do so. The Licensor waives and/or agrees not to assert any right or authority to forbid You from making technical modifications necessary to exercise the Licensed Rights, including technical modifications necessary to circumvent Effective Technological Measures. For purposes of this Public License, simply making modifications authorized by this Section 2(a)(4) never produces Adapted Material.
        Downstream recipients.
            Offer from the Licensor -- Licensed Material. Every recipient of the Licensed Material automatically receives an offer from the Licensor to exercise the Licensed Rights under the terms and conditions of this Public License.
            No downstream restrictions. You may not offer or impose any additional or different terms or conditions on, or apply any Effective Technological Measures to, the Licensed Material if doing so restricts exercise of the Licensed Rights by any recipient of the Licensed Material.
        No endorsement. Nothing in this Public License constitutes or may be construed as permission to assert or imply that You are, or that Your use of the Licensed Material is, connected with, or sponsored, endorsed, or granted official status by, the Licensor or others designated to receive attribution as provided in Section 3(a)(1)(A)(i).

    Other rights.
        Moral rights, such as the right of integrity, are not licensed under this Public License, nor are publicity, privacy, and/or other similar personality rights; however, to the extent possible, the Licensor waives and/or agrees not to assert any such rights held by the Licensor to the limited extent necessary to allow You to exercise the Licensed Rights, but not otherwise.
        Patent and trademark rights are not licensed under this Public License.
        To the extent possible, the Licensor waives any right to collect royalties from You for the exercise of the Licensed Rights, whether directly or through a collecting society under any voluntary or waivable statutory or compulsory licensing scheme. In all other cases the Licensor expressly reserves any right to collect such royalties.

Section 3 -- License Conditions.

Your exercise of the Licensed Rights is expressly made subject to the following conditions.

    Attribution.

        If You Share the Licensed Material (including in modified form), You must:
            retain the following if it is supplied by the Licensor with the Licensed Material:
                identification of the creator(s) of the Licensed Material and any others designated to receive attribution, in any reasonable manner requested by the Licensor (including by pseudonym if designated);
                a copyright notice;
                a notice that refers to this Public License;
                a notice that refers to the disclaimer of warranties;
                a URI or hyperlink to the Licensed Material to the extent reasonably practicable;
            indicate if You modified the Licensed Material and retain an indication of any previous modifications; and
            indicate the Licensed Material is licensed under this Public License, and include the text of, or the URI or hyperlink to, this Public License.
        You may satisfy the conditions in Section 3(a)(1) in any reasonable manner based on the medium, means, and context in which You Share the Licensed Material. For example, it may be reasonable to satisfy the conditions by providing a URI or hyperlink to a resource that includes the required information.
        If requested by the Licensor, You must remove any of the information required by Section 3(a)(1)(A) to the extent reasonably practicable.
        If You Share Adapted Material You produce, the Adapter's License You apply must not prevent recipients of the Adapted Material from complying with this Public License.

Section 4 -- Sui Generis Database Rights.

Where the Licensed Rights include Sui Generis Database Rights that apply to Your use of the Licensed Material:

    for the avoidance of doubt, Section 2(a)(1) grants You the right to extract, reuse, reproduce, and Share all or a substantial portion of the contents of the database;
    if You include all or a substantial portion of the database contents in a database in which You have Sui Generis Database Rights, then the database in which You have Sui Generis Database Rights (but not its individual contents) is Adapted Material; and
    You must comply with the conditions in Section 3(a) if You Share all or a substantial portion of the contents of the database.

For the avoidance of doubt, this Section 4 supplements and does not replace Your obligations under this Public License where the Licensed Rights include other Copyright and Similar Rights.

Section 5 -- Disclaimer of Warranties and Limitation of Liability.

    Unless otherwise separately undertaken by the Licensor, to the extent possible, the Licensor offers the Licensed Material as-is and as-available, and makes no representations or warranties of any kind concerning the Licensed Material, whether express, implied, statutory, or other. This includes, without limitation, warranties of title, merchantability, fitness for a particular purpose, non-infringement, absence of latent or other defects, accuracy, or the presence or absence of errors, whether or not known or discoverable. Where disclaimers of warranties are not allowed in full or in part, this disclaimer may not apply to You.
    To the extent possible, in no event will the Licensor be liable to You on any legal theory (including, without limitation, negligence) or otherwise for any direct, special, indirect, incidental, consequential, punitive, exemplary, or other losses, costs, expenses, or damages arising out of this Public License or use of the Licensed Material, even if the Licensor has been advised of the possibility of such losses, costs, expenses, or damages. Where a limitation of liability is not allowed in full or in part, this limitation may not apply to You.

    The disclaimer of warranties and limitation of liability provided above shall be interpreted in a manner that, to the extent possible, most closely approximates an absolute disclaimer and waiver of all liability.

Section 6 -- Term and Termination.

    This Public License applies for the term of the Copyright and Similar Rights licensed here. However, if You fail to comply with this Public License, then Your rights under this Public License terminate automatically.

    Where Your right to use the Licensed Material has terminated under Section 6(a), it reinstates:
        automatically as of the date the violation is cured, provided it is cured within 30 days of Your discovery of the violation; or
        upon express reinstatement by the Licensor.
    For the avoidance of doubt, this Section 6(b) does not affect any right the Licensor may have to seek remedies for Your violations of this Public License.
    For the avoidance of doubt, the Licensor may also offer the Licensed Material under separate terms or conditions or stop distributing the Licensed Material at any time; however, doing so will not terminate this Public License.
    Sections 1, 5, 6, 7, and 8 survive termination of this Public License.

Section 7 -- Other Terms and Conditions.

    The Licensor shall not be bound by any additional or different terms or conditions communicated by You unless expressly agreed.
    Any arrangements, understandings, or agreements regarding the Licensed Material not stated herein are separate from and independent of the terms and conditions of this Public License.

Section 8 -- Interpretation.

    For the avoidance of doubt, this Public License does not, and shall not be interpreted to, reduce, limit, restrict, or impose conditions on any use of the Licensed Material that could lawfully be made without permission under this Public License.
    To the extent possible, if any provision of this Public License is deemed unenforceable, it shall be automatically reformed to the minimum extent necessary to make it enforceable. If the provision cannot be reformed, it shall be severed from this Public License without affecting the enforceability of the remaining terms and conditions.
    No term or condition of this Public License will be waived and no failure to comply consented to unless expressly agreed to by the Licensor.
    Nothing in this Public License constitutes or may be interpreted as a limitation upon, or waiver of, any privileges and immunities that apply to the Licensor or You, including from the legal processes of any jurisdiction or authority.

Indices and tables

• Index
• Module Index
• Search Page