MSUM python toolbox

Contents

Overview

This package provides two types of functionality; only the first is likely to be of general interest.

Classes for managing a collection of FITS files

The class ImageFileCollection provides a table summarizing the values of FITS keywords in the files in a directory and provides easy ways to iterate over the HDUs, headers or data in those files. As a quick example:

>>> from msumastro import ImageFileCollection
>>> ic = ImageFileCollection('.', keywords=['imagetyp', 'filter'])
>>> for hdu in ic.hdus(imagetype='LIGHT', filter='I'):
...     print hdu.header, hdu.data.mean()

does what you would expect: it loops over all of the images in the collection whose image type is ‘LIGHT’ and filter is ‘I’. For more details see Image Management.

The TableTree constructs, from the summary table of an ImageFileCollection, a tree organized by values in the FITS headers of the collection. See Image Management for more details and examples.

Header processing of images from the Feder Observatory

Semi-automatic image processing

Command line scripts for automated updating of FITS header keywords. The intent with these is that they will rarely need to be used manually once a data preparation pipeline is set up.

The simplest option here is to use run_standard_header_process which will chain together all of the steps in data preparation for you. When using run_standard_header_process consider using the --scripts-only option, which generates bash scripts to carry out the data preparation. This gives you a complete record of the commands run in addition to the log files that are always generated.

All of these scripts can also be run from your python code if desired.

Manual header or image manipulation

Command lines scripts that easily automate a small number of tasks that occur frequently enough that it is convenient to have them available at the command line instead of requiring that new code to be written each time they are used.

There are currently two examples of this:

All of these scripts can also be run from your python code if desired.

Installation

This software

Users

This software requires a python distribution that includes numpy and other packages that support scientific work with python. The easiest way to get these is to download and install the Anaconda python distribution. Note that the Anaconda distribution includes astropy.

Install the way you install most python software:

pip install msumastro

followed (optionally) by:

pip install astropysics

only if you need the Feder Observatory stuff. You do not need astropysics for the image management features likely to be of broadest interest.

Developers

Install this software by downloading a copy from the github page for the code. On Mac/Linux do this by typing, in a terminal in the directory in which you want to run the code:

git clone https://github.com/mwcraig/msumastro.git

Navigate to the directory in which you downloaded it and run:

python setup.py develop

With this setup any changes you make to the source code will be immediately available to you without additional steps.

Dependencies

Python

This software has only been tested in python 2.7.x. It does not work in 3.x.

Note

Most of the requirements below will be taken care of automatically if you install msumastro with pip or setup.py as described above. The exceptions are numpy and scipy

Python packages
Required

Nothing will work without these:

  • numpy (included with anaconda): If you need to install it, please see the instructions at the SciPy download site. Some functionality may require SciPy.

  • astropy (included with anaconda): If you need to install it, do so with:

    pip install astropy
    
Required for some features

Most of the header patching functionality requires astropysics:

Required to build documentation

You only need to install the packages below if you want to build the documentation yourself:

(mostly) Non-python software: astrometry.net

Note

There is one piece of python software you need for astrometry.net and for now you need to install it manually:

pip install pyfits

If you want to be able to use the script Adding astrometry: run_astromtery.py you need a local installation of astrometry.net and sextractor (the latter works better than the source detection built into astrometry.net) The easiest way to do that (on a Mac) is with homebrew. Once you have installed homebrew the rest is easy (unless it fails, of course...):

  • brew tap homebrew/science (only needs to be done once; connects the set of homebrew science formulae)
  • brew install sextractor (note this can take a a few minutes)
  • brew install --env=std astrometry.net [Note the option --env=std. It is necessary to ensure homebrew sees your python installation.]

Automated header processing

Overview

The scripts described here are intended primarily to be run in an automated way whenever new images are uploaded to the physics server. Each can also be run manually, either from the command line or from python.

Both these ways of invoking the script (from the command line or from python) is are wrappers around the python functions that do the real work. References to those functions, which provide more control over what you can do at the expense of taking more effort to understand, are provided below where appropriate.

The purpose of the header processing is to:

  • Modify or add keywords to the FITS header to make working with other software easier:
    • Standardize names instead using the MaxIm DL defaults (e.g. RA and Dec instead of OBJCTRA and OBJCTDEC)
    • Set IMAGETYP to IRAF default, i.e. “BIAS”, “DARK”, “FLAT” or “LIGHT”
    • Add convenient keywords that MaxImDL does not always include (e.g. AIRMASS, HA, LST)
    • Add keywords indicating the overscan region, if any, in the image.
  • Add astrometry to the FITS header so that RA/Dec can be extracted/displayed for sources in the image.
  • Add object names to the HEADER where possible.
  • Identify images that need manual action to add any of the information above.
  • Create a table of images in each directory with columns for several FITS header keywords to facilitate indexing of images taken at Feder.

Intended workflow

The three primary scripts are, in the order in which they are intended (though not required, necessarily) to run:

  • generate a table summarizing properties of images in a directory. Each image is one row in the table and the columns are keywords from the FITS headers.
  • create files with lists of images missing key information.
  • Fix any problems identified by run_triage. The script quick_add_keys_to_file may be useful for this; it is an easy way to add/modify the values of keywords in FITS headers from the command line; see details at Manual intervention: quick_add_keys_to_file.py. After fixing these problems you may need to re-run patch, particularly if you have added pointing information or changed the IMAGETYP of any of the images.
  • run_astrometry to use astrometry.net to add WCS (astrometry) information to the file. See Adding astrometry: run_astromtery.py for details. Note that this requires a local installation of astrometry.net.
  • If desired, run_triage again to regenerate the table of image information.

The intended workflow will not work when...

The workflow above works great when the images that come off the telescope contain pointing information (i.e. RA/Dec), filter information and the image type in the header matches the actual image type.

Manual intervention will be required in any of these circumstances:

  • There is no pointing information in the files. In files that are produced at Feder Observatory the pointing information is contained in the FITS keywords OBJCTRA and OBJCTDEC. If there is no pointing information, astrometry will not be added to the image headers. astrometry.net can actually do blind astrometry, but it is fairly time consuming. Alternatives are suggested below.
    • How to identify this case: There are two ways this problem may be noted. If run_triage has been run (and it is run in the standard workflow) then a file called “NEEDS_POINTING_INFO.txt” will be created that lists all of the light files missing pointing information. In addition, one file with suffix .blind will be created for each light file which contains no pointing information.
  • Filter information is missing for light or flat images. All of the data preparation will occur if the FILTER keyword is missing from the headers for light or flat images, but the filter needs to be added to make the images useful.
    • How to identify this case: A file called “NEEDS_FILTER.txt” will be created as part of the standard workflow that lists each file that needs filter information.
  • Incorrect image type set for image. If the incorrect image type is set it can prevent some data preparation steps to be omitted that should actually occur or cause steps to be attempted that shouldn’t be. For example, if an image is really a LIGHT image but is labeled in the header as a FLAT, then no attempt will be made to calculate an apparent position (Alt/Az) for the frame or to add astrometry. If the mistake is reversed, with a FLAT image labeled as LIGHT an attempt will be made to add astrometry which will fail.
    • How to identify this case: Manual inspection of affected images is the only reliable way to do this. A good place to start looking is at light files for which adding astrometry failed, file names whose name implies a different type than its IMAGETYP in the FITS header (e.g. a file with IMAGETYP = LIGHT whose name is flat-001R.fit)
  • The object being observed is not in the master object list. The standard workflow has run but object names have not been add to all of the light files. This occurs when the object of the image was not in the list of objects used by run_patch.py or the object’s RA/Dec was too far from the center of the image to be matched.
    • How to identify this case: The script run_triage.py, part of the standard workflow, will produce a file called “NEEDS_OBJECT.txt” with a list of light files for which there is no object.

Fixes for cases that require intervention

The discussion below is deliberately broad. For some concrete examples see Manual header processing

  • Adding pointing information: There are a few options here:

    • Use quick_add_keys_to_file to add the OBJECT keyword to the header, then add_ra_dec_from_object_name() to add pointing information, then run_astrometry to add astrometry to the images.
    • Use quick_add_keys_to_file to add the OBJECT, RA, DEC, OBJCTRA and OBJCTDEC to the headers, then run_astrometry to add astrometry to the images. This route is not recommended because it is easy to use a format for RA/Dec that isn’t understood (or is misinterpreted) by the code that adds astrometry.
    • Do blind astrometry to add pointing information, then use add_object_info() to add object names. There are no inherent with with this approach, though it may be simpler to add the astrometry then re-run the standard processing workflow to add any missing keywords than it is to manually use add_object_info()
  • Adding filter information: The hard part here is not adding the filter keyword, it is figuring out what filter was used when the image was taken. You are on your own in figuring out that piece. Once you know what the filter should be, use quick_add_keys_to_file to add the keyword FILTER to the relevant files.

  • Adding filter information: As with adding filter information, the hard part is figuring out what the image type should be. In practice most cases of this are light images misidentified as flat and vice versa and it ought to be easy to determine which of those an image is at a glance (arguably, if you can’t tell at a glance then the image is probably useful as neither a light nor a flat image). Once you know what the image type should be, use quick_add_keys_to_file to set the keyword IMAGETYP to the appropriate value in the relevant files. Allowed values for IMAGETYP are “BIAS”, “DARK”, “FLAT” or “LIGHT”.

  • Adding object information: Assuming pointing information is already in the header for the images that need object information this is fairly straightforward. One way to do it is to add the object to the master object list and run add_object_info() (or even just re-run run_patch, which will end up re-doing some of the keyword-patching work). Another way to approach is to use quick_add_keys_to_file to set the OBJECT keyword directly. Either way you are encouraged to upadte the master object list.

Detailed list of keywords changed

Keywords purged before further processing

Some keywords are purged from the original headers because I don’t trust the values that MaxImDL v5 puts in:

OBJECT
JD
JD-HELIO
OBJCTALT
OBJCTAZ
OBJCTHA
AIRMASS
Keywords modified for all files

The keywords that are currently added/modified by patch_headers for all files are:

IMAGETYP: Type of image
LATITUDE: [degrees] Observatory latitude
LONGITUD: [degrees east] Observatory longitude
ALTITUDE: [meters] Observatory altitude
LST: Local Sidereal Time at start of observation
JD-OBS: Julian Date at start of observation
MJD-OBS: Modified Julian date at start of observation
BIASSEC: Region of the image useful for subtracting overscan
TRIMSEC: Region to which the image should be trimmed after removing overscan
Keywords modified only for light files

The keywords that are currently added/modified by patch_header for light files only are:

RA: Approximate RA at EQUINOX
DEC: Approximate DEC at EQUINOX
OBJECT: Target of the observations
HA: Hour angle
AIRMASS: Airmass (Sec(Z)) at start of observation
ALT-OBJ: [degrees] Altitude of object, no refraction
AZ-OBJ: [degrees] Azimuth of object, no refraction

Reference/API

Command-line scripts

Each of the command-line scripts described below is also callable from python. The details of how you call it from python are described below.

Both these ways of invoking the script (from the command line or from python) is are wrappers around the python functions that do the real work. References to those functions, which tend to provide more control over what you can do at the expense of taking more effort to understand, are provided below where appropriate.


Running the complete standard workflow: run_standard_header_process.py
Usage summary
usage: run_standard_header_process.py [-h]
                                      (--dest-root DEST_ROOT | --overwrite-source)
                                      [--scripts-only] [-r {a,t,p}]
                                      [--no-blind] [-o OBJECT_LIST]
                                      [--quiet-console] [--silent-console]
                                      [--debug] [--quiet-log]
                                      source_root
Positional arguments:
source_root All directories below this one that contain images will be processed
Options:
--dest-root If set, image directories below ``source-root`` will be copied into this directory tree. Only directories that contain image files will be copied; any intermediary directories required to contain directories that contain images will also be created.
--overwrite-source=False
 This flag must be used to overwrite images in the course directory.
--scripts-only=False
 This script will write a single shell script with the name provided in this option. No images will be modified or directories created, but the script can be run to do those things.
-r, --run-only

Select which scripts you want to run. This can be any combination of [p]atch, [a]strometry and [t]riage.

Possible choices: a, t, p

--no-blind=False
 Disable astrometry for images without pointing information
-o, --object-list
 Path to or URL of file containing list (and optionally coordinates of) objects that might be in these files. If not provided it defaults to looking for a file called obsinfo.txt in the directory being processed
--quiet-console=False
 Log only errors (or worse) to console while running scripts
--silent-console=False
 Turn off all logging output to console
--debug=False Turn on very detailed logging output
--quiet-log=False
 Log only warnings (or worse) to FILES AND CONSOLE while running scripts

Header patching: run_patch.py

For a detailed description of which header keywords are modified see Keywords purged before further processing.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Usage summary
usage: run_patch.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                    [--quiet-console] [--silent-console] [-o OBJECT_LIST]
                    [--overscan-only]
                    dir [dir ...]
Positional arguments:
dir Directory to process
Options:
-v=False, --verbose=False
 provide more information during processing
-d, --destination-dir
 Directory in which output from this script will be stored
--debug=False Turn on very detailed logging output
-n=False, --no-log-destination=False
 Do not write log files to destination directory
--quiet-console=False
 Log only errors (or worse) to console while running scripts
--silent-console=False
 Turn off all logging output to console
-o=https://raw.github.com/mwcraig/feder-object-list/master/feder_object_list.csv, --object-list=https://raw.github.com/mwcraig/feder-object-list/master/feder_object_list.csv
 Path to or URL of file containing list (and optionally coordinates of) objects that might be in these files. If not provided it defaults to looking for a file called obsinfo.txt in the directory being processed
--overscan-only=False
 Only add appropriate overscan keywords
DESCRIPTION

For each directory provided on the command line the headers all of the FITS files in that directory are modified to add information like LST, apparent object position, and more. See the full documentation for a list of the specific keywords that are modified.

Header patching

This is basically a wrapper around the function patch_headers.patch_headers() with the options set so that:

  • “Bad” keywords written by MaxImDL 5 are purged.
  • IMAGETYP keyword is changed from default MaxIM DL style to IRAF style (e.g. “Bias Frame” to “BIAS”)
  • Additional useful times like LST, JD are added to the header.
  • Apparent position (Alt/Az, hour angle) are added to the header.
  • Information about overscan is added to the header.
  • Files are overwritten.

For more control over what is patched and where the patched files are saved see the documentation for patch_headers at patch_headers.patch_headers().

Adding OBJECT keyword

run_patch also adds the name of the object being observed when appropriate (i.e. only for light files) and possible. It needs to be given a list of objects; looking up the coordinates for those objects requires an Internet connection. See

For a detailed description of the object list file see Object file format.

for a detailed description of the function that actually adds the object name see patch_headers.add_object_info().

If no object list is specified or present in the directory being processed the OBJECT keyword is simply not added to the FITS header.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

EXAMPLES

Invoking this script from the command line:

run_patch.py /my/folder/of/images

To work on the same folder from within python, do this:

from msumastro.scripts import run_patch
run_patch.main(['/my/folder/of/images'])

To use the same object list for several different directories do this:

run_patch.py --object-list path/to/list.txt dir1 dir2 dir3

where path/to/list.txt is the path to your object list and dir1, dir2, etc. are the directories you want to process.

From within python this would be:

from msumastro.scripts import run_patch
run_patch.main(['--object-list', 'path/to/list.txt',
               'dir1', 'dir2', 'dir3'])

Adding astrometry: run_astromtery.py

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

Usage summary
usage: run_astrometry.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                         [--quiet-console] [--silent-console] [-b]
                         dir [dir ...]
Positional arguments:
dir Directory to process
Options:
-v=False, --verbose=False
 provide more information during processing
-d, --destination-dir
 Directory in which output from this script will be stored
--debug=False Turn on very detailed logging output
-n=False, --no-log-destination=False
 Do not write log files to destination directory
--quiet-console=False
 Log only errors (or worse) to console while running scripts
--silent-console=False
 Turn off all logging output to console
-b=False, --blind=False
 Turn ON blind astrometry; disabled by default because it is so slow.
DESCRIPTION

For each directory provided on the command line add astrometry to the light files (those with IMAGETYP='LIGHT' in the FITS header).

By default, astrometry is added only for those files with pointing information in the header (specifically, RA and Dec) because blind astrometry is fairly slow. It may be faster to insert RA/Dec into those files before doing astrometry.

The functions called by this script set the WCS reference pixel to the center of the image, which turns out to make aligning images a little easier.

For more control over the parameters see add_astrometry() and for even more control, call_astrometry().

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

This script OVERWRITES the image files in the directories specified on the command line unless you use the –destination-dir option.

EXAMPLES

Invoking this script from the command line:

run_astrometry.py /my/folder/of/images

To work on the same folder from within python, do this:

from msumastro.scripts import run_astrometry
run_astrometry.main(['/my/folder/of/images'])

Find problems and create summary: run_triage.py
Usage summary
usage: run_triage.py [-h] [--debug] [-v] [-d DESTINATION_DIR] [-n]
                     [--quiet-console] [--silent-console] [-k KEY] [-l] [-a]
                     [-t TABLE_NAME] [-o OBJECT_NEEDED_LIST]
                     [-p POINTING_NEEDED_LIST] [-f FILTER_NEEDED_LIST]
                     [-y ASTROMETRY_NEEDED_LIST]
                     [dir [dir ...]]
Positional arguments:
dir Directory to process
Options:
--debug=False Turn on very detailed logging output
-v=False, --verbose=False
 provide more information during processing
-d, --destination-dir
 Directory in which output from this script will be stored
-n=False, --no-log-destination=False
 Do not write log files to destination directory
--quiet-console=False
 Log only errors (or worse) to console while running scripts
--silent-console=False
 Turn off all logging output to console
-k=[], --key=[]
 FITS keyword to add to table in addition to the defaults; for multiple keywords use this option multiple times.
-l=False, --list-default=False
 Print default list keywords put into table and exit
-a=False, --all=False
 Construct table from all FITS keywords present in headers and the list of default keywords.
-t=Manifest.txt, --table-name=Manifest.txt
 Name of file in which table is saved; default is Manifest.txt
-o=NEEDS_OBJECT_NAME.txt, --object-needed-list=NEEDS_OBJECT_NAME.txt
 Name of file to which list of files that need object name is saved; default is NEEDS_OBJECT_NAME.txt
-p=NEEDS_POINTING_INFO.txt, --pointing-needed-list=NEEDS_POINTING_INFO.txt
 Name of file to which list of files that need pointing name is saved; default is NEEDS_POINTING_INFO.txt
-f=NEEDS_FILTER.txt, --filter-needed-list=NEEDS_FILTER.txt
 Name of file to which list of files that need filter is saved; default is NEEDS_FILTER.txt
-y=NEEDS_ASTROMETRY.txt, --astrometry-needed-list=NEEDS_ASTROMETRY.txt
 Name of file to which list of files that need astrometry is saved; default is NEEDS_ASTROMETRY.txt
DESCRIPTION

For each directory provided on the command line create a table in that directory with one row for each FITS file in the directory. The columns are FITS keywords extracted from the header of each file.

The list of default keywords extracted is available through the command line option --list-default.

Note

This feature is available only from the command line.

For more control over the parameters see triage_fits_files()

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

EXAMPLES

Invoking this script from the command line:

python run_triage.py /my/folder/of/images

Get list of default keywords included in summary table:

python run_triage.py --list-default

To work on the same folder from within python, do this:

from msumastro.scripts import run_triage
run_triage.main(['/my/folder/of/images'])
# or...
run_triage.main(['--list-default'])

Manual intervention: quick_add_keys_to_file.py

Warning

This script OVERWRITES the image files in the directories specified on the command line. There is NO WAY TO DISABLE this behavior.

Usage summary
usage: quick_add_keys_to_file.py [-h] [--file-list FILE_LIST]
                                 (--key-file KEY_FILE | --key-value KEY_VALUE KEY_VALUE)
                                 [files [files ...]]
Positional arguments:
files Files in which to add/change keywords
Options:
--file-list File with list of files in which keywords are to be changed
--key-file File with keywords and values to be set
--key-value Keyword to add/change

Add/modify keywords in FITS files.

DESCRIPTION

Add each of the keywords in either the key_file or specified on the command line to each of the files either listed in the file file_list or specified on the command line. If the keyword is already present its value is updated to the value in key_file. A HISTORY comment is added to the header for each keyword indicating which keyword was modified.

Warning

This script OVERWRITES the image files in the list specified on the command line. There is NO WAY to override this behavior.


Sorting files based on image properties

Note

By default this script makes a copy of the images being sorted. There is an option for moving the files instead.

This type of sorting is handy for working with images in GUI software like AstroImageJ or MaxImDL, but will make it harder to process the data programmatically in python.

Usage summary
usage: sort_files.py [-h] [-v] [-d DESTINATION_DIR] [--debug] [-n]
                     [--quiet-console] [--silent-console] [--move]
                     dir [dir ...]
Positional arguments:
dir Directory to process
Options:
-v=False, --verbose=False
 provide more information during processing
-d, --destination-dir
 Directory in which output from this script will be stored
--debug=False Turn on very detailed logging output
-n=False, --no-log-destination=False
 Do not write log files to destination directory
--quiet-console=False
 Log only errors (or worse) to console while running scripts
--silent-console=False
 Turn off all logging output to console
--move=False, -m=False
 Move files instead of copying them.
DESCRIPTION

For the directory provided on the command line sort the FITS files in this way:

destination
    |
    |
    |
    |---'BIAS'
    |
    |---'DARK'
    |     |---exposure_time_1
    |     |---exposure_time_2, etc.
    |
    |---'FLAT'
    |      |---filter_1
    |      |    |---exposure_time_1
    |      |    |---exposure_time_2, etc.
    |      |
    |      |---filter_2, etc.
    |
    |---'LIGHT'
            |---object_1
            |       |---filter_1
            |       |    |---exposure_time_1
            |       |    |---exposure_time_2, etc.
            |       |
            |       |---filter_2, etc.
            |
            |---object_2
            |       |---filter_1
            |       |---filter_2, etc.
            |
            |---object_3, etc.
            |
            |---'no_object'
                    |---filter_1
                    |---filter_2, etc.

The names in single quotes, like ‘bias’, appear exactly as written in the directory tree created. Names like exposure_time_1 are replaced with a value, for example 30.0 if the first dark exposure time is 30.0 seconds.

The directory destination/calibration/flat/R will contain all of the FITS files that are R-band flats.

Note

This script is NOT RECURSIVE; it will not process files in subdirectories of the the directories supplied on the command line.

Warning

Unless you explicitly supply a destination using the –destination-dir option the files will be copied/moved in the directory in which they currently exist. While this should not lead to information loss, since files are moved or copied but never deleted, you have been warned.

EXAMPLES

Manual header processing

Overview

Sometimes the standard data preparation will fail at one stage or another, most often because pointing information is missing for an image or because no object was found matching the RA/Dec of the image. Your tool of choice in such cases, either to add pointing information or to add object names is quick_add_keys_to_file. A broad discussion of using it is at Fixes for cases that require intervention.

This document provides some examples of using quick_add_keys_to_file from the command line. See the documentation for add_keys() for use from python scripts.

Examples

Command line only

Add the keyword “OBJECT”, with value “EY UMa”, to the file image.fit:

quick_add_keys_to_file.py --key-value object "EY UMa" image.fits

The same, but for all of the files that match the pattern ey-uma*.fit:

quick_add_keys_to_file.py --key-value object "EY UMa" ey-uma*.fits

The rest of the command line examples you have created a file called keys.txt with a list of keyword/value pairs and a list of files called files.txt (you can call the files whatever you want, of course)

Command line and supporting files
Format of the keyword file

A keyword file looks like this (you need the header line):

Keyword,Value
OBJECT,"EY UMa"
RA,"09:02:20.79"
DEC,"+49:49:09.7"

You can include as many keywords as you want, and they can have numerical values instead of string values in appropriate. If the value has two words, like the value for the keyword “OBJECT” above, it must be in quotes, like “EY UMa”.

Keyword names are case insensitive because keywords in the FITS standard are case insensitive.

Format of the file list

A file list looks like this (yes, you need the header line):

File
MyFirstFile.fit
another_fits_file.fits
/or/even/the/full/path/to/a/fits/file.fit
Examples using keyword file/file list

Add all of the keywords in keys.txt to all of the files in files.txt:

quick_add_keys_to_file.py --key-file keys.txt --file-list files.txt

Add all of the keywords in keys.txt to the files image1.fit and image2.fit:

quick_add_keys_to_file.py --key-file keys.txt image1.fit image2.fit

Add keywords from the command line to all of the files in files.txt:

quick_add_keys_to_file.py --key-value my_key "some value" --file-list files.txt

Image Management

Working with a directory of images

For the sake of argument all of the examples below assume you are working in a directory that contains FITS images.

The class ImageFileCollection is meant to make working with a directory of FITS images easier by allowing you select the files you act on based on the values of FITS keywords in their headers.

It is initialized with the name of a directory containing FITS images and a list of FITS keywords you want the ImageFileCollection to be aware of. An example initialization looks like:

>>> from msumastro import ImageFileCollection
>>> keys = ['imagetyp', 'object', 'filter', 'exposure']
>>> ic1 = ImageFileCollection('.', keywords=keys) # only keep track of keys

You can use the wildcard * in place of a list to indicate you want the collection to use all keywords in the headers:

>>> ic_all = ImageFileCollection('.', keywords='*')

Most of the useful interaction with the image collection is via its .summary property, an `~astropy.table.Table`_ of the value of each keyword for each file in the collection:

>>> ic1.summary.colnames
['imagetyp', 'object', 'filter']
>>> ic_all.summary.colnames
# long list of keyword names omitted
Selecting files

Selecting the files that match a set of criteria, for example all images in the I band with exposure time less than 60 seconds you could do:

>>> matches = (ic1.summary['filter'] == 'I' & ic1.summary['exposure'] < 60)
>>> my_files = summary['file'][matches]

The column file is added automatically when the image collection is created.

For more simple selection, when you just want files whose keywords exactly match particular values, say all ‘I’ band images with exposure time of 30 seconds, there is a convenience method .files_filtered:

>>> my_files = ic1.files_filtered(filter='I', exposure=30)

The optional arguments to files_filtered are used to filter the list of files.

Iterating over hdus, headers or data

Three methods are provided for iterating over the images in the collection, optionally filtered by keyword values.

For example, to iterate over all of the I band light images with exposure of 30 seconds, performing some basic operation on the data (very contrived example):

>>> for hdu in ic1.hdus(imagetyp='LiGhT', filter='I', exposure=30):
...     hdu.header['exposure']
...     new_data = hdu.data - hdu.data.mean()

Note that the names of the arguments to hdus here are the names of FITS keywords in the collection and the values are the values of those keywords you want to select. Note also that string comparisons are not case sensitive.

The other iterators are headers and data.

All of them have the option to also provide the file name in addition to the hdu (or header or data):

>>> for hdu, fname in ic1.hdus(return_fname=True,
...                            imagetyp='LiGhT', filter='I', exposure=30):
...    hdu.header['meansub'] = True
...    hdu.data = hdu.data - hdu.data.mean()
...    hdu.writeto(fname + '.new')

That last use case, doing something to several files and wanting to same them somewhere afterwards, is common enough that the iterators provide arguments to automate it.

Automatic saving from the iterators

There are three ways of triggering automatic saving.

1. One is with the argument save_with_name; it adds the value of the argument to the file name between the original base name and extension. The example below has (almsot) the same effect of the example above, subtracting the mean from each image and saving to a new file:

 >>> for hdu in ic1.hdus(save_with_name='_new',
 ...                     imagetyp='LiGhT', filter='I', exposure=30):
 ...    hdu.header['meansub'] = True
 ...    hdu.data = hdu.data - hdu.data.mean()

It saves, in the ``location`` of the image collection, a new FITS file with
the mean subtracted from the data, with ``_new`` added to the name; as an
example, if one of the files iterated over was ``intput001.fit`` then a new
file, in the same directory, called ``input001_new.fit`` would be created.

2. You can also provide the directory to which you want to save the files with save_location; note that you do not need to actually do anything to the hdu (or header or data) to cause the copy to be made. The example below copies all of the I-band light files with 30 second exposure from the original location to "other_dir":

 >>> for hdu in ic1.hdus(save_location='other_dir',
 ...                     imagetyp='LiGhT', filter='I', exposure=30):
 ...     pass

This option can be combined with the previous one to also give the files a
new name.

3. Finally, if you want to live dangerously, you can overwrite the files in the location with the overwrite argument; use it carefully because it preserves no backup. The example below replaces each of the I-band light files with 30 second exposure with a file that has had the mean subtracted:

>>> for hdu in ic1.hdus(overwrite=True,
...                     imagetyp='LiGhT', filter='I', exposure=30):
...    hdu.header['meansub'] = True
...    hdu.data = hdu.data - hdu.data.mean()
msumastro.image_collection Module
Classes
ImageFileCollection(*arg, **kwd)

Turning an image collection into a tree

The class TableTree turns an Astropy Table into a tree based on the values in a particular column or columns.

TableTree(*args, **kwd) Base class for grouping images hierarchically into a tree based on metadata.
RecursiveTree() A dict-base recursive tree.

Header processing

Introduction

The msumastro.header_processing subpackage contains classes and functions that do the work of modifying headers. There are several scripts for Automated header processing and Manual header processing provided to carry out the most common types of headers processing.

Reference/API

msumastro.header_processing.fitskeyword Module
Classes
FITSKeyword([name, value, comment, synonyms]) Represents a FITS keyword, which may have several synonyms.
msumastro.header_processing.astrometry Module
Functions
call_astrometry(filename[, sextractor, ...]) Wrapper around astrometry.net solve-field.
add_astrometry(filename[, overwrite, ...]) Add WCS headers to FITS file using astrometry.net
msumastro.header_processing.feder Module
Classes
FederSite The Feder Observatory site.
ImageSoftware(name[, fits_name, ...]) Represents software that takes images at telescope.
Instrument(name[, fits_names, rows, ...]) Telescope instrument with simple properties.
ApogeeAltaU9() The Apogee Alta U9
MaximDL4() Represents MaximDL version 4, all sub-versions
MaximDL5() Represents MaximDL version 5, all sub-versions.
msumastro.header_processing.patchers Module
Functions
IRAF_image_type(image_type) Convert MaximDL default image type names to IRAF
add_image_unit(header[, history]) Add unit of image to header.
add_object_info([directory, object_list, ...]) Add object information to FITS files that contain pointing information given a list of objects.
add_object_pos_airmass(header[, history]) Add object information, such as RA/Dec and airmass.
add_overscan_header(header[, history]) Add overscan information to a FITS header.
add_ra_dec_from_object_name([directory, ...]) Add RA/Dec to FITS file that has object name but no pointing.
add_time_info(header[, history]) Add JD, MJD, LST to FITS header
change_imagetype_to_IRAF(header[, history]) Change IMAGETYP to default used by IRAF
get_software_name(header[, file_name, ...]) Determine the name of the software that created FITIS header
history(function[, mode, time]) Construct nicely formatted start/end markers in FITS history.
list_name_is_url(name)
patch_headers([dir, new_file_ext, ...]) Add minimal information to Feder FITS headers.
purge_bad_keywords(header[, history, force, ...]) Remove keywords from FITS header that may be incorrect
read_object_list([directory, input_list, ...]) Read a list of objects from a text file.