reducer: Friendly astronomical image reduction

Want to use reducer in a class?

Want to use reducer in a class and need some help getting started/advice on use? There is a very brief summary of how I’ve used with undergraduates at Using with reducer with students.

Please contact me at mattwcraig@gmail.com or open an issue on the GitHub site for reducer if you have questions, run into problems, or want to let me know how you are using it!

Short walkthrough video

The YouTube video below has a walkthrough of the reducer notebook. No explanatory audio or text, check out the Quickstart for that.

Contents:

Quickstart

The reducer package generates a widget-based Jupyter notebook for reducing astronomical images. The actual reduction steps are done by ccdproc.

Installation

The recommended way to install reducer, especially on Windows, is with the Anaconda python distribution. Several of the packages that reducer depends on need to be compiled…and most people haven’t installed a compiler on Windows.

Installing with anaconda
  1. Download and install the Anaconda python distribution.
  2. Depending on your platform:
  1. Windows: Open the “Anaconda Command Prompt” from the start menu.
  2. Mac: Open the Terminal app (it is in Applications/Utilities)
  3. Linux: Open a terminal windows.
  1. Install reducer by typing, in the terminal: conda install -c mwcraig -c astropy reducer
Installing with other python distributions

Remember, this route requires that you have a compiler installed and properly configured. On Windows, you do not have that unless you have set it up.

Install reducer with pip:

$ pip install reducer

Generate a template notebook

To generate a notebook, navigate to the directory in which you want the reduced data to end up. Then, at the command line, type:

$ reducer

That will create notebook called reduction.ipynb. Open that notebook with:

$ jupyter notebook

Using the notebook

The first rule of using the notebook is to read the text cells of the notebook.

There are three kinds of widgets in the reduction notebook.

Simple image browser

Images are arranged based on the values of keywords in their FITS headers. Clicking on a file name displays the image and a tab for displaying the header.

_images/file_browser_newer.gif

The FITS keywords used to construct the menu tree at the left are determined by this line in the notebook:

tt = msumastro.TableTree(images.summary_info,
                         ['imagetyp', 'exposure'],
                         'file')

In this example, images were grouped by imagetyp and exposure.

Reduction step

Each reduction step (bias subtraction, dark subtraction and flat correction) has a widget to go along with it. The example shown below is for a processing a light (science) image.

_images/reduction_step.gif

The key thing to understand here is how reducer is selecting the appropriate master (or synthetic) calibration image for each step. To be considered for matching an image file has to have a keyword MASTER=True and the correct IMAGETYP for the step (e.g. FLAT for flat correction). In addition, for dark subtraction, the master dark frame whose exposure most closely matches the image being reduced is selected. For flat frames the FILTER of the flat must match the FILTER of the image.

You can select the images to which the reduction step is applied. The widget is created in the notebook with this:

1
2
3
4
5
6
7
8
9
light_reduction = astro_gui.Reduction(description='Reduce light frames',
                                      toggle_type='button',
                                      allow_bias=True,
                                      master_source=reduced_collection,
                                      allow_dark=True,
                                      allow_flat=True,
                                      input_image_collection=images,
                                      destination=destination_dir,
                                      apply_to={'imagetyp': 'light'})

The apply_to argument selects the images to which the the reduction step will be applied. To reduce only V-band images of M101 you could (assuming the appropriate keywords are in the FITS header, of course) use:

apply_to={'imagetyp': 'light', 'filter': 'V', 'object': 'M101'}
Image combination

Calibration images can be combined to make a master (or synthetic) image. An example of the widget that does that is below, shown for creating master flats.

_images/image_combination.gif

Note well that this will create several flats. To understand which images in the source directory will be identified as flats, how they will be grouped, and what the output files will be called let’s look at the notebook code the created the widget above:

1
2
3
4
5
6
7
8
flat = astro_gui.Combiner(description="Make Master Flat",
                          toggle_type='button',
                          file_name_base='master_flat',
                          group_by='exposure, filter',
                          image_source=reduced_collection,
                          apply_to={'imagetyp': 'flat'},
                          destination=destination_dir)
flat.display()

The apply_to argument on line 6 controls which images in the directory of reduced files will be considered flat frames by this widget. It can be a dictionary with whatever keywords you want.

The group_by argument on line 4 sets the names of the FITS keywords that will be used to group the flat frames. The setting in this example makes sense for dome flats. For twilight flats you presumably want to group only by filter. This setting can also be modified in the widget.

The file_name_base argument on line 3 determines part of the output file name for the combined flats. One flat is produced for each unique combination and the file names generated include the values of the keywords used to group them. For the sample data set that comes with reducer, these files are produced:

master_flat_filter_B_exposure_120.0.fit
master_flat_filter_I_exposure_5.0.fit
master_flat_filter_R_exposure_15.0.fit
master_flat_filter_V_exposure_30.0.fit

It could also be used to combine science images in the unlikely case that you wanted to simply average the images without aligning them.

Short video walkthrough

The YouTube video below has a walkthrough of the reducer notebook. No explanatory audio or text, but it goes through the entire reduction process.

Using with reducer with students

Disclaimer: My experience using reducer is with undergraduates, primarily physics majors. It has been used by a couple of non-majors with no issues.

The first hurdle is installation and a really fast intro to the terminal on the platform of their choice, since you need to launch a jupyter (nee ipython) notebook from the terminal.

For installation and set up I point them to a set of notes about the mechanics of getting started.

I then have them walk through the notebook using the small dataset included with the notebook. This is data that I’ve reduced by two in each image dimension and converted from 16 bit to 8 bit. It is fine for learning how to use the notebook, but the reduced data will look awful because I was not very careful about converting the calibration images.

If you want the same data set, but at original resolution and 16-bit, download it here (WARNING: 1.5GB). It is images of part of the Landolt field SA112 SF1, taken over one night in July 2013 at the Feder Observatory. It covers a reasonably wide range of airmass, so can be used as an example calculating atmospheric extinction and for determining the transformation to the standard magnitude system. The images contain WCS information, so it shouldn’t be too hard to identify the Landolt stars.

GUI API

reducer.gui Module

Functions
set_color_for(a_widget)
Classes
ToggleContainer(*args, **kwd) A widget whose state controls the visibility of its chilren.
ToggleMinMax(*args, **kwd) Widget for setting a minimum and maximum integer value, controlled by a toggle.
ToggleGo(*args, **kwd) ToggleContainer whose state is linked to a button.
Class Inheritance Diagram
Inheritance diagram of reducer.gui.ToggleContainer, reducer.gui.ToggleMinMax, reducer.gui.ToggleGo

image browser API

reducer.image_browser Module

Functions
ndarray_to_png(x[, min_percent, max_percent])
Classes
ImageTree(tree) Create a tree view of a collection of images.
FitsViewer() Display the image and header from a single FITS file.
ImageBrowser(collection[, allow_missing]) Browse a tree of FITS images and view image/header.
Class Inheritance Diagram
Inheritance diagram of reducer.image_browser.ImageTree, reducer.image_browser.FitsViewer, reducer.image_browser.ImageBrowser

astro_gui API

reducer.astro_gui Module

Classes
Reduction(*arg, **kwd) Primary widget for performing a logical reduction step (e.g.
Combiner(*args, **kwd) Widget for displaying options for ccdproc.Combiner.
CosmicRaySettings(*args, **kwd)
Slice(*arg, **kwd)
CalibrationStep(*args, **kwd) Represents a calibration step that corresponds to a ccdproc command.
BiasSubtract([bias_image]) Subtract bias from an image using widget settings.
DarkSubtract([bias_image]) Subtract dark from an image using widget settings.
FlatCorrect([bias_image]) Subtract dark from an image using widget settings.
Overscan(*arg, **kwd) docstring for Overscan
Trim(*arg, **kwd) Controls and action for trimming a widget.
Class Inheritance Diagram
Inheritance diagram of reducer.astro_gui.Reduction, reducer.astro_gui.Combiner, reducer.astro_gui.CosmicRaySettings, reducer.astro_gui.Slice, reducer.astro_gui.CalibrationStep, reducer.astro_gui.BiasSubtract, reducer.astro_gui.DarkSubtract, reducer.astro_gui.FlatCorrect, reducer.astro_gui.Overscan, reducer.astro_gui.Trim

Contributors

Project Coordinator

  • Matt Craig (@mwcraig)

Contributors

All package contributors, listed alphabetically.

  • Juan Cabanela (@JuanCab)
  • Nathan Heidt (@heidtna)
  • Stuart Littlefair (@StuartLittlefair)
  • Paige Meyer (@meyerpa)
  • Thomas Robitaille (@astrofrog)
  • Christian Tismer (@ctismer)

Changes

1.0.0 (unreleased)

General
New Features
Other Changes
Bug fixes

0.3.0 (2016-07-17)

General
  • This version only supports IPython 4 or higher, and requires ipywidgets version 4.
  • The minimum required version of ccdproc is now 1.0.
New Features
  • Images can now simply be copied from the source to the destination directory. [#137]
Other Changes
Bug fixes

0.2.9 (2016-06-16)

General
New Features
Other Changes
  • Update package requirements to ipywidgets instead of ipython, and restrict version number.
Bug fixes
  • Use numpy dtype name instead of dtype itself to determine output dtype. [#129]

0.2.8 (2016-05-31)

General
New Features
Other Changes
Bug fixes
  • Check that the image collection for master images exists before refreshing it. [#128]

0.2.7 (2016-05-30)

General
New Features
Other Changes
Bug fixes
  • The ImageFileCollection used to find masters was out of date and not refreshed if a reduction widget was created before the masters were created. [#127]

0.2.6 (2016-05-27)

General
New Features
Other Changes
  • Use combine function for combining images to limit memory usage during image combination. [#120, #121]
  • Use median and median_absolute_deviation in sigma clipping instead of the default mean and std. [#106]
  • Discard mask/uncertainty from result of image combination unless input images have mask/uncertainty. [#119]
  • Choose sensible data type for reduced images based on data type of original images. [#122]
Bug fixes
  • Eliminate huge memory usage by reduction. [#118]

0.2.5 (2016-05-25)

General
New Features
Other Changes
  • Improve display of images in file browser.
Bug fixes
  • Work around a bug in ccdproc/astropy.nddata that incorrectly creates an uncertainty as a mask.
  • Work around a bug in astropy.io.fits that results in writing incorrect data values in some cases.

0.2.3 (2016-05-23)

General
New Features
Other Changes
Bug fixes
  • Ensure unsigned int images can be displayed. [#115, #116]
  • Ensure that combined images can be written. [#117]