rctnight - More automated reductions

Usage:

rctnight infiles outdir parfile

rctnight infiles outdir parfile a

rctnight indir outdir parfile

rctnight indir outdir parfile a

rctnight 'infilestemplate' outdir parfile

rctnight 'infilestemplate' outdir parfile a

Rctnight is a Perl script designed to perform the same type of data reductions provided by the combination of rctred and rctcom (reducing images and combining images). Furthermore, it doesn't require the user to organize images into lists prior to starting the script as do rctred and rctcom and can make automated decisions in order to operate in an automatic mode. Rctnight will examine the headers of all images provided to it to organize them into groups of images that can be reduced together (e.g., taken in the same filter, same gain setting, same readout mode, etc.).

Rctnight takes 3 required arguments. The first is one of three things: 1) the input data directory (where images to be reduced are stored), 2) a list of filenames (including full pathnames if not in the current directory), or 3) a template for the input filenames (or directory names) which can contain wildcard characters like '*' and '?' as used in many UNIX commands. If a template is used, it must be enclosed in single quotes ('')! If the first argument is a directory, the FITS files are assumed to end with a '.fits' suffix. If a list of filenames or a template for filenames is used, the user is responsible for seeing that only FITS images are input to the script. If a template is used, it is first assumed to be a template for filenames (not directory names). If, and only if, no files are found to match the template, rctnight attempts to find directories that match the template and looks for files in those directories that end with a '.fits' suffix. The second argument is the name of the output data directory (where reduced images are to be written). The third is the name of a parameters file. The fourth argument is optional. When it is set to 'a', the script runs automatically without pausing for user interaction (which it would normally do).

The parameters file (a simple text file) can exist in the user's current directory or anywhere else on the system if its full pathname is given. The format of this parameters file is one in which the name of a parameter appears on a line followed by one or more values. Spaces separate the parameter name from its values and the values from one another. The parameters can be listed in any order, but their names must be spelled as specified. They can be omitted from the file, in which case default values will be applied to them by the script, but this isn't necessarily advisable. If you list a parameter, you should supply at least one value to it unless there are specific directions telling you it is OK to have no values (e.g., for the comboname parameter described below). This is the case even if it is a parameter that has no effect because it is nullified by another parameter (supply a dumy name in such cases if you like).

The following parameters are recognized...

First there are a number of "restrictive" type parameters. They tell the script to restrict data reduction to only those images whose image headers or filenames match certain values. If the value is set to the special value 'all', then no restrictions are placed by that parameter on the images selected.

1. observer

Only images whose header keyword OBSERVER matches one of the values given are reduced.

2. date

The values should be 6-digit strings giving a date in the format YYMMDD. Only images whose header keyword DATE-OBS corresponding to one of the values given are reduced.

3. project

Only images whose header keyword PROJECT matches one of the values given are reduced.

4. propi

Only images whose header keyword PROPI matches one of the values given are reduced.

5. obstype

Only images whose header keyword OBSTYPE matches one of the values given are reduced.

6. object

Only images whose header keyword OBJECT matches one of the values given are reduced.

7. filter

Only images whose header keyword FILTER matches one of the values given are reduced.

8. exptime

Only images whose header keyword EXPTIME matches one of the values given are reduced.

9. clocking

Only images whose header keyword CLOCKING matches one of the values given are reduced.

10. ccdsum

Only images whose header keyword CCDSUM matches one of the values given are reduced.

11. gain

Only images whose header keyword GAIN matches one of the values given are reduced.

12. mppmode

Only images whose header keyword MPPMODE matches one of the values given are reduced.

13. filename

Only images whose filenames match one of the templates listed are reduced. The templates may include two wildcard characters as used by many UNIX commands: * and ?. Note that the use of this parameter is in addition to requirements the script may already have in where it should look for FITS images (by command-line arguments).

You need to be careful about what values you put in these parameters. This is because 1) each word listed after the parameter is considered to be a separate value and 2) the program will remove leading and trailing whitespace in both the parameters you input and string values read from the image headers. This means that if you want to match a keyword which is composed of more than one word, you must enclose your parameter value in single quotes (''). As an example for the second point, suppose the keyword PROJECT is set to 'Project X ' in an image header (note the whitepace). You can set the project parameter to something like 'Project X' or ' Project X ' for a match. A single space (in this case) between the words Project and X is necessary to get a match. Leading and trailing whitespace will be removed from both values before comparing them.

The other parameters are "operational" in nature. They tell the script what operations to perform on each image. They are quite similar to the parameters input to the rctred and rctcom scripts since rctnight calls on these scripts. It is up to the user to make sure that the operations performed on the images are appropriate for all of the images picked out by the restrictive parameters.

14. outputnames

is either 1) the name of a file containing the filenames (not full pathnames) of reduced images to be made, or 2) the "special" pair of values 'prefix' followed by 'r', which tells the script to create the output names automatically by prefixing the input names with a string, in this case an 'r'. The user must make sure that there is an output filename for each input filename if a list of names is used.

15. osbias

is either 'y' or 'n' telling the script "yes", subtract an overscan bias level from the image, or "no", don't do this.

16. fit

tells how to determine the overscan bias as a function of row in the image. If 'mean', the mean value of each row in the overscan region is subtracted from its corresponding row in the image. If 'constant', the mean value of the entire overscan region (a single value) is subtracted from the image. If 'cheby', a Chebyshev polynomial is fit through a running mean of the overscan region. Of course, if the CCD is readout through multiple amplifiers, a separate calculation is done for each area readout.

17. order

is an integer giving the order of the function fit to the overscan region(s) This parameter is applicable if fit='cheby'.

18. osrej

is either 'y' or 'n' telling the script "yes", reject overscan pixel values that deviate significantly from the other values in the same overscan row, or "no", don't do this.

19. osrejtype

tells what sort of method to use to reject deviant pixels from the overscan region before determining the overscan bias. If 'minmax', the high- and low-valued pixels in each row of the overscan are rejected. If 'sigclip', pixels deviating from the mean in the row by more than 3 times the standard deviation are rejected. This parameter is in effect when osrej='y'.

20. trim

is either 'y' or 'n' telling the script "yes", trim away the parts of the image that are the overscan bias, or "no", don't do this.

21. twodbias

is either 'y' or 'n' telling the script "yes", subtract a calibration frame from the image, or "no", don't do this. This operation is performed after any overscan bias subtraction and trimming is done. Any calibration frame used should have the same size as the image at this point in the process or an error will result. Probably the calibration frame will already have been reduced in some way. For instance, a 2-D pattern in the bias can be removed by providing the script with a bias image. The "bias" might also be made to contain dark counts by adding a dark into this image.

22. twodname

should be one of three things: 1) name of the calibration frame to subtract if requested by the twodbias parameter, 2) a directory name where that frame can be found, or 3) a template for the name of the calibration frame using wildcard characters like ? and *. This parameter only has an effect if twodbias is set to 'y'. The full pathname must be provided unless this image exists in the directory where rctred is invoked.

23. subdark

is either 'y' or 'n' telling the script "yes", subtract a dark frame from the image, or "no", don't do this. This operation is performed after any overscan bias subtraction, trimming, and 2-D bias pattern subtraction is done. Any dark frame used should have the same size as the image at this point in the process or an error will result.

24. darkname

should be one of three things: 1) name of the dark frame to subtract if requested by the subdark parameter, 2) a directory name where that dark frame can be found, or 3) a template for the name of the dark frame using wildcard characters like ? and *. This parameter only has an effect if subdark is set to 'y'. The full pathname must be provided unless this image exists in the directory where rctred is invoked.

25. flatten

is either 'y' or 'n' telling the script "yes", divide by a calibration image (e.g., a flat) or "no", don't do this. This operation is performed after the steps above have been taken (bias subtraction and trimming).

26. flatname

should be one of three things: 1) name of the flatfield frame to use if requested by the flatten parameter, 2) a directory name where that frame can be found, or 3) a template for the name of the flatfield frame using wildcard characters like ? and *. This parameter only has an effect if flatten is set to 'y'. The full pathname must be provided unless this image exists in the directory where rctred is invoked.

25. document

is either 'y' or 'n' telling the script "yes" write comments or history information into the image headers of the output images describing the reduction steps taken.

27. cleanup

is either 'y' or 'n' telling the script "yes" remove certain (somewhat extraneous) files from the output directory once the script is done using them, or "no" leave the files there. The files eligible to be cleaned up are those containing input and output image names created automatically by rctnight called i0.dat, i1.dat, r0.dat, r1.dat etc. and a file called rctnight.lists which is the file edited by the user when operating rctnight in interactive mode. Note that the lists of input and output images are created by the script as temporary files passed to rctred and rctcom and are not the same as any lists supplied by the user! A user might want to set this parameter to 'n' if it will help to debug certain problems. The default value is 'y'.

28. combine

is either 'y' or 'n' telling the script "yes" to combine the groups of images that are possible to combine or "no", don't do this. The script tries to decide which images to combine and ensures that they have the same size, gain, ccdsum, readout mode, etc. The remainder of the parameters control the combining of the images...

29. scaletype

is the method used to find factors that are used to scale pixel values in the individual images before deviant pixels are rejected and the images combined. The allowed values are 'none', 'median' and 'mean'. If you're using the IRAF version of rctcom, there's the additional value of 'mode'. If scaletype is set to something other than 'none', scaling factors are calculated so that each scaled image will have the same mode, median, or mean.

30. combtype

is the method used to combine the images. It is either 'average' or 'median'.

31. rejtype

is the method used to reject deviant pixels before combining them. This is performed after the images have been scaled. The allowed values are 'none', 'minmax', and 'sigclip'. If minmax rejection is chosen, a specific number of the highest and lowest valued pixels in the list of images is rejected. The number of highest and lowest pixels to be rejected are given by the parameters numhigh and numlow. The user is responsible for ensuring that the number of pixels rejected does not equal or exceed the total number of images being combined. If sigclip rejection is chosen, the standard deviation of values for each pixel is found for the images in the list. Pixels which deviate from the mean by an amount greater than a certain factor times the standard deviation are rejected. These factors are given by the parameters lowsigma and highsigma for deviations less than or greater than the mean respectively.

32. numlow

is an integer giving the number of lowest-valued pixels to be rejected if rejtype is set to 'minmax'.

33. numhigh

is an integer giving the number of highest-valued pixels to be rejected if rejtype is set to 'minmax'.

34. lowsigma

is a factor to be multiplied by the standard deviation of each pixel in the image list to find a threshold for deviant pixel rejection. Any pixel whose value is less than the mean by an amount greater than this threshold is rejected.

35. highsigma

is a factor to be multiplied by the standard deviation of each pixel in the image list to find a threshold for deviant pixel rejection. Any pixel whose value is greater than the mean by an amount greater than this threshold is rejected.

36. normalize

is set to 'yes' or 'no' telling the script "yes", normalize the combined image to have a final mean value of 1 or "no", do not do this.

37. comboname

establishes the name of the combined image. This parameter can be set to either a string or left unset (having the parameter name 'comboname' on a line by itself in the parameters file leaves it unset). The combined image name begins with a root word. If comboname is set, its value is taken as the root word. If it is left unset, the root word is picked automatically to start with a string denoting the object type as given by the OBSTYPE keyword followed by a 6-digit date string giving the UT date of observation in YYMMDD format (e.g., root words might be 'flat040512', 'bias040512', 'object040512', etc.). For flat and object type images, the root word is followed by a dot ('.') and then a string which designates the filter(s) used. No filter is shown in the case of darks or biases. Finally, there is a '.fits' suffix for each combined image name. However, in the case that the combined image name is already taken by an existing image on disk, the new combined image name is given an additional integer designation '.2', '.3', '.4' etc. to make it unique. The string which designates the filter(s) used can be derived by either using a special look-up table or in an automated fashion. See below for more details.

It is helpful to understand how rctnight treats each image given to it. The script first gathers together all FITS files given to it (either by directory name, by a list of filenames, or template of file or directory names). All of those files whose names match a pattern given by the filename parameter (if not set to 'all' or '*') are kept. It then looks through each image's FITS header for certain keywords that are useful for deciding what to do with each image. It culls from the list of images any that don't meet the qualifications imposed by the 'restrictive' type parameters (ie. observer, propi, date, obstype, object, filter, exptime, clocking, ccdsum, gain, mppmode, and project).

Next, the list of remaining images is split up into one or more sublists of images that can be reduced in the same manner (and combined together if need be). The images in the sublists must match one another in terms of their obstype, filter, gain, ccdsum, windowing, mppmode, overscan, clocking, image size, and (input) directory location. (The fact that the sublist images must all be located in the same directory is just an artificial software issue.) If dark frames are being combined or used to reduce other images, the exposure times for images in each sublist must also match. Any bias, dark, or flatfield calibration images needed for the reduction are found for each sublist according to the twodname, darkname, and flatname parameters. The calibration images are required to match the sublist images in terms of CCD readout parameters (and filter in the case of flats). Please note that the script does not require the time or date of observation for images in the sublists to be the same or similar. This means that if there are different sets of flats that the user intends to reduce and combine separately (but which match one another in terms of CCD readout mode and filter), the user must take extra steps to make sure they get reduced separately (e.g., by setting a restrictive parameter like date or object to distinguish between the sets or by placing the raw flats in different directories).

The filtername used for combined image names is normally derived using a look-up table found in the same directory as the rctnight script (its default name is 'filters.dat'). This table consists of two columns; the first column corresponds to strings that are possible values for the RCT image header keywords FILTNA1 and FILTNA2. The second column corresponds to strings intended for use in the filename (which should be free of odd characters like as ], /, (, $, * etc.). Lines in this file which begin with a '#' character in the first space are treated as comments. If this table does not exist or there is no entry which matches the filter as given by the image header, rctnight creates a name for the filter based on characters found in the keyword value (it takes the keyword value(s) and removes any "bad" characters like ], /, etc). If two filters are in place, the filtername is a combination of both separated by a '+' character; the first filter matched in the filter table is listed first, otherwise the filter in wheel 1 is listed first.

An example parameters 'filters.dat' is listed below:

	u		U
	b		B
	v		V
	r		R
	i		I
	4686a/heii		HeI4686
	4861a/hbeta		Hbeta
	5007a/[oiii]		OIII5007
	5300a/green_continuum_#2		Green2
	5755a/[nii]		NII5755
	6450a/red_continuum		Red
	6563a/halpha_1nm		Halpha1
	6584a/[nii]		NII6584
	6717a/[sii]		SII6717
	6731a/[sii]		SII6731

An example parameters file for rctnight is given below:

observer all

propi all

date all

obstype bias

object all

filter all

exptime all

clocking all

ccdsum all

gain all

mppmode all

project ESP

filename *.fits

outputnames prefix r

osbias y

fit mean

order 0

osrej y

osrejtype minmax

trim y

twodbias n

twodname not_used

subdark n

darkname not_used

flatten n

flatname not_used

document y

combine y

scaletype none

combtype average

rejtype minmax

numlow 1

numhigh 1

lowsigma 2.5

highsigma 2.5

normalize no

comboname

A good way to create this parameter file is to use a companion script to rctnight called setnight, which will help to ensure the proper format.