Masami Ouchi
National Astronomical Observatory of Japan/ Space Telescope Science
Institute
August 6, 2004 created
Translation by Catherine Ishida
National Astronomical Observatory of Japan
December 12, 2006 revised (Hisanori Furusawa, Yutaka Komiyama, YAGI Masafumi & SDFRED1 support team)
September 25, 2007 revised (Hisanori Furusawa, YAGI Masafumi & SDFRED1 support team)
May 1, 2008 revised (YAGI Masafumi & SDFRED1 support team)
September 1, 2008 revised (YAGI Masafumi & SDFRED1 support team)
April 6, 2009 revised (Ray S.
Furuya, YAGI Masafumi, & Hisanori Furusawa for SDFRED1)
Suprime-Cam is a mosaic CCD camera with ten 2048 x 4096-pixels CCDs. SDFRED1, the Suprime-cam Deep Field REDuction package 1, is the software package used for reducing Suprime-Cam data. SDFRED1 allows the immense data output from Suprime-Cam to be reduced semi-automatically, or even fully automatically, using standard parameters. After reduction, the reduced images are ready for scientific analysis.
SDFRED1 is optimized for photometry of deep field or blank field
images. Images that contain objects that occupy a significant portion of a chip may not reduce properly with this package. Moreover, we have not performed any quantitative tests for shape measurements to images produced by SDFRED1, such as those for weak lensing analysis. Special procedures and cautions for reducing such data will be described in each section.The copyright for SDFRED1 software belongs to Masami Ouchi who originally developed the package, and to Masafumi Yagi who supplied the basic code and made improvements. You may freely copy and distribute SDFRED1, but the following two papers should be cited in any scientific paper based on data reduced with SDFRED1:
Yagi et al., 2002, AJ, 123, 66
Ouchi et al., 2004, ApJ, 611, 660
You can get the latest version of the package at:
http://www.naoj.org/Observing/Instruments/SCam/sdfred/sdfred1.html.enWe strongly recommend that you visit
this web page from time to time to see updated information.
SExtractor
http://terapix.iap.fr/rubrique.php?id_rubrique=91
(Bertin & Arnouts 1996, A&AS,
117, 393)
IRAF(this is not necessarily required for SDFRED11, but is
pretty
handy to have)
http://iraf.nao.ac.jp/
or_ http://iraf.net/
The latest SDFRED1 (20080610) has been tested with version 2.12.2
SDFRED1 requires a UNIX-based computer equipped with 256MB of memory or more. Requirements for storage space depend on the quantity of data, but, typically you need several GB to several hundred GB. (To reduce the practice dataset, 10GB will besufficient.)
The latest version of SDFRED1 (06/10/2008) has been tested with Linux.
$ tar xvzf sdfred20080610.tar.gz
or
$ gunzip -c sdfred20080610.tar.gz | tar xvf -
(In this text, % or $ at the beginning of a line represents a shell prompt.)
A directory with a name like sdfred20080610/ will be created in the current directory. The eight-digit number represents the date of the software release.
Move (cd) into the sdfred20080610 directory and build the software as follows.
$ cd sdfred20080610/
$ ./configure
$ make all
Then programs are installed into sdfred20080610/bin/
Use an editor to add the sdfred20080610/bin directory (executables directory) to your PATH environment in your shell configuration file. In this example, /wa03a/subaru20/sdfred20080610/bin is the directory where SDFRED binaries are installed. You should modify to work with your environment.
Edit ~/.bashrc *and* ~/.bash_profile as follows;
Example:
$ emacs ~/.bashrc
$ emacs ~/.bash_profile
PATH=/wa03a/subaru20/sdfred20080610/bin:$PATH
export PATH
~/.bashrc is used when a new terminal is open, and ~/.bash_profile is used when you log into the computer. Therefore you need to modify both files.
After the files are updated, reflect the new setting to the current environment. This procedure is required only when you change the shell configuration files.
$ source ~/.bashrc
Edit ~/.cshrc to add the directory where SDFRED binaries are installed (executables directory) as follows;
Example:
% emacs ~/.cshrc
set path=( /wa03a/subaru20/sdfred20080610/bin $path )
After the files are updated, reflect the new setting to the current environment. This procedure is required only when you change the shell configuration files.
% source ~/.cshrc
% rehash
LANG, and LC_ALL should be set as "C", so that shell scripts and Perl scripts work correctly
Again, edit ~/.bashrc *and* ~/.bash_profile as follows;
$ emacs ~/.bashrc
$ emacs ~/.bash_profile
Add following two lines.
export LANG=C
export LC_ALL=C
And reflect the setting to the current session.
$ source ~/.bashrc
Again, edit ~/.cshrc as follows;
Example:
% emacs ~/.cshrc
Add following two lines.
setenv LANG C
setenv LC_ALL C
And reflect the setting to the current session.
% source ~/.cshrc
If you prefer to use IRAF, don't forget to execute mkiraf in your "work directory".
Additional settings may be required, depending on your computer environment.
When you finished with the settings, make sure your environment is as follows
Example:
$ env
$ which namechange.csh
In env command result, check LANG, and LC_ALL environment.
Make sure the "which" command can find the directory you have installed.
Once you have successfully finished
the software preparation,
you do not have to take these steps again.
Before staring a reduction, you need to sort data files into directories.
SDFRED requires that all input files must be in the "current" directory. If you want to process some files in a different directory, the files should be recognized as if they are in the "current" directory, by making symbolic links, or by other way.
A sample dataset (spcam_training_data.tar.gz; 753MB uncompressed) is
available at spcam_training_data.tar.gz.
We have checked that the software works well for this dataset. If you encounter any problem(s) during regular data reduction, we suggest you diagnose it using this well-tested sample data.
The specific examples given in this manual refer to this sample dataset.
Example:
extract sample data $ tar xvzf spcam_training_data.tar.gz
After the data extraction, check the images. Which are the object frames? Which can be used as flat flames? Which are the standard star frames?
Moreover, inspect images by eye using saoimage-ds9 or other your favorite FITS viewer. It is a good starting point to check for such issues as "Are there any files (i.e., exposures) showing distorted/elongated stars?", which allows you to eliminate bad data.
SUPA0020*.fits are the object frames (target object), and SUPA0019512*.fits are the standard frames in the sample dataset.
In this version (Ver.1.4), the data are assumed to be reduced/analyzed in two directories (object/ and standard/) which are in the same directory as spcam_training_data.
---(work directory root) - spcam_training_data/An example of making symbolic links:
- object/
- standard/
$ mkdir object standard
The result of using ls is;
$ ls
object spcam_training_data spcam_training_data.tar.gz standard
Then link object frames into object/ directory
$ cd object/
$ ln -s ../spcam_training_data/SUPA0020*.fits .
Copy blank map data
$ cp ../spcam_training_data/blankmap* .
Link standard object frames into standard/ directory
$ cd ../standard/Check that 10 FITS files are in standard/ directory, and 50 FITS files and six blankmaps are in object/ directory.
$ ln -s ../spcam_training_data/SUPA0019512?.fits .
All
data
obtained
with the Subaru Telescope
can be obtained through the archive system, SMOKA, after the 18-months of
the proprietary period has passed.
After getting the desired data, we suggest moving the target object and the standard object one into separate directories.
Check that data in a directory have observed with the same filter (e.g. V-band), and be the same data type (object/standard). Otherwise, you need to separate them into directories.
A typical procedure for reducing target object frames is:
Step | Command | time | Files Generated | |
(1) | Renaming | namechange.csh | 20s,1m | (H*.fits) |
(2) | Overscan and bias subtraction | overscansub.csh | 1s,1s | (To_TH*.fits) |
(3) | Making flat frames | mask_mkflat_HA.csh | 6m,20m | (Flat frames: *mflats*.fits) |
(4) | Flat fielding | ffield.csh | 30s,2m | (fTo_RH*.fits) |
(5) | Distortion correction and | |||
atmospheric dispersion correction | distcorr.csh | 1m,6m | (gfTo_RH*.fits) | |
(6) | PSF size measurement | fwhmpsf.csh | *,* | --- |
(7) | PSF size equalization | psfmatch_batch.csh | 40m,80m | (pgfTo_RH*.fits) |
(8) | Sky subtraction | skysb.csh | 1m,3m | (spgfTo_RH*.fits) |
(9) | Masking out AG probe | mask_AGX.csh | 15s,2m | (AspgfTo_RH*.fits) |
(10) | Masking out bad regions | blank.csh, .... | 15s,1m | (bAspgfTo_RH*.fits) |
(11) | Alignment | makemos.csh | 2m,15m | (Alignment file: *.mos) |
(12) | Coadding | imcio2a | 3m,12m | (Final Image) |
Standard Object frames:
Step | Command | Files Generated | |
(1S) | Renaming | namechange.csh | (H*.fits) |
(2S) | Overscan and bias subtraction | overscansub.csh | (To_TH*.fits) |
(3S) | Flat fielding | ffield.csh | (fTo_RH*.fits) |
(4S) | Distortion correction and | ||
atmospheric dispersion correction | distcorr.csh | (gfTo_RH*.fits) | |
(5S) | Relative gain correction | -- |
Each of the processes applies to a different subset of data frames. The basic data flow consists of making lists of these various subsets, and then providing these lists to various programs within the SDFRED package. Each step produces a new set of images or data files. The naming convention for the various new files are in parentheses after each step in the list above.
For reference, we provide typical times to reduce the sample dataset (five exposures), and a somewhat larger set (17 exposures) in the left- and right hand of the fourth columns, respectively. The time is measured using an Intel Xeon 3.0GHz WS with RedHat 5. The time is not linear to the number of the exposures. Although the actual time it will take to reduce any dataset will differ, the times should provide an idea of the relative amount of time one can expect each step to take.
To save disk space, you can delete files after they have been used
in the next step (except *.mos
and *mflats*.fits, since they
are also used in the standard object reduction). The users are
advised to keep the H*.fits
(renamed), fTo_RH*.fits
(flat
fielded), and AspgfTo_RH*.fits
(AG probe masked) files, since you never
know when you might need to re-reduce these data sets.
Note that other temporary files (tmp*; *.fits) are also produced in reduction processes. These can safely be removed.
$ namechange.csh [raw fits file list]
raw fits file list = names of raw data files
Prior to data reduction, it is useful to rename the data files using information associated with sensible parameters e.g., date of observations, exposure, and component CCD.
The filename such as SUPA... is changed as H[Date][type][ID]_[chipname].fits.
where
the
date,
YYMMDD, is one day
prior to DATE-OBS, and corresponds to Hawaiian Standard Time (HST)
of the first half of the night.
ID is the frame serial number of the observation day of each type(bias,dark,object). It should be noted that both target object(s) and standard star(s) have the same ID of "object".
Example:
$ cd object/
enters the directory of object frames
$ ls -1 SUPA*.fits > namechange.lis
Notice that the option of "ls -1" is "minus one".
$ namechange.csh namechange.lis
The namechange.lis should be like that
$ cat namechange.lis
SUPA00204610.fits
SUPA00204611.fits
SUPA00204612.fits
...
Result
The files are renamed as follows;H030425object025_si001s.fits
H030425object025_si002s.fits
H030425object025_si005s.fits
...
If you make symbolic links to the files in ../spcam_training_data/, they still points to SUPA***, but their names show up as H030425object025_si001s.fits. That is OK.
Note that each CCD of Suprime-Cam has a name:
-------- AG probe location ----------
w67c1 w6c1 si006s si002s w7c3
w93c2 w9c2 si005s si001s w4c5
$ overscansub.csh [overscansub.lis]
overscansub.lis = list of raw data files
The script overscansub.csh issues a command that subtracts the median value of the overscan region in each line, and trims the overscan region off from the frame. The Suprime-Cam CCDs typically have an overscan level of about 10000 ADU.
Since the CCDs in Suprime-Cam
have very little bias pattern, our experience suggests that
subtracting overscan should suffice for many cases.
Example:
$ ls -1 H*.fits > ovserscansub.lis
$ overscansub.csh ovserscansub.lis
The overscansub.lis should be like that
$ cat overscansub.lis
H030425object025_si001s.fits
H030425object025_si002s.fits
H030425object025_si005s.fits
...
Checkpoints:
$ mask_mkflat_HA.csh [mkflat.lis] [base name] [lower value] [upper value]
mkflat.lis = list of files to use to make flats
base name = basename for the flats
lower value = minimum value to accept (0.5 is recommended)
upper value = maximum value to accept (1.3 is recommended)
The script mask_mkflat_HA.csh creates a flat from files with objects. The flat file is used to correct the difference in sensitivities between pixels in a frame. Areas vignetted by the AG probe is masked out, normalized, and a median of them is taken.
There are three basic types of flats: sky flats (blank fields), twilight flats, and dome flats. Sky flats usually give the best result. This is because the optical path of the light for the night sky as well as the wavelength dependency are similar to those in your scientific frames. In fact, the target frames can be used to produce flats as long as there are no large objects that extend several hundred pixels in the frames.
Example:
$ ls -1 To_RH*.fits >mkflat.lis
$ mask_mkflat_HA.csh mkflat.lis obj 0.5 1.3
This is an example of making a sky flat by combining the object frames
After running the script, there should be flat files for the 10 CCDs.
obj_mflat_si001s.fits
obj_mflat_si002s.fits
...
The mflat files should have values around unity and should have a smooth pattern without much local structure. U-band and bands redder than z will have more structure than other bands. However, any local variations should be continuous. If there are abrupt changes in the flat values, consider creating another flat after eliminating (possible) bad exposures. Note that the value -32768 is the blank value used in SDFRED, and it is OK.
In principle, a flat can be produced with a minimum of three exposures. However, the smaller the number of frames used, the larger the noise and residual effects of objects in the frame. We recommend using at least six frames, ideally over 20 frames, to produce a (sensible) flat --- especially if you attempt to make a flat from sky frames. In the sample data, the flat is created from five exposures, yielding noticeable (but acceptable for robust reduction) error.
Keep in mind that the users should not mix the different types of flat exposures to make a flat frame because the background illuminations have intrinsically different slopes. For example, SDFRED may produce flats with discontinuous stripes when applied to frames with different illumination patterns. This is due to the algorithm used in SDFRED.
The SDFRED command uses a parameter file to mask out known bad columns and hot pixels. The default parameter file is optimized for data taken after August 2002 (Messia V). To reduce data taken before this, follow the procedures below.
$ cd sdfred20080610
For data taken before March 2001 (old CCD):
$ cp sdfredSH/mask_mkflat_HA/blankmap_oldCCD/* sdfredSH/mask_mkflat_HA/blankmap/
For data taken between April 2001 and August 2002 (Messia III):
$ cp sdfredSH/mask_mkflat_HA/blankmap_messiaIII/* sdfredSH/mask_mkflat_HA/blankmap/
To return to the default:
$ cp sdfredSH/mask_mkflat_HA/blankmap_messiaV/* sdfredSHmask_mkflat_HA/blankmap/
In the sample data, the flat is created in object/ directory, and will be shared for reducing standard-star data (by copying it into the standard/ directory). In many cases, several targets taken with an identical filter configuration can be mixed to create flats. If this is the case, it would be prudent to create another directory ../flat/. Subsequently, make symbolic links of To_R* files in all targets, and execute any commands in flat/ directory. (Note that the standard star frames that have shorter exposure times must not be included in the list.)
e.g.)
---(work directory root) - object1/
- object2/
- object3/
- flat/
- standard/
$ ffield.csh [ffiled_mf.lis] [ffield_im.lis]
ffield_mf.lis = list of flats to be used
ffield_im.lis = list of (overscan subtracted) files to be flat fielded
This command corrects the pixel-to-pixel variation in sensitivity, and the effect of vignetting of the telescope optics.
Example:
$ ls -1 obj_mflat*.fits > ffield_mf.lis
$ ls -1 To_*.fits > ffield_im.lis
$ ffield.csh ffield_mf.lis ffield_im.lis
After the flat fielding, the background in each file should be almost
flat.
The circular illumination pattern seen in the raw data at the
edge of the focal plane (w67c1, w93c2, w7c3, w4c5) should be nearly
gone at this stage. If the
flat is poor, you will see a low-level
(several percent of variation) illumination pattern. However, even if
you are not
happy with the residual pattern, you can safely ignore it and proceed
to Step 5 as long as you don't wish to achieve very high accuracy in
photometry (i.e., down to several percent). This is because such an
illumination pattern will be subtracted at the later, at Step 8.
On the other hand, if your scientific
goals require a rather high accuracy in photometry --- down to several
percent, consider going back to Step 3 to make another flat. The
"ffield.csh" used in Step 4 will help you to see the difference in the
patterns appeared in the frames listed in mkflat.lis. We suggest
eliminating the frames that have different patterns from those of
the object frames from mkflat.lis in the Step 3.
$ distcorr.csh [distcorr.lis]
distcorr.lis = list of (flat fielded) files to be corrected
The script distcorr.csh corrects the field distortion due to the telescope optics, and the differential atmospheric dispersion. The input frames are assumed to be flat-fielded images. The corrections are based on the airmass and other values recorded in the FITS header.
Example:
$ ls -1 fTo_RH030*.fits >distcorr.lis
$ distcorr.csh distcorr.lis
After the distortion correction, diagonal patterns may show up in the background. This is due to the fact that fractional pixel shifts smooth out the noise while integer pixel shifts leave the original noise characteristics intact.
$ fwhmpsf_batch.csh [fwhmpsf_batch.lis] [max number of objects]
[min peak flux] [max peak flux] [min FWHM] [max FWHM]
fwhmpsf_batch.lis = list of images to check PSF
max number of objects = the number of stars to use to measure
the PSF in each image
min peak flux = minimum peak flux of stars to use
max peak flux = maximum peak flux of stars to use
min FWHM = minimum FWHM of stars to use
max FWHM = maximum FWHM of stars to use
Before coadding, equalization of the PSF is required. The script fwhmpsf_batch.csh is used to determine an appropriate target PSF for a list of images. The script measures the FWHM of the PSF in several images. The script outputs a log and a histogram (exposure by exposure) to the standard output.
Example:
$ ls -1 gfTo_RH03042*.fits > fwhmpsf_batch.lis
$ fwhmpsf_batch.csh fwhmpsf_batch.lis 50 2000 30000 2.0 7.0 \
> fwhmpsf_batch.log
The log file should contain entries such as:
gfTo_RH030425object025_si001s.fits 3.60 1 6 15 13 0
gfTo_RH030425object025_si002s.fits 3.80 1 1 20 16 0
gfTo_RH030425object025_si005s.fits 3.60 2 13 19 0 0
...
3.3 |
3.4 |
3.5 |**
3.6 |**
3.7 |*
3.8 |**
3.9 |**
4.0 |
4.1 |*
The log format of each line is as follows:
[name of image]
[mean FWHM of PSF]
[number of objects with within 0.1'' of mean PSF-0.2]
[# within 0.1'' of mean PSF-0.1]
[# within 0.1'' of mean PSF]
[# within 0.1'' of mean PSF+0.1]
[# within 0.1'' of mean PSF+0.2]
An ASCII histogram following the log illustrates the distribution of mean PSFs.
Some training is required to determine the target PSF size.
Here, the sample images have FWHM values centered around 3.7 pixels. Adopting Target=4.1 would be the most conservative selection if you are interested in measuring fluxes of objects, but not in their structure. Target=3.9 would be a choice, if you believe the difference between 4.1 and 3.9 is negligible. Alternatively, if you have plenty of data, you may want to adopt a more strict threshold to exclude images having degraded PSFs (e.g, all data whose PSF are larger than 3.7). The selection depends on your scientific goals.
$ fwhmpsf.csh [image file] [max number of objects]
[min peak flux] [max peak flux] [min FWHM] [max FWHM]
image file = the image to check PSF
max number of objects = the number of stars to use to measure
the PSF in each image
min peak flux = minimum peak flux of stars to use
max peak flux = maximum peak flux of stars to use
min FWHM = minimum FWHM of stars to use
max FWHM = maximum FWHM of stars to use
Use the script fwhmpsf.csh to find the PSF of a single image. The parameters are the same as for fwhmpsf_batch.csh. Just supply the name of a image rather than a list of images.
Example:
$ fwhmpsf.csh gfTo_RH030425object025_si001s.fits 50 2000 30000 2.0 7.0
This produces output like:
gfTo_RH030425object025_si001s.fits 3.60 1 6 15 13 0
This output indicates that the image gfTo_RH030425object025_si001s.fits has a PSF FWHM of 3.6 pixels.
$ starselect.csh [image name][max number of objects][min peak flux]
[max peak flux][min FWHM][max FWHM][output file]
image name = name of image to check
max number of objects = the number of stars to use to measure the PSF
min peak flux = minimum peak flux of stars to use
max peak flux = maximum peak flux of stars to use
min FWHM = minimum FWHM of stars to use
max FWHM = maximum FWHM of stars to use
output file = name of file with location of selected stars
The script starselect.csh is useful for searching for the appropriate parameters ([max number of objects] [min peak flux] [max peak flux] [min FWHM] and [max FWHM]) for selecting stellar objects in an image.
$ starselect.csh gfTo_RH030425object025_si001s.fits 50 2000 \
30000 2.0 7.0 output.reg
The script will produce an output file (output.reg) that contains the location of stellar objects satisfying the given criteria. The output is formatted so that the stellar objects are plotted with green circles when you plot using saoimage-ds9. If the majority of the selected objects are "real stellar objects" (stars for many cases), then the parameters are appropriate for psf_match for a given image. If you realize that the quality of the data varies image by image, determine whether or not a single set of parameters can be applied for whole the data set. If it cannot, it is better to run psfmatch_batch multiple times using the appropriate criteria for each subset of data.
Using saoimage-ds9 is the easiest way to display an image and overlay the location of the selected stars.
$ ds9 gfTo_RH030425object025_si001s.fits
Select "Region", "Load", and select output.reg. Then green circles will be overlaid on the image. If more than half of the objects selected are stellar objects, the parameters you adopted are appropriate
The scripts fwhmpsf_batch.psf, starselect.csh and psfmatch_batch.csh(next step), may not work in crowded fields. In such fields, it may be necessary to estimate the PSF manually. If this is the case, to obtain the same results as the psfmatch script, each image that has a PSF more than 0.1'' smaller than the target PSF should be Gaussian smoothed with a Gaussian that has a sigma=sqrt(PSF_target^2 - PSF_image^2)/2.35482. The psfmatch program iterates around the estimated value of the sigma until the results converges to the best matched final PSF.
$ psfmatch_batch.csh [psfmatch_batch.lis] [max number of objects]
[min peak flux] [max peak flux] [min FWHM] [max FWHM] [target FWHM]
psfmatch_batch.lis = the list of images to match to a single PSF
max number of objects = the number of stars to use to measure the
PSF in each image
min peak flux = minimum peak flux of stars to use
max peak flux = maximum peak flux of stars to use
min FWHM = minimum FWHM of stars to use
max FWHM = maximum FWHM of stars to use
target FWHM = FWHM to smooth all the data to
The script psfmatch_batch.csh attempts to match the PSF of all images to be combined to a predetermined target FWHM. Images with PSFs smaller than the target (within a small range) are Gaussian smoothed, other images are simply copied. The target PSF should represent the typical PSF for the exposure, having the worst (i.e., the largest) PSF among the exposures to be combined.
The command prints a log to the standard output with the following columns:
[name of psf_matched_image]
[FWHM of PSF after matching]
[number of objects with within 0.1'' of target PSF-0.2]
[# within 0.1'' of target PSF-0.1]
[# within 0.1'' of target PSF]
[# within 0.1'' of target PSF+0.1]
[# within 0.1'' of target PSF+0.2]
The log can be used to check whether or not the PSF matching worked properly. To determine whether or not the command has ended successfully, make sure that the number of objects falling into (1) the bin of +/-0.1" of the final FWHM must be the peak of the distribution, and (2) outside bins e.g., PSF+0.2 arcsecond and PSF-0.2 are significantly small.
*If this is not the case, you are strongly encouraged to check the PSF manually.
*
*
*
**
**
***
***
***
*****
Example:
$ ls -1 gfTo_RH03042*.fits > psfmatch_batch.lis
$ psfmatch_batch.csh psfmatch_batch.lis 50 2000 30000 2.0 7.0 3.7 > psfmatch_batch.log &
The command produces output like:
pgfTo_RH030429object017_si001s.fits 3.70 0 5 38 6 0
The IRAF task "imexam" is handy for checking PSFs. (Display image; cl> imexam image.fits; place cursor above a star; type "r" or "a" to measure FWHM)
Keep in mind that each software routine may be using different fitting algorithms and may return different FWHM values. SDFRED adopts FWHM values generated by SExtractor, which are different from those produced by IRAF's imexam task. The purpose of checking with IRAF is not to find an exact match in the FWHM values, but to confirm that the output images have comparable PSF sizes after the matching.
The appropriate parameter values for psfmatch will change depending on the quality of the data. Different bandpasses, integration times, and weather conditions will require different parameters.
$ skysb.csh [skysb.lis] [sky-mesh]
skysb.lis = list of images to sky subtract
sky-mesh = size of mesh for determining sky values
The script skysb.csh (1) computes a mesh pattern that represents the sky background, (2) interpolates the pattern, and (3) subtracts it from the image. The script creates a grid --- referred to as "sky-mesh size squares" --- on the image with a grid spacing having the half of the "sky-mesh" size. An appropriate sky-mesh size will be selected for each mesh, and assigned to the pixel located at the center of the mesh. After rejecting the outliers, the sky values for other pixels will be given by interpolating bilineary from the surrounding meshes. Note that the sky-mesh size must be selected at least twice the largest object in interest due to the Nyquist sampling theorem.
Example:
$ ls -1 pgfTo_RH03042*.fits > skysb.lis
$ skysb.csh skysb.lis 64 > skysb.log
Once the sky background level is subtracted, the background in an
image should be
around zero without a spatial gradient.
If there is an extended object(s)
spreading over a large fraction of the
image,
the algorithm will most likely fail. Subtraction
of
sky
background in
crowded fields requires special data handling and you will need to
estimate the sky background manually.
$ mask_AGX.csh [mask_AGX.lis]
mask_AGX.lis = list of files to mask
The script mask_AGX.csh will mask areas vignetted by the AG (Auto-Guider) probe by the value -32768. The script should only affect the top few hundred rows of the data from chips w671, w6c1, si006s, si002s, and w7c3. Other files are not affected.
Example:
$ ls -1 spgfTo_RH03042*.fits > mask_AGX.lis
$ mask_AGX.csh mask_AGX.lis
Although only half the CCDs are potentially affected by the AG probe, the input file list should include all the object files so that files with the same naming convention exist to make list-making for subsequent steps easier.
Data in some pixels may be corrupted due to instrument trouble and/or other problems which may have occurred during the observation. Such regions must be common among the exposures (i.e., they are not time variable), and should be masked accordingly. For instance, we suggest masking the background areas where flattening fails and systematically deviates from zero. If plenty of exposures cover the observed region, we suggest not spending much time with this step. This is because outliers will be rejected automatically in Step 12.
The SDFRED1 package offers three
methods --- linear, circular, and rectangular regions --- to specify
regions to be masked
for eliminating bad pixels. Here, "Linear region" connects the two
points (x1,y1) - (x2,y2), extends the line to the edges of the
image, and masks the pixels within "width" from the line. The
"circular
region" masks the pixels in a circle. The "rectangular regions" masks
rectangular
regions aligned to the pixel coordinate.
$ line_bank [input image] [x1] [y1] [x2] [y2] [width]
[blank value] [output image]
input image = name of image to mask
x1 = x coordinate of start of line
y1 = y coordinate of start of line
x2 = x coordinate of end of line
y2 = y coordinate of end of line
width = width of line
blank value = mask value (usually -32768)
output image = name of masked image
The script line_bank masks a linear structure such as satellite trails.
Example:The example masks line which crosses (88,112) and (1940,837) and around 30 pixels width.
$ line_blank AspgfTo_RH030425object025_si001s.fits \
88 112 1940 837 30 -32768 lAspgfTo_RH030425object025_si001s.fits
$ circular_blanks [input image] [blanklist] [blank value] [output image]
input image = name of image to mask
blank list = a text file describing the x and y coordinates, as well as the radii of the areas to be masked
blank value = mask value (usually -32768)
output image = name of masked image
The script circular_blanks masks circular regions.
Example:
$ circular_blanks lAspgfTo_RH030425object025_si001.fits \
blanklist -32768 clAspgfTo_RH030425object025_si001s.fits
where blanklist looks like:
$ cat blanklistThe two lines correspond to circle of (x,y,r)=(365,1835,80) and (x,y,r)=(1202,3582,100).
365 1835 80
1202 3582 100
$ blank.csh [blank list]
blank list = list of images to be masked
For each image, xxx.fits, in the blank list the script blank.csh will look for a file named blankmap_xxx in the same directory, and mask rectangular regions specified in the file to -32768. Each line in the file blank_xxx should contain the x and y coordinates of two opposite corners of a rectangular area.
The IRAF routine imexam is useful for getting the coordinates. (cl> imexam; press "b" at two corners to define a rectangle; the coordinates of the corners will be printed to the screen in the order of x1 x2 y1 y2.)
Example:
$ ls -1 AspgfTo_RH03042*.fits > blank.lis
$ blank.csh blank.lis
Mask files have been included for a subset of images,
blankmap_AspgfTo_RH030425object025_si002s
...
These files have entries like:
$ cat blankmap_AspgfTo_RH030425object025_si002s
1974 2034 2356 2634
1528 1804 4024 4070
...
The script masks the regions specified in the corresponding blankmap_xxx file. If the blankmap_xxx file does not exist, the script will simply copy the image file to the output.
Note that the blankmap_* parameter files included in the sample dataset are from the Subaru Deep Field project and are not necessarily suitable for the sample dataset. If all of the sample files are applied to the sample data, areas that shouldn't normally be masked will be masked. These files are strictly for practice use. You may wish to create your own mask files to apply to the sample data.
$ makemos.csh [makemos.lis] [starsel nskysigma] [starsel npix]
[starsel peakmin] [starsel peakmax]
[aperture phot radius in pix] [output mos-file name]
makemos.lis = list of images to align
starsel nskysigma = signal to noise ratio of objects per pixel
to use for alignment
starsel npix = number of continuous pixels with [starsel nskysigma]
to identify object
starsel peakmin = minimum value of peak pixel of alignment stars
starsel peakmax = maximum value of peak pixel of alignment stars
aperture phot radius in pix = radius to use for aperture photometry
output mos-file name = file to record alignment and scaling
Signal-to-noise ratio (S/N) can be improved by combining multiple images (if you have them) to produce a final image. The script makemos.csh determines the shifts, rotations, and flux scales of different images. The script identifies stellar objects in each image and determines the the shifts, rotations, and flux scale from stellar objects common to multiple images. The first image in the list is used as the reference image.
Example:
$ ls -1 bAspgfTo_RH03042*.fits > makemos.lis
$ makemos.csh makemos.lis 5 20 500 10000 10 all.mos > makemos.log
The script will print to the standard output the number of stellar objects selected for alignment and scaling.
...
selected stars = 721
...
The script is likely to fail if the number of selected stars per image is either small (< 30) or very large (>1000). Optimizing key parameters such as [starsel nskysigma], [starsel npix], [starsel peakmin], and [starsel peakmax] will help the script to select appropriate stellar objects.
The best parameters for selecting objects in this step may be different from PSF measurement for many cases. This is because a different underlying algorithm is employed to find a wider range of objects to determine relative positions and flux scaling that works over a range of fluxes.
Example:
$ cat all.mos
bAspgfTo_RH030425object025_si001s.fits 0.000000 0.000000 0.000000 1.000000
bAspgfTo_RH030425object025_si002s.fits 1.601005 4088.551744 -0.000275 0.989406
bAspgfTo_RH030425object025_si005s.fits -2118.787371 1.104977 -0.000282 0.983195
...
As shown in the above, you will see five parameters (i.e., columns) in the output *.mos file: the name of the image, the x offset, the y offset, the counter clockwise rotation (radian), and flux ratio. If each result has four output parameters followed by the image name, the alignment or/and scaling have successfully finished. If the alignment and scaling have failed, the output file may not be produced at all, miss some parameters, or have unreasonable values.
We -- the SDFRED support team -- have been making continuous efforts to provide users more sophisticated method(s) that examines all.mos. Realizing the situation, however, we wish to share the following tips:
In the next step, each image is converted with the data in all.mos as follows;
x_mos = cos(theta) x -sin(theta) y + x_local
y_mos = sin(theta) x +cos(theta) y + y_local
$ imcio2a [parameters] [mos file] [result image]
parameters = parameters that define the combining algorithm usually
"-dist_clip -nline=20 -dtype=FITSFLOAT -pixignr=-32768"
mos file = file containing the alignment and scaling values
(output from makemos.csh)
result image= the name of the final image
imcio2a combines the images into a final combined image using the output from makemos.csh (*.mos). Using the parameter -dist_clip will combine the images using a clipped mean algorithm.
Example:
$ imcio2a -dist_clip -nline=20 -dtype=FITSFLOAT -pixignr=-32768 all.mos all.fits
The parameter -dist_clip can be replaced by -dist_med to get a weighted median combined image or -dist_add to use a weighted mean pixel values.
Note that the header of the output image is incomplete. Use the first file listed in makemos.lis in the previous step as a reference header.
Here are the meanings of the typical parameters:
-dist_clip : use a clipped mean algorithm for combining
-nline=20 : set the y direction buffer width to 20
-dtype=FITSFLOAT : make the output data floating point
-pixignr=-32768 : ignore pixels valued -32768
$ imcio2a -h
Steps 1S through 4S describe a typical procedure for reducing standard stars data. Since the physics behind this is the same as for reducing target objects, you can essentially repeat the procedure. Don't forget to work in the standard/ directory. The flat frames must be the same as those used for the objects, therefore be sure to copy them from the object/ directory.
$ namechange.csh [raw fits file list]
raw fits file list = names of raw data files
Renaming is done in the standard/ directory
Example:
$ cd standard/
enters into the directory of standard frames
$ ls -1 SUPA*.fits > namechange.lis
$ namechange.csh namechange.lis
The namechange.lis should be like that
$ cat namechange.lis
SUPA00195120.fits
SUPA00195121.fits
SUPA00195122.fits
...
Result
The files are renamed as follows;H030330object044_si001s.fits
H030330object044_si002s.fits
H030330object044_si005s.fits
...
$ overscansub.csh [overscansub.lis]
overscansub.lis = list of raw data files
In standard/ directory, overscan is subtracted from all the data as follows,
Example:
$ ls -1 H*.fits > overscansub.lis
$ overscansub.csh overscansub.lis
$ cat overscansub.lis
H030330object044_si001s.fits
H030330object044_si002s.fits
H030330object044_si005s.fits
...
and, To_RH030330object044_si001s.fits ... are created.
$ ffield.csh [ffiled_mf.lis] [ffield_im.lis]
ffield_mf.lis = list of flats to be used
ffield_im.lis = list of (overscan subtracted) files to be flat fielded
The flat frames used in this step
(ffield_mf.lis) must be identical to those used for the target(s) in
order to cancel out the uncertainty in the normalization.
Example:
$ cp ../object/obj_mflat*.fits .
$ ls -1 obj_mflat*.fits > ffield_mf.lis
$ ls -1 To_RH*.fits > ffield_im.lis
$ ffield.csh ffield_mf.lis ffield_im.lis
and fTo_RH030330object044_si001s.fits ... are created.
$ distcorr.csh [distcorr.lis]
distcorr.lis = list of (flat fielded) files to be corrected
The distortion correction is required since it slightly changes the sizes of the pixels, yielding slightly different flux value(s).
Example:
$ ls -1 fTo_RH*.fits >distcorr.lis
$ distcorr.csh distcorr.lis
Recall that the relative flux "scale" between different CCD chips has not yet been corrected even after the Step 4S. For example, the sensitivity of w67c1 is about one half in 2002-2008/06 data. If there is a star that has 10000 ADU in w67c1, it should have ~ 20000 ADU if it was observed with the other chip. The relative flux scale should be corrected according to the *.mos created in step (11) of the target object.
This step is unnecessary if the standard star is only in si001s. Since standard stars are distributed in several chips in the sample data, this process is needed.
In this step, the data is divided by a typical relative value of the chip to the reference chip. Currently (SDFRED ver1.*), the script for this step is not provided. Users should do this process manually.