(r4) CookBook2dfdr < Main

Tags: view all tags
---++!! !CookBook for AAOmega data reduction with 2dfdr

AAOmega data is processed using a revised version of the 2dfdr software package.  2dfdr currently has modes which reduce data for a number of instruments, AAOmega with the 2df and Spiral IFU top ends, 2dF, 6dF, FMOS and the older Spiral.  In upgrading the software to work with AAOmega data a number of fundamental changes to the software have been required.  At this time, the full 2dfdr manual 
([[http://local.aao.gov.au/AAO/2df/manual/UsersManual.doc][Word]],
[[http://local.aao.gov.au/AAO/2df/manual/UsersManual.pdf][pdf]])
is somewhat out of date and does not cover the modifications and improvements.  This !CookBook is aimed at providing a basic introduction to the reduction process.  An updated manual will be available shortly.  In the mean time, please contact your support astronomer with any questions about the software or problems with data reduction.

%TOC%

Installing the software

To install the software, download and unpack the relevant .tgz file from the ftp site.  Noting the path to the unpacked directory (2dfdr-linux_x86 for the RH3E version) first set an environment variable, then source the setup script :
setenv DRCONTROL_DIR /path/
source $DRCONTROL_DIR/2dfdr_setup

Run the software with

drcontrol aaomega.idx  &

Where aaomega.idx is the generic AAOmega setup file.  A number of specific .idx files are available (e.g. aaomega580V.idx, aaomega385R.idx) for individual gratings.



---+++ Download the software
2dfdr is rapidly evolving as more is learned about AAOmega data.  The high quality of AAOmega spectra, in comparison to what was achievable with the older 2dF spectrograph (which was, it must be stressed, principally designed to record low resolution redshifts of bright galaxies) means that there are many improvements required in the reduction system to do justice to the data recorded with AAOmega.  The user should check the AAOmega software web page before starting a campaign of data reduction to ensure they have the every latest version of the software.  This rate of evolution will deminish once a small number of outstanding issues are addressed.

The software is only available as a set of binary executables, and the source code is not readily available to the public.  The system is created with the Starlink software library and cannot be built without a Starlink installation.  While this software is not proprietary, we cannot support external builds at this time.  With the demise of the Starlink project in 2005, it may become necessary to remove the Starlink components in the future.

The executables are currently only available for the !RedHat Enterprise3 Linux.  This system has been tested and works well under current implementations of !RedHat Enterprise4 and most installations of Fedora, Scientific Linux and other popular flavors.  Spme problems have been tracked to older Tcl libraries being used. Limited support can be offered for other operating systems.  A Solaris version can be made available on request, but due to speed problems is unlikely to receive much use.  A version for Mac OSX is not available at this time due to complications with running elements of the Starlink libraries on the Mac.

Download the latest tar file from the [[ftp://ftp.aao.gov.au/pub/2df/][FTP site]]

The file will look something like: 2dfdr-RH3E_060610.tgz

Unpack the tar file and to extract the software to your chosen working directory

<verbatim>
tar -zxvf 2dfdr-RH3E_060610.tgz
</verbatim>

---+++ Running 2dfdr for AAOmega
First we need to set an environment variable and run the setup script

<verbatim>
    tcsh
    setenv DRCONTROL_DIR /path/to/software/2dfdr-linux_x86
    source $DRCONTROL_DIR/2dfdr_setup
</verbatim>

2dfdr should be run in a separate working directory for each set of observations with a particular field plate.  A meaningful directory structure for your observing run can save a lot of heartache later on.  An example directory structure might be:

<verbatim>
    Observing_june05/
        night1/
            field1/
            field1b/
            field2/
            field3/
        night2/
            field1/
            field4/
</verbatim>

Note that due to the way the flat and arc frames are used, each independent observation (i.e. with a different configuration of the fibres on the field plate) will require a new directory, even if all you have done is tweak the positions of fibre on a previously observed configuration.  Once the AAOmega slit wheel is moved, a new set of flats and arcs are required for the reduction.  Data from multiple repeats of the same field, or for fields that contain some repeat observations can be automatically combined, but this is done after the full reduction of data for each field.

Data from the blue and red arms can be reduced in the same directory, but this is often not easy to work with and so most users create separate ccd_1 and ccd_2 sub directories with blue and red data, respectively.

Move to your working directory of choice and then the software can be started with the command

<verbatim>
    drcontrol aaomega.idx &
</verbatim>

which selects defaults for AAOmega, or

<verbatim>
    drcontrol aaomega###.idx &
</verbatim>

where ### is the grating name (e.g. 385R, 580V or 1700D etc.) with which the data was taken.  The .idx files are all stored at

/path/to/software/2dfdr-linux_x86/*.idx

Users, if required, can make a copy of these instrument (.idx) files in the local directory and modify them to set their own reduction preferences.  Not all grating configurations currently have .idx files created.

These commands should bring up the main 2dfdr control window shown here.

<img src="%ATTACHURLPATH%/2dfdr_main_gui.jpg" alt="2dfdr Main GUI" width='1009' height='579' />

---+++ Starting a reduction

The basic files needed to reduce AAOmega data are:

   * a multi-fibre flat field exposure (class MFFFF) These exposures are made with a quartz lamp that provides a uniform spectrum. They are used to calibrate the detector, and to find the centre and profile of the spectra.

   * an arc exposure (class MFARC) These exposures are made with lamps having various known strong lines. They are used to calibrate the central wavelength and dispersion.

   * one or more science frames

Additional frames of various types may be needed, depending on the nature of your data.  To start the automatic reduction process, hit the *Setup* button in the upper right hand section of the GUI.  The window shown here will appear.

<img src="%ATTACHURLPATH%/setup_gui.jpg" alt="Setup GUI" width='530' height='189' />

The automatic data reduction depends on the use of a file naming convention in which the name has a root that is the same for all files. The root name is followed by a four-digit integer run number. Raw data from the AAT conforms to this convention with names of the form 13apr0001.fits, 13apr0002.fits etc . Data from the archive also conforms to the convention though the names are changed to run0001.fts, etc.

It is possible to reduce individual files that do not conform to the naming convention. They can be loaded into the system using the Reduce item in the Commands menu. However such files cannot form part of the automatic reduction of a sequence of files, unless they are renamed to conform to the other files in the sequence.

This Setup Automatic Reduction window is asking for the root name which prefixes the data to be reduced.  In this example the prefix is 28jun2 which indicates the software has found data taken on June 28th and for ccd2, the red arm.  If one wants to reduce data from the blue and red arms in the same directory then the root name can be altered to 28jun1 to pick up ccd1 data.

Once Setup has run, the file information will be available in the left *Data* tab section of the main GUI as shown here.

<img src="%ATTACHURLPATH%/file_info.jpg" alt="File info GUI.jpg" width='468' height='358' />

Here we see the the first file, Run 1, is file 28jun20001.fits (which is run 1 for ccd2 from 28th June).  The file  is a Multi-Fibre Fibre Flat Field (MFFFF, please refer to the full manual for an explanation of this insane classification convention).  The file is *Not Reduced* and so there is no *Reduced File*.  We can use the up and down spinner buttons to right of *Run*. or select a file from the *File* chooser to cycle through the current available data to view the other files with the same root name in the local directory.  If we hit the *Plot* button below the file information, we can see the 2D image to check that everything looks okay.  Note that the full CCD is 2kx4k and so many of the displays you will see during reduction are heavily aliased and will often show strange artifacts which are simply not in the data.

<img src="%ATTACHURLPATH%/a_flat.jpg" alt="Reduced flat with tramline overlaid" width='1025' height='693' />

---+++ Find Fibres

For older 2dF data it was neccessary to run the Find Fibres command from the main GUI commands menu.  This allowed the user to asign a fibre identifiaction to the flat field and prevented the extraction loosing fibres later on.  For AAOmega data this step is not required (and infact will not work and will crash the reduction if you try it).  AAOmega has eight well spaced gaps in the data, at the positions of guide fibres bundles on the field plates.  These gaps allow the accurate first order fit to the fibre positions and so the fibres are found automaticaly.

---+++ Reduce the data

The user should now be able to simply hit the Start button, on the upper right of the main GUI, to reduce the data in the current working directory.

The process runs as follows:

   * Reduce any and all multi-fibre flat field frame
   * Reduce any and all arc frame
   * Re-reduce the flat field frames using the accurate wavelength solution to compute a better average illumination correction
   * Reduce any and all science frames
   * Combine the science frames

The final step, the combination of the science frames, will, of course, fail if there is only one science frame.  A Tcl bug means that the pop-up window which informs you of this currently crashes the reduction software.  This will be fix as soon as possible.  Your working directory should now contain a set of *red.fits files which are the reduced data. The combined data will be in the combined_frames.fits file if it was created.

---+++ Note on extraction methods

There are currently three extraction options for AAOmega data, the option can be set under the Extract tab on the main GUI window:

   1. Tram - a simple summation of the data either side of the tram line fit.  This is quick, but a little noisy
   1. Gauss - a weighted summation of the data based on the predicted Gaussian PSF either side of the tram line fit
   1. Fit - a full PSF fitting of the data, which takes into account fibre-to-fibre cross talk fully.

Ultimately Fit extraction will give the best results, although the fitting process is rather slow.  At this time however, the FIT routine has not been correctly implemented and we suggest you use Tram extraction for a quick look reduction and Gauss extraction for science.

---+++ Known problems

See the current [[http://local.aao.gov.au/AAO/2df/manual/UsersManual.doc][bug list]].

Release notes: There are two outstanding issues with the new 2dfdr implementation. Both are related to the use of STARLINK libraries and are currently under review.

1. The software will not run correctly if the path to the executables is to long (greater than 40 characters). The error means that 2dfdr will appear to start okay, but on attempting to plot data the system will report that it cannot find the xwindow to display to.

2. A bug in the STARLINK ndf2fits routine means that the software will not run correctly when using .ndf as the output file type. Until this is corrected it is suggested that .fits is used as the output file type.

Note for users using Linux : You may have problems trying to reduce data on SUN/Solaris disks running 2dfdr under Linux: 2dfdr may simply fail giving an error message like this: "Task drexec2 has crashed while executing an action" This seems to be a problem related to accessing data over NFS between SUN and Linux machines. We don't have a solution for this, but the workaround is to copy the data to a local Linux disk, and reduce from there.

-- Main.RonHeald - 18 Jan 2007
Attachments
Topic attachments
I	Attachment	History	Action	Size	Date	Who	Comment
jpg	2dfdr_gui.jpg	r1	manage	83.6 K	2007-01-11 - 03:53	RonHeald	2dfdr Main GUI
jpg	2dfdr_main_gui.jpg	r1	manage	67.0 K	2007-01-11 - 04:31	RonHeald	2dfdr Main GUI
jpg	a_flat.jpg	r1	manage	64.1 K	2007-01-11 - 03:57	RonHeald	flat
jpg	file_info.jpg	r1	manage	24.9 K	2007-01-11 - 03:56	RonHeald	data tab file info
jpg	setup_gui.jpg	r1	manage	14.2 K	2007-01-11 - 03:55	RonHeald	setup GUI
Topic revision: r4 - 2007-01-18 - RonHeald
Main Web
~~Edit~~
~~Attach~~