AAOmegaFileFormat < Main

Tags: view all tags
---++!! !AAOmega File Format (or, which spectrum is which object?)


When analysing data from an AAOmega run, one needs to map the combined spectra produced by the reduction back to
individual objects from the input catalogue. All the relevant information is contained within the combined output file(s). This page describes how to recover this information. The process is _simple_, once one follows the _logic_. It can seem rather convoluted at first sight. Please send any questions, comments and suggestion you may have on how to make this information more digestible, to [[mailto:rgs@aao.aao.gov.au][Rob Sharp]] or [[mailto:rwh@aao.aao.gov.au][Ron Heald]].

%TOC%


---+++ Reduced File Extensions

This table gives a summary of a AAOmega 2dfdr output file content, for either an individual reduced frame (=*red.fits= files like ='21apr20006red.fits'=) or a combined reduced frame (=combined_frames.fits= files). The file is a standard [[http://archive.stsci.edu/fits/fits_standard][Multi-Extension FITS]] file (FITS MEF). The extensions can be accessed in a number of ways given [[#AccessBinary][below]].


| *Extension* | *IRAF format* | *Contents* |
| Primary | .fits[0] | The primary extension in the FITS file is a !WxN image where W is the number of pixels in each spectrum, and N is the number of spectra represented. This is 400 for AAOmega data, 392 science fibres and 8 guide fibres. Unused science fibres and Sky spectra, are included in the output file along with the guide fibres spectra, even though the spectra contain no information, as this simplifies book keeping, and is a small disc space overhead. In the case where multiple sets of AAOmega datasets, which contained a subset of common objects, have been combined, the format is a little more complex, and is explained separately [[#CombData][below]].|
| Variance | .fits[1] | The variance extension is also a !WxN array identical in size to the primary extension. Each member contains the variance for the corresponding element in the primary extension. |
| Fibre Table | .fits[2] | FITS binary table, with N rows, one for each fibre. Each row contains information for the fibre such as object name, position, etc. Full details are given [[#FibreTable][below]]. |
| Other Extensions |  | The files contain several other extensions which are generally only used when deeper analysis of the data is required. They are not necessarily in order and are accessed by name. They are described [[TwodfdrFileFormat][here]]. |

The files output by 2dfdr also include data at intermediate processing steps. They are not normally used by scientist. They are described [[TwodfdrFileFormat][here]].


#CombData
---+++ Combining Multiple AAOmega Data Sets containing a Common Subset of Targets

Combining of reduced files occurs in 'Auto Reduction' mode when all local object frames have been processed. It also can be done 'manually' using the =Combine Reduced Runs...= item in the 2dfdr =Commands= menu.

The 2dfdr combine algorithm combines data based on either object name or object location. That is, fibres having the same name (or location) are added and normalised to produce the output. This is to include all objects, whether they are contained within every frame or only a sub-set of the frames. The combine has the following features:

   * Multiple configurations of the same field can be combined together when objects are in common. Notice this can result in more spectra than the instrument can produce in one exposure.

   * Only fibre types 'S' (sky) and 'P' (program) fibres are combined. This includes cases in which a fibre has been disabled part way though a field observation, so only good data is combined. Other fibres such as unused and parked fibres have all values set to zero.

   * The first spectra will be all those from the first frame in the combine including unused/parked and sky fibres. Any additional spectra will be only sky and program spectra from objects in subsequent frames and not present in the first frame. If the data combined are all from the same configuration there will be no difference in the fibre count. 

   * All the fibre table extension information is properly propagated. Additional fibres are numbered beginning from the last fibre of the first frame. So for AAOmega, the first 400 fibres will be from the first frame, and fibre 401 and beyond will be additional fibres from subsequent frames (if any).

   * Variances are handled properly.

*N.B.* Currently the combined file exposure time is *NOT* set properly. Exposure time is given in only one place for a file, the value of FITS header keyword 'EXPOSED'. This exposure time applies to all fibres within the file. When files are combined, this keyword is copied from the first frame - no attempt is made at setting it properly. To fix this situation the fibre table structure must be modified to have a new column. This column would give the actual exposure time (in seconds) for each fibre spectrum. See [[AAOSoftwareWeb/Bugs/Item15][Exposure Time Handling]] in the bug list.

---+++ Combining Control

Control of the combining process is done using the widget on the  2dfdr GUI _Combine_ tab.

Sometimes morons write a script to make a configuration file for a large input catalogue but name the objects randomly (with some non-unique running ID number) rather than naming everything in advance. When this happens the combining should be based on matching object location. This is selected with the =COMBNAME= argument list member. This appears as the =Combine Matching Name= widget. Setting it to =true= causes match on name. Setting it =false= causes the combine to be based on match object locations.


#FibreTable
---+++ Fibre Table Columns

| *Column* | *Column name* | *Description* |
| 1 | Name| Object name from the =.fld= file |
| 2 | RA | J2000 Mean Right Ascension (Radians) from the =.fld= file |
| 3 | Dec | J2000 Mean Declination (Radians) from the =.fld= file |
| 4 | X | 2dF field plate X co-ordinate (microns) |
| 5 | Y | 2dF field plate Y co-ordinate (microns) |
| 6 | Xerr | Reported X error in final fibre placement (microns) |
| 7 | Yerr | Reported Y error in final fibre placement (microns) |
| 8 | Theta | Angle of fibre on field plate |
| 9 | Type | Fibre type, F-guide, N-broken, P-program, S-sky, U-unused |
| 10 | [[#PivotNote][Pivot]] | 2dF fibre pivot number |
| 11 | Magnitude | Object magnitude from the =.fld= file |
| 12 | PID | Program Id |
| 13 | Comment | Comment from the  =.fld= file |
| 14 | Retractor | 2dF retractor number |
| 15 | Switch Field | A or B if beam switched data |
| 16 | Switch Partner | Paired fibre if beam switched data (fibre slit position number) |
| 17 | Wlen | ??? |


#PivotNote
---+++ Important Note on 2dF Fibre-Pivot Number and 2dfdr Fibre Number

There are two very important, and very different, numbers needed to recover the information for which object each fibre was allocated: *Fibre slit position* AND *2dF Fibre-Pivot position*. For the most part there is a one-to-one correspondence between these number sets. i.e. the fibre at AAOmega slit position 1 (at the bottom of the CCD image, ...red.fits[0][*,1]) will map directly to 2dF Pivot position 1, and fibre 400 will map to Pivot position 400. (Note 400 is a guide fibre and so maps to a blank space at the top of the CCD image, ...red.fits[0][*,400]). However, during manufacture or repair of the AAOmega slit units, it is possible for the order of fibres in each of the AAOmega slits to fall out of synchronization with the 2dF Pivot numbering (2 slit blocks on each plate are like this as of April 2007). It is not practical to mechanically alter either position so each of the conversion betwen the two is propagated in the binary table extension (Pivot position).

In the primary image .fits[0] (and also the variance array .fits[1]) the fibre at the bottom of the image, which is the fibre at *slit position 1* (...red.fits[0][*,1]), corresponds to the first row in the binary table extension .fits[2]). The table contains a column entry, *PIVOT*, which gives the *2dF pivot position* for this fibre. This is the fibre number seen be the Configure software. The very top fibre in a CCD image corresponds to the very last entry in the binary table (which will be an AAOmega guide fibre in the case of a single AAOmega data set). There is *ALWAYS* a one-to-one correspondence between each spectrum position in the image and the binary table. There is typically a one-to-one correspondence between slit position and *2dF pivot position* but with a number of known mismatches and discontinuities which are tracked via the *PIVOT* column of the binary table.

#AccessBinary
---+++ How to Access the Fibre Table Information

This list is not exhaustive. If your favorite option is missing, send an e-mail to [[mailto:rgs@aao.aao.gov.au][Rob Sharp]] and it will be added.


---++++ With Configure

One can save a list file (file menu -> ..list) which contains the allocated 2dF [[#PivotNote][fibre Pivot number]] for each allocated fibre. Notice this is the Pivot number for 2dF, NOT the fibre number in the reduced 2D spectra file.


---++++ Within 2dfdr
To be added


---++++ 2dfinfo

The =2dfinfo= program comes packaged with =2dfdr=. It can be used to recover information on the fibre from either a .fits or .sdf file. The syntax for the command is :

=2dfinfo file.fits <option>=

If the =option= is omitted then the list of options is given. To recover the fibre table information one would use:

=2dfinfo file.fits fibres=


---++++ IRAF

The <span style IRAF/STSDAS package TABLES has a number of routines designed, unsurprisingly, for manipulating tables. A simple example might be:

=IRAF> tdump combined_frame.fits[2] >; output.txt=

This would create a complete, if rather inelegant, listing of the fibre info binary table and pipe it to a ascii text file. Formating the output can be achieved with:

=IRAF> tprint combined_frame.fits[2] columns="NAME,RA,DEC" > output.txt=


---++++ IDL

For users of IDL, the <a href="http://idlastro.gsfc.nasa.gov/">NASA IDL astronomy library</a> has some excellent fits data access routines.

Starting from a combined fits frame, =combined_frame.fits=, one might use the following code extracts to manipulate AAOmega data.

Notes, there are much cleverer (and quicker) ways to perform the operations below with the NASA astrolib tasks, the code here is given
as a set of simple examples.

<pre>
file='combined_frame.fits'

;; Read in the spectral image, store the header information
spec=mrdfits(dir+file_comb,0,header0)

;; And the variance array
spec_var=mrdfits(dir+file_comb,1)

;; Make a wavelength vector, note the use of CRPIX1, which is often not expected by many users.
;; If missed, the wavelength solution will tend to be wrong by half a CCD width
crpix=fxpar(header0,'crpix1')-1.0 ; The -1.0 is needed as IDL is ZERO indexed 
crval=fxpar(header0,'crval1')
cdelt=fxpar(header0,'cdelt1')
wave=((findgen(n_elements(spec[*,0]))-crpix)*cdelt)+crval

;; Read in the object identification information
fxbopen,unit,file,2
fxbreadm,unit $
  ,['name','ra','dec','x','y','xerr','yerr','theta','type','pivot','magnitude'] $
  ,id,ra,dec,x,y,xerr,yerr,theta,type,pivot,mag
fxbclose,unit

;; And read a copy of the sky spectrum subtracted from the data.
;; Note, for a combined frame, this is the sky spectrum from the first file in the list of combined frames.
;; It is a good representative sky spectrum, but should be used with caution for the combined spectral data.
fxbopen,unit,file,7
fxbreadm,unit,['SKY'],sky
fxbclose,unit
</pre>


-- Main.RonHeald - 023 Oct 2009
Topic revision: r18 - 2010-02-26 - RobSharp
Main Web