3.2 Existing Pipe-lines

There is currently more data available to astronomers than can be efficiently used. With the increasing complexity of instrumentation and the limited opportunities observers have to use instruments, reduction and analysis tools are the key to getting observations published, indeed reduction and analysis should be part of the instrument design (Bohannan, 2001). With such a backlog of data there is increasing pressure for observatories to supply pipe-line reduction (see Greimel et al., 2001).

Generally inherent in any automated procedures written will be the stage of “human intervention”, where it is far quicker and easier to manually interact with the reduction process than engineer software to carry out the same process, even though this quicker process may take weeks to accomplish. Even those astronomers who have the programming skills necessary to create such a complex package are resistant to do so due to the foreseeable time constraints in such an undertaking, which takes time away from their scientific research.

The model offered for pipe-line processing of the ING Wide Field Survey (WFS) images (Irwin and Lewis, 2001) is a model for all observations obtained in a standardised fashion (Bohannan, 2001). As such the following section will explore and contrast the infrastructure of the ING WFS PL with that of the LT PL.

3.2.1 The ING WFS Pipe-line

The ING WFS is major multi-colour, multi-epoch, CCD based wide field survey over an area of ~100^o2. The PL reduced data is only available after a period of one month (McMahon et al., 2001). The LT data will be fully reduced and available within 24hrs for normal observations with object of opportunity reductions taking place in real-time, but only at the request of the LT project scientist.

The ING WFS PL is based on IRAF routines. The pipe-line is essentially a series of pipes linked together in a modular way to form the PL. This modular way of creating a software package increases the upgrade-ability and robustness of the PL. Any pipe can be rewritten and replaced without the rest of the PL being effected. This modular infrastructure is also the basis that the LT PL has been built around, it allows for much greater flexibility, upgrading the system is also easier when done in this modular fashion.

The camera used on the ING WFS has a mosaic of 5 CCDs with a current readout time of around 160s, which in practical terms leads to a dead-time of around 3mins between exposures. A typical nights data is between 5 and 10Gbytes of data, all of which must be transfered to the Institute of Astronomy, Cambridge, on DAT tape for reduction.

The LT contains only one CCD which produces an 8 Megabyte file per readout, Bode (1995) maintains that there could be several hundred such frames each night giving upwards of 5 Giga-bytes of data. This data will be reduced at the LT site leaving only a series of text files as the fully reduced data which can be efficiently transfered electronically (see McNerney and Steele, 2000).

Although most of the ING WFS processing is automatic, creating updating and maintaining master flats is difficult if not impossible to automate as there is no direct control over the survey, however flat-fields have been found to be stable for periods of around 1 week. (Irwin and Lewis, 2001).

The LT will also maintain a library of flat-field images, however algorithms developed at the Astrophysics Research Institute in Liverpool will allow these calibration frames to be automatically generated and updated.

The data processing system of the ING WFS is heavily dependent on the integrity of the FITS headers. They are also a repository of information derived and generated by the processing stages. We cannot emphasise enough the vital importance of following this information for automated PL processing (Irwin and Lewis, 2001).

This method of extracting information and recording necessary data to file is also used in the LT PL, with the name of each calibration file recorded to the FITS header. It this way it is possible to accurately repeat the reduction of data files at a much later date than it originally occurred.

The ING WFS PL follows these stages:

• De-biasing and trimming: Stacked bias frames show repeatable structure, due to electronic transient effects and slow readout times - therefore a full 2D bias removal is necessary.

  This effect has not, to date, been noticed on the LT bias frames, therefore a simple bias level subtraction will be used, however the capability exists to do full 2D bias removal should it become necessary.

• Non-linearity correction: The CCDs are found to be non-linear and are corrected using look up tables, believed to be caused by the ADCs.

  The LT CCD has been tested and shown to be linear to >> 99% (Pittock, 1998), non-linearity therefore will not be an issue.

• Flat-fielding: A sequence of flat fields is created manually, as it is not possible to take flats in all filters each night due to time constraints.

  Time constraints for creating flat-fields will also affect the LT, however the scheduling algorithms will take into account the number of different filters to be used each night as part of its calculation, it is expected therefore that flats for each necessary filter will be observed each night. However with flat-fields being stable for ~ 1 week, the scheduler will ensure that flat-fields are taken for any filter, for which a flat has not previously been taken, within this time-scale.

• The ING WFS has mosaic of 5 CCDs, as such it is necessary to normalise the gain between them. The LT has only 1 CCD and so this step is unnecessary.
• De-fringing: The ING WFS finds high fringing in I and Z, de-fringing occurs using fixed fringe pattern frames and “in-house” developed algorithms.

  The LT PL will create de-fringing frames for each image that requires it, the de-fringing being carried out using the techniques discussed in Section 2.4.4.

• Astrometry: The WCS of the ING WFS is initially derived by adding the telescope pointing parameters, which is accurate to 5-10 arc-sec, to the data file. A more accurate fit is then attempted by matching stars visible in the frames to those in the guide star catalogue.

  This technique is the same as that used in the LT PL, however the zeroth order WCS gained from the telescope pointing will be accurate to < 2arc-secs. The catalogue used to create a more accurate WCS will be the USNO-A2.0 (see Monet, 1998).

• Photometry: The ING WFS uses Landolt (1992) standards which are observed each night. The zeropoint (see Section 2.5) is derived on a per photometric night per filter basis with results consistent within a few % of the true zeropoint.

  This process will also be used by the LT PL however, whilst initially the LT will use Landolt (1992) standards there will also be a catalogue of LT standards built up with the aim of improving accuracy.

3.2.2 ORAC-DR

Observatory Reduction and Acquisition Control Data Reduction (ORAC-DR) is a flexible and modular pipe-line originally developed by UKIRT for data reduction from IR telescopes (Currie et al., 1999a). ORAC-DR is an extension of a telescope control project which has combined many different facets of work with both UKIRT and JCMT collaborating. It is an infrastructure set-up with the intent of using other people’s software code (see Economou et al., 1999), with a major constituent of its algorithm engine coming from SL (see Economou et al., 1998). This infrastructure is, as has been stated above also evident in both the ING WFS PL and the infrastructure of the LT PL.