4.4 Problems with the Prototype PL

The following section critically evaluates the prototype PL, it seeks to find those aspects of the prototype which will be essential to include in the production PL, those which will need further work and to eliminate those which are surplus to requirements.

4.4.1 Pipe 1: Pros and Cons

Changing the format of the data from FITS to ndf  meant that the prototype PL could be written quickly and efficiently using existing programs. However the conversion takes time (~ 1.5secs per 1024×1024 pixel image) to perform, which is doubled if the data is converted back to FITS.

The re-conversion (FITS  ndf ) is possible, however additional information is imparted into the file making it a non-standard FITS format and therefore under NOST (1999) not actually FITS. This additional material is a quality variance array, essentially an error on the value in the image array. This data impedes some programs written to use the FITS from working.

There are also international problems in using ndf  files. While SL (British) uses an ndf  format its American counterpart IRAF uses its own image format - the IRAF format².

A FITS file may be an “INTEGER” or “FLOAT” (see Section 5.2 for a summary), in terms of physical size the FLOAT file is larger, (at least double). SL reads in a FITS file as a FLOAT, regardless of the initial level of the file. This means that any output file is also at FLOAT level, this is undesirable as it will double the amount of storage required.

While the prototype PL uses ndf  files, the production PL will eliminate this step by operating directly on the FITS image.

4.4.2 Pipe 2: Pros and Cons

The conversion of the filename from the telescope written name, a name which betrays nothing of the contents of the file (e.g., flat, bias, run, etc.), to one which is useful is a removable step. This is achieved by simply imparting useful information in the file name at the telescope. The LT filename system has been defined (Howells et al., 1999) such that each file produced by the telescope is uniquely named. The filename construction is: I_T_yyyymmdd_n_m_p_s.e, where:

• I = Instrument code (c=RATCam CCD Camera, n=Nu-View II Spectorgraph, m=MES Spectrograph, s=SupIRCam IR Camera etc., however only RATCam files will be reduced by the PL).
• T = Exposure code (e=exposure, b=bias, f=skyflat, w=lampflat, a=arc-flat, d=dark, s=photometric standard).
• yyyymmdd = Date at start of night (i.e., change-over at 12-noon).
• n = multrun number (=1 at start of night).
• m = run number (=1 at start of each multrun).
• p = pipe-line processing flag.
  • 0 will identify a raw-data-file, a file than has not been successfully processed by the PL.
  • 1 will identify a data-file that has been successfully processed by PL.
• e = file extension (.fits, .dat etc.)

In doing so it is no longer necessary to spend time interrogating the FITS file, for the data which is now contained in the filename and is therefore quicker to access.

4.4.3 Pipe 3: Pros and Cons

The software used to write this pipe has a useful feature, the image file is checked to see if it has been de-biased before flat-fielding is attempted. This is achieved by writing a new data line to the image file containing the name of the function performed on the image and the calibration file used. This feature will be written into the production PL.

There is however a fundamental problem with combining two files, at least one of the files may be corrupt. To counter this the production PL will have algorithms to detect problems with the files. These will include (i) ensuring the image file rigorously follows the FITS file standard using the FITS checksum bits. (ii) Checking for saturation - where pixels have reached their maximum data value, (iii) removal and flagging of any file that appears to be rogue.

While the image file is appended with the calibration files used in its reduction, the fact that it has been normalised to 1 second is not, therefore without prior knowledge of this fact incorrect reduction may be performed. To overcome this all reduction carried out by the production PL will be recorded to the header.

4.4.4 Pipe 4: Pros and Cons

The LT PL will produce FITS files with data values conforming to the 32-bit twos-complement format (see Section 5.2). The PL will be written to use these values directly with no need for reduction in data size.

4.4.5 Pipe 5: Pros and Cons

An object must be constructed of at least 8 pixels, each with a intensity greater than 3 above the calculated background. While this simple “algorithm” works for non-crowded stellar fields there is a problem when trying to identify, for example, objects in the spiral arms of a galaxy, where the spiral arm is also above the residual background, or in crowded fields where the background level is raised and faint objects are no longer detected.

To counter this a “local” background calculation is planned. The frame will be subdivided into a grid and the background of each grid square calculated, all operations within a grid square will use the associated grid background level.

4.4.6 Pipe 6: Pros and Cons

The co-ordinates of an object on a CCD frame are returned in a cartesian (x,y) system. In terms of identifying any particular object on the sky the co-ordinates are senseless. The test data lacks a world Co-ordinate System (WCS, see Section 5.5), briefly, this specifies the co-ordinate system of the celestial sphere and can be imparted into the FITS file directly, identifying the location of objects within. The co-ordinate system used to identify an object on the sky is Right Ascension (RA) and Declination (Dec). The LT will also produce FITS files without a WCS and it will be an additional job of the PL to “map” a WCS to the image.

The production PL will then be able implement a system to “map” between x,y and RA and DEC, making obsolete the need to use finding charts and automating one of the most time consuming pipes of the prototype. This process will involve pattern matching the objects on the CCD frame and a stellar catalogue which contains RA and Dec.

When a telescope is pointed at an object it will generally be in the middle of the field of view, however, over time the JKT’s pointing (its ability to aim at the centre of the target) has become degraded. The target object of a JKT observation can therefore be anywhere on the image and as such even manual location is time consuming.

4.4.7 Pipes 7 & 10: Pros and Cons

Since different work has a different standard filter system, which will not be known to the PL, and all observations performed with the LT will be with the LT filter set, (see Table 2.2) which will not change, all data reduction performed by the PL will be carried out in the LT system. During commissioning of the LT colour transform equations, between the LT filter system and a range of standard filter systems, will be determined and made available to the astronomical community.

4.4.8 Pipe 8: Pros and Cons

The automation of a line fitting algorithm is a simplistic task, the scientific removal of data points that are incorrect is not. Neither the prototype nor the production PLs have access to weather information and so are unable to make a decision on that basis. The prototype relies on manual intervention to evaluate which points are incorrect, this is a slow process. The process of re-fitting a best fit line is also time consuming.

The production PL will be able to search the HDU of the FITS files for a standard comment about vignetting or any number of problems (e.g., camera shutter staying closed) that can be located, however this standard comment is not available to the prototype. If a complex system problem occurs at the LT which is not recorded to the observation file then it will not always be possible to locate a problem based solely upon information in the file. A method to find a more reliable way of removing points needs to be investigated before the production PL can be written.

4.4.9 Pipe 9: Pros and Cons

The calculation of the final magnitudes in the production PL will be performed in the same way as the prototype. An improvement to the production PL will be to make the airmass correction variables (i.e., intercept, gradient and errors) dynamic, meaning that they will be automatically generated from the data; the prototype must have the variables hard coded in to the program, meaning it must be re-set for each new set of data.

4.4.10 Prototype Pipe-line Overview

Generally the PL will need to reduce one night’s observations in the same amount of time it took to produce the data. If conditions are good one night’s observing can last 12 hours. The PL will therefore have a maximum of 12 hours to reduce the data, or the average observation time of the night, a time which will be dependent on object and science to be performed.

The prototype has too much manual interaction to be considered a truly automated system, and is as close to manually reducing the data that is possible without repeatedly inputing the same command numerous times.

The programs used to construct the prototype output a considerable amount of information to the screen, a process which slows the PL.

The prototype PL showed that a major source of wasted time is disc access, each time a file or information from a file is need the disc needs to be activated, this takes a length of time equivalent to the actual reduction. This is a problem that can however be overcome, by allocating an area of memory that the FITS file can be held in throughout its processing. This will drastically reduce disc usage and hence run-time, and the only disc access needed will be to write files that are to be retained.