Progress report from UKIDSS

Memorandum by the UKIDSS Consortium Survey Scientist, Dr S J Warren

1. Science Verification

At the time of the Oct 2005 Board meeting the first phase of Science
Verification (SV), using the first version of the pipeline, was
complete. The new SV data, i.e. processed with the second version of
the pipeline, were released on 17 Nov, and the results were discussed at
a meeting in Edinburgh on 15 Dec. The second version of the pipeline
clearly resulted in improved sky subtraction. A number of issues were
raised by the second round of SV, of which the main two were the
problems of moon ghosts, and bad channels. These are dealt with
elsewhere in this paper.

On the basis of the SV results a set of quality control (QC)
procedures were implemented for the Early Data Release. The same QC
procedures, with some minor improvements, are being used for Data
Release 1 (DR1), the first large release. The QC procedures are
described later.

2. Early Data Release, 10 Feb 2006

The Early Data Release (EDR) was set for 31 Jan 2006. In the event the
release was delayed for 10 days, and took place on 10 Feb
2006. Announcements were made through PPARC, and on the ESO web front
page. The release included all good data from 05A (May, Jun 2005), and
UDS and DXS data from 05B up to the end of Sep 2005. It was decided to
provide two databases for the EDR. The EDR database contains fields
for each survey where the full complement of filters exists, and is a
subset of the EDR+ database which contains all data that passes the QC
procedures.

The accompanying paper by Dye et al., has been submitted to MN, and
appeared as astro-ph/0603608 on 22 March. The paper provides full
details of the observing and pipeline procedures, data quirks, QC
procedures, areas surveyed, filters, and depths.

The EDR appears to have gone very well. Very few errors have been
found, and people seem happy with the data format and the archive
interface. We almost met our planned release date, and this augurs
well for future releases. The only major issue that has come out of
the EDR is the problem of source detection in the UDS. Briefly, this
arises as a consequence of the 3x3 microstepping, which oversamples
the data. The CASU source detection algorithm does not work well on
such data, and tends to break up large objects into many sources. This
can be traced to the size of a convolution filter. CASU plan to
upgrade the detection software to cope with oversampled data, in time
to complete source detection for the UDS well in advance of the
planned DR1 release data (14 July, see below).

3. Quality control

The QC procedures have proved timeconsuming to implement, and this has
been the source of some concern within the consortium i.e. why does it
take so long after completion of pipelining to public release of the
data? For this reason a brief outline of the QC procedures,
highlighting the main issues, is provided here.

The aim of the QC procedures is to remove corrupt data (where the data
are meaningless e.g. a detector frame is empty, or no catalogue was
generated), bad data (where the data are meaningful but are of such
poor quality that they are valueless e.g. badly trailed data, bright
moon ghosts), and unacceptable data (i.e. do not conform to uniform
survey quality criteria e.g. cuts on sky brightness, ellipticity,
depth). The general rule is to eliminate all corrupt and bad data, and
then apply gentle cuts on most QC parameters (i.e. remove the extreme
tail), with the exception of depth, where a somewhat stricter cut is
applied. This is on the assumption that uniformity of depth is the
primary QC criterion.

The aim is to produce a data set of uniform quality, with minimal
errors. But errors creep into the datatrain at all points, including MSB
preparation, the observations themselves, in the pipeline, and at
archive ingest. We have taken time to track down the sources of errors
when they are discovered, with the aim of implementing procedures
which prevent their reoccurence. In the long term this will mean more
efficient use of the telescope, better quality data, and faster QC
turnaround.

At present about 20% of the data is being rejected. There are
currently four issues that are cause for concern:

i.) Ghosts. A variety of ghost images, principally due to the moon,
have been noted. By making a visual check of all frames, we have a
catalogue of examples, and have been able to investigate the observing
geometry that results in ghosts. The main problem is a bright 'pupil'
ghost that appears to occur when the moon lies within 30deg of the
field, and also abs(delta(RA)) and abs(delta(dec)) are both greater
than 5deg. We have recommended that JAC create a detailed map of the
problem, by undertaking a set of observations placing the moon on a
grid of points, abs(delta(RA)) and abs(delta(dec)).

ii.) Repeated observations. Some frames are repeated if guiding is
lost, but the failed frame is kept, and must be identified and ignored
in the pipeline. Errors creep into this process at various stages. For
example, in some cases the wrong frame was repeated, in a number of
cases the repeat was not logged, and sometimes the wrong frame was
rejected in the pipeline. Nevertheless, where a jitter/microstep
combination has been repeated it is always correct to take the later
observation, so the QC checking now identifies potential problems by
picking out examples where this is not the case.

In addition in several cases a whole or part MSB has been
repeated. Since there is no clear logging of why this has occurred, we
are implementing a procedure which (within the constraints of the
parameters of the surveys) results in the better of two observations
being kept.

iii.) Bad channels. Individual channels can suffer from a random bias
offset. There are two particular channels in array 4 which are
especially variable, but less severe examples are seen in all the
arrays. These offsets are not corrected for in the pipeline. They are
particularly unwelcome in microstepped data, causing a bed-of-nails
structure in the data.

iv.) There is a noticeable correlation with declination of average
stellar ellipticity in a frame i.e. guiding is not optimal in northern
fields.

4. Survey progress

DR1 will include all 05A data (reprocessed with the latest version of
the pipeline) and all 05B data. The date for DR1 has been set at July
14. We will aim to complete the accompanying journal paper as close as
possible to this date. QC is progressing at a rate compatible with
this date. CASU are aiming to complete the source detection software revision
by mid June. We are deferring quantification of survey progress until
after DR1 since we need to know how much data are being rejected by the
QC. At that time we can establish the overall efficiency of the
observations. Nevertheless it is clear that the surveys are
progressing significantly slower than originally anticipated. Two
important effects are the fraction of data being rejected, which is
still about 20%, and the observing overheads. Significant progress is
being made on both issues. Very roughly we have probably been surveying
slower than anticipated by a factor of about 1.5 at the moment, but
this remains to be properly quantified.

5. Registration

So far 60 institutions have registered for access to UKIDSS data, of
which 30 are from outside the UK.

6. Publicity etc.

DR1 will be much larger than 2MASS in terms of photons, and about
equal to 2MASS in terms of volume, meriting a press release. Some ZJK
images of photogenic sources were taken in January as a first step to
building a WFCAM/UKIDSS gallery. Some of these images could be used
for the press release. Not much progress has been made on the idea of
an image gallery due to the workload for DR1, but this will be
restarted after the release.

Several UKIDSS talks were given at the November 2005 workshop on
'Panoramic near-infrared astronomy'.

There were articles in the ESO Messenger and the UKIRT Newsletter
accompanying the EDR.

A full session at NAM was devoted to UKIDSS.

Three UKIDSS baseline papers have appeared as follows:

Lawrence et al., 'The UKIRT Infrared Deep Sky Survey (UKIDSS)', MNRAS
submitted, astro-ph/0604426

Dye et al., 'The UKIRT Infrared Deep Sky Survey Early Data Release',
MNRAS submitted, astro-ph/0603608

Hewett et al., 2006, 'The UKIRT Infrared Deep Sky Survey ZYJHK
photometric system: passbands and synthetic colours', MNRAS, 367, 454