KiDS survey processing

Within the KiDS team two data processing chains are used. The Astro-WISE pipeline is used as the main tool for data processing, data management, and monitoring of the telescope and instrument. This pipeline is also used to produce the public survey data products.
Secondly, a lensing-optimized processing chain consisting of the THELI data reduction pipeline and the LensFit shape measurement code are used for specific weak lensing analyses.

Astro-WISE

KiDS uses Astro-WISE as its main tool for data processing and survey data management, and all public survey data products are being produced using this system. The name Astro-WISE stands for Astronomical Wide-field Imaging System for Europe. Astro-WISE is an environment consisting of hardware and software federated over several institutions in Europe, and was originally specifically developed to process data from the OmegaCAM wide-field camera on he VST. An all-in-one system, Astro-WISE allows a scientist to archive raw data, calibrate data, perform data analyis and archive all results in one environment. The system architecture links together all steps in the data processing chain, based on an underlying data model, see Fig. 1.

Figure 1: Astro-WISE data model. Each box represents a class of survey objects. Survey objects not only contain the survey products denoted by familiar names in wide-field imaging, but also carry the information on how they can be created out of other survey objects, as illustrated by the arrows. The underlying data model captures the relationship between requested information and the physics of the atmosphere-to-detector observational system.

The federated architecture of Astro-WISE enables team members located in different countries to easily work together on the same data set and share their results instantaneously. Using concepts of 'projects' (of which users are members and to which data objects belong) and 'privileges' (to control which users can see which data objects) data management is integrated into the system. For example, a KiDS team member can experiment with data by creating new data products with privileges level 1 (the 'MyDB' level), where they are only visible to him/her. To share the data products with other team members, the objects can be 'published' to privileges level 2, where all project members have access to it. Final survey products can be published to privileges levels higher than 2 by the KiDS project managers, where they are visible to increasingly larger audiences, up to availability in the international virtual observatory.

Users can interact with the Astro-WISE system via a command-line interface, which also allows them to write their own scripts and methods or make changes in the code of their own check-out of the software. This makes it possible for all users to help improve the system and collaboratively work on new versions of the pipeline. Apart from the command-line interface there are also several web-interfaces available: for example a database viewer, a calibration data management system and a data processing interface to produce advanced data products in one go. This system allows users in any location to have an up-to-date overview of all available data, and to use hardware in any location to work on survey production.

For more information on Astro-WISE you can visit the Astro-WISE website. To learn more about the use of Astro-WISE for the production of the KiDS survey you can read the paper The Kilo-Degree Survey that will appear in the special edition of Experimental Astronomy on Astro-WISE.

THELI / Lensfit

The cosmic shear application of KiDS requires highly specialized analysis methods. To facilitate this the team is also engaged in a dedicated, lensing-optimized, data reducation and processing procedure of the data using the THELI pipeline and Lensfit shear measurement software that was also used for the weak lensing analysis of the CFHT Legacy Survey (CFHTLenS).