The data of the majority of these surveys is fully public: any astronomer is entitled to a copy of the data. The really novel aspect of this new paradigm is the long-term preservation of the raw data and the ability of re-calibrating it to the requirements of new science cases.
PERFORM ASTROMETRY WITH PYTHON STACK DITHERED FULL
The full quality control mechanisms are treated in complete detail in the Astro-WISE Quality Control paper. It requires an environment in which all non-manual qualification is automated and the scientist can graphically inspect where needed by easily going back and forth through the data (the pixels) and metadata (everything else) of the whole processing chain for large numbers of data products. Quality control is typically one of the largest challenges in the chain from raw data of the “sensor networks” to scientific papers.
Several large surveys plan to use the Astro-WISE information system to manage their data: the 1,500 deg 2 KIDS Survey, Footnote 3 the Vesuvio Survey Footnote 4 of nearby superclusters, the OmegaWhite Footnote 5 white dwarf binary survey and the OmegaTrans Footnote 6 search for transiting variables. Footnote 2 Hundreds of terabytes of data will start entering the system when ESO’s OmegaCAM camera starts operations in Chile in late-2011. The data rate of existing surveys is rapidly approaching terabytes per night, leading to survey volumes well into the petabyte regime and the new surveys will add many tens of petabytes to this. Many more are in progress or coming up with increasing spatial resolution, depth, and survey areas (OmegaCAM on VST, VIRCAM on VISTA, Pan-STARRS, LSST, etc.). Footnote 1 In recent years surveys have been performed which cover hundreds or thousands of square degrees up to the whole sky (SDSS, 2MASS, CFHTLS, etc.). Historical surveys have been digitized (POSS and its southern counterpart) or are in the process of being digitized. The rapid increase in the number of astronomical data sets and even faster increase of overall data volume demands a new paradigm for the scientific exploitation of optical and near-infrared imaging surveys. The advantages of this system for very large datasets are in the areas of: survey operations management, quality control, calibration analyses, and massive processing. The true integration enables a complete data processing cycle from the raw data up to the publication of science-ready catalogs. AWE is characterized by integration of archiving, data calibration, post-calibration analysis, and archiving of raw, intermediate, and final data products.
This information system is an environment of hardware resources and humanpower distributed over Europe. This paper describes the pipeline processing of optical wide-field astronomical data from the WFI ( ) and OmegaCAM ( ) instruments using the Astro-WISE information system (the Astro-WISE Environment or simply AWE).
It is a true integration thanks to complete linking of data lineage from the final catalogs back to the raw data. The system is characterized by integration of archiving, calibration, and post-calibration analysis of data from raw, through intermediate, to final data products.
We have designed and implemented a novel way to process wide-field astronomical data within a distributed environment of hardware resources and humanpower.
0 kommentar(er)
