The Imaging Pipeline
This page describes the SDSS imaging pipeline, which processes the raw telescope imaging to produce calibrated FITS files and catalogs of imaging parameters. Each step in the pipeline is outlined below.
More detail is available in the Algorithms section of this site. The Early Data Release (EDR) paper (Stoughton et al. 2002) is the fundamental resource for understanding the processing and data products from the SDSS, describing the pipelines and contents of generated data products. Successive data release papers describe changes between data releases - see SDSS Data Release Publications for a complete list, with links. The Data Release 9 paper (Ahn et al. 2012) describes some changes to astrometry algorithms that occurred with DR9 and later. The Data Release 13 paper (Albareti et al. 2017) describes changes to the definition of primary and the photometric calibration.
In addition to these data release papers, the SDSS technical summary provides more general information on the survey. The SDSS Project Book, written for a funding proposal early in the survey, is the most exhaustive description of the survey but was last updated in 1997. The SAS datamodel has a table of the most commonly useful files in the SAS. Truly in-depth details can be found on Robert Lupton's page of photo documents, including the draft photometric reduction paper as well as a treasure trove of photometric data reduction wisdom.
The remainder of this page starts with a brief overview of imaging data processing, followed by sections that describe the steps in data processing in detail. In addition to the more detailed description, each detail section provides references to papers that give additional details, and a table of the files associated with that step of the pipeline that can be found in the SAS. These tables include links to descriptions of the formats of those files and templates that can be used to generate SAS URLs for those files. The templates are in C printf() format, and can be used in C, bash, Python, and many other languages to automatically generate URLs.
Most of the catalog data (but not the images themselves) have been loaded into the Catalog Archive Server (CAS) database, which is accessible through SkyServer and CasJobs. Users are often better off obtaining SDSS data through a carefully constructed CAS query than they are downloading the data files from the SAS. Simple queries can be used to select just the objects and parameters of interest, while more complex queries can be used to do complex calculations on many objects, thereby avoiding the need to download the data on them at all.
Overview: Steps of the Imaging Pipeline
- Imaging Observing
- We used three instruments when collecting imaging data: the imaging camera mounted on the primary 2.5m SDSS telescope, which collected the imaging data themselves; the 0.5m photometric telescope, which collected images of photometric standard stars and reference fields; and a 10 micron all sky scanner
- Imaging Data Reduction
- The SDSS data processing factory used a collection of pipelines to process and calibrate the data from the imaging camera, ultimately producing a variety of data products including images with instrumental signatures removed, a photometric solution for the night, and a catalog of objects found in the data.
- Resolution of Multiple Detections
- The resolve step declares one observation on an object the primary observation, and others secondary, thereby avoiding unintentional duplication of objects. This step ultimately produces an astrometrically and photometrically calibrated catalog of objects found in the data from the imaging camera.
- Recalibration
- Improved photometric calibrations became possible at the end of the survey; we used a separate pipeline to refine the photometric calibrations.
- Final Photometric Catalog
- The results of the resolve and calibration stages are combined with the uncalibrated object catalogs to produce final catalogs.
Important Notes
- In all of the tables below, add 'https://data.sdss.org/sas/dr16/' to all 'URL format' values to get the full URL.
- Data reduction could occur multiple times for both images and spectra. Each time we repeated a data reduction, we labeled the output from that reduction process with a distinct rerun number.
- In the case of imaging rerun numbers, different decades in the rerun number designate significant differences in the data reduction software. For example, rerun 1 and rerun 2 of a given run would have been processed by identical (or at least very similar) versions of the data reduction software, while rerun 40 would have been reduced with a significantly different version of the pipeline. However, the only relevant reruns for DR8 and later are 157 and 301, with the bulk of the data in rerun 301.
Imaging Observing
Imager
The imager collected survey data from the imager in drift scan (also called TDI) mode. We use the term 'run' to designate a single scan along a great circle. The data acquisition system divided the data from each CCD into frames, stored as FITS files; the observatory sent data to be processed as a collection of FITS files, each corresponding to an arc along the great circle 1361 rows (539 arcseconds) long from a single CCD. In addition to the images themselves, the data acquisition system also generated a variety of metadata and other engineering files, which included pixel statistics and a catalog of bright stars for each field.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| idReport | out | records runs collected on a night | eboss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| idR | out | raw imaging frames | eboss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z | run, camcol, run, filter, camcol, field |
| idGang | out | assorted data gathered by the data acquisition system | eboss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit | run, run, r(ow) or c(ol), row number or column number, frame |
| idFrameLog | out | camera position information for each frame | eboss/photo/redux/%d/%d/logs/idFrameLog-%06d-%d.par | rerun, run, run, crate |
| opCamera | out | imaging camera geometry | eboss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | out | CCD readout configurations | eboss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | out | CCD electronic calibrations | eboss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
Infrared All-Sky Camera
Clouds can be clearly seen at night in a 10 micron all-sky camera. We used two such cameras at different times in the survey to monitor the sky for cloud (and therefore non-photometric conditions).
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| cR | out | A single image from the infrared camera | apo/ircam/%05d/cR%06d.fit | mjd, UTC time (HHMMSS) |
| irsc | out | A log of statistics on ircam frames | apo/ircam/%05d/irsc.log | mjd |
Other metadata
The observatory produces an assortment of engineering and other metadata.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| night log | out | A prose account of the night | apo/astrolog/%05d/manualLog-full.txt | mjd |
| idWeather | out | A log of weather data | eboss/photo/redux/%d/%d/logs/idWeather-%05d.par | rerun, run, mjd |
Imaging Data Reduction
Serial Stamp Collecting Pipeline (SSC)
The SSC repackages some of the data produced by the data acquisition system and produces postage stamps of bright reference stars.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| scPlan | in | the SSC processing plan | eboss/photo/redux/%d/%d/ssc/scPlan.par | run, rerun |
| idReport | in | records runs collected on a night | eboss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| opCamera | in | imaging camera geometry | eboss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | in | CCD readout configurations | eboss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | in | CCD electronic calibrations | eboss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
| opBC | in | CCD bad pixel file | eboss/photo/redux/%d/%d/log/opBC-%d.par | rerun, run, mjdbp (see fpPlan) |
| idR | in | raw imaging frames | eboss/photo/data/%d/fields/%d/idR-%06d-%s%d-%04d.fit.Z | run, camcol, run, filter, camcol, field |
| idGang | in | assorted data gathered by the data acquisition system | eboss/photo/data/%d/gangs/idGang-%06d-%s%d-%04d.fit | run, run, r(ow) or c(ol), row number or column number, frame |
| koCat | in | the catalog of known bright stars | eboss/photo/redux/%d/%d/ssc/koCat-%06d.fit | run, rerun, run |
Postage Stamp Pipeline (PSP)
References: Stoughton et al. (2002), section 4.3.
The PSP measures the bias and global sky level, and fits a model for the point spread function for each field.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| psParam | in | PSP tunable parameters | eboss/photo/redux/%d/%d/photo/psParam.par | run, rerun |
| psPlan | in | the PSP processing plan | eboss/photo/redux/%d/%d/photo/psPlan.par | run, rerun |
| opCamera | in | imaging camera geometry | eboss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | in | CCD readout configurations | eboss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | in | CCD electronic calibrations | eboss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
| opBC | in | CCD bad pixel file | eboss/photo/redux/%d/%d/log/opBC-%d.par | rerun, run, mjdbp (see fpPlan) |
| idB | in | bias vector | eboss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit | rerun, run, calibration run, filter, camcol |
| psField | out | the initial photometric calibration and point spread function fit by field (see the extractingPSF images page) | eboss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
Astrometric Pipeline (astrom)
References: Pier et al. (2003).
The astrometric pipeline calculates the astrometric solution. The astrometry page provides more information on the algorithms used and the proper interpretation of the output.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| asParam | in | astrom tunable parameters | eboss/photo/redux/%d/%d/astrom/asParam.par | rerun, run |
| asPlan | in | the astrom processing plan | eboss/photo/redux/%d/%d/astrom/asPlan.par | rerun, run |
| opCamera | in | imaging camera geometry | eboss/photo/redux/%d/%d/logs/opCamera-%05d.par | rerun, run, mjd (of generation) |
| opConfig | in | CCD readout configurations | eboss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| idReport | in | records runs collected on a night | eboss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| idWeather | in | A log of weather data | eboss/photo/redux/%d/%d/logs/idWeather-%05d.par | rerun, run, mjd |
| asTrans | out | transformation coefficients from row, column to great circle coordinates | eboss/photo/redux/%d/%d/astrom/asTrans-%06d.fit | rerun, run, run |
Frames Pipeline (frames)
References: Stoughton et al. (2002), section 4.4.
The frames pipeline applies flat field and bias corrections to each frame, and interpolates values for pixels in bad columns and bleed trails and those corrupted by cosmic rays. In previous data releases, this pipeline would produce a 'corrected frame' or fpC file. These files are no longer produced. Instead, a frame file is produced by a different method, though it contains substantially the same information. The frames pipeline described here also detects bright objects and estimates the varying sky background, detects fainter objects, and measures a variety of properties for each object. Several algorithms pages, linked below, describe details of the processing performed by the frames pipeline.
- Measures of flux and magnitudes describes the variety of methods the frames pipeline uses to measure the magnitude of each object, including radial profiles, fiber, model, cModel, Petrosian and PSF magnitudes and when to use each.
- Classification and Morphology describes other measurements of objects, including star/galaxy separation, model fits, surface brightnesses, ellipticities, adaptive moments and isophotal quantities.
- IMPORTANT! Photometric processing flags describes the variety of flags that the frames pipeline sets for each object, including quality flags. Understanding these flags is essential for properly interpreting the catalog.
- Flatfield creation and quality — See the Ubercal paper (Padmanabhan et al. 2008) for a more detailed discussion of flat-fielding.
- Sky measurement
- Deblending describes how the frames pipeline decides if an initial single detection is in fact a blend of multiple overlapping objects, and, if so, how it separates (or 'deblends') them.
- Creation of imaging masks
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| fpParam | in | frames tunable parameters | eboss/photo/redux/%d/%d/photo/fpParam.par | rerun, run |
| fpPlan | in | frames processing plan | eboss/photo/redux/%d/%d/photo/fpPlan.par | rerun, run |
| idReport | in | records runs collected on a night | eboss/photo/redux/%d/%d/logs/idReport-%05d.par | rerun, run, mjd |
| koCat | in | the catalog of known bright stars | eboss/photo/redux/%d/%d/ssc/koCat-%06d.fit | run, rerun, run |
| opConfig | in | CCD readout configurations | eboss/photo/redux/%d/%d/logs/opConfig-%05d.par | rerun, run, mjd (of generation) |
| opECalib | in | CCD electronic calibrations | eboss/photo/redux/%d/%d/logs/opECalib-%05d.par | rerun, run, mjd (of generation) |
| opBC | in | CCD bad pixel file | eboss/photo/redux/%d/%d/log/opBC-%d.par | rerun, run, mjdbp (see fpPlan) |
| idB | in | bias vector | eboss/photo/redux/%d/%d/photo/calib/idB-%06d-%c%d.fit | rerun, run, calibration run, filter, camcol |
| psField | in | the initial photometric calibration and point spread function fit by field (see the extractingPSF images page) | eboss/photo/redux/%d/%d/objcs/%d/psField-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
| asTrans | in | tranformation coefficients from row, column to great circle coordinates | eboss/photo/redux/%d/%d/astrom/asTrans-%06d.fit | rerun, run, run |
| fpFieldStat | out | field statistics | eboss/photo/redux/%d/%d/objcs/%d/fpFieldStat-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
| fpObjc | out | the (uncalibrated) object catalog | eboss/photo/redux/%d/%d/objcs/%d/fpObjc-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
| fpBIN | out | a 4x4 binned version of the corrected image after masking of objects and subtraction of sky, an image of the estimate of the sky, an image of the uncertainty in the sky estimate, and a table of bright star wings | eboss/photo/redux/%d/%d/objcs/%d/fpBIN-%06d-%c%d-%04d.fit.gz | rerun, run, camcol, run, filter, camcol, field |
| fpM | out | the frame masks (read using readAtlasImages) | eboss/photo/redux/%d/%d/objcs/%d/fpM-%06d-%c%d-%04d.fit.gz | rerun, run, camcol, run, filter, camcol, field |
| fpAtlas | out | the atlas images for all objects detected (read using readAtlasImages) | eboss/photo/redux/%d/%d/objcs/%d/fpAtlas-%06d-%d-%04d.fit | rerun, run, camcol, run, camcol, field |
Resolve Multiple Detections
A given area on the sky may be observed by multiple runs. The final calibrated catalogs contain a field declaring whether the specific observation of the object is 'primary', and the canonical measurement of the object, or 'secondary', a duplicate. For EDR data through DR7 data, the EDR paper, section 4.7 details on how objects are resolved. From DR8 onwards, a new, more general resolve process accounts for the full set of SDSS runs. The DR8 through DR12 data all use exactly the same resolve; DR13 uses the same overall procedure, but has implemented new decisions about the 'best' field to use in each location on the sky, with no further changes as of DR14.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| reObjGlobal | out | Global resolve status for every object in a field | eboss/resolve/%s/%d/%d/resolve/%d/reObjGlobal-%06d-%d-%04d.fit | resolve rerun, rerun, run, camcol, run, camcol, field |
| thingList | out | The full list of catalog entries for the resolved survey | eboss/resolve/%s/thingList.fits | resolve rerun |
| thingIndex | out | The full list of unique primary objects, gives position of an object in the thingList | eboss/resolve/%s/thingIndex.fits | resolve rerun |
| window_flist | out | The full list of fields used to determine the window function | eboss/resolve/%s/window_flist.fits | resolve rerun |
| window_unified | out | The full list of polygons determining the primary area of the window function | eboss/resolve/%s/window_unified.fits | resolve rerun |
| window_blist | out | The full list of balkans determining the primary area of the window function | eboss/resolve/%s/window_blist.fits | resolve rerun |
| window_bcaps | out | The full list of caps determining the primary area of the window function | eboss/resolve/%s/window_bcaps.fits | resolve rerun |
| window_bindx | out | Matches balkans to fields in the window function, sorted by balkans | eboss/resolve/%s/window_bindx.fits | resolve rerun |
| window_findx | out | Matches balkans to fields in the window function, sorted by sorted by fields | eboss/resolve/%s/window_findx.fits | resolve rerun |
Photometric Recalibration (Ubercal and Hypercal)
From DR7 through DR12, the "ubercal" recalibration recalibrated the survey by simultaneously solving for the calibration parameters and relative stellar fluxes using overlapping observations within SDSS. The algorithm decouples the problem of relative calibrations from that of absolute calibrations; the absolute calibration is reduced to determining a few numbers for the entire survey.
As of DR13, all of the g, r, i, z catalogs (but not frame images) were recalibrated to Pan-STARRS using a procedure known as hypercal ('hyper' is used to denote that it uses information outside SDSS data). For the u band, the flat field was updated but the zero point was not. The calibration files below retain the same format as for previous data releases. See the photometric calibration algorithms page for more information.
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| calibPhotomGlobal | out | Ubercal results (nanomaggies/count) and associated flat fields | eboss/calib/%s/%d/%d/nfcalib/calibPhotomGlobal-%06d-%d.fits | calibration rerun, rerun, run, run, camcol |
| calib | out | QA plots for calibration. | boss/calib/%s/plots/calib-%06d-%c-%s.png | calibration rerun, run, filter, type (flat, hist, run) |
Photometric Catalogs
| File Type | in/out | Description | URL format | format parameters |
|---|---|---|---|---|
| photoObj | out | Full, calibrated outputs of the imager photometric pipeline | eboss/photoObj/%d/%d/%d/photoObj-%06d-%d-%04d.fits | rerun, run, camcol, run, camcol, field |
| photoField | out | Summary outputs of the properties of every field | eboss/photoObj/%d/%d/photoField-%06d-%d.fits | rerun, run, run, camcol |
| photoRunAll | out | Summary information of the properties of every photometric run | eboss/photoObj/photoRunAll-%s.fits | release (dr12) |
References
- Ahn, C. P., et al. 2012, ApJ, 203, 21
- Albareti et al., 2017, ApJ, 233, 2
- Finkbeiner, D. P., et al. 2015, ApJ, 822, 66
- Padmanabhan, N., et al. 2008, ApJ, 674, 1217
- Pier, J. R., et al. 2003, AJ, 125, 1559
- Stoughton et al., 2002, AJ, 123, 485
