Imaging Data Caveats

In preparation for eBOSS targeting, several significant changes were made to the DR13 catalogs, including the calibration, the resolution of duplicate detections of the same sources, and the recommended extinction coefficients:

• The calibration has been now tied to Pan-STARRS 1 (PS1), as described in Finkbeiner et al. (2015). This procedure has led to new flat fields and zero points in the g, r, i, and z bands, and new flat fields (but not new zero points) in the u band. The changes affect essentially all calibrated photometric quantities in the catalog, at around the percent level.
• Due to the new calibration, users should determine that the data is photometrically well calibrated by checking the PS1_UNPHOT bit of the CALIB_STATUS, not the PHOTOMETRIC bit as previous.
• The corrected frames (described here), however, have NOT had these calibration changes applied in DR13.
• Due in part to the new calibration, new decisions were made in the resolve process regarding which detections of sources are considered "primary", for those sources detected more than once. The resolve algorithm itself was not changed. These differences will cause the parameters MODE, RESOLVE_STATUS, and THING_ID to change for some objects.
• The recommended extinction coefficients (extinction in each band relative to E(B-V)) has changed to that recommended by Schlafly et al. (2011).

This caveat is appropriate for users of the DR8 catalog, but irrelevant for DR9 and subsequent releases. The DR9/DR10 astrometry and proper motions are superior to both DR8 and DR7.

Because of several errors, the absolute astrometry in DR8 was degraded relative to that in DR7, and degraded substantially northward of a declination of around 40 deg. Note that the proper motions are considerably less degraded. See the full details on the DR8 astrometry algorithms page. Systematic errors introduced in DR8 southward of 40 deg declination are typically smaller than or comparable to the 45 mas systematic errors that characterize the SDSS astrometry for brighter stars. However, northward of 40 deg there is a systematic offset of around 250 mas in the declination direction.

For very precise astrometry it is necessary to use the DR9 version of the catalog or, if it is necessary to use DR8, two special CAS tables included in that release: astromDR9 and properMotionsDR9, described on the
DR8 astrometry algorithms page. These tables contain the values for positions and proper motions that are distributed in the DR9 release. (Note that there are some much smaller differences between DR8 and DR9 because of astrometry, in the sizes and position angles of objects for example; these are tiny and are not included in these tables).

Differences between the true and measured sizes and magnitudes of large galaxies, based on simulated images. The measured sizes are based on the better of the exponential and de Vaucouleurs models. The measured magnitudes are the cmodel magnitudes. The galaxies are measured by photo to be smaller and less bright than they truly are.

A number of investigators have shown that the sky subtraction algorithm used by the photometric pipeline causes it to systematically underestimate the brightness of large galaxies (Blanton et al. 2005, Lauer et al. 2007, Bernardi et al. 2007, and Lisker et al. 2007, among others). We quantified this by adding 1300 fake galaxies at random positions to SDSS imaging frames, reducing them with both the old (DR7) and new (DR8) versions of photo, and comparing the results with the truth. The galaxies, which have Sérsic radial profiles with a range of inclinations and Sérsic indices, follow the observed correlation between apparent magnitude and angular size seen for real galaxies. However, we biased the sample somewhat to larger and brighter objects, as this is the regime in which the sky subtraction errors are likely to be worst; in addition, the sample is approximately size-limited at r₅₀ ~ 5 arcsec.

Background galaxy counts around bright galaxies, for several different magnitude bins, in both DR7 and DR8. Except in the case of the brightest foreground galaxies, the DR8 counts are always closer to flat with distance (which is the desired result).

The results are shown in the adjacent figure, in which we plot the difference between measured and true magnitude in the r band for the simulated galaxies for the DR7 (red) and DR8 (blue) versions of the pipeline. Results in the other bands are similar. The new sky subtraction algorithm helps, but is not a panacea. The principal trend is with galaxy size, because that couples most directly to the sky measurement. The bias at the largest sizes (of well over a magnitude) is reduced in the new code by only about 0.25 magnitudes. The improvement as a function of half-light radius is also subtle at best, and is visible only for galaxies with r₅₀ > 30 arcsec. Some of the problem may be due not to sky subtraction, but rather to the deblender systematically assigning some of the light in the outer parts of galaxies to superposed fainter stars and galaxies.

A related problem reported by Mandelbaum et al. (2005) was an observed suppression in the number density of faint galaxies around bright galaxies, due to the sky misestimation caused by the latter. We have reinvestigated this problem for the new photometric pipeline. The results are shown in the adjacent figure, the number counts of faint galaxies around bright galaxies as a function of bright galaxy magnitude. In three of the panels, all potential background galaxies are included, and both DR7 (black) and DR8 (red) results are shown. In the bottom panel, faint galaxies with a possible physical association with the bright galaxies are excluded, and only DR8 is shown.

In all cases, the background galaxy counts are perturbed by the presence of the foreground galaxies; in particular, as the bottom right panel shows, the effect seems to be a net removal of sources. For DR8, the faint galaxy counts are substantially less affected by the bright galaxies than for DR7, particularly for foreground galaxies with r > 15. For the brightest set of foreground galaxies (very rare on the sky), the sense of the change is ambiguous, since it may be caused in an improvement in the deblending of the brightest galaxies.

The position angles devPhi and expPhi used in the model fits used by modelMag and cModelMag do not have their errors reported in either the photoObjAll table and its views or in the corresponding FITS files (photoObj). These errors are calculated as described by Stoughton et al. (2002) based on photon statistics and can be found in the uncalibrated fpObjc files, but are not propagated to the fully calibrated photoObj files or the database. We note that the devPhi and expPhi quantities are subject to large systematic errors (of order 15 degree) that are much larger than the photon statistical errors, and we do not recommend their use for precise position angle estimates in any case.

There are two major imaging bookkeeping changes between DR7 and all subsequent releases.

First, for DR8 and later releases the photometry was reduced using a new version of the photo pipeline, under rerun "301". The identifier depends on both the rerun value and the object number within each field (called id in flat files or obj in CAS). Both of these values can change under a new rerun. Thus, all of the objID values of every object changes from DR7 to DR8.

Second, the resolve algorithm changed between DR7 and DR8. This algorithm determines which fields are used as the "primary" observation of any given region of sky and which catalog entries are the "primary" detections of the given source. Thus, some fields (and objects) which were considered primary in DR7 are not primary in DR8 (and vice-versa). Thus, the set of runs that contribute to the photoPrimary table in CAS are different, even in areas where no new imaging was taken between DR7 and DR8. Note that one aspect of the new resolve algorithm is that there are no "special" runs or areas of the survey; SEGUE runs and other imaging outside the "Legacy" survey area is all treated the same way.

For users who need to match between DR7 and DR8 (or subsequent), we recommend using the photoObjDR7 and photoPrimaryDR7 tables in CAS> The photoObjDR7 stores a positional match between the DR7 and DR8 data within each run (that is, it only allows a match between the two data releases for two catalog entries within the same run). The photoPrimaryDR7 table matches between the "primary"photometry for each data release (so can contain a match between an object in run X of DR7 but run Y of DR8).

For DR13, the resolve algorithm was not changed, but it was rerun using different evaluations of field quality (due to the updated calibration). This will change the designation of which objects are primary, but does not change the ObjID values.

About 1% of fields in DR8 timed out, whereas that rate was about 0.1% in DR7. These cases are usually due to large galaxies or bright stars; on fields containing such objects, the new sky-subtraction techniques cause larger deblends and processing times. Many of the timed-out fields in DR8 are in the Galactic Plane; the rate at Galactic latitudes greater than 15 deg is about 0.5%.

One can identify a timed-out field in the field table or photoField files with the photoStatus flag, which is set to 3 for timed-out fields. See the image quality documentation for a more complete description.

Until June 2011, the CAS photoObj tables (and all related views), the flags variables were missing the last 32 bits of information. This included flags, flags_u, flags_g, flags_r, flags_i, and flags_z. This error occurred due to an error in preparing the inputs for the database load.

This problem has been repaired in the current version of the DR8 database.

Some chips have bad CCD columns which get interpolated over by the photometric pipeline, leading to noticeably correlated noise. The bad columns for each run are currently available in
fpM*.fits. These files can be found on the Science Archive Server in the objcs subdirectory of each run/rerun directory (e.g.,
https://data.sdss.org/sas/dr17/eboss/photo/redux/301/1740/objcs/1/.
They can be read with the read_mask command; see our description of reading atlas images for access to that software.

The u filter has a natural red leak around 7100 Å which is supposed to be blocked by an interference coating. However, under the vacuum in the camera, the wavelength cutoff of the interference coating has shifted redward (see the discussion in the EDR paper), allowing some of this red leak through. The extent of this contamination is different for each camera column. It is not completely clear if the effect is deterministic; there is some evidence that it is variable from one run to another with very similar conditions in a given camera column. Roughly speaking, however, this is a 0.02 magnitude effect in the u magnitudes for mid-K stars (and galaxies of similar color), increasing to 0.06 magnitude for M0 stars (r-i ~ 0.5), 0.2 magnitude at r-i ~ 1.2, and 0.3 magnitude at r-i = 1.5. There is a large dispersion in the red leak for the redder stars, caused by three effects:

• The differences in the detailed red leak response from column to column, beating with the complex red spectra of these objects.
• The almost certain time variability of the red leak.
• The red-leak images on the u chips are out of focus and are not centered at the same place as the u image because of lateral color in the optics and differential refraction - this means that the fraction of the red-leak flux recovered by the PSF fitting depends on the amount of centroid displacement.

To make matters even more complicated, this is a detector effect. This means that it is not the real i and z which drive the excess, but the instrumental colors (i.e., including the effects of atmospheric extinction), so the leak is worse at high airmass, when the true ultraviolet flux is heavily absorbed but the infrared flux is relatively unaffected. Given these complications, we cannot recommend a specific correction to the u-band magnitudes of red stars, and warn the user of these data about over-interpreting results on colors involving the u band for stars later than K.

There is a slight and only recently recognized downward bias in the determination of the sky level in the photometry, at the level of roughly 0.1 DN per pixel. This is apparent if one compares large-aperture and PSF photometry of faint stars; the bias is of order 29 mag arcsec^-2 in r. This, together with scattered light problems in the u band, can cause of order 10% errors in the u band Petrosian fluxes of large galaxies. As implied, this effect does perturb galaxy colors; that is the sky subtraction problem described above has slightly worse effects on the u band than on other bands.

Top panel: high proper motion stars from the Superblink survey that are missing in DR6.
Bottom panel: High proper motion stars recovered by SDSS.

A comparison of SDSS catalogs has shown that high proper motion stars from Sebastien Lepine's database (SUPERBLINK) are not registered as high proper motion stars in the DR6. For those stars, the ProperMotions table lists pm=0.0. The reason their motion is not registered in DR6 is because of the incompleteness of the USNO-B catalog, from which the DR6 proper motions are derived. Areas where the incompleteness is particularly severe include regions where there are bad SERC-I or POSS-II N plates (open squares, list from J. Munn).

If one tries to select off nearby stars with e.g. a pm<0.75 mas/yr proper motion cutoff, then the sample will be contaminated with these pm=0 high proper motion stars. The plot shows the stars with pm>100 mas/yr, which are relatively rare, but I suspect that a similar fraction of 10 mas/yr < pm < 100 mas/yr stars will be similarly unregistered in the DR6, which can add up a lot of foreground contaminants.

At the moment, the only mitigation strategy is to avoid the regions where contamination will be most severe.

DR8 includes a fair amount of imaging at low Galactic latitude |b| < 25 degrees, and as such, there are highly crowded fields, and regions of high extinction. These data were processed with the standard SDSS photo pipelines. Since these pipelines were not designed to work in such crowded regions, the quality of the photometry in these areas is not guaranteed to be accurate to the SDSS quoted limits of 2% in color and r magnitude, nor is each and every crowded frame fully deblended; i.e., many fields are incompletely cataloged. Some fields, in particular many at |b| < 5 degrees, time out completely and have no cataloged objects whatsoever. Using the image quality flags is often useful to identify such cases.

In DR9 and beyond, photometric object outlines are missing for some camcols of some imaging fields. They are missing because in these runs there are a handful of corrupted "fpAtlas" files output by the photometric pipeline. While these corruptions only affect a handful of fields, the process of creating the atlas outlines was interrupted for these runs. We do not plan on fixing this issue.

For these fields, the Outlines option will not be available in the SkyServer Navigate tool. The missing camcols are:

The DR13 photometric catalogs have had the new DR13 calibration applied. However, the images (described here) distributed in DR13 have not had the new calibration applied. The differences in zeropoints and calibrations are small (typically less than a percent).

For DR13 and later, SkyServer Navigate does not show the correct primary fields in a small number of areas of sky. Correspondingly, the primary-mask files in the $BOSS_PHOTOOBJ/atlasOutline directories are out of date; the other files in that directory tree are correct.

The original paper describing our WISE forced photometry catalog said that WISE bands W3 and W4 are included in the data. This is true of the "unWISE" catalog, but it is not true of the data released by SDSS, which only includes W1 and W2. These were the only bands used for targeting. The targeting is described correctly in the newer reference (Lang, Hogg, & Schlegel 2016).

Between DR8 and DR15, an error occurred in the loading of the Field table which affects the following columns that describe the 2-gaussian fit to the PSF: psf2GSigma1_g, psf2GSigma2_g, psf2GB_g, psf2GSigma1_r, psf2GSigma2_r, psf2GB_r, psf2GSigma1_i, psf2GSigma2_i, psf2GB_i, psf2GSigma1_z, psf2GSigma2_z, and psf2GB_z.

The values in the Field table for these columns in the griz-bands are all mistakenly equal to the corresponding u-band value.

This error does not affect the psfWidth_g values or the corresponding r, i, or z-band values based on the 2-gaussian fit. The other PSF fit available, which is a 2-Gaussian model plus a Moffat profile, is not affected. The photoField and psField files on SAS containing the original numbers are not affected.