[KIDS] Improved background subtraction to remove CCD edge vignetting

Fri Jan 18 13:20:55 CET 2013

Dear all,

My apologies for sending such large images. Attached are the same 
figures as jpg's. Their description is in the quoted mail below.

Hugo

On 18/01/13 13:13, Hugo Buddelmeijer wrote:
> Hi all,
>
> Thanks for doing the comparisons Nicola and Francesco. I did a similar
> comparison as well.
>
> Attached are the 4 KIDS_184.0_-1.5 with overplotted those sources where
> the new background subtraction has the largest effect.
>
> For every source the difference in the 20arcsec aperture magnitude
> before and after correction is derived. A source is included in the
> figure if the absolute difference is larger than 1.5 the error on the
> original magnitude. No magnitude limit is applied.
>
> The size of the circles is based on this offset:
> size = abs(MAG_APER_20_new - MAG_APER_20_old) - 1.5*MAGERR_APER_20_old
> The 10 circles on the bottom represent offsets of 0.1 to 1.0 magnitude.
> Red circles have IMAFLAGS_ISO > 0.
>
> In i and r there is a clear change in the bar regions, and in CCD 82. In
> u and g there is hardly any effect in the bar region. In all bands there
> are effects outside the bar region for a handful of sources. The biggest
> differences are in crowded areas (~concentric circles), so these might
> also be caused by misidentification.
>
> I'll put the code to make the figures on the CVS later. Perhaps it is
> worthwhile to make some magnitude cut instead of including also the very
> faintest sources.
>
> Hugo
>
>
> On 18/01/13 12:45, Francesco La Barbera wrote:
>> Hi Jelte,
>>
>> this is the difference of mags in the sense "after - before" IBSA.
>> As you can see there is a significant tail towards positive values, as
>> mentioned
>> in our previous email. These objects have too faint mags after IBSA,
>> possibly
>> because of a flux oversubtraction (as shown by the comparison to SDSS).
>>
>> So, we improve detection at the the price of somewhat worse photometry
>> for some (<20%) true objects in the *affected* regions.
>>
>> Cheers,
>> F.
>>
>> On Fri, Jan 18, 2013 at 12:25 PM, Jelte de Jong
>> <jelte at strw.leidenuniv.nl <mailto:jelte at strw.leidenuniv.nl>> wrote:
>>
>> Hi Nicola,
>>
>> thanks for performing this analysis, and good that the photometry
>> seems nearly not affected.
>>
>> Do you also have plots of the delta-mag between the KiDS-CAT
>> magnitudes before and after IBSA?
>> Comparing the difference on the KiDS photometry directly might give
>> us more detailed information than comparing both separately to SDSS.
>> That would also allow a comparison at fainter magnitudes.
>>
>> Cheers,
>> Jelte
>>
>>
>> On 18/01/13 12:09, Nicola R. Napolitano wrote:
>>>
>>> Hi Hugo and everybody,
>>>
>>> Francesco and I finally got the chance to check the photometry on
>>> the post improved background subtraction algorithm (IBSA) from
>>> Hugo. The IBSA is very well improving the object detection as the
>>> more regular background structure prevents spurious source
>>> extraction. It remained to figure out whether the new background
>>> affects object photometry.
>>>
>>> To assess this we have compared KIDSCAT magnitudes within 5''
>>> diameter apertures against SDSS petrosian magnitudes.
>>>
>>> We have considered the KIDS_175.0_-0.5i field (see Fig. 1 before
>>> and after IBSA on the left and right panel respectively) since all
>>> others coadds from Hugo were mostly showing less severe horizontal
>>> features.
>>>
>>> We have first concentrated on the area affected by the CCD82
>>> vignetting where the background correction has to have been
>>> stronger (green area in Fig. 1).
>>> Here we have considered all flag=0 sources from KIDSCAT matched to
>>> SDSS and computed the delta_mag for images before-IBSA and
>>> after-IBSA. We have selected about 900 sources in the three catalogs.
>>>
>>> The delta mags are plotted in Fig. 2 (delta_mag=
>>> mag_SDSS-mag_KCAT). We have selected only sources brighter than 21
>>> (SDSS i-band). Objects have generally consistent magnitudes
>>> (mean~-0.06 both before and after IBSA respectively, mostly due to
>>> an aperture effect) while the scatter of the histogram grows after
>>> IBSA (sig=0.20, 0.26 before and after IBSA, respectively), in
>>> particular because of an excess of sources after IBSA in the
>>> negative tail. We have checked that this tails is not mag
>>> dependent. Thus there seems that the IBSA possibly subtracts some
>>> of the source flux. This likely happens for ~20% of the sources in
>>> the given CCD82-selected area.
>>>
>>> We have made the same plot for the whole image in order to see
>>> whether this effect is present also in the area with less
>>> problematic background. The delta_mag plot is shown in Fig. 3.
>>> Here we see that the two distributions, although presenting yet a
>>> negative tail, are much more overlapping with each other, which
>>> means that the overall photometry on the whole field is almost
>>> unchanged. Thus the IBSA does introduces a difference in the
>>> photometry, but apparently this is mainly affecting sources in
>>> regions affected by strong horizontal features.
>>>
>>> Conclusions: IBSA improves source detection without basically
>>> affecting the source photometry in the regions with moderate BKG
>>> variation (e.g. normal CCD gaps), while it can introduce some
>>> photometry variation in the area of strong CCD vignetting. We
>>> propose to add a flag to the catalog for these latter cases in
>>> order to warn about photometric reliability due to CCD failures.
>>>
>>> Cheers,
>>>
>>> Nicola and Francesco
>>>
>>> On 1/10/13 11:30 AM, Hugo Buddelmeijer wrote:
>>>> Hi Nicola,
>>>>
>>>> Thanks for testing that the bar removal indeed improved
>>>> KIDS_175.0_-0.5
>>>> i as well, indeed a troublesome observation.
>>>>
>>>> We decided in the telecon to check the effect on the photometry on a
>>>> full tile. KIDS_184.0_-1.5 is a good test case, since it shows bar
>>>> problems in at least g, r and i. Could you compare the photometry
>>>> for
>>>> these coadds:
>>>>
>>>> band: old object_id new object_id
>>>> u: ce4fc4615d66ebf2e0407d81e60e07f4 D2DE8960273B568DE0407D81E60E7CAD
>>>> g: ce4cd66f33e9ad69e0407d81e60e6525 D2DE4D015103EF62E0407D81E60E7A67
>>>> r: ce4cfe555662aa8ee0407d81e60e6745 D2ED2945BFE943D9E0407D81E60E02BC
>>>> i: ce4ebb600dd0e946e0407d81e60e7a9d D2DE896026D9568DE0407D81E60E7CAD
>>>>
>>>> These are all three improved by the new algorithm, although i still
>>>> shows residual bars. The u band image is visually improved as well,
>>>> although this is a side-effect. Links to the dbviewer are on the
>>>> wiki:
>>>> http://wiki.astro-wise.org/projects:kids:tbarvignetting
>>>>
>>>> Greetings,
>>>> Hugo
>>>>
>>>>
>>>> On 09/01/13 13:52, Nicola R. Napolitano wrote:
>>>>> Hi Hugo and all,
>>>>>
>>>>> the corrected images look very good, nice job.
>>>>> We had a quick check to the image that looked worse than the
>>>>> others with
>>>>> clear severe features for CCD vignetting, i.e. KIDS_175.0_-0.5i
>>>>> from the
>>>>> wiki page.
>>>>> In attachment the catalogs before and after the correction. Red
>>>>> are all
>>>>> sources, green the "good" ones.
>>>>> The situation is indeed improved and also if there remains some
>>>>> slight
>>>>> horizontal features in the coadd in the area of CCD82, the
>>>>> catalogs are
>>>>> note significantly affected. Of course we want to look in more
>>>>> details
>>>>> at the extracted sources in that area.
>>>>>
>>>>> All other images you sent in your examples with residual CCD
>>>>> gaps did
>>>>> not actually present spurious detection in catalogs, thus we
>>>>> expect the
>>>>> improvement here is on the photometry measurement uncertainty.
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Nicola
>>>>>
>>>>>
>>>>>
>>>>> On 1/9/13 9:56 AM, Hugo Buddelmeijer wrote:
>>>>>> Dear KiDS members,
>>>>>>
>>>>>> An improved background subtraction algorithm for
>>>>>> RegriddedFrames was
>>>>>> created to remove the vignetting effect at the edges of the
>>>>>> CCDs. We
>>>>>> (Groningen) plan to use the method for KIDS ESO DR1.
>>>>>>
>>>>>> * Example *
>>>>>> See this example CoaddedRegriddedFrame of KIDS_184.0_-0.5 in r:
>>>>>> before: http://tinyurl.com/arce9oj
>>>>>> after: http://tinyurl.com/aso2q5k
>>>>>> And in the i-band, KIDS_131.0_-1.5:
>>>>>> before: http://tinyurl.com/b33pu47
>>>>>> after: http://tinyurl.com/a3muamz
>>>>>>
>>>>>> * Procedure *
>>>>>> The method is selected by choosing BACKGROUND_SUBTRACTION_TYPE
>>>>>> 4 for a
>>>>>> RegriddedFrame. (Only in 'current', not yet in 'AWBASE'.) This
>>>>>> performs
>>>>>> the following:
>>>>>> 1) Create a mask to flag all sources and bad pixels.
>>>>>> 2) Calculate the median value of the background pixels of each
>>>>>> row.
>>>>>> 3) Subtract this median value from the row.
>>>>>> 4) Let Swarp remove the rest of the background (as usual).
>>>>>>
>>>>>> * KIDS ESO DR1 *
>>>>>> Based on our experiments the Groningen team suggests to use the
>>>>>> new
>>>>>> method for the KIDS ESO DR1 release. We plan to start
>>>>>> processing next
>>>>>> week so there will be time to reprocess bad cases, if they
>>>>>> arise. Do you
>>>>>> agree?
>>>>>>
>>>>>> * Caveats *
>>>>>> There are currently no known cases where the quality of the
>>>>>> CoaddedRegriddedFrames decreases. Chip-filling galaxies will
>>>>>> probably be
>>>>>> a problem, but that was already the case with the original
>>>>>> method. The
>>>>>> vertical bar pattern seen occasionally at the corner CCDs is
>>>>>> not treated.
>>>>>>
>>>>>> * Future *
>>>>>> The 'bar pattern' arises because the vignetting of the
>>>>>> background is
>>>>>> slightly different in the science images than in the
>>>>>> calibration flats.
>>>>>> It might therefore be possible to improve the flatfielding to
>>>>>> remove the
>>>>>> bar already during the creation of ReducedScienceFrames. This
>>>>>> requires
>>>>>> more investigation and will not be achieved in time for ESO DR1.
>>>>>>
>>>>>> * Sources *
>>>>>> The flux of the sources is affected differently by the
>>>>>> vignetting than
>>>>>> the background. Correcting the source flux requires a separate
>>>>>> solution
>>>>>> and is therefore beyond the scope of this discussion. (There
>>>>>> can be a
>>>>>> flux error of about 1-2% in the affected regions of the individual
>>>>>> exposures, leading to a potential flux error of 0.2-0.4% in the
>>>>>> coadds.)
>>>>>>
>>>>>> * More Information *
>>>>>> The wiki contains more information about the bars in general and
>>>>>> examples of the improved background subtraction:
>>>>>> http://wiki.astro-wise.org/projects:kids:tbarvignetting
>>>>>> In particular in the section "Improved background subtraction".
>>>>>>
>>>>>> Greetings,
>>>>>> Hugo Buddelmeijer
>>>>>> _______________________________________________
>>>>>> KiDS mailing list
>>>>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>>>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>>>>
>>>>>
>>>>>
>>>>> _______________________________________________
>>>>> KiDS mailing list
>>>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>>> _______________________________________________
>>>> KiDS mailing list
>>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>>
>>>
>>>
>>>
>>> _______________________________________________
>>> KiDS mailing list
>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>
>> --
>> Dr. Jelte T. A. de Jong
>> KiDS project manager
>> Sterrewacht Leiden
>> Leiden University, Leiden, the Netherlands
>> E:jelte at strw.leidenuniv.nl <mailto:jelte at strw.leidenuniv.nl>
>> T:+31-(0)715275818 <tel:%2B31-%280%29715275818>
>> W:jelte.jdejong.net <http://jelte.jdejong.net>
>>
>>
>> _______________________________________________
>> KiDS mailing list
>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>> http://listman.astro-wise.org/mailman/listinfo/kids
>>
>>
>>
>>
>> _______________________________________________
>> KiDS mailing list
>> KiDS at astro-wise.org
>> http://listman.astro-wise.org/mailman/listinfo/kids
>
>
>
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/kids

-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_r_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5620178-70e5e4789b147f760e6c3782ebbdd740d8be7254.jpg
Type: image/jpeg
Size: 733542 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/5c3bf8f1/attachment-0004.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_g_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5373981-199e08c6ff9589293b1b5d2bbf44ae96bfb92ff2.jpg
Type: image/jpeg
Size: 702680 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/5c3bf8f1/attachment-0005.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_u_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5733178-545090237c92e3de73203b7328a245413d55ebc0.jpg
Type: image/jpeg
Size: 906740 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/5c3bf8f1/attachment-0006.jpg 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_i_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5482544-2ab264e3d3813adba8a780d01c1d3d46df318c4e.jpg
Type: image/jpeg
Size: 707808 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/5c3bf8f1/attachment-0007.jpg 