[KIDS] Improved background subtraction to remove CCD edge vignetting
Hugo Buddelmeijer
buddel at astro.rug.nl
Fri Jan 18 13:13:12 CET 2013
Hi all,
Thanks for doing the comparisons Nicola and Francesco. I did a similar
comparison as well.
Attached are the 4 KIDS_184.0_-1.5 with overplotted those sources where
the new background subtraction has the largest effect.
For every source the difference in the 20arcsec aperture magnitude
before and after correction is derived. A source is included in the
figure if the absolute difference is larger than 1.5 the error on the
original magnitude. No magnitude limit is applied.
The size of the circles is based on this offset:
size = abs(MAG_APER_20_new - MAG_APER_20_old) - 1.5*MAGERR_APER_20_old
The 10 circles on the bottom represent offsets of 0.1 to 1.0 magnitude.
Red circles have IMAFLAGS_ISO > 0.
In i and r there is a clear change in the bar regions, and in CCD 82. In
u and g there is hardly any effect in the bar region. In all bands there
are effects outside the bar region for a handful of sources. The biggest
differences are in crowded areas (~concentric circles), so these might
also be caused by misidentification.
I'll put the code to make the figures on the CVS later. Perhaps it is
worthwhile to make some magnitude cut instead of including also the very
faintest sources.
Hugo
On 18/01/13 12:45, Francesco La Barbera wrote:
> Hi Jelte,
>
> this is the difference of mags in the sense "after - before" IBSA.
> As you can see there is a significant tail towards positive values, as
> mentioned
> in our previous email. These objects have too faint mags after IBSA,
> possibly
> because of a flux oversubtraction (as shown by the comparison to SDSS).
>
> So, we improve detection at the the price of somewhat worse photometry
> for some (<20%) true objects in the *affected* regions.
>
> Cheers,
> F.
>
> On Fri, Jan 18, 2013 at 12:25 PM, Jelte de Jong
> <jelte at strw.leidenuniv.nl <mailto:jelte at strw.leidenuniv.nl>> wrote:
>
> Hi Nicola,
>
> thanks for performing this analysis, and good that the photometry
> seems nearly not affected.
>
> Do you also have plots of the delta-mag between the KiDS-CAT
> magnitudes before and after IBSA?
> Comparing the difference on the KiDS photometry directly might give
> us more detailed information than comparing both separately to SDSS.
> That would also allow a comparison at fainter magnitudes.
>
> Cheers,
> Jelte
>
>
> On 18/01/13 12:09, Nicola R. Napolitano wrote:
>>
>> Hi Hugo and everybody,
>>
>> Francesco and I finally got the chance to check the photometry on
>> the post improved background subtraction algorithm (IBSA) from
>> Hugo. The IBSA is very well improving the object detection as the
>> more regular background structure prevents spurious source
>> extraction. It remained to figure out whether the new background
>> affects object photometry.
>>
>> To assess this we have compared KIDSCAT magnitudes within 5''
>> diameter apertures against SDSS petrosian magnitudes.
>>
>> We have considered the KIDS_175.0_-0.5i field (see Fig. 1 before
>> and after IBSA on the left and right panel respectively) since all
>> others coadds from Hugo were mostly showing less severe horizontal
>> features.
>>
>> We have first concentrated on the area affected by the CCD82
>> vignetting where the background correction has to have been
>> stronger (green area in Fig. 1).
>> Here we have considered all flag=0 sources from KIDSCAT matched to
>> SDSS and computed the delta_mag for images before-IBSA and
>> after-IBSA. We have selected about 900 sources in the three catalogs.
>>
>> The delta mags are plotted in Fig. 2 (delta_mag=
>> mag_SDSS-mag_KCAT). We have selected only sources brighter than 21
>> (SDSS i-band). Objects have generally consistent magnitudes
>> (mean~-0.06 both before and after IBSA respectively, mostly due to
>> an aperture effect) while the scatter of the histogram grows after
>> IBSA (sig=0.20, 0.26 before and after IBSA, respectively), in
>> particular because of an excess of sources after IBSA in the
>> negative tail. We have checked that this tails is not mag
>> dependent. Thus there seems that the IBSA possibly subtracts some
>> of the source flux. This likely happens for ~20% of the sources in
>> the given CCD82-selected area.
>>
>> We have made the same plot for the whole image in order to see
>> whether this effect is present also in the area with less
>> problematic background. The delta_mag plot is shown in Fig. 3.
>> Here we see that the two distributions, although presenting yet a
>> negative tail, are much more overlapping with each other, which
>> means that the overall photometry on the whole field is almost
>> unchanged. Thus the IBSA does introduces a difference in the
>> photometry, but apparently this is mainly affecting sources in
>> regions affected by strong horizontal features.
>>
>> Conclusions: IBSA improves source detection without basically
>> affecting the source photometry in the regions with moderate BKG
>> variation (e.g. normal CCD gaps), while it can introduce some
>> photometry variation in the area of strong CCD vignetting. We
>> propose to add a flag to the catalog for these latter cases in
>> order to warn about photometric reliability due to CCD failures.
>>
>> Cheers,
>>
>> Nicola and Francesco
>>
>> On 1/10/13 11:30 AM, Hugo Buddelmeijer wrote:
>>> Hi Nicola,
>>>
>>> Thanks for testing that the bar removal indeed improved
>>> KIDS_175.0_-0.5
>>> i as well, indeed a troublesome observation.
>>>
>>> We decided in the telecon to check the effect on the photometry on a
>>> full tile. KIDS_184.0_-1.5 is a good test case, since it shows bar
>>> problems in at least g, r and i. Could you compare the photometry
>>> for
>>> these coadds:
>>>
>>> band: old object_id new object_id
>>> u: ce4fc4615d66ebf2e0407d81e60e07f4 D2DE8960273B568DE0407D81E60E7CAD
>>> g: ce4cd66f33e9ad69e0407d81e60e6525 D2DE4D015103EF62E0407D81E60E7A67
>>> r: ce4cfe555662aa8ee0407d81e60e6745 D2ED2945BFE943D9E0407D81E60E02BC
>>> i: ce4ebb600dd0e946e0407d81e60e7a9d D2DE896026D9568DE0407D81E60E7CAD
>>>
>>> These are all three improved by the new algorithm, although i still
>>> shows residual bars. The u band image is visually improved as well,
>>> although this is a side-effect. Links to the dbviewer are on the
>>> wiki:
>>> http://wiki.astro-wise.org/projects:kids:tbarvignetting
>>>
>>> Greetings,
>>> Hugo
>>>
>>>
>>> On 09/01/13 13:52, Nicola R. Napolitano wrote:
>>>> Hi Hugo and all,
>>>>
>>>> the corrected images look very good, nice job.
>>>> We had a quick check to the image that looked worse than the
>>>> others with
>>>> clear severe features for CCD vignetting, i.e. KIDS_175.0_-0.5i
>>>> from the
>>>> wiki page.
>>>> In attachment the catalogs before and after the correction. Red
>>>> are all
>>>> sources, green the "good" ones.
>>>> The situation is indeed improved and also if there remains some
>>>> slight
>>>> horizontal features in the coadd in the area of CCD82, the
>>>> catalogs are
>>>> note significantly affected. Of course we want to look in more
>>>> details
>>>> at the extracted sources in that area.
>>>>
>>>> All other images you sent in your examples with residual CCD
>>>> gaps did
>>>> not actually present spurious detection in catalogs, thus we
>>>> expect the
>>>> improvement here is on the photometry measurement uncertainty.
>>>>
>>>> Cheers,
>>>>
>>>> Nicola
>>>>
>>>>
>>>>
>>>> On 1/9/13 9:56 AM, Hugo Buddelmeijer wrote:
>>>>> Dear KiDS members,
>>>>>
>>>>> An improved background subtraction algorithm for
>>>>> RegriddedFrames was
>>>>> created to remove the vignetting effect at the edges of the
>>>>> CCDs. We
>>>>> (Groningen) plan to use the method for KIDS ESO DR1.
>>>>>
>>>>> * Example *
>>>>> See this example CoaddedRegriddedFrame of KIDS_184.0_-0.5 in r:
>>>>> before: http://tinyurl.com/arce9oj
>>>>> after: http://tinyurl.com/aso2q5k
>>>>> And in the i-band, KIDS_131.0_-1.5:
>>>>> before: http://tinyurl.com/b33pu47
>>>>> after: http://tinyurl.com/a3muamz
>>>>>
>>>>> * Procedure *
>>>>> The method is selected by choosing BACKGROUND_SUBTRACTION_TYPE
>>>>> 4 for a
>>>>> RegriddedFrame. (Only in 'current', not yet in 'AWBASE'.) This
>>>>> performs
>>>>> the following:
>>>>> 1) Create a mask to flag all sources and bad pixels.
>>>>> 2) Calculate the median value of the background pixels of each
>>>>> row.
>>>>> 3) Subtract this median value from the row.
>>>>> 4) Let Swarp remove the rest of the background (as usual).
>>>>>
>>>>> * KIDS ESO DR1 *
>>>>> Based on our experiments the Groningen team suggests to use the
>>>>> new
>>>>> method for the KIDS ESO DR1 release. We plan to start
>>>>> processing next
>>>>> week so there will be time to reprocess bad cases, if they
>>>>> arise. Do you
>>>>> agree?
>>>>>
>>>>> * Caveats *
>>>>> There are currently no known cases where the quality of the
>>>>> CoaddedRegriddedFrames decreases. Chip-filling galaxies will
>>>>> probably be
>>>>> a problem, but that was already the case with the original
>>>>> method. The
>>>>> vertical bar pattern seen occasionally at the corner CCDs is
>>>>> not treated.
>>>>>
>>>>> * Future *
>>>>> The 'bar pattern' arises because the vignetting of the
>>>>> background is
>>>>> slightly different in the science images than in the
>>>>> calibration flats.
>>>>> It might therefore be possible to improve the flatfielding to
>>>>> remove the
>>>>> bar already during the creation of ReducedScienceFrames. This
>>>>> requires
>>>>> more investigation and will not be achieved in time for ESO DR1.
>>>>>
>>>>> * Sources *
>>>>> The flux of the sources is affected differently by the
>>>>> vignetting than
>>>>> the background. Correcting the source flux requires a separate
>>>>> solution
>>>>> and is therefore beyond the scope of this discussion. (There
>>>>> can be a
>>>>> flux error of about 1-2% in the affected regions of the individual
>>>>> exposures, leading to a potential flux error of 0.2-0.4% in the
>>>>> coadds.)
>>>>>
>>>>> * More Information *
>>>>> The wiki contains more information about the bars in general and
>>>>> examples of the improved background subtraction:
>>>>> http://wiki.astro-wise.org/projects:kids:tbarvignetting
>>>>> In particular in the section "Improved background subtraction".
>>>>>
>>>>> Greetings,
>>>>> Hugo Buddelmeijer
>>>>> _______________________________________________
>>>>> KiDS mailing list
>>>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>>>
>>>>
>>>>
>>>> _______________________________________________
>>>> KiDS mailing list
>>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>> _______________________________________________
>>> KiDS mailing list
>>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>>> http://listman.astro-wise.org/mailman/listinfo/kids
>>
>>
>>
>>
>> _______________________________________________
>> KiDS mailing list
>> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
>> http://listman.astro-wise.org/mailman/listinfo/kids
>
> --
> Dr. Jelte T. A. de Jong
> KiDS project manager
> Sterrewacht Leiden
> Leiden University, Leiden, the Netherlands
> E:jelte at strw.leidenuniv.nl <mailto:jelte at strw.leidenuniv.nl>
> T:+31-(0)715275818 <tel:%2B31-%280%29715275818>
> W:jelte.jdejong.net <http://jelte.jdejong.net>
>
>
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org <mailto:KiDS at astro-wise.org>
> http://listman.astro-wise.org/mailman/listinfo/kids
>
>
>
>
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/kids
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_i_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5482544-2ab264e3d3813adba8a780d01c1d3d46df318c4e.png
Type: image/png
Size: 2749292 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/2ba0a16a/attachment-0004.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_u_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5733178-545090237c92e3de73203b7328a245413d55ebc0.png
Type: image/png
Size: 4002388 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/2ba0a16a/attachment-0005.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_g_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5373981-199e08c6ff9589293b1b5d2bbf44ae96bfb92ff2.png
Type: image/png
Size: 2737304 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/2ba0a16a/attachment-0006.png
-------------- next part --------------
A non-text attachment was scrubbed...
Name: barbackgroundcompare_QC-EHELMICH-OMEGACAM-------OCAM_r_SDSS-----Thumbnail-CoaddedRegriddedFrame-56252.5620178-70e5e4789b147f760e6c3782ebbdd740d8be7254.png
Type: image/png
Size: 2799253 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/2ba0a16a/attachment-0007.png
More information about the KiDS
mailing list