[KIDS] Improved background subtraction to remove CCD edge vignetting

Fri Jan 18 12:45:37 CET 2013

Hi Jelte,

this is the difference of mags in the sense "after - before" IBSA.
As you can see there is a significant tail towards positive values, as
mentioned
in our previous email. These objects have too faint mags  after IBSA,
possibly
because of a flux oversubtraction (as shown by the comparison to SDSS).

So, we improve detection at the  the price of somewhat worse photometry
for some (<20%) true objects in the *affected* regions.

Cheers,
F.

On Fri, Jan 18, 2013 at 12:25 PM, Jelte de Jong <jelte at strw.leidenuniv.nl>wrote:

>  Hi Nicola,
>
> thanks for performing this analysis, and good that the photometry seems
> nearly not affected.
>
> Do you also have plots of the delta-mag between the KiDS-CAT magnitudes
> before and after IBSA?
> Comparing the difference on the KiDS photometry directly might give us
> more detailed information than comparing both separately to SDSS.
> That would also allow a comparison at fainter magnitudes.
>
> Cheers,
> Jelte
>
>
> On 18/01/13 12:09, Nicola R. Napolitano wrote:
>
>
> Hi Hugo and everybody,
>
> Francesco and I finally got the chance to check the photometry on the post
> improved background subtraction algorithm (IBSA) from Hugo. The IBSA is
> very well improving the object detection as the more regular background
> structure prevents spurious source extraction. It remained to figure out
> whether the new background affects object photometry.
>
> To assess this we have compared KIDSCAT magnitudes within 5'' diameter
> apertures against SDSS petrosian magnitudes.
>
> We have considered the KIDS_175.0_-0.5i field (see Fig. 1 before and after
> IBSA on the left and right panel respectively) since all others coadds from
> Hugo were mostly showing less severe horizontal features.
>
> We have first concentrated on the area affected by the CCD82 vignetting
> where the background correction has to have been stronger (green area in
> Fig. 1).
> Here we have considered all flag=0 sources from KIDSCAT matched to SDSS
> and computed  the delta_mag  for images  before-IBSA and after-IBSA. We
> have selected about 900 sources in the three catalogs.
>
> The delta mags are plotted in Fig. 2 (delta_mag= mag_SDSS-mag_KCAT). We
> have selected only sources brighter than 21 (SDSS i-band). Objects have
> generally consistent magnitudes (mean~-0.06 both before and after IBSA
> respectively, mostly due to an aperture effect) while the scatter of the
> histogram grows after IBSA (sig=0.20, 0.26 before and after IBSA,
> respectively), in particular because of an excess of sources after IBSA in
> the negative tail. We have checked that this tails is not mag dependent.
> Thus there seems that the IBSA possibly subtracts some of the source flux.
> This likely happens for ~20% of the sources in the given CCD82-selected
> area.
>
> We have made the same plot for the whole image in order to see whether
> this effect is present also in the area with less problematic background.
> The delta_mag plot is shown in Fig. 3. Here we see that the two
> distributions, although presenting yet a negative tail, are much more
> overlapping with each other, which means that the overall photometry on the
> whole field is almost unchanged. Thus the IBSA does introduces a difference
> in the photometry, but apparently this is mainly affecting sources in
> regions affected by strong horizontal features.
>
> Conclusions: IBSA improves  source detection  without basically affecting
> the source photometry in the regions with moderate BKG variation (e.g.
> normal CCD gaps), while it can introduce some photometry variation in the
> area of strong CCD vignetting. We propose to add a flag to the catalog for
> these latter cases in order to warn about photometric reliability due to
> CCD failures.
>
> Cheers,
>
> Nicola and Francesco
>
> On 1/10/13 11:30 AM, Hugo Buddelmeijer wrote:
>
> Hi Nicola,
>
> Thanks for testing that the bar removal indeed improved KIDS_175.0_-0.5
> i as well, indeed a troublesome observation.
>
> We decided in the telecon to check the effect on the photometry on a
> full tile. KIDS_184.0_-1.5 is a good test case, since it shows bar
> problems in at least g, r and i. Could you compare the photometry for
> these coadds:
>
> band:  old object_id                new object_id
> u: ce4fc4615d66ebf2e0407d81e60e07f4 D2DE8960273B568DE0407D81E60E7CAD
> g: ce4cd66f33e9ad69e0407d81e60e6525 D2DE4D015103EF62E0407D81E60E7A67
> r: ce4cfe555662aa8ee0407d81e60e6745 D2ED2945BFE943D9E0407D81E60E02BC
> i: ce4ebb600dd0e946e0407d81e60e7a9d D2DE896026D9568DE0407D81E60E7CAD
>
> These are all three improved by the new algorithm, although i still
> shows residual bars. The u band image is visually improved as well,
> although this is a side-effect. Links to the dbviewer are on the wiki:
>     http://wiki.astro-wise.org/projects:kids:tbarvignetting
>
> Greetings,
> Hugo
>
>
> On 09/01/13 13:52, Nicola R. Napolitano wrote:
>
> Hi Hugo and all,
>
> the corrected images look very good, nice job.
> We had a quick check to the image that looked worse than the others with
> clear severe features for CCD vignetting, i.e. KIDS_175.0_-0.5i from the
> wiki page.
> In attachment the catalogs before and after the correction. Red are all
> sources, green the "good" ones.
> The situation is indeed improved and also if there remains some slight
> horizontal features in the coadd in the area of CCD82, the catalogs are
> note significantly affected. Of course we want to look in more details
> at the extracted sources in that area.
>
> All other images you sent in your examples with residual CCD gaps did
> not actually present spurious detection in catalogs, thus we expect the
> improvement here is on the photometry measurement uncertainty.
>
> Cheers,
>
> Nicola
>
>
>
> On 1/9/13 9:56 AM, Hugo Buddelmeijer wrote:
>
> Dear KiDS members,
>
> An improved background subtraction algorithm for RegriddedFrames was
> created to remove the vignetting effect at the edges of the CCDs. We
> (Groningen) plan to use the method for KIDS ESO DR1.
>
> * Example *
> See this example CoaddedRegriddedFrame of KIDS_184.0_-0.5 in r:
> before: http://tinyurl.com/arce9oj
> after: http://tinyurl.com/aso2q5k
> And in the i-band, KIDS_131.0_-1.5:
> before: http://tinyurl.com/b33pu47
> after: http://tinyurl.com/a3muamz
>
> * Procedure *
> The method is selected by choosing BACKGROUND_SUBTRACTION_TYPE 4 for a
> RegriddedFrame. (Only in 'current', not yet in 'AWBASE'.) This performs
> the following:
> 1) Create a mask to flag all sources and bad pixels.
> 2) Calculate the median value of the background pixels of each row.
> 3) Subtract this median value from the row.
> 4) Let Swarp remove the rest of the background (as usual).
>
> * KIDS ESO DR1 *
> Based on our experiments the Groningen team suggests to use the new
> method for the KIDS ESO DR1 release. We plan to start processing next
> week so there will be time to reprocess bad cases, if they arise. Do you
> agree?
>
> * Caveats *
> There are currently no known cases where the quality of the
> CoaddedRegriddedFrames decreases. Chip-filling galaxies will probably be
> a problem, but that was already the case with the original method. The
> vertical bar pattern seen occasionally at the corner CCDs is not treated.
>
> * Future *
> The 'bar pattern' arises because the vignetting of the background is
> slightly different in the science images than in the calibration flats.
> It might therefore be possible to improve the flatfielding to remove the
> bar already during the creation of ReducedScienceFrames. This requires
> more investigation and will not be achieved in time for ESO DR1.
>
> * Sources *
> The flux of the sources is affected differently by the vignetting than
> the background. Correcting the source flux requires a separate solution
> and is therefore beyond the scope of this discussion. (There can be a
> flux error of about 1-2% in the affected regions of the individual
> exposures, leading to a potential flux error of 0.2-0.4% in the coadds.)
>
> * More Information *
> The wiki contains more information about the bars in general and
> examples of the improved background subtraction:
> http://wiki.astro-wise.org/projects:kids:tbarvignetting
> In particular in the section "Improved background subtraction".
>
> Greetings,
> Hugo Buddelmeijer
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/kids
>
>
>
>
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/kids
>
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/kids
>
>
>
>
>
> _______________________________________________
> KiDS mailing listKiDS at astro-wise.orghttp://listman.astro-wise.org/mailman/listinfo/kids
>
>
> --
> Dr. Jelte T. A. de Jong
> KiDS project manager
> Sterrewacht Leiden
> Leiden University, Leiden, the Netherlands
> E: jelte at strw.leidenuniv.nl
> T: +31-(0)715275818
> W: jelte.jdejong.net
>
>
> _______________________________________________
> KiDS mailing list
> KiDS at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/kids
>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listman.astro-wise.org/pipermail/kids/attachments/20130118/56ea700e/attachment-0001.html 
-------------- next part --------------
A non-text attachment was scrubbed...
Name: diff_HB_175_-0.5_CCD82.ps
Type: application/postscript
Size: 9185 bytes
Desc: not available
Url : http://listman.astro-wise.org/pipermail/kids/attachments/20130118/56ea700e/attachment-0001.ps 