[Issues] why the illumination correction is underestimated in aw
Philippe Heraudeau
heraud at astro.uni-bonn.de
Tue Nov 24 17:38:41 CET 2009
Hi Ewout,
The problem in aw comes from the flat-field normalisation which is
derived per chip.
This breaks the relative zeropoints from chip to chip when
flat-fielding. Trying to get it back using a median
of the ZP for each chip (using SDSS) does not work because we broke the
continuity of the illumination correction over the field of view (see my
plots): the presence of the illumination correction itself prevents us
to derive the right relative ZPs.
If we would simply treat the whole camera as once (i.e. using one unique
normalisation number for all ccds)
we would not have problems (Imagine one single big chip instead of all
the chips...). This would give a flat background over the mosaic and
that's why the sky background trick can work. The moon should not be a
problem if it does not give a gradient over the mosaic but a _flat_
background...but I am not sure this is always true.
I think the best solution is:
- do not break the relative ZPs between chips during flat-fielding (e.g.
always use the same chip for the normalisation)
- do not calibrate chips individually on the SDSS but the whole mosaic
at once.
- In addition, using several pointings will improve the correction a lot!
Hope this helps...
Best regards,
Philippe
E. M. Helmich wrote:
> Oops, plots...
>
> E. M. Helmich wrote:
>> Hi Philippe, Fedor,
>>
>> I agree the procedure we use while fitting the 2-D polynomial is
>> wrong. It seems we indeed fit to this jig-saw puzzle your plots show
>> Philippe, and this is clearly cannot work. As to how to fix this; I'm
>> looking into it, but I don't have a good solution yet; I'm not sure I
>> see all the issues involved. One factor that influences the relative
>> (CCD-to-CCD) ADU -> magnitude scale is the gain. By replacing this line:
>>
>> sigclipped_rawzeros_list.append(sigclipped_rawzeros_dict[star][0] -
>> median)
>>
>> by this line:
>>
>> sigclipped_rawzeros_list.append(sigclipped_rawzeros_dict[star][0] +
>> 2.5*math.log10(gain.gain))
>>
>> (query for the gain object for the appropriate CCD first) in the "get
>> inputs_for_fit_routine" method, I think I'm getting a better result,
>> but I don't think this is the whole story (and there still seem to be
>> systematics in the distribution of offsets, especially along
>> X-direction of the CCD mosaic, see attached plots). Note that the
>> plots are for an exposure by MEGACAM of SA 107, where I made a fake
>> photometric reference catalog from SDSS DR5 sources for this field.
>>
>> I tried to use your method Fedor, but I get errors trying to get it
>> to work; you must have added more code than just this method to get
>> it to run. Also it seems to me if the moon is up during the exposure,
>> using the skybackground will give wrong results. Any further insight
>> would be appreciated!
>>
>> Regards,
>> Ewout
>>
>> Fedor I. Getman wrote:
>>
>>> Dear Philippe,
>>>
>>> the idea of my modification was to determinate
>>> IlluminationCorrection as best as possible, but with minimal
>>> modification of AW dataflow.
>>>
>>> PhotSrcCatalog object has skybackground already as magnitude value:
>>> skybackground = -2.50 * math.log10(frame.image.median/frame.EXPTIME)
>>>
>>> If you recall, in AW calibration flow the IlluminationCorrection
>>> produced from not many exposures, but only one. It doesn't allowed
>>> to store into DB more then 1 source exposure, used to produce
>>> IlluminationCorrection.
>>> The idea to use several exposures must be evaluated taking into
>>> account possible variation and stability of illumination over FOV.
>>>
>>> With best regars,
>>> Fedor Getman
>>>
>>> heraud at astro.uni-bonn.de wrote:
>>>
>>>> Dear Fedor,
>>>>
>>>> Thus you also noticed the problem! That's good!
>>>> Thanks for the code. I will test it when I find some time.
>>>> But I am wondering if you should not have a -2.5log10 somewhere
>>>> since I
>>>> think the "background" derived by Sextractor is in ADU/Frame, which
>>>> you
>>>> use to correct magnitude?
>>>> And does it work if we combine exposures with different exposure
>>>> times (I
>>>> do not see here that the exposure time is used)?
>>>>
>>>> Best regards,
>>>> Philippe
>>>>
>>>>
>>>>
>>>>> Dear Philippe and Gijs,
>>>>>
>>>>> first of all, my apologise for not commited to cvs my modification of
>>>>> illumination correction done several mounth before.
>>>>> Here the modified method "get_inputs_for_fit_routine()" wich must be
>>>>> invocked instead original one in "
>>>>>
>>>> ()" of IlluminationCorrection
>>>>
>>>>> class.
>>>>> The idea is to adjust before fitting the residuals into the same
>>>>> surface, using common value over FOV - sky background (as
>>>>> determined by
>>>>> sExtractor during source detection).
>>>>>
>>>>> In our tests with 3c273 mosaic, that modified method to derive
>>>>> IlluminationCorrection give reasonable results in comparision to SDSS
>>>>> mags.
>>>>>
>>>>> -------------------------------------------------------------------------------------------------
>>>>>
>>>>> def get_inputs_for_fit_routine_new(self):
>>>>> '''This routine accumulates source information from the input
>>>>> PhotSrcCatalog objects in preparation for the overall fit.
>>>>> '''
>>>>> sigclipped_rawzeros_list = []
>>>>> pixelpos_list = []
>>>>> error_list = []
>>>>>
>>>>> gridpositions_dict = {}
>>>>> sigclipped_rawzeros_dict = {}
>>>>>
>>>>> for photcat in self.photcats:
>>>>> for gridposition in
>>>>> self.gridpositions[photcat.chip.name]:
>>>>>
>>>>> gridpositions_dict[photcat.chip.name+"_"+str(gridposition.index)] =
>>>>> gridposition
>>>>>
>>>>> for src in photcat.photsourcelist:
>>>>> corrected_dmag = src.mag - src.instmag -
>>>>> (self.photcats[0].skybackground-photcat.Background)
>>>>> dmag_error = math.sqrt(src.mag_err**2 +
>>>>> src.instmag_err**2)
>>>>>
>>>>> sigclipped_rawzeros_dict[photcat.chip.name+"_"+str(src.index)] =
>>>>> (corrected_dmag, dmag_error)
>>>>>
>>>>> # do sigma-clipping
>>>>> mean, stdev =
>>>>> get_mean_and_stdev([sigclipped_rawzeros_dict[key][0] for key in
>>>>> sigclipped_rawzeros_dict.keys()])
>>>>> median =
>>>>> do_median([sigclipped_rawzeros_dict[key][0] for
>>>>> key in sigclipped_rawzeros_dict.keys()])
>>>>> sigma_clip = abs(self.process_params.SIGCLIP_LEVEL *
>>>>> stdev)
>>>>>
>>>>> for key in sigclipped_rawzeros_dict.keys():
>>>>> if abs( sigclipped_rawzeros_dict[key][0] - median ) >
>>>>> sigma_clip:
>>>>> sigclipped_rawzeros_dict.pop(key)
>>>>>
>>>>> # Add the gridpositions and the corrected raw zeropoints
>>>>> to the
>>>>> lists defined
>>>>> # above
>>>>> for star in sigclipped_rawzeros_dict.keys():
>>>>>
>>>>> sigclipped_rawzeros_list.append(sigclipped_rawzeros_dict[star][0] -
>>>>> median)
>>>>> pixelpos_list.append((gridpositions_dict[star].x,
>>>>> gridpositions_dict[star].y))
>>>>> error_list.append(1.)
>>>>>
>>>>> return pixelpos_list, sigclipped_rawzeros_list, error_list
>>>>> -------------------------------------------------------------------------------------------------
>>>>>
>>>>>
>>>>>
>>>>> Philippe Heraudeau wrote:
>>>>>
>>>>>> Dear Gijs,
>>>>>>
>>>>>> As we discussed yesterday the illumination correction is
>>>>>> underestimated
>>>>>> in awe because of the rescaling of each individual chip to the
>>>>>> median of
>>>>>> the zeropoints (see ./astro/main/IlluminationCorrection.py)
>>>>>> I simulated this in the 2 ps files you can download with:
>>>>>>
>>>>>> wget http://www.astro.uni-bonn.de/~heraud/illum_4_4.ps.gz
>>>>>> wget http://www.astro.uni-bonn.de/~heraud/illum_8_8.ps.gz
>>>>>>
>>>>>> which illustrate this effect for 4x4 chip and 8*8 chip cameras.
>>>>>> The higher number of chips the more "smoothing" and that's why you
>>>>>> notice more the underestimation with CFHTLS data than with WFI.
>>>>>>
>>>>>> The figures show:
>>>>>> a) the input illumination correction which is about 25% maximum from
>>>>>> peak to valley.
>>>>>> b) the 2D 2nd order polynomial fit of the input
>>>>>> c) the input illumination after dividing the field by the number of
>>>>>> chips and rescaling each chip to the median to simulate what is
>>>>>> done in
>>>>>> aw for 4x4 chips in illum_4_4.ps and 8x8 chips in illum_8_8.ps
>>>>>> d) the 2D 2nd order polynomial fit of c) which clearly
>>>>>> underestimate the
>>>>>> input: only a few percent (6-7%) are seen instead of 25%
>>>>>> Note: The minimum of the plotted surfaces is always set to 0 and the
>>>>>> median of c is 0.
>>>>>>
>>>>>> We should solve this issue to get the proper illumination
>>>>>> correction...
>>>>>>
>>>>>> Cheers,
>>>>>> Philippe
>>>>>>
>>>>>>
>>>>>>
>>>>
>>> ------------------------------------------------------------------------
>>>
>>>
>>> _______________________________________________
>>> Issues mailing list
>>> Issues at astro-wise.org
>>> http://listman.astro-wise.org/mailman/listinfo/issues
>>>
>>
>>
>
>
> ------------------------------------------------------------------------
>
>
> ------------------------------------------------------------------------
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Issues mailing list
> Issues at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/issues
>
--
==========================================================================
Philippe Heraudeau
Argelander-Institut für Astronomie Email: heraudeau at astro.uni-bonn.de
http://www.astro.uni-bonn.de/~heraud
Auf dem Hügel 71 Tel.: +49 (0)228 736729
D-53121 Bonn Fax.: +49 (0)228 731775
Germany
=========================================================================
More information about the Issues
mailing list