[Issues] why the illumination correction is underestimated in aw

[E. M. Helmich]

Hi Philippe, Fedor,

I agree the procedure we use while fitting the 2-D polynomial is wrong. 
It seems we indeed fit to this jig-saw puzzle your plots show Philippe, 
and this is clearly cannot work. As to how to fix this; I'm looking into 
it, but I don't have a good solution yet; I'm not sure I see all the 
issues involved. One factor that influences the relative (CCD-to-CCD) 
ADU -> magnitude scale is the gain. By replacing this line:

sigclipped_rawzeros_list.append(sigclipped_rawzeros_dict[star][0] - median)

by this line:

sigclipped_rawzeros_list.append(sigclipped_rawzeros_dict[star][0] + 
2.5*math.log10(gain.gain))

(query for the gain object for the appropriate CCD first) in the "get 
inputs_for_fit_routine" method, I think I'm getting a better result, but 
I don't think this is the whole story (and there still seem to be 
systematics in the distribution of offsets, especially along X-direction 
of the CCD mosaic, see attached plots). Note that the plots are for an 
exposure by MEGACAM of SA 107, where I made a fake photometric reference 
catalog from SDSS DR5 sources for this field.

I tried to use your method Fedor, but I get errors trying to get it to 
work; you must have added more code than just this method to get it to 
run. Also it seems to me if the moon is up during the exposure, using 
the skybackground will give wrong results. Any further insight would be 
appreciated!

Regards,
Ewout

Fedor I. Getman wrote:
> Dear Philippe,
>
> the idea of my modification was to determinate IlluminationCorrection 
> as best as possible, but with minimal modification of AW dataflow.
>
> PhotSrcCatalog object has skybackground already as magnitude value:
> skybackground = -2.50 * math.log10(frame.image.median/frame.EXPTIME)
>
> If you recall, in AW calibration flow the IlluminationCorrection 
> produced from not many exposures, but only one. It doesn't allowed to 
> store into DB more then 1 source exposure, used to produce 
> IlluminationCorrection.
> The idea to use several exposures must be evaluated taking into 
> account possible variation and stability of illumination over FOV.
>
> With best regars,
> Fedor Getman
>
> heraud at astro.uni-bonn.de wrote:
>> Dear Fedor,
>>
>> Thus you also noticed the problem! That's good!
>> Thanks for the code. I will test it when I find some time.
>> But I am wondering if you should not have a -2.5log10 somewhere since I
>> think the "background" derived by Sextractor is in ADU/Frame, which you
>> use to correct magnitude?
>> And does it work if we combine exposures with different exposure times (I
>> do not see here that the exposure time is used)?
>>
>> Best regards,
>> Philippe
>>
>>
>>   
>>> Dear Philippe and Gijs,
>>>
>>> first of all, my apologise for not commited to cvs my modification of
>>> illumination correction done several mounth before.
>>> Here the modified method "get_inputs_for_fit_routine()" wich must be
>>> invocked instead original one in "
>>>     
>>
>> ()" of IlluminationCorrection
>>   
>>> class.
>>> The idea is to adjust before fitting the residuals into the same
>>> surface, using common value over FOV - sky background (as determined by
>>> sExtractor during source detection).
>>>
>>> In our tests with 3c273 mosaic, that modified method to derive
>>> IlluminationCorrection give reasonable results in comparision to SDSS
>>> mags.
>>>
>>> -------------------------------------------------------------------------------------------------
>>>     def get_inputs_for_fit_routine_new(self):
>>>         '''This routine accumulates source information from the input
>>>         PhotSrcCatalog objects in preparation for the overall fit.
>>>         '''
>>>         sigclipped_rawzeros_list = []
>>>         pixelpos_list            = []
>>>         error_list               = []
>>>
>>>         gridpositions_dict = {}
>>>         sigclipped_rawzeros_dict = {}
>>>
>>>         for photcat in self.photcats:
>>>             for gridposition in self.gridpositions[photcat.chip.name]:
>>>
>>> gridpositions_dict[photcat.chip.name+"_"+str(gridposition.index)] =
>>> gridposition
>>>
>>>             for src in photcat.photsourcelist:
>>>                 corrected_dmag = src.mag - src.instmag -
>>> (self.photcats[0].skybackground-photcat.Background)
>>>                 dmag_error     = math.sqrt(src.mag_err**2 +
>>> src.instmag_err**2)
>>>
>>> sigclipped_rawzeros_dict[photcat.chip.name+"_"+str(src.index)] =
>>> (corrected_dmag, dmag_error)
>>>
>>>         # do sigma-clipping
>>>         mean, stdev   =
>>> get_mean_and_stdev([sigclipped_rawzeros_dict[key][0] for key in
>>> sigclipped_rawzeros_dict.keys()])
>>>         median        = do_median([sigclipped_rawzeros_dict[key][0] for
>>> key in sigclipped_rawzeros_dict.keys()])
>>>         sigma_clip    = abs(self.process_params.SIGCLIP_LEVEL * stdev)
>>>
>>>         for key in sigclipped_rawzeros_dict.keys():
>>>             if abs( sigclipped_rawzeros_dict[key][0] - median ) >
>>> sigma_clip:
>>>                 sigclipped_rawzeros_dict.pop(key)
>>>
>>>         # Add the gridpositions and the corrected raw zeropoints to the
>>> lists defined
>>>         # above
>>>         for star in sigclipped_rawzeros_dict.keys():
>>>
>>> sigclipped_rawzeros_list.append(sigclipped_rawzeros_dict[star][0] -
>>> median)
>>>                 pixelpos_list.append((gridpositions_dict[star].x,
>>> gridpositions_dict[star].y))
>>>                 error_list.append(1.)
>>>
>>>         return pixelpos_list, sigclipped_rawzeros_list, error_list
>>> -------------------------------------------------------------------------------------------------
>>>
>>>
>>> Philippe Heraudeau wrote:
>>>     
>>>> Dear Gijs,
>>>>
>>>> As we discussed yesterday the illumination correction is underestimated
>>>> in awe because of the rescaling of each individual chip to the median of
>>>> the zeropoints (see ./astro/main/IlluminationCorrection.py)
>>>> I simulated this in the 2 ps files you can download with:
>>>>
>>>> wget http://www.astro.uni-bonn.de/~heraud/illum_4_4.ps.gz
>>>> wget http://www.astro.uni-bonn.de/~heraud/illum_8_8.ps.gz
>>>>
>>>> which illustrate this effect for 4x4 chip and 8*8 chip cameras.
>>>> The higher number of chips the more "smoothing" and that's why you
>>>> notice more the underestimation with CFHTLS data than with WFI.
>>>>
>>>> The figures show:
>>>> a) the input illumination correction which is about 25% maximum from
>>>> peak to valley.
>>>> b) the 2D 2nd order polynomial fit of the input
>>>> c) the input illumination after dividing the field by the number of
>>>> chips and rescaling each chip to the median to simulate what is done in
>>>> aw for 4x4 chips in illum_4_4.ps and 8x8 chips in illum_8_8.ps
>>>> d) the 2D 2nd order polynomial fit of c) which clearly underestimate the
>>>> input: only a few percent (6-7%) are seen instead of 25%
>>>> Note: The minimum of the plotted surfaces is always set to 0 and the
>>>> median of c is 0.
>>>>
>>>> We should solve this issue to get the proper illumination correction...
>>>>
>>>> Cheers,
>>>> Philippe
>>>>
>>>>
>>>>       
>>
>>
>>   
>
> ------------------------------------------------------------------------
>
> _______________________________________________
> Issues mailing list
> Issues at astro-wise.org
> http://listman.astro-wise.org/mailman/listinfo/issues
>   

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=

Drs. Ewout Helmich
Kapteyn Astronomical Institute,
Astro-WISE/OmegaCEN

Landleven 12
P.O.Box 800
9700 AV Groningen
The Netherlands

email: helmich at astro.rug.nl
tel: +31(0)503634548

=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=