[News] sigmaclip algorithm
Roeland Rengelink
rengelin@strw.LeidenUniv.nl
Tue, 18 Mar 2003 13:39:52 +0100
Hi all,
I think the code that was recently submitted constitutes a considerable
improvement on the previous version. I like the flexibility of the
current implementation. Experience will teach us what parameters will
give the best result.
The next step would be to incoprorate this work into the astro-wise
framework. Let me discuss this in some detail for this particular case.
We work with several levels of abstraction with image processing
At the lowest level there is the eclipse C-code and the python-eclipse
wrapper c_eclipse.py. This is the level at which the implemented the
sigma-clipping algorithm was implemented.
At the next level there is the object-oriented wrapper of c_eclipse.py.
At this level we define several objects (image, cube, header, pixelmap)
whose methods define operations that can be performed on and with these
objects. For example: One can add and subtract images, one can normalize
them, extract subregions, perform statistics etc. etc. These are all
operations on individual images. In addition there are operations on
lists of images (cubes). These include arithmetic,normalization,
averageing etc. etc. It should come as no suprise that you can
implement an average_sigmaclip() method on cubes that uses
c_eclipse.cube_sigmaclip functon to perform the operation. I have made a
prelimenary implemention of this method in the cube object in the
eclipse.cube module in eclipse/lang/python/src/eclipse/cube.py
Note that the standard astro-wise interface to the low-level
sigma-clipping functionality would be through this OO wrapper
At the next higher level we combine low-level processing algorithms to
define data-reduction procedures for different data types. These include
BiasFrame, DomeFlatFrame, etc. etc. These procedures are invoked
through the make() method of these objects. Taking BiasFrame as an
example, we see that:
Given
bias = BiasFrame(...) # a target
bias.raw_bias_frames = [...] # a list of dependencies
bias.make() # do it...
the following will happen
bias.make() will invoke bias.make_image(). bias.make_image() does the
following:
1. construct a cube from a list of images created by extracting the trim
section from each raw bias image.
2. average the images in the cube using cube.average()
or in code:
images = []
for f in raw_bias_frames:
ima = eclipse.image.image(f)
ima = ima.extract_region(...)
images.append(ima)
c = eclipse.cube.cube(images)
bias_image = cube.average()
Note that only the last step needs to be replaced to implement bias
production with sigma-clipping
bias_image = cube.average_sigmaclip()
Finally, at the highest level there are user-interfaces to the process
targets. These are the recipes like the bias recipe. Note that recipes
are meant as interfaces to high-level operations, not low-level operations.
So, what are the next steps?
- I have implemented a cube.average_sigmaclip() method for your
consideration. It would be good if you tested this, and fleshed out the
interface. Things to consider:
o the documentation
o one method with flags as parameters to distinguish strategies, or
o more methods to distinguish strategies?, e.g:
cube.average_clip_measured_sigma(..., gain, rn)
cube.average_clip_computed_sigma(...)
- Update recipes to use the sigma clipping averageing as specified by
the CP. Questions to consider:
o Which parameters to fix.
o Which parameters to just give defaults for (through the
BiasFrameParameters object) but allow the user to change
o How to test the code (unittests)
Finally, as I remarked previously, the Recipes directory in opipe
contains command-line user-interfaces to high-level pipeline operations.
The sigma-clipping algorithm itself is not such a high-level operation,
and, hence, the SigmaClip recipe does not belong there. If you feel that
astro-wise should provide these interfaces to low-level functionality
too, I'd like to have a separate discussion on that.
Cheers,
Roeland
Roberto Silvotti wrote:
>Hi all,
>
>The files containing the sigma clip algorithm have been
>committed few days ago to the AW cvs.
>
>There are 4 new files:
>- opipe/Recipes/SigmaClip.py
>- opipe/eclipse/src/iproc/cube_sigmaclip.c
>- opipe/eclipse/src/include/cube_sigmaclip.h
>- opipe/eclipse/src/include/eclipse.h
>
>The algorithm is quite general and can be used in different
>situations: creation of the master bias, but also for master
>flats and scientific frames.
>Sigma is calculated considering a (squared) sum of poissonian
>noise + read-out noise, as it is done in IRAF.
>Although this method is certainly more robust, it is also
>possible, if one prefers, to derive sigma directly from
>the data (standard deviation).
>More info are given in the gzipped ps file attached.
>
>You are kindly invited to test this algorithm and let us
>know your comments and suggestions.
>Please consider that you have to rebuilt eclipse
>in order to include the new eclipse files.
>
>We also feel that it would be very usefull, to avoid
>confusion,to define a general list of key for the parameters
>of all the python scripts.
>For example, if -o means output file, then the key -o should
>be used in all scripts (with the same meaning !).
>We can discuss on this in the next telekon.
>
>
>Best regards,
> Roberto
>
>----------------------------------------
> Dr. Roberto Silvotti
>----------------------------------------
>INAF (Istituto Nazionale di AstroFisica)
>Osservatorio Astronomico di Capodimonte
>via Moiariello 16, I-80131 Napoli, Italy
>----------------------------------------
>tel/fax: +39-081-5575583/456710
>e-mail: silvotti@na.astro.it
>web: http://www.na.astro.it/~silvotti
>----------------------------------------
>