[News] Quality control- implementing subwindow statistics
Edwin A. Valentijn
valentyn at astro.rug.nl
Mon Nov 1 21:21:55 CET 2004
Implementing subwindows in AstroWise QC
This note summarizes the basic QC structures in the AstroWise system
and how one can use these structures to implement the statistics of
small subwindows as a tool for quality control evaluations.
There are two fundamentally different ways QC can be performed in the
AstroWise system,
( relevant for the implementation of multiple subwindows as a means to
assess quality of data):
Way one uses data committed to the db in the past. By quering the db on
particular items, such as Calfiles or object attributes like image
statistics on whole frames (imstat) one can interactively study the
behaviour of particular properties. Interactively, because this is
contrary to standard Pipelines.
When this is done as function of time we call this a form of trendanalysis.
In AstroWise this is mostly done in the Awe (python) prompt or by means
of the webinterface Objectviewer.
In special cases one could make scripts for standardized analysis or
tools, eg inspect trend of zeropoint .
For the subwindows, one could envisage QC analysis on the statistical
properties as function of time, or for a series of exposures.
The implementation of Way one is simply provided by
* the Class definition of SubWinStat which has a BaseFrame
dependency and a make() method.
* and the requirement that for every instantiation of a db object
of type frame the SubsWinStat.make() button is pushed
Way two is the more elementary form of QC. It is performed on any
instantiation of an object created by the pipelines for which it is
judged that a QC check has to be done. Here the QC is working on
individual object creation; the object QC flag is a property/attribute
of the instantiated object. When the object is labeled "bad" by QC it
will not be used by later Recipes which check on these flags.
The QC itself is performed by (implemented) in one, two or three of the
methods Verify, compare and inspect.
These methods can use subwindowstatistics and thus it is a requirement
that the SubWinStat button is pushed before the verify, compare or
inspect methods are instantiated.
Normally, verify would evaluate the statistics over different subwindows
in one frame, while compare would compare subwindows with a previous or
a reference frame.
The implementation of Way two is provided by:
* The Class definition of SubWinStat which has a BaseFrame
dependency and a make() method.
* The detailed Python script of the usages of subwindows in each
of the Classes ( the most relevant Classes are those who have a
related Recipe) for which it has been decided to push the
buttons verify and compare ( eg Recipe Bias -> Class BiasFrame
). This is most of the work.
Note - Way three : On one hand we can look at SubWinStat as a property
of a BaseFrame object and thus we instantiate it any time a BaseFrame
is created, on the other hand it would be nice to allow a later
execution of SubWinStat on many already existing BaseFrames (for
instance with different Subwindow dimensions) and then SubWinStat
should be threated as a Class in its own (and not as an attribute). The
factory construct allows the combination of both properties (Class and
attribute). Thus way three allows derivation of subwindowstatistics
with re-defined dimensions of the SubWindows.
The implementation of Way three is provided by:
* the SubWinStatFactory construct
<> Note - QC flag/index calculation : The calculation of the QC
flag/index, and the qualification of the frame, is done in its verify()
method, using the result from make_subwinstat(). For example, the check
on the flatness of the DomeFlatFrame is implemented in the verify() of
the DomeFlatFrame.
The scripts below provide an example of implementations at the various
places.
<>-----------------------------------------------------------------------
<>
-----------------------------------------------------------------------
Rename the Class producing the subwindow objects:
class QCStatistics(DBObject): -> class SubWinStat(DBObject)
-----------------------------------------------------------------------
# Recipe Bias example of QC structure in recipe
def make_masterbias(raw_bias_frames, read_noise, overscan = 0):
'''This routine derives Calfile 541.
Inputs :
raw_bias_frames -- list of raw bias frames
read_noise -- a read noise calibration
overscan -- overscan correction mode'''
filename = construct_filename(raw_bias_frames[0])
masterbias = BiasFrame(pathname = filename)
configure_process_parameters(masterbias, overscan)
masterbias.raw_bias_frames = raw_bias_frames
masterbias.read_noise = read_noise
masterbias.make() # rule: fixed order of execution for make, make_subwinstat, verify, compare, inspect
# store, commit and subwinstat.commit
masterbias.make_subwinstat() # calculates statistics per subwindow and creates objects
masterbias.verify() # could use subwindows when desired - various windows within one frame
masterbias.compare() # could use subwindows when desired - windows of different frames
# masterbias.inspect() # the inspect() method does not necessarily have to be executed
masterbias.store()
masterbias.commit() # commit CalFile in db
masterbias.commit_subwinstat() # commit window stats in db
return masterbias
-------------------------------
# add subwindow method to Baseframe
from SubWinStatFactory import create_subwinstat # Only import the factory function from the SubWinStatFactory module
class BaseFrame:
def make_subwinstat():
self.subwinstat = create_subwinstat() # to avoid circular dependencies
self.subwinstat.qcframe = self # SubWinStat object has to refer to this BaseFrame
self.subwinstat.make() # compute the statistics
def commit_subwinstat():
if self.subwinstat: # commit the SubWinStat object for this BaseFrame
self.subwinstat.commit()
------------------------------
# In module SubWinStatFactory
from SubWinStat import SubWinStat
def create_subwinstat():
'''Instantiate sub-window statistics object.'''
return SubWinStat()
------------------------------
# astro.main.BiasFrame example of verify and compare methods
class BiasFrame:
def verify():
''' QC by Internal arithmetics using sub-windows
Example showing how to access the sub-window statistics for this BiasFrame'''
for k in self.subwinstat.windows:
if k.mean > 5:
self.quality_flags = self.quality_flags | QC_BAD_SUBWINDOWFLAG
def compare():
''' QC Trend-analysis using sub-windows
Example showing how to access the sub-window statistics for this BiasFrame'''
previous = BiasFrame.select_for_date((self.raw_bias_frames[0].DATE_OBS - DateTime.oneWeek).strftime('%Y-%m-%d')
for x, y in zip(self.subwinstat.windows, previous.subwinstat.windows):
if abs(x.mean - y.mean) > 0.1:
self.quality_flags = self.quality_flags | QC_SUBWINDOW_CHANGED_FLAG
--------------------------------------------------------------------------------------------------------------
<><>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listman.astro-wise.org/pipermail/news/attachments/20041101/91ef80da/attachment.htm
More information about the News
mailing list