[Issues] Re: [Fwd: changing process parameters in dpu]

[Dave Wilman]

Thanks Ewout, yes, I think this is something which will have to be
changed, since the results in the database will become practically
untracable and unrepeatable if we have different versions of code floating
around. Actually this is probably even more important when it comes to the
SExtractor/SWARP stages than in the basic reduction, since these tasks are
even more intensive and will need to be run using the dpu. I hope someone
has an idea about how to implement this change.

Thanks,
Dave (on behalf of the Munich node)

David Wilman
Max Planck Institute for Extraterrestrial Physics
Garching bei Muenchen
Germany
Tel: +49 (0)893 0000 +3117
Fax: +49 (0)893 0000 +3495

On Fri, 7 Jul 2006, Ewout M. Helmich wrote:

> Hi,
>
> I've looked a little bit into the way parameters are handled, and I
> don't think it will be trivial to change all the parameters (especially
> Sextractor/Swarp parameters) in the way you suggest. I agree that it is
> preferable to have as much control as possible through parameters, which
> is something we can trace, as opposed to control through changes in the
> code. There's definitely room for improvement here.
>
> Regards,
> Ewout
>
>
>
>
> Dave Wilman wrote:
>
> >Dear Ewout & everybody,
> >
> >We are seriously concerned that this does not sound like a feasible
> >solution in the long term. In particular, surely it could get way out of
> >hand if everyone has his/her own version of astrowise code? Could it not
> >be included in the main AstroWise code (i.e. in the Reduce.py routine
> >there is a dictionary **kw passed in, which is not made use of: Perhaps
> >naively we would not expect it to be too difficult to make use of the kw
> >later on to pass any parameters). In particular, with federation of the
> >system over various nodes it is surely imperitive that there are not many
> >conflicting versions of code hanging around! Nonetheless, there are often
> >going to be occasions where it is necessary to change parameters, and yet
> >the job should be able to run remotely using the dpu object.
> >
> >Thanks,
> >Dave, Johannes & Jan
> >
> >David Wilman
> >Max Planck Institute for Extraterrestrial Physics
> >Garching bei Muenchen
> >Germany
> >Tel: +49 (0)893 0000 +3117
> >Fax: +49 (0)893 0000 +3495
> >
> >On Thu, 6 Jul 2006, Ewout M. Helmich wrote:
> >
> >
> >
> >>Hi Johannes, Dave,
> >>
> >>Unfortunately, no. Almost none of the parameters can be changed in the
> >>call to the dpu:
> >>
> >>I.e.
> >>
> >>dpu.run('Bias', i='WFI', d='2000-04-28', oc=8)
> >>
> >>This is pretty much as far as the control at this level goes. I.e. "oc"
> >>(OVERSCAN_CORRECTION) is a regular (but very important) process parameter.
> >>
> >>The typical way to change the process parameters in this case is to
> >>change the DBRecipe, in your case DBRecipes/HotPixels.py and add the
> >>following (the second) line after the instantiation of the HotPixelMap
> >>class:
> >>
> >>hotpixelmap = HotPixelMap()
> >>hotpixelmap.process_params.MAXIMUM_HOTPIXELCOUNT = 70000
> >>
> >>Since the user's own code is sent to the processing cluster (!), this
> >>approach works, but it requires more knowledge from the user.
> >>
> >>Regards,
> >>Ewout
> >>
> >>
> >>
> >>Gijs Verdoes Kleijn wrote:
> >>
> >>
> >>
> >>>Hi Ewout and Willem-Jan,
> >>>
> >>>Dave and Johannes (see cc for emails) are preparing an Astro-Wise
> >>>tutorial to be given next week in Munich. Could you help them out with
> >>>their question about th dpu below? Perhaps it is useful to include the
> >>>on-line "target processor" in this reply as well.
> >>>
> >>>
> >>>Thanks and cheers, Gijs
> >>>ps. please cc me in the reply.
> >>>
> >>>
> >>>-------- Original Message --------
> >>>Subject:     changing process parameters in dpu
> >>>Date:     Thu, 06 Jul 2006 10:23:22 +0200
> >>>From:     Johannes Koppenh�fer <koppenh at mpe.mpg.de>
> >>>Organization:     http://freemail.web.de/
> >>>To:     Gijs Verdoes Kleijn <g.verdoes at astro.rug.nl>
> >>>
> >>>
> >>>
> >>>Hi Gijs,
> >>>
> >>>we are working on the tutorials at the moment and wonder if it is
> >>>possible to change process parameters in dpu. It seems that there is
> >>>an optional keyword list **options you can specify in the dpu-call and
> >>>there is also an optional **kw list in Reduce.py but nothing is done
> >>>with it. So is it yet possible to change parameters in dpu? This is
> >>>important because else we would have to run any task with non-standard
> >>>parameters on the local cpu which would be pointless. E.g. we want to
> >>>change the MAXIMUM_HOT_PIXELS parameters in order to make the task
> >>>work on WFI images (where there are a lot of bad pixels). I wonder if
> >>>some of you have faced the same problem before.
> >>>
> >>>Could you comment quickly on this issue?
> >>>
> >>>Regards,
> >>>Johannes & Dave
> >>>_____________________________________________________________________
> >>>Der WEB.DE SmartSurfer hilft bis zu 70% Ihrer Onlinekosten zu sparen!
> >>>http://smartsurfer.web.de/?mc=100071&distributionid=000000000071
> >>>
> >>>
> >>--
> >>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >>Drs. Ewout Helmich             <><>
> >>Kapteyn Astronomical Institute <><> Astro-Wise/OmegaCEN
> >>Landleven 12                   <><>
> >>P.O.Box 800                    <><> email: helmich at astro.rug.nl
> >>9700 AV Groningen              <><> tel  : +31(0)503634548
> >>The Netherlands                <><>
> >>=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> >>
> >>
> >>
> >>
> >
> >
> >
>
> --
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
> Drs. Ewout Helmich             <><>
> Kapteyn Astronomical Institute <><> Astro-Wise/OmegaCEN
> Landleven 12                   <><>
> P.O.Box 800                    <><> email: helmich at astro.rug.nl
> 9700 AV Groningen              <><> tel  : +31(0)503634548
> The Netherlands                <><>
> =-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
>
>