[Issues] Memory Leak in reduction pipeline ....

[Ewout M. Helmich]

Hi,

Hmm. The error you get can occur for a variety of reasons. Running out 
of disk space on the nodes can cause it. If the pixelmaps in question 
don't exist on the dataserver, or can not be retrieved for some reason, 
it may happen. Can you retrieve the pixelmaps in question manually?

I.e:

awe> q=HotPixelMap.filename == 
'Cal-OCORDES-WFI---------ccd50---Hot-Pix-53929.5226041.fits.gz'
awe> hp = q[0]
awe> hp.retrieve()

Can you check whether these things are causing the problem?

Regards,
Ewout


Oliver-Mark Cordes wrote:

> Hi Guys!
>
> I found out that the complete reduction process has a big big memory 
> leak, which breaks the reduction process ...
>
> I started the following process:
>
> dpu.run( 'Reduce', i='WFI', d='2001-06-21', f='#844', oc=6, 
> o='DEEP1a-0?', C=0 )
>
>
> which means to reduce 20 RawScienceFrames one after another and 
> unfortunately during the reduction of the 20th image the pipeline 
> broke up with
>
> [node1] 18:52:28 - ERROR - Error loading pixelmap 
> Cal-OCORDES-WFI---------ccd50---Hot-Pix-53929.5226041.fits
>
> which worked fine for all other images (I attached the complete log!)
>
> During a second attempt I followed the memory consumption of the awe 
> process which exceeds 20?? MB at the end. I think this is a python 
> limit, Each node in the cluster has 2GB of memory which is OK so far.
>
> But the real problem is that some of the recipies do not free their 
> memory when one step is done. I think this may be a general problem of 
> eclipse and python? I'm not sure but this is a strong suspicion ;-)
>
> If you want to test the command above this should be possible from 
> Groningen but it will use some default ever calib. files which 
> shouldn't be a problem anyway.
>
> Hope you find the bug, because it is strange to split the reduction in 
> small steps ...
>
>
> Cheers,
>     Oliver
>
>------------------------------------------------------------------------
>
>_______________________________________________
>Issues mailing list
>Issues at astro-wise.org
>http://listman.astro-wise.org/mailman/listinfo/issues
>

-- 
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=
Drs. Ewout Helmich             <><>
Kapteyn Astronomical Institute <><> Astro-Wise/OmegaCEN
Landleven 12                   <><>
P.O.Box 800                    <><> email: helmich at astro.rug.nl
9700 AV Groningen              <><> tel  : +31(0)503634548
The Netherlands                <><>
=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=-=