[Issues] Memory Leak in reduction pipeline ....

[Oliver-Mark Cordes]

Hi!

On Fri, 14 Jul 2006, Ewout M. Helmich wrote:

> Hi,
>
> Hmm. The error you get can occur for a variety of reasons. Running out of 
> disk space on the nodes can cause it. If the pixelmaps in question don't 
> exist on the dataserver, or can not be retrieved for some reason, it may 
> happen. Can you retrieve the pixelmaps in question manually?
>

This was my first idea, because we the available disk space was a little 
bit small, but the disk was never full during the test.


> I.e:
>
> awe> q=HotPixelMap.filename == 
> 'Cal-OCORDES-WFI---------ccd50---Hot-Pix-53929.5226041.fits.gz'
> awe> hp = q[0]
> awe> hp.retrieve()
>
> Can you check whether these things are causing the problem?
>

As you can see in the logfile, the image was used before during the 
process of the other images and was stored locally, so that only the 
python process which loads the image failed. The error message does not 
indicate whether the problem is a disk or a memory problem. Anyway 
allocating memory which is not freed is not good!


Regards,
 	Oliver


> Regards,
> Ewout
>
>
> Oliver-Mark Cordes wrote:
>
>> Hi Guys!
>> 
>> I found out that the complete reduction process has a big big memory leak, 
>> which breaks the reduction process ...
>> 
>> I started the following process:
>> 
>> dpu.run( 'Reduce', i='WFI', d='2001-06-21', f='#844', oc=6, o='DEEP1a-0?', 
>> C=0 )
>> 
>> 
>> which means to reduce 20 RawScienceFrames one after another and 
>> unfortunately during the reduction of the 20th image the pipeline broke up 
>> with
>> 
>> [node1] 18:52:28 - ERROR - Error loading pixelmap 
>> Cal-OCORDES-WFI---------ccd50---Hot-Pix-53929.5226041.fits
>> 
>> which worked fine for all other images (I attached the complete log!)
>> 
>> During a second attempt I followed the memory consumption of the awe 
>> process which exceeds 20?? MB at the end. I think this is a python limit, 
>> Each node in the cluster has 2GB of memory which is OK so far.
>> 
>> But the real problem is that some of the recipies do not free their memory 
>> when one step is done. I think this may be a general problem of eclipse and 
>> python? I'm not sure but this is a strong suspicion ;-)
>> 
>> If you want to test the command above this should be possible from 
>> Groningen but it will use some default ever calib. files which shouldn't be 
>> a problem anyway.
>> 
>> Hope you find the bug, because it is strange to split the reduction in 
>> small steps ...
>> 
>> 
>> Cheers,
>>     Oliver
>> 
>> ------------------------------------------------------------------------
>> 
>> _______________________________________________
>> Issues mailing list
>> Issues at astro-wise.org
>> http://listman.astro-wise.org/mailman/listinfo/issues
>> 
>
>

-- 
  (O>  (0-   _o)   -O)
  //\  //\    /\\  /\\
  V_/_ V_/_  _\_V _\_V

  Dr. Oliver-M. Cordes       E-Mail:   ocordes at astro.uni-bonn.de
                             WWW-URL:  http://www.astro.uni-bonn.de/~ocordes/
  Argelander Institut fuer Astronomie
  Universitaet Bonn          Phone:    (49) 228 / 73-5656
  Auf dem Huegel 71          Fax:      (49) 228 / 73-3672
  D - 53121 Bonn    Germany

  Private address: Witterschlickerstr. 22 , D - 53123 Bonn
  Private phone:   (49) 228 / 7482582