[KIDS] R: Re: visual grading and manual mask production

Thu Apr 11 18:57:30 CEST 2013

Dear Hugo
All the masks (manual, automatic and automatic+manual) are available in:

http://vstportal.oacn.inaf.it/KIDSCAT/ESO_DR1_kc1.6_masks/

Cheers

   Mario

----- Messaggio originale -----
Da: Hugo Buddelmeijer <buddel at astro.rug.nl>
A: Mario Radovich <mario.radovich at oapd.inaf.it>
Cc: Francesco La Barbera <flabarber at gmail.com>, Gijs Verdoes Kleijn <verdoes at astro.rug.nl>, kids at astro-wise.org, Nicola R. Napolitano <napolita at na.astro.it>, Aniello Grado <agrado at na.astro.it>, aniello grado <aniello.grado at gmail.com>, Jelte de Jong <jelte at strw.leidenuniv.nl>
Inviato: Wed, 10 Apr 2013 18:50:19 +0200 (CEST)
Oggetto: Re: visual grading and manual mask production

Hi Mario,

Our focus should be on doing whatever is necessary for the data release. 
The mask separation is not strictly necessary for that, so by that 
definition: not urgent ;).

Supposedly it should be possible to make these files with Joey's scripts 
since he has created them before. (The 'all' files were indeed either 
combined 'starhalobpixman' or automatic 'starhalobpix' masks in earlier 
runs.)

Hugo

On 04/10/2013 06:28 PM, Mario Radovich wrote:
> Dear Hugo
>
> Automatic masks are available as FITS, but not manual masks at the
> moment (only the combined automatic+manual). I think that it should
> not be difficult modify Joey's scripts to do that: I can try to ask him,
> or give a look myself (not before next week, I am afraid...). Is that
> urgent ?
>
> Cheers
>
> Mario
>
>
> On 04/10/2013 06:22 PM, Hugo Buddelmeijer wrote:
>> Hi Mario,
>>
>> Okay, it wasn't clear to me that the *.msk.v1.0.all.* are the combined
>> masks.
>>
>> Then my question becomes: are the automatic and manual masks (as FITS)
>> available as well? [I suppose these could be reconstructed from the
>> combined masks..]
>>
>> Within Astro-WISE everything has full data lineage. That is, we ingest
>> the combined masks with references to the automatic and manual masks
>> from which it is created. Therefore, it is (formally) not possible to
>> ingest the combined masks without its dependencies. I can work around
>> this, but prefer not to, because that way it is not clear how the
>> masks are created, defeating purpose of the data lineage.
>>
>> E.g. these files, but for the latest run of KIDSCAT:
>> http://vstportal.oacn.inaf.it/KIDSCAT/ESO_DR1/Masks/
>>
>> Hugo
>>
>> On 04/10/2013 06:07 PM, Mario Radovich wrote:
>>> Dear Hugo
>>>
>>> the *.msk.v1.0.all.* are the automatic masks including the manual masks,
>>> when available.
>>>
>>> Cheers
>>>
>>> Mario
>>>
>>>
>>>
>>> On 04/10/2013 05:52 PM, Hugo Buddelmeijer wrote:
>>>> Thanks Francesco! Great work
>>>>
>>>> The files are being downloaded as we speak and I will start ingesting
>>>> them tomorrow.
>>>>
>>>> Are all the masks available as well? I can only find the
>>>> *.msk.v1.0.all.* ones. As per the naming convention these are only the
>>>> automatic masks, are there also manual masks and combined masks for
>>>> the ones where manual regions files have been created?
>>>>
>>>> [Perhaps I misunderstood things. I was under the impression that we
>>>> would create catalogs using the manual masks when available.]
>>>>
>>>>
>>>> A telecon tomorrow (see the rest of the mails) would be fine with me
>>>> if necessary. Preferably after 13:30.
>>>>
>>>> My most relevant contribution would be that I would prefer it if any
>>>> new files will be in new directories. That is, that it is certain that
>>>> a file at a specific url has to be downloaded only once because it
>>>> will never change. That makes bookkeeping for me much easier.
>>>>
>>>> Hugo
>>>>
>>>>
>>>> On 04/10/2013 02:53 PM, Francesco La Barbera wrote:
>>>>> Dear All,
>>>>>
>>>>> I'm happy to inform you that the new sourcelists have been produced
>>>>> now.
>>>>>
>>>>> Hugo, you can access them for AW ingestion at the same link as last
>>>>> time, i.e.
>>>>>
>>>>> http://vstportal.oacn.inaf.it/KIDSCAT/ESO_DR1_kc1.6/
>>>>>
>>>>> I checked that all files are in place, and there was no evident issue
>>>>> with all different data products.
>>>>> The sourcelists have been generated with the new, latest, masks passed
>>>>> me by Mario and Nicola,
>>>>> and using KIDS-CAT v.1.6 (that includes sat mag and ellipticity
>>>>> estimates).
>>>>>
>>>>> Pls, lemme know if everything is OK with this.
>>>>>
>>>>> Cheers,
>>>>> Francesco
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Tue, Apr 9, 2013 at 2:58 PM, Gijs Verdoes Kleijn
>>>>> <verdoes at astro.rug.nl <mailto:verdoes at astro.rug.nl>> wrote:
>>>>>
>>>>> Dear Nicola and Mario and Hugo,
>>>>>
>>>>> How is the automated mask + sourcelist production going?
>>>>>
>>>>> I am now coordinating the visual grading and manual mask
>>>>> production.
>>>>> My plan is to have that team start to work on all coadds at
>>>>> is_valid=1 and 2 asap.
>>>>> (NB: all coadds that will be re-processed this week are now set to
>>>>> is_valid=0)
>>>>>
>>>>> As far as I see this effort is decoupled (enough) from your
>>>>> mask+sourcelist effort and there can start now. I will tell the
>>>>> manual mask team to anticipate improvement on the current
>>>>> automated
>>>>> masks. And hence start no manually improving those.
>>>>>
>>>>> Please let me know if you see problems with this decoupled
>>>>> parallel
>>>>> effort.
>>>>>
>>>>> Best wishes, Gijs
>>>>>
>>>>>
>>>>>
>>>>> On 04/08/2013 11:04 PM, Nicola R. Napolitano wrote:
>>>>>>
>>>>>> OK Lino. Thanks
>>>>>>
>>>>>> N
>>>>>>
>>>>>>
>>>>>>
>>>>>> On 4/8/13 8:41 PM, aniello grado wrote:
>>>>>>> Hi Nicola,
>>>>>>>
>>>>>>> the manual masks are on beo3 in the dir:
>>>>>>>
>>>>>>> /opt/AW/current/astro/experimental/kids/regionfiles
>>>>>>>
>>>>>>> Lino
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Apr 8, 2013 at 3:37 PM, Nicola R. Napolitano
>>>>>>> <napolita at na.astro.it <mailto:napolita at na.astro.it>> wrote:
>>>>>>>
>>>>>>>
>>>>>>> Hi Lino,
>>>>>>>
>>>>>>> automatic masking is about to start. Mario and I are still
>>>>>>> struggling with the g-band but we should be able to work
>>>>>>> this
>>>>>>> out by the end of the day.
>>>>>>> Meanwhile r,i,u are ready to go after we have checked the
>>>>>>> spike length, which needs just some more very short visual
>>>>>>> test.
>>>>>>> This means that auto-masks will be possibly ready tomorrow
>>>>>>> morning.
>>>>>>>
>>>>>>> To start the sourcelists extraction, though, we need to
>>>>>>> incorporate the manual masks into the auto masks, since the
>>>>>>> idea is to provide a unique flag in the catalogs.
>>>>>>> Thus we need to know whether the final manual masks are also
>>>>>>> ready for the download. If so please give to Mario the
>>>>>>> necessary references for that.
>>>>>>>
>>>>>>> Cheers,
>>>>>>>
>>>>>>> Nicola
>>>>>>>
>>>>>>> On 4/6/13 3:55 PM, aniello grado wrote:
>>>>>>>> Hi Mario and Nicola,
>>>>>>>>
>>>>>>>> the coadds are ready for automatic masking.
>>>>>>>>
>>>>>>>> Lino
>>>>>>>>
>>>>>>>> ---------- Forwarded message ----------
>>>>>>>> From: <jelte at strw.leidenuniv.nl
>>>>>>>> <mailto:jelte at strw.leidenuniv.nl>>
>>>>>>>> Date: Sat, Apr 6, 2013 at 2:24 PM
>>>>>>>> Subject: Re: reprocessing and is_valid = 1 coadds
>>>>>>>> To: Gijs Verdoes Kleijn <verdoes at astro.rug.nl
>>>>>>>> <mailto:verdoes at astro.rug.nl>>, Lino Grado
>>>>>>>> <agrado at na.astro.it <mailto:agrado at na.astro.it>>
>>>>>>>>
>>>>>>>>
>>>>>>>> Hi Gijs and Lino,
>>>>>>>>
>>>>>>>> just to let you know that reprocessing is done now.
>>>>>>>> We now have again 200 DR1 coadds:
>>>>>>>>
>>>>>>>> awe> query = Comment.content == 'KiDS-ESO-DR1-Coadd'
>>>>>>>> awe> list =
>>>>>>>> [CoaddedRegriddedFrame(object_id=c.db_object_id)
>>>>>>>> for c in query]
>>>>>>>> awe> len(list)
>>>>>>>> 200
>>>>>>>>
>>>>>>>> The total number of reprocessed coadds is 50:
>>>>>>>> awe> query = Comment.content ==
>>>>>>>> 'KiDS-ESO-DR1-Coadd-Reprocessed-After-QA1'
>>>>>>>> awe> list =
>>>>>>>> [CoaddedRegriddedFrame(object_id=c.db_object_id)
>>>>>>>> for c in query]
>>>>>>>> awe> len(list)
>>>>>>>> 50
>>>>>>>>
>>>>>>>> Querying for the comment 'KiDS-ESO-DR1-Coadd-DISCARDED'
>>>>>>>> will
>>>>>>>> also give 50
>>>>>>>> coadds, namely the ones replaced by the reprocessed coadds.
>>>>>>>>
>>>>>>>> From the 74 is_valid=0 coadds, 24 were not reprocessed
>>>>>>>> since
>>>>>>>> according to
>>>>>>>> the reprocessing teams there was no improvement possible.
>>>>>>>> These coadds
>>>>>>>> that are retained in DR1 can be recognized by the comment
>>>>>>>> 'KiDS-ESO-DR1-Coadd-Retained-After-QA1':
>>>>>>>>
>>>>>>>> awe> query = Comment.content ==
>>>>>>>> 'KiDS-ESO-DR1-Coadd-Retained-After-QA1'
>>>>>>>> awe> list =
>>>>>>>> [CoaddedRegriddedFrame(object_id=c.db_object_id)
>>>>>>>> for c in query]
>>>>>>>> awe> len(list)
>>>>>>>> 24
>>>>>>>>
>>>>>>>> Cheers,
>>>>>>>> Jelte
>>>>>>>>
>>>>>>>> > Hi Gijs and Lino,
>>>>>>>> >
>>>>>>>> > reprocessing is almost done.
>>>>>>>> > There were still 3 coadds that needed to be reprocessed
>>>>>>>> and that is
>>>>>>>> > currently being done.
>>>>>>>> > Over the weekend I will update their comments etc, and
>>>>>>>> notify Mario so
>>>>>>>> > he can download the new ones for source extraction.
>>>>>>>> >
>>>>>>>> > I have also put a table on the wiki with the coadds that
>>>>>>>> were set to
>>>>>>>> > is_valid = 1 during the visual inspection (QA1):
>>>>>>>> >
>>>>>>>> http://wiki.astro-wise.org/projects:kids:data_deliveries:kids-eso-dr1:productiondashboard#coadd_set_to_is_valid_1_during_qa1
>>>>>>>>
>>>>>>>>
>>>>>>>> > Also included in the table (where it was clear to me) is
>>>>>>>> the reasons why
>>>>>>>> > the visual inspection team thinks these coadds could be
>>>>>>>> improved if
>>>>>>>> > there is enough time.
>>>>>>>> > We should set up a dedicated team that can have a look at
>>>>>>>> these coadds
>>>>>>>> > next week to see which ones can be improved by
>>>>>>>> reprocessing.
>>>>>>>> > In any case our goal should be to have all final
>>>>>>>> coadds by
>>>>>>>> next Friday
>>>>>>>> > (April 12).
>>>>>>>> >
>>>>>>>> > Cheers,
>>>>>>>> > Jelte
>>>>>>>> >
>>>>>>>> > --
>>>>>>>> > Dr. Jelte T. A. de Jong
>>>>>>>> > KiDS project manager
>>>>>>>> > Sterrewacht Leiden
>>>>>>>> > Leiden University, Leiden, the Netherlands
>>>>>>>> > E: jelte at strw.leidenuniv.nl
>>>>>>>> <mailto:jelte at strw.leidenuniv.nl>
>>>>>>>> > T: +31-(0)715275818 <tel:%2B31-%280%29715275818>
>>>>>>>> > W: jelte.jdejong.net <http://jelte.jdejong.net>
>>>>>>>> >
>>>>>>>> >
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> <<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>
>>>>>>>>
>>>>>>>> Aniello Grado, PhD
>>>>>>>>
>>>>>>>> Ricercatore Astronomo
>>>>>>>> INAF- Osservatorio Astronomico di Capodimonte
>>>>>>>> Via Moiariello 16 80131 Napoli
>>>>>>>> ITALIA
>>>>>>>> Tel: 0039 0815575547 <tel:0039%200815575547>
>>>>>>>> Fax: 0039 0815575433 <tel:0039%200815575433>
>>>>>>>>
>>>>>>>> <<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Nicola R. Napolitano
>>>>>>> Research Astronomer
>>>>>>> INAF - Observatory of Capodimonte
>>>>>>> Salita Moiariello, 16, 80131, Napoli (Italy)
>>>>>>> phone:+39 081 5575509 <tel:%2B39%20081%205575509>; fax:+39
>>>>>>> 081 456710 <tel:%2B39%20081%20456710>
>>>>>>> email:napolita at na.astro.it <mailto:napolita at na.astro.it>
>>>>>>> web page:http://www.na.astro.it/~napolita
>>>>>>> <http://www.na.astro.it/%7Enapolita>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> <<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>
>>>>>>>
>>>>>>> Aniello Grado, PhD
>>>>>>>
>>>>>>> Ricercatore Astronomo
>>>>>>> INAF- Osservatorio Astronomico di Capodimonte
>>>>>>> Via Moiariello 16 80131 Napoli
>>>>>>> ITALIA
>>>>>>> Tel: 0039 0815575547 <tel:0039%200815575547>
>>>>>>> Fax: 0039 0815575433 <tel:0039%200815575433>
>>>>>>>
>>>>>>> <<<<<<<<<<<<<<<<<<<<>>>>>>>>>>>>>>>>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> Nicola R. Napolitano
>>>>>> Research Astronomer
>>>>>> INAF - Observatory of Capodimonte
>>>>>> Salita Moiariello, 16, 80131, Napoli (Italy)
>>>>>> phone:+39 081 5575509 <tel:%2B39%20081%205575509>; fax:+39 081
>>>>>> 456710 <tel:%2B39%20081%20456710>
>>>>>> email:napolita at na.astro.it <mailto:napolita at na.astro.it>
>>>>>> web page:http://www.na.astro.it/~napolita
>>>>>> <http://www.na.astro.it/%7Enapolita>
>>>>>
>>>>> --
>>>>> --------------------------------------------------------------------
>>>>> |dr Gijs Verdoes Kleijn | astronomer |
>>>>> |e-mail:verdoes at astro.rug.nl <mailto:verdoes at astro.rug.nl> |
>>>>> OmegaCEN / Kapteyn Institute / |
>>>>> |www:www.astro.rug.nl/~verdoes
>>>>> <http://www.astro.rug.nl/~verdoes> | Target |
>>>>> |tel:+31-50-3638326 <tel:%2B31-50-3638326> |
>>>>> University of Groningen |
>>>>> |mobile:+31-654658050 <tel:%2B31-654658050> | postal
>>>>> address: |
>>>>> | | Kapteyn Astronomical
>>>>> Institute |
>>>>> | | Postbus 800, 9700 AV,
>>>>> Groningen|
>>>>> | | The
>>>>> Netherlands |
>>>>> --------------------------------------------------------------------
>>>>>
>>>>>
>>>>
>>