[KIDS] INTDR2 catalog ingestion

Fri Nov 23 12:33:31 CET 2012

Dear Hugo,

thanks for the great work to ingest the catalogs. Here goes the answer to
your questions:

On Fri, Nov 23, 2012 at 12:20 PM, Hugo Buddelmeijer <buddel at astro.rug.nl>wrote:

> Hi Nicola,
>
> As I understood it, there is a first Sextractor run to measure the
> SEEING_FWHM of the final Sextractor run. It would be useful to ingest these
> as well. However this does not have priority.
>

> Furthermore, I'm not sure which catalog this would be. For every final
> catalog there are two earlier runs, with the extensions ".cat" and ".cat.1"
> in the KIDSCAT_INTDR2_52FIELDS directory. From the KIDSCAT readme I figured
> that the ".cat.1" catalog is the FWHM catalog, correct?

The .cat.1 files are exactly the catalogs produced during first-run, i.e.
there is no FWHM optimization, and no 2DPHOT flag column in the catalogs.
Pls, lemme know if you need any further info/some modification about this
stuff.

> What is the difference with the "<ObservingBlockName>.cat" and
> "<CoaddFilename>.cat"?
>
>
They are the same. I just kept both at the moment, as I was not sure on the
best way to rename the catalog files for ingestion.
I will remove "<ObservingBlockName>.cat" files in the forthcoming versions,
OK?

Thanks!!

Francesco

Greetings,
> Hugo
>
>
>
> On 23/11/12 12:10, Nicola R. Napolitano wrote:
>
>>
>> Hi Hugo,
>>
>> I'm navigating within the catalogs. It works nice! Well done.
>>
>> I'm looking at the todo list: what do you mean by intermediate catalogs?
>>
>> Cheers,
>>
>> Nicola
>>
>>
>> On 11/23/12 11:58 AM, Hugo Buddelmeijer wrote:
>>
>>> Hi Nicola et al.,
>>>
>>> This night I've done a first ingestion of all the 52 catalogs made
>>> with KIDSCAT beta v1.0. They can be found by querying for SourceLists
>>> with the name "INTDR2test_*", like:
>>> awe> slsintdr2 = SourceList.name.like("**INTDR2test_*")
>>> awe> len(slsintdr2)
>>> 52
>>> Or through the dbviewer:
>>> http://socuteurl.com/**funnytumble <http://socuteurl.com/funnytumble>
>>>
>>> They have 'test' in the name because they are simply a conversion from
>>> the .cat files to AW SourceLists. No thorough assessment of the
>>> catalogs are done yet. However, they seem okay, and we will start
>>> using them here.
>>>
>>> What has not yet been done:
>>> - Ingesting the sexconf/sexparams (first thing to do)
>>> - Ingestion of the masks (the weights used are from AW right?)
>>> - Ingestion of the intermediate catalogs
>>> - Ingestion(/recreation) of QC plots
>>>
>>> What happened to the columns:
>>> - FLUX_APER and FLUXERR_APER are renamed to FLUX_APER_<aperturesize>
>>> and FLUXERR_APER_<aperturesize>. So FLUX_APER_10 is the flux in a
>>> 10 arcsecond aperture.
>>> - The VIGNET columns have been removed.
>>> - Some columns are renamed according to the FITSCOLUMN_NAMES dictionary
>>> in SourceList.py:
>>>
>>> http://www.astro-wise.org/**viewvc/awe/astro/main/**
>>> SourceList.py?annotate=1.228#**l2788<http://www.astro-wise.org/viewvc/awe/astro/main/SourceList.py?annotate=1.228#l2788>
>>>
>>> - The HTM column is added.
>>>
>>> I will improve the catalog ingestion, starting with the sexconf, since
>>> this will improve the data lineage and reproducebility. If there is
>>> anything wrong, unclear or missing, please notify me!
>>>
>>> Greetings,
>>> Hugo
>>>
>>> P.S. I moved this discussion to kids at astro-wise.org, that way everyone
>>> gets them. Contact Gijs if you're not on this list yet.
>>>
>>>
>>>
>>> On 21/11/12 10:48, Nicola R. Napolitano wrote:
>>>
>>>>
>>>> I forgot to include Francesco in the mailing list....reply to this
>>>> message if needed.
>>>>
>>>> Cheers,
>>>>
>>>> Nicola
>>>>
>>>>> Hi everyone,
>>>>>
>>>>> the catalogs for the 52 coadds selected for the INT-DR2 have been
>>>>> produced using KIDSCAT beta v1.0 and current products can be
>>>>> downloaded at the following link
>>>>>
>>>>> http://vstportal.oacn.inaf.it/**KIDSCAT/KIDSCAT_INTDR2_**52FIELDS/<http://vstportal.oacn.inaf.it/KIDSCAT/KIDSCAT_INTDR2_52FIELDS/>
>>>>>
>>>>> We have started to have a visual check to the catalogs in order to
>>>>> have a firs catalog QC before these are passed/ingested to AW for the
>>>>> QC of the coadds. In particular we are concentrating to the following
>>>>> checks:
>>>>>
>>>>> 1) look at the plot FWHM vs SNR in the PLOTS directory (Check that
>>>>> there is only one stellar sequence and that the spread is reasonably
>>>>> within a couple of pixels. Check that there are no weird features like
>>>>> clouds of points at low FWHM - e.g. satellite tracks?)
>>>>> 2) in the same directory there are now also the mag limit plots right
>>>>> implemented from Francesco (mag lim estimates are in
>>>>> OUTPUT_RES/MLIM.dat, see also Francesco's message);
>>>>> 3) look at a 2D X-Y plot of all sources versus the ones with col7
>>>>> (IMAFLAGS_ISO) <1 "bona fide" sources to see whether the distribution
>>>>> is uniform and cross-check with the fits image to see whether the
>>>>> major stars are correctly masked;
>>>>> 4) plot flux_radius (col9) vs mag_auto (col15) for an independent
>>>>> check of the stellar sequences;
>>>>> 5) report comments on the test above in a table with the following
>>>>> columns:
>>>>> #FIELD; BAND; FWHM vSNR; 2Dplot; 2Dflag0 plot; visual vs ds9;
>>>>> fluxRvsmag_A; extraComm
>>>>>
>>>>> I'll gather all you comments and I'll put in a table which will posted
>>>>> on the wiki
>>>>>
>>>>> http://wiki.astro-wise.org/**projects:kids:data_deliveries:**
>>>>> internaldelivery2.0_scratch#**quality_check_of_the_52_**
>>>>> catalog_for_int-dr2<http://wiki.astro-wise.org/projects:kids:data_deliveries:internaldelivery2.0_scratch#quality_check_of_the_52_catalog_for_int-dr2>
>>>>>
>>>>>
>>>>> Cheers,
>>>>>
>>>>> Nicola
>>>>>
>>>>> --
>>>>> Nicola R. Napolitano
>>>>> Research Astronomer
>>>>> INAF - Observatory of Capodimonte
>>>>> Salita Moiariello, 16, 80131, Napoli (Italy)
>>>>> phone: +39 081 5575509; fax: +39 081 456710
>>>>> email:napolita at na.astro.it <mailto:napolita at na.astro.it>
>>>>> web page:http://www.na.astro.it/~**napolita<http://www.na.astro.it/~napolita>
>>>>> <http://www.na.astro.it/%**7Enapolita<http://www.na.astro.it/%7Enapolita>
>>>>> >
>>>>>
>>>>
>>>>
>>>> --
>>>> Nicola R. Napolitano
>>>> Research Astronomer
>>>> INAF - Observatory of Capodimonte
>>>> Salita Moiariello, 16, 80131, Napoli (Italy)
>>>> phone: +39 081 5575509; fax: +39 081 456710
>>>> email:napolita at na.astro.it <mailto:napolita at na.astro.it>
>>>> web page:http://www.na.astro.it/~**napolita<http://www.na.astro.it/~napolita>
>>>>
>>>>
>>>
>>
>>
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: http://listman.astro-wise.org/pipermail/kids/attachments/20121123/dfc3ea78/attachment-0001.html 