AUDI jobs operating on the same dataset will result in some data products having identical file names and what to do in this situation needs to be addressed.
We should avoid determining the duplicity of a product by the filename because AUDI products will have the same file names, but possibly different angular resolution, pipeline versions, spectral coverage, etc. Instead, should focus on comparing metadata. The metadata of the new product should be compared to the metadata of all products contained within the archive.
Header keywords to check for the same values, which if all equal would mean the product is the same. Values are meant as examples and will not reflect the values that all datasets will have. If the new product does not contain all header fields as products already in archive (and vice-versa) it should be regarded as a new product. If metadata are identical, products are identical and should not be ingested.
NAXIS = 4
NAXIS1 = 1728
NAXIS2 = 1728
NAXIS3 = 1
NAXIS4 = 1
BMAJ = 3.401172359470E-05
BMIN = 2.806701517786E-05
BPA = -4.208995788542E+01
BTYPE = 'Intensity'
OBJECT = 'GW_Ori '
BUNIT = 'Jy/beam ' /Brightness (pixel) unit
CTYPE1 = 'RA---SIN'
CRVAL1 = 8.228496247106E+01
CDELT1 = -7.222222217153E-06
CRPIX1 = 8.650000000000E+02
CUNIT1 = 'deg '
CTYPE2 = 'DEC--SIN'
CRVAL2 = 1.187020728400E+01
CDELT2 = 7.222222217153E-06
CRPIX2 = 8.650000000000E+02
CUNIT2 = 'deg '
CTYPE3 = 'FREQ '
CRVAL3 = 2.248650462272E+11
CDELT3 = 1.810101696316E+10
CRPIX3 = 1.000000000000E+00
CTYPE4 = 'STOKES '
CRVAL4 = 1.000000000000E+00
CDELT4 = 1.000000000000E+00
CRPIX4 = 1.000000000000E+00
CUNIT4 = ' '
RESTFRQ = 2.305380000000E+11
SPECSYS = 'LSRK'
TELESCOP= 'ALMA '
DATE-OBS= '2017-12-10T04:53:09.359999'
PIPEVER = '43130 (Pipeline-CASA56-P2-B)'
CASAVER = '5.6.2-3 '
SPECMODE= 'cube '
We can also attempt to define criteria where a product would be a new version of the same product. This would attempt to keep products where some metadata change by very small values due to heuristic adjustments within the imaging pipelines, but are not fundamentally different.
Fields that should be identical:
NAXIS = 4
SPECMODE= 'cube '
TELESCOP= 'ALMA '
DATE-OBS= '2017-12-10T04:53:09.359999'
BTYPE = 'Intensity'
OBJECT = 'GW_Ori '
BUNIT = 'Jy/beam ' /Brightness (pixel) unit
SPECSYS = 'LSRK'
CTYPE1 = 'RA---SIN'
CUNIT1 = 'deg '
CTYPE2 = 'DEC--SIN'
CUNIT2 = 'deg '
CTYPE3 = 'FREQ '
CTYPE4 = 'STOKES '
CUNIT4 = ' '
RESTFRQ = 2.305380000000E+11
Fields same within 5%:
BMAJ = 3.401172359470E-05
BMIN = 2.806701517786E-05
BPA = -4.208995788542E+01
NAXIS1 = 1728
NAXIS2 = 1728
NAXIS3 = 1
NAXIS4 = 1
CRVAL1 = 8.228496247106E+01
CDELT1 = -7.222222217153E-06
CRPIX1 = 8.650000000000E+02
CRVAL2 = 1.187020728400E+01
CDELT2 = 7.222222217153E-06
CRPIX2 = 8.650000000000E+02
CRVAL3 = 2.248650462272E+11
CDELT3 = 1.810101696316E+10
CRPIX3 = 1.000000000000E+00
May or may not be the same:
PIPEVER = '43130 (Pipeline-CASA56-P2-B)'
CASAVER = '5.6.2-3 '
Add the steps involved:
| You may also want to use visual panels to communicate related information, tips or things users need to be aware of. |
Related articles appear here based on the labels you select. Click to edit the macro and add or change labels.
|
ORIGIN = 'CASA 5.6.2-3'