AUDI jobs operating on the same dataset will result in some data products having identical file names and what to do in this situation needs to be addressed.
Continuum/Cube products:
We should avoid determining the duplicity of a product by the filename because AUDI products will have the same file names, but possibly different angular resolution, pipeline versions, spectral coverage, etc. Instead, should focus on comparing metadata. The metadata of the new product should be compared to the metadata of all products contained within the archive.
Header keywords to check for the same values, which if all equal would mean the product is the same. Values are meant as examples and will not reflect the values that all datasets will have. If the new product does not contain all header fields as products already in archive (and vice-versa) it should be regarded as a new product. If metadata are identical, products are identical and should not be ingested.
NAXIS = 4
NAXIS1 = 1728
NAXIS2 = 1728
NAXIS3 = 1
NAXIS4 = 1
BMAJ = 3.401172359470E-05
BMIN = 2.806701517786E-05
BPA = -4.208995788542E+01
BTYPE = 'Intensity'
OBJECT = 'GW_Ori '
BUNIT = 'Jy/beam ' /Brightness (pixel) unit
CTYPE1 = 'RA---SIN'
CRVAL1 = 8.228496247106E+01
CDELT1 = -7.222222217153E-06
CRPIX1 = 8.650000000000E+02
CUNIT1 = 'deg '
CTYPE2 = 'DEC--SIN'
CRVAL2 = 1.187020728400E+01
CDELT2 = 7.222222217153E-06
CRPIX2 = 8.650000000000E+02
CUNIT2 = 'deg '
CTYPE3 = 'FREQ '
CRVAL3 = 2.248650462272E+11
CDELT3 = 1.810101696316E+10
CRPIX3 = 1.000000000000E+00
CTYPE4 = 'STOKES '
CRVAL4 = 1.000000000000E+00
CDELT4 = 1.000000000000E+00
CRPIX4 = 1.000000000000E+00
CUNIT4 = ' '
RESTFRQ = 2.305380000000E+11
SPECSYS = 'LSRK'
TELESCOP= 'ALMA '
DATE-OBS= '2017-12-10T04:53:09.359999'
PIPEVER = '43130 (Pipeline-CASA56-P2-B)'
CASAVER = '5.6.2-3 '
SPECMODE= 'cube '
New version of same product?
We can also attempt to define criteria where a product would be a new version of the same product. This would attempt to keep products where some metadata change by very small values due to heuristic adjustments within the imaging pipelines, but are not fundamentally different.
Fields that should be identical:
NAXIS = 4
SPECMODE= 'cube '
TELESCOP= 'ALMA '
DATE-OBS= '2017-12-10T04:53:09.359999'
BTYPE = 'Intensity'
OBJECT = 'GW_Ori '
BUNIT = 'Jy/beam ' /Brightness (pixel) unit
SPECSYS = 'LSRK'
CTYPE1 = 'RA---SIN'
CUNIT1 = 'deg '
CTYPE2 = 'DEC--SIN'
CUNIT2 = 'deg '
CTYPE3 = 'FREQ '
CTYPE4 = 'STOKES '
CUNIT4 = ' '
RESTFRQ = 2.305380000000E+11
Fields same within 5%:
BMAJ = 3.401172359470E-05
BMIN = 2.806701517786E-05
BPA = -4.208995788542E+01
NAXIS1 = 1728
NAXIS2 = 1728
NAXIS3 = 1
NAXIS4 = 1
CRVAL1 = 8.228496247106E+01
CDELT1 = -7.222222217153E-06
CRPIX1 = 8.650000000000E+02
CRVAL2 = 1.187020728400E+01
CDELT2 = 7.222222217153E-06
CRPIX2 = 8.650000000000E+02
CRVAL3 = 2.248650462272E+11
CDELT3 = 1.810101696316E+10
CRPIX3 = 1.000000000000E+00
May or may not be the same:
PIPEVER = '43130 (Pipeline-CASA56-P2-B)'
CASAVER = '5.6.2-3 '
Step-by-step guide
Related articles
ORIGIN = 'CASA 5.6.2-3'