SSA-6282 - Getting issue details... STATUS
Results
https://pypi.org/project/fitsio/
fitsio is a python module so it integrates into archive ingestion very well.
Some cons for using it:
- pip install of fitsio required python3.7
- I don't find it as easy to use as astropy.io.fits although I've had much more exerience using that library
- Would require significant code changes and regression testing. Essentially a re-write of ingestion
- The benefits may be somewhat sparse (see pros)
The pros for using it:
- It allows reading a subset of the data in the fits file (filtered by row, column) which might solve and issue with reading large file.
- So far I haven't identified any data that can't be extracted although the path is somewhat less straightforward than astropy fits
Plan
The primary purpose for using this is if fitsio can handle large files that astropy can't. Current, we have 47 VLBA files that are DEFERRED for file size. Given the number, an intermediate solution might be to create a utility that would do those ingestions as one-offs using a fitsio-based script. if this turns out to be a more prevalant issue, that script should be designed so that the fitsio data extraction is implemented in a way that would make it easier to plug into overall ingestion.
Action Item
Rick Lively [X] Retrieve one of the DEFERRED files and see if the data can be loaded with fitsio