The MUSICA IASI processing chain has been developed within several third-party funded projects during the last ten years. Here we propose to make this chain much more efficient in order to be able to process data in a timely and reliable manner and to reprocess data at a vast scale. The improved chain will guarantee a sustained, timely and reliable supply of highly consistent MUSICA IASI data. We expect the project to cause a substantial impact on atmospheric water and greenhouse gas cycle research, which are two very decisive fields of climate research (e.g., https://www.wcrp-climate.org/grand-challenges/grand-challenges-overview). In addition, the project will serve as a testbed for the processing of spectra from the upcoming Metop-SG mission, which will involve even more computational challenges (increased number of observations and higher resolution of measured spectra).
Using the reprocessing mode of the efficient chain, we will be able to document the dependency of the computational performance on different elements (code efficiency, processor architectures, usage of CPUs or GPUs, workflow management). This can serve as a guideline for other communities where large-scale data processing is essential and continuous performance improvements are crucial.

Figure: Flow chart of the MUSICA IASI processing chain (for details, refer to Schneider et al., 2021a). Blue symbols mark the six different processing components: (1) Preprocessing Stage, (2) PROFFIT-nadir Retrieval, (3) Output Generation, (4) A Posteriori Data Re-usage, (5) Regridding, and (6) Retrieval Simulator. The components (5) and (6) are not directly linked to the output products (green symbols) and are not relevant for this project.