Use of ASSISTT's Computing Cluster for R2O

The Algorithm Scientific Software Integration and System Transition Team’s (ASSISTT) computing cluster at NOAA/NESDIS/STAR hosts a wide variety of processing geared towards the generation of satellite level 2 (L2) meteorological products in the effort to support the research to operations (R2O) mission. Continuous delivery of products is dependent on the collaborative work of multiple teams to ensure efficient product generation in a timely manner. Currently, the ASSISTT cluster is employed to produce algorithm output in near-real time (NRT), run validation tests to ensure the quality of output, and to conduct regression tests. The cluster handles output production for a variety of satellites, such as the Himawari AHI, GOES-16 ABI, and JPSS VIIRS from the Suomi-NPP and NOAA-20 satellites. The NRT L2 product generation aids the detection of potential routine processing issues before algorithms are transitioned to operations. In addition, specific product validation runs provide the long-range of data needed for algorithms to advance through the Satellite Product and Services Review Board (SPSRB) reviews such as the Test Readiness Review and the Algorithm Readiness Review. Regression testing of the STAR Algorithm Processing Framework (SAPF) ensures that the software performs to the expected standards after any updates. ASSISTT has developed three types of software to manage cluster tasks in a timely and efficient manner: data staging, cluster management, and cluster processing. The data staging software ensures that all the required inputs are ready before a task is run by the cluster. Cluster management software controls the flow of tasks on the cluster. Finally, cluster processing software is run by the cluster to produce the L2 data, using tools such as SAPF. This software allows for a large volumes of data to be processed by the cluster each day. All developed software is validated to ensure the software can handle any anomalies that may arise when processing large amounts of data, and in the case of NRT, that latency requirements are being met. A more in-depth overview of these three processes and how the STAR Operational Cluster is utilized to handle them will be discussed.