NWS GOES-16 Operational Test and Evaluation (OT&E) and Forecaster User Survey Results

The National Weather Service (NWS) Office of Planning & Programming for Service Delivery (OPPSD) Systems Engineering, Integration, and Test (SEIT) Branch is responsible for the planning, conducting, and reporting the results of the GOES-16 NWS OT&E, and, in general, any OT&E conducted within the NWS at Weather Forecast Offices (WFO’s), River Forecast Centers (RFC’s), and National Centers (NC’s), in an operational environment prior to full national NWS deployment.

The NWS uses a life cycle model for the implementation of new or modified operational products to the NWS Field sites from the design phase, through the operations validation stage, production and deployment phase, to the operations and support stage. There are two major test phases in a program; Development Test and Evaluation (DT&E) and OT&E. During the DT&E phase, the system is tested to verify that the system requirements were satisfied. But with OT&E, we validate that the system meets the stakeholder’s needs and it is ready to support an operational mission. The OPPSD/SEIT branch is responsible for the operations validation stage or the OT&E for capabilities to be implemented into the NWS Mission.

The NWS GOES-16 OT&E was conducted in three “dynamic” phases at seven NWS sites, six WFO’s and one RFC from each of the six NWS regions, from March 2017-April 2018. The products received from NESDIS go through three major phases for operational readiness; Beta Validated Products (Post-Launch Beta Testing), Provisionally Validated Products (Post-Launch Provisional Testing), and Fully Validated Products (Post-Launch Full Validation Testing). Phase 1 was conducted from March -June 2017 with the Visible Infrared Imaging Radiometer Suite (VIIRS) Sectorized Cloud and Moisture Imagery (SCMI), beta validated imagery. Phase 2 was conducted from June-November 2017 with beta validated products flowing through the Product Distribution and Access (PDA) system producing derived products. Phase 3 was conducted from December 2017-April 2018 with provisionally validated SCMI imagery and PDA derived products.

The main objectives of the GOES-16 OT&E were, to validate end-to end delivery of SCMI Imagery and PDA derived products from NESDIS to NWS WFO’s and RFC’s through the Satellite Broadcast Network (SBN) into the AWIPS CAVE terminals. The OT&E also included executing test procedures at the OT&E sites and at NWS headquarters, tracking of the end-to-end latency and bandwidth, and the use of the SCMI Imagery and PDA products in an operational environment.

SEIT closely worked with the NWS Operations Proving Ground (OPG), NESDIS, and the WFO’s and RFC’s to resolve any issues that were found during the OT&E. A Test Review Group (TRG) consisting of subject matter experts from the NWS, NESDIS, OPG, and WFO’s met bi-weekly to discuss major issues and problems encountered during the OT&E. Issues were tracked with Test Trouble Reports (TTR’s) in a database shared with all users. The OPG represent NWS field offices in the Research to Operations (R2O) process by conducting pre-deployment readiness evaluations of promising new science and technology innovations in a realistic operational environment.

After the completion of the OT&E phases, SEIT asked the OT&E sites to fill out a GOES-16 OT&E Forecaster User Survey using Google Forms on-line. This survey was unique in that it was the first time the NWS and NESDIS had survey results from actual forecasters/users during an OT&E process on satellites. The results were very useful in summarizing the results of the OT&E, and summarizing the OT&E sites operational use of the GOES-16 Imagery and PDA derived products, particularly which products were most useful in an operational environment, and which products were not as useful. Survey results also included suggestions for improvement in the future.

The results of each phase of the OT&E, and recommendations for operational implementation were passed on to the NWS management for the final decision on whether to deploy the SCMI Imagery and PDA products operationally at all NWS WFO’s and RFC’s.

This paper summarizes the GOES-16 OT&E process (including OT&E prerequisites, test objectives and plan), and assessment of GOES-16 OT&E results (including the forecast field office site user survey results, test issues and resolution).