Using Parallel Python Tools to Postprocess Data for CMIP6

During CMIP5, NCAR was the first group to finish running our model simulations, but we were the last group to finish publication. This was the result of our post-processing tool chain being serial and there was no automation between tasks. As we looked ahead to CMIP6, it was estimated that we would need to post-process 30x the amount of data within a shorter time frame than in CMIP5. In order to meet these demands, we have been redesigning our workflow and creating new parallel Python tools that have increased our throughput. We have also been working on automating our workflow to eliminate the time between tasks. This presentation will discuss how we are using parallel Python to increase our throughput and the Python tools we will be using for our CMIP6 experiments.