Drilling down from Python Statistical Analyses to Rich Interactive Case Study Visualizations, within Jupyter Notebooks

Python (and its spinoff Jupyter) are great like duct tape: for connecting existing valuable software in other languages into a greater System or "stack".

In this EarthCube project (see https://brianmapes.github.io/EarthCube-DRILSDOWN/), we have coupled Jupyter notebooks running the iPython shell with the powerful GUI-driven data integration and visualization powers of the Java-based Integrated Data Viewer (IDV). Python widget extensions further make the system GUI-operable for discoverability of resources, or of course everything can be done as code with magic commands. The resulting scientific digital objects (Jypyter notebooks containing Python code and outputs, and IDV-derived imagery and IDV state bundles for reproducibility) can be stored and trafficked in a service-rich browser-accessible RAMADDA repository.

The larger vision is to serve the Visualization for Algorithm Design paradigm: Statistical analyses lead to curiosity about underlying details, DRILSDOWN uses the brain's visual hardware to see what the algorithm is doing, and resulting algorithm improvements can be cycled back into production codes -- all wrapped in replicable, provenanced objects (the notebooks).