SparkMonitor

Final Report | Installation | How it Works | Use Cases | Code | License

SparkMonitor - How the extension works

Jupyter Working

Jupyter Notebook is a web based application that follows a client-server architecture. It consists of a JavaScript browser client that renders the notebook interface and a web server process on the back end. The computation of the cells are outsourced to a separate kernel process running on the server. To extend the notebook, it is required to implement a separate extension component for each part.

The SparkMonitor extension for Jupyter Notebook has 4 components.

  1. Notebook Frontend extension written in JavaScript.
  2. IPython Kernel extension written in Python.
  3. Notebook web server extension written in Python.
  4. An implementation of SparkListener interface written in Scala.

The Frontend Extension

The Monitoring Display


IPython Kernel Extension

Kernel Extension


Scala SparkListener

SparkListener


The Notebook Webserver Extension - A Spark Web UI proxy

The Spark UI