{ "cells": [ { "cell_type": "markdown", "metadata": {}, "source": [ "# Navigating jupyter notebooks\n", "\n", "The typical workflow for of coffea uses [notebooks](https://jupyter.org/),\n", "which allows for rapid testing of code in development, as well as having a\n", "window that is can display analysis results without needing to switch a new\n", "window. The [installation and setup commands](../installation.html) already includes\n", "the command to initialize a server for notebooks. Here is mainly to help people\n", "who are unfamiliar with notebook to get started.\n", "\n", "## More information regarding starting a notebook server\n", "\n", "Notebooks servers generate the interface through network protocols, which can\n", "be access via your favorite browser. If you are running the commands one your\n", "personal machine, once you run the `jupy.sh` script, the command line should\n", "output something like:\n", "\n", "```\n", "http://127.0.0.1:8888/?token=62bfe8d8e84fe97f8da20\n", "```\n", "\n", "This line can be placed in your browsers URL bar to access the default jupyter\n", "interface. For security reasons, jupyter servers only allows network access\n", "from local connections. If you are using an LPC machine, you will need to setup\n", "ssh tunneling to pass traffic from the network port of your local machine to\n", "the LPC machine.\n", "\n", "```\n", "ssh -L localhost:8XXX:localhost:8XXX @\n", "```\n", "\n", "Or if you are using `~/.ssh/config` to make connection settings persistent:\n", "\n", "```\n", "Host \n", " user \n", " hostname \n", " LocalForward 8XXX :<8XXX>\n", "```\n", "\n", "Or if you are using the GUI based ssh connector, find the setting under the\n", "name of Connection/Tunneling or similar and set\n", "\n", "- source port: 8XXX\n", "- Destination: 127.0.0.1:8XXX\n", "\n", "Be sure to use these connections **before** starting the jupyter script! If\n", "not, stop the server session with `Ctl+C`, disconnect out of the ssh session\n", "and start again.\n", "\n", "## Navigating a server and notebook session\n", "\n", "After that you should be greeted with the Jupyter Notebook interface in your\n", "browser!\n", "\n", "![main browser page](../_static/notebook_mainpage.png)\n", "\n", "This is your typical directory tree explorer with the root directory set at where \n", "you initiated the jupyter server session. You can either click on a notebook\n", "file (extensions are typically `ipynb`) to open and edit an existing notebook, or \n", "click on the **New** button in the top right to create a new notebook.\n", "\n", "![kernel choice](../_static/kernel.png)\n", "\n", "One clicking the button, we will be greeted with the choice of [**kernels**][kernel]. \n", "Loosely speaking kernels referes to how the code in the notebook will be interpreted.\n", "\n", "[kernel]: https://jupyter-notebook-beginner-guide.readthedocs.io/en/latest/what_is_jupyter.html#kernel\n", "\n", "In our case we will be choosing the Python 3 kernel. By doing so we should be greeted with \n", "a notebook.\n", "\n", "![notebook page](../_static/notebook.png)\n", "\n", "You can change the name of the notebook file on the top title bar. Now lets look at \n", "the main structure of notebooks: **cells**, highlighted with the wide red box in \n", "the image above. Cells are chunks of (python) code to be executed by the server.\n", "After completed editing a cell, run the piece of code either by hitting the **Run**\n", "button, or hitting `Shift+Enter` with our cursor in the desired code. Any output of the \n", "code should also appear below the cell block by default. A new cell will always appear \n", "at the end of the notebook after executing the last cell in the notebook. Or you can \n", "use the **insert** button to create new cell anywhere within the notebook. \n", "\n", "Cells can also be set to be a markdown cell using the drop-down menu highlighed in green,\n", "in which case the output of the cell is a HTML render of the markdown text placed \n", "in the cell. This is extremely useful for alternative between documentation and code \n", "developments. Most of the pages in this tutorial will also follow a notebook style \n", "documentation: such as below\n" ] }, { "cell_type": "code", "execution_count": 1, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This is a demonstration code cell in the tutorial\n", "Hello world\n" ] } ], "source": [ "print(\"This is a demonstration code cell in the tutorial\")\n", "print(\"Hello world\")" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Understanding cells\n", "\n", "A notebooks can be thought of a cells seperating various code statements that share \n", "a common memory pool. This means that variables can be defined in one cells:" ] }, { "cell_type": "code", "execution_count": 2, "metadata": {}, "outputs": [], "source": [ "myvar = 'This is a test'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "And be called and maipuated being sepeared by documentation or file cells" ] }, { "cell_type": "code", "execution_count": 3, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "myvar is: This is a test\n" ] } ], "source": [ "print ('myvar is:', myvar)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Cells can also be exceuted out of order. We can define the statement in the following cell:" ] }, { "cell_type": "code", "execution_count": 4, "metadata": {}, "outputs": [], "source": [ "myvar = myvar + '.'" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "if reexecuted the cell containing the `print('myvar is:', myvar)` statement, it should now show:\n", "\n", "```\n", "myvar is: This is a test\n", "```" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "The small number to the left of the cell indicates the execution order, for the sake of reading clartiy\n", "it is usual practice to not abuse this function, but rather reserve it for code testing:\n", "\n", "Supposing we are currently developing a complicated function:\n" ] }, { "cell_type": "code", "execution_count": 5, "metadata": {}, "outputs": [], "source": [ "def my_difficult_calculation(x):\n", " return x*10" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "We can isolate this complicated function in a cell and use a new cell to check the results of a function" ] }, { "cell_type": "code", "execution_count": 6, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "This is a test.This is a test.This is a test.This is a test.This is a test.This is a test.This is a test.This is a test.This is a test.This is a test.\n" ] } ], "source": [ "print(my_difficult_calculation(myvar))" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Now if we found a some alterations to the `my_difficult_calculation` function, we can simply rerun \n", "the cell containing the definition of the function, forcing the kernel to update what \n", "`my_difficult_calculation` means for python cells, and rerun all cells containing `my_difficult_calculation`" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "## Library and imports \n", "\n", "All notebook cells should have the same access to any python libraries as typical python \n", "session running in the same (virtual) environment. Any import command that would work in\n", "python should work here: " ] }, { "cell_type": "code", "execution_count": 10, "metadata": {}, "outputs": [ { "name": "stdout", "output_type": "stream", "text": [ "Help on function process in module coffea.processor.processor:\n", "\n", "process(self, df)\n", " Processes a single DataFrame\n", " \n", " Returns a filled accumulator object which can be initialized via self.accumulator.identity()\n", "\n" ] } ], "source": [ "import coffea \n", "help(coffea.processor.ProcessorABC.process)" ] }, { "cell_type": "markdown", "metadata": {}, "source": [ "Again, because memory is shared across all cells, import statements can be added to the first executed \n", "cell in the notebook, and all cell will have access to said package. \n", "\n", "A small issue comes when using custom python packages developed specifically for some set of notebooks. \n", "If you have the directory structure like:\n", "\n", "```sh \n", "\n", "\n", "├── coffeaenv\n", "├── notebooks\n", "│   ├── mynotebook1.ipynb\n", "│   └── mynotebook2.ipynb\n", "├── utils\n", "│   ├── utils1.py\n", "│   ├── utils2.py\n", "│   └── utils3.py\n", "└── ...\n", "```\n", "\n", "You may find that the usual import statement in mynotebook1.ipynb such as below\n", "\n", "```python \n", "import utils.utils1 \n", "```\n", "\n", "**might fail**, with the system complaining that utils is not found. This is an issue for what \n", "the \"current working directory\" (`cwd`) is for a specific notebook as straighforwards as executing \n", "a python script in a command line. Depending on how the notebook is initialized, the `cwd` can \n", "be ``, `/notebooks` or neither of the above. \n", "\n", "The what to compensate for this is specific to how your intend to setup your environment, so there is \n", "so `global solution`. The typical solution is **before** starting the jupyter session, setup the \n", "environment variable `PYTHONPATH` to ensure the package path is included. In the case above, we can \n", "run:\n", "\n", "```sh \n", "export PYTHONPATH=${PYTHONPATH}:\n", "```\n", "\n", "If you have been using the `./init.sh` script included in the default setup commands, this variable \n", "should already be setup as such, so you can place python packages directly within your working \n", "directory, and the typical import commands should work as expected.\n", "\n" ] } ], "metadata": { "jupytext": { "main_language": "python" }, "kernelspec": { "display_name": "Python 3", "name": "python3" }, "language_info": { "codemirror_mode": { "name": "ipython", "version": 3 }, "file_extension": ".py", "mimetype": "text/x-python", "name": "python", "nbconvert_exporter": "python", "pygments_lexer": "ipython3", "version": "3.9.1-final" } }, "nbformat": 4, "nbformat_minor": 2 }