I really don't get (jupyter) notebooks. It always feels like an over-engineered solution to a simple problem. The only advantage they have over 'proper' code generating a 'proper' report is being able to modify code directly on the report (i.e. as opposed to having to change the code in a separate file instead). Which sounds good and all, but this is rarely what I see notebooks actually used for. For the most part, people use them as an IDE, and a notebook is a terrible choice of IDE compared to even something as simple as editing in nano. And the disadvantages of using one generally completely outweigh the benefit of live-editing in the first place, and encourage all sorts of bad programming habits / hacky code structure.
Whenever I've required a report which intermingles code, text, and outputs, a simple bash-script processing my (literate programming) codebase has always done a far more impressive job of generating a useful report. And, without having to sacrifice proper packaging / code-organisation semantics just for the sake of a report.
I find it a big shame that the iodide / pyodide notebooks didn't take off in the same way, at least. Those seemed tightly integrated to html in a way that was very elegant.
The main place that I find notebooks useful is for when I want to hack away at some decently sized dataset and poke around at its details.
A lot of transformations can be very time consuming to run, and the ability to cache(for lack of a better word) the computation without having to write to disk, or utilize the python repl (which is very obnoxious to use for anything that extends past a single line of code) really speeds the process up.
An example would be: pull a large json from a server. Then break out a new code block to do all of your different manipulations on that json. Lets you prototype around with the object you pulled from the server without having to worry about, among other things
- Dealing with how slow reading it from server is
- writing it to disk and then reading it from disk to deal with how slow that is
- the latency of parsing the file on each run of your script
- the list goes on.
These arn't things that you want to worry about when your current questions are "what does the structure of this look like" and "what are some of the basic statistical properties of this data". Notebooks are like the python repl with the benefits of having a proper multi-line text input.
I've never even heard of people using notebooks for report generation, and honestly I'd agree that sounds like a complete nightmare.
As a person who's often tasked with building reports for presentation, and more often, for passing on to analysts to expand on, I find that notebooks give a much more accessible workflow than other options. Excel is way too limiting, most of the downstream analysts aren't software devs and don't have, know, or want to know an IDE. Notebooks give a way to combine graphs, text and code (show your work) in a concise form.
Do you get REPL? Do you get debugger?
Notebooks are between REPL and "proper code" on this spectrum. They can be used even when nobody else sees the results. They are useful for experimentation, to figure out things you don't know you don't know as an extension of your working memory. They are less temporary and disposable as REPL but less reusable than proper code. Another example is Org Babel in Emacs.
Jupyter notebooks in the article are used for "learning-oriented tutorial" or "goal-oriented howtos" docs. You see explanation, code, results, and it is easy to try it. There are solutions with a single click that can get you the running editable copy.
Misleading docs may be worse than no docs, so using notebooks as executable documentation is a plus. Though it is not the best format for e2e tests in general. Tests may be too complex for that. Software (proper code) is better at handling complexity in the general case.
IMO notebooks are hugely overused and don't have any place in the development of software.
But they're fantastic for what they were designed for -- which is quite literally "notebooks". AFAIK the idea was first popularized by Mathematica, and I still reach for that when I have some highly iterative, undefined math/data problem to sketch on. IMO the real issue is that Python is used both for this purpose and for software development, which leads to people using notebooks inappropriately.
i mostly agree, but one way that i've found to make notebooks good/usable is this:
- move all of the code out of the notebook and into a nearby python module where it can be linted/mypy-d/version-controlled/code-reviewed easily
- tell the notebook to import that module and make a small number of function calls to get whatever data you need and make plots. at this point the notebook is really nothing more than a REPL with inlined plots/graphics
1) It sounds like you are maybe not in the target group, which largely comprises data scientists and the like.
2) Reports are one of many reasons people use notebooks.
3) You can work with notebooks in proper IDEs like PyCharm or VS Code or Emacs.
Having said that, I do agree with you that notebooks enable poor programming habits. (I have seen quite a few notebooks with > 10k lines of code, which is insanity.) Among other things, diffing notebooks is typically a big pain too. It is just that for many folks, notebooks' pros outweigh their cons.
Notebooks are not about producing a report. I've worked with hundreds of notebooks and very few of them were intended as polished documents.
Notebooks are literally for notes, for exploring new ideas - not creating production artifacts from existing ideas. You get a stateful kernel that can incrementally build up state instead of re-executing. And you get visual artifacts and user interfaces inline. The value proposition is faster iteration and immediate feedback, not report writing.
they work for me because of the self documenting interface for reporting. Then theres the built in project workspace that makes it clear which files are doing what. I can run them on a commandline as scripts, and build libraries or even us other notebooks as libraries.
This workflow works because my primary job is no developing software. It's developing solutions to solve multimodal industrial and science reporting issues, and as needs change, having a complicated stepping as just a series of code snippets is wonderful.
Sure, text is cleaner but I treat much of my work as "living documents" so notebooks do that wonderfully.
Whenever I've required a report which intermingles code, text, and outputs, a simple bash-script processing my (literate programming) codebase has always done a far more impressive job of generating a useful report. And, without having to sacrifice proper packaging / code-organisation semantics just for the sake of a report.
I find it a big shame that the iodide / pyodide notebooks didn't take off in the same way, at least. Those seemed tightly integrated to html in a way that was very elegant.
(they're not completely gone though, it was nice to see this example on the front page earlier: https://news.ycombinator.com/item?id=42425489 )