[Python] Run all jupyter notebooks inside folder and subfolders in an ordered manner and...

I want to run (in an alphabetical order, not in parallel) all jupyter notebooks inside a folder that contain many subfolders.

• Each subfolder may contain other folders.
• Folders contain jupyter notebooks and the resulting outcome (csv, json, excel, jpg files).
• It is important that files run in order, since the outcome of one notebook is used by others as input source.

As jupyter notebooks are executed, I would like the see a print stating the jupyter notebook and the path.

Until now I would create a jupyter notebook inside each folder, and run inside all the jupyter notebooks from the same folder using %run samplenotebook1.ipynb. However, this becomes tedious when there are numerous folders and subfolders and thus I need to speed up.

I have tried the solution from this post, the notebook seems to be running but if I open any of the folders where jupyter notebooks are supposed to run, I cannot see any file generated from running the jupyter notebooks.

Code below is the one I used, but the result was not the desired one.

import papermill as pm
from glob import glob

for nb in glob('*.ipynb'):
pm.execute_notebook(
input_path=nb,
output_path=nb,
engine_name='embedded',
)


I have also tried code below, I get a print as if it were done, but it only runs notebooks in the folder, not in subfolders.

import papermill as pm
from pathlib import Path

for nb in Path('../').glob('*.ipynb'):
pm.execute_notebook(
input_path=nb,
output_path=nb # Path to save executed notebook
)
print ('done')


Code above will run jupyter notebooks from the path, but not from inner folders. I tried adding /* in the path, but it would not work.

Continue reading...