Python question

[nevj (Neville Jackson)]

@pdecker

I think I’ve heard you can’t easily do that. The JupyterLabs like product called Marimo is able to do that.

Inside myProjects/jupyter-env/venv there is a file expt1.ipynb. It contains config info and some strange binary-like coding

{
 "cells": [
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "77c430a4-007b-42ec-b364-fcc23d6bf479",
   "metadata": {
    "scrolled": true
   },
   "outputs": [
    {
     "ename": "FileNotFoundError",
     "evalue": "[Errno 2] No such file or directory: 'expt1.csv'",
     "output_type": "error",
     "traceback": [
      "\u001b[31m---------------------------------------------------------------------------\u001b[39m",
      "\u001b[31mFileNotFoundError\u001b[39m                         Traceback (most recent call last)",
      "\u001b[36mCell\u001b[39m\u001b[36m \u001b[39m\u001b[32mIn[3]\u001b[39m\u001b[32m, line 2\u001b[39m\n\u001b[32m      1\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mpandas\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[38;5;28;01mas\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mpd\u001b[39;00m\n\u001b[32m----> \u001b[39m\u001b[32m2\u001b[39m df = \u001b[43mpd\u001b[49m\u001b[43m.\u001b[49m\u001b[43mread_csv\u001b[49m\u001b[43m(\u001b[49m\u001b[33;43m'\u001b[39;49m\u001b[33;43mexpt1.csv\u001b[39;49m\u001b[33;43m'\u001b[39;49m\u001b[43m)\u001b[49m\n\u001b[32m      4\u001b[39m \u001b[38;5;28;01mimport\u001b[39;00m\u001b[38;5;250m \u001b[39m\u001b[34;01mnumpy\u001
......

When I start jupyter

trinity:[nevj]:~/Projects/jupyter-env$ source venv/bin/activate
(venv) trinity:[nevj]:~/Projects/jupyter-env$ ls
expt1anal.py  Untitled1.ipynb  Untitled2.ipynb  untitled.py
expt1.csv     untitled1.py     Untitled.ipynb   venv
(venv) trinity:[nevj]:~/Projects/jupyter-env$ jupyter lab

I can do File → Open → Recents → expt1.ipynb
and it brings back my Notebook workspace … ie all the cells.

In the Jupyter tab, there seems to be an option to export

I tries Restructured Text and it made a file expt1.rst in the ~/Downloads directory… Why there?

It contains the code from my first cell, plus heaps of garbage.
If I choose Executable Script I get expt1.py in ~/Downloads, and it contains code only.

So it looks like one can recover code fron Jupyter.

[Rosika (Rosika Schreck)]

Hi Neville,

@nevj

Do you know, […]

So you´re asking how to take the code and documents you’ve created in JupyterLab and save them as standard text files - such as Python scripts or Markdown files - outside the notebook format, I guess.

Since I haven´t done any of this yet, let´s see what perplexity has to say about it:

Exporting from JupyterLab

JupyterLab provides built-in export options for notebooks:

• To export your notebook as a Python script (.py), Markdown (.md), or other formats:
  1. Open your notebook in JupyterLab.
  2. Go to the File menu.
  3. Select Export Notebook As….
  4. Choose your desired format, such as Export to Executable Script (for a .py file) or Export to Markdown (for a .md file)

This will create a standard text file containing your code or documentation, which you can then download, edit, or use elsewhere.

Command-Line Option

Alternatively, you can use the command line to export notebooks::
jupyter nbconvert my_notebook.ipynb --to script

This will generate a my_notebook.py file with all the code from your notebook

Export Formats

JupyterLab supports exporting to several formats, including:

Python script (.py)

Markdown (.md)

HTML (.html)

LaTeX (.tex)

PDF (may require extra setup)

ReStructuredText (.rst)

Summary Table

Export Option	File Type	Use Case
Executable Script	.py	Standard Python code file
Markdown	.md	Documentation, sharing notes
HTML	.html	Web viewing/sharing
PDF	.pdf	Printable, shareable document

In short:
Use the File > Export Notebook As… menu in JupyterLab to save your code and documents as normal text files like .py or .md. This makes it easy to use your work outside of JupyterLab

As always with AI the results have to be taken with a grain of salt, but the output looks pretty sensible so far.

Many greetings from Rosika

[nevj (Neville Jackson)]

@Rosika

Your Project Directory and File Access

If you started JupyterLab from your project directory (where expt1.csv and your virtual environment are), the notebook’s working directory should be correct. If you still get a “file not found” error, double-check the working directory in a notebook code cell:

import os
print(os.getcwd())

If it matches your project directory, your file path should work. If not, use a relative or absolute path to the CSV file.

I decided to go back to the manual venv environment, and try to access the file there

trinity:[nevj]:~/Projects/collagen$ source venv/bin/activate
(venv) trinity:[nevj]:~/Projects/collagen$ python
Python 3.13.3 (main, Apr  9 2025, 17:13:31) [GCC 14.2.1 20250207] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
>>> df = pd.read_csv('expt1.csv')
>>> df
            Farm       Grade  Sheep  Field  ...       Red     Green      Blue  Count
0       Glensloy      smooth   3457      1  ...  0.734539  0.304495  0.263945    732

Yes, it works there
I tried the collagenuv environment too,

trinity:[nevj]:~/Projects/collagenuv$ source .venv/bin/activate
(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ python
Python 3.13.3 (main, Apr  9 2025, 17:13:31) [GCC 14.2.1 20250207] on linux
>>> import pandas as pd
>>> df = pd.read_csv('expt1.csv')
>>> df
            Farm       Grade  Sheep  Field  ...       Red     Green      Blue  Count
0       Glensloy      smooth   3457      1  ...  0.734539  0.304495  0.263945    732
....
(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ deactivate
trinity:[nevj]:~/Projects/collagenuv$

So, it is not a virtual environment problem, it has to be a jupyter lab problem
So try jupyter again

trinity:[nevj]:~/Projects/jupyter-env$ source venv/bin/activate
(venv) trinity:[nevj]:~/Projects/jupyter-env$ jupyter lab
....

Now I am goint to run the following in a Notebook cell

OK, that is the problem…jupyter lab Notebook works in the directory
/home/nevj/Projects/jupyter-env/venv
whereas python REPL works in the directory I start it in.
So, I have to either

• change the filename to ../expt1.csv in the python script, or
• move the file into venv or make a link… I dont like that, or
• find some way to change the cwd in jupyter
  I think the last… any suggestions?

Outside of jupyter the pwd is

(venv) trinity:[nevj]:~/Projects/jupyter-env$ pwd
/home/nevj/Projects/jupyter-env

but if I start jupyter in that directory its cwd becomes
/home/nevj/Projects/jupyter-env/venv
Confusing?

I have a solution

Then it can read the file.
It remains to be seen if that affects other things in jupyter lab

Update:
I found this
" People also ask

How to change CWD in Jupyter notebook?

2:18

3:24

So we also provide the same convention. Okay in all the places. So that it can easily locate myMore

[image]

https://www.youtube.com · The Data Empower

Set working directory in jupyter notebook using Python … - YouTube

How do I change the default directory in JupyterLab?

How to Change Jupyter Notebook Default Folder

1. Open the terminal or command prompt on your computer.
2. Type jupyter notebook --generate-config and press Enter. …
3. Type jupyter --config-dir and press Enter. …
4. Navigate to the configuration directory using your file explorer.

More items…

•

27 Dec 2023 "
I dont know whether the default directory is the one you see with os.getcwd()

The there is this

Setting up JupyterLab and a virtual environment

which suggests I should be using ipykernel with jupyter lab ?

And some constructive criticism here

Using virtual environments with Jupyter is broken

[Rosika (Rosika Schreck)]

Hi Neville,

@nevj

I think the last… any suggestions?

O.K., let´s consider what peplexity suggests. As we´ve already taken this path we might as well follow it a bit further.
This way I hopefully can learn something new, too.

Your friend is correct that JupyterLab and the Python REPL can have different working directories, which affects how relative file paths are resolved. Here are the main ways to address this in JupyterLab:

Options for Changing or Handling the Working Directory

• Change the working directory in your notebook code:
  You can add this at the top of your notebook to set the working directory to where your data file is:

import os
os.chdir('/path/to/your/project')

Replace ‘/path/to/your/project’ with the directory containing expt1.csv. This is a straightforward solution and works immediately for that notebook session

• Start JupyterLab from the desired directory:
  Before launching JupyterLab, open your terminal, cd to the directory where your files are, and then run:
  jupyter lab
  JupyterLab will use this as its root directory, and relative paths will work as expected

• Edit the Jupyter configuration (advanced):
  You can set a default startup directory by editing the Jupyter configuration file.

1. Generate the config file if you haven’t:
   jupyter notebook --generate-config

2. Open ~/.jupyter/jupyter_notebook_config.py and find the line:
   #c.NotebookApp.notebook_dir = ''
   Remove the # and set it to your desired directory:
   c.NotebookApp.notebook_dir = '/home/nevj/Projects/jupyter-env'

3. Save and restart JupyterLab

Summary Table

Method	What to Do	When to Use
Change cwd in notebook code	os.chdir('/path/to/dir')	For quick, per-notebook changes
Start JupyterLab from project dir	cd /path/to/dir && jupyter lab	For session-wide consistency
Edit Jupyter config file	Set c.NotebookApp.notebook_dir in config	For permanent, default directory

Recommendation:

For most users, either starting JupyterLab from the desired directory or adding os.chdir() at the top of your notebook are the easiest and safest solutions

Personally I´d refrain from “Editing the Jupyter configuration (advanced)” and choose either option1 or 2.
That´d be my personal preference, of course.

Many greetings from Rosika

[nevj (Neville Jackson)]

@Rosika

Start JupyterLab from the desired directory:
Before launching JupyterLab, open your terminal, cd to the directory where your files are, and then run:
jupyter lab
JupyterLab will use this as its root directory, and relative paths will work as expected

Hi Rosika,

Well, AI got that wrong.
That is what anyone would expect, but I just demonstrated that jupyterlab chose venv as the cwd even though I started it in the parent directory.

Can you perhaps try your jupyterlab and see if it does the same? I may be doing something stupid. No hurry, I can shelve this.

Did you see the update on last reply?

Regards
Neville

[Rosika (Rosika Schreck)]

Hi Neville,

@nevj

Did you see the update on last reply?

No, it seems I missed it.
I´ll have to read it through first.

[pdecker]

I think I’d lean towards doing the cd in the notebook. “cd ~/Projects/”.
If you were working with a team of people on a project, that might not work very well. But, then again, it might work perfectly. Everyone would go to the same location for the project.

One of the things I don’t like about notebooks, and I don’t really use them, is that you can execute cells out of order. So, if you didn’t execute top to bottom, you won’t be in the correct directory.

[nevj (Neville Jackson)]

@pdecker

can execute cells out of order.

Yes. I think that pushes you towards an object oriented rather than functional code. You still need a main routine that has to be first.

[pdecker]

The newer Marimo notebook I mentioned before DOES execute top to bottom. It sure seems like that makes much more sense and would also be more reproducible.

It does some caching to avoid “expensive” function calls.

[nevj (Neville Jackson)]

@pdecker

newer Marimo notebook

I think that is worth a look.
Jupyter does not execute all cells… you choose the cell to run, but you can choose in any order.
It seems to me Notebooks are good for developing little snippets of code , or a couple of functions. I cant imagine it working for a large program.

[pdecker]

I don’t know that you need to use a notebook at all. When you were working with Julia it seemed your workflow was more suited to using whatever your preferred editor is and running the resulting program at a command prompt. I like working that way too, though I do like to use VS Code for the debugger.

[nevj (Neville Jackson)]

@pdecker

I don’t know that you need to use a notebook at all.

Right. I feel more in control with the python REPL and vi. Happy with UV though.
Python REPL is different to Julia. In Julia REPL I can copy in a slab of code and it runs.
In Python REPL it seems to only accept one line of code at time. … You may know a way around that.
THAT IS WRONG… it does accept a copied slab of code

(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ python
Python 3.13.3 (main, Apr  9 2025, 17:13:31) [GCC 14.2.1 20250207] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import pandas as pd
... df = pd.read_csv('expt1.csv')
... 
... import numpy as np
... #import matplotlib.pyplot as plt
... import scipy.stats as stats
... import statsmodels as sm
... import statsmodels.formula.api as smf
... 
... # model1 = ols('Area ~ C(Farm) + C(Grade)', df).fit()
... # model1 = smf.ols('Area ~ C(Farm) + C(Grade)', df).fit()
... 
... # model1 fit
... from statsmodels.formula.api import ols
... from statsmodels.stats.anova import anova_lm
... model1 = ols('Area ~ C(Farm) + C(Grade)', df).fit()
... model1.summary()
... anova_lm(model1)
... 
... 
... # model2 fit
... model2 = ols('Area ~ C(Farm) + C(Grade) + C(Farm):C(Grade)', df).fit()
... model2.summary()
... anova_lm(model2)
... 
                        df        sum_sq  ...            F        PR(>F)
C(Farm)                1.0  1.429054e+10  ...  2028.908896  0.000000e+00
C(Grade)               1.0  3.777585e+08  ...    53.632501  2.431477e-13
C(Farm):C(Grade)       1.0  2.928109e+09  ...   415.720047  2.930235e-92
Residual          127235.0  8.961750e+11  ...          NaN           NaN

[4 rows x 5 columns]
>>> 

I think those dots confused me… and it only prints the last line of output which in the above is from anova_lm(model2)… you dont get to see the output from model2.summary() or any previous lines.
I will get used to that. It actually helps because all I want to do is run the code down to the last working line, so I can debug the next step. The intermediate output is superfluous.
It has to be working code… an error in the middle will hang

Jupyter was more of a curiosity exercise… why does everyone use Notebook for R and Julia? I dont have the answer yet, but I at least know what a Notebook is.

[nevj (Neville Jackson)]

@pdecker

uv run main.py

Back in collagenuv environment.
Have already done uv venv
My setup does

source .venv/bin/activate
uv python install
uv pip install pandas
uv pip install scipy
uv pip install statsmodels

I have discovered that I can do the above before or after I do source .venv/bin/activate… it does not matter , uv pip still installs them in .venv/lib/....
and
I can run a script with uv run expt1anal.py either before or after doing source .venv/bin/activate… it still runs in the collagenuv environment.
but
to use the python REPL in the collagenuv environment, I have to do

source .venv/bin/activate
python

So UV acts like an automatic activate

[pdecker]

@nevj

So UV acts like an automatic activate

I found the same thing. I thought I was crazy at first. It says right there, “source .venv/bin/activate” to activate the virtual environment.

Did you look at using a comment section at the top of the script rather than a virtual environment? I guess if you’re using the REPL that wouldn’t work for you.

pdecker@dpd-u2404:~/python/zzz$ uv add --script nev.py pandas scipy statsmodels
Updated nev.py
pdecker@dpd-u2404:~/python/zzz$ cat nev.py
# /// script
# requires-python = “>=3.13”
# dependencies = [
# “pandas”,
# “scipy”,
# “statsmodels”,
# ]
# ///
import pandas
import scipy
import statsmodels
That way you can just run:

uv run nev.py

and it runs with no virtual environment needed. I use an alias py=’ uv run’ and that makes it:

py nev.py

[nevj (Neville Jackson)]

@pdecker

@nevj

So UV acts like an automatic activate

I found the same thing. I thought I was crazy at first. It says right there, “source .venv/bin/activate” to activate the virtual environment.

Did you look at using a comment section at the top of the script rather than a virtual environment? I guess if you’re using the REPL that wouldn’t work for you.

I forgot about that
but
my script works with uv run scriptname.py without any comments and it works whether I have activated the venv or not.
as long as I am in the project directory.
I guess putting in the comments does all the setup installs… OK I should use uv add to add the dependencies to my script. Thanks.

I use REPL when writing and debugging , plus vi to build a script file. Then if I want to use the debugged progam to process data, I run the script directly.

[pdecker]

@nevj

I use REPL when writing and debugging

You’ll need an activated virtual environment for that I believe. I just think it’s handy to use the “uv add” feature. During development and debugging that wouldn’t work well.

[nevj (Neville Jackson)]

@pdecker

You’ll need an activated virtual environment for that I believe.

Right.
If using UV, that is the only thing that requires me to activate.
BTW, what happened to.venv/bin/deactivate ?
Mine seems to have disappeared? I wonder has it been removed in an update?
It is the only way to terminate an activation.

[pdecker]

I’m confused by that too. I just created a venv and I do see a few versions of activate (for different platforms) along with one deactivate.bat. That isn’t what is executed, but when I execute the deactivate command it does deactivate the venv. I tried which deactivate but that didn’t find anything. Yet it works. Son of a gun.

[nevj (Neville Jackson)]

@pdecker

Yet it works

It seems to be a shell command …like cd… which cd finds nothing too
So it is only there when the venv shell is running and all it does is kill the shell.
Tricky.