Python question

Rosika · May 18, 2025, 12:30pm

Hi Neville,

Of course. No need to rush things.

That´s the most sensible and promising approach.

Right. Your academic background and professionality serve you well in this context.
That´s certainly to be admired.

Cheers from Rosika

pdecker · May 18, 2025, 12:33pm

They are compatible with each other. UV is just a more all-in-one way to do it and a bit quicker. You can create a venv using UV and use it via the traditional venv utility and vice versa.

nevj · May 18, 2025, 12:38pm

Thanks. I did not realize that. I like it…things working together.
Will do uv tomorrow.
I am getting the impression that Python is more organised than Julia… not surprising, it has been around longer.

pdecker · May 18, 2025, 12:41pm

There is a new alternative to Jupiter called Marimo. You might want to look into that

Here is a podcast that talks about it. There is also a YouTube video.

Rosika · May 18, 2025, 12:42pm

Hi, @pdecker

So sorry to hear that. I hope you´ll get better soon.
All the best.

Cheers from Rosika

pdecker · May 18, 2025, 12:51pm

Thanks. I do feel better, though it is easy to feel better than I did. Ugh. I’ll get back to normal in another day I think.

Here is the YouTube video I mentioned.

This person wrote Marimo during his graduate studies in Python. It looks really neat. I have not had much use for a notebook like that but maybe @nevj would. It seems a lot of science or statistics use them.

nevj · May 18, 2025, 12:59pm

Yeah, the numerical people today seem mad on notebooks. The R community are addicted to either Rstudio or Jupyter. I have never caught the bug… all these things seem to do is cram into one window what could be spread over several windows.
That amounts to reinventing the window system inside one window… see my comment elsewhere re pdf readers in browsers.

daniel.m.tripp · May 19, 2025, 2:15am

I kinda hate messing about with python and dependancies…

Some of the time - you’re supposed to run “pip” (e.g. what some online documentation suggests) - but - on some systems with both Python 2x and 3x installed - you’re supposed to run “pip3” to get something installed in Python 3x…

It’s messy…

I have stuff I’d previously got working - installed with “pip3” that’s now broken…

One is “bandcamp-dl” (AKA Bandcamp-Downlowder)…

It’s broken on my Pop!_OS environment on my Ryzen Desktop… and I can’t be arsed trying to fix it - because - it still works on my Pop!_OS Thinkpad - and - in Raspbian on one of my Pi4 systems… So I use it there… Both of them see the same storage as my desktop system… So it’s not a drama…

nevj · May 19, 2025, 3:33am

Are you using the system python3?
That might explain it breaking… eg an update might break things you add.

Everyone is telling me it is better to make a user python install in a virtual environment.

nevj · May 19, 2025, 11:45am

It is UV day.
I will create another separate project to be set up with UV

$ uv
An extremely fast Python package manager.

Usage: uv [OPTIONS] <COMMAND>
....

Yes it is installed
Make the venv

mkdir ~/Projects/collagenuv
cd ~/Projects/collagenuv

trinity:[nevj]:~/Projects/collagenuv$ uv venv
Using CPython 3.13.3 interpreter at: /usr/bin/python
Creating virtual environment at: .venv
Activate with: source .venv/bin/activate

trinity:[nevj]:~/Projects/collagenuv$ ls -a
.  ..  .venv
trinity:[nevj]:~/Projects/collagenuv$ ls .venv
bin  CACHEDIR.TAG  lib  lib64  pyvenv.cfg

Slightly different… it calls it .venv , but is has bin and lib and a .cfg file, just like venv
So lets do the same as before

$ .venv/bin/activate
bash: .venv/bin/activate: Permission denied

Lets investigate

trinity:[nevj]:~/Projects/collagenuv/.venv/bin$ ls -l activate
-rw-r--r-- 1 nevj nevj 4106 May 19 21:13 activate

OK it needs a chmod

$ chmod 755 activate

ls -l activate
-rwxr-xr-x 1 nevj nevj 4106 May 19 21:13 activate

cd ../..

trinity:[nevj]:~/Projects/collagenuv$ .venv/bin/activate
You must source this script: $ source .venv/bin/activate

trinity:[nevj]:~/Projects/collagenuv$ source .venv/bin/activate
(collagenuv) trinity:[nevj]:~/Projects/collagenuv$

Maybe I did not need to chmod it… needed to use source. Why?

Now some installs

(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ uv python install
Installed Python 3.13.3 in 6.63s
 + cpython-3.13.3-linux-x86_64-gnu




(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ uv pip install pandas
Resolved 6 packages in 1.08s
Prepared 6 packages in 3.25s
Installed 6 packages in 782ms
 + numpy==2.2.6
 + pandas==2.2.3
 + python-dateutil==2.9.0.post0
 + pytz==2025.2
 + six==1.17.0
 + tzdata==2025.2

It works
Now I have in .venv/bin

(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ ls .venv/bin
activate      activate.fish  activate_this.py  numpy-config  python3
activate.bat  activate.nu    deactivate.bat    pydoc.bat     python3.13
activate.csh  activate.ps1   f2py              python

The same as wih venv, except no pip ??
Does UV have its own pip?

The lib subdir is slightly different too

(collagenuv) trinity:[nevj]:~/Projects/collagenuv$ ls .venv/lib/python3.13/site-packages
dateutil                               pytz
numpy                                  pytz-2025.2.dist-info
numpy-2.2.6.dist-info                  six-1.17.0.dist-info
numpy.libs                             six.py
pandas                                 tzdata
pandas-2.2.3.dist-info                 tzdata-2025.2.dist-info
pandas.libs                            _virtualenv.pth
python_dateutil-2.9.0.post0.dist-info  _virtualenv.py

Those files with underscore are extra.

What I really like is
uv run main.py
running a script is something I need.
I suppose the venv equivalent is
python main.py

OK , venv and UV are nearly the same.
There is a uv init command … I suppose it combines the venv and activate steps?
What I might try next is putting jupyterlab in one of the environments.

pdecker · May 19, 2025, 1:29pm

Did you try exactly that? (Oops, I see maybe you did later)
That has always worked for me. Maybe it makes a difference what the parent directory permissions are.

If you use:

uv venv foo

It will create a virtual environment directory named foo.

Me too!
I created an alias py=‘uv run’
That way I can run my application with py main.py.

One of the things I like most about UV is that it can use a section of comments at the top of your python script instead of creating a virtual environment.

For example:.

uv add --script main.py pandas

Will add the following at the top of the script:

That way when you run: py main.py

It just works. No need to create a virtual environment, no need to activate one, no need to do a pip install, and no need to install python. UV sees the comments and processes them. Once it has been done, UV keeps a cache of things it has downloaded (which you can manage) and it’s super quick the next time you run your script.

nevj · May 19, 2025, 2:11pm

Oh, I get it… if I just run a script with python, and am not in an environment, I will get the system python, which is not what I want in most cases.
So I either

make a venv and be in it, or
use uv with comments in the script header.

Which way depends on whether I need interactive python .

pdecker · May 19, 2025, 2:47pm

For the most part, correct.

You can also get interactive Python using uv run python3.13.

You could have multiple versions of Python installed using UV. Use uv python list to list the versions of Python UV sees installed.

You and I probably don’t have much use for multiple versions of Python, but if you are a package maintainer, you want to make sure your package works with all current supported versions. And of course, avoid using the system Python so you don’t cause issues with the rest of your system.

The whole Python topic is pretty big. I’m sure you can imagine that pretty much everything you can do in R you can do in Python. I believe you have some C background too. There are lots of different areas you may never need to touch with Python. It’s also a fairly old language that has gone through more than a few cycles of attempts to make virtual environments and package management easier. For me, UV is probably the most straight forward. No sense muddying the water for you by mentioning all the rest.

nevj · May 19, 2025, 10:45pm

That is what I am doing… the same statistical analysis of my image data in R, Python, and Julia. There will be a topic when I finish. I am having immense trouble in Julia, but I am nearly there in Python.

nevj · May 20, 2025, 11:34am

Trying jupyterlab today.

trinity:[nevj]:~$ cd Projects
trinity:[nevj]:~/Projects$ mkdir -p jupyter-env
trinity:[nevj]:~/Projects$ cd jupyter-env
trinity:[nevj]:~/Projects/jupyter-env$ which python3
/usr/bin/python3
trinity:[nevj]:~/Projects/jupyter-env$ python3 -m venv venv
trinity:[nevj]:~/Projects/jupyter-env$ source venv/bin/activate
(venv) trinity:[nevj]:~/Projects/jupyter-env$ pip install jupyter
Collecting jupyter
  Downloading jupyter-1.1.1-py2.py3-none-any.whl.metadata (2.0 kB)
....

Collecting jupyterlab (from jupyter)
  Downloading jupyterlab-4.4.2-py3-none-any.whl.metadata (16 kB)
....

Installing collected packages: webencodings, wcwidth, pure-eval, ptyprocess, fastjsonschema, widgetsnbextension, websocket-client, webcolors, urllib3, uri-template, typing-extensions, types-python-dateutil, traitlets, tornado, tinycss2, soupsieve, sniffio, six, setuptools, send2trash, rpds-py, rfc3986-validator, pyzmq, pyyaml, python-json-logger, pygments, pycparser, psutil, prompt-toolkit, prometheus-client, platformdirs, pexpect, parso, pandocfilters, packaging, overrides, nest-asyncio, mistune, markupsafe, jupyterlab_widgets, jupyterlab-pygments, jsonpointer, json5, idna, h11, fqdn, executing, defusedxml, decorator, debugpy, charset-normalizer, certifi, bleach, babel, attrs, async-lru, asttokens, terminado, stack_data, rfc3339-validator, requests, referencing, python-dateutil, matplotlib-inline, jupyter-core, jinja2, jedi, ipython-pygments-lexers, httpcore, comm, cffi, beautifulsoup4, anyio, jupyter-server-terminals, jupyter-client, jsonschema-specifications, ipython, httpx, arrow, argon2-cffi-bindings, jsonschema, isoduration, ipywidgets, ipykernel, argon2-cffi, nbformat, jupyter-console, nbclient, jupyter-events, nbconvert, jupyter-server, notebook-shim, jupyterlab-server, jupyter-lsp, jupyterlab, notebook, jupyter
Successfully installed anyio-4.9.0 argon2-cffi-23.1.0 argon2-cffi-bindings-21.2.0 arrow-1.3.0 asttokens-3.0.0 async-lru-2.0.5 attrs-25.3.0 babel-2.17.0 beautifulsoup4-4.13.4 bleach-6.2.0 certifi-2025.4.26 cffi-1.17.1 charset-normalizer-3.4.2 comm-0.2.2 debugpy-1.8.14 decorator-5.2.1 defusedxml-0.7.1 executing-2.2.0 fastjsonschema-2.21.1 fqdn-1.5.1 h11-0.16.0 httpcore-1.0.9 httpx-0.28.1 idna-3.10 ipykernel-6.29.5 ipython-9.2.0 ipython-pygments-lexers-1.1.1 ipywidgets-8.1.7 isoduration-20.11.0 jedi-0.19.2 jinja2-3.1.6 json5-0.12.0 jsonpointer-3.0.0 jsonschema-4.23.0 jsonschema-specifications-2025.4.1 jupyter-1.1.1 jupyter-client-8.6.3 jupyter-console-6.6.3 jupyter-core-5.7.2 jupyter-events-0.12.0 jupyter-lsp-2.2.5 jupyter-server-2.16.0 jupyter-server-terminals-0.5.3 jupyterlab-4.4.2 jupyterlab-pygments-0.3.0 jupyterlab-server-2.27.3 jupyterlab_widgets-3.0.15 markupsafe-3.0.2 matplotlib-inline-0.1.7 mistune-3.1.3 nbclient-0.10.2 nbconvert-7.16.6 nbformat-5.10.4 nest-asyncio-1.6.0 notebook-7.4.2 notebook-shim-0.2.4 overrides-7.7.0 packaging-25.0 pandocfilters-1.5.1 parso-0.8.4 pexpect-4.9.0 platformdirs-4.3.8 prometheus-client-0.22.0 prompt-toolkit-3.0.51 psutil-7.0.0 ptyprocess-0.7.0 pure-eval-0.2.3 pycparser-2.22 pygments-2.19.1 python-dateutil-2.9.0.post0 python-json-logger-3.3.0 pyyaml-6.0.2 pyzmq-26.4.0 referencing-0.36.2 requests-2.32.3 rfc3339-validator-0.1.4 rfc3986-validator-0.1.1 rpds-py-0.25.0 send2trash-1.8.3 setuptools-80.7.1 six-1.17.0 sniffio-1.3.1 soupsieve-2.7 stack_data-0.6.3 terminado-0.18.1 tinycss2-1.4.0 tornado-6.5 traitlets-5.14.3 types-python-dateutil-2.9.0.20250516 typing-extensions-4.13.2 uri-template-1.3.0 urllib3-2.4.0 wcwidth-0.2.13 webcolors-24.11.1 webencodings-0.5.1 websocket-client-1.8.0 widgetsnbextension-4.0.14

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip

It is huge.

(venv) trinity:[nevj]:~/Projects/jupyter-env$ jupyter
usage: jupyter [-h] [--version] [--config-dir] [--data-dir] [--runtime-dir]
               [--paths] [--json] [--debug]
               [subcommand]

Jupyter: Interactive Computing

positional arguments:
  subcommand     the subcommand to launch

options:
  -h, --help     show this help message and exit
  --version      show the versions of core jupyter packages and exit
  --config-dir   show Jupyter config dir
  --data-dir     show Jupyter data dir
  --runtime-dir  show Jupyter runtime dir
  --paths        show all Jupyter paths. Add --json for machine-readable
                 format.
  --json         output paths as machine-readable json
  --debug        output debug information about paths

Available subcommands: console dejavu events execute kernel kernelspec lab
labextension labhub migrate nbconvert notebook run server troubleshoot trust

Please specify a subcommand or one of the optional arguments.

Hey, hang on, in Julia, I started jupyterlab from within the Julia REPL.
Is that what I do here?.. Consult the web page … it says to do

$ jupyter lab
[I 2025-05-20 20:35:02.576 ServerApp] jupyter_lsp | extension was successfully linked.
.....
[I 2025-05-20 20:35:03.122 ServerApp] Jupyter Server 2.16.0 is running at:
[I 2025-05-20 20:35:03.122 ServerApp] http://localhost:8888/lab?token=06026af9c625f8d34bfbff09f5faff82ffc4629ffa19c563
[I 2025-05-20 20:35:03.123 ServerApp]     http://127.0.0.1:8888/lab?token=06026af9c625f8d34bfbff09f5faff82ffc4629ffa19c563
[I 2025-05-20 20:35:03.123 ServerApp] Use Control-C to stop this server and shut down all kernels (twice to skip confirmation).
.....

    To access the server, open this file in a browser:
        file:///home/nevj/.local/share/jupyter/runtime/jpserver-3017-open.html
    Or copy and paste one of these URLs:
        http://localhost:8888/lab?token=06026af9c625f8d34bfbff09f5faff82ffc4629ffa19c563
        http://127.0.0.1:8888/lab?token=06026af9c625f8d34bfbff09f5faff82ffc4629ffa19c563

[I 2025-05-20 20:35:08.615 ServerApp] Skipped non-installed server(s): bash-language-server, dockerfile-language-server-nodejs, javascript-typescript-langserver, jedi-language-server, julia-language-server, pyright, python-language-server, python-lsp-server, r-languageserver, sql-language-server, texlab, typescript-language-server, unified-language-server, vscode-css-languageserver-bin, vscode-html-languageserver-bin, vscode-json-languageserver-bin, yaml-language-server
and it waits in that terminal window

A browser tab starts

There is a notebook and a console… I did not install python in this venv… did jupyter install it for me?.. No, making the venv automatically installs python.

I tried to use jupyter notebook.
Pasted my python script in to a cell , and ran it

It hangs on the first line
import pandas as pd
I guess that means I have to pip install pandas in this venv
So I did that in a second cell

Now go back to first cell and rerun

OK , now it gets to the next error… It cant find the file expt1.csv
That is funny… expt1.csv is there in the project directory… perhaps it needs a path?

That is enough to show what jupyterlab is like, at least as far as using Notebook is concerned.
My first impression is that cells have a tendency to fragment code
and it seems things have to be installed in any venv… a global install is not the way to go.

I will try a little more with Notebook… I need to see why everyone loves using notebook rather than a plain editor.

Rosika · May 20, 2025, 12:47pm

Hi Neville;

I submitted your problem to perplexity.
It suggested the following:

Your friend’s experience is common when using JupyterLab with Python scripts and external files. Here are the key points and solutions:

1. Importing Pandas Hangs or Errors

If import pandas as pd hangs or throws an error, it usually means the pandas library is not installed in the Python environment that JupyterLab is using

Simply running pip install pandas in a notebook cell might not always install pandas in the correct environment, especially if JupyterLab is running under a different Python interpreter or virtual environment than the terminal

To ensure pandas is installed in the right environment, use this command in a JupyterLab cell:
!pip install pandas
If the problem persists, check which Python executable JupyterLab is using by running:
import sys
print(sys.executable)
Compare this path with the one used in your terminal. If they differ, you may need to install pandas in the correct environment or adjust your Jupyter kernel

2. File Not Found: expt1.csv

If expt1.csv is reported as missing, but it exists in your project directory, the likely issue is the working directory from which JupyterLab is running.

JupyterLab’s working directory is typically where you launched it from, not necessarily where your notebook file is located.

To check your current working directory in a Jupyter cell:
import os
print(os.getcwd())
If this path does not contain expt1.csv, you have a few options:

Move expt1.csv to the working directory.

Use the full or relative path to expt1.csv in your code, e.g.:
pd.read_csv('/full/path/to/expt1.csv')
or
pd.read_csv('relative/path/from/cwd/expt1.csv')

Alternatively, you can change the working directory in your notebook:
import os
os.chdir('/path/where/csv/is')
These steps should resolve your friend’s issues with both importing pandas and accessing the CSV file

No idea if it helps at all.

Many greetings from Rosika

nevj · May 20, 2025, 1:00pm

Hi Rosika,
I can find a Notebook cell, and a Colsole cell, but where do I find a Jupyterlab cell.?

All these different types of cells have me confused.

I have expt1.csv in the directory where venv is… ie the top project directory.
I think I launched jupyter lab from that directory… will have to check

So far , to me, notebook looks like having several editors open at once.
And, what should I be using… Notebook or Console?

Regards
Neville

Rosika · May 20, 2025, 1:09pm

Hi Neville,

let´s see what perplexity says about your latest comments:

Your friend’s confusion is understandable—JupyterLab, Jupyter Notebook, and Jupyter Console all use the word “cell,” but in slightly different contexts.

What is a JupyterLab Cell?

In JupyterLab, a “cell” refers to a section of a notebook or console where you can enter and execute code or text. There is no special “JupyterLab cell”—the term just means a cell in a notebook or console opened within the JupyterLab interface. The main types of cells in a notebook are:

Code cells: Where you write and run Python (or other language) code.

Markdown cells: Where you write formatted text, documentation, etc.

Raw cells: For text not to be executed or formatted

When instructions say “use this command in a JupyterLab cell,” they mean: open a notebook in JupyterLab, click into a code cell, and type/run the command.

Notebook vs. Console

Notebook:

A notebook is a document made up of multiple cells (code, markdown, etc.), allowing you to mix code, results, and notes in a single file.

It’s ideal for data analysis, visualization, and sharing results.

You can run cells out of order and see outputs directly below each cell.

Notebooks are saved as .ipynb files and can be reopened, edited, and shared

Console:

A console in JupyterLab is more like a traditional REPL (Read-Eval-Print Loop) or interactive Python shell.

You type code and see results immediately, but you don’t get markdown, cell organization, or easy reproducibility.

Consoles are good for quick tests or debugging but aren’t meant for documenting workflows or sharing analyses

Which Should You Use?

For most data science, analysis, or reproducible research:
Use a notebook. It allows you to organize code, results, and documentation in one place. You can always open a console for quick experiments if needed

For quick experimentation or debugging:
The console is fine, but you lose the organizational and documentation benefits of notebooks.

Your Project Directory and File Access

If you started JupyterLab from your project directory (where expt1.csv and your virtual environment are), the notebook’s working directory should be correct. If you still get a “file not found” error, double-check the working directory in a notebook code cell:
import os
print(os.getcwd())
If it matches your project directory, your file path should work. If not, use a relative or absolute path to the CSV file.

Bottom line:
When someone says “run this in a JupyterLab cell,” they mean a code cell in a notebook you’ve opened in JupyterLab. For most workflows, stick with notebooks for organization, reproducibility, and sharing

Hope it helps…

Cheers from Rosika

nevj · May 20, 2025, 9:43pm

Hi Rosika,
It helps a lot, thank you,
More informative than the jupyter webpages.
Do you know, when I have finished a project in jupyterlab, how do I get my code and documents out of into to normat text files?
Regards
Neville

pdecker · May 21, 2025, 12:06am

I don’t have any hands-on use of notebooks so I can’t say for sure, but I think I’ve heard you can’t easily do that. The JupyterLabs like product called Marimo is able to do that. It’s just straight Python code. I don’t know that I would recommend chasing down the Marimo path, but maybe you want to give it a try.