Python question

Applications
1

I am having my first try at Python. I am working in Artix ( Arch based)
Python is installed by default

$ python
Python 3.13.3 (main, Apr  9 2025, 17:13:31) [GCC 14.2.1 20250207] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>>

I need to use some libraries from pandas

>>> import pandas as pd
Traceback (most recent call last):
  File "<python-input-0>", line 1, in <module>
    import pandas as pd
ModuleNotFoundError: No module named 'pandas'

I guess I need to install pandas first?
The documentation says I can install it with Anaconda ( what is that?) or from PyPI (what is that?) using pip.
I chose the latter
I found out what PyPI is
" PyPI, short for Python Package Index, is the official third-party package repository for the Python programming language. It is a central repository that hosts and distributes software packages for Python developers to use. Developers can upload Python packages to PyPI where they can be easily accessed by others.
"
Now where is ‘pip’?

trinity:[nevj]:~/juliawork/dermis.collagen.images$ which pip
which: no pip in (/usr/bin/vendor_perl/:/home/nevj/.juliaup/bin:/usr/local/bin:
/usr/bin:/bin:/usr/local/sbin:/usr/lib/jvm/default/bin:/usr/bin/site_perl:/usr/
bin/vendor_perl:/usr/bin/core_perl:/common/bin)

OK I need to install pip

# pacman -Ss pip
...
world/python-pip 25.0.1-1
    The PyPA recommended tool for installing Python packages
...

# pacman -Syu
# pacman -S python-pip
....
(8/8) installing python-pip

Now I try to use pip to install pandas

$ pip install pandas
error: externally-managed-environment

× This environment is externally managed
╰─> To install Python packages system-wide, try 'pacman -S
    python-xyz', where xyz is the package you are trying to
    install.
    
    If you wish to install a non-Arch-packaged Python package,
    create a virtual environment using 'python -m venv path/to/venv'.
    Then use path/to/venv/bin/python and path/to/venv/bin/pip.
    
    If you wish to install a non-Arch packaged Python application,
    it may be easiest to use 'pipx install xyz', which will manage a
    virtual environment for you. Make sure you have python-pipx
    installed via pacman.

I tried again as root … no luck
I tried using pipx … no luck
So I gave up on pip and used pacman to install pandas

$ pacman -Ss pandas
world/python-pandas 2.2.3-1
    High-performance, easy-to-use data structures and data analysis tools for
    Python
world/python-pandas-datareader 0.10.0-5
    Data readers extracted from the pandas codebase
world/python-numexpr 2.10.2-1
    Fast numerical array expression evaluator for Python, NumPy, PyTables,
    pandas

# pacman -S python-pandas
resolving dependencies...
....

That works

>>> import pandas as pd
>>> 
>>> df = pd.read_csv('expt1.csv')
>>> df
            Farm       Grade  Sheep  ...     Green      Blue  Count
0       Glensloy      smooth   3457  ...  0.304495  0.263945    732
1       Glensloy      smooth   3457  ...  0.233286  0.481588     41
2       Glensloy      smooth   3457  ...  0.285897  0.311324    374
3       Glensloy      smooth   3457  ...  0.232893  0.303401     49
4       Glensloy      smooth   3457  ...  0.204372  0.245076     87

I can import pandas and use pd.read_csv function

OK, I found a workaround
but
Question : What does this message mean

error: externally-managed-environment

Why is Artix stopping me from using pip and insisting I use the package system?

2 Likes
2

I tried to install pip awhile back and encountered the same thing. What I found here:

The “externally-managed-environment” error appears when pip is run in an environment that is not supported for direct package installation. This includes scenarios encountered in virtual environments created using tools like virtualenv or package managers like apt or dnf, which can create their own isolated environments that are not designed for pip installations.

Hope it helps!
Sheila

4 Likes
3

Hi Sheila,
Thanks, that link explains it all.
In my case I am using ‘system-python’, so I must use system installs of python packages.
It is protecting me from having pip damage my package system
Regards
Neville

4 Likes
4

Yes, that is what I found out when I could not install it : prevent damage to system from pip.

Sheila

3 Likes
5

So the obvious next question is… when can one use pip?
Is it only usable for a personally installed python, outside of the package system?

I am sure I have used pip before… probably in Debian… and it did not complain.
Is this something new?

1 Like
6

I don’t begin to understand all of this, do not remember what I was doing that said I needed pip, but I had put into my Linux notebook the following from Wikipedia;

Python is shipped with some basic built-in modules like the math module for extra maths operators and the re module for executing regular expressions. These are pre-included as they are considered quite useful and will be used quite a lot. However, more obscure or larger modules such as numpy (C based arrays) and others are not included - this is where pip comes in.

You can use the “package manager” to install, uninstall, update etc. any package within PyPI which is the Python Package Index. The result of this is that Python remains small, but there are immediately millions of free packages available for use if desired.

I thought since you are doing maths, etc. maybe you could understand it better than I.

Sheila

1 Like
7

I will try.
First find out where pip installs things?

Yeah… I am trying to replicate an analysis I did in R, with Python. I have already tried with Julia, and failed. I am doing better than that with Python.

2 Likes
8

Hi Neville, :waving_hand:

I did some research on this topic a while ago and came up with the following:

pip install vs apt install

pip install:

  • Used for: Installing Python packages from PyPI (Python Package Index).

  • Scope: Affects Python’s environment. Installs Python libraries, not system-wide software.

  • Example: pip install requests → installs the Python requests library.

  • Can be used in: A virtual environment (recommended), or globally (if needed).

  • Caution: Avoid using sudo pip install unless you really know what you’re doing—it can mess with system packages managed by apt.

apt install:

  • Used for: Installing system packages (e.g. programs, services, and some Python packages too) from Debian/Ubuntu repositories.

  • Scope: Installs to the entire system and is managed by the OS.

  • Example: apt install python3-requests → installs the system version of the requests package.

  • Safe: Because apt manages dependencies carefully and integrates with the system package manager.

When to use which?

  • Install a Python library in a project: pip install (ideally in a virtualenv)

  • Install a software tool like Git, Curl, or VLC: apt install

  • Install a Python module system-wide for scripts: apt install (if available), or pip in a virtualenv

  • Install latest version of a Python library: pip install (PyPI usually has newer versions)

Many greetings from Rosika :slightly_smiling_face:

3 Likes
9

Hi Rosika,
That is my case.
So I should use apt( or pacman) wherever possible, and only use pip if I need some weird package that is not in the package system … that is when it gets dangerous…
outside packzges are insecure

What is a virtualenv?

2 Likes
10

You’ve found most of the answer. Python is used by the distribution itself and they are trying to protect you from shooting yourself in the foot by preventing you from installing packages using pip to the global system. You can override this, but that is not recommended.

Instead, you should create a virtual environment for each project you are working on. There are different ways of doing that. The one I like is using a tool called UV.

I’ll try and gather some information rather than going off the top of my head.

6 Likes
11

So it would be like doing each R project in a separate directory.?

2 Likes
12

Hi Neville, :waving_hand:

I am using JupyterLab for working with/learning python.

  • I want Jupyter sandboxed and independent from system packages → Use a virtual environment

  • So, do I need Firejail if using venv?
    Yes, if I still want filesystem isolation. But venv only isolates Python packages, not the whole system, so Firejail is still useful.

Here´s my approach: Use venv + Firejail

  • Create a Virtual Environment (venv) for Jupyter
    By using a virtual environment, I ensure Jupyter’s dependencies are isolated from the rest of my system. This keeps things secure while giving control over Jupyter’s version and its packages.

  • Install Jupyter Inside the Virtual Environment
    This way, Jupyter will be isolated from system-wide Python packages, and one won’t have any conflicts.

  • Run Jupyter Inside a Firejail Sandbox
    Firejail will ensure that Jupyter operates in a confined environment, blocking access to other parts of my system and internet. The combination of venv and Firejail gives me double isolation.

Step-by-Step Setup:

  1. Install Python and pip (if not already installed):

sudo apt install python3 python3-pip python3-venv

  1. Create and Set Up the Virtual Environment:
mkdir -p ~/jupyter-env
cd ~/jupyter-env
python3 -m venv venv
source venv/bin/activate  # Activate the virtual environment
pip install jupyter      # Install Jupyter inside the venv
  1. Create the Firejail Profile:

Since I am using Firejail for isolation ChatGPT and I worked out a dedicated profile for it.

Create the Firejail profile:

mkdir -p ~/.config/firejail
nano ~/.config/firejail/jupyter.profile
[...]

I may provide it to you if you´re interested. Just let me know.

In the end I use it this way (my personal setup; I make use of fish):

  1. rosika@rosika-Lenovo-H520e ~> cd ~/jupyter-env; and sleep 1; and . venv/bin/activate.fish

  2. (venv) rosika@rosika-Lenovo-H520e ~/jupyter-env # from here I start the following:

  3. firejail --profile=~/.config/firejail/jupyter.profile jupyter lab --notebook-dir=/home/rosika/jupyter-test --no-browser

  4. then I´m presented with Jupyter Server´s output, which gives me URLs including tokens to access it via any browser on my system

  5. accessing JupyterLab via: firejail --private --dns=1.1.1.1 --dns=9.9.9.9 falkon -no-remote # making use of the URL/token JupyterLab provided via the command-line

So both falkon and JupyterLab are highly sandboxed.
It can´t get any more secure, I think. :wink:

Many greetings from Rosika :slightly_smiling_face:

P.S.:

  • exiting JupyterLab via the command line: 2 times “CTRL+C”
  • exiting venv: enter the command: deactivate
1 Like
13

That python is a real snake in the grass

3 Likes
14

It is named after Monty Python. Lots of snakes in logos and such though.

2 Likes
15

With my limited understanding of R, yes.

I am on vacation in New York City and today I have been sick. I will give a short answer and try to add more at some point.

The new utility I mentioned, UV, is nice because you can create a virtual environment, install pip packages, and even install multiple versions of Python. This way it will never interfere with the system version of Python.

First install uv with curl. It is safe in this case. Then create a virtual environment, install python, install pandas, and run your app named main.py. You can name it anything you want.

curl -LsSf https://astral.sh/uv/install.sh | sh
uv venv
.venv/bin/activate
uv python install
uv pip install pandas
uv run main.py

This is just a sketchy example of what you might do.

You can also use Anaconda to manage packages rather than pip or uv. I don’t have any hands on experience with Anaconda, so I’ll let you investigate that if you want.

3 Likes
16

That is really good, thanks.
And thank you for the effort while not feeling well. I hope you recover soon and can enjoy your vacation.

I like the idea of UV isolating me from the system python… there are too many things that depend on the system python being right… I should leave it alone.

Let me have a try with UV for a few days, before you bother with any more.

Not quite the same, I have found. A Python venv makes a new install of python, with its own libraries etc. Doing an R project in its own directory keeps the workspace separate , and enables multiple instances of R, but it does not duplicate the installed R.

3 Likes
17

Hi Rosika,
That is very detailed, thank you.
So you use Python-'s own venv utility… @pdecker uses UV
and you combine it with jupyterlab
and you sandbox the lot.

Looking for python3-venv

aur/py-mkvenv 1.6-1 (+2 0.00) 
    Automate creation of virtual environments in Python 3 projects

Is that it?

Not sure whether to go with venv or UV… I will try both.
Jupyterlab can wait… I am happy with the python prompt and an editor… dont like learning too many things at once.

I am getting somewhere with python… I was able to read in some data and do a statistical analysis … there will be a short writeup.

I think it is really important that I stop using the system python before I make a mess of the system. The documentation did not warn me about that. R does not have that problem because the system does not use R the way it uses Python.

I think you have really designed your python interface well.

Regards
Neville

4 Likes
18

Armed with all this help on virtual environments, I am going to try again.
This doc is worth a read

It would seem the main point of doing user projects in a virtual environment is to protect the system python from spurious installs that might interfere with it and hence with Linux.

I am working in Artix ( ie Arch)
I did a search for both venv and uv

world/python-uv 0.7.3-1 
    An extremely fast Python package installer and resolver written in Rust
aur/py-mkvenv 1.6-1 (+2 0.00) (Installed)
    Automate creation of virtual environments in Python 3 projects
aur/venv-wrapper-bin 2.0.2-1 (+0 0.00) 
    Simple python virtual environment management

They seem to be right, so lets install both

$ yay -S py-mkvenv
AUR Explicit (1): py-mkvenv-1.6-1  

$pacman -S python-uv
python-uv-0.7.3-1

Lets try venv first

trinity:[nevj]:~$ mkdir Projects
trinity:[nevj]:~$ cd Projects
trinity:[nevj]:~/Projects$ ls
trinity:[nevj]:~/Projects$ mkdir collagen
trinity:[nevj]:~/Projects$ cd collagen
trinity:[nevj]:~/Projects/collagen$ python -m venv venv
trinity:[nevj]:~/Projects/collagen$ ls
venv
trinity:[nevj]:~/Projects/collagen$ ls -a venv
.  ..  bin  .gitignore  include  lib  lib64  pyvenv.cfg

That looks OK. I made a project directory called collagen, and inside it python made a venv subdirectory.

I assume I need to be in the collagen directory, and to activate the virtual environment with

trinity:[nevj]:~/Projects/collagen$ source venv/bin/activate
(venv) trinity:[nevj]:~/Projects/collagen$

It did something … (venv) appears in the prompt.
I can test it by asking where python is

venv) trinity:[nevj]:~/Projects/collagen$ which python
/home/nevj/Projects/collagen/venv/bin/python

trinity:[nevj]:~$ which python
/usr/bin/python

so inside the collagen project directory I am using /home/nevj/Projects/collagen/venv/bin/python and outside it I am using /usr/bin/python, ie the system python.

Success!

Now I should be able to use ‘pip’ to install pandas

venv) trinity:[nevj]:~/Projects/collagen$ pip install pandas
Collecting pandas
  Using cached pandas-2.2.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (89 kB)
Collecting numpy>=1.26.0 (from pandas)
  Downloading numpy-2.2.6-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl.metadata (62 kB)
Collecting python-dateutil>=2.8.2 (from pandas)
  Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl.metadata (8.4 kB)
Collecting pytz>=2020.1 (from pandas)
  Using cached pytz-2025.2-py2.py3-none-any.whl.metadata (22 kB)
Collecting tzdata>=2022.7 (from pandas)
  Using cached tzdata-2025.2-py2.py3-none-any.whl.metadata (1.4 kB)
Collecting six>=1.5 (from python-dateutil>=2.8.2->pandas)
  Using cached six-1.17.0-py2.py3-none-any.whl.metadata (1.7 kB)
Using cached pandas-2.2.3-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.7 MB)
Downloading numpy-2.2.6-cp313-cp313-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (16.5 MB)
   ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 16.5/16.5 MB 9.5 MB/s eta 0:00:00
Using cached python_dateutil-2.9.0.post0-py2.py3-none-any.whl (229 kB)
Using cached pytz-2025.2-py2.py3-none-any.whl (509 kB)
Using cached tzdata-2025.2-py2.py3-none-any.whl (347 kB)
Using cached six-1.17.0-py2.py3-none-any.whl (11 kB)
Installing collected packages: pytz, tzdata, six, numpy, python-dateutil, pandas
Successfully installed numpy-2.2.6 pandas-2.2.3 python-dateutil-2.9.0.post0 pytz-2025.2 six-1.17.0 tzdata-2025.2

[notice] A new release of pip is available: 25.0.1 -> 25.1.1
[notice] To update, run: pip install --upgrade pip

It works.
So I seem to have a proper virtual environment.

What binarie do I now have in venv/bin ?

venv) trinity:[nevj]:~/Projects/collagen$ ls venv/bin
activate      activate.fish  f2py          pip   pip3.13  python3
activate.csh  Activate.ps1   numpy-config  pip3  python   python3.13

I can see python, pip, and activate… but not pandas? Pandas in in the lib subdir

(venv) trinity:[nevj]:~/Projects/collagen$ ls venv/lib/python3.13/site-packages
dateutil                __pycache__
numpy                   python_dateutil-2.9.0.post0.dist-info
numpy-2.2.6.dist-info   pytz
numpy.libs              pytz-2025.2.dist-info
pandas                  six-1.17.0.dist-info
pandas-2.2.3.dist-info  six.py
pip                     tzdata
pip-25.0.1.dist-info    tzdata-2025.2.dist-info

and it is also in venv/lib64/python3.13/site-packages.
There is also a config file venv/pyvenv.cfg

home = /usr/bin
include-system-site-packages = false
version = 3.13.3
executable = /usr/bin/python3.13
command = /usr/bin/python -m venv /home/nevj/Projects/collagen/venv

There is nothing in the include subdir.
So the whole isolated environment is there in the project directory.
I assume I should put project workfiles in ~/Projects/collagen and do all my work in there?

Next step… do it all again with uv

5 Likes
19

Hi Neville, :waving_hand:

Yes, exactly. That´s my setup. And the fact that the browser is sandboxed as well makes it extremely secure. It least that´s the hope. :blush:

Of course, Neville.
But there´s no big learning curve involved, it seems.
For me the advanteges I get with my setup outweigh any other considerations.

I don´t know what´s it like with Arch-based systems.
All I can tell is that I installed it thus:
sudo apt install python3 python3-pip python3-venv.

Perhaps someone else can help.

With me it´s like this:

cd ~/jupyter-env; and sleep 1; and . venv/bin/activate.fish
(venv) rosika@rosika-Lenovo-H520e ~/jupyter-env> which python
/home/rosika/jupyter-env/venv/bin/python

Many greetings from Rosika :slightly_smiling_face:

3 Likes
20

Hi Rosika,
I will get to jupyterlab.
As you can see, I want to do things one step at a time and understand what is underneath before building on it.
Did you notice how I got into trouble with my ‘dive straight in’ approach? That is when I learn… the problems have to surface before learning occurs for me.
I am fortunate to have a couple of nice rescuers.
Regards
Neville

4 Likes