I tested Font Manager in Ubuntu 22.04. I also have a whole bunch of Noto fonts for different languages (which is useful for me, as I do work with some Asian fonts in particular).
I tried to disable most of them with the Font Manager, and they disappeared from the Menu fonts list in Libre Office without restarting the OS or the session (but Libre Office needs to be restarded).
Strangely, they are still in the Fonts list in Inkscape, even after restarting the OS. At this point, I do not understand why.
Most fonts that I have added, I added them by double-clicking on a downloaded font file, and they are not linked to a package.
On the command line, I did
sudo apt list | grep fonts-noto
to check packages and I only get 11 noto fonts
So my guess is that one can safely delete unused fonts (or move the fonts files somewhere else) as long as they are not installed through a package manager, and use apt list to check if it is the case or not.