No.
I can’t, and actually, I wonder what it matters. The idea is not to have fonts cluttering dropdown menus when choosing a font, right? And this is what it achieves.
I just tried. Removed a Sundanese font, which I will never use in my life. Rebooted and it disappeared from LibreOffice, GnuIMP etc. Still, the ttf file is still on my computer, which makes sense, in case there’s a website or a document I get, where they are used for decorative purposes.