Of course there are always exceptions, but even those tend to be very limited.
For example, you were the one talking about your 400GB swap device — on a setup like that, the moment you start to consume swap, your memory speed no longer makes ANY difference because it’s going to spend all of its time waiting on paging. The memory will never be running at max speed, because even PCIe/NVME swap is so much slower.
That’s why it’s usually better to run with more memory, even if it’s slower, than it is to run with fast, constrained memory. Of course, a machine with 64 GB of faster RAM can (under ideal circumstances) outperform a machine with 64 GB of slower RAM, but a machine with 256 GB of even much slower RAM is generally going to run rings around them both, for all applications.