NFS filesystem export conundrum

nevj · May 26, 2024, 12:16pm

I have a linux sytem ( Antix) in a VM ( qemu/KVM)
I have the following in /etc/exports

# /etc/exports: the access control list for filesystems which may be exported
#		to NFS clients.  See exports(5).
/home/nevj trinity(insecure,rw,sync,no_subtree_check,no_root_squash)

trinity is the hostname of the VM host .
It does not work

mount.nfs: mount(2): Permission denied
mount.nfs: trying text-based options 'addr=192.168.122.109'
mount.nfs: prog 100003, trying vers=3, prot=6
mount.nfs: portmap query retrying: RPC: Program not registered
mount.nfs: prog 100003, trying vers=3, prot=17
mount.nfs: portmap query failed: RPC: Program not registered
mount.nfs: access denied by server while mounting 192.168.122.109:/home/nevj

but , if I change /etc/exports to

/home/nevj *(insecure,rw,sync,no_subtree_check,no_root_squash)

which means that any nfs client system can access the filesystem
it works.

That is crazy.  Why can I not export to a specific host from a VM.

If I do it in the other direction... ie export from the host system and access it from a VM with an NFS client, it works.  I only need to 
export to one specific VM system in that direction.

Why would an NFS server in a VM require exporting to everyone?

Rosika · May 26, 2024, 2:09pm

Hi Neville,

I did a bit of research on the subject:

“Permission Denied”:
The client doesn’t have the necessary permissions to mount the NFS share?
NFS server not recognizing the client?
Some misconfiguration in the NFS server’s access control?
“RPC Program Not Registered”:
NFS server’s RPC (Remote Procedure Call) program not registered?
NFS server not being properly started or configured?

Given that exporting to all clients (*) works but exporting to a specific client doesn’t…
…might there be an issue with how the NFS server is recognizing or authenticating the client?

Possible troubleshooting:

NFS server logs (/var/log/syslog or /var/log/messages):
are there any errors or warnings related to NFS exports or RPC registration?

Is the hostname of the client VM correctly specified in the /etc/exports file?

Or perhaps the firewall
Are ports for NFS (TCP/UDP ports 2049 and 111) open on both the server and client machines?

Or what about “Client Mount Options”:
Perhaps specify the NFS version explicitly when mounting the share on the client.
e.g.: mount -t nfs -o vers=3 server:/path/to/share /mnt

That´s all I could come up with.
It may not be of much help. Sorry, Neville.

Many greetings from Rosika

kovacslt · May 26, 2024, 2:16pm

Does the VM have a way to resolve “trinity”?
If you enter ping trinity, do you get a normal ping output with times, or something about host unreachable?

pdecker · May 26, 2024, 3:01pm

Is trinity the FQDN of the host or is it something like trinity.local?

Maybe it needs a reverse DNS entry so it can resolve the requesting IP to the name trinity?

nevj · May 26, 2024, 11:32pm

Hi Rosika,

I agree

I want to investigate that… I used hostname… it might need an fqdn

Your diagnosis is very systematic, thank you.
It may have nothing to do with the server being in a VM
Regards
Neville

nevj · May 26, 2024, 11:36pm

Yes I can do ping trinity or ssh trinity from the VM.
So it can resolve trinity
see my reply to Rosika… it may need an fqdn in the /etc/exports file. ,… the manpage is vague about that.
When I did it in the reverse direction (nfs server in host) I used an IP number … that works.

nevj · May 26, 2024, 11:41pm

No, trinity is the hostname only.
I think that may be it

What is that?
The manpage is vague on what it should be… an IP number or * works

nevj · May 26, 2024, 11:47pm

I need to file that for future reference.
The meaning of those error messages is rather obscure.

daniel.m.tripp · May 27, 2024, 12:03am

“reverse DNS” is a PTR record - this is generally default behaviour - i.e. if I create an A (authoritative) record, most “modern” DNS solutions will create a “reverse DNS” PTR record - e.g.

A record (in one bind database)
HOSTA : 192.168.1.253

PTR record (in yet another database)
PTR : 253.1.168.192

(or something like that - it’s kinda ugly - actually no - it’s VERY ugly!)

What this means is you can dig or nslookup BOTH - i.e. “nslookup 192.168.1.253” should return HOSTA.$fqdn

Note : hand crafting these things in bind databases is HIDEOUS! And incredibly prone to human error. I have one customer that still does it like this - and to make it worse - they host this bind stuff on SOLARIS!

ALL my other customers have moved ALL DNS to Windows servers - and much as I hate to say it - it’s an order of MAGNITUDE easier to manage.

Last time I tried to host my own internal DNS, I used a BananaPi ARM board, but I didn’t manage the bind databases by hand - they’re SO UGLY and unfriendly, I installed WebMin and managed them that way… But I found that aside from a few edge cases, Avahi (i.e. $HOSTNAME.local) worked where I needed it to work.

On my FreeBSD (TrueNAS) I allow NFS by network - i.e. (/etc/exports) :

/mnt/BARGEARSE -alldirs -network x.1.x.0/24
/mnt/BARGEARSE -alldirs -network x.2.x.0/24

I have two potential VLANs… I also don’t edit this file by hand - the TrueNAS Web UI does this - and it would be stupid to even try, as “/” and “/etc” are on readonly fileystem.

Have you considered allowing by network? i.e. x.x.x.0/$CIDR (where $CIDR is your netmask in shorthand, e.g. 255.255.255.0 = 24 )

I generally mount that NFS share from other Linux hosts using “baphomet.local” name - but - I don’t / can’t 100% rely on Avahi / Zeroconf / Bonjour resolving that for me - so I have an entry in /etc/hosts for “baphomet.local” (e.g. in Red Hat land Avahi is woeful - no idea why - works mostly flawlessly in Debian based distros, and MacOS).

x.x.x.x baphomet.local baphomet

Managing multiple /etc/hosts files across my systems with lots of entries would be a PITA - I only have just that one above…

Do you know how your systems are able to resolve “trinity” - what about “nslookup trinity”?

nevj · May 27, 2024, 12:32am

No, I have not considered that , but it is a good idea.
VM’s in particular are a nuisance because they get random IP numbers, but they are all on the same network.

I can remember bind from SunOS days. What I remember most is the swearing from my offsider .

I have a local domain name, but I dont have a local nameserver, so I am not sure how it would resolve an fqdn, unless I setup the hosts file.
In general VM’s seem to know the hostname of the system hosting the VM’s, without me having to set up anything, but not in this case with NFS exports?

daniel.m.tripp · May 27, 2024, 6:15am

When you say “local domain name” - is that just something arbitrary you’ve made up to call your network?

I suspect some of your machines are using Avahi.

Note also - how naming works, priorities, is usually set in the /etc/nsswitch.conf (which is something UNIX inherited from Sun Microsystems) :

╭─x@titan ~/bin  ‹main*› 
╰─➤  cat /etc/nsswitch.conf 
# /etc/nsswitch.conf
#
# Example configuration of GNU Name Service Switch functionality.
# If you have the `glibc-doc-reference' and `info' packages installed, try:
# `info libc "Name Service Switch"' for information about this file.

passwd:         files systemd
group:          files systemd
shadow:         files
gshadow:        files

hosts:          files mdns4_minimal [NOTFOUND=return] dns myhostname
networks:       files

protocols:      db files
services:       db files
ethers:         db files
rpc:            db files

netgroup:       nis

Don’t ask me how that last line got in there - I haven’t used NIS in anger for a few years now - and NEVER ever implemented it in my home network - that’s just how my nsswitch.conf file came from Pop!_OS install…

Anyway - that line :

hosts:          files mdns4_minimal [NOTFOUND=return] dns myhostname

tells it to first try “files” (i.e. /etc/hosts), then “mdns4_minimual” which is essentially Avahi (Microsoft “Zeroconf”), then “dns” (which I don’t have internally)… please don’t ask me to explain what “[NOTFOUND=return]” means

Ever since around Debian 9 and Ubuntu 16, “avahi” just seems to work - and is usually installed “out of the box” (you had to install it manually on Debian 8 Jessie). But I’ve never had much luck with Avahi on RPM based distros…

nevj · May 27, 2024, 8:14am

You are right, the host (Void) has avahi running.
The VM (Antix) does not have avahi running

When you say “local domain name” - is that just something arbitrary you’ve made up to call your network?

Yes I have always called it trinity.colrose.com.au and that is set in
the /etc/hosts file of the VM host.

nsswitch.conf has

hosts: 		files mdns dns

in Void,and

hosts:          files mdns4_minimal [NOTFOUND=return] dns myhostname

in the VM.
I suspect it only ever uses files ( ie /etc/hosts)

What puzzles me is that I can ping and ssh trinity from within the VM, but nfs seems not to want to export a filesystem to it.
Trinity ( the Void host) is not defined in /etc/hosts in the VM… it does not seem to need it for ping or ssh?

I added the following to /etc/hosts in the VM

192.168.122.1 trinity.colrose.com.au trinity

It makes no difference.

The VM (Antix) has s6 init system
I suspect some problems there not starting nfsd properly
All the daemons are running

root@antix-init-diversity:~# ps ax | grep rpc
 2441 ?        S      0:00 s6-supervise rpc.nfsd-log
 2442 ?        S      0:02 s6-supervise rpc.nfsd
 2443 ?        S      0:00 s6-supervise rpc.mountd-log
 2444 ?        S      0:00 s6-supervise rpcbind-log
 2445 ?        S      0:00 s6-supervise rpcbind
 2446 ?        S      0:00 s6-supervise rpc.mountd
 2465 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/rpc.mountd
 2466 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/rpcbind
 2475 ?        Ss     0:00 /sbin/rpcbind -f
 2479 ?        Ss     0:00 /sbin/rpc.mountd -F
 2485 ?        I<     0:00 [rpciod]
 6179 ?        S      0:00 s6-supervise rpc.idmapd-log
 6180 ?        S      0:00 s6-supervise rpc.idmapd
 6181 ?        S      0:00 s6-supervise rpc.statd-log
 6182 ?        S      0:00 s6-supervise rpc.statd
 6196 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/rpc.statd
 6197 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/rpc.idmapd
 6204 ?        Ss     0:00 /sbin/rpc.idmapd -f
 6205 ?        Ss     0:00 /sbin/rpc.statd -Fd
11355 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/rpc.nfsd
14858 pts/1    S+     0:00 grep rpc
root@antix-init-diversity:~# ps ax | grep nfs
 2441 ?        S      0:00 s6-supervise rpc.nfsd-log
 2442 ?        S      0:02 s6-supervise rpc.nfsd
 2492 ?        I<     0:00 [nfsiod]
 2506 ?        S      0:00 [nfsd]
 2507 ?        S      0:00 [nfsd]
 2508 ?        S      0:00 [nfsd]
 2509 ?        S      0:00 [nfsd]
 2510 ?        S      0:00 [nfsd]
 2511 ?        S      0:00 [nfsd]
 2512 ?        S      0:00 [nfsd]
 2513 ?        S      0:00 [nfsd]
11355 ?        Ss     0:00 /usr/bin/s6-log -d3 n3 s1000000 /var/log/66/rpc.nfsd
14951 pts/1    S+     0:00 grep nfs
[nevj@trinity s6]$

It might be a matter of the order in which they are started… there are dependencies.

nevj · May 27, 2024, 9:44am

No it is not that.
There is a complicated relation between what is in /etc/exports and what is in /etc/hosts … in the system which is the nfs-server. ( in my case it was Antix in a VM)

If /etc/hosts only defines localhost
then
/etc/exports has to export either to * or to the fqdn (trinity.colrose.com.au)

but
if /etc/hosts contains the line

192.168.122.1 trinity.colrose.com.au trinity

then /etc/exports can export to *, or fqdn or hostname (trinity)

I discovered this by studying showmount -e after exportfs -a
It would seem exportfs , if given a hostname, looks up /etc/hosts and converts it to an fqdn ( it it is defined in /;etc/hosts)

So, original problem solved.
I still have some issue of S6 not reliably starting nfsd… it works sometimes.

Thanks to @Rosika , @pdecker , @kovacslt , and @daniel.m.tripp
for suggestions.
You were all correct… it is a name resolving problem… obscured by my not having S6 starting nfsd properly.

nevj · May 27, 2024, 12:59pm

I did work out what was wrong in S6.
It was indeed a dependency issue… when starting daemons it is important to start them in the correct order if there are dependencies. s6-66 has mechanisms for this and I am still learning the ropes there.

I also need to look at exporting to a network as suggested by @daniel.m.tripp .
That seems to be a good way to deal with VM’s… export to the virtual network as it will contain all the VM’s plus the host.

It is difficult diagnosing when there are 2 problems.

pdecker · May 27, 2024, 3:06pm

daniel.m.tripp · May 27, 2024, 11:45pm

nevj · May 27, 2024, 11:56pm

We could remove all this DNS nonsense if we used IP numbers only.
That might also be the solution to the autocorrect issue, see

give eveyone an IP number