NFS Woes!

[Charles Lindsey-2]

Sorry! It's me again :-( .

I am now running SlugosBE with the rootfs NFS-mounted from another machine.
That is mostly working fine. But there  are two problems.

1. It seems that NFS will not let you export files that have been NFS-imported from
elsewhere (well, exportfs will let you do it, but remote clients will decline to  mount
it with 'Permission Denied'. This is not an unreasonable restriction, but note that you
cannot even mount tmpfs files such as those in /tmp, because /tmp is _really_
/var/volatile/tmp, and it needs to go to the remotely mounted rootfs to disentangle
the /var/volatile bit.

But there seems to be a workaround. It you create a top-level tmpfs file system
(e.g. /mounts), then it seems it can be legitimately exported (so I in fact mounted my
USB stick in there, and the contents of the stick are visible from the slug). Next
problem was to make them visible to my main machine, and that brings me to the
next problem:

2. The client seems willing to mount that /mounts, and it persuades the slug to
respond to the MOUNT3 packet, but then it fails to make an NFS connection. It
seems that the nfsd on the nslug had failed to tell the portmapper what port number
it would like to be contacted upon. If I look in /var/run/portmap_mapping, I can see
100000 2 6 111 1
100000 2 17 111 1
100005 1 17 1022 1
100005 1 6 601 1
100005 2 17 1022 1
100005 2 6 601 1
100005 3 17 1022 1
100005 3 6 601 1
100024 1 17 605 1
100024 1 6 608 1
but 100003 (for NFS) is conspicuously absent from that list, and without it the client
refuses to complete the mount (the 100005 is the MOUNT itself, and is fine).

AFFAICT, nfsd is properly installed; modprobe is happy with it, and /proc/modules
contains:
nls_utf8 1120 1 - Live 0xbf0ad000
nls_cp437 4608 1 - Live 0xbf0aa000
vfat 10496 1 - Live 0xbf0a6000
fat 42908 1 vfat, Live 0xbf09a000
nls_base 5024 4 nls_utf8,nls_cp437,vfat,fat, Live 0xbf097000
nfsd 186588 0 - Live 0xbf068000
exportfs 4352 1 nfsd, Live 0xbf065000
ohci_hcd 16804 0 - Live 0xbf05f000
ehci_hcd 30252 0 - Live 0xbf056000
nfs 108576 1 - Live 0xbf03a000
lockd 51384 2 nfsd,nfs, Live 0xbf02c000
sunrpc 131920 4 nfsd,nfs,lockd, Live 0xbf00a000
ixp4xx_mac 14612 0 - Live 0xbf005000
ixp4xx_qmgr 5388 5 ixp4xx_mac, Live 0xbf002000
mii 3424 1 ixp4xx_mac, Live 0xbf000000

So what am I missing?

And a final niggle:

3. I installed module-init-tools-doc, but the list of files it brought over was empty.

[Charles Lindsey-2]

--- In nslu2-linux@..., "clerew5" <clerew5@...> wrote:

> 1. It seems that NFS will not let you export files that have been NFS-imported from
> elsewhere (well, exportfs will let you do it, but remote clients will decline to  mount
> it with 'Permission Denied'. This is not an unreasonable restriction, but note that you
> cannot even mount tmpfs files such as those in /tmp, because /tmp is _really_
> /var/volatile/tmp, and it needs to go to the remotely mounted rootfs to disentangle
> the /var/volatile bit.
>
> But there seems to be a workaround. It you create a top-level tmpfs file system
> (e.g. /mounts), then it seems it can be legitimately exported (so I in fact mounted my
> USB stick in there, and the contents of the stick are visible from the slug). Next
> problem was to make them visible to my main machine, and that brings me to the
> next problem:

I am now less convinced with that evaluation, or with the workaround proposed.
See below:

>
> 2. The client seems willing to mount that /mounts, and it persuades the slug to
> respond to the MOUNT3 packet, but then it fails to make an NFS connection. It
> seems that the nfsd on the nslug had failed to tell the portmapper what port number
> it would like to be contacted upon. If I look in /var/run/portmap_mapping, I can see
> 100000 2 6 111 1
> 100000 2 17 111 1
> 100005 1 17 1022 1
> 100005 1 6 601 1
> 100005 2 17 1022 1
> 100005 2 6 601 1
> 100005 3 17 1022 1
> 100005 3 6 601 1
> 100024 1 17 605 1
> 100024 1 6 608 1
> but 100003 (for NFS) is conspicuously absent from that list, and without it the client
> refuses to complete the mount (the 100005 is the MOUNT itself, and is fine).

OK, I am a bit further. I got out of that state by Rebooting. So now 100003 appears in
that list.

But Mount by a client still fails. Indeed, the very first command sent (PUTROOTFH),
which is supposed to obtain a 'file handle' for the root of the server's filesystem,
returns with NFS4ERR_NOENT which, according to RFC3530, is not even an
allowed response to PUTROOTFH. Here is some snoop output:

RPC:  Program = 100003 (NFS), version = 4, procedure = 0
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 0 (Null procedure)
NFS:  
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 0 (Null procedure)
NFS:  
RPC:  Program = 100003 (NFS), version = 4, procedure = 1
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 1 (Compound)
NFS:  Tag = secinfo    
NFS:  Minor version = 0
NFS:  Number of operations = 3
NFS:  
NFS:  Op = 24 (PUTROOTFH)
NFS:  
NFS:  Op = 15 (LOOKUP)
NFS:  media
NFS:  
NFS:  Op = 33 (SECINFO)
NFS:  Name = sda1
NFS:  
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 1 (Compound)
NFS:  Tag = secinfo    
NFS:  Status = 2 (NFS4ERR_NOENT)
NFS:  Number of results = 1
NFS:  
NFS:  Op = 24 (PUTROOTFH)
NFS:  Status = 2 (NFS4ERR_NOENT)
NFS:  
RPC:  Program = 100003 (NFS), version = 4, procedure = 1
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 1 (Compound)
NFS:  Tag = mount      
NFS:  Minor version = 0
NFS:  Number of operations = 8
NFS:  
NFS:  Op = 24 (PUTROOTFH)
NFS:  
NFS:  Op = 10 (GETFH)
NFS:  
NFS:  Op = 15 (LOOKUP)
NFS:  media
NFS:  
NFS:  Op = 10 (GETFH)
NFS:  
NFS:  Op = 9 (GETATTR)
NFS:    0x67   SYMLINK_SUPPORT LINK_SUPPORT FH_EXPIRE_TYPE TYPE
SUPPORTED_ATTRS
NFS:    0x01   FSID
NFS:    0x00  
NFS:    0xc8   MAXWRITE MAXREAD MAXFILESIZE
NFS:    0x00  
NFS:    0x00  
NFS:    0x00  
NFS:    0x00  
NFS:  
NFS:  Op = 15 (LOOKUP)
NFS:  sda1
NFS:  
NFS:  Op = 10 (GETFH)
NFS:  
NFS:  Op = 9 (GETATTR)
NFS:    0x67   SYMLINK_SUPPORT LINK_SUPPORT FH_EXPIRE_TYPE TYPE
SUPPORTED_ATTRS
NFS:    0x01   FSID
NFS:    0x00  
NFS:    0xc8   MAXWRITE MAXREAD MAXFILESIZE
NFS:    0x00  
NFS:    0x00  
NFS:    0x00  
NFS:    0x00  
NFS:  
NFS:  ----- Sun NFS -----
NFS:  
NFS:  Proc = 1 (Compound)
NFS:  Tag = mount      
NFS:  Status = 2 (NFS4ERR_NOENT)
NFS:  Number of results = 1
NFS:  
NFS:  Op = 24 (PUTROOTFH)
NFS:  Status = 2 (NFS4ERR_NOENT)
NFS:  

And it continues in that vein, always rejecting the PUTROOTFH with NFS4ERR_NOENT

So can you suggest what I might try next?

[Charles Lindsey-2]

--- In nslu2-linux@..., "clerew5" <clerew5@...> wrote:
>
> --- In nslu2-linux@..., "clerew5" <clerew5@> wrote:

Hmmmmm! I seem to get to answer my own question again :-( .
>
> > 1. It seems that NFS will not let you export files that have been NFS-imported from
> > elsewhere (well, exportfs will let you do it, but remote clients will decline to  mount
> > it with 'Permission Denied'. This is not an unreasonable restriction, but note that you
> > cannot even mount tmpfs files such as those in /tmp, because /tmp is _really_
> > /var/volatile/tmp, and it needs to go to the remotely mounted rootfs to disentangle
> > the /var/volatile bit.

It turns out to be a bit more complicated than that. In order to export something
successfully, you have three hurdles to surmount (and don't think that because
exportfs didn't complain and prints out a list of what it thinks it has exported
that the kernel believes it; you have to look at /proc/net/rpc/nfsd.export/content
and .../nfsd.fh/content to see what *really* worked, and you have to set debugging
level 4 in /proc/sys/sunrpc/nfsd_debug to see why things did not work).

Anyway, here are the three hurdles:

1. From the directory you want to export, it works back to the point where
it was mounted (it needs a File Handle for that point). If it is a device
(with majhor/minor numbers) AND the Filesystem on that device *always*
uses a device (FS_REQUIRES_DEV flag set), then fine. The 'usual suspects'
fat/ext/ufs/isofs and even romfs are fine, but nfs/jffs2/ramfs/tmpfs (and of course
procfs/sysfs/etc) are not.

For those others, there is an alternative which is to supply a unique 'fsid' number
by giving the option '-o fsid=<n>' in the exportfs cmmand (and there is a third
alternative to use a "uuid", which I have not investigated).

2. Once it is able to construct an FH (from major/minor, or fsid or uuid), it next
asks whether that FS *allows* exports. Again, most of the 'usual suspects' do
(but seemingly not ufs/romfs) and, somewhat surprisingly, tmpfs is OK. But nfs
and jffs2 are strictly verboten, so there is NoWay you are going to inspect
the Slug's builtin Flash over NFS.

3. And then you can access those files over NFS *provided* your client uses
(or can be told to use) NFS Version 3. But if your client is using NFS Version 4,
then it demands to see an FH for the root of your whole filesystem; moreover
it expects that root to have fsid==0 (seemingly so even if it is mounted on a
hard disc). So you have to do
      exportfs -o fsid=0 client:/
(apparently whether or not you actually intend to mount '/' over NFS).

And there is a further problem with Version 4 - you need to have the 'md5'
kernel module loaded (there seems to be a missing dependency there).

And so finally I got my system, which uses 'turnup nfs', to allow me to
mount things (even '/', though usbsticks on /media/sda1 was my real
need) from outside using NFS Version 4, though to do that I had to arrange
that '/' was actually mounted on tmpfs, with separate mountings of /usr, /etc,
/sbin etc (not /var) from my main machine.

So it looks like I have some more wiki writing to do, and I shall report the
script that created that strange setup to the development list.

[Charles Lindsey-2]

--- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:

>
> --- In nslu2-linux@..., "clerew5" <clerew5@> wrote:
> >
> > --- In nslu2-linux@..., "clerew5" <clerew5@> wrote:
>
> Hmmmmm! I seem to get to answer my own question again :-( .
> >
> > > 1. It seems that NFS will not let you export files that have been NFS-imported from
> 3. And then you can access those files over NFS *provided* your client uses
> (or can be told to use) NFS Version 3. But if your client is using NFS Version 4,
> then it demands .....
>
> And there is a further problem with Version 4 - you need to have the 'md5'
> kernel module loaded (there seems to be a missing dependency there).

And I now find that it needs the idmapd daemon to be running before it will
mount files using NFS Version 4.

Is this daemon available anywhere on this site (I can't find it), and if not
could someone with the necessary compiling facilities please put it up?

And that looks like another missing dependency :-( .

[Mike (mwester)]

Charles Lindsey wrote:
> And I now find that it needs the idmapd daemon to be running before it will
> mount files using NFS Version 4.
>
> Is this daemon available anywhere on this site (I can't find it), and if not
> could someone with the necessary compiling facilities please put it up?
>
> And that looks like another missing dependency :-( .

NFS v4 is not supported, so it's no surprise that there are missing
dependencies.  I took a look at it around the SlugOS 4.8 release
timeframe (December of last year), and it didn't look like it was yet
ported to OE -- and doing so was non-trivial.

In the spirit of open-source, what that usually means is that those with
the need should do the work :)  Things might be different in OE by now
(i.e. somebody may have put the necessary stuff to support the full NFS
v4 stuff in OE for one project or another, if so we just need for
someone to build it and test it.  Otherwise, someone will have to figure
out what's involved and build the initial support (i.e. bitbake recipes)
to get it working.

Mike (mwester)

P.S. -- a bit of digging on some of the other stuff you've reported
shows that NFS exporting of JFFS2 is simply not a supported thing to do.
 Kernel patches exist, but it seems that either nobody has sent the
requisite patches to the maintainers, or perhaps the patches have been
rejected for one reason or another.  If you feel strongly that NFS
exporting tmpfs or jffs2 partitions is important, you might try to
contact the authors of the patches to see what ever happened to some of
those.

[Charles Lindsey-2]

--- In nslu2-linux@..., "Mike (mwester)" <mwester@...> wrote:

>
> Charles Lindsey wrote:
> > And I now find that it needs the idmapd daemon to be running before it will
> > mount files using NFS Version 4.
> >
> > Is this daemon available anywhere on this site (I can't find it), and if not
> > could someone with the necessary compiling facilities please put it up?
> >
> > And that looks like another missing dependency :-( .
>
> NFS v4 is not supported, so it's no surprise that there are missing
> dependencies.  I took a look at it around the SlugOS 4.8 release
> timeframe (December of last year), and it didn't look like it was yet
> ported to OE -- and doing so was non-trivial.

Slugos and the NFSD that comes with it purport to support NFS Version 4
(incoming requests, that is) because it correctly accepts the NULL4
command when it arrives (which causes the client to continue with
Version 4 instead of dropping down to Version 3). Indeed, it works
with Version 4 so long as you are trying to mount '/', but for mounting
anything else it gives you
   Jul 22 19:01:27 (none) user.warn kernel: nfsd: nfsv4 idmapping
         failing: has idmapd not been started?
in the syslog.

> In the spirit of open-source, what that usually means is that those with
> the need should do the work :)

Indeed, but until I have fixed myself an ARM cross-compiler, I am not in
a position to fix it.

I can force my system to use V3 in the meantime, but as a matter of principle
it ought to be put right. Though I do get the impression that NFS V4 is a
fairly recent addition to Linux and there are other reports of its flakiness.

>  Things might be different in OE by now

OE == Outlook Express??? I don't allow any Microsoft products anywhere
near my system :-) .

> P.S. -- a bit of digging on some of the other stuff you've reported
> shows that NFS exporting of JFFS2 is simply not a supported thing to do.

Sure, I can manage without that, but it needs documenting in the relevant Wiki,
which I shall do.

[Mike (mwester)]

Charles Lindsey wrote:

> --- In nslu2-linux@..., "Mike (mwester)" <mwester@...> wrote:
>> Charles Lindsey wrote:
>>> And I now find that it needs the idmapd daemon to be running before it will
>>> mount files using NFS Version 4.
>>>
>>> Is this daemon available anywhere on this site (I can't find it), and if not
>>> could someone with the necessary compiling facilities please put it up?
>>>
>>> And that looks like another missing dependency :-( .
>> NFS v4 is not supported, so it's no surprise that there are missing
>> dependencies.  I took a look at it around the SlugOS 4.8 release
>> timeframe (December of last year), and it didn't look like it was yet
>> ported to OE -- and doing so was non-trivial.
>
> Slugos and the NFSD that comes with it purport to support NFS Version 4
> (incoming requests, that is) because it correctly accepts the NULL4
> command when it arrives (which causes the client to continue with
> Version 4 instead of dropping down to Version 3). Indeed, it works
> with Version 4 so long as you are trying to mount '/', but for mounting
> anything else it gives you
>    Jul 22 19:01:27 (none) user.warn kernel: nfsd: nfsv4 idmapping
>          failing: has idmapd not been started?
> in the syslog.

No, that does not indicate that it "purports to support NFS Version 4";
that only indicates that I didn't go through the sources and remove that
stuff!

>> In the spirit of open-source, what that usually means is that those with
>> the need should do the work :)
>
> Indeed, but until I have fixed myself an ARM cross-compiler, I am not in
> a position to fix it.

See the wiki section on the Master Makefile -- it works pretty well.

> I can force my system to use V3 in the meantime, but as a matter of principle
> it ought to be put right. Though I do get the impression that NFS V4 is a
> fairly recent addition to Linux and there are other reports of its flakiness.
>
>>  Things might be different in OE by now
>
> OE == Outlook Express??? I don't allow any Microsoft products anywhere
> near my system :-) .

See http://www.openembedded.org/ ; this is the environment used to build
SlugOS (and Unslung).

>> P.S. -- a bit of digging on some of the other stuff you've reported
>> shows that NFS exporting of JFFS2 is simply not a supported thing to do.
>
> Sure, I can manage without that, but it needs documenting in the relevant Wiki,
> which I shall do.

Certainly you can put a note in the wiki on this.  However, the general
case for SlugOS is that it is a standard Linux distro, and in that
regard the other places on the web that note the limitations of NFS
apply to SlugOS just as well.

Since the NFS v4 is a difference at this point (although it is still
considered "not ready for prime time", it is commonly available for
other distros), *that* point should definitely appear in the wiki.

Mike (mwester)

[Charles Lindsey-2]

--- In nslu2-linux@..., "Mike (mwester)" <mwester@...> wrote:
>
> Charles Lindsey wrote:
> > Slugos and the NFSD that comes with it purport to support NFS Version 4
> > (incoming requests, that is) because it correctly accepts the NULL4
> > command when it arrives (which causes the client to continue with
> > Version 4 instead of dropping down to Version 3)....

> No, that does not indicate that it "purports to support NFS Version 4";
> that only indicates that I didn't go through the sources and remove that
> stuff!

Ah! So it's
  "Do What I Mean"
as opposed to
  "Do What I Say" :-)
>

Though I do find it rather odd that Slugos 4.8, which is based on
Linux 2.6.21.7 (Aug 2007) still recommends the use of nfs-utils version
1.0.6, which dates from Sept 2004 - at which time Linux 2.6.8 was all
we had. A lot of new stuff went into version 1.0.7 (much of it, such as
idmapd, related to NFS4). I am quite sure that the people who put
Linux 2.6.21.7 together were working on the assumtion that it would
be used in conjunction with at least version 1.0.7 of nfs-utils,
if not version 1.0.8.

> > Indeed, but until I have fixed myself an ARM cross-compiler, I am not in
> > a position to fix it.
>
> See the wiki section on the Master Makefile -- it works pretty well.

Sure, I have seen that, but it is rather low on my list of "things to do"
at the moment.

> Certainly you can put a note in the wiki on this.  However, the general
> case for SlugOS is that it is a standard Linux distro, and in that
> regard the other places on the web that note the limitations of NFS
> apply to SlugOS just as well.

Sure, but it seems that Linux 2.6.21.7 claims to be good enough to
welcome NFS4 clients now (letting it act as an an NFS client to
external servers is a different matter, or course). In which case, it noone
exposes it to such clients (and they are becoming increasingly common)
then we shall never learn what final bugs remain.
>
> Since the NFS v4 is a difference at this point (although it is still
> considered "not ready for prime time", it is commonly available for
> other distros), *that* point should definitely appear in the wiki.

Sure, I shall try to cover all those angles. At least that IS my next job.

[Charles Lindsey-2]

--- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:
>
> --- In nslu2-linux@..., "Mike (mwester)" <mwester@> wrote:

> > Certainly you can put a note in the wiki on this.  However, the general
> > case for SlugOS is that it is a standard Linux distro, and in that
> > regard the other places on the web that note the limitations of NFS
> > apply to SlugOS just as well.
>
> Sure, but it seems that Linux 2.6.21.7 claims to be good enough to
> welcome NFS4 clients now (letting it act as an an NFS client to
> external servers is a different matter, or course). In which case, it noone
> exposes it to such clients (and they are becoming increasingly common)
> then we shall never learn what final bugs remain.
> >
> > Since the NFS v4 is a difference at this point (although it is still
> > considered "not ready for prime time", it is commonly available for
> > other distros), *that* point should definitely appear in the wiki.
>
> Sure, I shall try to cover all those angles. At least that IS my next job.
>

OK, in the course or preparing my wiki, I observed another problem.

It seems (see exportfs man page) that export can function in either 'legacy'
or 'new_cache' mode; Slugos seems to default to 'legacy' mode but,
if SlugOS is trying to be a "standard Linux distro" as you said above, then
'new_cache' mode ought to become the norm. So I tried it, and it worked
fine, except for one thing: it did not keep /var/lib/nfs/rmtab up to date.
I have just had some correspondence with Neil Brown, who looks after
that corner of Linux, and he admits that:

   Well, rmtab is a bit of a 'crock'.  The NFS/MOUNT protocols do not
   make it possible to keep it up-to-date accurately.  Best advice is to
   ignore it, but that isn't always an option I guess.

And indeed, rmtab is only used in order to re-export stuff that was mounted
at the time of a crash, and the worst that can happen is that it exports stuff
that it needn't have (though still consistent with /etc/exports). He has no
immediate plan to fix the problem (and it may well get worse in NFSv4), but
he did say that nfs-utils version 1.1.1 had fixed some problems in that area,
and might be better (and the latest is actually 1.1.3). Moreover, a later
version of nfs-utils might well make clients trying to use NFSv4 behave
(and make SlugOS more of a "standard Linux distro"). I grant you that
letting SlugOS behave as an NFSv4 client would be an entirely different
matter.

[Charles Lindsey-2]

--- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:
>
> --- In nslu2-linux@..., "Charles Lindsey" <clerew5@> wrote:
> >
> > --- In nslu2-linux@..., "Mike (mwester)" <mwester@> wrote:
>

> > > Since the NFS v4 is a difference at this point (although it is still
> > > considered "not ready for prime time", it is commonly available for
> > > other distros), *that* point should definitely appear in the wiki.
> >
> > Sure, I shall try to cover all those angles. At least that IS my next job.
> >
> OK, in the course or preparing my wiki, I observed another problem.
>

OK, I have now added the result of my various researches to the existing
"InstallNFS" page <http://www.nslu2-linux.org/wiki/OpenSlug/InstallNFS>

Also did a bit of tidying up on that page to make a clear distinction between
the 'nfs' and 'nfsd' modules.

Share and Enjoy!