Probing the Kernel

[Charles Lindsey-2]

I still cannot make contact with my slug's nfsd, although interestingly it seems to fail in
different ways according to whether the client uses NFSv3 or NFSv4. But either way, I
cannot persuade it to give me a root filehandle.

I have been peering at the source code of linux nfsd and mountd, and have some idea
of where it might be getting lost, but they are complex beasties, and I really need to
know *exactly* which bits of nfsd it has gotten itself into.

So my next question is "What tools are available for tracing progress through
function calls within kernel space?"

Yes, I could install gdb if I wanted to trace things within mountd (which runs in user
space), but that is not where the problem appears to lie. Can gdb be used in kernel
space?

And also, I note that packages (though not kernel ones) usually come in three
flavours, e.g.:

nfs-utils - 1.0.6-r13 - userspace utilities for kernel nfs
nfs-utils-dbg - 1.0.6-r13 - userspace utilities for kernel nfs
nfs-utils-dev - 1.0.6-r13 - userspace utilities for kernel nfs

What is the difference between these, and in particular what is the '-dbg' good for
(e.g. was it compiled with '-g')?

[Charles Lindsey-2]

--- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:

> So my next question is "What tools are available for tracing progress through
> function calls within kernel space?"

Hmmmm! No replies, so it seems I have to answer my own question :-(

After much Googling, it turns out that there are three tools around for kernel
debugging - KDB, KGDB and LTT - but they all require (small?) modifications
to the kernel, which have not been made in Slugos and friends (reasonable
perhaps, given the need to squash the whole thing into the Slug's limited
flash).

It is quite possible to use 'strace' to see what system calls are being made,
particularly calls to the RPC modules and how these are implemented,
behind the scenes, by writing stuff to strange places within /proc/*, and these
gave me the clue to what was happening and how to observe it in action.

If you study the code of kernel modules, you will see many calls to 'printk',
which allegedly will cause interesting messages to appear in your syslog
(see /var/log/messages, or use dmesg if that is not configured). However,
if you look carefully you will find that in many modules 'printk' has actually
been redefined as a macro which only works if you have made the right
incantations beforehand. There is a man page nfsd(7), which can be seen at
http://linux.die.net/man/7/nfsd (though not all linux manpage sites seem to
have it), which explains it all.

Essentially, the kernel in general, and sunrpc in particular, is full of caches
and, for the sunrpc ones at least, their contents can be viewed, in readable
form, in /proc/net/rpc/*/content. It is particularly interesting to compare what
has actually been exported in /proc/netrpc/nfsd.export/content with what
'exportfs -v' claims to have exported.

On top of that, if you write suitable numbers into /proc/sys/sunrpc/*_debug, you
can awake some of those dormant 'printk's. I particular,
   echo 4 > /proc/sys/sunrpc/nfsd_debug
will cause some explanation of why certain things didn't really get exported
to appear in the syslog.

OK, I think I ought to write this up in a Wiki as soon as I get a Round Tuit.

> And also, I note that packages (though not kernel ones) usually come in three
> flavours, e.g.:
>
> nfs-utils - 1.0.6-r13 - userspace utilities for kernel nfs
> nfs-utils-dbg - 1.0.6-r13 - userspace utilities for kernel nfs
> nfs-utils-dev - 1.0.6-r13 - userspace utilities for kernel nfs
>
> What is the difference between these, and in particular what is the '-dbg' good for
> (e.g. was it compiled with '-g')?

But I would still like a response to that question.

[Rod Whitby-2]

Charles Lindsey wrote:

> --- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:
>
>> So my next question is "What tools are available for tracing progress through
>> function calls within kernel space?"
>
> Hmmmm! No replies, so it seems I have to answer my own question :-(
>
> After much Googling, it turns out that there are three tools around for kernel
> debugging - KDB, KGDB and LTT - but they all require (small?) modifications
> to the kernel, which have not been made in Slugos and friends (reasonable
> perhaps, given the need to squash the whole thing into the Slug's limited
> flash).

If someone comes up with an easy way for people to use one of those
debugging techniques on a slug, we will be glad to modify the SlugOS
kernel to support them.  We won't be able to do it on Unslung, but we
can certainly do it on SlugOS.

-- Rod

[Charles Lindsey-2]

--- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:

> If you study the code of kernel modules, you will see many calls to 'printk',
> which allegedly will cause interesting messages to appear in your syslog
> (see /var/log/messages, or use dmesg if that is not configured)......
>
> Essentially, the kernel in general, and sunrpc in particular, is full of caches
> and, for the sunrpc ones at least, their contents can be viewed, in readable
> form, in /proc/net/rpc/*/content. ......
>
> On top of that, if you write suitable numbers into /proc/sys/sunrpc/*_debug, you
> can awake some of those dormant 'printk's. ......
>
> OK, I think I ought to write this up in a Wiki as soon as I get a Round Tuit.

And I have now documented my findings in a new wiki page which you will find at
http://www.nslu2-linux.org/wiki/HowTo/ProbingTheKernel

I am sure there is much I have left out, and so I hope others will add to it as
and when they identify other techniques that might be useful.

[Rod Whitby-2]

Charles Lindsey wrote:
> --- In nslu2-linux@..., "Charles Lindsey" <clerew5@...> wrote:
>> OK, I think I ought to write this up in a Wiki as soon as I get a Round Tuit.
>
> And I have now documented my findings in a new wiki page which you will find at
> http://www.nslu2-linux.org/wiki/HowTo/ProbingTheKernel
>
> I am sure there is much I have left out, and so I hope others will add to it as
> and when they identify other techniques that might be useful.

Thanks Charles - that's a great start.

-- Rod, 'TPTB' ;-)