Kernel Crash using FUSE

[Bryan Ischo-5]

Hi, I'd like to report a kernel crash that I experienced using a FUSE
filesystem.

Unfortunately I have very little data to report, as I was running a test
on a server in a data center that I have no physical access to and could
not see the console output of the system to view any kernel panic
diagnostics.  All I have is syslog messages and they are not very
detailed.

Basically, I was testing out a FUSE filesystem called 's3backer', at:

http://code.google.com/p/s3backer

This FUSE filesystem exposes a single file that it stores block-by-block
in an S3 bucket.  You can use a loopback mount to then mount this file as
a filesystem, thus obtaining an S3-backed Unix filesystem.

I was testing it by using cp -a of a largish directory (4,000+ files in
the directory tree, total of about 300 MB of data), and at the same time,
running find | xargs wc -l to watch how many files had been copied.  After
a few minutes of activity, my entire system froze up.  Symptoms were that
I could not do anything that seemed to require a process to hit the
kernel's VFS layer; I found that I couldn't create any processes, but that
the kernel was still managing its TCP/IP stack (I could ping it) and I
could even make connections to existing services (the TCP/IP connection
would complete) but that I couldn't get them to do anything.

After almost 2 hours I managed to get the data center staff to reboot the
system for me.  This is all I have from syslog to describe the event:

Jul 16 23:34:32 dima s3backer: s3backer process 25251 for /tmp/testmnt
started
Jul 16 23:34:32 dima s3backer: created s3backer using
http://s3.amazonaws.com/www.ischo.com.testmnt
Jul 16 23:36:42 dima kernel: loop: loaded (max 8 devices)
Jul 16 23:43:01 dima s3backer: rec'd 500 response: PUT
http://s3.amazonaws.com/www.ischo.com.testmnt/0009e1df
Jul 16 23:43:01 dima s3backer: retrying query (attempt #2): PUT
http://s3.amazonaws.com/www.ischo.com.testmnt/0009e1df
Jul 16 23:45:25 dima s3backer: rec'd 500 response: PUT
http://s3.amazonaws.com/www.ischo.com.testmnt/0009e5e5
Jul 16 23:45:25 dima s3backer: retrying query (attempt #2): PUT
http://s3.amazonaws.com/www.ischo.com.testmnt/0009e5e5
Jul 17 01:00:52 dima syslogd 1.4.1: restart.

I was really surprised to find that a FUSE filesystem could crash the
kernel.  Doesn't FUSE, since it's dealing with a userspace program, have
stringent protections in place so that if the FUSE userspace program
misbehaves, it doesn't crash the kernel?  Or perhaps this is just a bug in
FUSE itself?

Details of my system:

Linux dima.ischo.com 2.6.20-1.2320.fc5
fusermount version: 2.6.5

(yes, it's a fairly old Fedora Core 5 system, were there maybe known bugs
in FUSE that have been fixed since then?)

Thanks!
Bryan

------------------------------------------------------------------------
Bryan Ischo                bryan@...            2001 Mazda 626 GLX
Hamilton, New Zealand      http://www.ischo.com     RedHat Fedora Core 5
------------------------------------------------------------------------



-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
fuse-devel mailing list
fuse-devel@...
https://lists.sourceforge.net/lists/listinfo/fuse-devel

[Miklos Szeredi]

On Thu, 17 Jul 2008, Bryan Ischo wrote:
> Hi, I'd like to report a kernel crash that I experienced using a FUSE
> filesystem.

Thanks for the report.

> I was testing it by using cp -a of a largish directory (4,000+ files in
> the directory tree, total of about 300 MB of data), and at the same time,
> running find | xargs wc -l to watch how many files had been copied.  After
> a few minutes of activity, my entire system froze up.  Symptoms were that
> I could not do anything that seemed to require a process to hit the
> kernel's VFS layer; I found that I couldn't create any processes, but that
> the kernel was still managing its TCP/IP stack (I could ping it) and I
> could even make connections to existing services (the TCP/IP connection
> would complete) but that I couldn't get them to do anything.

That sounds like an Oops that happened with some important lock held.

Unfortunately it's very hard to say anything about it without seeing
the actual Oops report.

> I was really surprised to find that a FUSE filesystem could crash the
> kernel.  Doesn't FUSE, since it's dealing with a userspace program, have
> stringent protections in place so that if the FUSE userspace program
> misbehaves, it doesn't crash the kernel?

Yes.  Whatever the userspace program does, it should not be able to
make the kernel misbehave or crash.

>  Or perhaps this is just a bug in
> FUSE itself?
>
> Details of my system:
>
> Linux dima.ischo.com 2.6.20-1.2320.fc5
> fusermount version: 2.6.5
>
> (yes, it's a fairly old Fedora Core 5 system, were there maybe known bugs
> in FUSE that have been fixed since then?)

Quite possibly.

I don't really know what to recommend, it's next to impossible to
debug such a problem without physical access to the machine, or at
least a remote serial console.

Thanks,
Miklos

-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
fuse-devel mailing list
fuse-devel@...
https://lists.sourceforge.net/lists/listinfo/fuse-devel

[Bryan Ischo-5]

> On Thu, 17 Jul 2008, Bryan Ischo wrote:
>> Hi, I'd like to report a kernel crash that I experienced using a FUSE
>> filesystem.
>
> Thanks for the report.

You are welcome, and thank you for your reply!

I have signed up for a Xen instance on Linode.com, installing Debian 4.0
unstable (to get the newest FUSE stuff), to further test s3backer and FUSE
in a more modern setting.  Thus far I have not seen any similar problems,
so I would chalk it up to an old version of FUSE (or maybe even the
kernel) until I see any reproduction of the crash.  Oh, and on Linode I
have access to the console so I ought to be able to capture the OOPS
logging should such an event occur again.

I have been finding some interesting performance issues and I would like
to ask some questions about the interaction of FUSE, loopback mounts, the
kernel's block cache, and threading, but I will save these for a different
thread so as not to confuse this particular thread.

Thank you again, and best wishes,
Bryan




-------------------------------------------------------------------------
This SF.Net email is sponsored by the Moblin Your Move Developer's challenge
Build the coolest Linux based applications with Moblin SDK & win great prizes
Grand prize is a trip for two to an Open Source event anywhere in the world
http://moblin-contest.org/redirect.php?banner_id=100&url=/
_______________________________________________
fuse-devel mailing list
fuse-devel@...
https://lists.sourceforge.net/lists/listinfo/fuse-devel