Filesystem choice for fileserver?

[KRM]

I have used XFS in the enterprise (and at home) on several servers since 2001.  I did have one issue (no data loss), but it is believed that this arose from a RAID hardware failure and not the file system itself.

I have recently done some work on a RHEL box using ext3 and have had some significant issues with large files taking forever to delete.  This has come up with one single 80Gb file and in situations when deleting large directory trees.  Unfortunately, this is all anecdotal as i have not done much to try and determine if the source of the slowdown.

/KRM




On Thu, Apr 17, 2008 at 12:26:50PM +0100, Peter Corlett wrote:

> On Thu, Apr 17, 2008 at 12:09:14PM +0100, Michael-John Turner wrote:
> [...]
> > Naturally, my first choice would be ZFS, but it's only stable on Solaris
> > (and maybe FreeBSD), neither of which are supported as VMware host
> > platforms.
>
> But there *is* Xen dom0 support.
>
> > My Linux filesystem of choice used to be reiserfs version 3, but that's
> > pretty much in maintenance only mode these days.
>
> It was never particularly reliable even when it was being maintained. Never
> trust a filesystem that doesn't have good recovery tools.
>
> More than once I have found myself trying to fix a problem without the
> recovery tools Hans assured us were not necessary. Usually the only fix
> available was mke2fs and restore from backup.
>
> > I'm very keen on XFS, what with its solid SGI heritage, but I've read a
> > few horror stories of people who've lost data using it.
>
> I played with that too. What I discovered was that while it was good at
> keeping metadata consistent, this didn't apply to the data. Finding
> /etc/fstab full of NULs after an unplanned powercycle can ruin one's day.
>
> > There's also JFS, which I have no experience with and doesn't seem
> > particularly popular,
>
> I haven't used this and can't comment.
>
> > and ext3, which I'm not a fan of because of it's semi-async nature.
>
> This is my preferred filesystem on Linux because it is the default and thus
> well-tested. It's a superset of ext2, which is in turn somewhat based on
> Minix and UFS filesystems, and so the design is sound and simple. I *like*
> my filesystems to be simple rather than clever.
>
> ext3 has a number of options that you can use to tune between performance
> and reliability. I routinely use noatime and dir_index now and that really
> does make it rather faster than ext2, and no less reliable.
>
> > Anyone got any thoughts on this? What's everyone using for their
> > fileserver needs these days?
>
> I just have a scummy old Debian box with a couple of internal disks and a
> couple more USB disks crammed in the back. I use software RAID1 for extra
> reliability.
>
> Since it is now full, I am going to have to finally build its replacement,
> the parts of which have been lying around for a few months now waiting for
> tuits.
> _______________________________________________
> rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[KRM]

Actually... that happens about twice a week with this particular setup as it is a temp location for some large backups.  Normally it happens in the middle of the night so no one really cares, but on two occasions recently we have had to come in and do some work on the weekends and have had to wait for about 45 minutes, doing nothing but twiddling my thumbs until the files are deleted.  The fact that this was Saturday golden time (on both occasions) make it all the more annoying.

Plus, as i recall it took forever and a day to initialize the filesystem (500 Gigs) to begin with.

/KRM

> Yes, unlink *is* expensive on ext3. That's because the file's inode and
> indirect blocks all have to be walked to determine which blocks to mark in
> the free space map. I suspect that unlink also locks the filesystem until
> the operation is complete.
>
> Still, it's not as if you're regularly deleting 80GB files...
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Michael-John Turner-2]

Hi all,

I'm looking at building a new home fileserver over the coming weeks and one
of the things I'm still a bit unsure about is choice of filesystem. I'm
looking at using 3x750GB drives for data storage, in a RAID5 configuration
(ie about 1.5TB usable). The server itself will probably be a Core2Quad
with 4/8GiB RAM.

One slight fly in the ointment is that I'd like to use the server for some
virtualisation as well - my software choice there is VMware Server (I want
to share VMs with VMware Fusion on my MBP), which pretty much necessitates
the use of Linux.

Naturally, my first choice would be ZFS, but it's only stable on Solaris
(and maybe FreeBSD), neither of which are supported as VMware host
platforms.

My Linux filesystem of choice used to be reiserfs version 3, but that's
pretty much in maintenance only mode these days. I'm very keen on XFS, what
with its solid SGI heritage, but I've read a few horror stories of people
who've lost data using it. There's also JFS, which I have no experience
with and doesn't seem particularly popular, and ext3, which I'm not a fan
of because of it's semi-async nature.

Anyone got any thoughts on this? What's everyone using for their fileserver
needs these days?

-mj
--
Michael-John Turner
mj@...     | http://mjturner.net/
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Peter Corlett]

On Thu, Apr 17, 2008 at 12:09:14PM +0100, Michael-John Turner wrote:
[...]
> Naturally, my first choice would be ZFS, but it's only stable on Solaris
> (and maybe FreeBSD), neither of which are supported as VMware host
> platforms.

But there *is* Xen dom0 support.

> My Linux filesystem of choice used to be reiserfs version 3, but that's
> pretty much in maintenance only mode these days.

It was never particularly reliable even when it was being maintained. Never
trust a filesystem that doesn't have good recovery tools.

More than once I have found myself trying to fix a problem without the
recovery tools Hans assured us were not necessary. Usually the only fix
available was mke2fs and restore from backup.

> I'm very keen on XFS, what with its solid SGI heritage, but I've read a
> few horror stories of people who've lost data using it.

I played with that too. What I discovered was that while it was good at
keeping metadata consistent, this didn't apply to the data. Finding
/etc/fstab full of NULs after an unplanned powercycle can ruin one's day.

> There's also JFS, which I have no experience with and doesn't seem
> particularly popular,

I haven't used this and can't comment.

> and ext3, which I'm not a fan of because of it's semi-async nature.

This is my preferred filesystem on Linux because it is the default and thus
well-tested. It's a superset of ext2, which is in turn somewhat based on
Minix and UFS filesystems, and so the design is sound and simple. I *like*
my filesystems to be simple rather than clever.

ext3 has a number of options that you can use to tune between performance
and reliability. I routinely use noatime and dir_index now and that really
does make it rather faster than ext2, and no less reliable.

> Anyone got any thoughts on this? What's everyone using for their
> fileserver needs these days?

I just have a scummy old Debian box with a couple of internal disks and a
couple more USB disks crammed in the back. I use software RAID1 for extra
reliability.

Since it is now full, I am going to have to finally build its replacement,
the parts of which have been lying around for a few months now waiting for
tuits.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Timothy Baldridge]

> I'm very keen on XFS, what with its solid SGI heritage, but I've read a
> few horror stories of people who've lost data using it.

And I've heard horror stories about NTFS, ext3, reiserfs, but I've
never had issues with those either. I've used XFS a fair amount and I
love it. Rock solid and blazing fast. However, allot of GRUBs don't
support it, so make your OS partition ext3 and XFS on the data drives.

XFS really excells with large files. Here you'll find its performance
to be way faster than ext3 and maybe even faster than reiserfs.

Of course all this comes from a self proclaimed SGI addict, so take it
as you will.

Timothy

--
Two wrights don't make a rong, they make an airplane. Or bicycles.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Joshua Boyd]

On Apr 17, 2008, at 7:38 AM, Timothy Baldridge wrote:

>> I'm very keen on XFS, what with its solid SGI heritage, but I've  
>> read a
>> few horror stories of people who've lost data using it.
>
> And I've heard horror stories about NTFS, ext3, reiserfs, but I've
> never had issues with those either. I've used XFS a fair amount and I
> love it. Rock solid and blazing fast. However, allot of GRUBs don't
> support it, so make your OS partition ext3 and XFS on the data drives.

It seems that swapping one grub for another should be easy.   Just  
drop the new grub files in the proper place in /boot, then run grub-
install with appropriate arguments.

This assumes that when you say a lot of grubs you don't really mean  
all grubs.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Phil Stracchino-3]

Michael-John Turner wrote:
> My Linux filesystem of choice used to be reiserfs version 3, but that's
> pretty much in maintenance only mode these days. I'm very keen on XFS, what
> with its solid SGI heritage, but I've read a few horror stories of people
> who've lost data using it. There's also JFS, which I have no experience
> with and doesn't seem particularly popular, and ext3, which I'm not a fan
> of because of it's semi-async nature.
>
> Anyone got any thoughts on this? What's everyone using for their fileserver
> needs these days?

The general consensus seems to be that jfs and xfs are the best of the
bunch.  I used jfs on Cymru's laptop, and it's working great there.
I've never been a fan of reiserfs.


--
   Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
   alaric@...   alaric@...   phil@...
          Renaissance Man, Unix ronin, Perl hacker, Free Stater
                  It's not the years, it's the mileage.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Patrick Giagnocavo 717-201-3366]

Michael-John Turner wrote:
> Hi all,
>
> I'm looking at building a new home fileserver over the coming weeks and one
> of the things I'm still a bit unsure about is choice of filesystem. I'm
> looking at using 3x750GB drives for data storage, in a RAID5 configuration
> (ie about 1.5TB usable). The server itself will probably be a Core2Quad
> with 4/8GiB RAM.
>

Will you be using ECC RAM on the server?

> One slight fly in the ointment is that I'd like to use the server for some
> virtualisation as well - my software choice there is VMware Server (I want
> to share VMs with VMware Fusion on my MBP), which pretty much necessitates
> the use of Linux.

You can look at Solaris with ZFS plus VirtualBox (www.virtualbox.org)
which I have used to virtualize Debian 4.0 on top of Solaris .  I have
not tried different OSes because I ran into an issue with networking.

The issue with networking is that Solaris uses VNICs (virtual NICs) for
virtualizing network interfaces, and apparently VBox does not support
that yet for the case where you want to have your VM available on the
network; it insteads serves DHCP to the VM and does NAT translation
(useful if you wanted to run XP as a client OS, less useful if you want
to have a VM that you can SSH into).

Instead, you have to install TUN and TAP drivers, create the tap0
device, then tell VBox to use it.  If you don't mind the extra work,
then Solaris plus ZFS plus VBox is a possible solution.

I have had good success in the past with XFS on linux as well, but that
was a few years ago.


--Patrick
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Meelis Roos]

> XFS really excells with large files. Here you'll find its performance
> to be way faster than ext3 and maybe even faster than reiserfs.

Yes, http://oss.oracle.com/~mason/seekwatcher/ext3_vs_btrfs_vs_xfs.png 
is a recent visual indication that comes to mind.

I mostly prefer ext3 because it has never really failed me (only once
when I forced a fsck on a mounted FS - got about 2% of files back). XFS
is my filesystem of choice when large files and/or parallel streaming
reads-writes are used. One gotcha is default yournal size (or metadata
buffer size ir RAM) - default used to be smallish and this resulted in
relatively slow metadata operations (like deleting a huge source tree).
There's a mount option to increase the RAM buffer size, or maybe
creatinga bugger journal during mkfs also helps.

--
Meelis Roos (mroos@...)
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Kevin Foote]

Why not something like

freenas --- > http://www.freenas.org
openfiler --- > http://www.openfiler.com

I personally like freenas for the footprint.

--
:wq!
kevin.foote

On Thu, Apr 17, 2008 at 7:09 AM, Michael-John Turner <mj@...> wrote:

> Hi all,
>
>  I'm looking at building a new home fileserver over the coming weeks and one
>  of the things I'm still a bit unsure about is choice of filesystem. I'm
>  looking at using 3x750GB drives for data storage, in a RAID5 configuration
>  (ie about 1.5TB usable). The server itself will probably be a Core2Quad
>  with 4/8GiB RAM.
>
>  One slight fly in the ointment is that I'd like to use the server for some
>  virtualisation as well - my software choice there is VMware Server (I want
>  to share VMs with VMware Fusion on my MBP), which pretty much necessitates
>  the use of Linux.
>
>  Naturally, my first choice would be ZFS, but it's only stable on Solaris
>  (and maybe FreeBSD), neither of which are supported as VMware host
>  platforms.
>
>  My Linux filesystem of choice used to be reiserfs version 3, but that's
>  pretty much in maintenance only mode these days. I'm very keen on XFS, what
>  with its solid SGI heritage, but I've read a few horror stories of people
>  who've lost data using it. There's also JFS, which I have no experience
>  with and doesn't seem particularly popular, and ext3, which I'm not a fan
>  of because of it's semi-async nature.
>
>  Anyone got any thoughts on this? What's everyone using for their fileserver
>  needs these days?
>
>  -mj
>  --
>  Michael-John Turner
>  mj@...     | http://mjturner.net/
>  _______________________________________________
>  rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Peter Corlett]

On Thu, Apr 17, 2008 at 10:46:01AM +0900, kevin r. marhsall wrote:
[...]
> I have recently done some work on a RHEL box using ext3 and have had some
> significant issues with large files taking forever to delete. This has
> come up with one single 80Gb file and in situations when deleting large
> directory trees. Unfortunately, this is all anecdotal as i have not done
> much to try and determine if the source of the slowdown.

Yes, unlink *is* expensive on ext3. That's because the file's inode and
indirect blocks all have to be walked to determine which blocks to mark in
the free space map. I suspect that unlink also locks the filesystem until
the operation is complete.

Still, it's not as if you're regularly deleting 80GB files...
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Sridhar Ayengar]

Phil Stracchino wrote:

> Michael-John Turner wrote:
>> My Linux filesystem of choice used to be reiserfs version 3, but that's
>> pretty much in maintenance only mode these days. I'm very keen on XFS,
>> what
>> with its solid SGI heritage, but I've read a few horror stories of people
>> who've lost data using it. There's also JFS, which I have no experience
>> with and doesn't seem particularly popular, and ext3, which I'm not a fan
>> of because of it's semi-async nature.
>>
>> Anyone got any thoughts on this? What's everyone using for their
>> fileserver
>> needs these days?
>
> The general consensus seems to be that jfs and xfs are the best of the
> bunch.  I used jfs on Cymru's laptop, and it's working great there. I've
> never been a fan of reiserfs.

I don't use JFS on Linux, but it's always been great on AIX and OS/2.

Peace...  Sridhar
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Sridhar Ayengar]

Kevin Foote wrote:
> Why not something like
>
> freenas --- > http://www.freenas.org
> openfiler --- > http://www.openfiler.com
>
> I personally like freenas for the footprint.

I don't think either of these fit his requirements, do they?

Peace...  Sridhar
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Michael-John Turner-2]

On Thu, Apr 17, 2008 at 10:47:33AM -0400, Sridhar Ayengar wrote:
> Kevin Foote wrote:
>> freenas --- > http://www.freenas.org
>> openfiler --- > http://www.openfiler.com
>
> I don't think either of these fit his requirements, do they?

Correct. I'm not looking for a lightweight NAS-type device, but rather a
server with a robust, fairly high-performance filesystem.

-mj
--
Michael-John Turner
mj@...     | http://mjturner.net/
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Michael-John Turner-2]

On Thu, Apr 17, 2008 at 08:41:07AM -0400, Patrick Giagnocavo wrote:
> Will you be using ECC RAM on the server?

Unfortunately, probably not - it's going to be using a consumer-grade
motherboard (model still to be decided), which I doubt will support ECC.

-mj
--
Michael-John Turner
mj@...     | http://mjturner.net/
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Sridhar Ayengar]

Michael-John Turner wrote:
>>> freenas --- > http://www.freenas.org
>>> openfiler --- > http://www.openfiler.com
>> I don't think either of these fit his requirements, do they?
>
> Correct. I'm not looking for a lightweight NAS-type device, but rather a
> server with a robust, fairly high-performance filesystem.

In any case, I'm just about done building out my own new file server
(4TB after RAID overhead, expandable to 12TB after overhead in the
chassis), and I've decided to go with FreeBSD + ZFS + NFS + Samba.  I'm
also using hardware RAID.  It's a Highpoint RocketRAID, for which I've
seen mixed reviews, but I'll see how it goes, and if I need to later,
I'm going to go with 3ware, but that would be a much more expen$$$ive
proposition.  It's a dual-processor dual-core Opteron with 4GB ECC RAM.
  Less than $2000 for the whole thing, including the drives.

I was thinking about using some sort of iSCSI target code, but I decided
that just wasn't for me.

Peace...  Sridhar
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Mike Shields-2]

On Thu, Apr 17, 2008 at 9:59 AM, Peter Corlett <abuse@...> wrote:

> On Thu, Apr 17, 2008 at 10:46:01AM +0900, kevin r. marhsall wrote:
> [...]
> > I have recently done some work on a RHEL box using ext3 and have had
> some
> > significant issues with large files taking forever to delete. This has
> > come up with one single 80Gb file and in situations when deleting large
> > directory trees. Unfortunately, this is all anecdotal as i have not done
> > much to try and determine if the source of the slowdown.
>
> Yes, unlink *is* expensive on ext3. That's because the file's inode and
> indirect blocks all have to be walked to determine which blocks to mark in
> the free space map. I suspect that unlink also locks the filesystem until
> the operation is complete.
>
> Still, it's not as if you're regularly deleting 80GB files...

I have a machine serving as a PVR and regularly delete files in the 2 to 4
gb range. I don't have any detailed metrics but it would take 5 - 10 seconds
to delete each file when I used ext3 as my filesystem. I switched over to
xfs after finding that this was a well-known sore spot in ext3, and have
never looked back -- although I still use ext3 for my root filesystem.

I'll grant though that my choice of xfs also had something to do with my
mild SGI fanaticism...

Speaking of rescue and SGI's, I'm on the hunt for some RM5 modules to beef
up my the RE2 in my deskside Onyx...
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Meelis Roos]

> I've decided to go with FreeBSD + ZFS + NFS + Samba.  I'm also using hardware
> RAID.  It's a Highpoint RocketRAID, for which I've seen mixed reviews, but

Out of curiosity - what RocketRAID model, and what driver are you using
in FreeBSD for drive it? I have met ones with Marvell chips - these seem
to be just fakeraid - and ones with HPT IOP message passing interface.
Are you using the latter?

--
Meelis Roos (mroos@...)
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Patrick Finnegan]

On Thursday 17 April 2008, Timothy Baldridge wrote:
> > I'm very keen on XFS, what with its solid SGI heritage, but I've
> > read a few horror stories of people who've lost data using it.
>
> And I've heard horror stories about NTFS, ext3, reiserfs, but I've
> never had issues with those either. I've used XFS a fair amount and I
> love it. Rock solid and blazing fast. However, allot of GRUBs don't
> support it, so make your OS partition ext3 and XFS on the data
> drives.

At work, we've locked up machines, and lost data due on fileservers that
used XFS under heavy load.

The main problem I had when I was using JFS (which was back before ext3
was in a "stable" kernel line), was that JFS removed any files that
were open when the machine didn't shut down cleanly.  After having to
rebuild my .gaimrc a few times, I gave up and switched to ext3 on an
unstable series kernel (2.3.something I think).

> XFS really excells with large files. Here you'll find its performance
> to be way faster than ext3 and maybe even faster than reiserfs.

It's not bad, but I've seen data loss with it.

On the other hand, we've never had data loss in the same environment at
work with an ext3 filesystem (at least where a hardware failure wasn't
to blame).  I don't think I've ever had any real problems with ext3,
except where I f'd up the filesysme by hand, or the disk/raid
controller failed.  FWIW, we've noticed at work that the raid
controller on Dell 2650s and Dell 1850s tends to lock up under high
utilization.  

This is also partly why I strongly prefer software raid to hardware
raid.  I've seen a number of times where hardware raid has caused us
data loss.  The only one time I've lost data on a software raid, was
because I was trying to piece it back together, and reassembled the
RAID-5 in the wrong order.  It's really easy to mash the chunks back
together with software raid if it becomes unhappy, but there's rarely
the same user accessible tools you'd need to do the same with hardware
raid.

Pat
--
Purdue University Research Computing ---  http://www.rcac.purdue.edu/
The Computer Refuge                  ---  http://computer-refuge.org
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Robert Darlington]

I have several RM4's for you if you need them.  No RM5's though.

On Thu, Apr 17, 2008 at 9:21 AM, Mike Shields <shieldsm@...> wrote:

> On Thu, Apr 17, 2008 at 9:59 AM, Peter Corlett <abuse@...> wrote:
>
>  > On Thu, Apr 17, 2008 at 10:46:01AM +0900, kevin r. marhsall wrote:
>  > [...]
>  > > I have recently done some work on a RHEL box using ext3 and have had
>  > some
>  > > significant issues with large files taking forever to delete. This has
>  > > come up with one single 80Gb file and in situations when deleting large
>  > > directory trees. Unfortunately, this is all anecdotal as i have not done
>  > > much to try and determine if the source of the slowdown.
>  >
>  > Yes, unlink *is* expensive on ext3. That's because the file's inode and
>  > indirect blocks all have to be walked to determine which blocks to mark in
>  > the free space map. I suspect that unlink also locks the filesystem until
>  > the operation is complete.
>  >
>  > Still, it's not as if you're regularly deleting 80GB files...
>
>
>  I have a machine serving as a PVR and regularly delete files in the 2 to 4
>  gb range. I don't have any detailed metrics but it would take 5 - 10 seconds
>  to delete each file when I used ext3 as my filesystem. I switched over to
>  xfs after finding that this was a well-known sore spot in ext3, and have
>  never looked back -- although I still use ext3 for my root filesystem.
>
>  I'll grant though that my choice of xfs also had something to do with my
>  mild SGI fanaticism...
>
>  Speaking of rescue and SGI's, I'm on the hunt for some RM5 modules to beef
>  up my the RE2 in my deskside Onyx...
>
>
> _______________________________________________
>  rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue