Filesystem choice for fileserver?

[Sridhar Ayengar]

Meelis Roos wrote:
>> I've decided to go with FreeBSD + ZFS + NFS + Samba.  I'm also using hardware
>> RAID.  It's a Highpoint RocketRAID, for which I've seen mixed reviews, but
>
> Out of curiosity - what RocketRAID model, and what driver are you using
> in FreeBSD for drive it? I have met ones with Marvell chips - these seem
> to be just fakeraid - and ones with HPT IOP message passing interface.
> Are you using the latter?

I'm using a RocketRAID 2220, which is an eight-channel 133MHz PCI-X
card.  I'm not exactly sure what kind it is, but I know the 2000-series
isn't their top-of-the-line.  That's the 3000-series.  However, there's
also a 1000-series which is quite a bit cheaper, which I would fully
expect to be a regular Marvell-based card.

If I were going to spend the extra money for a 3000-series card, I would
have just spent even a few bucks more for the 3ware.

Peace...  Sridhar
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Meelis Roos]

> I'm using a RocketRAID 2220, which is an eight-channel 133MHz PCI-X card.  I'm
> not exactly sure what kind it is, but I know the 2000-series isn't their
> top-of-the-line.  That's the 3000-series.  However, there's also a 1000-series
> which is quite a bit cheaper, which I would fully expect to be a regular
> Marvell-based card.

Hmm. Some googling tells it's _probably_ a Marvell MV88SX6081 8-port
SATA II PCI-X Controller and that one is definitely software RAID - but
done by vendor software RAID stack insted of FreeBSD's.

--
Meelis Roos (mroos@...)
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Sridhar Ayengar]

Meelis Roos wrote:
>> I'm using a RocketRAID 2220, which is an eight-channel 133MHz PCI-X card.  I'm
>> not exactly sure what kind it is, but I know the 2000-series isn't their
>> top-of-the-line.  That's the 3000-series.  However, there's also a 1000-series
>> which is quite a bit cheaper, which I would fully expect to be a regular
>> Marvell-based card.
>
> Hmm. Some googling tells it's _probably_ a Marvell MV88SX6081 8-port
> SATA II PCI-X Controller and that one is definitely software RAID - but
> done by vendor software RAID stack insted of FreeBSD's.

Yeah, I verified that, so I changed my order to an LSI Logic MegaRAID,
which is $180 more, but is definitely hardware RAID.

Peace...  Sridhar
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Jonathan Guthrie-3]

On Thu, Apr 17, 2008 at 12:26:50PM +0100, Peter Corlett wrote:
> On Thu, Apr 17, 2008 at 12:09:14PM +0100, Michael-John Turner wrote:
> > I'm very keen on XFS, what with its solid SGI heritage, but I've read a
> > few horror stories of people who've lost data using it.
 
> I played with that too. What I discovered was that while it was good at
> keeping metadata consistent, this didn't apply to the data. Finding
> /etc/fstab full of NULs after an unplanned powercycle can ruin one's day.

I have lost data in XFS files systems, but that hasn't happened to me
in a decade, and I'm not particularly careful about powercycling.  I
use XFS exclusively for those filesystems that are going to be shared
out remotely because I can extend XFS filesystems without umounting
the disks.  So, my file services all come from XFS build on LVM lv's.
Run out of space on /home?   Well, then, I do an "lvextend" and a
xfs_growfs and the problem has gone away.
--
Jonathan Guthrie (jguthrie@...)
Sto pro veritate
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Lionel Peterson]

>From: Kevin Foote <kevin.foote@...>
>Date: 2008/04/17 Thu AM 08:58:39 CDT
>To: The Rescue List <rescue@...>, Michael-John Turner <mj@...>
>Subject: Re: [rescue] Filesystem choice for fileserver?

>Why not something like
>
>freenas --- > http://www.freenas.org
>openfiler --- > http://www.openfiler.com

Just starting to play with FreeNAS at work, just to get a feel for what we want in a filer for a bunch of Windows Server 2003 boxes...

>I personally like freenas for the footprint.

I like it for the iSCSI support - and the whole "boot off a 128 Meg USB thumb drive" thing (aka footprint)...

Lionel
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Jonathan Groll-2]

On Thu, Apr 17, 2008 at 11:34:07AM -0400, Patrick Finnegan wrote:
> > XFS really excells with large files. Here you'll find its performance
> > to be way faster than ext3 and maybe even faster than reiserfs.
>
> It's not bad, but I've seen data loss with it.
>
> On the other hand, we've never had data loss in the same environment at
> work with an ext3 filesystem (at least where a hardware failure wasn't
> to blame).  I don't think I've ever had any real problems with ext3,

Another thing to consider: it seems to me that different filesystems
report significantly different amounts of available space.

For instance, in the test that I've done below, for a 100M disk image
file, I get 87M available space for ext3 and 68M for reiserfs(3). I
was under the impression however that the situation was reversed for
larger disks.

Cheers,
Jonathan

Test:
jonathan@speedy:/exports/test$ dd if=/dev/zero of=test bs=1M count=100
100+0 records in
100+0 records out
104857600 bytes (105 MB) copied, 1.35582 seconds, 77.3 MB/s
jonathan@speedy:/exports/test$ yes | mkfs.ext3 test
mke2fs 1.40.2 (12-Jul-2007)
test is not a block special device.
Proceed anyway? (y,n) Filesystem label=
OS type: Linux
Block size=1024 (log=0)
Fragment size=1024 (log=0)
25688 inodes, 102400 blocks
5120 blocks (5.00%) reserved for the super user
First data block=1
Maximum filesystem blocks=67371008
13 block groups
8192 blocks per group, 8192 fragments per group
1976 inodes per group
Superblock backups stored on blocks:
        8193, 24577, 40961, 57345, 73729

Writing inode tables: done
Creating journal (4096 blocks): done
Writing superblocks and filesystem accounting information: done

This filesystem will be automatically checked every 36 mounts or
180 days, whichever comes first.  Use tune2fs -c or -i to override.
jonathan@speedy:/exports/test$ sudo mount -o loop test /mnt/
[sudo] password for jonathan:
jonathan@speedy:/exports/test$ df -h
Filesystem            Size  Used Avail Use% Mounted on
<snipped>
/exports/test/test     97M  5.6M   87M   7% /mnt
jonathan@speedy:/exports/test$ sudo umount /exports/test/test
jonathan@speedy:/exports/test$ mkreiserfs -f test >/dev/null
mkreiserfs 3.6.19 (2003 www.namesys.com)

test is not a block special device
Continue (y/n):y
jonathan@speedy:/exports/test$ sudo mount -o loop test /mnt/
jonathan@speedy:/exports/test$ df -h
Filesystem            Size  Used Avail Use% Mounted on
<snipped>
/exports/test/test    100M   33M   68M  33% /mnt
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Michael-John Turner-2]

On Thu, Apr 17, 2008 at 11:34:07AM -0400, Patrick Finnegan wrote:
> This is also partly why I strongly prefer software raid to hardware
> raid.  I've seen a number of times where hardware raid has caused us
> data loss.  The only one time I've lost data on a software raid, was
> because I was trying to piece it back together, and reassembled the
> RAID-5 in the wrong order.  It's really easy to mash the chunks back
> together with software raid if it becomes unhappy, but there's rarely
> the same user accessible tools you'd need to do the same with hardware
> raid.

Agree with you on this one. In addition, the ability to be able to just pop
the disks into a replacement machine running the same OS, as you can with
software RAID, is a huge advantage. With hardware RAID, you'd need to find
the same model of controller, etc.

-mj
--
Michael-John Turner
mj@...     | http://mjturner.net/
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Michael-John Turner-2]

On Thu, Apr 17, 2008 at 11:13:44AM -0400, Sridhar Ayengar wrote:
> In any case, I'm just about done building out my own new file server (4TB
> after RAID overhead, expandable to 12TB after overhead in the chassis), and
> I've decided to go with FreeBSD + ZFS + NFS + Samba.  I'm also using
> hardware RAID.  It's a Highpoint RocketRAID, for which I've seen mixed
> reviews, but I'll see how it goes, and if I need to later, I'm going to go
> with 3ware, but that would be a much more expen$$$ive proposition.  It's a
> dual-processor dual-core Opteron with 4GB ECC RAM.  Less than $2000 for the
> whole thing, including the drives.

Nice. I'd be very curious to hear of your experiences with ZFS under
FreeBSD - I'd still be a little wary of trusting my 'production' data to
it, seeing as 7.0 is the first release to support it.

What disks will you be using, BTW? I'm edging towards Seagates (good luck
with them in the past and the five year warranty), but I've been hearing a
lot of good things about Samsung's latest 750GB/1TB drives (the three
platter series).

-mj
--
Michael-John Turner
mj@...     | http://mjturner.net/
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Peter Corlett]

On Fri, Apr 18, 2008 at 09:07:29AM +0100, Michael-John Turner wrote:
[...]
> Agree with you on this one. In addition, the ability to be able to just
> pop the disks into a replacement machine running the same OS, as you can
> with software RAID, is a huge advantage. With hardware RAID, you'd need to
> find the same model of controller, etc.

When I last bought a hardware RAID card, I bought *two* and then swapped the
spare card in after building the array to see if it was still readable.

... and then it turned out that Linux software RAID was faster anyway.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Lionel Peterson]

>From: Peter Corlett <abuse@...>
>Date: 2008/04/18 Fri AM 07:10:49 CDT
>To: The Rescue List <rescue@...>
>Subject: Re: [rescue] Filesystem choice for fileserver?

>On Fri, Apr 18, 2008 at 09:07:29AM +0100, Michael-John Turner wrote:
>[...]
>> Agree with you on this one. In addition, the ability to be able to just
>> pop the disks into a replacement machine running the same OS, as you can
>> with software RAID, is a huge advantage. With hardware RAID, you'd need to
>> find the same model of controller, etc.
>
>When I last bought a hardware RAID card, I bought *two* and then swapped the
>spare card in after building the array to see if it was still readable.
>
>... and then it turned out that Linux software RAID was faster anyway.

<homer Simpson>
   DOH!
</Homer Simpson>

(that has made it across the ocean, right?)

Lionel
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Phil Stracchino-3]

Peter Corlett wrote:

> On Fri, Apr 18, 2008 at 09:07:29AM +0100, Michael-John Turner wrote:
> [...]
>> Agree with you on this one. In addition, the ability to be able to just
>> pop the disks into a replacement machine running the same OS, as you can
>> with software RAID, is a huge advantage. With hardware RAID, you'd need to
>> find the same model of controller, etc.
>
> When I last bought a hardware RAID card, I bought *two* and then swapped the
> spare card in after building the array to see if it was still readable.
>
> ... and then it turned out that Linux software RAID was faster anyway.

A lot of hardware RAID cards have horribly slow processors on them.


--
   Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
   alaric@...   alaric@...   phil@...
          Renaissance Man, Unix ronin, Perl hacker, Free Stater
                  It's not the years, it's the mileage.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Phil Stracchino-3]

Phil Stracchino wrote:

> Peter Corlett wrote:
>> On Fri, Apr 18, 2008 at 09:07:29AM +0100, Michael-John Turner wrote:
>> [...]
>>> Agree with you on this one. In addition, the ability to be able to just
>>> pop the disks into a replacement machine running the same OS, as you can
>>> with software RAID, is a huge advantage. With hardware RAID, you'd
>>> need to
>>> find the same model of controller, etc.
>>
>> When I last bought a hardware RAID card, I bought *two* and then
>> swapped the
>> spare card in after building the array to see if it was still readable.
>>
>> ... and then it turned out that Linux software RAID was faster anyway.
>
> A lot of hardware RAID cards have horribly slow processors on them.

Oops ...  premature sendage.

What I meant to say was that the big win of *good* hardware RAID is
that, as long as the manufacturer did the job properly, the host OS
doesn't have to be up for you to rebuild or reconfigure the array (or to
configure it in the first place).


--
   Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
   alaric@...   alaric@...   phil@...
          Renaissance Man, Unix ronin, Perl hacker, Free Stater
                  It's not the years, it's the mileage.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Dan Sikorski]

Phil Stracchino wrote:

>>
>> A lot of hardware RAID cards have horribly slow processors on them.
>
> Oops ...  premature sendage.
>
> What I meant to say was that the big win of *good* hardware RAID is
> that, as long as the manufacturer did the job properly, the host OS
> doesn't have to be up for you to rebuild or reconfigure the array (or
> to configure it in the first place).
>
>

I've found that good hardware RAID also has good processors and is not
slow.  I much prefer having the RAID processing done by a dedicated
processor, there are plenty of other things for the CPU to do.  Also,
you have less data being transferred over the host bus, which can easily
be the bottleneck when you're talking about pc based systems with 32bit
PCI.  This was a strong consideration of mine when i built my fileserver
with a PCI-X 3ware 9500.  In  my testing, RAID 5 was faster than RAID 10
on the 3ware.  Clearly my bottleneck is disk speed, not controller or
host bus speed.

    -Dan Sikorski
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[der Mouse]

>> A lot of hardware RAID cards have horribly slow processors on them.
> [..."hardware RAID"...]

Does anyone actually do hardware RAID?  I've never seen any.  All the
supposedly-hardware RAID I've seen is actually software RAID running on
a dedicated processor (what would more honestly be called firmware
RAID).  Usually it doesn't even try to hide that - for example, I've
seen POST-time messages like "booting controller kernel".

Have I missed something?  Is there anyone who actually does RAID in
hardware?

/~\ The ASCII der Mouse
\ / Ribbon Campaign
 X  Against HTML       mouse@...
/ \ Email!     7D C8 61 52 5D E7 2D 39  4E F1 31 3E E8 B3 27 4B
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Dan Sikorski]

der Mouse wrote:
> Does anyone actually do hardware RAID?  I've never seen any.  All the
> supposedly-hardware RAID I've seen is actually software RAID running on
> a dedicated processor (what would more honestly be called firmware
> RAID).  Usually it doesn't even try to hide that - for example, I've
> seen POST-time messages like "booting controller kernel".
>
> Have I missed something?  Is there anyone who actually does RAID in
> hardware?
>  
I guess when i say hardware RAID, what i really mean is that there is
dedicated hardware for the RAID processing so that it is offloaded from
the CPU.  You're right, perhaps "firmware raid" would be a more accurate
term.

    -Dan Sikorski
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Phil Stracchino-3]

der Mouse wrote:

>>> A lot of hardware RAID cards have horribly slow processors on them.
>> [..."hardware RAID"...]
>
> Does anyone actually do hardware RAID?  I've never seen any.  All the
> supposedly-hardware RAID I've seen is actually software RAID running on
> a dedicated processor (what would more honestly be called firmware
> RAID).  Usually it doesn't even try to hide that - for example, I've
> seen POST-time messages like "booting controller kernel".
>
> Have I missed something?  Is there anyone who actually does RAID in
> hardware?

"Firmware RAID" is, I'm sure, probably a more accurate term.

True hardware raid implemented in, say, an FPGA could probably be
blindingly fast.


--
   Phil Stracchino, CDK#2     DoD#299792458     ICBM: 43.5607, -71.355
   alaric@...   alaric@...   phil@...
          Renaissance Man, Unix ronin, Perl hacker, Free Stater
                  It's not the years, it's the mileage.
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Sridhar Ayengar]

Phil Stracchino wrote:

>> Does anyone actually do hardware RAID?  I've never seen any.  All the
>> supposedly-hardware RAID I've seen is actually software RAID running on
>> a dedicated processor (what would more honestly be called firmware
>> RAID).  Usually it doesn't even try to hide that - for example, I've
>> seen POST-time messages like "booting controller kernel".
>>
>> Have I missed something?  Is there anyone who actually does RAID in
>> hardware?
>
> "Firmware RAID" is, I'm sure, probably a more accurate term.
>
> True hardware raid implemented in, say, an FPGA could probably be
> blindingly fast.

I've seen older systems implemented in that fashion.  (At least on the
mainframe.)  Are any of the current systems run that way?

Peace...  Sridhar
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Ron Wickersham]

i'm also planning to put a ZFS file server together (an E450 so i'll have
good disk bandwidth) and have read lots (but certainly not all) of the
Sun blogs and papers on ZFS.   there were several recommendations to use
JBOD because this ensures that the ZFS does its job and that when you use
other software or hardware RAID you don't get the full ZFS advantage for
bit-rot and other features that ensure data integrity.     so after a
failure, the other software or hardware RAID system makes sure that the
copies are consistent (in the case of full mirror, both copies will be
identical even if both have errors) and ZFS can't fix the error if i
understand the point correctly.

-ron
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Jonathan C. Patschke]

On Fri, 18 Apr 2008, Phil Stracchino wrote:

>> Have I missed something?  Is there anyone who actually does RAID in
>> hardware?
>
> "Firmware RAID" is, I'm sure, probably a more accurate term.
>
> True hardware raid implemented in, say, an FPGA could probably be blindingly
> fast.

Wouldn't that still be firmware RAID?  The FPGA usually has to download
its configuration from a serial EEPROM.

I mean, as long as we're going to be pedantic....

--
Jonathan Patschke | "There is no such thing as a short of reserves...
Elgin, TX         |  one bank can have a problem...the Fed can print
USA               |  money, there is no shortage."
.                 |     --Jim Glassman, US Economist, JPMorgan Chase
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue

[Shannon Hendrix]

On Apr 18, 2008, at 10:10 , der Mouse wrote:

>>> A lot of hardware RAID cards have horribly slow processors on them.
>> [..."hardware RAID"...]
>
> Does anyone actually do hardware RAID?  I've never seen any.  All the
> supposedly-hardware RAID I've seen is actually software RAID running  
> on
> a dedicated processor (what would more honestly be called firmware
> RAID).  Usually it doesn't even try to hide that - for example, I've
> seen POST-time messages like "booting controller kernel".
>
> Have I missed something?  Is there anyone who actually does RAID in
> hardware?

Well, thinking that way:

I've never seen a hardware CPU.  They all are programmed in microcode  
or "fixed logic instructions".

Has anyone actually done a CPU in hardware?

:)

--
"Where some they sell their dreams for small desires."
_______________________________________________
rescue list - http://www.sunhelp.org/mailman/listinfo/rescue