Slow write performance on Compaq Smart Array 64xx (ciss0)

[Joe S-3]

Why is the write performance of my RAID controller so slow? I've noticed
the problem on this server and another Intel Xeon based server with the
same controller. Both are running OpenBSD 4.0 generic. I will focus on
the amd64 box in this email. If this is fixed in current, great, I'll
upgrade. If not, will it be fixed in 4.1?

Here is a basic write test:

(write test running bsd kernel)
# dd if=/dev/zero of=/data/testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 113.978 secs (22999380 bytes/sec)

(write test running bsd.mp kernel)
# dd if=/dev/zero of=/data/testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 115.765 secs (22644445 bytes/sec)

Here is some information about the system:

# sysctl hw.sensors

hw.sensors.0=ciss0, sd0, drive online, OK

# bioctl -h ciss0
Volume  Status               Size Device
ciss0 0 Online               1.4T sd0     RAID5
       0 Online               279G 1:0.0   noencl <COMPAQ  BD3008A4C6
       1 Online               279G 1:1.0   noencl <COMPAQ  BD30089BBA
       2 Online               279G 1:2.0   noencl <COMPAQ  BD30089BBA
       3 Online               279G 1:3.0   noencl <COMPAQ  BD30089BBA
       4 Online               279G 1:4.0   noencl <COMPAQ  BD30089BBA
       5 Online               279G 1:5.0   noencl <COMPAQ  BD30089BBA



# dmesg
OpenBSD 4.0 (GENERIC.MP) #967: Sat Sep 16 20:38:15 MDT 2006
     deraadt@...:/usr/src/sys/arch/amd64/compile/GENERIC.MP
real mem = 4118376448 (4021852K)
avail mem = 3536994304 (3454096K)
using 22937 buffers containing 412045312 bytes (402388K) of memory
mainbus0 (root)
bios0 at mainbus0: SMBIOS rev. 2.3 @ 0xec000 (64 entries)
bios0: HP ProLiant DL385 G1
mainbus0: Intel MP Specification (Version 1.4) (HP       PROLIANT    )
cpu0 at mainbus0: apid 0 (boot processor)
cpu0: AMD Opteron(tm) Processor 285, 2606.23 MHz
cpu0:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu0: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu0: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu0: apic clock running at 200MHz
cpu1 at mainbus0: apid 2 (application processor)
cpu1: AMD Opteron(tm) Processor 285, 2605.91 MHz
cpu1:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu1: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu1: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu1: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2 at mainbus0: apid 1 (application processor)
cpu2: AMD Opteron(tm) Processor 285, 2605.91 MHz
cpu2:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu2: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu2: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu2: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3 at mainbus0: apid 3 (application processor)
cpu3: AMD Opteron(tm) Processor 285, 2605.91 MHz
cpu3:
FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CFLUSH,MMX,FXSR,SSE,SSE2,HTT,SSE3,NXE,MMXX,FFXSR,LONG,3DNOW2,3DNOW
cpu3: 64KB 64b/line 2-way I-cache, 64KB 64b/line 2-way D-cache, 1MB
64b/line 16-way L2 cache
cpu3: ITLB 32 4KB entries fully associative, 8 4MB entries fully associative
cpu3: DTLB 32 4KB entries fully associative, 8 4MB entries fully associative
mpbios: bus 0 is type PCI
mpbios: bus 1 is type PCI
mpbios: bus 2 is type PCI
mpbios: bus 3 is type PCI
mpbios: bus 4 is type PCI
mpbios: bus 5 is type PCI
mpbios: bus 6 is type PCI
mpbios: bus 32 is type ISA
ioapic0 at mainbus0 apid 4 pa 0xfec00000, version 11, 24 pins
ioapic1 at mainbus0 apid 5 pa 0xfec10000, version 11, 4 pins
ioapic2 at mainbus0 apid 6 pa 0xfec20000, version 11, 4 pins
ioapic3 at mainbus0 apid 7 pa 0xfdc00000, version 11, 4 pins
ioapic4 at mainbus0 apid 8 pa 0xfdc10000, version 11, 4 pins
pci0 at mainbus0 bus 0: configuration mode 1
ppb0 at pci0 dev 3 function 0 "AMD 8111 PCI-PCI" rev 0x07
pci1 at ppb0 bus 1
ohci0 at pci1 dev 0 function 0 "AMD 8111 USB" rev 0x0b: apic 4 int 19
(irq 5), version 1.0, legacy support
usb0 at ohci0: USB revision 1.0
uhub0 at usb0
uhub0: AMD OHCI root hub, rev 1.00/1.00, addr 1
uhub0: 3 ports with 3 removable, self powered
ohci1 at pci1 dev 0 function 1 "AMD 8111 USB" rev 0x0b: apic 4 int 19
(irq 5), version 1.0, legacy support
usb1 at ohci1: USB revision 1.0
uhub1 at usb1
uhub1: AMD OHCI root hub, rev 1.00/1.00, addr 1
uhub1: 3 ports with 3 removable, self powered
"Compaq iLO" rev 0x01 at pci1 dev 2 function 0 not configured
"Compaq iLO" rev 0x01 at pci1 dev 2 function 2 not configured
vga1 at pci1 dev 3 function 0 "ATI Rage XL" rev 0x27
wsdisplay0 at vga1 mux 1: console (80x25, vt100 emulation)
wsdisplay0: screen 1-5 added (80x25, vt100 emulation)
pcib0 at pci0 dev 4 function 0 "AMD AMD8111 LPC" rev 0x05
pciide0 at pci0 dev 4 function 1 "AMD 8111 IDE" rev 0x03: DMA, channel 0
configured to compatibility, channel 1 configured to compatibility
atapiscsi0 at pciide0 channel 0 drive 0
scsibus0 at atapiscsi0: 2 targets
cd0 at scsibus0 targ 0 lun 0: <TSSTcorp, CD-ROM TS-L162C, N204> SCSI0
5/cdrom removable
cd0(pciide0:0:0): using PIO mode 4, Ultra-DMA mode 2
pciide0: channel 1 ignored (disabled)
amdpm0 at pci0 dev 4 function 3 "AMD 8111 Power" rev 0x05: rng active
iic0 at amdpm0
ppb1 at pci0 dev 7 function 0 "AMD 8131 PCIX" rev 0x12
pci2 at ppb1 bus 2
ciss0 at pci2 dev 4 function 0 "Compaq Smart Array 64xx" rev 0x01: apic
5 int 0 (irq 7)
ciss0: 1 LD, HW rev 1, FW 2.58/2.58
scsibus1 at ciss0: 1 targets
sd0 at scsibus1 targ 0 lun 0: <HP, LOGICAL VOLUME, 2.58> SCSI0 0/direct
fixed
sd0: 1430488MB, 1430488 cyl, 64 head, 32 sec, 512 bytes/sec, 2929640988
sec total
aapic0 at pci0 dev 7 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
ppb2 at pci0 dev 8 function 0 "AMD 8131 PCIX" rev 0x12
pci3 at ppb2 bus 3
bge0 at pci3 dev 6 function 0 "Broadcom BCM5704C" rev 0x10, BCM5704 B0
(0x2100): apic 6 int 0 (irq 7), address xx:xx:xx:xx:xx:xx
brgphy0 at bge0 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
bge1 at pci3 dev 6 function 1 "Broadcom BCM5704C" rev 0x10, BCM5704 B0
(0x2100): apic 6 int 1 (irq 10), address xx:xx:xx:xx:xx:xx
brgphy1 at bge1 phy 1: BCM5704 10/100/1000baseT PHY, rev. 0
aapic1 at pci0 dev 8 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
pchb0 at pci0 dev 24 function 0 "AMD AMD64 HyperTransport" rev 0x00
pci4 at pchb0 bus 4
ppb3 at pci4 dev 9 function 0 "AMD 8131 PCIX" rev 0x12
pci5 at ppb3 bus 5
aapic2 at pci4 dev 9 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
ppb4 at pci4 dev 10 function 0 "AMD 8131 PCIX" rev 0x12
pci6 at ppb4 bus 6
aapic3 at pci4 dev 10 function 1 "AMD 8131 PCIX IOAPIC" rev 0x01
pchb1 at pci0 dev 24 function 1 "AMD AMD64 Address Map" rev 0x00
pchb2 at pci0 dev 24 function 2 "AMD AMD64 DRAM Cfg" rev 0x00
pchb3 at pci0 dev 24 function 3 "AMD AMD64 Misc Cfg" rev 0x00
pchb4 at pci0 dev 25 function 0 "AMD AMD64 HyperTransport" rev 0x00
pchb5 at pci0 dev 25 function 1 "AMD AMD64 Address Map" rev 0x00
pchb6 at pci0 dev 25 function 2 "AMD AMD64 DRAM Cfg" rev 0x00
pchb7 at pci0 dev 25 function 3 "AMD AMD64 Misc Cfg" rev 0x00
isa0 at pcib0
isadma0 at isa0
com0 at isa0 port 0x3f8/8 irq 4: ns16550a, 16 byte fifo
com1 at isa0 port 0x2f8/8 irq 3: ns16550a, 16 byte fifo
pckbc0 at isa0 port 0x60/5
pckbd0 at pckbc0 (kbd slot)
pckbc0: using irq 1 for kbd slot
wskbd0 at pckbd0: console keyboard, using wsdisplay0
pmsi0 at pckbc0 (aux slot)
pckbc0: using irq 12 for aux slot
wsmouse0 at pmsi0 mux 0
pcppi0 at isa0 port 0x61
midi0 at pcppi0: <PC speaker>
spkr0 at pcppi0
fdc0 at isa0 port 0x3f0/6 irq 6 drq 2
dkcsum: sd0 matches BIOS drive 0x80
root on sd0a
rootdev=0x400 rrootdev=0xd00 rawdev=0xd02

[Brian Candler]

On Sun, Jan 28, 2007 at 12:29:21AM -0800, Joe wrote:
> Why is the write performance of my RAID controller so slow?
...
> (write test running bsd kernel)
> # dd if=/dev/zero of=/data/testfile count=20000 bs=128k
> 20000+0 records in
> 20000+0 records out
> 2621440000 bytes transferred in 113.978 secs (22999380 bytes/sec)

That's about 22MB per second. Sounds plausible to me, given that RAID5
arrays intrinsically have very poor write performance. What value were you
expecting to see?

For comparison, what do you get for the same test but for read speed? i.e.

    # dd if=/data/testfile of=/dev/null bs=128k

RAID5 writes are slow because a single block write involves 4 disk
operations (read old data block, read old parity, write new data block,
write new parity). A non-volatile write-through cache can speed things up to
a degree though.

If you want high write performace, try setting up your six-disk array as
three separate mirrored pairs, or as a single RAID-01 (strip/mirror) and see
what you get. Of course your available storage size will be reduced to
3/5ths of what it was.

Regards,

Brian.

[Vijay Sankar]

On Sunday 28 January 2007 08:45, Brian Candler wrote:

> On Sun, Jan 28, 2007 at 12:29:21AM -0800, Joe wrote:
> > Why is the write performance of my RAID controller so slow?
>
> ...
>
> > (write test running bsd kernel)
> > # dd if=/dev/zero of=/data/testfile count=20000 bs=128k
> > 20000+0 records in
> > 20000+0 records out
> > 2621440000 bytes transferred in 113.978 secs (22999380 bytes/sec)
>
> That's about 22MB per second. Sounds plausible to me, given that RAID5
> arrays intrinsically have very poor write performance. What value were you
> expecting to see?

This is what I see on a DL 380 G4. It has two 300GB drives in RAID1
configuration and 4 300 GB drives on RAID 5 configuration.:

RAID5 with 4 300GB drives
# dd if=/dev/zero of=testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 31.360 secs (83589627 bytes/sec)

RAID 1 with 2 300 GB drives
#dd if=/dev/zero of=testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 43.803 secs (59845997 bytes/sec)

bioctl -h ciss0 gives me

bioctl: Can't locate ciss0 device via /dev/bio

If possible, please let me know what has to be set up to get the info from
bioctl.

sysctl hw gives me

hw.machine=i386
hw.model=Intel(R) Xeon(TM) CPU 3.80GHz ("GenuineIntel" 686-class)
hw.ncpu=1
hw.byteorder=1234
hw.physmem=2147000320
hw.usermem=2146467840
hw.pagesize=4096
hw.disknames=sd0,sd1,cd0,fd0
hw.diskcount=4
hw.cpuspeed=3801
hw.vendor=HP
hw.product=ProLiant DL380 G4

It does not give me the type of output hw.sensors.0 etc. If possible, please
let me know how to get this also. I can then run another test that makes it
more comparable.

>
> For comparison, what do you get for the same test but for read speed? i.e.
>
>     # dd if=/data/testfile of=/dev/null bs=128k
>
> RAID5 writes are slow because a single block write involves 4 disk
> operations (read old data block, read old parity, write new data block,
> write new parity). A non-volatile write-through cache can speed things up
> to a degree though.
>
> If you want high write performace, try setting up your six-disk array as
> three separate mirrored pairs, or as a single RAID-01 (strip/mirror) and
> see what you get. Of course your available storage size will be reduced to
> 3/5ths of what it was.
>
> Regards,
>
> Brian.
>
>
> !DSPAM:1,45bcb760150711234514069!

--
Vijay Sankar
ForeTell Technologies Limited
59 Flamingo Avenue, Winnipeg, MB, Canada R3J 0X6
Phone: [phone], E-Mail: vsankar@...

[Henning Brauer]

* Vijay Sankar <vsankar@...> [2007-01-28 16:07]:
> bioctl -h ciss0 gives me
>
> bioctl: Can't locate ciss0 device via /dev/bio

ciss doesn't support bio yet.

--
Henning Brauer, hb@..., henning@...
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam

[Vijay Sankar]

On Sunday 28 January 2007 10:32, Henning Brauer wrote:
> * Vijay Sankar <vsankar@...> [2007-01-28 16:07]:
> > bioctl -h ciss0 gives me
> >
> > bioctl: Can't locate ciss0 device via /dev/bio
>
> ciss doesn't support bio yet.

Thank you very much, I will wait, for sure.

I thought it was supported because of the dmesg submitted by joe
<js.lists@...> and the output quoted in his original message

# sysctl hw.sensors

hw.sensors.0=ciss0, sd0, drive online, OK

# bioctl -h ciss0
Volume  Status               Size Device
ciss0 0 Online               1.4T sd0     RAID5
       0 Online               279G 1:0.0   noencl <COMPAQ  BD3008A4C6
       1 Online               279G 1:1.0   noencl <COMPAQ  BD30089BBA
       2 Online               279G 1:2.0   noencl <COMPAQ  BD30089BBA
       3 Online               279G 1:3.0   noencl <COMPAQ  BD30089BBA
       4 Online               279G 1:4.0   noencl <COMPAQ  BD30089BBA
       5 Online               279G 1:5.0   noencl <COMPAQ  BD30089BBA

Vijay

--
Vijay Sankar
ForeTell Technologies Limited
59 Flamingo Avenue, Winnipeg, MB, Canada R3J 0X6
Phone: [phone], E-Mail: vsankar@...

[Joe S-3]

Brian Candler wrote:

> On Sun, Jan 28, 2007 at 12:29:21AM -0800, Joe wrote:
>> Why is the write performance of my RAID controller so slow?
> ...
>> (write test running bsd kernel)
>> # dd if=/dev/zero of=/data/testfile count=20000 bs=128k
>> 20000+0 records in
>> 20000+0 records out
>> 2621440000 bytes transferred in 113.978 secs (22999380 bytes/sec)
>
> That's about 22MB per second. Sounds plausible to me, given that RAID5
> arrays intrinsically have very poor write performance. What value were you
> expecting to see?
>
> For comparison, what do you get for the same test but for read speed? i.e.
>
>     # dd if=/data/testfile of=/dev/null bs=128k
>
> RAID5 writes are slow because a single block write involves 4 disk
> operations (read old data block, read old parity, write new data block,
> write new parity). A non-volatile write-through cache can speed things up to
> a degree though.
>
> If you want high write performace, try setting up your six-disk array as
> three separate mirrored pairs, or as a single RAID-01 (strip/mirror) and see
> what you get. Of course your available storage size will be reduced to
> 3/5ths of what it was.
>
> Regards,
>
> Brian.
>

Some more tests:

# dd if=/dev/zero of=./testfile count=1000000
1000000+0 records in
1000000+0 records out
512000000 bytes transferred in 16.354 secs (31306797 bytes/sec)

# dd if=./testfile of=/dev/null count=1000000

1000000+0 records in
1000000+0 records out
512000000 bytes transferred in 6.013 secs (85137347 bytes/sec)

So is 30MBps acceptable write speed for RAID 5 on a Compaq Smart Array
64xx controller?

Could this be a driver issue?





BTW,

I have another box with the same controller, but in 2 disks in RAID 0.


# bioctl -h ciss0
Volume  Status               Size Device
ciss0 0 Online               136G sd0     RAID0
       0 Online              67.8G 0:0.0   noencl <COMPAQ  BD0728A4B4
       1 Online              67.8G 0:1.0   noencl <COMPAQ  BD0728A4B4


# dd if=/dev/zero of=/nsm/testfile count=20000 bs=128k
20000+0 records in
20000+0 records out
2621440000 bytes transferred in 29.696 secs (88274982 bytes/sec)

[Bill Marquette]

On 1/28/07, Henning Brauer <lists-openbsd@...> wrote:
> * Vijay Sankar <vsankar@...> [2007-01-28 16:07]:
> > bioctl -h ciss0 gives me
> >
> > bioctl: Can't locate ciss0 device via /dev/bio
>
> ciss doesn't support bio yet.

Unless I'm mistaken, mickey@ added it pre-4.0 here:
http://marc.theaimsgroup.com/?l=openbsd-cvs&m=115671197617717&w=2

and bio(4) man page claims it's supported as does ciss(4) (albeit with a caveat)

--Bill

[Brian Candler]

On Sun, Jan 28, 2007 at 11:28:27AM -0800, Joe wrote:

> Some more tests:
>
> # dd if=/dev/zero of=./testfile count=1000000
> 1000000+0 records in
> 1000000+0 records out
> 512000000 bytes transferred in 16.354 secs (31306797 bytes/sec)
>
> # dd if=./testfile of=/dev/null count=1000000
>
> 1000000+0 records in
> 1000000+0 records out
> 512000000 bytes transferred in 6.013 secs (85137347 bytes/sec)
>
> So is 30MBps acceptable write speed for RAID 5 on a Compaq Smart Array
> 64xx controller?
>
> Could this be a driver issue?

I doubt it: clearly it can transfer data at 85MBps, and it's unlikely that
the SCSI bus can transfer data faster in one direction than the other.

I don't know this controller specifically, but maybe a better controller
would give you better RAID5 write performance. Or maybe something isn't
quite set up correctly on the card (e.g. if there's NVRAM write-through
cache, maybe the battery isn't present or it's disabled for some other
reason)

> I have another box with the same controller, but in 2 disks in RAID 0.
>
>
> # bioctl -h ciss0
> Volume  Status               Size Device
> ciss0 0 Online               136G sd0     RAID0
>       0 Online              67.8G 0:0.0   noencl <COMPAQ  BD0728A4B4
>       1 Online              67.8G 0:1.0   noencl <COMPAQ  BD0728A4B4
>
>
> # dd if=/dev/zero of=/nsm/testfile count=20000 bs=128k
> 20000+0 records in
> 20000+0 records out
> 2621440000 bytes transferred in 29.696 secs (88274982 bytes/sec)

RAID 0 is just striping, so half the data gets written to one disk while
half gets written to the other, so that would be expected to have better
performance than a single disk.

Regards,

Brian.

[Henning Brauer]

* Henning Brauer <lists-openbsd@...> [2007-01-28 17:37]:
> * Vijay Sankar <vsankar@...> [2007-01-28 16:07]:
> > bioctl -h ciss0 gives me
> >
> > bioctl: Can't locate ciss0 device via /dev/bio
>
> ciss doesn't support bio yet.

err, sorry, I missed that it is now, to some extent.

--
Henning Brauer, hb@..., henning@...
BS Web Services, http://bsws.de
Full-Service ISP - Secure Hosting, Mail and DNS Services
Dedicated Servers, Rootservers, Application Hosting - Hamburg & Amsterdam

[Michael Shalayeff-2]

On Sun, Jan 28, 2007 at 05:32:41PM +0100, Henning Brauer wrote:
> * Vijay Sankar <vsankar@...> [2007-01-28 16:07]:
> > bioctl -h ciss0 gives me
> >
> > bioctl: Can't locate ciss0 device via /dev/bio
>
> ciss doesn't support bio yet.

rtfm
bio(4) support is only possible for one volume right now.
so if you'd have only sd0 there it'd work.
cu
--
    paranoic mickey       (my employers have changed but, the name has remained)

[Johan M:son Lindman-2]

On Sunday 28 January 2007 22:21, you wrote:

> On Sun, Jan 28, 2007 at 11:28:27AM -0800, Joe wrote:
> > Some more tests:
> >
> > # dd if=/dev/zero of=./testfile count=1000000
> > 1000000+0 records in
> > 1000000+0 records out
> > 512000000 bytes transferred in 16.354 secs (31306797 bytes/sec)
> >
> > # dd if=./testfile of=/dev/null count=1000000
> >
> > 1000000+0 records in
> > 1000000+0 records out
> > 512000000 bytes transferred in 6.013 secs (85137347 bytes/sec)
> >
> > So is 30MBps acceptable write speed for RAID 5 on a Compaq Smart Array
> > 64xx controller?
> >
> > Could this be a driver issue?
>
> I doubt it: clearly it can transfer data at 85MBps, and it's unlikely that
> the SCSI bus can transfer data faster in one direction than the other.
>
> I don't know this controller specifically, but maybe a better controller
> would give you better RAID5 write performance. Or maybe something isn't
> quite set up correctly on the card (e.g. if there's NVRAM write-through
> cache, maybe the battery isn't present or it's disabled for some other
> reason)

You're right.
Most probably this guy joe is lacking Battery Backed Write Cache enabler.
HP ships their boxes without it as default.
This will likely give some performance penalty.

I told him to check if he has a BBWC in a previous mail but as most idiots on
misc he ignored it and sent another mail asking the same question. :-)


Regards
Johan M:son