Thinkpad T43 cached read Speed(hdparm -T) mystery

[jqian-2]

Hi All:
        This is not strictly an thinkpad issue, but I'm running out of
idea to check and people here are knowlegeable.
        I have a T43 with 1.86GHz, 553MHz FSB Pentium M Dothan(750)
processor and 2G of PC-4200 RAM. It also has a 60GB Hitachi 7200 rpm
drive (HTS726060M9AT00)(PATA-to-SATA bridge, showing up as /dev/sda).  
When it was running a Ubuntu hoary a few months ago(too lazy to
update), with vanilla 2.6.18 kernel, hdparm -T shows about 1800MB/sec.  
However, now I'm in hardy, with ubuntu 2.6.24-18 kernel, the hdparm -T
show only less than 900MB/sec. The disk read speed hdparm -t shows the
same result(about 37-38MB/sec).
        My questions are:
        0. If you have a T43(p) with comparable set up, what's your
number?
        1. Does this mean my computer's bus speed is suddenly cut in
half, as seen by linux? What's is the performance implication?
        2.What exactly does hdparm -T measure(processor bus speed?)?  
Is it a good gauge for performance? If it is not a good guage for
performance, what is?
        3.What cause this dramatic reduction and performance penalty?  
Kernel? Fglrx?
        Thanks in advance for enlightenment!
                                                                Jiang
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Michael Karcher-2]

Am Mittwoch, den 25.06.2008, 06:52 -0400 schrieb
jqian@...:
[...]
> update), with vanilla 2.6.18 kernel, hdparm -T shows about 1800MB/sec.  
> However, now I'm in hardy, with ubuntu 2.6.24-18 kernel, the hdparm -T
> show only less than 900MB/sec. The disk read speed hdparm -t shows the
> same result(about 37-38MB/sec).
> My questions are:
> 0. If you have a T43(p) with comparable set up, what's your
> number?
I don't have a T43p. On a 2GHz Core2Duo (T7200), FSB533 IIRC, I get
approximately 2200 MB/sec, kernel 2.6.25, x64-kernel with CONFIG_MCORE2.

> 1. Does this mean my computer's bus speed is suddenly cut in
> half, as seen by linux? What's is the performance implication?
> 2.What exactly does hdparm -T measure(processor bus speed?)?  
> Is it a good gauge for performance? If it is not a good guage for
> performance, what is?
> 3.What cause this dramatic reduction and performance penalty?  
> Kernel? Fglrx?
Collective answer to the three questions: hdparm -T measures, how fast
the data is copied from the disk cache (in memory) to application
memory, so it is a "simple" memory-to-memory copy. I wrote simple into
quotes, because one has to be careful to use the first/second level
caches of the processor in an optimal way according to the use case. For
example, in the disk cache copying case, it makes no sense to put the
disk cache data into processor caches, as it will not be used that quick
again, whereas the destination buffer would be quite useful in the
cache, as most applications access the data read by the read() system
call (thats what hdparm measures; I don't know whether hdparm touches
the data).

You are really seeing a drop of memory-to-memory-copying performance to
one half. The probable cause is, that the optimal cache management is
buried into SSE/SSE2/SSE3/SSE4 instructions, and for really
memory-intensive task, each Intel processor prefers a slight different
way to get optimal performance. Your vanilla kernel was probably
optimized for the Pentium M architecture, whereas the ubunty kernel is
optimized for another architecture. Compile your own kernel to get up to
the old performance again.

The performance implications in day-to-day use are quite low (as long as
your day-to-day use is not copying gigagbytes of data or video cutting),
so you won't notice the difference. Memory-to-memory copies are avoided
if possible, for example by doing a read-only mmap of all code. That
means that the processor is set up in a way that applications can not
overwrite their own code (they get a segfault instead), and thus it is
safe to directly make the disk cache visible to the application instead
of making a copy before. The same holds for swap-in/swap-out: The hard
disk controller directly accesses the data in the memory range that is
visible to the application, and thus no memory-to-memory copy is needed.

Memory-to-memory copies within applications (or mem-to-video-mem in the
X-Server) is not using the kernel functions, and thus independent of
kernel compilation options (but an aptly optimized libc might help
there).

> Thanks in advance for enlightenment!
I hope the explanations cause enlightenment.

Regards,
  Michael Karcher

jqian@... wrote:
> 2.What exactly does hdparm -T measure(processor bus speed?)?  

It shows how fast an application can read data from kernel's block
device cache.

I don't have answers for rest of your quetsions, sorry.

Cheers,
-jkt
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Henrique de Moraes Holschuh-2]

On Wed, 25 Jun 2008, jqian@... wrote:
> When it was running a Ubuntu hoary a few months ago(too lazy to
> update), with vanilla 2.6.18 kernel, hdparm -T shows about 1800MB/sec.  
> However, now I'm in hardy, with ubuntu 2.6.24-18 kernel, the hdparm -T

Might be the ZEROPAGE removal stuff that went in in 2.6.24-18.  Try 2.6.25.9
just released, it is supposed to fix a performance regression in the
ZEROPAGE issue.

If the regression in hdparm -T speed is indeed related to ZEROPAGE, that
means hdparm -T is a completely meaningless test.

> 2.What exactly does hdparm -T measure(processor bus speed?)?  

Probably nothing useful.

> Is it a good gauge for performance? If it is not a good guage for
> performance, what is?

Your real workload is the only gauge for performance that matters.

--
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Henrique de Moraes Holschuh-2]

On Wed, 25 Jun 2008, Henrique de Moraes Holschuh wrote:
> On Wed, 25 Jun 2008, jqian@... wrote:
> > When it was running a Ubuntu hoary a few months ago(too lazy to
> > update), with vanilla 2.6.18 kernel, hdparm -T shows about 1800MB/sec.  
> > However, now I'm in hardy, with ubuntu 2.6.24-18 kernel, the hdparm -T
>
> Might be the ZEROPAGE removal stuff that went in in 2.6.24-18.  Try 2.6.25.9

I mean 2.6.24.

--
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[jqian-2]

On Wed, Jun 25, 2008 at 02:01:52PM -0300, Henrique de Moraes Holschuh wrote:

> On Wed, 25 Jun 2008, jqian@... wrote:
> > When it was running a Ubuntu hoary a few months ago(too lazy to
> > update), with vanilla 2.6.18 kernel, hdparm -T shows about 1800MB/sec.  
> > However, now I'm in hardy, with ubuntu 2.6.24-18 kernel, the hdparm -T
>
> Might be the ZEROPAGE removal stuff that went in in 2.6.24-18.  Try 2.6.25.9
> just released, it is supposed to fix a performance regression in the
> ZEROPAGE issue.
>
> If the regression in hdparm -T speed is indeed related to ZEROPAGE, that
> means hdparm -T is a completely meaningless test.

I'm sorry to be a bit dense here, but if 2.6.25.9 fix a "performance
regression", then what hdparm -T detects is a real regression, not a
meaningless test, right?
>
> > 2.What exactly does hdparm -T measure(processor bus speed?)?  
>
> Probably nothing useful.
>
> > Is it a good gauge for performance? If it is not a good guage for
> > performance, what is?
>
> Your real workload is the only gauge for performance that matters.
It's hard for me to judge performances when I had distro change, is
there some benchmark that more sensibly reflect the bus speed of my
computer?
Thanks a lot.
Jiang
>
> --
>   "One disk to rule them all, One disk to find them. One disk to bring
>   them all and in the darkness grind them. In the Land of Redmond
>   where the shadows lie." -- The Silicon Valley Tarot
>   Henrique Holschuh
> --
> The linux-thinkpad mailing list home page is at:
> http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[jqian-2]

On Wed, Jun 25, 2008 at 02:05:48PM +0200, Michael Karcher wrote:

> Am Mittwoch, den 25.06.2008, 06:52 -0400 schrieb
> jqian@...:
> [...]
> > update), with vanilla 2.6.18 kernel, hdparm -T shows about 1800MB/sec.  
> > However, now I'm in hardy, with ubuntu 2.6.24-18 kernel, the hdparm -T
> > show only less than 900MB/sec. The disk read speed hdparm -t shows the
> > same result(about 37-38MB/sec).
> > My questions are:
> > 0. If you have a T43(p) with comparable set up, what's your
> > number?
> I don't have a T43p. On a 2GHz Core2Duo (T7200), FSB533 IIRC, I get
> approximately 2200 MB/sec, kernel 2.6.25, x64-kernel with CONFIG_MCORE2.
>
> > 1. Does this mean my computer's bus speed is suddenly cut in
> > half, as seen by linux? What's is the performance implication?
> > 2.What exactly does hdparm -T measure(processor bus speed?)?  
> > Is it a good gauge for performance? If it is not a good guage for
> > performance, what is?
> > 3.What cause this dramatic reduction and performance penalty?  
> > Kernel? Fglrx?
> Collective answer to the three questions: hdparm -T measures, how fast
> the data is copied from the disk cache (in memory) to application
> memory, so it is a "simple" memory-to-memory copy. I wrote simple into
> quotes, because one has to be careful to use the first/second level
> caches of the processor in an optimal way according to the use case. For
> example, in the disk cache copying case, it makes no sense to put the
> disk cache data into processor caches, as it will not be used that quick
> again, whereas the destination buffer would be quite useful in the
> cache, as most applications access the data read by the read() system
> call (thats what hdparm measures; I don't know whether hdparm touches
> the data).
>
> You are really seeing a drop of memory-to-memory-copying performance to
> one half. The probable cause is, that the optimal cache management is
> buried into SSE/SSE2/SSE3/SSE4 instructions, and for really
> memory-intensive task, each Intel processor prefers a slight different
> way to get optimal performance. Your vanilla kernel was probably
> optimized for the Pentium M architecture, whereas the ubunty kernel is
> optimized for another architecture. Compile your own kernel to get up to
> the old performance again.
>
> The performance implications in day-to-day use are quite low (as long as
> your day-to-day use is not copying gigagbytes of data or video cutting),
> so you won't notice the difference. Memory-to-memory copies are avoided
> if possible, for example by doing a read-only mmap of all code. That
> means that the processor is set up in a way that applications can not
> overwrite their own code (they get a segfault instead), and thus it is
> safe to directly make the disk cache visible to the application instead
> of making a copy before. The same holds for swap-in/swap-out: The hard
> disk controller directly accesses the data in the memory range that is
> visible to the application, and thus no memory-to-memory copy is needed.
>
> Memory-to-memory copies within applications (or mem-to-video-mem in the
> X-Server) is not using the kernel functions, and thus independent of
> kernel compilation options (but an aptly optimized libc might help
> there).

Thanks so much for this really detailed explanation! Though I don't
understand everything you said, it really helps!

I guess from what I understand, I can recover the old performance by
recompiling the kernel myself. Now I used to roll my own vanilla kernel
because there are patches I need to apply to(undervolting, atop per
process accounting etc), but now I'm too busy in doing "real work" to
mess with my kernel too often:)
If, as I understand from what you said, the performance gain from day
to day computing is small, I would rather forgo this recompilation for
now, let along recompilation of libc. As a bumper sticker in  Silicon
Valley allegedly said, "life is too short to compile Gentoo"!(No
offense to gentoo users).
>
> > Thanks in advance for enlightenment!
> I hope the explanations cause enlightenment.
It sure does!
Regards
Jiang
>
> Regards,
>   Michael Karcher


--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Tino Keitel-3]

On Wed, Jun 25, 2008 at 06:52:53 -0400, jqian@... wrote:

[...]

> 2.What exactly does hdparm -T measure(processor bus speed?)?  
> Is it a good gauge for performance? If it is not a good guage for
> performance, what is?

It reads from the kernel cache memeory, so no transfers to/from the
hard disk or over the IDE/SATA bus take place. It's more a number to
indicate the memory throughput of your hardware/kernel combo.

Regards,
Tino
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Henrique de Moraes Holschuh-2]

On Wed, 25 Jun 2008, jqian@... wrote:
> > If the regression in hdparm -T speed is indeed related to ZEROPAGE, that
> > means hdparm -T is a completely meaningless test.
> I'm sorry to be a bit dense here, but if 2.6.25.9 fix a "performance
> regression", then what hdparm -T detects is a real regression, not a
> meaningless test, right?

Sort of :-)  It is a regression on pages filled with zeros, you usually
don't have to *read* a lot from such pages, hmm?

The big part of the regression was in memory wastage, not speed to access
the data... but it might have made read access to them slower too (due to
cache).

> > Your real workload is the only gauge for performance that matters.
> It's hard for me to judge performances when I had distro change, is
> there some benchmark that more sensibly reflect the bus speed of my
> computer?

bonie++ for filesystems, and a few other benchmarking stuff for certain
styles of workloads (such as hackbench, etc).  But really, if you can't feel
or measure that the computer is doing what waitn it to do to be any slower,
what is the point?

--
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Henrique de Moraes Holschuh-2]

On Wed, 25 Jun 2008, Henrique de Moraes Holschuh wrote:

> On Wed, 25 Jun 2008, jqian@... wrote:
> > > If the regression in hdparm -T speed is indeed related to ZEROPAGE, that
> > > means hdparm -T is a completely meaningless test.
> > I'm sorry to be a bit dense here, but if 2.6.25.9 fix a "performance
> > regression", then what hdparm -T detects is a real regression, not a
> > meaningless test, right?
>
> Sort of :-)  It is a regression on pages filled with zeros, you usually
> don't have to *read* a lot from such pages, hmm?
>
> The big part of the regression was in memory wastage, not speed to access
> the data... but it might have made read access to them slower too (due to
> cache).
>
> > > Your real workload is the only gauge for performance that matters.
> > It's hard for me to judge performances when I had distro change, is
> > there some benchmark that more sensibly reflect the bus speed of my
> > computer?
>
> bonie++ for filesystems, and a few other benchmarking stuff for certain
> styles of workloads (such as hackbench, etc).  But really, if you can't feel
> or measure that the computer is doing what waitn it to do to be any slower,
> what is the point?

I sincerely don't know what happened to that paragraph :)  Here's a
corrected version...

But really, if you cannot feel or measure that whatever you need the
computer do to is any slower, what is the point of bothering with
benchmarks?

--
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh
--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad

[Michael Karcher-2]

Am Mittwoch, den 25.06.2008, 14:05 -0400 schrieb
jqian@...:
[...]
> Thanks so much for this really detailed explanation! Though I don't
> understand everything you said, it really helps!
>
> I guess from what I understand, I can recover the old performance by
> recompiling the kernel myself.
I guess so. I didn't try myself, but it seems very probable.

> If, as I understand from what you said, the performance gain from day
> to day computing is small, I would rather forgo this recompilation for
> now,
You got it right. At best I would estimate you can get at best a 3%
performance gain in day-to-day-computing for tasks that do lot of big
read() calls. A rule of thumb is, that performance gains below 30% a
hardly noticable. So just let it as it is now.

>  let along recompilation of libc.
I never even wanted to recommend that to you, I just wanted to point out
that the kernel memory copy performance does even less to your system
performance than you might expect, as userspace programs use libc
instead.

As you run a 32 bit system, you might want to install libc6-i686 (if it
is not already installed) to obtain better performance than libc6 would
deliver.

>  As a bumper sticker in  Silicon
> Valley allegedly said, "life is too short to compile Gentoo"!(No
> offense to gentoo users).
Compiling gentoo is a hobby, IMHO, but not the right approach if you
want to use the computer for your work *now*.

Regards,
  Michael Karcher

[Stefan Monnier]

> It's hard for me to judge performances when I had distro change, is
> there some benchmark that more sensibly reflect the bus speed of my
> computer?

If you can't tell the difference then the difference is insignificant,
and there isn't much point is trying to measure it.


        Stefan

--
The linux-thinkpad mailing list home page is at:
http://mailman.linux-thinkpad.org/mailman/listinfo/linux-thinkpad