EM and TSO

[Jack Vogel]

I introduced a change yesterday that limited TSO to PCI Express
adapters, I did this more for avoidance rather than a bug fix, and
I'm not 100% sure its the right thing, so I thought I would poll
everyone, do you have a PCI-X adapter and are using TSO without
problems and wish to keep the support in?

If no one is then I'll just leave it as is.

Jack
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Andre Oppermann-2]

Jack Vogel wrote:
> I introduced a change yesterday that limited TSO to PCI Express
> adapters, I did this more for avoidance rather than a bug fix, and
> I'm not 100% sure its the right thing, so I thought I would poll
> everyone, do you have a PCI-X adapter and are using TSO without
> problems and wish to keep the support in?

I'm using the dual-port PCI-X adapter and would like to keep
using TSO on it.  There are a lot of those adapters out there
in servers.  TSO is scheduled to be MFC'd really soon now and
not having it work with em(4) on PCI-X would be quite limiting.

--
Andre

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Jack Vogel]

On 5/16/07, Andre Oppermann <andre@...> wrote:

> Jack Vogel wrote:
> > I introduced a change yesterday that limited TSO to PCI Express
> > adapters, I did this more for avoidance rather than a bug fix, and
> > I'm not 100% sure its the right thing, so I thought I would poll
> > everyone, do you have a PCI-X adapter and are using TSO without
> > problems and wish to keep the support in?
>
> I'm using the dual-port PCI-X adapter and would like to keep
> using TSO on it.  There are a lot of those adapters out there
> in servers.  TSO is scheduled to be MFC'd really soon now and
> not having it work with em(4) on PCI-X would be quite limiting.

OK, I'll put it back, you also answered a question I had, namely
the MFC of TSO :) I was hoping that got into 6.3.

I have another bug I'm working on, will be a delta today or tomorrow
and I'll put it back to what it was.

Jack
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Tom Judge]

Jack Vogel wrote:
> I introduced a change yesterday that limited TSO to PCI Express
> adapters, I did this more for avoidance rather than a bug fix, and
> I'm not 100% sure its the right thing, so I thought I would poll
> everyone, do you have a PCI-X adapter and are using TSO without
> problems and wish to keep the support in?
>
> If no one is then I'll just leave it as is.
>
> Jack

Remember to CC the lists this time.

I have a number of systems with built in em adapters (on the
motherboard)  is there any easy way to find out what type of bus these
are connected to?

Tom



_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Thomas Hurst]

* Jack Vogel (jfvogel@...) wrote:

>  I introduced a change yesterday that limited TSO to PCI Express
>  adapters, I did this more for avoidance rather than a bug fix, and
>  I'm not 100% sure its the right thing, so I thought I would poll
>  everyone, do you have a PCI-X adapter and are using TSO without
>  problems and wish to keep the support in?
>
>  If no one is then I'll just leave it as is.

We put CURRENT on one of our Sun X4200's for some MySQL tests a few days
ago and haven't noticed any problems, though it's only handled a few
million packets so far.

Does this change remove support or just disable it by default?

--
Thomas 'Freaky' Hurst
    http://hur.st/
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Jack Vogel]

On 5/16/07, Thomas Hurst <tom.hurst@...> wrote:

> * Jack Vogel (jfvogel@...) wrote:
>
> >  I introduced a change yesterday that limited TSO to PCI Express
> >  adapters, I did this more for avoidance rather than a bug fix, and
> >  I'm not 100% sure its the right thing, so I thought I would poll
> >  everyone, do you have a PCI-X adapter and are using TSO without
> >  problems and wish to keep the support in?
> >
> >  If no one is then I'll just leave it as is.
>
> We put CURRENT on one of our Sun X4200's for some MySQL tests a few days
> ago and haven't noticed any problems, though it's only handled a few
> million packets so far.
>
> Does this change remove support or just disable it by default?

It removed it, but Andre and my own guilty conscience has convinced
me that I should change it back, once I get an MSI/X bug that was
discovered with the code solved there will be another delta that will
reenable it for PCI-X again.

Jack
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Julian Elischer]

Jack Vogel wrote:

> On 5/16/07, Andre Oppermann <andre@...> wrote:
>> Jack Vogel wrote:
>> > I introduced a change yesterday that limited TSO to PCI Express
>> > adapters, I did this more for avoidance rather than a bug fix, and
>> > I'm not 100% sure its the right thing, so I thought I would poll
>> > everyone, do you have a PCI-X adapter and are using TSO without
>> > problems and wish to keep the support in?
>>
>> I'm using the dual-port PCI-X adapter and would like to keep
>> using TSO on it.  There are a lot of those adapters out there
>> in servers.  TSO is scheduled to be MFC'd really soon now and
>> not having it work with em(4) on PCI-X would be quite limiting.
>
> OK, I'll put it back, you also answered a question I had, namely
> the MFC of TSO :) I was hoping that got into 6.3.
>
> I have another bug I'm working on, will be a delta today or tomorrow
> and I'll put it back to what it was.
>
> Jack
> _______________________________________________
> freebsd-net@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-net
> To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

give a sysctl to change the option.

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Jack Vogel]

You mean other than net.inet.tcp.tso that is there now?

On 5/16/07, Julian Elischer <julian@...> wrote:

> Jack Vogel wrote:
> > On 5/16/07, Andre Oppermann <andre@...> wrote:
> >> Jack Vogel wrote:
> >> > I introduced a change yesterday that limited TSO to PCI Express
> >> > adapters, I did this more for avoidance rather than a bug fix, and
> >> > I'm not 100% sure its the right thing, so I thought I would poll
> >> > everyone, do you have a PCI-X adapter and are using TSO without
> >> > problems and wish to keep the support in?
> >>
> >> I'm using the dual-port PCI-X adapter and would like to keep
> >> using TSO on it.  There are a lot of those adapters out there
> >> in servers.  TSO is scheduled to be MFC'd really soon now and
> >> not having it work with em(4) on PCI-X would be quite limiting.
> >
> > OK, I'll put it back, you also answered a question I had, namely
> > the MFC of TSO :) I was hoping that got into 6.3.
> >
> > I have another bug I'm working on, will be a delta today or tomorrow
> > and I'll put it back to what it was.
> >
> > Jack
> > _______________________________________________
> > freebsd-net@... mailing list
> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."
>
> give a sysctl to change the option.
>
>

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Sam Leffler]

Jack Vogel wrote:
> I introduced a change yesterday that limited TSO to PCI Express
> adapters, I did this more for avoidance rather than a bug fix, and
> I'm not 100% sure its the right thing, so I thought I would poll
> everyone, do you have a PCI-X adapter and are using TSO without
> problems and wish to keep the support in?
>
> If no one is then I'll just leave it as is.

It might be better to enable it by default on pci-e adapters and require
manual enable on other adapters that are capable but may not function
correctly.

        Sam

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Andre Oppermann-2]

Jack Vogel wrote:
> You mean other than net.inet.tcp.tso that is there now?

I don't think another sysctl is necessary for this.

--
Andre

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Sam Leffler]

Jack Vogel wrote:

> You mean other than net.inet.tcp.tso that is there now?
>
> On 5/16/07, Julian Elischer <julian@...> wrote:
>> Jack Vogel wrote:
>> > On 5/16/07, Andre Oppermann <andre@...> wrote:
>> >> Jack Vogel wrote:
>> >> > I introduced a change yesterday that limited TSO to PCI Express
>> >> > adapters, I did this more for avoidance rather than a bug fix, and
>> >> > I'm not 100% sure its the right thing, so I thought I would poll
>> >> > everyone, do you have a PCI-X adapter and are using TSO without
>> >> > problems and wish to keep the support in?
>> >>
>> >> I'm using the dual-port PCI-X adapter and would like to keep
>> >> using TSO on it.  There are a lot of those adapters out there
>> >> in servers.  TSO is scheduled to be MFC'd really soon now and
>> >> not having it work with em(4) on PCI-X would be quite limiting.
>> >
>> > OK, I'll put it back, you also answered a question I had, namely
>> > the MFC of TSO :) I was hoping that got into 6.3.
>> >
>> > I have another bug I'm working on, will be a delta today or tomorrow
>> > and I'll put it back to what it was.
>> >
>> > Jack
>> > _______________________________________________
>> > freebsd-net@... mailing list
>> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
>> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."
>>
>> give a sysctl to change the option.

ifconfig can be used to ena/dis tso.  As I said before just give all the
devices capable of tso the capability but only enable it by default on
those devices you really know work (e.g. pci-e cards).  On other devices
users that want to turn it on can do so with ifconfig.

        Sam
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Jack Vogel]

On 5/16/07, Sam Leffler <sam@...> wrote:

> Jack Vogel wrote:
> > You mean other than net.inet.tcp.tso that is there now?
> >
> > On 5/16/07, Julian Elischer <julian@...> wrote:
> >> Jack Vogel wrote:
> >> > On 5/16/07, Andre Oppermann <andre@...> wrote:
> >> >> Jack Vogel wrote:
> >> >> > I introduced a change yesterday that limited TSO to PCI Express
> >> >> > adapters, I did this more for avoidance rather than a bug fix, and
> >> >> > I'm not 100% sure its the right thing, so I thought I would poll
> >> >> > everyone, do you have a PCI-X adapter and are using TSO without
> >> >> > problems and wish to keep the support in?
> >> >>
> >> >> I'm using the dual-port PCI-X adapter and would like to keep
> >> >> using TSO on it.  There are a lot of those adapters out there
> >> >> in servers.  TSO is scheduled to be MFC'd really soon now and
> >> >> not having it work with em(4) on PCI-X would be quite limiting.
> >> >
> >> > OK, I'll put it back, you also answered a question I had, namely
> >> > the MFC of TSO :) I was hoping that got into 6.3.
> >> >
> >> > I have another bug I'm working on, will be a delta today or tomorrow
> >> > and I'll put it back to what it was.
> >> >
> >> > Jack
> >> > _______________________________________________
> >> > freebsd-net@... mailing list
> >> > http://lists.freebsd.org/mailman/listinfo/freebsd-net
> >> > To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."
> >>
> >> give a sysctl to change the option.
>
> ifconfig can be used to ena/dis tso.  As I said before just give all the
> devices capable of tso the capability but only enable it by default on
> those devices you really know work (e.g. pci-e cards).  On other devices
> users that want to turn it on can do so with ifconfig.

I like this, good compromise. Thanks Sam :)

Should be coming into tree shortly.

Jack
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Danny Braniss]

> On 5/16/07, Thomas Hurst <tom.hurst@...> wrote:
> > * Jack Vogel (jfvogel@...) wrote:
> >
> > >  I introduced a change yesterday that limited TSO to PCI Express
> > >  adapters, I did this more for avoidance rather than a bug fix, and
> > >  I'm not 100% sure its the right thing, so I thought I would poll
> > >  everyone, do you have a PCI-X adapter and are using TSO without
> > >  problems and wish to keep the support in?
> > >
> > >  If no one is then I'll just leave it as is.
> >
> > We put CURRENT on one of our Sun X4200's for some MySQL tests a few days
> > ago and haven't noticed any problems, though it's only handled a few
> > million packets so far.
> >
> > Does this change remove support or just disable it by default?
>
> It removed it, but Andre and my own guilty conscience has convinced
> me that I should change it back, once I get an MSI/X bug that was
> discovered with the code solved there will be another delta that will
> reenable it for PCI-X again.

em0@pci1:1:0:   class=0x020000 card=0x10118086 chip=0x10108086 rev=0x03
hdr=0x00
    vendor     = 'Intel Corporation'
    device     = '82546EB Dual Port Gigabit Ethernet Controller (Copper)'
    class      = network
    subclass   = ethernet
    cap 01[dc] = powerspec 2  supports D0 D3  current D0
    cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split
transaction
    cap 05[f0] = MSI supports 1 message, 64 bit

I have been testing TSO on a SunFire X4100, and though it works, I notice
that setting it on actually reduces throughput, from an average of 55MB/s
to 24MB/s

any explanation?

danny


_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Jack Vogel]

On 5/17/07, Danny Braniss <danny@...> wrote:

> > On 5/16/07, Thomas Hurst <tom.hurst@...> wrote:
> > > * Jack Vogel (jfvogel@...) wrote:
> > >
> > > >  I introduced a change yesterday that limited TSO to PCI Express
> > > >  adapters, I did this more for avoidance rather than a bug fix, and
> > > >  I'm not 100% sure its the right thing, so I thought I would poll
> > > >  everyone, do you have a PCI-X adapter and are using TSO without
> > > >  problems and wish to keep the support in?
> > > >
> > > >  If no one is then I'll just leave it as is.
> > >
> > > We put CURRENT on one of our Sun X4200's for some MySQL tests a few days
> > > ago and haven't noticed any problems, though it's only handled a few
> > > million packets so far.
> > >
> > > Does this change remove support or just disable it by default?
> >
> > It removed it, but Andre and my own guilty conscience has convinced
> > me that I should change it back, once I get an MSI/X bug that was
> > discovered with the code solved there will be another delta that will
> > reenable it for PCI-X again.
>
> em0@pci1:1:0:   class=0x020000 card=0x10118086 chip=0x10108086 rev=0x03
> hdr=0x00
>     vendor     = 'Intel Corporation'
>     device     = '82546EB Dual Port Gigabit Ethernet Controller (Copper)'
>     class      = network
>     subclass   = ethernet
>     cap 01[dc] = powerspec 2  supports D0 D3  current D0
>     cap 07[e4] = PCI-X 64-bit supports 133MHz, 2048 burst read, 1 split
> transaction
>     cap 05[f0] = MSI supports 1 message, 64 bit
>
> I have been testing TSO on a SunFire X4100, and though it works, I notice
> that setting it on actually reduces throughput, from an average of 55MB/s
> to 24MB/s
>
> any explanation?

TSO is workload sensitive and its not always the right answer.
The whole realm of performance tweaking has been outside my
scope right now, but it needs to be done at some point.

Most the testing I did while developing the code was on PCI-E,
and on say an 82571 I saw no significant reduction of throughput,
with a noticeable improvement in CPU. The adapter you are testing
on is one that I just disabled by default because I dont think it is
capable of showing TSO to its best advantage.

The has been no real work on tweaking this stuff to understand its
best usage yet, it would be a great task for someone to take on :)

Jack
_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."

[Steven Hartland]

Even on PCI-e devices this has some nasty side effects i.e. on a PFSence
firewall box based on 7.0-RELEASE-p3 with TSO enabled, access via the
public network to the web interface is almost impossible. Simply disabling
TSO and all was good.

This may not be strictly down to the HW or driver but is another
reason to not enable it by default if it causes such poor performance.

    Regards
    Steve

----- Original Message -----
From: "Sam Leffler" <sam@...>
To: "Jack Vogel" <jfvogel@...>
Cc: "freebsd-net" <freebsd-net@...>; "FreeBSD Current" <freebsd-current@...>
Sent: Wednesday, May 16, 2007 10:00 PM
Subject: Re: EM and TSO

> Jack Vogel wrote:
>> I introduced a change yesterday that limited TSO to PCI Express
>> adapters, I did this more for avoidance rather than a bug fix, and
>> I'm not 100% sure its the right thing, so I thought I would poll
>> everyone, do you have a PCI-X adapter and are using TSO without
>> problems and wish to keep the support in?
>>
>> If no one is then I'll just leave it as is.
>
> It might be better to enable it by default on pci-e adapters and require
> manual enable on other adapters that are capable but may not function
> correctly.
>
> Sam
>
> _______________________________________________
> freebsd-current@... mailing list
> http://lists.freebsd.org/mailman/listinfo/freebsd-current
> To unsubscribe, send any mail to "freebsd-current-unsubscribe@..."
>

================================================
This e.mail is private and confidential between Multiplay (UK) Ltd. and the person or entity to whom it is addressed. In the event of misdirection, the recipient is prohibited from using, copying, printing or otherwise disseminating it or any information contained in it.

In the event of misdirection, illegible or incomplete transmission please telephone [phone]
or return the E.mail to postmaster@....

_______________________________________________
freebsd-net@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-net
To unsubscribe, send any mail to "freebsd-net-unsubscribe@..."