V7 High CPU Usage on swi5:+, what is this process?

[Aminuddin Abdullah]

I have just upgraded 5 of my machines to V7 from 6.3 and then realized that
all the machines has a high CPU usage. Almost all of them using 80%-90% CPU
with more than 8000 connections. Using previous 6.3, it only uses 40-50% CPU
with the same kind of connections.

Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU
time. What is this process?  

All the machines are Intel C2D 6300 except one which is a AMD 4000+.

Is this normal for V7? How do I downgrade to 6.3 if this V7 killing the CPU?

TIA

_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Ivan Voras-7]

Aminuddin Abdullah wrote:
> I have just upgraded 5 of my machines to V7 from 6.3 and then realized that
> all the machines has a high CPU usage. Almost all of them using 80%-90% CPU
> with more than 8000 connections. Using previous 6.3, it only uses 40-50% CPU
> with the same kind of connections.
>
> Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU
> time. What is this process?  

"swi" stands for "software interrupt", but to find out which one you'll
need to give more information about the systems. What are they running?
Any routing, firewall? Check with "vmstat -ia" to see if you have an
interrupt storm.

[rwatson-2]

On Mon, 17 Mar 2008, Aminuddin Abdullah wrote:

> I have just upgraded 5 of my machines to V7 from 6.3 and then realized that
> all the machines has a high CPU usage. Almost all of them using 80%-90% CPU
> with more than 8000 connections. Using previous 6.3, it only uses 40-50% CPU
> with the same kind of connections.
>
> Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU
> time. What is this process?
>
> All the machines are Intel C2D 6300 except one which is a AMD 4000+.
>
> Is this normal for V7? How do I downgrade to 6.3 if this V7 killing the CPU?

'+' is used in a swi name to indicate that the names of the interrupts to put
in the thread name are too long, and the code looks like it was written under
the assumption that at least one name would fit.  It sounds like in this case,
none fit.  We should fix this code, but in the mean time, what you might
consider doing is hacking intr_event_update() in kern_intr.c to print out
overflowing names to the console using printf(9) so you can at least see what
they are.  This is the somewhat suspect bit of code:

212         /*
213          * If the handler names were too long, add +'s to indicate missing
214          * names. If we run out of room and still have +'s to add, change
215          * the last character from a + to a *.
216          */
217         last = &ie->ie_fullname[sizeof(ie->ie_fullname) - 2];
218         while (missed-- > 0) {
219                 if (strlen(ie->ie_fullname) + 1 ==
sizeof(ie->ie_fullname)) {
220                         if (*last == '+') {
221                                 *last = '*';
222                                 break;
223                         } else
224                                 *last = '+';
225                 } else if (space) {
226                         strcat(ie->ie_fullname, " +");
227                         space = 0;
228                 } else
229                         strcat(ie->ie_fullname, "+");
230         }

I've CC'd John, who might have views on what we should do about this.  It
would be nice if we had a way to export information on all the interrupt event
sources, including soft ones, and their mappings to ithreads, including swis,
using sysctl.  Or maybe we do already and he'll point us at it. :-)

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[John Baldwin]

On Tuesday 18 March 2008 07:21:23 am Robert Watson wrote:

> On Mon, 17 Mar 2008, Aminuddin Abdullah wrote:
> > I have just upgraded 5 of my machines to V7 from 6.3 and then realized
> > that all the machines has a high CPU usage. Almost all of them using
> > 80%-90% CPU with more than 8000 connections. Using previous 6.3, it only
> > uses 40-50% CPU with the same kind of connections.
> >
> > Using top -S, I can see that swi5: +, PID 17 process is using 30% of CPU
> > time. What is this process?
> >
> > All the machines are Intel C2D 6300 except one which is a AMD 4000+.
> >
> > Is this normal for V7? How do I downgrade to 6.3 if this V7 killing the
> > CPU?
>
> '+' is used in a swi name to indicate that the names of the interrupts to
> put in the thread name are too long, and the code looks like it was written
> under the assumption that at least one name would fit.  It sounds like in
> this case, none fit.  We should fix this code, but in the mean time, what
> you might consider doing is hacking intr_event_update() in kern_intr.c to
> print out overflowing names to the console using printf(9) so you can at
> least see what they are.  This is the somewhat suspect bit of code:

The code is not suspect as p_comm is of fixed length.  Someone just used too
long of a name for a swi handler.

> I've CC'd John, who might have views on what we should do about this.  It
> would be nice if we had a way to export information on all the interrupt
> event sources, including soft ones, and their mappings to ithreads,
> including swis, using sysctl.  Or maybe we do already and he'll point us at
> it. :-)

We don't and that is what we need for a userland interrupt binding interface
to make sense.

--
John Baldwin
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[rwatson-2]

On Tue, 18 Mar 2008, John Baldwin wrote:

>> '+' is used in a swi name to indicate that the names of the interrupts to
>> put in the thread name are too long, and the code looks like it was written
>> under the assumption that at least one name would fit.  It sounds like in
>> this case, none fit.  We should fix this code, but in the mean time, what
>> you might consider doing is hacking intr_event_update() in kern_intr.c to
>> print out overflowing names to the console using printf(9) so you can at
>> least see what they are.  This is the somewhat suspect bit of code:
>
> The code is not suspect as p_comm is of fixed length.  Someone just used too
> long of a name for a swi handler.

I was wondering whether we might not do better to put as much in as we can but
truncate with a '*', so you at least get a fractional swi name.  Under what
situations do we use a single ithread for multiple swi's?

Robert N M Watson
Computer Laboratory
University of Cambridge
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[John Baldwin]

On Tuesday 18 March 2008 09:04:05 am Robert Watson wrote:

> On Tue, 18 Mar 2008, John Baldwin wrote:
> >> '+' is used in a swi name to indicate that the names of the interrupts
> >> to put in the thread name are too long, and the code looks like it was
> >> written under the assumption that at least one name would fit.  It
> >> sounds like in this case, none fit.  We should fix this code, but in the
> >> mean time, what you might consider doing is hacking intr_event_update()
> >> in kern_intr.c to print out overflowing names to the console using
> >> printf(9) so you can at least see what they are.  This is the somewhat
> >> suspect bit of code:
> >
> > The code is not suspect as p_comm is of fixed length.  Someone just used
> > too long of a name for a swi handler.
>
> I was wondering whether we might not do better to put as much in as we can
> but truncate with a '*', so you at least get a fractional swi name.  Under
> what situations do we use a single ithread for multiple swi's?

The softclock one gets overloaded with some tty handlers.  This code is also
just generic ithread code common to swi's and hardware interrupts.

--
John Baldwin
_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."

[Ivan Voras-7]

Robert Watson wrote:

> I've CC'd John, who might have views on what we should do about this.
> It would be nice if we had a way to export information on all the
> interrupt event sources, including soft ones, and their mappings to
> ithreads, including swis, using sysctl.  Or maybe we do already and
> he'll point us at it. :-)

How about, for starts, the truncating loop gets overriden / replaced for
ithread process names?

_______________________________________________
freebsd-performance@... mailing list
http://lists.freebsd.org/mailman/listinfo/freebsd-performance
To unsubscribe, send any mail to "freebsd-performance-unsubscribe@..."