The perl 5.10.0 headache

After talking it over with stephen and arranging perl 5.10 to be built on real metal vs aranym, we are pleased to report that perl passes its test suite with the exception of one test, a threading stress test which launchs multiple threads, and then tries to make sure that they close in order.

This test is broken on both Perl 5.8, and 5.10, although it was only included in the later's base package by default. It appears its caused by the antique version of glibc we use, and the fact that we are using the old linuxthreads package vs. nptl. A lot of packages are currently broken because Perl 5.10 isn't available, so I personally think, since its the same situation with 5.8 that we force perl through (by applying a patch to skip this test, or make it a non-fatal fail).

What are your opinions on it?
Michael

[Stephen R Marenka]

On Mon, Jul 07, 2008 at 09:56:19AM -0400, Michael Casadevall wrote:

> After talking it over with stephen and arranging perl 5.10 to be built on
> real metal vs aranym, we are pleased to report that perl passes its test
> suite with the exception of one test, a threading stress test which launchs
> multiple threads, and then tries to make sure that they close in order.
>
> This test is broken on both Perl 5.8, and 5.10, although it was only
> included in the later's base package by default. It appears its caused by
> the antique version of glibc we use, and the fact that we are using the old
> linuxthreads package vs. nptl. A lot of packages are currently broken
> because Perl 5.10 isn't available, so I personally think, since its the same
> situation with 5.8 that we force perl through (by applying a patch to skip
> this test, or make it a non-fatal fail).
>
> What are your opinions on it?

On a perhaps related note, we've had a number of perl threads related
self-test failures (libthreads-shared-perl comes to mind).

If threads are broken, should we instead have a non-threaded perl? (and
what will *that* break.)

:)

Peace,

Stephen

--
Stephen R. Marenka     If life's not fun, you're not doing it right!
<stephen@...>

[Petr Salinger-3]

Hi from GNU/kFreeBSD porter.

> After talking it over with stephen and arranging perl 5.10 to be built on
> real metal vs aranym, we are pleased to report that perl passes its test
> suite with the exception of one test, a threading stress test which
> launchs multiple threads, and then tries to make sure that they close in
> order.

> It appears its caused by the antique version of glibc we use, and the
> fact that we are using the old linuxthreads package vs. nptl.

The test passes even under linuxthreads, but real-time variant of
signals have to be used, i.e. they should properly queue in kernel.
I know it, because on GNU/kFreeBSD it needs 7.x kernel.

You can use attached code to perform the same test in C.
Problem would be rather missing memory barier, or non-atomic operation.
This should be fixable even with 2.5 glibc and linuxthreads.

Petr
[stress.c]

---

#include <sys/time.h>
#include <time.h>
#include <errno.h>
#include <pthread.h>
#include <stdlib.h>


int x,y;
pthread_mutex_t mut = PTHREAD_MUTEX_INITIALIZER;
pthread_cond_t cond = PTHREAD_COND_INITIALIZER;

void * job(void * arg)
{
        int a = (int) arg;

        struct timeval now;
        struct timespec timeout;
       
        int retcode;
        printf("Started %d\n", a);
        usleep((rand() % 32777));

        pthread_mutex_lock(&mut);
        while (x != a)
        {
        printf("wait %d %d\n", x, a);
                gettimeofday(&now,NULL);
                timeout.tv_sec = now.tv_sec + 1;
                timeout.tv_nsec = now.tv_usec * 1000;
        retcode = pthread_cond_timedwait(&cond, &mut, &timeout);
                if (retcode == ETIMEDOUT)
                        printf("Timeout %d\n", a);
        }
       
        x++;
        printf("exit a %d %d\n", x, a);
        pthread_cond_broadcast(&cond);
        printf("exit b %d %d\n", x, a);
        pthread_mutex_unlock(&mut);
        printf("exit c %d %d\n", x, a);
}

int main()
{
        pthread_t t;
        int i;
        for (i = 0; i < 50; i++)
                pthread_create(&t, NULL, job, (void *) i);
        sleep (10);
        printf("FINAL %d %d\n", x, i);
       
}

---

[Michael Schmitz-6]

>> After talking it over with stephen and arranging perl 5.10 to be built on
>> real metal vs aranym, we are pleased to report that perl passes its test
>> suite with the exception of one test, a threading stress test which launchs
>> multiple threads, and then tries to make sure that they close in order.
>
>> It appears its caused by the antique version of glibc we use, and the fact
>> that we are using the old linuxthreads package vs. nptl.
>
> The test passes even under linuxthreads, but real-time variant of
> signals have to be used, i.e. they should properly queue in kernel.
> I know it, because on GNU/kFreeBSD it needs 7.x kernel.
>
> You can use attached code to perform the same test in C.
> Problem would be rather missing memory barier, or non-atomic operation.
> This should be fixable even with 2.5 glibc and linuxthreads.

Do we need a fix in linuxthreads, or the kernel?

  Michael


--
To UNSUBSCRIBE, email to debian-68k-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Petr Salinger-3]

>> Problem would be rather missing memory barier, or non-atomic operation.
>> This should be fixable even with 2.5 glibc and linuxthreads.

> Do we need a fix in linuxthreads, or the kernel?

I don't know.

The linuxthreads add-on is currently used
by hppa and kfreebsd-* for glibc 2.7.
But the latest entry in linuxthreads ChangeLog is dated 2006-10-31.
So codebase might be even the same as with your 2.5-11 glibc.

On GNU/kFreeBSD, the test fails with 6.x (FreeBSD) kernel, it passes with
7.x kernel (after I added runtime detection of presence of rt
capable FreeBSD kernel). It is due to realtime behaviour of signals, in
7.x they properly queue in kernel. I assume 2.6 linux kernel (and may
be also 2.4) even on 68k queues them properly.

IMHO, the problem is not in pthread add-on, but lies somewhere else,
i.e. non-atomic operation, wrong optimization by gcc, ...

I suggets to use attached C code and try strace and so on.

Petr


--
To UNSUBSCRIBE, email to debian-68k-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

I haven't had an opportunity to run the test yet (my laptop currently grinding away building the new release of GCC for m68k) but threads on Linux when using linuxthreads to my knowledge are almost completely in userland; linuxthreads uses clone() to create them. If it requires real-time signals on kFreeBSD, then won't it in theory require a realtime Linux kernel to get the same effect?
Michael

On Mon, Jul 7, 2008 at 2:46 PM, Petr Salinger <Petr.Salinger@...> wrote:

Hi from GNU/kFreeBSD porter.

After talking it over with stephen and arranging perl 5.10 to be built on
real metal vs aranym, we are pleased to report that perl passes its test
suite with the exception of one test, a threading stress test which launchs multiple threads, and then tries to make sure that they close in order.

It appears its caused by the antique version of glibc we use, and the fact that we are using the old linuxthreads package vs. nptl.

The test passes even under linuxthreads, but real-time variant of
signals have to be used, i.e. they should properly queue in kernel.
I know it, because on GNU/kFreeBSD it needs 7.x kernel.

You can use attached code to perform the same test in C.
Problem would be rather missing memory barier, or non-atomic operation.
This should be fixable even with 2.5 glibc and linuxthreads.

Petr

[Petr Salinger-3]

> I haven't had an opportunity to run the test yet (my laptop currently
> grinding away building the new release of GCC for m68k) but threads on Linux
> when using linuxthreads to my knowledge are almost completely in userland;
> linuxthreads uses clone() to create them. If it requires real-time signals
> on kFreeBSD, then won't it in theory require a realtime Linux kernel to get
> the same effect?

With non-rt behaviour of signals is possible to "drop" sent signals.
Pending signals are in kernel represented by bitmask, i.e. at most one
signal with given number is pending. The rt behaviour of signals uses
queue in kernel, i.e. there is no lost of pending signals.
The linuxthreads detect operating mode of signals. Under non-rt
behaviour it tries to count sent and received signals in userspace,
but there is a race condition, despite claims in lt code.

See pthread_restart, pthread_suspend, pthread_timedsuspend variants
in linuxthreads/pthread.c (comments claims rt variant is in linux kernel
since 2.2 series, soo FreeBSd have been really behind).

The wrong optimisation can lead to similar failures even with
signals queued by kernel.

You can test whether attached C code works in sarge and etch.

Petr


--
To UNSUBSCRIBE, email to debian-68k-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Michael Schmitz-4]

Hi,

>> After talking it over with stephen and arranging perl 5.10 to be built on
>> real metal vs aranym, we are pleased to report that perl passes its test
>> suite with the exception of one test, a threading stress test which launchs
>> multiple threads, and then tries to make sure that they close in order.
>
>> It appears its caused by the antique version of glibc we use, and the fact
>> that we are using the old linuxthreads package vs. nptl.
>
> The test passes even under linuxthreads, but real-time variant of
> signals have to be used, i.e. they should properly queue in kernel.
> I know it, because on GNU/kFreeBSD it needs 7.x kernel.
>
> You can use attached code to perform the same test in C.
> Problem would be rather missing memory barier, or non-atomic operation.
> This should be fixable even with 2.5 glibc and linuxthreads.

I've tested your code on 2.6.26-rc8 and I do see the exit c message happen
out of sequence quite a bit there (usually x = a+1 holds). Is that
supposed to happen? exit c is after the mutex unlock so it is expected to
not serialize properly if I understand the code right.

No timeout, no mismatch at the FINAL message though. Fairly old gcc
(3.3.5-3) and libc (2.3.2.ds1-22) on that system - it's my kernel hacking
disk only. The system is otherwise idle.

  Michael


--
To UNSUBSCRIBE, email to debian-68k-REQUEST@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...