[1.37.0] Compilers for release testing?

[Beman Dawes]

For 1.36.0, our release test primary compilers were:

    * GCC 4.01 on Mac OS X 10.4.10 with both Intel and Power PC
    * GCC 4.2.3 on Ubuntu Linux 8.04.1
    * HP C/aC++ B3910B A.06.17 on HP-UX 64-bit
    * Visual C++ 9.0 SP1 beta, 8.0 SP1, and 7.1, all on Windows XP SP-2

Could we have volunteers to run the Ubuntu Linux/GCC tests and the
Windows/msvc-8.0 tests? If someone could take those off my hands, it
would free resources to add additional compilers.
 
If I don't have to run any tests, it will free resources enough to
expand the list by two or three compilers for 1.37.0. The idea is to
provide coverage for popular compiler/operating system combinations not
currently being tested. Anyone care to volunteer?

I'd prefer volunteers who have experience testing on trunk, and been
able to keep their test setup running reliably.  They also need to have
a bit of time to monitor the Boost testing list.

No more than two tests per tester, to limit the inconvenience if a
tester goes down.

Thanks,

--Beman


_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[David Abrahams-3]

on Sun Aug 24 2008, Beman Dawes <bdawes-AT-acm.org> wrote:

> For 1.36.0, our release test primary compilers were:
>
>    * GCC 4.01 on Mac OS X 10.4.10 with both Intel and Power PC
>    * GCC 4.2.3 on Ubuntu Linux 8.04.1
>    * HP C/aC++ B3910B A.06.17 on HP-UX 64-bit
>    * Visual C++ 9.0 SP1 beta, 8.0 SP1, and 7.1, all on Windows XP SP-2

Just a terminology nit: IMO these should be called "test platforms," not
"test compilers," and they should include the machine's ISA.  Testing on
Darwin PPC is different from testing on Darwin x86 is different from
testing on Darwin amd64, etc.  That kind of variation applies to all the
OSes we test on, IIRC.

> Could we have volunteers to run the Ubuntu Linux/GCC tests and the
> Windows/msvc-8.0 tests? If someone could take those off my hands, it
> would free resources to add additional compilers.

I could, but I really want to dedicate my testing resources to a system
using the cmake toolchain, but IIUC the regular testing process is not
doing that yet.

> If I don't have to run any tests, it will free resources enough to
> expand the list by two or three compilers for 1.37.0. The idea is to
> provide coverage for popular compiler/operating system combinations
> not currently being tested. Anyone care to volunteer?
>
> I'd prefer volunteers who have experience testing on trunk, and been
> able to keep their test setup running reliably.  They also need to
> have a bit of time to monitor the Boost testing list.
>
> No more than two tests per tester, to limit the inconvenience if a
> tester goes down.

Do you mean two platforms per tester? ;-)

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Beman Dawes]

David Abrahams wrote:

> on Sun Aug 24 2008, Beman Dawes <bdawes-AT-acm.org> wrote:
>
>> For 1.36.0, our release test primary compilers were:
>>
>>    * GCC 4.01 on Mac OS X 10.4.10 with both Intel and Power PC
>>    * GCC 4.2.3 on Ubuntu Linux 8.04.1
>>    * HP C/aC++ B3910B A.06.17 on HP-UX 64-bit
>>    * Visual C++ 9.0 SP1 beta, 8.0 SP1, and 7.1, all on Windows XP SP-2
>
> Just a terminology nit: IMO these should be called "test platforms," not
> "test compilers," and they should include the machine's ISA.  Testing on
> Darwin PPC is different from testing on Darwin x86 is different from
> testing on Darwin amd64, etc.  That kind of variation applies to all the
> OSes we test on, IIRC.

Yes, agreed.

>
>> Could we have volunteers to run the Ubuntu Linux/GCC tests and the
>> Windows/msvc-8.0 tests? If someone could take those off my hands, it
>> would free resources to add additional compilers.
>
> I could, but I really want to dedicate my testing resources to a system
> using the cmake toolchain, but IIUC the regular testing process is not
> doing that yet.

IIUC, CMake builds and tests are working, but reporting hasn't gotten
attention yet and I'm not sure that anyone is even working on reporting.

Dave, I'd really like to see you apply your skills to helping get a
reporting system going for CMake based testing. We've got plenty of
volunteers who can do a great job with bjam testing, but no so many with
your strong sense of the needs for test reporting.

>> If I don't have to run any tests, it will free resources enough to
>> expand the list by two or three compilers for 1.37.0. The idea is to
>> provide coverage for popular compiler/operating system combinations
>> not currently being tested. Anyone care to volunteer?
>>
>> I'd prefer volunteers who have experience testing on trunk, and been
>> able to keep their test setup running reliably.  They also need to
>> have a bit of time to monitor the Boost testing list.
>>
>> No more than two tests per tester, to limit the inconvenience if a
>> tester goes down.
>
> Do you mean two platforms per tester? ;-)

Right:-)

--Beman

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[David Abrahams-3]

on Mon Aug 25 2008, Beman Dawes <bdawes-AT-acm.org> wrote:

> Dave, I'd really like to see you apply your skills to helping get a
> reporting system going for CMake based testing. We've got plenty of
> volunteers who can do a great job with bjam testing, but no so many
> with your strong sense of the needs for test reporting.

I'd be happy to try.  I sorta thought Troy's friend Evan Wheeler was
working on that?  He's apparently got much better web-UI chops than I
ever will (Ajax, etc.) but I'd be more than happy to collaborate.

Hmm, http://boost.resophonic.com/trac/traash?builds=all doesn't look too
encouraging right now.  What's going on, Troy?


--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[troy d. straszheim]

David Abrahams wrote:

> on Mon Aug 25 2008, Beman Dawes <bdawes-AT-acm.org> wrote:
>
>> Dave, I'd really like to see you apply your skills to helping get a
>> reporting system going for CMake based testing. We've got plenty of
>> volunteers who can do a great job with bjam testing, but no so many
>> with your strong sense of the needs for test reporting.
>
> I'd be happy to try.  I sorta thought Troy's friend Evan Wheeler was
> working on that?  He's apparently got much better web-UI chops than I
> ever will (Ajax, etc.) but I'd be more than happy to collaborate.

He got buried in day-job deadlines, as have I.  We're in a release cycle,
and I'm going to have to turn back to this stuff in the next week or so,
but at the moment I can't do any better than this mail.

> Hmm, http://boost.resophonic.com/trac/traash?builds=all doesn't look too
> encouraging right now.  What's going on, Troy?

I stopped running builds, since the reaction to it was so superficial and
lukewarm.  For instance, all of the uses cases put forward by Beman
(and other important ones not put forward) were reachable
in ~4 clicks from the front page..  well addressed by the interface, I thought.
I was hoping to get back round to this and make another effort to explain
things, but I'm willing to forget it as well.

The code is here:

    http://svn.resophonic.com/svn/trac-dev

And I'm sure it won't run right out of the box.  There is a lot of site-specific
hackery in there.  I wouldn't normally show code to anybody in that condition,
but if you are anxious to start hacking, there you go.

There are also some changes that really need to be made on the boost-cmake side, which
I can do when my orbit comes back around: the one-cmake-target-per-test model
doesn't work as well as just having the various boost_test_*() macros write
the test commandlines to a file, and then having a separate python
script do the running. (this is more like how ctest does it, but you're better
off, since you're not scraping logfiles and can submit results directly
from this script via xmlrpc.)  Another problem this addresses is when a library
has a set of tests that are meant to be run in order: if they are pure make targets,
one has to add dependencies (test_c depends on test_b depends on test_a), and this
is tricky and wordy and dumb.  With such a test-runner script you can just code it
such that they run in alphabetical order, and you're done.
It also makes makefile generation and time-to-build-first-target *significantly*
faster and reduces the amount of configuration. Luckily this requires no modification
on the reporting-site side.  Here is what a test-runner-script might look like:

   http://code.icecube.wisc.edu/projects/icecube/browser/projects/cmake/trunk/runtests.py.in

Here a sample slave-runner-script for regression testers is already in boost svn:

   http://svn.boost.org/trac/boost/browser/branches/CMake/release/tools/build/CMake/run_continuous_slave.py.in

it should get written to your build directory when BOOST_BUILD_SLAVE is turned on.

-t
_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Gubenko, Boris]

Can we add GCC 4.2.1 on HP-UX/ia64 to the list of release platforms for 1.37.0? I'm testing this platform against trunk since 1.35 and have the resources to do release branch testing as well.

Thanks,
  Boris

> -----Original Message-----
> From: boost-testing-bounces@...
> [mailto:boost-testing-bounces@...] On Behalf Of
> Beman Dawes
> Sent: Sunday, August 24, 2008 4:59 PM
> To: Boost testing list
> Subject: [Boost-testing] [1.37.0] Compilers for release testing?
>
> For 1.36.0, our release test primary compilers were:
>
>     * GCC 4.01 on Mac OS X 10.4.10 with both Intel and Power PC
>     * GCC 4.2.3 on Ubuntu Linux 8.04.1
>     * HP C/aC++ B3910B A.06.17 on HP-UX 64-bit
>     * Visual C++ 9.0 SP1 beta, 8.0 SP1, and 7.1, all on
> Windows XP SP-2
>
> Could we have volunteers to run the Ubuntu Linux/GCC tests and the
> Windows/msvc-8.0 tests? If someone could take those off my hands, it
> would free resources to add additional compilers.
>
> If I don't have to run any tests, it will free resources enough to
> expand the list by two or three compilers for 1.37.0. The idea is to
> provide coverage for popular compiler/operating system
> combinations not
> currently being tested. Anyone care to volunteer?
>
> I'd prefer volunteers who have experience testing on trunk, and been
> able to keep their test setup running reliably.  They also
> need to have
> a bit of time to monitor the Boost testing list.
>
> No more than two tests per tester, to limit the inconvenience if a
> tester goes down.
>
> Thanks,
>
> --Beman
>
>
> _______________________________________________
> Boost-Testing mailing list
> Boost-Testing@...
> http://lists.boost.org/mailman/listinfo.cgi/boost-testing
>

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Beman Dawes]

Gubenko, Boris wrote:
> Can we add GCC 4.2.1 on HP-UX/ia64 to the list of release platforms
> for 1.37.0? I'm testing this platform against trunk since 1.35 and
> have the resources to do release branch testing as well.

Yes, please do.

Thanks,

--Beman

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Beman Dawes]

troy d. straszheim wrote:
> David Abrahams wrote:

>> Hmm, http://boost.resophonic.com/trac/traash?builds=all doesn't look too
>> encouraging right now.  What's going on, Troy?
>
> I stopped running builds, since the reaction to it was so superficial and
> lukewarm.  For instance, all of the uses cases put forward by Beman
> (and other important ones not put forward) were reachable
> in ~4 clicks from the front page..  well addressed by the interface, I
> thought.

I must have missed that. Where should I be looking? The URL Dave gives
above doesn't seem to be active.

--Beman

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Darryl Green-4]

On Sun, 2008-08-24 at 16:58 -0400, Beman Dawes wrote:

> For 1.36.0, our release test primary compilers were:
>
>     * GCC 4.01 on Mac OS X 10.4.10 with both Intel and Power PC
>     * GCC 4.2.3 on Ubuntu Linux 8.04.1
>     * HP C/aC++ B3910B A.06.17 on HP-UX 64-bit
>     * Visual C++ 9.0 SP1 beta, 8.0 SP1, and 7.1, all on Windows XP SP-2
>
> Could we have volunteers to run the Ubuntu Linux/GCC tests and the
> Windows/msvc-8.0 tests? If someone could take those off my hands, it
> would free resources to add additional compilers.

Does it have to be ubuntu (I'm running Debian testing)?
What version(s) of GCC?

>  
> If I don't have to run any tests, it will free resources enough to
> expand the list by two or three compilers for 1.37.0. The idea is to
> provide coverage for popular compiler/operating system combinations not
> currently being tested. Anyone care to volunteer?

Um - yes. Assuming that you consider a reasonable range of GCC versions
on utterly ordinary linux to be somewhere in the "popular" space?

>
> I'd prefer volunteers who have experience testing on trunk

I haven't been because there is planty of gcc coverage on trunk and I
didn't want to create extra work for you guys - as per prev threads...

> , and been
> able to keep their test setup running reliably.

It stays up except for kernel upgrades and rare (so far) power outages.
It is running RAID1.

>   They also need to have
> a bit of time to monitor the Boost testing list.

Well, I suspect I have more time to do that than you do :)

>
> No more than two tests per tester, to limit the inconvenience if a
> tester goes down.

I was going to test 4.1, 4.3 because these were the gaps in 1.36 so far
as the 4.x series goes, moving to 4.4 once it branches (next few
months). I was intending to keep the point release level up to date for
the (so currently that would be 4.3.2).

I thought it was important to continue to test with 4.1 because its a
common "system" compiler version on stable/production systems. Testing
4.3 as the curent gcc release series seems kinda important too.


_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Richard Webb]

Beman Dawes <bdawes <at> acm.org> writes:

>
> Could we have volunteers to run the Ubuntu Linux/GCC tests and the
> Windows/msvc-8.0 tests? If someone could take those off my hands, it
> would free resources to add additional compilers.
>

I could switch from vc7.1 to 8.0 (then David Walthall could run the 7.1 tests
instead?).


_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[David Walthall-2]

Richard Webb wrote:
> I could switch from vc7.1 to 8.0 (then David Walthall could run the 7.1 tests
> instead?).

That would be fine with me.

David

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Richard Webb]

Richard Webb <richard.webb <at> boldonjames.com> writes:

>
> Beman Dawes <bdawes <at> acm.org> writes:
>
> >
> > Could we have volunteers to run the Ubuntu Linux/GCC tests and the
> > Windows/msvc-8.0 tests? If someone could take those off my hands, it
> > would free resources to add additional compilers.
> >

I've run some release tests on VC8 - results are on the website now.

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[David Abrahams-3]

on Tue Aug 26 2008, "troy d. straszheim" <troy-AT-resophonic.com> wrote:

> David Abrahams wrote:
>> on Mon Aug 25 2008, Beman Dawes <bdawes-AT-acm.org> wrote:
>>
>>> Dave, I'd really like to see you apply your skills to helping get a
>>> reporting system going for CMake based testing. We've got plenty of
>>> volunteers who can do a great job with bjam testing, but no so many
>>> with your strong sense of the needs for test reporting.
>>
>> I'd be happy to try.  I sorta thought Troy's friend Evan Wheeler was
>> working on that?  He's apparently got much better web-UI chops than I
>> ever will (Ajax, etc.) but I'd be more than happy to collaborate.
>
> He got buried in day-job deadlines, as have I.  We're in a release cycle,
> and I'm going to have to turn back to this stuff in the next week or so,
> but at the moment I can't do any better than this mail.

Thanks for taking the time anyway.  I'm sorry I couldn't reply to this sooner.

>> Hmm, http://boost.resophonic.com/trac/traash?builds=all doesn't look too
>> encouraging right now.  What's going on, Troy?
>
> I stopped running builds, since the reaction to it was so superficial and
> lukewarm.  For instance, all of the uses cases put forward by Beman
> (and other important ones not put forward) were reachable
> in ~4 clicks from the front page..  well addressed by the interface, I
> thought.

I know you probably don't have much time to deal with this right now,
but IMO the first and most-important change we need in the interface is
that we need a grid where the time axis runs down the page and the
latest results are always visible in the top row.  I think we probably
want to be able to choose whether columns represent libraries or
platforms or compiler families or...

> I was hoping to get back round to this and make another effort to
> explain things, but I'm willing to forget it as well.

Well, I can't say I understand the interface yet, but once I start
running my own server it should become clearer.

> The code is here:
>
>    http://svn.resophonic.com/svn/trac-dev
>
> And I'm sure it won't run right out of the box.  There is a lot of
> site-specific hackery in there.  I wouldn't normally show code to
> anybody in that condition, but if you are anxious to start hacking,
> there you go.

Eager but not anxious ;-)

> There are also some changes that really need to be made on the
> boost-cmake side, which I can do when my orbit comes back around: the
> one-cmake-target-per-test model doesn't work as well as just having
> the various boost_test_*() macros write the test commandlines to a
> file, and then having a separate python script do the running.

When you get a moment:

1. Why not?
2. Can we do does-it-compile tests this way also?

> (this is more like how ctest does it, but you're better off, since
> you're not scraping logfiles and can submit results directly from this
> script via xmlrpc.)  Another problem this addresses is when a library
> has a set of tests that are meant to be run in order: if they are pure
> make targets, one has to add dependencies (test_c depends on test_b
> depends on test_a), and this is tricky and wordy and dumb.  With such
> a test-runner script you can just code it such that they run in
> alphabetical order, and you're done.

But then don't you lose an opportunity for parallelism?

> It also makes makefile generation and time-to-build-first-target
> *significantly* faster and reduces the amount of
> configuration. Luckily this requires no modification on the
> reporting-site side.  Here is what a test-runner-script might look
> like:
>
>   http://code.icecube.wisc.edu/projects/icecube/browser/projects/cmake/trunk/runtests.py.in

I don't have FILE_VIEW permission.  What would you think about keeping
this stuff in Boost's SVN where all the people you want to collaborate
with will also have access?

> Here a sample slave-runner-script for regression testers is already in
> boost svn:
>
>   http://svn.boost.org/trac/boost/browser/branches/CMake/release/tools/build/CMake/run_continuous_slave.py.in
>
> it should get written to your build directory when BOOST_BUILD_SLAVE
> is turned on.

I'll have to play with this stuff before I really understand what you're
talking about, but thanks for the pointers.

--
Dave Abrahams
BoostPro Computing
http://www.boostpro.com
_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing

[Beman Dawes]

Richard Webb wrote:

> Richard Webb <richard.webb <at> boldonjames.com> writes:
>
>> Beman Dawes <bdawes <at> acm.org> writes:
>>
>>> Could we have volunteers to run the Ubuntu Linux/GCC tests and the
>>> Windows/msvc-8.0 tests? If someone could take those off my hands, it
>>> would free resources to add additional compilers.
>>>
>
> I've run some release tests on VC8 - results are on the website now.

Thanks! Between you and David Walthall we are getting good release
branch Windows 32-bit coverage.

--Beman

_______________________________________________
Boost-Testing mailing list
Boost-Testing@...
http://lists.boost.org/mailman/listinfo.cgi/boost-testing