Matrix projects and executor deadlock

[John Casey-5]

Hi,

I've been trying to configure a build farm using Hudson, so that a
particular job can be built on multiple machine types to validate that
all tests succeed in a variety of environments. In this case, it's more
important that all selected environments be tested, rather than that the
slave instances be used for load balancing.

I should add that in our CI environment, we have dozens of projects.

To convert our old configuration to a matrix-based setup, I've modified
the job definitions from our old server to conform with a sample
definition I created using the Hudson UI...then, I pre-populated the
.hudson/jobs directory with these configuration files in their
respective job-name folders.

When the server starts up, it triggers a build in all of these jobs (I'm
guessing that's because there is no history or other information on any
of the jobs). This is where things get a little wild.

It seems that the matrix-style project consumes one executor solely for
orchestration of the matrix build. Initially, I had setup each matrix
project to run on the master machine and the single slave I'd setup (one
CentOS, one Ubuntu jeos, just for testing). What I found was the whole
system quickly deadlocked, as orchestration jobs filled all available
executors and stopped any of the child builds from happening on either
the master or the slave.

The second thing I tried was to make the master instance an
orchestration-only machine (to take it out of the matrix axes, and
prevent the child builds from running there). In this case, the executor
list of the master was quickly filled with orchestrator jobs again.
HOWEVER, it still seemed that other orchestrator jobs (not the child
builds) were being shipped over to the slave, where they quickly
deadlocked the 2 executors I had there.

Is there any way to keep these orchestration threads on the master, and
keep the slave executor queue clear for child builds that use that axis?
I've been through what documentation I could find regarding matrix
projects and distributed Hudson instances, but clearly I'm missing
something.

Can anyone suggest a fix, or some documentation to help me out? Has
anyone tried this sort of configuration before?

Thanks in advance,

-john

---
John Casey
Developer and PMC Member, Apache Maven (http://maven.apache.org)
Member, Apache Software Foundation
Blog: http://www.ejlife.net/blogs/buildchimp

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[Aleksandar Kostadinov]

Create one more slave with many executors(not necessary a separate
machine). Define it as "load as much as possible".

Configure all other slaves to "for tied jobs only".

Then tie all your jobs to nodes/labels.

Not too much pretty though. We use that setup. We have all slaves
labeled anyways so we didn't really need the situation improved.

John Casey wrote, On 12/23/-28158 09:59 PM (EEST):

> <div class="moz-text-flowed" style="font-family: -moz-fixed">Hi,
>
> I've been trying to configure a build farm using Hudson, so that a
> particular job can be built on multiple machine types to validate that
> all tests succeed in a variety of environments. In this case, it's more
> important that all selected environments be tested, rather than that the
> slave instances be used for load balancing.
>
> I should add that in our CI environment, we have dozens of projects.
>
> To convert our old configuration to a matrix-based setup, I've modified
> the job definitions from our old server to conform with a sample
> definition I created using the Hudson UI...then, I pre-populated the
> .hudson/jobs directory with these configuration files in their
> respective job-name folders.
>
> When the server starts up, it triggers a build in all of these jobs (I'm
> guessing that's because there is no history or other information on any
> of the jobs). This is where things get a little wild.
>
> It seems that the matrix-style project consumes one executor solely for
> orchestration of the matrix build. Initially, I had setup each matrix
> project to run on the master machine and the single slave I'd setup (one
> CentOS, one Ubuntu jeos, just for testing). What I found was the whole
> system quickly deadlocked, as orchestration jobs filled all available
> executors and stopped any of the child builds from happening on either
> the master or the slave.
>
> The second thing I tried was to make the master instance an
> orchestration-only machine (to take it out of the matrix axes, and
> prevent the child builds from running there). In this case, the executor
> list of the master was quickly filled with orchestrator jobs again.
> HOWEVER, it still seemed that other orchestrator jobs (not the child
> builds) were being shipped over to the slave, where they quickly
> deadlocked the 2 executors I had there.
>
> Is there any way to keep these orchestration threads on the master, and
> keep the slave executor queue clear for child builds that use that axis?
> I've been through what documentation I could find regarding matrix
> projects and distributed Hudson instances, but clearly I'm missing
> something.
>
> Can anyone suggest a fix, or some documentation to help me out? Has
> anyone tried this sort of configuration before?
>
> Thanks in advance,
>
> -john
>
> ---
> John Casey
> Developer and PMC Member, Apache Maven (http://maven.apache.org)
> Member, Apache Software Foundation
> Blog: http://www.ejlife.net/blogs/buildchimp
> </div>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...

[John Casey-5]

Thanks! That appears to work well.

-john

---
John Casey
Developer and PMC Member, Apache Maven (http://maven.apache.org)
Member, Apache Software Foundation
Blog: http://www.ejlife.net/blogs/buildchimp

Aleksandar Kostadinov wrote:

> Create one more slave with many executors(not necessary a separate
> machine). Define it as "load as much as possible".
>
> Configure all other slaves to "for tied jobs only".
>
> Then tie all your jobs to nodes/labels.
>
> Not too much pretty though. We use that setup. We have all slaves
> labeled anyways so we didn't really need the situation improved.
>
> John Casey wrote, On 12/23/-28158 09:59 PM (EEST):
>> <div class="moz-text-flowed" style="font-family: -moz-fixed">Hi,
>>
>> I've been trying to configure a build farm using Hudson, so that a
>> particular job can be built on multiple machine types to validate that
>> all tests succeed in a variety of environments. In this case, it's
>> more important that all selected environments be tested, rather than
>> that the slave instances be used for load balancing.
>>
>> I should add that in our CI environment, we have dozens of projects.
>>
>> To convert our old configuration to a matrix-based setup, I've
>> modified the job definitions from our old server to conform with a
>> sample definition I created using the Hudson UI...then, I
>> pre-populated the .hudson/jobs directory with these configuration
>> files in their respective job-name folders.
>>
>> When the server starts up, it triggers a build in all of these jobs
>> (I'm guessing that's because there is no history or other information
>> on any of the jobs). This is where things get a little wild.
>>
>> It seems that the matrix-style project consumes one executor solely
>> for orchestration of the matrix build. Initially, I had setup each
>> matrix project to run on the master machine and the single slave I'd
>> setup (one CentOS, one Ubuntu jeos, just for testing). What I found
>> was the whole system quickly deadlocked, as orchestration jobs filled
>> all available executors and stopped any of the child builds from
>> happening on either the master or the slave.
>>
>> The second thing I tried was to make the master instance an
>> orchestration-only machine (to take it out of the matrix axes, and
>> prevent the child builds from running there). In this case, the
>> executor list of the master was quickly filled with orchestrator jobs
>> again. HOWEVER, it still seemed that other orchestrator jobs (not the
>> child builds) were being shipped over to the slave, where they quickly
>> deadlocked the 2 executors I had there.
>>
>> Is there any way to keep these orchestration threads on the master,
>> and keep the slave executor queue clear for child builds that use that
>> axis? I've been through what documentation I could find regarding
>> matrix projects and distributed Hudson instances, but clearly I'm
>> missing something.
>>
>> Can anyone suggest a fix, or some documentation to help me out? Has
>> anyone tried this sort of configuration before?
>>
>> Thanks in advance,
>>
>> -john
>>
>> ---
>> John Casey
>> Developer and PMC Member, Apache Maven (http://maven.apache.org)
>> Member, Apache Software Foundation
>> Blog: http://www.ejlife.net/blogs/buildchimp
>> </div>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@...
> For additional commands, e-mail: users-help@...
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@...
For additional commands, e-mail: users-help@...