Do not publish daily build if tests are not compilable

[Jaroslav Tulach]

Hi all.
I think that we have made big step forward compared to our engineering
practices a year ago. Now we have the http://hg.netbeans.org/main-golden 
repository which is always compilable. This is really good.

For example, those of us working on http://hg.netbeans.org/simpletests/ can
synchronize with that repository day by day, without risk of taking into
our "branch" any bad, uncompilable code. It really improves our productivity
to have "fixed point" known to work, that we can always use.

However, as the name of our repository suggests, we also care about tests. In
fact, the whole repository is about testing. And here we have just faced very
unfriendly situation. I did the merge with main-golden on Monday and since
then our tests became uncompilable. As I have found out, I did not do
anything wrong, the broken, uncompilable tests were present in main-golden.

I have not invented anything better than to wait till the main-golden gets
updated - so today, after five days of failures in the most important part of
our branch, we finally have working tests again. I hope I do not need to
point out how much that hurts our productivity.

I know how to fix this, and prevent similar situation from appearing ever
again. I'd like Michal to run the test compilation before publishing the
daily build and promoting the code changes to main-golden. Imho, having
compilable tests is as important as having compilable product code. If there
are no objections, I'll ask Michal to turn this check on after a week or so.

-jst

[Dmitry Lipin-2]

Hi Jarda,

Just my $0.02...

Jaroslav Tulach wrote:

> Hi all.
> I think that we have made big step forward compared to our engineering
> practices a year ago. Now we have the http://hg.netbeans.org/main-golden 
> repository which is always compilable. This is really good.
>
> For example, those of us working on http://hg.netbeans.org/simpletests/ can
> synchronize with that repository day by day, without risk of taking into
> our "branch" any bad, uncompilable code. It really improves our productivity
> to have "fixed point" known to work, that we can always use.
>
> However, as the name of our repository suggests, we also care about tests. In
> fact, the whole repository is about testing. And here we have just faced very
> unfriendly situation. I did the merge with main-golden on Monday and since
> then our tests became uncompilable. As I have found out, I did not do
> anything wrong, the broken, uncompilable tests were present in main-golden.
>
> I have not invented anything better than to wait till the main-golden gets
> updated - so today, after five days of failures in the most important part of
> our branch, we finally have working tests again. I hope I do not need to
> point out how much that hurts our productivity.
>
> I know how to fix this, and prevent similar situation from appearing ever
> again. I'd like Michal to run the test compilation before publishing the
> daily build and promoting the code changes to main-golden.

My personal opinion is that these two tasks - publishing daily build and
pushing changes to main-golden - are independent.
Yes, some teams depends on the state of main-golden repository, but
there are also teams that need the production build.
I agree that pushing changes to main-golden is inappropriate if some
tests fail but I don`t see a reason why we should not publish daily build.
In particual, Installer Team does not really depend on the quality of
NetBeans IDE itself - often we have no other way to check our work other
than check the daily (production) build.
Certainly, we verify the changes in our development environment but
sometimes, e.g. after significant changes, we always check again what we
did in the installers that were made during production build .
Additionally, afaik modules from main/contrib that goes to Alpha UC are
build only during production build.
I should also say that some teams perform automatic testing of NetBeans
IDE after its silent installation.
I`ve also got a number of negative feedback from QE after we treated
commit-validation tests failures as the build errors so, I guess, the
amount of such feedback would increase if we start treating every test
failure as the build failure.
If it is a problem and we want to alert users about the build status
("hey, this build didn`t pass our minimal quality level, use it on your
own risk") then we can mark it somehow with "Unstable" sign on the
download page.
[I shouldn`t say that in respect to our community but...] As the
consensus, we can publish the build internally, but do not upload it to
public server.

To sum up: I respect your team (as well as other teams who depends on
main-golden) work and the certainly understand the dependency of your
team productivity on the state of main-golden.
Let`s also try to respect those teams who doesn`t work with this repo.
I vote not to push changes to main-golden if tests fail but nevertheless
do the production build anyway.

Thanks,
Dmitry
> Imho, having
> compilable tests is as important as having compilable product code. If there
> are no objections, I'll ask Michal to turn this check on after a week or so.
>  



--
Dmitry Lipin
NBI Team Lead
Sun Microsystems, Inc.
Phone EMEA/APAC x33354 AMER x7033354
Email: Dmitry.Lipin@...

[Dmitry Lipin-2]

Hi All,

Please disregard my message, I was inattentive and confused test
compilation and test failure.
Sorry for misunderstanding.

Dmitry

Dmitry Lipin wrote:

> Hi Jarda,
>
> Just my $0.02...
>
> Jaroslav Tulach wrote:
>> Hi all.
>> I think that we have made big step forward compared to our
>> engineering practices a year ago. Now we have the
>> http://hg.netbeans.org/main-golden repository which is always
>> compilable. This is really good.
>>
>> For example, those of us working on
>> http://hg.netbeans.org/simpletests/ can synchronize with that
>> repository day by day, without risk of taking into our "branch" any
>> bad, uncompilable code. It really improves our productivity to have
>> "fixed point" known to work, that we can always use.
>>
>> However, as the name of our repository suggests, we also care about
>> tests. In fact, the whole repository is about testing. And here we
>> have just faced very unfriendly situation. I did the merge with
>> main-golden on Monday and since then our tests became uncompilable.
>> As I have found out, I did not do anything wrong, the broken,
>> uncompilable tests were present in main-golden.
>> I have not invented anything better than to wait till the main-golden
>> gets updated - so today, after five days of failures in the most
>> important part of our branch, we finally have working tests again. I
>> hope I do not need to point out how much that hurts our productivity.
>>
>> I know how to fix this, and prevent similar situation from appearing
>> ever again. I'd like Michal to run the test compilation before
>> publishing the daily build and promoting the code changes to
>> main-golden.
> My personal opinion is that these two tasks - publishing daily build
> and pushing changes to main-golden - are independent.
> Yes, some teams depends on the state of main-golden repository, but
> there are also teams that need the production build.
> I agree that pushing changes to main-golden is inappropriate if some
> tests fail but I don`t see a reason why we should not publish daily
> build.
> In particual, Installer Team does not really depend on the quality of
> NetBeans IDE itself - often we have no other way to check our work
> other than check the daily (production) build.
> Certainly, we verify the changes in our development environment but
> sometimes, e.g. after significant changes, we always check again what
> we did in the installers that were made during production build .
> Additionally, afaik modules from main/contrib that goes to Alpha UC
> are build only during production build.
> I should also say that some teams perform automatic testing of
> NetBeans IDE after its silent installation.
> I`ve also got a number of negative feedback from QE after we treated
> commit-validation tests failures as the build errors so, I guess, the
> amount of such feedback would increase if we start treating every test
> failure as the build failure.
> If it is a problem and we want to alert users about the build status
> ("hey, this build didn`t pass our minimal quality level, use it on
> your own risk") then we can mark it somehow with "Unstable" sign on
> the download page.
> [I shouldn`t say that in respect to our community but...] As the
> consensus, we can publish the build internally, but do not upload it
> to public server.
>
> To sum up: I respect your team (as well as other teams who depends on
> main-golden) work and the certainly understand the dependency of your
> team productivity on the state of main-golden.
> Let`s also try to respect those teams who doesn`t work with this repo.
> I vote not to push changes to main-golden if tests fail but
> nevertheless do the production build anyway.
>
> Thanks,
> Dmitry
>> Imho, having compilable tests is as important as having compilable
>> product code. If there are no objections, I'll ask Michal to turn
>> this check on after a week or so.
>>  
>
>
>

[Jaroslav Tulach]

Hello Dmitry.
Thanks for speaking up. I tend to have agreed with your opinion, however as
the tests are broken again, I also feel we need to find some solution.

> My personal opinion is that these two tasks - publishing daily build and
> pushing changes to main-golden - are independent.

I can understand that you need some build daily. Or a build which is produced
daily. You do not care much about its quality. Well, for that purpose you can
take deadlock's trunk build. It is usually produced at least once a day and
sort of works.

You said that pushing to main golden can be independent from producing build
for your purposes. I think this is OK, it would solve my immediate problems.

On the other hand, I do not believe we should publish anything else than a
build that qualifies for main-golden repository to any of our end users. I do
not think we can call a build "production ready" when it is known to be
broken. It scares me to imagine that we publish M1 build while we know it is
not even good enough for main-golden. That is why I believe we shall still
continue to have "golden" daily builds that pass as many sanity checks as
possible.

Btw. I've noticed one "little" assumption that imho makes the difference. You
talk about failed tests:

> amount of such feedback would increase if we start treating every test
> failure as the build failure.

I do not advocate test failures to prevent the build. I just want the tests to
be compilable. That is significantly easier to satisfy and should minimize
your fear.
-jst


Dne Saturday 07 June 2008 18:29:56 Dmitry Lipin napsal(a):

> Hi Jarda,
>
> Just my $0.02...
>
> Jaroslav Tulach wrote:
> > Hi all.
> > I think that we have made big step forward compared to our engineering
> > practices a year ago. Now we have the http://hg.netbeans.org/main-golden
> > repository which is always compilable. This is really good.
> >
> > For example, those of us working on http://hg.netbeans.org/simpletests/
> > can synchronize with that repository day by day, without risk of taking
> > into our "branch" any bad, uncompilable code. It really improves our
> > productivity to have "fixed point" known to work, that we can always use.
> >
> > However, as the name of our repository suggests, we also care about
> > tests. In fact, the whole repository is about testing. And here we have
> > just faced very unfriendly situation. I did the merge with main-golden on
> > Monday and since then our tests became uncompilable. As I have found out,
> > I did not do anything wrong, the broken, uncompilable tests were present
> > in main-golden.
> >
> > I have not invented anything better than to wait till the main-golden
> > gets updated - so today, after five days of failures in the most
> > important part of our branch, we finally have working tests again. I hope
> > I do not need to point out how much that hurts our productivity.
> >
> > I know how to fix this, and prevent similar situation from appearing ever
> > again. I'd like Michal to run the test compilation before publishing the
> > daily build and promoting the code changes to main-golden.
>
> My personal opinion is that these two tasks - publishing daily build and
> pushing changes to main-golden - are independent.
> Yes, some teams depends on the state of main-golden repository, but
> there are also teams that need the production build.
> I agree that pushing changes to main-golden is inappropriate if some
> tests fail but I don`t see a reason why we should not publish daily build.
> In particual, Installer Team does not really depend on the quality of
> NetBeans IDE itself - often we have no other way to check our work other
> than check the daily (production) build.
> Certainly, we verify the changes in our development environment but
> sometimes, e.g. after significant changes, we always check again what we
> did in the installers that were made during production build .
> Additionally, afaik modules from main/contrib that goes to Alpha UC are
> build only during production build.
> I should also say that some teams perform automatic testing of NetBeans
> IDE after its silent installation.
> I`ve also got a number of negative feedback from QE after we treated
> commit-validation tests failures as the build errors so, I guess, the
> amount of such feedback would increase if we start treating every test
> failure as the build failure.
> If it is a problem and we want to alert users about the build status
> ("hey, this build didn`t pass our minimal quality level, use it on your
> own risk") then we can mark it somehow with "Unstable" sign on the
> download page.
> [I shouldn`t say that in respect to our community but...] As the
> consensus, we can publish the build internally, but do not upload it to
> public server.
>
> To sum up: I respect your team (as well as other teams who depends on
> main-golden) work and the certainly understand the dependency of your
> team productivity on the state of main-golden.
> Let`s also try to respect those teams who doesn`t work with this repo.
> I vote not to push changes to main-golden if tests fail but nevertheless
> do the production build anyway.
>
> Thanks,
> Dmitry
>
> > Imho, having
> > compilable tests is as important as having compilable product code. If
> > there are no objections, I'll ask Michal to turn this check on after a
> > week or so.

[Jaroslav Tulach]

Dne Sunday 08 June 2008 11:24:27 Dmitry Lipin napsal(a):
> Hi All,
>
> Please disregard my message, I was inattentive and confused test
> compilation and test failure.
> Sorry for misunderstanding.

Still, it was an interesting mental exercise to find the right answers to your
valid requirements. Anyway I guess, we can proceed with this.

Guys in RE, please add "build-test-dist" into the daily build sequence of
targets and if it fails, please do not promote the build nor push changes to
the main-golden repository.
-jst

[Robert Novak]

Jaroslav Tulach wrote:

> Dne Sunday 08 June 2008 11:24:27 Dmitry Lipin napsal(a):
>> Hi All,
>>
>> Please disregard my message, I was inattentive and confused test
>> compilation and test failure.
>> Sorry for misunderstanding.
>
> Still, it was an interesting mental exercise to find the right answers to your
> valid requirements. Anyway I guess, we can proceed with this.
>
> Guys in RE, please add "build-test-dist" into the daily build sequence of
> targets and if it fails, please do not promote the build nor push changes to
> the main-golden repository.
> -jst
>

"build-test-dist" was a standard part of the build sequence.
I added a test to check whether this build step was successful and
stop the build process if it failed.

Robert

[Jaroslav Tulach]

Dne Monday 23 June 2008 18:45:22 Robert Novak napsal(a):
> > Guys in RE, please add "build-test-dist" into the daily build sequence of
> > targets and if it fails, please do not promote the build nor push changes
> > to the main-golden repository.
> > -jst
>
> "build-test-dist" was a standard part of the build sequence.
> I added a test to check whether this build step was successful and
> stop the build process if it failed.

Thanks a lot. A change like this will increase the quality of our tests. That
is for sure.
-jst