You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Benjamin Graf <be...@gmx.net> on 2020/05/01 08:32:33 UTC
About NetBeans build quality
Hi everybody,
I'm a bit concerned about the current build quality of NetBeans master branch and PRs done in the past. I have the feeling we're facing many broken PR builds over the last time causing/risking master to get unstable especially in the current process of building a LTS release.
IMHO we all should have a look and taking more care after our PRs being green on Travis, too. Sometimes periodically rebasing against master does help getting green again if build is broken because of Travis bugs or someone else broken commits.
This is not meant for blaming someone but to keep quality high and make NetBeans the best experience for everyone.
Kind regards
Benjamin
Re: About NetBeans build quality
Posted by Neil C Smith <ne...@apache.org>.
On Sat, 2 May 2020 at 08:20, Benjamin Graf <be...@gmx.net> wrote:
> Yes, and they are green because work has been done to get them green
> again. :-)
Thanks! :-)
> The problem is the big bunch of PRs based on old master
> snapshots that where unstable in time of creation. That's the reason
> their Travis builds are broken.
Retriggering should work OK (for most) here? That reapplies the
changes on top of current master, not a snapshot, AFAIK?
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: About NetBeans build quality
Posted by Benjamin Graf <be...@gmx.net>.
Hi Eric,
> Travis Master build has the same testing rule that a PR build has ? because it's green and the sync to release too.
Yes, and they are green because work has been done to get them green
again. :-) The problem is the big bunch of PRs based on old master
snapshots that where unstable in time of creation. That's the reason
their Travis builds are broken. IMHO that makes it hard to check if they
are ready to merge especially in your role as a release manager. That's
the reason why I like to emphasize that people should have a look on
their open PRs for periodically rebase. Less possible merging conflicts
and more valid status concerning test status. I don't want to say that
this is the only item to look after for taking the decision whether to
merge or not. But it helps a lot.
> PR check doing only test is not enough. A pr with invalid mail, invalid name should be flagged too before review code. It cost Matthias hours :p
> Hector Espert do a lot of addition to travis test but I'm unsure the coverage of travis test.
There is still room for improvement. ;-)
Kind regards,
Benjamin
On 02.05.2020 00:57, Eric Barboni wrote:
> Hi,
>
> I agree with the big picture. But
>
> For what I encounter since 2 months I guess, travis job is not more re triggerable (partial job too).
> During 11.3 travis build had 1 download issue for 4 builds and was red for no reason. So travis trust is low for me.
> Travis Master build has the same testing rule that a PR build has ? because it's green and the sync to release too.
>
> PR check doing only test is not enough. A pr with invalid mail, invalid name should be flagged too before review code. It cost Matthias hours :p
> Hector Espert do a lot of addition to travis test but I'm unsure the coverage of travis test.
> Best Regards
> Eric
>
> -----Message d'origine-----
> De : Neil C Smith <ne...@apache.org>
> Envoyé : vendredi 1 mai 2020 12:23
> À : dev <de...@netbeans.apache.org>
> Objet : Re: About NetBeans build quality
>
> On Fri, 1 May 2020 at 09:32, Benjamin Graf <be...@gmx.net> wrote:
>> IMHO we all should have a look and taking more care after our PRs being green on Travis, too. Sometimes periodically rebasing against master does help getting green again if build is broken because of Travis bugs or someone else broken commits.
> +1 And sometimes just retriggering Travis gets things working - to
> pick up fixes in master, and there are a few spurious failures at times unfortunately.
>
> My general approach when RM'ing was all PRs must be green before merging to master, the sync PR from master to release branch must be green before merging, and the release branch must be green before triggering a beta / release build.
>
> This isn't a criticism of Eric's work here because I think aspects of the Travis tests have been broken for a while now. Be good to get them back on track, ideally before considering 12.0 release.
>
> Best wishes,
>
> Neil
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
Re: About NetBeans build quality
Posted by Hector Espert <he...@gmail.com>.
>
> Hector, for the github action, maybe double check with Apache infra on
> their asf slack channel.
>
I asked for it but i don't receive any response, but there ara various
entries about GitHub actions in the apache wiki. Seems that some projects
are using this or are begin to use it.
https://cwiki.apache.org/confluence/display/INFRA/GitHub+Actions+and+Secrets+Policies
https://cwiki.apache.org/confluence/display/AIRFLOW/AIP-23+Migrate+out+of+Travis+CI
Would be nice to expose mail and name(it's only available to people logged
> and this information should be public )
>
I updated the action behaviour to show the name and the email for every
commit. It's a public information.
El lun., 4 may. 2020 a las 10:12, Eric Barboni (<sk...@apache.org>)
escribió:
> Hi,
> Thanks for the reloging tip it allow me to retrigger the build.
>
> Hector, for the github action, maybe double check with Apache infra on
> their asf slack channel. They may have done one or yours can help other
> project. Would be nice to expose mail and name(it's only available to
> people logged and this information should be public )
>
> Best Regards
> Eric
>
>
> -----Message d'origine-----
> De : Hector Espert <he...@gmail.com>
> Envoyé : samedi 2 mai 2020 17:19
> À : dev <de...@netbeans.apache.org>
> Objet : Re: About NetBeans build quality
>
> Hi,
>
> About a pull request with an invalid mail or an invalid name.
>
> I've started experimenting with GitHub actions and I've created a simple
> action to check commit author name and email.
> Currently anly check if commit is not created using a GitHub private
> email, like xxxxxx@users.noreply.github.com or if the name matches a
> regex.
> It is published in GitHub actions marketplace:
> https://github.com/marketplace/actions/check-author-name-and-email
> I've created a draft pr in the netbeans repo to test it:
> https://github.com/apache/netbeans/pull/212
>
> Is there any objection to use GitHub actions for that?
>
> El sáb., 2 may. 2020 a las 0:57, Eric Barboni (<sk...@apache.org>)
> escribió:
>
> > Hi,
> >
> > I agree with the big picture. But
> >
> > For what I encounter since 2 months I guess, travis job is not more
> > re triggerable (partial job too).
> > During 11.3 travis build had 1 download issue for 4 builds and was red
> > for no reason. So travis trust is low for me.
> > Travis Master build has the same testing rule that a PR build has ?
> > because it's green and the sync to release too.
> >
> > PR check doing only test is not enough. A pr with invalid mail,
> > invalid name should be flagged too before review code. It cost
> > Matthias hours :p Hector Espert do a lot of addition to travis test
> > but I'm unsure the coverage of travis test.
> > Best Regards
> > Eric
> >
> > -----Message d'origine-----
> > De : Neil C Smith <ne...@apache.org> Envoyé : vendredi 1 mai 2020
> > 12:23 À : dev <de...@netbeans.apache.org> Objet : Re: About NetBeans
> > build quality
> >
> > On Fri, 1 May 2020 at 09:32, Benjamin Graf <be...@gmx.net>
> wrote:
> > > IMHO we all should have a look and taking more care after our PRs
> > > being
> > green on Travis, too. Sometimes periodically rebasing against master
> > does help getting green again if build is broken because of Travis
> > bugs or someone else broken commits.
> >
> > +1 And sometimes just retriggering Travis gets things working - to
> > pick up fixes in master, and there are a few spurious failures at
> > times unfortunately.
> >
> > My general approach when RM'ing was all PRs must be green before
> > merging to master, the sync PR from master to release branch must be
> > green before merging, and the release branch must be green before
> > triggering a beta / release build.
> >
> > This isn't a criticism of Eric's work here because I think aspects of
> > the Travis tests have been broken for a while now. Be good to get
> > them back on track, ideally before considering 12.0 release.
> >
> > Best wishes,
> >
> > Neil
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> > For additional commands, e-mail: dev-help@netbeans.apache.org
> >
> > For further information about the NetBeans mailing lists, visit:
> > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >
> >
> >
> >
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> > For additional commands, e-mail: dev-help@netbeans.apache.org
> >
> > For further information about the NetBeans mailing lists, visit:
> > https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
> >
> >
> >
> >
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
RE: About NetBeans build quality
Posted by Eric Barboni <sk...@apache.org>.
Hi,
Thanks for the reloging tip it allow me to retrigger the build.
Hector, for the github action, maybe double check with Apache infra on their asf slack channel. They may have done one or yours can help other project. Would be nice to expose mail and name(it's only available to people logged and this information should be public )
Best Regards
Eric
-----Message d'origine-----
De : Hector Espert <he...@gmail.com>
Envoyé : samedi 2 mai 2020 17:19
À : dev <de...@netbeans.apache.org>
Objet : Re: About NetBeans build quality
Hi,
About a pull request with an invalid mail or an invalid name.
I've started experimenting with GitHub actions and I've created a simple action to check commit author name and email.
Currently anly check if commit is not created using a GitHub private email, like xxxxxx@users.noreply.github.com or if the name matches a regex.
It is published in GitHub actions marketplace:
https://github.com/marketplace/actions/check-author-name-and-email
I've created a draft pr in the netbeans repo to test it:
https://github.com/apache/netbeans/pull/212
Is there any objection to use GitHub actions for that?
El sáb., 2 may. 2020 a las 0:57, Eric Barboni (<sk...@apache.org>) escribió:
> Hi,
>
> I agree with the big picture. But
>
> For what I encounter since 2 months I guess, travis job is not more
> re triggerable (partial job too).
> During 11.3 travis build had 1 download issue for 4 builds and was red
> for no reason. So travis trust is low for me.
> Travis Master build has the same testing rule that a PR build has ?
> because it's green and the sync to release too.
>
> PR check doing only test is not enough. A pr with invalid mail,
> invalid name should be flagged too before review code. It cost
> Matthias hours :p Hector Espert do a lot of addition to travis test
> but I'm unsure the coverage of travis test.
> Best Regards
> Eric
>
> -----Message d'origine-----
> De : Neil C Smith <ne...@apache.org> Envoyé : vendredi 1 mai 2020
> 12:23 À : dev <de...@netbeans.apache.org> Objet : Re: About NetBeans
> build quality
>
> On Fri, 1 May 2020 at 09:32, Benjamin Graf <be...@gmx.net> wrote:
> > IMHO we all should have a look and taking more care after our PRs
> > being
> green on Travis, too. Sometimes periodically rebasing against master
> does help getting green again if build is broken because of Travis
> bugs or someone else broken commits.
>
> +1 And sometimes just retriggering Travis gets things working - to
> pick up fixes in master, and there are a few spurious failures at
> times unfortunately.
>
> My general approach when RM'ing was all PRs must be green before
> merging to master, the sync PR from master to release branch must be
> green before merging, and the release branch must be green before
> triggering a beta / release build.
>
> This isn't a criticism of Eric's work here because I think aspects of
> the Travis tests have been broken for a while now. Be good to get
> them back on track, ideally before considering 12.0 release.
>
> Best wishes,
>
> Neil
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: About NetBeans build quality
Posted by Hector Espert <he...@gmail.com>.
Hi,
About a pull request with an invalid mail or an invalid name.
I've started experimenting with GitHub actions and I've created a simple
action to check commit author name and email.
Currently anly check if commit is not created using a GitHub private email,
like xxxxxx@users.noreply.github.com or if the name matches a regex.
It is published in GitHub actions marketplace:
https://github.com/marketplace/actions/check-author-name-and-email
I've created a draft pr in the netbeans repo to test it:
https://github.com/apache/netbeans/pull/212
Is there any objection to use GitHub actions for that?
El sáb., 2 may. 2020 a las 0:57, Eric Barboni (<sk...@apache.org>) escribió:
> Hi,
>
> I agree with the big picture. But
>
> For what I encounter since 2 months I guess, travis job is not more re
> triggerable (partial job too).
> During 11.3 travis build had 1 download issue for 4 builds and was red for
> no reason. So travis trust is low for me.
> Travis Master build has the same testing rule that a PR build has ?
> because it's green and the sync to release too.
>
> PR check doing only test is not enough. A pr with invalid mail, invalid
> name should be flagged too before review code. It cost Matthias hours :p
> Hector Espert do a lot of addition to travis test but I'm unsure the
> coverage of travis test.
> Best Regards
> Eric
>
> -----Message d'origine-----
> De : Neil C Smith <ne...@apache.org>
> Envoyé : vendredi 1 mai 2020 12:23
> À : dev <de...@netbeans.apache.org>
> Objet : Re: About NetBeans build quality
>
> On Fri, 1 May 2020 at 09:32, Benjamin Graf <be...@gmx.net> wrote:
> > IMHO we all should have a look and taking more care after our PRs being
> green on Travis, too. Sometimes periodically rebasing against master does
> help getting green again if build is broken because of Travis bugs or
> someone else broken commits.
>
> +1 And sometimes just retriggering Travis gets things working - to
> pick up fixes in master, and there are a few spurious failures at times
> unfortunately.
>
> My general approach when RM'ing was all PRs must be green before merging
> to master, the sync PR from master to release branch must be green before
> merging, and the release branch must be green before triggering a beta /
> release build.
>
> This isn't a criticism of Eric's work here because I think aspects of the
> Travis tests have been broken for a while now. Be good to get them back on
> track, ideally before considering 12.0 release.
>
> Best wishes,
>
> Neil
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
Re: About NetBeans build quality
Posted by Neil C Smith <ne...@apache.org>.
On Sat, 2 May 2020, 10:02 Matthias Bläsing, <mb...@doppel-helix.eu>
wrote:
> a) an external change causes test failures - the recent failure of the
> vscode extension build is one such sample
> b) an external outage causes test failures - neighther github, nor
> apache apache gitbox, nor the ousol, nor maven central are perfect
> c) travis changes infrastructure changes - I have seen this in other
> projects
>
> I don't say travis should be ignored, but putting it on a pedastal is
> also wrong.
>
Which part of what I said implies putting it on a pedastal?! It's a PITA!
;-) But it picks up problems, and is currently the thing we have.
Of your scenarios, a) and c) require prioritising fixes before further PRs
being merged, and b) means the RM has to be good at retriggering and
reading the build logs!
I also think in and out of freeze approaches to this could be different.
Best wishes,
Neil
>
Re: About NetBeans build quality
Posted by Matthias Bläsing <mb...@doppel-helix.eu>.
Hi,
Am Samstag, den 02.05.2020, 09:49 +0100 schrieb Neil C Smith:
> If master testing is broken, that should be fixed before other
> merges. If it's a flaky test, and we have a few, then it should be
> retriggered to see if it passes.
I'm not a fan of "this is always right in all situations", these rules
have the tendency to become religious.
Some samples:
a) an external change causes test failures - the recent failure of the
vscode extension build is one such sample
b) an external outage causes test failures - neighther github, nor
apache apache gitbox, nor the ousol, nor maven central are perfect
c) travis changes infrastructure changes - I have seen this in other
projects
I don't say travis should be ignored, but putting it on a pedastal is
also wrong.
Greetings
Matthias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: About NetBeans build quality
Posted by Neil C Smith <ne...@apache.org>.
On Sat, 2 May 2020 at 08:01, Matthias Bläsing <mb...@doppel-helix.eu> wrote:
> Am Samstag, den 02.05.2020, 00:57 +0200 schrieb Eric Barboni:
> > For what I encounter since 2 months I guess, travis job is not more
> > re triggerable (partial job too).
>
> I saw that in the past, what helped was logging out of travis-ci and
> logging back in. I don't know how they authenticate the rebuild
> priviledge, but I assume, that somewhere along the way a token expires
> or gets lost and you loose access.
Yes, I can definitely retrigger, but have recently relogged in due to
same issue with an external project.
> I see travis as another tool. In an ideal world all unittests would
> always work, but world is not ideal. So there can be valid reasons to
> ignore failing travis tests - that is a decision of the committer.
?! Not sure I agree with the second bit. If the only thing that
triggers the failure is the change in the PR, why would we ignore it?
If expected the PR should fix or (temporarily) remove the test. If
master testing is broken, that should be fixed before other merges.
If it's a flaky test, and we have a few, then it should be retriggered
to see if it passes. If it consistently breaks the build after merge,
it should be fixed first or reverted.
There was not a single PR during 11.1 and 11.2 that could not be made
to go green after retriggers that did not prove to have a valid
problem in it.
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: About NetBeans build quality
Posted by Matthias Bläsing <mb...@doppel-helix.eu>.
Hi Eric,
Am Samstag, den 02.05.2020, 00:57 +0200 schrieb Eric Barboni:
> For what I encounter since 2 months I guess, travis job is not more
> re triggerable (partial job too).
I saw that in the past, what helped was logging out of travis-ci and
logging back in. I don't know how they authenticate the rebuild
priviledge, but I assume, that somewhere along the way a token expires
or gets lost and you loose access.
> During 11.3 travis build had 1 download issue for 4 builds and was red for no reason. So travis trust is low for me.
I see travis as another tool. In an ideal world all unittests would
always work, but world is not ideal. So there can be valid reasons to
ignore failing travis tests - that is a decision of the committer.
Greetings
Matthias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
RE: About NetBeans build quality
Posted by Eric Barboni <sk...@apache.org>.
Hi,
I agree with the big picture. But
For what I encounter since 2 months I guess, travis job is not more re triggerable (partial job too).
During 11.3 travis build had 1 download issue for 4 builds and was red for no reason. So travis trust is low for me.
Travis Master build has the same testing rule that a PR build has ? because it's green and the sync to release too.
PR check doing only test is not enough. A pr with invalid mail, invalid name should be flagged too before review code. It cost Matthias hours :p
Hector Espert do a lot of addition to travis test but I'm unsure the coverage of travis test.
Best Regards
Eric
-----Message d'origine-----
De : Neil C Smith <ne...@apache.org>
Envoyé : vendredi 1 mai 2020 12:23
À : dev <de...@netbeans.apache.org>
Objet : Re: About NetBeans build quality
On Fri, 1 May 2020 at 09:32, Benjamin Graf <be...@gmx.net> wrote:
> IMHO we all should have a look and taking more care after our PRs being green on Travis, too. Sometimes periodically rebasing against master does help getting green again if build is broken because of Travis bugs or someone else broken commits.
+1 And sometimes just retriggering Travis gets things working - to
pick up fixes in master, and there are a few spurious failures at times unfortunately.
My general approach when RM'ing was all PRs must be green before merging to master, the sync PR from master to release branch must be green before merging, and the release branch must be green before triggering a beta / release build.
This isn't a criticism of Eric's work here because I think aspects of the Travis tests have been broken for a while now. Be good to get them back on track, ideally before considering 12.0 release.
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: About NetBeans build quality
Posted by Neil C Smith <ne...@apache.org>.
On Fri, 1 May 2020 at 09:32, Benjamin Graf <be...@gmx.net> wrote:
> IMHO we all should have a look and taking more care after our PRs being green on Travis, too. Sometimes periodically rebasing against master does help getting green again if build is broken because of Travis bugs or someone else broken commits.
+1 And sometimes just retriggering Travis gets things working - to
pick up fixes in master, and there are a few spurious failures at
times unfortunately.
My general approach when RM'ing was all PRs must be green before
merging to master, the sync PR from master to release branch must be
green before merging, and the release branch must be green before
triggering a beta / release build.
This isn't a criticism of Eric's work here because I think aspects of
the Travis tests have been broken for a while now. Be good to get
them back on track, ideally before considering 12.0 release.
Best wishes,
Neil
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists