You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@netbeans.apache.org by Jaroslav Tulach <ja...@gmail.com> on 2022/06/06 07:05:39 UTC
Re: Kill ValidateLayerConsistencyTest?
Removing unstable tests from CI is proper attitude.
There is a value in `ValidateLayerConsistencyTest` - the problems it detects
(when working properly) would manifest themselves in incorrect runtime
behavior.
Up to everyone to judge what is more valuable - stable CI or stable runtime?
In this case I'd just restart the CI job, but I understand your frustration,
Matthias.
> 2022-05-29T17:06:31.9092439Z [junit] java.lang.AssertionError: Has to be
> NbRepository: org.openide.filesystems.Repository@744b325b
> 2022-05-29T17:06:31.9093232Z [junit] at
> org.netbeans.core.startup.Main.start(Main.java:298)
> 2022-05-29T17:06:31.9094120Z [junit] at
> org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:98)
> 2022-05-29T17:06:31.9094571Z [junit] at
> java.lang.Thread.run(Thread.java:750) 2022-05-29T17:06:31.9095235Z
> [junit]
Isn't this caused by some race condition? Temporarily adding `this.when = new
Exception("initialized")` into `Repository` constructor and printing the
stacktraces when the `AssertionError` happens might tell us who's responsible
for the misconfiguration of the test.
-jt
Dne neděle 29. května 2022 19:19:42 CEST, Matthias Bläsing napsal(a):
> Hi all,
>
> for me the testing of NetBeans shows its ugly dark side and I'm
> seriously thinking about killing the ValidateLayerConsistencyTest. It
> is flaky and causes untraceable errors.
>
> This is the PR:
>
> https://github.com/apache/netbeans/pull/4058
>
> The flakiness can be observed when comparing the commit validation
> runs. 1 fails whiel 3 work ok. In this instance the failing instance is
> JDK8, but I saw others.
>
> I use this PR to try to fix this:
>
> https://github.com/apache/netbeans/tree/try_fix_commit_validation2
>
> One problem could be tracked down, that the maven module has a entry in
> the configuration, that is a template, but is checked for instablility.
> Fixed here:
>
> https://github.com/apache/netbeans/commit/40a60e311000977b490935c6bf762464d3
> ddaa96
>
> Currently it fails with an error that also seems to be a race condition
> (line 23766-2769):
>
> https://pipelines.actions.githubusercontent.com/serviceHosts/3cf2747f-b5d6-4
> e8e-a999-150ff8ae61e1/_apis/pipelines/1/runs/14001/signedlogcontent/7?urlExp
> ires=2022-05-29T17%3A16%3A59.2143803Z&urlSigningMethod=HMACV1&urlSignature=f
> %2B%2BJVlE0n3fHCqA4wJkq4iKPdD%2B502hC2csZJFs5keE%3D
>
> 2022-05-29T17:06:31.9092439Z [junit] java.lang.AssertionError: Has to be
> NbRepository: org.openide.filesystems.Repository@744b325b
> 2022-05-29T17:06:31.9093232Z [junit] at
> org.netbeans.core.startup.Main.start(Main.java:298)
> 2022-05-29T17:06:31.9094120Z [junit] at
> org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:98)
> 2022-05-29T17:06:31.9094571Z [junit] at
> java.lang.Thread.run(Thread.java:750) 2022-05-29T17:06:31.9095235Z
> [junit]
> ---------------------------------------------------------------------------
> ---- 2022-05-29T17:06:31.9095602Z [junit] >Log Session: Sunday, May 29,
> 2022 5:06:31 PM UTC 2022-05-29T17:06:31.9095875Z [junit] >System Info:
> 2022-05-29T17:06:31.9096375Z [junit] Product Version = Apache
> NetBeans Platform Dev (Build dev-40a60e311000977b490935c6bf762464d3ddaa96)
>
> Given that I changed code in the JS area, this does not seem to related
> to my changes.
>
> So at this point this Test does harm and so is a candidate to be
> removed.
>
> I would like to fix this tough, but I put literally days into this and
> am at the end of my ideas. Anyone wants to chime in?
>
> Greetings
>
> Matthias
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Kill ValidateLayerConsistencyTest?
Posted by Michael Bien <mb...@gmail.com>.
sure, we can run like this for a while but it will be painful (for
everyone) to have often failing tests in gh actions additionally to
travis. It can also cause other issues. If someone isn't restarting them
we might max out our artifact storage. And if they are not restarted
within a day, we have to restart EVERYTHING to get new build artifacts.
We are lucky here that the job itself is short. If it would be in a 1h+
job that would be a showstopper IMO.
Matthias already spent time to try to fix this, there is no lack of
trying. We don't have much room left to make tests more unreliable.
best regards,
michael
On 09.06.22 20:13, Svata Dedic wrote:
> I have very bad experience with turning off integration tests. And
> consistency of the registrations is ultimately important. We can
> easily go unnoticed to the situation when the IDE simply does not
> start because of broken layer.
>
> I've fixed similar situation in the past (after some time without
> proper tests) and would appreciate not to go through the hell again.
>
> -S.
>
>
> Dne 09. 06. 22 v 19:10 Michael Bien napsal(a):
>> On 08.06.22 20:54, Matthias Bläsing wrote:
>>> Hi again,
>>>
>>> Am Montag, dem 06.06.2022 um 09:05 +0200 schrieb Jaroslav Tulach:
>>>>> 2022-05-29T17:06:31.9092439Z [junit] java.lang.AssertionError: Has
>>>>> to be NbRepository: org.openide.filesystems.Repository@744b325b
>>>>> 2022-05-29T17:06:31.9093232Z [junit] at
>>>>> org.netbeans.core.startup.Main.start(Main.java:298)
>>>>> 2022-05-29T17:06:31.9094120Z [junit] at
>>>>> org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:98)
>>>>> 2022-05-29T17:06:31.9094571Z [junit] at
>>>>> java.lang.Thread.run(Thread.java:750) 2022-05-29T17:06:31.9095235Z
>>>>> [junit]
>>>> Isn't this caused by some race condition? Temporarily adding
>>>> `this.when = new
>>>> Exception("initialized")` into `Repository` constructor and printing
>>>> the
>>>> stacktraces when the `AssertionError` happens might tell us who's
>>>> responsible
>>>> for the misconfiguration of the test.
>>> This is a very nice way to catch this - thank you for the idea. I had
>>> to slightly alter it, but I got a stack trace.
>>>
>>> I decided to go ahead with merging the PR and see how it develops in
>>> the next few days.
>>>
>>> I filed an issue to hold information about that problem:
>>>
>>> https://github.com/apache/netbeans/issues/4211
>>>
>>> Then at least there is a record of it and it can be further analysed.
>>>
>>> Greetings
>>>
>>> Matthias
>>>
>> Hi,
>>
>> we are lucky that the test is on a job which doesn't take very long
>> to run, but It might be good to consider to disable it anyway for now.
>>
>> I had to run it 4 times here:
>>
>> https://github.com/apache/netbeans/actions/runs/2465738672
>>
>> and twice here:
>>
>> https://github.com/apache/netbeans/actions/runs/2468718859
>>
>> since this is in a 4x matrix, basically every commit will likely fail
>> and someone has to restart something.
>>
>> you can see it here:
>>
>> https://github.com/apache/netbeans/commits/master
>>
>>
>> no hurry since this is still somewhat manageable i believe, but it
>> would be better to turn it off until resolved IMO.
>>
>> best regards,
>>
>> michael
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
>> For additional commands, e-mail: dev-help@netbeans.apache.org
>>
>> For further information about the NetBeans mailing lists, visit:
>> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>>
>>
>>
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Kill ValidateLayerConsistencyTest?
Posted by antonio <an...@vieiro.net>.
Hi there,
The link you posted a while back [1] has now expired, but I recall
seeing a "synchronized(CantRememberClass.class) {}" there (two, in fact,
intermingled with what looked to me as an async code call).
Can you please post the link again?
Thanks,
Antonio
[1]
https://pipelines.actions.githubusercontent.com/serviceHosts/3cf2747f-b5d6-4e8e-a999-150ff8ae61e1/_apis/pipelines/1/runs/14001/signedlogcontent/7?urlExpires=2022-05-29T17%3A16%3A59.2143803Z&urlSigningMethod=HMACV1&urlSignature=f%2B%2BJVlE0n3fHCqA4wJkq4iKPdD%2B502hC2csZJFs5keE%3D
El 9/6/22 a las 20:31, Matthias Bläsing escribió:
> I don't even know where to start with this - maybe you have an idea
> what could happen here?
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Kill ValidateLayerConsistencyTest?
Posted by Matthias Bläsing <mb...@doppel-helix.eu>.
Hi,
Am Donnerstag, dem 09.06.2022 um 20:13 +0200 schrieb Svata Dedic:
> I have very bad experience with turning off integration tests. And
> consistency of the registrations is ultimately important. We can easily
> go unnoticed to the situation when the IDE simply does not start because
> of broken layer.
>
> I've fixed similar situation in the past (after some time without proper
> tests) and would appreciate not to go through the hell again.
I understand that, but flaky tests put these into the "just ignore
them" category in my mind. They just delay merging and raise my
annoyance level with people claiming "test-driven-development" is a
good and always working ideal.
I don't even know where to start with this - maybe you have an idea
what could happen here?
Greetings
Matthias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Kill ValidateLayerConsistencyTest?
Posted by Svata Dedic <sv...@gmail.com>.
I have very bad experience with turning off integration tests. And
consistency of the registrations is ultimately important. We can easily
go unnoticed to the situation when the IDE simply does not start because
of broken layer.
I've fixed similar situation in the past (after some time without proper
tests) and would appreciate not to go through the hell again.
-S.
Dne 09. 06. 22 v 19:10 Michael Bien napsal(a):
> On 08.06.22 20:54, Matthias Bläsing wrote:
>> Hi again,
>>
>> Am Montag, dem 06.06.2022 um 09:05 +0200 schrieb Jaroslav Tulach:
>>>> 2022-05-29T17:06:31.9092439Z [junit] java.lang.AssertionError:
>>>> Has to be NbRepository: org.openide.filesystems.Repository@744b325b
>>>> 2022-05-29T17:06:31.9093232Z [junit] at
>>>> org.netbeans.core.startup.Main.start(Main.java:298)
>>>> 2022-05-29T17:06:31.9094120Z [junit] at
>>>> org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:98)
>>>> 2022-05-29T17:06:31.9094571Z [junit] at
>>>> java.lang.Thread.run(Thread.java:750) 2022-05-29T17:06:31.9095235Z
>>>> [junit]
>>> Isn't this caused by some race condition? Temporarily adding
>>> `this.when = new
>>> Exception("initialized")` into `Repository` constructor and printing
>>> the
>>> stacktraces when the `AssertionError` happens might tell us who's
>>> responsible
>>> for the misconfiguration of the test.
>> This is a very nice way to catch this - thank you for the idea. I had
>> to slightly alter it, but I got a stack trace.
>>
>> I decided to go ahead with merging the PR and see how it develops in
>> the next few days.
>>
>> I filed an issue to hold information about that problem:
>>
>> https://github.com/apache/netbeans/issues/4211
>>
>> Then at least there is a record of it and it can be further analysed.
>>
>> Greetings
>>
>> Matthias
>>
> Hi,
>
> we are lucky that the test is on a job which doesn't take very long to
> run, but It might be good to consider to disable it anyway for now.
>
> I had to run it 4 times here:
>
> https://github.com/apache/netbeans/actions/runs/2465738672
>
> and twice here:
>
> https://github.com/apache/netbeans/actions/runs/2468718859
>
> since this is in a 4x matrix, basically every commit will likely fail
> and someone has to restart something.
>
> you can see it here:
>
> https://github.com/apache/netbeans/commits/master
>
>
> no hurry since this is still somewhat manageable i believe, but it would
> be better to turn it off until resolved IMO.
>
> best regards,
>
> michael
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
> For additional commands, e-mail: dev-help@netbeans.apache.org
>
> For further information about the NetBeans mailing lists, visit:
> https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
>
>
>
>
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Kill ValidateLayerConsistencyTest?
Posted by Michael Bien <mb...@gmail.com>.
On 08.06.22 20:54, Matthias Bläsing wrote:
> Hi again,
>
> Am Montag, dem 06.06.2022 um 09:05 +0200 schrieb Jaroslav Tulach:
>>> 2022-05-29T17:06:31.9092439Z [junit] java.lang.AssertionError: Has to be NbRepository: org.openide.filesystems.Repository@744b325b
>>> 2022-05-29T17:06:31.9093232Z [junit] at org.netbeans.core.startup.Main.start(Main.java:298)
>>> 2022-05-29T17:06:31.9094120Z [junit] at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:98)
>>> 2022-05-29T17:06:31.9094571Z [junit] at java.lang.Thread.run(Thread.java:750) 2022-05-29T17:06:31.9095235Z
>>> [junit]
>> Isn't this caused by some race condition? Temporarily adding
>> `this.when = new
>> Exception("initialized")` into `Repository` constructor and printing
>> the
>> stacktraces when the `AssertionError` happens might tell us who's
>> responsible
>> for the misconfiguration of the test.
> This is a very nice way to catch this - thank you for the idea. I had
> to slightly alter it, but I got a stack trace.
>
> I decided to go ahead with merging the PR and see how it develops in
> the next few days.
>
> I filed an issue to hold information about that problem:
>
> https://github.com/apache/netbeans/issues/4211
>
> Then at least there is a record of it and it can be further analysed.
>
> Greetings
>
> Matthias
>
Hi,
we are lucky that the test is on a job which doesn't take very long to
run, but It might be good to consider to disable it anyway for now.
I had to run it 4 times here:
https://github.com/apache/netbeans/actions/runs/2465738672
and twice here:
https://github.com/apache/netbeans/actions/runs/2468718859
since this is in a 4x matrix, basically every commit will likely fail
and someone has to restart something.
you can see it here:
https://github.com/apache/netbeans/commits/master
no hurry since this is still somewhat manageable i believe, but it would
be better to turn it off until resolved IMO.
best regards,
michael
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists
Re: Kill ValidateLayerConsistencyTest?
Posted by Matthias Bläsing <mb...@doppel-helix.eu>.
Hi again,
Am Montag, dem 06.06.2022 um 09:05 +0200 schrieb Jaroslav Tulach:
> > 2022-05-29T17:06:31.9092439Z [junit] java.lang.AssertionError: Has to be NbRepository: org.openide.filesystems.Repository@744b325b
> > 2022-05-29T17:06:31.9093232Z [junit] at org.netbeans.core.startup.Main.start(Main.java:298)
> > 2022-05-29T17:06:31.9094120Z [junit] at org.netbeans.core.startup.TopThreadGroup.run(TopThreadGroup.java:98)
> > 2022-05-29T17:06:31.9094571Z [junit] at java.lang.Thread.run(Thread.java:750) 2022-05-29T17:06:31.9095235Z
> > [junit]
>
> Isn't this caused by some race condition? Temporarily adding
> `this.when = new
> Exception("initialized")` into `Repository` constructor and printing
> the
> stacktraces when the `AssertionError` happens might tell us who's
> responsible
> for the misconfiguration of the test.
This is a very nice way to catch this - thank you for the idea. I had
to slightly alter it, but I got a stack trace.
I decided to go ahead with merging the PR and see how it develops in
the next few days.
I filed an issue to hold information about that problem:
https://github.com/apache/netbeans/issues/4211
Then at least there is a record of it and it can be further analysed.
Greetings
Matthias
---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@netbeans.apache.org
For additional commands, e-mail: dev-help@netbeans.apache.org
For further information about the NetBeans mailing lists, visit:
https://cwiki.apache.org/confluence/display/NETBEANS/Mailing+lists