You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@beam.apache.org by Alex Amato <aj...@google.com> on 2020/06/30 17:47:18 UTC
Is there an easy way to figure out why my build failed?
Often I see the build failing, but on the next page there are no warnings
and no errors.
Then when you dive into the full log, it slows down the browser and there
is no obvious ctrl-f keyword to find the error ("error" yields over 100
results, and the error isn't always at the bottom). Is there a
faster/better way to do it?
There is a log about the build timing out, but I don't really know what
timed out or where to look next.
Is 120 min a long enough time? Did something recently happen? If so Can we
increase the timeout until we debug the regression?
https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/
https://issues.apache.org/jira/browse/BEAM-10390
Thanks, I would appreciate any ideas :)
Alex
Re: Is there an easy way to figure out why my build failed?
Posted by Luke Cwik <lc...@google.com>.
I think --stacktrace prints gradle task stack traces and not the stack
trace of things like tests. If I'm wrong and it provides additional details
that are useful, feel free to add it.
Finally, in this case, the test was getting stuck and waiting forever and
the tests stacktrace was never being shown.
On Tue, Jul 7, 2020 at 9:09 AM Tyson Hamilton <ty...@google.com> wrote:
> Is there a reason we don't always run with the --stacktrace flag? Having
> this enabled all the time could be useful and wouldn't add any additional
> output for successful runs (right?).
>
> On Tue, Jun 30, 2020 at 1:23 PM Luke Cwik <lc...@google.com> wrote:
>
>> This rabbitmq:test stuckness has been a common pain point for me
>> recently. I did some investigation and found one issue with the test and
>> also added a class level timeout for the test. Opened up
>> https://github.com/apache/beam/pull/12137
>>
>> On Tue, Jun 30, 2020 at 11:22 AM Maximilian Michels <mx...@apache.org>
>> wrote:
>>
>>> Hi Alex,
>>>
>>> Fully agree with you that it can be hard to find the cause for a failing
>>> build. You basically need to know the exact keyword to grep for. The
>>> reason is that Jenkins does not understand all build logs to display the
>>> error directly in the UI.
>>>
>>> I often do the following for large logs:
>>>
>>> $ curl
>>>
>>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/consoleText
>>> | less
>>>
>>> Then I can use '/' to search in the log quickly without my browser
>>> slowing down.
>>>
>>> In the linked build log, I searched for ' FAILED':
>>>
>>> 09:18:26 > Task :sdks:java:io:rabbitmq:test FAILED
>>> 09:18:26
>>> 09:18:26 FAILURE: Build failed with an exception.
>>> 09:18:26
>>> 09:18:26 * What went wrong:
>>> 09:18:26 Execution failed for task ':sdks:java:io:rabbitmq:test'.
>>> 09:18:26 > Process 'Gradle Test Executor 110' finished with non-zero
>>> exit value 143
>>> 09:18:26 This problem might be caused by incorrect test process
>>> configuration.
>>> 09:18:26 Please refer to the test execution section in the User
>>> Manual at
>>>
>>> https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
>>>
>>> Now, it appears that the rabbitmq tests are timing out but I'm not sure
>>> the issue if with rabbitmq because I'm also seeing:
>>>
>>> Build timed out (after 120 minutes). Marking the build as aborted.
>>> Build was aborted
>>> Recording test results
>>>
>>> So maybe some other test slowed down the build and when it reached
>>> rabbitmq it was killed. That can probably tested by running the build
>>> multiple times.
>>>
>>> -Max
>>>
>>> On 30.06.20 19:47, Alex Amato wrote:
>>> > Often I see the build failing, but on the next page there are no
>>> > warnings and no errors.
>>> >
>>> > Then when you dive into the full log, it slows down the browser and
>>> > there is no obvious ctrl-f keyword to find the error ("error" yields
>>> > over 100 results, and the error isn't always at the bottom). Is there
>>> a
>>> > faster/better way to do it?
>>> >
>>> > There is a log about the build timing out, but I don't really know
>>> what
>>> > timed out or where to look next.
>>> >
>>> > Is 120 min a long enough time? Did something recently happen? If so
>>> Can
>>> > we increase the timeout until we debug the regression?
>>> >
>>> > https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/
>>> >
>>> > https://issues.apache.org/jira/browse/BEAM-10390
>>> >
>>> > Thanks, I would appreciate any ideas :)
>>> > Alex
>>>
>>
Re: Is there an easy way to figure out why my build failed?
Posted by Tyson Hamilton <ty...@google.com>.
Is there a reason we don't always run with the --stacktrace flag? Having
this enabled all the time could be useful and wouldn't add any additional
output for successful runs (right?).
On Tue, Jun 30, 2020 at 1:23 PM Luke Cwik <lc...@google.com> wrote:
> This rabbitmq:test stuckness has been a common pain point for me recently.
> I did some investigation and found one issue with the test and also added a
> class level timeout for the test. Opened up
> https://github.com/apache/beam/pull/12137
>
> On Tue, Jun 30, 2020 at 11:22 AM Maximilian Michels <mx...@apache.org>
> wrote:
>
>> Hi Alex,
>>
>> Fully agree with you that it can be hard to find the cause for a failing
>> build. You basically need to know the exact keyword to grep for. The
>> reason is that Jenkins does not understand all build logs to display the
>> error directly in the UI.
>>
>> I often do the following for large logs:
>>
>> $ curl
>>
>> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/consoleText
>> | less
>>
>> Then I can use '/' to search in the log quickly without my browser
>> slowing down.
>>
>> In the linked build log, I searched for ' FAILED':
>>
>> 09:18:26 > Task :sdks:java:io:rabbitmq:test FAILED
>> 09:18:26
>> 09:18:26 FAILURE: Build failed with an exception.
>> 09:18:26
>> 09:18:26 * What went wrong:
>> 09:18:26 Execution failed for task ':sdks:java:io:rabbitmq:test'.
>> 09:18:26 > Process 'Gradle Test Executor 110' finished with non-zero
>> exit value 143
>> 09:18:26 This problem might be caused by incorrect test process
>> configuration.
>> 09:18:26 Please refer to the test execution section in the User
>> Manual at
>>
>> https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
>>
>> Now, it appears that the rabbitmq tests are timing out but I'm not sure
>> the issue if with rabbitmq because I'm also seeing:
>>
>> Build timed out (after 120 minutes). Marking the build as aborted.
>> Build was aborted
>> Recording test results
>>
>> So maybe some other test slowed down the build and when it reached
>> rabbitmq it was killed. That can probably tested by running the build
>> multiple times.
>>
>> -Max
>>
>> On 30.06.20 19:47, Alex Amato wrote:
>> > Often I see the build failing, but on the next page there are no
>> > warnings and no errors.
>> >
>> > Then when you dive into the full log, it slows down the browser and
>> > there is no obvious ctrl-f keyword to find the error ("error" yields
>> > over 100 results, and the error isn't always at the bottom). Is there a
>> > faster/better way to do it?
>> >
>> > There is a log about the build timing out, but I don't really know what
>> > timed out or where to look next.
>> >
>> > Is 120 min a long enough time? Did something recently happen? If so Can
>> > we increase the timeout until we debug the regression?
>> >
>> > https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/
>> >
>> > https://issues.apache.org/jira/browse/BEAM-10390
>> >
>> > Thanks, I would appreciate any ideas :)
>> > Alex
>>
>
Re: Is there an easy way to figure out why my build failed?
Posted by Luke Cwik <lc...@google.com>.
This rabbitmq:test stuckness has been a common pain point for me recently.
I did some investigation and found one issue with the test and also added a
class level timeout for the test. Opened up
https://github.com/apache/beam/pull/12137
On Tue, Jun 30, 2020 at 11:22 AM Maximilian Michels <mx...@apache.org> wrote:
> Hi Alex,
>
> Fully agree with you that it can be hard to find the cause for a failing
> build. You basically need to know the exact keyword to grep for. The
> reason is that Jenkins does not understand all build logs to display the
> error directly in the UI.
>
> I often do the following for large logs:
>
> $ curl
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/consoleText
> | less
>
> Then I can use '/' to search in the log quickly without my browser
> slowing down.
>
> In the linked build log, I searched for ' FAILED':
>
> 09:18:26 > Task :sdks:java:io:rabbitmq:test FAILED
> 09:18:26
> 09:18:26 FAILURE: Build failed with an exception.
> 09:18:26
> 09:18:26 * What went wrong:
> 09:18:26 Execution failed for task ':sdks:java:io:rabbitmq:test'.
> 09:18:26 > Process 'Gradle Test Executor 110' finished with non-zero
> exit value 143
> 09:18:26 This problem might be caused by incorrect test process
> configuration.
> 09:18:26 Please refer to the test execution section in the User
> Manual at
>
> https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
>
> Now, it appears that the rabbitmq tests are timing out but I'm not sure
> the issue if with rabbitmq because I'm also seeing:
>
> Build timed out (after 120 minutes). Marking the build as aborted.
> Build was aborted
> Recording test results
>
> So maybe some other test slowed down the build and when it reached
> rabbitmq it was killed. That can probably tested by running the build
> multiple times.
>
> -Max
>
> On 30.06.20 19:47, Alex Amato wrote:
> > Often I see the build failing, but on the next page there are no
> > warnings and no errors.
> >
> > Then when you dive into the full log, it slows down the browser and
> > there is no obvious ctrl-f keyword to find the error ("error" yields
> > over 100 results, and the error isn't always at the bottom). Is there a
> > faster/better way to do it?
> >
> > There is a log about the build timing out, but I don't really know what
> > timed out or where to look next.
> >
> > Is 120 min a long enough time? Did something recently happen? If so Can
> > we increase the timeout until we debug the regression?
> >
> > https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/
> >
> > https://issues.apache.org/jira/browse/BEAM-10390
> >
> > Thanks, I would appreciate any ideas :)
> > Alex
>
Re: Is there an easy way to figure out why my build failed?
Posted by Maximilian Michels <mx...@apache.org>.
Hi Alex,
Fully agree with you that it can be hard to find the cause for a failing
build. You basically need to know the exact keyword to grep for. The
reason is that Jenkins does not understand all build logs to display the
error directly in the UI.
I often do the following for large logs:
$ curl
https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/consoleText
| less
Then I can use '/' to search in the log quickly without my browser
slowing down.
In the linked build log, I searched for ' FAILED':
09:18:26 > Task :sdks:java:io:rabbitmq:test FAILED
09:18:26
09:18:26 FAILURE: Build failed with an exception.
09:18:26
09:18:26 * What went wrong:
09:18:26 Execution failed for task ':sdks:java:io:rabbitmq:test'.
09:18:26 > Process 'Gradle Test Executor 110' finished with non-zero
exit value 143
09:18:26 This problem might be caused by incorrect test process
configuration.
09:18:26 Please refer to the test execution section in the User
Manual at
https://docs.gradle.org/5.2.1/userguide/java_testing.html#sec:test_execution
Now, it appears that the rabbitmq tests are timing out but I'm not sure
the issue if with rabbitmq because I'm also seeing:
Build timed out (after 120 minutes). Marking the build as aborted.
Build was aborted
Recording test results
So maybe some other test slowed down the build and when it reached
rabbitmq it was killed. That can probably tested by running the build
multiple times.
-Max
On 30.06.20 19:47, Alex Amato wrote:
> Often I see the build failing, but on the next page there are no
> warnings and no errors.
>
> Then when you dive into the full log, it slows down the browser and
> there is no obvious ctrl-f keyword to find the error ("error" yields
> over 100 results, and the error isn't always at the bottom). Is there a
> faster/better way to do it?
>
> There is a log about the build timing out, but I don't really know what
> timed out or where to look next.
>
> Is 120 min a long enough time? Did something recently happen? If so Can
> we increase the timeout until we debug the regression?
>
> https://ci-beam.apache.org/job/beam_PreCommit_Java_Commit/12017/
>
> https://issues.apache.org/jira/browse/BEAM-10390
>
> Thanks, I would appreciate any ideas :)
> Alex