You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@kudu.apache.org by "Mahesh Reddy (Code Review)" <ge...@cloudera.org> on 2023/05/04 21:53:11 UTC
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Mahesh Reddy has uploaded this change for review. ( http://gerrit.cloudera.org:8080/19842
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
[tests] Fix flakiness in log-rolling-itest.cc
The test TestLogCleanupOnStartup within log-rolling-itest.cc
has been flaky with a 30% failure rate for the last 2+ weeks.
The timeout within the test was increased from 30 to 60 seconds
to allow the ExcessLogFileDeleterThread to have enough time to
properly clean up the log files.
Below are the dist-test stats for a DEBUG build.
Before: 13/100 failed.
http://dist-test.cloudera.org/job?job_id=mreddy.1683141191.148826
After: 0/100 failed.
http://dist-test.cloudera.org/job?job_id=mreddy.1683146152.19719
Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
---
M src/kudu/integration-tests/log-rolling-itest.cc
1 file changed, 4 insertions(+), 2 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/kudu refs/changes/42/19842/1
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1:
> Patch Set 1:
>
> > Patch Set 1:
> >
> > (1 comment)
>
> Running git bisect now, I'll update this patch with my findings.
Finished running git bisect, this commit: https://github.com/apache/kudu/commit/555854178b9b498701619f4bb0dbbbbeab8e69e7
was the first bad commit where the test started to become flaky. Looks like there were some changes in ServerBase::Init() that may have caused the initialization process to be longer. For context, the thread that cleans up the excess log files is ServerBase::StartExcessGlogDeleterThread().
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Mon, 08 May 2023 20:12:22 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/19842/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/19842/1//COMMIT_MSG@10
PS1, Line 10: with a 30% failure rate for the last 2+ weeks
Thank you for the fix.
What was the reason for the increase of the failure rate? I don't recall this test failing in March or earlier.
Just curious why it's now necessary to have 60 seconds timeout when it was enough to have just 30.
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Thu, 04 May 2023 22:43:50 +0000
Gerrit-HasComments: Yes
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1:
> > > Patch Set 1:
> > >
> > > > Patch Set 1:
> > > >
> > > > (1 comment)
> > >
> > > Running git bisect now, I'll update this patch with my
> findings.
> >
> > Finished running git bisect, this commit: https://github.com/apache/kudu/commit/555854178b9b498701619f4bb0dbbbbeab8e69e7
> >
> > was the first bad commit where the test started to become flaky.
> > Looks like there were some changes in ServerBase::Init() that may
> > have caused the initialization process to be longer. For context,
> > the thread that cleans up the excess log files is
> ServerBase::StartExcessGlogDeleterThread().
>
> Interesting: that doesn't seem to be the case, at least in my
> testing.
>
> I did a test run of the log-rolling-itest for the git commit
> 555854178 and commit just before. In both cases, all 256 tests
> passed for both commits, so my testing show that 555854178 isn't
> the changelist that introduced the issue.
>
> Test run for b00431df6:
> http://dist-test.cloudera.org//job?job_id=aserbin.1683691780.85751
>
> Test run for 555854178:
> http://dist-test.cloudera.org//job?job_id=aserbin.1683695402.105720
And running the test with the very latest sources is also passing in my case, interesting:
http://dist-test.cloudera.org//job?job_id=aserbin.1683697731.120325
I was running those tests using the following command:
../../build-support/dist_test.py loop -n 256 -- $test_fpath --stress_cpu_threads=16 $@
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 May 2023 05:56:28 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1:
> Patch Set 1:
>
> > > > Patch Set 1:
> > > >
> > > > > Patch Set 1:
> > > > >
> > > > > (1 comment)
> > > >
> > > > Running git bisect now, I'll update this patch with my
> > findings.
> > >
> > > Finished running git bisect, this commit: https://github.com/apache/kudu/commit/555854178b9b498701619f4bb0dbbbbeab8e69e7
> > >
> > > was the first bad commit where the test started to become flaky.
> > > Looks like there were some changes in ServerBase::Init() that may
> > > have caused the initialization process to be longer. For context,
> > > the thread that cleans up the excess log files is
> > ServerBase::StartExcessGlogDeleterThread().
> >
> > Interesting: that doesn't seem to be the case, at least in my
> > testing.
> >
> > I did a test run of the log-rolling-itest for the git commit
> > 555854178 and commit just before. In both cases, all 256 tests
> > passed for both commits, so my testing show that 555854178 isn't
> > the changelist that introduced the issue.
> >
> > Test run for b00431df6:
> > http://dist-test.cloudera.org//job?job_id=aserbin.1683691780.85751
> >
> > Test run for 555854178:
> > http://dist-test.cloudera.org//job?job_id=aserbin.1683695402.105720
>
> And running the test with the very latest sources is also passing in my case, interesting:
>
> http://dist-test.cloudera.org//job?job_id=aserbin.1683697731.120325
>
> I was running those tests using the following command:
>
> ../../build-support/dist_test.py loop -n 256 -- $test_fpath --stress_cpu_threads=16 $@
Interesting, what's the default value for --stress_cpu_threads? Could that parameter being set to 16 affect the result of this test? For context, the command I ran was
../../build-support/dist_test.py loop -n 100 ./bin/log-rolling-itest
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 May 2023 20:34:28 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Mahesh Reddy (Code Review)" <ge...@cloudera.org>.
Mahesh Reddy has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1:
> Patch Set 1:
>
> (1 comment)
Running git bisect now, I'll update this patch with my findings.
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Fri, 05 May 2023 20:22:51 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
[tests] Fix flakiness in log-rolling-itest.cc
The test TestLogCleanupOnStartup within log-rolling-itest.cc
has been flaky with a 30% failure rate for the last 2+ weeks.
The timeout within the test was increased from 30 to 60 seconds
to allow the ExcessLogFileDeleterThread to have enough time to
properly clean up the log files.
Below are the dist-test stats for a DEBUG build.
Before: 13/100 failed.
http://dist-test.cloudera.org/job?job_id=mreddy.1683141191.148826
After: 0/100 failed.
http://dist-test.cloudera.org/job?job_id=mreddy.1683146152.19719
Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Reviewed-on: http://gerrit.cloudera.org:8080/19842
Tested-by: Kudu Jenkins
Reviewed-by: Alexey Serbin <al...@apache.org>
---
M src/kudu/integration-tests/log-rolling-itest.cc
1 file changed, 4 insertions(+), 2 deletions(-)
Approvals:
Kudu Jenkins: Verified
Alexey Serbin: Looks good to me, approved
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 2
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 2:
> > Patch Set 1:
> >
> > > > > Patch Set 1:
> > > > >
> > > > > > Patch Set 1:
> > > > > >
> > > > > > (1 comment)
> > > > >
> > > > > Running git bisect now, I'll update this patch with my
> > > findings.
> > > >
> > > > Finished running git bisect, this commit: https://github.com/apache/kudu/commit/555854178b9b498701619f4bb0dbbbbeab8e69e7
> > > >
> > > > was the first bad commit where the test started to become
> flaky.
> > > > Looks like there were some changes in ServerBase::Init()
> that may
> > > > have caused the initialization process to be longer. For
> context,
> > > > the thread that cleans up the excess log files is
> > > ServerBase::StartExcessGlogDeleterThread().
> > >
> > > Interesting: that doesn't seem to be the case, at least in my
> > > testing.
> > >
> > > I did a test run of the log-rolling-itest for the git commit
> > > 555854178 and commit just before. In both cases, all 256
> tests
> > > passed for both commits, so my testing show that 555854178
> isn't
> > > the changelist that introduced the issue.
> > >
> > > Test run for b00431df6:
> > > http://dist-test.cloudera.org//job?job_id=aserbin.1683691780.85751
> > >
> > > Test run for 555854178:
> > > http://dist-test.cloudera.org//job?job_id=aserbin.1683695402.105720
> >
> > And running the test with the very latest sources is also passing
> in my case, interesting:
> >
> > http://dist-test.cloudera.org//job?job_id=aserbin.1683697731.120325
> >
> > I was running those tests using the following command:
> >
> > ../../build-support/dist_test.py loop -n 256 -- $test_fpath
> --stress_cpu_threads=16 $@
>
> Interesting, what's the default value for --stress_cpu_threads?
> Could that parameter being set to 16 affect the result of this
> test? For context, the command I ran was
> ../../build-support/dist_test.py loop -n 100 ./bin/log-rolling-itest
I guess there original timeout (30) seconds was in the grey area, and since the changelist 555854178 introduced an extra check for time source (that might add up to 1 extra second to the startup time), the ASSERT_EVENTUALLY started failing.
At least, introducing this fix doesn't hurt, and allows to have less flakiness in the test.
Thank you for the fix!
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 2
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 May 2023 16:10:40 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1:
> > Patch Set 1:
> >
> > > Patch Set 1:
> > >
> > > (1 comment)
> >
> > Running git bisect now, I'll update this patch with my findings.
>
> Finished running git bisect, this commit: https://github.com/apache/kudu/commit/555854178b9b498701619f4bb0dbbbbeab8e69e7
>
> was the first bad commit where the test started to become flaky.
> Looks like there were some changes in ServerBase::Init() that may
> have caused the initialization process to be longer. For context,
> the thread that cleans up the excess log files is ServerBase::StartExcessGlogDeleterThread().
Interesting: that doesn't seem to be the case, at least in my testing.
I did a test run of the log-rolling-itest for the git commit 555854178 and commit just before. In both cases, all 256 tests passed for both commits, so my testing show that 555854178 isn't the changelist that introduced the issue.
Test run for b00431df6:
http://dist-test.cloudera.org//job?job_id=aserbin.1683691780.85751
Test run for 555854178:
http://dist-test.cloudera.org//job?job_id=aserbin.1683695402.105720
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 May 2023 05:42:38 +0000
Gerrit-HasComments: No
[kudu-CR] [tests] Fix flakiness in log-rolling-itest.cc
Posted by "Alexey Serbin (Code Review)" <ge...@cloudera.org>.
Alexey Serbin has posted comments on this change. ( http://gerrit.cloudera.org:8080/19842 )
Change subject: [tests] Fix flakiness in log-rolling-itest.cc
......................................................................
Patch Set 1: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/19842
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: kudu
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I4d33559fa01c26d8c8b3ad12b8822d2cdd760fae
Gerrit-Change-Number: 19842
Gerrit-PatchSet: 1
Gerrit-Owner: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Reviewer: Abhishek Chennaka <ac...@cloudera.com>
Gerrit-Reviewer: Alexey Serbin <al...@apache.org>
Gerrit-Reviewer: Kudu Jenkins (120)
Gerrit-Reviewer: Mahesh Reddy <mr...@cloudera.com>
Gerrit-Comment-Date: Mon, 22 May 2023 16:07:21 +0000
Gerrit-HasComments: No