You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Riza Suminto (Code Review)" <ge...@cloudera.org> on 2021/02/10 07:33:25 UTC
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Riza Suminto has uploaded this change for review. ( http://gerrit.cloudera.org:8080/17054
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
test_no_fd_caching_on_cached_data has been flaky for not having all of
the data fully cached in warm up phase. This patch fix the test by:
1. Reduce the number of rows written to table cachefd.simple.
2. Repeat the warm up query 5 times.
3. Lower the cluster size to 1.
Testing:
- Loop the test manually 100 times and see no more failures.
Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
---
M tests/custom_cluster/test_hdfs_fd_caching.py
1 file changed, 31 insertions(+), 20 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/17054/1
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 5: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 00:18:46 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17054/2/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/17054/2/tests/custom_cluster/test_hdfs_fd_caching.py@168
PS2, Line 168: 500MB",
> Under the covers, custom cluster test uses bin/start-impala-cluster.py and
Done. Thanks for pointing out this solution.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 21:10:58 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Bikramjeet Vig, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17054
to look at the new patch set (#3).
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
test_no_fd_caching_on_cached_data has been flaky for not having all of
the data fully cached in the warm-up phase. There is a limit on
concurrency in writing to the cache such that we may fail to cache data
the first time read it. This patch fixes the test by repeating the
warm-up query 5 times. This patch also reduce the test cluster size to
just 1 to avoid potential flakiness from multiple impalad racing to
delete the same old fata cache files.
Testing:
- Loop the test manually 100 times and see no more failures.
Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
---
M tests/custom_cluster/test_hdfs_fd_caching.py
1 file changed, 8 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/17054/3
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 5:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6891/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 00:18:47 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 3:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17054/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17054/3//COMMIT_MSG@13
PS3, Line 13: This patch also reduce the test cluster size to
: just 1 to avoid potential flakiness from multiple impalad racing to
: delete the same old fata cache files.
Oh, can you revise this statement?
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 21:30:45 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
test_no_fd_caching_on_cached_data has been flaky for not having all of
the data fully cached in the warm-up phase. There is a limit on
concurrency in writing to the cache such that we may fail to cache data
the first time read it. This patch fixes the test by repeating the
warm-up query 5 times. This patch also add a proper start_args to the
test so that each impalad will write their data cache file in their own
directory.
Testing:
- Loop the test manually 100 times and see no more failures.
Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Reviewed-on: http://gerrit.cloudera.org:8080/17054
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M tests/custom_cluster/test_hdfs_fd_caching.py
1 file changed, 8 insertions(+), 5 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 6
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Bikramjeet Vig, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17054
to look at the new patch set (#4).
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
test_no_fd_caching_on_cached_data has been flaky for not having all of
the data fully cached in the warm-up phase. There is a limit on
concurrency in writing to the cache such that we may fail to cache data
the first time read it. This patch fixes the test by repeating the
warm-up query 5 times. This patch also add a proper start_args to the
test so that each impalad will write their data cache file in their own
directory.
Testing:
- Loop the test manually 100 times and see no more failures.
Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
---
M tests/custom_cluster/test_hdfs_fd_caching.py
1 file changed, 8 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/17054/4
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 3:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8141/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 21:29:14 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8113/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Feb 2021 07:53:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 4:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17054/3//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/17054/3//COMMIT_MSG@13
PS3, Line 13: This patch also add a proper start_args to the
: test so that each impalad will write their data cache file in their own
: directory.
> Oh, can you revise this statement?
Done. Thanks for catching this!
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 22:10:50 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/8125/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Feb 2021 03:19:06 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Hello Joe McDonnell, Bikramjeet Vig, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/17054
to look at the new patch set (#2).
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
test_no_fd_caching_on_cached_data has been flaky for not having all of
the data fully cached in the warm-up phase. There is a limit on
concurrency in writing to the cache such that we may fail to cache data
the first time read it. This patch fixes the test by repeating the
warm-up query 5 times. This patch also reduce the test cluster size to
just 1 to avoid potential flakiness from multiple impalad racing to
delete the same old fata cache files.
Testing:
- Loop the test manually 100 times and see no more failures.
Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
---
M tests/custom_cluster/test_hdfs_fd_caching.py
1 file changed, 7 insertions(+), 5 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/54/17054/2
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 3: Code-Review+2
Thanks for fixing this!
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 3
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 21:29:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/17054/2/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/17054/2/tests/custom_cluster/test_hdfs_fd_caching.py@168
PS2, Line 168: cluster_size=1
Under the covers, custom cluster test uses bin/start-impala-cluster.py and that script has an option for turning on the data cache (separate from impalad_args). It gives a different directory to each impalad. To use that, you can specify those arguments via start_args and omit the corresponding impalad_args. So, I think we can do something similar to what test_data_cache.py does and use this:
impalads_args="--max_cached_file_handles=16 --unused_file_handle_timeout_sec=5 " +
"--always_use_data_cache=true"
start_args="--data_cache_dir=/tmp --data_cache_size=500MB"
and then leave the rest as-is.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 20:30:22 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 1:
(1 comment)
Thanks Joe for the feedback.
I will revert most of these and leave only the extra warm up part.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@190
PS1, Line 190: cluster_size=1
> I think this should not impact the caching. Our scheduling algorithm maps f
I will remove this so the cluster size back to default.
Previously, when I kept cluster size as default (3), after several iteration of looping this test, one impalad failed with the following error.
I0210 17:50:00.781529 10340 mem_tracker.cc:83] Creating tracker /tmp-sharded_lru_cache->root
I0210 17:50:00.782321 10340 status.cc:129] Failed to delete old cache file /tmp/impala-cache-file-fc4060e3ab7e45b0:a680f0d3b05005a1: Not found: /tmp/impala-cache-file-fc4060e3ab7e45b0:a680f0d3b05005a1: No such file or directory (error 2)
@ 0x2c008d7 _ZN6impala13GetStackTraceB5cxx11Ev
@ 0x1c9003e impala::Status::Status()
@ 0x36016f9 impala::io::DataCache::Partition::DeleteExistingFiles()
@ 0x3601e22 impala::io::DataCache::Partition::Init()
@ 0x360975e impala::io::DataCache::Init()
@ 0x35b4db0 impala::io::DiskIoMgr::Init()
@ 0x251b229 impala::ExecEnv::Init()
@ 0x29a0412 ImpaladMain()
@ 0x1bdf4d3 main
@ 0x7faa8e3bebf7 __libc_start_main
@ 0x1ae81da _start
I0210 17:50:00.897989 10340 mem_tracker.cc:87] Destroying tracker /tmp-sharded_lru_cache->root
I was guessing that since all three write their data cache to common /tmp/ dir, one might accidentally delete cache file that belong to other impalad. Just running the test one time doesn't reveal this error.
So in this patchset 1, I follow examples from custom_cluster/test_data_cache.py to set custom_cluster=1.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Feb 2021 02:18:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 1:
(5 comments)
Thanks for looking into this.
My main thought here is that I think the change may be bigger than it needs to be. My theory is that the "Repeat the warm up query 5 times" is the part of the change that makes it work and it may not need the other parts. I would be interested in whether a stripped down version of this with only that part of the change would fix the issue.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@179
PS1, Line 179: TestHdfsFdCaching
One thing to watch out for here is if you inherit from TestHdfsFdCaching, you may inherit all of its test cases (e.g. test_caching_with_eviction()). I'm not sure how pytest deals with that.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@184
PS1, Line 184: NUM_ROWS = 10
I believe that the number of rows should not impact the behavior. The differences between 10 and 100 is small enough that it shouldn't impact the IO pattern. They are all in a single parquet files, and we would do an IO for the footer and one IO for each column.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@190
PS1, Line 190: cluster_size=1
I think this should not impact the caching. Our scheduling algorithm maps files to nodes consistently, so each node should be reading the same file each time.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@207
PS1, Line 207: Read 5 times to make
: # sure the data cache is fully warmed up.
: for x in range(5):
This makes a lot of sense to me as the explanation. There are times when data won't be written to the cache. In particular, there is a limit on concurrency in writing to the cache. I'm guessing we could hit that and fail to cache something the first time around.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@224
PS1, Line 224: for x in range(5):
I'm fine with either value here, but I also don't see this changing the behavior of the test.
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 1
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 10 Feb 2021 22:59:41 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 4: Code-Review+2
Carry +2
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 4
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Feb 2021 22:20:45 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 5: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 5
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Wed, 17 Feb 2021 05:57:20 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10497: Fix flakiness in test no fd caching on cached data.
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/17054 )
Change subject: IMPALA-10497: Fix flakiness in test_no_fd_caching_on_cached_data.
......................................................................
Patch Set 2:
(5 comments)
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@179
PS1, Line 179: le handles cached
> One thing to watch out for here is if you inherit from TestHdfsFdCaching, y
Done
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@184
PS1, Line 184: # handle ca
> I believe that the number of rows should not impact the behavior. The diffe
Done
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@190
PS1, Line 190:
> I will remove this so the cluster size back to default.
I change my mind, I think we should keep cluster_size=1 here.
On start up, impalad delete the old cache file by listing file matching CACHE_FILE_PREFIX and delete them one by one.
https://github.com/apache/impala/blob/b42c649/be/src/runtime/io/data-cache.cc#L415
So if cluster_size > 1, multiple impalad might race to delete the same old cache file and crash the losing ones.
Also, setting cluster_size=1 does not change the behavior being tested by this test, so I think it should be okay.
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@207
PS1, Line 207:
: def cached_handles(self):
: return self.get_ag
> This makes a lot of sense to me as the explanation. There are times when da
Done
http://gerrit.cloudera.org:8080/#/c/17054/1/tests/custom_cluster/test_hdfs_fd_caching.py@224
PS1, Line 224:
> I'm fine with either value here, but I also don't see this changing the beh
Done
--
To view, visit http://gerrit.cloudera.org:8080/17054
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I774f9dfea7dcc107c3c7f2b76db3aaf4b2dd7952
Gerrit-Change-Number: 17054
Gerrit-PatchSet: 2
Gerrit-Owner: Riza Suminto <ri...@cloudera.com>
Gerrit-Reviewer: Bikramjeet Vig <bi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Thu, 11 Feb 2021 03:09:44 +0000
Gerrit-HasComments: Yes