You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Sahil Takiar (Code Review)" <ge...@cloudera.org> on 2020/10/01 18:33:32 UTC
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Sahil Takiar has uploaded this change for review. ( http://gerrit.cloudera.org:8080/16532
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
IMPALA-10202: Enable file handle cache for ABFS files
Like IMPALA-8428, but for ABFS, instead of S3A. Adds support for adding
ABFS file handles to the file handle cache. Support for ABFSInputStream
unbuffer operations was added in HADOOP-16859.
Ran a full table scan of a 1GB store_sales table on ABFS, made
sure the file handles were cached (validated via the runtime
profile); did this multiple times, against several different
copies of the store_sales table, in order to increase the number
of file handles cached by an impalad.
Tested:
* Tested against a ABFS storage account I have access to
Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
3 files changed, 11 insertions(+), 6 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/32/16532/1
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 00:12:32 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
Patch Set 1: Code-Review+2
This looks good to me.
If you haven't already, there are a couple sanity checks I would do:
1. Cache an ABFS file handle and then remove the file and see how it behaves. For an object store, I don't expect any weirdness. If the file is gone, I expect the read calls to eventually fail, then subsequent open calls for that file would fail.
2. Take the cache up to its default max_cached_file_handles limit (20,000) and make sure things are ok. It sounds like you did something like this already.
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Oct 2020 22:08:01 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
Patch Set 2:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/6499/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 00:12:33 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/7336/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Comment-Date: Thu, 01 Oct 2020 19:04:13 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Sahil Takiar (Code Review)" <ge...@cloudera.org>.
Sahil Takiar has posted comments on this change. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
Patch Set 1:
> This looks good to me.
>
> If you haven't already, there are a couple sanity checks I would
> do:
> 1. Cache an ABFS file handle and then remove the file and see how
> it behaves. For an object store, I don't expect any weirdness. If
> the file is gone, I expect the read calls to eventually fail, then
> subsequent open calls for that file would fail.
> 2. Take the cache up to its default max_cached_file_handles limit
> (20,000) and make sure things are ok. It sounds like you did
> something like this already.
1: I did that, seems to work as expected. The query eventually fails with a FileNotFoundException. After I run invalidate metadata, the query works again.
2: I created a table with 500 files and ran a full table scan. Seems to work as expected. I checked the amount of untracked memory before + after I ran the query and it looked okay.
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 1
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 00:12:17 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
IMPALA-10202: Enable file handle cache for ABFS files
Like IMPALA-8428, but for ABFS, instead of S3A. Adds support for adding
ABFS file handles to the file handle cache. Support for ABFSInputStream
unbuffer operations was added in HADOOP-16859.
Ran a full table scan of a 1GB store_sales table on ABFS, made
sure the file handles were cached (validated via the runtime
profile); did this multiple times, against several different
copies of the store_sales table, in order to increase the number
of file handles cached by an impalad.
Tested:
* Tested against a ABFS storage account I have access to
Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Reviewed-on: http://gerrit.cloudera.org:8080/16532
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
3 files changed, 11 insertions(+), 6 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 3
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
[Impala-ASF-CR] IMPALA-10202: Enable file handle cache for ABFS files
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/16532 )
Change subject: IMPALA-10202: Enable file handle cache for ABFS files
......................................................................
Patch Set 2: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/16532
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I64f12f832980f4e0207af78368402dd09e370fc3
Gerrit-Change-Number: 16532
Gerrit-PatchSet: 2
Gerrit-Owner: Sahil Takiar <st...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Sahil Takiar <st...@cloudera.com>
Gerrit-Comment-Date: Fri, 02 Oct 2020 05:34:11 +0000
Gerrit-HasComments: No