You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Smith (Code Review)" <ge...@cloudera.org> on 2022/08/15 23:01:02 UTC

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Michael Smith has uploaded this change for review. ( http://gerrit.cloudera.org:8080/18853


Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................

IMPALA-10214: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching, similar to what's suggested in IMPALA-10375.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds a test that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
3 files changed, 52 insertions(+), 20 deletions(-)



  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/1
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 2:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11163/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Aug 2022 17:10:24 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#2).

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................

IMPALA-10214: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching, similar to what's suggested in IMPALA-10375.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds a test that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
3 files changed, 54 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/2
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 2:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8450/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Aug 2022 17:33:44 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 9: Code-Review+1

This makes sense to me. Assuming the tests pass and no other comments, I'll go to +2 tomorrow


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 02:34:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 6:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/6/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/6/tests/custom_cluster/test_hdfs_fd_caching.py@191
PS6, Line 191: @
flake8: F811 redefinition of unused 'test_remote_caching_disabled_by_param' from line 180



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 21:23:22 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#9).

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................

IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/scan-range.cc
A tests/common/network.py
M tests/custom_cluster/test_hdfs_fd_caching.py
5 files changed, 92 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/9
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#6).

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................

IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
4 files changed, 71 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/6
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#8).

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................

IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/scan-range.cc
A tests/common/network.py
M tests/custom_cluster/test_hdfs_fd_caching.py
5 files changed, 91 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/8
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 9:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11248/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 23:39:37 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 6:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11242/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 21:43:30 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 10:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8511/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 17:56:59 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#3).

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................

IMPALA-10214: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching, similar to what's suggested in IMPALA-10375.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds a test that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
3 files changed, 57 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/3
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 3:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11167/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 3
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Aug 2022 21:09:52 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py@127
PS4, Line 127:     s.connect(('10.254.254.254', 1))
> Hard-coded address?
Ah, the timeout means it doesn't matter if it resolves (we kind of don't want it to). I should've left a comment.



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Aug 2022 21:44:18 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 6:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18853/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18853/5//COMMIT_MSG@10
PS5, Line 10: Updates
            : file handle cache enablement to be more stringent about enabling
            : caching.
> Nit: I think this addresses IMPALA-10375, so we can modify the language her
Done


http://gerrit.cloudera.org:8080/#/c/18853/5/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/5/tests/custom_cluster/test_hdfs_fd_caching.py@182
PS5, Line 182: --cache_remote_file_handles
> If max_cached_file_handles=0, caching can't happen (is_file_handle_caching_
I don't remember why I included max_cached_file_handles here. Separate tests for each flag disabled with remote storage makes sense to me.



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 6
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 21:22:40 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 10: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 22:40:25 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 8:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11246/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 23:31:38 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 8:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/8/tests/common/network.py
File tests/common/network.py:

http://gerrit.cloudera.org:8080/#/c/18853/8/tests/common/network.py@24
PS8, Line 24: def get_external_ip():
flake8: E302 expected 2 blank lines, found 1



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 8
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 23:17:38 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 10:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11259/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 16:57:40 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 7:

Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/8506/ DRY_RUN=true


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 21:47:48 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 2: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 2
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Aug 2022 22:23:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 7: Verified+1


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 02:39:34 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 7:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11243/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 21:52:27 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 10: Code-Review+2


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Wed, 31 Aug 2022 11:03:51 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Kurt Deschler (Code Review)" <ge...@cloudera.org>.
Kurt Deschler has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 4:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/4/tests/custom_cluster/test_hdfs_fd_caching.py@127
PS4, Line 127:     s.connect(('10.254.254.254', 1))
Hard-coded address?



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Aug 2022 18:00:27 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 1:

(8 comments)

http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@131
PS1, Line 131: +
flake8: W504 line break after binary operator


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@132
PS1, Line 132: "
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@132
PS1, Line 132: +
flake8: W504 line break after binary operator


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@153
PS1, Line 153: +
flake8: W504 line break after binary operator


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@154
PS1, Line 154: "
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@179
PS1, Line 179: +
flake8: W504 line break after binary operator


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@180
PS1, Line 180: "
flake8: E131 continuation line unaligned for hanging indent


http://gerrit.cloudera.org:8080/#/c/18853/1/tests/custom_cluster/test_hdfs_fd_caching.py@180
PS1, Line 180: +
flake8: W504 line break after binary operator



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 23:01:57 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#5).

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................

IMPALA-10214: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching, similar to what's suggested in IMPALA-10375.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds a test that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
4 files changed, 60 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/5
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 5:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11193/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Fri, 19 Aug 2022 22:06:13 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 7:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/7/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/7/tests/custom_cluster/test_hdfs_fd_caching.py@124
PS7, Line 124:   def get_external_ip():
Since I used this in two different patches, it probably makes sense to pull into its own package.



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 23:12:43 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 9: Code-Review+1

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/9/be/src/runtime/io/scan-range.cc
File be/src/runtime/io/scan-range.cc:

http://gerrit.cloudera.org:8080/#/c/18853/9/be/src/runtime/io/scan-range.cc@175
PS9, Line 175:   if (is_file_handle_caching_enabled() && (
             :       (expected_local_ && IsHdfsPath(file())) ||
             :       (FLAGS_cache_remote_file_handles && disk_id_ == io_mgr_->RemoteDfsDiskId()) ||
             :       (FLAGS_cache_ozone_file_handles && (
             :         (expected_local_ && IsOzonePath(file())) ||
             :         (disk_id_ == io_mgr_->RemoteOzoneDiskId()))) ||
             :       (FLAGS_cache_s3_file_handles && disk_id_ == io_mgr_->RemoteS3DiskId()) ||
             :       (FLAGS_cache_abfs_file_handles && disk_id_ == io_mgr_->RemoteAbfsDiskId()))) {
             :     use_file_handle_cache = true;
             :   }
Optional: extract this to a new functions?
I think that it quite hard to read this condition now if you don't know already what it tries to express :)



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 11:41:14 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................

IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Reviewed-on: http://gerrit.cloudera.org:8080/18853
Tested-by: Impala Public Jenkins <im...@cloudera.com>
Reviewed-by: Csaba Ringhofer <cs...@cloudera.com>
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
A tests/common/network.py
M tests/custom_cluster/test_hdfs_fd_caching.py
6 files changed, 109 insertions(+), 24 deletions(-)

Approvals:
  Impala Public Jenkins: Verified
  Csaba Ringhofer: Looks good to me, approved

-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 11
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Joe McDonnell, Csaba Ringhofer, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#10).

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................

IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
A tests/common/network.py
M tests/custom_cluster/test_hdfs_fd_caching.py
6 files changed, 109 insertions(+), 24 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/10
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 10
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Michael Smith has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................


Patch Set 9:

(1 comment)

http://gerrit.cloudera.org:8080/#/c/18853/9/be/src/runtime/io/scan-range.cc
File be/src/runtime/io/scan-range.cc:

http://gerrit.cloudera.org:8080/#/c/18853/9/be/src/runtime/io/scan-range.cc@175
PS9, Line 175:   if (is_file_handle_caching_enabled() && (
             :       (expected_local_ && IsHdfsPath(file())) ||
             :       (FLAGS_cache_remote_file_handles && disk_id_ == io_mgr_->RemoteDfsDiskId()) ||
             :       (FLAGS_cache_ozone_file_handles && (
             :         (expected_local_ && IsOzonePath(file())) ||
             :         (disk_id_ == io_mgr_->RemoteOzoneDiskId()))) ||
             :       (FLAGS_cache_s3_file_handles && disk_id_ == io_mgr_->RemoteS3DiskId()) ||
             :       (FLAGS_cache_abfs_file_handles && disk_id_ == io_mgr_->RemoteAbfsDiskId()))) {
             :     use_file_handle_cache = true;
             :   }
> Optional: extract this to a new functions?
Maybe a separate function for each filesystem?



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 9
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Tue, 30 Aug 2022 15:51:29 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Joe McDonnell (Code Review)" <ge...@cloudera.org>.
Joe McDonnell has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 5:

(2 comments)

http://gerrit.cloudera.org:8080/#/c/18853/5//COMMIT_MSG
Commit Message:

http://gerrit.cloudera.org:8080/#/c/18853/5//COMMIT_MSG@10
PS5, Line 10: Updates
            : file handle cache enablement to be more stringent about enabling
            : caching, similar to what's suggested in IMPALA-10375.
Nit: I think this addresses IMPALA-10375, so we can modify the language here to say that this resolves IMPALA-10375. I assume we'll close IMPALA-10375 when this goes in. You can include IMPALA-10375 as another JIRA in the title if you want.


http://gerrit.cloudera.org:8080/#/c/18853/5/tests/custom_cluster/test_hdfs_fd_caching.py
File tests/custom_cluster/test_hdfs_fd_caching.py:

http://gerrit.cloudera.org:8080/#/c/18853/5/tests/custom_cluster/test_hdfs_fd_caching.py@182
PS5, Line 182: --max_cached_file_handles=0
If max_cached_file_handles=0, caching can't happen (is_file_handle_caching_enabled() is false) and none of the other flags matter. One option for a test would be to keep max_cached_file_handles non-zero, but then specify all the cache_X_file_handles=false. That would verify that those flags function.



-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 5
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>
Gerrit-Comment-Date: Mon, 29 Aug 2022 19:42:02 +0000
Gerrit-HasComments: Yes

[Impala-ASF-CR] IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Kurt Deschler, Joe McDonnell, Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#7).

Change subject: IMPALA-10214, IMPALA-10375: Ozone remote file handle caching
......................................................................

IMPALA-10214, IMPALA-10375: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds tests that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
4 files changed, 71 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/7
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 7
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Joe McDonnell <jo...@cloudera.com>
Gerrit-Reviewer: Kurt Deschler <kd...@cloudera.com>
Gerrit-Reviewer: Michael Smith <mi...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Michael Smith (Code Review)" <ge...@cloudera.org>.
Hello Impala Public Jenkins, 

I'd like you to reexamine a change. Please visit

    http://gerrit.cloudera.org:8080/18853

to look at the new patch set (#4).

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................

IMPALA-10214: Ozone remote file handle caching

Enables support for caching remote file handles for Ozone. Local file
handles were already cached unintentionally, similar to HDFS. Updates
file handle cache enablement to be more stringent about enabling
caching, similar to what's suggested in IMPALA-10375.

File handle caching is enabled if a max_cached_file_handles is non-zero
and any of the following are true
- HDFS file is local
- HDFS file is remote and cache_remote_file_handles is enabled
- Ozone file is local or remote and cache_ozone_file_handles is enabled
- S3 file is remote and cache_s3_file_handles is enabled
- ABFS file is remote and cache_abfs_file_handles is enabled

Enables testing Ozone in test_hdfs_fd_caching, and adds a test that
remote caching can be disabled using individual flags.

Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
---
M be/src/runtime/io/disk-io-mgr.cc
M be/src/runtime/io/handle-cache.inline.h
M be/src/runtime/io/scan-range.cc
M tests/custom_cluster/test_hdfs_fd_caching.py
4 files changed, 58 insertions(+), 21 deletions(-)


  git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/53/18853/4
-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 4:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11168/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 4
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Tue, 16 Aug 2022 22:26:32 +0000
Gerrit-HasComments: No

[Impala-ASF-CR] IMPALA-10214: Ozone remote file handle caching

Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/18853 )

Change subject: IMPALA-10214: Ozone remote file handle caching
......................................................................


Patch Set 1:

Build Successful 

https://jenkins.impala.io/job/gerrit-code-review-checks/11159/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.


-- 
To view, visit http://gerrit.cloudera.org:8080/18853
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings

Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I9df13208999c6d3b14f4c005a91ee2a92a05bdf9
Gerrit-Change-Number: 18853
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Smith <mi...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Mon, 15 Aug 2022 23:21:47 +0000
Gerrit-HasComments: No