You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Michael Ho (Code Review)" <ge...@cloudera.org> on 2019/02/27 20:27:32 UTC
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Michael Ho has uploaded this change for review. ( http://gerrit.cloudera.org:8080/12622
Change subject: IMPALA-4568: Cache parquet footer
......................................................................
IMPALA-4568: Cache parquet footer
Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
---
M be/src/exec/hdfs-scan-node-base.cc
M be/src/exec/hdfs-scan-node-base.h
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-text-scanner.cc
M be/src/exec/parquet/CMakeLists.txt
M be/src/exec/parquet/hdfs-parquet-scanner.cc
M be/src/exec/parquet/hdfs-parquet-scanner.h
M be/src/exec/parquet/parquet-column-readers.cc
A be/src/exec/parquet/parquet-footer-cache.cc
A be/src/exec/parquet/parquet-footer-cache.h
M be/src/runtime/exec-env.cc
M be/src/runtime/exec-env.h
M be/src/runtime/io/request-context.cc
M be/src/runtime/io/request-ranges.h
M be/src/runtime/io/scan-range.cc
15 files changed, 443 insertions(+), 126 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/22/12622/1
--
To view, visit http://gerrit.cloudera.org:8080/12622
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
Gerrit-Change-Number: 12622
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has abandoned this change. ( http://gerrit.cloudera.org:8080/12622 )
Change subject: IMPALA-4568: Cache parquet footer
......................................................................
Abandoned
--
To view, visit http://gerrit.cloudera.org:8080/12622
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: abandon
Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
Gerrit-Change-Number: 12622
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12622 )
Change subject: IMPALA-4568: Cache parquet footer
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/2278/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/12622
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
Gerrit-Change-Number: 12622
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Feb 2019 20:58:42 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Posted by "Michael Ho (Code Review)" <ge...@cloudera.org>.
Michael Ho has posted comments on this change. ( http://gerrit.cloudera.org:8080/12622 )
Change subject: IMPALA-4568: Cache parquet footer
......................................................................
Patch Set 1:
Sorry, pushing the wrong branch. This is WIP. Definitely not meant for review.
--
To view, visit http://gerrit.cloudera.org:8080/12622
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
Gerrit-Change-Number: 12622
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Feb 2019 20:28:58 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-4568: Cache parquet footer
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/12622 )
Change subject: IMPALA-4568: Cache parquet footer
......................................................................
Patch Set 1:
(10 comments)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-scanner.cc
File be/src/exec/hdfs-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-scanner.cc@828
PS1, Line 828: BufferOpts(cache_footer ? BufferOpts::EXT_CACHED : BufferOpts::UNCACHED, -1),
line too long (96 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-text-scanner.cc
File be/src/exec/hdfs-text-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/hdfs-text-scanner.cc@120
PS1, Line 120: BufferOpts(split->cache_tags(), files[i]->mtime));
tab used for whitespace
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc
File be/src/exec/parquet/hdfs-parquet-scanner.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1200
PS1, Line 1200: // We're processing the scan range issued in IssueInitialRanges(). The scan range should
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1202
PS1, Line 1202: // expect. Note we can't detect if the file is larger than we expect without attempting
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1237
PS1, Line 1237: string(reinterpret_cast<char*>(magic_number_ptr), sizeof(PARQUET_VERSION_NUMBER)),
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1265
PS1, Line 1265: "footer: $1 bytes. File size: $2 bytes.", filename(), metadata_size, file_len));
line too long (92 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/hdfs-parquet-scanner.cc@1271
PS1, Line 1271: return scan_node_->mem_tracker()->MemLimitExceeded(state_, details, metadata_size);
line too long (91 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/parquet-footer-cache.cc
File be/src/exec/parquet/parquet-footer-cache.cc:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/exec/parquet/parquet-footer-cache.cc@76
PS1, Line 76: void ParquetFooterCache::Partition::Insert(const string& fname, const FileMetaData& footer) {
line too long (93 > 90)
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/exec-env.h
File be/src/runtime/exec-env.h:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/exec-env.h@222
PS1, Line 222:
line has trailing whitespace
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/io/request-ranges.h
File be/src/runtime/io/request-ranges.h:
http://gerrit.cloudera.org:8080/#/c/12622/1/be/src/runtime/io/request-ranges.h@275
PS1, Line 275:
line has trailing whitespace
--
To view, visit http://gerrit.cloudera.org:8080/12622
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: Ib30dcc4a13050c11f011491dec3776ba9e402c78
Gerrit-Change-Number: 12622
Gerrit-PatchSet: 1
Gerrit-Owner: Michael Ho <kw...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Wed, 27 Feb 2019 20:28:22 +0000
Gerrit-HasComments: Yes