You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@impala.apache.org by "Huaisi Xu (Code Review)" <ge...@cloudera.org> on 2016/06/24 19:18:05 UTC
[Impala-CR](cdh5-2.3.0 5.5.x) CDH-41243: Parquet scanner regression on wide tables
Huaisi Xu has uploaded a new change for review.
http://gerrit.cloudera.org:8080/3484
Change subject: CDH-41243: Parquet scanner regression on wide tables
......................................................................
CDH-41243: Parquet scanner regression on wide tables
IMPALA-2473 introduced a check that prevent row batches growing
beyond 8MB, but it has a corner case that when an empty row
batch is larger than 8MB, it returns this row batch immediately
after it materialize one row, essentailly setting batch_size=1.
Revert "IMPALA-2473: reduce scanner memory usage"
This reverts commit 1635c0a8738daef1b283cb457fbd3bca227aa0b1.
Change-Id: If6728ed8facd305682d7dfd58f1210fa294bb232
---
M be/src/exec/data-source-scan-node.cc
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/row-batch.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
7 files changed, 26 insertions(+), 81 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala refs/changes/84/3484/1
--
To view, visit http://gerrit.cloudera.org:8080/3484
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: If6728ed8facd305682d7dfd58f1210fa294bb232
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.3.0_5.5.x
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
[Impala-CR](cdh5-2.3.0 5.5.x) CDH-41243: Parquet scanner regression on wide tables
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has submitted this change and it was merged.
Change subject: CDH-41243: Parquet scanner regression on wide tables
......................................................................
CDH-41243: Parquet scanner regression on wide tables
IMPALA-2473 introduced a check that prevent row batches growing
beyond 8MB, but it has a corner case that when an empty row
batch is larger than 8MB, it returns this row batch immediately
after it materialize one row, essentailly setting batch_size=1.
Revert "IMPALA-2473: reduce scanner memory usage"
This reverts commit 1635c0a8738daef1b283cb457fbd3bca227aa0b1.
Change-Id: If6728ed8facd305682d7dfd58f1210fa294bb232
Reviewed-on: http://gerrit.cloudera.org:8080/3484
Reviewed-by: Huaisi Xu <hx...@cloudera.com>
Tested-by: Huaisi Xu <hx...@cloudera.com>
---
M be/src/exec/data-source-scan-node.cc
M be/src/exec/hdfs-parquet-scanner.cc
M be/src/exec/hdfs-scanner.cc
M be/src/exec/hdfs-table-sink.cc
M be/src/exec/hdfs-table-sink.h
M be/src/runtime/row-batch.h
M testdata/workloads/functional-query/queries/QueryTest/nested-types-tpch.test
7 files changed, 26 insertions(+), 81 deletions(-)
Approvals:
Huaisi Xu: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/3484
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: If6728ed8facd305682d7dfd58f1210fa294bb232
Gerrit-PatchSet: 2
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.3.0_5.5.x
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
[Impala-CR](cdh5-2.3.0 5.5.x) CDH-41243: Parquet scanner regression on wide tables
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: CDH-41243: Parquet scanner regression on wide tables
......................................................................
Patch Set 1: Code-Review+2 Verified+1
tested here, http://sandbox.jenkins.cloudera.com/view/Impala/view/Maintenance-Releases/job/impala-cdh5-2.3.0_5.5.x-exhaustive/241/
and it is a clean revert. so merge now.
--
To view, visit http://gerrit.cloudera.org:8080/3484
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If6728ed8facd305682d7dfd58f1210fa294bb232
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.3.0_5.5.x
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No
[Impala-CR](cdh5-2.3.0 5.5.x) CDH-41243: Parquet scanner regression on wide tables
Posted by "Huaisi Xu (Code Review)" <ge...@cloudera.org>.
Huaisi Xu has posted comments on this change.
Change subject: CDH-41243: Parquet scanner regression on wide tables
......................................................................
Patch Set 1:
Tim, it is a clean revert. but I reverted all changes in this commit. Am I doing thsis correctly? I tested in a release build, and performance is similar to 5.4.x after revert.
--
To view, visit http://gerrit.cloudera.org:8080/3484
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: If6728ed8facd305682d7dfd58f1210fa294bb232
Gerrit-PatchSet: 1
Gerrit-Project: Impala
Gerrit-Branch: cdh5-2.3.0_5.5.x
Gerrit-Owner: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Huaisi Xu <hx...@cloudera.com>
Gerrit-Reviewer: Tim Armstrong <ta...@cloudera.com>
Gerrit-HasComments: No