You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@impala.apache.org by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org> on 2024/09/13 15:00:36 UTC
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Csaba Ringhofer has uploaded this change for review. ( http://gerrit.cloudera.org:8080/21797
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
The way the planner estimates mem usage for KrpcDataStreamSender
is very different than how the backend actually uses memory -
the planner assumes that batch_size number of rows are sent at
a time while the BE tries to limit it to
data_stream_sender_buffer_size_ (but doesn't consider var len data).
The Jira has more detail about differences and issues.
This change adds flag data_stream_sender_buffer_size_used_by_planner.
If this is set to 16K (data_stream_sender_buffer_size_ default)
then the estimation will work similarly to BE.
Tested that this can improve both under and overestimations:
peak mem / mem estimate of the first sender:
select distinct * from tpch_parquet.lineitem limit 100000
default:
284.04 KB 2.75 MB
--data_stream_sender_buffer_size_used_by_planner=16384:
282.04 KB 283.39 KB
select distinct l_comment from tpch_parquet.lineitem limit 100000;
default:
747.71 KB 509.94 KB
--data_stream_sender_buffer_size_used_by_planner=16384:
740.71 KB 627.46 KB
The default is not changed to avoid side effects. I would like
to change it once BE's handling of var len data is fixed, which
is a prerequisity to use mem reservation in KrpcDataStreamSender.
Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
6 files changed, 42 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/21797/1
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newchange
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG@16
PS1, Line 16: data_stream_sender_buffer_size_used_by_planner
> I thought that it can be useful to use a slightly smaller / larger value if
Ack
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java:
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@100
PS1, Line 100: Math.max(1, (long)Math.ceil(beBufferBytes/fixedLenRowSize));
> Even if use_be_logic is true, the calculation still uses getAvgSerializedRo
Done
http://gerrit.cloudera.org:8080/#/c/21797/2/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java:
http://gerrit.cloudera.org:8080/#/c/21797/2/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@98
PS2, Line 98: if (fixedLenRowSize==0) fixedLenRowSize = 1; // avoid division by 0
: long beRowsPerBuffer =
: Math.max(1, (long)Math.ceil(beBufferBytes/fixedLenRowSize));
This can blow up if all columns are var-len type, and data_stream_sender_buffer_size_used_by_planner is high. Is that OK? Should we bound it to batch_size max if fixedLenRowSize == 1?
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 16:41:54 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 2: Code-Review+2
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21797/2/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java:
http://gerrit.cloudera.org:8080/#/c/21797/2/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@98
PS2, Line 98: if (fixedLenRowSize==0) fixedLenRowSize = 1; // avoid division by 0
: long beRowsPerBuffer =
: Math.max(1, (long)Math.ceil(beBufferBytes/fixedLenRowSize));
> As we also discussed on another channel this shouldn't lead to much more ro
Ack
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Sep 2024 12:49:38 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 3: Code-Review+2
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Sep 2024 15:16:50 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 2:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21797/2/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java:
http://gerrit.cloudera.org:8080/#/c/21797/2/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@98
PS2, Line 98: if (fixedLenRowSize==0) fixedLenRowSize = 1; // avoid division by 0
: long beRowsPerBuffer =
: Math.max(1, (long)Math.ceil(beBufferBytes/fixedLenRowSize));
> This can blow up if all columns are var-len type, and data_stream_sender_bu
As we also discussed on another channel this shouldn't lead to much more rows than the original batch_size of 1024.
A string (or collection) column adds 12 bytes to fixed len size, while 1024 rows will be used when the the row size is 16 bytes (default data_stream_sender_buffer_size=16K = 1024*16), so a single large string column can raise the memory estimate, but not by much.
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Sep 2024 09:29:09 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 1:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16945/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 15:32:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Hello Riza Suminto, Impala Public Jenkins,
I'd like you to reexamine a change. Please visit
http://gerrit.cloudera.org:8080/21797
to look at the new patch set (#2).
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
The way the planner estimates mem usage for KrpcDataStreamSender
is very different than how the backend actually uses memory -
the planner assumes that batch_size number of rows are sent at
a time while the BE tries to limit it to
data_stream_sender_buffer_size_ (but doesn't consider var len data).
The Jira has more detail about differences and issues.
This change adds flag data_stream_sender_buffer_size_used_by_planner.
If this is set to 16K (data_stream_sender_buffer_size_ default)
then the estimation will work similarly to BE.
Tested that this can improve both under and overestimations:
peak mem / mem estimate of the first sender:
select distinct * from tpch_parquet.lineitem limit 100000
default:
284.04 KB 2.75 MB
--data_stream_sender_buffer_size_used_by_planner=16384:
282.04 KB 283.39 KB
select distinct l_comment from tpch_parquet.lineitem limit 100000;
default:
747.71 KB 509.94 KB
--data_stream_sender_buffer_size_used_by_planner=16384:
740.71 KB 627.46 KB
The default is not changed to avoid side effects. I would like
to change it once BE's handling of var len data is fixed, which
is a prerequisity to use mem reservation in KrpcDataStreamSender.
Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
6 files changed, 44 insertions(+), 3 deletions(-)
git pull ssh://gerrit.cloudera.org:29418/Impala-ASF refs/changes/97/21797/2
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 2:
Build Successful
https://jenkins.impala.io/job/gerrit-code-review-checks/16947/ : Initial code review checks passed. Use gerrit-verify-dryrun-external or gerrit-verify-dryrun to run full precommit tests.
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 16:52:07 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has submitted this change and it was merged. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
The way the planner estimates mem usage for KrpcDataStreamSender
is very different than how the backend actually uses memory -
the planner assumes that batch_size number of rows are sent at
a time while the BE tries to limit it to
data_stream_sender_buffer_size_ (but doesn't consider var len data).
The Jira has more detail about differences and issues.
This change adds flag data_stream_sender_buffer_size_used_by_planner.
If this is set to 16K (data_stream_sender_buffer_size_ default)
then the estimation will work similarly to BE.
Tested that this can improve both under and overestimations:
peak mem / mem estimate of the first sender:
select distinct * from tpch_parquet.lineitem limit 100000
default:
284.04 KB 2.75 MB
--data_stream_sender_buffer_size_used_by_planner=16384:
282.04 KB 283.39 KB
select distinct l_comment from tpch_parquet.lineitem limit 100000;
default:
747.71 KB 509.94 KB
--data_stream_sender_buffer_size_used_by_planner=16384:
740.71 KB 627.46 KB
The default is not changed to avoid side effects. I would like
to change it once BE's handling of var len data is fixed, which
is a prerequisity to use mem reservation in KrpcDataStreamSender.
Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Reviewed-on: http://gerrit.cloudera.org:8080/21797
Reviewed-by: Impala Public Jenkins <im...@cloudera.com>
Tested-by: Impala Public Jenkins <im...@cloudera.com>
---
M be/src/util/backend-gflag-util.cc
M common/thrift/BackendGflags.thrift
M fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
M fe/src/main/java/org/apache/impala/planner/EmptySetNode.java
M fe/src/main/java/org/apache/impala/planner/PlanNode.java
M fe/src/main/java/org/apache/impala/service/BackendConfig.java
6 files changed, 44 insertions(+), 3 deletions(-)
Approvals:
Impala Public Jenkins: Looks good to me, approved; Verified
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: merged
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 4
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java:
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@100
PS1, Line 100: boolean use_be_logic = be_buffer_bytes > 0 && outputPartition_.isPartitioned();
If getAvgSerializedRowSize() is significantly larger than getFixedLenRowSize(), should we favor prior carculation instead?
I'm thinking a flag, say, data_stream_sender_large_row_threshold, and then:
boolean use_be_logic = be_buffer_bytes > 0 && outputPartition_.isPartitioned() && (ExchangeNode.getAvgSerializedRowSize(exchNode_) < data_stream_sender_large_row_threshold * exchNode_.getFixedLenRowSize());
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 16:12:39 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/PlanNode.java@732
PS1, Line 732: getFixedLenRowSize_ += desc.getByteSize();
Can this be zero?
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 15:51:19 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Csaba Ringhofer (Code Review)" <ge...@cloudera.org>.
Csaba Ringhofer has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 2:
(3 comments)
http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG@16
PS1, Line 16: data_stream_sender_buffer_size_used_by_planner
> Why not utilize data_stream_sender_buffer_size flag straight away?
I thought that it can be useful to use a slightly smaller / larger value if this have to be tuned. + a new flag was needed anyway, as I didn't want this to be the default
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java
File fe/src/main/java/org/apache/impala/planner/DataStreamSink.java:
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java@100
PS1, Line 100: Math.max(1, (long)Math.ceil(beBufferBytes/fixedLenRowSize));
> If getAvgSerializedRowSize() is significantly larger than getFixedLenRowSiz
Even if use_be_logic is true, the calculation still uses getAvgSerializedRowSize() to calculate buffer sizes. The difference is in calculating the number of rows, where what matters is the formula in BE.
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/PlanNode.java
File fe/src/main/java/org/apache/impala/planner/PlanNode.java:
http://gerrit.cloudera.org:8080/#/c/21797/1/fe/src/main/java/org/apache/impala/planner/PlanNode.java@732
PS1, Line 732: getFixedLenRowSize_ += desc.getByteSize();
> Can this be zero?
I don't know. Added check for division by zero protection.
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 2
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 16:21:57 +0000
Gerrit-HasComments: Yes
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 3: Verified+1
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Sep 2024 20:24:40 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Impala Public Jenkins (Code Review)" <ge...@cloudera.org>.
Impala Public Jenkins has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 3:
Build started: https://jenkins.impala.io/job/gerrit-verify-dryrun/10950/ DRY_RUN=false
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 3
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Sat, 14 Sep 2024 15:16:51 +0000
Gerrit-HasComments: No
[Impala-ASF-CR] IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
Posted by "Riza Suminto (Code Review)" <ge...@cloudera.org>.
Riza Suminto has posted comments on this change. ( http://gerrit.cloudera.org:8080/21797 )
Change subject: IMPALA-12594: Add flag to tune KrpcDataStreamSender mem estimate
......................................................................
Patch Set 1:
(1 comment)
http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG
Commit Message:
http://gerrit.cloudera.org:8080/#/c/21797/1//COMMIT_MSG@16
PS1, Line 16: data_stream_sender_buffer_size_used_by_planner
Why not utilize data_stream_sender_buffer_size flag straight away?
--
To view, visit http://gerrit.cloudera.org:8080/21797
To unsubscribe, visit http://gerrit.cloudera.org:8080/settings
Gerrit-Project: Impala-ASF
Gerrit-Branch: master
Gerrit-MessageType: comment
Gerrit-Change-Id: I1e4b1db030be934cece565e3f2634ee7cbdb7c4f
Gerrit-Change-Number: 21797
Gerrit-PatchSet: 1
Gerrit-Owner: Csaba Ringhofer <cs...@cloudera.com>
Gerrit-Reviewer: Impala Public Jenkins <im...@cloudera.com>
Gerrit-Reviewer: Riza Suminto <ri...@cloudera.com>
Gerrit-Comment-Date: Fri, 13 Sep 2024 15:47:51 +0000
Gerrit-HasComments: Yes