You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Omid Shahidi (Jira)" <ji...@apache.org> on 2022/08/18 18:08:00 UTC
[jira] [Updated] (IMPALA-11510) Provide memory estimation for Exchange Sender's plan node
[ https://issues.apache.org/jira/browse/IMPALA-11510?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Omid Shahidi updated IMPALA-11510:
----------------------------------
Description:
Currently there is no memory estimation provided for DataStreamSink which is responsible for consuming the KRPC OutboundRowBatches which contain the tuple data for RowBatches. There is a hypothesis that providing such memory estimation can help to reduce memory usage without affecting the performance. A rough estimation for exchange sender is:
num_channel * 2 * (tuple_buffer_length + compressed_buffer_length)
With IMPALA-6684, two new runtime profile counters were added. TupleDataBytes and CompressionScratchBytes track the tuple buffer length and compressed buffer length within an OutboundRowBatch. These runtime profile counters can be used a research source and as experimentation for an estimation of tuple_buffer_length and comressed_buffer_length
[https://github.com/apache/impala/blob/26438d8e3e2cecfdab82643fcee7553df50198ca/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java#L60-L63]
was:
Currently there is no memory estimation provided for DataStreamSink which is responsible for consuming the KRPC OutboundRowBatches which contain the tuple data for RowBatches. There is a hypothesis that providing such memory estimation can help to reduce memory usage without affecting the performance. A rough estimation for exchange sender is:
num_channel * 2 * (tuple_buffer_length + compressed_buffer_length)
With IMPALA-6684, two new runtime profile counters were added. TupleDataBytes and CompressionScratchBytes track the tuple buffer length and compressed buffer length within an OutboundRowBatch. These runtime profile counters can be used a research source and as experimentation for an estimation of tuple_buffer_length and comressed_buffer_length
https://github.com/apache/impala/blob/26438d8e3e2cecfdab82643fcee7553df50198ca/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java#L60-L63
> Provide memory estimation for Exchange Sender's plan node
> ---------------------------------------------------------
>
> Key: IMPALA-11510
> URL: https://issues.apache.org/jira/browse/IMPALA-11510
> Project: IMPALA
> Issue Type: Improvement
> Components: Frontend
> Reporter: Omid Shahidi
> Priority: Major
>
> Currently there is no memory estimation provided for DataStreamSink which is responsible for consuming the KRPC OutboundRowBatches which contain the tuple data for RowBatches. There is a hypothesis that providing such memory estimation can help to reduce memory usage without affecting the performance. A rough estimation for exchange sender is:
> num_channel * 2 * (tuple_buffer_length + compressed_buffer_length)
> With IMPALA-6684, two new runtime profile counters were added. TupleDataBytes and CompressionScratchBytes track the tuple buffer length and compressed buffer length within an OutboundRowBatch. These runtime profile counters can be used a research source and as experimentation for an estimation of tuple_buffer_length and comressed_buffer_length
> [https://github.com/apache/impala/blob/26438d8e3e2cecfdab82643fcee7553df50198ca/fe/src/main/java/org/apache/impala/planner/DataStreamSink.java#L60-L63]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org