You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues-all@impala.apache.org by "Tim Armstrong (JIRA)" <ji...@apache.org> on 2018/09/17 21:51:00 UTC
[jira] [Updated] (IMPALA-4179) Remove RowBatch::MarkNeedsDeepCopy()
memory management API
[ https://issues.apache.org/jira/browse/IMPALA-4179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Tim Armstrong updated IMPALA-4179:
----------------------------------
Description:
We should remove this API as a simplification of the memory transfer model.
It was used in various places to manage lifetime of BufferHandles, but those can be replaced with attaching the BufferHandle + MarkFlushResources().
It is also used to work around the fact that some memory is never returned from ExecNodes and is freed in Close(). Part of the solution is probably to add a way to attach all resources to a RowBatch before Close() and during Reset().
was:
We should remove this API as a simplification of the memory transfer model.
Here is a list of the cases I know of where it is used and why:
* Sorter::Run and BufferedTupleStream need it when iterating over an unpinned sequence of blocks so they can trade in the previous block for the next one. This can't be fixed until after the switch to the new BufferPool API.
* PartitionedAggregationNode needs it because it returns RowBatches that reference data from a Partition's aggregate_row_stream. We need a way to attach that memory.
The MarkFlushResources() API will help here - once fully implemented it will force operators like NestedLoopJoin to either take over ownership of memory or free it.
Part of the solution is probably to add a way to attach all resources to a RowBatch on Close() and Reset().
> Remove RowBatch::MarkNeedsDeepCopy() memory management API
> ----------------------------------------------------------
>
> Key: IMPALA-4179
> URL: https://issues.apache.org/jira/browse/IMPALA-4179
> Project: IMPALA
> Issue Type: Improvement
> Components: Backend
> Affects Versions: Impala 2.8.0
> Reporter: Tim Armstrong
> Assignee: Tim Armstrong
> Priority: Major
> Labels: resource-management
>
> We should remove this API as a simplification of the memory transfer model.
> It was used in various places to manage lifetime of BufferHandles, but those can be replaced with attaching the BufferHandle + MarkFlushResources().
> It is also used to work around the fact that some memory is never returned from ExecNodes and is freed in Close(). Part of the solution is probably to add a way to attach all resources to a RowBatch before Close() and during Reset().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-all-unsubscribe@impala.apache.org
For additional commands, e-mail: issues-all-help@impala.apache.org