[jira] [Commented] (HIVE-15221) Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception

["Barna Zsombor Klara (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HIVE-15221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15673923#comment-15673923 ] 

Barna Zsombor Klara commented on HIVE-15221:
--------------------------------------------

Hi [~ferhui],
thanks for the patch, but I'm not sure how I feel about an explicit call to System.gc.
I understand your problem that a portion of the used heap at the point of the exception could be garbage, but a call to System.gc will not guarantee a garbage collection cycle, the jvm may ignore it. So the change would not guarantee that we only abort if the heap is over utilised.
Sporadic and unexpected calls for a garbage collection would also ravage attempts at fine tuning garbage collection cycles.
If a user is worried about false MemoryExhaustionExceptions then I would recommend to try and fine tune the collector in use to increase the garbage collection frequency of the jvm.

> Improvement for MapJoin checkMemoryStatus, adding gc before throwing Exception
> ------------------------------------------------------------------------------
>
>                 Key: HIVE-15221
>                 URL: https://issues.apache.org/jira/browse/HIVE-15221
>             Project: Hive
>          Issue Type: Improvement
>          Components: Query Processor
>    Affects Versions: 2.1.0, 2.0.1
>            Reporter: Fei Hui
>            Assignee: Fei Hui
>         Attachments: HIVE-15221.1.patch
>
>
> i see in the current master version
>  percentage = (double) usedMemory / (double) maxHeapSize;
> if  percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException
> in my opinion, running is better than fail. after System.gc, ' if percentage > maxMemoryUsage, then throw MapJoinMemoryExhaustionException' maybe better
> And original checking way has a problem: 1) consuming much memory cause gc (e.g young gc), then check after adding row and pass. 2) consuming much memory does not cause gc, then check after adding rows but throw Exception
> sometimes 2) occurs, but it contians less rows than 1).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)