You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "xuchuanyin (JIRA)" <ji...@apache.org> on 2018/04/09 02:52:00 UTC
[jira] [Resolved] (CARBONDATA-2246) Fix not-enough-memory bugs in
unsafe data loading
[ https://issues.apache.org/jira/browse/CARBONDATA-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xuchuanyin resolved CARBONDATA-2246.
------------------------------------
Resolution: Fixed
> Fix not-enough-memory bugs in unsafe data loading
> -------------------------------------------------
>
> Key: CARBONDATA-2246
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2246
> Project: CarbonData
> Issue Type: Bug
> Reporter: xuchuanyin
> Assignee: xuchuanyin
> Priority: Major
> Time Spent: 3h 20m
> Remaining Estimate: 0h
>
> Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.
> The key properties to reproduce the bug are as following:
> ```
> 01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
> 02:
> 03: // unsafe sort memory manager
> 04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
> 05:
> 06: // unsafe working memory manager
> 07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
> 08:
> 09: // one unsafe page, better if loading_cores * this < memory
> 10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")
> ```
>
> Notice that the `OFFHEAP_SORT_CHUNK_SIZE_IN_MB` are exactly the same as `UNSAFE_WORKING_MEMORY_IN_MB` which will cause problem
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)