You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by "xuchuanyin (JIRA)" <ji...@apache.org> on 2018/03/12 07:57:00 UTC
[jira] [Updated] (CARBONDATA-2246) Fix out-of-memory bugs in unsafe
data loading
[ https://issues.apache.org/jira/browse/CARBONDATA-2246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
xuchuanyin updated CARBONDATA-2246:
-----------------------------------
Description:
Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.
The key properties to reproduce the bug are as following:
```
01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
02:
03: // unsafe sort memory manager
04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
05:
06: // unsafe working memory manager
07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
08:
09: // one unsafe page, better if loading_cores * this < memory
10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")
```
was:
Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.
The key properties to reproduce the bug are as following:
```
```
> Fix out-of-memory bugs in unsafe data loading
> ---------------------------------------------
>
> Key: CARBONDATA-2246
> URL: https://issues.apache.org/jira/browse/CARBONDATA-2246
> Project: CarbonData
> Issue Type: Bug
> Reporter: xuchuanyin
> Assignee: xuchuanyin
> Priority: Major
>
> Currently in carbon data loading, if we enable the unsafe loading and specify corresponding properties, data loading will end in OOM.
> The key properties to reproduce the bug are as following:
> ```
> 01: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_INMEMORY_MERGE_SORT, "true") CarbonProperties.getInstance().addProperty(CarbonCommonConstants.ENABLE_UNSAFE_SORT, "true")
> 02:
> 03: // unsafe sort memory manager
> 04: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.IN_MEMORY_STORAGE_FOR_SORTED_DATA_IN_MB, "1024")
> 05:
> 06: // unsafe working memory manager
> 07: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB, "512")
> 08:
> 09: // one unsafe page, better if loading_cores * this < memory
> 10: CarbonProperties.getInstance().addProperty(CarbonCommonConstants.OFFHEAP_SORT_CHUNK_SIZE_IN_MB, "512")
> ```
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)