You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by kumarvishal09 <gi...@git.apache.org> on 2018/10/22 18:27:04 UTC
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
GitHub user kumarvishal09 opened a pull request:
https://github.com/apache/carbondata/pull/2841
[WIP] Unsafe fallback to heap and unsafe query fix
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kumarvishal09/incubator-carbondata master_17
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2841.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2841
----
commit d8a89afb317cd8acec061be03ee9a81232479b75
Author: kumarvishal09 <ku...@...>
Date: 2018-10-18T07:31:28Z
fixed unsafe problem
----
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9190/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9237/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/935/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1164/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r228694154
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -125,13 +117,24 @@ private synchronized MemoryBlock allocateMemory(MemoryType memoryType, String ta
}
listOfMemoryBlock.add(memoryBlock);
if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(String.format("Creating working Memory block (%s) with size %d."
+ LOGGER.debug(String.format("Creating Offheap working Memory block (%s) with size %d."
+ " Total memory used %d Bytes, left %d Bytes.",
memoryBlock.toString(), memoryBlock.size(), memoryUsed, totalMemory - memoryUsed));
}
- return memoryBlock;
+ } else {
+ memoryBlock = MemoryAllocator.HEAP.allocate(memoryRequested);
+ Set<MemoryBlock> listOfMemoryBlock = taskIdToMemoryBlockMap.get(taskId);
+ if (null == listOfMemoryBlock) {
+ listOfMemoryBlock = new HashSet<>();
+ taskIdToMemoryBlockMap.put(taskId, listOfMemoryBlock);
--- End diff --
updated the variable name
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/955/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227241237
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/VarLengthColumnPageBase.java ---
@@ -125,46 +126,48 @@ public void setDoublePage(double[] doubleData) {
* Create a new column page for decimal page
*/
static ColumnPage newDecimalColumnPage(TableSpec.ColumnSpec columnSpec, byte[] lvEncodedBytes,
- String compressorName) throws MemoryException {
+ String compressorName, boolean isUnsafe) throws MemoryException {
DecimalConverterFactory.DecimalConverter decimalConverter =
DecimalConverterFactory.INSTANCE.getDecimalConverter(columnSpec.getPrecision(),
columnSpec.getScale());
int size = decimalConverter.getSize();
if (size < 0) {
return getLVBytesColumnPage(columnSpec, lvEncodedBytes,
DataTypes.createDecimalType(columnSpec.getPrecision(), columnSpec.getScale()),
- CarbonCommonConstants.INT_SIZE_IN_BYTE, compressorName);
+ CarbonCommonConstants.INT_SIZE_IN_BYTE, compressorName, isUnsafe);
} else {
// Here the size is always fixed.
- return getDecimalColumnPage(columnSpec, lvEncodedBytes, size, compressorName);
+ return getDecimalColumnPage(columnSpec, lvEncodedBytes, size, compressorName, isUnsafe);
}
}
/**
* Create a new column page based on the LV (Length Value) encoded bytes
*/
static ColumnPage newLVBytesColumnPage(TableSpec.ColumnSpec columnSpec, byte[] lvEncodedBytes,
- int lvLength, String compressorName) throws MemoryException {
+ int lvLength, String compressorName, boolean isUnsafe) throws MemoryException {
return getLVBytesColumnPage(columnSpec, lvEncodedBytes, DataTypes.BYTE_ARRAY,
- lvLength, compressorName);
+ lvLength, compressorName, isUnsafe);
}
/**
* Create a new column page based on the LV (Length Value) encoded bytes
*/
static ColumnPage newComplexLVBytesColumnPage(TableSpec.ColumnSpec columnSpec,
- byte[] lvEncodedBytes, int lvLength, String compressorName) throws MemoryException {
+ byte[] lvEncodedBytes, int lvLength, String compressorName, boolean isUnsafe)
+ throws MemoryException {
return getComplexLVBytesColumnPage(columnSpec, lvEncodedBytes, DataTypes.BYTE_ARRAY,
- lvLength, compressorName);
+ lvLength, compressorName, isUnsafe);
}
private static ColumnPage getDecimalColumnPage(TableSpec.ColumnSpec columnSpec,
- byte[] lvEncodedBytes, int size, String compressorName) throws MemoryException {
+ byte[] lvEncodedBytes, int size, String compressorName, boolean unsafe)
+ throws MemoryException {
TableSpec.ColumnSpec spec = TableSpec.ColumnSpec
.newInstance(columnSpec.getFieldName(), DataTypes.INT, ColumnType.MEASURE);
ColumnPage rowOffset = ColumnPage.newPage(
new ColumnPageEncoderMeta(spec, DataTypes.INT, compressorName),
- CarbonV3DataFormatConstants.NUMBER_OF_ROWS_PER_BLOCKLET_COLUMN_PAGE_DEFAULT);
+ CarbonV3DataFormatConstants.NUMBER_OF_ROWS_PER_BLOCKLET_COLUMN_PAGE_DEFAULT, unsafe);
--- End diff --
change the flag name `unsafe --> isUnsafe`, same as done in the above the changes. Keep the flag name consistent everywhere
---
[GitHub] carbondata issue #2841: [CARBONDATA-3047] Added fallback mechanism when offh...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9338/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r228694135
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -125,13 +117,24 @@ private synchronized MemoryBlock allocateMemory(MemoryType memoryType, String ta
}
listOfMemoryBlock.add(memoryBlock);
if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(String.format("Creating working Memory block (%s) with size %d."
+ LOGGER.debug(String.format("Creating Offheap working Memory block (%s) with size %d."
+ " Total memory used %d Bytes, left %d Bytes.",
memoryBlock.toString(), memoryBlock.size(), memoryUsed, totalMemory - memoryUsed));
}
- return memoryBlock;
+ } else {
+ memoryBlock = MemoryAllocator.HEAP.allocate(memoryRequested);
+ Set<MemoryBlock> listOfMemoryBlock = taskIdToMemoryBlockMap.get(taskId);
+ if (null == listOfMemoryBlock) {
+ listOfMemoryBlock = new HashSet<>();
+ taskIdToMemoryBlockMap.put(taskId, listOfMemoryBlock);
--- End diff --
This is not required updated the same
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r228694119
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -77,21 +77,12 @@
if (takenSize < defaultSize) {
takenSize = defaultSize;
LOGGER.warn(String.format(
- "It is not recommended to set unsafe working memory size less than %sMB,"
+ "It is not recommended to set offheap working memory size less than %sMB,"
+ " so setting default value to %d",
CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT, defaultSize));
}
takenSize = takenSize * 1024 * 1024;
} else {
- long maxMemory = Runtime.getRuntime().maxMemory() * 60 / 100;
- if (takenSize == 0L) {
- takenSize = maxMemory;
- } else {
- takenSize = takenSize * 1024 * 1024;
- if (takenSize > maxMemory) {
- takenSize = maxMemory;
- }
- }
--- End diff --
add comment
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227235905
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/DimensionRawColumnChunk.java ---
@@ -168,13 +169,16 @@ public CarbonDictionary getLocalDictionary() {
*/
public static CarbonDictionary getDictionary(LocalDictionaryChunk localDictionaryChunk,
Compressor compressor) throws IOException, MemoryException {
+ boolean isUnsafe = Boolean.parseBoolean(CarbonProperties.getInstance()
+ .getProperty(CarbonCommonConstants.ENABLE_UNSAFE_IN_QUERY_EXECUTION,
+ CarbonCommonConstants.ENABLE_UNSAFE_IN_QUERY_EXECUTION_DEFAULTVALUE));
--- End diff --
Move this flag inside if null check loop
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227246446
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -124,8 +116,21 @@ private synchronized MemoryBlock allocateMemory(MemoryType memoryType, String ta
+ "Bytes");
}
return allocate;
+ } else {
+ MemoryBlock allocate = MemoryAllocator.HEAP.allocate(memoryRequested);
+ Set<MemoryBlock> listOfMemoryBlock = taskIdToMemoryBlockMap.get(taskId);
+ if (null == listOfMemoryBlock) {
+ listOfMemoryBlock = new HashSet<>();
+ taskIdToMemoryBlockMap.put(taskId, listOfMemoryBlock);
+ }
+ listOfMemoryBlock.add(allocate);
+ if (LOGGER.isDebugEnabled()) {
+ LOGGER.debug("Memory block (" + allocate + ") is created with size " + allocate.size()
+ + ". Total memory used " + memoryUsed + "Bytes, left " + (totalMemory - memoryUsed)
+ + "Bytes");
--- End diff --
for On-Heap case totalMemory can be 0 also if not configured...so bytes left cannot be printed in the debug logs. For printing memory used add the allocated size to memory used for on-heap case
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1183/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/920/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r228692328
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -125,13 +117,24 @@ private synchronized MemoryBlock allocateMemory(MemoryType memoryType, String ta
}
listOfMemoryBlock.add(memoryBlock);
if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(String.format("Creating working Memory block (%s) with size %d."
+ LOGGER.debug(String.format("Creating Offheap working Memory block (%s) with size %d."
+ " Total memory used %d Bytes, left %d Bytes.",
memoryBlock.toString(), memoryBlock.size(), memoryUsed, totalMemory - memoryUsed));
}
- return memoryBlock;
+ } else {
+ memoryBlock = MemoryAllocator.HEAP.allocate(memoryRequested);
+ Set<MemoryBlock> listOfMemoryBlock = taskIdToMemoryBlockMap.get(taskId);
+ if (null == listOfMemoryBlock) {
+ listOfMemoryBlock = new HashSet<>();
+ taskIdToMemoryBlockMap.put(taskId, listOfMemoryBlock);
--- End diff --
Is it required to keep tracking of onheap blocks in map.
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1129/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227239817
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java ---
@@ -73,8 +73,9 @@ public ComplexColumnPage(List<ComplexColumnInfo> complexColumnInfoList) {
* if memory is not sufficient
*/
public void initialize(Map<String, LocalDictionaryGenerator> columnToDictMap, int pageSize,
- String columnCompressor) throws MemoryException {
+ String columnCompressor, boolean isUnsafe) throws MemoryException {
DataType dataType;
+
--- End diff --
Remove extra line
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9199/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9333/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1141/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/967/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9219/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227238938
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/impl/DimensionRawColumnChunk.java ---
@@ -168,13 +169,16 @@ public CarbonDictionary getLocalDictionary() {
*/
public static CarbonDictionary getDictionary(LocalDictionaryChunk localDictionaryChunk,
Compressor compressor) throws IOException, MemoryException {
+ boolean isUnsafe = Boolean.parseBoolean(CarbonProperties.getInstance()
--- End diff --
similar to other places below this is also a query execution flow...so here also the parameter `CarbonCommonConstants.ENABLE_UNSAFE_IN_QUERY_EXECUTION` need to be considered for deciding on unsafe or only one parameter is sufficient?
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r228692401
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -125,13 +117,24 @@ private synchronized MemoryBlock allocateMemory(MemoryType memoryType, String ta
}
listOfMemoryBlock.add(memoryBlock);
if (LOGGER.isDebugEnabled()) {
- LOGGER.debug(String.format("Creating working Memory block (%s) with size %d."
+ LOGGER.debug(String.format("Creating Offheap working Memory block (%s) with size %d."
+ " Total memory used %d Bytes, left %d Bytes.",
memoryBlock.toString(), memoryBlock.size(), memoryUsed, totalMemory - memoryUsed));
}
- return memoryBlock;
+ } else {
+ memoryBlock = MemoryAllocator.HEAP.allocate(memoryRequested);
+ Set<MemoryBlock> listOfMemoryBlock = taskIdToMemoryBlockMap.get(taskId);
+ if (null == listOfMemoryBlock) {
+ listOfMemoryBlock = new HashSet<>();
+ taskIdToMemoryBlockMap.put(taskId, listOfMemoryBlock);
--- End diff --
And also please change the `taskIdToMemoryBlockMap` to `taskIdToOffHeapBlockMap`
---
[GitHub] carbondata issue #2841: [CARBONDATA-3047] Added fallback mechanism when offh...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2841
LGTM
---
[GitHub] carbondata pull request #2841: [CARBONDATA-3047] Added fallback mechanism wh...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2841
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r228692174
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -77,21 +77,12 @@
if (takenSize < defaultSize) {
takenSize = defaultSize;
LOGGER.warn(String.format(
- "It is not recommended to set unsafe working memory size less than %sMB,"
+ "It is not recommended to set offheap working memory size less than %sMB,"
+ " so setting default value to %d",
CarbonCommonConstants.UNSAFE_WORKING_MEMORY_IN_MB_DEFAULT, defaultSize));
}
takenSize = takenSize * 1024 * 1024;
} else {
- long maxMemory = Runtime.getRuntime().maxMemory() * 60 / 100;
- if (takenSize == 0L) {
- takenSize = maxMemory;
- } else {
- takenSize = takenSize * 1024 * 1024;
- if (takenSize > maxMemory) {
- takenSize = maxMemory;
- }
- }
--- End diff --
Please add a comment why memory limit is not required to take for ONHEAP
---
[GitHub] carbondata issue #2841: [CARBONDATA-3047] Added fallback mechanism when offh...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1286/
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227339904
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -124,8 +116,21 @@ private synchronized MemoryBlock allocateMemory(MemoryType memoryType, String ta
+ "Bytes");
}
return allocate;
+ } else {
+ MemoryBlock allocate = MemoryAllocator.HEAP.allocate(memoryRequested);
--- End diff --
Its not actually fall back once offheap memory is available it will switch to offheap ...depends on offheap memory , i will add debug log
---
[GitHub] carbondata pull request #2841: [WIP] Unsafe fallback to heap and unsafe quer...
Posted by manishgupta88 <gi...@git.apache.org>.
Github user manishgupta88 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2841#discussion_r227246688
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -292,7 +300,7 @@ private void addComplexColumn(int index, int rowId, byte[] complexColumns) {
complexDimensionPages[index] = new ComplexColumnPage(complexColumnInfoList);
try {
complexDimensionPages[index].initialize(
- model.getColumnLocalDictGenMap(), pageSize, columnCompressor);
+ model.getColumnLocalDictGenMap(), pageSize, columnCompressor, unsafe);
--- End diff --
replace `unsafe --> isUnsafe`
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1068/
---
[GitHub] carbondata issue #2841: [CARBONDATA-3047] Added fallback mechanism when offh...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/1073/
---
[GitHub] carbondata issue #2841: [WIP] Unsafe fallback to heap and unsafe query fix
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2841
Build Success with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1281/
---