You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by kevinjmh <gi...@git.apache.org> on 2018/07/11 11:57:27 UTC
[GitHub] carbondata pull request #2490: [CARBONDATA-2727] Support create bloom datama...
GitHub user kevinjmh opened a pull request:
https://github.com/apache/carbondata/pull/2490
[CARBONDATA-2727] Support create bloom datamap on newly added column
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
- Add a result collector with rowId infomation for datamap rebuild if table schema is changed;
- Use keygenerator to retrieve surrogate value of dictIndexColumn from query result;
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/kevinjmh/carbondata bloom_new_column
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/2490.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #2490
----
commit 6ed28f07360ef88cbd5079eaf7d6b76ea69cddf1
Author: Manhua <ke...@...>
Date: 2018-07-11T11:39:31Z
support create bloom datamap on newly added column
----
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2490
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5892/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7230/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2490
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5886/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5960/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5816/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5803/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7243/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2490
LGTM
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202886338
--- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/AbstractBloomDataMapWriter.java ---
@@ -281,4 +287,13 @@ protected void releaseResouce() {
}
}
+ /**
+ * BloomDataMapBuilder(called when datamap rebuild) is set to true;
+ * BloomDataMapWriter(called when data load) is set to false;
+ *
+ * The reason for this is dict index column is already decoded to surrogate key when rebuild
+ * but it is still byte array if build datamap when loading
+ */
+ abstract boolean isRebuildProcess();
--- End diff --
fix by abstract method `convertDictionaryValue`
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202706208
--- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/AbstractBloomDataMapWriter.java ---
@@ -281,4 +287,13 @@ protected void releaseResouce() {
}
}
+ /**
+ * BloomDataMapBuilder(called when datamap rebuild) is set to true;
+ * BloomDataMapWriter(called when data load) is set to false;
+ *
+ * The reason for this is dict index column is already decoded to surrogate key when rebuild
+ * but it is still byte array if build datamap when loading
+ */
+ abstract boolean isRebuildProcess();
--- End diff --
@xuchuanyin has merged Writer and Rebuilder for datamap, if you add this function, then it is still like having different implementations for Writer and Rebuilder
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2490
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5779/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5949/
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202230960
--- Diff: integration/spark2/src/main/scala/org/apache/carbondata/datamap/IndexDataMapRebuildRDD.scala ---
@@ -154,89 +160,112 @@ class OriginalReadSupport(dataTypes: Array[DataType]) extends CarbonReadSupport[
*/
class RawBytesReadSupport(segmentProperties: SegmentProperties, indexColumns: Array[CarbonColumn])
extends CarbonReadSupport[Array[Object]] {
- var columnarSplitter: ColumnarSplitter = _
+ var dimensionKeyGenerator: KeyGenerator = _
+ // for the dictionary dimensions
+ var indexCol2IdxInDictArray: Map[String, Int] = Map()
// for the non dictionary dimensions
var indexCol2IdxInNoDictArray: Map[String, Int] = Map()
// for the measures
var indexCol2IdxInMeasureArray: Map[String, Int] = Map()
- // for the dictionary/date dimensions
- var dictIndexCol2MdkIndex: Map[String, Int] = Map()
- var mdkIndex2DictIndexCol: Map[Int, String] = Map()
- var existDim = false
+
+ /**
+ * use same way as restructure based result collector to prepare
+ * key generator to get surrogate value of dict column result.
+ * Refer to `RestructureBasedRowIdRawResultCollector`
+ */
--- End diff --
please add description to explain why this modification is needed compared with previous version. what's the scenario or intention?
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2490
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5827/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7028/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/2490
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/5845/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6017/
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202705370
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/RestructureBasedRowIdRawResultCollector.java ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.scan.collector.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.block.SegmentProperties;
+import org.apache.carbondata.core.keygenerator.KeyGenException;
+import org.apache.carbondata.core.keygenerator.KeyGenerator;
+import org.apache.carbondata.core.keygenerator.mdkey.MultiDimKeyVarLengthGenerator;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.encoder.Encoding;
+import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;
+import org.apache.carbondata.core.scan.executor.infos.BlockExecutionInfo;
+import org.apache.carbondata.core.scan.model.ProjectionDimension;
+import org.apache.carbondata.core.scan.model.ProjectionMeasure;
+import org.apache.carbondata.core.scan.result.BlockletScannedResult;
+import org.apache.carbondata.core.scan.wrappers.ByteArrayWrapper;
+import org.apache.carbondata.core.stats.QueryStatistic;
+import org.apache.carbondata.core.stats.QueryStatisticsConstants;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.core.util.DataTypeUtil;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+/**
+ * It is not a collector it is just a scanned result holder.
+ * most of the lines are copyied from `RestructureBasedRawResultCollector`, the difference in
+ * function is that this class return all the dimensions in a ByteArrayWrapper and append
+ * blockletNo/PageId/RowId at end of the row.
+ * This implementation refers to `RestructureBasedRawResultCollector`
+ */
+@InterfaceAudience.Internal
+public class RestructureBasedRowIdRawResultCollector extends RowIdRawBasedResultCollector {
--- End diff --
There are many duplicated code from `RestructureBasedRawResultCollector`, it is not good for maintenance. Can you optimize it.
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6031/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest this please
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2490
LGTM
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7255/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest sdv please
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest this please
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7173/
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202706832
--- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/AbstractBloomDataMapWriter.java ---
@@ -281,4 +287,13 @@ protected void releaseResouce() {
}
}
+ /**
+ * BloomDataMapBuilder(called when datamap rebuild) is set to true;
+ * BloomDataMapWriter(called when data load) is set to false;
+ *
+ * The reason for this is dict index column is already decoded to surrogate key when rebuild
+ * but it is still byte array if build datamap when loading
+ */
+ abstract boolean isRebuildProcess();
--- End diff --
It is better to abstract the difference of behavior instead of adding this.
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202230753
--- Diff: datamap/bloom/src/main/java/org/apache/carbondata/datamap/bloom/AbstractBloomDataMapWriter.java ---
@@ -173,7 +173,13 @@ protected void addValue2BloomIndex(int indexColIdx, Object value) {
} else {
if (indexColumns.get(indexColIdx).hasEncoding(Encoding.DICTIONARY)
|| indexColumns.get(indexColIdx).hasEncoding(Encoding.DIRECT_DICTIONARY)) {
- indexValue = convertDictionaryValue(indexColIdx, (byte[]) value);
+ if (value.getClass().isArray()) {
--- End diff --
Maybe we can add a variable to indicate whether it is from 'Rebuild(builder0' or 'DirectWrite(writer)'. Use the class type as condition is not proper.
Do not forget to add comments for that variable.
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest sdv please
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest this please
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by xuchuanyin <gi...@git.apache.org>.
Github user xuchuanyin commented on the issue:
https://github.com/apache/carbondata/pull/2490
please fix the comments
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest sdv please
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/2490
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by brijoobopanna <gi...@git.apache.org>.
Github user brijoobopanna commented on the issue:
https://github.com/apache/carbondata/pull/2490
retest sdv please
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7184/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7039/
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202707191
--- Diff: integration/spark2/src/test/scala/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapSuite.scala ---
@@ -377,6 +377,102 @@ class BloomCoarseGrainDataMapSuite extends QueryTest with BeforeAndAfterAll with
checkQuery("fakeDm", shouldHit = false)
}
+ test("test create bloom datamap on newly added column") {
--- End diff --
Can this PR work for Lucene datamap? I think it should be a general fix for all index datamap, right?
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202883458
--- Diff: integration/spark2/src/test/scala/org/apache/carbondata/datamap/bloom/BloomCoarseGrainDataMapSuite.scala ---
@@ -377,6 +377,102 @@ class BloomCoarseGrainDataMapSuite extends QueryTest with BeforeAndAfterAll with
checkQuery("fakeDm", shouldHit = false)
}
+ test("test create bloom datamap on newly added column") {
--- End diff --
mostly yes, it depends on the way lucene deals with dictionary column. This PR enables to get default value of newly added column by a new result collector in `IndexDataMapRebuildRDD` and other codes are worked for getting surrogate value of dictionary column relying on PR2425
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5859/
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/6003/
---
[GitHub] carbondata pull request #2490: [CARBONDATA-2727][BloomDataMap] Support creat...
Posted by kevinjmh <gi...@git.apache.org>.
Github user kevinjmh commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/2490#discussion_r202886715
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/collector/impl/RestructureBasedRowIdRawResultCollector.java ---
@@ -0,0 +1,265 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.core.scan.collector.impl;
+
+import java.util.ArrayList;
+import java.util.List;
+
+import org.apache.carbondata.common.annotations.InterfaceAudience;
+import org.apache.carbondata.core.constants.CarbonCommonConstants;
+import org.apache.carbondata.core.datastore.block.SegmentProperties;
+import org.apache.carbondata.core.keygenerator.KeyGenException;
+import org.apache.carbondata.core.keygenerator.KeyGenerator;
+import org.apache.carbondata.core.keygenerator.mdkey.MultiDimKeyVarLengthGenerator;
+import org.apache.carbondata.core.metadata.datatype.DataTypes;
+import org.apache.carbondata.core.metadata.encoder.Encoding;
+import org.apache.carbondata.core.metadata.schema.table.column.CarbonDimension;
+import org.apache.carbondata.core.scan.executor.infos.BlockExecutionInfo;
+import org.apache.carbondata.core.scan.model.ProjectionDimension;
+import org.apache.carbondata.core.scan.model.ProjectionMeasure;
+import org.apache.carbondata.core.scan.result.BlockletScannedResult;
+import org.apache.carbondata.core.scan.wrappers.ByteArrayWrapper;
+import org.apache.carbondata.core.stats.QueryStatistic;
+import org.apache.carbondata.core.stats.QueryStatisticsConstants;
+import org.apache.carbondata.core.util.CarbonUtil;
+import org.apache.carbondata.core.util.DataTypeUtil;
+
+import org.apache.commons.lang3.ArrayUtils;
+
+/**
+ * It is not a collector it is just a scanned result holder.
+ * most of the lines are copyied from `RestructureBasedRawResultCollector`, the difference in
+ * function is that this class return all the dimensions in a ByteArrayWrapper and append
+ * blockletNo/PageId/RowId at end of the row.
+ * This implementation refers to `RestructureBasedRawResultCollector`
+ */
+@InterfaceAudience.Internal
+public class RestructureBasedRowIdRawResultCollector extends RowIdRawBasedResultCollector {
--- End diff --
change to extends RestructureBasedRawResultCollector and add row id info when filling data
---
[GitHub] carbondata issue #2490: [CARBONDATA-2727][BloomDataMap] Support create bloom...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/2490
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7082/
---