You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by jackylk <gi...@git.apache.org> on 2017/05/30 00:41:45 UTC
[GitHub] carbondata pull request #971: [WIP] Refactor writer to use ColumnPage/TableS...
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/971
[WIP] Refactor writer to use ColumnPage/TableStatistics/TableSpec
This PR extracts interface that used for:
- make ColumnPage unsafe
- Encoding override
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata write
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/971.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #971
----
commit 9523b1796abd560fe8fcec1d869d4044cf7e7449
Author: jackylk <ja...@huawei.com>
Date: 2017-05-27T18:14:48Z
use ColumnPage in writer
commit 4df3861cf7e3eb124ad45106185bf70f68c0c319
Author: jackylk <ja...@huawei.com>
Date: 2017-05-27T18:22:31Z
remove WriterCompressModel
commit 7ffd5098008847bf7fb7c7651ea8693efed65bfd
Author: jackylk <ja...@huawei.com>
Date: 2017-05-28T14:55:29Z
add PrimitiveColumnPage
commit ce81299766e449f02f00420c26ce16bcbd915ac9
Author: jackylk <ja...@huawei.com>
Date: 2017-05-30T00:36:20Z
add TableSpec
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119887721
--- Diff: docs/faq.md ---
@@ -123,7 +123,7 @@ id city name
3 davi shenzhen
```
-As result shows, the second column is city in carbon table, but what inside is name, such as jack. This phenomenon is same with insert data into hive table.
+As encodedData shows, the second column is city in carbon table, but what inside is name, such as jack. This phenomenon is same with insert data into hive table.
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r120013941
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/measure/v3/CompressedMeasureChunkFileBasedReaderV3.java ---
@@ -220,13 +223,25 @@ public CompressedMeasureChunkFileBasedReaderV3(BlockletInfo blockletInfo, String
valueEncodeMeta.add(CarbonUtil
.deserializeEncoderMetaNew(measureColumnChunk.getEncoder_meta().get(i).array()));
}
- WriterCompressModel compressionModel = CarbonUtil.getValueCompressionModel(valueEncodeMeta);
- ValueCompressionHolder values = compressionModel.getValueCompressionHolder()[0];
+
+ MeasurePageStatistics stats = CarbonUtil.getMeasurePageStats(valueEncodeMeta);
--- End diff --
I will change this code in #987, both encode and decode will use `EncodingStrategy` class to create the Encoding
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119885662
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java ---
@@ -74,4 +75,70 @@ public void putComplexData(int rowId, int depth, List<byte[]> value) {
public int getDepth() {
return depth;
}
+
+ @Override
+ public void putInt(int rowId, int value) {
--- End diff --
How about add default implementation in ColumnPage and override the one that is needed in ComplexColumnPage
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/209/<h2>Failed Tests: <span class='status-failure'>2</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/209/org.apache.carbondata$carbondata-spark/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/209/org.apache.carbondata$carbondata-spark/testReport/org.apache.carbondata.spark.util/AutoHighCardinalityIdentifyTestCase/skip_auto_identify_high_cardinality_column_for_column_group/'><strong>org.apache.carbondata.spark.util.AutoHighCardinalityIdentifyTestCase.skip auto identify high cardinality column for column group</strong></a></li></ul><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/
209/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/209/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2274/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/147/<h2>Failed Tests: <span class='status-failure'>13</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>13</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_hive/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from hive</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allq
ueries/InsertIntoCarbonTableTestCase/insert_from_carbon_select_columns/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from carbon-select columns</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_carbon_select___columns/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from carbon-select * columns</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_functions/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with functions</strong></a></li><li><a href='https
://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_different_names_and_aliases/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with different names and aliases</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_existing_load_pass/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into existing load-pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_in
to_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_select_from_same_table/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert select from same table</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Less_Than_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Less Than Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/c
arbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Les_Than_equal_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Les Than equal Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Greater_Than_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Greater Than Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Greater_Than_equal_to_Filter_all_null/'><strong>org.apache.carbondata.spark.te
stsuite.filterexpr.GrtLtFilterProcessorTestCase.Greater Than equal to Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/147/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.iud/HorizontalCompactionTestCase/test_IUD_Horizontal_Compaction_Check_Column_Cardinality/'><strong>org.apache.carbondata.spark.testsuite.iud.HorizontalCompactionTestCase.test IUD Horizontal Compaction Check Column Cardinality</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119909916
--- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -841,17 +837,14 @@ public static WriterCompressModel getValueCompressionModel(
*/
for (int i = 0; i < dataTypeSelected.length; i++) { // always 1
ValueEncoderMeta valueEncoderMeta = encodeMetaList.get(i);
- maxValue[i] = valueEncoderMeta.getMaxValue();
- minValue[i] = valueEncoderMeta.getMinValue();
- uniqueValue[i] = valueEncoderMeta.getUniqueValue();
+ max[i] = valueEncoderMeta.getMaxValue();
+ min[i] = valueEncoderMeta.getMinValue();
+ nonExistValue[i] = valueEncoderMeta.getUniqueValue();
decimal[i] = valueEncoderMeta.getDecimal();
type[i] = valueEncoderMeta.getType();
dataTypeSelected[i] = valueEncoderMeta.getDataTypeSelected();
}
- MeasureMetaDataModel measureMetadataModel =
- new MeasureMetaDataModel(minValue, maxValue, decimal, dataTypeSelected.length, uniqueValue,
- type, dataTypeSelected);
- return ValueCompressionUtil.getWriterCompressModel(measureMetadataModel);
+ return MeasurePageStatistics.build(min, max, nonExistValue, decimal, type, dataTypeSelected);
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119783944
--- Diff: docs/faq.md ---
@@ -123,7 +123,7 @@ id city name
3 davi shenzhen
```
-As result shows, the second column is city in carbon table, but what inside is name, such as jack. This phenomenon is same with insert data into hive table.
+As encodedData shows, the second column is city in carbon table, but what inside is name, such as jack. This phenomenon is same with insert data into hive table.
--- End diff --
I think changed this file by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2160/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2332/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/971
@jackylk Test cases are failing as it is aborting because of error, please check.
```
17/06/06 00:43:20 ERROR CarbonFactDataHandlerColumnar: pool-694-thread-1 Error in producer
java.lang.ArrayIndexOutOfBoundsException
at java.lang.System.arraycopy(Native Method)
at org.apache.carbondata.core.util.ByteUtil.flatten(ByteUtil.java:697)
at org.apache.carbondata.processing.store.DefaultEncoder.encodeAndCompressDimensions(DefaultEncoder.java:204)
at org.apache.carbondata.processing.store.DefaultEncoder.encode(DefaultEncoder.java:64)
at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.processDataRows(CarbonFactDataHandlerColumnar.java:352)
at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar.access$500(CarbonFactDataHandlerColumnar.java:60)
at org.apache.carbondata.processing.store.CarbonFactDataHandlerColumnar$Producer.call(CarbonFactDataHandlerColumnar.java:736)
```
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119895320
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/measure/v3/CompressedMeasureChunkFileBasedReaderV3.java ---
@@ -220,13 +223,25 @@ public CompressedMeasureChunkFileBasedReaderV3(BlockletInfo blockletInfo, String
valueEncodeMeta.add(CarbonUtil
.deserializeEncoderMetaNew(measureColumnChunk.getEncoder_meta().get(i).array()));
}
- WriterCompressModel compressionModel = CarbonUtil.getValueCompressionModel(valueEncodeMeta);
- ValueCompressionHolder values = compressionModel.getValueCompressionHolder()[0];
+
+ MeasurePageStatistics stats = CarbonUtil.getMeasurePageStats(valueEncodeMeta);
--- End diff --
This code is repeated in CompressedMeasureChunkFileBasedReaderV2 and CompressedMeasureChunkFileBasedReaderV3, if we move it to a shared function, a result object need to be created, since multiple results are needed (stats, convertedType, values).
Actually in #987 , usage of `CompressionFinder` is removed in write path, I think another PR is needed to remove it for read path also, after this is done, `CompressionFinder` related class and the whole `org.apache.carbondata.core.datastore.compression` package.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2273/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/971
@ravipesala I have rebased
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119887697
--- Diff: docs/dml-operation-on-carbondata.md ---
@@ -211,7 +211,7 @@ By default the above configuration will be false.
### Examples
```
-INSERT INTO table1 SELECT item1 ,sum(item2 + 1000) as result FROM
+INSERT INTO table1 SELECT item1 ,sum(item2 + 1000) as encodedData FROM
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2222/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2281/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119887373
--- Diff: dev/scalastyle-config.xml ---
@@ -193,12 +193,12 @@ This file is divided into 3 sections:
</check>
<check customId="awaitresult" level="error" class="org.scalastyle.file.RegexChecker" enabled="true">
- <parameters><parameter name="regex">Await\.result</parameter></parameters>
+ <parameters><parameter name="regex">Await\.encodedData</parameter></parameters>
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119783933
--- Diff: docs/dml-operation-on-carbondata.md ---
@@ -211,7 +211,7 @@ By default the above configuration will be false.
### Examples
```
-INSERT INTO table1 SELECT item1 ,sum(item2 + 1000) as result FROM
+INSERT INTO table1 SELECT item1 ,sum(item2 + 1000) as encodedData FROM
--- End diff --
I think changed this file by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/971
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119804061
--- Diff: dev/findbugs-exclude.xml ---
@@ -143,7 +143,7 @@
This method returns a value that is not checked. The return value should be checked since
it can indicate an unusual or unexpected function execution. For example, the
File.delete() method returns false if the file could not be successfully deleted
- (rather than throwing an Exception). If you don't check the result, you won't notice
+ (rather than throwing an Exception). If you don't check the encodedData, you won't notice
--- End diff --
I think added by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119890152
--- Diff: dev/findbugs-exclude.xml ---
@@ -143,7 +143,7 @@
This method returns a value that is not checked. The return value should be checked since
it can indicate an unusual or unexpected function execution. For example, the
File.delete() method returns false if the file could not be successfully deleted
- (rather than throwing an Exception). If you don't check the result, you won't notice
+ (rather than throwing an Exception). If you don't check the encodedData, you won't notice
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119887844
--- Diff: docs/release-guide.md ---
@@ -109,7 +109,7 @@ staging repository and promote the artifacts to Maven Central.
4. Choose `User Token` from the dropdown, then click `Access User Token`. Copy a snippet of the
Maven XML configuration block.
5. Insert this snippet twice into your global Maven `settings.xml` file, typically `${HOME]/
-.m2/settings.xml`. The end result should look like this, where `TOKEN_NAME` and `TOKEN_PASSWORD`
+.m2/settings.xml`. The end encodedData should look like this, where `TOKEN_NAME` and `TOKEN_PASSWORD`
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119782797
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/reader/measure/v3/CompressedMeasureChunkFileBasedReaderV3.java ---
@@ -220,13 +223,25 @@ public CompressedMeasureChunkFileBasedReaderV3(BlockletInfo blockletInfo, String
valueEncodeMeta.add(CarbonUtil
.deserializeEncoderMetaNew(measureColumnChunk.getEncoder_meta().get(i).array()));
}
- WriterCompressModel compressionModel = CarbonUtil.getValueCompressionModel(valueEncodeMeta);
- ValueCompressionHolder values = compressionModel.getValueCompressionHolder()[0];
+
+ MeasurePageStatistics stats = CarbonUtil.getMeasurePageStats(valueEncodeMeta);
--- End diff --
it seems this code is repeated in all readers, so better move abstract class or utility
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/145/<h2>Failed Tests: <span class='status-failure'>13</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>13</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_hive/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from hive</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allq
ueries/InsertIntoCarbonTableTestCase/insert_from_carbon_select_columns/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from carbon-select columns</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_carbon_select___columns/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from carbon-select * columns</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_functions/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with functions</strong></a></li><li><a href='https
://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_different_names_and_aliases/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with different names and aliases</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_existing_load_pass/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into existing load-pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_in
to_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_select_from_same_table/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert select from same table</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Less_Than_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Less Than Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/c
arbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Les_Than_equal_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Les Than equal Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Greater_Than_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Greater Than Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Greater_Than_equal_to_Filter_all_null/'><strong>org.apache.carbondata.spark.te
stsuite.filterexpr.GrtLtFilterProcessorTestCase.Greater Than equal to Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/145/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.iud/HorizontalCompactionTestCase/test_IUD_Horizontal_Compaction_Check_Column_Cardinality/'><strong>org.apache.carbondata.spark.testsuite.iud.HorizontalCompactionTestCase.test IUD Horizontal Compaction Check Column Cardinality</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/153/<h2>Build result: FAILURE</span></h2>[...truncated 54.27 KB...][JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Processing #152 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Spark Common #152 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examp
les-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Common #152 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Parent #
152 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Core #152 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Hadoop #152 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom
.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Spark #152 to compare, so performing full copy of artifactschannel stoppedSetting status of eab163752334887b0dc9c6b947fbb79e3dba5214 to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/153/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119782441
--- Diff: LICENSE ---
@@ -157,7 +157,7 @@
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
- result of this License or out of the use or inability to use the
+ encodedData of this License or out of the use or inability to use the
--- End diff --
I think changed by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119889985
--- Diff: core/src/test/java/org/apache/carbondata/core/datastore/chunk/reader/measure/CompressedMeasureChunkFileBasedReaderTest.java ---
@@ -1,90 +0,0 @@
-/*
--- End diff --
ok. And I will remove `CompressedDimensionChunkFileBasedReaderTest` also
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2218/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119783972
--- Diff: docs/release-guide.md ---
@@ -109,7 +109,7 @@ staging repository and promote the artifacts to Maven Central.
4. Choose `User Token` from the dropdown, then click `Access User Token`. Copy a snippet of the
Maven XML configuration block.
5. Insert this snippet twice into your global Maven `settings.xml` file, typically `${HOME]/
-.m2/settings.xml`. The end result should look like this, where `TOKEN_NAME` and `TOKEN_PASSWORD`
+.m2/settings.xml`. The end encodedData should look like this, where `TOKEN_NAME` and `TOKEN_PASSWORD`
--- End diff --
I think changed this file by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119887960
--- Diff: docs/useful-tips-on-carbondata.md ---
@@ -127,7 +127,7 @@ query performance. The create table command can be modified as below :
TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI',
'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME');
```
- The result of performance analysis of test-case shows reduction in query execution time from 15 to 3 seconds, thereby improving performance by nearly 5 times.
+ The encodedData of performance analysis of test-case shows reduction in query execution time from 15 to 3 seconds, thereby improving performance by nearly 5 times.
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/54/<h2>Failed Tests: <span class='status-failure'>3</span></h2><h3><a name='carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/54/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>3</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/54/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/54/org.apache.carbondata$carbondata-spark-common-test/testReport/org.ap
ache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_offer_column_dictionary_file/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when offer column dictionary file</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/54/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_do_incremental_load/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when do incremental load</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/23/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/971
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119888798
--- Diff: core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java ---
@@ -670,4 +671,23 @@ public static int putBytes(byte[] tgtBytes, int tgtOffset, byte[] srcBytes, int
System.arraycopy(srcBytes, srcOffset, tgtBytes, tgtOffset, srcLength);
return tgtOffset + srcLength;
}
+
+ /**
+ * flatten the byte[][] to byte[] and return data after applying compression by compressor
+ * @param compressor compressor to use
+ * @return compressed data
+ */
+ public static byte[] flattenAndCompress(Compressor compressor, byte[][] byteArrayData) {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119801431
--- Diff: core/src/main/java/org/apache/carbondata/core/util/ByteUtil.java ---
@@ -670,4 +671,23 @@ public static int putBytes(byte[] tgtBytes, int tgtOffset, byte[] srcBytes, int
System.arraycopy(srcBytes, srcOffset, tgtBytes, tgtOffset, srcLength);
return tgtOffset + srcLength;
}
+
+ /**
+ * flatten the byte[][] to byte[] and return data after applying compression by compressor
+ * @param compressor compressor to use
+ * @return compressed data
+ */
+ public static byte[] flattenAndCompress(Compressor compressor, byte[][] byteArrayData) {
--- End diff --
Since it is ByteUtil, better use only flatten not compress, compress can be done in caller method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119781493
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -65,20 +68,20 @@
this.pageSize = pageSize;
keyColumnPage = new KeyColumnPage(pageSize,
model.getSegmentProperties().getDimensionPartitions().length);
- noDictDimensionPage = new VarLengthColumnPage[model.getNoDictionaryCount()];
+ noDictDimensionPage = new PrimitiveColumnPage[model.getNoDictionaryCount()];
for (int i = 0; i < noDictDimensionPage.length; i++) {
- noDictDimensionPage[i] = new VarLengthColumnPage(pageSize);
+ noDictDimensionPage[i] = new PrimitiveColumnPage(DataType.STRING, pageSize);
--- End diff --
Primitive column page should only contains primitive data types, it should not contain string or decimal. So better create another page for string and decimal.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/152/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119783989
--- Diff: docs/useful-tips-on-carbondata.md ---
@@ -127,7 +127,7 @@ query performance. The create table command can be modified as below :
TBLPROPERTIES ( 'DICTIONARY_EXCLUDE'='MSISDN,HOST,IMSI',
'DICTIONARY_INCLUDE'='Dime_1,END_TIME,BEGIN_TIME');
```
- The result of performance analysis of test-case shows reduction in query execution time from 15 to 3 seconds, thereby improving performance by nearly 5 times.
+ The encodedData of performance analysis of test-case shows reduction in query execution time from 15 to 3 seconds, thereby improving performance by nearly 5 times.
--- End diff --
I think changed this file by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119783237
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/MeasurePageStatistics.java ---
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.statistics;
+
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+public class MeasurePageStatistics {
--- End diff --
I think better give other name, it confuses. it is just holds the data like holder. so better name as VO or holder
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119909710
--- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java ---
@@ -549,24 +562,23 @@ private static ValueEncoderMeta deserializeValueEncoderMeta(ByteBuffer byteBuffe
}
- private static WriterCompressModel getValueCompressionModel(ValueEncoderMeta[] encoderMetas) {
- Object[] maxValue = new Object[encoderMetas.length];
- Object[] minValue = new Object[encoderMetas.length];
- int[] decimalLength = new int[encoderMetas.length];
- Object[] uniqueValue = new Object[encoderMetas.length];
- DataType[] aggType = new DataType[encoderMetas.length];
+ private static MeasurePageStatistics getMeasurePageStats(ValueEncoderMeta[] encoderMetas) {
+ Object[] max = new Object[encoderMetas.length];
+ Object[] min = new Object[encoderMetas.length];
+ int[] decimal = new int[encoderMetas.length];
+ Object[] nonExistValue = new Object[encoderMetas.length];
+ DataType[] types = new DataType[encoderMetas.length];
byte[] dataTypeSelected = new byte[encoderMetas.length];
for (int i = 0; i < encoderMetas.length; i++) {
- maxValue[i] = encoderMetas[i].getMaxValue();
- minValue[i] = encoderMetas[i].getMinValue();
- decimalLength[i] = encoderMetas[i].getDecimal();
- uniqueValue[i] = encoderMetas[i].getUniqueValue();
- aggType[i] = encoderMetas[i].getType();
+ max[i] = encoderMetas[i].getMaxValue();
+ min[i] = encoderMetas[i].getMinValue();
+ decimal[i] = encoderMetas[i].getDecimal();
+ nonExistValue[i] = encoderMetas[i].getUniqueValue();
+ types[i] = encoderMetas[i].getType();
dataTypeSelected[i] = encoderMetas[i].getDataTypeSelected();
}
- return ValueCompressionUtil
- .getWriterCompressModel(maxValue, minValue, decimalLength, uniqueValue, aggType,
- dataTypeSelected);
+
+ return MeasurePageStatistics.build(min, max, nonExistValue, decimal, types, dataTypeSelected);
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/212/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2272/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119781980
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/ColumnPageStatistics.java ---
@@ -114,6 +115,46 @@ private int getDecimalCount(double value) {
return decimalPlaces;
}
+ /**
+ * return min value as byte array
+ */
+ public byte[] minBytes() {
+ return getValueAsBytes(getMin());
+ }
+
+ /**
+ * return max value as byte array
+ */
+ public byte[] maxBytes() {
+ return getValueAsBytes(getMax());
+ }
+
+ /**
+ * convert value to byte array
+ */
+ private byte[] getValueAsBytes(Object value) {
+ ByteBuffer b = null;
+ Object max = getMax();
--- End diff --
We should use `value` not `getMax()`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119886685
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/ColumnPageStatistics.java ---
@@ -114,6 +115,46 @@ private int getDecimalCount(double value) {
return decimalPlaces;
}
+ /**
+ * return min value as byte array
+ */
+ public byte[] minBytes() {
+ return getValueAsBytes(getMin());
+ }
+
+ /**
+ * return max value as byte array
+ */
+ public byte[] maxBytes() {
+ return getValueAsBytes(getMax());
+ }
+
+ /**
+ * convert value to byte array
+ */
+ private byte[] getValueAsBytes(Object value) {
+ ByteBuffer b = null;
+ Object max = getMax();
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119802768
--- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonUtil.java ---
@@ -841,17 +837,14 @@ public static WriterCompressModel getValueCompressionModel(
*/
for (int i = 0; i < dataTypeSelected.length; i++) { // always 1
ValueEncoderMeta valueEncoderMeta = encodeMetaList.get(i);
- maxValue[i] = valueEncoderMeta.getMaxValue();
- minValue[i] = valueEncoderMeta.getMinValue();
- uniqueValue[i] = valueEncoderMeta.getUniqueValue();
+ max[i] = valueEncoderMeta.getMaxValue();
+ min[i] = valueEncoderMeta.getMinValue();
+ nonExistValue[i] = valueEncoderMeta.getUniqueValue();
decimal[i] = valueEncoderMeta.getDecimal();
type[i] = valueEncoderMeta.getType();
dataTypeSelected[i] = valueEncoderMeta.getDataTypeSelected();
}
- MeasureMetaDataModel measureMetadataModel =
- new MeasureMetaDataModel(minValue, maxValue, decimal, dataTypeSelected.length, uniqueValue,
- type, dataTypeSelected);
- return ValueCompressionUtil.getWriterCompressModel(measureMetadataModel);
+ return MeasurePageStatistics.build(min, max, nonExistValue, decimal, type, dataTypeSelected);
--- End diff --
Better pass `ValueEncoderMeta` list to `build` method to avoid duplicate code in other places
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2158/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119886349
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/ColumnPageStatistics.java ---
@@ -33,30 +34,30 @@
* the unique value is the non-exist value in the row,
* and will be used as storage key for null values of measures
*/
- private Object uniqueValue;
+ private Object nonExistValue;
--- End diff --
But it will be written into file by `CarbonMetadataUtil.serializeEncodeMetaUsingByteBuffer`, you mean read path is not using it?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119783901
--- Diff: dev/scalastyle-config.xml ---
@@ -193,12 +193,12 @@ This file is divided into 3 sections:
</check>
<check customId="awaitresult" level="error" class="org.scalastyle.file.RegexChecker" enabled="true">
- <parameters><parameter name="regex">Await\.result</parameter></parameters>
+ <parameters><parameter name="regex">Await\.encodedData</parameter></parameters>
--- End diff --
I think changed this file by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/25/<h2>Failed Tests: <span class='status-failure'>5</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/25/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>5</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/25/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/25/org.apache.carbondata$carbondata-spark-common-test/testReport/org.ap
ache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_offer_column_dictionary_file/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when offer column dictionary file</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/25/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_do_incremental_load/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when do incremental load</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/25/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_hive_sum
_expression/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from hive-sum expression</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/25/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/88/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119781704
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/ColumnPageStatistics.java ---
@@ -33,30 +34,30 @@
* the unique value is the non-exist value in the row,
* and will be used as storage key for null values of measures
*/
- private Object uniqueValue;
+ private Object nonExistValue;
--- End diff --
I don't think we are using this now, better can be removed
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119905806
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/MeasurePageStatistics.java ---
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.statistics;
+
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+public class MeasurePageStatistics {
--- End diff --
ok, change to `MeasurePageStatsVO`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119802447
--- Diff: core/src/main/java/org/apache/carbondata/core/util/CarbonMetadataUtil.java ---
@@ -549,24 +562,23 @@ private static ValueEncoderMeta deserializeValueEncoderMeta(ByteBuffer byteBuffe
}
- private static WriterCompressModel getValueCompressionModel(ValueEncoderMeta[] encoderMetas) {
- Object[] maxValue = new Object[encoderMetas.length];
- Object[] minValue = new Object[encoderMetas.length];
- int[] decimalLength = new int[encoderMetas.length];
- Object[] uniqueValue = new Object[encoderMetas.length];
- DataType[] aggType = new DataType[encoderMetas.length];
+ private static MeasurePageStatistics getMeasurePageStats(ValueEncoderMeta[] encoderMetas) {
+ Object[] max = new Object[encoderMetas.length];
+ Object[] min = new Object[encoderMetas.length];
+ int[] decimal = new int[encoderMetas.length];
+ Object[] nonExistValue = new Object[encoderMetas.length];
+ DataType[] types = new DataType[encoderMetas.length];
byte[] dataTypeSelected = new byte[encoderMetas.length];
for (int i = 0; i < encoderMetas.length; i++) {
- maxValue[i] = encoderMetas[i].getMaxValue();
- minValue[i] = encoderMetas[i].getMinValue();
- decimalLength[i] = encoderMetas[i].getDecimal();
- uniqueValue[i] = encoderMetas[i].getUniqueValue();
- aggType[i] = encoderMetas[i].getType();
+ max[i] = encoderMetas[i].getMaxValue();
+ min[i] = encoderMetas[i].getMinValue();
+ decimal[i] = encoderMetas[i].getDecimal();
+ nonExistValue[i] = encoderMetas[i].getUniqueValue();
+ types[i] = encoderMetas[i].getType();
dataTypeSelected[i] = encoderMetas[i].getDataTypeSelected();
}
- return ValueCompressionUtil
- .getWriterCompressModel(maxValue, minValue, decimalLength, uniqueValue, aggType,
- dataTypeSelected);
+
+ return MeasurePageStatistics.build(min, max, nonExistValue, decimal, types, dataTypeSelected);
--- End diff --
Better pass `ValueEncoderMeta` list to `build` method to avoid duplicate code in other places
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119803990
--- Diff: core/src/test/java/org/apache/carbondata/core/datastore/chunk/reader/measure/CompressedMeasureChunkFileBasedReaderTest.java ---
@@ -1,90 +0,0 @@
-/*
--- End diff --
Better remove the file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/971
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119781585
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ComplexColumnPage.java ---
@@ -74,4 +75,70 @@ public void putComplexData(int rowId, int depth, List<byte[]> value) {
public int getDepth() {
return depth;
}
+
+ @Override
+ public void putInt(int rowId, int value) {
--- End diff --
Create one abstract class to move this default implementations.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/94/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/52/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/154/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2280/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/971
@jackylk Can you rebase , I will merge it.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2279/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119887399
--- Diff: LICENSE ---
@@ -157,7 +157,7 @@
negligent acts) or agreed to in writing, shall any Contributor be
liable to You for damages, including any direct, indirect, special,
incidental, or consequential damages of any character arising as a
- result of this License or out of the use or inability to use the
+ encodedData of this License or out of the use or inability to use the
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/971
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2335/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #971: [CARBONDATA-1015] Extract interface in data lo...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/971#discussion_r119885006
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -65,20 +68,20 @@
this.pageSize = pageSize;
keyColumnPage = new KeyColumnPage(pageSize,
model.getSegmentProperties().getDimensionPartitions().length);
- noDictDimensionPage = new VarLengthColumnPage[model.getNoDictionaryCount()];
+ noDictDimensionPage = new PrimitiveColumnPage[model.getNoDictionaryCount()];
for (int i = 0; i < noDictDimensionPage.length; i++) {
- noDictDimensionPage[i] = new VarLengthColumnPage(pageSize);
+ noDictDimensionPage[i] = new PrimitiveColumnPage(DataType.STRING, pageSize);
--- End diff --
If create another class for string and decimal, then it will be more complex for measure handling. And actually in future I think all simple column handling should be similar, so better to keep them in one class. I think a better class name can solve your comment? How about SimpleColumnPage and ComplexColumnPage?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #971: [CARBONDATA-1015] Extract interface in data load writ...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/971
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/146/<h2>Failed Tests: <span class='status-failure'>13</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>13</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_hive/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from hive</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allq
ueries/InsertIntoCarbonTableTestCase/insert_from_carbon_select_columns/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from carbon-select columns</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_from_carbon_select___columns/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert from carbon-select * columns</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_functions/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with functions</strong></a></li><li><a href='https
://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_different_names_and_aliases/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with different names and aliases</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_existing_load_pass/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into existing load-pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_in
to_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union query</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_select_from_same_table/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert select from same table</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Less_Than_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Less Than Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/c
arbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Les_Than_equal_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Les Than equal Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Greater_Than_Filter_all_null/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.Greater Than Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/Greater_Than_equal_to_Filter_all_null/'><strong>org.apache.carbondata.spark.te
stsuite.filterexpr.GrtLtFilterProcessorTestCase.Greater Than equal to Filter all null</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/146/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.iud/HorizontalCompactionTestCase/test_IUD_Horizontal_Compaction_Check_Column_Cardinality/'><strong>org.apache.carbondata.spark.testsuite.iud.HorizontalCompactionTestCase.test IUD Horizontal Compaction Check Column Cardinality</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---