You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by jackylk <gi...@git.apache.org> on 2017/06/01 18:24:49 UTC
[GitHub] carbondata pull request #987: [WIP] Use ColumnPage for measure encoding and ...
GitHub user jackylk opened a pull request:
https://github.com/apache/carbondata/pull/987
[WIP] Use ColumnPage for measure encoding and compression
[WIP] This PR should be merged after #971
In this PR, following interfaces are added:
###ColumnPageEncoder
encoder for a ColumnPage
###EncoderStrategy, DefaultEncoderStrategy
EncoderStrategy defines the strategy to choose encoding for a ColumnPage.
DefaultEncoderStrategy is a default implementation which choose encoding based on data type and statistics
Other changes:
1. encoder for measure will use ColumnPage
2. compression for measure will use ColumnPage
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jackylk/incubator-carbondata encode
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/987.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #987
----
commit d8a4a2b74373f1f9b1c15780231958d176a6cf93
Author: jackylk <ja...@huawei.com>
Date: 2017-05-27T18:14:48Z
use ColumnPage in writer
commit 705469a4c5983d832b5b4e5504155dfb934425d7
Author: jackylk <ja...@huawei.com>
Date: 2017-05-27T18:22:31Z
remove WriterCompressModel
commit 418278d6e93c3d2bbfd4096893c2711e1e8ee9b8
Author: jackylk <ja...@huawei.com>
Date: 2017-05-28T14:55:29Z
add PrimitiveColumnPage
commit f2bb1267de866fb8fe72ea51aa8b117cdc2ceb4e
Author: jackylk <ja...@huawei.com>
Date: 2017-05-30T00:36:20Z
add TableSpec
commit 80bc3e4551a45d01386bbbd05d307354f5da5c25
Author: jackylk <ja...@huawei.com>
Date: 2017-05-30T10:11:21Z
add TablePageKey
commit 2ed4fbe38b98a6002cd4106d7274cdd88ef2b112
Author: jackylk <ja...@huawei.com>
Date: 2017-05-30T15:21:19Z
fix testcase
commit 0bc0bad2fe98f8719a2dfd311bf0bd379c3683de
Author: jackylk <ja...@huawei.com>
Date: 2017-05-30T17:37:38Z
fix style:
commit 224abbd9e2137ed1f16dad3c4ad737a0d3e80d37
Author: jackylk <ja...@huawei.com>
Date: 2017-05-31T08:58:16Z
make PrimitiveColumnPage generic
commit 545273ac7321f1e3c44e3c96b49083ede709a0aa
Author: jackylk <ja...@huawei.com>
Date: 2017-05-31T11:23:35Z
fix style
commit d6a60b4e500ad9a02463b7599d0caad8dff522cd
Author: jackylk <ja...@huawei.com>
Date: 2017-05-31T15:45:14Z
fix style
commit 210c9cba566b6208f65302b8d5c687ea279b7a2c
Author: jackylk <ja...@huawei.com>
Date: 2017-06-01T18:17:33Z
add ColumnPageEncoder
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Use ColumnPage for measure encoding and compres...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2143/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121616959
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -123,4 +121,29 @@ public boolean isMemoryAvailable() {
public long getUsableMemory() {
return totalMemory - minimumMemory;
}
+
+ /**
+ * It tries to allocate memory of `size` bytes, keep retry until it allocates successfully.
+ */
+ public static MemoryBlock allocateMemoryBlocking(long size) throws MemoryException {
--- End diff --
Actually it is not a typo, I want to express this function will block (retry) until it allocate memory successfully. Since this name is not good, I will change it to `allocateMemoryWithRetry`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/284/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/99/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/214/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121613640
--- Diff: core/src/main/java/org/apache/carbondata/core/memory/UnsafeMemoryManager.java ---
@@ -123,4 +121,29 @@ public boolean isMemoryAvailable() {
public long getUsableMemory() {
return totalMemory - minimumMemory;
}
+
+ /**
+ * It tries to allocate memory of `size` bytes, keep retry until it allocates successfully.
+ */
+ public static MemoryBlock allocateMemoryBlocking(long size) throws MemoryException {
--- End diff --
typo `allocateMemoryBlock`
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/329/<h2>Build result: FAILURE</span></h2>[...truncated 52.16 KB...][ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn <goals> -rf :carbondata-processing[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-
slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examples-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspa
ce/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /h
ome/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomchannel stoppedSetting status of b85a8bf927acf7716b8d454f0e6b1d187d3d3906 to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/329/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/316/<h2>Failed Tests: <span class='status-failure'>9</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>9</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-
test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_when_delimiter_is_____and_data_with_header/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadData
WithHiveSyntaxV2Format/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV2Format/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV2Format/test_data_loading_when_delimiter_is_____and_data_with_header/'><strong>org.apache.carb
ondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_no_dictionary_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single no dictionary column</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_complex_struct_type_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single complex struct type column</strong></a></li><li><a href='https://bui
lds.apache.org/job/carbondata-pr-spark-1.6/316/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_complex_array_type_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single complex array type column</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/318/<h2>Failed Tests: <span class='status-failure'>9</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>9</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-
test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_when_delimiter_is_____and_data_with_header/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadData
WithHiveSyntaxV2Format/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV2Format/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV2Format/test_data_loading_when_delimiter_is_____and_data_with_header/'><strong>org.apache.carb
ondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_no_dictionary_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single no dictionary column</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_complex_struct_type_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single complex struct type column</strong></a></li><li><a href='https://bui
lds.apache.org/job/carbondata-pr-spark-1.6/318/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_complex_array_type_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single complex array type column</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/200/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/21/<h2>Failed Tests: <span class='status-failure'>7</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>7</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinInteger/test_carbon_table_data_loading_when_the_int_column_contains_min_integer_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinInteger.test carbon table data loading when the int column contains min integer value</strong></a></li><li><a href='https://builds.apache.org
/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_offer_column_dictionary_file/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when offer column dictionary file</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite
.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_do_incremental_load/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when do incremental load</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.bigdecimal/TestBigInt/test_big_int_data_type_storage_for_boundary_values/'><strong>org.apache.carbondata.spark.testsuite.bigdecimal.TestBigInt.test big int data type storage for boundary values</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/When_the_values_of_Double_datatype_are_negative_values/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataType
TestCase.When the values of Double datatype are negative values</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/21/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/When_the_values_of_Double_datatype_have_both_postive_and_negative_values/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataTypeTestCase.When the values of Double datatype have both postive and negative values</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2298/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/184/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/95/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/14/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/327/<h2>Build result: FAILURE</span></h2>[...truncated 54.09 KB...][JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Processing #326 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Spark Common #326 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examp
les-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Common #326 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Parent #
326 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Core #326 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Hadoop #326 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom
.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Spark #326 to compare, so performing full copy of artifactschannel stoppedSetting status of 708f54cdca0ab573715c183aabf0da51286db830 to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/327/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2226/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2446/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2437/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/330/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/331/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/987
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/71/<h2>Failed Tests: <span class='status-failure'>17</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>17</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinInteger/test_carbon_table_data_loading_when_the_int_column_contains_min_integer_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinInteger.test carbon table data loading when the int column contains min integer value</strong></a></li><li><a href='https://builds.apache.o
rg/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_offer_column_dictionary_file/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when offer column dictionary file</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsui
te.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_do_incremental_load/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when do incremental load</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.primitiveTypes/FloatDataTypeTestCase/select_row_whose_rating_is_more_than_2_8_from_tfloat/'><strong>org.apache.carbondata.integration.spark.testsuite.primitiveTypes.FloatDataTypeTestCase.select row whose rating is more than 2.8 from tfloat</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.primitiveTypes/FloatDataTypeTestCase/select_row_whose_rating_is_3_5_from_tfloat/'><strong>org.apache.carbondata.
integration.spark.testsuite.primitiveTypes.FloatDataTypeTestCase.select row whose rating is 3.5 from tfloat</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.primitiveTypes/FloatDataTypeTestCase/select_sum_of_rating_column_from_tfloat/'><strong>org.apache.carbondata.integration.spark.testsuite.primitiveTypes.FloatDataTypeTestCase.select sum of rating column from tfloat</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert__insert_with_functions/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert->insert with functions</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.c
arbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntax/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntax.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntax/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntax.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntax/test_data_loading_w
hen_delimiter_is_____and_data_with_header/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntax.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/ActualDataType_double_ChangedDatatype_Short_CompressionType_NonDecimalMaxMin/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataTypeTestCase.ActualDataType:double,ChangedDatatype:Short,CompressionType:NonDecimalMaxMin</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/ActualDataType_double_ChangedDatatype_byte_CompressionType_NonDecimalMaxMin/'><strong>org.apache.ca
rbondata.spark.testsuite.detailquery.ValueCompressionDataTypeTestCase.ActualDataType:double,ChangedDatatype:byte,CompressionType:NonDecimalMaxMin</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.iud/HorizontalCompactionTestCase/test_IUD_Horizontal_Compaction_Check_Column_Cardinality/'><strong>org.apache.carbondata.spark.testsuite.iud.HorizontalCompactionTestCase.test IUD Horizontal Compaction Check Column Cardinality</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.windowsexpr/WindowsExprTestCase/SELECT_country_name_salary_FROM__SELECT_country_name_salary_dense_rank___OVER__PARTITION_BY_country_ORDER_BY_salary_DESC__as_rank_FROM_windowstable__tmp_WHERE_rank____2_order_by_country/'><strong>org.apache.carbondata.s
park.testsuite.windowsexpr.WindowsExprTestCase.SELECT country,name,salary FROM (SELECT country,name,salary,dense_rank() OVER (PARTITION BY country ORDER BY salary DESC) as rank FROM windowstable) tmp WHERE rank <= 2 order by country</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.windowsexpr/WindowsExprTestCase/SELECT_ID__country__SUM_salary__OVER__PARTITION_BY_country___AS_TopBorcT_FROM_windowstable/'><strong>org.apache.carbondata.spark.testsuite.windowsexpr.WindowsExprTestCase.SELECT ID, country, SUM(salary) OVER (PARTITION BY country ) AS TopBorcT FROM windowstable</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/71/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.windowsexpr/WindowsExprTestCase/SELECT_country_name_salary_ROW_NUMBER___OVER__PARTITION_BY_count
ry_ORDER_BY_salary_DESC__as_rownum_FROM_windowstable/'><strong>org.apache.carbondata.spark.testsuite.windowsexpr.WindowsExprTestCase.SELECT country,name,salary,ROW_NUMBER() OVER (PARTITION BY country ORDER BY salary DESC) as rownum FROM windowstable</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2149/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/24/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121619385
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/UpscaleFloatingCodec.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for floating point (float, double) data type page.
+ * This codec will upscale the diff from page max value to integer value,
+ * and do type casting to make storage minimum.
+ */
+public class UpscaleFloatingCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+ private double factor;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new UpscaleFloatingCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private UpscaleFloatingCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ this.factor = Math.pow(10, stats.getDecimal());
+ }
+
+ @Override public String getName() {
+ return "UpscaleFloatingCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
--- End diff --
I thought about it earlier, but in that case the log message will not be as good as now, like "internal error: " + debugInfo().
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121577179
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -22,22 +22,38 @@
import java.util.BitSet;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatistics;
+import org.apache.carbondata.core.constants.CarbonV3DataFormatConstants;
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.util.ByteUtil;
+import org.apache.carbondata.core.util.CarbonProperties;
import org.apache.carbondata.core.util.DataTypeUtil;
-// Represent a columnar data in one page for one column.
+import static org.apache.carbondata.core.metadata.datatype.DataType.BYTE;
+import static org.apache.carbondata.core.metadata.datatype.DataType.DECIMAL;
+import static org.apache.carbondata.core.metadata.datatype.DataType.DOUBLE;
+import static org.apache.carbondata.core.metadata.datatype.DataType.FLOAT;
+import static org.apache.carbondata.core.metadata.datatype.DataType.INT;
+import static org.apache.carbondata.core.metadata.datatype.DataType.LONG;
+import static org.apache.carbondata.core.metadata.datatype.DataType.SHORT;
+import static org.apache.carbondata.core.metadata.datatype.DataType.STRING;
+
+/**
+ * Represent a columnar data in one page for one column.
+ */
public class ColumnPage {
- protected final DataType dataType;
- protected final int pageSize;
- protected ColumnPageStatistics stats;
+ private final int pageSize;
+ private DataType dataType;
+ private ColumnPageStatsVO stats;
--- End diff --
I think ColumnPageStats supposed to be interface so that user can implement to have his own stats
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2436/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/182/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/179/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2449/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/987
@jackylk can you the failed tests
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2405/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2323/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121608358
--- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/DoubleArrayTrieDictionary.java ---
@@ -43,7 +43,7 @@
private static final int UUSD_ROOM_VALUE = -2; // unused position, only for zero
private static final int EPTY_BACK_VALUE = 0; // value of empty position
- private static final int ENCODE_BASE_VALUE = 10; // encode start number
+ private static final int ENCODE_BASE_VALUE = 10; // apply start number
--- End diff --
ok, it is modified by mistake
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2306/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2156/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2450/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/167/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121509773
--- Diff: core/src/main/java/org/apache/carbondata/core/cache/dictionary/DoubleArrayTrieDictionary.java ---
@@ -43,7 +43,7 @@
private static final int UUSD_ROOM_VALUE = -2; // unused position, only for zero
private static final int EPTY_BACK_VALUE = 0; // value of empty position
- private static final int ENCODE_BASE_VALUE = 10; // encode start number
+ private static final int ENCODE_BASE_VALUE = 10; // apply start number
--- End diff --
I think no need to change this file
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/43/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2285/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/50/<h2>Failed Tests: <span class='status-failure'>7</span></h2><h3><a name='carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-2.1/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>7</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinInteger/test_carbon_table_data_loading_when_the_int_column_contains_min_integer_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinInteger.test carbon table data loading when the int column contains min integer value</strong></a></li><li><a href='https://builds.apache.org
/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_offer_column_dictionary_file/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when offer column dictionary file</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite
.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_do_incremental_load/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when do incremental load</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.bigdecimal/TestBigInt/test_big_int_data_type_storage_for_boundary_values/'><strong>org.apache.carbondata.spark.testsuite.bigdecimal.TestBigInt.test big int data type storage for boundary values</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/When_the_values_of_Double_datatype_are_negative_values/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataType
TestCase.When the values of Double datatype are negative values</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-2.1/50/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/When_the_values_of_Double_datatype_have_both_postive_and_negative_values/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataTypeTestCase.When the values of Double datatype have both postive and negative values</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/987
apache:retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncoderStrategy and use ColumnPage for meas...
Posted by chenliang613 <gi...@git.apache.org>.
Github user chenliang613 commented on the issue:
https://github.com/apache/carbondata/pull/987
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/987
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2310/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121608648
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/AdaptiveIntegerCodec.java ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for integer (byte, short, int, long) data type page.
+ * This codec will do type casting on page data to make storage minimum.
+ */
+class AdaptiveIntegerCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new AdaptiveIntegerCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private AdaptiveIntegerCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ }
+
+ @Override
+ public String getName() {
+ return "AdaptiveIntegerCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ if (srcDataType.equals(targetDataType)) {
+ return input.compress(compressor);
+ } else {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+ }
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ if (srcDataType.equals(targetDataType)) {
+ return ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ } else {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
+ @Override
+ public void encode(int rowId, byte value) {
+ switch (targetDataType) {
+ default:
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+ }
+
+ @Override
+ public void encode(int rowId, short value) {
+ switch (targetDataType) {
+ case BYTE:
+ encodedPage.putByte(rowId, (byte) value);
+ break;
+ default:
+ assert (false);
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121579950
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -46,51 +62,157 @@
// The index of the rowId whose value is null, will be set to 1
private BitSet nullBitSet;
- public ColumnPage(DataType dataType, int pageSize) {
- this.dataType = dataType;
+ protected ColumnPage(DataType dataType, int pageSize) {
this.pageSize = pageSize;
- this.stats = new ColumnPageStatistics(dataType);
- this.nullBitSet = new BitSet(pageSize);
+ this.dataType = dataType;
+ }
+
+ // create a new page
+ public static ColumnPage newPage(DataType dataType, int pageSize) {
+ ColumnPage instance;
switch (dataType) {
+ case BYTE:
+ instance = newBytePage(new byte[pageSize]);
+ break;
case SHORT:
+ instance = newShortPage(new short[pageSize]);
+ break;
case INT:
+ instance = newIntPage(new int[pageSize]);
+ break;
case LONG:
- longData = new long[pageSize];
+ instance = newLongPage(new long[pageSize]);
+ break;
+ case FLOAT:
+ instance = newFloatPage(new float[pageSize]);
break;
case DOUBLE:
- doubleData = new double[pageSize];
+ instance = newDoublePage(new double[pageSize]);
break;
case DECIMAL:
- byteArrayData = new byte[pageSize][];
+ instance = newDecimalPage(new byte[pageSize][]);
break;
case STRING:
- byteArrayData = new byte[pageSize][];
+ instance = newStringPage(new byte[pageSize][]);
break;
default:
throw new RuntimeException("Unsupported data dataType: " + dataType);
}
+ instance.stats = new ColumnPageStatsVO(dataType);
+ instance.nullBitSet = new BitSet(pageSize);
+ return instance;
+ }
+
+ // create a new page and set data with input `pageData`
+ public static ColumnPage newPage(DataType dataType, Object pageData) {
+ // This is used in read path, since statistics and nullBitSet is not required, not creating
+ // them to make object minimum
+ switch (dataType) {
+ case BYTE:
+ byte[] byteData = (byte[]) pageData;
+ return newBytePage(byteData);
+ case SHORT:
+ short[] shortData = (short[]) pageData;
+ return newShortPage(shortData);
+ case INT:
+ int[] intData = (int[]) pageData;
+ return newIntPage(intData);
+ case LONG:
+ long[] longData = (long[]) pageData;
+ return newLongPage(longData);
+ case FLOAT:
+ float[] floatData = (float[]) pageData;
+ return newFloatPage(floatData);
+ case DOUBLE:
+ double[] doubleData = (double[]) pageData;
+ return newDoublePage(doubleData);
+ case DECIMAL:
+ byte[][] decimalData = (byte[][]) pageData;
+ return newDecimalPage(decimalData);
+ case STRING:
--- End diff --
Better change this to BYTE_ARRAY and bring KeyColumnPage also here
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/987
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2223/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2309/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121625468
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -22,22 +22,38 @@
import java.util.BitSet;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatistics;
+import org.apache.carbondata.core.constants.CarbonV3DataFormatConstants;
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.util.ByteUtil;
+import org.apache.carbondata.core.util.CarbonProperties;
import org.apache.carbondata.core.util.DataTypeUtil;
-// Represent a columnar data in one page for one column.
+import static org.apache.carbondata.core.metadata.datatype.DataType.BYTE;
+import static org.apache.carbondata.core.metadata.datatype.DataType.DECIMAL;
+import static org.apache.carbondata.core.metadata.datatype.DataType.DOUBLE;
+import static org.apache.carbondata.core.metadata.datatype.DataType.FLOAT;
+import static org.apache.carbondata.core.metadata.datatype.DataType.INT;
+import static org.apache.carbondata.core.metadata.datatype.DataType.LONG;
+import static org.apache.carbondata.core.metadata.datatype.DataType.SHORT;
+import static org.apache.carbondata.core.metadata.datatype.DataType.STRING;
+
+/**
+ * Represent a columnar data in one page for one column.
+ */
public class ColumnPage {
- protected final DataType dataType;
- protected final int pageSize;
- protected ColumnPageStatistics stats;
+ private final int pageSize;
+ private DataType dataType;
+ private ColumnPageStatsVO stats;
--- End diff --
We better design this feature before changing it to interface. I think both write and read need to change.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2337/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121580390
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/PrimitiveCodec.java ---
@@ -0,0 +1,50 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page;
+
+// no operation, new value = page value
+// NO_OP,
+//
+// // new value = (max value of page) - (page value)
+// MAX_DELTA,
+//
+// // new value = (10 power of decimal) * (page value)
+// UPSCALE,
+//
+// // new value = (10 power of decimal) * ((max value of page) - (page value))
+// UPSCALE_MAX_DELTA
+
+// Transformation type that can be applied to ColumnPage
+public interface PrimitiveCodec {
--- End diff --
We better bring RLE of mdk also to similar interfaces, may we can work on future PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2302/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/210/<h2>Failed Tests: <span class='status-failure'>3</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/210/org.apache.carbondata$carbondata-spark/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark</a>: <span class='status-failure'>2</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/210/org.apache.carbondata$carbondata-spark/testReport/org.apache.carbondata.spark.util/AutoHighCardinalityIdentifyTestCase/auto_identify_high_cardinality_column_in_first_load__396/'><strong>org.apache.carbondata.spark.util.AutoHighCardinalityIdentifyTestCase.auto identify high cardinality column in first load #396</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/210/org.apache.carbondata$carbondata-spark/testReport/org.apache.carbondata.spark.util/AutoHighCar
dinalityIdentifyTestCase/skip_auto_identify_high_cardinality_column_for_column_group/'><strong>org.apache.carbondata.spark.util.AutoHighCardinalityIdentifyTestCase.skip auto identify high cardinality column for column group</strong></a></li></ul><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/210/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>1</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/210/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.allqueries/InsertIntoCarbonTableTestCase/insert_into_carbon_table_from_carbon_table_union_query/'><strong>org.apache.carbondata.spark.testsuite.allqueries.InsertIntoCarbonTableTestCase.insert into carbon table from carbon table union
query</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2303/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121608594
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -22,22 +22,38 @@
import java.util.BitSet;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatistics;
--- End diff --
Yes, you are right. To make this PR smaller, I suggest to do it in another PR (EncodingStrategy for dimension)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/987
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121612421
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/UpscaleDeltaFloatingCodec.java ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.math.BigDecimal;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for floating point (float, double) data type page.
+ * This codec will upscale (multiple page value by decimal) to integer value,
+ * and do type casting to make storage minimum.
+ */
+public class UpscaleDeltaFloatingCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+
+ private BigDecimal max;
+ private double factor;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new UpscaleDeltaFloatingCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private UpscaleDeltaFloatingCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ this.max = BigDecimal.valueOf((double) stats.getMax());
+ this.factor = Math.pow(10, stats.getDecimal());
+ }
+
+ @Override
+ public String getName() {
+ return "UpscaleDeltaFloatingCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
+ @Override
+ public void encode(int rowId, byte value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, short value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, int value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, long value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, float value) {
+ double diff = max.subtract(BigDecimal.valueOf(value)).doubleValue();
+ switch (targetDataType) {
+ case BYTE:
+ encodedPage.putByte(rowId, (byte)(Math.round(factor * diff)));
+ break;
+ case SHORT:
+ encodedPage.putShort(rowId, (short)(Math.round(factor * diff)));
+ break;
+ case INT:
+ encodedPage.putInt(rowId, (int)(Math.round(factor * diff)));
+ break;
+ case LONG:
+ encodedPage.putLong(rowId, (long)(Math.round(factor * diff)));
+ break;
+ default:
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+ }
+
+ @Override
+ public void encode(int rowId, double value) {
+ double diff = max.subtract(BigDecimal.valueOf(value)).doubleValue();
+ switch (targetDataType) {
+ case BYTE:
+ encodedPage.putByte(rowId, (byte)(Math.round(factor * diff)));
+ break;
+ case SHORT:
+ encodedPage.putShort(rowId, (short)(Math.round(factor * diff)));
+ break;
+ case INT:
+ encodedPage.putInt(rowId, (int)(Math.round(factor * diff)));
+ break;
+ case LONG:
+ encodedPage.putLong(rowId, (long)(Math.round(factor * diff)));
+ break;
+ default:
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+ }
+
+ @Override
+ public long decodeLong(byte value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public long decodeLong(short value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public long decodeLong(int value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public double decodeDouble(byte value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(short value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(int value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(long value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(float value) {
+ assert (false);
--- End diff --
I think assert is not required as exception is throwing
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-2.1/53/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2448/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2197/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/175/<h2>Build result: FAILURE</span></h2>[...truncated 53.79 KB...][JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Processing #173 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Spark Common #173 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examp
les-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Common #173 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Parent #
173 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Core #173 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Hadoop #173 to compare, so performing full copy of artifacts[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom
.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomNo artifacts from carbondata-pr-spark-1.6 » Apache CarbonData :: Spark #173 to compare, so performing full copy of artifactschannel stoppedSetting status of 43b4ef0c88bd97eddafe253d9efb890d8b6dcb76 to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/175/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2333/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2204/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/987
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121580802
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/ColumnPageCodec.java ---
@@ -21,15 +21,28 @@
/**
* Codec for a column page data, implementation should not keep state across pages,
- * caller will use the same object to encode multiple pages.
+ * caller may use the same object to apply multiple pages.
*/
-public interface ColumnCodec {
+public interface ColumnPageCodec {
--- End diff --
We better bring RLE of mdk also to similar interfaces, may we can work on future PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/64/<h2>Failed Tests: <span class='status-failure'>5</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/64/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>5</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/64/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinInteger/test_carbon_table_data_loading_when_the_int_column_contains_min_integer_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinInteger.test carbon table data loading when the int column contains min integer value</strong></a></li><li><a href='https://builds.apache.org
/job/carbondata-pr-spark-1.6/64/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/64/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_offer_column_dictionary_file/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when offer column dictionary file</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/64/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite
.dataload/TestLoadDataWithSinglePass/test_data_loading_use_one_pass_when_do_incremental_load/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithSinglePass.test data loading use one pass when do incremental load</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/64/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.filterexpr/GrtLtFilterProcessorTestCase/0_0_and__0_0_equality_check_for_double_data_type_applying_log_function/'><strong>org.apache.carbondata.spark.testsuite.filterexpr.GrtLtFilterProcessorTestCase.0.0 and -0.0 equality check for double data type applying log function</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/183/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121615305
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/UpscaleDeltaFloatingCodec.java ---
@@ -0,0 +1,197 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import java.math.BigDecimal;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for floating point (float, double) data type page.
+ * This codec will upscale (multiple page value by decimal) to integer value,
+ * and do type casting to make storage minimum.
+ */
+public class UpscaleDeltaFloatingCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+
+ private BigDecimal max;
+ private double factor;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new UpscaleDeltaFloatingCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private UpscaleDeltaFloatingCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ this.max = BigDecimal.valueOf((double) stats.getMax());
+ this.factor = Math.pow(10, stats.getDecimal());
+ }
+
+ @Override
+ public String getName() {
+ return "UpscaleDeltaFloatingCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
+ @Override
+ public void encode(int rowId, byte value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, short value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, int value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, long value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public void encode(int rowId, float value) {
+ double diff = max.subtract(BigDecimal.valueOf(value)).doubleValue();
+ switch (targetDataType) {
+ case BYTE:
+ encodedPage.putByte(rowId, (byte)(Math.round(factor * diff)));
+ break;
+ case SHORT:
+ encodedPage.putShort(rowId, (short)(Math.round(factor * diff)));
+ break;
+ case INT:
+ encodedPage.putInt(rowId, (int)(Math.round(factor * diff)));
+ break;
+ case LONG:
+ encodedPage.putLong(rowId, (long)(Math.round(factor * diff)));
+ break;
+ default:
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+ }
+
+ @Override
+ public void encode(int rowId, double value) {
+ double diff = max.subtract(BigDecimal.valueOf(value)).doubleValue();
+ switch (targetDataType) {
+ case BYTE:
+ encodedPage.putByte(rowId, (byte)(Math.round(factor * diff)));
+ break;
+ case SHORT:
+ encodedPage.putShort(rowId, (short)(Math.round(factor * diff)));
+ break;
+ case INT:
+ encodedPage.putInt(rowId, (int)(Math.round(factor * diff)));
+ break;
+ case LONG:
+ encodedPage.putLong(rowId, (long)(Math.round(factor * diff)));
+ break;
+ default:
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+ }
+
+ @Override
+ public long decodeLong(byte value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public long decodeLong(short value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public long decodeLong(int value) {
+ // this codec is for floating point type only
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+
+ @Override
+ public double decodeDouble(byte value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(short value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(int value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(long value) {
+ return max.subtract(BigDecimal.valueOf(value / factor)).doubleValue();
+ }
+
+ @Override
+ public double decodeDouble(float value) {
+ assert (false);
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/317/<h2>Failed Tests: <span class='status-failure'>9</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>9</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-
test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV1Format/test_data_loading_when_delimiter_is_____and_data_with_header/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV1Format.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadData
WithHiveSyntaxV2Format/test_data_loading_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV2Format/test_data_loading_with_different_case_file_header_and_validate_query_output/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading with different case file header and validate query output</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithHiveSyntaxV2Format/test_data_loading_when_delimiter_is_____and_data_with_header/'><strong>org.apache.carb
ondata.spark.testsuite.dataload.TestLoadDataWithHiveSyntaxV2Format.test data loading when delimiter is '|' and data with header</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_no_dictionary_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single no dictionary column</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_complex_struct_type_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single complex struct type column</strong></a></li><li><a href='https://bui
lds.apache.org/job/carbondata-pr-spark-1.6/317/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestLoadDataWithNoMeasure/test_data_loading_with_single_complex_array_type_column/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestLoadDataWithNoMeasure.test data loading with single complex array type column</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121608684
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/ColumnPageCodec.java ---
@@ -21,15 +21,28 @@
/**
* Codec for a column page data, implementation should not keep state across pages,
- * caller will use the same object to encode multiple pages.
+ * caller may use the same object to apply multiple pages.
*/
-public interface ColumnCodec {
+public interface ColumnPageCodec {
--- End diff --
Yes, you are right. To make this PR smaller, I suggest to do it in another PR (EncodingStrategy for dimension)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/158/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/176/<h2>Failed Tests: <span class='status-failure'>2</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/176/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>2</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/176/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/ActualDataType_double_ChangedDatatype_Short_CompressionType_NonDecimalMaxMin/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataTypeTestCase.ActualDataType:double,ChangedDatatype:Short,CompressionType:NonDecimalMaxMin</strong></a></li><li><a href='https://builds.apache.org/job/carbon
data-pr-spark-1.6/176/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.detailquery/ValueCompressionDataTypeTestCase/ActualDataType_double_ChangedDatatype_byte_CompressionType_NonDecimalMaxMin/'><strong>org.apache.carbondata.spark.testsuite.detailquery.ValueCompressionDataTypeTestCase.ActualDataType:double,ChangedDatatype:byte,CompressionType:NonDecimalMaxMin</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121618486
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/UpscaleFloatingCodec.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for floating point (float, double) data type page.
+ * This codec will upscale the diff from page max value to integer value,
+ * and do type casting to make storage minimum.
+ */
+public class UpscaleFloatingCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+ private double factor;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new UpscaleFloatingCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private UpscaleFloatingCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ this.factor = Math.pow(10, stats.getDecimal());
+ }
+
+ @Override public String getName() {
+ return "UpscaleFloatingCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/97/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/987
LGTM
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/315/<h2>Build result: FAILURE</span></h2>[...truncated 51.50 KB...][ERROR] Re-run Maven using the -X switch to enable full debug logging.[ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles:[ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoExecutionException[ERROR] [ERROR] After correcting the problems, you can resume the build with the command[ERROR] mvn <goals> -rf :carbondata-processing[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/processing/pom.xml to org.apache.carbondata/carbondata-processing/1.2.0-SNAPSHOT/carbondata-processing-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common/pom.xml to org.apache.carbondata/carbondata-spark-common/1.2.0-SNAPSHOT/carbondata-spark-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-
slave/workspace/carbondata-pr-spark-1.6/examples/spark/pom.xml to org.apache.carbondata/carbondata-examples-spark/1.2.0-SNAPSHOT/carbondata-examples-spark-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/hive/pom.xml to org.apache.carbondata/carbondata-hive/1.2.0-SNAPSHOT/carbondata-hive-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/common/pom.xml to org.apache.carbondata/carbondata-common/1.2.0-SNAPSHOT/carbondata-common-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/assembly/pom.xml to org.apache.carbondata/carbondata-assembly/1.2.0-SNAPSHOT/carbondata-assembly-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/pom.xml to org.apache.carbondata/carbondata-parent/1.2.0-SNAPSHOT/carbondata-parent-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspa
ce/carbondata-pr-spark-1.6/examples/flink/pom.xml to org.apache.carbondata/carbondata-examples-flink/1.2.0-SNAPSHOT/carbondata-examples-flink-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/core/pom.xml to org.apache.carbondata/carbondata-core/1.2.0-SNAPSHOT/carbondata-core-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/hadoop/pom.xml to org.apache.carbondata/carbondata-hadoop/1.2.0-SNAPSHOT/carbondata-hadoop-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/presto/pom.xml to org.apache.carbondata/carbondata-presto/1.2.0-SNAPSHOT/carbondata-presto-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /home/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark-common-test/pom.xml to org.apache.carbondata/carbondata-spark-common-test/1.2.0-SNAPSHOT/carbondata-spark-common-test-1.2.0-SNAPSHOT.pom[JENKINS] Archiving /h
ome/jenkins/jenkins-slave/workspace/carbondata-pr-spark-1.6/integration/spark/pom.xml to org.apache.carbondata/carbondata-spark/1.2.0-SNAPSHOT/carbondata-spark-1.2.0-SNAPSHOT.pomchannel stoppedSetting status of 56d21b1db66e70758b44e386659e3c670531c57e to FAILURE with url https://builds.apache.org/job/carbondata-pr-spark-1.6/315/ and message: 'Tests Failed for Spark1.6 'Using context: Jenkins(Spark1.6): mvn clean test -Pspark-1.6
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2294/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2236/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121628329
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -46,51 +62,157 @@
// The index of the rowId whose value is null, will be set to 1
private BitSet nullBitSet;
- public ColumnPage(DataType dataType, int pageSize) {
- this.dataType = dataType;
+ protected ColumnPage(DataType dataType, int pageSize) {
this.pageSize = pageSize;
- this.stats = new ColumnPageStatistics(dataType);
- this.nullBitSet = new BitSet(pageSize);
+ this.dataType = dataType;
+ }
+
+ // create a new page
+ public static ColumnPage newPage(DataType dataType, int pageSize) {
+ ColumnPage instance;
switch (dataType) {
+ case BYTE:
+ instance = newBytePage(new byte[pageSize]);
+ break;
case SHORT:
+ instance = newShortPage(new short[pageSize]);
+ break;
case INT:
+ instance = newIntPage(new int[pageSize]);
+ break;
case LONG:
- longData = new long[pageSize];
+ instance = newLongPage(new long[pageSize]);
+ break;
+ case FLOAT:
+ instance = newFloatPage(new float[pageSize]);
break;
case DOUBLE:
- doubleData = new double[pageSize];
+ instance = newDoublePage(new double[pageSize]);
break;
case DECIMAL:
- byteArrayData = new byte[pageSize][];
+ instance = newDecimalPage(new byte[pageSize][]);
break;
case STRING:
- byteArrayData = new byte[pageSize][];
+ instance = newStringPage(new byte[pageSize][]);
break;
default:
throw new RuntimeException("Unsupported data dataType: " + dataType);
}
+ instance.stats = new ColumnPageStatsVO(dataType);
+ instance.nullBitSet = new BitSet(pageSize);
+ return instance;
+ }
+
+ // create a new page and set data with input `pageData`
+ public static ColumnPage newPage(DataType dataType, Object pageData) {
+ // This is used in read path, since statistics and nullBitSet is not required, not creating
+ // them to make object minimum
+ switch (dataType) {
+ case BYTE:
+ byte[] byteData = (byte[]) pageData;
+ return newBytePage(byteData);
+ case SHORT:
+ short[] shortData = (short[]) pageData;
+ return newShortPage(shortData);
+ case INT:
+ int[] intData = (int[]) pageData;
+ return newIntPage(intData);
+ case LONG:
+ long[] longData = (long[]) pageData;
+ return newLongPage(longData);
+ case FLOAT:
+ float[] floatData = (float[]) pageData;
+ return newFloatPage(floatData);
+ case DOUBLE:
+ double[] doubleData = (double[]) pageData;
+ return newDoublePage(doubleData);
+ case DECIMAL:
+ byte[][] decimalData = (byte[][]) pageData;
+ return newDecimalPage(decimalData);
+ case STRING:
--- End diff --
ok
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121613071
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/UpscaleFloatingCodec.java ---
@@ -0,0 +1,190 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for floating point (float, double) data type page.
+ * This codec will upscale the diff from page max value to integer value,
+ * and do type casting to make storage minimum.
+ */
+public class UpscaleFloatingCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+ private double factor;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new UpscaleFloatingCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private UpscaleFloatingCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ this.factor = Math.pow(10, stats.getDecimal());
+ }
+
+ @Override public String getName() {
+ return "UpscaleFloatingCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
--- End diff --
Better use some abstract class and implement all methods with default implementation, and extended classes can just use required methods
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/171/<h2>Failed Tests: <span class='status-failure'>7</span></h2><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark</a>: <span class='status-failure'>2</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark/testReport/org.apache.carbondata.spark.util/AutoHighCardinalityIdentifyTestCase/auto_identify_high_cardinality_column_in_first_load__396/'><strong>org.apache.carbondata.spark.util.AutoHighCardinalityIdentifyTestCase.auto identify high cardinality column in first load #396</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark/testReport/org.apache.carbondata.spark.util/AutoHighCar
dinalityIdentifyTestCase/skip_auto_identify_high_cardinality_column_for_column_group/'><strong>org.apache.carbondata.spark.util.AutoHighCardinalityIdentifyTestCase.skip auto identify high cardinality column for column group</strong></a></li></ul><h3><a name='carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test' /><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark-common-test/testReport'>carbondata-pr-spark-1.6/org.apache.carbondata:carbondata-spark-common-test</a>: <span class='status-failure'>5</span></h3><ul><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinBigInt/test_carbon_table_data_loading_when_the_bigint_column_contains_min_bigint_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinBigInt.test
carbon table data loading when the bigint column contains min bigint value</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinBigInt/test_carbon_table_data_loading_when_the_bigint_column_contains_max_bigint_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinBigInt.test carbon table data loading when the bigint column contains max bigint value</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.integration.spark.testsuite.dataload/TestLoadDataWithMaxMinBigInt/test_carbon_table_data_loading_when_the_bigint_column_contains_min_and_max_bigint_value/'><strong>org.apache.carbondata.integration.spark.testsuite.dataload.TestLoadDataWithMaxMinBigInt.te
st carbon table data loading when the bigint column contains min and max bigint value</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.blockprune/CarbonCustomBlockDistributionTest/test_block_prune_query/'><strong>org.apache.carbondata.spark.testsuite.blockprune.CarbonCustomBlockDistributionTest.test block prune query</strong></a></li><li><a href='https://builds.apache.org/job/carbondata-pr-spark-1.6/171/org.apache.carbondata$carbondata-spark-common-test/testReport/org.apache.carbondata.spark.testsuite.dataload/TestDataLoadWithColumnsMoreThanSchema/test_for_duplicate_column_name_in_the_Fileheader_options_in_load_command/'><strong>org.apache.carbondata.spark.testsuite.dataload.TestDataLoadWithColumnsMoreThanSchema.test for duplicate column name in the Fileheader options in load command</strong></a></li></ul>
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2435/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2434/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121580534
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/AdaptiveIntegerCodec.java ---
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding;
+
+import org.apache.carbondata.core.datastore.compression.Compressor;
+import org.apache.carbondata.core.datastore.page.ColumnPage;
+import org.apache.carbondata.core.datastore.page.LazyColumnPage;
+import org.apache.carbondata.core.datastore.page.PrimitiveCodec;
+import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatsVO;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+
+/**
+ * Codec for integer (byte, short, int, long) data type page.
+ * This codec will do type casting on page data to make storage minimum.
+ */
+class AdaptiveIntegerCodec extends AdaptiveCompressionCodec {
+
+ private ColumnPage encodedPage;
+
+ public static ColumnPageCodec newInstance(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ return new AdaptiveIntegerCodec(srcDataType, targetDataType, stats, compressor);
+ }
+
+ private AdaptiveIntegerCodec(DataType srcDataType, DataType targetDataType,
+ ColumnPageStatsVO stats, Compressor compressor) {
+ super(srcDataType, targetDataType, stats, compressor);
+ }
+
+ @Override
+ public String getName() {
+ return "AdaptiveIntegerCodec";
+ }
+
+ @Override
+ public byte[] encode(ColumnPage input) {
+ if (srcDataType.equals(targetDataType)) {
+ return input.compress(compressor);
+ } else {
+ encodedPage = ColumnPage.newPage(targetDataType, input.getPageSize());
+ input.encode(codec);
+ return encodedPage.compress(compressor);
+ }
+ }
+
+ @Override
+ public ColumnPage decode(byte[] input, int offset, int length) {
+ if (srcDataType.equals(targetDataType)) {
+ return ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ } else {
+ ColumnPage page = ColumnPage.decompress(compressor, targetDataType, input, offset, length);
+ return LazyColumnPage.newPage(page, codec);
+ }
+ }
+
+ private PrimitiveCodec codec = new PrimitiveCodec() {
+ @Override
+ public void encode(int rowId, byte value) {
+ switch (targetDataType) {
+ default:
+ assert (false);
+ throw new RuntimeException("internal error: " + debugInfo());
+ }
+ }
+
+ @Override
+ public void encode(int rowId, short value) {
+ switch (targetDataType) {
+ case BYTE:
+ encodedPage.putByte(rowId, (byte) value);
+ break;
+ default:
+ assert (false);
--- End diff --
This assertion is not required as we are already throwing exception
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata pull request #987: [CARBONDATA-1017] Add EncodingStrategy and use...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/987#discussion_r121579752
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -22,22 +22,38 @@
import java.util.BitSet;
import org.apache.carbondata.core.constants.CarbonCommonConstants;
-import org.apache.carbondata.core.datastore.page.statistics.ColumnPageStatistics;
--- End diff --
I think we should unify KeyColumnPage and ColumnPage or atleast one is derived from another.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit commented on the issue:
https://github.com/apache/carbondata/pull/987
Refer to this link for build results (access rights to CI server needed):
https://builds.apache.org/job/carbondata-pr-spark-1.6/109/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [WIP] Add EncodingStrategy and use ColumnPage for mea...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2159/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] carbondata issue #987: [CARBONDATA-1017] Add EncodingStrategy and use Column...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/987
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/2224/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---