You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by mayunSaicmotor <gi...@git.apache.org> on 2017/04/21 06:20:22 UTC
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
GitHub user mayunSaicmotor opened a pull request:
https://github.com/apache/incubator-carbondata/pull/833
[CARBONDATA-951] create table ddl can specify a bitmap option and add a new Encoding: BITMAP
create table ddl can specify a bitmap option
add a new Encoding: BITMAP\u2028\u2028
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/mayunSaicmotor/incubator-carbondata 12-dev
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-carbondata/pull/833.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #833
----
commit ec1b7b25c8b5f1a09603d42673ada04efcdc5b06
Author: mayun <si...@163.com>
Date: 2017-04-21T06:16:57Z
1. create table ddl can specify a bitmap option
2. add a new Encoding: BITMAP\u2028\u2028
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1848/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by mayunSaicmotor <gi...@git.apache.org>.
Github user mayunSaicmotor commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r114095485
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -130,6 +130,7 @@ struct DataChunk2{
9: optional list<binary> encoder_meta; // Extra information required by encoders
10: optional BlockletMinMaxIndex min_max;
11: optional i32 numberOfRowsInpage;
+ 12: optional list<i32> bitmap_encoded_dictionaries;
--- End diff --
store each chunk' dictionaries, for example if in 2 chunks one, have dictionary 2 and 3, and the other have 3 and 4. There are different between the 2 chunks. So bitmap_encoded_dictionaries to store the dictionaries for each chunk.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1731/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by mayunSaicmotor <gi...@git.apache.org>.
Github user mayunSaicmotor commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
fixed for the below items
1.the column should use DICTIONARY encoding also.
2.the column will be a dimension, maybe sorted column or unsorted column
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r113092141
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -130,6 +130,7 @@ struct DataChunk2{
9: optional list<binary> encoder_meta; // Extra information required by encoders
10: optional BlockletMinMaxIndex min_max;
11: optional i32 numberOfRowsInpage;
+ 12: optional list<i32> bitmap_encoded_dictionaries;
--- End diff --
What's the purpose?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by mayunSaicmotor <gi...@git.apache.org>.
Github user mayunSaicmotor commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r114114799
--- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CarbonBitMapEncodingExample.scala ---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.examples.util.ExampleUtils
+
+object CarbonBitMapEncodingExample {
--- End diff --
added just test case please review
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r113092685
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala ---
@@ -385,6 +391,13 @@ class TableNewProcessor(cm: TableModel) {
field.scale,
field.schemaOrdinal)
columnSchema.setSortColumn(true)
+ if (bitmapCols.contains(field.column)) {
+ if (columnSchema.getEncodingList.contains(Encoding.DICTIONARY)) {
--- End diff --
please use columnSchema.hasEncoding(Encoding encoding) method
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r113092306
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala ---
@@ -366,6 +366,12 @@ class TableNewProcessor(cm: TableModel) {
val LOGGER = LogServiceFactory.getLogService(TableNewProcessor.getClass.getName)
var allColumns = Seq[ColumnSchema]()
var index = 0
+ var bitmapStr: String = cm.tableProperties
+ .get(CarbonCommonConstants.BITMAP_ENCODING).getOrElse(null)
--- End diff --
please use Option directly.
change bitmapStr != null to bitmapOption.isDefine
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Please check whether BITMAP column use dictionary encoding or not.
If the column don't user dictionary encoding, throw error and stop flow.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Regarding BITMAP encoding:
1. the column should use DICTIONARY encoding also.
2. the column will be a dimension, maybe sorted column or unsorted column
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r113093434
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala ---
@@ -385,6 +391,13 @@ class TableNewProcessor(cm: TableModel) {
field.scale,
field.schemaOrdinal)
columnSchema.setSortColumn(true)
+ if (bitmapCols.contains(field.column)) {
+ if (columnSchema.getEncodingList.contains(Encoding.DICTIONARY)) {
--- End diff --
why not add these code into getColumnSchema method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r113092142
--- Diff: integration/spark-common/src/main/scala/org/apache/spark/sql/execution/command/carbonTableSchema.scala ---
@@ -273,7 +273,7 @@ class AlterTableProcessor(
columnSchema.setDataType(dataType)
columnSchema.setColumnName(colName)
if (alterTableModel.highCardinalityDims.contains(colName)) {
- encoders.remove(encoders.remove(Encoding.DICTIONARY))
+ encoders.remove(Encoding.DICTIONARY)
--- End diff --
please keep old code, avoid conflict when rebase master
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by mayunSaicmotor <gi...@git.apache.org>.
Github user mayunSaicmotor closed the pull request at:
https://github.com/apache/incubator-carbondata/pull/833
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1762/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1777/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1847/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata pull request #833: [CARBONDATA-951] create table ddl ca...
Posted by QiangCai <gi...@git.apache.org>.
Github user QiangCai commented on a diff in the pull request:
https://github.com/apache/incubator-carbondata/pull/833#discussion_r113093677
--- Diff: examples/spark/src/main/scala/org/apache/carbondata/examples/CarbonBitMapEncodingExample.scala ---
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.examples
+
+import org.apache.carbondata.core.constants.CarbonCommonConstants
+import org.apache.carbondata.core.util.CarbonProperties
+import org.apache.carbondata.examples.util.ExampleUtils
+
+object CarbonBitMapEncodingExample {
--- End diff --
please add test case to check function also.
Example is just a demo, will not be run during building.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1873/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-carbondata issue #833: [CARBONDATA-951] create table ddl can speci...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/incubator-carbondata/pull/833
Build Failed with Spark 1.6.2, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder/1846/
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---