You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by xubo245 <gi...@git.apache.org> on 2017/09/17 10:12:20 UTC
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
GitHub user xubo245 opened a pull request:
https://github.com/apache/carbondata/pull/1362
[CARBONDATA-1444]Support boolean
CarbonData should support boolean data type in following aspects:
1. create table
2.insert into table values and select
3.load data
4.filter
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/xubo245/carbondata supportBoolean
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1362.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1362
----
commit 581288d82c53247e4f68e37d638bfe7cc7fb7ded
Author: xubo245 <60...@qq.com>
Date: 2017-09-15T15:42:44Z
support Boolean data type
commit e7d279a41f6e58b6640f828d9fcb1c9962097cb8
Author: xubo245 <60...@qq.com>
Date: 2017-09-17T07:57:40Z
fix findbugs
commit 6e8f5da599038649b22a1866425941f706650135
Author: xubo245 <60...@qq.com>
Date: 2017-09-17T10:01:06Z
add test case
----
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890685
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -530,6 +546,11 @@ private void putNull(int rowId) {
public abstract byte[] getShortIntPage();
/**
+ * Get boolean value page
+ */
+ public abstract byte[] getBooleanPage();
--- End diff --
Ok, I have changed it.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1121/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1015/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/368/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/217/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r141994786
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -707,6 +711,10 @@
*/
public static final char BIG_INT_MEASURE = 'd';
/**
+ * BOOLEAN_VALUE_MEASURE
+ */
+ public static final char BOOLEAN_MEASURE = 'a';
--- End diff --
ok, I have removed it.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143614716
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -206,6 +199,7 @@ public static ColumnPage newPage(TableSpec.ColumnSpec columnSpec, DataType dataT
}
} else {
switch (dataType) {
+ case BOOLEAN:
--- End diff --
It is better to add a `newBooleanPage`, otherwise `newBytePage` internal will store the data type as byte instead of boolean
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1362
LGTM
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/399/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/340/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/308/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1083/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307603
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java ---
@@ -92,6 +90,10 @@ public ColumnPageDecoder createDecoder(List<Encoding> encodings, List<ByteBuffer
RLEEncoderMeta metadata = new RLEEncoderMeta();
metadata.readFields(in);
return new RLECodec().createDecoder(metadata);
+ } else if (encoding == BOOL_BYTE) {
+ BooleanEncoderMeta metadata=new BooleanEncoderMeta();
--- End diff --
Not compliant to coding standard. Add space before and after `=`
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307631
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java ---
@@ -92,6 +90,10 @@ public ColumnPageDecoder createDecoder(List<Encoding> encodings, List<ByteBuffer
RLEEncoderMeta metadata = new RLEEncoderMeta();
metadata.readFields(in);
return new RLECodec().createDecoder(metadata);
+ } else if (encoding == BOOL_BYTE) {
+ BooleanEncoderMeta metadata=new BooleanEncoderMeta();
--- End diff --
I think it is better to use RLE on byte, since it just has two values
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444]Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/830/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/214/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/967/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/433/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r141512380
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -23,6 +23,10 @@
public final class CarbonCommonConstants {
/**
+ * Boolean size in bytes
+ */
+ public static final int BOOLEAN_SIZE_IN_BYTE = 1;
--- End diff --
ok
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/207/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890027
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java ---
@@ -92,6 +90,10 @@ public ColumnPageDecoder createDecoder(List<Encoding> encodings, List<ByteBuffer
RLEEncoderMeta metadata = new RLEEncoderMeta();
metadata.readFields(in);
return new RLECodec().createDecoder(metadata);
+ } else if (encoding == BOOL_BYTE) {
+ BooleanEncoderMeta metadata=new BooleanEncoderMeta();
--- End diff --
ok, I have changed the encoding from snappy to RLE
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890183
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+/**
+ * convert tools for boolean data type
+ */
+public class BooleanConvert {
--- End diff --
I think it should be reserved. There are some place invoke it.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890406
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -321,6 +315,11 @@ private static ColumnPage newLVBytesPage(TableSpec.ColumnSpec columnSpec,
public abstract void setShortIntPage(byte[] shortIntData);
/**
+ * Set boolean values to page
+ */
+ public abstract void setBooleanPage(byte[] booleanData);
--- End diff --
ok, I have changed it
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/329/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143617076
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -361,6 +360,10 @@ public void putData(int rowId, Object value) {
return;
}
switch (dataType) {
+ case BOOLEAN:
+ putByte(rowId, (byte) value);
--- End diff --
use `putBoolean`
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143617246
--- Diff: integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/booleantype/BooleanDataTypesLoadTest.scala ---
@@ -0,0 +1,655 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.booleantype
+
+import java.io.File
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+/**
+ * Created by root on 9/17/17.
+ */
+class BooleanDataTypesLoadTest extends QueryTest with BeforeAndAfterEach with BeforeAndAfterAll {
+ val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+
--- End diff --
add some testcase to use unsafe while loading
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/966/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/910/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/279/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143616906
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -187,6 +188,12 @@ private void convertToColumnarAndAddToPages(int rowId, CarbonRow row, byte[] mdk
value != null) {
value = ((Decimal) value).toJavaBigDecimal();
}
+ if (measurePages[i].getColumnSpec().getSchemaDataType()
+ == DataType.BOOLEAN && value != null) {
--- End diff --
incorrect line wrap
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/155/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307225
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -707,6 +711,10 @@
*/
public static final char BIG_INT_MEASURE = 'd';
/**
+ * BOOLEAN_VALUE_MEASURE
+ */
+ public static final char BOOLEAN_MEASURE = 'a';
--- End diff --
This is not required. Others are for backward compatible only.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/278/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890823
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java ---
@@ -215,7 +215,7 @@ private void initMeasureBlockIndexes() {
} else {
// specific for restructure case where default values need to be filled
pageNumbers = blockChunkHolder.getDataBlock().numberOfPages();
- numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize() };
+ numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize()};
--- End diff --
Ok.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
I changed the format module.Other module can not invoke the new code in format module.
Could you help me to deal with it? @ravipesala
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/335/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/329/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143616554
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+/**
+ * convert tools for boolean data type
+ */
+public class BooleanConvert {
+
+ public static final byte trueValue = 1;
+ public static final byte falseValue = 0;
+ /**
+ * convert boolean to byte
+ *
+ * @param data data of boolean data type
+ * @return byte type data by convert
+ */
+ public static byte boolean2Byte(boolean data) {
+ return data ? (byte) 1 : (byte) 0;
--- End diff --
use constant
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143614968
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -436,6 +439,11 @@ public void putData(int rowId, Object value) {
public abstract void putShortInt(int rowId, int value);
/**
+ * Set boolean value at rowId
+ */
+ public abstract void putBoolean(int rowId, boolean value);
--- End diff --
Since internally boolean is stored as byte, so you can implement this function to call `putByte` instead of abstract function
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/212/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/337/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/328/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/262/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/953/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/957/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/334/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444]Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/75/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143615434
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java ---
@@ -38,11 +38,7 @@
import org.apache.carbondata.core.util.CarbonUtil;
import org.apache.carbondata.format.Encoding;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_DELTA_INTEGRAL;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_FLOATING;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_INTEGRAL;
-import static org.apache.carbondata.format.Encoding.DIRECT_COMPRESS;
-import static org.apache.carbondata.format.Encoding.RLE_INTEGRAL;
+import static org.apache.carbondata.format.Encoding.*;
--- End diff --
Do not use wildcard
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/216/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/382/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1062/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1362
Thanks for working on this
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/965/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/959/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1082/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143616465
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+/**
+ * convert tools for boolean data type
+ */
+public class BooleanConvert {
+
+ public static final byte trueValue = 1;
+ public static final byte falseValue = 0;
+ /**
+ * convert boolean to byte
+ *
+ * @param data data of boolean data type
+ * @return byte type data by convert
+ */
+ public static byte boolean2Byte(boolean data) {
+ return data ? (byte) 1 : (byte) 0;
+ }
+
+ /**
+ * convert byte to boolean
+ *
+ * @param data byte type data
+ * @return boolean type data
+ */
+ public static boolean byte2Boolean(int data) {
+ return data == 1 ? true : false;
--- End diff --
simply it
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1013/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/277/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1033/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/279/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143614990
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -321,6 +315,11 @@ private static ColumnPage newLVBytesPage(TableSpec.ColumnSpec columnSpec,
public abstract void setShortIntPage(byte[] shortIntData);
/**
+ * Set boolean values to page
+ */
+ public abstract void setBooleanPage(byte[] booleanData);
--- End diff --
Since internally boolean is stored as byte, so you can implement this function to call setBytePage instead of abstract function
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
@xubo245 Please use `mvn clean -Pbuild-with-format install` to build fomat module
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1362
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
@jackylk The conflicts have been solved
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/280/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/962/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/339/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890387
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -436,6 +439,11 @@ public void putData(int rowId, Object value) {
public abstract void putShortInt(int rowId, int value);
/**
+ * Set boolean value at rowId
+ */
+ public abstract void putBoolean(int rowId, boolean value);
--- End diff --
ok, I have changed it
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143891060
--- Diff: integration/spark2/src/test/scala/org/apache/carbondata/spark/testsuite/booleantype/BooleanDataTypesLoadTest.scala ---
@@ -0,0 +1,655 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.carbondata.spark.testsuite.booleantype
+
+import java.io.File
+
+import org.apache.spark.sql.Row
+import org.apache.spark.sql.test.util.QueryTest
+import org.scalatest.{BeforeAndAfterAll, BeforeAndAfterEach}
+
+/**
+ * Created by root on 9/17/17.
+ */
+class BooleanDataTypesLoadTest extends QueryTest with BeforeAndAfterEach with BeforeAndAfterAll {
+ val rootPath = new File(this.getClass.getResource("/").getPath
+ + "../../../..").getCanonicalPath
+
--- End diff --
OK, I add some testcase to use unsafe in BooleanDataTypesLoadTest.scala and BooleanDataTypesBigFileTest.scala
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890779
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java ---
@@ -38,11 +38,7 @@
import org.apache.carbondata.core.util.CarbonUtil;
import org.apache.carbondata.format.Encoding;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_DELTA_INTEGRAL;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_FLOATING;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_INTEGRAL;
-import static org.apache.carbondata.format.Encoding.DIRECT_COMPRESS;
-import static org.apache.carbondata.format.Encoding.RLE_INTEGRAL;
+import static org.apache.carbondata.format.Encoding.*;
--- End diff --
I also changed similar problem in columnPage
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143615152
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -485,6 +496,11 @@ private void putNull(int rowId) {
public abstract int getShortInt(int rowId);
/**
+ * Get boolean value at rowId
+ */
+ public abstract boolean getBoolean(int rowId);
--- End diff --
The same as above, you can implement this function here instead of abstract
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/963/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307286
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -271,6 +267,18 @@ private static ColumnPage newIntPage(TableSpec.ColumnSpec columnSpec, int[] intD
return columnPage;
}
+ private static ColumnPage newBooleanPage(TableSpec.ColumnSpec columnSpec, byte[] booleanData) {
+ ColumnPage columnPage = createPage(columnSpec, BOOLEAN, booleanData.length);
--- End diff --
To make it simple, I think it is better to store boolean as byte in ColumnPage, so you can create a ColumnPage of Byte type.
When come to encoding, you still can do proper encoding of this column page.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/201/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/454/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890454
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -321,6 +315,11 @@ private static ColumnPage newLVBytesPage(TableSpec.ColumnSpec columnSpec,
public abstract void setShortIntPage(byte[] shortIntData);
/**
+ * Set boolean values to page
+ */
+ public abstract void setBooleanPage(byte[] booleanData);
--- End diff --
ok, I have removed it
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/331/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890342
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanEncoderMeta.java ---
@@ -0,0 +1,40 @@
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.TableSpec;
+import org.apache.carbondata.core.datastore.page.encoding.ColumnPageEncoderMeta;
+import org.apache.carbondata.core.datastore.page.statistics.SimpleStatsResult;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.table.Writable;
+
+public class BooleanEncoderMeta extends ColumnPageEncoderMeta implements Writable {
+ private String compressorName;
+
+ public BooleanEncoderMeta() {
+ }
+
+ public BooleanEncoderMeta(TableSpec.ColumnSpec columnSpec, DataType storeDataType,
+ SimpleStatsResult stats, String compressorName) {
+ super(columnSpec,storeDataType,stats,compressorName);
+ this.compressorName = compressorName;
--- End diff --
ok
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890810
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java ---
@@ -203,7 +203,7 @@ private void initMeasureBlockIndexes() {
} else {
// specific for restructure case where default values need to be filled
pageNumbers = blockChunkHolder.getDataBlock().numberOfPages();
- numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize() };
+ numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize()};
--- End diff --
Ok.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444]Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/198/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/205/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/213/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
@jackylk please review it
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890228
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -187,6 +179,7 @@ public static ColumnPage newPage(TableSpec.ColumnSpec columnSpec, DataType dataT
case BYTE:
case SHORT:
case SHORT_INT:
+ case BOOLEAN:
--- End diff --
ok
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/154/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/968/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890910
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -361,6 +360,10 @@ public void putData(int rowId, Object value) {
return;
}
switch (dataType) {
+ case BOOLEAN:
+ putByte(rowId, (byte) value);
--- End diff --
I have deleted it.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890419
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -485,6 +496,11 @@ private void putNull(int rowId) {
public abstract int getShortInt(int rowId);
/**
+ * Get boolean value at rowId
+ */
+ public abstract boolean getBoolean(int rowId);
--- End diff --
ok, I have changed it
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/956/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/969/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/911/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/493/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/260/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307667
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanEncoderMeta.java ---
@@ -0,0 +1,40 @@
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+import java.io.DataInput;
+import java.io.DataOutput;
+import java.io.IOException;
+
+import org.apache.carbondata.core.datastore.TableSpec;
+import org.apache.carbondata.core.datastore.page.encoding.ColumnPageEncoderMeta;
+import org.apache.carbondata.core.datastore.page.statistics.SimpleStatsResult;
+import org.apache.carbondata.core.metadata.datatype.DataType;
+import org.apache.carbondata.core.metadata.schema.table.Writable;
+
+public class BooleanEncoderMeta extends ColumnPageEncoderMeta implements Writable {
+ private String compressorName;
+
+ public BooleanEncoderMeta() {
+ }
+
+ public BooleanEncoderMeta(TableSpec.ColumnSpec columnSpec, DataType storeDataType,
+ SimpleStatsResult stats, String compressorName) {
+ super(columnSpec,storeDataType,stats,compressorName);
+ this.compressorName = compressorName;
--- End diff --
compressorName is already stored in ColumnPageEncoderMeta, this BooleanEncoderMeta is not required.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890703
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/EncodingFactory.java ---
@@ -38,11 +38,7 @@
import org.apache.carbondata.core.util.CarbonUtil;
import org.apache.carbondata.format.Encoding;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_DELTA_INTEGRAL;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_FLOATING;
-import static org.apache.carbondata.format.Encoding.ADAPTIVE_INTEGRAL;
-import static org.apache.carbondata.format.Encoding.DIRECT_COMPRESS;
-import static org.apache.carbondata.format.Encoding.RLE_INTEGRAL;
+import static org.apache.carbondata.format.Encoding.*;
--- End diff --
Ok, I have changed it.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143615171
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -530,6 +546,11 @@ private void putNull(int rowId) {
public abstract byte[] getShortIntPage();
/**
+ * Get boolean value page
+ */
+ public abstract byte[] getBooleanPage();
--- End diff --
The same as above, you can implement this function here instead of abstract
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/964/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
@jackylk Please review it.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
Please update jar of format module @ravipesala
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/338/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/330/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890886
--- Diff: processing/src/main/java/org/apache/carbondata/processing/store/TablePage.java ---
@@ -187,6 +188,12 @@ private void convertToColumnarAndAddToPages(int rowId, CarbonRow row, byte[] mdk
value != null) {
value = ((Decimal) value).toJavaBigDecimal();
}
+ if (measurePages[i].getColumnSpec().getSchemaDataType()
+ == DataType.BOOLEAN && value != null) {
--- End diff --
Ok, I have changed it.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143615593
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java ---
@@ -215,7 +215,7 @@ private void initMeasureBlockIndexes() {
} else {
// specific for restructure case where default values need to be filled
pageNumbers = blockChunkHolder.getDataBlock().numberOfPages();
- numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize() };
+ numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize()};
--- End diff --
do not modify this
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1362
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/1030/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/402/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
@jackylk I have added some descriptions for test cases in this PR.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890833
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+/**
+ * convert tools for boolean data type
+ */
+public class BooleanConvert {
+
+ public static final byte trueValue = 1;
+ public static final byte falseValue = 0;
+ /**
+ * convert boolean to byte
+ *
+ * @param data data of boolean data type
+ * @return byte type data by convert
+ */
+ public static byte boolean2Byte(boolean data) {
+ return data ? (byte) 1 : (byte) 0;
+ }
+
+ /**
+ * convert byte to boolean
+ *
+ * @param data byte type data
+ * @return boolean type data
+ */
+ public static boolean byte2Boolean(int data) {
+ return data == 1 ? true : false;
--- End diff --
Ok.
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307244
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -187,6 +179,7 @@ public static ColumnPage newPage(TableSpec.ColumnSpec columnSpec, DataType dataT
case BYTE:
case SHORT:
case SHORT_INT:
+ case BOOLEAN:
--- End diff --
Move it before BYTE. Please modify the same in all switch case
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
Conflicts has been fixed， please review it @jackylk
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/336/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/210/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/455/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890846
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java ---
@@ -0,0 +1,63 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+/**
+ * convert tools for boolean data type
+ */
+public class BooleanConvert {
+
+ public static final byte trueValue = 1;
+ public static final byte falseValue = 0;
+ /**
+ * convert boolean to byte
+ *
+ * @param data data of boolean data type
+ * @return byte type data by convert
+ */
+ public static byte boolean2Byte(boolean data) {
+ return data ? (byte) 1 : (byte) 0;
--- End diff --
Ok.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/215/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
@jackylk CI pass
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/384/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support Boolean data type
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143890311
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/ColumnPage.java ---
@@ -271,6 +267,18 @@ private static ColumnPage newIntPage(TableSpec.ColumnSpec columnSpec, int[] intD
return columnPage;
}
+ private static ColumnPage newBooleanPage(TableSpec.ColumnSpec columnSpec, byte[] booleanData) {
+ ColumnPage columnPage = createPage(columnSpec, BOOLEAN, booleanData.length);
--- End diff --
ok, I use newBytePage to replace newBooleanPage
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/211/
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444] Support boolean data type
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r143615585
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelFilterExecuterImpl.java ---
@@ -203,7 +203,7 @@ private void initMeasureBlockIndexes() {
} else {
// specific for restructure case where default values need to be filled
pageNumbers = blockChunkHolder.getDataBlock().numberOfPages();
- numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize() };
+ numberOfRows = new int[] { blockChunkHolder.getDataBlock().nodeSize()};
--- End diff --
do not modify this
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
I can run success in my computer, but I can not fix this problem in github/jenkins. How to add "mvn clean -Pbuild-with-format install" to run my branch in github after pushing request to github? @ravipesala
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307234
--- Diff: core/src/main/java/org/apache/carbondata/core/constants/CarbonCommonConstants.java ---
@@ -23,6 +23,10 @@
public final class CarbonCommonConstants {
/**
+ * Boolean size in bytes
+ */
+ public static final int BOOLEAN_SIZE_IN_BYTE = 1;
--- End diff --
This is not required, it should be added in DataType enum
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 1.6, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/204/
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by xubo245 <gi...@git.apache.org>.
Github user xubo245 commented on the issue:
https://github.com/apache/carbondata/pull/1362
@jackylk please review it
---
[GitHub] carbondata pull request #1362: [CARBONDATA-1444]Support boolean
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1362#discussion_r139307650
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/encoding/bool/BooleanConvert.java ---
@@ -0,0 +1,61 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.carbondata.core.datastore.page.encoding.bool;
+
+/**
+ * convert tools for boolean data type
+ */
+public class BooleanConvert {
--- End diff --
I think this is not required, it is simple conversion only.
---
[GitHub] carbondata issue #1362: [CARBONDATA-1444] Support boolean
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1362
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/325/
---