You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by sounakr <gi...@git.apache.org> on 2018/01/03 03:10:35 UTC
[GitHub] carbondata pull request #1751: [WIP]Measure null
GitHub user sounakr opened a pull request:
https://github.com/apache/carbondata/pull/1751
[WIP]Measure null
Recognise Null values in blocklets so that it can help in filter prunning.
- [ ] Any interfaces changed?
- [ ] Any backward compatibility impacted?
- [ ] Document update required?
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/sounakr/incubator-carbondata measure_null
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1751.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1751
----
commit 7d0f9709578651ea29fda335663d1d0075091b62
Author: sounakr <so...@...>
Date: 2017-12-29T13:44:03Z
Measure Null Filter
commit 82df440e98234661b8027f14c464da30e407f762
Author: sounakr <so...@...>
Date: 2018-01-03T02:43:06Z
Measure Null Value
commit 67ea82d752605902fb94c73aa9b9bf31d2e9dafb
Author: sounakr <so...@...>
Date: 2018-01-03T03:06:39Z
Measure Null Value
----
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431317
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java ---
@@ -34,7 +34,7 @@
* @return
*/
BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minValue,
- String uniqueBlockPath);
+ byte[] nullValue, String uniqueBlockPath);
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1602/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832035
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
@@ -728,9 +746,9 @@ private DataMapRow convertToRow(IndexKey key) {
private List<String> getPartitions() {
DataMapRow unsafeRow = unsafeMemorySummaryDMStore.getUnsafeRow(0);
- if (unsafeRow.getColumnCount() > 2) {
+ if (unsafeRow.getColumnCount() > 3) {
List<String> partitions = new ArrayList<>();
- DataMapRow row = unsafeRow.getRow(2);
+ DataMapRow row = unsafeRow.getRow(3);
--- End diff --
Please add constant to partition index
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/557/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159368023
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/DataRefNode.java ---
@@ -71,6 +71,8 @@
*/
byte[][] getColumnsMinValue();
+ byte[] getColumnsNullValue();
--- End diff --
please add comment
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431288
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/FilterExecuter.java ---
@@ -46,7 +46,7 @@ boolean applyFilter(RowIntf value, int dimOrdinalMax)
* @param blockMinValue
* @return BitSet
*/
- BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue);
+ BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue, byte[] blockNullValue);
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/735/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/4294/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
retest this please
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751
Retest this please.
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162289924
--- Diff: hadoop/src/test/java/org/apache/carbondata/hadoop/test/util/StoreCreator.java ---
@@ -520,7 +522,19 @@ public static String readCurrentTime() {
}
public static void main(String[] args) {
- StoreCreator.createCarbonStore();
+
+ SimpleDateFormat parser = null;
+ parser = new SimpleDateFormat("yyyy-MM-dd");
+ parser.setTimeZone(TimeZone.getTimeZone("IST"));
+
+ String v = "2011-11-11";
+ try {
+ long value = parser.parse(v).getTime() * 1000L;
+ System.out.print(value);
+ } catch (ParseException pe) {
+
+ }
+ // StoreCreator.createCarbonStore();
--- End diff --
Removed
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162124981
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanEqualFilterExecuterImpl.java ---
@@ -150,14 +155,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
DataType dataType) {
Object value =
DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
- // TODO handle min and max for null values.
if (filterValue[i] == null) {
- return true;
+ if (nullValue == true) {
+ return true;
+ } else {
+ return false;
+ }
--- End diff --
Just return nullValue no need if condition
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2851/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2653/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
retest this please
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431581
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala ---
@@ -39,8 +39,8 @@ object ExampleUtils {
val warehouse = s"$rootPath/examples/spark2/target/warehouse"
CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss")
- .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd HH:mm:ss")
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2847/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2992/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159831719
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/TablePageStatistics.java ---
@@ -71,6 +82,10 @@ private void updateMeasureMinMax(EncodedColumnPage[] measures) {
return dimensionMaxValue;
}
+ public byte[] getDimentionNullValue() {
--- End diff --
typo getDimensionNullValue
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/5901/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162290297
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
@@ -196,6 +201,10 @@ private DataMapRowImpl loadToUnsafe(DataFileFooter fileFooter,
unsafeMemorySummaryDMStore.getSchema()[taskMinMaxOrdinal], maxValues,
TASK_MAX_VALUES_INDEX, false);
ordinal++;
+ taskMinMaxOrdinal++;
+
+ summaryRow.setByteArray(minMaxIndex.getNullValues().toByteArray(), taskMinMaxOrdinal);
--- End diff --
Added new routine addTaskNullValues which is going to OR all the Null Bitsets from all the blocklets and place the final value into the summary row. Like the one done for minMax.
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2672/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2732/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162290027
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGrtrThanEquaToFilterExecuterImpl.java ---
@@ -149,13 +154,17 @@ private boolean isScanRequired(byte[] blockMaxValue, byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] maxValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] maxValue, boolean nullValue, Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(maxValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
- return true;
+ if (nullValue == true) {
+ return true;
+ } else {
+ return false;
+ }
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2811/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751
retest this please
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2671/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1485/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/1781/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
retest sdv please
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3432/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.3.2, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/10257/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2959/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
retest sdv please
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/4466/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1271/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829285
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala ---
@@ -62,9 +62,9 @@ object CarbonSessionExample {
spark.sql(
s"""
- | SELECT *
- | FROM carbon_table
- | WHERE stringfield = 'spark' AND decimalField > 40
+ | SELECT *
--- End diff --
Don't change this example.
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1643/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162053147
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
@@ -196,6 +201,10 @@ private DataMapRowImpl loadToUnsafe(DataFileFooter fileFooter,
unsafeMemorySummaryDMStore.getSchema()[taskMinMaxOrdinal], maxValues,
TASK_MAX_VALUES_INDEX, false);
ordinal++;
+ taskMinMaxOrdinal++;
+
+ summaryRow.setByteArray(minMaxIndex.getNullValues().toByteArray(), taskMinMaxOrdinal);
--- End diff --
We need to calculate null bits based on whole blocks and then set to the summary
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/5183/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2724/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161366923
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
@@ -25,6 +25,8 @@
Object getMax();
+ byte getNull();
--- End diff --
SimpleStatsResult is the statistics for a ColumnPage, what does getNull() means for a page?
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2560/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2850/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162125695
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFiterExecuterImpl.java ---
@@ -153,13 +158,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
- return true;
+ if (nullValue == true) {
+ return true;
+ } else {
+ return false;
+ }
--- End diff --
just return nullValue
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2859/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832789
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java ---
@@ -34,7 +34,7 @@
* @return
*/
BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minValue,
- String uniqueBlockPath);
+ byte[] nullValue, String uniqueBlockPath);
--- End diff --
Here also use BlockletStats object
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1544/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431217
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataRefNodeWrapper.java ---
@@ -99,6 +99,10 @@ public BlockletDataRefNodeWrapper(List<TableBlockInfo> blockInfos, int index,
return null;
}
+ @Override public byte[] getColumnsNullValue() {
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829085
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
struct BlockletMinMaxIndex{
1: required list<binary> min_values; //Min value of all columns of one blocklet Bit-Packed
2: required list<binary> max_values; //Max value of all columns of one blocklet Bit-Packed
+ 3: optional list<byte> null_value; // Null Bool turn on if null value present.
--- End diff --
Better use binary and try to use BitSet for null values
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1612/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2971/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832676
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/FilterExecuter.java ---
@@ -46,7 +46,7 @@ boolean applyFilter(RowIntf value, int dimOrdinalMax)
* @param blockMinValue
* @return BitSet
*/
- BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue);
+ BitSet isScanRequired(byte[][] blockMaxValue, byte[][] blockMinValue, byte[] blockNullValue);
--- End diff --
Better wrap these parameters to new BlockletStats object
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2658/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
retest this please
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431299
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java ---
@@ -43,5 +43,6 @@ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minVal
*
* @return
*/
- Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue);
+ Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue,
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/3031/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1491/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1607/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2507/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2723/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2496/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1279/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by kumarvishal09 <gi...@git.apache.org>.
Github user kumarvishal09 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r160157245
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
struct BlockletMinMaxIndex{
1: required list<binary> min_values; //Min value of all columns of one blocklet Bit-Packed
2: required list<binary> max_values; //Max value of all columns of one blocklet Bit-Packed
+ 3: optional list<byte> null_value; // Null Bool turn on if null value present.
--- End diff --
we can use PresenceMeta to store null value
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/9846/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161963305
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
@@ -25,6 +25,8 @@
Object getMax();
+ byte getNull();
--- End diff --
We can do it. But in later phase of execution the Boolean will again get converted to Byte[]. So it doesn't make that much difference.
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162289956
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanFiterExecuterImpl.java ---
@@ -153,13 +158,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
- return true;
+ if (nullValue == true) {
+ return true;
+ } else {
+ return false;
+ }
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/53/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2837/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2880/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2857/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2503/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.3.1, Please check CI http://136.243.101.176:8080/job/carbondataprbuilder2.3/8802/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431335
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/TablePageStatistics.java ---
@@ -71,6 +82,10 @@ private void updateMeasureMinMax(EncodedColumnPage[] measures) {
return dimensionMaxValue;
}
+ public byte[] getDimentionNullValue() {
--- End diff --
Rectified
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832127
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataRefNodeWrapper.java ---
@@ -99,6 +99,10 @@ public BlockletDataRefNodeWrapper(List<TableBlockInfo> blockInfos, int index,
return null;
}
+ @Override public byte[] getColumnsNullValue() {
--- End diff --
Change byte[] to BitSet
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431568
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/CarbonSessionExample.scala ---
@@ -62,9 +62,9 @@ object CarbonSessionExample {
spark.sql(
s"""
- | SELECT *
- | FROM carbon_table
- | WHERE stringfield = 'spark' AND decimalField > 40
+ | SELECT *
--- End diff --
Rectified
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2932/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/7125/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2842/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/3036/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2777/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1762/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162124619
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeGrtrThanEquaToFilterExecuterImpl.java ---
@@ -149,13 +154,17 @@ private boolean isScanRequired(byte[] blockMaxValue, byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] maxValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] maxValue, boolean nullValue, Object[] filterValue,
DataType dataType) {
Object value = DataTypeUtil.getMeasureObjectFromDataType(maxValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
// TODO handle min and max for null values.
if (filterValue[i] == null) {
- return true;
+ if (nullValue == true) {
+ return true;
+ } else {
+ return false;
+ }
--- End diff --
you no need use if condition, just return nullValue
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1618/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1426/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2660/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2875/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2968/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751
Retest this please
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161366903
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
@@ -25,6 +25,8 @@
Object getMax();
+ byte getNull();
--- End diff --
SimpleStatsResult is the statistics for a ColumnPage, what does getNull() means for a page?
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3050/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159368019
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/chunk/AbstractRawColumnChunk.java ---
@@ -29,6 +29,8 @@
private byte[][] maxValues;
+ private byte[] nullValues;
--- End diff --
please add comment
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1647/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751
Retest this please.
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://95.216.28.178:8080/job/ApacheCarbonPRBuilder1/2182/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2846/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1730/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162289986
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/RowLevelRangeLessThanEqualFilterExecuterImpl.java ---
@@ -150,14 +155,17 @@ private boolean isScanRequired(byte[] blockMinValue, byte[][] filterValues) {
return isScanRequired;
}
- private boolean isScanRequired(byte[] minValue, Object[] filterValue,
+ private boolean isScanRequired(byte[] minValue, boolean nullValue, Object[] filterValue,
DataType dataType) {
Object value =
DataTypeUtil.getMeasureObjectFromDataType(minValue, dataType);
for (int i = 0; i < filterValue.length; i++) {
- // TODO handle min and max for null values.
if (filterValue[i] == null) {
- return true;
+ if (nullValue == true) {
+ return true;
+ } else {
+ return false;
+ }
--- End diff --
Done
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2718/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431169
--- Diff: core/src/main/java/org/apache/carbondata/core/indexstore/blockletindex/BlockletDataMap.java ---
@@ -728,9 +746,9 @@ private DataMapRow convertToRow(IndexKey key) {
private List<String> getPartitions() {
DataMapRow unsafeRow = unsafeMemorySummaryDMStore.getUnsafeRow(0);
- if (unsafeRow.getColumnCount() > 2) {
+ if (unsafeRow.getColumnCount() > 3) {
List<String> partitions = new ArrayList<>();
- DataMapRow row = unsafeRow.getRow(2);
+ DataMapRow row = unsafeRow.getRow(3);
--- End diff --
Done
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161430749
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
@@ -25,6 +25,8 @@
Object getMax();
+ byte getNull();
--- End diff --
getNull() will returns Null value present in page or not.
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1322/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2917/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2810/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2879/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1751
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2881/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751
Retest this please.
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1501/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1805/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1421/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1282/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829185
--- Diff: examples/spark2/src/main/scala/org/apache/carbondata/examples/ExampleUtils.scala ---
@@ -39,8 +39,8 @@ object ExampleUtils {
val warehouse = s"$rootPath/examples/spark2/target/warehouse"
CarbonProperties.getInstance()
- .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy/MM/dd HH:mm:ss")
- .addProperty(CarbonCommonConstants.CARBON_DATE_FORMAT, "yyyy/MM/dd")
+ .addProperty(CarbonCommonConstants.CARBON_TIMESTAMP_FORMAT, "yyyy-MM-dd HH:mm:ss")
--- End diff --
Don't change the example class
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder2.1/2005/
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1751
Build Success with Spark 2.2.1, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/3973/
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159829745
--- Diff: core/src/main/java/org/apache/carbondata/core/datastore/page/statistics/SimpleStatsResult.java ---
@@ -25,6 +25,8 @@
Object getMax();
+ byte getNull();
--- End diff --
Better use boolean and change method to isNull
---
[GitHub] carbondata issue #1751: [CARBONDATA-1971][Blocklet Prunning] Measure Null va...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1751
Retest this please
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r161431619
--- Diff: format/src/main/thrift/carbondata.thrift ---
@@ -45,6 +45,7 @@ struct BlockletBTreeIndex{
struct BlockletMinMaxIndex{
1: required list<binary> min_values; //Min value of all columns of one blocklet Bit-Packed
2: required list<binary> max_values; //Max value of all columns of one blocklet Bit-Packed
+ 3: optional list<byte> null_value; // Null Bool turn on if null value present.
--- End diff --
Done. Stored as a Bitset
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r162126330
--- Diff: hadoop/src/test/java/org/apache/carbondata/hadoop/test/util/StoreCreator.java ---
@@ -520,7 +522,19 @@ public static String readCurrentTime() {
}
public static void main(String[] args) {
- StoreCreator.createCarbonStore();
+
+ SimpleDateFormat parser = null;
+ parser = new SimpleDateFormat("yyyy-MM-dd");
+ parser.setTimeZone(TimeZone.getTimeZone("IST"));
+
+ String v = "2011-11-11";
+ try {
+ long value = parser.parse(v).getTime() * 1000L;
+ System.out.print(value);
+ } catch (ParseException pe) {
+
+ }
+ // StoreCreator.createCarbonStore();
--- End diff --
why this class i changed
---
[GitHub] carbondata pull request #1751: [CARBONDATA-1971][Blocklet Prunning] Measure ...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1751#discussion_r159832781
--- Diff: core/src/main/java/org/apache/carbondata/core/scan/filter/executer/ImplicitColumnFilterExecutor.java ---
@@ -43,5 +43,6 @@ BitSet isFilterValuesPresentInBlockOrBlocklet(byte[][] maxValue, byte[][] minVal
*
* @return
*/
- Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue);
+ Boolean isFilterValuesPresentInAbstractIndex(byte[][] maxValue, byte[][] minValue,
--- End diff --
Here also use BlockletStats object
---