You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by geetikagupta16 <gi...@git.apache.org> on 2017/12/13 06:36:40 UTC
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
GitHub user geetikagupta16 opened a pull request:
https://github.com/apache/carbondata/pull/1650
[CARBONDATA-1703] Refactored code for creation of filter for IN expression
Refactored code for creation of filter for IN expression in case of null values
Be sure to do all of the following checklist to help us incorporate
your contribution quickly and easily:
- [ ] Any interfaces changed? No
- [ ] Any backward compatibility impacted? No
- [ ] Document update required? No
- [ ] Testing done
Please provide details on
- Whether new unit test cases have been added or why no new tests are required?
- How it is tested? Please attach test report.
- Is it a performance related change? Please attach the performance test report.
- Any additional information to help reviewers in testing this change.
- [ ] For large changes, please consider breaking it into sub-tasks under an umbrella JIRA.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/geetikagupta16/incubator-carbondata CARBONDATA-1703
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/carbondata/pull/1650.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1650
----
commit 81743ac431587cd7eb89b8218203811aa63a8762
Author: Geetika Gupta <ge...@knoldus.in>
Date: 2017-12-13T06:33:22Z
Refactored code for creation of filter for IN expression
----
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1972/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1650
@jackylk I have updated the PR description please check
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by jackylk <gi...@git.apache.org>.
Github user jackylk commented on the issue:
https://github.com/apache/carbondata/pull/1650
@geetikagupta16 can you add more description of this bug? what is the bug and what is the root analysis of it?
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/703/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by sraghunandan <gi...@git.apache.org>.
Github user sraghunandan commented on the issue:
https://github.com/apache/carbondata/pull/1650
ok to test
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r158826908
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/detailquery/ExpressionWithNullTestCase.scala ---
@@ -75,6 +78,9 @@ class ExpressionWithNullTestCase extends QueryTest with BeforeAndAfterAll {
checkAnswer(sql("select * from expression_test where id not in ('2')"), sql("select * from expression_test_hive where id not in ('2')"))
checkAnswer(sql("select * from expression_test where id not in (cast('2' as int))"), sql("select * from expression_test_hive where id not in (cast('2' as int))"))
// checkAnswer(sql("select * from expression_test where id not in (cast('null' as int))"), sql("select * from expression_test_hive where id not in (cast('null' as int))"))
+ checkAnswer(sql("select * from expression_test where id not in (1,2,NULL)"), sql("select * from expression_test_hive where id not in (1,2,NULL)"))
+ checkAnswer(sql("select * from expression_test where id not in (NULL)"), sql("select * from expression_test_hive where id not in (NULL)"))
--- End diff --
Please add some more test cases with explicit cast and with Nulls
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/875/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1914/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2263/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/861/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2022/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1650
@gvramana Please review again, as I have made the required changes
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2594/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r158826141
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,13 +78,25 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
--- End diff --
In case there are only one value in In list and that is also qualifies as Null, then why cant we return False expression just like Not In case. i.e. Some(new FalseExpression(getCarbonExpression(name))).
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by sraghunandan <gi...@git.apache.org>.
Github user sraghunandan commented on the issue:
https://github.com/apache/carbondata/pull/1650
retest sdv please
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2086/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r158826285
--- Diff: integration/spark-common-test/src/test/scala/org/apache/carbondata/spark/testsuite/detailquery/ExpressionWithNullTestCase.scala ---
@@ -75,6 +78,9 @@ class ExpressionWithNullTestCase extends QueryTest with BeforeAndAfterAll {
checkAnswer(sql("select * from expression_test where id not in ('2')"), sql("select * from expression_test_hive where id not in ('2')"))
checkAnswer(sql("select * from expression_test where id not in (cast('2' as int))"), sql("select * from expression_test_hive where id not in (cast('2' as int))"))
// checkAnswer(sql("select * from expression_test where id not in (cast('null' as int))"), sql("select * from expression_test_hive where id not in (cast('null' as int))"))
--- End diff --
Please uncomment the test cases and rerun it once.
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on the issue:
https://github.com/apache/carbondata/pull/1650
LGTM
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2351/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r157195978
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,9 +78,17 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
--- End diff --
I have made the null check for IN expression in line no. 89
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r157336895
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,9 +78,17 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
--- End diff --
1. why two conditions length=1 and others when logic is same. only else condition should be sufficient
2. Not In case when null comes, in method TransformExpression (as sounak pointed), the behaviour was similar to falseExpression. So we have to conform not IN null case with hive to decide the behaviour.
3. Please add test cases for In and Not in case with null expression and compare behaviour with hive.
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2291/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/carbondata/pull/1650
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/809/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/788/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1650
retest this please
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/741/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2394/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2696/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r157137308
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,9 +78,17 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
--- End diff --
Please make the In and Not In filter as par of TransformExpression. For e.g. Please look into line no 321-323..when it was corrected there it got missed in createFilter.
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2436/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2174/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1185/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2334/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/1932/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2409/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2387/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r157416588
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,9 +78,17 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
--- End diff --
The 1 condition was added when we have only NULL in our IN expression for that case, the Left expression was becoming NULL. So to avoid that we added this case.
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2593/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by gvramana <gi...@git.apache.org>.
Github user gvramana commented on the issue:
https://github.com/apache/carbondata/pull/1650
LGTM
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/685/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1212/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1650
retest this please
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r158903934
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,13 +78,25 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
+ convertToJavaList(values.map(filterValues =>
+ getCarbonLiteralExpression(name, filterValues)).toList))))
+ } else {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
+ convertToJavaList(values.filterNot(_ == null)
+ .map(filterValues => getCarbonLiteralExpression(name, filterValues)).toList))))
+ }
case sources.Not(sources.In(name, values)) =>
- Some(new NotInExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.contains(null)) {
+ Some(new FalseExpression(getCarbonExpression(name)))
+ } else {
+ Some(new NotInExpression(getCarbonExpression(name),
--- End diff --
Can you please provide scenario for the same
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2040/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Fail , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2392/
---
[GitHub] carbondata pull request #1650: [CARBONDATA-1703] Refactored code for creatio...
Posted by sounakr <gi...@git.apache.org>.
Github user sounakr commented on a diff in the pull request:
https://github.com/apache/carbondata/pull/1650#discussion_r158827952
--- Diff: integration/spark2/src/main/scala/org/apache/spark/sql/optimizer/CarbonFilters.scala ---
@@ -78,13 +78,25 @@ object CarbonFilters {
Some(new LessThanEqualToExpression(getCarbonExpression(name),
getCarbonLiteralExpression(name, value)))
case sources.In(name, values) =>
- Some(new InExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.length == 1 && values(0) == null) {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
+ convertToJavaList(values.map(filterValues =>
+ getCarbonLiteralExpression(name, filterValues)).toList))))
+ } else {
+ Some(new InExpression(getCarbonExpression(name),
+ new ListExpression(
+ convertToJavaList(values.filterNot(_ == null)
+ .map(filterValues => getCarbonLiteralExpression(name, filterValues)).toList))))
+ }
case sources.Not(sources.In(name, values)) =>
- Some(new NotInExpression(getCarbonExpression(name),
- new ListExpression(
- convertToJavaList(values.map(f => getCarbonLiteralExpression(name, f)).toList))))
+ if (values.contains(null)) {
+ Some(new FalseExpression(getCarbonExpression(name)))
+ } else {
+ Some(new NotInExpression(getCarbonExpression(name),
--- End diff --
I think similar changes are required in transformExpression also. In case Cast is not resolved by Spark then transformExpression will handle those expression. But prior doing any changes we need supporting test cases.
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2100/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2247/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/878/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by ravipesala <gi...@git.apache.org>.
Github user ravipesala commented on the issue:
https://github.com/apache/carbondata/pull/1650
SDV Build Success , Please check CI http://144.76.159.231:8080/job/ApacheSDVTests/2384/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by geetikagupta16 <gi...@git.apache.org>.
Github user geetikagupta16 commented on the issue:
https://github.com/apache/carbondata/pull/1650
retest this please
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.1.0, Please check CI http://136.243.101.176:8080/job/ApacheCarbonPRBuilder1/2103/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Failed with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/1171/
---
[GitHub] carbondata issue #1650: [CARBONDATA-1703] Refactored code for creation of fi...
Posted by CarbonDataQA <gi...@git.apache.org>.
Github user CarbonDataQA commented on the issue:
https://github.com/apache/carbondata/pull/1650
Build Success with Spark 2.2.0, Please check CI http://88.99.58.216:8080/job/ApacheCarbonPRBuilder/945/
---