You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@carbondata.apache.org by GitBox <gi...@apache.org> on 2021/02/22 13:14:31 UTC
[GitHub] [carbondata] QiangCai opened a new pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
QiangCai opened a new pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097
### Why is this PR needed?
1. in spark version 3, org.apache.spark.sql.sources.Filter is sealed
2. CarbonLateDecodeStrategy class's name is incorrect
3. CarbonDataSourceScan should be the same for 2.3 and 2.4
### What changes were proposed in this PR?
1. translate spark Expression to carbon Expression directly, skip spark Filter step. Remove all spark Filter in carbon code.
2. separate CarbonLateDecodeStrategy to CarbonSourceStrategy and DMLStrategy, and simplify the code of CarbonSourceStrategy
3. move CarbonDataSourceScan back to the source folder
### Does this PR introduce any user interface change?
- No
### Is any new testcase added?
- No
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786479544
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3755/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai closed pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
QiangCai closed pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784962948
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5511/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786477934
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5520/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-789546202
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3349/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
QiangCai commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786371903
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] MarvinLitt commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
MarvinLitt commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-791226234
LGTM
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784040262
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3739/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-785231539
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5514/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-785239855
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3749/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] MarvinLitt commented on a change in pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
MarvinLitt commented on a change in pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#discussion_r587998973
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/index/TextMatchUDF.scala
##########
@@ -34,13 +32,3 @@ class TextMatchMaxDocUDF extends ((String, Int) => Boolean) with Serializable {
v1.length > 0
}
}
Review comment:
the same as the Spatial index, if removed, how to use the fun.
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/geo/InPolygonUDF.scala
##########
@@ -59,23 +58,3 @@ class InPolygonRangeListUDF extends ((String, String) => Boolean) with Serializa
true // Carbon applies the filter. So, Spark do not have to apply filter.
}
}
Review comment:
this filter is use for Spatial index， if remove how about that?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-788605734
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12545/job/ApacheCarbon_PR_Builder_2.4.5/3339/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-785541812
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5515/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-788600828
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5098/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-785544498
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3750/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] QiangCai commented on a change in pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
QiangCai commented on a change in pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#discussion_r588018676
##########
File path: integration/spark/src/main/scala/org/apache/carbondata/geo/InPolygonUDF.scala
##########
@@ -59,23 +58,3 @@ class InPolygonRangeListUDF extends ((String, String) => Boolean) with Serializa
true // Carbon applies the filter. So, Spark do not have to apply filter.
}
}
Review comment:
1. it will still keep the udf and expression for Spatial index
2. In the old flow, carbon converts spark expression to spark filter, and then converts spark filter to carbon expression.
So, after creating carbon expression, this filter will be removed from memory.
Now in the new flow, it will convert spark expression to carbon expression directly, and not need this filter.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-789545650
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12545/job/ApacheCarbonPRBuilder2.3/5108/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-783401216
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5503/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786551057
Build Success with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5521/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784270298
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3741/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784270040
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5506/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786552446
Build Success with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3756/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784798556
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5509/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-783406803
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3738/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784963097
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3746/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784035438
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5504/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] MarvinLitt commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
MarvinLitt commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-791242231
merged
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan for spark3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786339618
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786409127
Build Failed with Spark 2.3.4, Please check CI http://121.244.95.60:12444/job/ApacheCarbonPRBuilder2.3/5519/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [WIP] Refactor CarbonLateDecodeStrategy and CarbonDataSourceScan
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-784788475
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3744/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [carbondata] CarbonDataQA2 commented on pull request #4097: [CARBONDATA-4137] Refactor CarbonDataSourceScan without the soruces.Filter of Spark 3
Posted by GitBox <gi...@apache.org>.
CarbonDataQA2 commented on pull request #4097:
URL: https://github.com/apache/carbondata/pull/4097#issuecomment-786408503
Build Failed with Spark 2.4.5, Please check CI http://121.244.95.60:12444/job/ApacheCarbon_PR_Builder_2.4.5/3754/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org