You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@carbondata.apache.org by qiuchenjian <80...@qq.com> on 2019/03/09 09:14:00 UTC
[Discuss]Adapt MV datamap to spark 2.1 version
Hi, all
It's a summary that using spark 2.1 version to run MV datamap, the all MV
test cases have passed, detail information see the MvAdaptSpark2.1, any
suggestion?
1,The Class we cann’t access in Spark 2.1 version
(1). org.apache.spark.internal.Logging
(2). org.apache.spark.sql.internal.SQLConf
Solution:Create class extends above classed in spark2.1 modular
2,The Class that Spark 2.1 version doesn’t have
(1). org.apache.spark.sql.catalyst.plans.logical.Subquery
(2). org.apache.spark.sql.catalyst.catalog.interface.HiveTableRelation
Solution: Use CatalogRelation instead and don’t use (in
LogicalPlanSignatureGenerator)
Mv the Subquery code to carbon project
3,The method that we can’t access in Spark 2.1 version
(1). sparkSession.sessionState.catalog.lookupRelation
Solution: Solution:Add this method to CarbonToSparkAdapter in spark2.1
modular
4,The changes of some class
(1). org.apache.spark.sql.catalyst.expressions.SortOrder
(2). org.apache.spark.sql.catalyst.expressions.Cast
(3). org.apache.spark.sql.catalyst.plans.Statistics
Solution: Adapt the new interface
5,The method that Spark 2.1 version doesn’t have
(1). normalizeExprId,canonicalized of
org.apache.spark.sql.catalyst.plans.QueryPlan
(2). CASE_SENSITIVE of SQLConf
(3). STARSCHEMA_DETECTION of SQLConf
Solution:Don’t use normalize , canonicalize and the CASE_SENSITIVE,
STARSCHEMA_DETECTION
6,Some logicplan optimization rules that Spark 2.1 version doesn’t have
(1). SimplifyCreateMapOps
(2). SimplifyCreateArrayOps
(3). SimplifyCreateStructOps
(4). RemoveRedundantProject
(5). RemoveRedundantAliases
(6). PullupCorrelatedPredicates
(7). ReplaceDeduplicateWithAggregate
(8). EliminateView
Solution: Delete or move the code to carbon project
7,Generate the instance in SparkSQLUtil to adapt Spark 2.1 version
8,Query SQL pass the MV check in Spark 2.1 version(CarbonSessionState)
MvAdaptSpark2.pdf
<http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/file/t378/MvAdaptSpark2.pdf>
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discuss]Adapt MV datamap to spark 2.1 version
Posted by qiuchenjian <80...@qq.com>.
Hi, Akash R
If MV datamap supports spark 2.1, we must modify the mv internal code.
But mv datamap framework is independent, it's decoupled with spark 2.1
if we decide to remove 2.1 , mv will still be work with spark-2.2 or above
without modify
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/
Re: [Discuss]Adapt MV datamap to spark 2.1 version
Posted by akashrn5 <ak...@gmail.com>.
hi,
is the changes intrusive for support to 2.1 or you are going to use the
decoupling strategy?
I hope decoupling will be better as once we decide to remove 2.1 from
carbondata code, it will be easy to remove.
Thanks
--
Sent from: http://apache-carbondata-dev-mailing-list-archive.1130556.n5.nabble.com/