You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Todd <bi...@163.com> on 2015/11/16 15:35:01 UTC
How 'select name,age from TBL_STUDENT where age = 37' is optimized
when caching it
Hi,
When I cache the dataframe and run the query,
val df = sqlContext.sql("select name,age from TBL_STUDENT where age = 37")
df.cache()
df.show
println(df.queryExecution)
I got the following execution plan,from the optimized logical plan,I can see the whole analyzed logical plan is totally replaced with the InMemoryRelation logical plan. But when I look into the Optimizer, I didn't see any optimizer that relates to the InMemoryRelation.
Could you please explain how the optimization works?
== Parsed Logical Plan ==
'<Project>, argString:< [unresolvedalias(UnresolvedAttribute: 'name),unresolvedalias(UnresolvedAttribute: 'age)]>
'<Filter>, argString:< (UnresolvedAttribute: 'age = 37)>
'<UnresolvedRelation>, argString:< [TBL_STUDENT], None>
== Analyzed Logical Plan ==
name: string, age: int
<Project>, argString:< [AttributeReference:name#1,AttributeReference:age#3]>
<Filter>, argString:< (AttributeReference:age#3 = 37)>
<Subquery>, argString:< TBL_STUDENT>
<LogicalRDD>, argString:< [AttributeReference:id#0,AttributeReference:name#1,AttributeReference:classId#2,AttributeReference:age#3], MapPartitionsRDD[4] at main at NativeMethodAccessorImpl.java:-2>
== Optimized Logical Plan ==
<InMemoryRelation>, argString:< [AttributeReference:name#1,AttributeReference:age#3], true, 10000, StorageLevel(true, true, false, true, 1), (<TungstenProject>, argString:< [AttributeReference:name#1,AttributeReference:age#3]>), None>
== Physical Plan ==
<InMemoryColumnarTableScan>, argString:< [AttributeReference:name#1,AttributeReference:age#3], (<InMemoryRelation>, argString:< [AttributeReference:name#1,AttributeReference:age#3], true, 10000, StorageLevel(true, true, false, true, 1), (<TungstenProject>, argString:< [AttributeReference:name#1,AttributeReference:age#3]>), None>)>
Re: How 'select name,age from TBL_STUDENT where age = 37' is
optimized when caching it
Posted by Xiao Li <ga...@gmail.com>.
Your dataframe is cached. Thus, your plan is stored as an InMemoryRelation.
You can read the logics in CacheManager.scala.
Good luck,
Xiao Li
2015-11-16 6:35 GMT-08:00 Todd <bi...@163.com>:
> Hi,
>
> When I cache the dataframe and run the query,
>
> val df = sqlContext.sql("select name,age from TBL_STUDENT where age =
> 37")
> df.cache()
> df.show
> println(df.queryExecution)
>
> I got the following execution plan,from the optimized logical plan,I can
> see the whole analyzed logical plan is totally replaced with the
> InMemoryRelation logical plan. But when I look into the Optimizer, I didn't
> see any optimizer that relates to the InMemoryRelation.
>
> Could you please explain how the optimization works?
>
>
>
>
> == Parsed Logical Plan ==
> '<Project>, argString:< [unresolvedalias(UnresolvedAttribute:
> 'name),unresolvedalias(UnresolvedAttribute: 'age)]>
> '<Filter>, argString:< (UnresolvedAttribute: 'age = 37)>
> '<UnresolvedRelation>, argString:< [TBL_STUDENT], None>
>
> == Analyzed Logical Plan ==
> name: string, age: int
> <Project>, argString:<
> [AttributeReference:name#1,AttributeReference:age#3]>
> <Filter>, argString:< (AttributeReference:age#3 = 37)>
> <Subquery>, argString:< TBL_STUDENT>
> <LogicalRDD>, argString:<
> [AttributeReference:id#0,AttributeReference:name#1,AttributeReference:classId#2,AttributeReference:age#3],
> MapPartitionsRDD[4] at main at NativeMethodAccessorImpl.java:-2>
>
> == Optimized Logical Plan ==
> <InMemoryRelation>, argString:<
> [AttributeReference:name#1,AttributeReference:age#3], true, 10000,
> StorageLevel(true, true, false, true, 1), (<TungstenProject>, argString:<
> [AttributeReference:name#1,AttributeReference:age#3]>), None>
>
> == Physical Plan ==
> <InMemoryColumnarTableScan>, argString:<
> [AttributeReference:name#1,AttributeReference:age#3], (<InMemoryRelation>,
> argString:< [AttributeReference:name#1,AttributeReference:age#3], true,
> 10000, StorageLevel(true, true, false, true, 1), (<TungstenProject>,
> argString:< [AttributeReference:name#1,AttributeReference:age#3]>), None>)>
>
>
>
>
>
>
>
>
>
>
>