You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Kousuke Saruta (JIRA)" <ji...@apache.org> on 2019/07/27 18:09:00 UTC
[jira] [Updated] (SPARK-28520) WholeStageCodegen does not work
property for LocalTableScanExec
[ https://issues.apache.org/jira/browse/SPARK-28520?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kousuke Saruta updated SPARK-28520:
-----------------------------------
Issue Type: Bug (was: Improvement)
> WholeStageCodegen does not work property for LocalTableScanExec
> ---------------------------------------------------------------
>
> Key: SPARK-28520
> URL: https://issues.apache.org/jira/browse/SPARK-28520
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 3.0.0
> Reporter: Kousuke Saruta
> Assignee: Kousuke Saruta
> Priority: Minor
>
> Code is not generated for LocalTableScanExec although proper situations.
> If a LocalTableScanExec plan has the direct parent plan which supports WholeStageCodegen,
> the LocalTableScanExec plan also should be within a WholeStageCodegen domain.
> But code is not generated for LocalTableScanExec and InputAdapter is inserted for now.
> {code}
> val df1 = spark.createDataset(1 to 10).toDF
> val df2 = spark.createDataset(1 to 10).toDF
> val df3 = df1.join(df2, df1("value") === df2("value"))
> df3.explain(true)
> ...
> == Physical Plan ==
> *(1) BroadcastHashJoin [value#1], [value#6], Inner, BuildRight
> :- LocalTableScan [value#1] // LocalTableScanExec is not within a WholeStageCodegen domain
> +- BroadcastExchange HashedRelationBroadcastMode(List(cast(input[0, int, false] as bigint)))
> +- LocalTableScan [value#6]
> {code}
> {code}
> scala> df3.queryExecution.executedPlan.children.head.children.head.getClass
> res4: Class[_ <: org.apache.spark.sql.execution.SparkPlan] = class org.apache.spark.sql.execution.InputAdapter
> {code}
> For the current implementation of LocalTableScanExec, codegen is enabled in case `parent` is not null
> but `parent` is set in `consume`, which is called after `insertInputAdapter` so it doesn't work as intended.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org