You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Sean Owen (JIRA)" <ji...@apache.org> on 2017/06/25 14:19:00 UTC
[jira] [Resolved] (SPARK-20413) New Optimizer Hint to prevent
collapsing of adjacent projections
[ https://issues.apache.org/jira/browse/SPARK-20413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sean Owen resolved SPARK-20413.
-------------------------------
Resolution: Won't Fix
> New Optimizer Hint to prevent collapsing of adjacent projections
> ----------------------------------------------------------------
>
> Key: SPARK-20413
> URL: https://issues.apache.org/jira/browse/SPARK-20413
> Project: Spark
> Issue Type: Improvement
> Components: Optimizer, PySpark, SQL
> Affects Versions: 2.1.0
> Reporter: Michael Styles
>
> I am proposing that a new optimizer hint called NO_COLLAPSE be introduced. This hint is essentially identical to Oracle's NO_MERGE hint.
> Let me first give an example of why I am proposing this.
> {noformat}
> df1 = sc.sql.createDataFrame([(1, "abc")], ["id", "user_agent"])
> df2 = df1.withColumn("ua", user_agent_details(df1["user_agent"]))
> df3 = df2.select(df2["ua"].device_form_factor.alias("c1"), df2["ua"].browser_version.alias("c2"))
> df3.explain(True)
> == Parsed Logical Plan ==
> 'Project [ua#85[device_form_factor] AS c1#90, ua#85[browser_version] AS c2#91]
> +- Project [id#80L, user_agent#81, UDF(user_agent#81) AS ua#85]
> +- LogicalRDD [id#80L, user_agent#81]
> == Analyzed Logical Plan ==
> c1: string, c2: string
> Project [ua#85.device_form_factor AS c1#90, ua#85.browser_version AS c2#91]
> +- Project [id#80L, user_agent#81, UDF(user_agent#81) AS ua#85]
> +- LogicalRDD [id#80L, user_agent#81]
> == Optimized Logical Plan ==
> Project [UDF(user_agent#81).device_form_factor AS c1#90, UDF(user_agent#81).browser_version AS c2#91]
> +- LogicalRDD [id#80L, user_agent#81]
> == Physical Plan ==
> *Project [UDF(user_agent#81).device_form_factor AS c1#90, UDF(user_agent#81).browser_version AS c2#91]
> +- Scan ExistingRDD[id#80L,user_agent#81]
> {noformat}
> user_agent_details is a user-defined function that returns a struct. As can be seen from the generated query plan, the function is being executed multiple times which could lead to performance issues. This is due to the CollapseProject optimizer rule that collapses adjacent projections.
> I'm proposing a hint that will prevent the optimizer from collapsing adjacent projections. A new function called 'no_collapse' would be introduced for this purpose. Consider the following example and generated query plan.
> {noformat}
> df1 = sc.sql.createDataFrame([(1, "abc")], ["id", "user_agent"])
> df2 = F.no_collapse(df1.withColumn("ua", user_agent_details(df1["user_agent"])))
> df3 = df2.select(df2["ua"].device_form_factor.alias("c1"), df2["ua"].browser_version.alias("c2"))
> df3.explain(True)
> == Parsed Logical Plan ==
> 'Project [ua#69[device_form_factor] AS c1#75, ua#69[browser_version] AS c2#76]
> +- NoCollapseHint
> +- Project [id#64L, user_agent#65, UDF(user_agent#65) AS ua#69]
> +- LogicalRDD [id#64L, user_agent#65]
> == Analyzed Logical Plan ==
> c1: string, c2: string
> Project [ua#69.device_form_factor AS c1#75, ua#69.browser_version AS c2#76]
> +- NoCollapseHint
> +- Project [id#64L, user_agent#65, UDF(user_agent#65) AS ua#69]
> +- LogicalRDD [id#64L, user_agent#65]
> == Optimized Logical Plan ==
> Project [ua#69.device_form_factor AS c1#75, ua#69.browser_version AS c2#76]
> +- NoCollapseHint
> +- Project [UDF(user_agent#65) AS ua#69]
> +- LogicalRDD [id#64L, user_agent#65]
> == Physical Plan ==
> *Project [ua#69.device_form_factor AS c1#75, ua#69.browser_version AS c2#76]
> +- *Project [UDF(user_agent#65) AS ua#69]
> +- Scan ExistingRDD[id#64L,user_agent#65]
> {noformat}
> As can be seen from the query plan, the user-defined function is now evaluated once per row.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org