[jira] [Commented] (SPARK-21691) Accessing canonicalized plan for query with limit throws exception

["Anton Okolnychyi (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-21691?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16124532#comment-16124532 ] 

Anton Okolnychyi commented on SPARK-21691:
------------------------------------------

The issue is related to `Project \[\*\]` but not to `Limit`. You will always have this exception if you have a non-top level `Project \[\*\]` in your logical plan. For instance, the following query will also produce the same exception:

{noformat}
spark.sql("select * from (select * from (values 0, 1)) as v").queryExecution.logical.canonicalized
{noformat}

In the failing example from the ticket description, the non-canonicalized logical plan looks like:

{noformat}
'GlobalLimit 1
+- 'LocalLimit 1
   +- 'Project [*]
      +- 'SubqueryAlias __auto_generated_subquery_name
         +- 'UnresolvedInlineTable [col1], [List(0), List(1)]
{noformat}

Once Spark tries to canonicalize it and processes `LocalLimit 1`, it will get all attributes by calling `children.flatMap(_.output)`, which triggers the problem. `Project#output` will try to convert its project list to attributes, which will fail for `UnresolvedStar` with the aforementioned exception.

I see that `UnresolvedRelation` and `UnresolvedInlineTable` return Nil as output. Therefore, one option to fix this problem is to return `Nil` as output from `Project` if it is unresolved.

{noformat}
override def output: Seq[Attribute] = if (resolved) projectList.map(_.toAttribute) else Nil
{noformat}

I can fix it once we agree on a solution.


> Accessing canonicalized plan for query with limit throws exception
> ------------------------------------------------------------------
>
>                 Key: SPARK-21691
>                 URL: https://issues.apache.org/jira/browse/SPARK-21691
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 2.2.0
>            Reporter: Bjoern Toldbod
>
> Accessing the logical, canonicalized plan fails for queries with limits.
> The following demonstrates the issue:
> {code:java}
> val session = SparkSession.builder.master("local").getOrCreate()
> // This works
> session.sql("select * from (values 0, 1)").queryExecution.logical.canonicalized
> // This fails
> session.sql("select * from (values 0, 1) limit 1").queryExecution.logical.canonicalized
> {code}
> The message in the thrown exception is somewhat confusing (or at least not directly related to the limit):
> "Invalid call to toAttribute on unresolved object, tree: *"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org