You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Apache Spark (Jira)" <ji...@apache.org> on 2022/12/16 02:59:00 UTC

[jira] [Commented] (SPARK-41539) stats and constraints in LogicalRDD may not be in sync with output attributes

    [ https://issues.apache.org/jira/browse/SPARK-41539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17648312#comment-17648312 ] 

Apache Spark commented on SPARK-41539:
--------------------------------------

User 'HeartSaVioR' has created a pull request for this issue:
https://github.com/apache/spark/pull/39082

> stats and constraints in LogicalRDD may not be in sync with output attributes
> -----------------------------------------------------------------------------
>
>                 Key: SPARK-41539
>                 URL: https://issues.apache.org/jira/browse/SPARK-41539
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 3.4.0
>            Reporter: Jungtaek Lim
>            Priority: Major
>
> We encountered the case where the output of logical plan and optimized plan were different in LogicalRDD (the difference was exprId for the case), led the situation that stats and constraints are out of sync with output attributes, eventually failed the query.
> We should remap stats and constraints based on the output of logical plan, assuming that the output of logical plan and optimized plan are "slightly" different (e.g. exprId) but "semantically" same.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org