You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@spark.apache.org by "Burak Yavuz (JIRA)" <ji...@apache.org> on 2017/10/10 22:07:00 UTC

[jira] [Created] (SPARK-22238) EnsureStatefulOpPartitioning shouldn't ask for the child RDD before planning is completed

Burak Yavuz created SPARK-22238:
-----------------------------------

             Summary: EnsureStatefulOpPartitioning shouldn't ask for the child RDD before planning is completed
                 Key: SPARK-22238
                 URL: https://issues.apache.org/jira/browse/SPARK-22238
             Project: Spark
          Issue Type: Bug
          Components: Structured Streaming
    Affects Versions: 2.3.0
            Reporter: Burak Yavuz
            Assignee: Burak Yavuz


In EnsureStatefulOpPartitioning, we check that the inputRDD to a SparkPlan has the expected partitioning for Streaming Stateful Operators. The problem is that we are not allowed to access this information during planning.

The reason we added that check was because CoalesceExec could actually create RDDs with 0 partitions. We should fix it such that when CoalesceExec says that there is a SinglePartition, there is in fact an inputRDD of 1 partition instead of 0 partitions.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org