You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2019/10/11 01:51:00 UTC
[jira] [Resolved] (SPARK-29284) df.distinct.count throw
NoSuchElementException when enabled daptive executor
[ https://issues.apache.org/jira/browse/SPARK-29284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hyukjin Kwon resolved SPARK-29284.
----------------------------------
Resolution: Cannot Reproduce
> df.distinct.count throw NoSuchElementException when enabled daptive executor
> -----------------------------------------------------------------------------
>
> Key: SPARK-29284
> URL: https://issues.apache.org/jira/browse/SPARK-29284
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.4.4
> Reporter: yiming.xu
> Priority: Minor
>
> This case:
> spark.sql("SET spark.sql.adaptive.enabled=true")
> spark.sql("SET spark.sql.shuffle.partitions=1")
> val result = spark.range(0).distinct().count
> or spark.table("empty table").distinct.count
> throw java.util.NoSuchElementException: next on empty iterator
> If current stage partition is 0,org.apache.spark.sql.execution.exchange.ExchangeCoordinator#doEstimationIfNecessary->partitionStartIndices is empty (https://issues.apache.org/jira/browse/SPARK-22144)
> next stage rdd partition is empty.
> To solve the problem I will change partitionStartIndices to Array(0) when the parent rdd partition is 0;
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org