You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by ir...@apache.org on 2017/05/31 15:53:53 UTC
spark git commit: [SPARK-20633][SQL] FileFormatWriter should not wrap
FetchFailedException
Repository: spark
Updated Branches:
refs/heads/master ac7fc3075 -> d0f36bcb1
[SPARK-20633][SQL] FileFormatWriter should not wrap FetchFailedException
## What changes were proposed in this pull request?
Explicitly handle the FetchFailedException in FileFormatWriter, so it does not get wrapped.
Note that this is no longer strictly necessary after SPARK-19276, but it improves error messages and also will help avoid others stumbling across this in the future.
## How was this patch tested?
Existing unit tests.
Closes https://github.com/apache/spark/pull/17893
Author: Liu Shaohui <li...@xiaomi.com>
Closes #18145 from squito/SPARK-20633.
Project: http://git-wip-us.apache.org/repos/asf/spark/repo
Commit: http://git-wip-us.apache.org/repos/asf/spark/commit/d0f36bcb
Tree: http://git-wip-us.apache.org/repos/asf/spark/tree/d0f36bcb
Diff: http://git-wip-us.apache.org/repos/asf/spark/diff/d0f36bcb
Branch: refs/heads/master
Commit: d0f36bcb10c3f424e87a6a38def0c0a3b60c03d1
Parents: ac7fc30
Author: Liu Shaohui <li...@xiaomi.com>
Authored: Wed May 31 10:53:31 2017 -0500
Committer: Imran Rashid <ir...@cloudera.com>
Committed: Wed May 31 10:53:31 2017 -0500
----------------------------------------------------------------------
.../spark/sql/execution/datasources/FileFormatWriter.scala | 5 ++++-
1 file changed, 4 insertions(+), 1 deletion(-)
----------------------------------------------------------------------
http://git-wip-us.apache.org/repos/asf/spark/blob/d0f36bcb/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
----------------------------------------------------------------------
diff --git a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
index afe454f..0daffa9 100644
--- a/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
+++ b/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/FileFormatWriter.scala
@@ -31,6 +31,7 @@ import org.apache.spark._
import org.apache.spark.internal.Logging
import org.apache.spark.internal.io.{FileCommitProtocol, SparkHadoopWriterUtils}
import org.apache.spark.internal.io.FileCommitProtocol.TaskCommitMessage
+import org.apache.spark.shuffle.FetchFailedException
import org.apache.spark.sql.SparkSession
import org.apache.spark.sql.catalyst.catalog.{BucketSpec, ExternalCatalogUtils}
import org.apache.spark.sql.catalyst.catalog.CatalogTypes.TablePartitionSpec
@@ -259,8 +260,10 @@ object FileFormatWriter extends Logging {
}
})
} catch {
+ case e: FetchFailedException =>
+ throw e
case t: Throwable =>
- throw new SparkException("Task failed while writing rows", t)
+ throw new SparkException("Task failed while writing rows.", t)
}
}
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org