[GitHub] [spark] HyukjinKwon commented on a change in pull request #24677: [SPARK-27805][PYTHON] Propagate SparkExceptions during toPandas with arrow enabled

[GitBox <gi...@apache.org>]

HyukjinKwon commented on a change in pull request #24677: [SPARK-27805][PYTHON] Propagate SparkExceptions during toPandas with arrow enabled
URL: https://github.com/apache/spark/pull/24677#discussion_r291858273
 
 

 ##########
 File path: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala
 ##########
 @@ -3313,20 +3313,34 @@ class Dataset[T] private[sql](
             }
           }
 
-        val arrowBatchRdd = toArrowBatchRdd(plan)
-        sparkSession.sparkContext.runJob(
-          arrowBatchRdd,
-          (it: Iterator[Array[Byte]]) => it.toArray,
-          handlePartitionBatches)
+        var sparkException: Option[SparkException] = None
+        try {
+          val arrowBatchRdd = toArrowBatchRdd(plan)
+          sparkSession.sparkContext.runJob(
+            arrowBatchRdd,
+            (it: Iterator[Array[Byte]]) => it.toArray,
+            handlePartitionBatches)
+        } catch {
+          case e: SparkException =>
+            sparkException = Some(e)
+        }
 
-        // After processing all partitions, end the stream and write batch order indices
+        // After processing all partitions, end the batch stream
         batchWriter.end()
-        out.writeInt(batchOrder.length)
-        // Sort by (index of partition, batch index in that partition) tuple to get the
-        // overall_batch_index from 0 to N-1 batches, which can be used to put the
-        // transferred batches in the correct order
-        batchOrder.zipWithIndex.sortBy(_._1).foreach { case (_, overallBatchIndex) =>
-          out.writeInt(overallBatchIndex)
+        sparkException match {
+          case Some(exception) =>
+            // Signal failure and write error message
+            out.writeInt(-1)
+            PythonRDD.writeUTF(exception.getMessage, out)
 
 Review comment:
   Sorry for late response, @BryanCutler 
   
   Yea, I had the same question https://github.com/apache/spark/pull/24677#issuecomment-494879041. Thanks for details at https://github.com/apache/spark/pull/24677#issuecomment-494884105. It would have been better if those are commented since at least two committers raised the same questions :-).
   
   > @HyukjinKwon do you think would it make sense to patch branch-2.4 with a manual fix?
   
   Yea, I don't mind backporting it (don't strongly feel we should do too).

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org