[GitHub] spark pull request #21546: [SPARK-23030][SQL][PYTHON] Use Arrow stream forma...

[HyukjinKwon <gi...@git.apache.org>]

Github user HyukjinKwon commented on a diff in the pull request:

    https://github.com/apache/spark/pull/21546#discussion_r203760078
  
    --- Diff: sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala ---
    @@ -3349,20 +3385,20 @@ class Dataset[T] private[sql](
         }
       }
     
    -  /** Convert to an RDD of ArrowPayload byte arrays */
    -  private[sql] def toArrowPayload(plan: SparkPlan): RDD[ArrowPayload] = {
    +  /** Convert to an RDD of serialized ArrowRecordBatches. */
    +  private[sql] def getArrowBatchRdd(plan: SparkPlan): RDD[Array[Byte]] = {
    --- End diff --
    
    @BryanCutler, not a big deal at all but how about `to` like the before? This reminds me of QueryExecution's toRdd and the previous one looks slightly better.


---

---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org