You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Yuchen Liu (Jira)" <ji...@apache.org> on 2025/03/02 05:38:00 UTC

[jira] [Created] (SPARK-51362) change toJSON to use NextIterator API to reduce latency

Yuchen Liu created SPARK-51362:
----------------------------------

             Summary: change toJSON to use NextIterator API to reduce latency
                 Key: SPARK-51362
                 URL: https://issues.apache.org/jira/browse/SPARK-51362
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 4.0.0
            Reporter: Yuchen Liu


The current toJSON operation uses the Iterator API where iter.hasNext is called after iter.next, which means the return of current row depends on the next row to arrive. If we change it to use the NextIterator API, iter.next will be called after iter.hasNext, so the current row will return immediately. This eliminates the dependency between adjacent rows, which reduces record-level latency.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org