[jira] [Assigned] (SPARK-18658) Writing to a text DataSource buffers one or more lines in memory

["Apache Spark (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-18658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Apache Spark reassigned SPARK-18658:
------------------------------------

    Assignee: Apache Spark

> Writing to a text DataSource buffers one or more lines in memory
> ----------------------------------------------------------------
>
>                 Key: SPARK-18658
>                 URL: https://issues.apache.org/jira/browse/SPARK-18658
>             Project: Spark
>          Issue Type: Improvement
>          Components: SQL
>    Affects Versions: 2.0.2
>            Reporter: Nathan Howell
>            Assignee: Apache Spark
>            Priority: Minor
>
> The JSON and CSV writing paths buffer entire lines (or multiple lines) in memory prior to writing to disk. For large rows this is inefficient. It may make sense to skip the {{TextOutputFormat}} record writer and go directly to the underlying {{FSDataOutputStream}}, allowing the writers to append arbitrary byte arrays (fractions of a row) instead of a full row.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org