You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marius Van Niekerk (JIRA)" <ji...@apache.org> on 2015/11/20 15:29:11 UTC

[jira] [Created] (SPARK-11881) [SQL] JDBC Postgres fetchsize parameter ignored

Marius Van Niekerk created SPARK-11881:
------------------------------------------

             Summary: [SQL] JDBC Postgres fetchsize parameter ignored 
                 Key: SPARK-11881
                 URL: https://issues.apache.org/jira/browse/SPARK-11881
             Project: Spark
          Issue Type: Improvement
          Components: SQL
    Affects Versions: 1.6.0
            Reporter: Marius Van Niekerk


Reference: https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor

In order for PostgreSQL to honor the fetchSize non-zero setting, its Connection.autoCommit needs to be set to false. Otherwise, it will just quietly ignore the fetchSize setting. 

Since the JDBC driver doesn’t serialize the result data in its memory, we don’t want to have 2-3 times of the memory created in the driver first before Spark can fetch anything into its serialized data storage. 




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org