You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Marius Van Niekerk (JIRA)" <ji...@apache.org> on 2015/11/20 15:29:11 UTC
[jira] [Created] (SPARK-11881) [SQL] JDBC Postgres fetchsize
parameter ignored
Marius Van Niekerk created SPARK-11881:
------------------------------------------
Summary: [SQL] JDBC Postgres fetchsize parameter ignored
Key: SPARK-11881
URL: https://issues.apache.org/jira/browse/SPARK-11881
Project: Spark
Issue Type: Improvement
Components: SQL
Affects Versions: 1.6.0
Reporter: Marius Van Niekerk
Reference: https://jdbc.postgresql.org/documentation/head/query.html#query-with-cursor
In order for PostgreSQL to honor the fetchSize non-zero setting, its Connection.autoCommit needs to be set to false. Otherwise, it will just quietly ignore the fetchSize setting.
Since the JDBC driver doesn’t serialize the result data in its memory, we don’t want to have 2-3 times of the memory created in the driver first before Spark can fetch anything into its serialized data storage.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org