You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Christian Kurz (JIRA)" <ji...@apache.org> on 2015/11/25 15:20:11 UTC

[jira] [Created] (SPARK-11989) Spark JDBC write only works on techologies with transaction support

Christian Kurz created SPARK-11989:
--------------------------------------

             Summary: Spark JDBC write only works on techologies with transaction support
                 Key: SPARK-11989
                 URL: https://issues.apache.org/jira/browse/SPARK-11989
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 1.5.2
            Reporter: Christian Kurz


Writing DataFrames out to a JDBC destination currently requires the JDBC driver/ database to support transaction. This is because spark/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala *always* calls commit()/rollback().

Some technologies do not support transactions and their drivers will throw an exception if commit()/rollback() is used. For these technologies (like Progress JDBC Driver for Cassandra) this is a blocking problem.

Prior to using transaction support JdbcUtils.scala needs to check whether the drivers does support transaction. Check can be done via 

conn.getMetaData().supportsDataManipulationTransactionsOnly() ||
conn.getMetaData().supportsDataDefinitionAndDataManipulationTransactions()

The working code change can be seen [here|https://github.com/CK50/spark/blob/branch-1.6_non-transactional/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org