[jira] [Resolved] (SPARK-12048) JDBCRDD calls close() twice - SQLite then throws an exception

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen resolved SPARK-12048.
-------------------------------
       Resolution: Fixed
    Fix Version/s: 1.6.1
                   1.5.3
                   2.0.0

Issue resolved by pull request 10101
[https://github.com/apache/spark/pull/10101]

> JDBCRDD calls close() twice - SQLite then throws an exception
> -------------------------------------------------------------
>
>                 Key: SPARK-12048
>                 URL: https://issues.apache.org/jira/browse/SPARK-12048
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.1
>            Reporter: R. H.
>            Priority: Minor
>              Labels: starter
>             Fix For: 2.0.0, 1.5.3, 1.6.1
>
>
> The following code works:
> {quote}
>   val tableData = sqlContext.read.format("jdbc")
>     .options(
>       Map(
>         "url" -> "jdbc:sqlite:/tmp/test.db",
>         "dbtable" -> "testtable")).load()
> {quote}
> but an exception gets reported. From the log:
> {quote}
> 15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
> 15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
> {quote}
> So Spark succeeded to close the JDBC connection, and then it fails to close the JDBC statement.
> Looking at the source, close() seems to be called twice: 
> {quote}
> context.addTaskCompletionListener\{ context => close() \}
> {quote}
> and in
> {quote}
> def hasNext
> {quote}
> If you look at the close() method (around line 443)
> {quote}
>   def close() {
>     if (closed) return
> {quote}
> you can see that it checks the variable closed, but that value is never set to true.
> So a trivial fix should be to set "closed = true" at the end of close().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org