[jira] [Updated] (SPARK-12048) JDBCRDD calls close() twice - SQLite then throws an exception

["Sean Owen (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/SPARK-12048?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Sean Owen updated SPARK-12048:
------------------------------
    Assignee: R. H.

> JDBCRDD calls close() twice - SQLite then throws an exception
> -------------------------------------------------------------
>
>                 Key: SPARK-12048
>                 URL: https://issues.apache.org/jira/browse/SPARK-12048
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.1
>            Reporter: R. H.
>            Assignee: R. H.
>            Priority: Minor
>              Labels: starter
>             Fix For: 1.5.3, 1.6.1, 2.0.0
>
>
> The following code works:
> {quote}
>   val tableData = sqlContext.read.format("jdbc")
>     .options(
>       Map(
>         "url" -> "jdbc:sqlite:/tmp/test.db",
>         "dbtable" -> "testtable")).load()
> {quote}
> but an exception gets reported. From the log:
> {quote}
> 15/11/30 12:13:02 INFO jdbc.JDBCRDD: closed connection
> 15/11/30 12:13:02 WARN jdbc.JDBCRDD: Exception closing statement java.sql.SQLException: [SQLITE_ERROR] SQL error or missing database (Connection is closed) at org.sqlite.core.DB.newSQLException(DB.java:890) at org.sqlite.core.CoreStatement.internalClose(CoreStatement.java:109) at org.sqlite.jdbc3.JDBC3Statement.close(JDBC3Statement.java:35) at org.apache.spark.sql.execution.datasources.jdbc.JDBCRDD$$anon$1.org$apache$spark$sql$execution$datasources$jdbc$JDBCRDD$$anon$$close(JDBCRDD.scala:454)
> {quote}
> So Spark succeeded to close the JDBC connection, and then it fails to close the JDBC statement.
> Looking at the source, close() seems to be called twice: 
> {quote}
> context.addTaskCompletionListener\{ context => close() \}
> {quote}
> and in
> {quote}
> def hasNext
> {quote}
> If you look at the close() method (around line 443)
> {quote}
>   def close() {
>     if (closed) return
> {quote}
> you can see that it checks the variable closed, but that value is never set to true.
> So a trivial fix should be to set "closed = true" at the end of close().



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org