You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Stephen Samuel (JIRA)" <ji...@apache.org> on 2015/11/03 10:43:27 UTC

[jira] [Created] (SPARK-11474) Options to jdbc load are lower cased

Stephen Samuel created SPARK-11474:
--------------------------------------

             Summary: Options to jdbc load are lower cased
                 Key: SPARK-11474
                 URL: https://issues.apache.org/jira/browse/SPARK-11474
             Project: Spark
          Issue Type: Bug
          Components: Input/Output
    Affects Versions: 1.5.1
         Environment: Linux & Mac
            Reporter: Stephen Samuel
            Priority: Minor


We recently upgraded from spark 1.3.0 to 1.5.1 and one of the features we wanted to take advantage of was the fetchSize added to the jdbc data frame reader.

In 1.5.1 there appears to be a bug or regression, whereby an options map has its keys lowercased. This means the existing properties from prior to 1.4 are ok, such as dbtable, url and driver, but the newer fetchSize gets converted to fetchsize.

To re-produce:

val conf = new SparkConf(true).setMaster("local").setAppName("fetchtest")
val sc = new SparkContext(conf)
val sql = new SQLContext(sc)

val options = Map("url" -> ...., "driver" -> ...., "fetchSize" -> ....)
val df = sql.load("jdbc", options)

Breakpoint at line 371 in JDBCRDD and you'll see the options are all lowercased, so:
val fetchSize = properties.getProperty("fetchSize", "0").toInt
results in 0

Now I know sql.load is deprecated, but this might be occuring on other methods too. The workaround is to use the java.util.Properties overload, which keeps the case sensitive keys.






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org