You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Xiao Li (JIRA)" <ji...@apache.org> on 2016/05/31 23:59:13 UTC

[jira] [Updated] (SPARK-11474) Options to jdbc load are lower cased

     [ https://issues.apache.org/jira/browse/SPARK-11474?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Xiao Li updated SPARK-11474:
----------------------------
    Component/s:     (was: Input/Output)
                 SQL

> Options to jdbc load are lower cased
> ------------------------------------
>
>                 Key: SPARK-11474
>                 URL: https://issues.apache.org/jira/browse/SPARK-11474
>             Project: Spark
>          Issue Type: Bug
>          Components: SQL
>    Affects Versions: 1.5.1
>         Environment: Linux & Mac
>            Reporter: Stephen Samuel
>            Assignee: Huaxin Gao
>            Priority: Minor
>             Fix For: 1.5.3, 1.6.0
>
>
> We recently upgraded from spark 1.3.0 to 1.5.1 and one of the features we wanted to take advantage of was the fetchSize added to the jdbc data frame reader.
> In 1.5.1 there appears to be a bug or regression, whereby an options map has its keys lowercased. This means the existing properties from prior to 1.4 are ok, such as dbtable, url and driver, but the newer fetchSize gets converted to fetchsize.
> To re-produce:
> val conf = new SparkConf(true).setMaster("local").setAppName("fetchtest")
> val sc = new SparkContext(conf)
> val sql = new SQLContext(sc)
> val options = Map("url" -> ...., "driver" -> ...., "fetchSize" -> ....)
> val df = sql.load("jdbc", options)
> Breakpoint at line 371 in JDBCRDD and you'll see the options are all lowercased, so:
> val fetchSize = properties.getProperty("fetchSize", "0").toInt
> results in 0
> Now I know sql.load is deprecated, but this might be occuring on other methods too. The workaround is to use the java.util.Properties overload, which keeps the case sensitive keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org