[jira] [Commented] (SPARK-17351) Refactor JDBCRDD to expose JDBC -> SparkSQL conversion functionality

["Rafael (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/SPARK-17351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17107338#comment-17107338 ] 

Rafael commented on SPARK-17351:
--------------------------------

Hey guys, 
I know that it is a very old ticket but I encountered an issue related to these changes so let me ask my question it here.

Now the code expects the for Decimal type we need to have in JDBC metadata precision and scale. 

[https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/jdbc/JdbcUtils.scala#L402-L414]

 

I found out that in the OracleDB it is valid to have Decimal without these data. When I do a query read metadata for such column I'm getting DATA_PRECISION = Null, and DATA_SCALE = Null.

Then when I run the `spark-sql` I'm getting such error:
{code:java}
java.lang.IllegalArgumentException: requirement failed: Decimal precision 45 exceeds max precision 38
        at scala.Predef$.require(Predef.scala:224)
        at org.apache.spark.sql.types.Decimal.set(Decimal.scala:114)
        at org.apache.spark.sql.types.Decimal$.apply(Decimal.scala:465)
        at org.apache.spark.sql.execution.datasources.jdbc.JdbcUtils$$anonfun$org$apache$spark$sql$execution$datasources$jdbc$JdbcUtils$$makeGetter$3$$anonfun$12.apply(JdbcUtils.scala:407)
{code}
Do you have a work around how spark-sql can work with such cases?

> Refactor JDBCRDD to expose JDBC -> SparkSQL conversion functionality
> --------------------------------------------------------------------
>
>                 Key: SPARK-17351
>                 URL: https://issues.apache.org/jira/browse/SPARK-17351
>             Project: Spark
>          Issue Type: New Feature
>          Components: SQL
>            Reporter: Josh Rosen
>            Assignee: Josh Rosen
>            Priority: Major
>             Fix For: 2.1.0
>
>
> It would be useful if more of JDBCRDD's JDBC -> Spark SQL functionality was usable from outside of JDBCRDD; this would make it easier to write test harnesses comparing Spark output against other JDBC databases. 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org