[jira] [Created] (SPARK-30100) Decimal Precision Inferred from JDBC via Spark

["Joby Joje (Jira)" <ji...@apache.org>]

Joby Joje created SPARK-30100:
---------------------------------

             Summary: Decimal Precision Inferred from JDBC via Spark
                 Key: SPARK-30100
                 URL: https://issues.apache.org/jira/browse/SPARK-30100
             Project: Spark
          Issue Type: Bug
          Components: PySpark
    Affects Versions: 2.4.0
            Reporter: Joby Joje


When trying to load data from JDBC(Oracle) into Spark, there seems to be precision loss in the decimal field, as per my understanding Spark supports *DECIMAL(38,18)*. The field from the Oracle is DECIMAL(38,14), whereas Spark rounds off the last four digits making it a precision of DECIMAL(38,10). This is happening to few fields in the dataframe where the column is fetched using a CASE statement whereas in the same query another field populates the right schema.

Tried to pass the {{}}
{code:java}
spark.sql.decimalOperations.allowPrecisionLoss=false{code}
{{}} conf in the Spark-submit though didn't get the desired results.

{{}}
{code:java}
jdbcDF = spark.read \ .format("jdbc") \ .option("url", "ORACLE") \ .option("dbtable", "QUERY") \ .option("user", "USERNAME") \ .option("password", "PASSWORD") \ .load(){code}
{{}}

So considering that the Spark infers the schema from a sample records, how does this work here? Does it use the results of the query i.e (SELECT * FROM TABLE_NAME JOIN ...) or does it take a different route to guess the schema for itself? Can someone throw some light on this and advise how to achieve the right decimal precision on this regards without manipulating the query as doing a CAST on the query does solve the issue, but would prefer to get some alternatives.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org