[jira] [Created] (SPARK-38846) Teradata's Number is either converted to its floor value or ceiling value despite its fractional part.

["eugene (Jira)" <ji...@apache.org>]

eugene created SPARK-38846:
------------------------------

             Summary: Teradata's Number is either converted to its floor value or ceiling value despite its fractional part.
                 Key: SPARK-38846
                 URL: https://issues.apache.org/jira/browse/SPARK-38846
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.3.0
         Environment: Spark2.3.0 on Yarn

Teradata 16.20.32.59
            Reporter: eugene


I'm trying to load data from Teradata, the code using is:
    

    sparkSession.read
          .format("jdbc")
          .options(
            Map(
              "url" -> "jdbc:teradata://hostname, user=$username, password=$password",
              "MAYBENULL" -> "ON",
              "SIP_SUPPORT" -> "ON",
              "driver" -> "com.teradata.jdbc.TeraDriver",
              "dbtable" -> $table_name
            )
          )
          .load()

However, some data lost its fractional part after loading. To be more concise, the column in Teradata is in the type of [Number][1] and after loading, the data type in Spark is `DecimalType(38,0)`, the scale value is 0 which means no digits after decimal point. 

Data in Teradata is something like,

    id column1 column2
    1   50.23    100.23
    2   25.8     20.669
    3   30.2     19.23

The `dataframe` of Spark is like,

    id column1 column2
    1   50     100
    2   26     21
    3   30     19

The meta data of the table in Teradata is like:

    CREATE SET TABLE table_name (id BIGINT, column1 NUMBER, column2 NUMBER) PRIMARY INDEX (id);

The Spark version is 2.3.0 and Teradata is 16.20.32.59. 

 



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org