[ https://issues.apache.org/jira/browse/SPARK-38846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-38846:
------------------------------------
Assignee: (was: Apache Spark)
> Teradata's Number is either converted to its floor value or ceiling value despite its fractional part.
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-38846
> URL: https://issues.apache.org/jira/browse/SPARK-38846
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0, 3.2.1
> Environment: Spark2.3.0/Spark3.2.1 on Yarn
> Teradata 16.20.32.59
> Reporter: eugene
> Priority: Major
>
> I'm trying to load data from Teradata, the code using is:
>
> sparkSession.read
> .format("jdbc")
> .options(
> Map(
> "url" -> "jdbc:teradata://hostname, user=$username, password=$password",
> "MAYBENULL" -> "ON",
> "SIP_SUPPORT" -> "ON",
> "driver" -> "com.teradata.jdbc.TeraDriver",
> "dbtable" -> $table_name
> )
> )
> .load()
> However, some data lost its fractional part after loading. To be more concise, the column in Teradata is in the type of [Number][1] and after loading, the data type in Spark is `DecimalType(38,0)`, the scale value is 0 which means no digits after decimal point.
> Data in Teradata is something like,
> id column1 column2
> 1 50.23 100.23
> 2 25.8 20.669
> 3 30.2 19.23
> The `dataframe` of Spark is like,
> id column1 column2
> 1 50 100
> 2 26 21
> 3 30 19
> The meta data of the table in Teradata is like:
> CREATE SET TABLE table_name (id BIGINT, column1 NUMBER, column2 NUMBER) PRIMARY INDEX (id);
> The Spark version is 2.3.0/3.2.1 and Teradata is 16.20.32.59.
>
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org