You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hyukjin Kwon (Jira)" <ji...@apache.org> on 2022/04/11 00:12:00 UTC
[jira] [Commented] (SPARK-38846) Teradata's Number is either converted to its floor value or ceiling value despite its fractional part.
[ https://issues.apache.org/jira/browse/SPARK-38846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17520249#comment-17520249 ]
Hyukjin Kwon commented on SPARK-38846:
--------------------------------------
Spark 2.x is EOL. Mind trying Spark 3+ out?
> Teradata's Number is either converted to its floor value or ceiling value despite its fractional part.
> ------------------------------------------------------------------------------------------------------
>
> Key: SPARK-38846
> URL: https://issues.apache.org/jira/browse/SPARK-38846
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 2.3.0
> Environment: Spark2.3.0 on Yarn
> Teradata 16.20.32.59
> Reporter: eugene
> Priority: Major
>
> I'm trying to load data from Teradata, the code using is:
>
> sparkSession.read
> .format("jdbc")
> .options(
> Map(
> "url" -> "jdbc:teradata://hostname, user=$username, password=$password",
> "MAYBENULL" -> "ON",
> "SIP_SUPPORT" -> "ON",
> "driver" -> "com.teradata.jdbc.TeraDriver",
> "dbtable" -> $table_name
> )
> )
> .load()
> However, some data lost its fractional part after loading. To be more concise, the column in Teradata is in the type of [Number][1] and after loading, the data type in Spark is `DecimalType(38,0)`, the scale value is 0 which means no digits after decimal point.
> Data in Teradata is something like,
> id column1 column2
> 1 50.23 100.23
> 2 25.8 20.669
> 3 30.2 19.23
> The `dataframe` of Spark is like,
> id column1 column2
> 1 50 100
> 2 26 21
> 3 30 19
> The meta data of the table in Teradata is like:
> CREATE SET TABLE table_name (id BIGINT, column1 NUMBER, column2 NUMBER) PRIMARY INDEX (id);
> The Spark version is 2.3.0 and Teradata is 16.20.32.59.
>
--
This message was sent by Atlassian Jira
(v8.20.1#820001)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org