You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Apache Spark (JIRA)" <ji...@apache.org> on 2016/08/24 21:21:21 UTC
[jira] [Assigned] (SPARK-17230) Writing decimal to csv will result
empty string if the decimal exceeds (20, 18)
[ https://issues.apache.org/jira/browse/SPARK-17230?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Apache Spark reassigned SPARK-17230:
------------------------------------
Assignee: Apache Spark (was: Davies Liu)
> Writing decimal to csv will result empty string if the decimal exceeds (20, 18)
> -------------------------------------------------------------------------------
>
> Key: SPARK-17230
> URL: https://issues.apache.org/jira/browse/SPARK-17230
> Project: Spark
> Issue Type: Bug
> Components: SQL
> Affects Versions: 1.6.2, 2.0.0
> Reporter: Davies Liu
> Assignee: Apache Spark
>
> {code}
> // file content
> spark.read.csv("/mnt/djiang/test-case.csv").show
> // read in as string and create temp view
> spark.read.csv("/mnt/djiang/test-case.csv").createOrReplaceTempView("test")
> // confirm schema
> spark.table("test").printSchema
> // apply decimal calculation, confirm the result is correct
> spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) from test").show(false)
> // run the same query, and write out as csv
> spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) from test").write.csv("/mnt/djiang/test-case-result")
> // show the content of the result file, particularly, for number exceeded decimal(20, 18), the csv is not writing anything or failing silently
> spark.read.csv("/mnt/djiang/test-case-result").show
> +------+
> | _c0|
> +------+
> | 1|
> | 10|
> | 100|
> | 1000|
> | 10000|
> |100000|
> +------+
> root
> |-- _c0: string (nullable = true)
> +------+-------------------------------------------------------------------------------------------------------------------------------------------------+
> |_c0 |(CAST(CAST(CAST(CAST(_c0 AS DECIMAL(20,0)) AS BIGINT) AS DECIMAL(20,0)) AS DECIMAL(38,18)) * CAST(CAST(1.0 AS DECIMAL(38,18)) AS DECIMAL(38,18)))|
> +------+-------------------------------------------------------------------------------------------------------------------------------------------------+
> |1 |1.000000000000000000 |
> |10 |10.000000000000000000 |
> |100 |100.000000000000000000 |
> |1000 |1000.000000000000000000 |
> |10000 |10000.000000000000000000 |
> |100000|100000.000000000000000000 |
> +------+-------------------------------------------------------------------------------------------------------------------------------------------------+
> +------+--------------------+
> | _c0| _c1|
> +------+--------------------+
> | 1|1.000000000000000000|
> | 10|10.00000000000000...|
> | 100| |
> | 1000| |
> | 10000| |
> |100000| |
> +------+--------------------+
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org