[jira] [Created] (SPARK-17230) Writing decimal to csv will result empty string if the decimal exceeds (20, 18)

["Davies Liu (JIRA)" <ji...@apache.org>]

Davies Liu created SPARK-17230:
----------------------------------

             Summary: Writing decimal to csv will result empty string if the decimal exceeds (20, 18)
                 Key: SPARK-17230
                 URL: https://issues.apache.org/jira/browse/SPARK-17230
             Project: Spark
          Issue Type: Bug
          Components: SQL
    Affects Versions: 2.0.0, 1.6.2
            Reporter: Davies Liu
            Assignee: Davies Liu


{code}
// file content 
spark.read.csv("/mnt/djiang/test-case.csv").show 
// read in as string and create temp view 
spark.read.csv("/mnt/djiang/test-case.csv").createOrReplaceTempView("test") 
// confirm schema 
spark.table("test").printSchema 
// apply decimal calculation, confirm the result is correct 
spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) from test").show(false) 
// run the same query, and write out as csv 
spark.sql("select _c0, cast(_c0 as long) * cast('1.0' as decimal(38, 18)) from test").write.csv("/mnt/djiang/test-case-result") 
// show the content of the result file, particularly, for number exceeded decimal(20, 18), the csv is not writing anything or failing silently 
spark.read.csv("/mnt/djiang/test-case-result").show

+------+ 
| _c0| 
+------+ 
| 1| 
| 10| 
| 100| 
| 1000| 
| 10000| 
|100000| 
+------+

root 
|-- _c0: string (nullable = true)

+------+-------------------------------------------------------------------------------------------------------------------------------------------------+ 
|_c0 |(CAST(CAST(CAST(CAST(_c0 AS DECIMAL(20,0)) AS BIGINT) AS DECIMAL(20,0)) AS DECIMAL(38,18)) * CAST(CAST(1.0 AS DECIMAL(38,18)) AS DECIMAL(38,18)))| 
+------+-------------------------------------------------------------------------------------------------------------------------------------------------+ 
|1 |1.000000000000000000 | 
|10 |10.000000000000000000 | 
|100 |100.000000000000000000 | 
|1000 |1000.000000000000000000 | 
|10000 |10000.000000000000000000 |
|100000|100000.000000000000000000 | 
+------+-------------------------------------------------------------------------------------------------------------------------------------------------+

+------+--------------------+ 
| _c0| _c1| 
+------+--------------------+ 
| 1|1.000000000000000000| 
| 10|10.00000000000000...| 
| 100| | 
| 1000| | 
| 10000| | 
|100000| | 
+------+--------------------+
{code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org