You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Damian Momot (JIRA)" <ji...@apache.org> on 2016/11/17 07:54:58 UTC
[jira] [Created] (SPARK-18484) case class datasets - ability to
specify decimal precision and scale
Damian Momot created SPARK-18484:
------------------------------------
Summary: case class datasets - ability to specify decimal precision and scale
Key: SPARK-18484
URL: https://issues.apache.org/jira/browse/SPARK-18484
Project: Spark
Issue Type: Improvement
Affects Versions: 2.0.1, 2.0.0
Reporter: Damian Momot
Currently when using decimal type (BigDecimal in scala case class) there's no way to enforce precision and scale. This is quite critical when saving data - regarding space usage and compatibility with external systems (for example Hive table) because spark saves data as Decimal(38,18)
{code:scala}
val spark: SparkSession = ???
case class TestClass(id: String, money: BigDecimal)
val testDs = spark.createDataset(Seq(
TestClass("1", BigDecimal("22.50")),
TestClass("2", BigDecimal("500.66"))
))
testDs.printSchema()
{code}
{code}
root
|-- id: string (nullable = true)
|-- money: decimal(38,18) (nullable = true)
{code}
Workaround is to convert dataset to dataframe before saving and manually cast to specific decimal scale/precision:
{code:scala}
import org.apache.spark.sql.types.DecimalType
val testDf = testDs.toDF()
testDf
.withColumn("money", testDf("money").cast(DecimalType(10,2)))
.printSchema()
{code}
{code}
root
|-- id: string (nullable = true)
|-- money: decimal(10,2) (nullable = true)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org