You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by BryanCutler <gi...@git.apache.org> on 2017/12/22 23:10:24 UTC
[GitHub] spark pull request #18754: [SPARK-21552][SQL] Add DecimalType support to Arr...
Github user BryanCutler commented on a diff in the pull request:
https://github.com/apache/spark/pull/18754#discussion_r158567491
--- Diff: python/pyspark/sql/types.py ---
@@ -1617,7 +1617,7 @@ def to_arrow_type(dt):
elif type(dt) == DoubleType:
arrow_type = pa.float64()
elif type(dt) == DecimalType:
- arrow_type = pa.decimal(dt.precision, dt.scale)
+ arrow_type = pa.decimal128(dt.precision, dt.scale)
--- End diff --
yes, that's the right way - it is now fixed at 128 bits internally. I believe the Arrow Java limit is the same as Spark 38/38, not sure if pyarrow is the same but I think so.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org