You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Uroš Bojanić <ur...@apache.org> on 2026/08/18 16:00:13 UTC
[DISCUSS] SPIP: Add the DECFLOAT data type
Hi all,
I would like to start a discussion on the SPIP to add a new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 floating-point decimals with per-value exponents.
JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the gap between DECIMAL, which caps at precision 38 with a fixed per-column scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact: each value keeps its own exponent, arithmetic is decimal, and it supports signed zero, Inf, and NaN. It is additive, round-trips through a proposed Parquet logical type for cross-engine interop, and would ship config-gated during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior unchanged.
Additional information is available in the SPIP document:
https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
Please provide your feedback on the approach, scope, and the proposal described in the document.
Thank you!
Best,
Uroš
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Uroš Bojanić <ur...@apache.org>.
Hi all,
Thank you for your comments, here is where we land with the revised DECFLOAT SPIP:
- There is a single arithmetic path in DECFLOAT v1 proposal - the pure Java implementation. Any native kernels are completely out of scope for the purpose of this DECFLOAT SPIP.
- Regarding the pure-Java in-tree implementation, of course the ported arithmetic library will land together with the actual downstream consumer code, Serge's PR was just a POC.
- Persistence is conditional on the logical type in Parquet, and Spark will not ship any private encoding in the meantime; but, in-memory engine representation is independent.
- There are no more open items regarding the public API surface. For Arrow we propose an extension type over fixed-width binary, and a Spark-owned value class for the JVM type.
- To address the incubation concern, the SPIP now defines measurable exit criteria for enabling the type by default and provides better estimates against the larger scope.
- The notes regarding coercion, AtomicType, and ordering, are all corrected. DECFLOAT is a FractionalType, ordering will follow DOUBLE's conventions, and DECFLOAT(34) is widest.
This should resolve all SPIP contracts, whereas the exact implementation details and deeper technical discussions can be carried out further during code review in appropriate PRs.
Best,
Uroš
On 2026/09/01 16:07:31 Uroš Bojanić wrote:
> Hi all,
>
> In accordance with concerns raised in the first SPIP vote for DECFLOAT (https://lists.apache.org/thread/nozkdbsc7oy69j581qtp9v69wtzv4tdl), please note that the SPIP proposal has been revised and updated accordingly: https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E.
>
> In short, we agree to move away from the initial JNI / libbid proposal. The new proposal is to use a custom Java native IEEE 754 compliant port.
>
> Given that this DECFLOAT proposal is still actively evolving, we can keep this DISCUSS thread open to any additional feedback and substantive concerns about this SPIP to add the DECFLOAT data type!
>
> Best,
> Uroš
>
> On 2026/08/18 16:00:13 Uroš Bojanić wrote:
> > Hi all,
> >
> > I would like to start a discussion on the SPIP to add a new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 floating-point decimals with per-value exponents.
> >
> > JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
> >
> > Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the gap between DECIMAL, which caps at precision 38 with a fixed per-column scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact: each value keeps its own exponent, arithmetic is decimal, and it supports signed zero, Inf, and NaN. It is additive, round-trips through a proposed Parquet logical type for cross-engine interop, and would ship config-gated during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior unchanged.
> >
> > Additional information is available in the SPIP document:
> > https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
> >
> > Please provide your feedback on the approach, scope, and the proposal described in the document.
> >
> > Thank you!
> >
> > Best,
> > Uroš
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Uroš Bojanić <ur...@apache.org>.
Hi all,
In accordance with concerns raised in the first SPIP vote for DECFLOAT (https://lists.apache.org/thread/nozkdbsc7oy69j581qtp9v69wtzv4tdl), please note that the SPIP proposal has been revised and updated accordingly: https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E.
In short, we agree to move away from the initial JNI / libbid proposal. The new proposal is to use a custom Java native IEEE 754 compliant port.
Given that this DECFLOAT proposal is still actively evolving, we can keep this DISCUSS thread open to any additional feedback and substantive concerns about this SPIP to add the DECFLOAT data type!
Best,
Uroš
On 2026/08/18 16:00:13 Uroš Bojanić wrote:
> Hi all,
>
> I would like to start a discussion on the SPIP to add a new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 floating-point decimals with per-value exponents.
>
> JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
>
> Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the gap between DECIMAL, which caps at precision 38 with a fixed per-column scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact: each value keeps its own exponent, arithmetic is decimal, and it supports signed zero, Inf, and NaN. It is additive, round-trips through a proposed Parquet logical type for cross-engine interop, and would ship config-gated during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior unchanged.
>
> Additional information is available in the SPIP document:
> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
>
> Please provide your feedback on the approach, scope, and the proposal described in the document.
>
> Thank you!
>
> Best,
> Uroš
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Uroš Bojanić <ur...@apache.org>.
Hi Shrirang,
Both your points are well-taken and folded into the SPIP document.
1) Kernel: added pros/cons & trade-offs for JNI vs Java impl (expanded Appendix E, #6).
2) Coercion: agreed and tightened the Note 3 for type precedence (under Appendix C).
Thank you for reviewing the doc! Further feedback is still welcome.
Best,
Uroš
On 2026/08/18 18:53:05 Shrirang Mhalgi wrote:
> Hi Uroš.
> Thank you so much for the thorough SPIP. The IEEE 754 alignment, the
> config-gated incubation pattern, and the explicit type table are really
> helpful - excited to see this move forward.
>
> I had two questions:
> 1. Has a pure - Java arithmetic implementation been evaluated as an
> alternative to JNI / libbid? Spark's existing native dependencies (snappy,
> zstd, lz4) are compression codecs that ship as multi-platform fat JARs.
> libbid would be the first native dependency on the arithmetic hot path with
> no fallback - every DECFLOAT operation would be a JNI call. There is a
> potential risk of introducing platform-specific build complexity,
> crash-mode failures (SIGSEGV vs exceptions), and distribution overhead for
> downstream packagers. Curious if the tradeoff was considered.
> 2. On type precedence (Note 3): is the DECIMAL(35..38) -> DECFLOAT(34)
> coercion implicit in a UNION or JOIN? Users could silently lose precision.
> Should ANSI mode reject this implicit widening (require explicit CAST)?
> Silent precision loss seems inconsistent with ANSI mode's fail-fast
> behavior for other numeric overflows (CAST_OVERFLOW).
>
> Thanks and regards,
> Shrirang
>
> On Tue, Aug 18, 2026 at 9:01 AM Uroš Bojanić <ur...@apache.org> wrote:
>
> > Hi all,
> >
> > I would like to start a discussion on the SPIP to add a new Spark SQL data
> > type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10
> > floating-point decimals with per-value exponents.
> >
> > JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
> >
> > Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type
> > (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the
> > gap between DECIMAL, which caps at precision 38 with a fixed per-column
> > scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact:
> > each value keeps its own exponent, arithmetic is decimal, and it supports
> > signed zero, Inf, and NaN. It is additive, round-trips through a proposed
> > Parquet logical type for cross-engine interop, and would ship config-gated
> > during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior
> > unchanged.
> >
> > Additional information is available in the SPIP document:
> >
> > https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
> >
> > Please provide your feedback on the approach, scope, and the proposal
> > described in the document.
> >
> > Thank you!
> >
> > Best,
> > Uroš
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Shrirang Mhalgi <sh...@gmail.com>.
Hi Uroš.
Thank you so much for the thorough SPIP. The IEEE 754 alignment, the
config-gated incubation pattern, and the explicit type table are really
helpful - excited to see this move forward.
I had two questions:
1. Has a pure - Java arithmetic implementation been evaluated as an
alternative to JNI / libbid? Spark's existing native dependencies (snappy,
zstd, lz4) are compression codecs that ship as multi-platform fat JARs.
libbid would be the first native dependency on the arithmetic hot path with
no fallback - every DECFLOAT operation would be a JNI call. There is a
potential risk of introducing platform-specific build complexity,
crash-mode failures (SIGSEGV vs exceptions), and distribution overhead for
downstream packagers. Curious if the tradeoff was considered.
2. On type precedence (Note 3): is the DECIMAL(35..38) -> DECFLOAT(34)
coercion implicit in a UNION or JOIN? Users could silently lose precision.
Should ANSI mode reject this implicit widening (require explicit CAST)?
Silent precision loss seems inconsistent with ANSI mode's fail-fast
behavior for other numeric overflows (CAST_OVERFLOW).
Thanks and regards,
Shrirang
On Tue, Aug 18, 2026 at 9:01 AM Uroš Bojanić <ur...@apache.org> wrote:
> Hi all,
>
> I would like to start a discussion on the SPIP to add a new Spark SQL data
> type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10
> floating-point decimals with per-value exponents.
>
> JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
>
> Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type
> (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the
> gap between DECIMAL, which caps at precision 38 with a fixed per-column
> scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact:
> each value keeps its own exponent, arithmetic is decimal, and it supports
> signed zero, Inf, and NaN. It is additive, round-trips through a proposed
> Parquet logical type for cross-engine interop, and would ship config-gated
> during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior
> unchanged.
>
> Additional information is available in the SPIP document:
>
> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
>
> Please provide your feedback on the approach, scope, and the proposal
> described in the document.
>
> Thank you!
>
> Best,
> Uroš
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Serge Rielau <se...@rielau.com>.
Dongjoon,
You make many valid points, and we’ll address them.
Some I do not understand however, and as the author of the Decimal128 PR let me ask for clarification:
It is my understanding that it is frowned upon to roll in “unused APIs” (i.e. dead code).
The PR remains in draft mode because this SPIP has not passed yet. The PR’s initial purpose is to support THIS SPIP.
By the same token, there ought not be a DecFloatType without this SPIP.
Are you advocating:
a) Roll in this library as a precondition to approve the SPIP
b) Make it clear that this is the specific library being used,
c) Sssert that there will be “some” java native library. Whether that is BigDecimal based, libbid, or something else - possibly home grown.
I’m also confused by the asserted dependency on Parquet and the transport layer. Aside from the obvious chicken-egg problem I wonder why the in-memory layout used by the DECFLOAT type in Spark must be aligned with the on-disk layout of Parquet.
TIMESTAMP nano being one example where that is not case. Certainly it is “nice” to just copy through, but, in my opinion semantics are way more important. Which is why we have been focusing on this.
Cheers
Serge
On Sep 22, 2026, at 8:29 AM, Thomas Kissinger via dev <de...@spark.apache.org> wrote:
Hi all,
Thanks, Uroš, for revising the proposal and for all the work invested. I support adding a decimal-floating-point type to Spark, and I agree with Dongjoon that the proposal is not yet ready for another vote.
My concern is that IEEE BID seems to be a predetermined choice rather than the result of comparing alternatives against Spark's requirements. IEEE BID is standardized, but its implementation ecosystem is very limited: neither Java nor Python provides a native BID value type. The proposed libbid-based arithmetic stops at 34 digits, leaving the established 38-digit SQL precision unsupported. Wider formats would require significant additional implementation work, not just selecting a larger precision.
Porting libbid to Java reuses established algorithms and test vectors, but does not automatically transfer the original implementation's maturity. Roughly 20K lines of newly ported arithmetic still create a substantial obligation for correctness, review, and long-term maintenance.
Java and Python already have mature arbitrary-precision decimal arithmetic. I would first evaluate a Spark-owned wrapper around BigDecimal for finite arithmetic, with explicit handling of special values and error semantics. This could preserve wider precision and much less custom arithmetic code—a particularly important tradeoff for the main use case of decimal-floating point types: financial workloads.
Best,
Thomas
On Tue, Sep 22, 2026 at 4:17 PM Dongjoon Hyun <do...@apache.org>> wrote:
Thank you for the revised proposal and for the follow-up.
Let me start with my overall assessment, since I said the same thing
on the vote thread in August: this SPIP is still not mature enough to
go to a vote. The revision fixes one architectural point, but the
document still carries [OPEN] markers on its public API surface,
assumes a Parquet encoding that the Parquet community has explicitly
deferred, and leans on a config gate and a timeline whose precedent
does not support them. Details below.
On DB's specific question: yes, Revision 1 addresses the concern
behind my -1. The arithmetic path no longer depends on a native
library, and the fallback story is now a pure-Java, in-tree
implementation (Appendix E, item 6; SPARK-59111). That was the
blocking factor for me, and it is resolved at the SPIP level. One
follow-up on the same item: it still says native kernels "remain an
optional optimization under the same IEEE contract". I'd like the SPIP
to state plainly that the pure-Java path is the only arithmetic path
in v1, and that any native or accelerator path is out of scope and
would need its own SPIP. Otherwise the Java implementation risks being
demoted to a reference path that only the test suite exercises.
That said, I want to be precise about what exists today. The only code
on the table is PR #58410, which adds a ported BID arithmetic library
under a non-Spark package (org.bidfp): about 21K lines of Java, 10K
lines of tests, and 129K lines of Intel test vectors. By its own
description it "does not yet connect it to a user-facing Spark API or
execution path". There is still no DecFloatType, no parser or Catalyst
integration, no cast or coercion code, and no data source path. In
other words, the SPIP has been revised on paper, but the implementation
that would let us judge the pure-Java claim in practice has not been
provided. The PR itself has not been reviewed yet either. For that
reason I don't think the library should be merged as a standalone
module with no consumer in the tree. It should land together with the
first code that actually uses it, so that it is reviewed and exercised
as part of the feature rather than parked as dead code.
The SPIP-level items I'd want settled before a new vote are contracts
rather than implementation details:
1. Parquet. Appendix D assumes a BID little-endian FIXED_LEN_BYTE_ARRAY
of 8/16 bytes. That is not where the parquet-dev thread is. As of
the Sep 5 and Sep 21 messages, the points both sides agree on are:
precision parameterized rather than fixed at 38, with narrower
encodings allowed and a path to wider ones; coverage of the full
finite decimal128 range; at minimum +/-Inf and a canonical NaN; and,
explicitly, that the physical representation is to be selected only
after public benchmarks and validation vectors. Canonicalization and
sNaN/-0 handling are still open, and a joint proposal has not been
written. I'd suggest the SPIP make Parquet persistence conditional
on the Parquet logical type being adopted, and state that Spark will
not ship a private encoding in the meantime. Russell raised the
alignment question on the vote thread and I share it.
2. External types and Arrow. Appendix B (Java/Scala external type,
JDBC) and the Arrow transport question are still marked [OPEN].
These define what Row.get, Spark Connect, and Arrow-based collect in
PySpark return, so they are public API. Ian's -0 asked for the Arrow
half of this, and the revision does not answer it yet.
3. Gate and timeline. The SPIP cites the TIME type as the incubation
pattern and bases the 9-12 month estimate on it. TIME is the worst
possible example to lean on. Its SPIP passed on 2025-02-26. The type
was developed without a gate for about nine months, and
spark.sql.timeType.enabled was added on 2025-12-10, six days before
the 4.1.0 release and during the RC phase, because support was
incomplete (SPARK-54609). Nineteen months after the vote it is still
off by default everywhere: in 4.1.0, in 4.2.0, in branch-4.3 whose
RC1 was cut on 2026-08-31, in branch-4.x (4.4.0-SNAPSHOT), and on
master, which is already Spark 5.0.0-SNAPSHOT. Coverage commits for
TIME are still landing this month. So a config gate by itself is not
a mitigation; it defers the risk across an entire major version
line. DECFLOAT must not follow that precedent. We should not accept
a second data type that lands piecemeal behind a flag with no
defined end state, and I would treat repeating the TIME pattern as
a reason to block rather than as an established practice to cite.
Concretely, I'd ask the SPIP to define measurable exit criteria for
the gate (function list, data sources, clients, plus the Parquet and
Arrow contracts above), name who owns reaching them, and revisit the
estimate, since DECFLOAT is strictly larger than TIME.
A few smaller design points in the current text should also be fixed:
- Coercion note (3) claims that lossy DECIMAL(35..38) -> DECFLOAT(34)
widening is not implicit in ANSI mode, "consistent with ANSI
coercion". Spark's ANSI mode already widens DECIMAL(38) + DOUBLE
implicitly to DOUBLE, so the note describes a rule Spark does not
have. The precedence also stops being a total order, so the result of
DOUBLE + DECIMAL(38) + DECFLOAT(34) would depend on operand order.
- Appendix A declares DecFloatType as an AtomicType. Arithmetic and
aggregates such as SUM and ABS accept NumericType, so it needs to be
a FractionalType like DecimalType and DoubleType.
- Ordering is described as IEEE totalOrder, which distinguishes -0 from
+0. Spark's DOUBLE treats -0.0 = 0.0 and NaN = NaN for equality,
grouping, and joins, and sorts NaN after +Inf. DECFLOAT should follow
those conventions, or equality and ORDER BY will disagree.
Thanks,
Dongjoon.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org<ma...@spark.apache.org>
--
THOMAS KISSINGER
Staff Software Engineer
MOBILE +49 174-2195270
EMAIL thomas.kissinger@snowflake.com<ma...@snowflake.com>
[https://www.snowflake.com/wp-content/themes/snowflake/img/snowflake-logo-blue@2x.png]
Snowflake Inc.
135 Constitution Drive
Menlo Park, CA 94025, USA
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Thomas Kissinger via dev <de...@spark.apache.org>.
Hi all,
Thanks, Uroš, for revising the proposal and for all the work invested. I
support adding a decimal-floating-point type to Spark, and I agree with
Dongjoon that the proposal is not yet ready for another vote.
My concern is that IEEE BID seems to be a predetermined choice rather than
the result of comparing alternatives against Spark's requirements. IEEE BID
is standardized, but its implementation ecosystem is very limited: neither
Java nor Python provides a native BID value type. The proposed libbid-based
arithmetic stops at 34 digits, leaving the established 38-digit SQL
precision unsupported. Wider formats would require significant additional
implementation work, not just selecting a larger precision.
Porting libbid to Java reuses established algorithms and test vectors, but
does not automatically transfer the original implementation's maturity.
Roughly 20K lines of newly ported arithmetic still create a substantial
obligation for correctness, review, and long-term maintenance.
Java and Python already have mature arbitrary-precision decimal arithmetic.
I would first evaluate a Spark-owned wrapper around BigDecimal for finite
arithmetic, with explicit handling of special values and error semantics.
This could preserve wider precision and much less custom arithmetic code—a
particularly important tradeoff for the main use case of decimal-floating
point types: financial workloads.
Best,
Thomas
On Tue, Sep 22, 2026 at 4:17 PM Dongjoon Hyun <do...@apache.org> wrote:
> Thank you for the revised proposal and for the follow-up.
>
> Let me start with my overall assessment, since I said the same thing
> on the vote thread in August: this SPIP is still not mature enough to
> go to a vote. The revision fixes one architectural point, but the
> document still carries [OPEN] markers on its public API surface,
> assumes a Parquet encoding that the Parquet community has explicitly
> deferred, and leans on a config gate and a timeline whose precedent
> does not support them. Details below.
>
> On DB's specific question: yes, Revision 1 addresses the concern
> behind my -1. The arithmetic path no longer depends on a native
> library, and the fallback story is now a pure-Java, in-tree
> implementation (Appendix E, item 6; SPARK-59111). That was the
> blocking factor for me, and it is resolved at the SPIP level. One
> follow-up on the same item: it still says native kernels "remain an
> optional optimization under the same IEEE contract". I'd like the SPIP
> to state plainly that the pure-Java path is the only arithmetic path
> in v1, and that any native or accelerator path is out of scope and
> would need its own SPIP. Otherwise the Java implementation risks being
> demoted to a reference path that only the test suite exercises.
>
> That said, I want to be precise about what exists today. The only code
> on the table is PR #58410, which adds a ported BID arithmetic library
> under a non-Spark package (org.bidfp): about 21K lines of Java, 10K
> lines of tests, and 129K lines of Intel test vectors. By its own
> description it "does not yet connect it to a user-facing Spark API or
> execution path". There is still no DecFloatType, no parser or Catalyst
> integration, no cast or coercion code, and no data source path. In
> other words, the SPIP has been revised on paper, but the implementation
> that would let us judge the pure-Java claim in practice has not been
> provided. The PR itself has not been reviewed yet either. For that
> reason I don't think the library should be merged as a standalone
> module with no consumer in the tree. It should land together with the
> first code that actually uses it, so that it is reviewed and exercised
> as part of the feature rather than parked as dead code.
>
> The SPIP-level items I'd want settled before a new vote are contracts
> rather than implementation details:
>
> 1. Parquet. Appendix D assumes a BID little-endian FIXED_LEN_BYTE_ARRAY
> of 8/16 bytes. That is not where the parquet-dev thread is. As of
> the Sep 5 and Sep 21 messages, the points both sides agree on are:
> precision parameterized rather than fixed at 38, with narrower
> encodings allowed and a path to wider ones; coverage of the full
> finite decimal128 range; at minimum +/-Inf and a canonical NaN; and,
> explicitly, that the physical representation is to be selected only
> after public benchmarks and validation vectors. Canonicalization and
> sNaN/-0 handling are still open, and a joint proposal has not been
> written. I'd suggest the SPIP make Parquet persistence conditional
> on the Parquet logical type being adopted, and state that Spark will
> not ship a private encoding in the meantime. Russell raised the
> alignment question on the vote thread and I share it.
>
> 2. External types and Arrow. Appendix B (Java/Scala external type,
> JDBC) and the Arrow transport question are still marked [OPEN].
> These define what Row.get, Spark Connect, and Arrow-based collect in
> PySpark return, so they are public API. Ian's -0 asked for the Arrow
> half of this, and the revision does not answer it yet.
>
> 3. Gate and timeline. The SPIP cites the TIME type as the incubation
> pattern and bases the 9-12 month estimate on it. TIME is the worst
> possible example to lean on. Its SPIP passed on 2025-02-26. The type
> was developed without a gate for about nine months, and
> spark.sql.timeType.enabled was added on 2025-12-10, six days before
> the 4.1.0 release and during the RC phase, because support was
> incomplete (SPARK-54609). Nineteen months after the vote it is still
> off by default everywhere: in 4.1.0, in 4.2.0, in branch-4.3 whose
> RC1 was cut on 2026-08-31, in branch-4.x (4.4.0-SNAPSHOT), and on
> master, which is already Spark 5.0.0-SNAPSHOT. Coverage commits for
> TIME are still landing this month. So a config gate by itself is not
> a mitigation; it defers the risk across an entire major version
> line. DECFLOAT must not follow that precedent. We should not accept
> a second data type that lands piecemeal behind a flag with no
> defined end state, and I would treat repeating the TIME pattern as
> a reason to block rather than as an established practice to cite.
> Concretely, I'd ask the SPIP to define measurable exit criteria for
> the gate (function list, data sources, clients, plus the Parquet and
> Arrow contracts above), name who owns reaching them, and revisit the
> estimate, since DECFLOAT is strictly larger than TIME.
>
> A few smaller design points in the current text should also be fixed:
>
> - Coercion note (3) claims that lossy DECIMAL(35..38) -> DECFLOAT(34)
> widening is not implicit in ANSI mode, "consistent with ANSI
> coercion". Spark's ANSI mode already widens DECIMAL(38) + DOUBLE
> implicitly to DOUBLE, so the note describes a rule Spark does not
> have. The precedence also stops being a total order, so the result of
> DOUBLE + DECIMAL(38) + DECFLOAT(34) would depend on operand order.
> - Appendix A declares DecFloatType as an AtomicType. Arithmetic and
> aggregates such as SUM and ABS accept NumericType, so it needs to be
> a FractionalType like DecimalType and DoubleType.
> - Ordering is described as IEEE totalOrder, which distinguishes -0 from
> +0. Spark's DOUBLE treats -0.0 = 0.0 and NaN = NaN for equality,
> grouping, and joins, and sorts NaN after +Inf. DECFLOAT should follow
> those conventions, or equality and ORDER BY will disagree.
>
> Thanks,
> Dongjoon.
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
--
THOMAS KISSINGER
Staff Software Engineer
MOBILE +49 174-2195270
EMAIL thomas.kissinger@snowflake.com
Snowflake Inc.
135 Constitution Drive
Menlo Park, CA 94025, USA
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Dongjoon Hyun <do...@apache.org>.
Thank you for the revised proposal and for the follow-up.
Let me start with my overall assessment, since I said the same thing
on the vote thread in August: this SPIP is still not mature enough to
go to a vote. The revision fixes one architectural point, but the
document still carries [OPEN] markers on its public API surface,
assumes a Parquet encoding that the Parquet community has explicitly
deferred, and leans on a config gate and a timeline whose precedent
does not support them. Details below.
On DB's specific question: yes, Revision 1 addresses the concern
behind my -1. The arithmetic path no longer depends on a native
library, and the fallback story is now a pure-Java, in-tree
implementation (Appendix E, item 6; SPARK-59111). That was the
blocking factor for me, and it is resolved at the SPIP level. One
follow-up on the same item: it still says native kernels "remain an
optional optimization under the same IEEE contract". I'd like the SPIP
to state plainly that the pure-Java path is the only arithmetic path
in v1, and that any native or accelerator path is out of scope and
would need its own SPIP. Otherwise the Java implementation risks being
demoted to a reference path that only the test suite exercises.
That said, I want to be precise about what exists today. The only code
on the table is PR #58410, which adds a ported BID arithmetic library
under a non-Spark package (org.bidfp): about 21K lines of Java, 10K
lines of tests, and 129K lines of Intel test vectors. By its own
description it "does not yet connect it to a user-facing Spark API or
execution path". There is still no DecFloatType, no parser or Catalyst
integration, no cast or coercion code, and no data source path. In
other words, the SPIP has been revised on paper, but the implementation
that would let us judge the pure-Java claim in practice has not been
provided. The PR itself has not been reviewed yet either. For that
reason I don't think the library should be merged as a standalone
module with no consumer in the tree. It should land together with the
first code that actually uses it, so that it is reviewed and exercised
as part of the feature rather than parked as dead code.
The SPIP-level items I'd want settled before a new vote are contracts
rather than implementation details:
1. Parquet. Appendix D assumes a BID little-endian FIXED_LEN_BYTE_ARRAY
of 8/16 bytes. That is not where the parquet-dev thread is. As of
the Sep 5 and Sep 21 messages, the points both sides agree on are:
precision parameterized rather than fixed at 38, with narrower
encodings allowed and a path to wider ones; coverage of the full
finite decimal128 range; at minimum +/-Inf and a canonical NaN; and,
explicitly, that the physical representation is to be selected only
after public benchmarks and validation vectors. Canonicalization and
sNaN/-0 handling are still open, and a joint proposal has not been
written. I'd suggest the SPIP make Parquet persistence conditional
on the Parquet logical type being adopted, and state that Spark will
not ship a private encoding in the meantime. Russell raised the
alignment question on the vote thread and I share it.
2. External types and Arrow. Appendix B (Java/Scala external type,
JDBC) and the Arrow transport question are still marked [OPEN].
These define what Row.get, Spark Connect, and Arrow-based collect in
PySpark return, so they are public API. Ian's -0 asked for the Arrow
half of this, and the revision does not answer it yet.
3. Gate and timeline. The SPIP cites the TIME type as the incubation
pattern and bases the 9-12 month estimate on it. TIME is the worst
possible example to lean on. Its SPIP passed on 2025-02-26. The type
was developed without a gate for about nine months, and
spark.sql.timeType.enabled was added on 2025-12-10, six days before
the 4.1.0 release and during the RC phase, because support was
incomplete (SPARK-54609). Nineteen months after the vote it is still
off by default everywhere: in 4.1.0, in 4.2.0, in branch-4.3 whose
RC1 was cut on 2026-08-31, in branch-4.x (4.4.0-SNAPSHOT), and on
master, which is already Spark 5.0.0-SNAPSHOT. Coverage commits for
TIME are still landing this month. So a config gate by itself is not
a mitigation; it defers the risk across an entire major version
line. DECFLOAT must not follow that precedent. We should not accept
a second data type that lands piecemeal behind a flag with no
defined end state, and I would treat repeating the TIME pattern as
a reason to block rather than as an established practice to cite.
Concretely, I'd ask the SPIP to define measurable exit criteria for
the gate (function list, data sources, clients, plus the Parquet and
Arrow contracts above), name who owns reaching them, and revisit the
estimate, since DECFLOAT is strictly larger than TIME.
A few smaller design points in the current text should also be fixed:
- Coercion note (3) claims that lossy DECIMAL(35..38) -> DECFLOAT(34)
widening is not implicit in ANSI mode, "consistent with ANSI
coercion". Spark's ANSI mode already widens DECIMAL(38) + DOUBLE
implicitly to DOUBLE, so the note describes a rule Spark does not
have. The precedence also stops being a total order, so the result of
DOUBLE + DECIMAL(38) + DECFLOAT(34) would depend on operand order.
- Appendix A declares DecFloatType as an AtomicType. Arithmetic and
aggregates such as SUM and ABS accept NumericType, so it needs to be
a FractionalType like DecimalType and DoubleType.
- Ordering is described as IEEE totalOrder, which distinguishes -0 from
+0. Spark's DOUBLE treats -0.0 = 0.0 and NaN = NaN for equality,
grouping, and joins, and sorts NaN after +Inf. DECFLOAT should follow
those conventions, or equality and ORDER BY will disagree.
Thanks,
Dongjoon.
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by DB Tsai <db...@dbtsai.com>.
Thanks for updating the proposal and working through the feedback. I
support adding decimal floating-point support to Spark.
The revised proposal now specifies an in-tree, pure-Java implementation
with no native runtime dependency. This appears to address the
architectural concern behind Dongjoon’s original -1. Dongjoon, could you
confirm whether this addresses your concern about having a safe Java
execution path?
Thanks,
On Mon, Sep 21, 2026 at 3:40 PM Hyukjin Kwon <gu...@apache.org> wrote:
> Let's separate the implementation specific part from SPIP. For
> implementation things, I think we can discuss details in the PR and make
> sure, during the reviews, to involve people who had concerns about it.
>
> On Fri, 18 Sept 2026 at 02:16, Gengliang Wang <lt...@gmail.com> wrote:
>
>> Thanks for the update. I agree with separating the value domain from
>> exception handling and the arithmetic implementation.
>>
>> I support defining DECFLOAT with signed zero, positive and negative
>> infinity, and quiet NaN. These are numeric values rather than SQL NULL, and
>> ppreserving them enables faithful interchange and round-tripping across
>> language and engine boundaries, including between Spark and Python clients.
>> A finite-only definition would lose that information and would be difficult
>> to extend compatibly later.
>>
>> Detailed coercion rules, the execution implementation, external APIs, and
>> the Parquet encoding still require further review, but they should not
>> block agreement on the logical value domain.
>>
>> With this separation clarified, I support moving the SPIP forward.
>>
>> Best,
>> Gengliang
>>
>> On Thu, Sep 17, 2026 at 7:44 AM Uroš Bojanić <ur...@apache.org> wrote:
>>
>>> Hi all,
>>>
>>> We've had some great input and discussion in the DECFLOAT SPIP document (
>>> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E/edit?tab=t.0#heading=h.zarpams0nxzq),
>>> so I'm writing an update here to keep the DISCUSS thread active and invite
>>> further feedback and discussion regarding this SPIP.
>>>
>>> It's really important to separate the irreversible decisions from the
>>> reversible ones. The one-way door here is the value domain - DECFLOAT
>>> should carry IEEE 754's non-finite values (+Inf, -Inf, NaN,
>>> signed zero), the same way DOUBLE already does. Digit count (e.g. 34 vs
>>> 38) and arithmetic implementation can easily be changed later - for
>>> example, digits can always be widened later and retain full backwards
>>> compatibility; also, the execution kernel can change under a fixed IEEE
>>> contract. However, specials are at the opposite end - we can't retrofit
>>> them onto a shipped finite-only type without breaking it.
>>>
>>> Why this matters for big data: decimal pipelines produce unbounded and
>>> undefined quantities as legitimate business states, not data bugs (e.g. an
>>> uncapped credit line, a threshold that means "no maximum," a magnitude that
>>> has run past any finite bound). IEEE already names & defines these
>>> precisely, and DOUBLE carries them today. DECFLOAT is the base-10 DOUBLE,
>>> its domain should match accordingly.
>>>
>>> A finite-only decimal type can't hold those values, so applications
>>> disguise them (almost always as NULL). However, NULL in Spark means
>>> missing/unknown, so "unbounded" and "never loaded" would become the same
>>> value, and SUM, COUNT, range filters, etc. could no longer telling them
>>> apart. The loss here runs one way, we can always collapse Inf/NaN to NULL
>>> if we want SQL null semantics, but we can never rebuild meaning already
>>> flattened into NULL. This reaches past Spark too, Parquet's decimal-float
>>> work and existing C++/Rust/Python engines already emit Inf/NaN, so a
>>> finite-only type can't round-trip that data faithfully across different
>>> languages / engines.
>>>
>>> Let's agree that DECFLOAT's domain is IEEE 754's, non-finite values
>>> included. Without this, DECFLOAT isn't a better DECIMAL - it's a worse
>>> DOUBLE.
>>>
>>> Best,
>>> Uroš
>>>
>>> On 2026/08/18 16:00:13 Uroš Bojanić wrote:
>>> > Hi all,
>>> >
>>> > I would like to start a discussion on the SPIP to add a new Spark SQL
>>> data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10
>>> floating-point decimals with per-value exponents.
>>> >
>>> > JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
>>> >
>>> > Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type
>>> (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the
>>> gap between DECIMAL, which caps at precision 38 with a fixed per-column
>>> scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact:
>>> each value keeps its own exponent, arithmetic is decimal, and it supports
>>> signed zero, Inf, and NaN. It is additive, round-trips through a proposed
>>> Parquet logical type for cross-engine interop, and would ship config-gated
>>> during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior
>>> unchanged.
>>> >
>>> > Additional information is available in the SPIP document:
>>> >
>>> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
>>> >
>>> > Please provide your feedback on the approach, scope, and the proposal
>>> described in the document.
>>> >
>>> > Thank you!
>>> >
>>> > Best,
>>> > Uroš
>>> >
>>> > ---------------------------------------------------------------------
>>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>> >
>>> >
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>>
>>>
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Hyukjin Kwon <gu...@apache.org>.
Let's separate the implementation specific part from SPIP. For
implementation things, I think we can discuss details in the PR and make
sure, during the reviews, to involve people who had concerns about it.
On Fri, 18 Sept 2026 at 02:16, Gengliang Wang <lt...@gmail.com> wrote:
> Thanks for the update. I agree with separating the value domain from
> exception handling and the arithmetic implementation.
>
> I support defining DECFLOAT with signed zero, positive and negative
> infinity, and quiet NaN. These are numeric values rather than SQL NULL, and
> ppreserving them enables faithful interchange and round-tripping across
> language and engine boundaries, including between Spark and Python clients.
> A finite-only definition would lose that information and would be difficult
> to extend compatibly later.
>
> Detailed coercion rules, the execution implementation, external APIs, and
> the Parquet encoding still require further review, but they should not
> block agreement on the logical value domain.
>
> With this separation clarified, I support moving the SPIP forward.
>
> Best,
> Gengliang
>
> On Thu, Sep 17, 2026 at 7:44 AM Uroš Bojanić <ur...@apache.org> wrote:
>
>> Hi all,
>>
>> We've had some great input and discussion in the DECFLOAT SPIP document (
>> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E/edit?tab=t.0#heading=h.zarpams0nxzq),
>> so I'm writing an update here to keep the DISCUSS thread active and invite
>> further feedback and discussion regarding this SPIP.
>>
>> It's really important to separate the irreversible decisions from the
>> reversible ones. The one-way door here is the value domain - DECFLOAT
>> should carry IEEE 754's non-finite values (+Inf, -Inf, NaN,
>> signed zero), the same way DOUBLE already does. Digit count (e.g. 34 vs
>> 38) and arithmetic implementation can easily be changed later - for
>> example, digits can always be widened later and retain full backwards
>> compatibility; also, the execution kernel can change under a fixed IEEE
>> contract. However, specials are at the opposite end - we can't retrofit
>> them onto a shipped finite-only type without breaking it.
>>
>> Why this matters for big data: decimal pipelines produce unbounded and
>> undefined quantities as legitimate business states, not data bugs (e.g. an
>> uncapped credit line, a threshold that means "no maximum," a magnitude that
>> has run past any finite bound). IEEE already names & defines these
>> precisely, and DOUBLE carries them today. DECFLOAT is the base-10 DOUBLE,
>> its domain should match accordingly.
>>
>> A finite-only decimal type can't hold those values, so applications
>> disguise them (almost always as NULL). However, NULL in Spark means
>> missing/unknown, so "unbounded" and "never loaded" would become the same
>> value, and SUM, COUNT, range filters, etc. could no longer telling them
>> apart. The loss here runs one way, we can always collapse Inf/NaN to NULL
>> if we want SQL null semantics, but we can never rebuild meaning already
>> flattened into NULL. This reaches past Spark too, Parquet's decimal-float
>> work and existing C++/Rust/Python engines already emit Inf/NaN, so a
>> finite-only type can't round-trip that data faithfully across different
>> languages / engines.
>>
>> Let's agree that DECFLOAT's domain is IEEE 754's, non-finite values
>> included. Without this, DECFLOAT isn't a better DECIMAL - it's a worse
>> DOUBLE.
>>
>> Best,
>> Uroš
>>
>> On 2026/08/18 16:00:13 Uroš Bojanić wrote:
>> > Hi all,
>> >
>> > I would like to start a discussion on the SPIP to add a new Spark SQL
>> data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10
>> floating-point decimals with per-value exponents.
>> >
>> > JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
>> >
>> > Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type
>> (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the
>> gap between DECIMAL, which caps at precision 38 with a fixed per-column
>> scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact:
>> each value keeps its own exponent, arithmetic is decimal, and it supports
>> signed zero, Inf, and NaN. It is additive, round-trips through a proposed
>> Parquet logical type for cross-engine interop, and would ship config-gated
>> during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior
>> unchanged.
>> >
>> > Additional information is available in the SPIP document:
>> >
>> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
>> >
>> > Please provide your feedback on the approach, scope, and the proposal
>> described in the document.
>> >
>> > Thank you!
>> >
>> > Best,
>> > Uroš
>> >
>> > ---------------------------------------------------------------------
>> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>> >
>> >
>>
>> ---------------------------------------------------------------------
>> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>>
>>
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Gengliang Wang <lt...@gmail.com>.
Thanks for the update. I agree with separating the value domain from
exception handling and the arithmetic implementation.
I support defining DECFLOAT with signed zero, positive and negative
infinity, and quiet NaN. These are numeric values rather than SQL NULL, and
ppreserving them enables faithful interchange and round-tripping across
language and engine boundaries, including between Spark and Python clients.
A finite-only definition would lose that information and would be difficult
to extend compatibly later.
Detailed coercion rules, the execution implementation, external APIs, and
the Parquet encoding still require further review, but they should not
block agreement on the logical value domain.
With this separation clarified, I support moving the SPIP forward.
Best,
Gengliang
On Thu, Sep 17, 2026 at 7:44 AM Uroš Bojanić <ur...@apache.org> wrote:
> Hi all,
>
> We've had some great input and discussion in the DECFLOAT SPIP document (
> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E/edit?tab=t.0#heading=h.zarpams0nxzq),
> so I'm writing an update here to keep the DISCUSS thread active and invite
> further feedback and discussion regarding this SPIP.
>
> It's really important to separate the irreversible decisions from the
> reversible ones. The one-way door here is the value domain - DECFLOAT
> should carry IEEE 754's non-finite values (+Inf, -Inf, NaN,
> signed zero), the same way DOUBLE already does. Digit count (e.g. 34 vs
> 38) and arithmetic implementation can easily be changed later - for
> example, digits can always be widened later and retain full backwards
> compatibility; also, the execution kernel can change under a fixed IEEE
> contract. However, specials are at the opposite end - we can't retrofit
> them onto a shipped finite-only type without breaking it.
>
> Why this matters for big data: decimal pipelines produce unbounded and
> undefined quantities as legitimate business states, not data bugs (e.g. an
> uncapped credit line, a threshold that means "no maximum," a magnitude that
> has run past any finite bound). IEEE already names & defines these
> precisely, and DOUBLE carries them today. DECFLOAT is the base-10 DOUBLE,
> its domain should match accordingly.
>
> A finite-only decimal type can't hold those values, so applications
> disguise them (almost always as NULL). However, NULL in Spark means
> missing/unknown, so "unbounded" and "never loaded" would become the same
> value, and SUM, COUNT, range filters, etc. could no longer telling them
> apart. The loss here runs one way, we can always collapse Inf/NaN to NULL
> if we want SQL null semantics, but we can never rebuild meaning already
> flattened into NULL. This reaches past Spark too, Parquet's decimal-float
> work and existing C++/Rust/Python engines already emit Inf/NaN, so a
> finite-only type can't round-trip that data faithfully across different
> languages / engines.
>
> Let's agree that DECFLOAT's domain is IEEE 754's, non-finite values
> included. Without this, DECFLOAT isn't a better DECIMAL - it's a worse
> DOUBLE.
>
> Best,
> Uroš
>
> On 2026/08/18 16:00:13 Uroš Bojanić wrote:
> > Hi all,
> >
> > I would like to start a discussion on the SPIP to add a new Spark SQL
> data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10
> floating-point decimals with per-value exponents.
> >
> > JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
> >
> > Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type
> (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the
> gap between DECIMAL, which caps at precision 38 with a fixed per-column
> scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact:
> each value keeps its own exponent, arithmetic is decimal, and it supports
> signed zero, Inf, and NaN. It is additive, round-trips through a proposed
> Parquet logical type for cross-engine interop, and would ship config-gated
> during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior
> unchanged.
> >
> > Additional information is available in the SPIP document:
> >
> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
> >
> > Please provide your feedback on the approach, scope, and the proposal
> described in the document.
> >
> > Thank you!
> >
> > Best,
> > Uroš
> >
> > ---------------------------------------------------------------------
> > To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
> >
> >
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
Re: [DISCUSS] SPIP: Add the DECFLOAT data type
Posted by Uroš Bojanić <ur...@apache.org>.
Hi all,
We've had some great input and discussion in the DECFLOAT SPIP document (https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E/edit?tab=t.0#heading=h.zarpams0nxzq), so I'm writing an update here to keep the DISCUSS thread active and invite further feedback and discussion regarding this SPIP.
It's really important to separate the irreversible decisions from the reversible ones. The one-way door here is the value domain - DECFLOAT should carry IEEE 754's non-finite values (+Inf, -Inf, NaN,
signed zero), the same way DOUBLE already does. Digit count (e.g. 34 vs 38) and arithmetic implementation can easily be changed later - for example, digits can always be widened later and retain full backwards compatibility; also, the execution kernel can change under a fixed IEEE contract. However, specials are at the opposite end - we can't retrofit them onto a shipped finite-only type without breaking it.
Why this matters for big data: decimal pipelines produce unbounded and undefined quantities as legitimate business states, not data bugs (e.g. an uncapped credit line, a threshold that means "no maximum," a magnitude that has run past any finite bound). IEEE already names & defines these precisely, and DOUBLE carries them today. DECFLOAT is the base-10 DOUBLE, its domain should match accordingly.
A finite-only decimal type can't hold those values, so applications disguise them (almost always as NULL). However, NULL in Spark means missing/unknown, so "unbounded" and "never loaded" would become the same value, and SUM, COUNT, range filters, etc. could no longer telling them apart. The loss here runs one way, we can always collapse Inf/NaN to NULL if we want SQL null semantics, but we can never rebuild meaning already flattened into NULL. This reaches past Spark too, Parquet's decimal-float work and existing C++/Rust/Python engines already emit Inf/NaN, so a finite-only type can't round-trip that data faithfully across different languages / engines.
Let's agree that DECFLOAT's domain is IEEE 754's, non-finite values included. Without this, DECFLOAT isn't a better DECIMAL - it's a worse DOUBLE.
Best,
Uroš
On 2026/08/18 16:00:13 Uroš Bojanić wrote:
> Hi all,
>
> I would like to start a discussion on the SPIP to add a new Spark SQL data type: DECFLOAT (IEEE 754 decimal64 / decimal128), for base-10 floating-point decimals with per-value exponents.
>
> JIRA ID: https://issues.apache.org/jira/browse/SPARK-58820
>
> Brief summary: DECFLOAT is a new IEEE 754 decimal floating-point type (decimal64 / decimal128, i.e. DECFLOAT(16) / DECFLOAT(34)) that closes the gap between DECIMAL, which caps at precision 38 with a fixed per-column scale, and DOUBLE, whose binary rounding makes fractions like 0.1 inexact: each value keeps its own exponent, arithmetic is decimal, and it supports signed zero, Inf, and NaN. It is additive, round-trips through a proposed Parquet logical type for cross-engine interop, and would ship config-gated during incubation like TIME, leaving existing DECIMAL / DOUBLE behavior unchanged.
>
> Additional information is available in the SPIP document:
> https://docs.google.com/document/d/1qnLXm0ldHSwPSJs_Q5SzGyyTKMMEqeX5rm8Fh4rvV3E
>
> Please provide your feedback on the approach, scope, and the proposal described in the document.
>
> Thank you!
>
> Best,
> Uroš
>
> ---------------------------------------------------------------------
> To unsubscribe e-mail: dev-unsubscribe@spark.apache.org
>
>
---------------------------------------------------------------------
To unsubscribe e-mail: dev-unsubscribe@spark.apache.org