You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "huberylee (via GitHub)" <gi...@apache.org> on 2024/07/10 10:07:06 UTC
[I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
huberylee opened a new issue, #43205:
URL: https://github.com/apache/arrow/issues/43205
### Describe the enhancement requested
Currently, there is no definitive standard for decimal NaN encoding in the industry, and Arrow does not support decimal NaN values. Is it possible to support decimal NaN values in Arrow, including the Arrow read/write Parquet path and the decimal cast path?
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "zeroshade (via GitHub)" <gi...@apache.org>.
zeroshade commented on issue #43205:
URL: https://github.com/apache/arrow/issues/43205#issuecomment-2221402091
Currently I believe the issue ultimately is that there's no current standard for how you could encode an NaN for decimal128/256 values. In most cases though, it's not *technically* necessary for Arrow to explicitly support it as the values are carried as-is throughout a system. Thus a particular system/group of systems/organization could accept a particular bit-pattern for decimal128/256 to indicate NaN and it should pass through any Arrow-based or Parquet-based libraries unmolested.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #43205:
URL: https://github.com/apache/arrow/issues/43205#issuecomment-3547060995
This issue has been marked as stale because it has had no activity in the past 365 days. Please remove the stale label or comment below, or this issue will be closed in 14 days. If this improvement is still desired but has no current owner, please add the 'Status: needs champion' label.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "huberylee (via GitHub)" <gi...@apache.org>.
huberylee commented on issue #43205:
URL: https://github.com/apache/arrow/issues/43205#issuecomment-2224297057
PostgreSQL supports decimal NaN by using a sign bit in its internal decimal format to mark decimal NaN values.
> In addition to ordinary numeric values, the numeric type has several special values:
> Infinity
> -Infinity
> NaN
https://www.postgresql.org/docs/current/datatype-numeric.html#DATATYPE-NUMERIC-DECIMAL
We can use a similar method as with floating point types to represent NaN values by using an unused value at a specific precision. For details, refer to: https://www3.ntu.edu.sg/home/ehchua/programming/java/datarepresentation.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic commented on issue #43205:
URL: https://github.com/apache/arrow/issues/43205#issuecomment-3921276874
This issue has been closed because there has been no activity for the 14 days since the stale warning. Feel free to reopen if this has been closed in error.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "huberylee (via GitHub)" <gi...@apache.org>.
huberylee commented on issue #43205:
URL: https://github.com/apache/arrow/issues/43205#issuecomment-2221875600
> Currently I believe the issue ultimately is that there's no current standard for how you could encode an NaN for decimal128/256 values. In most cases though, it's not _technically_ necessary for Arrow to explicitly support it as the values are carried as-is throughout a system. Thus a particular system/group of systems/organization could accept a particular bit-pattern for decimal128/256 to indicate NaN and it should pass through any Arrow-based or Parquet-based libraries unmolested.
>
> The only place where it would matter is in the functions for converting to/from float/double explicitly
Is it possible to provide some extension interface somewhere like the Cast function, where the user can decide for themselves whether to convert a specific Decimal value to NaN, or vice versa?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43205:
URL: https://github.com/apache/arrow/issues/43205#issuecomment-2223404568
Is there any example that decimal NaN? I've a quick check in snowflake and it doesn't support that
> Values of infinity and NaN (not-a-number) result in conversion errors.
https://docs.snowflake.com/en/sql-reference/functions/to_decimal
It would be interesting to hear how other system supports this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++][Arrow][Parquet] Are there any plans to support decimal NaN? [arrow]
Posted by "thisisnic (via GitHub)" <gi...@apache.org>.
thisisnic closed issue #43205: [C++][Arrow][Parquet] Are there any plans to support decimal NaN?
URL: https://github.com/apache/arrow/issues/43205
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org