You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2022/09/17 07:35:09 UTC
[GitHub] [arrow-datafusion] liukun4515 opened a new issue, #3523: Decimal issue
liukun4515 opened a new issue, #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
A clear and concise description of what the problem is. Ex. I'm always frustrated when [...]
(This section helps Arrow developers understand the context and *why* for this feature, in addition to the *what*)
I have implemented the feature about decimal in the datafusion. But didn't take care about some special case.
Specially when do the arithmetic operation:
1. loss of the precision, In many case, the intermediate result will be overflow, So we use the float32 or float64 as the intermediate which will cause the the issue of loss of precision
2. overflow/out of the range
We can list all the bug issue in there
We can list all the behavior which should be changed in the datafusion for the decimal
**Describe the solution you'd like**
A clear and concise description of what you want to happen.
**Describe alternatives you've considered**
A clear and concise description of any alternative solutions or features you've considered.
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] kmitchener commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
kmitchener commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1255382492
@liukun4515 @alamb Here are the 2 PRs. The one for arrow-rs is ready for review. The one for DF will remain in draft until arrow-rs 24 is released.
https://github.com/apache/arrow-rs/pull/2770
https://github.com/apache/arrow-datafusion/pull/3590
If/once https://github.com/apache/arrow-rs/pull/2770 is merged in, I'll update the DF PR to use git revs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #3523:
URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2491369491
I think I am going to claim this epic is complete and that DataFusion supports decimals. We can track additional work in other tickets if needed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1250023517
cc @liukun4515 @kmitchener
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1251601672
> Also, how do you deal with the synchronization between the projects? for example, some DF issues are due to the decimal cast code in arrow-rs .. how do we test the fixes? now that we're at the beginning of the dev cycle for 13, can we change DF to pin it to arrow-rs git revs?
@kmitchener I have seen two basic patterns for this:
1. Make a PR that pins to a arrow-rs git revision and complete the changes you want (and this basically leaves the PR as a draft for a while). Example: https://github.com/apache/arrow-datafusion/pull/3386
2. Make a "wrapper" / "adapter" function for the arrow one, adding whatever new special logic is needed and calling into the arrow kernel if needed. For example https://github.com/apache/arrow-datafusion/blob/master/datafusion/physical-expr/src/expressions/binary/adapter.rs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] kmitchener commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
kmitchener commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1254032843
@liukun4515 I'll submit a couple PRs to move the decimal kernels to arrow-rs. I have the code moved locally and it's mostly working -- enough that I'm confident I can complete it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "lostmygithubaccount (via GitHub)" <gi...@apache.org>.
lostmygithubaccount commented on issue #3523:
URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2758717587
just noting this page still says it's experimental: https://datafusion.apache.org/user-guide/sql/data_types.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #3523:
URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2069169178
I wonder if we should claim this ticket is complete? I wonder if there is anything else this ticket is tracking
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1254448138
> I'll submit a couple PRs to move the decimal kernels to arrow-rs. I have the code moved locally and it's mostly working -- enough that I'm confident I can complete it.
I think it is not easy work to move the decimal operation to arrow-rs.
We can not do the comparation and arithmetic operation based on the I128, we should support the decimal256 also.
I try it in my local, it works.
But We need do more work to make it grace, there are some pr about decimal in the arrow-rs you can take a look it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] alamb commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1254052534
That would be awesome @kmitchener -- and a long time coming 👏
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] kmitchener commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
kmitchener commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1250082133
Hm, I see the situation in arrow-rs is complicated re decimals and I want to retract all opinions at this time. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] kmitchener commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
kmitchener commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1250051220
Thanks @liukun4515. It's all the issues listed in #3480 as well:
* #3437
* #3479
Per other discussions, I'd like to help here. I think the general plan should be something like:
* stabilize and add tests
* move the decimal implementation code to arrow-rs
* have the discussion about Decimal128/256 implementation that's going on arrow-rs and agree on path forward for implementation details
* meanwhile in DataFusion, continue to improve use of Decimals in the math functions, sql literals, scientific notation support, etc.
I think there will be some churn .. for example, to fix #3521 , we could use the `num` crate that arrow-rs already uses and fix the implementation of the decimal divide method here. However, that's only a fine short-term solution for stabilization and we'd need to remove the `num` crate again when we finally move the Decimal kernel implementation over to arrow-rs. Is that acceptable?
Also, how do you deal with the synchronization between the projects? for example, some DF issues are due to the decimal cast code in arrow-rs .. how do we test the fixes? now that we're at the beginning of the dev cycle for 13, can we change DF to pin it to arrow-rs git revs?
cc @alamb
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1250486554
> Also, how do you deal with the synchronization between the projects? for example, some DF issues are due to the decimal cast code in arrow-rs .. how do we test the fixes? now that we're at the beginning of the dev cycle for 13, can we change DF to pin it to arrow-rs git revs?
Hi @kmitchener
I think we can resolve the arithmetic operation for decimal128 data type in this sprint, I will do this and move the operation to the arrow-rs.
At the same time, I will fix the issue about decimal in datafusion. From the bug issue list, I think the most issue about decimal operation in the datafusion are about the `divide` arithmetic operation.
And other issue is about feature, like casting between string and decimal which can be implement in the arrow-rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #3523: [EPIC] Add Decimal support
URL: https://github.com/apache/datafusion/issues/3523
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
[GitHub] [arrow-datafusion] liukun4515 commented on issue #3523: Decimal issue
Posted by GitBox <gi...@apache.org>.
liukun4515 commented on issue #3523:
URL: https://github.com/apache/arrow-datafusion/issues/3523#issuecomment-1250487745
@kmitchener if the casting feature is important for you, it's great to add it in the arrow-rs kernel.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "liukun4515 (via GitHub)" <gi...@apache.org>.
liukun4515 commented on issue #3523:
URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2071292078
> I wonder if we should claim this ticket is complete? I wonder if there is anything else this ticket is tracking
I will check above unclosed issue in this week.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@datafusion.apache.org
For additional commands, e-mail: commits-help@datafusion.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #3523:
URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2759323169
Thanks @lostmygithubaccount -- I made a ticket to fix that:
- https://github.com/apache/datafusion/issues/15464
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] [EPIC] Add Decimal support [datafusion]
Posted by "niebayes (via GitHub)" <gi...@apache.org>.
niebayes commented on issue #3523:
URL: https://github.com/apache/datafusion/issues/3523#issuecomment-2760010836
This https://github.com/apache/arrow-rs/issues/7343 might help write tests involving decimal data type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org