You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by GitBox <gi...@apache.org> on 2021/10/06 11:16:25 UTC
[GitHub] [arrow-rs] alamb opened a new issue #814: Add Dense/Sparse annotation to DataType::Union
alamb opened a new issue #814:
URL: https://github.com/apache/arrow-rs/issues/814
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
While working on validation for ArrayData (https://github.com/apache/arrow-rs/pull/810), I was trying to validate that the number of `Buffers` passed to a `UnionArray` were accurate.
However, since number of Buffers for a `UnionArray` differs based on if it has a "sparse" or "dense" layout (1 or 2, respectively), the sparseness for the UnionArray is not encoded in the type`DataType::UnionArray`; This means we can't easily validate that the UnionArray has the correct number of buffers for its type
It also means that the UnionArray can not be used without first checking to see how many buffers it has (what type it is)
**Describe the solution you'd like**
I propose changing
```rust
enum DataType {
..
Union(Vec<Field>),
..
}
```
To something like
```rust
enum UnionMode {
Sparse,
Dense
}
enum DataType {
..
Union(UnionMode, Vec<Field>),
..
}
```
which is both consistent with the C++ implementation as well as allows `UnionArray` to be statically typechecked: https://github.com/apache/arrow/blob/661c7d749150905a63dd3b52e0a04dac39030d95/cpp/src/arrow/type.h#L1028
**Describe alternatives you've considered**
Can keep the sparse/dense
**Additional context**
Add any other context or screenshots about the feature request here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #814: Add Dense/Sparse annotation to DataType::Union
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #814:
URL: https://github.com/apache/arrow-rs/issues/814#issuecomment-950316683
Maybe related to https://github.com/apache/arrow-rs/issues/85
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] paddyhoran commented on issue #814: Add Dense/Sparse annotation to DataType::Union
Posted by GitBox <gi...@apache.org>.
paddyhoran commented on issue #814:
URL: https://github.com/apache/arrow-rs/issues/814#issuecomment-952177881
Sorry for taking so long. I modeled the original impl on the C++ implementation, it's possible that the C++ impl added this after I wrote the initial implementation. I agree with the change.
> It may be the case that we simply "wait for arrow2" and not do this ticket
I've been out of the loop, is there a plan for adopting `arrow2`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb closed issue #814: Add Dense/Sparse annotation to DataType::Union
Posted by GitBox <gi...@apache.org>.
alamb closed issue #814:
URL: https://github.com/apache/arrow-rs/issues/814
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #814: Add Dense/Sparse annotation to DataType::Union
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #814:
URL: https://github.com/apache/arrow-rs/issues/814#issuecomment-936031238
@nevi-me and @paddyhoran since it looks like you involved in the initial implementation of `UnionArray` I wonder if you have thoughts.
the arrow2 implementation appears to use a `bool` to encode sparseness: https://github.com/jorgecarleitao/arrow2/blob/55c3f9cf31f8fd0bc9c49f48d6265f07e2367ec8/src/datatypes/mod.rs#L95-L96
cc @jorgecarleitao
It may be the case that we simply "wait for arrow2" and not do this ticket
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow-rs] alamb commented on issue #814: Add Dense/Sparse annotation to DataType::Union
Posted by GitBox <gi...@apache.org>.
alamb commented on issue #814:
URL: https://github.com/apache/arrow-rs/issues/814#issuecomment-952299104
> I've been out of the loop, is there a plan for adopting arrow2?
I would say there are no plans -- lots of hope and ideas though :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org