You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "alamb (via GitHub)" <gi...@apache.org> on 2024/03/14 17:25:07 UTC
[I] Cast support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
alamb opened a new issue, #5508:
URL: https://github.com/apache/arrow-rs/issues/5508
**Is your feature request related to a problem or challenge? Please describe what you are trying to do.**
This is part of the larger project to implement `StringViewArray` -- see https://github.com/apache/arrow-rs/issues/5374
In https://github.com/apache/arrow-rs/pull/5481 we added support for `StringViewArray` and `ByteViewArray`.
This ticket tracks supporting `StringViewArray` and `ByteViewArray` in the `cast` kernel: https://docs.rs/arrow/latest/arrow/compute/kernels/cast/index.html
**Describe the solution you'd like**
Specifically the following conversions should be supported in the cast kernels:
* `StringViewArray` <--> `StringArray`
* `StringViewArray` <--> `LargeStringArray`
And similarly for `Binary`:
* `BinaryViewArray` <--> `BinaryArray`
* `BinaryViewArray` <--> `LargeBinaryArray`
Notes:
1. Good test coverage is the most important part of this ticket
2. I recommend 2 PRs (one for StringView and one for BinaryView) to make review easier -- once we have one then the other will largely follow the example of the first
3. We can track other features in follow on tickets (e.g. DictionaryArray <--> StringViewArray)
**Describe alternatives you've considered**
**Additional context**
<!--
Add any other context or screenshots about the feature request here.
-->
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `cast` kernel support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5508:
URL: https://github.com/apache/arrow-rs/issues/5508#issuecomment-2088296642
> As we know, we can use ViewArray for random access of byte buffers. So, when converting ViewArray to ByteArray, memory copy is unavoidable. I can't come up with a zero-copy way.
> If I implement this operation, I will allocate brand new buffers for the target ByteArray. I want to discuss if there is a better way?
I agree there is unlikely to be a zero copy way. The only potential exception I can imagine is if the underlying buffer was already pre-packed (though this would require all strings to be longer than 12 bytes and in order and contiguous). I suspect the time necessary to detect if this was the case would be substantial
Thus I suggest we start with the simple case (copy to new buffer) and we can optimize later of it turns out that is an important usecase.
I think some part of #5513 might be relevant here (namely the operation to compact the strings) 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `cast` kernel support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb closed issue #5508: `cast` kernel support for `StringViewArray` and `BinaryViewArray`
URL: https://github.com/apache/arrow-rs/issues/5508
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `cast` kernel support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5508:
URL: https://github.com/apache/arrow-rs/issues/5508#issuecomment-2158920045
Sorry @XiangpengHao corrected me -- it seems that @RinChanNOWWW actually implemented both directions
So I think we can claim this ticket is done.
https://github.com/apache/arrow-rs/issues/5861 tracks adding support for dictionary
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `cast` kernel support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5508:
URL: https://github.com/apache/arrow-rs/issues/5508#issuecomment-2079120361
@RinChanNOWWW has a PR to add initial support here: https://github.com/apache/arrow-rs/pull/5686
I updated this ticket with other potential subtasks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `cast` kernel support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
Posted by "RinChanNOWWW (via GitHub)" <gi...@apache.org>.
RinChanNOWWW commented on issue #5508:
URL: https://github.com/apache/arrow-rs/issues/5508#issuecomment-2087927159
I want to discuss about casting from ViewArray to ByteArray.
As we know, we can use ViewArray for random access of byte buffers. So, when converting ViewArray to ByteArray, memory copy is unavoidable. I can't come up with a zero-copy way.
If I implement this operation, I will allocate brand new buffers for the target ByteArray. I want to discuss if there is a better way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] `cast` kernel support for `StringViewArray` and `BinaryViewArray` [arrow-rs]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on issue #5508:
URL: https://github.com/apache/arrow-rs/issues/5508#issuecomment-2158635138
I think what is remaining for this ticket is support for
* `StringViewArray` --> `StringArray` / `LargeStringArray`
* `BinaryViewArray` --> `BinaryArray` / `LargeBinaryArray`
As @RinChanNOWWW says in https://github.com/apache/arrow-rs/issues/5508#issuecomment-2087927159, these casts are going to require copying the string data into the compacted form required of `StringArray`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org