You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "felipecrv (via GitHub)" <gi...@apache.org> on 2024/08/05 22:29:53 UTC
[I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
felipecrv opened a new issue, #43573:
URL: https://github.com/apache/arrow/issues/43573
### Describe the enhancement requested
Builders append one bit to the validity bitmap on every value, but that is unnecessarily slow when casting arrays as casts don't have to transform the validity bitmap in any way.
https://github.com/apache/arrow/pull/43302#discussion_r1686274741
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2757601881
Issue resolved by pull request 44822
https://github.com/apache/arrow/pull/44822
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2283097538
@buaazhwb
> I think the builder->Resize() operation has pre-allocate space for offset and validity buffer. Will it be much slower than use individual offsets and data buffers and copy the validity buffer?
I think this is due to "append bit to bitmap", Copy Bitmap is just like a memcpy, however, append bit to bitmap is a bit-or for everybit, which would be slower
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "CrystalZhou0529 (via GitHub)" <gi...@apache.org>.
CrystalZhou0529 commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2495775404
Hello, since this feature has not been worked on actively for a while, I just opened a PR for it. Please take a look and let me know if you have any suggestions!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU closed issue #43573: [C++] Copy bitmap all at once when casting from string-view to offset string and binary types
URL: https://github.com/apache/arrow/issues/43573
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2294029051
> @felipecrv
>
> Sorry for late replying, should I start this after #43302 is merged ?
Yes. This issue is about improving part of what #43302 does. And I recommend not trying to do it right away because aas @buaazhwb already noticed, it will require building arrays without the array builders and custom code will be needed for classic string arrays and string view arrays.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2283143841
Just go ahead 👍
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "buaazhwb (via GitHub)" <gi...@apache.org>.
buaazhwb commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2362566240
> @buaazhwb the cast is merged, this can be move forward, would you still like to try this?
@mapleFU Yes,I will continue it later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2350724189
@buaazhwb the cast is merged, this can be move forward, would you still like to try this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "felipecrv (via GitHub)" <gi...@apache.org>.
felipecrv commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2271578321
Implementation tip from the linked comment thread:
>> How can I do that while still using the builders? I will create an issue and link it from the comments.
>
> You could instead use individual TypedBufferBuilders for the offsets and data buffers. That shouldn't add a lot of complexity.
@llama90 @mapleFU, any of you interested in implementing this improvement?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "buaazhwb (via GitHub)" <gi...@apache.org>.
buaazhwb commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2282653001
@felipecrv I think the `builder->Resize()` operation has pre-allocate space for offset and validity buffer. Will it be much slower than use individual offsets and data buffers and copy the validity buffer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2283098091
@felipecrv
Sorry for late replying, should I start this after https://github.com/apache/arrow/pull/43302 is merged ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "buaazhwb (via GitHub)" <gi...@apache.org>.
buaazhwb commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2283143123
> @buaazhwb
>
> > I think the builder->Resize() operation has pre-allocate space for offset and validity buffer. Will it be much slower than use individual offsets and data buffers and copy the validity buffer?
>
> I think this is due to "append bit to bitmap", Copy Bitmap is just like a memcpy, however, append bit to bitmap is a bit-or for everybit, which would be slower
@mapleFU got it. It seems like we need to build array data without using builder. Can i have a chance to optimize it after [#43302](https://github.com/apache/arrow/pull/43302) merged?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [C++] Copy bitmap all at once when casting from string-view to offset string and binary types [arrow]
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #43573:
URL: https://github.com/apache/arrow/issues/43573#issuecomment-2757603335
@CrystalZhou0529 would you mind reply a message here? I cannot assign task without a message
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org