You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2023/05/18 11:45:23 UTC
[GitHub] [arrow] pitrou opened a new issue, #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
pitrou opened a new issue, #35665:
URL: https://github.com/apache/arrow/issues/35665
### Describe the bug, including details regarding any error messages, version, and platform.
`DeltaLengthByteArrayEncoder::Put` passes the total desired capacity (`encoded_size_`) to `BufferBuilder::Reserve`. However, `BufferBuilder::Reserve` expects the _additional_ desired capacity. This will therefore request much more memory than desired, and perhaps reallocate more often than expected.
https://github.com/apache/arrow/blob/3e4eaa917fa9b09a923d255adee520aa68a4e78c/cpp/src/parquet/encoding.cc#L2651-L2675
### Component(s)
C++
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] rok commented on issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "rok (via GitHub)" <gi...@apache.org>.
rok commented on issue #35665:
URL: https://github.com/apache/arrow/issues/35665#issuecomment-1553292114
Thanks for catching this @pitrou! I seem to have mixed up `Resize` and `Reserve`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #35665:
URL: https://github.com/apache/arrow/issues/35665#issuecomment-1552942252
Maybe I can take a look and fix them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35665:
URL: https://github.com/apache/arrow/issues/35665#issuecomment-1552998021
@mapleFU Feel free to.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] mapleFU commented on issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "mapleFU (via GitHub)" <gi...@apache.org>.
mapleFU commented on issue #35665:
URL: https://github.com/apache/arrow/issues/35665#issuecomment-1553301877
(Actually it's a bit misleading. C++ `std::vector<T>::reserve` interface works like rok does. But our arrow code use it as incremental...)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou closed issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou closed issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
URL: https://github.com/apache/arrow/issues/35665
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35665:
URL: https://github.com/apache/arrow/issues/35665#issuecomment-1552937596
It would be a good idea to look for similar mistakes in other Parquet encoders/decoders.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [arrow] pitrou commented on issue #35665: `DeltaLengthByteArrayEncoder::Put` may reserve too much space
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on issue #35665:
URL: https://github.com/apache/arrow/issues/35665#issuecomment-1552936890
@felipecrv Do you want to take a look at this?
Also @rok FYI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org