You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@arrow.apache.org by "pitrou (via GitHub)" <gi...@apache.org> on 2026/07/07 13:57:06 UTC
[PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
pitrou opened a new pull request, #50408:
URL: https://github.com/apache/arrow/pull/50408
### Rationale for this change
Add guidance about handling or avoiding memory bombs in file formats that involve optional compression.
### Are these changes tested?
N/A.
### Are there any user-facing changes?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou merged PR #50408:
URL: https://github.com/apache/arrow/pull/50408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "alamb (via GitHub)" <gi...@apache.org>.
alamb commented on code in PR #50408:
URL: https://github.com/apache/arrow/pull/50408#discussion_r3540057960
##########
docs/source/cpp/security.rst:
##########
@@ -120,6 +120,22 @@ variants, but these typically fall into two categories:
arguments (this is typical of the :ref:`array builders <cpp-api-array-builders>`
and :ref:`buffer builders <cpp-api-buffer-builders>`).
+Controlling and restricting memory allocation
+---------------------------------------------
+
+By construction, many Arrow C++ APIs can allocate large amounts of memory, depending
+on their input parameters. Arrow C++ allows customizing the memory allocator for
+such large data requests through the :ref:`MemoryPool <cpp_memory_pool>` interface.
+
+You can therefore implement a MemoryPool class enforcing the restrictions
+of your choise (for example to limit the total number of allocated bytes), and pass
Review Comment:
```suggestion
of your choice (for example to limit the total number of allocated bytes), and pass
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50408:
URL: https://github.com/apache/arrow/pull/50408#issuecomment-4912179333
Also @zanmato1984 @HuaHuaY FYI
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #50408:
URL: https://github.com/apache/arrow/pull/50408#issuecomment-4921855978
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6d06317ab1fe3d5be478e4de282365d7021f073b.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/86046553159) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50408:
URL: https://github.com/apache/arrow/pull/50408#issuecomment-4905047000
Preview docs:
* https://s3.amazonaws.com/arrow-data/pr_docs/50408/format/Security.html
* https://s3.amazonaws.com/arrow-data/pr_docs/50408/cpp/security.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "paleolimbot (via GitHub)" <gi...@apache.org>.
paleolimbot commented on code in PR #50408:
URL: https://github.com/apache/arrow/pull/50408#discussion_r3537544836
##########
docs/source/cpp/security.rst:
##########
@@ -120,6 +120,22 @@ variants, but these typically fall into two categories:
arguments (this is typical of the :ref:`array builders <cpp-api-array-builders>`
and :ref:`buffer builders <cpp-api-buffer-builders>`).
+Controlling and restricting memory allocation
+---------------------------------------------
+
+By construction, many Arrow C++ APIs can allocate large amounts of memory, depending
+on their input parameters. Arrow C++ allows customizing the memory allocator for
+such large data areads through the :ref:`MemoryPool <cpp_memory_pool>` interface.
Review Comment:
```suggestion
such large data requests through the :ref:`MemoryPool <cpp_memory_pool>` interface.
```
##########
docs/source/format/Security.rst:
##########
@@ -180,6 +180,11 @@ their own risks. For example, buffer offsets and sizes encoded in IPC messages
may be out of bounds for the IPC stream; Flatbuffers-encoded metadata payloads
may carry incorrect offsets pointing outside of the designated metadata area.
+Besides, the IPC format provides optional :ref:`buffer compression <buffer-compression>`
Review Comment:
```suggestion
In addition, the IPC format provides optional :ref:`buffer compression <buffer-compression>`
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #50408:
URL: https://github.com/apache/arrow/pull/50408#issuecomment-4904635068
Revision: fbd25003e16e967d63ee92a93d6db257528cc5ce
Submitted crossbow builds: [ursacomputing/crossbow @ actions-1c56e5737d](https://github.com/ursacomputing/crossbow/branches/all?query=actions-1c56e5737d)
|Task|Status|
|----|------|
|preview-docs|[](https://github.com/ursacomputing/crossbow/actions/runs/28871982903/job/85636922282)|
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "pitrou (via GitHub)" <gi...@apache.org>.
pitrou commented on PR #50408:
URL: https://github.com/apache/arrow/pull/50408#issuecomment-4904610165
@github-actions crossbow submit preview-docs
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] GH-50313: [C++][Docs] Add guidance about memory bombs [arrow]
Posted by "conbench-apache-arrow[bot] (via GitHub)" <gi...@apache.org>.
conbench-apache-arrow[bot] commented on PR #50408:
URL: https://github.com/apache/arrow/pull/50408#issuecomment-4921879237
After merging your PR, Conbench analyzed the 4 benchmarking runs that have been run so far on merge-commit 6d06317ab1fe3d5be478e4de282365d7021f073b.
There were no benchmark performance regressions. 🎉
The [full Conbench report](https://github.com/apache/arrow/runs/86047056798) has more details.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@arrow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org