You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@avro.apache.org by "siddharthmittal13 (via GitHub)" <gi...@apache.org> on 2026/07/14 10:41:41 UTC

[I] Add optional custom allocator support (CustomAllocator + CountingAllocator) [avro-rs]

siddharthmittal13 opened a new issue, #589:
URL: https://github.com/apache/avro-rs/issues/589

   ### Problem/motivation
   
   `apache-avro` currently offers no first-class hook for customizing or observing heap allocation. The existing `util::max_allocation_bytes` guard only rejects a single oversized length prefix at decode time; it gives no visibility into the actual live/peak memory a workload uses, and no reusable building block for users who want to plug in custom allocation behavior.
   
   ### Proposed solution
   
   Add an **opt-in** cargo feature `custom_allocator` (off by default) exposing a new `apache_avro::alloc` module with two custom `GlobalAlloc` implementations:
   
   1. **`CustomAllocator`** — a minimal allocator that delegates to the `System` allocator. A ready-made base users can install as `#[global_allocator]`   or extend to add their own behavior (logging, pooling, etc.) without reimplementing platform allocation logic.
   2. **`CountingAllocator<A = System>`** — wraps any inner allocator and tracks live bytes (`allocated()`), peak (`peak_allocated()`), cumulative
   (`total_allocated()`), and allocation/deallocation counts, with `reset_peak()` for scoped measurement. `const fn` constructors let it back a `#[global_allocator]` static.
   
   Both are opt-in and inert unless the user installs one as the global allocator, so there is zero impact when the feature is disabled. `CountingAllocator`
   complements `max_allocation_bytes` by measuring real process memory, including pressure from many individually-small allocations.
   
   ### Alternatives considered
   
   - A plain user-side `#[global_allocator]`: works, but every user must re-implement the counting/peak logic; std provides no counting allocator.
   - Re-exporting a third-party allocator (jemalloc/mimalloc): low novelty and users can already do that themselves.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add optional custom allocator support (CustomAllocator + CountingAllocator) [avro-rs]

Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 commented on issue #589:
URL: https://github.com/apache/avro-rs/issues/589#issuecomment-4968360788

   This is not something that should be in this crate. This is just a custom allocator that tracks memory usage and does not interact with the rest of the crate at all.
   If you want to provide this to other users just create your own crate.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add optional custom allocator support (CustomAllocator + CountingAllocator) [avro-rs]

Posted by "Kriskras99 (via GitHub)" <gi...@apache.org>.
Kriskras99 closed issue #589: Add optional custom allocator support (CustomAllocator + CountingAllocator)
URL: https://github.com/apache/avro-rs/issues/589


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] Add optional custom allocator support (CustomAllocator + CountingAllocator) [avro-rs]

Posted by "siddharthmittal13 (via GitHub)" <gi...@apache.org>.
siddharthmittal13 commented on issue #589:
URL: https://github.com/apache/avro-rs/issues/589#issuecomment-4968415236

   > This is not something that should be in this crate. This is just a custom allocator that tracks memory usage and does not interact with the rest of the crate at all. If you want to provide this to other users just create your own crate.
   
   Tracking custom memory was an add on feature but we faced issue since we wanted to use our own custom allocator to go through and not the default memory manager thought would be beneficial.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: issues-unsubscribe@avro.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org