You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@opendal.apache.org by "tbEgg (via GitHub)" <gi...@apache.org> on 2024/11/28 09:10:48 UTC

[I] new feature: provides a way to control memory allocation and deallocation [opendal]

tbEgg opened a new issue, #5365:
URL: https://github.com/apache/opendal/issues/5365

   ### Feature Description
   
   OpenDAL currently does not support allowing callers to take control of memory management, which is crucial for programs based on C and C++.
   
   ### Problem and Solution
   
   Enable users to define a custom memory allocator that can replace the default Rust memory allocation mechanisms.
   
   ### Additional Context
   
   _No response_
   
   ### Are you willing to contribute to the development of this feature?
   
   - [X] Yes, I am willing to contribute to the development of this feature.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: provides a way to control memory allocation and deallocation [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5365:
URL: https://github.com/apache/opendal/issues/5365#issuecomment-2505741347

   cc @PragmaTwice @xyjixyjixyji @silver-ymz, is it possible to pass an Allactor from the C side? I haven't seen such an attempt before; perhaps we are the pioneers in this area. It should be exciting.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: provides a way to control memory allocation and deallocation [opendal]

Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #5365:
URL: https://github.com/apache/opendal/issues/5365#issuecomment-2680598707

   > If everyone agree on this I could do this in a few days.
   
   Hi, I suggest that we don't rush this. I want to understand how users will integrate this feature into their own app.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: provides a way to control memory allocation and deallocation [opendal]

Posted by "xyjixyjixyji (via GitHub)" <gi...@apache.org>.
xyjixyjixyji commented on issue #5365:
URL: https://github.com/apache/opendal/issues/5365#issuecomment-2678441892

   I think at this point there is only one solution if we want to use a **custom allocator**.
   
   We expose an API like this
   
   ```Rust
   type AllocFn = unsafe extern "C" fn(size: usize) -> *mut u8;
   type DeallocFn = unsafe extern "C" fn(ptr: *mut u8, size: usize);
   
   #[no_mangle]
   pub extern "C" fn opendal_set_global_allocator(alloc: AllocFn, dealloc: DeallocFn);
   ```
   
   And users call `opendal_set_global_allocator()` before any dynamic memory allocation, we can also force this so that this can only be called once.
   
   If everyone agree on this I could do this in a few days. \cc @Xuanwo @tbEgg 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: provides a way to control memory allocation and deallocation [opendal]

Posted by "xyjixyjixyji (via GitHub)" <gi...@apache.org>.
xyjixyjixyji commented on issue #5365:
URL: https://github.com/apache/opendal/issues/5365#issuecomment-2506735970

   I think it is possible, we pass 2/3 pointers (malloc, free, option<realloc>) to the rust side, and rust side set it as the global allocator.
   
   But user has to guarantee that if this is expected it should be called before any opendal internal function that might have dynamic memory allocation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [I] new feature: provides a way to control memory allocation and deallocation [opendal]

Posted by "silver-ymz (via GitHub)" <gi...@apache.org>.
silver-ymz commented on issue #5365:
URL: https://github.com/apache/opendal/issues/5365#issuecomment-2678790402

   I have 2 questions.
   
   The [GlobalAlloc](https://doc.rust-lang.org/std/alloc/trait.GlobalAlloc.html) trait in standard library is defined as follows:
   
   ```rust
   pub unsafe trait GlobalAlloc {
       // Required methods
       unsafe fn alloc(&self, layout: Layout) -> *mut u8;
       unsafe fn dealloc(&self, ptr: *mut u8, layout: Layout);
   
       // Provided methods
       unsafe fn alloc_zeroed(&self, layout: Layout) -> *mut u8;
       unsafe fn realloc(&self, ptr: *mut u8, layout: Layout, new_size: usize) -> *mut u8;
   }
   ```
   
   ## 1. Should alignment information be exposed to C functions?
   
   Should we expose the alignment parameter to C functions in the following way?
   
   ```rust
   type AllocFn = unsafe extern "C" fn(size: usize, alignment: usize) -> *mut u8;
   type DeallocFn = unsafe extern "C" fn(ptr: *mut u8, size: usize, alignment: usize);
   ```
   
   For example, the GNU malloc function aligns to 8 bytes on 32-bit systems and 16 bytes on 64-bit systems. However, Rust allows more explicit alignment specifications such as `#[repr(align(32))]`. If we do not expose the alignment parameter, it seems that we would need to perform additional work to handle alignment internally.
   
   ## 2. Should `alloc_zeroed` and `realloc` be exposed for further optimization?
   
   Modern allocators often have specialized functions to optimize memory management. For example, [`mimalloc`](https://github.com/microsoft/mimalloc) provides [`mi_zalloc`](https://microsoft.github.io/mimalloc/group__malloc.html#gae6e38c4403247a7b40d80419e093bfb8) and [`mi_realloc`](https://microsoft.github.io/mimalloc/group__malloc.html#ga0621af6a5e3aa384e6a1b548958bf583). Should we expose these functions as optional arguments, allowing them to be nullptr if not implemented?
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@opendal.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org