[C++] Shared memory

[Louis C <lc...@outlook.fr>]

Hello,

For IPC purposes, I was interested in "shared memory" to pass data between processes. The idea would be to create a named shared memory zone (not backed by any file), and write Arrow data in it, then read back this data in another process. On linux this would probably correspond to something like calling "shm_open" and then mmap this region, before writing Arrow data in it.
Looking at the docs and the code of Arrow, I think that it is not possible using only Arrow. The only class that comes close to what I would like is "MemoryMappedFile" which is an input/output class to a real file on disk (but mapped in memory at the openning).
Is this correct ? Is there any plan in the future to directly support this feature ?

Best regards,
Louis C.

RE: [C++] Shared memory

[Louis C <lc...@outlook.fr>]

Hi,
Thanks everyone for your answer. I was aware of Plasma and knew it seemed to be discontinued (and I think only working on UNIX ?), so I don't think it is in an usable state.
For UCX, I was not aware this could pass data over shared memory. I will look into it and see if this is usable.
Else I might also implement something on my side or use another library to do what I want.

Thanks,
Louis
________________________________
De : Yibo Cai <yi...@arm.com>
Envoyé : mercredi 20 juillet 2022 04:51
À : user@arrow.apache.org <us...@arrow.apache.org>
Objet : Re: [C++] Shared memory

Hi Louis,

Arrow FlightRPC supports UCX transport, which supports IPC over shared
memory. Please note UCX support is still experimental, with few
documents now.

Yibo

On 7/19/22 16:51, Louis C wrote:
>
> Hello,
>
> For IPC purposes, I was interested in "shared memory" to pass data
> between processes. The idea would be to create a named shared memory
> zone (not backed by any file), and write Arrow data in it, then read
> back this data in another process. On linux this would probably
> correspond to something like calling "shm_open" and then mmap this
> region, before writing Arrow data in it.
> Looking at the docs and the code of Arrow, I think that it is not
> possible using only Arrow. The only class that comes close to what I
> would like is "MemoryMappedFile" which is an input/output class to a
> real file on disk (but mapped in memory at the openning).
> Is this correct ? Is there any plan in the future to directly support
> this feature ?
>
> Best regards,
> Louis C.

Re: [C++] Shared memory

[Yibo Cai <yi...@arm.com>]

Hi Louis,

Arrow FlightRPC supports UCX transport, which supports IPC over shared 
memory. Please note UCX support is still experimental, with few 
documents now.

Yibo

On 7/19/22 16:51, Louis C wrote:
> 
> Hello,
> 
> For IPC purposes, I was interested in "shared memory" to pass data 
> between processes. The idea would be to create a named shared memory 
> zone (not backed by any file), and write Arrow data in it, then read 
> back this data in another process. On linux this would probably 
> correspond to something like calling "shm_open" and then mmap this 
> region, before writing Arrow data in it.
> Looking at the docs and the code of Arrow, I think that it is not 
> possible using only Arrow. The only class that comes close to what I 
> would like is "MemoryMappedFile" which is an input/output class to a 
> real file on disk (but mapped in memory at the openning).
> Is this correct ? Is there any plan in the future to directly support 
> this feature ?
> 
> Best regards,
> Louis C.

Re: [C++] Shared memory

[Kevin Gurney <kg...@mathworks.com>]

Hi Louis,

It sounds like your use case is very similar to the kind of shared memory IPC use cases that the Plasma In-Memory Object Store [1] is/was intended to support.

However, my understanding is that there have been multiple mailing list discussions about Plasma being unmaintained over the last few years (e.g. see [2]). Others in the community may be able to provide more context.

[1] https://arrow.apache.org/docs/python/plasma.html
[2] https://lists.apache.org/thread/8zd8j8l2w0xlwznzjnzd2s228v31ld61

Best Regards,

Kevin Gurney
________________________________
From: Louis C <lc...@outlook.fr>
Sent: Tuesday, July 19, 2022 4:51 AM
To: user@arrow.apache.org <us...@arrow.apache.org>
Subject: [C++] Shared memory

Hello,

For IPC purposes, I was interested in "shared memory" to pass data between processes. The idea would be to create a named shared memory zone (not backed by any file), and write Arrow data in it, then read back this data in another process. On linux this would probably correspond to something like calling "shm_open" and then mmap this region, before writing Arrow data in it.
Looking at the docs and the code of Arrow, I think that it is not possible using only Arrow. The only class that comes close to what I would like is "MemoryMappedFile" which is an input/output class to a real file on disk (but mapped in memory at the openning).
Is this correct ? Is there any plan in the future to directly support this feature ?

Best regards,
Louis C.