[jira] [Commented] (MNEMONIC-2) Integration Mnemonic and Apache Arrow, Phase 1

["Yanping Wang (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MNEMONIC-2?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15192563#comment-15192563 ] 

Yanping Wang commented on MNEMONIC-2:
-------------------------------------

Move Arrow integration discussion here as part of record. 
===============================

From: Jacques Nadeau [mailto:jacques@apache.org] 
Sent: Friday, March 11, 2016 11:41 AM
To: Wang, Gang1 <ga...@intel.com>
Cc: Jacques Nadeau <ja...@apache.org>; Wang, Yanping <ya...@intel.com>
Subject: Re: [DISCUSS] Mnemonic - Arrow Integration

Yup, that should work. 

On Thu, Mar 10, 2016 at 5:28 PM, Wang, Gang1 <ga...@intel.com> wrote:
Hi Jacques,
 
   Thank you very much for your guidance and your advices are very helpful for me to understand the internals of Apache Arrow, we will follow up and try to create a MnmonicByteBuf similar to the class io.netty.buffer.UnpooledUnsafeDirectByteBuf for class LargeBuffer if I understand correctly, Thanks. 
 
Best Regards
Gary (Wang, Gang)
 
From: Jacques Nadeau [mailto:jacques@apache.org] 
Sent: Thursday, March 10, 2016 4:05 PM
To: Wang, Gang1 <ga...@intel.com>
Cc: Jacques Nadeau <ja...@apache.org>; Wang, Yanping <ya...@intel.com>
Subject: Re: [DISCUSS] Mnemonic - Arrow Integration
 
We should actually link lower. ArrowBuf is a finalized single implementation for performance reasons.  We should stitch in at the constructor level here:
 
https://github.com/apache/arrow/blob/master/java/memory/src/main/java/io/netty/buffer/UnsafeDirectLittleEndian.java
 
This allows us to manage accounting, sharing etc above the physical layer. Note how this class has a number of different representations of data it can wrap including PooledUnsafeDirectByteBuf, LargeBuffer and DuplicatedByteBuf.
 
The main things that we need to have inside the new implementation are: how to release the memory, the starting address and the length. Read and write is done directly with memory addresses.
 
On Thu, Mar 10, 2016 at 3:54 PM, Wang, Gang1 <ga...@intel.com> wrote:
Hi Jacques,
 
After further code investigation, we found the interface BufferAllocator is a better point to back Apache Arrow using Mnemonic infrastructure as an alternatives. 
 
However, the interface of BufferAllocator in turn depends on the class ArrowBuf that is  not an interface so, for the purpose of extension and separating of concerns, is it possible to convert the ArrowBuf to an interface officially for Apache Arrow ?  surely, we can do it if desired,  Thanks.
 
Best Regards
Gary (Wang, Gang)

> Integration Mnemonic and Apache Arrow, Phase 1
> ----------------------------------------------
>
>                 Key: MNEMONIC-2
>                 URL: https://issues.apache.org/jira/browse/MNEMONIC-2
>             Project: Mnemonic
>          Issue Type: Improvement
>            Reporter: Wang, Gang
>              Labels: features
>
> as we have discussed, Mnemonic is able to improve the generality of Apache Arrow, so I create this issue to address this new feature for discussion and task tracking, Thanks.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)