You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "Kontinuation (via GitHub)" <gi...@apache.org> on 2024/08/29 08:09:07 UTC
[I] Following the memory management semantics stated in the Arrow C Data Interface Specification [datafusion-comet]
Kontinuation opened a new issue, #885:
URL: https://github.com/apache/datafusion-comet/issues/885
I came through a native memory leak (https://github.com/apache/datafusion-comet/issues/884) caused by the misuse of the [Java Implementation of the Arrow C data interface](https://arrow.apache.org/docs/java/cdata.html). I'm wondering why Datafusion Comet is not following the memory management semantics stated in the [Arrow C Data Interface Specification](https://arrow.apache.org/docs/format/CDataInterface.html#semantics)? The specification says that:
> ## Member allocation
> It is intended for the base structure to be stack- or heap-allocated by the consumer. In this case, the producer API should take a pointer to the consumer-allocated structure.
The base structures (`ArrowSchema` and `ArrowVector`) should be allocated by the consumer of Arrow vectors, the consumer passes 2 memory locations to the producer, and producer fills the base structures with pointers to the buffers allocated by the producer, as well as a release callback function for freeing up the resources allocated by the producer.
Datafusion Comet always allocates base structures in the producer and returns the base structures to the consumer. The producer may have a hard time figuring out when to deallocate the base structures: it has to happen after the consumer moves the Arrow vectors and takes ownership of them:
* On the native side, the allocated base structures were kept in the [`ffi_arrays` field of the execution context](https://github.com/apache/datafusion-comet/blob/0.2.0/native/core/src/execution/jni_api.rs#L311-L312), and will be released when the next batch is produced or when the execution context is released by `Native.releasePlan`.
* On the JVM side, the lifetime of allocated base structures was not taken care of, thus causing native memory leaks.
If we follow the memory management semantics stated by the specification, we can always free the base structures immediately after calling the producer and importing the vector, thus having a simpler model of base structure lifetime, and reducing the cost of adapting the mindset from one memory management semantics to another for contributors familiar with the Arrow C Data Interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Following the memory management semantics stated in the Arrow C Data Interface Specification [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #885:
URL: https://github.com/apache/datafusion-comet/issues/885#issuecomment-2318704751
I think it is because the query execution is always triggered from JVM side (the producer). If the array and schema structures are allocated by native, the process will become:
1. JVM side calls native side to get array and schema structures for new batch
2. JVM fills array and schema structures
3. JVM calls native side again to execute the query
> On the native side, the allocated base structures were kept in the [ffi_arrays field of the execution context](https://github.com/apache/datafusion-comet/blob/0.2.0/native/core/src/execution/jni_api.rs#L311-L312), and will be released when the next batch is produced or when the execution context is released by Native.releasePlan.
For output batch, we can provide array and schema from JVM to native when JVM calls native side to execute the query, and use them for importing. This won't change query process but have additional JNI parameters in above step 3.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Following the memory management semantics stated in the Arrow C Data Interface Specification [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya commented on issue #885:
URL: https://github.com/apache/datafusion-comet/issues/885#issuecomment-2319728224
> * On the JVM side, the lifetime of allocated base structures was not taken care of, thus causing native memory leaks.
I revised the current approach in https://github.com/apache/datafusion-comet/pull/893 for that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Following the memory management semantics stated in the Arrow C Data Interface Specification [datafusion-comet]
Posted by "viirya (via GitHub)" <gi...@apache.org>.
viirya closed issue #885: Following the memory management semantics stated in the Arrow C Data Interface Specification
URL: https://github.com/apache/datafusion-comet/issues/885
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org