You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/08/17 14:05:40 UTC

[I] Implement zero-copy FFI transfer of batches from JVM to Native [datafusion-comet]

andygrove opened a new issue, #2171:
URL: https://github.com/apache/datafusion-comet/issues/2171

   ### What is the problem the feature request solves?
   
   Comet doesn't currently follow Arrow best practice when transferring batches from JVM to native code. Ownership is retained by the JVM after batches have been made available in FFI, requiring a copy to be made on the native side. 
   
   It would be better to pass ownership of batches to native rather than make copies.
   
   This design was necessary originally due to the use of mutable Parquet buffers in the `native_comet` Parquet scan.
   
   
   
   
   
   ### Describe the potential solution
   
   _No response_
   
   ### Additional context
   
   _No response_


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Implement zero-copy FFI transfer of batches from JVM to Native [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2171:
URL: https://github.com/apache/datafusion-comet/issues/2171#issuecomment-4548510284

   this is implemented now


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Implement zero-copy FFI transfer of batches from JVM to Native [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2171: Implement zero-copy FFI transfer of batches from JVM to Native
URL: https://github.com/apache/datafusion-comet/issues/2171


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Implement zero-copy FFI transfer of batches from JVM to Native [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2171:
URL: https://github.com/apache/datafusion-comet/issues/2171#issuecomment-3194486905

   Thanks @jatin510. I think the next steps are:
   
   - https://github.com/apache/datafusion-comet/issues/2172
   - https://github.com/apache/datafusion-comet/issues/2177
   - https://github.com/apache/datafusion-comet/issues/2152


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org

Re: [I] Implement zero-copy FFI transfer of batches from JVM to Native [datafusion-comet]

Posted by "jatin510 (via GitHub)" <gi...@apache.org>.
jatin510 commented on issue #2171:
URL: https://github.com/apache/datafusion-comet/issues/2171#issuecomment-3194420683

   i would love to work on this issue @andygrove 
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org