You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "andygrove (via GitHub)" <gi...@apache.org> on 2025/08/13 20:13:57 UTC
[I] Determine why deep copies are needed for native_iceberg_compat scan in some cases [datafusion-comet]
andygrove opened a new issue, #2147:
URL: https://github.com/apache/datafusion-comet/issues/2147
### What is the problem the feature request solves?
If I remove the CopyExecs that perform a deep copy, then the Spark SQL test `filtering ratio policy fallback` fails with memory corruption issues. It is using `native_iceberg_compat` scan, so the issue cannot be related to reuse of mutable buffers in the Parquet path. Presumably it is related to a use-after-free FFI issue.
I plan on debugging this to find the exact root cause.
### Describe the potential solution
_No response_
### Additional context
_No response_
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Determine why deep copies are needed for native_iceberg_compat scan in some cases [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove closed issue #2147: Determine why deep copies are needed for native_iceberg_compat scan in some cases
URL: https://github.com/apache/datafusion-comet/issues/2147
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Determine why deep copies are needed for native_iceberg_compat scan in some cases [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2147:
URL: https://github.com/apache/datafusion-comet/issues/2147#issuecomment-3186142878
So, the `native_iceberg_compat` scan does still create mutable buffers.
```
[info] Cause: org.apache.comet.CometNativeException: oops
[info] at comet::errors::init::{{closure}}(__internal__:0)
[info] at std::panicking::rust_panic_with_hook(__internal__:0)
[info] at std::panicking::begin_panic_handler::{{closure}}(__internal__:0)
[info] at std::sys::backtrace::__rust_end_short_backtrace(__internal__:0)
[info] at __rustc::rust_begin_unwind(__internal__:0)
[info] at core::panicking::panic_fmt(__internal__:0)
[info] at comet::parquet::read::column::TypedColumnReader<T>::new(__internal__:0)
[info] at comet::parquet::read::column::ColumnReader::get(__internal__:0)
[info] at Java_org_apache_comet_parquet_Native_initColumnReader(__internal__:0)
[info] at <unknown>(__internal__:0)
[info] at org.apache.comet.parquet.Native.initColumnReader(Native Method)
[info] at org.apache.comet.parquet.Utils.initColumnReader(Utils.java:156)
[info] at org.apache.comet.parquet.AbstractColumnReader.initNative(AbstractColumnReader.java:127)
[info] at org.apache.comet.parquet.ConstantColumnReader.<init>(ConstantColumnReader.java:76)
[info] at org.apache.comet.parquet.ConstantColumnReader.<init>(ConstantColumnReader.java:51)
[info] at org.apache.comet.parquet.NativeBatchReader.init(NativeBatchReader.java:388)
[info] at org.apache.comet.parquet.CometParquetFileFormat.$anonfun$buildReaderWithPartitionValues$1(CometParquetFileFormat.scala:169)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.org$apache$spark$sql$execution$datasources$FileScanRDD$$anon$$readCurrentFile(FileScanRDD.scala:219)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.nextIterator(FileScanRDD.scala:282)
[info] at org.apache.spark.sql.execution.datasources.FileScanRDD$$anon$1.hasNext(FileScanRDD.scala:131)
[info] at org.apache.spark.sql.comet.CometScanExec$$anon$1.hasNext(CometScanExec.scala:268)
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Determine why deep copies are needed for native_iceberg_compat scan in some cases [datafusion-comet]
Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on issue #2147:
URL: https://github.com/apache/datafusion-comet/issues/2147#issuecomment-3188508069
The conclusion of this investigation is that `native_iceberg_compat` does use mutable Parquet buffers due to its use of `ConstantColumnReader`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org