You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "rich7420 (via GitHub)" <gi...@apache.org> on 2026/09/21 10:29:22 UTC
[I] Parquet writer tests can capture a previous write's plan [datafusion-comet]
rich7420 opened a new issue, #6075:
URL: https://github.com/apache/datafusion-comet/issues/6075
### Describe the bug
`CometParquetWriterTestBase.captureWritePlan` registers a listener without draining pending events, then returns once any matching callback arrives. A delayed callback from a setup write can be mistaken for the write under test.
In this [fork CI failure](https://github.com/rich7420/datafusion-comet/actions/runs/35569382593/job/106241292041), `INSERT INTO ... SELECT writes the target table's column names` captured the earlier insert into `comet_rename_source`, where Comet was deliberately disabled. The intended write was into `comet_rename_target`, so the assertion incorrectly reported zero native writers.
### Steps to reproduce
Observed in the Spark 4.0.4 / JDK 21 scans job at `46ef424d0`. The same test passed locally on the unchanged commit when run alone; I have not reproduced the failing schedule locally.
### Expected behavior
Capture the requested write's plan, without accepting callbacks from earlier writes.
### Additional context
The [helper](https://github.com/apache/datafusion-comet/blob/5ca149928f7743bfe7a96feadea5e0f9bed1412f/spark/src/test/scala/org/apache/comet/parquet/CometParquetWriterTestBase.scala#L63-L102) and failing test are unchanged on current upstream main (`5ca149928`).
Drain the listener bus before registration and after the write using `CometListenerBusUtils.waitUntilEmpty`, replacing the polling loop. The columnar-to-row benchmark already uses this pattern.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet writer tests can capture a previous write's plan [datafusion-comet]
Posted by "rich7420 (via GitHub)" <gi...@apache.org>.
rich7420 commented on issue #6075:
URL: https://github.com/apache/datafusion-comet/issues/6075#issuecomment-5759104437
take
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org
Re: [I] Parquet writer tests can capture a previous write's plan [datafusion-comet]
Posted by "sunchao (via GitHub)" <gi...@apache.org>.
sunchao closed issue #6075: Parquet writer tests can capture a previous write's plan
URL: https://github.com/apache/datafusion-comet/issues/6075
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org