You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "goutamadwant (via GitHub)" <gi...@apache.org> on 2026/08/06 02:56:51 UTC
[PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
goutamadwant opened a new pull request, #57809:
URL: https://github.com/apache/spark/pull/57809
### What changes were proposed in this pull request?
Run `SchemaPruning` after `PushVariantIntoScan` in the early scan pushdown batch. This removes variant reconstruction projections that become unnecessary during variant pushdown and allows the `Once` batch to reach the same plan on its first application.
Add a Parquet V1 regression test covering a query that reads `_metadata.file_path` while a VARIANT column is referenced below a nondeterministic filter.
Closes #57659.
### Why are the changes needed?
`PushVariantIntoScan` can make a variant reconstruction projection unnecessary after the earlier schema-pruning passes have completed. Reapplying the optimizer batch then removes that projection, which violates the idempotence requirement for a `Once` batch and causes the reported query to fail during optimization.
Running `SchemaPruning` once after variant pushdown makes the first application reach the stable plan.
### Does this PR introduce _any_ user-facing change?
Yes. Queries that select file metadata while pruning a VARIANT column below a nondeterministic filter no longer fail during optimization with a `Once` batch idempotence error.
### How was this patch tested?
The new regression test was confirmed to fail before the optimizer change and pass afterward.
The following tests and checks passed:
- `./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite -- -z "SPARK-57659"'`
- `./build/sbt 'sql/testOnly org.apache.spark.sql.execution.datasources.parquet.ParquetV1SchemaPruningSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanSuite org.apache.spark.sql.execution.datasources.PushVariantIntoScanVectorizedSuite'` (349 tests)
- `./build/sbt 'sql/scalastyle' 'sql/Test/scalastyle'`
- `git diff --check`
The full `./dev/run-tests` suite was not run locally.
### Was this patch authored or co-authored using generative AI tooling?
AI was used to review the code and understand the existing codebase.
This contribution is my original work, and I license it under the project's open source license.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5537076523
I previously left a comment for this in https://issues.apache.org/jira/browse/SPARK-59171, and left the ticket Unassigned until we figure it out. @goutamadwant Please ping when you see this, so that we can attribute the ticket accordingly!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5506717042
https://issues.apache.org/jira/browse/SPARK-57659 seems unrelated to this PR @goutamadwant, could you please fix the PR title accordingly?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5220832525
The docker integration test failed with an error that seems unrelated:
`[info] Cause: oracle.net.ns.NetException: ORA-12516: Cannot connect to database. Listener at host 10.1.0.191 port 32769 does not have a protocol handler for TCP ready or registered for service freepdb1.`
@uros-b can we ignore and merge it, or do we need to re-run?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5248788471
@felipepessoto @uros-b this does not seem related to changes in this PR. Let me know if there is anything to address. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5270238632
gentle ping @uros-b
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5507044602
We don't have this tracker on JIRA. Which tag we use for GitHub issues? https://github.com/apache/spark/issues/57659
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5541746705
@uros-b @felipepessoto goutamadwant is the user id in Jira. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5211739831
> @uros-b , @goutamadwant any idea why CI is not running?
@uros-b @felipepessoto Thanks for the links. I enabled the fork workflows and followed the steps. I see upstream build check has been detected
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5296624880
@goutamadwant, @uros-b, just checking if we need to re-run CI pipeline? Or just wait for merge?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5507243968
> UPDATE: There is a discussion about it in the email list, no answer yet: "[DISCUSS] Next step for github issues"
Yeah there is some discussion at the moment, we'll see how it evolves over time, but as of now let's stick with Jira
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5523871619
**Merge Summary:**
- merged into master https://github.com/apache/spark/commit/a5a9e7baddbe8109190fa3bac38bd9a8abcb2330
- merged into branch-4.x https://github.com/apache/spark/commit/809b7e7e859d0caa872c230d68464fabbaac2daa
*Posted by `merge_spark_pr.py`*
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5521283152
> Btw, the PR description seems a bit overstated? IIUC, there is no production query failure here because the idempotence throw is gated on Utils.isTesting (only true when SPARK_TESTING or spark.testing is set, so basically Spark unit tests only). In production, the batch still runs once and the plan is correct. The leftover reconstruction Project is wasted work, not wrong results, and it does not fail the query.
>
> I understand this showed up in Delta’s tests because those run in a Spark test JVM, even though Delta’s UPDATE/DELETE uses this plan shape in production too.
>
> So while this is a reasonable fix, I don't think we should proceed with backports to Spark 4.2 & 4.3, as the ticket suggests.
@uros-b thats true. I updated the PR description to clarify the test-only failure and no user-facing correctness impact, and I agree that this should target master only without backports to 4.2 or 4.3. Let me know. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5512433937
Btw, the PR description seems a bit overstated? IIUC, there is no production query failure here because the idempotence throw is gated on Utils.isTesting (only true when SPARK_TESTING or spark.testing is set, so basically Spark unit tests only). In production, the batch still runs once and the plan is correct. The leftover reconstruction Project is wasted work, not wrong results, and it does not fail the query.
I understand this showed up in Delta’s tests because those run in a Spark test JVM, even though Delta’s UPDATE/DELETE uses this plan shape in production too.
So while this is a reasonable fix, I don't think we should proceed with backports to Spark 4.2 & 4.3, as the ticket suggests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5511814330
> https://issues.apache.org/jira/browse/SPARK-57659 seems unrelated to this PR @goutamadwant, could you please fix the PR title accordingly?
@uros-b @felipepessoto done. thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b closed pull request #57809: [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown
URL: https://github.com/apache/spark/pull/57809
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5504753222
@goutamadwant, @uros-b, I think we forgot to merge the PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5202484089
For more information, please see section "Pull request" in https://spark.apache.org/contributing.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5201258273
@uros-b , @goutamadwant any idea why CI is not running?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "uros-b (via GitHub)" <gi...@apache.org>.
uros-b commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5202475816
@goutamadwant @felipepessoto Please try to follow the instructions in https://github.com/apache/spark/pull/57809/checks?check_run_id=92506854083.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "goutamadwant (via GitHub)" <gi...@apache.org>.
goutamadwant commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5504820445
@felipepessoto Thanks for the reminder! I’ve rerun the failed Docker integration job and also the failure was the unrelated Oracle listener error ORA-12516. Thanks!
https://github.com/goutamadwant/spark/actions/runs/31142946517
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-57659][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5507132488
@goutamadwant I created this: https://issues.apache.org/jira/browse/SPARK-59171
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-59171][SQL] Make SchemaPruning idempotent after variant pushdown [spark]
Posted by "felipepessoto (via GitHub)" <gi...@apache.org>.
felipepessoto commented on PR #57809:
URL: https://github.com/apache/spark/pull/57809#issuecomment-5530858275
@goutamadwant do you have a JIRA user id? to assign the fix to you
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org