You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@datafusion.apache.org by "mizulun (via GitHub)" <gi...@apache.org> on 2026/09/29 11:07:52 UTC

[PR] fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [datafusion-comet]

mizulun opened a new pull request, #6382:
URL: https://github.com/apache/datafusion-comet/pull/6382

   ## Which issue does this PR close?
   
   Closes #6122.
   
   ## Rationale for this change
   
   In `AdaptiveQueryExecSuite`, the test `Reuse the default parallelism in local shuffle read` had its assertion changed from Spark's `assert(localReads.length == 2)` to `== 1` in the 3.5.9, 4.0.4 and 4.1.3 diffs. The next statement reads `localReads(1)`, so the test cannot pass for any length. With one local read it throws `IndexOutOfBoundsException`, and with two the assertion fails.
   
   CI stays green because the test carries `IgnoreComet`, so it only runs when Comet is disabled. That is the Spark-only baseline run. The 4.2.0 diff already keeps Spark's value, and this PR brings the other three diffs in line with it.
   
   ## What changes are included in this PR?
   
   - `dev/diffs/3.5.9.diff`, `dev/diffs/4.0.4.diff`, `dev/diffs/4.1.3.diff`: drop the
     `// Comet shuffle changes shuffle metrics` comment and restore `assert(localReads.length == 2)`.
     The `IgnoreComet` tag on the test stays.
   - Each diff was regenerated as described in `spark-sql-tests.md`: apply it to the Spark tag, edit the source, then run `git diff`. Apart from the removed hunk, the only changes are the line offsets of later hunks in `AdaptiveQueryExecSuite.scala`.
   
   ## How are these changes tested?
   
   For each Spark version, the diff was applied to its tag checkout and the test was run in both modes:
   
       NOLINT_ON_COMPILE=true ENABLE_COMET=<false|true> build/sbt \
         "sql/testOnly org.apache.spark.sql.execution.adaptive.AdaptiveQueryExecSuite -- -z \"Reuse the default parallelism in local shuffle read\""
   
   | Spark | `ENABLE_COMET=false` | `ENABLE_COMET=true` |
   |---|---|---|
   | 3.5.9 | succeeded 1 | ignored 1 |
   | 4.0.4 | succeeded 1 | ignored 1 |
   | 4.1.3 | succeeded 1 | ignored 1 |
   
   The baseline run now exercises the restored assertion, and the Comet run still skips the test through the existing `IgnoreComet` tag.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6382:
URL: https://github.com/apache/datafusion-comet/pull/6382#issuecomment-5915639293

   `dev/diffs/3.4.3.diff` has the same broken edit, and this PR doesn't touch it. In `Reuse the default parallelism in local shuffle read` it changes `assert(localReads.length == 2)` to `== 1` (around line 1681 of the diff on `main`), and `localReads(1)` still follows two lines later. I missed it when I filed #6122 and only listed three diffs. Could we fix 3.4.3 in this PR as well, using the same apply, edit and `git diff` steps from `spark-sql-tests.md` against `v3.4.3`? Then `Closes #6122` won't leave a broken diff behind.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [datafusion-comet]

Posted by "mizulun (via GitHub)" <gi...@apache.org>.
mizulun commented on PR #6382:
URL: https://github.com/apache/datafusion-comet/pull/6382#issuecomment-6079083856

   Sure, I’ll fix the conflicts as soon as possible. Thanks for the reminder!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [datafusion-comet]

Posted by "mizulun (via GitHub)" <gi...@apache.org>.
mizulun commented on PR #6382:
URL: https://github.com/apache/datafusion-comet/pull/6382#issuecomment-5925980684

   Thanks for catching this, @andygrove ! I’ve updated the 3.4.3 diff and pushed the fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove merged PR #6382:
URL: https://github.com/apache/datafusion-comet/pull/6382


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org


Re: [PR] fix: restore localReads assertion in AdaptiveQueryExecSuite diffs [datafusion-comet]

Posted by "andygrove (via GitHub)" <gi...@apache.org>.
andygrove commented on PR #6382:
URL: https://github.com/apache/datafusion-comet/pull/6382#issuecomment-6063492582

   @mizulun could you fix conflicts?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: github-unsubscribe@datafusion.apache.org
For additional commands, e-mail: github-help@datafusion.apache.org