You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "siying (via GitHub)" <gi...@apache.org> on 2024/11/06 23:19:24 UTC
[PR] [SPARK-50253] Stream-Stream Join should not fetch checkpoint ID if not supported. [spark]
siying opened a new pull request, #48783:
URL: https://github.com/apache/spark/pull/48783
### What changes were proposed in this pull request?
In stream-stream join, only call getLatestCheckpointInfo() when the format version indicates that checkpoint version is supported.
### Why are the changes needed?
We are hitting flaky test, failing with:
org.apache.spark.sql.streaming.StreamingQueryException: Job aborted due to stage failure: Task 0 in stage 10.0 failed 1 times, most recent failure: Lost task 0.0 in stage 10.0 (TID 12) (1b41214d-1151-4535-aa1b-60e66b974850 executor driver): java.lang.AssertionError: assertion failed: two state stores in a stream-stream joiner don't return the same batch version
at scala.Predef$.assert(Predef.scala:223)
at org.apache.spark.sql.execution.streaming.state.SymmetricHashJoinStateManager.getLatestCheckpointInfo(SymmetricHashJoinStateManager.scala)
This is because when not supported, what is returned might not pass the check and fail the assertion.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Run the flaky test multiple times and make sure it's not failing anymore.
### Was this patch authored or co-authored using generative AI tooling?
No.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50253][SS] Stream-Stream Join should not fetch checkpoint ID if not supported. [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #48783: [SPARK-50253][SS] Stream-Stream Join should not fetch checkpoint ID if not supported.
URL: https://github.com/apache/spark/pull/48783
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
Re: [PR] [SPARK-50253][SS] Stream-Stream Join should not fetch checkpoint ID if not supported. [spark]
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #48783:
URL: https://github.com/apache/spark/pull/48783#issuecomment-2461353433
Thanks! Merging to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org