You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by "HeartSaVioR (via GitHub)" <gi...@apache.org> on 2023/06/09 07:30:21 UTC
[GitHub] [spark] HeartSaVioR opened a new pull request, #41530: [SPARK-43404][SS][3.4] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
HeartSaVioR opened a new pull request, #41530:
URL: https://github.com/apache/spark/pull/41530
NOTE: This ports back the commit https://github.com/apache/spark/commit/d3b9f4e7b1be7d89466cf80c3e38890c7add7625 (PR https://github.com/apache/spark/pull/41089) to branch-3.4. The cherry-pick is clean.
### What changes were proposed in this pull request?
Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
### Why are the changes needed?
In case of task retry on the same executor, its possible that the original task completed the phase of creating the SST files and uploading them to the object store. In this case, we also might have added an entry to the in-memory map for `versionToRocksDBFiles` for the given version. When the retry task creates the local checkpoint, its possible the file name and size is the same, but the metadata ID embedded within the file may be different. So, when we try to load this version on successful commit, the metadata zip file points to the old SST file which results in a RocksDB mismatch id error.
```
Mismatch in unique ID on table file 24220. Expected: {9692563551998415634,4655083329411385714} Actual: {9692563551998415639,10299185534092933087} in file /local_disk0/spark-f58a741d-576f-400c-9b56-53497745ac01/executor-18e08e59-20e8-4a00-bd7e-94ad4599150b/spark-5d980399-3425-4951-894a-808b943054ea/StateStoreId(opId=2147483648,partId=53,name=default)-d89e082e-4e33-4371-8efd-78d927ad3ba3/workingDir-9928750e-f648-4013-a300-ac96cb6ec139/MANIFEST-024212
```
This change avoids reusing files for the same version on the same host based on the map entries to reduce the chance of running into the error above.
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
Unit test
RocksDBSuite
```
[info] Run completed in 35 seconds, 995 milliseconds.
[info] Total number of tests run: 33
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 33, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #41530: [SPARK-43404][SS][3.4] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #41530:
URL: https://github.com/apache/spark/pull/41530#issuecomment-1584449345
Thanks! Merging to branch-3.4.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #41530: [SPARK-43404][SS][3.4] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #41530:
URL: https://github.com/apache/spark/pull/41530#issuecomment-1584238066
Sure, nice finding! Let me fix the type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR commented on pull request #41530: [SPARK-43404][SS][3.4] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR commented on PR #41530:
URL: https://github.com/apache/spark/pull/41530#issuecomment-1584107947
Since this is a clean cherry-pick, I'll merge once the CI passes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HeartSaVioR closed pull request #41530: [SPARK-43404][SS][3.4] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
Posted by "HeartSaVioR (via GitHub)" <gi...@apache.org>.
HeartSaVioR closed pull request #41530: [SPARK-43404][SS][3.4] Skip reusing sst file for same version of RocksDB state store to avoid id mismatch error
URL: https://github.com/apache/spark/pull/41530
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org