You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "zuston (via GitHub)" <gi...@apache.org> on 2025/11/19 06:40:59 UTC
[PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
zuston opened a new pull request, #2680:
URL: https://github.com/apache/uniffle/pull/2680
### What changes were proposed in this pull request?
This PR is to fix the potential data mismatch when encountering the duplicate blockIds when the overlapping decompression is enabled.
In the one batch remote fetching, if partial blocks should be filtered out due to duplicating or unneed, we should skip it to inc the `segmentIndex`
### Why are the changes needed?
fix the data mismatch cases, that is found by the uniffle integrity validation mechanism.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Unit tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551433297
One question, do we encounter the similar issue if we don't enable overlapping decompression?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551532622
Overlapping compression is disabled by default, isn't it?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551111511
## Test Results
3 150 files ± 0 3 150 suites ±0 6h 54m 28s ⏱️ + 5m 22s
1 229 tests + 16 1 228 ✅ + 16 1 💤 ±0 0 ❌ ±0
15 584 runs +240 15 569 ✅ +240 15 💤 ±0 0 ❌ ±0
Results for commit 922511ff. ± Comparison against base commit afe1b9a5.
<details>
<summary>This pull request <b>removes</b> 16 and <b>adds</b> 32 tests. <i>Note that renamed tests count towards both.</i></summary>
```
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest1
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest10
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest11
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest12
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest13
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest13b
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest14
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest15
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest16
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest2
…
```
```
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest10{Supplier}[1]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest10{Supplier}[2]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest11{Supplier}[1]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest11{Supplier}[2]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest12{Supplier}[1]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest12{Supplier}[2]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest13b{Supplier}[1]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest13b{Supplier}[2]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest13{Supplier}[1]
org.apache.uniffle.client.impl.ShuffleReadClientImplTest ‑ readTest13{Supplier}[2]
…
```
</details>
[test-results]:data:application/gzip;base64,H4sIAENsHWkC/03MTQ6DIBCG4asY1l04IyPQyzQUISH1p0FYmd69aJWy/J6ZvBtzfrQruzcdUHtr2Jp8rPeQgo5+mbMgl73MlO9x/wBEdc3Hmoz5mfzby793K+C0HzO0BWwISzglpPmoEkl+zlIl6lWxs0oXVNVj11GzTJOPeTCFSADOPVthFXDH1cDN4FAIAOpQAjeklbXs8wWywIhzFAEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551678031
> Overlapping compression is disabled by default, isn't it?
Overlapping compression is enabled by default. Decompression is not enabled
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston merged PR #2680:
URL: https://github.com/apache/uniffle/pull/2680
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551717820
> > Overlapping compression is disabled by default, isn't it?
>
> Overlapping compression is enabled by default. Decompression is not enabled
It means that 0.10.0 won't have the similar issue by default, it's ok for me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "zuston (via GitHub)" <gi...@apache.org>.
zuston commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551460828
> One question, do we encounter the similar issue if we don't enable overlapping decompression?
Maybe existing. Anyway integrity validation could recognize this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#2679] fix(spark): Potential data mismatch on overlapping decompression [uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #2680:
URL: https://github.com/apache/uniffle/pull/2680#issuecomment-3551504209
> > One question, do we encounter the similar issue if we don't enable overlapping decompression?
>
> Maybe existing. Anyway integrity validation could recognize this
OK.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org