You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@uniffle.apache.org by "qijiale76 (via GitHub)" <gi...@apache.org> on 2024/04/19 11:33:12 UTC
[PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
qijiale76 opened a new pull request, #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666
<!--
1. Title: [#<issue>] <type>(<scope>): <subject>
Examples:
- "[#123] feat(operator): support xxx"
- "[#233] fix: check null before access result in xxx"
- "[MINOR] refactor: fix typo in variable name"
- "[MINOR] docs: fix typo in README"
- "[#255] test: fix flaky test NameOfTheTest"
Reference: https://www.conventionalcommits.org/en/v1.0.0/
2. Contributor guidelines:
https://github.com/apache/incubator-uniffle/blob/master/CONTRIBUTING.md
3. If the PR is unfinished, please mark this PR as draft.
-->
### What changes were proposed in this pull request?
The current implementation of the SortBufferIterator.getKey() and .getValue() methods in the SortWriteBuffer class assumes that keys and values are always stored within a single buffer. This assumption can lead to runtime exceptions, specifically ArrayIndexOutOfBoundsException, when keys or values span multiple WrappedBuffer instances.
This PR update getKey() to use fetchDataFromBuffers() to retrieve the key data. And this PR update getValue() to first adjust the starting index and offset based on the length of the key and then use fetchDataFromBuffers() to retrieve the value data.
### Why are the changes needed?
Fix: #1341
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Tested manually. I'll submit a new integration test later.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2084292731
> > This pull request will copy data one more time. Will it only copy data when we don't use the combine?
>
> `SortBufferIterator` is only used by `Combiner`. When `Combiner` is `null`, these methods are never called.
OK, I got it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2066417912
## Test Results
2 329 files +29 2 329 suites +29 3h 55m 33s :stopwatch: - 5m 45s
920 tests + 1 917 :white_check_mark: ± 0 1 :zzz: ±0 1 :x: +1 1 :fire: ±0
10 550 runs +55 10 531 :white_check_mark: +53 14 :zzz: ±0 1 :x: +1 4 :fire: +1
For more details on these failures and errors, see [this check](https://github.com/apache/incubator-uniffle/runs/24022156161).
Results for commit b6b6c582. ± Comparison against base commit 45ad0b80.
[test-results]:data:application/gzip;base64,H4sIAO1bImYC/1XMSw6DIBSF4a0Yxh1weWo30whIQqrS8BiZ7r3QgLXD83H5D2TdukR0Hwgl021AMbt03SaHOTm/FwEGlBYq76leTAT39YhZ60ogf/R0r/rrBDu79Q+WEHxoEvJem4A5x232aDEKp7Uq63CpfneP1gPtt82lMpASSmg+EsuFoAyMFkoJGCWWIIEzDBYk5kaj9wdSKydREgEAAA==
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "qijiale76 (via GitHub)" <gi...@apache.org>.
qijiale76 commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2071325477
@Lobo2008 Thanks so much for your review, This bug of overflowed nextSeqNo is discussed in #1418. I'll solve it in that PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2082323050
LGTM+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2084293139
Merged to master & branch 0.9 . Thanks @qijiale76 @Lobo2008 @zhengchenyu
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "qijiale76 (via GitHub)" <gi...@apache.org>.
qijiale76 commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2066390108
@sunshineJK Would you please review this PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi merged PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2081990827
@qijiale76 Can you add a UT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "qijiale76 (via GitHub)" <gi...@apache.org>.
qijiale76 commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2084220209
> This pull request will copy data one more time. Will it only copy data when we don't use the combine?
`SortBufferIterator` is only used by `Combiner`. When `Combiner` is `null`, these methods are never called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2082594217
This pull request will copy data one more time. Will it only copy data when we don't use the combine?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "Lobo2008 (via GitHub)" <gi...@apache.org>.
Lobo2008 commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2069273131
@qijiale76 hi, i made some tests and found out that this PR works fine with input datas-212.txt (212MB) or any file size <= 212MB, but failed with datas-213.txt (213MB ).
here is the exception:
~~~shell
org.apache.hadoop.mapred.MapTask$NewOutputCollector@66971f6b
org.apache.uniffle.common.exception.RssException: Can't support sequence [4097], the max value should be 4095
at org.apache.hadoop.mapreduce.RssMRUtils.getBlockId(RssMRUtils.java:237)
at org.apache.hadoop.mapred.SortWriteBufferManager.createShuffleBlock(SortWriteBufferManager.java:390)
~~~
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "jerqi (via GitHub)" <gi...@apache.org>.
jerqi commented on PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#issuecomment-2066453364
Could you a ut for this case?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org
Re: [PR] [#1341] fix(mr): Fix MR Combiner ArrayIndexOutOfBoundsException Bug. [incubator-uniffle]
Posted by "zhengchenyu (via GitHub)" <gi...@apache.org>.
zhengchenyu commented on code in PR #1666:
URL: https://github.com/apache/incubator-uniffle/pull/1666#discussion_r1582706355
##########
client-mr/core/src/main/java/org/apache/hadoop/mapred/SortWriteBuffer.java:
##########
@@ -341,22 +341,57 @@ public SortBufferIterator(SortWriteBuffer<K, V> sortWriteBuffer) {
this.iterator = sortWriteBuffer.records.iterator();
}
+ private byte[] fetchDataFromBuffers(int startIndex, int offset, int length) {
+ int bufferIndex = startIndex;
+ int currentOffset = offset;
+ int remainingLength = length;
+ byte[] data = new byte[length]; // Create a new array to store the complete data
+ int copyDestPos = 0;
+
+ while (remainingLength > 0) {
+ WrappedBuffer currentBuffer = sortWriteBuffer.buffers.get(bufferIndex);
+ byte[] currentBufferData = currentBuffer.getBuffer();
+ int currentBufferCapacity = currentBuffer.getSize();
+ int copyLength = Math.min(currentBufferCapacity - currentOffset, remainingLength);
+
+ // Copy data from the current buffer to the data array
+ System.arraycopy(currentBufferData, currentOffset, data, copyDestPos, copyLength);
+ remainingLength -= copyLength;
+ copyDestPos += copyLength;
+
+ // Move to the next buffer
+ bufferIndex++;
+ currentOffset = 0; // Start position in the new buffer is 0
+ }
+ return data;
+ }
+
@Override
public DataInputBuffer getKey() {
- SortWriteBuffer.WrappedBuffer keyWrappedBuffer =
- sortWriteBuffer.buffers.get(currentRecord.getKeyIndex());
- byte[] rawData = keyWrappedBuffer.getBuffer();
- keyBuffer.reset(rawData, currentRecord.getKeyOffSet(), currentRecord.getKeyLength());
+ int keyIndex = currentRecord.getKeyIndex();
+ int keyOffset = currentRecord.getKeyOffSet();
+ int keyLength = currentRecord.getKeyLength();
+ byte[] keyData = fetchDataFromBuffers(keyIndex, keyOffset, keyLength);
+
+ // Reset keyBuffer with the assembled complete key and return
+ keyBuffer.reset(keyData, 0, keyLength);
return keyBuffer;
}
@Override
public DataInputBuffer getValue() {
- SortWriteBuffer.WrappedBuffer valueWrappedBuffer =
- sortWriteBuffer.buffers.get(currentRecord.getKeyIndex());
- byte[] rawData = valueWrappedBuffer.getBuffer();
+ int valueStartIndex = currentRecord.getKeyIndex();
int valueOffset = currentRecord.getKeyOffSet() + currentRecord.getKeyLength();
- valueBuffer.reset(rawData, valueOffset, currentRecord.getValueLength());
+ int valueLength = currentRecord.getValueLength();
+
+ // Adjust start index and offset for the start of the value
+ while (valueOffset >= sortWriteBuffer.buffers.get(valueStartIndex).getSize()) {
Review Comment:
Can I integrate this code into fetchDataFromBuffers?
##########
client-mr/core/src/main/java/org/apache/hadoop/mapred/SortWriteBuffer.java:
##########
@@ -341,22 +341,57 @@ public SortBufferIterator(SortWriteBuffer<K, V> sortWriteBuffer) {
this.iterator = sortWriteBuffer.records.iterator();
}
+ private byte[] fetchDataFromBuffers(int startIndex, int offset, int length) {
+ int bufferIndex = startIndex;
Review Comment:
Since startIndex, offset, length only used here, do we need thees variables? Here is extra data copy.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@uniffle.apache.org
For additional commands, e-mail: issues-help@uniffle.apache.org