You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by jinxing64 <gi...@git.apache.org> on 2017/07/05 10:25:23 UTC
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
GitHub user jinxing64 opened a pull request:
https://github.com/apache/spark/pull/18541
[SPARK-21315][SQL]Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.
## What changes were proposed in this pull request?
In current code, it is expensive to use `UnboundedFollowingWindowFunctionFrame`, because it is iterating from the start to lower bound every time calling `write` method. When traverse the iterator, it's possible to skip some spilled files thus to save some time.
## How was this patch tested?
Added unit test
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/jinxing64/spark SPARK-21315
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/spark/pull/18541.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #18541
----
commit 442abb32cc08a3311ce6540e7e40c4b62e51a0c1
Author: jinxing <ji...@126.com>
Date: 2017-07-05T06:22:19Z
Skip some spill files when generateIterator(startIndex) in ExternalAppendOnlyUnsafeRowArray.
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79213/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r125632002
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -596,17 +600,37 @@ public long getKeyPrefix() {
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
--- End diff --
Can you update the documentation and add the parameter?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126345627
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,49 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns an iterator starts from startIndex, which will return the rows in the order as
+ * inserted.
*
* It is the caller's responsibility to call `cleanupResources()`
* after consuming this iterator.
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, startIndex);
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() > startIndex) {
+ UnsafeSorterIterator iter = spillWriter.getReader(serializerManager);
+ moveOver(iter, startIndex - i);
+ queue.add(iter);
+ }
+ i += spillWriter.recordsSpilled();
}
if (inMemSorter != null) {
- queue.add(inMemSorter.getSortedIterator());
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, startIndex - i);
+ queue.add(iter);
}
return new ChainedIterator(queue);
}
}
+ private void moveOver(UnsafeSorterIterator iter, int steps)
+ throws IOException {
+ if (steps > 0) {
+ for (int i = 0; i < steps && iter.hasNext(); i++) {
--- End diff --
nit: we should move `iter.hasNext` to the loop body with an assert, to match the original behavior: https://github.com/apache/spark/pull/18541/files#diff-a17b7c6b3521a3e93f958a634501b7f6L219
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r125631732
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -588,6 +588,10 @@ public long getKeyPrefix() {
}
}
+ public UnsafeSorterIterator getIterator() throws IOException {
--- End diff --
Shall we remove this method?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79282 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79282/testReport)** for PR 18541 at commit [`8425aae`](https://github.com/apache/spark/commit/8425aaec77cd6611e88930034bdd285e7e3cac17).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79216 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79216/testReport)** for PR 18541 at commit [`442abb3`](https://github.com/apache/spark/commit/442abb32cc08a3311ce6540e7e40c4b62e51a0c1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79411 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79411/testReport)** for PR 18541 at commit [`a0d90c9`](https://github.com/apache/spark/commit/a0d90c9953798808295ea5fb1f0b69be6247b79f).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by jinxing64 <gi...@git.apache.org>.
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18541
@jiangxb1987
Thanks for approving !
Already updated to PR description.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79411 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79411/testReport)** for PR 18541 at commit [`a0d90c9`](https://github.com/apache/spark/commit/a0d90c9953798808295ea5fb1f0b69be6247b79f).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79444/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/spark/pull/18541
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79213 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79213/testReport)** for PR 18541 at commit [`442abb3`](https://github.com/apache/spark/commit/442abb32cc08a3311ce6540e7e40c4b62e51a0c1).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79444 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79444/testReport)** for PR 18541 at commit [`3ab157d`](https://github.com/apache/spark/commit/3ab157d89f240ac014bc48ef67fdde859e040ab6).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r125631060
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -596,17 +600,37 @@ public long getKeyPrefix() {
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ for (int i = 0; i < startIndex && iter.hasNext(); i++) {
+ iter.loadNext();
+ }
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() <= startIndex) {
+ i += spillWriter.recordsSpilled();
+ continue;
+ } else {
+ UnsafeSorterIterator iter = spillWriter.getReader(serializerManager);
+ for (int j = i; j < startIndex && iter.hasNext(); j++) {
+ iter.loadNext();
+ }
+ queue.add(iter);
+ i += spillWriter.recordsSpilled();
+ }
}
if (inMemSorter != null) {
- queue.add(inMemSorter.getSortedIterator());
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
--- End diff --
This is almost similar to the code in the first if clause. Let's try to get rid of this duplication, and move this logic into a separate function.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79216/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by dongjoon-hyun <gi...@git.apache.org>.
Github user dongjoon-hyun commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126297686
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,50 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns a iterator starts from startIndex, which will return the rows in the order as inserted.
--- End diff --
nit: `a iterator` -> `an iterator`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by kiszk <gi...@git.apache.org>.
Github user kiszk commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126340610
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,51 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns an iterator starts from startIndex, which will return the rows in the order as
+ * inserted.
*
* It is the caller's responsibility to call `cleanupResources()`
* after consuming this iterator.
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, startIndex);
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() <= startIndex) {
+ i += spillWriter.recordsSpilled();
--- End diff --
nit: Can we move lines 611 and 617 after line 618 since they are the same? If we could do this, we can update line 610, too.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79213 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79213/testReport)** for PR 18541 at commit [`442abb3`](https://github.com/apache/spark/commit/442abb32cc08a3311ce6540e7e40c4b62e51a0c1).
* This patch **fails Spark unit tests**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test FAILed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79437/
Test FAILed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126330934
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,51 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns an iterator starts from startIndex, which will return the rows in the order as
+ * inserted.
*
* It is the caller's responsibility to call `cleanupResources()`
* after consuming this iterator.
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, 0, startIndex);
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() <= startIndex) {
+ i += spillWriter.recordsSpilled();
+ } else {
+ UnsafeSorterIterator iter = spillWriter.getReader(serializerManager);
+ moveOver(iter, i, startIndex);
+ queue.add(iter);
+ i += spillWriter.recordsSpilled();
+ }
}
if (inMemSorter != null) {
- queue.add(inMemSorter.getSortedIterator());
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, i, startIndex);
+ queue.add(iter);
}
return new ChainedIterator(queue);
}
}
+ private void moveOver(UnsafeSorterIterator iter, int currentIndex, int targetIndex)
--- End diff --
instead of having 2 parameters, I think we have just have one `int steps`.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79282/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on the issue:
https://github.com/apache/spark/pull/18541
retest this please
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79453/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Test PASSed.
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/79411/
Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by jinxing64 <gi...@git.apache.org>.
Github user jinxing64 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126342950
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,51 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns an iterator starts from startIndex, which will return the rows in the order as
+ * inserted.
*
* It is the caller's responsibility to call `cleanupResources()`
* after consuming this iterator.
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, startIndex);
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() <= startIndex) {
+ i += spillWriter.recordsSpilled();
--- End diff --
Sure
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126331052
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,51 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns an iterator starts from startIndex, which will return the rows in the order as
+ * inserted.
*
* It is the caller's responsibility to call `cleanupResources()`
* after consuming this iterator.
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ moveOver(iter, 0, startIndex);
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() <= startIndex) {
+ i += spillWriter.recordsSpilled();
+ } else {
+ UnsafeSorterIterator iter = spillWriter.getReader(serializerManager);
+ moveOver(iter, i, startIndex);
+ queue.add(iter);
+ i += spillWriter.recordsSpilled();
--- End diff --
shall we break the loop here?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79437 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79437/testReport)** for PR 18541 at commit [`473483d`](https://github.com/apache/spark/commit/473483d541bfe6b18106f9b7f1fd571ff23fad48).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79453 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79453/testReport)** for PR 18541 at commit [`c319dc9`](https://github.com/apache/spark/commit/c319dc98db6cbb7a25b7fed4d8528c8a9c5d1e59).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18541
LGTM except some minor comments, good catch!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by jinxing64 <gi...@git.apache.org>.
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18541
@cloud-fan
Would you mind take a look when you have time :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79437 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79437/testReport)** for PR 18541 at commit [`473483d`](https://github.com/apache/spark/commit/473483d541bfe6b18106f9b7f1fd571ff23fad48).
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by AmplabJenkins <gi...@git.apache.org>.
Github user AmplabJenkins commented on the issue:
https://github.com/apache/spark/pull/18541
Merged build finished. Test PASSed.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79444 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79444/testReport)** for PR 18541 at commit [`3ab157d`](https://github.com/apache/spark/commit/3ab157d89f240ac014bc48ef67fdde859e040ab6).
* This patch **fails due to an unknown error code, -9**.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79453 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79453/testReport)** for PR 18541 at commit [`c319dc9`](https://github.com/apache/spark/commit/c319dc98db6cbb7a25b7fed4d8528c8a9c5d1e59).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79282 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79282/testReport)** for PR 18541 at commit [`8425aae`](https://github.com/apache/spark/commit/8425aaec77cd6611e88930034bdd285e7e3cac17).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by jinxing64 <gi...@git.apache.org>.
Github user jinxing64 commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r126299265
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -589,29 +589,50 @@ public long getKeyPrefix() {
}
/**
- * Returns a iterator, which will return the rows in the order as inserted.
+ * Returns a iterator starts from startIndex, which will return the rows in the order as inserted.
--- End diff --
Sure thing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by cloud-fan <gi...@git.apache.org>.
Github user cloud-fan commented on the issue:
https://github.com/apache/spark/pull/18541
thanks, merging to master!
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by jinxing64 <gi...@git.apache.org>.
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18541
Did a small test:
2000200 rows in the `UnsafeExternalSorter`: 2 spill files(each contains 1000000 rows) and `inMemSorter` contains 200 rows.
I want to target the iterator to index=2000001.
With this change: `getIterator(2000001)`, it will cost almost 0ms~1ms;
Without this change:
`for(int i=0; i<2000001; i++)geIterator().loadNext()`, it will cost 300ms;
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark pull request #18541: [SPARK-21315][SQL]Skip some spill files when gene...
Posted by hvanhovell <gi...@git.apache.org>.
Github user hvanhovell commented on a diff in the pull request:
https://github.com/apache/spark/pull/18541#discussion_r125631249
--- Diff: core/src/main/java/org/apache/spark/util/collection/unsafe/sort/UnsafeExternalSorter.java ---
@@ -596,17 +600,37 @@ public long getKeyPrefix() {
*
* TODO: support forced spilling
*/
- public UnsafeSorterIterator getIterator() throws IOException {
+ public UnsafeSorterIterator getIterator(int startIndex) throws IOException {
if (spillWriters.isEmpty()) {
assert(inMemSorter != null);
- return inMemSorter.getSortedIterator();
+ UnsafeSorterIterator iter = inMemSorter.getSortedIterator();
+ for (int i = 0; i < startIndex && iter.hasNext(); i++) {
+ iter.loadNext();
+ }
+ return iter;
} else {
LinkedList<UnsafeSorterIterator> queue = new LinkedList<>();
+ int i = 0;
for (UnsafeSorterSpillWriter spillWriter : spillWriters) {
- queue.add(spillWriter.getReader(serializerManager));
+ if (i + spillWriter.recordsSpilled() <= startIndex) {
+ i += spillWriter.recordsSpilled();
+ continue;
--- End diff --
Nit: remove continue. It is useless and confusing.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by jinxing64 <gi...@git.apache.org>.
Github user jinxing64 commented on the issue:
https://github.com/apache/spark/pull/18541
@hvanhovell
I refined according to your comments. Please take another look when you have time :)
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] spark issue #18541: [SPARK-21315][SQL]Skip some spill files when generateIte...
Posted by SparkQA <gi...@git.apache.org>.
Github user SparkQA commented on the issue:
https://github.com/apache/spark/pull/18541
**[Test build #79216 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/79216/testReport)** for PR 18541 at commit [`442abb3`](https://github.com/apache/spark/commit/442abb32cc08a3311ce6540e7e40c4b62e51a0c1).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org