You are viewing a plain text version of this content. The canonical link for it is here.
Posted to notifications@asterixdb.apache.org by "Taewoo Kim (Code Review)" <do...@asterixdb.incubator.apache.org> on 2016/12/18 21:23:46 UTC
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Taewoo Kim has uploaded a new change for review.
https://asterix-gerrit.ics.uci.edu/1395
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
- Properly closes the spilled parition file during the build phase of hash join
so that the file will be deleted after the hash join finishes.
Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
---
M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
1 file changed, 8 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/95/1395/1
diff --git a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
index 0770784..4f85e57 100644
--- a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
+++ b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
@@ -247,10 +247,15 @@
break;
}
- for (int pid = spilledStatus.nextSetBit(0); pid >= 0; pid = spilledStatus.nextSetBit(pid + 1)) {
+ for (int pid = spilledStatus.nextSetBit(0); pid >= 0
+ && pid < numOfPartitions; pid = spilledStatus.nextSetBit(pid + 1)) {
if (bufferManager.getNumTuples(pid) > 0) {
bufferManager.flushPartition(pid, getSpillWriterOrCreateNewOneIfNotExist(pid, whichSide));
bufferManager.clearPartition(pid);
+ }
+ // It doesn't matter whether a spilled partition currently holds a tuple in memory or not.
+ // The file that holds the corresponding spilled partition needs to be closed.
+ if (runFileWriters[pid] != null) {
runFileWriters[pid].close();
}
}
@@ -299,7 +304,8 @@
}
private int selectPartitionsToReload(int freeSpace, int pid) {
- for (int i = spilledStatus.nextSetBit(pid); i >= 0; i = spilledStatus.nextSetBit(i + 1)) {
+ for (int i = spilledStatus.nextSetBit(pid); i >= 0
+ && i < numOfPartitions; i = spilledStatus.nextSetBit(i + 1)) {
assert buildRFWriters[i].getFileSize() > 0 : "How comes a spilled partition have size 0?";
if (freeSpace >= buildRFWriters[i].getFileSize()) {
return i;
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newchange
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
Patch Set 2: Integration-Tests+1
Integration Tests Successful
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1343/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Taewoo Kim (Code Review)" <do...@asterixdb.incubator.apache.org>.
Taewoo Kim has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1:
Triggered build again, but the integration test is not being executed again.
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1:
BAD Compatibility Tests Started https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/67/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Ian Maxon (Code Review)" <do...@asterixdb.incubator.apache.org>.
Ian Maxon has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
Patch Set 2: Code-Review+2
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1: BAD+1
BAD Compatibility Tests Successful
https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/67/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
Patch Set 2:
Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1343/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1: Integration-Tests-1
Integration Tests Failed
https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1340/ : UNSTABLE
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Taewoo Kim (Code Review)" <do...@asterixdb.incubator.apache.org>.
Hello Jenkins,
I'd like you to reexamine a change. Please visit
https://asterix-gerrit.ics.uci.edu/1395
to look at the new patch set (#2).
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
ASTERIXDB-1751: Spilled partition files will be cleaned properly.
- Properly closes the spilled parition file during the build phase of hash join
so that the file will be deleted after the hash join finishes.
Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
---
M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
1 file changed, 8 insertions(+), 2 deletions(-)
git pull ssh://asterix-gerrit.ics.uci.edu:29418/asterixdb refs/changes/95/1395/2
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: newpatchset
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Taewoo Kim (Code Review)" <do...@asterixdb.incubator.apache.org>.
Taewoo Kim has submitted this change and it was merged.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
ASTERIXDB-1751: Spilled partition files will be cleaned properly.
- Properly closes the spilled parition file during the build phase of hash join
so that the file will be deleted after the hash join finishes.
Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Reviewed-on: https://asterix-gerrit.ics.uci.edu/1395
Sonar-Qube: Jenkins <je...@fulliautomatix.ics.uci.edu>
Tested-by: Jenkins <je...@fulliautomatix.ics.uci.edu>
BAD: Jenkins <je...@fulliautomatix.ics.uci.edu>
Integration-Tests: Jenkins <je...@fulliautomatix.ics.uci.edu>
Reviewed-by: Ian Maxon <im...@apache.org>
---
M hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
1 file changed, 8 insertions(+), 2 deletions(-)
Approvals:
Ian Maxon: Looks good to me, approved
Jenkins: Verified; No violations found; No violations found; Verified
diff --git a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
index 0770784..4f85e57 100644
--- a/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
+++ b/hyracks-fullstack/hyracks/hyracks-dataflow-std/src/main/java/org/apache/hyracks/dataflow/std/join/OptimizedHybridHashJoin.java
@@ -247,10 +247,15 @@
break;
}
- for (int pid = spilledStatus.nextSetBit(0); pid >= 0; pid = spilledStatus.nextSetBit(pid + 1)) {
+ for (int pid = spilledStatus.nextSetBit(0); pid >= 0
+ && pid < numOfPartitions; pid = spilledStatus.nextSetBit(pid + 1)) {
if (bufferManager.getNumTuples(pid) > 0) {
bufferManager.flushPartition(pid, getSpillWriterOrCreateNewOneIfNotExist(pid, whichSide));
bufferManager.clearPartition(pid);
+ }
+ // It doesn't matter whether a spilled partition currently holds a tuple in memory or not.
+ // The file that holds the corresponding spilled partition needs to be closed.
+ if (runFileWriters[pid] != null) {
runFileWriters[pid].close();
}
}
@@ -299,7 +304,8 @@
}
private int selectPartitionsToReload(int freeSpace, int pid) {
- for (int i = spilledStatus.nextSetBit(pid); i >= 0; i = spilledStatus.nextSetBit(i + 1)) {
+ for (int i = spilledStatus.nextSetBit(pid); i >= 0
+ && i < numOfPartitions; i = spilledStatus.nextSetBit(i + 1)) {
assert buildRFWriters[i].getFileSize() > 0 : "How comes a spilled partition have size 0?";
if (freeSpace >= buildRFWriters[i].getFileSize()) {
return i;
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: merged
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 3
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3581/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3582/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
Patch Set 2:
BAD Compatibility Tests Started https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/70/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
Patch Set 2: BAD+1
BAD Compatibility Tests Successful
https://asterix-jenkins.ics.uci.edu/job/asterixbad-compat/70/ : SUCCESS
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled parititon files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled parititon files will be cleaned properly.
......................................................................
Patch Set 1:
Integration Tests Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-integration-tests/1340/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 1
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-HasComments: No
Change in asterixdb[master]: ASTERIXDB-1751: Spilled partition files will be cleaned prop...
Posted by "Jenkins (Code Review)" <do...@asterixdb.incubator.apache.org>.
Jenkins has posted comments on this change.
Change subject: ASTERIXDB-1751: Spilled partition files will be cleaned properly.
......................................................................
Patch Set 2:
Build Started https://asterix-jenkins.ics.uci.edu/job/asterix-gerrit-notopic/3585/
--
To view, visit https://asterix-gerrit.ics.uci.edu/1395
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings
Gerrit-MessageType: comment
Gerrit-Change-Id: Ibdf0baa060efa35a173d00e15f0a348937f2e7f5
Gerrit-PatchSet: 2
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Taewoo Kim <wa...@yahoo.com>
Gerrit-Reviewer: Ian Maxon <im...@apache.org>
Gerrit-Reviewer: Jenkins <je...@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Taewoo Kim <wa...@yahoo.com>
Gerrit-HasComments: No