You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@drill.apache.org by adeneche <gi...@git.apache.org> on 2015/09/05 02:11:19 UTC
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
GitHub user adeneche opened a pull request:
https://github.com/apache/drill/pull/147
DRILL-3732: Drill leaks memory if external sort hits out of disk spac…
…e exception
- ExternalSort.mergeAndSpill() cleans all it's data in case an errors occurs while it's spilling to disk
- made BatchGroup AutoCloseable so it can easily be closed with AutoCloseables.close() if an error occurs
- added injection site while External sort is spilling to disk
- added unit test that forces a 2 batch query to spill to disk and injects an exception while it does so
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/adeneche/incubator-drill DRILL-3732
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/drill/pull/147.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #147
----
commit a113d9a5e5d434715159c01e97ff7ffa4bad9b38
Author: adeneche <ad...@gmail.com>
Date: 2015-09-05T00:10:17Z
DRILL-3732: Drill leaks memory if external sort hits out of disk space exception
- ExternalSort.mergeAndSpill() cleans all it's data in case an errors occurs while it's spilling to disk
- made BatchGroup AutoCloseable so it can easily be closed with AutoCloseables.close() if an error occurs
- added injection site while External sort is spilling to disk
- added unit test that forces a 2 batch query to spill to disk and injects an exception while it does so
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
Posted by asfgit <gi...@git.apache.org>.
Github user asfgit closed the pull request at:
https://github.com/apache/drill/pull/147
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
Posted by adeneche <gi...@git.apache.org>.
Github user adeneche commented on a diff in the pull request:
https://github.com/apache/drill/pull/147#discussion_r39662668
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java ---
@@ -514,13 +516,20 @@ public BatchGroup mergeAndSpill(LinkedList<BatchGroup> batchGroups) throws Schem
outputContainer.setRecordCount(count);
newGroup.addBatch(outputContainer);
}
+ injector.injectChecked(context.getExecutionControls(), INTERRUPTION_WHILE_SPILLING, IOException.class);
newGroup.closeOutputStream();
- for (BatchGroup group : batchGroupList) {
- group.cleanup();
- }
- hyperBatch.clear();
+ threw = false;
--- End diff --
`threw = false;` needs to be after any statement that could throw. I will add this in it's comment
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on the pull request:
https://github.com/apache/drill/pull/147#issuecomment-141143211
+1 . Rest looks good. Pls add the comment for the 'threw = false' assignment.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/147#discussion_r39115553
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java ---
@@ -515,13 +517,23 @@ public BatchGroup mergeAndSpill(LinkedList<BatchGroup> batchGroups) throws Schem
outputContainer.setRecordCount(count);
newGroup.addBatch(outputContainer);
}
+ injector.injectChecked(context.getExecutionControls(), INTERRUPTION_WHILE_SPILLING, IOException.class);
newGroup.closeOutputStream();
+ success = true;
+ } catch (IOException e) {
+ throw UserException.resourceError(e)
+ .message("External Sort encountered an error while spilling to disk")
+ .build(logger);
+ } finally {
+ // make sure we properly release merged batch groups
for (BatchGroup group : batchGroupList) {
- group.cleanup();
+ AutoCloseables.close(group, logger);
--- End diff --
If there is an exception during this close operation that would get ignored ...
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/147#discussion_r39660969
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java ---
@@ -506,7 +508,7 @@ public BatchGroup mergeAndSpill(LinkedList<BatchGroup> batchGroups) throws Schem
String outputFile = Joiner.on("/").join(dirs.next(), fileName, spillCount++);
BatchGroup newGroup = new BatchGroup(c1, fs, outputFile, oContext.getAllocator());
-
+ boolean threw = true;
--- End diff --
Can you add comments on what is the significance of this flag ?
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] drill pull request: DRILL-3732: Drill leaks memory if external sor...
Posted by amansinha100 <gi...@git.apache.org>.
Github user amansinha100 commented on a diff in the pull request:
https://github.com/apache/drill/pull/147#discussion_r39662237
--- Diff: exec/java-exec/src/main/java/org/apache/drill/exec/physical/impl/xsort/ExternalSortBatch.java ---
@@ -514,13 +516,20 @@ public BatchGroup mergeAndSpill(LinkedList<BatchGroup> batchGroups) throws Schem
outputContainer.setRecordCount(count);
newGroup.addBatch(outputContainer);
}
+ injector.injectChecked(context.getExecutionControls(), INTERRUPTION_WHILE_SPILLING, IOException.class);
newGroup.closeOutputStream();
- for (BatchGroup group : batchGroupList) {
- group.cleanup();
- }
- hyperBatch.clear();
+ threw = false;
--- End diff --
I think there is an implicit assumption here that threw = false; statement will be the last statement in the try {} block. Is that right ? If so, that could be prone to some future problems if new code gets added here.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---