You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hawq.apache.org by foyzur <gi...@git.apache.org> on 2016/04/13 03:17:01 UTC
[GitHub] incubator-hawq pull request: HAWQ-665. Dumping memory usage during...
GitHub user foyzur opened a pull request:
https://github.com/apache/incubator-hawq/pull/599
HAWQ-665. Dumping memory usage during runaway query termination.
Previously when we ran out of memory, we logged the memory usage of
all the running queries on the segment where OOM happened. However,
if runaway termination successfully terminates the biggest offender,
before we hit OOM, we did not have any logging mechanism. This left
us with insufficient information for root cause analysis of memory leaks.
This PR is introducing logging of memory usage for only the runaway query.
Note, we do not log usage for all the other queries, like we do for out
of memory. Rather we restrict ourselves to only the biggest violator that
is getting terminated. Moreover, because of the sensitive nature of runaway
termination, we need to terminate as fast as possible, to prevent hitting
an out of memory in other processes. Therefore, we cannot log the full memory
context dump at the time of runaway cleanup. Rather, we attempt to dump
memory usage after the cleanup. This gives us the details of memory accounting p
eak usage for all the operators. For memory context, however, we only dump partial
memory context tree as most of the memory contexts will have been dropped
by then. This is still a valuable piece of information to do root cause
analysis as the memory accounting tree represents the execution plan closely
and helps us pinpoint the operator where we might have excessive memory consumption.
Signed-off-by: Nikos Armenatzoglou <ni...@gmail.com>
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/foyzur/incubator-hawq runaway
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/incubator-hawq/pull/599.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #599
----
commit 8290db59b258446277a4e3e0db1c86d595ddddfe
Author: Foyzur Rahman <fo...@gmail.com>
Date: 2016-04-06T19:47:29Z
HAWQ-665. Dumping memory usage during runaway query termination.
Previously when we ran out of memory, we logged the memory usage of
all the running queries on the segment where OOM happened. However,
if runaway termination successfully terminates the biggest offender,
before we hit OOM, we did not have any logging mechanism. This left
us with insufficient information for root cause analysis of memory leaks.
This PR is introducing logging of memory usage for only the runaway query.
Note, we do not log usage for all the other queries, like we do for out
of memory. Rather we restrict ourselves to only the biggest violator that
is getting terminated. Moreover, because of the sensitive nature of runaway
termination, we need to terminate as fast as possible, to prevent hitting
an out of memory in other processes. Therefore, we cannot log the full memory
context dump at the time of runaway cleanup. Rather, we attempt to dump
memory usage after the cleanup. This gives us the details of memory accounting p
eak usage for all the operators. For memory context, however, we only dump partial
memory context tree as most of the memory contexts will have been dropped
by then. This is still a valuable piece of information to do root cause
analysis as the memory accounting tree represents the execution plan closely
and helps us pinpoint the operator where we might have excessive memory consumption.
Signed-off-by: Nikos Armenatzoglou <ni...@gmail.com>
----
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-665. Dumping memory usage during...
Posted by liming01 <gi...@git.apache.org>.
Github user liming01 commented on the pull request:
https://github.com/apache/incubator-hawq/pull/599#issuecomment-209749629
@foyzur , code merged, please close this PR.
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---
[GitHub] incubator-hawq pull request: HAWQ-665. Dumping memory usage during...
Posted by foyzur <gi...@git.apache.org>.
Github user foyzur closed the pull request at:
https://github.com/apache/incubator-hawq/pull/599
---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastructure@apache.org or file a JIRA ticket
with INFRA.
---