You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by Navis Ryu <na...@nexr.com> on 2015/09/11 07:54:52 UTC
Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
-----------------------------------------------------------
Review request for hive.
Repository: hive-git
Description
-------
More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
Most of the paths contains a suffix of ".pipeout".
Diffs
-----
common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 7ed8e5f
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java bcc66cf
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java cc9df76
Diff: https://reviews.apache.org/r/38292/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Xuefu Zhang <xz...@cloudera.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review98616
-----------------------------------------------------------
common/src/java/org/apache/hive/common/util/ShutdownHookManager.java (line 59)
<https://reviews.apache.org/r/38292/#comment155142>
Side question: I'm wondering why we need two methods which are the same thing. Could we get rid of one?
- Xuefu Zhang
On Sept. 11, 2015, 5:54 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> -----------------------------------------------------------
>
> (Updated Sept. 11, 2015, 5:54 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
> Most of the paths contains a suffix of ".pipeout".
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 7ed8e5f
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java bcc66cf
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java cc9df76
>
> Diff: https://reviews.apache.org/r/38292/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review102728
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (line 302)
<https://reviews.apache.org/r/38292/#comment160469>
the comment is no longer applicable
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (line 307)
<https://reviews.apache.org/r/38292/#comment160470>
the comment is no longer applicable here
common/src/java/org/apache/hadoop/hive/common/FileUtils.java (line 776)
<https://reviews.apache.org/r/38292/#comment160471>
Can you also please add javadoc to this method saying that it registers for deletion on exit with shutdown hook. Also that deleteTmpFile method should be called for temp files created using this to free up the memory.
Can you also please add a unit test case ? It can check if the shutdown hook entry is getting freed on the deletion.
- Thejas Nair
On Oct. 14, 2015, 12:49 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> -----------------------------------------------------------
>
> (Updated Oct. 14, 2015, 12:49 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
> Most of the paths contains a suffix of ".pipeout".
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386
> common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
> common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java fa30f15
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
>
> Diff: https://reviews.apache.org/r/38292/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review102808
-----------------------------------------------------------
Ship it!
Ship It!
- Thejas Nair
On Oct. 15, 2015, 12:35 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> -----------------------------------------------------------
>
> (Updated Oct. 15, 2015, 12:35 a.m.)
>
>
> Review request for hive.
>
>
> Bugs: HIVE-11768
> https://issues.apache.org/jira/browse/HIVE-11768
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
> Most of the paths contains a suffix of ".pipeout".
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386
> common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
> common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java fa30f15
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 92ac209
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
>
> Diff: https://reviews.apache.org/r/38292/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
-----------------------------------------------------------
(Updated 10 15, 2015, 12:35 오전)
Review request for hive.
Bugs: HIVE-11768
https://issues.apache.org/jira/browse/HIVE-11768
Repository: hive-git
Description
-------
More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
Most of the paths contains a suffix of ".pipeout".
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386
common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java fa30f15
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 92ac209
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
Diff: https://reviews.apache.org/r/38292/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
-----------------------------------------------------------
(Updated 10 14, 2015, 12:49 오전)
Review request for hive.
Changes
-------
Addressed comments & fixed test fails
Repository: hive-git
Description
-------
More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
Most of the paths contains a suffix of ".pipeout".
Diffs (updated)
-----
common/src/java/org/apache/hadoop/hive/common/FileUtils.java 7e4f386
common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
common/src/test/org/apache/hive/common/util/TestShutdownHookManager.java fa30f15
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
Diff: https://reviews.apache.org/r/38292/diff/
Testing
-------
Thanks,
Navis Ryu
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Navis Ryu <na...@nexr.com>.
> On 10 13, 2015, 9:35 오후, Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 309
> > <https://reviews.apache.org/r/38292/diff/2/?file=1096900#file1096900line309>
> >
> > I think this method could be re-used in other parts of hive as well. How about adding this a FileUtils.deleteTmpFile and a new FileUtils.createTmpFile method ?
> > Can you also add a unit test for those methods ?
Sure
> On 10 13, 2015, 9:35 오후, Thejas Nair wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java, line 871
> > <https://reviews.apache.org/r/38292/diff/2/?file=1096900#file1096900line871>
> >
> > same as above comment . A method in FileUtils for use throughout hive would be useful.
Consider it done
- Navis
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review102537
-----------------------------------------------------------
On 10 13, 2015, 1:19 오전, Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> -----------------------------------------------------------
>
> (Updated 10 13, 2015, 1:19 오전)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
> Most of the paths contains a suffix of ".pipeout".
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
>
> Diff: https://reviews.apache.org/r/38292/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Thejas Nair <th...@hortonworks.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/#review102537
-----------------------------------------------------------
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (line 309)
<https://reviews.apache.org/r/38292/#comment160236>
I think this method could be re-used in other parts of hive as well. How about adding this a FileUtils.deleteTmpFile and a new FileUtils.createTmpFile method ?
Can you also add a unit test for those methods ?
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java (line 871)
<https://reviews.apache.org/r/38292/#comment160237>
same as above comment . A method in FileUtils for use throughout hive would be useful.
- Thejas Nair
On Oct. 13, 2015, 1:19 a.m., Navis Ryu wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/38292/
> -----------------------------------------------------------
>
> (Updated Oct. 13, 2015, 1:19 a.m.)
>
>
> Review request for hive.
>
>
> Repository: hive-git
>
>
> Description
> -------
>
> More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
> Most of the paths contains a suffix of ".pipeout".
>
>
> Diffs
> -----
>
> common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
> ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1
> service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
> service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
>
> Diff: https://reviews.apache.org/r/38292/diff/
>
>
> Testing
> -------
>
>
> Thanks,
>
> Navis Ryu
>
>
Re: Review Request 38292: HIVE-11768 java.io.DeleteOnExitHook leaks
memory on long running Hive Server2 Instances
Posted by Navis Ryu <na...@nexr.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/38292/
-----------------------------------------------------------
(Updated 10 13, 2015, 1:19 오전)
Review request for hive.
Repository: hive-git
Description
-------
More than 490,000 paths was added to java.io.DeleteOnExitHook on one of our long running HiveServer2 instances,taken up more than 100MB on heap.
Most of the paths contains a suffix of ".pipeout".
Diffs (updated)
-----
common/src/java/org/apache/hive/common/util/ShutdownHookManager.java fd2f20a
ql/src/java/org/apache/hadoop/hive/ql/session/SessionState.java 41b4bb1
service/src/java/org/apache/hive/service/cli/operation/HiveCommandOperation.java 1d1e995
service/src/java/org/apache/hive/service/cli/operation/SQLOperation.java 175348b
Diff: https://reviews.apache.org/r/38292/diff/
Testing
-------
Thanks,
Navis Ryu