You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@hive.apache.org by ch...@huawei.com on 2011/08/12 21:10:40 UTC
Review Request: Clean up the scratch.dir (tmp/hive-root) while restarting
Hive server.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/
-----------------------------------------------------------
Review request for hive.
Summary
-------
Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
This addresses bug HIVE-2181.
https://issues.apache.org/jira/browse/HIVE-2181
Diffs
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1156493
trunk/conf/hive-default.xml 1156493
trunk/data/conf/hive-site.xml 1156493
trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1156493
trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1156493
Diff: https://reviews.apache.org/r/1481/diff
Testing
-------
Added test case for this scenario.
Thanks,
chinna
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by ch...@huawei.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/
-----------------------------------------------------------
(Updated 2011-09-22 18:26:19.812602)
Review request for hive.
Changes
-------
In testcases server startup is removed and added unit tests for cleanUpScratchDir().
Summary
-------
Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
This addresses bug HIVE-2181.
https://issues.apache.org/jira/browse/HIVE-2181
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/common/ServerUtils.java PRE-CREATION
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1174277
trunk/conf/hive-default.xml 1174277
trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1174277
trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1174277
Diff: https://reviews.apache.org/r/1481/diff
Testing
-------
Added test case for this scenario.
Thanks,
chinna
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by ch...@huawei.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/
-----------------------------------------------------------
(Updated 2011-09-21 13:30:22.283181)
Review request for hive.
Changes
-------
spelling mistake corrected
Summary
-------
Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
This addresses bug HIVE-2181.
https://issues.apache.org/jira/browse/HIVE-2181
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1173539
trunk/conf/hive-default.xml 1173539
trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1173539
trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1173539
Diff: https://reviews.apache.org/r/1481/diff
Testing
-------
Added test case for this scenario.
Thanks,
chinna
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by John Sichi <js...@fb.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/#review1977
-----------------------------------------------------------
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java
<https://reviews.apache.org/r/1481/#comment4467>
SCRATCH needs a T in it (everywhere).
- John
On 2011-09-16 17:30:40, chinna wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/1481/
> -----------------------------------------------------------
>
> (Updated 2011-09-16 17:30:40)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
>
> Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
>
>
> This addresses bug HIVE-2181.
> https://issues.apache.org/jira/browse/HIVE-2181
>
>
> Diffs
> -----
>
> trunk/conf/hive-default.xml 1171488
> trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1171488
> trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1171488
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171488
>
> Diff: https://reviews.apache.org/r/1481/diff
>
>
> Testing
> -------
>
> Added test case for this scenario.
>
>
> Thanks,
>
> chinna
>
>
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by ch...@huawei.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/
-----------------------------------------------------------
(Updated 2011-09-16 17:30:40.980271)
Review request for hive.
Changes
-------
Rebased the patch on latest trunk
Summary
-------
Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
This addresses bug HIVE-2181.
https://issues.apache.org/jira/browse/HIVE-2181
Diffs (updated)
-----
trunk/conf/hive-default.xml 1171488
trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1171488
trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1171488
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1171488
Diff: https://reviews.apache.org/r/1481/diff
Testing
-------
Added test case for this scenario.
Thanks,
chinna
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by ch...@huawei.com.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/
-----------------------------------------------------------
(Updated 2011-08-22 18:23:15.749654)
Review request for hive.
Changes
-------
Fixed review comments and updated the patch.
Summary
-------
Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
This addresses bug HIVE-2181.
https://issues.apache.org/jira/browse/HIVE-2181
Diffs (updated)
-----
trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1158629
trunk/conf/hive-default.xml 1158629
trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1158629
trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1158629
Diff: https://reviews.apache.org/r/1481/diff
Testing
-------
Added test case for this scenario.
Thanks,
chinna
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by ch...@huawei.com.
> On 2011-08-13 14:08:14, M IS wrote:
> > trunk/data/conf/hive-site.xml, line 181
> > <https://reviews.apache.org/r/1481/diff/1/?file=32387#file32387line181>
> >
> > This overrides the value set in hive-default.xml. And also deviates from the existing behavior. So, the property needn't be mentioned in hive-site.xml
This is added for the testing purpose. Now it is removed.
> On 2011-08-13 14:08:14, M IS wrote:
> > trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java, line 713
> > <https://reviews.apache.org/r/1481/diff/1/?file=32388#file32388line713>
> >
> > Why create a reference, when the same can be logged directly and easily readable as well.
Creating a reference for the log message is removed
> On 2011-08-13 14:08:14, M IS wrote:
> > trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java, line 394
> > <https://reviews.apache.org/r/1481/diff/1/?file=32389#file32389line394>
> >
> > There needs to be a test case to test the other use-case also, where the user would have chosen not to set the property to "true".
Added a testcase for other user-case (Test should not clear scratchDir)
- chinna
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/#review1444
-----------------------------------------------------------
On 2011-08-12 19:10:40, chinna wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/1481/
> -----------------------------------------------------------
>
> (Updated 2011-08-12 19:10:40)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
>
> Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
>
>
> This addresses bug HIVE-2181.
> https://issues.apache.org/jira/browse/HIVE-2181
>
>
> Diffs
> -----
>
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1156493
> trunk/conf/hive-default.xml 1156493
> trunk/data/conf/hive-site.xml 1156493
> trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1156493
> trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1156493
>
> Diff: https://reviews.apache.org/r/1481/diff
>
>
> Testing
> -------
>
> Added test case for this scenario.
>
>
> Thanks,
>
> chinna
>
>
Re: Review Request: Clean up the scratch.dir (tmp/hive-root) while
restarting Hive server.
Posted by M IS <mi...@gmail.com>.
-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/1481/#review1444
-----------------------------------------------------------
trunk/data/conf/hive-site.xml
<https://reviews.apache.org/r/1481/#comment3342>
This overrides the value set in hive-default.xml. And also deviates from the existing behavior. So, the property needn't be mentioned in hive-site.xml
trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java
<https://reviews.apache.org/r/1481/#comment3343>
Why create a reference, when the same can be logged directly and easily readable as well.
trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java
<https://reviews.apache.org/r/1481/#comment3344>
There needs to be a test case to test the other use-case also, where the user would have chosen not to set the property to "true".
- M
On 2011-08-12 19:10:40, chinna wrote:
>
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/1481/
> -----------------------------------------------------------
>
> (Updated 2011-08-12 19:10:40)
>
>
> Review request for hive.
>
>
> Summary
> -------
>
> Now queries leaves the map outputs under scratch.dir after execution. If the hive server is stopped we need not keep the stopped server's map oputputs. So whle starting the server we can clear the scratch.dir. This can help in improved disk usage.
>
> Implemented a cleanup method in HiveServer and it will trigger based on the property value "hive.start.cleanup.scrachdir"
>
>
> This addresses bug HIVE-2181.
> https://issues.apache.org/jira/browse/HIVE-2181
>
>
> Diffs
> -----
>
> trunk/common/src/java/org/apache/hadoop/hive/conf/HiveConf.java 1156493
> trunk/conf/hive-default.xml 1156493
> trunk/data/conf/hive-site.xml 1156493
> trunk/service/src/java/org/apache/hadoop/hive/service/HiveServer.java 1156493
> trunk/service/src/test/org/apache/hadoop/hive/service/TestHiveServer.java 1156493
>
> Diff: https://reviews.apache.org/r/1481/diff
>
>
> Testing
> -------
>
> Added test case for this scenario.
>
>
> Thanks,
>
> chinna
>
>