You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hive.apache.org by shangan <sh...@corp.kaixin001.com> on 2010/08/24 10:35:33 UTC
scrach space grows too fast
what's the parameter "hive.exec.scratchdir" for ? save the job plan ?
the space of this directory is growing too fast and I don't know why files under this directory is not removed after the job is finished. Do they have other functions ?
2010-08-24
shangan
RE: RE: scrach space grows too fast
Posted by Bennie Schut <bs...@ebuddy.com>.
Probably not the end result but the intermediate results are there (which are usually bigger then the end result).
I never used hwi myself.
________________________________
From: shangan [mailto:shangan@corp.kaixin001.com]
Sent: Tuesday, August 24, 2010 11:00 AM
To: hive-user
Subject: Re: RE: scrach space grows too fast
thank you for your so quick reply. You mean the job result is saved under this directory ? This reminds me another question. When I use hwi, the result is saved under the conf directory by default, do you know how the change the default directory, it seems that I didn't find any parameter related to this in the config file.
2010-08-24
________________________________
shangan
________________________________
发件人： Bennie Schut
发送时间： 2010-08-24 16:44:43
收件人： 'hive-user@hadoop.apache.org'
抄送：
主题： RE: scrach space grows too fast
I was actually planning on filing a bug for this. I think it is supposed to remove the temp. hadoop results from this dir but it doesn't seem to do that anymore. For now I use a workaround to simply remove anything from there older then x days however that still means you have results from really large jobs laying around there for x days.
________________________________
From: shangan [mailto:shangan@corp.kaixin001.com]
Sent: Tuesday, August 24, 2010 10:36 AM
To: hive-user
Subject: scrach space grows too fast
what's the parameter "hive.exec.scratchdir" for ? save the job plan ?
the space of this directory is growing too fast and I don't know why files under this directory is not removed after the job is finished. Do they have other functions ?
2010-08-24
________________________________
shangan
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5391 (20100823) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
Re: RE: scrach space grows too fast
Posted by shangan <sh...@corp.kaixin001.com>.
thank you for your so quick reply. You mean the job result is saved under this directory ? This reminds me another question. When I use hwi, the result is saved under the conf directory by default, do you know how the change the default directory, it seems that I didn't find any parameter related to this in the config file.
2010-08-24
shangan
发件人： Bennie Schut
发送时间： 2010-08-24 16:44:43
收件人： 'hive-user@hadoop.apache.org'
抄送：
主题： RE: scrach space grows too fast
I was actually planning on filing a bug for this. I think it is supposed to remove the temp. hadoop results from this dir but it doesn't seem to do that anymore. For now I use a workaround to simply remove anything from there older then x days however that still means you have results from really large jobs laying around there for x days.
From: shangan [mailto:shangan@corp.kaixin001.com]
Sent: Tuesday, August 24, 2010 10:36 AM
To: hive-user
Subject: scrach space grows too fast
what's the parameter "hive.exec.scratchdir" for ? save the job plan ?
the space of this directory is growing too fast and I don't know why files under this directory is not removed after the job is finished. Do they have other functions ?
2010-08-24
shangan
__________ Information from ESET NOD32 Antivirus, version of virus signature database 5391 (20100823) __________
The message was checked by ESET NOD32 Antivirus.
http://www.eset.com
RE: scrach space grows too fast
Posted by Bennie Schut <bs...@ebuddy.com>.
I was actually planning on filing a bug for this. I think it is supposed to remove the temp. hadoop results from this dir but it doesn't seem to do that anymore. For now I use a workaround to simply remove anything from there older then x days however that still means you have results from really large jobs laying around there for x days.
________________________________
From: shangan [mailto:shangan@corp.kaixin001.com]
Sent: Tuesday, August 24, 2010 10:36 AM
To: hive-user
Subject: scrach space grows too fast
what's the parameter "hive.exec.scratchdir" for ? save the job plan ?
the space of this directory is growing too fast and I don't know why files under this directory is not removed after the job is finished. Do they have other functions ?
2010-08-24
________________________________
shangan