You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@hbase.apache.org by iwannaplay games <fu...@gmail.com> on 2012/11/23 07:58:35 UTC
Log files occupy lot of Disk size
Hi,
Everytime i query hbase or hive ,there is a significant growth in my
log files and it consumes lot of space from my hard disk....(Approx 40
gb)
So i stop the cluster ,delete all the logs and free the space and then
again start the cluster to start my work.
Is there any other solution coz i cannot restart the cluster everyday.
Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
Hi,
Everytime i query hbase or hive ,there is a significant growth in my
log files and it consumes lot of space from my hard disk....(Approx 40
gb)
So i stop the cluster ,delete all the logs and free the space and then
again start the cluster to start my work.
Is there any other solution coz i cannot restart the cluster everyday.
Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
Hi,
Everytime i query hbase or hive ,there is a significant growth in my
log files and it consumes lot of space from my hard disk....(Approx 40
gb)
So i stop the cluster ,delete all the logs and free the space and then
again start the cluster to start my work.
Is there any other solution coz i cannot restart the cluster everyday.
Re: Log files occupy lot of Disk size
Posted by yonghu <yo...@gmail.com>.
I think you can set setWriteToWAL() method to false to reduce the
amount of log infor. But you may get risks when your cluster is down.
regards!
yong
On Fri, Nov 23, 2012 at 7:58 AM, iwannaplay games
<fu...@gmail.com> wrote:
> Hi,
>
> Everytime i query hbase or hive ,there is a significant growth in my
> log files and it consumes lot of space from my hard disk....(Approx 40
> gb)
> So i stop the cluster ,delete all the logs and free the space and then
> again start the cluster to start my work.
>
> Is there any other solution coz i cannot restart the cluster everyday.
RE: Log files occupy lot of Disk size
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Every so often I simply wipe out the whole directory content:
/var/log/hadoop-hdfs/*
/var/log/hadoop-0.20-mapreduce/*
I also clean /home/mapred/local/* and /tmp
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, November 23, 2012 2:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Log files occupy lot of Disk size
Lower your log levels if you do not need all that verbosity. You can control log retention, max sizes to keep, max number of files to keep, and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games <fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically
>> checks for the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx
>>> 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and
>>> then again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: Log files occupy lot of Disk size
Posted by shashwat shriparv <dw...@gmail.com>.
When you run a hive query it internally runs lot of map reduce tasks, which
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?
∞
Shashwat Shriparv
On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>> > If i delete the log file without stopping the cluster won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> >> Hi there,
>> >>
>> >> You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> funnlearnforkids@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx 40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Log files occupy lot of Disk size
Posted by shashwat shriparv <dw...@gmail.com>.
When you run a hive query it internally runs lot of map reduce tasks, which
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?
∞
Shashwat Shriparv
On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>> > If i delete the log file without stopping the cluster won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> >> Hi there,
>> >>
>> >> You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> funnlearnforkids@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx 40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Log files occupy lot of Disk size
Posted by shashwat shriparv <dw...@gmail.com>.
When you run a hive query it internally runs lot of map reduce tasks, which
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?
∞
Shashwat Shriparv
On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>> > If i delete the log file without stopping the cluster won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> >> Hi there,
>> >>
>> >> You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> funnlearnforkids@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx 40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Log files occupy lot of Disk size
Posted by shashwat shriparv <dw...@gmail.com>.
When you run a hive query it internally runs lot of map reduce tasks, which
intern generates lot of temporary files, so your disk uses grows, so can
you tell which folder is taking most of the spaces?
∞
Shashwat Shriparv
On Fri, Nov 23, 2012 at 1:24 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Harsh has got a point. I was thinking the same, but then I thought maybe
> you need all these log files. If not then do as Harsh has suggested. And
> deleting log files won't affect your Hdfs working, but it will not write
> logs for any operation until the next Hdfs restart.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
>
>> Lower your log levels if you do not need all that verbosity. You can
>> control log retention, max sizes to keep, max number of files to keep,
>> and logging levels, etc. via each components' log4j.properties file.
>>
>> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
>> <fu...@gmail.com> wrote:
>> > If i delete the log file without stopping the cluster won't it
>> > terminate the session.
>> >
>> >
>> >
>> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> >> Hi there,
>> >>
>> >> You can write a small job or some script which periodically checks
>> for
>> >> the log growth and performs the delete after certain threshold.
>> >>
>> >> Regards,
>> >> Mohammad Tariq
>> >>
>> >>
>> >>
>> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> >> funnlearnforkids@gmail.com> wrote:
>> >>
>> >>> Hi,
>> >>>
>> >>> Everytime i query hbase or hive ,there is a significant growth in my
>> >>> log files and it consumes lot of space from my hard disk....(Approx 40
>> >>> gb)
>> >>> So i stop the cluster ,delete all the logs and free the space and then
>> >>> again start the cluster to start my work.
>> >>>
>> >>> Is there any other solution coz i cannot restart the cluster everyday.
>> >>>
>> >>
>>
>>
>>
>> --
>> Harsh J
>>
>
>
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Harsh has got a point. I was thinking the same, but then I thought maybe
you need all these log files. If not then do as Harsh has suggested. And
deleting log files won't affect your Hdfs working, but it will not write
logs for any operation until the next Hdfs restart.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
> Lower your log levels if you do not need all that verbosity. You can
> control log retention, max sizes to keep, max number of files to keep,
> and logging levels, etc. via each components' log4j.properties file.
>
> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
> <fu...@gmail.com> wrote:
> > If i delete the log file without stopping the cluster won't it
> > terminate the session.
> >
> >
> >
> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> >> Hi there,
> >>
> >> You can write a small job or some script which periodically checks
> for
> >> the log growth and performs the delete after certain threshold.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >>
> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> >> funnlearnforkids@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Everytime i query hbase or hive ,there is a significant growth in my
> >>> log files and it consumes lot of space from my hard disk....(Approx 40
> >>> gb)
> >>> So i stop the cluster ,delete all the logs and free the space and then
> >>> again start the cluster to start my work.
> >>>
> >>> Is there any other solution coz i cannot restart the cluster everyday.
> >>>
> >>
>
>
>
> --
> Harsh J
>
RE: Log files occupy lot of Disk size
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Every so often I simply wipe out the whole directory content:
/var/log/hadoop-hdfs/*
/var/log/hadoop-0.20-mapreduce/*
I also clean /home/mapred/local/* and /tmp
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, November 23, 2012 2:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Log files occupy lot of Disk size
Lower your log levels if you do not need all that verbosity. You can control log retention, max sizes to keep, max number of files to keep, and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games <fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically
>> checks for the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx
>>> 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and
>>> then again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Harsh has got a point. I was thinking the same, but then I thought maybe
you need all these log files. If not then do as Harsh has suggested. And
deleting log files won't affect your Hdfs working, but it will not write
logs for any operation until the next Hdfs restart.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
> Lower your log levels if you do not need all that verbosity. You can
> control log retention, max sizes to keep, max number of files to keep,
> and logging levels, etc. via each components' log4j.properties file.
>
> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
> <fu...@gmail.com> wrote:
> > If i delete the log file without stopping the cluster won't it
> > terminate the session.
> >
> >
> >
> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> >> Hi there,
> >>
> >> You can write a small job or some script which periodically checks
> for
> >> the log growth and performs the delete after certain threshold.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >>
> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> >> funnlearnforkids@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Everytime i query hbase or hive ,there is a significant growth in my
> >>> log files and it consumes lot of space from my hard disk....(Approx 40
> >>> gb)
> >>> So i stop the cluster ,delete all the logs and free the space and then
> >>> again start the cluster to start my work.
> >>>
> >>> Is there any other solution coz i cannot restart the cluster everyday.
> >>>
> >>
>
>
>
> --
> Harsh J
>
RE: Log files occupy lot of Disk size
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Every so often I simply wipe out the whole directory content:
/var/log/hadoop-hdfs/*
/var/log/hadoop-0.20-mapreduce/*
I also clean /home/mapred/local/* and /tmp
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, November 23, 2012 2:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Log files occupy lot of Disk size
Lower your log levels if you do not need all that verbosity. You can control log retention, max sizes to keep, max number of files to keep, and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games <fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically
>> checks for the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx
>>> 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and
>>> then again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
RE: Log files occupy lot of Disk size
Posted by "Kartashov, Andy" <An...@mpac.ca>.
Every so often I simply wipe out the whole directory content:
/var/log/hadoop-hdfs/*
/var/log/hadoop-0.20-mapreduce/*
I also clean /home/mapred/local/* and /tmp
-----Original Message-----
From: Harsh J [mailto:harsh@cloudera.com]
Sent: Friday, November 23, 2012 2:43 AM
To: <us...@hadoop.apache.org>
Subject: Re: Log files occupy lot of Disk size
Lower your log levels if you do not need all that verbosity. You can control log retention, max sizes to keep, max number of files to keep, and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games <fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically
>> checks for the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx
>>> 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and
>>> then again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
NOTICE: This e-mail message and any attachments are confidential, subject to copyright and may be privileged. Any unauthorized use, copying or disclosure is prohibited. If you are not the intended recipient, please delete and contact the sender immediately. Please consider the environment before printing this e-mail. AVIS : le présent courriel et toute pièce jointe qui l'accompagne sont confidentiels, protégés par le droit d'auteur et peuvent être couverts par le secret professionnel. Toute utilisation, copie ou divulgation non autorisée est interdite. Si vous n'êtes pas le destinataire prévu de ce courriel, supprimez-le et contactez immédiatement l'expéditeur. Veuillez penser à l'environnement avant d'imprimer le présent courriel
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Harsh has got a point. I was thinking the same, but then I thought maybe
you need all these log files. If not then do as Harsh has suggested. And
deleting log files won't affect your Hdfs working, but it will not write
logs for any operation until the next Hdfs restart.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
> Lower your log levels if you do not need all that verbosity. You can
> control log retention, max sizes to keep, max number of files to keep,
> and logging levels, etc. via each components' log4j.properties file.
>
> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
> <fu...@gmail.com> wrote:
> > If i delete the log file without stopping the cluster won't it
> > terminate the session.
> >
> >
> >
> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> >> Hi there,
> >>
> >> You can write a small job or some script which periodically checks
> for
> >> the log growth and performs the delete after certain threshold.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >>
> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> >> funnlearnforkids@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Everytime i query hbase or hive ,there is a significant growth in my
> >>> log files and it consumes lot of space from my hard disk....(Approx 40
> >>> gb)
> >>> So i stop the cluster ,delete all the logs and free the space and then
> >>> again start the cluster to start my work.
> >>>
> >>> Is there any other solution coz i cannot restart the cluster everyday.
> >>>
> >>
>
>
>
> --
> Harsh J
>
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Harsh has got a point. I was thinking the same, but then I thought maybe
you need all these log files. If not then do as Harsh has suggested. And
deleting log files won't affect your Hdfs working, but it will not write
logs for any operation until the next Hdfs restart.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 1:12 PM, Harsh J <ha...@cloudera.com> wrote:
> Lower your log levels if you do not need all that verbosity. You can
> control log retention, max sizes to keep, max number of files to keep,
> and logging levels, etc. via each components' log4j.properties file.
>
> On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
> <fu...@gmail.com> wrote:
> > If i delete the log file without stopping the cluster won't it
> > terminate the session.
> >
> >
> >
> > On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> >> Hi there,
> >>
> >> You can write a small job or some script which periodically checks
> for
> >> the log growth and performs the delete after certain threshold.
> >>
> >> Regards,
> >> Mohammad Tariq
> >>
> >>
> >>
> >> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> >> funnlearnforkids@gmail.com> wrote:
> >>
> >>> Hi,
> >>>
> >>> Everytime i query hbase or hive ,there is a significant growth in my
> >>> log files and it consumes lot of space from my hard disk....(Approx 40
> >>> gb)
> >>> So i stop the cluster ,delete all the logs and free the space and then
> >>> again start the cluster to start my work.
> >>>
> >>> Is there any other solution coz i cannot restart the cluster everyday.
> >>>
> >>
>
>
>
> --
> Harsh J
>
Re: Log files occupy lot of Disk size
Posted by Harsh J <ha...@cloudera.com>.
Lower your log levels if you do not need all that verbosity. You can
control log retention, max sizes to keep, max number of files to keep,
and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
<fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically checks for
>> the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and then
>>> again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
Re: Log files occupy lot of Disk size
Posted by Harsh J <ha...@cloudera.com>.
Lower your log levels if you do not need all that verbosity. You can
control log retention, max sizes to keep, max number of files to keep,
and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
<fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically checks for
>> the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and then
>>> again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
Re: Log files occupy lot of Disk size
Posted by Harsh J <ha...@cloudera.com>.
Lower your log levels if you do not need all that verbosity. You can
control log retention, max sizes to keep, max number of files to keep,
and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
<fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically checks for
>> the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and then
>>> again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
Re: Log files occupy lot of Disk size
Posted by Harsh J <ha...@cloudera.com>.
Lower your log levels if you do not need all that verbosity. You can
control log retention, max sizes to keep, max number of files to keep,
and logging levels, etc. via each components' log4j.properties file.
On Fri, Nov 23, 2012 at 12:42 PM, iwannaplay games
<fu...@gmail.com> wrote:
> If i delete the log file without stopping the cluster won't it
> terminate the session.
>
>
>
> On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
>> Hi there,
>>
>> You can write a small job or some script which periodically checks for
>> the log growth and performs the delete after certain threshold.
>>
>> Regards,
>> Mohammad Tariq
>>
>>
>>
>> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
>> funnlearnforkids@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> Everytime i query hbase or hive ,there is a significant growth in my
>>> log files and it consumes lot of space from my hard disk....(Approx 40
>>> gb)
>>> So i stop the cluster ,delete all the logs and free the space and then
>>> again start the cluster to start my work.
>>>
>>> Is there any other solution coz i cannot restart the cluster everyday.
>>>
>>
--
Harsh J
Re: Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
If i delete the log file without stopping the cluster won't it
terminate the session.
On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> Hi there,
>
> You can write a small job or some script which periodically checks for
> the log growth and performs the delete after certain threshold.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> funnlearnforkids@gmail.com> wrote:
>
>> Hi,
>>
>> Everytime i query hbase or hive ,there is a significant growth in my
>> log files and it consumes lot of space from my hard disk....(Approx 40
>> gb)
>> So i stop the cluster ,delete all the logs and free the space and then
>> again start the cluster to start my work.
>>
>> Is there any other solution coz i cannot restart the cluster everyday.
>>
>
Re: Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
If i delete the log file without stopping the cluster won't it
terminate the session.
On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> Hi there,
>
> You can write a small job or some script which periodically checks for
> the log growth and performs the delete after certain threshold.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> funnlearnforkids@gmail.com> wrote:
>
>> Hi,
>>
>> Everytime i query hbase or hive ,there is a significant growth in my
>> log files and it consumes lot of space from my hard disk....(Approx 40
>> gb)
>> So i stop the cluster ,delete all the logs and free the space and then
>> again start the cluster to start my work.
>>
>> Is there any other solution coz i cannot restart the cluster everyday.
>>
>
Re: Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
If i delete the log file without stopping the cluster won't it
terminate the session.
On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> Hi there,
>
> You can write a small job or some script which periodically checks for
> the log growth and performs the delete after certain threshold.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> funnlearnforkids@gmail.com> wrote:
>
>> Hi,
>>
>> Everytime i query hbase or hive ,there is a significant growth in my
>> log files and it consumes lot of space from my hard disk....(Approx 40
>> gb)
>> So i stop the cluster ,delete all the logs and free the space and then
>> again start the cluster to start my work.
>>
>> Is there any other solution coz i cannot restart the cluster everyday.
>>
>
Re: Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
If i delete the log file without stopping the cluster won't it
terminate the session.
On 11/23/12, Mohammad Tariq <do...@gmail.com> wrote:
> Hi there,
>
> You can write a small job or some script which periodically checks for
> the log growth and performs the delete after certain threshold.
>
> Regards,
> Mohammad Tariq
>
>
>
> On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
> funnlearnforkids@gmail.com> wrote:
>
>> Hi,
>>
>> Everytime i query hbase or hive ,there is a significant growth in my
>> log files and it consumes lot of space from my hard disk....(Approx 40
>> gb)
>> So i stop the cluster ,delete all the logs and free the space and then
>> again start the cluster to start my work.
>>
>> Is there any other solution coz i cannot restart the cluster everyday.
>>
>
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
You can write a small job or some script which periodically checks for
the log growth and performs the delete after certain threshold.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
funnlearnforkids@gmail.com> wrote:
> Hi,
>
> Everytime i query hbase or hive ,there is a significant growth in my
> log files and it consumes lot of space from my hard disk....(Approx 40
> gb)
> So i stop the cluster ,delete all the logs and free the space and then
> again start the cluster to start my work.
>
> Is there any other solution coz i cannot restart the cluster everyday.
>
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
You can write a small job or some script which periodically checks for
the log growth and performs the delete after certain threshold.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
funnlearnforkids@gmail.com> wrote:
> Hi,
>
> Everytime i query hbase or hive ,there is a significant growth in my
> log files and it consumes lot of space from my hard disk....(Approx 40
> gb)
> So i stop the cluster ,delete all the logs and free the space and then
> again start the cluster to start my work.
>
> Is there any other solution coz i cannot restart the cluster everyday.
>
Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
Hi,
Everytime i query hbase or hive ,there is a significant growth in my
log files and it consumes lot of space from my hard disk....(Approx 40
gb)
So i stop the cluster ,delete all the logs and free the space and then
again start the cluster to start my work.
Is there any other solution coz i cannot restart the cluster everyday.
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
You can write a small job or some script which periodically checks for
the log growth and performs the delete after certain threshold.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
funnlearnforkids@gmail.com> wrote:
> Hi,
>
> Everytime i query hbase or hive ,there is a significant growth in my
> log files and it consumes lot of space from my hard disk....(Approx 40
> gb)
> So i stop the cluster ,delete all the logs and free the space and then
> again start the cluster to start my work.
>
> Is there any other solution coz i cannot restart the cluster everyday.
>
Re: Log files occupy lot of Disk size
Posted by Mohammad Tariq <do...@gmail.com>.
Hi there,
You can write a small job or some script which periodically checks for
the log growth and performs the delete after certain threshold.
Regards,
Mohammad Tariq
On Fri, Nov 23, 2012 at 12:28 PM, iwannaplay games <
funnlearnforkids@gmail.com> wrote:
> Hi,
>
> Everytime i query hbase or hive ,there is a significant growth in my
> log files and it consumes lot of space from my hard disk....(Approx 40
> gb)
> So i stop the cluster ,delete all the logs and free the space and then
> again start the cluster to start my work.
>
> Is there any other solution coz i cannot restart the cluster everyday.
>
Log files occupy lot of Disk size
Posted by iwannaplay games <fu...@gmail.com>.
Hi,
Everytime i query hbase or hive ,there is a significant growth in my
log files and it consumes lot of space from my hard disk....(Approx 40
gb)
So i stop the cluster ,delete all the logs and free the space and then
again start the cluster to start my work.
Is there any other solution coz i cannot restart the cluster everyday.