You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Arko Provo Mukherjee <ar...@gmail.com> on 2013/02/13 20:32:20 UTC
Managing space in Master Node
Hello Gurus,
I am managing a Hadoop Cluster to run some experiments.
The issue I am continuously facing is that the Master Node runs out of
disk space due to logs and data files.
I can monitor and delete log files. However, I cannot delete the HDFS data.
Thus, is there a way to force Hadoop not to save any HDFS data in the
Master Node?
Then I can use my master to handle the metadata only and store the logs.
Thanks & regards
Arko
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
Hello Arko,
Add the dfs.data.dir property in your hdfs-site.xml file and
point it to
some other location.
For logs, do the same thing by modifying the following line in
hadoop-env.sh file :
# Where log files are stored. $HADOOP_HOME/logs by default.
export HADOOP_LOG_DIR=/hadoop/hdfs/logs
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:02 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Hello Gurus,
>
> I am managing a Hadoop Cluster to run some experiments.
>
> The issue I am continuously facing is that the Master Node runs out of
> disk space due to logs and data files.
>
> I can monitor and delete log files. However, I cannot delete the HDFS data.
>
> Thus, is there a way to force Hadoop not to save any HDFS data in the
> Master Node?
>
> Then I can use my master to handle the metadata only and store the logs.
>
> Thanks & regards
> Arko
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
Yes.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:24 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Sure, thanks!
> One last question before I go ahead and make the changes.
> My hdfs-site.xml doesn't have an entry for dfs.replication.
> So can I just assume that the system is using the default of 3?
> I ask this because if it is 1, then I will first have to increase it
> so as to not loose data when I stop the NN being a DN.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:51 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> > Decommissioning is for removing DNs and not the NN.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee
> > <ar...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I was just wondering if it is okay to exclude the NN IP.
> >>
> >> I hope it won't create any "side effects".
> >>
> >> This is because I don't want to actually decommission the NN node, but
> >> just make it stop working as a DataNode.
> >>
> >> However, if you are sure there are no problems, I will go ahead and
> >> make the changes.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <do...@gmail.com>
> >> wrote:
> >> > It should.
> >> >
> >> > BTW what's the problem with decommissioning?It's the correct way of
> >> > removing
> >> > the DN, as said by Charles earlier. Create the exclude file and add
> the
> >> > property in the hdfs-site.xml file.
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> >> > <ar...@gmail.com> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> I see that the NameNode IP is present in conf/slaves file.
> >> >>
> >> >> Will removing the IP of the NN from the conf/slaves file and
> >> >> restarting hadoop help?
> >> >>
> >> >> Thanks & regards
> >> >> Arko
> >> >>
> >> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> >> >> <ar...@gmail.com> wrote:
> >> >> > Hi,
> >> >> >
> >> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave
> >> >> > Nodes
> >> >> > (running DN and Task Tracker).
> >> >> >
> >> >> > My issue is that the my NN is getting filled up pretty fast as the
> NN
> >> >> > acts as a DN.
> >> >> >
> >> >> > I want my NN not to function as the DN.
> >> >> >
> >> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> >> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >> >> >
> >> >> > Thanks & regards
> >> >> > Arko
> >> >> >
> >> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <
> dontariq@gmail.com>
> >> >> > wrote:
> >> >> >> Stopping DN on one machine means excluding it. What exactly is the
> >> >> >> requirement? How many nodes do you have right now?
> >> >> >>
> >> >> >> Warm Regards,
> >> >> >> Tariq
> >> >> >> https://mtariq.jux.com/
> >> >> >> cloudfront.blogspot.com
> >> >> >>
> >> >> >>
> >> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> >> >> <ar...@gmail.com> wrote:
> >> >> >>>
> >> >> >>> Hi,
> >> >> >>>
> >> >> >>> I am not sure the provided link matches my situation.
> >> >> >>>
> >> >> >>> That is because, I am not trying to exclude any node but just
> >> >> >>> trying
> >> >> >>> to stop the NN act as a DataNode.
> >> >> >>>
> >> >> >>> Thanks & regards
> >> >> >>> Arko
> >> >> >>>
> >> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >> >> >>> <ar...@gmail.com> wrote:
> >> >> >>> > Thanks! Let me study and find out if it works for me.
> >> >> >>> > Thanks & regards
> >> >> >>> > Arko
> >> >> >>> >
> >> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >> >> >>> > <ar...@gmail.com> wrote:
> >> >> >>> >> Hi,
> >> >> >>> >>
> >> >> >>> >> But the new location will also be the same NN physical server.
> >> >> >>> >> So how does that help?
> >> >> >>> >>
> >> >> >>> >> I would like to clear up the space in the master by removing
> all
> >> >> >>> >> the
> >> >> >>> >> HDFS data to the slave nodes.
> >> >> >>> >>
> >> >> >>> >> Thanks & regards
> >> >> >>> >> Arko
> >> >> >>> >>
> >> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
> >> >> >>> >> <do...@gmail.com>
> >> >> >>> >> wrote:
> >> >> >>> >>> No, copy this directory to the new location of your choice.
> >> >> >>> >>>
> >> >> >>> >>> Warm Regards,
> >> >> >>> >>> Tariq
> >> >> >>> >>> https://mtariq.jux.com/
> >> >> >>> >>> cloudfront.blogspot.com
> >> >> >>> >>>
> >> >> >>> >>>
> >> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >> >> >>> >>> <ar...@gmail.com> wrote:
> >> >> >>> >>>>
> >> >> >>> >>>> Thanks!
> >> >> >>> >>>> However, in that case, how do I handle the current data that
> >> >> >>> >>>> is
> >> >> >>> >>>> already in
> >> >> >>> >>>> NN?
> >> >> >>> >>>> Do I have to reformat HDFS?
> >> >> >>> >>>> Thanks & regards
> >> >> >>> >>>> Arko
> >> >> >>> >>>>
> >> >> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq
> >> >> >>> >>>> <do...@gmail.com>
> >> >> >>> >>>> wrote:
> >> >> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate
> that
> >> >> >>> >>>> > machine
> >> >> >>> >>>> > to
> >> >> >>> >>>> > NN.
> >> >> >>> >>>> > Remove the hostname of the NN machine from the 'slaves'
> >> >> >>> >>>> > file.
> >> >> >>> >>>> >
> >> >> >>> >>>> > Warm Regards,
> >> >> >>> >>>> > Tariq
> >> >> >>> >>>> > https://mtariq.jux.com/
> >> >> >>> >>>> > cloudfront.blogspot.com
> >> >> >>> >>>> >
> >> >> >>> >>>> >
> >> >> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >> >> >>> >>>> > <ar...@gmail.com> wrote:
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Hi,
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Thanks for the help!
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> However, I am still unsure about how to "turn off" the
> >> >> >>> >>>> >> datanode
> >> >> >>> >>>> >> feature on the NN.
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir
> is
> >> >> >>> >>>> >> pointed
> >> >> >>> >>>> >> to
> >> >> >>> >>>> >> a directory.
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Should I just comment out that property? What would
> happen
> >> >> >>> >>>> >> to
> >> >> >>> >>>> >> the
> >> >> >>> >>>> >> current data that there is in the Master? Will it get
> >> >> >>> >>>> >> removed
> >> >> >>> >>>> >> automatically?
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> Thanks & regards
> >> >> >>> >>>> >> Arko
> >> >> >>> >>>> >>
> >> >> >>> >>>> >>
> >> >> >>> >>>> >>
> >> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
> >> >> >>> >>>> >> <do...@gmail.com>
> >> >> >>> >>>> >> wrote:
> >> >> >>> >>>> >> > You can specify the logging level as specified by
> >> >> >>> >>>> >> > Charles.
> >> >> >>> >>>> >> > But
> >> >> >>> >>>> >> > turning
> >> >> >>> >>>> >> > logs
> >> >> >>> >>>> >> > off is never a good idea. Logs are really helpful in
> >> >> >>> >>>> >> > problem
> >> >> >>> >>>> >> > diagnosis,
> >> >> >>> >>>> >> > which are eventual.
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > Warm Regards,
> >> >> >>> >>>> >> > Tariq
> >> >> >>> >>>> >> > https://mtariq.jux.com/
> >> >> >>> >>>> >> > cloudfront.blogspot.com
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >> >> >>> >>>> >> > <ar...@gmail.com> wrote:
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Hi,
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> I would like to "turn off" this feature.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Request help regarding the same.
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> Thanks & regards
> >> >> >>> >>>> >> >> Arko
> >> >> >>> >>>> >> >>
> >> >> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker
> >> >> >>> >>>> >> >> <cb...@sdl.com>
> >> >> >>> >>>> >> >> wrote:
> >> >> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode
> on
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > NameNode
> >> >> >>> >>>> >> >> > which
> >> >> >>> >>>> >> >> > is
> >> >> >>> >>>> >> >> > not recommended practice. Normally, the only files
> the
> >> >> >>> >>>> >> >> > NN
> >> >> >>> >>>> >> >> > stores
> >> >> >>> >>>> >> >> > are
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > image and edits files. It does not store any actual
> >> >> >>> >>>> >> >> > HDFS
> >> >> >>> >>>> >> >> > data. If
> >> >> >>> >>>> >> >> > you
> >> >> >>> >>>> >> >> > must
> >> >> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in
> >> >> >>> >>>> >> >> > /conf/log4j.properties:
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
> >> >> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
> >> >> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Depending on the logging information you require, of
> >> >> >>> >>>> >> >> > course.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > -Chuck
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > -----Original Message-----
> >> >> >>> >>>> >> >> > From: Arko Provo Mukherjee
> >> >> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
> >> >> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >> >> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >> >> >>> >>>> >> >> > Subject: Managing space in Master Node
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Hello Gurus,
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some
> >> >> >>> >>>> >> >> > experiments.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > The issue I am continuously facing is that the
> Master
> >> >> >>> >>>> >> >> > Node
> >> >> >>> >>>> >> >> > runs
> >> >> >>> >>>> >> >> > out
> >> >> >>> >>>> >> >> > of
> >> >> >>> >>>> >> >> > disk
> >> >> >>> >>>> >> >> > space due to logs and data files.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > I can monitor and delete log files. However, I
> cannot
> >> >> >>> >>>> >> >> > delete
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > HDFS
> >> >> >>> >>>> >> >> > data.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any
> >> >> >>> >>>> >> >> > HDFS
> >> >> >>> >>>> >> >> > data in
> >> >> >>> >>>> >> >> > the
> >> >> >>> >>>> >> >> > Master
> >> >> >>> >>>> >> >> > Node?
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Then I can use my master to handle the metadata only
> >> >> >>> >>>> >> >> > and
> >> >> >>> >>>> >> >> > store the
> >> >> >>> >>>> >> >> > logs.
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >> > Thanks & regards
> >> >> >>> >>>> >> >> > Arko
> >> >> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights
> >> >> >>> >>>> >> >> > reserved.
> >> >> >>> >>>> >> >> > The
> >> >> >>> >>>> >> >> > information contained in this email may be
> >> >> >>> >>>> >> >> > confidential
> >> >> >>> >>>> >> >> > and/or
> >> >> >>> >>>> >> >> > legally
> >> >> >>> >>>> >> >> > privileged. It has been sent for the sole use of the
> >> >> >>> >>>> >> >> > intended
> >> >> >>> >>>> >> >> > recipient(s).
> >> >> >>> >>>> >> >> > If you are not the intended recipient of this mail,
> >> >> >>> >>>> >> >> > you
> >> >> >>> >>>> >> >> > are
> >> >> >>> >>>> >> >> > hereby
> >> >> >>> >>>> >> >> > notified
> >> >> >>> >>>> >> >> > that any unauthorized review, use, disclosure,
> >> >> >>> >>>> >> >> > dissemination,
> >> >> >>> >>>> >> >> > distribution,
> >> >> >>> >>>> >> >> > or copying of this communication, or any of its
> >> >> >>> >>>> >> >> > contents,
> >> >> >>> >>>> >> >> > is
> >> >> >>> >>>> >> >> > strictly
> >> >> >>> >>>> >> >> > prohibited. If you have received this communication
> in
> >> >> >>> >>>> >> >> > error,
> >> >> >>> >>>> >> >> > please
> >> >> >>> >>>> >> >> > reply
> >> >> >>> >>>> >> >> > to the sender and destroy all copies of the message.
> >> >> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
> >> >> >>> >>>> >> >> > Wakefield, MA
> >> >> >>> >>>> >> >> > 01880,
> >> >> >>> >>>> >> >> > USA
> >> >> >>> >>>> >> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >> >
> >> >> >>> >>>> >
> >> >> >>> >>>> >
> >> >> >>> >>>
> >> >> >>> >>>
> >> >> >>
> >> >> >>
> >> >
> >> >
> >
> >
>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Sure, thanks!
One last question before I go ahead and make the changes.
My hdfs-site.xml doesn't have an entry for dfs.replication.
So can I just assume that the system is using the default of 3?
I ask this because if it is 1, then I will first have to increase it
so as to not loose data when I stop the NN being a DN.
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 5:51 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Decommissioning is for removing DNs and not the NN.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I was just wondering if it is okay to exclude the NN IP.
>>
>> I hope it won't create any "side effects".
>>
>> This is because I don't want to actually decommission the NN node, but
>> just make it stop working as a DataNode.
>>
>> However, if you are sure there are no problems, I will go ahead and
>> make the changes.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <do...@gmail.com>
>> wrote:
>> > It should.
>> >
>> > BTW what's the problem with decommissioning?It's the correct way of
>> > removing
>> > the DN, as said by Charles earlier. Create the exclude file and add the
>> > property in the hdfs-site.xml file.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
>> > <ar...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> I see that the NameNode IP is present in conf/slaves file.
>> >>
>> >> Will removing the IP of the NN from the conf/slaves file and
>> >> restarting hadoop help?
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
>> >> <ar...@gmail.com> wrote:
>> >> > Hi,
>> >> >
>> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave
>> >> > Nodes
>> >> > (running DN and Task Tracker).
>> >> >
>> >> > My issue is that the my NN is getting filled up pretty fast as the NN
>> >> > acts as a DN.
>> >> >
>> >> > I want my NN not to function as the DN.
>> >> >
>> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
>> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>> >> >
>> >> > Thanks & regards
>> >> > Arko
>> >> >
>> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <do...@gmail.com>
>> >> > wrote:
>> >> >> Stopping DN on one machine means excluding it. What exactly is the
>> >> >> requirement? How many nodes do you have right now?
>> >> >>
>> >> >> Warm Regards,
>> >> >> Tariq
>> >> >> https://mtariq.jux.com/
>> >> >> cloudfront.blogspot.com
>> >> >>
>> >> >>
>> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> >> >> <ar...@gmail.com> wrote:
>> >> >>>
>> >> >>> Hi,
>> >> >>>
>> >> >>> I am not sure the provided link matches my situation.
>> >> >>>
>> >> >>> That is because, I am not trying to exclude any node but just
>> >> >>> trying
>> >> >>> to stop the NN act as a DataNode.
>> >> >>>
>> >> >>> Thanks & regards
>> >> >>> Arko
>> >> >>>
>> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> >> >>> <ar...@gmail.com> wrote:
>> >> >>> > Thanks! Let me study and find out if it works for me.
>> >> >>> > Thanks & regards
>> >> >>> > Arko
>> >> >>> >
>> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> >> >>> > <ar...@gmail.com> wrote:
>> >> >>> >> Hi,
>> >> >>> >>
>> >> >>> >> But the new location will also be the same NN physical server.
>> >> >>> >> So how does that help?
>> >> >>> >>
>> >> >>> >> I would like to clear up the space in the master by removing all
>> >> >>> >> the
>> >> >>> >> HDFS data to the slave nodes.
>> >> >>> >>
>> >> >>> >> Thanks & regards
>> >> >>> >> Arko
>> >> >>> >>
>> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
>> >> >>> >> <do...@gmail.com>
>> >> >>> >> wrote:
>> >> >>> >>> No, copy this directory to the new location of your choice.
>> >> >>> >>>
>> >> >>> >>> Warm Regards,
>> >> >>> >>> Tariq
>> >> >>> >>> https://mtariq.jux.com/
>> >> >>> >>> cloudfront.blogspot.com
>> >> >>> >>>
>> >> >>> >>>
>> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >> >>> >>> <ar...@gmail.com> wrote:
>> >> >>> >>>>
>> >> >>> >>>> Thanks!
>> >> >>> >>>> However, in that case, how do I handle the current data that
>> >> >>> >>>> is
>> >> >>> >>>> already in
>> >> >>> >>>> NN?
>> >> >>> >>>> Do I have to reformat HDFS?
>> >> >>> >>>> Thanks & regards
>> >> >>> >>>> Arko
>> >> >>> >>>>
>> >> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq
>> >> >>> >>>> <do...@gmail.com>
>> >> >>> >>>> wrote:
>> >> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
>> >> >>> >>>> > machine
>> >> >>> >>>> > to
>> >> >>> >>>> > NN.
>> >> >>> >>>> > Remove the hostname of the NN machine from the 'slaves'
>> >> >>> >>>> > file.
>> >> >>> >>>> >
>> >> >>> >>>> > Warm Regards,
>> >> >>> >>>> > Tariq
>> >> >>> >>>> > https://mtariq.jux.com/
>> >> >>> >>>> > cloudfront.blogspot.com
>> >> >>> >>>> >
>> >> >>> >>>> >
>> >> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> >> >>> >>>> > <ar...@gmail.com> wrote:
>> >> >>> >>>> >>
>> >> >>> >>>> >> Hi,
>> >> >>> >>>> >>
>> >> >>> >>>> >> Thanks for the help!
>> >> >>> >>>> >>
>> >> >>> >>>> >> However, I am still unsure about how to "turn off" the
>> >> >>> >>>> >> datanode
>> >> >>> >>>> >> feature on the NN.
>> >> >>> >>>> >>
>> >> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is
>> >> >>> >>>> >> pointed
>> >> >>> >>>> >> to
>> >> >>> >>>> >> a directory.
>> >> >>> >>>> >>
>> >> >>> >>>> >> Should I just comment out that property? What would happen
>> >> >>> >>>> >> to
>> >> >>> >>>> >> the
>> >> >>> >>>> >> current data that there is in the Master? Will it get
>> >> >>> >>>> >> removed
>> >> >>> >>>> >> automatically?
>> >> >>> >>>> >>
>> >> >>> >>>> >> Thanks & regards
>> >> >>> >>>> >> Arko
>> >> >>> >>>> >>
>> >> >>> >>>> >>
>> >> >>> >>>> >>
>> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>> >> >>> >>>> >> <do...@gmail.com>
>> >> >>> >>>> >> wrote:
>> >> >>> >>>> >> > You can specify the logging level as specified by
>> >> >>> >>>> >> > Charles.
>> >> >>> >>>> >> > But
>> >> >>> >>>> >> > turning
>> >> >>> >>>> >> > logs
>> >> >>> >>>> >> > off is never a good idea. Logs are really helpful in
>> >> >>> >>>> >> > problem
>> >> >>> >>>> >> > diagnosis,
>> >> >>> >>>> >> > which are eventual.
>> >> >>> >>>> >> >
>> >> >>> >>>> >> > Warm Regards,
>> >> >>> >>>> >> > Tariq
>> >> >>> >>>> >> > https://mtariq.jux.com/
>> >> >>> >>>> >> > cloudfront.blogspot.com
>> >> >>> >>>> >> >
>> >> >>> >>>> >> >
>> >> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >> >>> >>>> >> > <ar...@gmail.com> wrote:
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Hi,
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> I would like to "turn off" this feature.
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Request help regarding the same.
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> Thanks & regards
>> >> >>> >>>> >> >> Arko
>> >> >>> >>>> >> >>
>> >> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker
>> >> >>> >>>> >> >> <cb...@sdl.com>
>> >> >>> >>>> >> >> wrote:
>> >> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on
>> >> >>> >>>> >> >> > the
>> >> >>> >>>> >> >> > NameNode
>> >> >>> >>>> >> >> > which
>> >> >>> >>>> >> >> > is
>> >> >>> >>>> >> >> > not recommended practice. Normally, the only files the
>> >> >>> >>>> >> >> > NN
>> >> >>> >>>> >> >> > stores
>> >> >>> >>>> >> >> > are
>> >> >>> >>>> >> >> > the
>> >> >>> >>>> >> >> > image and edits files. It does not store any actual
>> >> >>> >>>> >> >> > HDFS
>> >> >>> >>>> >> >> > data. If
>> >> >>> >>>> >> >> > you
>> >> >>> >>>> >> >> > must
>> >> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in
>> >> >>> >>>> >> >> > /conf/log4j.properties:
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
>> >> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
>> >> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > Depending on the logging information you require, of
>> >> >>> >>>> >> >> > course.
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > -Chuck
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > -----Original Message-----
>> >> >>> >>>> >> >> > From: Arko Provo Mukherjee
>> >> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
>> >> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
>> >> >>> >>>> >> >> > Subject: Managing space in Master Node
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > Hello Gurus,
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some
>> >> >>> >>>> >> >> > experiments.
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > The issue I am continuously facing is that the Master
>> >> >>> >>>> >> >> > Node
>> >> >>> >>>> >> >> > runs
>> >> >>> >>>> >> >> > out
>> >> >>> >>>> >> >> > of
>> >> >>> >>>> >> >> > disk
>> >> >>> >>>> >> >> > space due to logs and data files.
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > I can monitor and delete log files. However, I cannot
>> >> >>> >>>> >> >> > delete
>> >> >>> >>>> >> >> > the
>> >> >>> >>>> >> >> > HDFS
>> >> >>> >>>> >> >> > data.
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any
>> >> >>> >>>> >> >> > HDFS
>> >> >>> >>>> >> >> > data in
>> >> >>> >>>> >> >> > the
>> >> >>> >>>> >> >> > Master
>> >> >>> >>>> >> >> > Node?
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > Then I can use my master to handle the metadata only
>> >> >>> >>>> >> >> > and
>> >> >>> >>>> >> >> > store the
>> >> >>> >>>> >> >> > logs.
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >> > Thanks & regards
>> >> >>> >>>> >> >> > Arko
>> >> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights
>> >> >>> >>>> >> >> > reserved.
>> >> >>> >>>> >> >> > The
>> >> >>> >>>> >> >> > information contained in this email may be
>> >> >>> >>>> >> >> > confidential
>> >> >>> >>>> >> >> > and/or
>> >> >>> >>>> >> >> > legally
>> >> >>> >>>> >> >> > privileged. It has been sent for the sole use of the
>> >> >>> >>>> >> >> > intended
>> >> >>> >>>> >> >> > recipient(s).
>> >> >>> >>>> >> >> > If you are not the intended recipient of this mail,
>> >> >>> >>>> >> >> > you
>> >> >>> >>>> >> >> > are
>> >> >>> >>>> >> >> > hereby
>> >> >>> >>>> >> >> > notified
>> >> >>> >>>> >> >> > that any unauthorized review, use, disclosure,
>> >> >>> >>>> >> >> > dissemination,
>> >> >>> >>>> >> >> > distribution,
>> >> >>> >>>> >> >> > or copying of this communication, or any of its
>> >> >>> >>>> >> >> > contents,
>> >> >>> >>>> >> >> > is
>> >> >>> >>>> >> >> > strictly
>> >> >>> >>>> >> >> > prohibited. If you have received this communication in
>> >> >>> >>>> >> >> > error,
>> >> >>> >>>> >> >> > please
>> >> >>> >>>> >> >> > reply
>> >> >>> >>>> >> >> > to the sender and destroy all copies of the message.
>> >> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
>> >> >>> >>>> >> >> > Wakefield, MA
>> >> >>> >>>> >> >> > 01880,
>> >> >>> >>>> >> >> > USA
>> >> >>> >>>> >> >> >
>> >> >>> >>>> >> >
>> >> >>> >>>> >> >
>> >> >>> >>>> >
>> >> >>> >>>> >
>> >> >>> >>>
>> >> >>> >>>
>> >> >>
>> >> >>
>> >
>> >
>
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
Decommissioning is for removing DNs and not the NN.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:18 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Hi,
>
> I was just wondering if it is okay to exclude the NN IP.
>
> I hope it won't create any "side effects".
>
> This is because I don't want to actually decommission the NN node, but
> just make it stop working as a DataNode.
>
> However, if you are sure there are no problems, I will go ahead and
> make the changes.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> > It should.
> >
> > BTW what's the problem with decommissioning?It's the correct way of
> removing
> > the DN, as said by Charles earlier. Create the exclude file and add the
> > property in the hdfs-site.xml file.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> > <ar...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> I see that the NameNode IP is present in conf/slaves file.
> >>
> >> Will removing the IP of the NN from the conf/slaves file and
> >> restarting hadoop help?
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> >> <ar...@gmail.com> wrote:
> >> > Hi,
> >> >
> >> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> >> > (running DN and Task Tracker).
> >> >
> >> > My issue is that the my NN is getting filled up pretty fast as the NN
> >> > acts as a DN.
> >> >
> >> > I want my NN not to function as the DN.
> >> >
> >> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> >> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >> >
> >> > Thanks & regards
> >> > Arko
> >> >
> >> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <do...@gmail.com>
> >> > wrote:
> >> >> Stopping DN on one machine means excluding it. What exactly is the
> >> >> requirement? How many nodes do you have right now?
> >> >>
> >> >> Warm Regards,
> >> >> Tariq
> >> >> https://mtariq.jux.com/
> >> >> cloudfront.blogspot.com
> >> >>
> >> >>
> >> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> >> <ar...@gmail.com> wrote:
> >> >>>
> >> >>> Hi,
> >> >>>
> >> >>> I am not sure the provided link matches my situation.
> >> >>>
> >> >>> That is because, I am not trying to exclude any node but just trying
> >> >>> to stop the NN act as a DataNode.
> >> >>>
> >> >>> Thanks & regards
> >> >>> Arko
> >> >>>
> >> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >> >>> <ar...@gmail.com> wrote:
> >> >>> > Thanks! Let me study and find out if it works for me.
> >> >>> > Thanks & regards
> >> >>> > Arko
> >> >>> >
> >> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >> >>> > <ar...@gmail.com> wrote:
> >> >>> >> Hi,
> >> >>> >>
> >> >>> >> But the new location will also be the same NN physical server.
> >> >>> >> So how does that help?
> >> >>> >>
> >> >>> >> I would like to clear up the space in the master by removing all
> >> >>> >> the
> >> >>> >> HDFS data to the slave nodes.
> >> >>> >>
> >> >>> >> Thanks & regards
> >> >>> >> Arko
> >> >>> >>
> >> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
> >> >>> >> <do...@gmail.com>
> >> >>> >> wrote:
> >> >>> >>> No, copy this directory to the new location of your choice.
> >> >>> >>>
> >> >>> >>> Warm Regards,
> >> >>> >>> Tariq
> >> >>> >>> https://mtariq.jux.com/
> >> >>> >>> cloudfront.blogspot.com
> >> >>> >>>
> >> >>> >>>
> >> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >> >>> >>> <ar...@gmail.com> wrote:
> >> >>> >>>>
> >> >>> >>>> Thanks!
> >> >>> >>>> However, in that case, how do I handle the current data that is
> >> >>> >>>> already in
> >> >>> >>>> NN?
> >> >>> >>>> Do I have to reformat HDFS?
> >> >>> >>>> Thanks & regards
> >> >>> >>>> Arko
> >> >>> >>>>
> >> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq
> >> >>> >>>> <do...@gmail.com>
> >> >>> >>>> wrote:
> >> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
> >> >>> >>>> > machine
> >> >>> >>>> > to
> >> >>> >>>> > NN.
> >> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
> >> >>> >>>> >
> >> >>> >>>> > Warm Regards,
> >> >>> >>>> > Tariq
> >> >>> >>>> > https://mtariq.jux.com/
> >> >>> >>>> > cloudfront.blogspot.com
> >> >>> >>>> >
> >> >>> >>>> >
> >> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >> >>> >>>> > <ar...@gmail.com> wrote:
> >> >>> >>>> >>
> >> >>> >>>> >> Hi,
> >> >>> >>>> >>
> >> >>> >>>> >> Thanks for the help!
> >> >>> >>>> >>
> >> >>> >>>> >> However, I am still unsure about how to "turn off" the
> >> >>> >>>> >> datanode
> >> >>> >>>> >> feature on the NN.
> >> >>> >>>> >>
> >> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is
> >> >>> >>>> >> pointed
> >> >>> >>>> >> to
> >> >>> >>>> >> a directory.
> >> >>> >>>> >>
> >> >>> >>>> >> Should I just comment out that property? What would happen
> to
> >> >>> >>>> >> the
> >> >>> >>>> >> current data that there is in the Master? Will it get
> removed
> >> >>> >>>> >> automatically?
> >> >>> >>>> >>
> >> >>> >>>> >> Thanks & regards
> >> >>> >>>> >> Arko
> >> >>> >>>> >>
> >> >>> >>>> >>
> >> >>> >>>> >>
> >> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
> >> >>> >>>> >> <do...@gmail.com>
> >> >>> >>>> >> wrote:
> >> >>> >>>> >> > You can specify the logging level as specified by Charles.
> >> >>> >>>> >> > But
> >> >>> >>>> >> > turning
> >> >>> >>>> >> > logs
> >> >>> >>>> >> > off is never a good idea. Logs are really helpful in
> problem
> >> >>> >>>> >> > diagnosis,
> >> >>> >>>> >> > which are eventual.
> >> >>> >>>> >> >
> >> >>> >>>> >> > Warm Regards,
> >> >>> >>>> >> > Tariq
> >> >>> >>>> >> > https://mtariq.jux.com/
> >> >>> >>>> >> > cloudfront.blogspot.com
> >> >>> >>>> >> >
> >> >>> >>>> >> >
> >> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >> >>> >>>> >> > <ar...@gmail.com> wrote:
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Hi,
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> I would like to "turn off" this feature.
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Request help regarding the same.
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> Thanks & regards
> >> >>> >>>> >> >> Arko
> >> >>> >>>> >> >>
> >> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker
> >> >>> >>>> >> >> <cb...@sdl.com>
> >> >>> >>>> >> >> wrote:
> >> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on
> the
> >> >>> >>>> >> >> > NameNode
> >> >>> >>>> >> >> > which
> >> >>> >>>> >> >> > is
> >> >>> >>>> >> >> > not recommended practice. Normally, the only files the
> NN
> >> >>> >>>> >> >> > stores
> >> >>> >>>> >> >> > are
> >> >>> >>>> >> >> > the
> >> >>> >>>> >> >> > image and edits files. It does not store any actual
> HDFS
> >> >>> >>>> >> >> > data. If
> >> >>> >>>> >> >> > you
> >> >>> >>>> >> >> > must
> >> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in
> >> >>> >>>> >> >> > /conf/log4j.properties:
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
> >> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
> >> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > Depending on the logging information you require, of
> >> >>> >>>> >> >> > course.
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > -Chuck
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > -----Original Message-----
> >> >>> >>>> >> >> > From: Arko Provo Mukherjee
> >> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
> >> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >> >>> >>>> >> >> > Subject: Managing space in Master Node
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > Hello Gurus,
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > The issue I am continuously facing is that the Master
> >> >>> >>>> >> >> > Node
> >> >>> >>>> >> >> > runs
> >> >>> >>>> >> >> > out
> >> >>> >>>> >> >> > of
> >> >>> >>>> >> >> > disk
> >> >>> >>>> >> >> > space due to logs and data files.
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > I can monitor and delete log files. However, I cannot
> >> >>> >>>> >> >> > delete
> >> >>> >>>> >> >> > the
> >> >>> >>>> >> >> > HDFS
> >> >>> >>>> >> >> > data.
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any
> HDFS
> >> >>> >>>> >> >> > data in
> >> >>> >>>> >> >> > the
> >> >>> >>>> >> >> > Master
> >> >>> >>>> >> >> > Node?
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > Then I can use my master to handle the metadata only
> and
> >> >>> >>>> >> >> > store the
> >> >>> >>>> >> >> > logs.
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >> > Thanks & regards
> >> >>> >>>> >> >> > Arko
> >> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights
> reserved.
> >> >>> >>>> >> >> > The
> >> >>> >>>> >> >> > information contained in this email may be confidential
> >> >>> >>>> >> >> > and/or
> >> >>> >>>> >> >> > legally
> >> >>> >>>> >> >> > privileged. It has been sent for the sole use of the
> >> >>> >>>> >> >> > intended
> >> >>> >>>> >> >> > recipient(s).
> >> >>> >>>> >> >> > If you are not the intended recipient of this mail, you
> >> >>> >>>> >> >> > are
> >> >>> >>>> >> >> > hereby
> >> >>> >>>> >> >> > notified
> >> >>> >>>> >> >> > that any unauthorized review, use, disclosure,
> >> >>> >>>> >> >> > dissemination,
> >> >>> >>>> >> >> > distribution,
> >> >>> >>>> >> >> > or copying of this communication, or any of its
> contents,
> >> >>> >>>> >> >> > is
> >> >>> >>>> >> >> > strictly
> >> >>> >>>> >> >> > prohibited. If you have received this communication in
> >> >>> >>>> >> >> > error,
> >> >>> >>>> >> >> > please
> >> >>> >>>> >> >> > reply
> >> >>> >>>> >> >> > to the sender and destroy all copies of the message.
> >> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
> >> >>> >>>> >> >> > Wakefield, MA
> >> >>> >>>> >> >> > 01880,
> >> >>> >>>> >> >> > USA
> >> >>> >>>> >> >> >
> >> >>> >>>> >> >
> >> >>> >>>> >> >
> >> >>> >>>> >
> >> >>> >>>> >
> >> >>> >>>
> >> >>> >>>
> >> >>
> >> >>
> >
> >
>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
I was just wondering if it is okay to exclude the NN IP.
I hope it won't create any "side effects".
This is because I don't want to actually decommission the NN node, but
just make it stop working as a DataNode.
However, if you are sure there are no problems, I will go ahead and
make the changes.
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 5:43 PM, Mohammad Tariq <do...@gmail.com> wrote:
> It should.
>
> BTW what's the problem with decommissioning?It's the correct way of removing
> the DN, as said by Charles earlier. Create the exclude file and add the
> property in the hdfs-site.xml file.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I see that the NameNode IP is present in conf/slaves file.
>>
>> Will removing the IP of the NN from the conf/slaves file and
>> restarting hadoop help?
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
>> <ar...@gmail.com> wrote:
>> > Hi,
>> >
>> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
>> > (running DN and Task Tracker).
>> >
>> > My issue is that the my NN is getting filled up pretty fast as the NN
>> > acts as a DN.
>> >
>> > I want my NN not to function as the DN.
>> >
>> > Does including the IP for the NN in the "dfs.hosts.exclude" and
>> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>> >
>> > Thanks & regards
>> > Arko
>> >
>> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <do...@gmail.com>
>> > wrote:
>> >> Stopping DN on one machine means excluding it. What exactly is the
>> >> requirement? How many nodes do you have right now?
>> >>
>> >> Warm Regards,
>> >> Tariq
>> >> https://mtariq.jux.com/
>> >> cloudfront.blogspot.com
>> >>
>> >>
>> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> >> <ar...@gmail.com> wrote:
>> >>>
>> >>> Hi,
>> >>>
>> >>> I am not sure the provided link matches my situation.
>> >>>
>> >>> That is because, I am not trying to exclude any node but just trying
>> >>> to stop the NN act as a DataNode.
>> >>>
>> >>> Thanks & regards
>> >>> Arko
>> >>>
>> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> >>> <ar...@gmail.com> wrote:
>> >>> > Thanks! Let me study and find out if it works for me.
>> >>> > Thanks & regards
>> >>> > Arko
>> >>> >
>> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> >>> > <ar...@gmail.com> wrote:
>> >>> >> Hi,
>> >>> >>
>> >>> >> But the new location will also be the same NN physical server.
>> >>> >> So how does that help?
>> >>> >>
>> >>> >> I would like to clear up the space in the master by removing all
>> >>> >> the
>> >>> >> HDFS data to the slave nodes.
>> >>> >>
>> >>> >> Thanks & regards
>> >>> >> Arko
>> >>> >>
>> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq
>> >>> >> <do...@gmail.com>
>> >>> >> wrote:
>> >>> >>> No, copy this directory to the new location of your choice.
>> >>> >>>
>> >>> >>> Warm Regards,
>> >>> >>> Tariq
>> >>> >>> https://mtariq.jux.com/
>> >>> >>> cloudfront.blogspot.com
>> >>> >>>
>> >>> >>>
>> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >>> >>> <ar...@gmail.com> wrote:
>> >>> >>>>
>> >>> >>>> Thanks!
>> >>> >>>> However, in that case, how do I handle the current data that is
>> >>> >>>> already in
>> >>> >>>> NN?
>> >>> >>>> Do I have to reformat HDFS?
>> >>> >>>> Thanks & regards
>> >>> >>>> Arko
>> >>> >>>>
>> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq
>> >>> >>>> <do...@gmail.com>
>> >>> >>>> wrote:
>> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
>> >>> >>>> > machine
>> >>> >>>> > to
>> >>> >>>> > NN.
>> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >>> >>>> >
>> >>> >>>> > Warm Regards,
>> >>> >>>> > Tariq
>> >>> >>>> > https://mtariq.jux.com/
>> >>> >>>> > cloudfront.blogspot.com
>> >>> >>>> >
>> >>> >>>> >
>> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> >>> >>>> > <ar...@gmail.com> wrote:
>> >>> >>>> >>
>> >>> >>>> >> Hi,
>> >>> >>>> >>
>> >>> >>>> >> Thanks for the help!
>> >>> >>>> >>
>> >>> >>>> >> However, I am still unsure about how to "turn off" the
>> >>> >>>> >> datanode
>> >>> >>>> >> feature on the NN.
>> >>> >>>> >>
>> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is
>> >>> >>>> >> pointed
>> >>> >>>> >> to
>> >>> >>>> >> a directory.
>> >>> >>>> >>
>> >>> >>>> >> Should I just comment out that property? What would happen to
>> >>> >>>> >> the
>> >>> >>>> >> current data that there is in the Master? Will it get removed
>> >>> >>>> >> automatically?
>> >>> >>>> >>
>> >>> >>>> >> Thanks & regards
>> >>> >>>> >> Arko
>> >>> >>>> >>
>> >>> >>>> >>
>> >>> >>>> >>
>> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>> >>> >>>> >> <do...@gmail.com>
>> >>> >>>> >> wrote:
>> >>> >>>> >> > You can specify the logging level as specified by Charles.
>> >>> >>>> >> > But
>> >>> >>>> >> > turning
>> >>> >>>> >> > logs
>> >>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>> >>> >>>> >> > diagnosis,
>> >>> >>>> >> > which are eventual.
>> >>> >>>> >> >
>> >>> >>>> >> > Warm Regards,
>> >>> >>>> >> > Tariq
>> >>> >>>> >> > https://mtariq.jux.com/
>> >>> >>>> >> > cloudfront.blogspot.com
>> >>> >>>> >> >
>> >>> >>>> >> >
>> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >>> >>>> >> > <ar...@gmail.com> wrote:
>> >>> >>>> >> >>
>> >>> >>>> >> >> Hi,
>> >>> >>>> >> >>
>> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>> >>>> >> >>
>> >>> >>>> >> >> I would like to "turn off" this feature.
>> >>> >>>> >> >>
>> >>> >>>> >> >> Request help regarding the same.
>> >>> >>>> >> >>
>> >>> >>>> >> >> Thanks & regards
>> >>> >>>> >> >> Arko
>> >>> >>>> >> >>
>> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker
>> >>> >>>> >> >> <cb...@sdl.com>
>> >>> >>>> >> >> wrote:
>> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the
>> >>> >>>> >> >> > NameNode
>> >>> >>>> >> >> > which
>> >>> >>>> >> >> > is
>> >>> >>>> >> >> > not recommended practice. Normally, the only files the NN
>> >>> >>>> >> >> > stores
>> >>> >>>> >> >> > are
>> >>> >>>> >> >> > the
>> >>> >>>> >> >> > image and edits files. It does not store any actual HDFS
>> >>> >>>> >> >> > data. If
>> >>> >>>> >> >> > you
>> >>> >>>> >> >> > must
>> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in
>> >>> >>>> >> >> > /conf/log4j.properties:
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
>> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
>> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > Depending on the logging information you require, of
>> >>> >>>> >> >> > course.
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > -Chuck
>> >>> >>>> >> >> >
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > -----Original Message-----
>> >>> >>>> >> >> > From: Arko Provo Mukherjee
>> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
>> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
>> >>> >>>> >> >> > Subject: Managing space in Master Node
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > Hello Gurus,
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > The issue I am continuously facing is that the Master
>> >>> >>>> >> >> > Node
>> >>> >>>> >> >> > runs
>> >>> >>>> >> >> > out
>> >>> >>>> >> >> > of
>> >>> >>>> >> >> > disk
>> >>> >>>> >> >> > space due to logs and data files.
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > I can monitor and delete log files. However, I cannot
>> >>> >>>> >> >> > delete
>> >>> >>>> >> >> > the
>> >>> >>>> >> >> > HDFS
>> >>> >>>> >> >> > data.
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS
>> >>> >>>> >> >> > data in
>> >>> >>>> >> >> > the
>> >>> >>>> >> >> > Master
>> >>> >>>> >> >> > Node?
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > Then I can use my master to handle the metadata only and
>> >>> >>>> >> >> > store the
>> >>> >>>> >> >> > logs.
>> >>> >>>> >> >> >
>> >>> >>>> >> >> > Thanks & regards
>> >>> >>>> >> >> > Arko
>> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved.
>> >>> >>>> >> >> > The
>> >>> >>>> >> >> > information contained in this email may be confidential
>> >>> >>>> >> >> > and/or
>> >>> >>>> >> >> > legally
>> >>> >>>> >> >> > privileged. It has been sent for the sole use of the
>> >>> >>>> >> >> > intended
>> >>> >>>> >> >> > recipient(s).
>> >>> >>>> >> >> > If you are not the intended recipient of this mail, you
>> >>> >>>> >> >> > are
>> >>> >>>> >> >> > hereby
>> >>> >>>> >> >> > notified
>> >>> >>>> >> >> > that any unauthorized review, use, disclosure,
>> >>> >>>> >> >> > dissemination,
>> >>> >>>> >> >> > distribution,
>> >>> >>>> >> >> > or copying of this communication, or any of its contents,
>> >>> >>>> >> >> > is
>> >>> >>>> >> >> > strictly
>> >>> >>>> >> >> > prohibited. If you have received this communication in
>> >>> >>>> >> >> > error,
>> >>> >>>> >> >> > please
>> >>> >>>> >> >> > reply
>> >>> >>>> >> >> > to the sender and destroy all copies of the message.
>> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
>> >>> >>>> >> >> > Wakefield, MA
>> >>> >>>> >> >> > 01880,
>> >>> >>>> >> >> > USA
>> >>> >>>> >> >> >
>> >>> >>>> >> >
>> >>> >>>> >> >
>> >>> >>>> >
>> >>> >>>> >
>> >>> >>>
>> >>> >>>
>> >>
>> >>
>
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
It should.
BTW what's the problem with decommissioning?It's the correct way of
removing the DN, as said by Charles earlier. Create the exclude file and
add the property in the hdfs-site.xml file.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 5:05 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Hi,
>
> I see that the NameNode IP is present in conf/slaves file.
>
> Will removing the IP of the NN from the conf/slaves file and
> restarting hadoop help?
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
> > Hi,
> >
> > I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> > (running DN and Task Tracker).
> >
> > My issue is that the my NN is getting filled up pretty fast as the NN
> > acts as a DN.
> >
> > I want my NN not to function as the DN.
> >
> > Does including the IP for the NN in the "dfs.hosts.exclude" and
> > running "$ hadoop dfsadmin -refreshNodes" solve my problem?
> >
> > Thanks & regards
> > Arko
> >
> > On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> >> Stopping DN on one machine means excluding it. What exactly is the
> >> requirement? How many nodes do you have right now?
> >>
> >> Warm Regards,
> >> Tariq
> >> https://mtariq.jux.com/
> >> cloudfront.blogspot.com
> >>
> >>
> >> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> >> <ar...@gmail.com> wrote:
> >>>
> >>> Hi,
> >>>
> >>> I am not sure the provided link matches my situation.
> >>>
> >>> That is because, I am not trying to exclude any node but just trying
> >>> to stop the NN act as a DataNode.
> >>>
> >>> Thanks & regards
> >>> Arko
> >>>
> >>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> >>> <ar...@gmail.com> wrote:
> >>> > Thanks! Let me study and find out if it works for me.
> >>> > Thanks & regards
> >>> > Arko
> >>> >
> >>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> >>> > <ar...@gmail.com> wrote:
> >>> >> Hi,
> >>> >>
> >>> >> But the new location will also be the same NN physical server.
> >>> >> So how does that help?
> >>> >>
> >>> >> I would like to clear up the space in the master by removing all the
> >>> >> HDFS data to the slave nodes.
> >>> >>
> >>> >> Thanks & regards
> >>> >> Arko
> >>> >>
> >>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <dontariq@gmail.com
> >
> >>> >> wrote:
> >>> >>> No, copy this directory to the new location of your choice.
> >>> >>>
> >>> >>> Warm Regards,
> >>> >>> Tariq
> >>> >>> https://mtariq.jux.com/
> >>> >>> cloudfront.blogspot.com
> >>> >>>
> >>> >>>
> >>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >>> >>> <ar...@gmail.com> wrote:
> >>> >>>>
> >>> >>>> Thanks!
> >>> >>>> However, in that case, how do I handle the current data that is
> >>> >>>> already in
> >>> >>>> NN?
> >>> >>>> Do I have to reformat HDFS?
> >>> >>>> Thanks & regards
> >>> >>>> Arko
> >>> >>>>
> >>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <
> dontariq@gmail.com>
> >>> >>>> wrote:
> >>> >>>> > Just don't run the DN daemon on that machine. Dedicate that
> machine
> >>> >>>> > to
> >>> >>>> > NN.
> >>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
> >>> >>>> >
> >>> >>>> > Warm Regards,
> >>> >>>> > Tariq
> >>> >>>> > https://mtariq.jux.com/
> >>> >>>> > cloudfront.blogspot.com
> >>> >>>> >
> >>> >>>> >
> >>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >>> >>>> > <ar...@gmail.com> wrote:
> >>> >>>> >>
> >>> >>>> >> Hi,
> >>> >>>> >>
> >>> >>>> >> Thanks for the help!
> >>> >>>> >>
> >>> >>>> >> However, I am still unsure about how to "turn off" the datanode
> >>> >>>> >> feature on the NN.
> >>> >>>> >>
> >>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is
> pointed
> >>> >>>> >> to
> >>> >>>> >> a directory.
> >>> >>>> >>
> >>> >>>> >> Should I just comment out that property? What would happen to
> the
> >>> >>>> >> current data that there is in the Master? Will it get removed
> >>> >>>> >> automatically?
> >>> >>>> >>
> >>> >>>> >> Thanks & regards
> >>> >>>> >> Arko
> >>> >>>> >>
> >>> >>>> >>
> >>> >>>> >>
> >>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
> >>> >>>> >> <do...@gmail.com>
> >>> >>>> >> wrote:
> >>> >>>> >> > You can specify the logging level as specified by Charles.
> But
> >>> >>>> >> > turning
> >>> >>>> >> > logs
> >>> >>>> >> > off is never a good idea. Logs are really helpful in problem
> >>> >>>> >> > diagnosis,
> >>> >>>> >> > which are eventual.
> >>> >>>> >> >
> >>> >>>> >> > Warm Regards,
> >>> >>>> >> > Tariq
> >>> >>>> >> > https://mtariq.jux.com/
> >>> >>>> >> > cloudfront.blogspot.com
> >>> >>>> >> >
> >>> >>>> >> >
> >>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >>> >>>> >> > <ar...@gmail.com> wrote:
> >>> >>>> >> >>
> >>> >>>> >> >> Hi,
> >>> >>>> >> >>
> >>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >>> >>>> >> >>
> >>> >>>> >> >> I would like to "turn off" this feature.
> >>> >>>> >> >>
> >>> >>>> >> >> Request help regarding the same.
> >>> >>>> >> >>
> >>> >>>> >> >> Thanks & regards
> >>> >>>> >> >> Arko
> >>> >>>> >> >>
> >>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <
> cbaker@sdl.com>
> >>> >>>> >> >> wrote:
> >>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the
> >>> >>>> >> >> > NameNode
> >>> >>>> >> >> > which
> >>> >>>> >> >> > is
> >>> >>>> >> >> > not recommended practice. Normally, the only files the NN
> >>> >>>> >> >> > stores
> >>> >>>> >> >> > are
> >>> >>>> >> >> > the
> >>> >>>> >> >> > image and edits files. It does not store any actual HDFS
> >>> >>>> >> >> > data. If
> >>> >>>> >> >> > you
> >>> >>>> >> >> > must
> >>> >>>> >> >> > run a DN on the NN, try turning down the logging in
> >>> >>>> >> >> > /conf/log4j.properties:
> >>> >>>> >> >> >
> >>> >>>> >> >> > #hadoop.root.logger=INFO,console
> >>> >>>> >> >> > #hadoop.root.logger=WARN,console
> >>> >>>> >> >> > hadoop.root.logger=ERROR,console
> >>> >>>> >> >> >
> >>> >>>> >> >> > Depending on the logging information you require, of
> course.
> >>> >>>> >> >> >
> >>> >>>> >> >> > -Chuck
> >>> >>>> >> >> >
> >>> >>>> >> >> >
> >>> >>>> >> >> > -----Original Message-----
> >>> >>>> >> >> > From: Arko Provo Mukherjee
> >>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
> >>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >>> >>>> >> >> > Subject: Managing space in Master Node
> >>> >>>> >> >> >
> >>> >>>> >> >> > Hello Gurus,
> >>> >>>> >> >> >
> >>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
> >>> >>>> >> >> >
> >>> >>>> >> >> > The issue I am continuously facing is that the Master Node
> >>> >>>> >> >> > runs
> >>> >>>> >> >> > out
> >>> >>>> >> >> > of
> >>> >>>> >> >> > disk
> >>> >>>> >> >> > space due to logs and data files.
> >>> >>>> >> >> >
> >>> >>>> >> >> > I can monitor and delete log files. However, I cannot
> delete
> >>> >>>> >> >> > the
> >>> >>>> >> >> > HDFS
> >>> >>>> >> >> > data.
> >>> >>>> >> >> >
> >>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS
> >>> >>>> >> >> > data in
> >>> >>>> >> >> > the
> >>> >>>> >> >> > Master
> >>> >>>> >> >> > Node?
> >>> >>>> >> >> >
> >>> >>>> >> >> > Then I can use my master to handle the metadata only and
> >>> >>>> >> >> > store the
> >>> >>>> >> >> > logs.
> >>> >>>> >> >> >
> >>> >>>> >> >> > Thanks & regards
> >>> >>>> >> >> > Arko
> >>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved.
> The
> >>> >>>> >> >> > information contained in this email may be confidential
> >>> >>>> >> >> > and/or
> >>> >>>> >> >> > legally
> >>> >>>> >> >> > privileged. It has been sent for the sole use of the
> intended
> >>> >>>> >> >> > recipient(s).
> >>> >>>> >> >> > If you are not the intended recipient of this mail, you
> are
> >>> >>>> >> >> > hereby
> >>> >>>> >> >> > notified
> >>> >>>> >> >> > that any unauthorized review, use, disclosure,
> dissemination,
> >>> >>>> >> >> > distribution,
> >>> >>>> >> >> > or copying of this communication, or any of its contents,
> is
> >>> >>>> >> >> > strictly
> >>> >>>> >> >> > prohibited. If you have received this communication in
> error,
> >>> >>>> >> >> > please
> >>> >>>> >> >> > reply
> >>> >>>> >> >> > to the sender and destroy all copies of the message.
> >>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
> >>> >>>> >> >> > Wakefield, MA
> >>> >>>> >> >> > 01880,
> >>> >>>> >> >> > USA
> >>> >>>> >> >> >
> >>> >>>> >> >
> >>> >>>> >> >
> >>> >>>> >
> >>> >>>> >
> >>> >>>
> >>> >>>
> >>
> >>
>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
I see that the NameNode IP is present in conf/slaves file.
Will removing the IP of the NN from the conf/slaves file and
restarting hadoop help?
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 5:32 PM, Arko Provo Mukherjee
<ar...@gmail.com> wrote:
> Hi,
>
> I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
> (running DN and Task Tracker).
>
> My issue is that the my NN is getting filled up pretty fast as the NN
> acts as a DN.
>
> I want my NN not to function as the DN.
>
> Does including the IP for the NN in the "dfs.hosts.exclude" and
> running "$ hadoop dfsadmin -refreshNodes" solve my problem?
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <do...@gmail.com> wrote:
>> Stopping DN on one machine means excluding it. What exactly is the
>> requirement? How many nodes do you have right now?
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
>> <ar...@gmail.com> wrote:
>>>
>>> Hi,
>>>
>>> I am not sure the provided link matches my situation.
>>>
>>> That is because, I am not trying to exclude any node but just trying
>>> to stop the NN act as a DataNode.
>>>
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>>> <ar...@gmail.com> wrote:
>>> > Thanks! Let me study and find out if it works for me.
>>> > Thanks & regards
>>> > Arko
>>> >
>>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>>> > <ar...@gmail.com> wrote:
>>> >> Hi,
>>> >>
>>> >> But the new location will also be the same NN physical server.
>>> >> So how does that help?
>>> >>
>>> >> I would like to clear up the space in the master by removing all the
>>> >> HDFS data to the slave nodes.
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <do...@gmail.com>
>>> >> wrote:
>>> >>> No, copy this directory to the new location of your choice.
>>> >>>
>>> >>> Warm Regards,
>>> >>> Tariq
>>> >>> https://mtariq.jux.com/
>>> >>> cloudfront.blogspot.com
>>> >>>
>>> >>>
>>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> >>> <ar...@gmail.com> wrote:
>>> >>>>
>>> >>>> Thanks!
>>> >>>> However, in that case, how do I handle the current data that is
>>> >>>> already in
>>> >>>> NN?
>>> >>>> Do I have to reformat HDFS?
>>> >>>> Thanks & regards
>>> >>>> Arko
>>> >>>>
>>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
>>> >>>> wrote:
>>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
>>> >>>> > to
>>> >>>> > NN.
>>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >>>> >
>>> >>>> > Warm Regards,
>>> >>>> > Tariq
>>> >>>> > https://mtariq.jux.com/
>>> >>>> > cloudfront.blogspot.com
>>> >>>> >
>>> >>>> >
>>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> >>>> > <ar...@gmail.com> wrote:
>>> >>>> >>
>>> >>>> >> Hi,
>>> >>>> >>
>>> >>>> >> Thanks for the help!
>>> >>>> >>
>>> >>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >>>> >> feature on the NN.
>>> >>>> >>
>>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
>>> >>>> >> to
>>> >>>> >> a directory.
>>> >>>> >>
>>> >>>> >> Should I just comment out that property? What would happen to the
>>> >>>> >> current data that there is in the Master? Will it get removed
>>> >>>> >> automatically?
>>> >>>> >>
>>> >>>> >> Thanks & regards
>>> >>>> >> Arko
>>> >>>> >>
>>> >>>> >>
>>> >>>> >>
>>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>>> >>>> >> <do...@gmail.com>
>>> >>>> >> wrote:
>>> >>>> >> > You can specify the logging level as specified by Charles. But
>>> >>>> >> > turning
>>> >>>> >> > logs
>>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >>>> >> > diagnosis,
>>> >>>> >> > which are eventual.
>>> >>>> >> >
>>> >>>> >> > Warm Regards,
>>> >>>> >> > Tariq
>>> >>>> >> > https://mtariq.jux.com/
>>> >>>> >> > cloudfront.blogspot.com
>>> >>>> >> >
>>> >>>> >> >
>>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>> >>>> >> > <ar...@gmail.com> wrote:
>>> >>>> >> >>
>>> >>>> >> >> Hi,
>>> >>>> >> >>
>>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>> >>>> >> >>
>>> >>>> >> >> I would like to "turn off" this feature.
>>> >>>> >> >>
>>> >>>> >> >> Request help regarding the same.
>>> >>>> >> >>
>>> >>>> >> >> Thanks & regards
>>> >>>> >> >> Arko
>>> >>>> >> >>
>>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
>>> >>>> >> >> wrote:
>>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the
>>> >>>> >> >> > NameNode
>>> >>>> >> >> > which
>>> >>>> >> >> > is
>>> >>>> >> >> > not recommended practice. Normally, the only files the NN
>>> >>>> >> >> > stores
>>> >>>> >> >> > are
>>> >>>> >> >> > the
>>> >>>> >> >> > image and edits files. It does not store any actual HDFS
>>> >>>> >> >> > data. If
>>> >>>> >> >> > you
>>> >>>> >> >> > must
>>> >>>> >> >> > run a DN on the NN, try turning down the logging in
>>> >>>> >> >> > /conf/log4j.properties:
>>> >>>> >> >> >
>>> >>>> >> >> > #hadoop.root.logger=INFO,console
>>> >>>> >> >> > #hadoop.root.logger=WARN,console
>>> >>>> >> >> > hadoop.root.logger=ERROR,console
>>> >>>> >> >> >
>>> >>>> >> >> > Depending on the logging information you require, of course.
>>> >>>> >> >> >
>>> >>>> >> >> > -Chuck
>>> >>>> >> >> >
>>> >>>> >> >> >
>>> >>>> >> >> > -----Original Message-----
>>> >>>> >> >> > From: Arko Provo Mukherjee
>>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
>>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
>>> >>>> >> >> > Subject: Managing space in Master Node
>>> >>>> >> >> >
>>> >>>> >> >> > Hello Gurus,
>>> >>>> >> >> >
>>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>> >>>> >> >> >
>>> >>>> >> >> > The issue I am continuously facing is that the Master Node
>>> >>>> >> >> > runs
>>> >>>> >> >> > out
>>> >>>> >> >> > of
>>> >>>> >> >> > disk
>>> >>>> >> >> > space due to logs and data files.
>>> >>>> >> >> >
>>> >>>> >> >> > I can monitor and delete log files. However, I cannot delete
>>> >>>> >> >> > the
>>> >>>> >> >> > HDFS
>>> >>>> >> >> > data.
>>> >>>> >> >> >
>>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS
>>> >>>> >> >> > data in
>>> >>>> >> >> > the
>>> >>>> >> >> > Master
>>> >>>> >> >> > Node?
>>> >>>> >> >> >
>>> >>>> >> >> > Then I can use my master to handle the metadata only and
>>> >>>> >> >> > store the
>>> >>>> >> >> > logs.
>>> >>>> >> >> >
>>> >>>> >> >> > Thanks & regards
>>> >>>> >> >> > Arko
>>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>>> >>>> >> >> > information contained in this email may be confidential
>>> >>>> >> >> > and/or
>>> >>>> >> >> > legally
>>> >>>> >> >> > privileged. It has been sent for the sole use of the intended
>>> >>>> >> >> > recipient(s).
>>> >>>> >> >> > If you are not the intended recipient of this mail, you are
>>> >>>> >> >> > hereby
>>> >>>> >> >> > notified
>>> >>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>>> >>>> >> >> > distribution,
>>> >>>> >> >> > or copying of this communication, or any of its contents, is
>>> >>>> >> >> > strictly
>>> >>>> >> >> > prohibited. If you have received this communication in error,
>>> >>>> >> >> > please
>>> >>>> >> >> > reply
>>> >>>> >> >> > to the sender and destroy all copies of the message.
>>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
>>> >>>> >> >> > Wakefield, MA
>>> >>>> >> >> > 01880,
>>> >>>> >> >> > USA
>>> >>>> >> >> >
>>> >>>> >> >
>>> >>>> >> >
>>> >>>> >
>>> >>>> >
>>> >>>
>>> >>>
>>
>>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
I have 10 nodes. 1 Master Node (running all deamons) and 9 Slave Nodes
(running DN and Task Tracker).
My issue is that the my NN is getting filled up pretty fast as the NN
acts as a DN.
I want my NN not to function as the DN.
Does including the IP for the NN in the "dfs.hosts.exclude" and
running "$ hadoop dfsadmin -refreshNodes" solve my problem?
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 5:22 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Stopping DN on one machine means excluding it. What exactly is the
> requirement? How many nodes do you have right now?
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> I am not sure the provided link matches my situation.
>>
>> That is because, I am not trying to exclude any node but just trying
>> to stop the NN act as a DataNode.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
>> <ar...@gmail.com> wrote:
>> > Thanks! Let me study and find out if it works for me.
>> > Thanks & regards
>> > Arko
>> >
>> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
>> > <ar...@gmail.com> wrote:
>> >> Hi,
>> >>
>> >> But the new location will also be the same NN physical server.
>> >> So how does that help?
>> >>
>> >> I would like to clear up the space in the master by removing all the
>> >> HDFS data to the slave nodes.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <do...@gmail.com>
>> >> wrote:
>> >>> No, copy this directory to the new location of your choice.
>> >>>
>> >>> Warm Regards,
>> >>> Tariq
>> >>> https://mtariq.jux.com/
>> >>> cloudfront.blogspot.com
>> >>>
>> >>>
>> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> >>> <ar...@gmail.com> wrote:
>> >>>>
>> >>>> Thanks!
>> >>>> However, in that case, how do I handle the current data that is
>> >>>> already in
>> >>>> NN?
>> >>>> Do I have to reformat HDFS?
>> >>>> Thanks & regards
>> >>>> Arko
>> >>>>
>> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
>> >>>> wrote:
>> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
>> >>>> > to
>> >>>> > NN.
>> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >>>> >
>> >>>> > Warm Regards,
>> >>>> > Tariq
>> >>>> > https://mtariq.jux.com/
>> >>>> > cloudfront.blogspot.com
>> >>>> >
>> >>>> >
>> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> >>>> > <ar...@gmail.com> wrote:
>> >>>> >>
>> >>>> >> Hi,
>> >>>> >>
>> >>>> >> Thanks for the help!
>> >>>> >>
>> >>>> >> However, I am still unsure about how to "turn off" the datanode
>> >>>> >> feature on the NN.
>> >>>> >>
>> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
>> >>>> >> to
>> >>>> >> a directory.
>> >>>> >>
>> >>>> >> Should I just comment out that property? What would happen to the
>> >>>> >> current data that there is in the Master? Will it get removed
>> >>>> >> automatically?
>> >>>> >>
>> >>>> >> Thanks & regards
>> >>>> >> Arko
>> >>>> >>
>> >>>> >>
>> >>>> >>
>> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq
>> >>>> >> <do...@gmail.com>
>> >>>> >> wrote:
>> >>>> >> > You can specify the logging level as specified by Charles. But
>> >>>> >> > turning
>> >>>> >> > logs
>> >>>> >> > off is never a good idea. Logs are really helpful in problem
>> >>>> >> > diagnosis,
>> >>>> >> > which are eventual.
>> >>>> >> >
>> >>>> >> > Warm Regards,
>> >>>> >> > Tariq
>> >>>> >> > https://mtariq.jux.com/
>> >>>> >> > cloudfront.blogspot.com
>> >>>> >> >
>> >>>> >> >
>> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >>>> >> > <ar...@gmail.com> wrote:
>> >>>> >> >>
>> >>>> >> >> Hi,
>> >>>> >> >>
>> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>>> >> >>
>> >>>> >> >> I would like to "turn off" this feature.
>> >>>> >> >>
>> >>>> >> >> Request help regarding the same.
>> >>>> >> >>
>> >>>> >> >> Thanks & regards
>> >>>> >> >> Arko
>> >>>> >> >>
>> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
>> >>>> >> >> wrote:
>> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the
>> >>>> >> >> > NameNode
>> >>>> >> >> > which
>> >>>> >> >> > is
>> >>>> >> >> > not recommended practice. Normally, the only files the NN
>> >>>> >> >> > stores
>> >>>> >> >> > are
>> >>>> >> >> > the
>> >>>> >> >> > image and edits files. It does not store any actual HDFS
>> >>>> >> >> > data. If
>> >>>> >> >> > you
>> >>>> >> >> > must
>> >>>> >> >> > run a DN on the NN, try turning down the logging in
>> >>>> >> >> > /conf/log4j.properties:
>> >>>> >> >> >
>> >>>> >> >> > #hadoop.root.logger=INFO,console
>> >>>> >> >> > #hadoop.root.logger=WARN,console
>> >>>> >> >> > hadoop.root.logger=ERROR,console
>> >>>> >> >> >
>> >>>> >> >> > Depending on the logging information you require, of course.
>> >>>> >> >> >
>> >>>> >> >> > -Chuck
>> >>>> >> >> >
>> >>>> >> >> >
>> >>>> >> >> > -----Original Message-----
>> >>>> >> >> > From: Arko Provo Mukherjee
>> >>>> >> >> > [mailto:arkoprovomukherjee@gmail.com]
>> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >>>> >> >> > To: hdfs-user@hadoop.apache.org
>> >>>> >> >> > Subject: Managing space in Master Node
>> >>>> >> >> >
>> >>>> >> >> > Hello Gurus,
>> >>>> >> >> >
>> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>> >>>> >> >> >
>> >>>> >> >> > The issue I am continuously facing is that the Master Node
>> >>>> >> >> > runs
>> >>>> >> >> > out
>> >>>> >> >> > of
>> >>>> >> >> > disk
>> >>>> >> >> > space due to logs and data files.
>> >>>> >> >> >
>> >>>> >> >> > I can monitor and delete log files. However, I cannot delete
>> >>>> >> >> > the
>> >>>> >> >> > HDFS
>> >>>> >> >> > data.
>> >>>> >> >> >
>> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS
>> >>>> >> >> > data in
>> >>>> >> >> > the
>> >>>> >> >> > Master
>> >>>> >> >> > Node?
>> >>>> >> >> >
>> >>>> >> >> > Then I can use my master to handle the metadata only and
>> >>>> >> >> > store the
>> >>>> >> >> > logs.
>> >>>> >> >> >
>> >>>> >> >> > Thanks & regards
>> >>>> >> >> > Arko
>> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>> >>>> >> >> > information contained in this email may be confidential
>> >>>> >> >> > and/or
>> >>>> >> >> > legally
>> >>>> >> >> > privileged. It has been sent for the sole use of the intended
>> >>>> >> >> > recipient(s).
>> >>>> >> >> > If you are not the intended recipient of this mail, you are
>> >>>> >> >> > hereby
>> >>>> >> >> > notified
>> >>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>> >>>> >> >> > distribution,
>> >>>> >> >> > or copying of this communication, or any of its contents, is
>> >>>> >> >> > strictly
>> >>>> >> >> > prohibited. If you have received this communication in error,
>> >>>> >> >> > please
>> >>>> >> >> > reply
>> >>>> >> >> > to the sender and destroy all copies of the message.
>> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
>> >>>> >> >> > Wakefield, MA
>> >>>> >> >> > 01880,
>> >>>> >> >> > USA
>> >>>> >> >> >
>> >>>> >> >
>> >>>> >> >
>> >>>> >
>> >>>> >
>> >>>
>> >>>
>
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
Stopping DN on one machine means excluding it. What exactly is the
requirement? How many nodes do you have right now?
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 4:36 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Hi,
>
> I am not sure the provided link matches my situation.
>
> That is because, I am not trying to exclude any node but just trying
> to stop the NN act as a DataNode.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
> > Thanks! Let me study and find out if it works for me.
> > Thanks & regards
> > Arko
> >
> > On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> > <ar...@gmail.com> wrote:
> >> Hi,
> >>
> >> But the new location will also be the same NN physical server.
> >> So how does that help?
> >>
> >> I would like to clear up the space in the master by removing all the
> >> HDFS data to the slave nodes.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> >>> No, copy this directory to the new location of your choice.
> >>>
> >>> Warm Regards,
> >>> Tariq
> >>> https://mtariq.jux.com/
> >>> cloudfront.blogspot.com
> >>>
> >>>
> >>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> >>> <ar...@gmail.com> wrote:
> >>>>
> >>>> Thanks!
> >>>> However, in that case, how do I handle the current data that is
> already in
> >>>> NN?
> >>>> Do I have to reformat HDFS?
> >>>> Thanks & regards
> >>>> Arko
> >>>>
> >>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
> >>>> wrote:
> >>>> > Just don't run the DN daemon on that machine. Dedicate that machine
> to
> >>>> > NN.
> >>>> > Remove the hostname of the NN machine from the 'slaves' file.
> >>>> >
> >>>> > Warm Regards,
> >>>> > Tariq
> >>>> > https://mtariq.jux.com/
> >>>> > cloudfront.blogspot.com
> >>>> >
> >>>> >
> >>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> >>>> > <ar...@gmail.com> wrote:
> >>>> >>
> >>>> >> Hi,
> >>>> >>
> >>>> >> Thanks for the help!
> >>>> >>
> >>>> >> However, I am still unsure about how to "turn off" the datanode
> >>>> >> feature on the NN.
> >>>> >>
> >>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed
> to
> >>>> >> a directory.
> >>>> >>
> >>>> >> Should I just comment out that property? What would happen to the
> >>>> >> current data that there is in the Master? Will it get removed
> >>>> >> automatically?
> >>>> >>
> >>>> >> Thanks & regards
> >>>> >> Arko
> >>>> >>
> >>>> >>
> >>>> >>
> >>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <
> dontariq@gmail.com>
> >>>> >> wrote:
> >>>> >> > You can specify the logging level as specified by Charles. But
> >>>> >> > turning
> >>>> >> > logs
> >>>> >> > off is never a good idea. Logs are really helpful in problem
> >>>> >> > diagnosis,
> >>>> >> > which are eventual.
> >>>> >> >
> >>>> >> > Warm Regards,
> >>>> >> > Tariq
> >>>> >> > https://mtariq.jux.com/
> >>>> >> > cloudfront.blogspot.com
> >>>> >> >
> >>>> >> >
> >>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >>>> >> > <ar...@gmail.com> wrote:
> >>>> >> >>
> >>>> >> >> Hi,
> >>>> >> >>
> >>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >>>> >> >>
> >>>> >> >> I would like to "turn off" this feature.
> >>>> >> >>
> >>>> >> >> Request help regarding the same.
> >>>> >> >>
> >>>> >> >> Thanks & regards
> >>>> >> >> Arko
> >>>> >> >>
> >>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
> >>>> >> >> wrote:
> >>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the
> NameNode
> >>>> >> >> > which
> >>>> >> >> > is
> >>>> >> >> > not recommended practice. Normally, the only files the NN
> stores
> >>>> >> >> > are
> >>>> >> >> > the
> >>>> >> >> > image and edits files. It does not store any actual HDFS
> data. If
> >>>> >> >> > you
> >>>> >> >> > must
> >>>> >> >> > run a DN on the NN, try turning down the logging in
> >>>> >> >> > /conf/log4j.properties:
> >>>> >> >> >
> >>>> >> >> > #hadoop.root.logger=INFO,console
> >>>> >> >> > #hadoop.root.logger=WARN,console
> >>>> >> >> > hadoop.root.logger=ERROR,console
> >>>> >> >> >
> >>>> >> >> > Depending on the logging information you require, of course.
> >>>> >> >> >
> >>>> >> >> > -Chuck
> >>>> >> >> >
> >>>> >> >> >
> >>>> >> >> > -----Original Message-----
> >>>> >> >> > From: Arko Provo Mukherjee [mailto:
> arkoprovomukherjee@gmail.com]
> >>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >>>> >> >> > To: hdfs-user@hadoop.apache.org
> >>>> >> >> > Subject: Managing space in Master Node
> >>>> >> >> >
> >>>> >> >> > Hello Gurus,
> >>>> >> >> >
> >>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
> >>>> >> >> >
> >>>> >> >> > The issue I am continuously facing is that the Master Node
> runs
> >>>> >> >> > out
> >>>> >> >> > of
> >>>> >> >> > disk
> >>>> >> >> > space due to logs and data files.
> >>>> >> >> >
> >>>> >> >> > I can monitor and delete log files. However, I cannot delete
> the
> >>>> >> >> > HDFS
> >>>> >> >> > data.
> >>>> >> >> >
> >>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS
> data in
> >>>> >> >> > the
> >>>> >> >> > Master
> >>>> >> >> > Node?
> >>>> >> >> >
> >>>> >> >> > Then I can use my master to handle the metadata only and
> store the
> >>>> >> >> > logs.
> >>>> >> >> >
> >>>> >> >> > Thanks & regards
> >>>> >> >> > Arko
> >>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
> >>>> >> >> > information contained in this email may be confidential and/or
> >>>> >> >> > legally
> >>>> >> >> > privileged. It has been sent for the sole use of the intended
> >>>> >> >> > recipient(s).
> >>>> >> >> > If you are not the intended recipient of this mail, you are
> hereby
> >>>> >> >> > notified
> >>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
> >>>> >> >> > distribution,
> >>>> >> >> > or copying of this communication, or any of its contents, is
> >>>> >> >> > strictly
> >>>> >> >> > prohibited. If you have received this communication in error,
> >>>> >> >> > please
> >>>> >> >> > reply
> >>>> >> >> > to the sender and destroy all copies of the message.
> >>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225,
> Wakefield, MA
> >>>> >> >> > 01880,
> >>>> >> >> > USA
> >>>> >> >> >
> >>>> >> >
> >>>> >> >
> >>>> >
> >>>> >
> >>>
> >>>
>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
I am not sure the provided link matches my situation.
That is because, I am not trying to exclude any node but just trying
to stop the NN act as a DataNode.
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 2:31 PM, Arko Provo Mukherjee
<ar...@gmail.com> wrote:
> Thanks! Let me study and find out if it works for me.
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>> Hi,
>>
>> But the new location will also be the same NN physical server.
>> So how does that help?
>>
>> I would like to clear up the space in the master by removing all the
>> HDFS data to the slave nodes.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <do...@gmail.com> wrote:
>>> No, copy this directory to the new location of your choice.
>>>
>>> Warm Regards,
>>> Tariq
>>> https://mtariq.jux.com/
>>> cloudfront.blogspot.com
>>>
>>>
>>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>>> <ar...@gmail.com> wrote:
>>>>
>>>> Thanks!
>>>> However, in that case, how do I handle the current data that is already in
>>>> NN?
>>>> Do I have to reformat HDFS?
>>>> Thanks & regards
>>>> Arko
>>>>
>>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
>>>> wrote:
>>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>>> > NN.
>>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>>> >
>>>> > Warm Regards,
>>>> > Tariq
>>>> > https://mtariq.jux.com/
>>>> > cloudfront.blogspot.com
>>>> >
>>>> >
>>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>>> > <ar...@gmail.com> wrote:
>>>> >>
>>>> >> Hi,
>>>> >>
>>>> >> Thanks for the help!
>>>> >>
>>>> >> However, I am still unsure about how to "turn off" the datanode
>>>> >> feature on the NN.
>>>> >>
>>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>>> >> a directory.
>>>> >>
>>>> >> Should I just comment out that property? What would happen to the
>>>> >> current data that there is in the Master? Will it get removed
>>>> >> automatically?
>>>> >>
>>>> >> Thanks & regards
>>>> >> Arko
>>>> >>
>>>> >>
>>>> >>
>>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
>>>> >> wrote:
>>>> >> > You can specify the logging level as specified by Charles. But
>>>> >> > turning
>>>> >> > logs
>>>> >> > off is never a good idea. Logs are really helpful in problem
>>>> >> > diagnosis,
>>>> >> > which are eventual.
>>>> >> >
>>>> >> > Warm Regards,
>>>> >> > Tariq
>>>> >> > https://mtariq.jux.com/
>>>> >> > cloudfront.blogspot.com
>>>> >> >
>>>> >> >
>>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>>> >> > <ar...@gmail.com> wrote:
>>>> >> >>
>>>> >> >> Hi,
>>>> >> >>
>>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>>> >> >>
>>>> >> >> I would like to "turn off" this feature.
>>>> >> >>
>>>> >> >> Request help regarding the same.
>>>> >> >>
>>>> >> >> Thanks & regards
>>>> >> >> Arko
>>>> >> >>
>>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
>>>> >> >> wrote:
>>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>>>> >> >> > which
>>>> >> >> > is
>>>> >> >> > not recommended practice. Normally, the only files the NN stores
>>>> >> >> > are
>>>> >> >> > the
>>>> >> >> > image and edits files. It does not store any actual HDFS data. If
>>>> >> >> > you
>>>> >> >> > must
>>>> >> >> > run a DN on the NN, try turning down the logging in
>>>> >> >> > /conf/log4j.properties:
>>>> >> >> >
>>>> >> >> > #hadoop.root.logger=INFO,console
>>>> >> >> > #hadoop.root.logger=WARN,console
>>>> >> >> > hadoop.root.logger=ERROR,console
>>>> >> >> >
>>>> >> >> > Depending on the logging information you require, of course.
>>>> >> >> >
>>>> >> >> > -Chuck
>>>> >> >> >
>>>> >> >> >
>>>> >> >> > -----Original Message-----
>>>> >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>>> >> >> > To: hdfs-user@hadoop.apache.org
>>>> >> >> > Subject: Managing space in Master Node
>>>> >> >> >
>>>> >> >> > Hello Gurus,
>>>> >> >> >
>>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>>> >> >> >
>>>> >> >> > The issue I am continuously facing is that the Master Node runs
>>>> >> >> > out
>>>> >> >> > of
>>>> >> >> > disk
>>>> >> >> > space due to logs and data files.
>>>> >> >> >
>>>> >> >> > I can monitor and delete log files. However, I cannot delete the
>>>> >> >> > HDFS
>>>> >> >> > data.
>>>> >> >> >
>>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
>>>> >> >> > the
>>>> >> >> > Master
>>>> >> >> > Node?
>>>> >> >> >
>>>> >> >> > Then I can use my master to handle the metadata only and store the
>>>> >> >> > logs.
>>>> >> >> >
>>>> >> >> > Thanks & regards
>>>> >> >> > Arko
>>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>>>> >> >> > information contained in this email may be confidential and/or
>>>> >> >> > legally
>>>> >> >> > privileged. It has been sent for the sole use of the intended
>>>> >> >> > recipient(s).
>>>> >> >> > If you are not the intended recipient of this mail, you are hereby
>>>> >> >> > notified
>>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>>>> >> >> > distribution,
>>>> >> >> > or copying of this communication, or any of its contents, is
>>>> >> >> > strictly
>>>> >> >> > prohibited. If you have received this communication in error,
>>>> >> >> > please
>>>> >> >> > reply
>>>> >> >> > to the sender and destroy all copies of the message.
>>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
>>>> >> >> > 01880,
>>>> >> >> > USA
>>>> >> >> >
>>>> >> >
>>>> >> >
>>>> >
>>>> >
>>>
>>>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Thanks! Let me study and find out if it works for me.
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 2:28 PM, Arko Provo Mukherjee
<ar...@gmail.com> wrote:
> Hi,
>
> But the new location will also be the same NN physical server.
> So how does that help?
>
> I would like to clear up the space in the master by removing all the
> HDFS data to the slave nodes.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <do...@gmail.com> wrote:
>> No, copy this directory to the new location of your choice.
>>
>> Warm Regards,
>> Tariq
>> https://mtariq.jux.com/
>> cloudfront.blogspot.com
>>
>>
>> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
>> <ar...@gmail.com> wrote:
>>>
>>> Thanks!
>>> However, in that case, how do I handle the current data that is already in
>>> NN?
>>> Do I have to reformat HDFS?
>>> Thanks & regards
>>> Arko
>>>
>>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
>>> wrote:
>>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>>> > NN.
>>> > Remove the hostname of the NN machine from the 'slaves' file.
>>> >
>>> > Warm Regards,
>>> > Tariq
>>> > https://mtariq.jux.com/
>>> > cloudfront.blogspot.com
>>> >
>>> >
>>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>>> > <ar...@gmail.com> wrote:
>>> >>
>>> >> Hi,
>>> >>
>>> >> Thanks for the help!
>>> >>
>>> >> However, I am still unsure about how to "turn off" the datanode
>>> >> feature on the NN.
>>> >>
>>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>>> >> a directory.
>>> >>
>>> >> Should I just comment out that property? What would happen to the
>>> >> current data that there is in the Master? Will it get removed
>>> >> automatically?
>>> >>
>>> >> Thanks & regards
>>> >> Arko
>>> >>
>>> >>
>>> >>
>>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
>>> >> wrote:
>>> >> > You can specify the logging level as specified by Charles. But
>>> >> > turning
>>> >> > logs
>>> >> > off is never a good idea. Logs are really helpful in problem
>>> >> > diagnosis,
>>> >> > which are eventual.
>>> >> >
>>> >> > Warm Regards,
>>> >> > Tariq
>>> >> > https://mtariq.jux.com/
>>> >> > cloudfront.blogspot.com
>>> >> >
>>> >> >
>>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>>> >> > <ar...@gmail.com> wrote:
>>> >> >>
>>> >> >> Hi,
>>> >> >>
>>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>>> >> >>
>>> >> >> I would like to "turn off" this feature.
>>> >> >>
>>> >> >> Request help regarding the same.
>>> >> >>
>>> >> >> Thanks & regards
>>> >> >> Arko
>>> >> >>
>>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
>>> >> >> wrote:
>>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>>> >> >> > which
>>> >> >> > is
>>> >> >> > not recommended practice. Normally, the only files the NN stores
>>> >> >> > are
>>> >> >> > the
>>> >> >> > image and edits files. It does not store any actual HDFS data. If
>>> >> >> > you
>>> >> >> > must
>>> >> >> > run a DN on the NN, try turning down the logging in
>>> >> >> > /conf/log4j.properties:
>>> >> >> >
>>> >> >> > #hadoop.root.logger=INFO,console
>>> >> >> > #hadoop.root.logger=WARN,console
>>> >> >> > hadoop.root.logger=ERROR,console
>>> >> >> >
>>> >> >> > Depending on the logging information you require, of course.
>>> >> >> >
>>> >> >> > -Chuck
>>> >> >> >
>>> >> >> >
>>> >> >> > -----Original Message-----
>>> >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>>> >> >> > To: hdfs-user@hadoop.apache.org
>>> >> >> > Subject: Managing space in Master Node
>>> >> >> >
>>> >> >> > Hello Gurus,
>>> >> >> >
>>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>>> >> >> >
>>> >> >> > The issue I am continuously facing is that the Master Node runs
>>> >> >> > out
>>> >> >> > of
>>> >> >> > disk
>>> >> >> > space due to logs and data files.
>>> >> >> >
>>> >> >> > I can monitor and delete log files. However, I cannot delete the
>>> >> >> > HDFS
>>> >> >> > data.
>>> >> >> >
>>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
>>> >> >> > the
>>> >> >> > Master
>>> >> >> > Node?
>>> >> >> >
>>> >> >> > Then I can use my master to handle the metadata only and store the
>>> >> >> > logs.
>>> >> >> >
>>> >> >> > Thanks & regards
>>> >> >> > Arko
>>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>>> >> >> > information contained in this email may be confidential and/or
>>> >> >> > legally
>>> >> >> > privileged. It has been sent for the sole use of the intended
>>> >> >> > recipient(s).
>>> >> >> > If you are not the intended recipient of this mail, you are hereby
>>> >> >> > notified
>>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>>> >> >> > distribution,
>>> >> >> > or copying of this communication, or any of its contents, is
>>> >> >> > strictly
>>> >> >> > prohibited. If you have received this communication in error,
>>> >> >> > please
>>> >> >> > reply
>>> >> >> > to the sender and destroy all copies of the message.
>>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
>>> >> >> > 01880,
>>> >> >> > USA
>>> >> >> >
>>> >> >
>>> >> >
>>> >
>>> >
>>
>>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
But the new location will also be the same NN physical server.
So how does that help?
I would like to clear up the space in the master by removing all the
HDFS data to the slave nodes.
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 2:14 PM, Mohammad Tariq <do...@gmail.com> wrote:
> No, copy this directory to the new location of your choice.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Thanks!
>> However, in that case, how do I handle the current data that is already in
>> NN?
>> Do I have to reformat HDFS?
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
>> wrote:
>> > Just don't run the DN daemon on that machine. Dedicate that machine to
>> > NN.
>> > Remove the hostname of the NN machine from the 'slaves' file.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
>> > <ar...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Thanks for the help!
>> >>
>> >> However, I am still unsure about how to "turn off" the datanode
>> >> feature on the NN.
>> >>
>> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> >> a directory.
>> >>
>> >> Should I just comment out that property? What would happen to the
>> >> current data that there is in the Master? Will it get removed
>> >> automatically?
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >>
>> >>
>> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
>> >> wrote:
>> >> > You can specify the logging level as specified by Charles. But
>> >> > turning
>> >> > logs
>> >> > off is never a good idea. Logs are really helpful in problem
>> >> > diagnosis,
>> >> > which are eventual.
>> >> >
>> >> > Warm Regards,
>> >> > Tariq
>> >> > https://mtariq.jux.com/
>> >> > cloudfront.blogspot.com
>> >> >
>> >> >
>> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> >> > <ar...@gmail.com> wrote:
>> >> >>
>> >> >> Hi,
>> >> >>
>> >> >> Yeah, my NameNode is also seconding as a DataNode.
>> >> >>
>> >> >> I would like to "turn off" this feature.
>> >> >>
>> >> >> Request help regarding the same.
>> >> >>
>> >> >> Thanks & regards
>> >> >> Arko
>> >> >>
>> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
>> >> >> wrote:
>> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>> >> >> > which
>> >> >> > is
>> >> >> > not recommended practice. Normally, the only files the NN stores
>> >> >> > are
>> >> >> > the
>> >> >> > image and edits files. It does not store any actual HDFS data. If
>> >> >> > you
>> >> >> > must
>> >> >> > run a DN on the NN, try turning down the logging in
>> >> >> > /conf/log4j.properties:
>> >> >> >
>> >> >> > #hadoop.root.logger=INFO,console
>> >> >> > #hadoop.root.logger=WARN,console
>> >> >> > hadoop.root.logger=ERROR,console
>> >> >> >
>> >> >> > Depending on the logging information you require, of course.
>> >> >> >
>> >> >> > -Chuck
>> >> >> >
>> >> >> >
>> >> >> > -----Original Message-----
>> >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> >> > To: hdfs-user@hadoop.apache.org
>> >> >> > Subject: Managing space in Master Node
>> >> >> >
>> >> >> > Hello Gurus,
>> >> >> >
>> >> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >> >
>> >> >> > The issue I am continuously facing is that the Master Node runs
>> >> >> > out
>> >> >> > of
>> >> >> > disk
>> >> >> > space due to logs and data files.
>> >> >> >
>> >> >> > I can monitor and delete log files. However, I cannot delete the
>> >> >> > HDFS
>> >> >> > data.
>> >> >> >
>> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
>> >> >> > the
>> >> >> > Master
>> >> >> > Node?
>> >> >> >
>> >> >> > Then I can use my master to handle the metadata only and store the
>> >> >> > logs.
>> >> >> >
>> >> >> > Thanks & regards
>> >> >> > Arko
>> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>> >> >> > information contained in this email may be confidential and/or
>> >> >> > legally
>> >> >> > privileged. It has been sent for the sole use of the intended
>> >> >> > recipient(s).
>> >> >> > If you are not the intended recipient of this mail, you are hereby
>> >> >> > notified
>> >> >> > that any unauthorized review, use, disclosure, dissemination,
>> >> >> > distribution,
>> >> >> > or copying of this communication, or any of its contents, is
>> >> >> > strictly
>> >> >> > prohibited. If you have received this communication in error,
>> >> >> > please
>> >> >> > reply
>> >> >> > to the sender and destroy all copies of the message.
>> >> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
>> >> >> > 01880,
>> >> >> > USA
>> >> >> >
>> >> >
>> >> >
>> >
>> >
>
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
No, copy this directory to the new location of your choice.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:42 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Thanks!
> However, in that case, how do I handle the current data that is already in
> NN?
> Do I have to reformat HDFS?
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> > Just don't run the DN daemon on that machine. Dedicate that machine to
> NN.
> > Remove the hostname of the NN machine from the 'slaves' file.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> > <ar...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> Thanks for the help!
> >>
> >> However, I am still unsure about how to "turn off" the datanode
> >> feature on the NN.
> >>
> >> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
> >> a directory.
> >>
> >> Should I just comment out that property? What would happen to the
> >> current data that there is in the Master? Will it get removed
> >> automatically?
> >>
> >> Thanks & regards
> >> Arko
> >>
> >>
> >>
> >> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
> >> wrote:
> >> > You can specify the logging level as specified by Charles. But turning
> >> > logs
> >> > off is never a good idea. Logs are really helpful in problem
> diagnosis,
> >> > which are eventual.
> >> >
> >> > Warm Regards,
> >> > Tariq
> >> > https://mtariq.jux.com/
> >> > cloudfront.blogspot.com
> >> >
> >> >
> >> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> >> > <ar...@gmail.com> wrote:
> >> >>
> >> >> Hi,
> >> >>
> >> >> Yeah, my NameNode is also seconding as a DataNode.
> >> >>
> >> >> I would like to "turn off" this feature.
> >> >>
> >> >> Request help regarding the same.
> >> >>
> >> >> Thanks & regards
> >> >> Arko
> >> >>
> >> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com>
> wrote:
> >> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
> >> >> > which
> >> >> > is
> >> >> > not recommended practice. Normally, the only files the NN stores
> are
> >> >> > the
> >> >> > image and edits files. It does not store any actual HDFS data. If
> you
> >> >> > must
> >> >> > run a DN on the NN, try turning down the logging in
> >> >> > /conf/log4j.properties:
> >> >> >
> >> >> > #hadoop.root.logger=INFO,console
> >> >> > #hadoop.root.logger=WARN,console
> >> >> > hadoop.root.logger=ERROR,console
> >> >> >
> >> >> > Depending on the logging information you require, of course.
> >> >> >
> >> >> > -Chuck
> >> >> >
> >> >> >
> >> >> > -----Original Message-----
> >> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> >> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >> >> > To: hdfs-user@hadoop.apache.org
> >> >> > Subject: Managing space in Master Node
> >> >> >
> >> >> > Hello Gurus,
> >> >> >
> >> >> > I am managing a Hadoop Cluster to run some experiments.
> >> >> >
> >> >> > The issue I am continuously facing is that the Master Node runs out
> >> >> > of
> >> >> > disk
> >> >> > space due to logs and data files.
> >> >> >
> >> >> > I can monitor and delete log files. However, I cannot delete the
> HDFS
> >> >> > data.
> >> >> >
> >> >> > Thus, is there a way to force Hadoop not to save any HDFS data in
> the
> >> >> > Master
> >> >> > Node?
> >> >> >
> >> >> > Then I can use my master to handle the metadata only and store the
> >> >> > logs.
> >> >> >
> >> >> > Thanks & regards
> >> >> > Arko
> >> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
> >> >> > information contained in this email may be confidential and/or
> >> >> > legally
> >> >> > privileged. It has been sent for the sole use of the intended
> >> >> > recipient(s).
> >> >> > If you are not the intended recipient of this mail, you are hereby
> >> >> > notified
> >> >> > that any unauthorized review, use, disclosure, dissemination,
> >> >> > distribution,
> >> >> > or copying of this communication, or any of its contents, is
> strictly
> >> >> > prohibited. If you have received this communication in error,
> please
> >> >> > reply
> >> >> > to the sender and destroy all copies of the message.
> >> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
> >> >> > 01880,
> >> >> > USA
> >> >> >
> >> >
> >> >
> >
> >
>
RE: Managing space in Master Node
Posted by Charles Baker <cb...@sdl.com>.
The best way is to decommission the DN so the blocks get replicated to the
other nodes in the cluster. Manually copying the data is not recommended. See
if this works for you:
http://wiki.apache.org/hadoop/FAQ#I_want_to_make_a_large_cluster_smaller_by_t
aking_out_a_bunch_of_nodes_simultaneously._How_can_this_be_done.3F
-----Original Message-----
From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
Sent: Wednesday, February 13, 2013 12:13 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Managing space in Master Node
Thanks!
However, in that case, how do I handle the current data that is already in
NN?
Do I have to reformat HDFS?
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Just don't run the DN daemon on that machine. Dedicate that machine to NN.
> Remove the hostname of the NN machine from the 'slaves' file.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> Thanks for the help!
>>
>> However, I am still unsure about how to "turn off" the datanode
>> feature on the NN.
>>
>> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> a directory.
>>
>> Should I just comment out that property? What would happen to the
>> current data that there is in the Master? Will it get removed
>> automatically?
>>
>> Thanks & regards
>> Arko
>>
>>
>>
>> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
>> wrote:
>> > You can specify the logging level as specified by Charles. But
>> > turning logs off is never a good idea. Logs are really helpful in
>> > problem diagnosis, which are eventual.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> > <ar...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>
>> >> I would like to "turn off" this feature.
>> >>
>> >> Request help regarding the same.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com> wrote:
>> >> > Hi Arko. Sounds like you may be running a DataNode on the
>> >> > NameNode which is not recommended practice. Normally, the only
>> >> > files the NN stores are the image and edits files. It does not
>> >> > store any actual HDFS data. If you must run a DN on the NN, try
>> >> > turning down the logging in
>> >> > /conf/log4j.properties:
>> >> >
>> >> > #hadoop.root.logger=INFO,console
>> >> > #hadoop.root.logger=WARN,console
>> >> > hadoop.root.logger=ERROR,console
>> >> >
>> >> > Depending on the logging information you require, of course.
>> >> >
>> >> > -Chuck
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> > To: hdfs-user@hadoop.apache.org
>> >> > Subject: Managing space in Master Node
>> >> >
>> >> > Hello Gurus,
>> >> >
>> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >
>> >> > The issue I am continuously facing is that the Master Node runs
>> >> > out of disk space due to logs and data files.
>> >> >
>> >> > I can monitor and delete log files. However, I cannot delete the
>> >> > HDFS data.
>> >> >
>> >> > Thus, is there a way to force Hadoop not to save any HDFS data
>> >> > in the Master Node?
>> >> >
>> >> > Then I can use my master to handle the metadata only and store
>> >> > the logs.
>> >> >
>> >> > Thanks & regards
>> >> > Arko
>> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>> >> > information contained in this email may be confidential and/or
>> >> > legally privileged. It has been sent for the sole use of the
>> >> > intended recipient(s).
>> >> > If you are not the intended recipient of this mail, you are
>> >> > hereby notified that any unauthorized review, use, disclosure,
>> >> > dissemination, distribution, or copying of this communication,
>> >> > or any of its contents, is strictly prohibited. If you have
>> >> > received this communication in error, please reply to the sender
>> >> > and destroy all copies of the message.
>> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield,
>> >> > MA 01880, USA
>> >> >
>> >
>> >
>
>
SDL Enterprise Technologies, Inc. - all rights reserved. The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Thanks!
However, in that case, how do I handle the current data that is already in NN?
Do I have to reformat HDFS?
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 2:06 PM, Mohammad Tariq <do...@gmail.com> wrote:
> Just don't run the DN daemon on that machine. Dedicate that machine to NN.
> Remove the hostname of the NN machine from the 'slaves' file.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> Thanks for the help!
>>
>> However, I am still unsure about how to "turn off" the datanode
>> feature on the NN.
>>
>> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
>> a directory.
>>
>> Should I just comment out that property? What would happen to the
>> current data that there is in the Master? Will it get removed
>> automatically?
>>
>> Thanks & regards
>> Arko
>>
>>
>>
>> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
>> wrote:
>> > You can specify the logging level as specified by Charles. But turning
>> > logs
>> > off is never a good idea. Logs are really helpful in problem diagnosis,
>> > which are eventual.
>> >
>> > Warm Regards,
>> > Tariq
>> > https://mtariq.jux.com/
>> > cloudfront.blogspot.com
>> >
>> >
>> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
>> > <ar...@gmail.com> wrote:
>> >>
>> >> Hi,
>> >>
>> >> Yeah, my NameNode is also seconding as a DataNode.
>> >>
>> >> I would like to "turn off" this feature.
>> >>
>> >> Request help regarding the same.
>> >>
>> >> Thanks & regards
>> >> Arko
>> >>
>> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com> wrote:
>> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
>> >> > which
>> >> > is
>> >> > not recommended practice. Normally, the only files the NN stores are
>> >> > the
>> >> > image and edits files. It does not store any actual HDFS data. If you
>> >> > must
>> >> > run a DN on the NN, try turning down the logging in
>> >> > /conf/log4j.properties:
>> >> >
>> >> > #hadoop.root.logger=INFO,console
>> >> > #hadoop.root.logger=WARN,console
>> >> > hadoop.root.logger=ERROR,console
>> >> >
>> >> > Depending on the logging information you require, of course.
>> >> >
>> >> > -Chuck
>> >> >
>> >> >
>> >> > -----Original Message-----
>> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> >> > Sent: Wednesday, February 13, 2013 11:32 AM
>> >> > To: hdfs-user@hadoop.apache.org
>> >> > Subject: Managing space in Master Node
>> >> >
>> >> > Hello Gurus,
>> >> >
>> >> > I am managing a Hadoop Cluster to run some experiments.
>> >> >
>> >> > The issue I am continuously facing is that the Master Node runs out
>> >> > of
>> >> > disk
>> >> > space due to logs and data files.
>> >> >
>> >> > I can monitor and delete log files. However, I cannot delete the HDFS
>> >> > data.
>> >> >
>> >> > Thus, is there a way to force Hadoop not to save any HDFS data in the
>> >> > Master
>> >> > Node?
>> >> >
>> >> > Then I can use my master to handle the metadata only and store the
>> >> > logs.
>> >> >
>> >> > Thanks & regards
>> >> > Arko
>> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>> >> > information contained in this email may be confidential and/or
>> >> > legally
>> >> > privileged. It has been sent for the sole use of the intended
>> >> > recipient(s).
>> >> > If you are not the intended recipient of this mail, you are hereby
>> >> > notified
>> >> > that any unauthorized review, use, disclosure, dissemination,
>> >> > distribution,
>> >> > or copying of this communication, or any of its contents, is strictly
>> >> > prohibited. If you have received this communication in error, please
>> >> > reply
>> >> > to the sender and destroy all copies of the message.
>> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
>> >> > 01880,
>> >> > USA
>> >> >
>> >
>> >
>
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
Just don't run the DN daemon on that machine. Dedicate that machine to NN.
Remove the hostname of the NN machine from the 'slaves' file.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:31 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Hi,
>
> Thanks for the help!
>
> However, I am still unsure about how to "turn off" the datanode
> feature on the NN.
>
> I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
> a directory.
>
> Should I just comment out that property? What would happen to the
> current data that there is in the Master? Will it get removed
> automatically?
>
> Thanks & regards
> Arko
>
>
>
> On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com>
> wrote:
> > You can specify the logging level as specified by Charles. But turning
> logs
> > off is never a good idea. Logs are really helpful in problem diagnosis,
> > which are eventual.
> >
> > Warm Regards,
> > Tariq
> > https://mtariq.jux.com/
> > cloudfront.blogspot.com
> >
> >
> > On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> > <ar...@gmail.com> wrote:
> >>
> >> Hi,
> >>
> >> Yeah, my NameNode is also seconding as a DataNode.
> >>
> >> I would like to "turn off" this feature.
> >>
> >> Request help regarding the same.
> >>
> >> Thanks & regards
> >> Arko
> >>
> >> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com> wrote:
> >> > Hi Arko. Sounds like you may be running a DataNode on the NameNode
> which
> >> > is
> >> > not recommended practice. Normally, the only files the NN stores are
> the
> >> > image and edits files. It does not store any actual HDFS data. If you
> >> > must
> >> > run a DN on the NN, try turning down the logging in
> >> > /conf/log4j.properties:
> >> >
> >> > #hadoop.root.logger=INFO,console
> >> > #hadoop.root.logger=WARN,console
> >> > hadoop.root.logger=ERROR,console
> >> >
> >> > Depending on the logging information you require, of course.
> >> >
> >> > -Chuck
> >> >
> >> >
> >> > -----Original Message-----
> >> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> >> > Sent: Wednesday, February 13, 2013 11:32 AM
> >> > To: hdfs-user@hadoop.apache.org
> >> > Subject: Managing space in Master Node
> >> >
> >> > Hello Gurus,
> >> >
> >> > I am managing a Hadoop Cluster to run some experiments.
> >> >
> >> > The issue I am continuously facing is that the Master Node runs out of
> >> > disk
> >> > space due to logs and data files.
> >> >
> >> > I can monitor and delete log files. However, I cannot delete the HDFS
> >> > data.
> >> >
> >> > Thus, is there a way to force Hadoop not to save any HDFS data in the
> >> > Master
> >> > Node?
> >> >
> >> > Then I can use my master to handle the metadata only and store the
> logs.
> >> >
> >> > Thanks & regards
> >> > Arko
> >> > SDL Enterprise Technologies, Inc. - all rights reserved. The
> >> > information contained in this email may be confidential and/or legally
> >> > privileged. It has been sent for the sole use of the intended
> recipient(s).
> >> > If you are not the intended recipient of this mail, you are hereby
> notified
> >> > that any unauthorized review, use, disclosure, dissemination,
> distribution,
> >> > or copying of this communication, or any of its contents, is strictly
> >> > prohibited. If you have received this communication in error, please
> reply
> >> > to the sender and destroy all copies of the message.
> >> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
> 01880,
> >> > USA
> >> >
> >
> >
>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
Thanks for the help!
However, I am still unsure about how to "turn off" the datanode
feature on the NN.
I checked the "hdfs-site.xml" file and the dfs.data.dir is pointed to
a directory.
Should I just comment out that property? What would happen to the
current data that there is in the Master? Will it get removed
automatically?
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 1:55 PM, Mohammad Tariq <do...@gmail.com> wrote:
> You can specify the logging level as specified by Charles. But turning logs
> off is never a good idea. Logs are really helpful in problem diagnosis,
> which are eventual.
>
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
>
>
> On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>>
>> Hi,
>>
>> Yeah, my NameNode is also seconding as a DataNode.
>>
>> I would like to "turn off" this feature.
>>
>> Request help regarding the same.
>>
>> Thanks & regards
>> Arko
>>
>> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com> wrote:
>> > Hi Arko. Sounds like you may be running a DataNode on the NameNode which
>> > is
>> > not recommended practice. Normally, the only files the NN stores are the
>> > image and edits files. It does not store any actual HDFS data. If you
>> > must
>> > run a DN on the NN, try turning down the logging in
>> > /conf/log4j.properties:
>> >
>> > #hadoop.root.logger=INFO,console
>> > #hadoop.root.logger=WARN,console
>> > hadoop.root.logger=ERROR,console
>> >
>> > Depending on the logging information you require, of course.
>> >
>> > -Chuck
>> >
>> >
>> > -----Original Message-----
>> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> > Sent: Wednesday, February 13, 2013 11:32 AM
>> > To: hdfs-user@hadoop.apache.org
>> > Subject: Managing space in Master Node
>> >
>> > Hello Gurus,
>> >
>> > I am managing a Hadoop Cluster to run some experiments.
>> >
>> > The issue I am continuously facing is that the Master Node runs out of
>> > disk
>> > space due to logs and data files.
>> >
>> > I can monitor and delete log files. However, I cannot delete the HDFS
>> > data.
>> >
>> > Thus, is there a way to force Hadoop not to save any HDFS data in the
>> > Master
>> > Node?
>> >
>> > Then I can use my master to handle the metadata only and store the logs.
>> >
>> > Thanks & regards
>> > Arko
>> > SDL Enterprise Technologies, Inc. - all rights reserved. The
>> > information contained in this email may be confidential and/or legally
>> > privileged. It has been sent for the sole use of the intended recipient(s).
>> > If you are not the intended recipient of this mail, you are hereby notified
>> > that any unauthorized review, use, disclosure, dissemination, distribution,
>> > or copying of this communication, or any of its contents, is strictly
>> > prohibited. If you have received this communication in error, please reply
>> > to the sender and destroy all copies of the message.
>> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880,
>> > USA
>> >
>
>
Re: Managing space in Master Node
Posted by Mohammad Tariq <do...@gmail.com>.
You can specify the logging level as specified by Charles. But turning logs
off is never a good idea. Logs are really helpful in problem diagnosis,
which are eventual.
Warm Regards,
Tariq
https://mtariq.jux.com/
cloudfront.blogspot.com
On Thu, Feb 14, 2013 at 1:22 AM, Arko Provo Mukherjee <
arkoprovomukherjee@gmail.com> wrote:
> Hi,
>
> Yeah, my NameNode is also seconding as a DataNode.
>
> I would like to "turn off" this feature.
>
> Request help regarding the same.
>
> Thanks & regards
> Arko
>
> On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com> wrote:
> > Hi Arko. Sounds like you may be running a DataNode on the NameNode which
> is
> > not recommended practice. Normally, the only files the NN stores are the
> > image and edits files. It does not store any actual HDFS data. If you
> must
> > run a DN on the NN, try turning down the logging in
> /conf/log4j.properties:
> >
> > #hadoop.root.logger=INFO,console
> > #hadoop.root.logger=WARN,console
> > hadoop.root.logger=ERROR,console
> >
> > Depending on the logging information you require, of course.
> >
> > -Chuck
> >
> >
> > -----Original Message-----
> > From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> > Sent: Wednesday, February 13, 2013 11:32 AM
> > To: hdfs-user@hadoop.apache.org
> > Subject: Managing space in Master Node
> >
> > Hello Gurus,
> >
> > I am managing a Hadoop Cluster to run some experiments.
> >
> > The issue I am continuously facing is that the Master Node runs out of
> disk
> > space due to logs and data files.
> >
> > I can monitor and delete log files. However, I cannot delete the HDFS
> data.
> >
> > Thus, is there a way to force Hadoop not to save any HDFS data in the
> Master
> > Node?
> >
> > Then I can use my master to handle the metadata only and store the logs.
> >
> > Thanks & regards
> > Arko
> > SDL Enterprise Technologies, Inc. - all rights reserved. The
> information contained in this email may be confidential and/or legally
> privileged. It has been sent for the sole use of the intended recipient(s).
> If you are not the intended recipient of this mail, you are hereby notified
> that any unauthorized review, use, disclosure, dissemination, distribution,
> or copying of this communication, or any of its contents, is strictly
> prohibited. If you have received this communication in error, please reply
> to the sender and destroy all copies of the message.
> > Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880,
> USA
> >
>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hi,
Yeah, my NameNode is also seconding as a DataNode.
I would like to "turn off" this feature.
Request help regarding the same.
Thanks & regards
Arko
On Wed, Feb 13, 2013 at 1:38 PM, Charles Baker <cb...@sdl.com> wrote:
> Hi Arko. Sounds like you may be running a DataNode on the NameNode which is
> not recommended practice. Normally, the only files the NN stores are the
> image and edits files. It does not store any actual HDFS data. If you must
> run a DN on the NN, try turning down the logging in /conf/log4j.properties:
>
> #hadoop.root.logger=INFO,console
> #hadoop.root.logger=WARN,console
> hadoop.root.logger=ERROR,console
>
> Depending on the logging information you require, of course.
>
> -Chuck
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> Sent: Wednesday, February 13, 2013 11:32 AM
> To: hdfs-user@hadoop.apache.org
> Subject: Managing space in Master Node
>
> Hello Gurus,
>
> I am managing a Hadoop Cluster to run some experiments.
>
> The issue I am continuously facing is that the Master Node runs out of disk
> space due to logs and data files.
>
> I can monitor and delete log files. However, I cannot delete the HDFS data.
>
> Thus, is there a way to force Hadoop not to save any HDFS data in the Master
> Node?
>
> Then I can use my master to handle the metadata only and store the logs.
>
> Thanks & regards
> Arko
> SDL Enterprise Technologies, Inc. - all rights reserved. The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
> Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
>
RE: Managing space in Master Node
Posted by Charles Baker <cb...@sdl.com>.
Hi Arko. Sounds like you may be running a DataNode on the NameNode which is
not recommended practice. Normally, the only files the NN stores are the
image and edits files. It does not store any actual HDFS data. If you must
run a DN on the NN, try turning down the logging in /conf/log4j.properties:
#hadoop.root.logger=INFO,console
#hadoop.root.logger=WARN,console
hadoop.root.logger=ERROR,console
Depending on the logging information you require, of course.
-Chuck
-----Original Message-----
From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
Sent: Wednesday, February 13, 2013 11:32 AM
To: hdfs-user@hadoop.apache.org
Subject: Managing space in Master Node
Hello Gurus,
I am managing a Hadoop Cluster to run some experiments.
The issue I am continuously facing is that the Master Node runs out of disk
space due to logs and data files.
I can monitor and delete log files. However, I cannot delete the HDFS data.
Thus, is there a way to force Hadoop not to save any HDFS data in the Master
Node?
Then I can use my master to handle the metadata only and store the logs.
Thanks & regards
Arko
SDL Enterprise Technologies, Inc. - all rights reserved. The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
RE: Managing space in Master Node
Posted by Charles Baker <cb...@sdl.com>.
Hey Arko. It should be safe to delete then.
-Chuck
-----Original Message-----
From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
Sent: Friday, February 15, 2013 11:56 AM
To: hdfs-user@hadoop.apache.org
Subject: Re: Managing space in Master Node
Hello Charles,
Thanks a lot for your reply and help!
Yes, the NN data (image, edit files) is kept separate from the data files.
"dfs.name.dir=/hadoop/hdfs/name" whereas "dfs.data.dir=/hadoop/hdfs/data".
Their contents match with the description you specified
Can I safely go ahead and delete all contents from the data dir?
Thanks & regards
Arko
On Thu, Feb 14, 2013 at 8:16 PM, Charles Baker <cb...@sdl.com> wrote:
> Hey Arko. Glad to hear it worked out for you. Just make sure that your
> NN's image and edit files aren't also (perhaps inadvertently) stored
> in that directory structure. The NameNode image/edit dir contains the
following:
>
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 14 17:48 current drwxr-xr-x 2 hdfs
> hdfs 4096 Jul 17 2012 image
> -rw-r--r-- 1 hdfs hdfs 0 Sep 27 12:28 in_use.lock
> drwxr-xr-x 2 hdfs hdfs 4096 Sep 17 23:25 previous.checkpoint
>
> whereas, the DN data dir should contain:
>
> drwxr-xr-x 2 hdfs hdfs 57344 Feb 14 14:04 blocksBeingWritten
> drwxr-xr-x 66 hdfs hdfs 8192 Jan 22 02:44 current
> drwxr-xr-x 2 hdfs hdfs 6 Aug 10 2012 detach
> -rw-r--r-- 1 hdfs hdfs 0 Sep 27 11:19 in_use.lock
> -rw-r--r-- 1 hdfs hdfs 157 Aug 10 2012 storage
> drwxr-xr-x 2 hdfs hdfs 6 Feb 4 17:34 tmp
>
>
> It's a good practice to locate the JobTracker and SecondaryNameNode
> services on a different machine than the NN. If you have an
> opportunity to do so, I recommend getting a separate machine for those two
services as well.
>
> -Chuck
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> Sent: Thursday, February 14, 2013 4:21 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Managing space in Master Node
>
> Hello Everyone - I have decommissioned my NN so that only the
> JobTracker, NameNode and SecondaryNameNodes are running on it now.
> I would like to thank everyone who helped me to perform this task!
>
> I have just one more issue: The data files created in the Master Node
> seem to be still there. in /hdfs/data/current Now that this server is
> not being used as a DataNode, can I safely delete the files under this
> folder to clear up space?
>
> Thanks & regards
> Arko
>
> On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>> Thanks! I have already requested for downtime. Will do the changes soon!
>> Warm regards
>> Arko
>>
>> On Thu, Feb 14, 2013 at 3:31 AM, <ra...@accenture.com> wrote:
>>> Hi Arko.
>>>
>>> Only thing you need to do is not running the TaskTracker and
>>> DataNode
> demons on your master machine. Ensure you do not have this on your
> slaves file con hadoop's config directory when you start the system.
> I'm supposing you are on Open Source release. For other distributions
> look at its documentation about how to remove those demons from your master
node machine.
>>>
>>>
>>> -----Original Message-----
>>> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>> Sent: miércoles, 13 de febrero de 2013 20:32
>>> To: hdfs-user@hadoop.apache.org
>>> Subject: Managing space in Master Node
>>>
>>> Hello Gurus,
>>>
>>> I am managing a Hadoop Cluster to run some experiments.
>>>
>>> The issue I am continuously facing is that the Master Node runs out
>>> of disk space due to logs and data files.
>>>
>>> I can monitor and delete log files. However, I cannot delete the
>>> HDFS
> data.
>>>
>>> Thus, is there a way to force Hadoop not to save any HDFS data in
>>> the Master Node?
>>>
>>> Then I can use my master to handle the metadata only and store the logs.
>>>
>>> Thanks & regards
>>> Arko
>>>
>>>
>>> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete
> the original. Any other use of the e-mail by you is prohibited.
>>>
>>> Where allowed by local law, electronic communications with Accenture
>>> and
> its affiliates, including e-mail and instant messaging (including
> content), may be scanned by our systems for the purposes of
> information security and assessment of internal compliance with Accenture
policy.
>>>
>>> ____________________________________________________________________
>>> _
>>> _________________
>>>
>>> www.accenture.com
>>>
> SDL Enterprise Technologies, Inc. - all rights reserved. The information
contained in this email may be confidential and/or legally privileged. It has
been sent for the sole use of the intended recipient(s). If you are not the
intended recipient of this mail, you are hereby notified that any
unauthorized review, use, disclosure, dissemination, distribution, or copying
of this communication, or any of its contents, is strictly prohibited. If you
have received this communication in error, please reply to the sender and
destroy all copies of the message.
> Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA
> 01880, USA
>
SDL Enterprise Technologies, Inc. - all rights reserved. The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hello Charles,
Thanks a lot for your reply and help!
Yes, the NN data (image, edit files) is kept separate from the data files.
"dfs.name.dir=/hadoop/hdfs/name" whereas "dfs.data.dir=/hadoop/hdfs/data".
Their contents match with the description you specified
Can I safely go ahead and delete all contents from the data dir?
Thanks & regards
Arko
On Thu, Feb 14, 2013 at 8:16 PM, Charles Baker <cb...@sdl.com> wrote:
> Hey Arko. Glad to hear it worked out for you. Just make sure that your NN's
> image and edit files aren't also (perhaps inadvertently) stored in that
> directory structure. The NameNode image/edit dir contains the following:
>
> drwxr-xr-x 2 hdfs hdfs 4096 Feb 14 17:48 current
> drwxr-xr-x 2 hdfs hdfs 4096 Jul 17 2012 image
> -rw-r--r-- 1 hdfs hdfs 0 Sep 27 12:28 in_use.lock
> drwxr-xr-x 2 hdfs hdfs 4096 Sep 17 23:25 previous.checkpoint
>
> whereas, the DN data dir should contain:
>
> drwxr-xr-x 2 hdfs hdfs 57344 Feb 14 14:04 blocksBeingWritten
> drwxr-xr-x 66 hdfs hdfs 8192 Jan 22 02:44 current
> drwxr-xr-x 2 hdfs hdfs 6 Aug 10 2012 detach
> -rw-r--r-- 1 hdfs hdfs 0 Sep 27 11:19 in_use.lock
> -rw-r--r-- 1 hdfs hdfs 157 Aug 10 2012 storage
> drwxr-xr-x 2 hdfs hdfs 6 Feb 4 17:34 tmp
>
>
> It's a good practice to locate the JobTracker and SecondaryNameNode services
> on a different machine than the NN. If you have an opportunity to do so, I
> recommend getting a separate machine for those two services as well.
>
> -Chuck
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> Sent: Thursday, February 14, 2013 4:21 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: Managing space in Master Node
>
> Hello Everyone - I have decommissioned my NN so that only the JobTracker,
> NameNode and SecondaryNameNodes are running on it now.
> I would like to thank everyone who helped me to perform this task!
>
> I have just one more issue: The data files created in the Master Node seem to
> be still there. in /hdfs/data/current Now that this server is not being used
> as a DataNode, can I safely delete the files under this folder to clear up
> space?
>
> Thanks & regards
> Arko
>
> On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee
> <ar...@gmail.com> wrote:
>> Thanks! I have already requested for downtime. Will do the changes soon!
>> Warm regards
>> Arko
>>
>> On Thu, Feb 14, 2013 at 3:31 AM, <ra...@accenture.com> wrote:
>>> Hi Arko.
>>>
>>> Only thing you need to do is not running the TaskTracker and DataNode
> demons on your master machine. Ensure you do not have this on your slaves
> file con hadoop's config directory when you start the system. I'm supposing
> you are on Open Source release. For other distributions look at its
> documentation about how to remove those demons from your master node machine.
>>>
>>>
>>> -----Original Message-----
>>> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>>> Sent: miércoles, 13 de febrero de 2013 20:32
>>> To: hdfs-user@hadoop.apache.org
>>> Subject: Managing space in Master Node
>>>
>>> Hello Gurus,
>>>
>>> I am managing a Hadoop Cluster to run some experiments.
>>>
>>> The issue I am continuously facing is that the Master Node runs out
>>> of disk space due to logs and data files.
>>>
>>> I can monitor and delete log files. However, I cannot delete the HDFS
> data.
>>>
>>> Thus, is there a way to force Hadoop not to save any HDFS data in the
>>> Master Node?
>>>
>>> Then I can use my master to handle the metadata only and store the logs.
>>>
>>> Thanks & regards
>>> Arko
>>>
>>>
>>> This message is for the designated recipient only and may contain
> privileged, proprietary, or otherwise private information. If you have
> received it in error, please notify the sender immediately and delete the
> original. Any other use of the e-mail by you is prohibited.
>>>
>>> Where allowed by local law, electronic communications with Accenture and
> its affiliates, including e-mail and instant messaging (including content),
> may be scanned by our systems for the purposes of information security and
> assessment of internal compliance with Accenture policy.
>>>
>>> _____________________________________________________________________
>>> _________________
>>>
>>> www.accenture.com
>>>
> SDL Enterprise Technologies, Inc. - all rights reserved. The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
> Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
>
RE: Managing space in Master Node
Posted by Charles Baker <cb...@sdl.com>.
Hey Arko. Glad to hear it worked out for you. Just make sure that your NN's
image and edit files aren't also (perhaps inadvertently) stored in that
directory structure. The NameNode image/edit dir contains the following:
drwxr-xr-x 2 hdfs hdfs 4096 Feb 14 17:48 current
drwxr-xr-x 2 hdfs hdfs 4096 Jul 17 2012 image
-rw-r--r-- 1 hdfs hdfs 0 Sep 27 12:28 in_use.lock
drwxr-xr-x 2 hdfs hdfs 4096 Sep 17 23:25 previous.checkpoint
whereas, the DN data dir should contain:
drwxr-xr-x 2 hdfs hdfs 57344 Feb 14 14:04 blocksBeingWritten
drwxr-xr-x 66 hdfs hdfs 8192 Jan 22 02:44 current
drwxr-xr-x 2 hdfs hdfs 6 Aug 10 2012 detach
-rw-r--r-- 1 hdfs hdfs 0 Sep 27 11:19 in_use.lock
-rw-r--r-- 1 hdfs hdfs 157 Aug 10 2012 storage
drwxr-xr-x 2 hdfs hdfs 6 Feb 4 17:34 tmp
It's a good practice to locate the JobTracker and SecondaryNameNode services
on a different machine than the NN. If you have an opportunity to do so, I
recommend getting a separate machine for those two services as well.
-Chuck
-----Original Message-----
From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
Sent: Thursday, February 14, 2013 4:21 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: Managing space in Master Node
Hello Everyone - I have decommissioned my NN so that only the JobTracker,
NameNode and SecondaryNameNodes are running on it now.
I would like to thank everyone who helped me to perform this task!
I have just one more issue: The data files created in the Master Node seem to
be still there. in /hdfs/data/current Now that this server is not being used
as a DataNode, can I safely delete the files under this folder to clear up
space?
Thanks & regards
Arko
On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee
<ar...@gmail.com> wrote:
> Thanks! I have already requested for downtime. Will do the changes soon!
> Warm regards
> Arko
>
> On Thu, Feb 14, 2013 at 3:31 AM, <ra...@accenture.com> wrote:
>> Hi Arko.
>>
>> Only thing you need to do is not running the TaskTracker and DataNode
demons on your master machine. Ensure you do not have this on your slaves
file con hadoop's config directory when you start the system. I'm supposing
you are on Open Source release. For other distributions look at its
documentation about how to remove those demons from your master node machine.
>>
>>
>> -----Original Message-----
>> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> Sent: miércoles, 13 de febrero de 2013 20:32
>> To: hdfs-user@hadoop.apache.org
>> Subject: Managing space in Master Node
>>
>> Hello Gurus,
>>
>> I am managing a Hadoop Cluster to run some experiments.
>>
>> The issue I am continuously facing is that the Master Node runs out
>> of disk space due to logs and data files.
>>
>> I can monitor and delete log files. However, I cannot delete the HDFS
data.
>>
>> Thus, is there a way to force Hadoop not to save any HDFS data in the
>> Master Node?
>>
>> Then I can use my master to handle the metadata only and store the logs.
>>
>> Thanks & regards
>> Arko
>>
>>
>> This message is for the designated recipient only and may contain
privileged, proprietary, or otherwise private information. If you have
received it in error, please notify the sender immediately and delete the
original. Any other use of the e-mail by you is prohibited.
>>
>> Where allowed by local law, electronic communications with Accenture and
its affiliates, including e-mail and instant messaging (including content),
may be scanned by our systems for the purposes of information security and
assessment of internal compliance with Accenture policy.
>>
>> _____________________________________________________________________
>> _________________
>>
>> www.accenture.com
>>
SDL Enterprise Technologies, Inc. - all rights reserved. The information contained in this email may be confidential and/or legally privileged. It has been sent for the sole use of the intended recipient(s). If you are not the intended recipient of this mail, you are hereby notified that any unauthorized review, use, disclosure, dissemination, distribution, or copying of this communication, or any of its contents, is strictly prohibited. If you have received this communication in error, please reply to the sender and destroy all copies of the message.
Registered address: 201 Edgewater Drive, Suite 225, Wakefield, MA 01880, USA
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Hello Everyone - I have decommissioned my NN so that only the
JobTracker, NameNode and SecondaryNameNodes are running on it now.
I would like to thank everyone who helped me to perform this task!
I have just one more issue: The data files created in the Master Node
seem to be still there. in /hdfs/data/current
Now that this server is not being used as a DataNode, can I safely
delete the files under this folder to clear up space?
Thanks & regards
Arko
On Thu, Feb 14, 2013 at 11:48 AM, Arko Provo Mukherjee
<ar...@gmail.com> wrote:
> Thanks! I have already requested for downtime. Will do the changes soon!
> Warm regards
> Arko
>
> On Thu, Feb 14, 2013 at 3:31 AM, <ra...@accenture.com> wrote:
>> Hi Arko.
>>
>> Only thing you need to do is not running the TaskTracker and DataNode demons on your master machine. Ensure you do not have this on your slaves file con hadoop's config directory when you start the system. I'm supposing you are on Open Source release. For other distributions look at its documentation about how to remove those demons from your master node machine.
>>
>>
>> -----Original Message-----
>> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
>> Sent: miércoles, 13 de febrero de 2013 20:32
>> To: hdfs-user@hadoop.apache.org
>> Subject: Managing space in Master Node
>>
>> Hello Gurus,
>>
>> I am managing a Hadoop Cluster to run some experiments.
>>
>> The issue I am continuously facing is that the Master Node runs out of
>> disk space due to logs and data files.
>>
>> I can monitor and delete log files. However, I cannot delete the HDFS data.
>>
>> Thus, is there a way to force Hadoop not to save any HDFS data in the
>> Master Node?
>>
>> Then I can use my master to handle the metadata only and store the logs.
>>
>> Thanks & regards
>> Arko
>>
>>
>> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
>>
>> Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>>
>> ______________________________________________________________________________________
>>
>> www.accenture.com
>>
Re: Managing space in Master Node
Posted by Arko Provo Mukherjee <ar...@gmail.com>.
Thanks! I have already requested for downtime. Will do the changes soon!
Warm regards
Arko
On Thu, Feb 14, 2013 at 3:31 AM, <ra...@accenture.com> wrote:
> Hi Arko.
>
> Only thing you need to do is not running the TaskTracker and DataNode demons on your master machine. Ensure you do not have this on your slaves file con hadoop's config directory when you start the system. I'm supposing you are on Open Source release. For other distributions look at its documentation about how to remove those demons from your master node machine.
>
>
> -----Original Message-----
> From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
> Sent: miércoles, 13 de febrero de 2013 20:32
> To: hdfs-user@hadoop.apache.org
> Subject: Managing space in Master Node
>
> Hello Gurus,
>
> I am managing a Hadoop Cluster to run some experiments.
>
> The issue I am continuously facing is that the Master Node runs out of
> disk space due to logs and data files.
>
> I can monitor and delete log files. However, I cannot delete the HDFS data.
>
> Thus, is there a way to force Hadoop not to save any HDFS data in the
> Master Node?
>
> Then I can use my master to handle the metadata only and store the logs.
>
> Thanks & regards
> Arko
>
>
> This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
>
> Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
>
> ______________________________________________________________________________________
>
> www.accenture.com
>
RE: Managing space in Master Node
Posted by ra...@accenture.com.
Hi Arko.
Only thing you need to do is not running the TaskTracker and DataNode demons on your master machine. Ensure you do not have this on your slaves file con hadoop's config directory when you start the system. I'm supposing you are on Open Source release. For other distributions look at its documentation about how to remove those demons from your master node machine.
-----Original Message-----
From: Arko Provo Mukherjee [mailto:arkoprovomukherjee@gmail.com]
Sent: miércoles, 13 de febrero de 2013 20:32
To: hdfs-user@hadoop.apache.org
Subject: Managing space in Master Node
Hello Gurus,
I am managing a Hadoop Cluster to run some experiments.
The issue I am continuously facing is that the Master Node runs out of
disk space due to logs and data files.
I can monitor and delete log files. However, I cannot delete the HDFS data.
Thus, is there a way to force Hadoop not to save any HDFS data in the
Master Node?
Then I can use my master to handle the metadata only and store the logs.
Thanks & regards
Arko
This message is for the designated recipient only and may contain privileged, proprietary, or otherwise private information. If you have received it in error, please notify the sender immediately and delete the original. Any other use of the e-mail by you is prohibited.
Where allowed by local law, electronic communications with Accenture and its affiliates, including e-mail and instant messaging (including content), may be scanned by our systems for the purposes of information security and assessment of internal compliance with Accenture policy.
______________________________________________________________________________________
www.accenture.com