You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Harsh J <ha...@cloudera.com> on 2011/09/23 05:48:30 UTC
Re: formatting hdfs without user interaction
Ivan,
Writing your own program was overkill.
The 'yes' coreutil is pretty silly, but nifty at the same time. It
accepts an argument, which it would repeat infinitely.
So:
$ yes Y | hadoop namenode -format
Would do it for you.
(Note that in the future release, saner answers will be acceptable,
i.e. y instead of strictly Y, etc.)
Also, two other things:
- What do you mean by 'Yeah I have a secondary namenode as well so 2
directories.'? A secondary namenode uses different directories than
dfs.name.dir.
- The prompt only appears when it detects a 'reformat' being happening
- which is very dangerous to do non-interactively. If you do the
-format the first time, on clean dfs.name.dir setups, you will never
receive a prompt.
On Fri, Sep 23, 2011 at 3:53 AM, <Iv...@emc.com> wrote:
> Hi Adam,
>
> Well the "yes" program prints lower case y's and apparently only captial Y
> is accepted.
>
> But by creating my out "Yes" program that spews Y's to stdout it works :)
>
> Cheers,
> Ivan
>
> On 9/22/11 3:02 PM, "Adam Shook" <as...@clearedgeit.com> wrote:
>
>>yes | hadoop namenode -format
>>
>>The "yes" program simply outputs 'y' a bunch of times. "echo yes" will
>>just print "yes" to stdout.
>>
>>-- Adam
>>
>>-----Original Message-----
>>From: Ivan.Novick@emc.com [mailto:Ivan.Novick@emc.com]
>>Sent: Thursday, September 22, 2011 6:00 PM
>>To: common-user@hadoop.apache.org; rajvish@yahoo.com
>>Subject: Re: formatting hdfs without user interaction
>>
>>Hi Raj and Arpit,
>>
>>I think the fact that there are 2 confirmations is causing it not to work:
>>
>>echo "Y" | hadoop namenode -format
>>11/09/22 14:57:10 INFO namenode.NameNode: STARTUP_MSG:
>>/************************************************************
>>STARTUP_MSG: Starting NameNode
>>STARTUP_MSG: host = hdm1/172.28.9.250
>>STARTUP_MSG: args = [-format]
>>STARTUP_MSG: version = 0.20.204.0
>>STARTUP_MSG: build = git://hrt8n35.cc1.ygridcore.net/ on branch
>>branch-0.20-security-204 -r 65e258bf0813ac2b15bb4c954660eaf9e8fba141;
>>compiled by 'hortonow' on Thu Aug 25 23:25:52 UTC 2011
>>************************************************************/
>>Re-format filesystem in /data/local/hdfs/name ? (Y or N) Re-format
>>filesystem in /data/remote/hdfs/name ? (Y or N) Format aborted in
>>/data/remote/hdfs/name
>>11/09/22 14:57:10 INFO namenode.NameNode: SHUTDOWN_MSG:
>>/************************************************************
>>SHUTDOWN_MSG: Shutting down NameNode at hdm1/172.28.9.250
>>************************************************************/
>>Heap
>> par new generation total 2831168K, used 150996K [0x00002aaaae210000,
>>0x00002aab6e210000, 0x00002aab6e210000)
>> eden space 2516608K, 6% used [0x00002aaaae210000, 0x00002aaab75852d0,
>>0x00002aab47bb0000)
>> from space 314560K, 0% used [0x00002aab47bb0000, 0x00002aab47bb0000,
>>0x00002aab5aee0000)
>> to space 314560K, 0% used [0x00002aab5aee0000, 0x00002aab5aee0000,
>>0x00002aab6e210000)
>> concurrent mark-sweep generation total 30408704K, used 0K
>>[0x00002aab6e210000, 0x00002ab2ae210000, 0x00002ab2ae210000)
>> concurrent-mark-sweep perm gen total 21248K, used 7333K
>>[0x00002ab2ae210000, 0x00002ab2af6d0000, 0x00002ab2b3610000)
>>
>>
>>Cheers,
>>Ivan
>>
>>On 9/22/11 2:52 PM, "Raj V" <ra...@yahoo.com> wrote:
>>
>>>Ian
>>>"echo 'Y' | hadoop namenode -format"
>>>
>>>should work.,
>>>
>>>Raj
>>>
>>>
>>>>________________________________
>>>>From: "Ivan.Novick@emc.com" <Iv...@emc.com>
>>>>To: common-user@hadoop.apache.org
>>>>Sent: Thursday, September 22, 2011 2:43 PM
>>>>Subject: formatting hdfs without user interaction
>>>>
>>>>Hello,
>>>>
>>>>I am trying to automate formatting an HDFS volume. Is there any way to
>>>>do this without the interaction (and using expect)?
>>>>
>>>>Cheers,
>>>>Ivan
>>>>
>>>>
>>
>>
>>-----
>>No virus found in this message.
>>Checked by AVG - www.avg.com
>>Version: 10.0.1410 / Virus Database: 1520/3912 - Release Date: 09/22/11
>>
>
>
--
Harsh J
Re: formatting hdfs without user interaction
Posted by Iv...@emc.com.
On 9/23/11 9:46 AM, "Harsh J" <ha...@cloudera.com> wrote:
>Ivan,
>
>On Fri, Sep 23, 2011 at 9:22 PM, <Iv...@emc.com> wrote:
>[snip]
>> Which parameter are you referring to? I am planning on using 2
>>directories
>> in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
>> machine running the secondary namenode.
>
>I'm slightly confused here. Do you mean to say a 'SecondaryNameNode',
>which is a form of checkpointing daemon available as part of Hadoop,
>or do you mean to say, loosely, 'secondary namenode', which would
>imply a machine that can step in as a NameNode once you start it (in
>case the primary fails)?
>
>For the former, the properties are the ones that begin with
>"fs.checkpoint.*" in 0.20.x versions, which you can checkout.
I mean both... A 2nd node which can step in as a name node and also that
node will run the checkpointing daemon. I have 2 directories in
dfs.name.dir one is local and one is remote on the 2nd machine so we have
a copy of the data if we lose the first machine.
Cheers,
Ivan
Re: formatting hdfs without user interaction
Posted by Harsh J <ha...@cloudera.com>.
Ivan,
On Fri, Sep 23, 2011 at 9:22 PM, <Iv...@emc.com> wrote:
[snip]
> Which parameter are you referring to? I am planning on using 2 directories
> in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
> machine running the secondary namenode.
I'm slightly confused here. Do you mean to say a 'SecondaryNameNode',
which is a form of checkpointing daemon available as part of Hadoop,
or do you mean to say, loosely, 'secondary namenode', which would
imply a machine that can step in as a NameNode once you start it (in
case the primary fails)?
For the former, the properties are the ones that begin with
"fs.checkpoint.*" in 0.20.x versions, which you can checkout.
--
Harsh J
Re: formatting hdfs without user interaction
Posted by Iv...@emc.com.
On 9/23/11 9:01 AM, "Edward Capriolo" <ed...@gmail.com> wrote:
>On Fri, Sep 23, 2011 at 11:52 AM, <Iv...@emc.com> wrote:
>
>> Hi Harsh,
>>
>> On 9/22/11 8:48 PM, "Harsh J" <ha...@cloudera.com> wrote:
>>
>> >Ivan,
>> >
>> >Writing your own program was overkill.
>> >
>> >The 'yes' coreutil is pretty silly, but nifty at the same time. It
>> >accepts an argument, which it would repeat infinitely.
>> >
>> >So:
>> >
>> >$ yes Y | hadoop namenode -format
>> >
>> >Would do it for you.
>>
>> Nice! I read the man page for yes too quickly and did not see that
>> option. Thanks!
>>
>>...... SNIP ...............
>>
>You might want to try expect scripting. You open a stream to the process
>and
>then can wait for prompts and print replies. expect also has this feature
>autoexpect which is a shell that records the streams and turns your
>teminal
>interaction into an expect script.
Yeah, I have used expect via a python library numerous times, but expect
is a bit brittle and error prone, so I try to avoid it whenever possible.
Cheers,
Ivan
Re: formatting hdfs without user interaction
Posted by Edward Capriolo <ed...@gmail.com>.
On Fri, Sep 23, 2011 at 11:52 AM, <Iv...@emc.com> wrote:
> Hi Harsh,
>
> On 9/22/11 8:48 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
> >Ivan,
> >
> >Writing your own program was overkill.
> >
> >The 'yes' coreutil is pretty silly, but nifty at the same time. It
> >accepts an argument, which it would repeat infinitely.
> >
> >So:
> >
> >$ yes Y | hadoop namenode -format
> >
> >Would do it for you.
>
> Nice! I read the man page for yes too quickly and did not see that
> option. Thanks!
>
>
> >(Note that in the future release, saner answers will be acceptable,
> >i.e. y instead of strictly Y, etc.)
>
> Y/y/yes/YES would all seem like good things to accept :)
>
>
> >Also, two other things:
> >
> >- What do you mean by 'Yeah I have a secondary namenode as well so 2
> >directories.'? A secondary namenode uses different directories than
> >dfs.name.dir.
>
> Which parameter are you referring to? I am planning on using 2 directories
> in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
> machine running the secondary namenode.
>
>
> >- The prompt only appears when it detects a 'reformat' being happening
> >- which is very dangerous to do non-interactively. If you do the
> >-format the first time, on clean dfs.name.dir setups, you will never
> >receive a prompt.
>
> Yeah I am creating some automation, so it needs to be able to wipe out an
> existing filesystem and start over......
>
>
> Cheers,
> Ivan
>
>
>
> >
>
>
You might want to try expect scripting. You open a stream to the process and
then can wait for prompts and print replies. expect also has this feature
autoexpect which is a shell that records the streams and turns your teminal
interaction into an expect script.
Re: formatting hdfs without user interaction
Posted by Iv...@emc.com.
Hi Harsh,
On 9/22/11 8:48 PM, "Harsh J" <ha...@cloudera.com> wrote:
>Ivan,
>
>Writing your own program was overkill.
>
>The 'yes' coreutil is pretty silly, but nifty at the same time. It
>accepts an argument, which it would repeat infinitely.
>
>So:
>
>$ yes Y | hadoop namenode -format
>
>Would do it for you.
Nice! I read the man page for yes too quickly and did not see that
option. Thanks!
>(Note that in the future release, saner answers will be acceptable,
>i.e. y instead of strictly Y, etc.)
Y/y/yes/YES would all seem like good things to accept :)
>Also, two other things:
>
>- What do you mean by 'Yeah I have a secondary namenode as well so 2
>directories.'? A secondary namenode uses different directories than
>dfs.name.dir.
Which parameter are you referring to? I am planning on using 2 directories
in dfs.name.dir, one is local and the other is an NFS mount of a 2nd
machine running the secondary namenode.
>- The prompt only appears when it detects a 'reformat' being happening
>- which is very dangerous to do non-interactively. If you do the
>-format the first time, on clean dfs.name.dir setups, you will never
>receive a prompt.
Yeah I am creating some automation, so it needs to be able to wipe out an
existing filesystem and start over......
Cheers,
Ivan
>