You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by tim robertson <ti...@gmail.com> on 2008/06/27 20:27:06 UTC
EC2 public AMI
Hi all,
I have been battling EC2 all day and getting nowhere (see other message)
Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small instances
successfully?
Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately doesn't work
as the slaves don't come up (details in my earlier message).
Thanks for any help or pointers to getting it running on EC2.
Tim
Re: EC2 public AMI
Posted by tim robertson <ti...@gmail.com>.
Thanks to Tom and Mike for taking the time to respond. I really appreciate
your help
Tom - good call on the groups!. I installed elasticfox (thanks for the tip)
removed the groups and they were recreated on startup fine, and the PI
example ran without issues.
Thanks again,
Tim
On Sat, Jun 28, 2008 at 9:56 AM, Tom White <to...@gmail.com> wrote:
> Hi Tim,
>
> Could it be a problem with the security groups? The scripts create two
> groups: one for the master and one for the slaves, and they must be
> authorized to talk to each other. You can inspect the groups that have
> been created using standard EC2 tools (e.g. elasticfox) to see if they
> have the right permissions. Also, try deleting the groups and run the
> Hadoop script again as that will recreate the groups (BTW my access ID
> *doesn't* have hyphens in it - not sure if that makes a difference).
>
> If none of that works, could you look in the datanode and tasktracker
> logs for errors?
>
> Hope this helps.
>
> Tom
>
> On Sat, Jun 28, 2008 at 8:17 AM, tim robertson
> <ti...@gmail.com> wrote:
> > Hi Mike,
> > Thanks again. I am at a loss too.
> >
> > (yeah I meant hadoop-ec2-env.sh)
> >
> > I have also stepped through the scripts and can't work out either what
> could
> > be up - the userdata passed to the slave seems ok, and he slave starts
> fine,
> > just doesn't join the master.
> >
> > Could you please tell me what I should go looking at on the slave which
> > instructs it to join the master?
> > I am new to Hadoop... but I presume there is a config file on there holds
> > the masters location to join on startup, since it gets the masters IP
> > address passed in.
> >
> > I will build an image tomorrow if needbe.
> > I have been trying with both the hadoop 0.17.0 and 0.17.1 downloads and
> get
> > the same issue
> >
> > Thanks,
> >
> > Tim
> >
> >
> >
> > On Fri, Jun 27, 2008 at 11:36 PM, montag <mk...@about.com> wrote:
> >
> >>
> >> Yes, that's the correct format. I have to ask, you meant
> >> hadoop-ec2-env.sh, right?
> >>
> >> In any case, I just successfully tested running the scripts from Mac
> OS
> >> 10.5. It don't think it could be an issue with your key pair or
> >> hadoop-ec2-env.sh . Otherwise, I don't believe you would be able to
> boot a
> >> master successfully. To be honest, I'm kind of at a loss. I would
> suggest
> >> breaking the scripts down into their key launch components (i.e. run
> >> launch-master and launch-slaves manually instead of just typing in
> >> launch-cluster).
> >>
> >> Were you able to build your own image using the hadoop scripts?
> >>
> >> Cheers,
> >> Mike
> >>
> >>
> >> TimRobertson100 wrote:
> >> >
> >> > A daft question:
> >> >
> >> > Is this the correct format for the AWS account id in the
> >> > hadoop-ec2-env.txt
> >> > ?
> >> > AWS_ACCOUNT_ID=XXXX-XXXX-0848
> >> >
> >> > It doesn't seem to matter if I remove the "-" as always slaves and
> master
> >> > start...
> >> >
> >> >
> >> >
> >> > Thanks,
> >> >
> >> > Tim
> >> >
> >> >
> >> >
> >> > On Fri, Jun 27, 2008 at 9:44 PM, tim robertson <
> >> timrobertson100@gmail.com>
> >> > wrote:
> >> >
> >> >> Also in my .bashrc:
> >> >> EC2_HOME=~/dev/ibiodiversity/aws/ec2-api-tools-1.3-19403
> >> >> export EC2_HOME
> >> >> EC2_PRIVATE_KEY=~/.ec2/pk-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
> >> >> export EC2_PRIVATE_KEY
> >> >> EC2_CERT=~/.ec2/cert-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
> >> >> export EC2_CERT
> >> >> EC2_KEYDIR=~/.ec2
> >> >> export EC2_KEYDIR
> >> >>
> >> >>
> >> >>
> >> >>
> >> >> On Fri, Jun 27, 2008 at 9:33 PM, tim robertson
> >> >> <ti...@gmail.com>
> >> >> wrote:
> >> >>
> >> >>> Hi Mike,
> >> >>> Thanks for taking the time to try that! I appreciate it.
> >> >>>
> >> >>> I am running
> >> >>>
> >> >>> - Mac OSX 10.4
> >> >>>
> >> >>>
> >>
> PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
> >> >>>
> >> >>> To my knowledge there is nothing quirky about my network... I tried
> >> >>> several times to start it up and always got the same. I am not sure
> >> why
> >> >>> my
> >> >>> OS should make a difference - am I not just issuing a request for an
> >> >>> Amazon
> >> >>> instance to be created using an image from S3? Or do some of my key
> >> >>> files
> >> >>> get copied across?
> >> >>>
> >> >>> I did get an error on the key file the very first time, saying it's
> >> >>> permission were too open - so I ran "chmod 600" on the keypair,
> after
> >> a
> >> >>> quick google search... could it be that some files are not getting
> >> >>> copied
> >> >>> correctly to the master?
> >> >>>
> >> >>> Cheers again,
> >> >>>
> >> >>> Tim
> >> >>>
> >> >>>
> >> >>> On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
> >> >>>
> >> >>>>
> >> >>>> Hi Tim,
> >> >>>>
> >> >>>> That's really strange. I launched a cluster of two and ran the pi
> >> >>>> example
> >> >>>> just now with no issue. This is most likely a dead end, but what
> OS
> >> >>>> are
> >> >>>> you
> >> >>>> running the scripts on, what's your $PATH, and is there anything
> >> quirky
> >> >>>> about your network? It's very weird that you are able to boot up
> the
> >> >>>> master, but not the slaves.
> >> >>>> If you have an S3 account, I would suggest trying to create your
> own
> >> >>>> image
> >> >>>> using the hadoop bundling scripts and launching a cluster based off
> of
> >> >>>> that
> >> >>>> image. You'll need to download the AMI tools from here:
> >> >>>>
> >> >>>>
> >>
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
> >> >>>>
> >> >>>> Cheers,
> >> >>>> Mike
> >> >>>>
> >> >>>>
> >> >>>>
> >> >>>> TimRobertson100 wrote:
> >> >>>> >
> >> >>>> > Hi Mike,
> >> >>>> > Thanks for the reply!
> >> >>>> >
> >> >>>> > I am using ec2-api-tools-1.3-19403 which I got from
> >> >>>> >
> >> >>>>
> >>
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
> >> >>>> >
> >> >>>> > The master seems to start, and there is an instance started for a
> >> >>>> slave
> >> >>>> I
> >> >>>> > can see from the EC2 command line tools, but not through the
> hadoop
> >> >>>> > console... but it doesn't seem to function (error log is
> attached)
> >> >>>> >
> >> >>>> > Thanks for taking time to reply,
> >> >>>> >
> >> >>>> > Tim
> >> >>>> > (I am new to hadoop and ec2 so please don't hesitate to point the
> >> >>>> obvious
> >> >>>> > out to me. I have my own mapreduce code running on single server
> >> >>>> mode
> >> >>>> > fine...)
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com>
> wrote:
> >> >>>> >
> >> >>>> >>
> >> >>>> >> Hi Tim,
> >> >>>> >>
> >> >>>> >> I've had no problems launching clusters using the small hadoop
> >> >>>> 0.17.0
> >> >>>> >> images. Just curious, what version of the Amazon EC2 command
> line
> >> >>>> tools
> >> >>>> >> are
> >> >>>> >> you using?
> >> >>>> >>
> >> >>>> >> Thanks,
> >> >>>> >> Mike
> >> >>>> >>
> >> >>>> >>
> >> >>>> >> TimRobertson100 wrote:
> >> >>>> >> >
> >> >>>> >> > Hi all,
> >> >>>> >> > I have been battling EC2 all day and getting nowhere (see
> other
> >> >>>> >> message)
> >> >>>> >> >
> >> >>>> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for
> small
> >> >>>> >> > instances
> >> >>>> >> > successfully?
> >> >>>> >> >
> >> >>>> >> > Following http://wiki.apache.org/hadoop/AmazonEC2unfortunately
> >> >>>> doesn't
> >> >>>> >> > work
> >> >>>> >> > as the slaves don't come up (details in my earlier message).
> >> >>>> >> >
> >> >>>> >> > Thanks for any help or pointers to getting it running on EC2.
> >> >>>> >> >
> >> >>>> >> > Tim
> >> >>>> >> >
> >> >>>> >> >
> >> >>>> >>
> >> >>>> >> --
> >> >>>> >> View this message in context:
> >> >>>> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
> >> >>>> >> Sent from the Hadoop core-user mailing list archive at
> Nabble.com.
> >> >>>> >>
> >> >>>> >>
> >> >>>> >
> >> >>>> > Start the cluster
> >> >>>> >
> >> >>>> > $ pwd
> >> >>>> >
> /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
> >> >>>> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
> >> >>>> > Testing for existing master in group: test-cluster
> >> >>>> > Starting master with AMI ami-ee53b687
> >> >>>> > Waiting for instance i-62ea3c0b to start
> >> >>>> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
> >> >>>> > Warning: Permanently added
> >> >>>> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA)
> to
> >> >>>> the
> >> >>>> > list of known hosts.
> >> >>>> > Copying private key to master
> >> >>>> > id_rsa-gsg-keypair
> >> >>>> > 100% 1670 1.6KB/s 00:00
> >> >>>> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is
> >> >>>> 67.202.25.117,
> >> >>>> > zone is us-east-1b.
> >> >>>> > Adding test-cluster node(s) to cluster group test-cluster with
> AMI
> >> >>>> > ami-ee53b687
> >> >>>> > i-78ea3c11
> >> >>>> > i-7bea3c12
> >> >>>> >
> >> >>>> >
> >> >>>> > Wait for the nodes to start
> >> >>>> > RESERVATION r-cc8449a5 544208310848
> test-cluster-master
> >> >>>> > INSTANCE i-62ea3c0b ami-ee53b687
> >> >>>> > ec2-67-202-25-117.compute-1.amazonaws.com
> >> >>>> > domU-12-31-39-00-8A-03.compute-1.internal running
> gsg-keypair
> >> >>>> 0
> >> >>>> > m1.small 2008-06-27T09:48:00+0000
> >> >>>> us-east-1baki-a71cf9ce
> >> >>>> > ari-a51cf9cc
> >> >>>> > RESERVATION r-c28449ab 544208310848 test-cluster
> >> >>>> > INSTANCE i-78ea3c11 ami-ee53b687
> >> >>>> > ec2-75-101-232-80.compute-1.amazonaws.com
> >> >>>> > domU-12-31-39-00-84-23.compute-1.internal running
> gsg-keypair
> >> >>>> 0
> >> >>>> > m1.small 2008-06-27T09:49:19+0000
> >> >>>> us-east-1baki-a71cf9ce
> >> >>>> > ari-a51cf9cc
> >> >>>> > INSTANCE i-7bea3c12 ami-ee53b687
> >> >>>> > ec2-67-202-0-48.compute-1.amazonaws.com
> >> >>>> > domU-12-31-39-00-85-C4.compute-1.internal running
> gsg-keypair
> >> >>>> 1
> >> >>>> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
> >> >>>> > aki-a71cf9ce ari-a51cf9cc
> >> >>>> >
> >> >>>> >
> >> >>>> > Go to the console:
> >> >>>> >
> >> http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
> >> >>>> > It says here Cluster Summary - Nodes = 0
> >> >>>> > ???
> >> >>>> >
> >> >>>> >
> >> >>>> >
> >> >>>> > Login and run the test:
> >> >>>> > $ bin/hadoop-ec2 login test-cluster
> >> >>>> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
> >> >>>> >
> >> >>>> > __| __|_ ) Fedora 8
> >> >>>> > _| ( / 32-bit
> >> >>>> > ___|\___|___|
> >> >>>> >
> >> >>>> > Welcome to an EC2 Public Image
> >> >>>> > :-)
> >> >>>> > Base
> >> >>>> >
> >> >>>> > --[ see /etc/ec2/release-notes ]--
> >> >>>> >
> >> >>>> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
> >> >>>> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
> >> >>>> > hadoop-*-examples.jar pi 10 10000000
> >> >>>> >
> >> >>>> > Output is as follows:
> >> >>>> >
> >> >>>> >
> >> >>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
> >> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> >> >>>> > filesystem name. Use
> >> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/"
> instead.
> >> >>>> > Number of Maps = 10 Samples per Map = 10000000
> >> >>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
> >> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> >> >>>> > filesystem name. Use
> >> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/"
> instead.
> >> >>>> > 08/06/27 05:53:48 WARN fs.FileSystem:
> >> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> >> >>>> > filesystem name. Use
> >> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/"
> instead.
> >> >>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part0 retries left 4
> >> >>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part0 retries left 3
> >> >>>> > 08/06/27 05:53:49 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part0 retries left 2
> >> >>>> > 08/06/27 05:53:51 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part0 retries left 1
> >> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #0
> >> >>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part1 retries left 4
> >> >>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part1 retries left 3
> >> >>>> > 08/06/27 05:53:55 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part1 retries left 2
> >> >>>> > 08/06/27 05:53:57 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part1 retries left 1
> >> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #1
> >> >>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part2 retries left 4
> >> >>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part2 retries left 3
> >> >>>> > 08/06/27 05:54:01 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part2 retries left 2
> >> >>>> > 08/06/27 05:54:03 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part2 retries left 1
> >> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #2
> >> >>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >> >>>> Method)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part3 retries left 4
> >> >>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part3 retries left 3
> >> >>>> > 08/06/27 05:54:07 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part3 retries left 2
> >> >>>> > 08/06/27 05:54:09 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part3 retries left 1
> >> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #3
> >> >>>> > 08/06/27 05:54:12 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part4 retries left 4
> >> >>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part4 retries left 3
> >> >>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part4 retries left 2
> >> >>>> > 08/06/27 05:54:15 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part4 retries left 1
> >> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #4
> >> >>>> > 08/06/27 05:54:18 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part5 retries left 4
> >> >>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part5 retries left 3
> >> >>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part5 retries left 2
> >> >>>> > 08/06/27 05:54:21 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part5 retries left 1
> >> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #5
> >> >>>> > 08/06/27 05:54:24 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part6 retries left 4
> >> >>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part6 retries left 3
> >> >>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part6 retries left 2
> >> >>>> > 08/06/27 05:54:27 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part6 retries left 1
> >> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block
> null
> >> >>>> bad
> >> >>>> > datanode[0]
> >> >>>> > Wrote input for Map #6
> >> >>>> > 08/06/27 05:54:30 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part7 retries left 4
> >> >>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> > at
> org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >> >>>> >
> >> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >> >>>> >
> >> >>>> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException
> >> >>>> sleeping
> >> >>>> > /user/root/test-mini-mr/in/part7 retries left 3
> >> >>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
> >> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
> >> nodes,
> >> >>>> > instead of 1
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >> >>>> > at
> >> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >> >>>> Source)
> >> >>>> > at
> >> >>>> >
> >> >>>>
> >>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >> >>>> >
> >> >>>
> >> >>>
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18164284.html
> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >>
> >>
> >
>
Re: EC2 public AMI
Posted by Tom White <to...@gmail.com>.
Hi Tim,
Could it be a problem with the security groups? The scripts create two
groups: one for the master and one for the slaves, and they must be
authorized to talk to each other. You can inspect the groups that have
been created using standard EC2 tools (e.g. elasticfox) to see if they
have the right permissions. Also, try deleting the groups and run the
Hadoop script again as that will recreate the groups (BTW my access ID
*doesn't* have hyphens in it - not sure if that makes a difference).
If none of that works, could you look in the datanode and tasktracker
logs for errors?
Hope this helps.
Tom
On Sat, Jun 28, 2008 at 8:17 AM, tim robertson
<ti...@gmail.com> wrote:
> Hi Mike,
> Thanks again. I am at a loss too.
>
> (yeah I meant hadoop-ec2-env.sh)
>
> I have also stepped through the scripts and can't work out either what could
> be up - the userdata passed to the slave seems ok, and he slave starts fine,
> just doesn't join the master.
>
> Could you please tell me what I should go looking at on the slave which
> instructs it to join the master?
> I am new to Hadoop... but I presume there is a config file on there holds
> the masters location to join on startup, since it gets the masters IP
> address passed in.
>
> I will build an image tomorrow if needbe.
> I have been trying with both the hadoop 0.17.0 and 0.17.1 downloads and get
> the same issue
>
> Thanks,
>
> Tim
>
>
>
> On Fri, Jun 27, 2008 at 11:36 PM, montag <mk...@about.com> wrote:
>
>>
>> Yes, that's the correct format. I have to ask, you meant
>> hadoop-ec2-env.sh, right?
>>
>> In any case, I just successfully tested running the scripts from Mac OS
>> 10.5. It don't think it could be an issue with your key pair or
>> hadoop-ec2-env.sh . Otherwise, I don't believe you would be able to boot a
>> master successfully. To be honest, I'm kind of at a loss. I would suggest
>> breaking the scripts down into their key launch components (i.e. run
>> launch-master and launch-slaves manually instead of just typing in
>> launch-cluster).
>>
>> Were you able to build your own image using the hadoop scripts?
>>
>> Cheers,
>> Mike
>>
>>
>> TimRobertson100 wrote:
>> >
>> > A daft question:
>> >
>> > Is this the correct format for the AWS account id in the
>> > hadoop-ec2-env.txt
>> > ?
>> > AWS_ACCOUNT_ID=XXXX-XXXX-0848
>> >
>> > It doesn't seem to matter if I remove the "-" as always slaves and master
>> > start...
>> >
>> >
>> >
>> > Thanks,
>> >
>> > Tim
>> >
>> >
>> >
>> > On Fri, Jun 27, 2008 at 9:44 PM, tim robertson <
>> timrobertson100@gmail.com>
>> > wrote:
>> >
>> >> Also in my .bashrc:
>> >> EC2_HOME=~/dev/ibiodiversity/aws/ec2-api-tools-1.3-19403
>> >> export EC2_HOME
>> >> EC2_PRIVATE_KEY=~/.ec2/pk-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
>> >> export EC2_PRIVATE_KEY
>> >> EC2_CERT=~/.ec2/cert-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
>> >> export EC2_CERT
>> >> EC2_KEYDIR=~/.ec2
>> >> export EC2_KEYDIR
>> >>
>> >>
>> >>
>> >>
>> >> On Fri, Jun 27, 2008 at 9:33 PM, tim robertson
>> >> <ti...@gmail.com>
>> >> wrote:
>> >>
>> >>> Hi Mike,
>> >>> Thanks for taking the time to try that! I appreciate it.
>> >>>
>> >>> I am running
>> >>>
>> >>> - Mac OSX 10.4
>> >>>
>> >>>
>> PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
>> >>>
>> >>> To my knowledge there is nothing quirky about my network... I tried
>> >>> several times to start it up and always got the same. I am not sure
>> why
>> >>> my
>> >>> OS should make a difference - am I not just issuing a request for an
>> >>> Amazon
>> >>> instance to be created using an image from S3? Or do some of my key
>> >>> files
>> >>> get copied across?
>> >>>
>> >>> I did get an error on the key file the very first time, saying it's
>> >>> permission were too open - so I ran "chmod 600" on the keypair, after
>> a
>> >>> quick google search... could it be that some files are not getting
>> >>> copied
>> >>> correctly to the master?
>> >>>
>> >>> Cheers again,
>> >>>
>> >>> Tim
>> >>>
>> >>>
>> >>> On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
>> >>>
>> >>>>
>> >>>> Hi Tim,
>> >>>>
>> >>>> That's really strange. I launched a cluster of two and ran the pi
>> >>>> example
>> >>>> just now with no issue. This is most likely a dead end, but what OS
>> >>>> are
>> >>>> you
>> >>>> running the scripts on, what's your $PATH, and is there anything
>> quirky
>> >>>> about your network? It's very weird that you are able to boot up the
>> >>>> master, but not the slaves.
>> >>>> If you have an S3 account, I would suggest trying to create your own
>> >>>> image
>> >>>> using the hadoop bundling scripts and launching a cluster based off of
>> >>>> that
>> >>>> image. You'll need to download the AMI tools from here:
>> >>>>
>> >>>>
>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
>> >>>>
>> >>>> Cheers,
>> >>>> Mike
>> >>>>
>> >>>>
>> >>>>
>> >>>> TimRobertson100 wrote:
>> >>>> >
>> >>>> > Hi Mike,
>> >>>> > Thanks for the reply!
>> >>>> >
>> >>>> > I am using ec2-api-tools-1.3-19403 which I got from
>> >>>> >
>> >>>>
>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
>> >>>> >
>> >>>> > The master seems to start, and there is an instance started for a
>> >>>> slave
>> >>>> I
>> >>>> > can see from the EC2 command line tools, but not through the hadoop
>> >>>> > console... but it doesn't seem to function (error log is attached)
>> >>>> >
>> >>>> > Thanks for taking time to reply,
>> >>>> >
>> >>>> > Tim
>> >>>> > (I am new to hadoop and ec2 so please don't hesitate to point the
>> >>>> obvious
>> >>>> > out to me. I have my own mapreduce code running on single server
>> >>>> mode
>> >>>> > fine...)
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
>> >>>> >
>> >>>> >>
>> >>>> >> Hi Tim,
>> >>>> >>
>> >>>> >> I've had no problems launching clusters using the small hadoop
>> >>>> 0.17.0
>> >>>> >> images. Just curious, what version of the Amazon EC2 command line
>> >>>> tools
>> >>>> >> are
>> >>>> >> you using?
>> >>>> >>
>> >>>> >> Thanks,
>> >>>> >> Mike
>> >>>> >>
>> >>>> >>
>> >>>> >> TimRobertson100 wrote:
>> >>>> >> >
>> >>>> >> > Hi all,
>> >>>> >> > I have been battling EC2 all day and getting nowhere (see other
>> >>>> >> message)
>> >>>> >> >
>> >>>> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
>> >>>> >> > instances
>> >>>> >> > successfully?
>> >>>> >> >
>> >>>> >> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately
>> >>>> doesn't
>> >>>> >> > work
>> >>>> >> > as the slaves don't come up (details in my earlier message).
>> >>>> >> >
>> >>>> >> > Thanks for any help or pointers to getting it running on EC2.
>> >>>> >> >
>> >>>> >> > Tim
>> >>>> >> >
>> >>>> >> >
>> >>>> >>
>> >>>> >> --
>> >>>> >> View this message in context:
>> >>>> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
>> >>>> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>> >>>> >>
>> >>>> >>
>> >>>> >
>> >>>> > Start the cluster
>> >>>> >
>> >>>> > $ pwd
>> >>>> > /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
>> >>>> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
>> >>>> > Testing for existing master in group: test-cluster
>> >>>> > Starting master with AMI ami-ee53b687
>> >>>> > Waiting for instance i-62ea3c0b to start
>> >>>> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
>> >>>> > Warning: Permanently added
>> >>>> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to
>> >>>> the
>> >>>> > list of known hosts.
>> >>>> > Copying private key to master
>> >>>> > id_rsa-gsg-keypair
>> >>>> > 100% 1670 1.6KB/s 00:00
>> >>>> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is
>> >>>> 67.202.25.117,
>> >>>> > zone is us-east-1b.
>> >>>> > Adding test-cluster node(s) to cluster group test-cluster with AMI
>> >>>> > ami-ee53b687
>> >>>> > i-78ea3c11
>> >>>> > i-7bea3c12
>> >>>> >
>> >>>> >
>> >>>> > Wait for the nodes to start
>> >>>> > RESERVATION r-cc8449a5 544208310848 test-cluster-master
>> >>>> > INSTANCE i-62ea3c0b ami-ee53b687
>> >>>> > ec2-67-202-25-117.compute-1.amazonaws.com
>> >>>> > domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair
>> >>>> 0
>> >>>> > m1.small 2008-06-27T09:48:00+0000
>> >>>> us-east-1baki-a71cf9ce
>> >>>> > ari-a51cf9cc
>> >>>> > RESERVATION r-c28449ab 544208310848 test-cluster
>> >>>> > INSTANCE i-78ea3c11 ami-ee53b687
>> >>>> > ec2-75-101-232-80.compute-1.amazonaws.com
>> >>>> > domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair
>> >>>> 0
>> >>>> > m1.small 2008-06-27T09:49:19+0000
>> >>>> us-east-1baki-a71cf9ce
>> >>>> > ari-a51cf9cc
>> >>>> > INSTANCE i-7bea3c12 ami-ee53b687
>> >>>> > ec2-67-202-0-48.compute-1.amazonaws.com
>> >>>> > domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair
>> >>>> 1
>> >>>> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
>> >>>> > aki-a71cf9ce ari-a51cf9cc
>> >>>> >
>> >>>> >
>> >>>> > Go to the console:
>> >>>> >
>> http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
>> >>>> > It says here Cluster Summary - Nodes = 0
>> >>>> > ???
>> >>>> >
>> >>>> >
>> >>>> >
>> >>>> > Login and run the test:
>> >>>> > $ bin/hadoop-ec2 login test-cluster
>> >>>> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
>> >>>> >
>> >>>> > __| __|_ ) Fedora 8
>> >>>> > _| ( / 32-bit
>> >>>> > ___|\___|___|
>> >>>> >
>> >>>> > Welcome to an EC2 Public Image
>> >>>> > :-)
>> >>>> > Base
>> >>>> >
>> >>>> > --[ see /etc/ec2/release-notes ]--
>> >>>> >
>> >>>> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
>> >>>> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
>> >>>> > hadoop-*-examples.jar pi 10 10000000
>> >>>> >
>> >>>> > Output is as follows:
>> >>>> >
>> >>>> >
>> >>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>> >>>> > filesystem name. Use
>> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>> >>>> > Number of Maps = 10 Samples per Map = 10000000
>> >>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>> >>>> > filesystem name. Use
>> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>> >>>> > 08/06/27 05:53:48 WARN fs.FileSystem:
>> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>> >>>> > filesystem name. Use
>> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>> >>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part0 retries left 4
>> >>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part0 retries left 3
>> >>>> > 08/06/27 05:53:49 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part0 retries left 2
>> >>>> > 08/06/27 05:53:51 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part0 retries left 1
>> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #0
>> >>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part1 retries left 4
>> >>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part1 retries left 3
>> >>>> > 08/06/27 05:53:55 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part1 retries left 2
>> >>>> > 08/06/27 05:53:57 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part1 retries left 1
>> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #1
>> >>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part2 retries left 4
>> >>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part2 retries left 3
>> >>>> > 08/06/27 05:54:01 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part2 retries left 2
>> >>>> > 08/06/27 05:54:03 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part2 retries left 1
>> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #2
>> >>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> >>>> Method)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part3 retries left 4
>> >>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part3 retries left 3
>> >>>> > 08/06/27 05:54:07 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part3 retries left 2
>> >>>> > 08/06/27 05:54:09 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part3 retries left 1
>> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #3
>> >>>> > 08/06/27 05:54:12 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part4 retries left 4
>> >>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part4 retries left 3
>> >>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part4 retries left 2
>> >>>> > 08/06/27 05:54:15 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part4 retries left 1
>> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #4
>> >>>> > 08/06/27 05:54:18 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part5 retries left 4
>> >>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part5 retries left 3
>> >>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part5 retries left 2
>> >>>> > 08/06/27 05:54:21 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part5 retries left 1
>> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #5
>> >>>> > 08/06/27 05:54:24 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part6 retries left 4
>> >>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part6 retries left 3
>> >>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part6 retries left 2
>> >>>> > 08/06/27 05:54:27 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part6 retries left 1
>> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null
>> >>>> bad
>> >>>> > datanode[0]
>> >>>> > Wrote input for Map #6
>> >>>> > 08/06/27 05:54:30 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part7 retries left 4
>> >>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >>>> >
>> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >>>> >
>> >>>> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException
>> >>>> sleeping
>> >>>> > /user/root/test-mini-mr/in/part7 retries left 3
>> >>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
>> nodes,
>> >>>> > instead of 1
>> >>>> > at
>> >>>> >
>> >>>>
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> >>>> > at
>> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>> >>>> Source)
>> >>>> > at
>> >>>> >
>> >>>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> >>>> >
>> >>>
>> >>>
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/EC2-public-AMI-tp18161182p18164284.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>
Re: EC2 public AMI
Posted by tim robertson <ti...@gmail.com>.
Hi Mike,
Thanks again. I am at a loss too.
(yeah I meant hadoop-ec2-env.sh)
I have also stepped through the scripts and can't work out either what could
be up - the userdata passed to the slave seems ok, and he slave starts fine,
just doesn't join the master.
Could you please tell me what I should go looking at on the slave which
instructs it to join the master?
I am new to Hadoop... but I presume there is a config file on there holds
the masters location to join on startup, since it gets the masters IP
address passed in.
I will build an image tomorrow if needbe.
I have been trying with both the hadoop 0.17.0 and 0.17.1 downloads and get
the same issue
Thanks,
Tim
On Fri, Jun 27, 2008 at 11:36 PM, montag <mk...@about.com> wrote:
>
> Yes, that's the correct format. I have to ask, you meant
> hadoop-ec2-env.sh, right?
>
> In any case, I just successfully tested running the scripts from Mac OS
> 10.5. It don't think it could be an issue with your key pair or
> hadoop-ec2-env.sh . Otherwise, I don't believe you would be able to boot a
> master successfully. To be honest, I'm kind of at a loss. I would suggest
> breaking the scripts down into their key launch components (i.e. run
> launch-master and launch-slaves manually instead of just typing in
> launch-cluster).
>
> Were you able to build your own image using the hadoop scripts?
>
> Cheers,
> Mike
>
>
> TimRobertson100 wrote:
> >
> > A daft question:
> >
> > Is this the correct format for the AWS account id in the
> > hadoop-ec2-env.txt
> > ?
> > AWS_ACCOUNT_ID=XXXX-XXXX-0848
> >
> > It doesn't seem to matter if I remove the "-" as always slaves and master
> > start...
> >
> >
> >
> > Thanks,
> >
> > Tim
> >
> >
> >
> > On Fri, Jun 27, 2008 at 9:44 PM, tim robertson <
> timrobertson100@gmail.com>
> > wrote:
> >
> >> Also in my .bashrc:
> >> EC2_HOME=~/dev/ibiodiversity/aws/ec2-api-tools-1.3-19403
> >> export EC2_HOME
> >> EC2_PRIVATE_KEY=~/.ec2/pk-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
> >> export EC2_PRIVATE_KEY
> >> EC2_CERT=~/.ec2/cert-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
> >> export EC2_CERT
> >> EC2_KEYDIR=~/.ec2
> >> export EC2_KEYDIR
> >>
> >>
> >>
> >>
> >> On Fri, Jun 27, 2008 at 9:33 PM, tim robertson
> >> <ti...@gmail.com>
> >> wrote:
> >>
> >>> Hi Mike,
> >>> Thanks for taking the time to try that! I appreciate it.
> >>>
> >>> I am running
> >>>
> >>> - Mac OSX 10.4
> >>>
> >>>
> PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
> >>>
> >>> To my knowledge there is nothing quirky about my network... I tried
> >>> several times to start it up and always got the same. I am not sure
> why
> >>> my
> >>> OS should make a difference - am I not just issuing a request for an
> >>> Amazon
> >>> instance to be created using an image from S3? Or do some of my key
> >>> files
> >>> get copied across?
> >>>
> >>> I did get an error on the key file the very first time, saying it's
> >>> permission were too open - so I ran "chmod 600" on the keypair, after
> a
> >>> quick google search... could it be that some files are not getting
> >>> copied
> >>> correctly to the master?
> >>>
> >>> Cheers again,
> >>>
> >>> Tim
> >>>
> >>>
> >>> On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
> >>>
> >>>>
> >>>> Hi Tim,
> >>>>
> >>>> That's really strange. I launched a cluster of two and ran the pi
> >>>> example
> >>>> just now with no issue. This is most likely a dead end, but what OS
> >>>> are
> >>>> you
> >>>> running the scripts on, what's your $PATH, and is there anything
> quirky
> >>>> about your network? It's very weird that you are able to boot up the
> >>>> master, but not the slaves.
> >>>> If you have an S3 account, I would suggest trying to create your own
> >>>> image
> >>>> using the hadoop bundling scripts and launching a cluster based off of
> >>>> that
> >>>> image. You'll need to download the AMI tools from here:
> >>>>
> >>>>
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
> >>>>
> >>>> Cheers,
> >>>> Mike
> >>>>
> >>>>
> >>>>
> >>>> TimRobertson100 wrote:
> >>>> >
> >>>> > Hi Mike,
> >>>> > Thanks for the reply!
> >>>> >
> >>>> > I am using ec2-api-tools-1.3-19403 which I got from
> >>>> >
> >>>>
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
> >>>> >
> >>>> > The master seems to start, and there is an instance started for a
> >>>> slave
> >>>> I
> >>>> > can see from the EC2 command line tools, but not through the hadoop
> >>>> > console... but it doesn't seem to function (error log is attached)
> >>>> >
> >>>> > Thanks for taking time to reply,
> >>>> >
> >>>> > Tim
> >>>> > (I am new to hadoop and ec2 so please don't hesitate to point the
> >>>> obvious
> >>>> > out to me. I have my own mapreduce code running on single server
> >>>> mode
> >>>> > fine...)
> >>>> >
> >>>> >
> >>>> >
> >>>> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
> >>>> >
> >>>> >>
> >>>> >> Hi Tim,
> >>>> >>
> >>>> >> I've had no problems launching clusters using the small hadoop
> >>>> 0.17.0
> >>>> >> images. Just curious, what version of the Amazon EC2 command line
> >>>> tools
> >>>> >> are
> >>>> >> you using?
> >>>> >>
> >>>> >> Thanks,
> >>>> >> Mike
> >>>> >>
> >>>> >>
> >>>> >> TimRobertson100 wrote:
> >>>> >> >
> >>>> >> > Hi all,
> >>>> >> > I have been battling EC2 all day and getting nowhere (see other
> >>>> >> message)
> >>>> >> >
> >>>> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
> >>>> >> > instances
> >>>> >> > successfully?
> >>>> >> >
> >>>> >> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately
> >>>> doesn't
> >>>> >> > work
> >>>> >> > as the slaves don't come up (details in my earlier message).
> >>>> >> >
> >>>> >> > Thanks for any help or pointers to getting it running on EC2.
> >>>> >> >
> >>>> >> > Tim
> >>>> >> >
> >>>> >> >
> >>>> >>
> >>>> >> --
> >>>> >> View this message in context:
> >>>> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
> >>>> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >>>> >>
> >>>> >>
> >>>> >
> >>>> > Start the cluster
> >>>> >
> >>>> > $ pwd
> >>>> > /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
> >>>> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
> >>>> > Testing for existing master in group: test-cluster
> >>>> > Starting master with AMI ami-ee53b687
> >>>> > Waiting for instance i-62ea3c0b to start
> >>>> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
> >>>> > Warning: Permanently added
> >>>> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to
> >>>> the
> >>>> > list of known hosts.
> >>>> > Copying private key to master
> >>>> > id_rsa-gsg-keypair
> >>>> > 100% 1670 1.6KB/s 00:00
> >>>> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is
> >>>> 67.202.25.117,
> >>>> > zone is us-east-1b.
> >>>> > Adding test-cluster node(s) to cluster group test-cluster with AMI
> >>>> > ami-ee53b687
> >>>> > i-78ea3c11
> >>>> > i-7bea3c12
> >>>> >
> >>>> >
> >>>> > Wait for the nodes to start
> >>>> > RESERVATION r-cc8449a5 544208310848 test-cluster-master
> >>>> > INSTANCE i-62ea3c0b ami-ee53b687
> >>>> > ec2-67-202-25-117.compute-1.amazonaws.com
> >>>> > domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair
> >>>> 0
> >>>> > m1.small 2008-06-27T09:48:00+0000
> >>>> us-east-1baki-a71cf9ce
> >>>> > ari-a51cf9cc
> >>>> > RESERVATION r-c28449ab 544208310848 test-cluster
> >>>> > INSTANCE i-78ea3c11 ami-ee53b687
> >>>> > ec2-75-101-232-80.compute-1.amazonaws.com
> >>>> > domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair
> >>>> 0
> >>>> > m1.small 2008-06-27T09:49:19+0000
> >>>> us-east-1baki-a71cf9ce
> >>>> > ari-a51cf9cc
> >>>> > INSTANCE i-7bea3c12 ami-ee53b687
> >>>> > ec2-67-202-0-48.compute-1.amazonaws.com
> >>>> > domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair
> >>>> 1
> >>>> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
> >>>> > aki-a71cf9ce ari-a51cf9cc
> >>>> >
> >>>> >
> >>>> > Go to the console:
> >>>> >
> http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
> >>>> > It says here Cluster Summary - Nodes = 0
> >>>> > ???
> >>>> >
> >>>> >
> >>>> >
> >>>> > Login and run the test:
> >>>> > $ bin/hadoop-ec2 login test-cluster
> >>>> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
> >>>> >
> >>>> > __| __|_ ) Fedora 8
> >>>> > _| ( / 32-bit
> >>>> > ___|\___|___|
> >>>> >
> >>>> > Welcome to an EC2 Public Image
> >>>> > :-)
> >>>> > Base
> >>>> >
> >>>> > --[ see /etc/ec2/release-notes ]--
> >>>> >
> >>>> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
> >>>> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
> >>>> > hadoop-*-examples.jar pi 10 10000000
> >>>> >
> >>>> > Output is as follows:
> >>>> >
> >>>> >
> >>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> >>>> > filesystem name. Use
> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> >>>> > Number of Maps = 10 Samples per Map = 10000000
> >>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> >>>> > filesystem name. Use
> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> >>>> > 08/06/27 05:53:48 WARN fs.FileSystem:
> >>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> >>>> > filesystem name. Use
> >>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> >>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part0 retries left 4
> >>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part0 retries left 3
> >>>> > 08/06/27 05:53:49 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part0 retries left 2
> >>>> > 08/06/27 05:53:51 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part0 retries left 1
> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #0
> >>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part1 retries left 4
> >>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part1 retries left 3
> >>>> > 08/06/27 05:53:55 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part1 retries left 2
> >>>> > 08/06/27 05:53:57 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part1 retries left 1
> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #1
> >>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part2 retries left 4
> >>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part2 retries left 3
> >>>> > 08/06/27 05:54:01 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part2 retries left 2
> >>>> > 08/06/27 05:54:03 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part2 retries left 1
> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #2
> >>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> >>>> Method)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part3 retries left 4
> >>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part3 retries left 3
> >>>> > 08/06/27 05:54:07 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part3 retries left 2
> >>>> > 08/06/27 05:54:09 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part3 retries left 1
> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #3
> >>>> > 08/06/27 05:54:12 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part4 retries left 4
> >>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part4 retries left 3
> >>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part4 retries left 2
> >>>> > 08/06/27 05:54:15 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part4 retries left 1
> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #4
> >>>> > 08/06/27 05:54:18 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part5 retries left 4
> >>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part5 retries left 3
> >>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part5 retries left 2
> >>>> > 08/06/27 05:54:21 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part5 retries left 1
> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #5
> >>>> > 08/06/27 05:54:24 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part6 retries left 4
> >>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part6 retries left 3
> >>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part6 retries left 2
> >>>> > 08/06/27 05:54:27 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part6 retries left 1
> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null
> >>>> bad
> >>>> > datanode[0]
> >>>> > Wrote input for Map #6
> >>>> > 08/06/27 05:54:30 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part7 retries left 4
> >>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >>>> >
> >>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> >>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> >>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >>>> >
> >>>> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException
> >>>> sleeping
> >>>> > /user/root/test-mini-mr/in/part7 retries left 3
> >>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
> >>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> >>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0
> nodes,
> >>>> > instead of 1
> >>>> > at
> >>>> >
> >>>>
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> >>>> > at
> org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> >>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
> >>>> Source)
> >>>> > at
> >>>> >
> >>>>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> >>>> > at java.lang.reflect.Method.invoke(Method.java:597)
> >>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> >>>> >
> >>>
> >>>
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/EC2-public-AMI-tp18161182p18164284.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: EC2 public AMI
Posted by montag <mk...@about.com>.
Yes, that's the correct format. I have to ask, you meant
hadoop-ec2-env.sh, right?
In any case, I just successfully tested running the scripts from Mac OS
10.5. It don't think it could be an issue with your key pair or
hadoop-ec2-env.sh . Otherwise, I don't believe you would be able to boot a
master successfully. To be honest, I'm kind of at a loss. I would suggest
breaking the scripts down into their key launch components (i.e. run
launch-master and launch-slaves manually instead of just typing in
launch-cluster).
Were you able to build your own image using the hadoop scripts?
Cheers,
Mike
TimRobertson100 wrote:
>
> A daft question:
>
> Is this the correct format for the AWS account id in the
> hadoop-ec2-env.txt
> ?
> AWS_ACCOUNT_ID=XXXX-XXXX-0848
>
> It doesn't seem to matter if I remove the "-" as always slaves and master
> start...
>
>
>
> Thanks,
>
> Tim
>
>
>
> On Fri, Jun 27, 2008 at 9:44 PM, tim robertson <ti...@gmail.com>
> wrote:
>
>> Also in my .bashrc:
>> EC2_HOME=~/dev/ibiodiversity/aws/ec2-api-tools-1.3-19403
>> export EC2_HOME
>> EC2_PRIVATE_KEY=~/.ec2/pk-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
>> export EC2_PRIVATE_KEY
>> EC2_CERT=~/.ec2/cert-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
>> export EC2_CERT
>> EC2_KEYDIR=~/.ec2
>> export EC2_KEYDIR
>>
>>
>>
>>
>> On Fri, Jun 27, 2008 at 9:33 PM, tim robertson
>> <ti...@gmail.com>
>> wrote:
>>
>>> Hi Mike,
>>> Thanks for taking the time to try that! I appreciate it.
>>>
>>> I am running
>>>
>>> - Mac OSX 10.4
>>>
>>> PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
>>>
>>> To my knowledge there is nothing quirky about my network... I tried
>>> several times to start it up and always got the same. I am not sure why
>>> my
>>> OS should make a difference - am I not just issuing a request for an
>>> Amazon
>>> instance to be created using an image from S3? Or do some of my key
>>> files
>>> get copied across?
>>>
>>> I did get an error on the key file the very first time, saying it's
>>> permission were too open - so I ran "chmod 600" on the keypair, after a
>>> quick google search... could it be that some files are not getting
>>> copied
>>> correctly to the master?
>>>
>>> Cheers again,
>>>
>>> Tim
>>>
>>>
>>> On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
>>>
>>>>
>>>> Hi Tim,
>>>>
>>>> That's really strange. I launched a cluster of two and ran the pi
>>>> example
>>>> just now with no issue. This is most likely a dead end, but what OS
>>>> are
>>>> you
>>>> running the scripts on, what's your $PATH, and is there anything quirky
>>>> about your network? It's very weird that you are able to boot up the
>>>> master, but not the slaves.
>>>> If you have an S3 account, I would suggest trying to create your own
>>>> image
>>>> using the hadoop bundling scripts and launching a cluster based off of
>>>> that
>>>> image. You'll need to download the AMI tools from here:
>>>>
>>>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
>>>>
>>>> Cheers,
>>>> Mike
>>>>
>>>>
>>>>
>>>> TimRobertson100 wrote:
>>>> >
>>>> > Hi Mike,
>>>> > Thanks for the reply!
>>>> >
>>>> > I am using ec2-api-tools-1.3-19403 which I got from
>>>> >
>>>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
>>>> >
>>>> > The master seems to start, and there is an instance started for a
>>>> slave
>>>> I
>>>> > can see from the EC2 command line tools, but not through the hadoop
>>>> > console... but it doesn't seem to function (error log is attached)
>>>> >
>>>> > Thanks for taking time to reply,
>>>> >
>>>> > Tim
>>>> > (I am new to hadoop and ec2 so please don't hesitate to point the
>>>> obvious
>>>> > out to me. I have my own mapreduce code running on single server
>>>> mode
>>>> > fine...)
>>>> >
>>>> >
>>>> >
>>>> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
>>>> >
>>>> >>
>>>> >> Hi Tim,
>>>> >>
>>>> >> I've had no problems launching clusters using the small hadoop
>>>> 0.17.0
>>>> >> images. Just curious, what version of the Amazon EC2 command line
>>>> tools
>>>> >> are
>>>> >> you using?
>>>> >>
>>>> >> Thanks,
>>>> >> Mike
>>>> >>
>>>> >>
>>>> >> TimRobertson100 wrote:
>>>> >> >
>>>> >> > Hi all,
>>>> >> > I have been battling EC2 all day and getting nowhere (see other
>>>> >> message)
>>>> >> >
>>>> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
>>>> >> > instances
>>>> >> > successfully?
>>>> >> >
>>>> >> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately
>>>> doesn't
>>>> >> > work
>>>> >> > as the slaves don't come up (details in my earlier message).
>>>> >> >
>>>> >> > Thanks for any help or pointers to getting it running on EC2.
>>>> >> >
>>>> >> > Tim
>>>> >> >
>>>> >> >
>>>> >>
>>>> >> --
>>>> >> View this message in context:
>>>> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
>>>> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>>> >>
>>>> >>
>>>> >
>>>> > Start the cluster
>>>> >
>>>> > $ pwd
>>>> > /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
>>>> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
>>>> > Testing for existing master in group: test-cluster
>>>> > Starting master with AMI ami-ee53b687
>>>> > Waiting for instance i-62ea3c0b to start
>>>> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
>>>> > Warning: Permanently added
>>>> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to
>>>> the
>>>> > list of known hosts.
>>>> > Copying private key to master
>>>> > id_rsa-gsg-keypair
>>>> > 100% 1670 1.6KB/s 00:00
>>>> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is
>>>> 67.202.25.117,
>>>> > zone is us-east-1b.
>>>> > Adding test-cluster node(s) to cluster group test-cluster with AMI
>>>> > ami-ee53b687
>>>> > i-78ea3c11
>>>> > i-7bea3c12
>>>> >
>>>> >
>>>> > Wait for the nodes to start
>>>> > RESERVATION r-cc8449a5 544208310848 test-cluster-master
>>>> > INSTANCE i-62ea3c0b ami-ee53b687
>>>> > ec2-67-202-25-117.compute-1.amazonaws.com
>>>> > domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair
>>>> 0
>>>> > m1.small 2008-06-27T09:48:00+0000
>>>> us-east-1baki-a71cf9ce
>>>> > ari-a51cf9cc
>>>> > RESERVATION r-c28449ab 544208310848 test-cluster
>>>> > INSTANCE i-78ea3c11 ami-ee53b687
>>>> > ec2-75-101-232-80.compute-1.amazonaws.com
>>>> > domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair
>>>> 0
>>>> > m1.small 2008-06-27T09:49:19+0000
>>>> us-east-1baki-a71cf9ce
>>>> > ari-a51cf9cc
>>>> > INSTANCE i-7bea3c12 ami-ee53b687
>>>> > ec2-67-202-0-48.compute-1.amazonaws.com
>>>> > domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair
>>>> 1
>>>> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
>>>> > aki-a71cf9ce ari-a51cf9cc
>>>> >
>>>> >
>>>> > Go to the console:
>>>> > http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
>>>> > It says here Cluster Summary - Nodes = 0
>>>> > ???
>>>> >
>>>> >
>>>> >
>>>> > Login and run the test:
>>>> > $ bin/hadoop-ec2 login test-cluster
>>>> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
>>>> >
>>>> > __| __|_ ) Fedora 8
>>>> > _| ( / 32-bit
>>>> > ___|\___|___|
>>>> >
>>>> > Welcome to an EC2 Public Image
>>>> > :-)
>>>> > Base
>>>> >
>>>> > --[ see /etc/ec2/release-notes ]--
>>>> >
>>>> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
>>>> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
>>>> > hadoop-*-examples.jar pi 10 10000000
>>>> >
>>>> > Output is as follows:
>>>> >
>>>> >
>>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>>>> > filesystem name. Use
>>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>>>> > Number of Maps = 10 Samples per Map = 10000000
>>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>>>> > filesystem name. Use
>>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>>>> > 08/06/27 05:53:48 WARN fs.FileSystem:
>>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>>>> > filesystem name. Use
>>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part0 retries left 4
>>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part0 retries left 3
>>>> > 08/06/27 05:53:49 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part0 retries left 2
>>>> > 08/06/27 05:53:51 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part0 retries left 1
>>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #0
>>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part1 retries left 4
>>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part1 retries left 3
>>>> > 08/06/27 05:53:55 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part1 retries left 2
>>>> > 08/06/27 05:53:57 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part1 retries left 1
>>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #1
>>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part2 retries left 4
>>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part2 retries left 3
>>>> > 08/06/27 05:54:01 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part2 retries left 2
>>>> > 08/06/27 05:54:03 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part2 retries left 1
>>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #2
>>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>>> Method)
>>>> > at
>>>> >
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part3 retries left 4
>>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part3 retries left 3
>>>> > 08/06/27 05:54:07 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part3 retries left 2
>>>> > 08/06/27 05:54:09 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part3 retries left 1
>>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #3
>>>> > 08/06/27 05:54:12 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part4 retries left 4
>>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part4 retries left 3
>>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part4 retries left 2
>>>> > 08/06/27 05:54:15 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part4 retries left 1
>>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #4
>>>> > 08/06/27 05:54:18 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part5 retries left 4
>>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part5 retries left 3
>>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part5 retries left 2
>>>> > 08/06/27 05:54:21 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part5 retries left 1
>>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #5
>>>> > 08/06/27 05:54:24 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part6 retries left 4
>>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part6 retries left 3
>>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part6 retries left 2
>>>> > 08/06/27 05:54:27 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part6 retries left 1
>>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null
>>>> bad
>>>> > datanode[0]
>>>> > Wrote input for Map #6
>>>> > 08/06/27 05:54:30 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part7 retries left 4
>>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>>> >
>>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>>> > at
>>>> >
>>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>>> >
>>>> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException
>>>> sleeping
>>>> > /user/root/test-mini-mr/in/part7 retries left 3
>>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>>>> > instead of 1
>>>> > at
>>>> >
>>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown
>>>> Source)
>>>> > at
>>>> >
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>>> >
>>>
>>>
>
>
--
View this message in context: http://www.nabble.com/EC2-public-AMI-tp18161182p18164284.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: EC2 public AMI
Posted by tim robertson <ti...@gmail.com>.
A daft question:
Is this the correct format for the AWS account id in the hadoop-ec2-env.txt
?
AWS_ACCOUNT_ID=XXXX-XXXX-0848
It doesn't seem to matter if I remove the "-" as always slaves and master
start...
Thanks,
Tim
On Fri, Jun 27, 2008 at 9:44 PM, tim robertson <ti...@gmail.com>
wrote:
> Also in my .bashrc:
> EC2_HOME=~/dev/ibiodiversity/aws/ec2-api-tools-1.3-19403
> export EC2_HOME
> EC2_PRIVATE_KEY=~/.ec2/pk-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
> export EC2_PRIVATE_KEY
> EC2_CERT=~/.ec2/cert-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
> export EC2_CERT
> EC2_KEYDIR=~/.ec2
> export EC2_KEYDIR
>
>
>
>
> On Fri, Jun 27, 2008 at 9:33 PM, tim robertson <ti...@gmail.com>
> wrote:
>
>> Hi Mike,
>> Thanks for taking the time to try that! I appreciate it.
>>
>> I am running
>>
>> - Mac OSX 10.4
>>
>> PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
>>
>> To my knowledge there is nothing quirky about my network... I tried
>> several times to start it up and always got the same. I am not sure why my
>> OS should make a difference - am I not just issuing a request for an Amazon
>> instance to be created using an image from S3? Or do some of my key files
>> get copied across?
>>
>> I did get an error on the key file the very first time, saying it's
>> permission were too open - so I ran "chmod 600" on the keypair, after a
>> quick google search... could it be that some files are not getting copied
>> correctly to the master?
>>
>> Cheers again,
>>
>> Tim
>>
>>
>> On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
>>
>>>
>>> Hi Tim,
>>>
>>> That's really strange. I launched a cluster of two and ran the pi
>>> example
>>> just now with no issue. This is most likely a dead end, but what OS are
>>> you
>>> running the scripts on, what's your $PATH, and is there anything quirky
>>> about your network? It's very weird that you are able to boot up the
>>> master, but not the slaves.
>>> If you have an S3 account, I would suggest trying to create your own
>>> image
>>> using the hadoop bundling scripts and launching a cluster based off of
>>> that
>>> image. You'll need to download the AMI tools from here:
>>>
>>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
>>>
>>> Cheers,
>>> Mike
>>>
>>>
>>>
>>> TimRobertson100 wrote:
>>> >
>>> > Hi Mike,
>>> > Thanks for the reply!
>>> >
>>> > I am using ec2-api-tools-1.3-19403 which I got from
>>> >
>>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
>>> >
>>> > The master seems to start, and there is an instance started for a slave
>>> I
>>> > can see from the EC2 command line tools, but not through the hadoop
>>> > console... but it doesn't seem to function (error log is attached)
>>> >
>>> > Thanks for taking time to reply,
>>> >
>>> > Tim
>>> > (I am new to hadoop and ec2 so please don't hesitate to point the
>>> obvious
>>> > out to me. I have my own mapreduce code running on single server mode
>>> > fine...)
>>> >
>>> >
>>> >
>>> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
>>> >
>>> >>
>>> >> Hi Tim,
>>> >>
>>> >> I've had no problems launching clusters using the small hadoop 0.17.0
>>> >> images. Just curious, what version of the Amazon EC2 command line
>>> tools
>>> >> are
>>> >> you using?
>>> >>
>>> >> Thanks,
>>> >> Mike
>>> >>
>>> >>
>>> >> TimRobertson100 wrote:
>>> >> >
>>> >> > Hi all,
>>> >> > I have been battling EC2 all day and getting nowhere (see other
>>> >> message)
>>> >> >
>>> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
>>> >> > instances
>>> >> > successfully?
>>> >> >
>>> >> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately
>>> doesn't
>>> >> > work
>>> >> > as the slaves don't come up (details in my earlier message).
>>> >> >
>>> >> > Thanks for any help or pointers to getting it running on EC2.
>>> >> >
>>> >> > Tim
>>> >> >
>>> >> >
>>> >>
>>> >> --
>>> >> View this message in context:
>>> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
>>> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>> >>
>>> >>
>>> >
>>> > Start the cluster
>>> >
>>> > $ pwd
>>> > /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
>>> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
>>> > Testing for existing master in group: test-cluster
>>> > Starting master with AMI ami-ee53b687
>>> > Waiting for instance i-62ea3c0b to start
>>> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
>>> > Warning: Permanently added
>>> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to the
>>> > list of known hosts.
>>> > Copying private key to master
>>> > id_rsa-gsg-keypair
>>> > 100% 1670 1.6KB/s 00:00
>>> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is
>>> 67.202.25.117,
>>> > zone is us-east-1b.
>>> > Adding test-cluster node(s) to cluster group test-cluster with AMI
>>> > ami-ee53b687
>>> > i-78ea3c11
>>> > i-7bea3c12
>>> >
>>> >
>>> > Wait for the nodes to start
>>> > RESERVATION r-cc8449a5 544208310848 test-cluster-master
>>> > INSTANCE i-62ea3c0b ami-ee53b687
>>> > ec2-67-202-25-117.compute-1.amazonaws.com
>>> > domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair
>>> 0
>>> > m1.small 2008-06-27T09:48:00+0000 us-east-1baki-a71cf9ce
>>> > ari-a51cf9cc
>>> > RESERVATION r-c28449ab 544208310848 test-cluster
>>> > INSTANCE i-78ea3c11 ami-ee53b687
>>> > ec2-75-101-232-80.compute-1.amazonaws.com
>>> > domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair
>>> 0
>>> > m1.small 2008-06-27T09:49:19+0000 us-east-1baki-a71cf9ce
>>> > ari-a51cf9cc
>>> > INSTANCE i-7bea3c12 ami-ee53b687
>>> > ec2-67-202-0-48.compute-1.amazonaws.com
>>> > domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair
>>> 1
>>> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
>>> > aki-a71cf9ce ari-a51cf9cc
>>> >
>>> >
>>> > Go to the console:
>>> > http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
>>> > It says here Cluster Summary - Nodes = 0
>>> > ???
>>> >
>>> >
>>> >
>>> > Login and run the test:
>>> > $ bin/hadoop-ec2 login test-cluster
>>> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
>>> >
>>> > __| __|_ ) Fedora 8
>>> > _| ( / 32-bit
>>> > ___|\___|___|
>>> >
>>> > Welcome to an EC2 Public Image
>>> > :-)
>>> > Base
>>> >
>>> > --[ see /etc/ec2/release-notes ]--
>>> >
>>> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
>>> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
>>> > hadoop-*-examples.jar pi 10 10000000
>>> >
>>> > Output is as follows:
>>> >
>>> >
>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>>> > filesystem name. Use
>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>>> > Number of Maps = 10 Samples per Map = 10000000
>>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>>> > filesystem name. Use
>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>>> > 08/06/27 05:53:48 WARN fs.FileSystem:
>>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>>> > filesystem name. Use
>>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part0 retries left 4
>>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part0 retries left 3
>>> > 08/06/27 05:53:49 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part0 retries left 2
>>> > 08/06/27 05:53:51 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part0 retries left 1
>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #0
>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part1 retries left 4
>>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part1 retries left 3
>>> > 08/06/27 05:53:55 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part1 retries left 2
>>> > 08/06/27 05:53:57 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part1 retries left 1
>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #1
>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part2 retries left 4
>>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part2 retries left 3
>>> > 08/06/27 05:54:01 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part2 retries left 2
>>> > 08/06/27 05:54:03 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part2 retries left 1
>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #2
>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>> > at
>>> >
>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part3 retries left 4
>>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part3 retries left 3
>>> > 08/06/27 05:54:07 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part3 retries left 2
>>> > 08/06/27 05:54:09 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part3 retries left 1
>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #3
>>> > 08/06/27 05:54:12 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part4 retries left 4
>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part4 retries left 3
>>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part4 retries left 2
>>> > 08/06/27 05:54:15 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part4 retries left 1
>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #4
>>> > 08/06/27 05:54:18 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part5 retries left 4
>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part5 retries left 3
>>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part5 retries left 2
>>> > 08/06/27 05:54:21 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part5 retries left 1
>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #5
>>> > 08/06/27 05:54:24 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part6 retries left 4
>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part6 retries left 3
>>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part6 retries left 2
>>> > 08/06/27 05:54:27 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part6 retries left 1
>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null bad
>>> > datanode[0]
>>> > Wrote input for Map #6
>>> > 08/06/27 05:54:30 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part7 retries left 4
>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>>> >
>>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>>> > at
>>> >
>>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>>> > at
>>> >
>>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>>> >
>>> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException
>>> sleeping
>>> > /user/root/test-mini-mr/in/part7 retries left 3
>>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>>> > instead of 1
>>> > at
>>> >
>>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>>> > at
>>> >
>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>>> > at java.lang.reflect.Method.invoke(Method.java:597)
>>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>>> >
>>
>>
Re: EC2 public AMI
Posted by tim robertson <ti...@gmail.com>.
Also in my .bashrc:
EC2_HOME=~/dev/ibiodiversity/aws/ec2-api-tools-1.3-19403
export EC2_HOME
EC2_PRIVATE_KEY=~/.ec2/pk-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
export EC2_PRIVATE_KEY
EC2_CERT=~/.ec2/cert-MNKG53RUHACIXW4M5SRVIVNGXO5LIBV6.pem
export EC2_CERT
EC2_KEYDIR=~/.ec2
export EC2_KEYDIR
On Fri, Jun 27, 2008 at 9:33 PM, tim robertson <ti...@gmail.com>
wrote:
> Hi Mike,
> Thanks for taking the time to try that! I appreciate it.
>
> I am running
>
> - Mac OSX 10.4
>
> PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
>
> To my knowledge there is nothing quirky about my network... I tried several
> times to start it up and always got the same. I am not sure why my OS
> should make a difference - am I not just issuing a request for an Amazon
> instance to be created using an image from S3? Or do some of my key files
> get copied across?
>
> I did get an error on the key file the very first time, saying it's
> permission were too open - so I ran "chmod 600" on the keypair, after a
> quick google search... could it be that some files are not getting copied
> correctly to the master?
>
> Cheers again,
>
> Tim
>
>
> On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
>
>>
>> Hi Tim,
>>
>> That's really strange. I launched a cluster of two and ran the pi
>> example
>> just now with no issue. This is most likely a dead end, but what OS are
>> you
>> running the scripts on, what's your $PATH, and is there anything quirky
>> about your network? It's very weird that you are able to boot up the
>> master, but not the slaves.
>> If you have an S3 account, I would suggest trying to create your own
>> image
>> using the hadoop bundling scripts and launching a cluster based off of
>> that
>> image. You'll need to download the AMI tools from here:
>>
>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
>>
>> Cheers,
>> Mike
>>
>>
>>
>> TimRobertson100 wrote:
>> >
>> > Hi Mike,
>> > Thanks for the reply!
>> >
>> > I am using ec2-api-tools-1.3-19403 which I got from
>> >
>> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
>> >
>> > The master seems to start, and there is an instance started for a slave
>> I
>> > can see from the EC2 command line tools, but not through the hadoop
>> > console... but it doesn't seem to function (error log is attached)
>> >
>> > Thanks for taking time to reply,
>> >
>> > Tim
>> > (I am new to hadoop and ec2 so please don't hesitate to point the
>> obvious
>> > out to me. I have my own mapreduce code running on single server mode
>> > fine...)
>> >
>> >
>> >
>> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
>> >
>> >>
>> >> Hi Tim,
>> >>
>> >> I've had no problems launching clusters using the small hadoop 0.17.0
>> >> images. Just curious, what version of the Amazon EC2 command line
>> tools
>> >> are
>> >> you using?
>> >>
>> >> Thanks,
>> >> Mike
>> >>
>> >>
>> >> TimRobertson100 wrote:
>> >> >
>> >> > Hi all,
>> >> > I have been battling EC2 all day and getting nowhere (see other
>> >> message)
>> >> >
>> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
>> >> > instances
>> >> > successfully?
>> >> >
>> >> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately
>> doesn't
>> >> > work
>> >> > as the slaves don't come up (details in my earlier message).
>> >> >
>> >> > Thanks for any help or pointers to getting it running on EC2.
>> >> >
>> >> > Tim
>> >> >
>> >> >
>> >>
>> >> --
>> >> View this message in context:
>> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
>> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>> >>
>> >>
>> >
>> > Start the cluster
>> >
>> > $ pwd
>> > /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
>> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
>> > Testing for existing master in group: test-cluster
>> > Starting master with AMI ami-ee53b687
>> > Waiting for instance i-62ea3c0b to start
>> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
>> > Warning: Permanently added
>> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to the
>> > list of known hosts.
>> > Copying private key to master
>> > id_rsa-gsg-keypair
>> > 100% 1670 1.6KB/s 00:00
>> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is
>> 67.202.25.117,
>> > zone is us-east-1b.
>> > Adding test-cluster node(s) to cluster group test-cluster with AMI
>> > ami-ee53b687
>> > i-78ea3c11
>> > i-7bea3c12
>> >
>> >
>> > Wait for the nodes to start
>> > RESERVATION r-cc8449a5 544208310848 test-cluster-master
>> > INSTANCE i-62ea3c0b ami-ee53b687
>> > ec2-67-202-25-117.compute-1.amazonaws.com
>> > domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair
>> 0
>> > m1.small 2008-06-27T09:48:00+0000 us-east-1baki-a71cf9ce
>> > ari-a51cf9cc
>> > RESERVATION r-c28449ab 544208310848 test-cluster
>> > INSTANCE i-78ea3c11 ami-ee53b687
>> > ec2-75-101-232-80.compute-1.amazonaws.com
>> > domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair
>> 0
>> > m1.small 2008-06-27T09:49:19+0000 us-east-1baki-a71cf9ce
>> > ari-a51cf9cc
>> > INSTANCE i-7bea3c12 ami-ee53b687
>> > ec2-67-202-0-48.compute-1.amazonaws.com
>> > domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair
>> 1
>> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
>> > aki-a71cf9ce ari-a51cf9cc
>> >
>> >
>> > Go to the console:
>> > http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
>> > It says here Cluster Summary - Nodes = 0
>> > ???
>> >
>> >
>> >
>> > Login and run the test:
>> > $ bin/hadoop-ec2 login test-cluster
>> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
>> >
>> > __| __|_ ) Fedora 8
>> > _| ( / 32-bit
>> > ___|\___|___|
>> >
>> > Welcome to an EC2 Public Image
>> > :-)
>> > Base
>> >
>> > --[ see /etc/ec2/release-notes ]--
>> >
>> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
>> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
>> > hadoop-*-examples.jar pi 10 10000000
>> >
>> > Output is as follows:
>> >
>> >
>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>> > filesystem name. Use
>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>> > Number of Maps = 10 Samples per Map = 10000000
>> > 08/06/27 05:53:47 WARN fs.FileSystem:
>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>> > filesystem name. Use
>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>> > 08/06/27 05:53:48 WARN fs.FileSystem:
>> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
>> > filesystem name. Use
>> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part0 retries left 4
>> > 08/06/27 05:53:48 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part0 retries left 3
>> > 08/06/27 05:53:49 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part0 retries left 2
>> > 08/06/27 05:53:51 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part0 retries left 1
>> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #0
>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part1 retries left 4
>> > 08/06/27 05:53:54 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part1 retries left 3
>> > 08/06/27 05:53:55 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part1 retries left 2
>> > 08/06/27 05:53:57 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part1 retries left 1
>> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #1
>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part2 retries left 4
>> > 08/06/27 05:54:00 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part2 retries left 3
>> > 08/06/27 05:54:01 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part2 retries left 2
>> > 08/06/27 05:54:03 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part2 retries left 1
>> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #2
>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>> > at
>> >
>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part3 retries left 4
>> > 08/06/27 05:54:06 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part3 retries left 3
>> > 08/06/27 05:54:07 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part3 retries left 2
>> > 08/06/27 05:54:09 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part3 retries left 1
>> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #3
>> > 08/06/27 05:54:12 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part4 retries left 4
>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part4 retries left 3
>> > 08/06/27 05:54:13 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part4 retries left 2
>> > 08/06/27 05:54:15 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part4 retries left 1
>> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #4
>> > 08/06/27 05:54:18 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part5 retries left 4
>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part5 retries left 3
>> > 08/06/27 05:54:19 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part5 retries left 2
>> > 08/06/27 05:54:21 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part5 retries left 1
>> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #5
>> > 08/06/27 05:54:24 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part6 retries left 4
>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part6 retries left 3
>> > 08/06/27 05:54:25 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part6 retries left 2
>> > 08/06/27 05:54:27 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part6 retries left 1
>> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null bad
>> > datanode[0]
>> > Wrote input for Map #6
>> > 08/06/27 05:54:30 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part7 retries left 4
>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
>> > at
>> >
>> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
>> > at
>> >
>> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>> >
>> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException sleeping
>> > /user/root/test-mini-mr/in/part7 retries left 3
>> > 08/06/27 05:54:31 INFO dfs.DFSClient:
>> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
>> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
>> > instead of 1
>> > at
>> >
>> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
>> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
>> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>> >
>> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
>> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
>> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
>> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
>> > at
>> >
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
>> > at java.lang.reflect.Method.invoke(Method.java:597)
>> > at
>> > org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(Ret
>
> ...
>
> [Message clipped]
Re: EC2 public AMI
Posted by tim robertson <ti...@gmail.com>.
Hi Mike,
Thanks for taking the time to try that! I appreciate it.
I am running
- Mac OSX 10.4
PATH=$PATH:/usr/local/bin:$OPT_HOME/bin:/sw/bin:$MYSQL_HOME/bin:$POSTGRES_HOME/bin:$ANT_HOME/bin:$JBOSS_HOME/bin:$JAVA_HOME/bin:$ECLIPSE_HOME:$TOMCAT_HOME/bin:$MAVEN_HOME/bin:$EC2_HOME/bin
To my knowledge there is nothing quirky about my network... I tried several
times to start it up and always got the same. I am not sure why my OS
should make a difference - am I not just issuing a request for an Amazon
instance to be created using an image from S3? Or do some of my key files
get copied across?
I did get an error on the key file the very first time, saying it's
permission were too open - so I ran "chmod 600" on the keypair, after a
quick google search... could it be that some files are not getting copied
correctly to the master?
Cheers again,
Tim
On Fri, Jun 27, 2008 at 9:25 PM, montag <mk...@about.com> wrote:
>
> Hi Tim,
>
> That's really strange. I launched a cluster of two and ran the pi example
> just now with no issue. This is most likely a dead end, but what OS are
> you
> running the scripts on, what's your $PATH, and is there anything quirky
> about your network? It's very weird that you are able to boot up the
> master, but not the slaves.
> If you have an S3 account, I would suggest trying to create your own image
> using the hadoop bundling scripts and launching a cluster based off of that
> image. You'll need to download the AMI tools from here:
>
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
>
> Cheers,
> Mike
>
>
>
> TimRobertson100 wrote:
> >
> > Hi Mike,
> > Thanks for the reply!
> >
> > I am using ec2-api-tools-1.3-19403 which I got from
> >
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
> >
> > The master seems to start, and there is an instance started for a slave I
> > can see from the EC2 command line tools, but not through the hadoop
> > console... but it doesn't seem to function (error log is attached)
> >
> > Thanks for taking time to reply,
> >
> > Tim
> > (I am new to hadoop and ec2 so please don't hesitate to point the obvious
> > out to me. I have my own mapreduce code running on single server mode
> > fine...)
> >
> >
> >
> > On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
> >
> >>
> >> Hi Tim,
> >>
> >> I've had no problems launching clusters using the small hadoop 0.17.0
> >> images. Just curious, what version of the Amazon EC2 command line tools
> >> are
> >> you using?
> >>
> >> Thanks,
> >> Mike
> >>
> >>
> >> TimRobertson100 wrote:
> >> >
> >> > Hi all,
> >> > I have been battling EC2 all day and getting nowhere (see other
> >> message)
> >> >
> >> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
> >> > instances
> >> > successfully?
> >> >
> >> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately
> doesn't
> >> > work
> >> > as the slaves don't come up (details in my earlier message).
> >> >
> >> > Thanks for any help or pointers to getting it running on EC2.
> >> >
> >> > Tim
> >> >
> >> >
> >>
> >> --
> >> View this message in context:
> >> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
> >> Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >>
> >>
> >
> > Start the cluster
> >
> > $ pwd
> > /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
> > $ bin/hadoop-ec2 launch-cluster test-cluster 2
> > Testing for existing master in group: test-cluster
> > Starting master with AMI ami-ee53b687
> > Waiting for instance i-62ea3c0b to start
> > .............Started as domU-12-31-39-00-8A-03.compute-1.internal
> > Warning: Permanently added
> > 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to the
> > list of known hosts.
> > Copying private key to master
> > id_rsa-gsg-keypair
> > 100% 1670 1.6KB/s 00:00
> > Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is 67.202.25.117
> ,
> > zone is us-east-1b.
> > Adding test-cluster node(s) to cluster group test-cluster with AMI
> > ami-ee53b687
> > i-78ea3c11
> > i-7bea3c12
> >
> >
> > Wait for the nodes to start
> > RESERVATION r-cc8449a5 544208310848 test-cluster-master
> > INSTANCE i-62ea3c0b ami-ee53b687
> > ec2-67-202-25-117.compute-1.amazonaws.com
> > domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair 0
> > m1.small 2008-06-27T09:48:00+0000 us-east-1baki-a71cf9ce
> > ari-a51cf9cc
> > RESERVATION r-c28449ab 544208310848 test-cluster
> > INSTANCE i-78ea3c11 ami-ee53b687
> > ec2-75-101-232-80.compute-1.amazonaws.com
> > domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair 0
> > m1.small 2008-06-27T09:49:19+0000 us-east-1baki-a71cf9ce
> > ari-a51cf9cc
> > INSTANCE i-7bea3c12 ami-ee53b687
> > ec2-67-202-0-48.compute-1.amazonaws.com
> > domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair 1
> > m1.small 2008-06-27T09:49:19+0000 us-east-1b
> > aki-a71cf9ce ari-a51cf9cc
> >
> >
> > Go to the console:
> > http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
> > It says here Cluster Summary - Nodes = 0
> > ???
> >
> >
> >
> > Login and run the test:
> > $ bin/hadoop-ec2 login test-cluster
> > Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
> >
> > __| __|_ ) Fedora 8
> > _| ( / 32-bit
> > ___|\___|___|
> >
> > Welcome to an EC2 Public Image
> > :-)
> > Base
> >
> > --[ see /etc/ec2/release-notes ]--
> >
> > [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
> > [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
> > hadoop-*-examples.jar pi 10 10000000
> >
> > Output is as follows:
> >
> >
> > 08/06/27 05:53:47 WARN fs.FileSystem:
> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> > filesystem name. Use
> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> > Number of Maps = 10 Samples per Map = 10000000
> > 08/06/27 05:53:47 WARN fs.FileSystem:
> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> > filesystem name. Use
> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> > 08/06/27 05:53:48 WARN fs.FileSystem:
> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> > filesystem name. Use
> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> > 08/06/27 05:53:48 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part0 retries left 4
> > 08/06/27 05:53:48 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part0 retries left 3
> > 08/06/27 05:53:49 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part0 retries left 2
> > 08/06/27 05:53:51 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part0 retries left 1
> > 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #0
> > 08/06/27 05:53:54 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part1 retries left 4
> > 08/06/27 05:53:54 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part1 retries left 3
> > 08/06/27 05:53:55 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part1 retries left 2
> > 08/06/27 05:53:57 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part1 retries left 1
> > 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #1
> > 08/06/27 05:54:00 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part2 retries left 4
> > 08/06/27 05:54:00 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part2 retries left 3
> > 08/06/27 05:54:01 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part2 retries left 2
> > 08/06/27 05:54:03 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part2 retries left 1
> > 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #2
> > 08/06/27 05:54:06 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > at
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part3 retries left 4
> > 08/06/27 05:54:06 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part3 retries left 3
> > 08/06/27 05:54:07 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part3 retries left 2
> > 08/06/27 05:54:09 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part3 retries left 1
> > 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #3
> > 08/06/27 05:54:12 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part4 retries left 4
> > 08/06/27 05:54:13 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part4 retries left 3
> > 08/06/27 05:54:13 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part4 retries left 2
> > 08/06/27 05:54:15 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part4 retries left 1
> > 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #4
> > 08/06/27 05:54:18 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part5 retries left 4
> > 08/06/27 05:54:19 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part5 retries left 3
> > 08/06/27 05:54:19 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part5 retries left 2
> > 08/06/27 05:54:21 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part5 retries left 1
> > 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #5
> > 08/06/27 05:54:24 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part6 retries left 4
> > 08/06/27 05:54:25 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part6 retries left 3
> > 08/06/27 05:54:25 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part6 retries left 2
> > 08/06/27 05:54:27 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part6 retries left 1
> > 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #6
> > 08/06/27 05:54:30 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part7 retries left 4
> > 08/06/27 05:54:31 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part7 retries left 3
> > 08/06/27 05:54:31 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part7 retries left 2
> > 08/06/27 05:54:33 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:33 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part7 retries left 1
> > 08/06/27 05:54:36 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:36 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #7
> > 08/06/27 05:54:36 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:36 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part8 retries left 4
> > 08/06/27 05:54:37 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:37 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part8 retries left 3
> > 08/06/27 05:54:37 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:37 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part8 retries left 2
> > 08/06/27 05:54:39 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:39 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part8 retries left 1
> > 08/06/27 05:54:42 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:42 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #8
> > 08/06/27 05:54:42 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:42 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part9 retries left 4
> > 08/06/27 05:54:43 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:43 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part9 retries left 3
> > 08/06/27 05:54:43 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:43 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part9 retries left 2
> > 08/06/27 05:54:45 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:45 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /user/root/test-mini-mr/in/part9 retries left 1
> > 08/06/27 05:54:48 WARN dfs.DFSClient: DataStreamer Exception:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> > instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > 08/06/27 05:54:48 WARN dfs.DFSClient: Error Recovery for block null bad
> > datanode[0]
> > Wrote input for Map #9
> > Starting Job
> > 08/06/27 05:54:48 WARN fs.FileSystem:
> > "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> > filesystem name. Use
> > "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> > 08/06/27 05:54:48 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> > replicated to 0 nodes, instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar retries left 4
> > 08/06/27 05:54:49 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> > replicated to 0 nodes, instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
> >
> > 08/06/27 05:54:49 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> > /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar retries left 3
> > 08/06/27 05:54:50 INFO dfs.DFSClient:
> > org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> > /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> > replicated to 0 nodes, instead of 1
> > at
> >
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> > at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> > at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
> >
> > at org.apache.hadoop.ipc.Client.call(Client.java:557)
> > at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> > at
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> > at java.lang.reflect.Method.invoke(Method.java:597)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> > at
> >
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> > at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> > at
> >
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:233...
>
> [Message clipped]
Re: EC2 public AMI
Posted by montag <mk...@about.com>.
Hi Tim,
That's really strange. I launched a cluster of two and ran the pi example
just now with no issue. This is most likely a dead end, but what OS are you
running the scripts on, what's your $PATH, and is there anything quirky
about your network? It's very weird that you are able to boot up the
master, but not the slaves.
If you have an S3 account, I would suggest trying to create your own image
using the hadoop bundling scripts and launching a cluster based off of that
image. You'll need to download the AMI tools from here:
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=368&categoryID=88
Cheers,
Mike
TimRobertson100 wrote:
>
> Hi Mike,
> Thanks for the reply!
>
> I am using ec2-api-tools-1.3-19403 which I got from
> http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
>
> The master seems to start, and there is an instance started for a slave I
> can see from the EC2 command line tools, but not through the hadoop
> console... but it doesn't seem to function (error log is attached)
>
> Thanks for taking time to reply,
>
> Tim
> (I am new to hadoop and ec2 so please don't hesitate to point the obvious
> out to me. I have my own mapreduce code running on single server mode
> fine...)
>
>
>
> On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
>
>>
>> Hi Tim,
>>
>> I've had no problems launching clusters using the small hadoop 0.17.0
>> images. Just curious, what version of the Amazon EC2 command line tools
>> are
>> you using?
>>
>> Thanks,
>> Mike
>>
>>
>> TimRobertson100 wrote:
>> >
>> > Hi all,
>> > I have been battling EC2 all day and getting nowhere (see other
>> message)
>> >
>> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
>> > instances
>> > successfully?
>> >
>> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately doesn't
>> > work
>> > as the slaves don't come up (details in my earlier message).
>> >
>> > Thanks for any help or pointers to getting it running on EC2.
>> >
>> > Tim
>> >
>> >
>>
>> --
>> View this message in context:
>> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
>> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>>
>>
>
> Start the cluster
>
> $ pwd
> /Users/timrobertson/dev/ibiodiversity/hadoop-0.17.1/src/contrib/ec2
> $ bin/hadoop-ec2 launch-cluster test-cluster 2
> Testing for existing master in group: test-cluster
> Starting master with AMI ami-ee53b687
> Waiting for instance i-62ea3c0b to start
> .............Started as domU-12-31-39-00-8A-03.compute-1.internal
> Warning: Permanently added
> 'ec2-67-202-25-117.compute-1.amazonaws.com,67.202.25.117' (RSA) to the
> list of known hosts.
> Copying private key to master
> id_rsa-gsg-keypair
> 100% 1670 1.6KB/s 00:00
> Master is ec2-67-202-25-117.compute-1.amazonaws.com, ip is 67.202.25.117,
> zone is us-east-1b.
> Adding test-cluster node(s) to cluster group test-cluster with AMI
> ami-ee53b687
> i-78ea3c11
> i-7bea3c12
>
>
> Wait for the nodes to start
> RESERVATION r-cc8449a5 544208310848 test-cluster-master
> INSTANCE i-62ea3c0b ami-ee53b687
> ec2-67-202-25-117.compute-1.amazonaws.com
> domU-12-31-39-00-8A-03.compute-1.internal running gsg-keypair 0
> m1.small 2008-06-27T09:48:00+0000 us-east-1baki-a71cf9ce
> ari-a51cf9cc
> RESERVATION r-c28449ab 544208310848 test-cluster
> INSTANCE i-78ea3c11 ami-ee53b687
> ec2-75-101-232-80.compute-1.amazonaws.com
> domU-12-31-39-00-84-23.compute-1.internal running gsg-keypair 0
> m1.small 2008-06-27T09:49:19+0000 us-east-1baki-a71cf9ce
> ari-a51cf9cc
> INSTANCE i-7bea3c12 ami-ee53b687
> ec2-67-202-0-48.compute-1.amazonaws.com
> domU-12-31-39-00-85-C4.compute-1.internal running gsg-keypair 1
> m1.small 2008-06-27T09:49:19+0000 us-east-1b
> aki-a71cf9ce ari-a51cf9cc
>
>
> Go to the console:
> http://ec2-67-202-25-117.compute-1.amazonaws.com:50030/jobtracker.jsp
> It says here Cluster Summary - Nodes = 0
> ???
>
>
>
> Login and run the test:
> $ bin/hadoop-ec2 login test-cluster
> Logging in to host ec2-67-202-25-117.compute-1.amazonaws.com.
>
> __| __|_ ) Fedora 8
> _| ( / 32-bit
> ___|\___|___|
>
> Welcome to an EC2 Public Image
> :-)
> Base
>
> --[ see /etc/ec2/release-notes ]--
>
> [root@domU-12-31-39-00-8A-03 ~]# cd /usr/local/hadoop-*
> [root@domU-12-31-39-00-8A-03 hadoop-0.17.0]# bin/hadoop jar
> hadoop-*-examples.jar pi 10 10000000
>
> Output is as follows:
>
>
> 08/06/27 05:53:47 WARN fs.FileSystem:
> "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> filesystem name. Use
> "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> Number of Maps = 10 Samples per Map = 10000000
> 08/06/27 05:53:47 WARN fs.FileSystem:
> "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> filesystem name. Use
> "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> 08/06/27 05:53:48 WARN fs.FileSystem:
> "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> filesystem name. Use
> "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> 08/06/27 05:53:48 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part0 retries left 4
> 08/06/27 05:53:48 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part0 retries left 3
> 08/06/27 05:53:49 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:49 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part0 retries left 2
> 08/06/27 05:53:51 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:51 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part0 retries left 1
> 08/06/27 05:53:54 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part0 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:53:54 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #0
> 08/06/27 05:53:54 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part1 retries left 4
> 08/06/27 05:53:54 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:54 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part1 retries left 3
> 08/06/27 05:53:55 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:55 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part1 retries left 2
> 08/06/27 05:53:57 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:53:57 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part1 retries left 1
> 08/06/27 05:54:00 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part1 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:00 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #1
> 08/06/27 05:54:00 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part2 retries left 4
> 08/06/27 05:54:00 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:00 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part2 retries left 3
> 08/06/27 05:54:01 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:01 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part2 retries left 2
> 08/06/27 05:54:03 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:03 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part2 retries left 1
> 08/06/27 05:54:06 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part2 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:06 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #2
> 08/06/27 05:54:06 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part3 retries left 4
> 08/06/27 05:54:06 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:06 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part3 retries left 3
> 08/06/27 05:54:07 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:07 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part3 retries left 2
> 08/06/27 05:54:09 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:09 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part3 retries left 1
> 08/06/27 05:54:12 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part3 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:12 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #3
> 08/06/27 05:54:12 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:12 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part4 retries left 4
> 08/06/27 05:54:13 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part4 retries left 3
> 08/06/27 05:54:13 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:13 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part4 retries left 2
> 08/06/27 05:54:15 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:15 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part4 retries left 1
> 08/06/27 05:54:18 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part4 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:18 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #4
> 08/06/27 05:54:18 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:18 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part5 retries left 4
> 08/06/27 05:54:19 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part5 retries left 3
> 08/06/27 05:54:19 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:19 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part5 retries left 2
> 08/06/27 05:54:21 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:21 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part5 retries left 1
> 08/06/27 05:54:24 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part5 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:24 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #5
> 08/06/27 05:54:24 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:24 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part6 retries left 4
> 08/06/27 05:54:25 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part6 retries left 3
> 08/06/27 05:54:25 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:25 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part6 retries left 2
> 08/06/27 05:54:27 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:27 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part6 retries left 1
> 08/06/27 05:54:30 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part6 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:30 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #6
> 08/06/27 05:54:30 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:30 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part7 retries left 4
> 08/06/27 05:54:31 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part7 retries left 3
> 08/06/27 05:54:31 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:31 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part7 retries left 2
> 08/06/27 05:54:33 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:33 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part7 retries left 1
> 08/06/27 05:54:36 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part7 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:36 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #7
> 08/06/27 05:54:36 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:36 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part8 retries left 4
> 08/06/27 05:54:37 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:37 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part8 retries left 3
> 08/06/27 05:54:37 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:37 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part8 retries left 2
> 08/06/27 05:54:39 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:39 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part8 retries left 1
> 08/06/27 05:54:42 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part8 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:42 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #8
> 08/06/27 05:54:42 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:42 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part9 retries left 4
> 08/06/27 05:54:43 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:43 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part9 retries left 3
> 08/06/27 05:54:43 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:43 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part9 retries left 2
> 08/06/27 05:54:45 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:45 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /user/root/test-mini-mr/in/part9 retries left 1
> 08/06/27 05:54:48 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /user/root/test-mini-mr/in/part9 could only be replicated to 0 nodes,
> instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:48 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> Wrote input for Map #9
> Starting Job
> 08/06/27 05:54:48 WARN fs.FileSystem:
> "domU-12-31-39-00-8A-03.compute-1.internal:50001" is a deprecated
> filesystem name. Use
> "hdfs://domU-12-31-39-00-8A-03.compute-1.internal:50001/" instead.
> 08/06/27 05:54:48 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:48 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar retries left 4
> 08/06/27 05:54:49 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:49 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar retries left 3
> 08/06/27 05:54:50 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:50 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar retries left 2
> 08/06/27 05:54:51 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:51 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar retries left 1
> 08/06/27 05:54:55 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.jar could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:54:55 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> 08/06/27 05:54:55 INFO mapred.FileInputFormat: Total input paths to
> process : 10
> 08/06/27 05:54:55 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:55 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split retries left 4
> 08/06/27 05:54:55 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:55 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split retries left 3
> 08/06/27 05:54:56 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:56 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split retries left 2
> 08/06/27 05:54:58 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:54:58 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split retries left 1
> 08/06/27 05:55:01 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.split could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:55:01 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> 08/06/27 05:55:01 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:55:01 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml retries left 4
> 08/06/27 05:55:01 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:55:01 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml retries left 3
> 08/06/27 05:55:02 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:55:02 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml retries left 2
> 08/06/27 05:55:04 INFO dfs.DFSClient:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at org.apache.hadoop.dfs.$Proxy0.addBlock(Unknown Source)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.locateFollowingBlock(DFSClient.java:2334)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.nextBlockOutputStream(DFSClient.java:2219)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream.access$1700(DFSClient.java:1702)
> at
> org.apache.hadoop.dfs.DFSClient$DFSOutputStream$DataStreamer.run(DFSClient.java:1842)
>
> 08/06/27 05:55:04 WARN dfs.DFSClient: NotReplicatedYetException sleeping
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml retries left 1
> 08/06/27 05:55:07 WARN dfs.DFSClient: DataStreamer Exception:
> org.apache.hadoop.ipc.RemoteException: java.io.IOException: File
> /mnt/hadoop/mapred/system/job_200806270549_0001/job.xml could only be
> replicated to 0 nodes, instead of 1
> at
> org.apache.hadoop.dfs.FSNamesystem.getAdditionalBlock(FSNamesystem.java:1145)
> at org.apache.hadoop.dfs.NameNode.addBlock(NameNode.java:300)
> at sun.reflect.GeneratedMethodAccessor2.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> 08/06/27 05:55:07 WARN dfs.DFSClient: Error Recovery for block null bad
> datanode[0]
> org.apache.hadoop.ipc.RemoteException: java.io.IOException:
> java.lang.RuntimeException: org.xml.sax.SAXParseException: Premature end
> of file.
> at
> org.apache.hadoop.conf.Configuration.loadResource(Configuration.java:901)
> at
> org.apache.hadoop.conf.Configuration.loadResources(Configuration.java:804)
> at
> org.apache.hadoop.conf.Configuration.getProps(Configuration.java:772)
> at
> org.apache.hadoop.conf.Configuration.get(Configuration.java:272)
> at
> org.apache.hadoop.mapred.JobConf.getJobPriority(JobConf.java:1137)
> at
> org.apache.hadoop.mapred.JobInProgress.<init>(JobInProgress.java:177)
> at
> org.apache.hadoop.mapred.JobTracker.submitJob(JobTracker.java:1755)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:446)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:896)
>
> at org.apache.hadoop.ipc.Client.call(Client.java:557)
> at org.apache.hadoop.ipc.RPC$Invoker.invoke(RPC.java:212)
> at $Proxy1.submitJob(Unknown Source)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invokeMethod(RetryInvocationHandler.java:82)
> at
> org.apache.hadoop.io.retry.RetryInvocationHandler.invoke(RetryInvocationHandler.java:59)
> at $Proxy1.submitJob(Unknown Source)
> at
> org.apache.hadoop.mapred.JobClient.submitJob(JobClient.java:758)
> at org.apache.hadoop.mapred.JobClient.runJob(JobClient.java:973)
> at
> org.apache.hadoop.examples.PiEstimator.launch(PiEstimator.java:212)
> at
> org.apache.hadoop.examples.PiEstimator.run(PiEstimator.java:245)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at
> org.apache.hadoop.examples.PiEstimator.main(PiEstimator.java:252)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at
> org.apache.hadoop.util.ProgramDriver$ProgramDescription.invoke(ProgramDriver.java:68)
> at
> org.apache.hadoop.util.ProgramDriver.driver(ProgramDriver.java:139)
> at
> org.apache.hadoop.examples.ExampleDriver.main(ExampleDriver.java:53)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> at java.lang.reflect.Method.invoke(Method.java:597)
> at org.apache.hadoop.util.RunJar.main(RunJar.java:155)
> at org.apache.hadoop.mapred.JobShell.run(JobShell.java:194)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:65)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:79)
> at org.apache.hadoop.mapred.JobShell.main(JobShell.java:220)
>
>
>
--
View this message in context: http://www.nabble.com/EC2-public-AMI-tp18161182p18162200.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: EC2 public AMI
Posted by tim robertson <ti...@gmail.com>.
Hi Mike,
Thanks for the reply!
I am using ec2-api-tools-1.3-19403 which I got from
http://developer.amazonwebservices.com/connect/entry.jspa?externalID=351today
The master seems to start, and there is an instance started for a slave I
can see from the EC2 command line tools, but not through the hadoop
console... but it doesn't seem to function (error log is attached)
Thanks for taking time to reply,
Tim
(I am new to hadoop and ec2 so please don't hesitate to point the obvious
out to me. I have my own mapreduce code running on single server mode
fine...)
On Fri, Jun 27, 2008 at 8:43 PM, montag <mk...@about.com> wrote:
>
> Hi Tim,
>
> I've had no problems launching clusters using the small hadoop 0.17.0
> images. Just curious, what version of the Amazon EC2 command line tools
> are
> you using?
>
> Thanks,
> Mike
>
>
> TimRobertson100 wrote:
> >
> > Hi all,
> > I have been battling EC2 all day and getting nowhere (see other message)
> >
> > Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
> > instances
> > successfully?
> >
> > Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately doesn't
> > work
> > as the slaves don't come up (details in my earlier message).
> >
> > Thanks for any help or pointers to getting it running on EC2.
> >
> > Tim
> >
> >
>
> --
> View this message in context:
> http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>
Re: EC2 public AMI
Posted by montag <mk...@about.com>.
Hi Tim,
I've had no problems launching clusters using the small hadoop 0.17.0
images. Just curious, what version of the Amazon EC2 command line tools are
you using?
Thanks,
Mike
TimRobertson100 wrote:
>
> Hi all,
> I have been battling EC2 all day and getting nowhere (see other message)
>
> Does anyone use the hadoop-ec2-images/hadoop-0.17.0 AMI for small
> instances
> successfully?
>
> Following http://wiki.apache.org/hadoop/AmazonEC2 unfortunately doesn't
> work
> as the slaves don't come up (details in my earlier message).
>
> Thanks for any help or pointers to getting it running on EC2.
>
> Tim
>
>
--
View this message in context: http://www.nabble.com/EC2-public-AMI-tp18161182p18161469.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.