You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ken Krugler <kk...@transpac.com> on 2010/07/08 23:35:54 UTC
FAQ for "New to Hadoop"
Hi all,
I recently hosted an "Intro to Hadoop" session at the BigDataCamp
unconference last week. I later wrote down questions from the audience
that seemed useful to other Hadoop beginners, and the compared this to
the Hadoop project FAQ at http://wiki.apache.org/hadoop/FAQ
There was overlap, but not as much as I expected - the Hadoop FAQ has
more "how do I do X" versus "can I do X" or "why should I do X".
I posted these questions to http://www.scaleunlimited.com/blog/intro-to-hadoop-at-bigdatacamp
, and would appreciate any input - e.g. questions you think should
be there, answers you think aren't very clear (though mea culpa in
advance, I jotted these down quickly so I realize they're pretty rough).
Thanks,
-- Ken
--------------------------------------------
Ken Krugler
+1 530-210-6378
http://bixolabs.com
e l a s t i c w e b m i n i n g
Re: FAQ for "New to Hadoop"
Posted by Alex Baranau <al...@gmail.com>.
Ken,
You can also take a look at the FAQ section in the posts we publish
periodically. It started with
http://blog.sematext.com/2010/02/16/hadoop-digest-february-2010/. The
frequently asked questions are mainly retrieved from the project's user
mailing lists.
We also cover HBase (you can find posts on http://blog.sematext.com as well.
Alex Baranau
----
Sematext :: http://sematext.com/ :: Solr - Lucene - Nutch - Hadoop - HBase
Hadoop ecosystem search :: http://search-hadoop.com/
On Fri, Jul 9, 2010 at 1:35 AM, Mark Kerzner <ma...@gmail.com> wrote:
> Cool, Ken, thank you, I think it is very useful.
>
> Mark
>
> On Thu, Jul 8, 2010 at 4:35 PM, Ken Krugler <kkrugler_lists@transpac.com
> >wrote:
>
> > Hi all,
> >
> > I recently hosted an "Intro to Hadoop" session at the BigDataCamp
> > unconference last week. I later wrote down questions from the audience
> that
> > seemed useful to other Hadoop beginners, and the compared this to the
> Hadoop
> > project FAQ at http://wiki.apache.org/hadoop/FAQ
> >
> > There was overlap, but not as much as I expected - the Hadoop FAQ has
> more
> > "how do I do X" versus "can I do X" or "why should I do X".
> >
> > I posted these questions to
> > http://www.scaleunlimited.com/blog/intro-to-hadoop-at-bigdatacamp , and
> > would appreciate any input - e.g. questions you think should be there,
> > answers you think aren't very clear (though mea culpa in advance, I
> jotted
> > these down quickly so I realize they're pretty rough).
> >
> > Thanks,
> >
> > -- Ken
> >
> > --------------------------------------------
> > Ken Krugler
> > +1 530-210-6378
> > http://bixolabs.com
> > e l a s t i c w e b m i n i n g
> >
> >
>
Re: Help with Hadoop runtime error
Posted by Ted Yu <yu...@gmail.com>.
Do you happen to see something similar to:
10/03/17 15:47:58 WARN hdfs.DFSClient: NotReplicatedYetException sleeping
/user/perserver/data/575Gb/ps_
es_mstore_events_fact.txt retries left 4
10/03/17 15:47:58 INFO hdfs.DFSClient:
org.apache.hadoop.ipc.RemoteException:
org.apache.hadoop.hdfs.server.namenode.NotReplicatedYetException: Not
replicated yet:/user/perserver/data/575Gb/ps_es_mstore_events_fact.txt
Other people saw the above along with Bad connect ack error.
On Fri, Jul 9, 2010 at 2:06 PM, Raymond Jennings III
<ra...@yahoo.com>wrote:
> Hi Ted, thanks for your replay. That does not seem to make a difference
> though. I put that property in the xml file, restarted everything, tried
> to
> transfer the file again but the same thing occurred.
>
> I had my cluster working perfectly for about a year but I recently had some
> disk
> failures and scrubbed all of my machines reinstalled linux (same version)
> and
> moved from hadoop 0.20.1 to 0.20.2.
>
>
>
> ----- Original Message ----
> From: Ted Yu <yu...@gmail.com>
> To: common-user@hadoop.apache.org
> Sent: Fri, July 9, 2010 4:26:30 PM
> Subject: Re: Help with Hadoop runtime error
>
> Please see the description about xcievers at:
> http://hbase.apache.org/docs/r0.20.5/api/overview-summary.html#requirements
>
> You can confirm that you have a xcievers problem by grepping the
> datanode logs with the error message pasted in the last bullet point.
>
> On Fri, Jul 9, 2010 at 1:10 PM, Raymond Jennings III
> <ra...@yahoo.com>wrote:
>
> > Does anyone know what might be causing this error? I am using version
> > Hadoop
> > 0.20.2 and it happens when I run bin/hadoop dfs -copyFromLocal ...
> >
> > 10/07/09 15:51:45 INFO hdfs.DFSClient: Exception in
> createBlockOutputStream
> > java.io.IOException: Bad connect ack with firstBadLink
> 128.238.55.43:50010
> > 10/07/09 15:51:45 INFO hdfs.DFSClient: Abandoning block
> > blk_2932625575574450984_1002
> >
> >
> >
> >
>
>
>
>
>
Re: Help with Hadoop runtime error
Posted by Raymond Jennings III <ra...@yahoo.com>.
Hi Ted, thanks for your replay. That does not seem to make a difference
though. I put that property in the xml file, restarted everything, tried to
transfer the file again but the same thing occurred.
I had my cluster working perfectly for about a year but I recently had some disk
failures and scrubbed all of my machines reinstalled linux (same version) and
moved from hadoop 0.20.1 to 0.20.2.
----- Original Message ----
From: Ted Yu <yu...@gmail.com>
To: common-user@hadoop.apache.org
Sent: Fri, July 9, 2010 4:26:30 PM
Subject: Re: Help with Hadoop runtime error
Please see the description about xcievers at:
http://hbase.apache.org/docs/r0.20.5/api/overview-summary.html#requirements
You can confirm that you have a xcievers problem by grepping the
datanode logs with the error message pasted in the last bullet point.
On Fri, Jul 9, 2010 at 1:10 PM, Raymond Jennings III
<ra...@yahoo.com>wrote:
> Does anyone know what might be causing this error? I am using version
> Hadoop
> 0.20.2 and it happens when I run bin/hadoop dfs -copyFromLocal ...
>
> 10/07/09 15:51:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.io.IOException: Bad connect ack with firstBadLink 128.238.55.43:50010
> 10/07/09 15:51:45 INFO hdfs.DFSClient: Abandoning block
> blk_2932625575574450984_1002
>
>
>
>
Re: Help with Hadoop runtime error
Posted by Ted Yu <yu...@gmail.com>.
Please see the description about xcievers at:
http://hbase.apache.org/docs/r0.20.5/api/overview-summary.html#requirements
You can confirm that you have a xcievers problem by grepping the
datanode logs with the error message pasted in the last bullet point.
On Fri, Jul 9, 2010 at 1:10 PM, Raymond Jennings III
<ra...@yahoo.com>wrote:
> Does anyone know what might be causing this error? I am using version
> Hadoop
> 0.20.2 and it happens when I run bin/hadoop dfs -copyFromLocal ...
>
> 10/07/09 15:51:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream
> java.io.IOException: Bad connect ack with firstBadLink 128.238.55.43:50010
> 10/07/09 15:51:45 INFO hdfs.DFSClient: Abandoning block
> blk_2932625575574450984_1002
>
>
>
>
Help with Hadoop runtime error
Posted by Raymond Jennings III <ra...@yahoo.com>.
Does anyone know what might be causing this error? I am using version Hadoop
0.20.2 and it happens when I run bin/hadoop dfs -copyFromLocal ...
10/07/09 15:51:45 INFO hdfs.DFSClient: Exception in createBlockOutputStream
java.io.IOException: Bad connect ack with firstBadLink 128.238.55.43:50010
10/07/09 15:51:45 INFO hdfs.DFSClient: Abandoning block
blk_2932625575574450984_1002
Re: FAQ for "New to Hadoop"
Posted by Mark Kerzner <ma...@gmail.com>.
Cool, Ken, thank you, I think it is very useful.
Mark
On Thu, Jul 8, 2010 at 4:35 PM, Ken Krugler <kk...@transpac.com>wrote:
> Hi all,
>
> I recently hosted an "Intro to Hadoop" session at the BigDataCamp
> unconference last week. I later wrote down questions from the audience that
> seemed useful to other Hadoop beginners, and the compared this to the Hadoop
> project FAQ at http://wiki.apache.org/hadoop/FAQ
>
> There was overlap, but not as much as I expected - the Hadoop FAQ has more
> "how do I do X" versus "can I do X" or "why should I do X".
>
> I posted these questions to
> http://www.scaleunlimited.com/blog/intro-to-hadoop-at-bigdatacamp , and
> would appreciate any input - e.g. questions you think should be there,
> answers you think aren't very clear (though mea culpa in advance, I jotted
> these down quickly so I realize they're pretty rough).
>
> Thanks,
>
> -- Ken
>
> --------------------------------------------
> Ken Krugler
> +1 530-210-6378
> http://bixolabs.com
> e l a s t i c w e b m i n i n g
>
>