You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ahmad Shahzad <as...@gmail.com> on 2010/10/22 16:31:31 UTC
what happens inside hadoop !!
Hi ALL,
Is there any documentation or guide or any presentation about
what happens inside hadoop. I mean, there are different documentation about
map-reduce and hdfs and they tell what the do, but what is happening inside
is not mentioned in those articles. Any idea !!
Ahmad
Re: what happens inside hadoop !!
Posted by Null Ecksor <nu...@gmail.com>.
http://hadoop.apache.org/hdfs/docs/current/api/index.html
Javadocs is helpful. You can see how/which methods are called i.e., how
things are happening.
On Fri, Oct 22, 2010 at 10:31 AM, Ahmad Shahzad <as...@gmail.com> wrote:
> Hi ALL,
> Is there any documentation or guide or any presentation about
> what happens inside hadoop. I mean, there are different documentation about
> map-reduce and hdfs and they tell what the do, but what is happening inside
> is not mentioned in those articles. Any idea !!
>
> Ahmad
>
RC
Re: what happens inside hadoop !!
Posted by Null Ecksor <nu...@gmail.com>.
http://hadoop.apache.org/hdfs/docs/current/api/index.html
Javadocs is helpful. You can see how/which methods are called i.e., how
things are happening.
On Fri, Oct 22, 2010 at 10:31 AM, Ahmad Shahzad <as...@gmail.com> wrote:
> Hi ALL,
> Is there any documentation or guide or any presentation about
> what happens inside hadoop. I mean, there are different documentation about
> map-reduce and hdfs and they tell what the do, but what is happening inside
> is not mentioned in those articles. Any idea !!
>
> Ahmad
>
RC
Re: what happens inside hadoop !!
Posted by George Romantsow <g....@gmail.com>.
Thanks to everybody for their answers. That was very helpfull although
that wasn't me who asked the question.
George
Re: what happens inside hadoop !!
Posted by Greg Roelofs <ro...@yahoo-inc.com>.
Harsh J <qw...@gmail.com> wrote:
> The source is your friend.
In particular, start with JobTracker.java: initJob() for Job and
TaskInProgress creation/init, and heartbeat() for Task creation.
TaskTracker.java, JobInProgress.java, TaskInProgress.java, *Task.java,
*TaskRunner.java, *TaskController.java, and JvmManager.java are the
other core MR files, I think. Internal communication protocols include
InterTrackerProtocol (JT-TT), TaskUmbilicalProtocol (TT-Task), and
HTTP/Jetty (MapTask-ReduceTask and others, I believe). There are
probably more of them, but those are the ones I've come across so far.
I haven't dug into HDFS.
Greg
Re: what happens inside hadoop !!
Posted by Harsh J <qw...@gmail.com>.
The source is your friend. And perhaps a good Java IDE too. I use Eclipse + F3.
But since you ask, you may begin at the wiki:
http://wiki.apache.org/hadoop/FrontPage
There's stuff there not many see, and those pretty much cover enough
to get you started at the right places :)
About a document, I guess O'Malley's Hadoop MR Arch one would help,
but nothing beats reading sources the way its supposed to be done:
http://docs.huihoo.com/apache/hadoop/HadoopMapReduceArch.pdf
On Fri, Oct 22, 2010 at 8:01 PM, Ahmad Shahzad <as...@gmail.com> wrote:
> Hi ALL,
> Is there any documentation or guide or any presentation about
> what happens inside hadoop. I mean, there are different documentation about
> map-reduce and hdfs and they tell what the do, but what is happening inside
> is not mentioned in those articles. Any idea !!
>
> Ahmad
>
--
Harsh J
www.harshj.com