You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by David Gkogkritsiani <da...@gmail.com> on 2013/04/23 01:01:59 UTC

Hadoop MapReduce

Helllo,


I have undertaken my diploma thesis on Hadoop MapReduce and I have been
requested to I do an application written in MapReduce.
I found on internet this code and I ran the code :

http://paste.ubuntu.com/5591999/

How can I add the code to stores the pages somewhere locally (text only,
not Images) and then have to be processed . ie,I should a Mapreduce code,
which would download pages from the web and store on the local file system
and not the HDFS.
After ,I run the quest (program) in order to not depend on network speed.

Because ,my network is so slow.

I do this to improvement performance.

I am running Hadoop Version 0.20.2 .
I am new to Hadoop and am kinda lost and any help would be greatly
appreciated.

Thanks in advance for any assistance !

Re: Hadoop MapReduce

Posted by Daryn Sharp <da...@yahoo-inc.com>.
MR has a "local mode" that does what you want.  Pig has the ability to use this mode.  I did a quick search but didn't immediately find a good link to documentation, but hopefully this gets you going in the right direction.

Daryn

On Apr 22, 2013, at 6:01 PM, David Gkogkritsiani wrote:

Helllo,

I have undertaken my diploma thesis on Hadoop MapReduce and I have been requested to I do an application written in MapReduce.
I found on internet this code and I ran the code :

http://paste.ubuntu.com/5591999/

How can I add the code to stores the pages somewhere locally (text only, not Images) and then have to be processed . ie,I should a Mapreduce code, which would download pages from the web and store on the local file system and not the HDFS.
After ,I run the quest (program) in order to not depend on network speed.

Because ,my network is so slow.

I do this to improvement performance.

I am running Hadoop Version 0.20.2 .
I am new to Hadoop and am kinda lost and any help would be greatly appreciated.

Thanks in advance for any assistance !

Re: Hadoop MapReduce

Posted by Daryn Sharp <da...@yahoo-inc.com>.
MR has a "local mode" that does what you want.  Pig has the ability to use this mode.  I did a quick search but didn't immediately find a good link to documentation, but hopefully this gets you going in the right direction.

Daryn

On Apr 22, 2013, at 6:01 PM, David Gkogkritsiani wrote:

Helllo,

I have undertaken my diploma thesis on Hadoop MapReduce and I have been requested to I do an application written in MapReduce.
I found on internet this code and I ran the code :

http://paste.ubuntu.com/5591999/

How can I add the code to stores the pages somewhere locally (text only, not Images) and then have to be processed . ie,I should a Mapreduce code, which would download pages from the web and store on the local file system and not the HDFS.
After ,I run the quest (program) in order to not depend on network speed.

Because ,my network is so slow.

I do this to improvement performance.

I am running Hadoop Version 0.20.2 .
I am new to Hadoop and am kinda lost and any help would be greatly appreciated.

Thanks in advance for any assistance !

Re: Hadoop MapReduce

Posted by Daryn Sharp <da...@yahoo-inc.com>.
MR has a "local mode" that does what you want.  Pig has the ability to use this mode.  I did a quick search but didn't immediately find a good link to documentation, but hopefully this gets you going in the right direction.

Daryn

On Apr 22, 2013, at 6:01 PM, David Gkogkritsiani wrote:

Helllo,

I have undertaken my diploma thesis on Hadoop MapReduce and I have been requested to I do an application written in MapReduce.
I found on internet this code and I ran the code :

http://paste.ubuntu.com/5591999/

How can I add the code to stores the pages somewhere locally (text only, not Images) and then have to be processed . ie,I should a Mapreduce code, which would download pages from the web and store on the local file system and not the HDFS.
After ,I run the quest (program) in order to not depend on network speed.

Because ,my network is so slow.

I do this to improvement performance.

I am running Hadoop Version 0.20.2 .
I am new to Hadoop and am kinda lost and any help would be greatly appreciated.

Thanks in advance for any assistance !

Re: Hadoop MapReduce

Posted by Daryn Sharp <da...@yahoo-inc.com>.
MR has a "local mode" that does what you want.  Pig has the ability to use this mode.  I did a quick search but didn't immediately find a good link to documentation, but hopefully this gets you going in the right direction.

Daryn

On Apr 22, 2013, at 6:01 PM, David Gkogkritsiani wrote:

Helllo,

I have undertaken my diploma thesis on Hadoop MapReduce and I have been requested to I do an application written in MapReduce.
I found on internet this code and I ran the code :

http://paste.ubuntu.com/5591999/

How can I add the code to stores the pages somewhere locally (text only, not Images) and then have to be processed . ie,I should a Mapreduce code, which would download pages from the web and store on the local file system and not the HDFS.
After ,I run the quest (program) in order to not depend on network speed.

Because ,my network is so slow.

I do this to improvement performance.

I am running Hadoop Version 0.20.2 .
I am new to Hadoop and am kinda lost and any help would be greatly appreciated.

Thanks in advance for any assistance !