You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Bharati <bh...@mparallelo.com> on 2013/05/22 23:25:44 UTC
Eclipse plugin
Hi,
I am trying to get or build eclipse plugin for 1.2.0
All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
I need to build a hadoop map reduce project and be able to debug in eclipse.
Thanks,
Bharati
Sent from my iPad
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Excerpts from a Development Arch doc I had created
2 Architecture
We will use Core Java , MySQL, Hadoop, SVN, Junit, Maven.
For webservices will use Spring MVC & JBOSS.
2.1 Hardware Requirements
Either Mac/Snow Leopard(or higher) or PC/Laptop/Windows
8GB RAM recommended
2 X 500GB disc recommended
2.2 Software Requirements
For Windows 64bit
Java
Java, java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
Eclipse - version eclipse-SDK-3.6.2-win32-x86_64 http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-win32-x86_64.zip
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
You can, for example, use the Sun 64-bit 1.5 JVM for AMD64. Note
that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be
used to run this build.
apache-maven-3.0.3 : http://archive.apache.org/dist/maven/binaries/apache-maven-3.0.3-bin.zip
add maven bin to the path
q4e maven plugin - http://q4e.googlecode.com/svn/trunk/updatesite-iam/
(if q4e does not work then u can use http://www.sonatype.org/m2eclipse/
subclipe 1.8x plugin - http://subclipse.tigris.org/update_1.8.x<http://subclipse.tigris.org/update_1.8.x>
Cygwin : required to run Hadoop on a Windows Box. Add cygwin executable to path
For Mac
Java
same version as mentioned for Windows
Eclipse - version eclipse-SDK-3.6.2
http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-macosx-cocoa-x86_64.tar.gz
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
From: Bharati Adkar <bh...@mparallelo.com>>
Date: Wednesday, May 22, 2013 4:37 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Sanjay Subramanian <sa...@wizecommerce.com>>
Subject: Re: Eclipse plugin
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Excerpts from a Development Arch doc I had created
2 Architecture
We will use Core Java , MySQL, Hadoop, SVN, Junit, Maven.
For webservices will use Spring MVC & JBOSS.
2.1 Hardware Requirements
Either Mac/Snow Leopard(or higher) or PC/Laptop/Windows
8GB RAM recommended
2 X 500GB disc recommended
2.2 Software Requirements
For Windows 64bit
Java
Java, java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
Eclipse - version eclipse-SDK-3.6.2-win32-x86_64 http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-win32-x86_64.zip
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
You can, for example, use the Sun 64-bit 1.5 JVM for AMD64. Note
that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be
used to run this build.
apache-maven-3.0.3 : http://archive.apache.org/dist/maven/binaries/apache-maven-3.0.3-bin.zip
add maven bin to the path
q4e maven plugin - http://q4e.googlecode.com/svn/trunk/updatesite-iam/
(if q4e does not work then u can use http://www.sonatype.org/m2eclipse/
subclipe 1.8x plugin - http://subclipse.tigris.org/update_1.8.x<http://subclipse.tigris.org/update_1.8.x>
Cygwin : required to run Hadoop on a Windows Box. Add cygwin executable to path
For Mac
Java
same version as mentioned for Windows
Eclipse - version eclipse-SDK-3.6.2
http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-macosx-cocoa-x86_64.tar.gz
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
From: Bharati Adkar <bh...@mparallelo.com>>
Date: Wednesday, May 22, 2013 4:37 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Sanjay Subramanian <sa...@wizecommerce.com>>
Subject: Re: Eclipse plugin
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Excerpts from a Development Arch doc I had created
2 Architecture
We will use Core Java , MySQL, Hadoop, SVN, Junit, Maven.
For webservices will use Spring MVC & JBOSS.
2.1 Hardware Requirements
Either Mac/Snow Leopard(or higher) or PC/Laptop/Windows
8GB RAM recommended
2 X 500GB disc recommended
2.2 Software Requirements
For Windows 64bit
Java
Java, java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
Eclipse - version eclipse-SDK-3.6.2-win32-x86_64 http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-win32-x86_64.zip
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
You can, for example, use the Sun 64-bit 1.5 JVM for AMD64. Note
that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be
used to run this build.
apache-maven-3.0.3 : http://archive.apache.org/dist/maven/binaries/apache-maven-3.0.3-bin.zip
add maven bin to the path
q4e maven plugin - http://q4e.googlecode.com/svn/trunk/updatesite-iam/
(if q4e does not work then u can use http://www.sonatype.org/m2eclipse/
subclipe 1.8x plugin - http://subclipse.tigris.org/update_1.8.x<http://subclipse.tigris.org/update_1.8.x>
Cygwin : required to run Hadoop on a Windows Box. Add cygwin executable to path
For Mac
Java
same version as mentioned for Windows
Eclipse - version eclipse-SDK-3.6.2
http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-macosx-cocoa-x86_64.tar.gz
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
From: Bharati Adkar <bh...@mparallelo.com>>
Date: Wednesday, May 22, 2013 4:37 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Sanjay Subramanian <sa...@wizecommerce.com>>
Subject: Re: Eclipse plugin
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Excerpts from a Development Arch doc I had created
2 Architecture
We will use Core Java , MySQL, Hadoop, SVN, Junit, Maven.
For webservices will use Spring MVC & JBOSS.
2.1 Hardware Requirements
Either Mac/Snow Leopard(or higher) or PC/Laptop/Windows
8GB RAM recommended
2 X 500GB disc recommended
2.2 Software Requirements
For Windows 64bit
Java
Java, java version "1.6.0_29"
Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
Eclipse - version eclipse-SDK-3.6.2-win32-x86_64 http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-win32-x86_64.zip
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
You can, for example, use the Sun 64-bit 1.5 JVM for AMD64. Note
that the Sun 1.4.2 JVM for AMD64 is 32-bit and therefore cannot be
used to run this build.
apache-maven-3.0.3 : http://archive.apache.org/dist/maven/binaries/apache-maven-3.0.3-bin.zip
add maven bin to the path
q4e maven plugin - http://q4e.googlecode.com/svn/trunk/updatesite-iam/
(if q4e does not work then u can use http://www.sonatype.org/m2eclipse/
subclipe 1.8x plugin - http://subclipse.tigris.org/update_1.8.x<http://subclipse.tigris.org/update_1.8.x>
Cygwin : required to run Hadoop on a Windows Box. Add cygwin executable to path
For Mac
Java
same version as mentioned for Windows
Eclipse - version eclipse-SDK-3.6.2
http://www.eclipse.org/downloads/download.php?file=/eclipse/downloads/drops/R-3.6.2-201102101200/eclipse-SDK-3.6.2-macosx-cocoa-x86_64.tar.gz
Note: This build requires a 64-bit JVM, and will not run with a 32-bit JVM.
From: Bharati Adkar <bh...@mparallelo.com>>
Date: Wednesday, May 22, 2013 4:37 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>, Sanjay Subramanian <sa...@wizecommerce.com>>
Subject: Re: Eclipse plugin
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com>> wrote:
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Bharati Adkar <bh...@mparallelo.com>.
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com> wrote:
> Hi
>
> I don't use any need any special plugin to walk thru the code
>
> All my map reduce jobs have a
>
> JobMapper.java
> JobReducer.java
> JobProcessor.java (set any configs u like)
>
> I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
>
> Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
>
>
> MAVEN REPO INFO
> ================
> <repositories>
> <repository>
> <id>Cloudera repository</id>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> </repositories>
>
> <properties>
> <cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
> </properties>
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-mapreduce-client-core</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
>
> WordCountNew (please modify as needed)
> ======================================
>
> public class WordCountNew {
>
>
>
> public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
>
> private final static IntWritable one = new IntWritable(1);
>
> private Text word = new Text();
>
>
>
> public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
> FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
> // System.out.println(value.toString());
> String fileName = fileSplit.getPath().toString();
> String line = value.toString();
> StringTokenizer tokenizer = new StringTokenizer(line);
> while (tokenizer.hasMoreTokens()) {
> word.set(tokenizer.nextToken());
> ctxt.write(word, one);
>
> }
>
> }
>
> }
>
>
>
> public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
>
> public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
>
> int sum = 0;
>
> for (IntWritable value : values) {
>
> sum += value.get();
>
> }
>
> ctxt.write(key, new IntWritable(sum));
>
> }
>
> }
>
>
>
> public static void main(String[] args) throws Exception {
> org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
> hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
>
>
> Job job = new Job(hadoopConf);
>
> job.setJobName("wordcountNEW");
>
> job.setJarByClass(WordCountNew.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
> job.setMapOutputKeyClass(Text.class);
>
> job.setMapOutputValueClass(IntWritable.class);
>
>
>
> job.setMapperClass(WordCountNew.Map.class);
>
> job.setCombinerClass(WordCountNew.Reduce.class);
>
> job.setReducerClass(Reduce.class);
>
> //
> job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
>
> job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
>
>
> job.setOutputFormatClass(TextOutputFormat.class);
>
>
> if (FileUtils.doesFileOrDirectoryExist(args[1])){
>
> org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
>
> }
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>
>
> job.waitForCompletion(true);
>
> System.out.println();
>
> }
> }
>
>
>
>
>
> From: Bharati <bh...@mparallelo.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, May 22, 2013 3:39 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Eclipse plugin
>
> Hi Jing,
>
> I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
>
> For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
> How can I use eclipse to rewrite the code.
>
> The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
>
> Thanks,
> Bharati
>
> Sent from my iPad
>
> On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
>
> > Hi Bharati,
> >
> > Usually you only need to run "ant clean jar jar-test" and "ant
> > eclipse" on your code base, and then import the project into your
> > eclipse. Can you provide some more detailed description about the
> > problem you met?
> >
> > Thanks,
> > -Jing
> >
> > On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> >> Hi,
> >>
> >> I am trying to get or build eclipse plugin for 1.2.0
> >>
> >> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
> >>
> >> I need to build a hadoop map reduce project and be able to debug in eclipse.
> >>
> >> Thanks,
> >> Bharati
> >> Sent from my iPad
> >> Fortigate Filtered
> >>
> Fortigate Filtered
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Bharati Adkar <bh...@mparallelo.com>.
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com> wrote:
> Hi
>
> I don't use any need any special plugin to walk thru the code
>
> All my map reduce jobs have a
>
> JobMapper.java
> JobReducer.java
> JobProcessor.java (set any configs u like)
>
> I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
>
> Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
>
>
> MAVEN REPO INFO
> ================
> <repositories>
> <repository>
> <id>Cloudera repository</id>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> </repositories>
>
> <properties>
> <cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
> </properties>
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-mapreduce-client-core</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
>
> WordCountNew (please modify as needed)
> ======================================
>
> public class WordCountNew {
>
>
>
> public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
>
> private final static IntWritable one = new IntWritable(1);
>
> private Text word = new Text();
>
>
>
> public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
> FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
> // System.out.println(value.toString());
> String fileName = fileSplit.getPath().toString();
> String line = value.toString();
> StringTokenizer tokenizer = new StringTokenizer(line);
> while (tokenizer.hasMoreTokens()) {
> word.set(tokenizer.nextToken());
> ctxt.write(word, one);
>
> }
>
> }
>
> }
>
>
>
> public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
>
> public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
>
> int sum = 0;
>
> for (IntWritable value : values) {
>
> sum += value.get();
>
> }
>
> ctxt.write(key, new IntWritable(sum));
>
> }
>
> }
>
>
>
> public static void main(String[] args) throws Exception {
> org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
> hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
>
>
> Job job = new Job(hadoopConf);
>
> job.setJobName("wordcountNEW");
>
> job.setJarByClass(WordCountNew.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
> job.setMapOutputKeyClass(Text.class);
>
> job.setMapOutputValueClass(IntWritable.class);
>
>
>
> job.setMapperClass(WordCountNew.Map.class);
>
> job.setCombinerClass(WordCountNew.Reduce.class);
>
> job.setReducerClass(Reduce.class);
>
> //
> job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
>
> job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
>
>
> job.setOutputFormatClass(TextOutputFormat.class);
>
>
> if (FileUtils.doesFileOrDirectoryExist(args[1])){
>
> org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
>
> }
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>
>
> job.waitForCompletion(true);
>
> System.out.println();
>
> }
> }
>
>
>
>
>
> From: Bharati <bh...@mparallelo.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, May 22, 2013 3:39 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Eclipse plugin
>
> Hi Jing,
>
> I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
>
> For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
> How can I use eclipse to rewrite the code.
>
> The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
>
> Thanks,
> Bharati
>
> Sent from my iPad
>
> On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
>
> > Hi Bharati,
> >
> > Usually you only need to run "ant clean jar jar-test" and "ant
> > eclipse" on your code base, and then import the project into your
> > eclipse. Can you provide some more detailed description about the
> > problem you met?
> >
> > Thanks,
> > -Jing
> >
> > On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> >> Hi,
> >>
> >> I am trying to get or build eclipse plugin for 1.2.0
> >>
> >> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
> >>
> >> I need to build a hadoop map reduce project and be able to debug in eclipse.
> >>
> >> Thanks,
> >> Bharati
> >> Sent from my iPad
> >> Fortigate Filtered
> >>
> Fortigate Filtered
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Bharati Adkar <bh...@mparallelo.com>.
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com> wrote:
> Hi
>
> I don't use any need any special plugin to walk thru the code
>
> All my map reduce jobs have a
>
> JobMapper.java
> JobReducer.java
> JobProcessor.java (set any configs u like)
>
> I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
>
> Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
>
>
> MAVEN REPO INFO
> ================
> <repositories>
> <repository>
> <id>Cloudera repository</id>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> </repositories>
>
> <properties>
> <cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
> </properties>
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-mapreduce-client-core</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
>
> WordCountNew (please modify as needed)
> ======================================
>
> public class WordCountNew {
>
>
>
> public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
>
> private final static IntWritable one = new IntWritable(1);
>
> private Text word = new Text();
>
>
>
> public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
> FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
> // System.out.println(value.toString());
> String fileName = fileSplit.getPath().toString();
> String line = value.toString();
> StringTokenizer tokenizer = new StringTokenizer(line);
> while (tokenizer.hasMoreTokens()) {
> word.set(tokenizer.nextToken());
> ctxt.write(word, one);
>
> }
>
> }
>
> }
>
>
>
> public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
>
> public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
>
> int sum = 0;
>
> for (IntWritable value : values) {
>
> sum += value.get();
>
> }
>
> ctxt.write(key, new IntWritable(sum));
>
> }
>
> }
>
>
>
> public static void main(String[] args) throws Exception {
> org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
> hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
>
>
> Job job = new Job(hadoopConf);
>
> job.setJobName("wordcountNEW");
>
> job.setJarByClass(WordCountNew.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
> job.setMapOutputKeyClass(Text.class);
>
> job.setMapOutputValueClass(IntWritable.class);
>
>
>
> job.setMapperClass(WordCountNew.Map.class);
>
> job.setCombinerClass(WordCountNew.Reduce.class);
>
> job.setReducerClass(Reduce.class);
>
> //
> job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
>
> job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
>
>
> job.setOutputFormatClass(TextOutputFormat.class);
>
>
> if (FileUtils.doesFileOrDirectoryExist(args[1])){
>
> org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
>
> }
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>
>
> job.waitForCompletion(true);
>
> System.out.println();
>
> }
> }
>
>
>
>
>
> From: Bharati <bh...@mparallelo.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, May 22, 2013 3:39 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Eclipse plugin
>
> Hi Jing,
>
> I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
>
> For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
> How can I use eclipse to rewrite the code.
>
> The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
>
> Thanks,
> Bharati
>
> Sent from my iPad
>
> On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
>
> > Hi Bharati,
> >
> > Usually you only need to run "ant clean jar jar-test" and "ant
> > eclipse" on your code base, and then import the project into your
> > eclipse. Can you provide some more detailed description about the
> > problem you met?
> >
> > Thanks,
> > -Jing
> >
> > On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> >> Hi,
> >>
> >> I am trying to get or build eclipse plugin for 1.2.0
> >>
> >> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
> >>
> >> I need to build a hadoop map reduce project and be able to debug in eclipse.
> >>
> >> Thanks,
> >> Bharati
> >> Sent from my iPad
> >> Fortigate Filtered
> >>
> Fortigate Filtered
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Bharati Adkar <bh...@mparallelo.com>.
Hi,
I am using a mac.
I have not used maven before, I am new to hadoop and eclipse.
Any directions to start a project as map reuce as per all the videos on youtube.
Thanks,
Bharati
On May 22, 2013, at 4:23 PM, Sanjay Subramanian <Sa...@wizecommerce.com> wrote:
> Hi
>
> I don't use any need any special plugin to walk thru the code
>
> All my map reduce jobs have a
>
> JobMapper.java
> JobReducer.java
> JobProcessor.java (set any configs u like)
>
> I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
>
> Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
>
>
> MAVEN REPO INFO
> ================
> <repositories>
> <repository>
> <id>Cloudera repository</id>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
> </repository>
> </repositories>
>
> <properties>
> <cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
> </properties>
>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-mapreduce-client-core</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-common</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
> <dependency>
> <groupId>org.apache.hadoop</groupId>
> <artifactId>hadoop-client</artifactId>
> <version>${cloudera_version}</version>
> <scope>compile</scope>
> </dependency>
>
> WordCountNew (please modify as needed)
> ======================================
>
> public class WordCountNew {
>
>
>
> public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
>
> private final static IntWritable one = new IntWritable(1);
>
> private Text word = new Text();
>
>
>
> public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
> FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
> // System.out.println(value.toString());
> String fileName = fileSplit.getPath().toString();
> String line = value.toString();
> StringTokenizer tokenizer = new StringTokenizer(line);
> while (tokenizer.hasMoreTokens()) {
> word.set(tokenizer.nextToken());
> ctxt.write(word, one);
>
> }
>
> }
>
> }
>
>
>
> public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
>
> public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
>
> int sum = 0;
>
> for (IntWritable value : values) {
>
> sum += value.get();
>
> }
>
> ctxt.write(key, new IntWritable(sum));
>
> }
>
> }
>
>
>
> public static void main(String[] args) throws Exception {
> org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
> hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
> hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
>
>
> Job job = new Job(hadoopConf);
>
> job.setJobName("wordcountNEW");
>
> job.setJarByClass(WordCountNew.class);
>
> job.setOutputKeyClass(Text.class);
>
> job.setOutputValueClass(IntWritable.class);
>
> job.setMapOutputKeyClass(Text.class);
>
> job.setMapOutputValueClass(IntWritable.class);
>
>
>
> job.setMapperClass(WordCountNew.Map.class);
>
> job.setCombinerClass(WordCountNew.Reduce.class);
>
> job.setReducerClass(Reduce.class);
>
> //
> job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
>
> job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
>
>
> job.setOutputFormatClass(TextOutputFormat.class);
>
>
> if (FileUtils.doesFileOrDirectoryExist(args[1])){
>
> org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
>
> }
>
> org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
> org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
>
>
>
> job.waitForCompletion(true);
>
> System.out.println();
>
> }
> }
>
>
>
>
>
> From: Bharati <bh...@mparallelo.com>
> Reply-To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Date: Wednesday, May 22, 2013 3:39 PM
> To: "user@hadoop.apache.org" <us...@hadoop.apache.org>
> Subject: Re: Eclipse plugin
>
> Hi Jing,
>
> I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
>
> For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
> How can I use eclipse to rewrite the code.
>
> The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
>
> Thanks,
> Bharati
>
> Sent from my iPad
>
> On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
>
> > Hi Bharati,
> >
> > Usually you only need to run "ant clean jar jar-test" and "ant
> > eclipse" on your code base, and then import the project into your
> > eclipse. Can you provide some more detailed description about the
> > problem you met?
> >
> > Thanks,
> > -Jing
> >
> > On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> >> Hi,
> >>
> >> I am trying to get or build eclipse plugin for 1.2.0
> >>
> >> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
> >>
> >> I need to build a hadoop map reduce project and be able to debug in eclipse.
> >>
> >> Thanks,
> >> Bharati
> >> Sent from my iPad
> >> Fortigate Filtered
> >>
> Fortigate Filtered
>
> CONFIDENTIALITY NOTICE
> ======================
> This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Forgot to add, if u run Windows and Eclipse and want to do Hadoop u have to setup Cygwin and add $CYGWIN_PATH/bin to PATH
Good Luck
Sanjay
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 4:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Forgot to add, if u run Windows and Eclipse and want to do Hadoop u have to setup Cygwin and add $CYGWIN_PATH/bin to PATH
Good Luck
Sanjay
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 4:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Forgot to add, if u run Windows and Eclipse and want to do Hadoop u have to setup Cygwin and add $CYGWIN_PATH/bin to PATH
Good Luck
Sanjay
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 4:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Forgot to add, if u run Windows and Eclipse and want to do Hadoop u have to setup Cygwin and add $CYGWIN_PATH/bin to PATH
Good Luck
Sanjay
From: Sanjay Subramanian <sa...@wizecommerce.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 4:23 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
publicclass WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
public class WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
public class WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
public class WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Sanjay Subramanian <Sa...@wizecommerce.com>.
Hi
I don't use any need any special plugin to walk thru the code
All my map reduce jobs have a
JobMapper.java
JobReducer.java
JobProcessor.java (set any configs u like)
I create a new maven project in eclipse (easier to manage dependencies) ….the elements are in the order as they should appear in the POM
Then In Eclipse Debug Configurations I create a new JAVA application and then I start debugging ! That’s it…..
MAVEN REPO INFO
================
<repositories>
<repository>
<id>Cloudera repository</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<properties>
<cloudera_version>2.0.0-cdh4.1.2</cloudera_version>
</properties>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-mapreduce-client-core</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-common</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-client</artifactId>
<version>${cloudera_version}</version>
<scope>compile</scope>
</dependency>
WordCountNew (please modify as needed)
======================================
public class WordCountNew {
public static class Map extends org.apache.hadoop.mapreduce.Mapper<LongWritable, Text, Text, IntWritable> {
private final static IntWritable one = new IntWritable(1);
private Text word = new Text();
public void map(LongWritable key, Text value, Context ctxt) throws IOException, InterruptedException {
FileSplit fileSplit = (FileSplit)ctxt.getInputSplit();
// System.out.println(value.toString());
String fileName = fileSplit.getPath().toString();
String line = value.toString();
StringTokenizer tokenizer = new StringTokenizer(line);
while (tokenizer.hasMoreTokens()) {
word.set(tokenizer.nextToken());
ctxt.write(word, one);
}
}
}
public static class Reduce extends org.apache.hadoop.mapreduce.Reducer<Text, IntWritable, Text, IntWritable> {
public void reduce(Text key, Iterable<IntWritable> values, Context ctxt) throws IOException, InterruptedException {
int sum = 0;
for (IntWritable value : values) {
sum += value.get();
}
ctxt.write(key, new IntWritable(sum));
}
}
public static void main(String[] args) throws Exception {
org.apache.hadoop.conf.Configuration hadoopConf = new org.apache.hadoop.conf.Configuration();
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_END.getVal());
hadoopConf.set(MapredConfEnum.IMPRESSIONS_LOG_REC_CACHED_SEPARATOR.getVal(), MapredConfEnum.PRODUCT_IMPR_LOG_REC_CACHED.getVal());
hadoopConf.set("io.compression.codecs", "org.apache.hadoop.io.compress.GzipCodec");
Job job = new Job(hadoopConf);
job.setJobName("wordcountNEW");
job.setJarByClass(WordCountNew.class);
job.setOutputKeyClass(Text.class);
job.setOutputValueClass(IntWritable.class);
job.setMapOutputKeyClass(Text.class);
job.setMapOutputValueClass(IntWritable.class);
job.setMapperClass(WordCountNew.Map.class);
job.setCombinerClass(WordCountNew.Reduce.class);
job.setReducerClass(Reduce.class);
// job.setInputFormatClass(ZipMultipleLineRecordInputFormat.class);
job.setInputFormatClass(org.apache.hadoop.mapreduce.lib.input.TextInputFormat.class);
job.setOutputFormatClass(TextOutputFormat.class);
if (FileUtils.doesFileOrDirectoryExist(args[1])){
org.apache.commons.io.FileUtils.deleteDirectory(new File(args[1]));
}
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.setInputPaths(job, new Path(args[0]));
org.apache.hadoop.mapreduce.lib.output.FileOutputFormat.setOutputPath(job, new Path(args[1]));
job.waitForCompletion(true);
System.out.println();
}
}
From: Bharati <bh...@mparallelo.com>>
Reply-To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Date: Wednesday, May 22, 2013 3:39 PM
To: "user@hadoop.apache.org<ma...@hadoop.apache.org>" <us...@hadoop.apache.org>>
Subject: Re: Eclipse plugin
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com>> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com>> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Fortigate Filtered
CONFIDENTIALITY NOTICE
======================
This email message and any attachments are for the exclusive use of the intended recipient(s) and may contain confidential and privileged information. Any unauthorized review, use, disclosure or distribution is prohibited. If you are not the intended recipient, please contact the sender by reply email and destroy all copies of the original message along with any attachments, from your computer system. If you are the intended recipient, please be advised that the content of this message is subject to access, review and disclosure by the sender's Email System Administrator.
Re: Eclipse plugin
Posted by Bharati <bh...@mparallelo.com>.
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Re: Eclipse plugin
Posted by Bharati <bh...@mparallelo.com>.
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Re: Eclipse plugin
Posted by Bharati <bh...@mparallelo.com>.
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Re: Eclipse plugin
Posted by Bharati <bh...@mparallelo.com>.
Hi Jing,
I want to be able to open a project as map reduce project in eclipse instead of java project as per some of the videos on youtube.
For now let us say I want to write a wordcount program and step through it with hadoop 1.2.0
How can I use eclipse to rewrite the code.
The goal here is to setup the development env to start project as mad reduce right in eclipse or netbeans which ever works better. The idea is to be able to step through the code.
Thanks,
Bharati
Sent from my iPad
On May 22, 2013, at 2:42 PM, Jing Zhao <ji...@hortonworks.com> wrote:
> Hi Bharati,
>
> Usually you only need to run "ant clean jar jar-test" and "ant
> eclipse" on your code base, and then import the project into your
> eclipse. Can you provide some more detailed description about the
> problem you met?
>
> Thanks,
> -Jing
>
> On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
>> Hi,
>>
>> I am trying to get or build eclipse plugin for 1.2.0
>>
>> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>>
>> I need to build a hadoop map reduce project and be able to debug in eclipse.
>>
>> Thanks,
>> Bharati
>> Sent from my iPad
>> Fortigate Filtered
>>
Re: Eclipse plugin
Posted by Jing Zhao <ji...@hortonworks.com>.
Hi Bharati,
Usually you only need to run "ant clean jar jar-test" and "ant
eclipse" on your code base, and then import the project into your
eclipse. Can you provide some more detailed description about the
problem you met?
Thanks,
-Jing
On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> Hi,
>
> I am trying to get or build eclipse plugin for 1.2.0
>
> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>
> I need to build a hadoop map reduce project and be able to debug in eclipse.
>
> Thanks,
> Bharati
> Sent from my iPad
> Fortigate Filtered
>
Re: Eclipse plugin
Posted by Jing Zhao <ji...@hortonworks.com>.
Hi Bharati,
Usually you only need to run "ant clean jar jar-test" and "ant
eclipse" on your code base, and then import the project into your
eclipse. Can you provide some more detailed description about the
problem you met?
Thanks,
-Jing
On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> Hi,
>
> I am trying to get or build eclipse plugin for 1.2.0
>
> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>
> I need to build a hadoop map reduce project and be able to debug in eclipse.
>
> Thanks,
> Bharati
> Sent from my iPad
> Fortigate Filtered
>
Re: Eclipse plugin
Posted by Jing Zhao <ji...@hortonworks.com>.
Hi Bharati,
Usually you only need to run "ant clean jar jar-test" and "ant
eclipse" on your code base, and then import the project into your
eclipse. Can you provide some more detailed description about the
problem you met?
Thanks,
-Jing
On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> Hi,
>
> I am trying to get or build eclipse plugin for 1.2.0
>
> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>
> I need to build a hadoop map reduce project and be able to debug in eclipse.
>
> Thanks,
> Bharati
> Sent from my iPad
> Fortigate Filtered
>
Re: Eclipse plugin
Posted by Jing Zhao <ji...@hortonworks.com>.
Hi Bharati,
Usually you only need to run "ant clean jar jar-test" and "ant
eclipse" on your code base, and then import the project into your
eclipse. Can you provide some more detailed description about the
problem you met?
Thanks,
-Jing
On Wed, May 22, 2013 at 2:25 PM, Bharati <bh...@mparallelo.com> wrote:
> Hi,
>
> I am trying to get or build eclipse plugin for 1.2.0
>
> All the methods I found on the web did not work for me. Any tutorial, methods to build the plugin will help.
>
> I need to build a hadoop map reduce project and be able to debug in eclipse.
>
> Thanks,
> Bharati
> Sent from my iPad
> Fortigate Filtered
>