You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Mohit Singh <mo...@gmail.com> on 2014/08/22 02:08:01 UTC
Running giraph custom code on CDH5.1.0
Hi,
So, I have Hadoop 2.3.0-cdh5.1.0 installed on my cluster... and I
compiled giraph thru the following command.. mvn clean package -DskipTests
-Dhadoop=non_secure -Phadoop_2.0.0
So, the compilation worked just fine and then the pagerank benchmark also
succeeded..
So, I am guessing giraph compiled just fine..
Now, I am trying to run my code..
hadoop jar graphAnalytics-0.0.1-SNAPSHOT-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
-Dgiraph.zkList=srv-110-07:2181,srv-110-08:2181,srv-210-08:2181
org.examples.graphAnalytics.PageRankVertex -w 1 -vif
org.examples.graphAnalytics.PageRankInputFormat
And I am getting this error:
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at....
I know, its an incomplete command but its already complaining.. :-/
My POM file looks like :
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.rdio.graphanalytics</groupId>
<artifactId>graphAnalytics</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>Graph Analytics Algorithms</name>
<description>Package for Graph Analytics Algorithms
</description>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.giraph</groupId>
<artifactId>giraph-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit-dep</artifactId>
<version>4.8.2</version>
</dependency>
<dependency>
* <groupId>org.apache.hadoop</groupId>*
* <artifactId>hadoop-client</artifactId>*
* <version>2.3.0-cdh5.1.0</version>*
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
How do i resolve this?
Thanks
--
Mohit
"When you want success as badly as you want the air, then you will get it.
There is no other secret of success."
-Socrates
Fwd: Running giraph custom code on CDH5.1.0
Posted by Mohit Singh <mo...@gmail.com>.
---------- Forwarded message ----------
From: Mohit Singh <mo...@gmail.com>
Date: Thu, Aug 21, 2014 at 5:08 PM
Subject: Running giraph custom code on CDH5.1.0
To: user <us...@giraph.apache.org>
Hi,
So, I have Hadoop 2.3.0-cdh5.1.0 installed on my cluster... and I
compiled giraph thru the following command.. mvn clean package -DskipTests
-Dhadoop=non_secure -Phadoop_2.0.0
So, the compilation worked just fine and then the pagerank benchmark also
succeeded..
So, I am guessing giraph compiled just fine..
Now, I am trying to run my code..
hadoop jar graphAnalytics-0.0.1-SNAPSHOT-jar-with-dependencies.jar
org.apache.giraph.GiraphRunner
-Dgiraph.zkList=srv-110-07:2181,srv-110-08:2181,srv-210-08:2181
org.examples.graphAnalytics.PageRankVertex -w 1 -vif
org.examples.graphAnalytics.PageRankInputFormat
And I am getting this error:
Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
interface org.apache.hadoop.mapreduce.JobContext, but class was expected
at
org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
at
org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at....
I know, its an incomplete command but its already complaining.. :-/
My POM file looks like :
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<modelVersion>4.0.0</modelVersion>
<groupId>org.rdio.graphanalytics</groupId>
<artifactId>graphAnalytics</artifactId>
<version>0.0.1-SNAPSHOT</version>
<name>Graph Analytics Algorithms</name>
<description>Package for Graph Analytics Algorithms
</description>
<repositories>
<repository>
<id>cloudera</id>
<url>https://repository.cloudera.com/artifactory/cloudera-repos/</url>
</repository>
</repositories>
<dependencies>
<dependency>
<groupId>org.apache.giraph</groupId>
<artifactId>giraph-core</artifactId>
<version>1.0.0</version>
</dependency>
<dependency>
<groupId>junit</groupId>
<artifactId>junit-dep</artifactId>
<version>4.8.2</version>
</dependency>
<dependency>
* <groupId>org.apache.hadoop</groupId>*
* <artifactId>hadoop-client</artifactId>*
* <version>2.3.0-cdh5.1.0</version>*
</dependency>
</dependencies>
<build>
<plugins>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-compiler-plugin</artifactId>
<version>2.1</version>
<configuration>
<source>1.6</source>
<target>1.6</target>
</configuration>
</plugin>
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-assembly-plugin</artifactId>
<version>2.4</version>
<configuration>
<descriptorRefs>
<descriptorRef>jar-with-dependencies</descriptorRef>
</descriptorRefs>
</configuration>
<executions>
<execution>
<id>assemble-all</id>
<phase>package</phase>
<goals>
<goal>single</goal>
</goals>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
How do i resolve this?
Thanks
--
Mohit
"When you want success as badly as you want the air, then you will get it.
There is no other secret of success."
-Socrates
--
Mohit
"When you want success as badly as you want the air, then you will get it.
There is no other secret of success."
-Socrates
Re: Running giraph custom code on CDH5.1.0
Posted by Eli Reisman <ap...@gmail.com>.
The interface/class change there means you are running a Hadoop cluster
that doesn't match your Giraph build. You might take a look at the Giraph
POM and see if there's a Maven build profile thats better suited to your
your Hadoop version. On CDH5 Hadoop cluster, the hadoop_yarn profile is
likely to be your only option.
Good luck,
Eli
On Thu, Aug 21, 2014 at 5:08 PM, Mohit Singh <mo...@gmail.com> wrote:
> Hi,
> So, I have Hadoop 2.3.0-cdh5.1.0 installed on my cluster... and I
> compiled giraph thru the following command.. mvn clean package -DskipTests
> -Dhadoop=non_secure -Phadoop_2.0.0
> So, the compilation worked just fine and then the pagerank benchmark also
> succeeded..
> So, I am guessing giraph compiled just fine..
>
>
> Now, I am trying to run my code..
> hadoop jar graphAnalytics-0.0.1-SNAPSHOT-jar-with-dependencies.jar
> org.apache.giraph.GiraphRunner
> -Dgiraph.zkList=srv-110-07:2181,srv-110-08:2181,srv-210-08:2181
> org.examples.graphAnalytics.PageRankVertex -w 1 -vif
> org.examples.graphAnalytics.PageRankInputFormat
>
> And I am getting this error:
> Exception in thread "main" java.lang.IncompatibleClassChangeError: Found
> interface org.apache.hadoop.mapreduce.JobContext, but class was expected
> at
> org.apache.giraph.bsp.BspOutputFormat.checkOutputSpecs(BspOutputFormat.java:43)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.checkSpecs(JobSubmitter.java:458)
> at
> org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:343)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1295)
> at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1292)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at....
>
> I know, its an incomplete command but its already complaining.. :-/
>
> My POM file looks like :
>
> <project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="
> http://www.w3.org/2001/XMLSchema-instance" xsi:schemaLocation="
> http://maven.apache.org/POM/4.0.0
> http://maven.apache.org/xsd/maven-4.0.0.xsd">
>
> <modelVersion>4.0.0</modelVersion>
>
> <groupId>org.rdio.graphanalytics</groupId>
>
> <artifactId>graphAnalytics</artifactId>
>
> <version>0.0.1-SNAPSHOT</version>
>
> <name>Graph Analytics Algorithms</name>
>
> <description>Package for Graph Analytics Algorithms
>
> </description>
>
> <repositories>
>
> <repository>
>
> <id>cloudera</id>
>
> <url>https://repository.cloudera.com/artifactory/cloudera-repos/</
> url>
>
> </repository>
>
> </repositories>
>
> <dependencies>
>
> <dependency>
>
> <groupId>org.apache.giraph</groupId>
>
> <artifactId>giraph-core</artifactId>
>
> <version>1.0.0</version>
>
> </dependency>
>
> <dependency>
>
> <groupId>junit</groupId>
>
> <artifactId>junit-dep</artifactId>
>
> <version>4.8.2</version>
>
> </dependency>
>
> <dependency>
>
> * <groupId>org.apache.hadoop</groupId>*
>
> * <artifactId>hadoop-client</artifactId>*
>
> * <version>2.3.0-cdh5.1.0</version>*
>
> </dependency>
>
> </dependencies>
>
> <build>
>
> <plugins>
>
> <plugin>
>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-compiler-plugin</artifactId>
>
> <version>2.1</version>
>
> <configuration>
>
> <source>1.6</source>
>
> <target>1.6</target>
>
> </configuration>
>
> </plugin>
>
> <plugin>
>
> <groupId>org.apache.maven.plugins</groupId>
>
> <artifactId>maven-assembly-plugin</artifactId>
>
> <version>2.4</version>
>
> <configuration>
>
> <descriptorRefs>
>
> <descriptorRef>jar-with-dependencies</descriptorRef>
>
> </descriptorRefs>
>
>
> </configuration>
>
> <executions>
>
> <execution>
>
> <id>assemble-all</id>
>
> <phase>package</phase>
>
> <goals>
>
> <goal>single</goal>
>
> </goals>
>
> </execution>
>
> </executions>
>
> </plugin>
>
> </plugins>
>
> </build>
>
> </project>
>
>
> How do i resolve this?
> Thanks
>
> --
> Mohit
>
> "When you want success as badly as you want the air, then you will get it.
> There is no other secret of success."
> -Socrates
>