You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Christoph Böhm <li...@gmx.net> on 2011/12/23 14:11:09 UTC

Re: Packaging a Giraph application in a jar

Hi all,

first, I cannot find this node runner "bin/giraph" mentioned in GIRAPH-64 in the 0.7 trunk and, second, I'm not quite sure about the essence from that ueber-fat-jar discussion.
Is there a better/best practice way to run my implementation (vertex, io, of, etc) other then the ueber jar?

What am I missing?
Thanks.
Christoph




--- original msg ---
Gianmarco-
   GIRAPH-64 has been committed, so it should be much easier to treat
Giraph as a library and link your vertices against it.  Try it out and
see if it meets your needs.  If not, be sure to open a JIRA so we can
improve it.

Thanks,
Jakob


On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales
<gd...@apache.org> wrote:
> Nice!
> It looks cleaner than my ant+ivy hack :)
> Thanks for sharing,
> --
> Gianmarco
>
>
> On Thu, Nov 10, 2011 at 16:18, Claudio Martella <cl...@gmail.com>
> wrote:
>>
>> Ok,
>>
>> quite basically this is my pom.xml:
>>
>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>> � � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>> http://maven.apache.org/maven-v4_0_0.xsd">
>> � � � �<modelVersion>4.0.0</modelVersion>
>> � � � �<groupId>org.acaro.graffiti</groupId>
>> � � � �<artifactId>graffiti</artifactId>
>> � � � �<packaging>jar</packaging>
>> � � � �<version>1.0-SNAPSHOT</version>
>> � � � �<name>graffiti</name>
>> � � � �<url>http://maven.apache.org</url>
>> � � � �<build>
>> � � � � � � � �<plugins>
>> � � � � � � � � � � � �<plugin>
>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>
>> �<artifactId>maven-assembly-plugin</artifactId>
>> � � � � � � � � � � � � � � � �<version>2.2</version>
>> � � � � � � � � � � � � � � � �<configuration>
>>
>> �<outputDirectory>target</outputDirectory>
>> � � � � � � � � � � � � � � � � � � � �<descriptorRefs>
>>
>> �<descriptorRef>jar-with-dependencies</descriptorRef>
>> � � � � � � � � � � � � � � � � � � � �</descriptorRefs>
>> � � � � � � � � � � � � � � � �</configuration>
>> � � � � � � � � � � � � � � � �<executions>
>> � � � � � � � � � � � � � � � � � � � �<execution>
>> � � � � � � � � � � � � � � � � � � � � � � � �<id>make-assembly</id>
>> � � � � � � � � � � � � � � � � � � � � � � � �<phase>compile</phase>
>> � � � � � � � � � � � � � � � � � � � � � � � �<goals>
>> � � � � � � � � � � � � � � � � � � � � � � � � � � � �<goal>single</goal>
>> � � � � � � � � � � � � � � � � � � � � � � � �</goals>
>> � � � � � � � � � � � � � � � � � � � �</execution>
>> � � � � � � � � � � � � � � � �</executions>
>> � � � � � � � � � � � �</plugin>
>> � � � � � � � � � � � �<plugin>
>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>
>> �<artifactId>maven-eclipse-plugin</artifactId>
>> � � � � � � � � � � � � � � � �<configuration>
>>
>> �<downloadSources>true</downloadSources>
>>
>> �<downloadJavadocs>true</downloadJavadocs>
>> � � � � � � � � � � � � � � � �</configuration>
>> � � � � � � � � � � � �</plugin>
>> � � � � � � � � � � � �<plugin>
>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>> � � � � � � � � � � � � � � � �<artifactId>maven-shade-plugin</artifactId>
>> � � � � � � � � � � � � � � � �<version>1.5</version>
>> � � � � � � � � � � � � � � � �<executions>
>> � � � � � � � � � � � � � � � � � � � �<execution>
>> � � � � � � � � � � � � � � � � � � � � � � � �<phase>package</phase>
>> � � � � � � � � � � � � � � � � � � � � � � � �<goals>
>> � � � � � � � � � � � � � � � � � � � � � � � � � � � �<goal>shade</goal>
>> � � � � � � � � � � � � � � � � � � � � � � � �</goals>
>> � � � � � � � � � � � � � � � � � � � � � � � �<configuration>
>> � � � � � � � � � � � � � � � � � � � � � � � �</configuration>
>> � � � � � � � � � � � � � � � � � � � �</execution>
>> � � � � � � � � � � � � � � � �</executions>
>> � � � � � � � � � � � �</plugin>
>> � � � � � � � �</plugins>
>> � � � �</build>
>> � � � �<dependencies>
>> � � � � � � � �<dependency>
>> � � � � � � � � � � � �<groupId>junit</groupId>
>> � � � � � � � � � � � �<artifactId>junit</artifactId>
>> � � � � � � � � � � � �<version>4.8.2</version>
>> � � � � � � � � � � � �<scope>test</scope>
>> � � � � � � � �</dependency>
>> � � � � � � � �<dependency>
>> � � � � � � � � � � � �<groupId>ch.qos.logback</groupId>
>> � � � � � � � � � � � �<artifactId>logback-classic</artifactId>
>> � � � � � � � � � � � �<version>0.9.27</version>
>> � � � � � � � �</dependency>
>> � � � � � � � �<dependency>
>> � � � � � � � � � � � �<groupId>com.google.guava</groupId>
>> � � � � � � � � � � � �<artifactId>guava</artifactId>
>> � � � � � � � � � � � �<version>r09</version>
>> � � � � � � � �</dependency>
>> � � � � � � � �<dependency>
>> � � � � � � � � � � � �<groupId>org.antlr</groupId>
>> � � � � � � � � � � � �<artifactId>antlr</artifactId>
>> � � � � � � � � � � � �<version>3.3</version>
>> � � � � � � � �</dependency>
>> � � � � � � � �<dependency>
>> � � � � � � � � � � � �<groupId>org.apache.giraph</groupId>
>> � � � � � � � � � � � �<artifactId>giraph</artifactId>
>> � � � � � � � � � � � �<version>0.70</version>
>> � � � � � � � �</dependency>
>> � � � � � � � �<dependency>
>> � � � � � � � � � � � �<groupId>org.apache.hadoop</groupId>
>> � � � � � � � � � � � �<artifactId>hadoop-core</artifactId>
>> � � � � � � � � � � � �<version>0.20.204.0</version>
>> � � � � � � � � � � � �<scope>provided</scope>
>> � � � � � � � �</dependency>
>> � � � �</dependencies>
>> </project>
>>
>>
>> You set your project dependecies as usual, set hadoop-core (you don't
>> really need it in the job jar) as <provided> and then use the
>> maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
>> package goal it will upack your dependencies and re-pack them along
>> with your project classes in an uberjar.
>>
>> maven-shade allows you to set excludes in case of conflicts.
>>
>> Easy, automatic and clean.
>>
>>
>> Hope this helps,
>> Claudio
>>
>> On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching <ac...@apache.org> wrote:
>> > Would be great if you can document what you did. =)
>> >
>> > Thanks,
>> >
>> > Avery
>> >
>> > On 11/8/11 3:13 PM, Claudio Martella wrote:
>> >>
>> >> Sorry guys, may bad.
>> >>
>> >> Was calling job.waitForCompletion() directly. I've been coding
>> >> standard mapreduce whole weekend...
>> >>
>> >> Anyway I got a solution for clean packaging of your own application
>> >> over giraph, and that is exactly using maven-shade-plugin. it will
>> >> prepare the uberjar for you.
>> >>
>> >> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
>> >> <cl...@gmail.com> �wrote:
>> >>>
>> >>> Hello list,
>> >>>
>> >>> I'm actually having troubles as well to get my application running.
>> >>>
>> >>> I've give a shot to maven-shade plugin which unpacks my dependencies
>> >>> and packs them all together with my classes in a new jar.
>> >>>
>> >>> I attach the hierarchy of the jar so that somebody can maybe spot
>> >>> what's missing, because i can't get it working. I get an identity
>> >>> map-reduce job with jobconf complaining about no job jar being set.
>> >>>
>> >>> Any idea?
>> >>>
>> >>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<ac...@apache.org> �wrote:
>> >>>>
>> >>>> Hi Gianmarco,
>> >>>>
>> >>>> You're right, most of us (to my knowledge) have been using Giraph
>> >>>> with
>> >>>> an
>> >>>> uberjar as you've put it. �However, Jakob has been doing some work to
>> >>>> make
>> >>>> this easier. �See the below issue:
>> >>>>
>> >>>> https://issues.apache.org/jira/browse/GIRAPH-64
>> >>>>
>> >>>> If you can suggest a better approach, please add to the issue or
>> >>>> create
>> >>>> a
>> >>>> new one if appropriate.
>> >>>>
>> >>>> Thanks,
>> >>>>
>> >>>> Avery
>> >>>>
>> >>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
>> >>>>>
>> >>>>> Hi community,
>> >>>>>
>> >>>>> I was wondering what is the current best practice to package an
>> >>>>> application in a jar for deployment.
>> >>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib
>> >>>>> directory
>> >>>>> of
>> >>>>> my jar, and using the -libjars option but none of them worked. It
>> >>>>> looks
>> >>>>> like
>> >>>>> the backend classloader is doing some mess and it doesn't find my
>> >>>>> own
>> >>>>> classes in the jar.
>> >>>>>
>> >>>>> I resorted to uncompressing the giraph-*.jar and repackaging my
>> >>>>> classes
>> >>>>> with it, all at the same level (an uber-fat jar), but even though it
>> >>>>> works
>> >>>>> it doesn't sound like the right approach.
>> >>>>>
>> >>>>> Any suggestions?
>> >>>>>
>> >>>>> Thanks,
>> >>>>> --
>> >>>>> Gianmarco
>> >>>>>
>> >>>>>

Re: Packaging a Giraph application in a jar

Posted by Christoph Böhm <li...@gmx.net>.
Thanks Avery!
I don't like the uber-fat-jar since I work remote and doploy on the cluster via a not-super-fast-connection.
The first approach I actually tried was to "put the jar in the library path" which didn't work.
Think that I got to deal with the fat jar for a while.
Christoph


-------- Original-Nachricht --------
> Datum: Fri, 23 Dec 2011 09:19:37 -0800
> Von: Avery Ching <ac...@apache.org>
> An: giraph-user@incubator.apache.org
> Betreff: Re: Packaging a Giraph application in a jar

> Hi Christoph,
> 
> Is your svn up to date?  I see bin/giraph in my svn checkout.
> 
> I am not sure that we have totally figured out the uber-fat-jar 
> discussion.  It's still evolving.  Here's some quick possibilities you 
> can try out though.
> 
> 1)  Add your application to the Giraph code and compile the uber fat jar
> 
> 2)  mvn install your giraph jar into your local maven repo and then 
> include it as a dependency for your applications.  Then you can either 
> include the application jar (uber jar approach) or you can put the jar 
> in the library path for your hadoop cluster to reduce your application 
> jar size.
> 
> Hope that helps!
> 
> Avery
> 
> On 12/23/11 5:11 AM, "Christoph Böhm" wrote:
> > Hi all,
> >
> > first, I cannot find this node runner "bin/giraph" mentioned in
> GIRAPH-64 in the 0.7 trunk and, second, I'm not quite sure about the essence from
> that ueber-fat-jar discussion.
> > Is there a better/best practice way to run my implementation (vertex,
> io, of, etc) other then the ueber jar?
> >
> > What am I missing?
> > Thanks.
> > Christoph
> >
> >
> >
> >
> > --- original msg ---
> > Gianmarco-
> >     GIRAPH-64 has been committed, so it should be much easier to treat
> > Giraph as a library and link your vertices against it.  Try it out and
> > see if it meets your needs.  If not, be sure to open a JIRA so we can
> > improve it.
> >
> > Thanks,
> > Jakob
> >
> >
> > On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales
> > <gd...@apache.org>  wrote:
> >> Nice!
> >> It looks cleaner than my ant+ivy hack :)
> >> Thanks for sharing,
> >> --
> >> Gianmarco
> >>
> >>
> >> On Thu, Nov 10, 2011 at 16:18, Claudio
> Martella<cl...@gmail.com>
> >> wrote:
> >>> Ok,
> >>>
> >>> quite basically this is my pom.xml:
> >>>
> >>> <project xmlns="http://maven.apache.org/POM/4.0.0"
> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >>> � � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> >>> http://maven.apache.org/maven-v4_0_0.xsd">
> >>> � � � �<modelVersion>4.0.0</modelVersion>
> >>> � � � �<groupId>org.acaro.graffiti</groupId>
> >>> � � � �<artifactId>graffiti</artifactId>
> >>> � � � �<packaging>jar</packaging>
> >>> � � � �<version>1.0-SNAPSHOT</version>
> >>> � � � �<name>graffiti</name>
> >>> � � � �<url>http://maven.apache.org</url>
> >>> � � � �<build>
> >>> � � � � � � � �<plugins>
> >>> � � � � � � � � � � � �<plugin>
> >>> � � � � � � � � � � � � � � �
> �<groupId>org.apache.maven.plugins</groupId>
> >>>
> >>> �<artifactId>maven-assembly-plugin</artifactId>
> >>> � � � � � � � � � � � � � � �
> �<version>2.2</version>
> >>> � � � � � � � � � � � � � � �
> �<configuration>
> >>>
> >>> �<outputDirectory>target</outputDirectory>
> >>> � � � � � � � � � � � � � � � � �
> � � �<descriptorRefs>
> >>>
> >>> �<descriptorRef>jar-with-dependencies</descriptorRef>
> >>> � � � � � � � � � � � � � � � � �
> � � �</descriptorRefs>
> >>> � � � � � � � � � � � � � � �
> �</configuration>
> >>> � � � � � � � � � � � � � � �
> �<executions>
> >>> � � � � � � � � � � � � � � � � �
> � � �<execution>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<id>make-assembly</id>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<phase>compile</phase>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<goals>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � � � � � �<goal>single</goal>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �</goals>
> >>> � � � � � � � � � � � � � � � � �
> � � �</execution>
> >>> � � � � � � � � � � � � � � �
> �</executions>
> >>> � � � � � � � � � � � �</plugin>
> >>> � � � � � � � � � � � �<plugin>
> >>> � � � � � � � � � � � � � � �
> �<groupId>org.apache.maven.plugins</groupId>
> >>>
> >>> �<artifactId>maven-eclipse-plugin</artifactId>
> >>> � � � � � � � � � � � � � � �
> �<configuration>
> >>>
> >>> �<downloadSources>true</downloadSources>
> >>>
> >>> �<downloadJavadocs>true</downloadJavadocs>
> >>> � � � � � � � � � � � � � � �
> �</configuration>
> >>> � � � � � � � � � � � �</plugin>
> >>> � � � � � � � � � � � �<plugin>
> >>> � � � � � � � � � � � � � � �
> �<groupId>org.apache.maven.plugins</groupId>
> >>> � � � � � � � � � � � � � � �
> �<artifactId>maven-shade-plugin</artifactId>
> >>> � � � � � � � � � � � � � � �
> �<version>1.5</version>
> >>> � � � � � � � � � � � � � � �
> �<executions>
> >>> � � � � � � � � � � � � � � � � �
> � � �<execution>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<phase>package</phase>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<goals>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � � � � � �<goal>shade</goal>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �</goals>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<configuration>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �</configuration>
> >>> � � � � � � � � � � � � � � � � �
> � � �</execution>
> >>> � � � � � � � � � � � � � � �
> �</executions>
> >>> � � � � � � � � � � � �</plugin>
> >>> � � � � � � � �</plugins>
> >>> � � � �</build>
> >>> � � � �<dependencies>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>junit</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>junit</artifactId>
> >>> � � � � � � � � � � �
> �<version>4.8.2</version>
> >>> � � � � � � � � � � � �<scope>test</scope>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>ch.qos.logback</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>logback-classic</artifactId>
> >>> � � � � � � � � � � �
> �<version>0.9.27</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>com.google.guava</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>guava</artifactId>
> >>> � � � � � � � � � � � �<version>r09</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>org.antlr</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>antlr</artifactId>
> >>> � � � � � � � � � � � �<version>3.3</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>org.apache.giraph</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>giraph</artifactId>
> >>> � � � � � � � � � � � �<version>0.70</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>org.apache.hadoop</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>hadoop-core</artifactId>
> >>> � � � � � � � � � � �
> �<version>0.20.204.0</version>
> >>> � � � � � � � � � � � �<scope>provided</scope>
> >>> � � � � � � � �</dependency>
> >>> � � � �</dependencies>
> >>> </project>
> >>>
> >>>
> >>> You set your project dependecies as usual, set hadoop-core (you don't
> >>> really need it in the job jar) as<provided>  and then use the
> >>> maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
> >>> package goal it will upack your dependencies and re-pack them along
> >>> with your project classes in an uberjar.
> >>>
> >>> maven-shade allows you to set excludes in case of conflicts.
> >>>
> >>> Easy, automatic and clean.
> >>>
> >>>
> >>> Hope this helps,
> >>> Claudio
> >>>
> >>> On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching<ac...@apache.org> 
> wrote:
> >>>> Would be great if you can document what you did. =)
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Avery
> >>>>
> >>>> On 11/8/11 3:13 PM, Claudio Martella wrote:
> >>>>> Sorry guys, may bad.
> >>>>>
> >>>>> Was calling job.waitForCompletion() directly. I've been coding
> >>>>> standard mapreduce whole weekend...
> >>>>>
> >>>>> Anyway I got a solution for clean packaging of your own application
> >>>>> over giraph, and that is exactly using maven-shade-plugin. it will
> >>>>> prepare the uberjar for you.
> >>>>>
> >>>>> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
> >>>>> <cl...@gmail.com>  �wrote:
> >>>>>> Hello list,
> >>>>>>
> >>>>>> I'm actually having troubles as well to get my application running.
> >>>>>>
> >>>>>> I've give a shot to maven-shade plugin which unpacks my
> dependencies
> >>>>>> and packs them all together with my classes in a new jar.
> >>>>>>
> >>>>>> I attach the hierarchy of the jar so that somebody can maybe spot
> >>>>>> what's missing, because i can't get it working. I get an identity
> >>>>>> map-reduce job with jobconf complaining about no job jar being set.
> >>>>>>
> >>>>>> Any idea?
> >>>>>>
> >>>>>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<ac...@apache.org> 
> �wrote:
> >>>>>>> Hi Gianmarco,
> >>>>>>>
> >>>>>>> You're right, most of us (to my knowledge) have been using Giraph
> >>>>>>> with
> >>>>>>> an
> >>>>>>> uberjar as you've put it. �However, Jakob has been doing some
> work to
> >>>>>>> make
> >>>>>>> this easier. �See the below issue:
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/GIRAPH-64
> >>>>>>>
> >>>>>>> If you can suggest a better approach, please add to the issue or
> >>>>>>> create
> >>>>>>> a
> >>>>>>> new one if appropriate.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Avery
> >>>>>>>
> >>>>>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
> >>>>>>>> Hi community,
> >>>>>>>>
> >>>>>>>> I was wondering what is the current best practice to package an
> >>>>>>>> application in a jar for deployment.
> >>>>>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib
> >>>>>>>> directory
> >>>>>>>> of
> >>>>>>>> my jar, and using the -libjars option but none of them worked. It
> >>>>>>>> looks
> >>>>>>>> like
> >>>>>>>> the backend classloader is doing some mess and it doesn't find my
> >>>>>>>> own
> >>>>>>>> classes in the jar.
> >>>>>>>>
> >>>>>>>> I resorted to uncompressing the giraph-*.jar and repackaging my
> >>>>>>>> classes
> >>>>>>>> with it, all at the same level (an uber-fat jar), but even though
> it
> >>>>>>>> works
> >>>>>>>> it doesn't sound like the right approach.
> >>>>>>>>
> >>>>>>>> Any suggestions?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> --
> >>>>>>>> Gianmarco
> >>>>>>>>
> >>>>>>>>
> 

Re: Packaging a Giraph application in a jar

Posted by Avery Ching <ac...@apache.org>.
Hi Christoph,

Is your svn up to date?  I see bin/giraph in my svn checkout.

I am not sure that we have totally figured out the uber-fat-jar 
discussion.  It's still evolving.  Here's some quick possibilities you 
can try out though.

1)  Add your application to the Giraph code and compile the uber fat jar

2)  mvn install your giraph jar into your local maven repo and then 
include it as a dependency for your applications.  Then you can either 
include the application jar (uber jar approach) or you can put the jar 
in the library path for your hadoop cluster to reduce your application 
jar size.

Hope that helps!

Avery

On 12/23/11 5:11 AM, "Christoph Böhm" wrote:
> Hi all,
>
> first, I cannot find this node runner "bin/giraph" mentioned in GIRAPH-64 in the 0.7 trunk and, second, I'm not quite sure about the essence from that ueber-fat-jar discussion.
> Is there a better/best practice way to run my implementation (vertex, io, of, etc) other then the ueber jar?
>
> What am I missing?
> Thanks.
> Christoph
>
>
>
>
> --- original msg ---
> Gianmarco-
>     GIRAPH-64 has been committed, so it should be much easier to treat
> Giraph as a library and link your vertices against it.  Try it out and
> see if it meets your needs.  If not, be sure to open a JIRA so we can
> improve it.
>
> Thanks,
> Jakob
>
>
> On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales
> <gd...@apache.org>  wrote:
>> Nice!
>> It looks cleaner than my ant+ivy hack :)
>> Thanks for sharing,
>> --
>> Gianmarco
>>
>>
>> On Thu, Nov 10, 2011 at 16:18, Claudio Martella<cl...@gmail.com>
>> wrote:
>>> Ok,
>>>
>>> quite basically this is my pom.xml:
>>>
>>> <project xmlns="http://maven.apache.org/POM/4.0.0"
>>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
>>> � � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
>>> http://maven.apache.org/maven-v4_0_0.xsd">
>>> � � � �<modelVersion>4.0.0</modelVersion>
>>> � � � �<groupId>org.acaro.graffiti</groupId>
>>> � � � �<artifactId>graffiti</artifactId>
>>> � � � �<packaging>jar</packaging>
>>> � � � �<version>1.0-SNAPSHOT</version>
>>> � � � �<name>graffiti</name>
>>> � � � �<url>http://maven.apache.org</url>
>>> � � � �<build>
>>> � � � � � � � �<plugins>
>>> � � � � � � � � � � � �<plugin>
>>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>>
>>> �<artifactId>maven-assembly-plugin</artifactId>
>>> � � � � � � � � � � � � � � � �<version>2.2</version>
>>> � � � � � � � � � � � � � � � �<configuration>
>>>
>>> �<outputDirectory>target</outputDirectory>
>>> � � � � � � � � � � � � � � � � � � � �<descriptorRefs>
>>>
>>> �<descriptorRef>jar-with-dependencies</descriptorRef>
>>> � � � � � � � � � � � � � � � � � � � �</descriptorRefs>
>>> � � � � � � � � � � � � � � � �</configuration>
>>> � � � � � � � � � � � � � � � �<executions>
>>> � � � � � � � � � � � � � � � � � � � �<execution>
>>> � � � � � � � � � � � � � � � � � � � � � � � �<id>make-assembly</id>
>>> � � � � � � � � � � � � � � � � � � � � � � � �<phase>compile</phase>
>>> � � � � � � � � � � � � � � � � � � � � � � � �<goals>
>>> � � � � � � � � � � � � � � � � � � � � � � � � � � � �<goal>single</goal>
>>> � � � � � � � � � � � � � � � � � � � � � � � �</goals>
>>> � � � � � � � � � � � � � � � � � � � �</execution>
>>> � � � � � � � � � � � � � � � �</executions>
>>> � � � � � � � � � � � �</plugin>
>>> � � � � � � � � � � � �<plugin>
>>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>>
>>> �<artifactId>maven-eclipse-plugin</artifactId>
>>> � � � � � � � � � � � � � � � �<configuration>
>>>
>>> �<downloadSources>true</downloadSources>
>>>
>>> �<downloadJavadocs>true</downloadJavadocs>
>>> � � � � � � � � � � � � � � � �</configuration>
>>> � � � � � � � � � � � �</plugin>
>>> � � � � � � � � � � � �<plugin>
>>> � � � � � � � � � � � � � � � �<groupId>org.apache.maven.plugins</groupId>
>>> � � � � � � � � � � � � � � � �<artifactId>maven-shade-plugin</artifactId>
>>> � � � � � � � � � � � � � � � �<version>1.5</version>
>>> � � � � � � � � � � � � � � � �<executions>
>>> � � � � � � � � � � � � � � � � � � � �<execution>
>>> � � � � � � � � � � � � � � � � � � � � � � � �<phase>package</phase>
>>> � � � � � � � � � � � � � � � � � � � � � � � �<goals>
>>> � � � � � � � � � � � � � � � � � � � � � � � � � � � �<goal>shade</goal>
>>> � � � � � � � � � � � � � � � � � � � � � � � �</goals>
>>> � � � � � � � � � � � � � � � � � � � � � � � �<configuration>
>>> � � � � � � � � � � � � � � � � � � � � � � � �</configuration>
>>> � � � � � � � � � � � � � � � � � � � �</execution>
>>> � � � � � � � � � � � � � � � �</executions>
>>> � � � � � � � � � � � �</plugin>
>>> � � � � � � � �</plugins>
>>> � � � �</build>
>>> � � � �<dependencies>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>junit</groupId>
>>> � � � � � � � � � � � �<artifactId>junit</artifactId>
>>> � � � � � � � � � � � �<version>4.8.2</version>
>>> � � � � � � � � � � � �<scope>test</scope>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>ch.qos.logback</groupId>
>>> � � � � � � � � � � � �<artifactId>logback-classic</artifactId>
>>> � � � � � � � � � � � �<version>0.9.27</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>com.google.guava</groupId>
>>> � � � � � � � � � � � �<artifactId>guava</artifactId>
>>> � � � � � � � � � � � �<version>r09</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>org.antlr</groupId>
>>> � � � � � � � � � � � �<artifactId>antlr</artifactId>
>>> � � � � � � � � � � � �<version>3.3</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>org.apache.giraph</groupId>
>>> � � � � � � � � � � � �<artifactId>giraph</artifactId>
>>> � � � � � � � � � � � �<version>0.70</version>
>>> � � � � � � � �</dependency>
>>> � � � � � � � �<dependency>
>>> � � � � � � � � � � � �<groupId>org.apache.hadoop</groupId>
>>> � � � � � � � � � � � �<artifactId>hadoop-core</artifactId>
>>> � � � � � � � � � � � �<version>0.20.204.0</version>
>>> � � � � � � � � � � � �<scope>provided</scope>
>>> � � � � � � � �</dependency>
>>> � � � �</dependencies>
>>> </project>
>>>
>>>
>>> You set your project dependecies as usual, set hadoop-core (you don't
>>> really need it in the job jar) as<provided>  and then use the
>>> maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
>>> package goal it will upack your dependencies and re-pack them along
>>> with your project classes in an uberjar.
>>>
>>> maven-shade allows you to set excludes in case of conflicts.
>>>
>>> Easy, automatic and clean.
>>>
>>>
>>> Hope this helps,
>>> Claudio
>>>
>>> On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching<ac...@apache.org>  wrote:
>>>> Would be great if you can document what you did. =)
>>>>
>>>> Thanks,
>>>>
>>>> Avery
>>>>
>>>> On 11/8/11 3:13 PM, Claudio Martella wrote:
>>>>> Sorry guys, may bad.
>>>>>
>>>>> Was calling job.waitForCompletion() directly. I've been coding
>>>>> standard mapreduce whole weekend...
>>>>>
>>>>> Anyway I got a solution for clean packaging of your own application
>>>>> over giraph, and that is exactly using maven-shade-plugin. it will
>>>>> prepare the uberjar for you.
>>>>>
>>>>> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
>>>>> <cl...@gmail.com>  �wrote:
>>>>>> Hello list,
>>>>>>
>>>>>> I'm actually having troubles as well to get my application running.
>>>>>>
>>>>>> I've give a shot to maven-shade plugin which unpacks my dependencies
>>>>>> and packs them all together with my classes in a new jar.
>>>>>>
>>>>>> I attach the hierarchy of the jar so that somebody can maybe spot
>>>>>> what's missing, because i can't get it working. I get an identity
>>>>>> map-reduce job with jobconf complaining about no job jar being set.
>>>>>>
>>>>>> Any idea?
>>>>>>
>>>>>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<ac...@apache.org>  �wrote:
>>>>>>> Hi Gianmarco,
>>>>>>>
>>>>>>> You're right, most of us (to my knowledge) have been using Giraph
>>>>>>> with
>>>>>>> an
>>>>>>> uberjar as you've put it. �However, Jakob has been doing some work to
>>>>>>> make
>>>>>>> this easier. �See the below issue:
>>>>>>>
>>>>>>> https://issues.apache.org/jira/browse/GIRAPH-64
>>>>>>>
>>>>>>> If you can suggest a better approach, please add to the issue or
>>>>>>> create
>>>>>>> a
>>>>>>> new one if appropriate.
>>>>>>>
>>>>>>> Thanks,
>>>>>>>
>>>>>>> Avery
>>>>>>>
>>>>>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
>>>>>>>> Hi community,
>>>>>>>>
>>>>>>>> I was wondering what is the current best practice to package an
>>>>>>>> application in a jar for deployment.
>>>>>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib
>>>>>>>> directory
>>>>>>>> of
>>>>>>>> my jar, and using the -libjars option but none of them worked. It
>>>>>>>> looks
>>>>>>>> like
>>>>>>>> the backend classloader is doing some mess and it doesn't find my
>>>>>>>> own
>>>>>>>> classes in the jar.
>>>>>>>>
>>>>>>>> I resorted to uncompressing the giraph-*.jar and repackaging my
>>>>>>>> classes
>>>>>>>> with it, all at the same level (an uber-fat jar), but even though it
>>>>>>>> works
>>>>>>>> it doesn't sound like the right approach.
>>>>>>>>
>>>>>>>> Any suggestions?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> --
>>>>>>>> Gianmarco
>>>>>>>>
>>>>>>>>