You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@giraph.apache.org by Christoph Böhm <li...@gmx.net> on 2012/01/03 13:00:44 UTC

Re: Packaging a Giraph application in a jar

Thanks Avery!
I don't like the uber-fat-jar since I work remote and doploy on the cluster via a not-super-fast-connection.
The first approach I actually tried was to "put the jar in the library path" which didn't work.
Think that I got to deal with the fat jar for a while.
Christoph


-------- Original-Nachricht --------
> Datum: Fri, 23 Dec 2011 09:19:37 -0800
> Von: Avery Ching <ac...@apache.org>
> An: giraph-user@incubator.apache.org
> Betreff: Re: Packaging a Giraph application in a jar

> Hi Christoph,
> 
> Is your svn up to date?  I see bin/giraph in my svn checkout.
> 
> I am not sure that we have totally figured out the uber-fat-jar 
> discussion.  It's still evolving.  Here's some quick possibilities you 
> can try out though.
> 
> 1)  Add your application to the Giraph code and compile the uber fat jar
> 
> 2)  mvn install your giraph jar into your local maven repo and then 
> include it as a dependency for your applications.  Then you can either 
> include the application jar (uber jar approach) or you can put the jar 
> in the library path for your hadoop cluster to reduce your application 
> jar size.
> 
> Hope that helps!
> 
> Avery
> 
> On 12/23/11 5:11 AM, "Christoph Böhm" wrote:
> > Hi all,
> >
> > first, I cannot find this node runner "bin/giraph" mentioned in
> GIRAPH-64 in the 0.7 trunk and, second, I'm not quite sure about the essence from
> that ueber-fat-jar discussion.
> > Is there a better/best practice way to run my implementation (vertex,
> io, of, etc) other then the ueber jar?
> >
> > What am I missing?
> > Thanks.
> > Christoph
> >
> >
> >
> >
> > --- original msg ---
> > Gianmarco-
> >     GIRAPH-64 has been committed, so it should be much easier to treat
> > Giraph as a library and link your vertices against it.  Try it out and
> > see if it meets your needs.  If not, be sure to open a JIRA so we can
> > improve it.
> >
> > Thanks,
> > Jakob
> >
> >
> > On Thu, Nov 10, 2011 at 7:31 AM, Gianmarco De Francisci Morales
> > <gd...@apache.org>  wrote:
> >> Nice!
> >> It looks cleaner than my ant+ivy hack :)
> >> Thanks for sharing,
> >> --
> >> Gianmarco
> >>
> >>
> >> On Thu, Nov 10, 2011 at 16:18, Claudio
> Martella<cl...@gmail.com>
> >> wrote:
> >>> Ok,
> >>>
> >>> quite basically this is my pom.xml:
> >>>
> >>> <project xmlns="http://maven.apache.org/POM/4.0.0"
> >>> xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
> >>> � � � �xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
> >>> http://maven.apache.org/maven-v4_0_0.xsd">
> >>> � � � �<modelVersion>4.0.0</modelVersion>
> >>> � � � �<groupId>org.acaro.graffiti</groupId>
> >>> � � � �<artifactId>graffiti</artifactId>
> >>> � � � �<packaging>jar</packaging>
> >>> � � � �<version>1.0-SNAPSHOT</version>
> >>> � � � �<name>graffiti</name>
> >>> � � � �<url>http://maven.apache.org</url>
> >>> � � � �<build>
> >>> � � � � � � � �<plugins>
> >>> � � � � � � � � � � � �<plugin>
> >>> � � � � � � � � � � � � � � �
> �<groupId>org.apache.maven.plugins</groupId>
> >>>
> >>> �<artifactId>maven-assembly-plugin</artifactId>
> >>> � � � � � � � � � � � � � � �
> �<version>2.2</version>
> >>> � � � � � � � � � � � � � � �
> �<configuration>
> >>>
> >>> �<outputDirectory>target</outputDirectory>
> >>> � � � � � � � � � � � � � � � � �
> � � �<descriptorRefs>
> >>>
> >>> �<descriptorRef>jar-with-dependencies</descriptorRef>
> >>> � � � � � � � � � � � � � � � � �
> � � �</descriptorRefs>
> >>> � � � � � � � � � � � � � � �
> �</configuration>
> >>> � � � � � � � � � � � � � � �
> �<executions>
> >>> � � � � � � � � � � � � � � � � �
> � � �<execution>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<id>make-assembly</id>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<phase>compile</phase>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<goals>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � � � � � �<goal>single</goal>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �</goals>
> >>> � � � � � � � � � � � � � � � � �
> � � �</execution>
> >>> � � � � � � � � � � � � � � �
> �</executions>
> >>> � � � � � � � � � � � �</plugin>
> >>> � � � � � � � � � � � �<plugin>
> >>> � � � � � � � � � � � � � � �
> �<groupId>org.apache.maven.plugins</groupId>
> >>>
> >>> �<artifactId>maven-eclipse-plugin</artifactId>
> >>> � � � � � � � � � � � � � � �
> �<configuration>
> >>>
> >>> �<downloadSources>true</downloadSources>
> >>>
> >>> �<downloadJavadocs>true</downloadJavadocs>
> >>> � � � � � � � � � � � � � � �
> �</configuration>
> >>> � � � � � � � � � � � �</plugin>
> >>> � � � � � � � � � � � �<plugin>
> >>> � � � � � � � � � � � � � � �
> �<groupId>org.apache.maven.plugins</groupId>
> >>> � � � � � � � � � � � � � � �
> �<artifactId>maven-shade-plugin</artifactId>
> >>> � � � � � � � � � � � � � � �
> �<version>1.5</version>
> >>> � � � � � � � � � � � � � � �
> �<executions>
> >>> � � � � � � � � � � � � � � � � �
> � � �<execution>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<phase>package</phase>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<goals>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � � � � � �<goal>shade</goal>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �</goals>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �<configuration>
> >>> � � � � � � � � � � � � � � � � �
> � � � � � � �</configuration>
> >>> � � � � � � � � � � � � � � � � �
> � � �</execution>
> >>> � � � � � � � � � � � � � � �
> �</executions>
> >>> � � � � � � � � � � � �</plugin>
> >>> � � � � � � � �</plugins>
> >>> � � � �</build>
> >>> � � � �<dependencies>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>junit</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>junit</artifactId>
> >>> � � � � � � � � � � �
> �<version>4.8.2</version>
> >>> � � � � � � � � � � � �<scope>test</scope>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>ch.qos.logback</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>logback-classic</artifactId>
> >>> � � � � � � � � � � �
> �<version>0.9.27</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>com.google.guava</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>guava</artifactId>
> >>> � � � � � � � � � � � �<version>r09</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>org.antlr</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>antlr</artifactId>
> >>> � � � � � � � � � � � �<version>3.3</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>org.apache.giraph</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>giraph</artifactId>
> >>> � � � � � � � � � � � �<version>0.70</version>
> >>> � � � � � � � �</dependency>
> >>> � � � � � � � �<dependency>
> >>> � � � � � � � � � � �
> �<groupId>org.apache.hadoop</groupId>
> >>> � � � � � � � � � � �
> �<artifactId>hadoop-core</artifactId>
> >>> � � � � � � � � � � �
> �<version>0.20.204.0</version>
> >>> � � � � � � � � � � � �<scope>provided</scope>
> >>> � � � � � � � �</dependency>
> >>> � � � �</dependencies>
> >>> </project>
> >>>
> >>>
> >>> You set your project dependecies as usual, set hadoop-core (you don't
> >>> really need it in the job jar) as<provided>  and then use the
> >>> maven-shade (http://maven.apache.org/plugins/maven-shade-plugin/). At
> >>> package goal it will upack your dependencies and re-pack them along
> >>> with your project classes in an uberjar.
> >>>
> >>> maven-shade allows you to set excludes in case of conflicts.
> >>>
> >>> Easy, automatic and clean.
> >>>
> >>>
> >>> Hope this helps,
> >>> Claudio
> >>>
> >>> On Wed, Nov 9, 2011 at 12:19 PM, Avery Ching<ac...@apache.org> 
> wrote:
> >>>> Would be great if you can document what you did. =)
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Avery
> >>>>
> >>>> On 11/8/11 3:13 PM, Claudio Martella wrote:
> >>>>> Sorry guys, may bad.
> >>>>>
> >>>>> Was calling job.waitForCompletion() directly. I've been coding
> >>>>> standard mapreduce whole weekend...
> >>>>>
> >>>>> Anyway I got a solution for clean packaging of your own application
> >>>>> over giraph, and that is exactly using maven-shade-plugin. it will
> >>>>> prepare the uberjar for you.
> >>>>>
> >>>>> On Tue, Nov 8, 2011 at 9:33 PM, Claudio Martella
> >>>>> <cl...@gmail.com>  �wrote:
> >>>>>> Hello list,
> >>>>>>
> >>>>>> I'm actually having troubles as well to get my application running.
> >>>>>>
> >>>>>> I've give a shot to maven-shade plugin which unpacks my
> dependencies
> >>>>>> and packs them all together with my classes in a new jar.
> >>>>>>
> >>>>>> I attach the hierarchy of the jar so that somebody can maybe spot
> >>>>>> what's missing, because i can't get it working. I get an identity
> >>>>>> map-reduce job with jobconf complaining about no job jar being set.
> >>>>>>
> >>>>>> Any idea?
> >>>>>>
> >>>>>> On Sat, Nov 5, 2011 at 5:09 PM, Avery Ching<ac...@apache.org> 
> �wrote:
> >>>>>>> Hi Gianmarco,
> >>>>>>>
> >>>>>>> You're right, most of us (to my knowledge) have been using Giraph
> >>>>>>> with
> >>>>>>> an
> >>>>>>> uberjar as you've put it. �However, Jakob has been doing some
> work to
> >>>>>>> make
> >>>>>>> this easier. �See the below issue:
> >>>>>>>
> >>>>>>> https://issues.apache.org/jira/browse/GIRAPH-64
> >>>>>>>
> >>>>>>> If you can suggest a better approach, please add to the issue or
> >>>>>>> create
> >>>>>>> a
> >>>>>>> new one if appropriate.
> >>>>>>>
> >>>>>>> Thanks,
> >>>>>>>
> >>>>>>> Avery
> >>>>>>>
> >>>>>>> On 11/5/11 4:11 AM, Gianmarco De Francisci Morales wrote:
> >>>>>>>> Hi community,
> >>>>>>>>
> >>>>>>>> I was wondering what is the current best practice to package an
> >>>>>>>> application in a jar for deployment.
> >>>>>>>> I tried the 'hadoop way' by putting giraph-*.jar in the /lib
> >>>>>>>> directory
> >>>>>>>> of
> >>>>>>>> my jar, and using the -libjars option but none of them worked. It
> >>>>>>>> looks
> >>>>>>>> like
> >>>>>>>> the backend classloader is doing some mess and it doesn't find my
> >>>>>>>> own
> >>>>>>>> classes in the jar.
> >>>>>>>>
> >>>>>>>> I resorted to uncompressing the giraph-*.jar and repackaging my
> >>>>>>>> classes
> >>>>>>>> with it, all at the same level (an uber-fat jar), but even though
> it
> >>>>>>>> works
> >>>>>>>> it doesn't sound like the right approach.
> >>>>>>>>
> >>>>>>>> Any suggestions?
> >>>>>>>>
> >>>>>>>> Thanks,
> >>>>>>>> --
> >>>>>>>> Gianmarco
> >>>>>>>>
> >>>>>>>>
>