You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@twill.apache.org by "Peterson, Michael" <mi...@truvenhealth.com> on 2014/08/08 15:59:22 UTC
Newbie: how to deploy/launch
Hello,
Just joined the list yesterday. I have read over the documentation I could find and downloaded and built the code. I have a few questions:
1. How does one deploy and launch a Twill-based app? I am familiar with Slider, so is it something like that, where you have to put your jars into HDFS launch the jar with the YarnClient and point it to the jar you put in HDFS? Any config or properties files required?
2. I see that the EchoMain example you provide uses OSGi, but it looks like the other example "yarn" does not and uses a shaded jar instead. Does that mean that running Twill with OSGi is optional or does it require OSGi?
BTW - the code builds on Window but at least one of the tests fails.
Thank you,
-Michael
Re: Newbie: how to deploy/launch
Posted by Terence Yim <ch...@gmail.com>.
Hi Michael,
Looks like the example poms are wrong. Will look into that.
Terence
On Aug 14, 2014, at 2:01 PM, Peterson, Michael <mi...@truvenhealth.com> wrote:
> Terence and Alvin -
>
> Thanks for the help. Changing from guava 16 to 13 did ultimately fix the problem (along with changing the pom, see below). The reason I tried 16 is that guava 16 is what is included in the other example project (echo):
>
> $ jar -tf twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar
> [...]
> echo/
> echo/EchoMain$1.class
> echo/EchoMain$TestConverter.class
> echo/EchoMain.class
> lib/
> lib/guava-16.0.1.jar
> lib/slf4j-api-1.7.5.jar
>
>
>
> Also, as I mentioned in the first email, even though the shaded plugin is included in the yarn example app pom, no dependent libraries are added to the shaded jar. This is because the dependencies in its pom are scoped as "provided". If you comment out that scope, now it includes everything needed:
>
> <dependency>
> <groupId>${project.groupId}</groupId>
> <artifactId>twill-yarn</artifactId>
> <version>${project.version}</version>
> <!-- <scope>provided</scope> -->
> </dependency>
> <dependency>
> <groupId>${project.groupId}</groupId>
> <artifactId>twill-ext</artifactId>
> <version>${project.version}</version>
> <!-- <scope>provided</scope> -->
> </dependency>
> <dependency>
> <groupId>com.google.guava</groupId>
> <artifactId>guava</artifactId>
> <!-- <scope>provided</scope> -->
> </dependency>
>
>
>
> And now it ran with this:
>
> $ export CP=twill-examples-yarn-shaded-0.4.0-incubating-SNAPSHOT.jar:`hadoop classpath`
> $ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181
>
>
> Should the "provided" scope be removed from the yarn example pom? Why is it there?
>
> -Michael
>
>
> -----Original Message-----
> From: Terence Yim [mailto:chtyim@gmail.com]
> Sent: Thursday, August 14, 2014 12:42 AM
> To: dev@twill.incubator.apache.org
> Subject: Re: Newbie: how to deploy/launch
>
> Try to have guava 13 jar at the beginning if your classpath, that should solve the problem you mentioned.
>
> Terence
>
> Sent from my iPhone
>
>> On Aug 11, 2014, at 1:17 PM, "Peterson, Michael" <mi...@truvenhealth.com> wrote:
>>
>> I did a little more research on the run time error.
>>
>> When I run it with -verbose:
>> $ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld
>> localhost:2181
>>
>> I see that the problem is with the guava libraries - the Service$Listener class in particular.
>>
>> [... lots of lines removed ...]
>> [Loaded com.google.common.util.concurrent.Service$Listener from
>> file:/home/yarn/twill2/guava-16.0.1.jar]
>> [Loaded java.lang.ClassFormatError from
>> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>> Exception in thread "main" [Loaded
>> java.lang.Throwable$PrintStreamOrWriter from
>> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>> [Loaded java.lang.Throwable$WrappedPrintStream from
>> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>> [Loaded java.util.IdentityHashMap from
>> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>> [Loaded java.util.IdentityHashMap$KeySet from
>> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>> java.lang.IncompatibleClassChangeError: Implementing class
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at java.lang.ClassLoader.defineClass1(Native Method)
>> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
>> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
>> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
>> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
>> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>> at java.security.AccessController.doPrivileged(Native Method)
>> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>> at
>> org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
>> [Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>> [Loaded java.lang.Shutdown$Lock from
>> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>>
>>
>> The version in my version of Hadoop is Guava 11.0.2:
>>
>> $ ls /usr/lib/hadoop/lib/guava*
>> /usr/lib/hadoop/lib/guava-11.0.2.jar
>>
>>
>> Has anyone hit this issue with Twill? And how should we resolve it?
>>
>> -Michael
>
RE: Newbie: how to deploy/launch
Posted by "Peterson, Michael" <mi...@truvenhealth.com>.
Terence and Alvin -
Thanks for the help. Changing from guava 16 to 13 did ultimately fix the problem (along with changing the pom, see below). The reason I tried 16 is that guava 16 is what is included in the other example project (echo):
$ jar -tf twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar
[...]
echo/
echo/EchoMain$1.class
echo/EchoMain$TestConverter.class
echo/EchoMain.class
lib/
lib/guava-16.0.1.jar
lib/slf4j-api-1.7.5.jar
Also, as I mentioned in the first email, even though the shaded plugin is included in the yarn example app pom, no dependent libraries are added to the shaded jar. This is because the dependencies in its pom are scoped as "provided". If you comment out that scope, now it includes everything needed:
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>twill-yarn</artifactId>
<version>${project.version}</version>
<!-- <scope>provided</scope> -->
</dependency>
<dependency>
<groupId>${project.groupId}</groupId>
<artifactId>twill-ext</artifactId>
<version>${project.version}</version>
<!-- <scope>provided</scope> -->
</dependency>
<dependency>
<groupId>com.google.guava</groupId>
<artifactId>guava</artifactId>
<!-- <scope>provided</scope> -->
</dependency>
And now it ran with this:
$ export CP=twill-examples-yarn-shaded-0.4.0-incubating-SNAPSHOT.jar:`hadoop classpath`
$ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181
Should the "provided" scope be removed from the yarn example pom? Why is it there?
-Michael
-----Original Message-----
From: Terence Yim [mailto:chtyim@gmail.com]
Sent: Thursday, August 14, 2014 12:42 AM
To: dev@twill.incubator.apache.org
Subject: Re: Newbie: how to deploy/launch
Try to have guava 13 jar at the beginning if your classpath, that should solve the problem you mentioned.
Terence
Sent from my iPhone
> On Aug 11, 2014, at 1:17 PM, "Peterson, Michael" <mi...@truvenhealth.com> wrote:
>
> I did a little more research on the run time error.
>
> When I run it with -verbose:
> $ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld
> localhost:2181
>
> I see that the problem is with the guava libraries - the Service$Listener class in particular.
>
> [... lots of lines removed ...]
> [Loaded com.google.common.util.concurrent.Service$Listener from
> file:/home/yarn/twill2/guava-16.0.1.jar]
> [Loaded java.lang.ClassFormatError from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> Exception in thread "main" [Loaded
> java.lang.Throwable$PrintStreamOrWriter from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.lang.Throwable$WrappedPrintStream from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.util.IdentityHashMap from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.util.IdentityHashMap$KeySet from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
> [Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.lang.Shutdown$Lock from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>
>
> The version in my version of Hadoop is Guava 11.0.2:
>
> $ ls /usr/lib/hadoop/lib/guava*
> /usr/lib/hadoop/lib/guava-11.0.2.jar
>
>
> Has anyone hit this issue with Twill? And how should we resolve it?
>
> -Michael
Re: Newbie: how to deploy/launch
Posted by Terence Yim <ch...@gmail.com>.
Try to have guava 13 jar at the beginning if your classpath, that should solve the problem you mentioned.
Terence
Sent from my iPhone
> On Aug 11, 2014, at 1:17 PM, "Peterson, Michael" <mi...@truvenhealth.com> wrote:
>
> I did a little more research on the run time error.
>
> When I run it with -verbose:
> $ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181
>
> I see that the problem is with the guava libraries - the Service$Listener class in particular.
>
> [... lots of lines removed ...]
> [Loaded com.google.common.util.concurrent.Service$Listener from file:/home/yarn/twill2/guava-16.0.1.jar]
> [Loaded java.lang.ClassFormatError from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> Exception in thread "main" [Loaded java.lang.Throwable$PrintStreamOrWriter from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.lang.Throwable$WrappedPrintStream from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.util.IdentityHashMap from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.util.IdentityHashMap$KeySet from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
> [Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.lang.Shutdown$Lock from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>
>
> The version in my version of Hadoop is Guava 11.0.2:
>
> $ ls /usr/lib/hadoop/lib/guava*
> /usr/lib/hadoop/lib/guava-11.0.2.jar
>
>
> Has anyone hit this issue with Twill? And how should we resolve it?
>
> -Michael
>
>
> -----Original Message-----
> From: Peterson, Michael
> Sent: Monday, August 11, 2014 11:16 AM
> To: dev@twill.incubator.apache.org
> Subject: RE: Newbie: how to deploy/launch
>
> Hi Terence,
>
> Thanks for the detailed write up and explanation how to set up the classpath to run a Twill app and around how Twill uses OSGi.
>
> Right now I'm trying to get the Hello example working without OSGi and still having problems.
>
> Issue #1:
>
>> In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies.
>
> When I build Twill on Windows (I have the latest code from https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it generates a shaded jar in the yarn-example directory, but when I build it on Linux, it does *not* create a shaded jar.
>
> But even the shaded jar on Windows doesn't have any of its dependencies in it, so I don't think the pom is set up correctly?
>
> u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master) $ jar -tf twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> META-INF/MANIFEST.MF
> META-INF/
> META-INF/DEPENDENCIES
> META-INF/LICENSE
> META-INF/NOTICE
> org/
> org/apache/
> org/apache/twill/
> org/apache/twill/example/
> org/apache/twill/example/yarn/
> org/apache/twill/example/yarn/BundledJarExample$1.class
> org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class
> org/apache/twill/example/yarn/BundledJarExample.class
> org/apache/twill/example/yarn/HelloWorld$1.class
> org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class
> org/apache/twill/example/yarn/HelloWorld.class
> META-INF/maven/
> META-INF/maven/org.apache.twill/
> META-INF/maven/org.apache.twill/twill-examples-yarn/
> META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml
> META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties
>
>
> So I had to copy all the binary jars to the Hadoop system and put them all into a classpath env var, similar to how you suggested:
>
> $ echo $CP | tr ':' '\n'
> guava-16.0.1.jar
> twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar
> original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
> twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar
> slf4j-api-1.7.5.jar
> twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
> twill-api-0.4.0-incubating-SNAPSHOT.jar
> twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> twill-common-0.4.0-incubating-SNAPSHOT.jar
> twill-ext-0.4.0-incubating-SNAPSHOT.jar
> twill-core-0.4.0-incubating-SNAPSHOT.jar
> twill-yarn-0.4.0-incubating-SNAPSHOT.jar
> twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar
> twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> /etc/hadoop/conf
> /usr/lib/hadoop/lib/*
> /usr/lib/hadoop/.//*
> /usr/lib/hadoop-hdfs/./
> /usr/lib/hadoop-hdfs/lib/*
> /usr/lib/hadoop-hdfs/.//*
> /usr/lib/hadoop-yarn/lib/*
> /usr/lib/hadoop-yarn/.//*
> /usr/lib/hadoop-mapreduce/lib/*
> /usr/lib/hadoop-mapreduce/.//*
> /usr/share/java/mysql-connector-java-5.1.17.jar
> /usr/share/java/mysql-connector-java.jar
> /usr/lib/hadoop-mapreduce/*
> /usr/lib/tez/*
> /usr/lib/tez/lib/*
> /etc/tez/conf
>
>
>
> Issue #2:
>
> When I run the HelloWorld after setting up the classpath, I get an error:
>
> $ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181 Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
>
>
> Researching this it looks like a binary incompatibility so I recompiled all the Twill jars on the Linux Hadoop machine that I'm running it on, recopied all the jars to a single directory and ran it again, but I get the same error.
>
> On Linux, I'm running:
> $ java -version
> java version "1.7.0_45"
> Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
>
> On Windows, I'm compiling with:
> $ java -version
> java version "1.7.0_67"
> Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
>
>
> -Michael
>
>
> -----Original Message-----
> From: Terence Yim [mailto:chtyim@gmail.com]
> Sent: Saturday, August 09, 2014 2:33 AM
> To: dev@twill.incubator.apache.org
> Subject: Re: Newbie: how to deploy/launch
>
> Hi Peterson,
>
> Welcome to Apache Twill !!
>
> The general principle to launch a Twill application in YARN is simply to run a regular Java main class and launch the Twill application from there. Twill takes care of finding the dependencies and building a "job jar" for you.
>
> If you take a closer look at the HelloWorld example (source code in http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a HelloWorldRunnable, which basically is the class that will get executed in a YARN container. The other piece is the main() method, which is a YARN client and can run from any box that can talk to YARN RM (think of it as where you submit a YARN job).
>
> To run the HelloWorld example, you can simply compile and run the class, with both hadoop conf, hadoop jars and twill related jars in the classpath. You can even run it from your IDE, by having the yarn configuration available through the classpath or provided to the TwillRunnerService directly.
>
> In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies. So, to run it, you can simply do something like:
>
> CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP org.apache.twill.example.yarn.HelloWorld
>
> You also mentioned OSGI bundle jar. In fact, that's another thing that Twill supports. It allows you to run any Java application in twill, without worrying about library versions conflict between your application and twill/hadoop. You could simply write an application, build an OSGI bundle jar (actually Twill only look for jar entries that are .class or .jar, hence you can create a jar in that format using any tool you wanted), and use the the bundle jar runner
> (http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN client and also has a generic TwillRunnable wrapper that will unjar the bundle jar when running in a YARN container, create a ClassLoader using the .class and .jar files from the bundle jar and launch the
> main() method through that ClassLoader.
>
> If you face any issues while trying to run a Twill application, please feel free to discuss in here :)
>
> Thanks,
> Terence
>
>> On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael <mi...@truvenhealth.com> wrote:
>> Based on the source code for TwillLauncher, I tried to launch this on a Hadoop cluster like so:
>>
>> $ java -cp
>> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
>> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
>> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
>> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
>> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
>> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
>> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
>> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
>> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
>> org.apache.twill.launcher.TwillLauncher
>> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
>> org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181
>>
>>
>> Or with the BundledJar:
>> $ java -cp
>> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
>> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
>> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
>> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
>> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
>> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
>> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
>> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
>> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
>> org.apache.twill.launcher.TwillLauncher
>> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
>> org.apache.twill.example.yarn.BundledJarExample localhost:2181
>> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar .
>> org.apache.twill.example.yarn.HelloWorld localhost:2181
>>
>> But both fail as not being able to find the hadoop jars. I imagine you have an easier way to launch this. Can we launch it via "hadoop jar" or "yarn jar" instead?
>>
>> Which leads back to my questions about how to set up the classpath - a fat (shaded) jar or some OSGi mechanism?
>>
>> -Michael
>>
>> -----Original Message-----
>> From: Peterson, Michael [mailto:michael.d.peterson@truvenhealth.com]
>> Sent: Friday, August 08, 2014 9:59 AM
>> To: dev@twill.incubator.apache.org
>> Subject: Newbie: how to deploy/launch
>>
>> Hello,
>>
>> Just joined the list yesterday. I have read over the documentation I could find and downloaded and built the code. I have a few questions:
>>
>> 1. How does one deploy and launch a Twill-based app? I am familiar with Slider, so is it something like that, where you have to put your jars into HDFS launch the jar with the YarnClient and point it to the jar you put in HDFS? Any config or properties files required?
>>
>> 2. I see that the EchoMain example you provide uses OSGi, but it looks like the other example "yarn" does not and uses a shaded jar instead. Does that mean that running Twill with OSGi is optional or does it require OSGi?
>>
>>
>> BTW - the code builds on Window but at least one of the tests fails.
>>
>> Thank you,
>> -Michael
Re: Newbie: how to deploy/launch
Posted by Alvin Wang <al...@continuuity.com>.
Twill uses Guava 13.0.1, which is likely incompatible with Guava 11.0.2. I
also see that you have Guava 16.0.1 in your classpath, which is
incompatible with Guava 13.0.1.
Could you try replacing the Guava 16.0.1 in your classpath with Guava
13.0.1? If that doesn't work, could you also try replacing Hadoop's Guava
jar with 13.0.1? (make sure to backup your other Guava jars as I haven't
actually tried this)
Alvin
On Mon, Aug 11, 2014 at 1:17 PM, Peterson, Michael <
michael.d.peterson@truvenhealth.com> wrote:
> I did a little more research on the run time error.
>
> When I run it with -verbose:
> $ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld
> localhost:2181
>
> I see that the problem is with the guava libraries - the Service$Listener
> class in particular.
>
> [... lots of lines removed ...]
> [Loaded com.google.common.util.concurrent.Service$Listener from
> file:/home/yarn/twill2/guava-16.0.1.jar]
> [Loaded java.lang.ClassFormatError from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> Exception in thread "main" [Loaded java.lang.Throwable$PrintStreamOrWriter
> from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.lang.Throwable$WrappedPrintStream from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.util.IdentityHashMap from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.util.IdentityHashMap$KeySet from
> /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> java.lang.IncompatibleClassChangeError: Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
> [Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
> [Loaded java.lang.Shutdown$Lock from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
>
>
> The version in my version of Hadoop is Guava 11.0.2:
>
> $ ls /usr/lib/hadoop/lib/guava*
> /usr/lib/hadoop/lib/guava-11.0.2.jar
>
>
> Has anyone hit this issue with Twill? And how should we resolve it?
>
> -Michael
>
>
> -----Original Message-----
> From: Peterson, Michael
> Sent: Monday, August 11, 2014 11:16 AM
> To: dev@twill.incubator.apache.org
> Subject: RE: Newbie: how to deploy/launch
>
> Hi Terence,
>
> Thanks for the detailed write up and explanation how to set up the
> classpath to run a Twill app and around how Twill uses OSGi.
>
> Right now I'm trying to get the Hello example working without OSGi and
> still having problems.
>
> Issue #1:
>
> > In the Twill example directory, we used the maven-shade-plugin to build
> a fat jar that contains the HelloWorld class, as well as Twill classes and
> its dependencies.
>
> When I build Twill on Windows (I have the latest code from
> https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it
> generates a shaded jar in the yarn-example directory, but when I build it
> on Linux, it does *not* create a shaded jar.
>
> But even the shaded jar on Windows doesn't have any of its dependencies in
> it, so I don't think the pom is set up correctly?
>
> u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master) $ jar -tf
> twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> META-INF/MANIFEST.MF
> META-INF/
> META-INF/DEPENDENCIES
> META-INF/LICENSE
> META-INF/NOTICE
> org/
> org/apache/
> org/apache/twill/
> org/apache/twill/example/
> org/apache/twill/example/yarn/
> org/apache/twill/example/yarn/BundledJarExample$1.class
> org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class
> org/apache/twill/example/yarn/BundledJarExample.class
> org/apache/twill/example/yarn/HelloWorld$1.class
> org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class
> org/apache/twill/example/yarn/HelloWorld.class
> META-INF/maven/
> META-INF/maven/org.apache.twill/
> META-INF/maven/org.apache.twill/twill-examples-yarn/
> META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml
> META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties
>
>
> So I had to copy all the binary jars to the Hadoop system and put them all
> into a classpath env var, similar to how you suggested:
>
> $ echo $CP | tr ':' '\n'
> guava-16.0.1.jar
> twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar
> original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
> twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar
> slf4j-api-1.7.5.jar
> twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
> twill-api-0.4.0-incubating-SNAPSHOT.jar
> twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> twill-common-0.4.0-incubating-SNAPSHOT.jar
> twill-ext-0.4.0-incubating-SNAPSHOT.jar
> twill-core-0.4.0-incubating-SNAPSHOT.jar
> twill-yarn-0.4.0-incubating-SNAPSHOT.jar
> twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar
> twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> /etc/hadoop/conf
> /usr/lib/hadoop/lib/*
> /usr/lib/hadoop/.//*
> /usr/lib/hadoop-hdfs/./
> /usr/lib/hadoop-hdfs/lib/*
> /usr/lib/hadoop-hdfs/.//*
> /usr/lib/hadoop-yarn/lib/*
> /usr/lib/hadoop-yarn/.//*
> /usr/lib/hadoop-mapreduce/lib/*
> /usr/lib/hadoop-mapreduce/.//*
> /usr/share/java/mysql-connector-java-5.1.17.jar
> /usr/share/java/mysql-connector-java.jar
> /usr/lib/hadoop-mapreduce/*
> /usr/lib/tez/*
> /usr/lib/tez/lib/*
> /etc/tez/conf
>
>
>
> Issue #2:
>
> When I run the HelloWorld after setting up the classpath, I get an error:
>
> $ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181
> Exception in thread "main" java.lang.IncompatibleClassChangeError:
> Implementing class
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at java.lang.ClassLoader.defineClass1(Native Method)
> at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
> at
> java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
> at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
> at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
> at java.security.AccessController.doPrivileged(Native Method)
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
> at
> org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
>
>
> Researching this it looks like a binary incompatibility so I recompiled
> all the Twill jars on the Linux Hadoop machine that I'm running it on,
> recopied all the jars to a single directory and ran it again, but I get the
> same error.
>
> On Linux, I'm running:
> $ java -version
> java version "1.7.0_45"
> Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM)
> 64-Bit Server VM (build 24.45-b08, mixed mode)
>
> On Windows, I'm compiling with:
> $ java -version
> java version "1.7.0_67"
> Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM)
> 64-Bit Server VM (build 24.65-b04, mixed mode)
>
>
> -Michael
>
>
> -----Original Message-----
> From: Terence Yim [mailto:chtyim@gmail.com]
> Sent: Saturday, August 09, 2014 2:33 AM
> To: dev@twill.incubator.apache.org
> Subject: Re: Newbie: how to deploy/launch
>
> Hi Peterson,
>
> Welcome to Apache Twill !!
>
> The general principle to launch a Twill application in YARN is simply to
> run a regular Java main class and launch the Twill application from there.
> Twill takes care of finding the dependencies and building a "job jar" for
> you.
>
> If you take a closer look at the HelloWorld example (source code in
> http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a
> HelloWorldRunnable, which basically is the class that will get executed in
> a YARN container. The other piece is the main() method, which is a YARN
> client and can run from any box that can talk to YARN RM (think of it as
> where you submit a YARN job).
>
> To run the HelloWorld example, you can simply compile and run the class,
> with both hadoop conf, hadoop jars and twill related jars in the classpath.
> You can even run it from your IDE, by having the yarn configuration
> available through the classpath or provided to the TwillRunnerService
> directly.
>
> In the Twill example directory, we used the maven-shade-plugin to build a
> fat jar that contains the HelloWorld class, as well as Twill classes and
> its dependencies. So, to run it, you can simply do something like:
>
> CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP
> org.apache.twill.example.yarn.HelloWorld
>
> You also mentioned OSGI bundle jar. In fact, that's another thing that
> Twill supports. It allows you to run any Java application in twill, without
> worrying about library versions conflict between your application and
> twill/hadoop. You could simply write an application, build an OSGI bundle
> jar (actually Twill only look for jar entries that are .class or .jar,
> hence you can create a jar in that format using any tool you wanted), and
> use the the bundle jar runner
> (http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN
> client and also has a generic TwillRunnable wrapper that will unjar the
> bundle jar when running in a YARN container, create a ClassLoader using the
> .class and .jar files from the bundle jar and launch the
> main() method through that ClassLoader.
>
> If you face any issues while trying to run a Twill application, please
> feel free to discuss in here :)
>
> Thanks,
> Terence
>
> On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael <
> michael.d.peterson@truvenhealth.com> wrote:
> > Based on the source code for TwillLauncher, I tried to launch this on a
> Hadoop cluster like so:
> >
> > $ java -cp
> > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
> > iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
> > s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
> > s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
> > OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
> > common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
> > OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
> > ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
> > will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> > org.apache.twill.launcher.TwillLauncher
> > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> > org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181
> >
> >
> > Or with the BundledJar:
> > $ java -cp
> > guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
> > iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
> > s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
> > s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
> > OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
> > common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
> > OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
> > ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
> > will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> > org.apache.twill.launcher.TwillLauncher
> > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> > org.apache.twill.example.yarn.BundledJarExample localhost:2181
> > ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar .
> > org.apache.twill.example.yarn.HelloWorld localhost:2181
> >
> > But both fail as not being able to find the hadoop jars. I imagine you
> have an easier way to launch this. Can we launch it via "hadoop jar" or
> "yarn jar" instead?
> >
> > Which leads back to my questions about how to set up the classpath - a
> fat (shaded) jar or some OSGi mechanism?
> >
> > -Michael
> >
> > -----Original Message-----
> > From: Peterson, Michael [mailto:michael.d.peterson@truvenhealth.com]
> > Sent: Friday, August 08, 2014 9:59 AM
> > To: dev@twill.incubator.apache.org
> > Subject: Newbie: how to deploy/launch
> >
> > Hello,
> >
> > Just joined the list yesterday. I have read over the documentation I
> could find and downloaded and built the code. I have a few questions:
> >
> > 1. How does one deploy and launch a Twill-based app? I am familiar
> with Slider, so is it something like that, where you have to put your jars
> into HDFS launch the jar with the YarnClient and point it to the jar you
> put in HDFS? Any config or properties files required?
> >
> > 2. I see that the EchoMain example you provide uses OSGi, but it
> looks like the other example "yarn" does not and uses a shaded jar instead.
> Does that mean that running Twill with OSGi is optional or does it require
> OSGi?
> >
> >
> > BTW - the code builds on Window but at least one of the tests fails.
> >
> > Thank you,
> > -Michael
>
RE: Newbie: how to deploy/launch
Posted by "Peterson, Michael" <mi...@truvenhealth.com>.
I did a little more research on the run time error.
When I run it with -verbose:
$ java -verbose -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181
I see that the problem is with the guava libraries - the Service$Listener class in particular.
[... lots of lines removed ...]
[Loaded com.google.common.util.concurrent.Service$Listener from file:/home/yarn/twill2/guava-16.0.1.jar]
[Loaded java.lang.ClassFormatError from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
Exception in thread "main" [Loaded java.lang.Throwable$PrintStreamOrWriter from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
[Loaded java.lang.Throwable$WrappedPrintStream from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
[Loaded java.util.IdentityHashMap from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
[Loaded java.util.IdentityHashMap$KeySet from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
[Loaded java.lang.Shutdown from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
[Loaded java.lang.Shutdown$Lock from /usr/jdk64/jdk1.7.0_45/jre/lib/rt.jar]
The version in my version of Hadoop is Guava 11.0.2:
$ ls /usr/lib/hadoop/lib/guava*
/usr/lib/hadoop/lib/guava-11.0.2.jar
Has anyone hit this issue with Twill? And how should we resolve it?
-Michael
-----Original Message-----
From: Peterson, Michael
Sent: Monday, August 11, 2014 11:16 AM
To: dev@twill.incubator.apache.org
Subject: RE: Newbie: how to deploy/launch
Hi Terence,
Thanks for the detailed write up and explanation how to set up the classpath to run a Twill app and around how Twill uses OSGi.
Right now I'm trying to get the Hello example working without OSGi and still having problems.
Issue #1:
> In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies.
When I build Twill on Windows (I have the latest code from https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it generates a shaded jar in the yarn-example directory, but when I build it on Linux, it does *not* create a shaded jar.
But even the shaded jar on Windows doesn't have any of its dependencies in it, so I don't think the pom is set up correctly?
u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master) $ jar -tf twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
META-INF/MANIFEST.MF
META-INF/
META-INF/DEPENDENCIES
META-INF/LICENSE
META-INF/NOTICE
org/
org/apache/
org/apache/twill/
org/apache/twill/example/
org/apache/twill/example/yarn/
org/apache/twill/example/yarn/BundledJarExample$1.class
org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class
org/apache/twill/example/yarn/BundledJarExample.class
org/apache/twill/example/yarn/HelloWorld$1.class
org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class
org/apache/twill/example/yarn/HelloWorld.class
META-INF/maven/
META-INF/maven/org.apache.twill/
META-INF/maven/org.apache.twill/twill-examples-yarn/
META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml
META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties
So I had to copy all the binary jars to the Hadoop system and put them all into a classpath env var, similar to how you suggested:
$ echo $CP | tr ':' '\n'
guava-16.0.1.jar
twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar
original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar
slf4j-api-1.7.5.jar
twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
twill-api-0.4.0-incubating-SNAPSHOT.jar
twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
twill-common-0.4.0-incubating-SNAPSHOT.jar
twill-ext-0.4.0-incubating-SNAPSHOT.jar
twill-core-0.4.0-incubating-SNAPSHOT.jar
twill-yarn-0.4.0-incubating-SNAPSHOT.jar
twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar
twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar
/etc/hadoop/conf
/usr/lib/hadoop/lib/*
/usr/lib/hadoop/.//*
/usr/lib/hadoop-hdfs/./
/usr/lib/hadoop-hdfs/lib/*
/usr/lib/hadoop-hdfs/.//*
/usr/lib/hadoop-yarn/lib/*
/usr/lib/hadoop-yarn/.//*
/usr/lib/hadoop-mapreduce/lib/*
/usr/lib/hadoop-mapreduce/.//*
/usr/share/java/mysql-connector-java-5.1.17.jar
/usr/share/java/mysql-connector-java.jar
/usr/lib/hadoop-mapreduce/*
/usr/lib/tez/*
/usr/lib/tez/lib/*
/etc/tez/conf
Issue #2:
When I run the HelloWorld after setting up the classpath, I get an error:
$ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181 Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
Researching this it looks like a binary incompatibility so I recompiled all the Twill jars on the Linux Hadoop machine that I'm running it on, recopied all the jars to a single directory and ran it again, but I get the same error.
On Linux, I'm running:
$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18) Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
On Windows, I'm compiling with:
$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01) Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
-Michael
-----Original Message-----
From: Terence Yim [mailto:chtyim@gmail.com]
Sent: Saturday, August 09, 2014 2:33 AM
To: dev@twill.incubator.apache.org
Subject: Re: Newbie: how to deploy/launch
Hi Peterson,
Welcome to Apache Twill !!
The general principle to launch a Twill application in YARN is simply to run a regular Java main class and launch the Twill application from there. Twill takes care of finding the dependencies and building a "job jar" for you.
If you take a closer look at the HelloWorld example (source code in http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a HelloWorldRunnable, which basically is the class that will get executed in a YARN container. The other piece is the main() method, which is a YARN client and can run from any box that can talk to YARN RM (think of it as where you submit a YARN job).
To run the HelloWorld example, you can simply compile and run the class, with both hadoop conf, hadoop jars and twill related jars in the classpath. You can even run it from your IDE, by having the yarn configuration available through the classpath or provided to the TwillRunnerService directly.
In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies. So, to run it, you can simply do something like:
CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP org.apache.twill.example.yarn.HelloWorld
You also mentioned OSGI bundle jar. In fact, that's another thing that Twill supports. It allows you to run any Java application in twill, without worrying about library versions conflict between your application and twill/hadoop. You could simply write an application, build an OSGI bundle jar (actually Twill only look for jar entries that are .class or .jar, hence you can create a jar in that format using any tool you wanted), and use the the bundle jar runner
(http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN client and also has a generic TwillRunnable wrapper that will unjar the bundle jar when running in a YARN container, create a ClassLoader using the .class and .jar files from the bundle jar and launch the
main() method through that ClassLoader.
If you face any issues while trying to run a Twill application, please feel free to discuss in here :)
Thanks,
Terence
On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael <mi...@truvenhealth.com> wrote:
> Based on the source code for TwillLauncher, I tried to launch this on a Hadoop cluster like so:
>
> $ java -cp
> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> org.apache.twill.launcher.TwillLauncher
> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181
>
>
> Or with the BundledJar:
> $ java -cp
> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> org.apache.twill.launcher.TwillLauncher
> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> org.apache.twill.example.yarn.BundledJarExample localhost:2181
> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar .
> org.apache.twill.example.yarn.HelloWorld localhost:2181
>
> But both fail as not being able to find the hadoop jars. I imagine you have an easier way to launch this. Can we launch it via "hadoop jar" or "yarn jar" instead?
>
> Which leads back to my questions about how to set up the classpath - a fat (shaded) jar or some OSGi mechanism?
>
> -Michael
>
> -----Original Message-----
> From: Peterson, Michael [mailto:michael.d.peterson@truvenhealth.com]
> Sent: Friday, August 08, 2014 9:59 AM
> To: dev@twill.incubator.apache.org
> Subject: Newbie: how to deploy/launch
>
> Hello,
>
> Just joined the list yesterday. I have read over the documentation I could find and downloaded and built the code. I have a few questions:
>
> 1. How does one deploy and launch a Twill-based app? I am familiar with Slider, so is it something like that, where you have to put your jars into HDFS launch the jar with the YarnClient and point it to the jar you put in HDFS? Any config or properties files required?
>
> 2. I see that the EchoMain example you provide uses OSGi, but it looks like the other example "yarn" does not and uses a shaded jar instead. Does that mean that running Twill with OSGi is optional or does it require OSGi?
>
>
> BTW - the code builds on Window but at least one of the tests fails.
>
> Thank you,
> -Michael
RE: Newbie: how to deploy/launch
Posted by "Peterson, Michael" <mi...@truvenhealth.com>.
Hi Terence,
Thanks for the detailed write up and explanation how to set up the classpath to run a Twill app and around how Twill uses OSGi.
Right now I'm trying to get the Hello example working without OSGi and still having problems.
Issue #1:
> In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies.
When I build Twill on Windows (I have the latest code from https://git-wip-us.apache.org/repos/asf/incubator-twill.git), it generates a shaded jar in the yarn-example directory, but when I build it on Linux, it does *not* create a shaded jar.
But even the shaded jar on Windows doesn't have any of its dependencies in it, so I don't think the pom is set up correctly?
u070072@DUR-MPETER2-2W7 /d/eval/twill/incubator-twill (master)
$ jar -tf twill-examples/yarn/target/twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
META-INF/MANIFEST.MF
META-INF/
META-INF/DEPENDENCIES
META-INF/LICENSE
META-INF/NOTICE
org/
org/apache/
org/apache/twill/
org/apache/twill/example/
org/apache/twill/example/yarn/
org/apache/twill/example/yarn/BundledJarExample$1.class
org/apache/twill/example/yarn/BundledJarExample$ExampleBundledJarApp.class
org/apache/twill/example/yarn/BundledJarExample.class
org/apache/twill/example/yarn/HelloWorld$1.class
org/apache/twill/example/yarn/HelloWorld$HelloWorldRunnable.class
org/apache/twill/example/yarn/HelloWorld.class
META-INF/maven/
META-INF/maven/org.apache.twill/
META-INF/maven/org.apache.twill/twill-examples-yarn/
META-INF/maven/org.apache.twill/twill-examples-yarn/pom.xml
META-INF/maven/org.apache.twill/twill-examples-yarn/pom.properties
So I had to copy all the binary jars to the Hadoop system and put them all into a classpath env var, similar to how you suggested:
$ echo $CP | tr ':' '\n'
guava-16.0.1.jar
twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar
original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar
slf4j-api-1.7.5.jar
twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar
twill-api-0.4.0-incubating-SNAPSHOT.jar
twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
twill-common-0.4.0-incubating-SNAPSHOT.jar
twill-ext-0.4.0-incubating-SNAPSHOT.jar
twill-core-0.4.0-incubating-SNAPSHOT.jar
twill-yarn-0.4.0-incubating-SNAPSHOT.jar
twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar
twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar
/etc/hadoop/conf
/usr/lib/hadoop/lib/*
/usr/lib/hadoop/.//*
/usr/lib/hadoop-hdfs/./
/usr/lib/hadoop-hdfs/lib/*
/usr/lib/hadoop-hdfs/.//*
/usr/lib/hadoop-yarn/lib/*
/usr/lib/hadoop-yarn/.//*
/usr/lib/hadoop-mapreduce/lib/*
/usr/lib/hadoop-mapreduce/.//*
/usr/share/java/mysql-connector-java-5.1.17.jar
/usr/share/java/mysql-connector-java.jar
/usr/lib/hadoop-mapreduce/*
/usr/lib/tez/*
/usr/lib/tez/lib/*
/etc/tez/conf
Issue #2:
When I run the HelloWorld after setting up the classpath, I get an error:
$ java -cp $CP org.apache.twill.example.yarn.HelloWorld localhost:2181
Exception in thread "main" java.lang.IncompatibleClassChangeError: Implementing class
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:800)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:449)
at java.net.URLClassLoader.access$100(URLClassLoader.java:71)
at java.net.URLClassLoader$1.run(URLClassLoader.java:361)
at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
at org.apache.twill.example.yarn.HelloWorld.main(HelloWorld.java:61)
Researching this it looks like a binary incompatibility so I recompiled all the Twill jars on the Linux Hadoop machine that I'm running it on, recopied all the jars to a single directory and ran it again, but I get the same error.
On Linux, I'm running:
$ java -version
java version "1.7.0_45"
Java(TM) SE Runtime Environment (build 1.7.0_45-b18)
Java HotSpot(TM) 64-Bit Server VM (build 24.45-b08, mixed mode)
On Windows, I'm compiling with:
$ java -version
java version "1.7.0_67"
Java(TM) SE Runtime Environment (build 1.7.0_67-b01)
Java HotSpot(TM) 64-Bit Server VM (build 24.65-b04, mixed mode)
-Michael
-----Original Message-----
From: Terence Yim [mailto:chtyim@gmail.com]
Sent: Saturday, August 09, 2014 2:33 AM
To: dev@twill.incubator.apache.org
Subject: Re: Newbie: how to deploy/launch
Hi Peterson,
Welcome to Apache Twill !!
The general principle to launch a Twill application in YARN is simply to run a regular Java main class and launch the Twill application from there. Twill takes care of finding the dependencies and building a "job jar" for you.
If you take a closer look at the HelloWorld example (source code in http://bit.ly/1yfMhDC), it actually contains two parts. First, there is a HelloWorldRunnable, which basically is the class that will get executed in a YARN container. The other piece is the main() method, which is a YARN client and can run from any box that can talk to YARN RM (think of it as where you submit a YARN job).
To run the HelloWorld example, you can simply compile and run the class, with both hadoop conf, hadoop jars and twill related jars in the classpath. You can even run it from your IDE, by having the yarn configuration available through the classpath or provided to the TwillRunnerService directly.
In the Twill example directory, we used the maven-shade-plugin to build a fat jar that contains the HelloWorld class, as well as Twill classes and its dependencies. So, to run it, you can simply do something like:
CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP org.apache.twill.example.yarn.HelloWorld
You also mentioned OSGI bundle jar. In fact, that's another thing that Twill supports. It allows you to run any Java application in twill, without worrying about library versions conflict between your application and twill/hadoop. You could simply write an application, build an OSGI bundle jar (actually Twill only look for jar entries that are .class or .jar, hence you can create a jar in that format using any tool you wanted), and use the the bundle jar runner
(http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN client and also has a generic TwillRunnable wrapper that will unjar the bundle jar when running in a YARN container, create a ClassLoader using the .class and .jar files from the bundle jar and launch the
main() method through that ClassLoader.
If you face any issues while trying to run a Twill application, please feel free to discuss in here :)
Thanks,
Terence
On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael <mi...@truvenhealth.com> wrote:
> Based on the source code for TwillLauncher, I tried to launch this on a Hadoop cluster like so:
>
> $ java -cp
> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> org.apache.twill.launcher.TwillLauncher
> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181
>
>
> Or with the BundledJar:
> $ java -cp
> guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:or
> iginal-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-example
> s-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-example
> s-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSH
> OT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-
> common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSH
> OT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incub
> ating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:t
> will-zookeeper-0.4.0-incubating-SNAPSHOT.jar
> org.apache.twill.launcher.TwillLauncher
> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar
> org.apache.twill.example.yarn.BundledJarExample localhost:2181
> ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar .
> org.apache.twill.example.yarn.HelloWorld localhost:2181
>
> But both fail as not being able to find the hadoop jars. I imagine you have an easier way to launch this. Can we launch it via "hadoop jar" or "yarn jar" instead?
>
> Which leads back to my questions about how to set up the classpath - a fat (shaded) jar or some OSGi mechanism?
>
> -Michael
>
> -----Original Message-----
> From: Peterson, Michael [mailto:michael.d.peterson@truvenhealth.com]
> Sent: Friday, August 08, 2014 9:59 AM
> To: dev@twill.incubator.apache.org
> Subject: Newbie: how to deploy/launch
>
> Hello,
>
> Just joined the list yesterday. I have read over the documentation I could find and downloaded and built the code. I have a few questions:
>
> 1. How does one deploy and launch a Twill-based app? I am familiar with Slider, so is it something like that, where you have to put your jars into HDFS launch the jar with the YarnClient and point it to the jar you put in HDFS? Any config or properties files required?
>
> 2. I see that the EchoMain example you provide uses OSGi, but it looks like the other example "yarn" does not and uses a shaded jar instead. Does that mean that running Twill with OSGi is optional or does it require OSGi?
>
>
> BTW - the code builds on Window but at least one of the tests fails.
>
> Thank you,
> -Michael
Re: Newbie: how to deploy/launch
Posted by Terence Yim <ch...@gmail.com>.
Hi Peterson,
Welcome to Apache Twill !!
The general principle to launch a Twill application in YARN is simply
to run a regular Java main class and launch the Twill application from
there. Twill takes care of finding the dependencies and building a
"job jar" for you.
If you take a closer look at the HelloWorld example (source code in
http://bit.ly/1yfMhDC), it actually contains two parts. First, there
is a HelloWorldRunnable, which basically is the class that will get
executed in a YARN container. The other piece is the main() method,
which is a YARN client and can run from any box that can talk to YARN
RM (think of it as where you submit a YARN job).
To run the HelloWorld example, you can simply compile and run the
class, with both hadoop conf, hadoop jars and twill related jars in
the classpath. You can even run it from your IDE, by having the yarn
configuration available through the classpath or provided to the
TwillRunnerService directly.
In the Twill example directory, we used the maven-shade-plugin to
build a fat jar that contains the HelloWorld class, as well as Twill
classes and its dependencies. So, to run it, you can simply do
something like:
CP=`hadoop classpath` java -cp twill-examples-yarn.jar:$CP
org.apache.twill.example.yarn.HelloWorld
You also mentioned OSGI bundle jar. In fact, that's another thing that
Twill supports. It allows you to run any Java application in twill,
without worrying about library versions conflict between your
application and twill/hadoop. You could simply write an application,
build an OSGI bundle jar (actually Twill only look for jar entries
that are .class or .jar, hence you can create a jar in that format
using any tool you wanted), and use the the bundle jar runner
(http://bit.ly/1osS4EN) to run it. The bundle jar runner act as a YARN
client and also has a generic TwillRunnable wrapper that will unjar
the bundle jar when running in a YARN container, create a ClassLoader
using the .class and .jar files from the bundle jar and launch the
main() method through that ClassLoader.
If you face any issues while trying to run a Twill application, please
feel free to discuss in here :)
Thanks,
Terence
On Fri, Aug 8, 2014 at 11:23 AM, Peterson, Michael
<mi...@truvenhealth.com> wrote:
> Based on the source code for TwillLauncher, I tried to launch this on a Hadoop cluster like so:
>
> $ java -cp guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSHOT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSHOT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar org.apache.twill.launcher.TwillLauncher ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181
>
>
> Or with the BundledJar:
> $ java -cp guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSHOT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSHOT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar org.apache.twill.launcher.TwillLauncher ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar org.apache.twill.example.yarn.BundledJarExample localhost:2181 ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar . org.apache.twill.example.yarn.HelloWorld localhost:2181
>
> But both fail as not being able to find the hadoop jars. I imagine you have an easier way to launch this. Can we launch it via "hadoop jar" or "yarn jar" instead?
>
> Which leads back to my questions about how to set up the classpath - a fat (shaded) jar or some OSGi mechanism?
>
> -Michael
>
> -----Original Message-----
> From: Peterson, Michael [mailto:michael.d.peterson@truvenhealth.com]
> Sent: Friday, August 08, 2014 9:59 AM
> To: dev@twill.incubator.apache.org
> Subject: Newbie: how to deploy/launch
>
> Hello,
>
> Just joined the list yesterday. I have read over the documentation I could find and downloaded and built the code. I have a few questions:
>
> 1. How does one deploy and launch a Twill-based app? I am familiar with Slider, so is it something like that, where you have to put your jars into HDFS launch the jar with the YarnClient and point it to the jar you put in HDFS? Any config or properties files required?
>
> 2. I see that the EchoMain example you provide uses OSGi, but it looks like the other example "yarn" does not and uses a shaded jar instead. Does that mean that running Twill with OSGi is optional or does it require OSGi?
>
>
> BTW - the code builds on Window but at least one of the tests fails.
>
> Thank you,
> -Michael
RE: Newbie: how to deploy/launch
Posted by "Peterson, Michael" <mi...@truvenhealth.com>.
Based on the source code for TwillLauncher, I tried to launch this on a Hadoop cluster like so:
$ java -cp guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSHOT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSHOT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar org.apache.twill.launcher.TwillLauncher ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar org.apache.twill.example.yarn.HelloWorld localhost:2181 localhost:2181
Or with the BundledJar:
$ java -cp guava-16.0.1.jar:twill-discovery-core-0.4.0-incubating-SNAPSHOT.jar:original-twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-examples-echo-0.4.0-incubating-SNAPSHOT.jar:slf4j-api-1.7.5.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-api-0.4.0-incubating-SNAPSHOT.jar:twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar:twill-common-0.4.0-incubating-SNAPSHOT.jar:twill-ext-0.4.0-incubating-SNAPSHOT.jar:twill-core-0.4.0-incubating-SNAPSHOT.jar:twill-yarn-0.4.0-incubating-SNAPSHOT.jar:twill-discovery-api-0.4.0-incubating-SNAPSHOT.jar:twill-zookeeper-0.4.0-incubating-SNAPSHOT.jar org.apache.twill.launcher.TwillLauncher ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar org.apache.twill.example.yarn.BundledJarExample localhost:2181 ./twill-examples-yarn-0.4.0-incubating-SNAPSHOT-shaded.jar . org.apache.twill.example.yarn.HelloWorld localhost:2181
But both fail as not being able to find the hadoop jars. I imagine you have an easier way to launch this. Can we launch it via "hadoop jar" or "yarn jar" instead?
Which leads back to my questions about how to set up the classpath - a fat (shaded) jar or some OSGi mechanism?
-Michael
-----Original Message-----
From: Peterson, Michael [mailto:michael.d.peterson@truvenhealth.com]
Sent: Friday, August 08, 2014 9:59 AM
To: dev@twill.incubator.apache.org
Subject: Newbie: how to deploy/launch
Hello,
Just joined the list yesterday. I have read over the documentation I could find and downloaded and built the code. I have a few questions:
1. How does one deploy and launch a Twill-based app? I am familiar with Slider, so is it something like that, where you have to put your jars into HDFS launch the jar with the YarnClient and point it to the jar you put in HDFS? Any config or properties files required?
2. I see that the EchoMain example you provide uses OSGi, but it looks like the other example "yarn" does not and uses a shaded jar instead. Does that mean that running Twill with OSGi is optional or does it require OSGi?
BTW - the code builds on Window but at least one of the tests fails.
Thank you,
-Michael