You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@twill.apache.org by Hendrik Lücke-Tieke <he...@hotmail.com> on 2014/06/05 20:19:54 UTC
Twill application development requirements for windows
Hi,
is it possible to develop and start a twill application on windows without having to install hadoop binaries? Or a node needed on the local machine?
Ty,
Hendrik
AW: Twill application development requirements for windows
Posted by helt <he...@outlook.com>.
So twill doesn’t check for "binary compatibility" with the cluster? If that is possible, I would (try to) provide a patch for that^^ If youre willing to lend me a hand and tell me which direction to search...
-----Ursprüngliche Nachricht-----
Von: Gary Helmling [mailto:ghelmling@gmail.com]
Gesendet: Donnerstag, 19. Juni 2014 19:27
An: dev@twill.incubator.apache.org
Betreff: Re: Twill application development requirements for windows
Ah, that would be a problem. Glad that you worked out the issue and have things up and running now!
On Thu, Jun 19, 2014 at 10:18 AM, helt <he...@outlook.com> wrote:
> Hi Gary,
>
> finally the hello world program started and finished successfully.
>
> The issue could be identified as a java version mismatch. The pom.xml
> was configured to java use 1.7 code and binaries whereas the cluster
> runs oracles jdk 1.6.
>
> DOH...
>
> Best regards,
> Hendrik
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: helt [mailto:helt@outlook.com]
> Gesendet: Sonntag, 15. Juni 2014 14:56
> An: dev@twill.incubator.apache.org
> Betreff: AW: Twill application development requirements for windows
>
> Hi Gary,
>
>
>
>
>
> Thank you for your response. In the short run, I want to develop a
> twill application on windows and then run it on a cluster. In the long
> run, I probably will develop & test on windows using a local Hadoop,
> but first, I would like to run hello world as a minimum working example.
>
> Then I can build on this to do my own stuff, but before I do that, I
> want to go through the examples to verify/fix that my setup works
> correctly, and that it will run on the cluster correctly…
>
>
>
> Which brings me to my actual question:
>
> What do I have to do to get the hello world example to run on the real
> cluster? I managed to get around a few issues. But it seems like the
> Hello World example is optimized towards a local cluster installation.
>
>
>
> So far I managed to fix (snippet at EOF):
>
> - Connection to yarn fixed by setting the yarn resource manager
> address to sth != 0.0.0.0:8032
>
> - Connection to hdfs fixed by providing a defaultFS.
>
> - Minimize the apps resource requirements, to make sure that yarn
> is able to provide the needed resources.
>
>
>
> Hello World will start, but not finish, nor print the log messages…
>
> Open issues are therefore (from what I can see from here with my
> limited
> experience):
>
> - Fix log4j configurations so it stops complaining about “No
> appenders” somehow.
>
>
>
> Wrap-up: Hello World configuration for running on a real cluster seems
> to be quite cumbersome and is still unkown. Any advice on this?
>
>
>
> Best regards,
>
> Hendrik
>
>
>
>
>
> Twill configuration to get it running:
>
> public static void main(String[] args) {
>
>
>
> String zkConnectStr = "machine1:2181";
>
> Configuration conf = new Configuration();
>
> conf.set("yarn.resourcemanager.address", "machine1:8032");
>
> conf.set("fs.defaultFS", "hdfs://machine1:8020");
>
>
>
> YarnConfiguration yc = new YarnConfiguration(conf);
>
> final TwillRunnerService twillRunner = new
> YarnTwillRunnerService(yc, zkConnectStr);
>
> twillRunner.startAndWait();
>
> TwillPreparer twillPreparer = twillRunner.prepare(new
> HelloWorldRunnable(), ResourceSpecification.Builder.with()
>
> .setVirtualCores(1)
>
> .setMemory(64,
> SizeUnit.MEGA)
>
> .setInstances(1)
>
> .build())
>
> .addLogHandler(
>
> new PrinterLogHandler(new PrintWriter(System.out, true)));
>
> final TwillController controller = twillPreparer.start();
>
>
>
>
>
>
>
> -----Ursprüngliche Nachricht-----
>
> Von: Gary Helmling [mailto:ghelmling@gmail.com]
>
> Gesendet: Freitag, 6. Juni 2014 18:39
>
> An: dev@twill.incubator.apache.org
>
> Betreff: Re: Twill application development requirements for windows
>
>
>
> Hi Hendrik,
>
>
>
> Are you talking about developing a twill application on windows and
> then running it on a cluster? That should be possible with just the
> Hadoop client library jars (dependencies of twill).
>
>
>
> Or are you talking about developing and testing locally on a
> pseudo-distributed Hadoop cluster running on your windows box?
>
>
>
> Please describe in a bit more detail what you want to do.
>
>
>
> Gary
>
>
>
>
>
>
>
> On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke
> <he...@hotmail.com>
>
> wrote:
>
>
>
> > Hi,
>
> >
>
> >
>
> > is it possible to develop and start a twill application on windows
>
> > without having to install hadoop binaries? Or a node needed on the
> > local
> machine?
>
> >
>
> >
>
> > Ty,
>
> >
>
> > Hendrik
>
>
Re: Twill application development requirements for windows
Posted by Gary Helmling <gh...@gmail.com>.
Ah, that would be a problem. Glad that you worked out the issue and have
things up and running now!
On Thu, Jun 19, 2014 at 10:18 AM, helt <he...@outlook.com> wrote:
> Hi Gary,
>
> finally the hello world program started and finished successfully.
>
> The issue could be identified as a java version mismatch. The pom.xml was
> configured to java use 1.7 code and binaries whereas the cluster runs
> oracles jdk 1.6.
>
> DOH...
>
> Best regards,
> Hendrik
>
>
>
> -----Ursprüngliche Nachricht-----
> Von: helt [mailto:helt@outlook.com]
> Gesendet: Sonntag, 15. Juni 2014 14:56
> An: dev@twill.incubator.apache.org
> Betreff: AW: Twill application development requirements for windows
>
> Hi Gary,
>
>
>
>
>
> Thank you for your response. In the short run, I want to develop a twill
> application on windows and then run it on a cluster. In the long run, I
> probably will develop & test on windows using a local Hadoop, but first, I
> would like to run hello world as a minimum working example.
>
> Then I can build on this to do my own stuff, but before I do that, I want
> to go through the examples to verify/fix that my setup works correctly, and
> that it will run on the cluster correctly…
>
>
>
> Which brings me to my actual question:
>
> What do I have to do to get the hello world example to run on the real
> cluster? I managed to get around a few issues. But it seems like the Hello
> World example is optimized towards a local cluster installation.
>
>
>
> So far I managed to fix (snippet at EOF):
>
> - Connection to yarn fixed by setting the yarn resource manager
> address to sth != 0.0.0.0:8032
>
> - Connection to hdfs fixed by providing a defaultFS.
>
> - Minimize the apps resource requirements, to make sure that yarn
> is able to provide the needed resources.
>
>
>
> Hello World will start, but not finish, nor print the log messages…
>
> Open issues are therefore (from what I can see from here with my limited
> experience):
>
> - Fix log4j configurations so it stops complaining about “No
> appenders” somehow.
>
>
>
> Wrap-up: Hello World configuration for running on a real cluster seems to
> be quite cumbersome and is still unkown. Any advice on this?
>
>
>
> Best regards,
>
> Hendrik
>
>
>
>
>
> Twill configuration to get it running:
>
> public static void main(String[] args) {
>
>
>
> String zkConnectStr = "machine1:2181";
>
> Configuration conf = new Configuration();
>
> conf.set("yarn.resourcemanager.address", "machine1:8032");
>
> conf.set("fs.defaultFS", "hdfs://machine1:8020");
>
>
>
> YarnConfiguration yc = new YarnConfiguration(conf);
>
> final TwillRunnerService twillRunner = new YarnTwillRunnerService(yc,
> zkConnectStr);
>
> twillRunner.startAndWait();
>
> TwillPreparer twillPreparer = twillRunner.prepare(new
> HelloWorldRunnable(), ResourceSpecification.Builder.with()
>
> .setVirtualCores(1)
>
> .setMemory(64, SizeUnit.MEGA)
>
> .setInstances(1)
>
> .build())
>
> .addLogHandler(
>
> new PrinterLogHandler(new PrintWriter(System.out, true)));
>
> final TwillController controller = twillPreparer.start();
>
>
>
>
>
>
>
> -----Ursprüngliche Nachricht-----
>
> Von: Gary Helmling [mailto:ghelmling@gmail.com]
>
> Gesendet: Freitag, 6. Juni 2014 18:39
>
> An: dev@twill.incubator.apache.org
>
> Betreff: Re: Twill application development requirements for windows
>
>
>
> Hi Hendrik,
>
>
>
> Are you talking about developing a twill application on windows and then
> running it on a cluster? That should be possible with just the Hadoop
> client library jars (dependencies of twill).
>
>
>
> Or are you talking about developing and testing locally on a
> pseudo-distributed Hadoop cluster running on your windows box?
>
>
>
> Please describe in a bit more detail what you want to do.
>
>
>
> Gary
>
>
>
>
>
>
>
> On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke <he...@hotmail.com>
>
> wrote:
>
>
>
> > Hi,
>
> >
>
> >
>
> > is it possible to develop and start a twill application on windows
>
> > without having to install hadoop binaries? Or a node needed on the local
> machine?
>
> >
>
> >
>
> > Ty,
>
> >
>
> > Hendrik
>
>
AW: Twill application development requirements for windows
Posted by helt <he...@outlook.com>.
Hi Gary,
finally the hello world program started and finished successfully.
The issue could be identified as a java version mismatch. The pom.xml was configured to java use 1.7 code and binaries whereas the cluster runs oracles jdk 1.6.
DOH...
Best regards,
Hendrik
-----Ursprüngliche Nachricht-----
Von: helt [mailto:helt@outlook.com]
Gesendet: Sonntag, 15. Juni 2014 14:56
An: dev@twill.incubator.apache.org
Betreff: AW: Twill application development requirements for windows
Hi Gary,
Thank you for your response. In the short run, I want to develop a twill application on windows and then run it on a cluster. In the long run, I probably will develop & test on windows using a local Hadoop, but first, I would like to run hello world as a minimum working example.
Then I can build on this to do my own stuff, but before I do that, I want to go through the examples to verify/fix that my setup works correctly, and that it will run on the cluster correctly…
Which brings me to my actual question:
What do I have to do to get the hello world example to run on the real cluster? I managed to get around a few issues. But it seems like the Hello World example is optimized towards a local cluster installation.
So far I managed to fix (snippet at EOF):
- Connection to yarn fixed by setting the yarn resource manager address to sth != 0.0.0.0:8032
- Connection to hdfs fixed by providing a defaultFS.
- Minimize the apps resource requirements, to make sure that yarn is able to provide the needed resources.
Hello World will start, but not finish, nor print the log messages…
Open issues are therefore (from what I can see from here with my limited experience):
- Fix log4j configurations so it stops complaining about “No appenders” somehow.
Wrap-up: Hello World configuration for running on a real cluster seems to be quite cumbersome and is still unkown. Any advice on this?
Best regards,
Hendrik
Twill configuration to get it running:
public static void main(String[] args) {
String zkConnectStr = "machine1:2181";
Configuration conf = new Configuration();
conf.set("yarn.resourcemanager.address", "machine1:8032");
conf.set("fs.defaultFS", "hdfs://machine1:8020");
YarnConfiguration yc = new YarnConfiguration(conf);
final TwillRunnerService twillRunner = new YarnTwillRunnerService(yc, zkConnectStr);
twillRunner.startAndWait();
TwillPreparer twillPreparer = twillRunner.prepare(new HelloWorldRunnable(), ResourceSpecification.Builder.with()
.setVirtualCores(1)
.setMemory(64, SizeUnit.MEGA)
.setInstances(1)
.build())
.addLogHandler(
new PrinterLogHandler(new PrintWriter(System.out, true)));
final TwillController controller = twillPreparer.start();
-----Ursprüngliche Nachricht-----
Von: Gary Helmling [mailto:ghelmling@gmail.com]
Gesendet: Freitag, 6. Juni 2014 18:39
An: dev@twill.incubator.apache.org
Betreff: Re: Twill application development requirements for windows
Hi Hendrik,
Are you talking about developing a twill application on windows and then running it on a cluster? That should be possible with just the Hadoop client library jars (dependencies of twill).
Or are you talking about developing and testing locally on a pseudo-distributed Hadoop cluster running on your windows box?
Please describe in a bit more detail what you want to do.
Gary
On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke <he...@hotmail.com>
wrote:
> Hi,
>
>
> is it possible to develop and start a twill application on windows
> without having to install hadoop binaries? Or a node needed on the local machine?
>
>
> Ty,
>
> Hendrik
AW: Twill application development requirements for windows
Posted by helt <he...@outlook.com>.
Hi Gary,
Thank you for your response. In the short run, I want to develop a twill application on windows and then run it on a cluster. In the long run, I probably will develop & test on windows using a local Hadoop, but first, I would like to run hello world as a minimum working example.
Then I can build on this to do my own stuff, but before I do that, I want to go through the examples to verify/fix that my setup works correctly, and that it will run on the cluster correctly…
Which brings me to my actual question:
What do I have to do to get the hello world example to run on the real cluster? I managed to get around a few issues. But it seems like the Hello World example is optimized towards a local cluster installation.
So far I managed to fix (snippet at EOF):
- Connection to yarn fixed by setting the yarn resource manager address to sth != 0.0.0.0:8032
- Connection to hdfs fixed by providing a defaultFS.
- Minimize the apps resource requirements, to make sure that yarn is able to provide the needed resources.
Hello World will start, but not finish, nor print the log messages…
Open issues are therefore (from what I can see from here with my limited experience):
- Fix log4j configurations so it stops complaining about “No appenders” somehow.
Wrap-up: Hello World configuration for running on a real cluster seems to be quite cumbersome and is still unkown. Any advice on this?
Best regards,
Hendrik
Twill configuration to get it running:
public static void main(String[] args) {
String zkConnectStr = "machine1:2181";
Configuration conf = new Configuration();
conf.set("yarn.resourcemanager.address", "machine1:8032");
conf.set("fs.defaultFS", "hdfs://machine1:8020");
YarnConfiguration yc = new YarnConfiguration(conf);
final TwillRunnerService twillRunner = new YarnTwillRunnerService(yc, zkConnectStr);
twillRunner.startAndWait();
TwillPreparer twillPreparer = twillRunner.prepare(new HelloWorldRunnable(), ResourceSpecification.Builder.with()
.setVirtualCores(1)
.setMemory(64, SizeUnit.MEGA)
.setInstances(1)
.build())
.addLogHandler(
new PrinterLogHandler(new PrintWriter(System.out, true)));
final TwillController controller = twillPreparer.start();
-----Ursprüngliche Nachricht-----
Von: Gary Helmling [mailto:ghelmling@gmail.com]
Gesendet: Freitag, 6. Juni 2014 18:39
An: dev@twill.incubator.apache.org
Betreff: Re: Twill application development requirements for windows
Hi Hendrik,
Are you talking about developing a twill application on windows and then running it on a cluster? That should be possible with just the Hadoop client library jars (dependencies of twill).
Or are you talking about developing and testing locally on a pseudo-distributed Hadoop cluster running on your windows box?
Please describe in a bit more detail what you want to do.
Gary
On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke <he...@hotmail.com>
wrote:
> Hi,
>
>
> is it possible to develop and start a twill application on windows
> without having to install hadoop binaries? Or a node needed on the local machine?
>
>
> Ty,
>
> Hendrik
Re: Twill application development requirements for windows
Posted by Gary Helmling <gh...@gmail.com>.
Hi Hendrik,
Are you talking about developing a twill application on windows and then
running it on a cluster? That should be possible with just the Hadoop
client library jars (dependencies of twill).
Or are you talking about developing and testing locally on a
pseudo-distributed Hadoop cluster running on your windows box?
Please describe in a bit more detail what you want to do.
Gary
On Thu, Jun 5, 2014 at 11:19 AM, Hendrik Lücke-Tieke <he...@hotmail.com>
wrote:
> Hi,
>
>
> is it possible to develop and start a twill application on windows without
> having to install hadoop binaries? Or a node needed on the local machine?
>
>
> Ty,
>
> Hendrik