You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Alexander Filipchik <af...@gmail.com> on 2015/10/20 20:31:40 UTC
Fwd: How to deploy Hello Samza app on ubuntu 15.04?
Hi,
I was trying to deploy samza hello app on my Ubuntu 15.04. All script
executed fine, but when I submitted the first job to the Yarn it died with:
Stack trace:
2015-10-19 13:19:58 SamzaTaskManager [INFO] Container
container_1445285902612_0001_01_000007 failed with exit code 1 -
Exception from container-launch.
Container id: container_1445285902612_0001_01_000007
Exit code: 1
Stack trace: ExitCodeException exitCode=1:
at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
at org.apache.hadoop.util.Shell.run(Shell.java:455)
at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Container exited with a non-zero exit code 1
stderr:
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
java version "1.8.0_60"
Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
Can't find anything obvious in the logs. Does someone experience it before?
Thank you,
Alex
Re: How to deploy Hello Samza app on ubuntu 15.04?
Posted by Alexander Filipchik <af...@gmail.com>.
Thank you for helping me! That is exactly what is happening.
Alex
On Tue, Oct 20, 2015 at 1:15 PM, Navina Ramesh <nramesh@linkedin.com.invalid
> wrote:
> Hi Alex,
> It looks like a connection issue. I found this in your logs:
>
> 2015-10-20 12:34:55 SamzaContainer [ERROR] Caught exception in process
> loop.
> *java.lang.RuntimeException: Unable to connect to irc.wikimedia.org:6667
> <http://irc.wikimedia.org:6667>.*
> at
> samza.examples.wikipedia.system.WikipediaFeed.start(WikipediaFeed.java:64)
> at
>
> samza.examples.wikipedia.system.WikipediaConsumer.start(WikipediaConsumer.java:62)
> at
>
> org.apache.samza.system.SystemConsumers$$anonfun$start$5.apply(SystemConsumers.scala:159)
> at
>
> org.apache.samza.system.SystemConsumers$$anonfun$start$5.apply(SystemConsumers.scala:159)
> at scala.collection.Iterator$class.foreach(Iterator.scala:727)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
> at
> scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
> at org.apache.samza.system.SystemConsumers.start(SystemConsumers.scala:159)
> at
>
> org.apache.samza.container.SamzaContainer.startConsumers(SamzaContainer.scala:674)
> at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:572)
> at
>
> org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:95)
> at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:69)
> at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
> Caused by: java.net.ConnectException: Connection refused
> at java.net.PlainSocketImpl.socketConnect(Native Method)
> at
>
> java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
> at
>
> java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
> at
> java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
> at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
> at java.net.Socket.connect(Socket.java:589)
> at java.net.Socket.connect(Socket.java:538)
> at java.net.Socket.<init>(Socket.java:434)
> at java.net.Socket.<init>(Socket.java:211)
> at org.schwering.irc.lib.IRCConnection.connect(IRCConnection.java:295)
> at
> samza.examples.wikipedia.system.WikipediaFeed.start(WikipediaFeed.java:62)
> ... 12 more
> 2015-10-20 12:34:55 SamzaContainer [INFO] Shutting down.
> 2015-10-20 12:34:55 SamzaContainer [INFO] Shutting down consumer
> multiplexer.
>
> This can happen if you are behind any firewall.. Can you verify that you
> are able to telnet to this host and port??
>
> Thanks!
> Navina
>
> On Tue, Oct 20, 2015 at 12:41 PM, Alexander Filipchik <
> afilipchik@gmail.com>
> wrote:
>
> > Hi,
> >
> > Thank you for a prompt response! I updated yarn config and gathered some
> > logs. See attached.
> >
> > Alex
> >
> >
> > On Tue, Oct 20, 2015 at 12:17 PM, Navina Ramesh <
> > nramesh@linkedin.com.invalid> wrote:
> >
> >> Hi Alex,
> >> I think the output you pasted is from the application-master.log. Can
> you
> >> send the link to the container logs? specifically, logs for container -
> >> container_1445285902612_0001_01_000007
> >>
> >> If a job fails in Yarn, the working directory usually gets deleted by
> NM's
> >> deletion service, unless you add a property - "
> >> *yarn.nodemanager.delete.debug-delay-sec*" to yarn-site.xml.
> >>
> >> This property indicates how long to keep the working directory of a
> failed
> >> yarn job. If you add this property with a value > 0, then you should be
> >> able to find your container logs as well.
> >> The container logs will have further details on the cause of the
> failure.
> >>
> >> Please give it a try and share your log output.
> >>
> >> HTH,
> >> Navina
> >>
> >>
> >> On Tue, Oct 20, 2015 at 11:31 AM, Alexander Filipchik <
> >> afilipchik@gmail.com>
> >> wrote:
> >>
> >> > Hi,
> >> >
> >> > I was trying to deploy samza hello app on my Ubuntu 15.04. All script
> >> > executed fine, but when I submitted the first job to the Yarn it died
> >> with:
> >> > Stack trace:
> >> >
> >> > 2015-10-19 13:19:58 SamzaTaskManager [INFO] Container
> >> > container_1445285902612_0001_01_000007 failed with exit code 1 -
> >> > Exception from container-launch.
> >> > Container id: container_1445285902612_0001_01_000007
> >> > Exit code: 1
> >> > Stack trace: ExitCodeException exitCode=1:
> >> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> >> > at org.apache.hadoop.util.Shell.run(Shell.java:455)
> >> > at
> >> >
> >>
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> >> > at
> >> >
> >>
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> >> > at
> >> >
> >>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> >> > at
> >> >
> >>
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> >> > at
> >> >
> >>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> >> > at java.lang.Thread.run(Thread.java:745)
> >> >
> >> > Container exited with a non-zero exit code 1
> >> >
> >> > stderr:
> >> >
> >> > Picked up JAVA_TOOL_OPTIONS:
> >> -javaagent:/usr/share/java/jayatanaag.jar
> >> > java version "1.8.0_60"
> >> > Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> >> > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> >> > Picked up JAVA_TOOL_OPTIONS:
> >> -javaagent:/usr/share/java/jayatanaag.jar
> >> >
> >> >
> >> > Can't find anything obvious in the logs. Does someone experience it
> >> before?
> >> >
> >> > Thank you,
> >> > Alex
> >> >
> >>
> >>
> >>
> >> --
> >> Navina R.
> >>
> >
> >
>
>
> --
> Navina R.
>
Re: How to deploy Hello Samza app on ubuntu 15.04?
Posted by Navina Ramesh <nr...@linkedin.com.INVALID>.
Hi Alex,
It looks like a connection issue. I found this in your logs:
2015-10-20 12:34:55 SamzaContainer [ERROR] Caught exception in process loop.
*java.lang.RuntimeException: Unable to connect to irc.wikimedia.org:6667
<http://irc.wikimedia.org:6667>.*
at
samza.examples.wikipedia.system.WikipediaFeed.start(WikipediaFeed.java:64)
at
samza.examples.wikipedia.system.WikipediaConsumer.start(WikipediaConsumer.java:62)
at
org.apache.samza.system.SystemConsumers$$anonfun$start$5.apply(SystemConsumers.scala:159)
at
org.apache.samza.system.SystemConsumers$$anonfun$start$5.apply(SystemConsumers.scala:159)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.MapLike$DefaultValuesIterable.foreach(MapLike.scala:206)
at org.apache.samza.system.SystemConsumers.start(SystemConsumers.scala:159)
at
org.apache.samza.container.SamzaContainer.startConsumers(SamzaContainer.scala:674)
at org.apache.samza.container.SamzaContainer.run(SamzaContainer.scala:572)
at
org.apache.samza.container.SamzaContainer$.safeMain(SamzaContainer.scala:95)
at org.apache.samza.container.SamzaContainer$.main(SamzaContainer.scala:69)
at org.apache.samza.container.SamzaContainer.main(SamzaContainer.scala)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at
java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:350)
at
java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at
java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at java.net.Socket.connect(Socket.java:538)
at java.net.Socket.<init>(Socket.java:434)
at java.net.Socket.<init>(Socket.java:211)
at org.schwering.irc.lib.IRCConnection.connect(IRCConnection.java:295)
at
samza.examples.wikipedia.system.WikipediaFeed.start(WikipediaFeed.java:62)
... 12 more
2015-10-20 12:34:55 SamzaContainer [INFO] Shutting down.
2015-10-20 12:34:55 SamzaContainer [INFO] Shutting down consumer
multiplexer.
This can happen if you are behind any firewall.. Can you verify that you
are able to telnet to this host and port??
Thanks!
Navina
On Tue, Oct 20, 2015 at 12:41 PM, Alexander Filipchik <af...@gmail.com>
wrote:
> Hi,
>
> Thank you for a prompt response! I updated yarn config and gathered some
> logs. See attached.
>
> Alex
>
>
> On Tue, Oct 20, 2015 at 12:17 PM, Navina Ramesh <
> nramesh@linkedin.com.invalid> wrote:
>
>> Hi Alex,
>> I think the output you pasted is from the application-master.log. Can you
>> send the link to the container logs? specifically, logs for container -
>> container_1445285902612_0001_01_000007
>>
>> If a job fails in Yarn, the working directory usually gets deleted by NM's
>> deletion service, unless you add a property - "
>> *yarn.nodemanager.delete.debug-delay-sec*" to yarn-site.xml.
>>
>> This property indicates how long to keep the working directory of a failed
>> yarn job. If you add this property with a value > 0, then you should be
>> able to find your container logs as well.
>> The container logs will have further details on the cause of the failure.
>>
>> Please give it a try and share your log output.
>>
>> HTH,
>> Navina
>>
>>
>> On Tue, Oct 20, 2015 at 11:31 AM, Alexander Filipchik <
>> afilipchik@gmail.com>
>> wrote:
>>
>> > Hi,
>> >
>> > I was trying to deploy samza hello app on my Ubuntu 15.04. All script
>> > executed fine, but when I submitted the first job to the Yarn it died
>> with:
>> > Stack trace:
>> >
>> > 2015-10-19 13:19:58 SamzaTaskManager [INFO] Container
>> > container_1445285902612_0001_01_000007 failed with exit code 1 -
>> > Exception from container-launch.
>> > Container id: container_1445285902612_0001_01_000007
>> > Exit code: 1
>> > Stack trace: ExitCodeException exitCode=1:
>> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
>> > at org.apache.hadoop.util.Shell.run(Shell.java:455)
>> > at
>> >
>> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
>> > at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
>> > at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
>> > at
>> >
>> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
>> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>> > at
>> >
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>> > at java.lang.Thread.run(Thread.java:745)
>> >
>> > Container exited with a non-zero exit code 1
>> >
>> > stderr:
>> >
>> > Picked up JAVA_TOOL_OPTIONS:
>> -javaagent:/usr/share/java/jayatanaag.jar
>> > java version "1.8.0_60"
>> > Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
>> > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
>> > Picked up JAVA_TOOL_OPTIONS:
>> -javaagent:/usr/share/java/jayatanaag.jar
>> >
>> >
>> > Can't find anything obvious in the logs. Does someone experience it
>> before?
>> >
>> > Thank you,
>> > Alex
>> >
>>
>>
>>
>> --
>> Navina R.
>>
>
>
--
Navina R.
Re: How to deploy Hello Samza app on ubuntu 15.04?
Posted by Alexander Filipchik <af...@gmail.com>.
Hi,
Thank you for a prompt response! I updated yarn config and gathered some
logs. See attached.
Alex
On Tue, Oct 20, 2015 at 12:17 PM, Navina Ramesh <
nramesh@linkedin.com.invalid> wrote:
> Hi Alex,
> I think the output you pasted is from the application-master.log. Can you
> send the link to the container logs? specifically, logs for container -
> container_1445285902612_0001_01_000007
>
> If a job fails in Yarn, the working directory usually gets deleted by NM's
> deletion service, unless you add a property - "
> *yarn.nodemanager.delete.debug-delay-sec*" to yarn-site.xml.
> This property indicates how long to keep the working directory of a failed
> yarn job. If you add this property with a value > 0, then you should be
> able to find your container logs as well.
> The container logs will have further details on the cause of the failure.
>
> Please give it a try and share your log output.
>
> HTH,
> Navina
>
>
> On Tue, Oct 20, 2015 at 11:31 AM, Alexander Filipchik <
> afilipchik@gmail.com>
> wrote:
>
> > Hi,
> >
> > I was trying to deploy samza hello app on my Ubuntu 15.04. All script
> > executed fine, but when I submitted the first job to the Yarn it died
> with:
> > Stack trace:
> >
> > 2015-10-19 13:19:58 SamzaTaskManager [INFO] Container
> > container_1445285902612_0001_01_000007 failed with exit code 1 -
> > Exception from container-launch.
> > Container id: container_1445285902612_0001_01_000007
> > Exit code: 1
> > Stack trace: ExitCodeException exitCode=1:
> > at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> > at org.apache.hadoop.util.Shell.run(Shell.java:455)
> > at
> > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> > at
> >
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> >
> > Container exited with a non-zero exit code 1
> >
> > stderr:
> >
> > Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/usr/share/java/jayatanaag.jar
> > java version "1.8.0_60"
> > Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> > Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> > Picked up JAVA_TOOL_OPTIONS:
> -javaagent:/usr/share/java/jayatanaag.jar
> >
> >
> > Can't find anything obvious in the logs. Does someone experience it
> before?
> >
> > Thank you,
> > Alex
> >
>
>
>
> --
> Navina R.
>
Re: How to deploy Hello Samza app on ubuntu 15.04?
Posted by Navina Ramesh <nr...@linkedin.com.INVALID>.
Hi Alex,
I think the output you pasted is from the application-master.log. Can you
send the link to the container logs? specifically, logs for container -
container_1445285902612_0001_01_000007
If a job fails in Yarn, the working directory usually gets deleted by NM's
deletion service, unless you add a property - "
*yarn.nodemanager.delete.debug-delay-sec*" to yarn-site.xml.
This property indicates how long to keep the working directory of a failed
yarn job. If you add this property with a value > 0, then you should be
able to find your container logs as well.
The container logs will have further details on the cause of the failure.
Please give it a try and share your log output.
HTH,
Navina
On Tue, Oct 20, 2015 at 11:31 AM, Alexander Filipchik <af...@gmail.com>
wrote:
> Hi,
>
> I was trying to deploy samza hello app on my Ubuntu 15.04. All script
> executed fine, but when I submitted the first job to the Yarn it died with:
> Stack trace:
>
> 2015-10-19 13:19:58 SamzaTaskManager [INFO] Container
> container_1445285902612_0001_01_000007 failed with exit code 1 -
> Exception from container-launch.
> Container id: container_1445285902612_0001_01_000007
> Exit code: 1
> Stack trace: ExitCodeException exitCode=1:
> at org.apache.hadoop.util.Shell.runCommand(Shell.java:538)
> at org.apache.hadoop.util.Shell.run(Shell.java:455)
> at
> org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:715)
> at
> org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:211)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
> at
> org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
>
> Container exited with a non-zero exit code 1
>
> stderr:
>
> Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
> java version "1.8.0_60"
> Java(TM) SE Runtime Environment (build 1.8.0_60-b27)
> Java HotSpot(TM) 64-Bit Server VM (build 25.60-b23, mixed mode)
> Picked up JAVA_TOOL_OPTIONS: -javaagent:/usr/share/java/jayatanaag.jar
>
>
> Can't find anything obvious in the logs. Does someone experience it before?
>
> Thank you,
> Alex
>
--
Navina R.