You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@samza.apache.org by Malcolm McFarland <mm...@cavulus.com> on 2019/05/07 22:22:55 UTC
Samza tasks aren't starting in YARN containers
Hey folks,
We're having some trouble running Samza under YARN. The YARN
containers are launching fully into the RUNNING state, and I can see
in the node manager logs that the containers are running, but my logs
are showing that Samza is not actually starting inside of the
container. What's really curious is that this is intermittent;
everything will launch fine one time, and then it will do this
RUNNING-but-no-Samza thing the next.
I've been trying to get into the AM UI to see what's going on, but I
see the following error when I try accessing it:
Problem accessing /proxy/application_1557265340810_0002/. Reason:
Cannot assign requested address (Bind failed)
Caused by:
java.net.BindException: Cannot assign requested address (Bind failed)
Has anybody seen this issue with the AM web interface? Also, are there
any other ways that I could introspect the YARN container to try and
deduce what's happening?
Cheers,
Malcolm
--
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of
the contents of this message is prohibited. The information contained
in this message is intended only for the personal and confidential use
of the recipient(s) named above. If you have received this message in
error, please notify the sender immediately and delete the original
message.
Re: Samza tasks aren't starting in YARN containers
Posted by Malcolm McFarland <mm...@cavulus.com>.
Hey Jeremiah,
That's basically what I mean by "all at once"; ie, a script to submits
multiple tasks in sequence without breaks. I'd be surprised if YARN
couldn't handle that, but just thought I'd ask.
It really looks like it's a problem with the way our infrastructure is
configured. I think I need to find which logs to look at in order to trace
the startup/localization steps. If anybody has advice on this, I'd
appreciate it!
Cheers,
Malcolm
On Mon, May 20, 2019 at 9:49 PM Malcolm McFarland <mm...@cavulus.com>
wrote:
>
> Hey Jagadish,
>
> Thanks for the tip. I'm including the stacktrace for a running, stuck
> process. Usually when I try to restart a single process manually, it
> works fine. This brings up the question: are there any
> resource-contention issues with submitting multiple jobs to YARN at
> once?
>
> Also, I was using the same tarball for launch multiple Samza tasks
> (but with different inputs and config). Can YARN handle this, or does
> each running application need to have its own tarball? I ask this
> because when I simulated using different applications by using
> tarballs with different names, the chance of the applications
> launching on the first try seemed to improve.
>
> Cheers,
> Malcolm
>
>
> ====== STACK TRACE FOR STUCK PROCESS ======
>
> "Attach Listener" #8 daemon prio=9 os_prio=0 tid=0x00007ffb6c753800
> nid=0x3a5 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007ffb6c0c2800
> nid=0x2c5 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffb6c0bd000
> nid=0x2c4 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffb6c0ba000
> nid=0x2c3 waiting on condition [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffb6c0b8000
> nid=0x2c2 runnable [0x0000000000000000]
> java.lang.Thread.State: RUNNABLE
>
> "Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffb6c091800
> nid=0x2c1 in Object.wait() [0x00007ffb5cdfc000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000e3000af0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
> - locked <0x00000000e3000af0> (a java.lang.ref.ReferenceQueue$Lock)
> at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
> at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
>
> "Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffb6c08f000
> nid=0x2c0 in Object.wait() [0x00007ffb5cefd000]
> java.lang.Thread.State: WAITING (on object monitor)
> at java.lang.Object.wait(Native Method)
> - waiting on <0x00000000e3008660> (a java.lang.ref.Reference$Lock)
> at java.lang.Object.wait(Object.java:502)
> at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
> - locked <0x00000000e3008660> (a java.lang.ref.Reference$Lock)
> at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
>
> "main" #1 prio=5 os_prio=0 tid=0x00007ffb6c01b800 nid=0x2bc waiting on
> condition [0x00007ffb75876000]
> java.lang.Thread.State: TIMED_WAITING (sleeping)
> at java.lang.Thread.sleep(Native Method)
> at
org.apache.samza.util.ExponentialSleepStrategy$RetryLoopState.sleep(ExponentialSleepStrategy.scala:113)
> at
org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:99)
> at org.apache.samza.util.HttpUtil$.read(HttpUtil.scala:42)
> at org.apache.samza.util.HttpUtil.read(HttpUtil.scala)
> at
org.apache.samza.logging.log4j.StreamAppender.getConfig(StreamAppender.java:272)
> at
org.apache.samza.logging.log4j.StreamAppender.setupSystem(StreamAppender.java:283)
> at
org.apache.samza.logging.log4j.StreamAppender.activateOptions(StreamAppender.java:154)
> at
org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
> at
org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:295)
> at
org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176)
> at
org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191)
> at
org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523)
> at
org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:492)
> - locked <0x00000000e3009558> (a org.apache.log4j.spi.RootLogger)
> at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1006)
> at
org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:872)
> at
org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778)
> at
org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
> at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
> at
org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
> - locked <0x00000000e3009788> (a org.slf4j.impl.Log4jLoggerFactory)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
> at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:265)
> at
org.apache.samza.runtime.AbstractApplicationRunner.<clinit>(AbstractApplicationRunner.java:50)
> <br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon,
> May 13, 2019 at 8:01 PM Jagadish Venkatraman
> <jagadish1989@gmail.com> wrote:<br></div><blockquote
> class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px
> solid rgb(204, 204, 204); padding-left: 1ex;">Can you attach<br>
> <br>
> (1) the full log file for the AM?<br>
> (2) the full log file for the container<br>
> <br>
> > Btw, how do I pull a thread-dump of the stuck container?<br>
> <br>
> Should not be any different than a thread dump for a Java process.<br>
> <br>
> Log in to the machine YARN shows the container as running, locate its<br>
> process-id and use jstack on the PID<br>
> <<a href="
https://confluence.atlassian.com/doc/generating-a-thread-dump-externally-182158040.html#GeneratingaThreadDumpExternally-GeneratingthreaddumpsonLinux
"
> rel="noreferrer"
> target="_blank">https://confluence.atlassian
.<wbr>com/doc/generating-a-thread-du<wbr>mp-externally-182158040.html#G<wbr>eneratingaThreadDumpExternally<wbr>-GeneratingthreaddumpsonLinux</a>><br>
> <br>
> > java.net.ConnectException<br>
> <br>
> Since this is a ConnectException, can you rule out network issues? Can
the<br>
> AM host and container host communicate?<br>
> <br>
> <br>
> <br>
> <br>
> <br>
> On Fri, May 10, 2019 at 9:47 AM Malcolm McFarland <<a
> href="mailto:mmcfarland@cavulus.com"
> target="_blank">mmcfarland@cavulus.com</a>><br>
> wrote:<br>
> <br>
> > Hey all,<br>
> ><br>
> > Logs are working, the AM process is running. I haven't hit a
> "known good<br>
> > version" yet; the deploy seems to be hitting this wall each time,
> which,<br>
> > once again, seems to fail slightly differently each time. Looking
> at the<br>
> > node manager logs, I am seeing this line being repeated:<br>
> ><br>
> > 2019-05-10 05:48:07,705 WARN
> [org.apache.samza.util.Util$:7<wbr>4] Error getting<br>
> > response from Job coordinator server. received IOException: class<br>
> > java.net.ConnectException. Retrying...<br>
> ><br>
> > There's no other information in the log about what is going on.
Does<br>
> > anybody have ideas on this?<br>
> ><br>
> > Btw, how do I pull a thread-dump of the stuck container?<br>
> ><br>
> > Cheers,<br>
> > Malcolm<br>
> ><br>
> ><br>
> > On Tue, May 7, 2019 at 10:48 PM Jagadish Venkatraman <<br>
> > <a href="mailto:jagadish1989@gmail.com"
> target="_blank">jagadish1989@gmail.com</a>><br>
> > wrote:<br>
> ><br>
> > > Malcolm,<br>
> > ><br>
> > > Did the AM-process come up? If so, can you attach its entire
> log-file?<br>
> > ><br>
> > > "> everything will launch fine one time, and then it will
> do this<br>
> > > RUNNING-but-no-Samza thing the next."<br>
> > ><br>
> > > IIUC, you believe your container is not making progress. If
> the issue is<br>
> > > recurs, can you attach a thread-dump & log-file(s) of
> the "stuck"<br>
> > > container?<br>
> > ><br>
> > > "> my logs are showing that Samza is not actually
> starting inside of the<br>
> > > container"<br>
> > ><br>
> > > Can you confirm that logging is actually working? eg: have
> you verified<br>
> > > there is only one log4j binding in your class-path?<br>
> > ><br>
> > > Did anything change on your end? eg: did you upgrade to a
> new Samza<br>
> > > version/ app-version/yarn-version?<br>
> > ><br>
> > > Can you roll-back to a known-good version to better isolate
> the issue?<br>
> > ><br>
> > > Best,<br>
> > > Jagadish<br>
> > ><br>
> > > On Tue, May 7, 2019 at 3:54 PM Malcolm McFarland <<a
> href="mailto:mmcfarland@cavulus.com"
> target="_blank">mmcfarland@cavulus.com</a><br>
> > ><br>
> > > wrote:<br>
> > ><br>
> > > > As a followup to this, here's what I see when the Samza
> app tries to<br>
> > > start;<br>
> > > > it actually seems to be getting to the run-container
> script, and then<br>
> > > > stops:<br>
> > > ><br>
> > > ><br>
> > > > Kafka version : 0.11.0.2<br>
> > > > Kafka commitId : 73be1e1168f91ee2<br>
> > > > Error registering AppInfo mbean<br>
> > > > Started coordinator stream writer.<br>
> > > > sent SetConfig message with key =
> samza.autoscaling.server.url and<br>
> > value<br>
> > > =<br>
> > > > <a href="http://ba6ecb67825e:34205/" rel="noreferrer"
> target="_blank">http://ba6ecb67825e:34205/</a><br>
> > > > Stopping the coordinator stream producer.<br>
> > > > Stopping coordinator stream producer.<br>
> > > > Stopping producer for system: kafka<br>
> > > > Closing the Kafka producer with timeoutMillis =
> 9223372036854775807 ms.<br>
> > > > Webapp is started at (rpc <a
> href="http://ba6ecb67825e:35629/" rel="noreferrer"
> target="_blank">http://ba6ecb67825e:35629/</a>, tracking<br>
> > > > <a href="http://ba6ecb67825e:34151/" rel="noreferrer"
> target="_blank">http://ba6ecb67825e:34151/</a>, coordinator <a
> href="http://ba6ecb67825e:34205/" rel="noreferrer"
> target="_blank">http://ba6ecb67825e:34205/</a>)<br>
> > > > Starting YarnContainerManager.<br>
> > > > Upper bound of the thread pool size is 500<br>
> > > > yarn.client.max-cached-nodeman<wbr>agers-proxies : 0<br>
> > > > Got AM register response. The YARN RM supports
> container requests with<br>
> > > > max-mem: 8192, max-cpu: 32<br>
> > > > Finished starting YarnContainerManager<br>
> > > > Starting the Samza task manager<br>
> > > > Resource Request created for 0 on ANY_HOST at
1557268807252<br>
> > > > Requesting resources on ANY_HOST for container 0<br>
> > > > Making a request for ANY_HOST<br>
> > > > Starting the container allocator thread<br>
> > > > Received new token for :<br>
> > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8032<br>
> > > > Container allocated from RM on<br>
> > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > > Container allocated from RM on<br>
> > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > > Host affinity not enabled. Saving the samzaResource<br>
> > > > container_e39_1557265340810_00<wbr>03_01_000002 in the
> buffer for ANY_HOST<br>
> > > > Returning a buffered resource:<br>
> > container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > > for ANY_HOST from preferred-host buffer.<br>
> > > > Returning a buffered resource:<br>
> > container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > > for ANY_HOST from preferred-host buffer.<br>
> > > > Cancelling request
> SamzaResourceRequest{numCores=<wbr>4, memoryMB=8192,<br>
> > > > preferredHost='ANY_HOST',<br>
> > > requestID='1507e2c5-e437-409b-<wbr>821c-ef505ee19b85',<br>
> > > > containerID=0, requestTimestampMs=15572688072<wbr>52}<br>
> > > > Found available resources on ANY_HOST. Assigning request
for<br>
> > > container_id 0<br>
> > > > with timestamp 1557268807252 to resource<br>
> > > > container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > > Received launch request for 0 on hostname<br>
> > > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > > Got available container ID (0) for container:
> Container: [ContainerId:<br>
> > > > container_e39_1557265340810_00<wbr>03_01_000002,
NodeId:<br>
> > > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8032,
> NodeHttpAddress:<br>
> > > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8088,
> Resource:<br>
> > <memory:8192,<br>
> > > > vCores:4>, Priority: 1, Token: Token { kind:
> ContainerToken, service:<br>
> > > > <a href="http://10.60.31.121:8032" rel="noreferrer"
> target="_blank">10.60.31.121:8032</a> }, ]<br>
> > > > In runContainer in util: fwkPath=
> ;cmdPath=./__package/;jobLib=<br>
> > > > Container ID 0 using command
> ./__package//bin/run-container<wbr>.sh<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Malcolm<br>
> > > ><br>
> > > ><br>
> > > > On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <<br>
> > <a href="mailto:mmcfarland@cavulus.com"
> target="_blank">mmcfarland@cavulus.com</a><br>
> > > ><br>
> > > > wrote:<br>
> > > ><br>
> > > > > Hey folks,<br>
> > > > ><br>
> > > > > We're having some trouble running Samza under
> YARN. The YARN<br>
> > > > > containers are launching fully into the RUNNING
> state, and I can see<br>
> > > > > in the node manager logs that the containers are
> running, but my logs<br>
> > > > > are showing that Samza is not actually starting
> inside of the<br>
> > > > > container. What's really curious is that this is
> intermittent;<br>
> > > > > everything will launch fine one time, and then it
> will do this<br>
> > > > > RUNNING-but-no-Samza thing the next.<br>
> > > > ><br>
> > > > > I've been trying to get into the AM UI to see
> what's going on, but I<br>
> > > > > see the following error when I try accessing it:<br>
> > > > ><br>
> > > > > Problem accessing
> /proxy/application_15572653408<wbr>10_0002/. Reason:<br>
> > > > > Cannot assign requested address
> (Bind failed)<br>
> > > > > Caused by:<br>
> > > > > java.net.BindException: Cannot assign requested
> address (Bind failed)<br>
> > > > ><br>
> > > > > Has anybody seen this issue with the AM web
> interface? Also, are<br>
> > there<br>
> > > > > any other ways that I could introspect the YARN
> container to try and<br>
> > > > > deduce what's happening?<br>
> > > > ><br>
> > > > > Cheers,<br>
> > > > > Malcolm<br>
> > > > ><br>
> > > > ><br>
> > > > > --<br>
> > > > > Malcolm McFarland<br>
> > > > > Cavulus<br>
> > > > ><br>
> > > > ><br>
> > > > > This correspondence is from HealthPlanCRM, LLC,
> d/b/a Cavulus. Any<br>
> > > > > unauthorized or improper disclosure, copying,
> distribution, or use of<br>
> > > > > the contents of this message is prohibited. The
> information contained<br>
> > > > > in this message is intended only for the personal
> and confidential<br>
> > use<br>
> > > > > of the recipient(s) named above. If you have
> received this message in<br>
> > > > > error, please notify the sender immediately and
> delete the original<br>
> > > > > message.<br>
> > > > ><br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Malcolm McFarland<br>
> > > > Cavulus<br>
> > > > 1-800-760-6915<br>
> > > > <a href="mailto:mmcfarland@cavulus.com"
> target="_blank">mmcfarland@cavulus.com</a><br>
> > > ><br>
> > > ><br>
> > > > This correspondence is from HealthPlanCRM, LLC, d/b/a
> Cavulus. Any<br>
> > > > unauthorized or improper disclosure, copying,
> distribution, or use of<br>
> > the<br>
> > > > contents of this message is prohibited. The information
> contained in<br>
> > this<br>
> > > > message is intended only for the personal and
> confidential use of the<br>
> > > > recipient(s) named above. If you have received this
> message in error,<br>
> > > > please notify the sender immediately and delete the
> original message.<br>
> > > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Jagadish V,<br>
> > > Graduate Student,<br>
> > > Department of Computer Science,<br>
> > > Stanford University<br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Malcolm McFarland<br>
> > Cavulus<br>
> ><br>
> ><br>
> > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus.
Any<br>
> > unauthorized or improper disclosure, copying, distribution, or
> use of the<br>
> > contents of this message is prohibited. The information contained
> in this<br>
> > message is intended only for the personal and confidential use of
the<br>
> > recipient(s) named above. If you have received this message in
error,<br>
> > please notify the sender immediately and delete the original
message.<br>
> ><br>
> <br>
> <br>
> -- <br>
> Jagadish V,<br>
> Graduate Student,<br>
> Department of Computer Science,<br>
> Stanford University<br>
> </blockquote></div><br clear="all"><div><br></div>-- <br><div
> dir="ltr" class="gmail_signature">Malcolm
> McFarland<br>Cavulus<br>1-800-760-6915<br><a
> href="mailto:mmcfarland@cavulus.com"
> target="_blank">mmcfarland@cavulus.com</a><br><br><br>This
> correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of
> the contents of this message is prohibited. The information contained
> in this message is intended only for the personal and confidential use
> of the recipient(s) named above. If you have received this message in
> error, please notify the sender immediately and delete the original
> message.<br></div>
--
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
Re: Samza tasks aren't starting in YARN containers
Posted by Jeremiah Adams <JA...@helixeducation.com>.
I have had no issues in running multiple tasks with the same tarball. I had four discrete Task implementations in the same jar running in production for months. Like you say - I had different inputs and configs for each Task but I don't imagine that this is necessary as far as Yarn is concerned. What exactly do you mean by 'submitting multiple jobs at once'? I start ours one at a time though do not verify they are 'Running' until I've submitted all three instances of the jar.
Jeremiah Adams
Software Engineer
www.helixeducation.com
Blog | Twitter | Facebook | LinkedIn
________________________________________
From: Malcolm McFarland <mm...@cavulus.com>
Sent: Monday, May 20, 2019 10:49 PM
To: dev@samza.apache.org
Subject: Re: Samza tasks aren't starting in YARN containers
Hey Jagadish,
Thanks for the tip. I'm including the stacktrace for a running, stuck
process. Usually when I try to restart a single process manually, it
works fine. This brings up the question: are there any
resource-contention issues with submitting multiple jobs to YARN at
once?
Also, I was using the same tarball for launch multiple Samza tasks
(but with different inputs and config). Can YARN handle this, or does
each running application need to have its own tarball? I ask this
because when I simulated using different applications by using
tarballs with different names, the chance of the applications
launching on the first try seemed to improve.
Cheers,
Malcolm
====== STACK TRACE FOR STUCK PROCESS ======
"Attach Listener" #8 daemon prio=9 os_prio=0 tid=0x00007ffb6c753800
nid=0x3a5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007ffb6c0c2800
nid=0x2c5 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffb6c0bd000
nid=0x2c4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffb6c0ba000
nid=0x2c3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffb6c0b8000
nid=0x2c2 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffb6c091800
nid=0x2c1 in Object.wait() [0x00007ffb5cdfc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e3000af0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000e3000af0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffb6c08f000
nid=0x2c0 in Object.wait() [0x00007ffb5cefd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e3008660> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e3008660> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007ffb6c01b800 nid=0x2bc waiting on
condition [0x00007ffb75876000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.samza.util.ExponentialSleepStrategy$RetryLoopState.sleep(ExponentialSleepStrategy.scala:113)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:99)
at org.apache.samza.util.HttpUtil$.read(HttpUtil.scala:42)
at org.apache.samza.util.HttpUtil.read(HttpUtil.scala)
at org.apache.samza.logging.log4j.StreamAppender.getConfig(StreamAppender.java:272)
at org.apache.samza.logging.log4j.StreamAppender.setupSystem(StreamAppender.java:283)
at org.apache.samza.logging.log4j.StreamAppender.activateOptions(StreamAppender.java:154)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:295)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191)
at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523)
at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:492)
- locked <0x00000000e3009558> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1006)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:872)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
- locked <0x00000000e3009788> (a org.slf4j.impl.Log4jLoggerFactory)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:265)
at org.apache.samza.runtime.AbstractApplicationRunner.<clinit>(AbstractApplicationRunner.java:50)
<br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon,
May 13, 2019 at 8:01 PM Jagadish Venkatraman
<jagadish1989@gmail.com> wrote:<br></div><blockquote
class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px
solid rgb(204, 204, 204); padding-left: 1ex;">Can you attach<br>
<br>
(1) the full log file for the AM?<br>
(2) the full log file for the container<br>
<br>
> Btw, how do I pull a thread-dump of the stuck container?<br>
<br>
Should not be any different than a thread dump for a Java process.<br>
<br>
Log in to the machine YARN shows the container as running, locate its<br>
process-id and use jstack on the PID<br>
<<a href="https://url.emailprotection.link/?bv19JQxVPAnuXqbKa8qDo-59VEELNEAitFviaVJS_IQaPHhxTYUFNsCxmZR6uHeOrFDpCD4f6_UwFnwg9fQuEumfd-i6TGJgYLKfLHdb7cxsdZ22xt-MrB2BPIBwsyhT4052f2gw3yGUg1neU21vB-uXKwdNFqG-9Ki4crSafVWrmmOyxMisk9QZ7G5_Se5Pi0MoA7BN7UvjHHl7gvzgYFA~~"
rel="noreferrer"
target="_blank">https://url.emailprotection.link/?bluiPjlOmEhuM9ClmzMDtLwRgO4XPMGXFvImTbyeiaK436U6V5zY7YJp-lwukY08HGPZwbtZ6LNOraLhkznzZxg~~.<wbr>com/doc/generating-a-thread-du<wbr>mp-externally-182158040.html#G<wbr>eneratingaThreadDumpExternally<wbr>-GeneratingthreaddumpsonLinux</a>><br>
<br>
> java.net.ConnectException<br>
<br>
Since this is a ConnectException, can you rule out network issues? Can the<br>
AM host and container host communicate?<br>
<br>
<br>
<br>
<br>
<br>
On Fri, May 10, 2019 at 9:47 AM Malcolm McFarland <<a
href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a>><br>
wrote:<br>
<br>
> Hey all,<br>
><br>
> Logs are working, the AM process is running. I haven't hit a
"known good<br>
> version" yet; the deploy seems to be hitting this wall each time,
which,<br>
> once again, seems to fail slightly differently each time. Looking
at the<br>
> node manager logs, I am seeing this line being repeated:<br>
><br>
> 2019-05-10 05:48:07,705 WARN
[org.apache.samza.util.Util$:7<wbr>4] Error getting<br>
> response from Job coordinator server. received IOException: class<br>
> java.net.ConnectException. Retrying...<br>
><br>
> There's no other information in the log about what is going on. Does<br>
> anybody have ideas on this?<br>
><br>
> Btw, how do I pull a thread-dump of the stuck container?<br>
><br>
> Cheers,<br>
> Malcolm<br>
><br>
><br>
> On Tue, May 7, 2019 at 10:48 PM Jagadish Venkatraman <<br>
> <a href="mailto:jagadish1989@gmail.com"
target="_blank">jagadish1989@gmail.com</a>><br>
> wrote:<br>
><br>
> > Malcolm,<br>
> ><br>
> > Did the AM-process come up? If so, can you attach its entire
log-file?<br>
> ><br>
> > "> everything will launch fine one time, and then it will
do this<br>
> > RUNNING-but-no-Samza thing the next."<br>
> ><br>
> > IIUC, you believe your container is not making progress. If
the issue is<br>
> > recurs, can you attach a thread-dump & log-file(s) of
the "stuck"<br>
> > container?<br>
> ><br>
> > "> my logs are showing that Samza is not actually
starting inside of the<br>
> > container"<br>
> ><br>
> > Can you confirm that logging is actually working? eg: have
you verified<br>
> > there is only one log4j binding in your class-path?<br>
> ><br>
> > Did anything change on your end? eg: did you upgrade to a
new Samza<br>
> > version/ app-version/yarn-version?<br>
> ><br>
> > Can you roll-back to a known-good version to better isolate
the issue?<br>
> ><br>
> > Best,<br>
> > Jagadish<br>
> ><br>
> > On Tue, May 7, 2019 at 3:54 PM Malcolm McFarland <<a
href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br>
> ><br>
> > wrote:<br>
> ><br>
> > > As a followup to this, here's what I see when the Samza
app tries to<br>
> > start;<br>
> > > it actually seems to be getting to the run-container
script, and then<br>
> > > stops:<br>
> > ><br>
> > ><br>
> > > Kafka version : 0.11.0.2<br>
> > > Kafka commitId : 73be1e1168f91ee2<br>
> > > Error registering AppInfo mbean<br>
> > > Started coordinator stream writer.<br>
> > > sent SetConfig message with key =
samza.autoscaling.server.url and<br>
> value<br>
> > =<br>
> > > <a href="https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1KnpFlm2kVzoqZuBzHEoAQz70QI07ZGzURDkc_xf_p2H9I9qHuPuLx46WiZ8SNNfNVAw~~" rel="noreferrer"
target="_blank">https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1KnpFlm2kVzoqZuBzHEoAQz70QI07ZGzURDkc_xf_p2H9I9qHuPuLx46WiZ8SNNfNVAw~~</a><br>
> > > Stopping the coordinator stream producer.<br>
> > > Stopping coordinator stream producer.<br>
> > > Stopping producer for system: kafka<br>
> > > Closing the Kafka producer with timeoutMillis =
9223372036854775807 ms.<br>
> > > Webapp is started at (rpc <a
href="https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1Knuqbc-sAzjrnvhpuB4SrjDOf6NKkg1BRh77V27VsFqDH0o3QbEu97-ybPJ1E8ztCqw~~" rel="noreferrer"
target="_blank">https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1Knuqbc-sAzjrnvhpuB4SrjDOf6NKkg1BRh77V27VsFqDH0o3QbEu97-ybPJ1E8ztCqw~~</a>, tracking<br>
> > > <a href="https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1KnitQCADD0WEAUAUuBwKhz9MTFVvIaGCUSIfHDcI36G2czUE8FH40u5x5oGCwNh5TiA~~" rel="noreferrer"
target="_blank">https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1KnitQCADD0WEAUAUuBwKhz9MTFVvIaGCUSIfHDcI36G2czUE8FH40u5x5oGCwNh5TiA~~</a>, coordinator <a
href="https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1KnpFlm2kVzoqZuBzHEoAQz70QI07ZGzURDkc_xf_p2H9I9qHuPuLx46WiZ8SNNfNVAw~~" rel="noreferrer"
target="_blank">https://url.emailprotection.link/?bv29xFXUGIevOCVmVcj1KnpFlm2kVzoqZuBzHEoAQz70QI07ZGzURDkc_xf_p2H9I9qHuPuLx46WiZ8SNNfNVAw~~</a>)<br>
> > > Starting YarnContainerManager.<br>
> > > Upper bound of the thread pool size is 500<br>
> > > yarn.client.max-cached-nodeman<wbr>agers-proxies : 0<br>
> > > Got AM register response. The YARN RM supports
container requests with<br>
> > > max-mem: 8192, max-cpu: 32<br>
> > > Finished starting YarnContainerManager<br>
> > > Starting the Samza task manager<br>
> > > Resource Request created for 0 on ANY_HOST at 1557268807252<br>
> > > Requesting resources on ANY_HOST for container 0<br>
> > > Making a request for ANY_HOST<br>
> > > Starting the container allocator thread<br>
> > > Received new token for :<br>
> ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8032<br>
> > > Container allocated from RM on<br>
> ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > Container allocated from RM on<br>
> ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > Host affinity not enabled. Saving the samzaResource<br>
> > > container_e39_1557265340810_00<wbr>03_01_000002 in the
buffer for ANY_HOST<br>
> > > Returning a buffered resource:<br>
> container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > for ANY_HOST from preferred-host buffer.<br>
> > > Returning a buffered resource:<br>
> container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > for ANY_HOST from preferred-host buffer.<br>
> > > Cancelling request
SamzaResourceRequest{numCores=<wbr>4, memoryMB=8192,<br>
> > > preferredHost='ANY_HOST',<br>
> > requestID='1507e2c5-e437-409b-<wbr>821c-ef505ee19b85',<br>
> > > containerID=0, requestTimestampMs=15572688072<wbr>52}<br>
> > > Found available resources on ANY_HOST. Assigning request for<br>
> > container_id 0<br>
> > > with timestamp 1557268807252 to resource<br>
> > > container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > Received launch request for 0 on hostname<br>
> > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > Got available container ID (0) for container:
Container: [ContainerId:<br>
> > > container_e39_1557265340810_00<wbr>03_01_000002, NodeId:<br>
> > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8032,
NodeHttpAddress:<br>
> > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8088,
Resource:<br>
> <memory:8192,<br>
> > > vCores:4>, Priority: 1, Token: Token { kind:
ContainerToken, service:<br>
> > > <a href="https://url.emailprotection.link/?bpf_NAktnnt5c2CVuAOvWCWC40fi_aKHeEBVbSr2mXbWKp8DUNmNXZkQYh2ItS6HGJQ69sC-nQyfV1wej03kDzA~~" rel="noreferrer"
target="_blank">10.60.31.121:8032</a> }, ]<br>
> > > In runContainer in util: fwkPath=
;cmdPath=./__package/;jobLib=<br>
> > > Container ID 0 using command
./__package//bin/run-container<wbr>.sh<br>
> > ><br>
> > > Cheers,<br>
> > > Malcolm<br>
> > ><br>
> > ><br>
> > > On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <<br>
> <a href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br>
> > ><br>
> > > wrote:<br>
> > ><br>
> > > > Hey folks,<br>
> > > ><br>
> > > > We're having some trouble running Samza under
YARN. The YARN<br>
> > > > containers are launching fully into the RUNNING
state, and I can see<br>
> > > > in the node manager logs that the containers are
running, but my logs<br>
> > > > are showing that Samza is not actually starting
inside of the<br>
> > > > container. What's really curious is that this is
intermittent;<br>
> > > > everything will launch fine one time, and then it
will do this<br>
> > > > RUNNING-but-no-Samza thing the next.<br>
> > > ><br>
> > > > I've been trying to get into the AM UI to see
what's going on, but I<br>
> > > > see the following error when I try accessing it:<br>
> > > ><br>
> > > > Problem accessing
/proxy/application_15572653408<wbr>10_0002/. Reason:<br>
> > > > Cannot assign requested address
(Bind failed)<br>
> > > > Caused by:<br>
> > > > java.net.BindException: Cannot assign requested
address (Bind failed)<br>
> > > ><br>
> > > > Has anybody seen this issue with the AM web
interface? Also, are<br>
> there<br>
> > > > any other ways that I could introspect the YARN
container to try and<br>
> > > > deduce what's happening?<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Malcolm<br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Malcolm McFarland<br>
> > > > Cavulus<br>
> > > ><br>
> > > ><br>
> > > > This correspondence is from HealthPlanCRM, LLC,
d/b/a Cavulus. Any<br>
> > > > unauthorized or improper disclosure, copying,
distribution, or use of<br>
> > > > the contents of this message is prohibited. The
information contained<br>
> > > > in this message is intended only for the personal
and confidential<br>
> use<br>
> > > > of the recipient(s) named above. If you have
received this message in<br>
> > > > error, please notify the sender immediately and
delete the original<br>
> > > > message.<br>
> > > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Malcolm McFarland<br>
> > > Cavulus<br>
> > > 1-800-760-6915<br>
> > > <a href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br>
> > ><br>
> > ><br>
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a
Cavulus. Any<br>
> > > unauthorized or improper disclosure, copying,
distribution, or use of<br>
> the<br>
> > > contents of this message is prohibited. The information
contained in<br>
> this<br>
> > > message is intended only for the personal and
confidential use of the<br>
> > > recipient(s) named above. If you have received this
message in error,<br>
> > > please notify the sender immediately and delete the
original message.<br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Jagadish V,<br>
> > Graduate Student,<br>
> > Department of Computer Science,<br>
> > Stanford University<br>
> ><br>
><br>
><br>
> --<br>
> Malcolm McFarland<br>
> Cavulus<br>
><br>
><br>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any<br>
> unauthorized or improper disclosure, copying, distribution, or
use of the<br>
> contents of this message is prohibited. The information contained
in this<br>
> message is intended only for the personal and confidential use of the<br>
> recipient(s) named above. If you have received this message in error,<br>
> please notify the sender immediately and delete the original message.<br>
><br>
<br>
<br>
-- <br>
Jagadish V,<br>
Graduate Student,<br>
Department of Computer Science,<br>
Stanford University<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div
dir="ltr" class="gmail_signature">Malcolm
McFarland<br>Cavulus<br>1-800-760-6915<br><a
href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br><br><br>This
correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of
the contents of this message is prohibited. The information contained
in this message is intended only for the personal and confidential use
of the recipient(s) named above. If you have received this message in
error, please notify the sender immediately and delete the original
message.<br></div>
Re: Samza tasks aren't starting in YARN containers
Posted by Malcolm McFarland <mm...@cavulus.com>.
Hey Jagadish,
Thanks for the tip. I'm including the stacktrace for a running, stuck
process. Usually when I try to restart a single process manually, it
works fine. This brings up the question: are there any
resource-contention issues with submitting multiple jobs to YARN at
once?
Also, I was using the same tarball for launch multiple Samza tasks
(but with different inputs and config). Can YARN handle this, or does
each running application need to have its own tarball? I ask this
because when I simulated using different applications by using
tarballs with different names, the chance of the applications
launching on the first try seemed to improve.
Cheers,
Malcolm
====== STACK TRACE FOR STUCK PROCESS ======
"Attach Listener" #8 daemon prio=9 os_prio=0 tid=0x00007ffb6c753800
nid=0x3a5 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Service Thread" #7 daemon prio=9 os_prio=0 tid=0x00007ffb6c0c2800
nid=0x2c5 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C1 CompilerThread1" #6 daemon prio=9 os_prio=0 tid=0x00007ffb6c0bd000
nid=0x2c4 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"C2 CompilerThread0" #5 daemon prio=9 os_prio=0 tid=0x00007ffb6c0ba000
nid=0x2c3 waiting on condition [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Signal Dispatcher" #4 daemon prio=9 os_prio=0 tid=0x00007ffb6c0b8000
nid=0x2c2 runnable [0x0000000000000000]
java.lang.Thread.State: RUNNABLE
"Finalizer" #3 daemon prio=8 os_prio=0 tid=0x00007ffb6c091800
nid=0x2c1 in Object.wait() [0x00007ffb5cdfc000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e3000af0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:144)
- locked <0x00000000e3000af0> (a java.lang.ref.ReferenceQueue$Lock)
at java.lang.ref.ReferenceQueue.remove(ReferenceQueue.java:165)
at java.lang.ref.Finalizer$FinalizerThread.run(Finalizer.java:216)
"Reference Handler" #2 daemon prio=10 os_prio=0 tid=0x00007ffb6c08f000
nid=0x2c0 in Object.wait() [0x00007ffb5cefd000]
java.lang.Thread.State: WAITING (on object monitor)
at java.lang.Object.wait(Native Method)
- waiting on <0x00000000e3008660> (a java.lang.ref.Reference$Lock)
at java.lang.Object.wait(Object.java:502)
at java.lang.ref.Reference.tryHandlePending(Reference.java:191)
- locked <0x00000000e3008660> (a java.lang.ref.Reference$Lock)
at java.lang.ref.Reference$ReferenceHandler.run(Reference.java:153)
"main" #1 prio=5 os_prio=0 tid=0x00007ffb6c01b800 nid=0x2bc waiting on
condition [0x00007ffb75876000]
java.lang.Thread.State: TIMED_WAITING (sleeping)
at java.lang.Thread.sleep(Native Method)
at org.apache.samza.util.ExponentialSleepStrategy$RetryLoopState.sleep(ExponentialSleepStrategy.scala:113)
at org.apache.samza.util.ExponentialSleepStrategy.run(ExponentialSleepStrategy.scala:99)
at org.apache.samza.util.HttpUtil$.read(HttpUtil.scala:42)
at org.apache.samza.util.HttpUtil.read(HttpUtil.scala)
at org.apache.samza.logging.log4j.StreamAppender.getConfig(StreamAppender.java:272)
at org.apache.samza.logging.log4j.StreamAppender.setupSystem(StreamAppender.java:283)
at org.apache.samza.logging.log4j.StreamAppender.activateOptions(StreamAppender.java:154)
at org.apache.log4j.config.PropertySetter.activate(PropertySetter.java:307)
at org.apache.log4j.xml.DOMConfigurator.parseAppender(DOMConfigurator.java:295)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByName(DOMConfigurator.java:176)
at org.apache.log4j.xml.DOMConfigurator.findAppenderByReference(DOMConfigurator.java:191)
at org.apache.log4j.xml.DOMConfigurator.parseChildrenOfLoggerElement(DOMConfigurator.java:523)
at org.apache.log4j.xml.DOMConfigurator.parseRoot(DOMConfigurator.java:492)
- locked <0x00000000e3009558> (a org.apache.log4j.spi.RootLogger)
at org.apache.log4j.xml.DOMConfigurator.parse(DOMConfigurator.java:1006)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:872)
at org.apache.log4j.xml.DOMConfigurator.doConfigure(DOMConfigurator.java:778)
at org.apache.log4j.helpers.OptionConverter.selectAndConfigure(OptionConverter.java:526)
at org.apache.log4j.LogManager.<clinit>(LogManager.java:127)
at org.slf4j.impl.Log4jLoggerFactory.getLogger(Log4jLoggerFactory.java:73)
- locked <0x00000000e3009788> (a org.slf4j.impl.Log4jLoggerFactory)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:253)
at org.slf4j.LoggerFactory.getLogger(LoggerFactory.java:265)
at org.apache.samza.runtime.AbstractApplicationRunner.<clinit>(AbstractApplicationRunner.java:50)
<br><div class="gmail_quote"><div dir="ltr" class="gmail_attr">On Mon,
May 13, 2019 at 8:01 PM Jagadish Venkatraman
<jagadish1989@gmail.com> wrote:<br></div><blockquote
class="gmail_quote" style="margin: 0px 0px 0px 0.8ex; border-left: 1px
solid rgb(204, 204, 204); padding-left: 1ex;">Can you attach<br>
<br>
(1) the full log file for the AM?<br>
(2) the full log file for the container<br>
<br>
> Btw, how do I pull a thread-dump of the stuck container?<br>
<br>
Should not be any different than a thread dump for a Java process.<br>
<br>
Log in to the machine YARN shows the container as running, locate its<br>
process-id and use jstack on the PID<br>
<<a href="https://confluence.atlassian.com/doc/generating-a-thread-dump-externally-182158040.html#GeneratingaThreadDumpExternally-GeneratingthreaddumpsonLinux"
rel="noreferrer"
target="_blank">https://confluence.atlassian.<wbr>com/doc/generating-a-thread-du<wbr>mp-externally-182158040.html#G<wbr>eneratingaThreadDumpExternally<wbr>-GeneratingthreaddumpsonLinux</a>><br>
<br>
> java.net.ConnectException<br>
<br>
Since this is a ConnectException, can you rule out network issues? Can the<br>
AM host and container host communicate?<br>
<br>
<br>
<br>
<br>
<br>
On Fri, May 10, 2019 at 9:47 AM Malcolm McFarland <<a
href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a>><br>
wrote:<br>
<br>
> Hey all,<br>
><br>
> Logs are working, the AM process is running. I haven't hit a
"known good<br>
> version" yet; the deploy seems to be hitting this wall each time,
which,<br>
> once again, seems to fail slightly differently each time. Looking
at the<br>
> node manager logs, I am seeing this line being repeated:<br>
><br>
> 2019-05-10 05:48:07,705 WARN
[org.apache.samza.util.Util$:7<wbr>4] Error getting<br>
> response from Job coordinator server. received IOException: class<br>
> java.net.ConnectException. Retrying...<br>
><br>
> There's no other information in the log about what is going on. Does<br>
> anybody have ideas on this?<br>
><br>
> Btw, how do I pull a thread-dump of the stuck container?<br>
><br>
> Cheers,<br>
> Malcolm<br>
><br>
><br>
> On Tue, May 7, 2019 at 10:48 PM Jagadish Venkatraman <<br>
> <a href="mailto:jagadish1989@gmail.com"
target="_blank">jagadish1989@gmail.com</a>><br>
> wrote:<br>
><br>
> > Malcolm,<br>
> ><br>
> > Did the AM-process come up? If so, can you attach its entire
log-file?<br>
> ><br>
> > "> everything will launch fine one time, and then it will
do this<br>
> > RUNNING-but-no-Samza thing the next."<br>
> ><br>
> > IIUC, you believe your container is not making progress. If
the issue is<br>
> > recurs, can you attach a thread-dump & log-file(s) of
the "stuck"<br>
> > container?<br>
> ><br>
> > "> my logs are showing that Samza is not actually
starting inside of the<br>
> > container"<br>
> ><br>
> > Can you confirm that logging is actually working? eg: have
you verified<br>
> > there is only one log4j binding in your class-path?<br>
> ><br>
> > Did anything change on your end? eg: did you upgrade to a
new Samza<br>
> > version/ app-version/yarn-version?<br>
> ><br>
> > Can you roll-back to a known-good version to better isolate
the issue?<br>
> ><br>
> > Best,<br>
> > Jagadish<br>
> ><br>
> > On Tue, May 7, 2019 at 3:54 PM Malcolm McFarland <<a
href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br>
> ><br>
> > wrote:<br>
> ><br>
> > > As a followup to this, here's what I see when the Samza
app tries to<br>
> > start;<br>
> > > it actually seems to be getting to the run-container
script, and then<br>
> > > stops:<br>
> > ><br>
> > ><br>
> > > Kafka version : 0.11.0.2<br>
> > > Kafka commitId : 73be1e1168f91ee2<br>
> > > Error registering AppInfo mbean<br>
> > > Started coordinator stream writer.<br>
> > > sent SetConfig message with key =
samza.autoscaling.server.url and<br>
> value<br>
> > =<br>
> > > <a href="http://ba6ecb67825e:34205/" rel="noreferrer"
target="_blank">http://ba6ecb67825e:34205/</a><br>
> > > Stopping the coordinator stream producer.<br>
> > > Stopping coordinator stream producer.<br>
> > > Stopping producer for system: kafka<br>
> > > Closing the Kafka producer with timeoutMillis =
9223372036854775807 ms.<br>
> > > Webapp is started at (rpc <a
href="http://ba6ecb67825e:35629/" rel="noreferrer"
target="_blank">http://ba6ecb67825e:35629/</a>, tracking<br>
> > > <a href="http://ba6ecb67825e:34151/" rel="noreferrer"
target="_blank">http://ba6ecb67825e:34151/</a>, coordinator <a
href="http://ba6ecb67825e:34205/" rel="noreferrer"
target="_blank">http://ba6ecb67825e:34205/</a>)<br>
> > > Starting YarnContainerManager.<br>
> > > Upper bound of the thread pool size is 500<br>
> > > yarn.client.max-cached-nodeman<wbr>agers-proxies : 0<br>
> > > Got AM register response. The YARN RM supports
container requests with<br>
> > > max-mem: 8192, max-cpu: 32<br>
> > > Finished starting YarnContainerManager<br>
> > > Starting the Samza task manager<br>
> > > Resource Request created for 0 on ANY_HOST at 1557268807252<br>
> > > Requesting resources on ANY_HOST for container 0<br>
> > > Making a request for ANY_HOST<br>
> > > Starting the container allocator thread<br>
> > > Received new token for :<br>
> ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8032<br>
> > > Container allocated from RM on<br>
> ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > Container allocated from RM on<br>
> ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > Host affinity not enabled. Saving the samzaResource<br>
> > > container_e39_1557265340810_00<wbr>03_01_000002 in the
buffer for ANY_HOST<br>
> > > Returning a buffered resource:<br>
> container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > for ANY_HOST from preferred-host buffer.<br>
> > > Returning a buffered resource:<br>
> container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > for ANY_HOST from preferred-host buffer.<br>
> > > Cancelling request
SamzaResourceRequest{numCores=<wbr>4, memoryMB=8192,<br>
> > > preferredHost='ANY_HOST',<br>
> > requestID='1507e2c5-e437-409b-<wbr>821c-ef505ee19b85',<br>
> > > containerID=0, requestTimestampMs=15572688072<wbr>52}<br>
> > > Found available resources on ANY_HOST. Assigning request for<br>
> > container_id 0<br>
> > > with timestamp 1557268807252 to resource<br>
> > > container_e39_1557265340810_00<wbr>03_01_000002<br>
> > > Received launch request for 0 on hostname<br>
> > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal<br>
> > > Got available container ID (0) for container:
Container: [ContainerId:<br>
> > > container_e39_1557265340810_00<wbr>03_01_000002, NodeId:<br>
> > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8032,
NodeHttpAddress:<br>
> > > ip-10-60-31-121.us-west-2.comp<wbr>ute.internal:8088,
Resource:<br>
> <memory:8192,<br>
> > > vCores:4>, Priority: 1, Token: Token { kind:
ContainerToken, service:<br>
> > > <a href="http://10.60.31.121:8032" rel="noreferrer"
target="_blank">10.60.31.121:8032</a> }, ]<br>
> > > In runContainer in util: fwkPath=
;cmdPath=./__package/;jobLib=<br>
> > > Container ID 0 using command
./__package//bin/run-container<wbr>.sh<br>
> > ><br>
> > > Cheers,<br>
> > > Malcolm<br>
> > ><br>
> > ><br>
> > > On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <<br>
> <a href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br>
> > ><br>
> > > wrote:<br>
> > ><br>
> > > > Hey folks,<br>
> > > ><br>
> > > > We're having some trouble running Samza under
YARN. The YARN<br>
> > > > containers are launching fully into the RUNNING
state, and I can see<br>
> > > > in the node manager logs that the containers are
running, but my logs<br>
> > > > are showing that Samza is not actually starting
inside of the<br>
> > > > container. What's really curious is that this is
intermittent;<br>
> > > > everything will launch fine one time, and then it
will do this<br>
> > > > RUNNING-but-no-Samza thing the next.<br>
> > > ><br>
> > > > I've been trying to get into the AM UI to see
what's going on, but I<br>
> > > > see the following error when I try accessing it:<br>
> > > ><br>
> > > > Problem accessing
/proxy/application_15572653408<wbr>10_0002/. Reason:<br>
> > > > Cannot assign requested address
(Bind failed)<br>
> > > > Caused by:<br>
> > > > java.net.BindException: Cannot assign requested
address (Bind failed)<br>
> > > ><br>
> > > > Has anybody seen this issue with the AM web
interface? Also, are<br>
> there<br>
> > > > any other ways that I could introspect the YARN
container to try and<br>
> > > > deduce what's happening?<br>
> > > ><br>
> > > > Cheers,<br>
> > > > Malcolm<br>
> > > ><br>
> > > ><br>
> > > > --<br>
> > > > Malcolm McFarland<br>
> > > > Cavulus<br>
> > > ><br>
> > > ><br>
> > > > This correspondence is from HealthPlanCRM, LLC,
d/b/a Cavulus. Any<br>
> > > > unauthorized or improper disclosure, copying,
distribution, or use of<br>
> > > > the contents of this message is prohibited. The
information contained<br>
> > > > in this message is intended only for the personal
and confidential<br>
> use<br>
> > > > of the recipient(s) named above. If you have
received this message in<br>
> > > > error, please notify the sender immediately and
delete the original<br>
> > > > message.<br>
> > > ><br>
> > ><br>
> > ><br>
> > > --<br>
> > > Malcolm McFarland<br>
> > > Cavulus<br>
> > > 1-800-760-6915<br>
> > > <a href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br>
> > ><br>
> > ><br>
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a
Cavulus. Any<br>
> > > unauthorized or improper disclosure, copying,
distribution, or use of<br>
> the<br>
> > > contents of this message is prohibited. The information
contained in<br>
> this<br>
> > > message is intended only for the personal and
confidential use of the<br>
> > > recipient(s) named above. If you have received this
message in error,<br>
> > > please notify the sender immediately and delete the
original message.<br>
> > ><br>
> ><br>
> ><br>
> > --<br>
> > Jagadish V,<br>
> > Graduate Student,<br>
> > Department of Computer Science,<br>
> > Stanford University<br>
> ><br>
><br>
><br>
> --<br>
> Malcolm McFarland<br>
> Cavulus<br>
><br>
><br>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any<br>
> unauthorized or improper disclosure, copying, distribution, or
use of the<br>
> contents of this message is prohibited. The information contained
in this<br>
> message is intended only for the personal and confidential use of the<br>
> recipient(s) named above. If you have received this message in error,<br>
> please notify the sender immediately and delete the original message.<br>
><br>
<br>
<br>
-- <br>
Jagadish V,<br>
Graduate Student,<br>
Department of Computer Science,<br>
Stanford University<br>
</blockquote></div><br clear="all"><div><br></div>-- <br><div
dir="ltr" class="gmail_signature">Malcolm
McFarland<br>Cavulus<br>1-800-760-6915<br><a
href="mailto:mmcfarland@cavulus.com"
target="_blank">mmcfarland@cavulus.com</a><br><br><br>This
correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of
the contents of this message is prohibited. The information contained
in this message is intended only for the personal and confidential use
of the recipient(s) named above. If you have received this message in
error, please notify the sender immediately and delete the original
message.<br></div>
Re: Samza tasks aren't starting in YARN containers
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Can you attach
(1) the full log file for the AM?
(2) the full log file for the container
> Btw, how do I pull a thread-dump of the stuck container?
Should not be any different than a thread dump for a Java process.
Log in to the machine YARN shows the container as running, locate its
process-id and use jstack on the PID
<https://confluence.atlassian.com/doc/generating-a-thread-dump-externally-182158040.html#GeneratingaThreadDumpExternally-GeneratingthreaddumpsonLinux>
> java.net.ConnectException
Since this is a ConnectException, can you rule out network issues? Can the
AM host and container host communicate?
On Fri, May 10, 2019 at 9:47 AM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Hey all,
>
> Logs are working, the AM process is running. I haven't hit a "known good
> version" yet; the deploy seems to be hitting this wall each time, which,
> once again, seems to fail slightly differently each time. Looking at the
> node manager logs, I am seeing this line being repeated:
>
> 2019-05-10 05:48:07,705 WARN [org.apache.samza.util.Util$:74] Error getting
> response from Job coordinator server. received IOException: class
> java.net.ConnectException. Retrying...
>
> There's no other information in the log about what is going on. Does
> anybody have ideas on this?
>
> Btw, how do I pull a thread-dump of the stuck container?
>
> Cheers,
> Malcolm
>
>
> On Tue, May 7, 2019 at 10:48 PM Jagadish Venkatraman <
> jagadish1989@gmail.com>
> wrote:
>
> > Malcolm,
> >
> > Did the AM-process come up? If so, can you attach its entire log-file?
> >
> > "> everything will launch fine one time, and then it will do this
> > RUNNING-but-no-Samza thing the next."
> >
> > IIUC, you believe your container is not making progress. If the issue is
> > recurs, can you attach a thread-dump & log-file(s) of the "stuck"
> > container?
> >
> > "> my logs are showing that Samza is not actually starting inside of the
> > container"
> >
> > Can you confirm that logging is actually working? eg: have you verified
> > there is only one log4j binding in your class-path?
> >
> > Did anything change on your end? eg: did you upgrade to a new Samza
> > version/ app-version/yarn-version?
> >
> > Can you roll-back to a known-good version to better isolate the issue?
> >
> > Best,
> > Jagadish
> >
> > On Tue, May 7, 2019 at 3:54 PM Malcolm McFarland <mmcfarland@cavulus.com
> >
> > wrote:
> >
> > > As a followup to this, here's what I see when the Samza app tries to
> > start;
> > > it actually seems to be getting to the run-container script, and then
> > > stops:
> > >
> > >
> > > Kafka version : 0.11.0.2
> > > Kafka commitId : 73be1e1168f91ee2
> > > Error registering AppInfo mbean
> > > Started coordinator stream writer.
> > > sent SetConfig message with key = samza.autoscaling.server.url and
> value
> > =
> > > http://ba6ecb67825e:34205/
> > > Stopping the coordinator stream producer.
> > > Stopping coordinator stream producer.
> > > Stopping producer for system: kafka
> > > Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
> > > Webapp is started at (rpc http://ba6ecb67825e:35629/, tracking
> > > http://ba6ecb67825e:34151/, coordinator http://ba6ecb67825e:34205/)
> > > Starting YarnContainerManager.
> > > Upper bound of the thread pool size is 500
> > > yarn.client.max-cached-nodemanagers-proxies : 0
> > > Got AM register response. The YARN RM supports container requests with
> > > max-mem: 8192, max-cpu: 32
> > > Finished starting YarnContainerManager
> > > Starting the Samza task manager
> > > Resource Request created for 0 on ANY_HOST at 1557268807252
> > > Requesting resources on ANY_HOST for container 0
> > > Making a request for ANY_HOST
> > > Starting the container allocator thread
> > > Received new token for :
> ip-10-60-31-121.us-west-2.compute.internal:8032
> > > Container allocated from RM on
> ip-10-60-31-121.us-west-2.compute.internal
> > > Container allocated from RM on
> ip-10-60-31-121.us-west-2.compute.internal
> > > Host affinity not enabled. Saving the samzaResource
> > > container_e39_1557265340810_0003_01_000002 in the buffer for ANY_HOST
> > > Returning a buffered resource:
> container_e39_1557265340810_0003_01_000002
> > > for ANY_HOST from preferred-host buffer.
> > > Returning a buffered resource:
> container_e39_1557265340810_0003_01_000002
> > > for ANY_HOST from preferred-host buffer.
> > > Cancelling request SamzaResourceRequest{numCores=4, memoryMB=8192,
> > > preferredHost='ANY_HOST',
> > requestID='1507e2c5-e437-409b-821c-ef505ee19b85',
> > > containerID=0, requestTimestampMs=1557268807252}
> > > Found available resources on ANY_HOST. Assigning request for
> > container_id 0
> > > with timestamp 1557268807252 to resource
> > > container_e39_1557265340810_0003_01_000002
> > > Received launch request for 0 on hostname
> > > ip-10-60-31-121.us-west-2.compute.internal
> > > Got available container ID (0) for container: Container: [ContainerId:
> > > container_e39_1557265340810_0003_01_000002, NodeId:
> > > ip-10-60-31-121.us-west-2.compute.internal:8032, NodeHttpAddress:
> > > ip-10-60-31-121.us-west-2.compute.internal:8088, Resource:
> <memory:8192,
> > > vCores:4>, Priority: 1, Token: Token { kind: ContainerToken, service:
> > > 10.60.31.121:8032 }, ]
> > > In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
> > > Container ID 0 using command ./__package//bin/run-container.sh
> > >
> > > Cheers,
> > > Malcolm
> > >
> > >
> > > On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <
> mmcfarland@cavulus.com
> > >
> > > wrote:
> > >
> > > > Hey folks,
> > > >
> > > > We're having some trouble running Samza under YARN. The YARN
> > > > containers are launching fully into the RUNNING state, and I can see
> > > > in the node manager logs that the containers are running, but my logs
> > > > are showing that Samza is not actually starting inside of the
> > > > container. What's really curious is that this is intermittent;
> > > > everything will launch fine one time, and then it will do this
> > > > RUNNING-but-no-Samza thing the next.
> > > >
> > > > I've been trying to get into the AM UI to see what's going on, but I
> > > > see the following error when I try accessing it:
> > > >
> > > > Problem accessing /proxy/application_1557265340810_0002/. Reason:
> > > > Cannot assign requested address (Bind failed)
> > > > Caused by:
> > > > java.net.BindException: Cannot assign requested address (Bind failed)
> > > >
> > > > Has anybody seen this issue with the AM web interface? Also, are
> there
> > > > any other ways that I could introspect the YARN container to try and
> > > > deduce what's happening?
> > > >
> > > > Cheers,
> > > > Malcolm
> > > >
> > > >
> > > > --
> > > > Malcolm McFarland
> > > > Cavulus
> > > >
> > > >
> > > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > > unauthorized or improper disclosure, copying, distribution, or use of
> > > > the contents of this message is prohibited. The information contained
> > > > in this message is intended only for the personal and confidential
> use
> > > > of the recipient(s) named above. If you have received this message in
> > > > error, please notify the sender immediately and delete the original
> > > > message.
> > > >
> > >
> > >
> > > --
> > > Malcolm McFarland
> > > Cavulus
> > > 1-800-760-6915
> > > mmcfarland@cavulus.com
> > >
> > >
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > unauthorized or improper disclosure, copying, distribution, or use of
> the
> > > contents of this message is prohibited. The information contained in
> this
> > > message is intended only for the personal and confidential use of the
> > > recipient(s) named above. If you have received this message in error,
> > > please notify the sender immediately and delete the original message.
> > >
> >
> >
> > --
> > Jagadish V,
> > Graduate Student,
> > Department of Computer Science,
> > Stanford University
> >
>
>
> --
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: Samza tasks aren't starting in YARN containers
Posted by Malcolm McFarland <mm...@cavulus.com>.
Hey all,
Logs are working, the AM process is running. I haven't hit a "known good
version" yet; the deploy seems to be hitting this wall each time, which,
once again, seems to fail slightly differently each time. Looking at the
node manager logs, I am seeing this line being repeated:
2019-05-10 05:48:07,705 WARN [org.apache.samza.util.Util$:74] Error getting
response from Job coordinator server. received IOException: class
java.net.ConnectException. Retrying...
There's no other information in the log about what is going on. Does
anybody have ideas on this?
Btw, how do I pull a thread-dump of the stuck container?
Cheers,
Malcolm
On Tue, May 7, 2019 at 10:48 PM Jagadish Venkatraman <ja...@gmail.com>
wrote:
> Malcolm,
>
> Did the AM-process come up? If so, can you attach its entire log-file?
>
> "> everything will launch fine one time, and then it will do this
> RUNNING-but-no-Samza thing the next."
>
> IIUC, you believe your container is not making progress. If the issue is
> recurs, can you attach a thread-dump & log-file(s) of the "stuck"
> container?
>
> "> my logs are showing that Samza is not actually starting inside of the
> container"
>
> Can you confirm that logging is actually working? eg: have you verified
> there is only one log4j binding in your class-path?
>
> Did anything change on your end? eg: did you upgrade to a new Samza
> version/ app-version/yarn-version?
>
> Can you roll-back to a known-good version to better isolate the issue?
>
> Best,
> Jagadish
>
> On Tue, May 7, 2019 at 3:54 PM Malcolm McFarland <mm...@cavulus.com>
> wrote:
>
> > As a followup to this, here's what I see when the Samza app tries to
> start;
> > it actually seems to be getting to the run-container script, and then
> > stops:
> >
> >
> > Kafka version : 0.11.0.2
> > Kafka commitId : 73be1e1168f91ee2
> > Error registering AppInfo mbean
> > Started coordinator stream writer.
> > sent SetConfig message with key = samza.autoscaling.server.url and value
> =
> > http://ba6ecb67825e:34205/
> > Stopping the coordinator stream producer.
> > Stopping coordinator stream producer.
> > Stopping producer for system: kafka
> > Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
> > Webapp is started at (rpc http://ba6ecb67825e:35629/, tracking
> > http://ba6ecb67825e:34151/, coordinator http://ba6ecb67825e:34205/)
> > Starting YarnContainerManager.
> > Upper bound of the thread pool size is 500
> > yarn.client.max-cached-nodemanagers-proxies : 0
> > Got AM register response. The YARN RM supports container requests with
> > max-mem: 8192, max-cpu: 32
> > Finished starting YarnContainerManager
> > Starting the Samza task manager
> > Resource Request created for 0 on ANY_HOST at 1557268807252
> > Requesting resources on ANY_HOST for container 0
> > Making a request for ANY_HOST
> > Starting the container allocator thread
> > Received new token for : ip-10-60-31-121.us-west-2.compute.internal:8032
> > Container allocated from RM on ip-10-60-31-121.us-west-2.compute.internal
> > Container allocated from RM on ip-10-60-31-121.us-west-2.compute.internal
> > Host affinity not enabled. Saving the samzaResource
> > container_e39_1557265340810_0003_01_000002 in the buffer for ANY_HOST
> > Returning a buffered resource: container_e39_1557265340810_0003_01_000002
> > for ANY_HOST from preferred-host buffer.
> > Returning a buffered resource: container_e39_1557265340810_0003_01_000002
> > for ANY_HOST from preferred-host buffer.
> > Cancelling request SamzaResourceRequest{numCores=4, memoryMB=8192,
> > preferredHost='ANY_HOST',
> requestID='1507e2c5-e437-409b-821c-ef505ee19b85',
> > containerID=0, requestTimestampMs=1557268807252}
> > Found available resources on ANY_HOST. Assigning request for
> container_id 0
> > with timestamp 1557268807252 to resource
> > container_e39_1557265340810_0003_01_000002
> > Received launch request for 0 on hostname
> > ip-10-60-31-121.us-west-2.compute.internal
> > Got available container ID (0) for container: Container: [ContainerId:
> > container_e39_1557265340810_0003_01_000002, NodeId:
> > ip-10-60-31-121.us-west-2.compute.internal:8032, NodeHttpAddress:
> > ip-10-60-31-121.us-west-2.compute.internal:8088, Resource: <memory:8192,
> > vCores:4>, Priority: 1, Token: Token { kind: ContainerToken, service:
> > 10.60.31.121:8032 }, ]
> > In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
> > Container ID 0 using command ./__package//bin/run-container.sh
> >
> > Cheers,
> > Malcolm
> >
> >
> > On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <mmcfarland@cavulus.com
> >
> > wrote:
> >
> > > Hey folks,
> > >
> > > We're having some trouble running Samza under YARN. The YARN
> > > containers are launching fully into the RUNNING state, and I can see
> > > in the node manager logs that the containers are running, but my logs
> > > are showing that Samza is not actually starting inside of the
> > > container. What's really curious is that this is intermittent;
> > > everything will launch fine one time, and then it will do this
> > > RUNNING-but-no-Samza thing the next.
> > >
> > > I've been trying to get into the AM UI to see what's going on, but I
> > > see the following error when I try accessing it:
> > >
> > > Problem accessing /proxy/application_1557265340810_0002/. Reason:
> > > Cannot assign requested address (Bind failed)
> > > Caused by:
> > > java.net.BindException: Cannot assign requested address (Bind failed)
> > >
> > > Has anybody seen this issue with the AM web interface? Also, are there
> > > any other ways that I could introspect the YARN container to try and
> > > deduce what's happening?
> > >
> > > Cheers,
> > > Malcolm
> > >
> > >
> > > --
> > > Malcolm McFarland
> > > Cavulus
> > >
> > >
> > > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > > unauthorized or improper disclosure, copying, distribution, or use of
> > > the contents of this message is prohibited. The information contained
> > > in this message is intended only for the personal and confidential use
> > > of the recipient(s) named above. If you have received this message in
> > > error, please notify the sender immediately and delete the original
> > > message.
> > >
> >
> >
> > --
> > Malcolm McFarland
> > Cavulus
> > 1-800-760-6915
> > mmcfarland@cavulus.com
> >
> >
> > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > unauthorized or improper disclosure, copying, distribution, or use of the
> > contents of this message is prohibited. The information contained in this
> > message is intended only for the personal and confidential use of the
> > recipient(s) named above. If you have received this message in error,
> > please notify the sender immediately and delete the original message.
> >
>
>
> --
> Jagadish V,
> Graduate Student,
> Department of Computer Science,
> Stanford University
>
--
Malcolm McFarland
Cavulus
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.
Re: Samza tasks aren't starting in YARN containers
Posted by Jagadish Venkatraman <ja...@gmail.com>.
Malcolm,
Did the AM-process come up? If so, can you attach its entire log-file?
"> everything will launch fine one time, and then it will do this
RUNNING-but-no-Samza thing the next."
IIUC, you believe your container is not making progress. If the issue is
recurs, can you attach a thread-dump & log-file(s) of the "stuck"
container?
"> my logs are showing that Samza is not actually starting inside of the
container"
Can you confirm that logging is actually working? eg: have you verified
there is only one log4j binding in your class-path?
Did anything change on your end? eg: did you upgrade to a new Samza
version/ app-version/yarn-version?
Can you roll-back to a known-good version to better isolate the issue?
Best,
Jagadish
On Tue, May 7, 2019 at 3:54 PM Malcolm McFarland <mm...@cavulus.com>
wrote:
> As a followup to this, here's what I see when the Samza app tries to start;
> it actually seems to be getting to the run-container script, and then
> stops:
>
>
> Kafka version : 0.11.0.2
> Kafka commitId : 73be1e1168f91ee2
> Error registering AppInfo mbean
> Started coordinator stream writer.
> sent SetConfig message with key = samza.autoscaling.server.url and value =
> http://ba6ecb67825e:34205/
> Stopping the coordinator stream producer.
> Stopping coordinator stream producer.
> Stopping producer for system: kafka
> Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
> Webapp is started at (rpc http://ba6ecb67825e:35629/, tracking
> http://ba6ecb67825e:34151/, coordinator http://ba6ecb67825e:34205/)
> Starting YarnContainerManager.
> Upper bound of the thread pool size is 500
> yarn.client.max-cached-nodemanagers-proxies : 0
> Got AM register response. The YARN RM supports container requests with
> max-mem: 8192, max-cpu: 32
> Finished starting YarnContainerManager
> Starting the Samza task manager
> Resource Request created for 0 on ANY_HOST at 1557268807252
> Requesting resources on ANY_HOST for container 0
> Making a request for ANY_HOST
> Starting the container allocator thread
> Received new token for : ip-10-60-31-121.us-west-2.compute.internal:8032
> Container allocated from RM on ip-10-60-31-121.us-west-2.compute.internal
> Container allocated from RM on ip-10-60-31-121.us-west-2.compute.internal
> Host affinity not enabled. Saving the samzaResource
> container_e39_1557265340810_0003_01_000002 in the buffer for ANY_HOST
> Returning a buffered resource: container_e39_1557265340810_0003_01_000002
> for ANY_HOST from preferred-host buffer.
> Returning a buffered resource: container_e39_1557265340810_0003_01_000002
> for ANY_HOST from preferred-host buffer.
> Cancelling request SamzaResourceRequest{numCores=4, memoryMB=8192,
> preferredHost='ANY_HOST', requestID='1507e2c5-e437-409b-821c-ef505ee19b85',
> containerID=0, requestTimestampMs=1557268807252}
> Found available resources on ANY_HOST. Assigning request for container_id 0
> with timestamp 1557268807252 to resource
> container_e39_1557265340810_0003_01_000002
> Received launch request for 0 on hostname
> ip-10-60-31-121.us-west-2.compute.internal
> Got available container ID (0) for container: Container: [ContainerId:
> container_e39_1557265340810_0003_01_000002, NodeId:
> ip-10-60-31-121.us-west-2.compute.internal:8032, NodeHttpAddress:
> ip-10-60-31-121.us-west-2.compute.internal:8088, Resource: <memory:8192,
> vCores:4>, Priority: 1, Token: Token { kind: ContainerToken, service:
> 10.60.31.121:8032 }, ]
> In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
> Container ID 0 using command ./__package//bin/run-container.sh
>
> Cheers,
> Malcolm
>
>
> On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <mm...@cavulus.com>
> wrote:
>
> > Hey folks,
> >
> > We're having some trouble running Samza under YARN. The YARN
> > containers are launching fully into the RUNNING state, and I can see
> > in the node manager logs that the containers are running, but my logs
> > are showing that Samza is not actually starting inside of the
> > container. What's really curious is that this is intermittent;
> > everything will launch fine one time, and then it will do this
> > RUNNING-but-no-Samza thing the next.
> >
> > I've been trying to get into the AM UI to see what's going on, but I
> > see the following error when I try accessing it:
> >
> > Problem accessing /proxy/application_1557265340810_0002/. Reason:
> > Cannot assign requested address (Bind failed)
> > Caused by:
> > java.net.BindException: Cannot assign requested address (Bind failed)
> >
> > Has anybody seen this issue with the AM web interface? Also, are there
> > any other ways that I could introspect the YARN container to try and
> > deduce what's happening?
> >
> > Cheers,
> > Malcolm
> >
> >
> > --
> > Malcolm McFarland
> > Cavulus
> >
> >
> > This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> > unauthorized or improper disclosure, copying, distribution, or use of
> > the contents of this message is prohibited. The information contained
> > in this message is intended only for the personal and confidential use
> > of the recipient(s) named above. If you have received this message in
> > error, please notify the sender immediately and delete the original
> > message.
> >
>
>
> --
> Malcolm McFarland
> Cavulus
> 1-800-760-6915
> mmcfarland@cavulus.com
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
--
Jagadish V,
Graduate Student,
Department of Computer Science,
Stanford University
Re: Samza tasks aren't starting in YARN containers
Posted by Malcolm McFarland <mm...@cavulus.com>.
As a followup to this, here's what I see when the Samza app tries to start;
it actually seems to be getting to the run-container script, and then stops:
Kafka version : 0.11.0.2
Kafka commitId : 73be1e1168f91ee2
Error registering AppInfo mbean
Started coordinator stream writer.
sent SetConfig message with key = samza.autoscaling.server.url and value =
http://ba6ecb67825e:34205/
Stopping the coordinator stream producer.
Stopping coordinator stream producer.
Stopping producer for system: kafka
Closing the Kafka producer with timeoutMillis = 9223372036854775807 ms.
Webapp is started at (rpc http://ba6ecb67825e:35629/, tracking
http://ba6ecb67825e:34151/, coordinator http://ba6ecb67825e:34205/)
Starting YarnContainerManager.
Upper bound of the thread pool size is 500
yarn.client.max-cached-nodemanagers-proxies : 0
Got AM register response. The YARN RM supports container requests with
max-mem: 8192, max-cpu: 32
Finished starting YarnContainerManager
Starting the Samza task manager
Resource Request created for 0 on ANY_HOST at 1557268807252
Requesting resources on ANY_HOST for container 0
Making a request for ANY_HOST
Starting the container allocator thread
Received new token for : ip-10-60-31-121.us-west-2.compute.internal:8032
Container allocated from RM on ip-10-60-31-121.us-west-2.compute.internal
Container allocated from RM on ip-10-60-31-121.us-west-2.compute.internal
Host affinity not enabled. Saving the samzaResource
container_e39_1557265340810_0003_01_000002 in the buffer for ANY_HOST
Returning a buffered resource: container_e39_1557265340810_0003_01_000002
for ANY_HOST from preferred-host buffer.
Returning a buffered resource: container_e39_1557265340810_0003_01_000002
for ANY_HOST from preferred-host buffer.
Cancelling request SamzaResourceRequest{numCores=4, memoryMB=8192,
preferredHost='ANY_HOST', requestID='1507e2c5-e437-409b-821c-ef505ee19b85',
containerID=0, requestTimestampMs=1557268807252}
Found available resources on ANY_HOST. Assigning request for container_id 0
with timestamp 1557268807252 to resource
container_e39_1557265340810_0003_01_000002
Received launch request for 0 on hostname
ip-10-60-31-121.us-west-2.compute.internal
Got available container ID (0) for container: Container: [ContainerId:
container_e39_1557265340810_0003_01_000002, NodeId:
ip-10-60-31-121.us-west-2.compute.internal:8032, NodeHttpAddress:
ip-10-60-31-121.us-west-2.compute.internal:8088, Resource: <memory:8192,
vCores:4>, Priority: 1, Token: Token { kind: ContainerToken, service:
10.60.31.121:8032 }, ]
In runContainer in util: fwkPath= ;cmdPath=./__package/;jobLib=
Container ID 0 using command ./__package//bin/run-container.sh
Cheers,
Malcolm
On Tue, May 7, 2019 at 3:22 PM Malcolm McFarland <mm...@cavulus.com>
wrote:
> Hey folks,
>
> We're having some trouble running Samza under YARN. The YARN
> containers are launching fully into the RUNNING state, and I can see
> in the node manager logs that the containers are running, but my logs
> are showing that Samza is not actually starting inside of the
> container. What's really curious is that this is intermittent;
> everything will launch fine one time, and then it will do this
> RUNNING-but-no-Samza thing the next.
>
> I've been trying to get into the AM UI to see what's going on, but I
> see the following error when I try accessing it:
>
> Problem accessing /proxy/application_1557265340810_0002/. Reason:
> Cannot assign requested address (Bind failed)
> Caused by:
> java.net.BindException: Cannot assign requested address (Bind failed)
>
> Has anybody seen this issue with the AM web interface? Also, are there
> any other ways that I could introspect the YARN container to try and
> deduce what's happening?
>
> Cheers,
> Malcolm
>
>
> --
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of
> the contents of this message is prohibited. The information contained
> in this message is intended only for the personal and confidential use
> of the recipient(s) named above. If you have received this message in
> error, please notify the sender immediately and delete the original
> message.
>
--
Malcolm McFarland
Cavulus
1-800-760-6915
mmcfarland@cavulus.com
This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.