apache-spark 1.3.0 and yarn integration and spring-boot as a container

[Nirav Patel <np...@xactlycorp.com>]

Hi, I was running spark application as a query service (much like
spark-shell but within my servlet container provided by spring-boot) with
spark 1.0.2 and standalone mode. Now After upgrading to spark 1.3.1 and
trying to use Yarn instead of standalone cluster things going south for me.
I created uber jar with all dependencies (spark-core, spark-yarn,
spring-boot) and tried to deploy my application.

    15/07/29 11:19:26 INFO client.RMProxy: Connecting to ResourceManager at
/0.0.0.0:8032

    15/07/29 11:19:27 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 0 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)

    15/07/29 11:19:28 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 1 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)

    15/07/29 11:19:29 INFO ipc.Client: Retrying connect to server:
0.0.0.0/0.0.0.0:8032. Already tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
MILLISECONDS)

I also tried to exclude spark-yarn dependencies and supplied it during
runtime but same exception. We use MapR distribution and they said it's not
possible to run spark jobs on yarn without using spark-submit script. I can
try to launch my webapp using that script as my build artifact is
spring-boot jar (not war) but that just doesn't feel right. I should be
able to init service from my container not other way around.

-- 


[image: What's New with Xactly] <http://www.xactlycorp.com/email-click/>

 <https://www.nyse.com/quote/XNYS:XTLY>  [image: LinkedIn] 
<https://www.linkedin.com/company/xactly-corporation>  [image: Twitter] 
<https://twitter.com/Xactly>  [image: Facebook] 
<https://www.facebook.com/XactlyCorp>  [image: YouTube] 
<http://www.youtube.com/xactlycorporation>

Re: apache-spark 1.3.0 and yarn integration and spring-boot as a container

[Steve Loughran <st...@hortonworks.com>]

you need to fix your configuration so that the resource manager hostname/URL is set...that address there is the "listen on any port" path
On 30 Jul 2015, at 10:47, Nirav Patel <np...@xactlycorp.com>> wrote:

15/07/29 11:19:26 INFO client.RMProxy: Connecting to ResourceManager at /0.0.0.0:8032<http://0.0.0.0:8032/>

Future versions of Hadoop will give you more meaningful diagnostics messages here ( HADOOP-9657<https://issues.apache.org/jira/browse/HADOOP-9657> ), but the fix is the same: get your RM hostname right.

-steve