You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Matt Narrell <ma...@gmail.com> on 2014/08/19 01:07:08 UTC
spark-submit with HA YARN
Hello,
I have an HA enabled YARN cluster with two resource mangers. When submitting jobs via “spark-submit —master yarn-cluster”. It appears that the driver is looking explicitly for the "yarn.resourcemanager.address” property rather than round robin-ing through the resource managers via the “yarn.client.failover-proxy-provider” property set to “org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider”
If I explicitly set the “yarn.resourcemanager.address” to the active resource manager, jobs will submit fine.
Is there a manner to set “spark-submit —master yarn-cluster” to respect the failover proxy?
Thanks in advance,
Matt
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit with HA YARN
Posted by Matt Narrell <ma...@gmail.com>.
Ok Marcelo,
Thanks for the quick and thorough replies. I’ll keep an eye on these tickets and the mailing list to see how things move along.
mn
On Aug 20, 2014, at 1:33 PM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Hi,
>
> On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell <ma...@gmail.com> wrote:
>> Specifying the driver-class-path yields behavior like
>> https://issues.apache.org/jira/browse/SPARK-2420 and
>> https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a
>> can of worms here if I also need to replace the guava dependencies.
>
> There's a PR open for the Guava dependency. I'm not sure at the moment
> if there will be a need to shade any other dependencies.
>
>> Wouldn’t calling “./make-distribution.sh —skip-java-test —hadoop 2.4.1
>> —with-yarn —tgz” include the appropriate versions of the hadoop libs into
>> the spark jar?
>
> Yes. I was suggesting a workaround that just excludes any Hadoop
> classes from your Spark jar; that way you're guaranteed to use
> whatever Hadoop classes you have in your cluster. :-)
>
>> I’m trying to rebuild using the hadoop-provided profile, but I’m getting
>> several build errors. Is this sufficient: mvm -Phadoop-provided clean
>> package -Phadoop-2.4 -Pyarn -Dyarn.version=2.4.1 -Dhadoop.version=2.4.1
>> -DskipTests
>
> Hmm, I'm pretty sure that's worked for me in the past, but then I
> haven't tried that in a while.
>
> --
> Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit with HA YARN
Posted by Marcelo Vanzin <va...@cloudera.com>.
Hi,
On Wed, Aug 20, 2014 at 11:59 AM, Matt Narrell <ma...@gmail.com> wrote:
> Specifying the driver-class-path yields behavior like
> https://issues.apache.org/jira/browse/SPARK-2420 and
> https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a
> can of worms here if I also need to replace the guava dependencies.
There's a PR open for the Guava dependency. I'm not sure at the moment
if there will be a need to shade any other dependencies.
> Wouldn’t calling “./make-distribution.sh —skip-java-test —hadoop 2.4.1
> —with-yarn —tgz” include the appropriate versions of the hadoop libs into
> the spark jar?
Yes. I was suggesting a workaround that just excludes any Hadoop
classes from your Spark jar; that way you're guaranteed to use
whatever Hadoop classes you have in your cluster. :-)
> I’m trying to rebuild using the hadoop-provided profile, but I’m getting
> several build errors. Is this sufficient: mvm -Phadoop-provided clean
> package -Phadoop-2.4 -Pyarn -Dyarn.version=2.4.1 -Dhadoop.version=2.4.1
> -DskipTests
Hmm, I'm pretty sure that's worked for me in the past, but then I
haven't tried that in a while.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit with HA YARN
Posted by Matt Narrell <ma...@gmail.com>.
Marcelo,
Specifying the driver-class-path yields behavior like https://issues.apache.org/jira/browse/SPARK-2420 and https://issues.apache.org/jira/browse/SPARK-2848 It feels like opening a can of worms here if I also need to replace the guava dependencies.
Wouldn’t calling “./make-distribution.sh —skip-java-test —hadoop 2.4.1 —with-yarn —tgz” include the appropriate versions of the hadoop libs into the spark jar?
I’m trying to rebuild using the hadoop-provided profile, but I’m getting several build errors. Is this sufficient: mvm -Phadoop-provided clean package -Phadoop-2.4 -Pyarn -Dyarn.version=2.4.1 -Dhadoop.version=2.4.1 -DskipTests
Or am I missing something completely? What is the time frame to have the above JIRA issues resolved?
mn
On Aug 20, 2014, at 11:25 AM, Marcelo Vanzin <va...@cloudera.com> wrote:
> Ah, sorry, forgot to talk about the second issue.
>
> On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell <ma...@gmail.com> wrote:
>> However, now the Spark jobs running in the ApplicationMaster on a given node
>> fails to find the active resourcemanager. Below is a log excerpt from one
>> of the assigned nodes. As all the jobs fail, eventually YARN will move this
>> to execute on the node that co-locates the active resourcemanager and a
>> nodemanager, where the job will proceed a bit further. Then, the Spark job
>> itself will fail attempting to access HDFS files via the virtualized HA HDFS
>> URI.
>
>> 14/08/20 11:34:27 INFO Client: Retrying connect to server:
>> 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
>> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
>> MILLISECONDS)
>
> Here it seems you're running into the same issue but on the AM side.
> If you're comfortable with using a custom build of Spark, I'd
> recommend building it with the "-Phadoop-provided" profile enabled
> (note: I think that only works with maven currently). That way the
> Spark assembly does not include the Hadoop classes.
>
> A quick look at the code seems to indicate the Spark assembly is
> always added before the local Hadoop jars in the classpath.
> Unfortunately there's no workaround for that at the moment (aside from
> the above built-time fix), although we might be able to do something
> when SPARK-2848 is fixed.
>
> --
> Marcelo
Re: spark-submit with HA YARN
Posted by Marcelo Vanzin <va...@cloudera.com>.
Ah, sorry, forgot to talk about the second issue.
On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell <ma...@gmail.com> wrote:
> However, now the Spark jobs running in the ApplicationMaster on a given node
> fails to find the active resourcemanager. Below is a log excerpt from one
> of the assigned nodes. As all the jobs fail, eventually YARN will move this
> to execute on the node that co-locates the active resourcemanager and a
> nodemanager, where the job will proceed a bit further. Then, the Spark job
> itself will fail attempting to access HDFS files via the virtualized HA HDFS
> URI.
> 14/08/20 11:34:27 INFO Client: Retrying connect to server:
> 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000
> MILLISECONDS)
Here it seems you're running into the same issue but on the AM side.
If you're comfortable with using a custom build of Spark, I'd
recommend building it with the "-Phadoop-provided" profile enabled
(note: I think that only works with maven currently). That way the
Spark assembly does not include the Hadoop classes.
A quick look at the code seems to indicate the Spark assembly is
always added before the local Hadoop jars in the classpath.
Unfortunately there's no workaround for that at the moment (aside from
the above built-time fix), although we might be able to do something
when SPARK-2848 is fixed.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit with HA YARN
Posted by Marcelo Vanzin <va...@cloudera.com>.
On Wed, Aug 20, 2014 at 8:54 AM, Matt Narrell <ma...@gmail.com> wrote:
> An “unaccepted” reply to this thread from Dean Chen suggested to build Spark
> with a newer version of Hadoop (2.4.1) and this has worked to some extent.
> I’m now able to submit jobs (omitting an explicit
> “yarn.resourcemanager.address” property) and the
> ConfiguredRMFailoverProxyProvider seems to submit this to the arbitrary,
> active resource manager. Thanks Dean!
That may point to the actual issue, I think. Your build of Spark
probably has an older version of the Yarn classes included in the jar
(btw I still think it's sort of a bad idea to include the Hadoop
classes in the Spark assembly jar, but well).
What you can try without resorting to a rebuild is force your Hadoop
jars to come before the Spark jar in the classpath (by using
--driver-class-path with spark-submit). Something like:
spark-submit --driver-class-path $(hadoop classpath)
Note: totally untested, but a quick look at the scripts seem to
indicate --driver-class-path comes before the assembly in the final
classpath.
--
Marcelo
---------------------------------------------------------------------
To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
For additional commands, e-mail: user-help@spark.apache.org
Re: spark-submit with HA YARN
Posted by Matt Narrell <ma...@gmail.com>.
Yes, I’m pretty sure my YARN and HDFS HA configuration is correct. I can use the UIs and HDFS command line tools with HA support as expected (failing over namenodes and resourcemanagers, etc) so I believe this to be a Spark issue.
Like I mentioned earlier, if i manipulate the “yarn.resourcemanager.address” to reflect the active resource manager, things work as expected, although that would not be an HA setup…
An “unaccepted” reply to this thread from Dean Chen suggested to build Spark with a newer version of Hadoop (2.4.1) and this has worked to some extent. I’m now able to submit jobs (omitting an explicit “yarn.resourcemanager.address” property) and the ConfiguredRMFailoverProxyProvider seems to submit this to the arbitrary, active resource manager. Thanks Dean!
However, now the Spark jobs running in the ApplicationMaster on a given node fails to find the active resourcemanager. Below is a log excerpt from one of the assigned nodes. As all the jobs fail, eventually YARN will move this to execute on the node that co-locates the active resourcemanager and a nodemanager, where the job will proceed a bit further. Then, the Spark job itself will fail attempting to access HDFS files via the virtualized HA HDFS URI.
I’m running Apache Spark 1.0.2 built against Hadoop 2.4.1. Is it verified that Spark is ready for HA YARN/HDFS?
===================================================
14/08/20 11:34:23 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
14/08/20 11:34:24 INFO ApplicationMaster: ApplicationAttemptId: appattempt_1408548063882_0002_000001
14/08/20 11:34:24 INFO RMProxy: Connecting to ResourceManager at /0.0.0.0:8030
14/08/20 11:34:24 INFO SecurityManager: Changing view acls to: hadoop
14/08/20 11:34:24 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users with view permissions: Set(hadoop)
14/08/20 11:34:24 INFO ApplicationMaster: Starting the user JAR in a separate Thread
14/08/20 11:34:24 INFO ApplicationMaster: Waiting for Spark context initialization
14/08/20 11:34:24 INFO ApplicationMaster: Waiting for Spark context initialization ... 0
14/08/20 11:34:24 WARN SparkConf: In Spark 1.0 and later spark.local.dir will be overridden by the value set by the cluster manager (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
14/08/20 11:34:25 INFO Slf4jLogger: Slf4jLogger started
14/08/20 11:34:25 INFO Remoting: Starting remoting
14/08/20 11:34:25 INFO Remoting: Remoting started; listening on addresses :[akka.tcp://spark@ip-10-0-5-106.us-west-2.compute.internal:41419]
14/08/20 11:34:25 INFO Remoting: Remoting now listens on addresses: [akka.tcp://spark@ip-10-0-5-106.us-west-2.compute.internal:41419]
14/08/20 11:34:27 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 0 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:28 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 1 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:29 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 2 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:30 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 3 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:31 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 4 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:32 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 5 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:33 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 6 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
14/08/20 11:34:34 INFO Client: Retrying connect to server: 0.0.0.0/0.0.0.0:8030. Already tried 7 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10, sleepTime=1000 MILLISECONDS)
On Aug 19, 2014, at 5:34 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Matt,
>
> I checked in the YARN code and I don't see any references to yarn.resourcemanager.address. Have you made sure that your YARN client configuration on the node you're launching from contains the right configs?
>
> -Sandy
>
>
> On Mon, Aug 18, 2014 at 4:07 PM, Matt Narrell <ma...@gmail.com> wrote:
> Hello,
>
> I have an HA enabled YARN cluster with two resource mangers. When submitting jobs via “spark-submit —master yarn-cluster”. It appears that the driver is looking explicitly for the "yarn.resourcemanager.address” property rather than round robin-ing through the resource managers via the “yarn.client.failover-proxy-provider” property set to “org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider”
>
> If I explicitly set the “yarn.resourcemanager.address” to the active resource manager, jobs will submit fine.
>
> Is there a manner to set “spark-submit —master yarn-cluster” to respect the failover proxy?
>
> Thanks in advance,
> Matt
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>
Re: spark-submit with HA YARN
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Matt,
I checked in the YARN code and I don't see any references to
yarn.resourcemanager.address. Have you made sure that your YARN client
configuration on the node you're launching from contains the right configs?
-Sandy
On Mon, Aug 18, 2014 at 4:07 PM, Matt Narrell <ma...@gmail.com>
wrote:
> Hello,
>
> I have an HA enabled YARN cluster with two resource mangers. When
> submitting jobs via “spark-submit —master yarn-cluster”. It appears that
> the driver is looking explicitly for the "yarn.resourcemanager.address”
> property rather than round robin-ing through the resource managers via the
> “yarn.client.failover-proxy-provider” property set to
> “org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider”
>
> If I explicitly set the “yarn.resourcemanager.address” to the active
> resource manager, jobs will submit fine.
>
> Is there a manner to set “spark-submit —master yarn-cluster” to respect
> the failover proxy?
>
> Thanks in advance,
> Matt
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
> For additional commands, e-mail: user-help@spark.apache.org
>
>