You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Ron Gonzalez <zl...@yahoo.com> on 2014/07/14 03:49:18 UTC
Possible bug in ClientBase.scala?
Hi,
I was doing programmatic submission of Spark yarn jobs and I saw code in ClientBase.getDefaultYarnApplicationClasspath():
val field = classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
MRJobConfig doesn't have this field so the created launch env is incomplete. Workaround is to set yarn.application.classpath with the value from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
This results in having the spark job hang if the submission config is different from the default config. For example, if my resource manager port is 8050 instead of 8030, then the spark app is not able to register itself and stays in ACCEPTED state.
I can easily fix this by changing this to YarnConfiguration instead of MRJobConfig but was wondering what the steps are for submitting a fix.
Thanks,
Ron
Sent from my iPhone
Re: Possible bug in ClientBase.scala?
Posted by Nicholas Chammas <ni...@gmail.com>.
On Sun, Jul 13, 2014 at 9:49 PM, Ron Gonzalez <zl...@yahoo.com> wrote:
> I can easily fix this by changing this to YarnConfiguration instead of
> MRJobConfig but was wondering what the steps are for submitting a fix.
>
Relevant links:
- https://cwiki.apache.org/confluence/display/SPARK/Contributing+to+Spark
- https://github.com/apache/spark/pulls
Nick
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
OK I will create PR.
thanks
On Thu, Jul 17, 2014 at 7:58 AM, Sean Owen <so...@cloudera.com> wrote:
> Looks like a real problem. I see it too. I think the same workaround
> found in ClientBase.scala needs to be used here. There, the fact that
> this field can be a String or String[] is handled explicitly. In fact
> I think you can just call to ClientBase for this? PR it, I say.
>
> On Thu, Jul 17, 2014 at 3:24 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
> > val knownDefMRAppCP: Seq[String] =
> > getFieldValue[String, Seq[String]](classOf[MRJobConfig],
> >
> > "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> > Seq[String]())(a =>
> a.split(","))
> >
> > will fail for yarn-alpha.
> >
> > sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
> >
>
Re: Possible bug in ClientBase.scala?
Posted by Sean Owen <so...@cloudera.com>.
Looks like a real problem. I see it too. I think the same workaround
found in ClientBase.scala needs to be used here. There, the fact that
this field can be a String or String[] is handled explicitly. In fact
I think you can just call to ClientBase for this? PR it, I say.
On Thu, Jul 17, 2014 at 3:24 PM, Chester Chen <ch...@alpinenow.com> wrote:
> val knownDefMRAppCP: Seq[String] =
> getFieldValue[String, Seq[String]](classOf[MRJobConfig],
>
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> Seq[String]())(a => a.split(","))
>
> will fail for yarn-alpha.
>
> sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
@Sean and @Sandy
Thanks for the reply. I used to be able to see yarn-alpha and yarn
directories which corresponding to the modules.
I guess due to the recent SparkBuild.scala changes, I did not see
yarn-alpha (by default) and I thought yarn-alpha is renamed to "yarn" and
"yarn-stable" is the old yarn. So I compiled "yarn" against the
hadoop.version = 2.0.5-alpha. My mistake.
I tried
export SPARK_HADOOP_VERSION=2.0.5-alpha
sbt/sbt -Pyarn-alpha yarn-alpha/test
the compilation errors are all gone.
sbt/sbt -Pyarn-alpha projects
does show the yarn-alpha project, I did not realize this is dynamically
enabled based on yarn flag. Thanks Sean for pointing that out.
To Sandy's point, I am not trying to use alpha version of Yarn. I am
experimenting some changes in Yarn Client and refactoring code and just
want to make sure I am passing tests for both yarn-alpha and yarn-stable.
The yarn-alpha tests actually failing due to the yarn API changes in
MRJobConfig class.
as I mentioned in earlier email
The field
DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH
returns String in yarn-stable, but returns String array in yarn-alpha API.
So the method in ClientBaseSuite.scala
val knownDefMRAppCP: Seq[String] =
getFieldValue[String, Seq[String]](classOf[MRJobConfig],
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
Seq[String]())(a => a.split(","))
will fail for yarn-alpha.
sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
...
4/07/17 07:07:16 INFO ClientBase: Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
[info] - default Yarn application classpath *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply$mcV$sp(ClientBaseSuite.scala:47)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply(ClientBaseSuite.scala:47)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply(ClientBaseSuite.scala:47)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - default MR application classpath *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply$mcV$sp(ClientBaseSuite.scala:51)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply(ClientBaseSuite.scala:51)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply(ClientBaseSuite.scala:51)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines a classpath
for YARN *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply$mcV$sp(ClientBaseSuite.scala:55)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply(ClientBaseSuite.scala:55)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply(ClientBaseSuite.scala:55)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines a classpath
for MR *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply$mcV$sp(ClientBaseSuite.scala:64)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply(ClientBaseSuite.scala:64)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply(ClientBaseSuite.scala:64)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines both
classpaths, YARN and MR *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply$mcV$sp(ClientBaseSuite.scala:73)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply(ClientBaseSuite.scala:73)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply(ClientBaseSuite.scala:73)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - Local jar URIs
On Thu, Jul 17, 2014 at 12:44 AM, Sandy Ryza <sa...@cloudera.com>
wrote:
> To add, we've made some effort to yarn-alpha to work with the 2.0.x line,
> but this was a time when YARN went through wild API changes. The only line
> that the yarn-alpha profile is guaranteed to work against is the 0.23 line.
>
>
> On Thu, Jul 17, 2014 at 12:40 AM, Sean Owen <so...@cloudera.com> wrote:
>
> > Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
> > "projects", shows it as a module. You should only build yarn-stable
> > *or* yarn-alpha at any given time.
> >
> > I don't remember the modules changing in a while. 'yarn-alpha' is for
> > YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
> > for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
> > is code common to both, so should compile with yarn-alpha.
> >
> > What's the compile error, and are you setting yarn.version? the
> > default is to use hadoop.version, but that defaults to 1.0.4 and there
> > is no such YARN.
> >
> > Unless I missed it, I only see compile errors in yarn-stable, and you
> > are trying to compile vs YARN alpha versions no?
> >
> > On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ch...@alpinenow.com>
> > wrote:
> > > Looking further, the yarn and yarn-stable are both for the stable
> version
> > > of Yarn, that explains the compilation errors when using 2.0.5-alpha
> > > version of hadoop.
> > >
> > > the module yarn-alpha ( although is still on SparkBuild.scala), is no
> > > longer there in sbt console.
> > >
> > >
> > >> projects
> > >
> > > [info] In file:/Users/chester/projects/spark/
> > >
> > > [info] assembly
> > >
> > > [info] bagel
> > >
> > > [info] catalyst
> > >
> > > [info] core
> > >
> > > [info] examples
> > >
> > > [info] graphx
> > >
> > > [info] hive
> > >
> > > [info] mllib
> > >
> > > [info] oldDeps
> > >
> > > [info] repl
> > >
> > > [info] spark
> > >
> > > [info] sql
> > >
> > > [info] streaming
> > >
> > > [info] streaming-flume
> > >
> > > [info] streaming-kafka
> > >
> > > [info] streaming-mqtt
> > >
> > > [info] streaming-twitter
> > >
> > > [info] streaming-zeromq
> > >
> > > [info] tools
> > >
> > > [info] yarn
> > >
> > > [info] * yarn-stable
> > >
> > >
> > > On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com>
> > wrote:
> > >
> > >> Hmm
> > >> looks like a Build script issue:
> > >>
> > >> I run the command with :
> > >>
> > >> sbt/sbt clean *yarn/*test:compile
> > >>
> > >> but errors came from
> > >>
> > >> [error] 40 errors found
> > >>
> > >> [error] (*yarn-stable*/compile:compile) Compilation failed
> > >>
> > >>
> > >> Chester
> > >>
> > >>
> > >> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
> > >> wrote:
> > >>
> > >>> Hi, Sandy
> > >>>
> > >>> We do have some issue with this. The difference is in Yarn-Alpha
> > and
> > >>> Yarn Stable ( I noticed that in the latest build, the module name has
> > >>> changed,
> > >>> yarn-alpha --> yarn
> > >>> yarn --> yarn-stable
> > >>> )
> > >>>
> > >>> For example: MRJobConfig.class
> > >>> the field:
> > >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
> > >>>
> > >>>
> > >>> In Yarn-Alpha : the field returns java.lang.String[]
> > >>>
> > >>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>
> > >>> while in Yarn-Stable, it returns a String
> > >>>
> > >>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>
> > >>> So in ClientBaseSuite.scala
> > >>>
> > >>> The following code:
> > >>>
> > >>> val knownDefMRAppCP: Seq[String] =
> > >>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
> > >>>
> > >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> > >>> Seq[String]())(a =>
> > >>> *a.split(",")*)
> > >>>
> > >>>
> > >>> works for yarn-stable, but doesn't work for yarn-alpha.
> > >>>
> > >>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago.
> I
> > >>> believe this can be refactored to yarn-alpha module and make
> different
> > >>> tests according different API signatures.
> > >>>
> > >>> I just update the master branch and build doesn't even compile for
> > >>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test
> > passed.
> > >>>
> > >>>
> > >>> Does the Spark Jenkins job run against yarn-alpha ?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Here is output from yarn-alpha compilation:
> > >>>
> > >>> I got the 40 compilation errors.
> > >>>
> > >>> sbt/sbt clean yarn/test:compile
> > >>>
> > >>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
> > as
> > >>> default JAVA_HOME.
> > >>>
> > >>> Note, this will be overridden by -java-home if it is set.
> > >>>
> > >>> [info] Loading project definition from
> > >>> /Users/chester/projects/spark/project/project
> > >>>
> > >>> [info] Loading project definition from
> > >>>
> >
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
> > >>>
> > >>> [warn] Multiple resolvers having different access mechanism
> configured
> > >>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove
> > duplicate
> > >>> project resolvers (`resolvers`) or rename publishing resolver
> > (`publishTo`).
> > >>>
> > >>> [info] Loading project definition from
> > >>> /Users/chester/projects/spark/project
> > >>>
> > >>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> > >>> -Dhadoop.version=2.0.5-alpha
> > >>>
> > >>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
> > >>>
> > >>> [info] Set current project to spark-parent (in build
> > >>> file:/Users/chester/projects/spark/)
> > >>>
> > >>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}core...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [info] Resolving commons-net#commons-net;3.1 ...
> > >>>
> > >>> [info] Compiling 358 Scala sources and 34 Java sources to
> > >>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> > >>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> > >>> corresponding Javadoc for more information.
> > >>>
> > >>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> > >>> attemptId)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewHadoopJob(conf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> > >>> constructor TaskID in class TaskID is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] new TaskAttemptID(new TaskID(jID.value, true,
> splitID),
> > >>> attemptID))
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> > >>> method makeQualified in class Path is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] outputPath.makeQualified(fs)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] if (!fs.getFileStatus(path).isDir) {
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] val logDirs = if (logStatus != null)
> > >>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
> > >>>
> > >>> [warn]
> ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] if (file.isDir) 0L else file.getLen
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> > >>> method getDefaultReplication in class FileSystem is deprecated: see
> > >>> corresponding Javadoc for more information.
> > >>>
> > >>> [warn] fs.create(tempOutputPath, false, bufferSize,
> > >>> fs.getDefaultReplication, blockSize)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> > >>> constructor TaskID in class TaskID is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
> > >>> splitId), attemptId)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new Job(conf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn] 14 warnings found
> > >>>
> > >>> [warn] Note:
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> > >>> uses unchecked or unsafe operations.
> > >>>
> > >>> [warn] Note: Recompile with -Xlint:unchecked for details.
> > >>>
> > >>> [info] Compiling 15 Scala sources to
> > >>>
> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> > >>> not found: value YarnClient
> > >>>
> > >>> [error] val yarnClient = YarnClient.createYarnClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> > >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> > >>> value RM_AM_MAX_ATTEMPTS is not a member of object
> > >>> org.apache.hadoop.yarn.conf.YarnConfiguration
> > >>>
> > >>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> > >>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> > >>> not found: value AMRMClient
> > >>>
> > >>> [error] amClient = AMRMClient.createAMRMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> > >>> not found: value WebAppUtils
> > >>>
> > >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> > >>> value CONTAINER_ID is not a member of object
> > >>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> > >>>
> > >>> [error] val containerIdString = System.getenv(
> > >>> ApplicationConstants.Environment.CONTAINER_ID.name())
> > >>>
> > >>> [error]
> > >>> ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> > >>> value setTokens is not a member of
> > >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> > >>>
> > >>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> > >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> > >>> not found: value AMRMClient
> > >>>
> > >>> [error] amClient = AMRMClient.createAMRMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> > >>> not found: value WebAppUtils
> > >>>
> > >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> > >>> object ProtoUtils is not a member of package
> > >>> org.apache.hadoop.yarn.api.records.impl.pb
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.NMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> > >>> not found: type NMClient
> > >>>
> > >>> [error] var nmClient: NMClient = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> > >>> not found: value NMClient
> > >>>
> > >>> [error] nmClient = NMClient.createNMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> > >>> value setTokens is not a member of
> > >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> > >>>
> > >>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> > >>> object ApplicationMasterProtocol is not a member of package
> > >>> org.apache.hadoop.yarn.api
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] val amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] hostContainers: ArrayBuffer[ContainerRequest]
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val requestedContainers = new
> > >>> ArrayBuffer[ContainerRequest](rackToCounts.size)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val containerRequests: List[ContainerRequest] =
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val hostContainerRequests = new
> > >>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val rackContainerRequests: List[ContainerRequest] =
> > >>> createRackResourceRequests(
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val containerRequestBuffer = new
> > >>> ArrayBuffer[ContainerRequest](
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> > >>> value newInstance is not a member of object
> > >>> org.apache.hadoop.yarn.api.records.Resource
> > >>>
> > >>> [error] val resource = Resource.newInstance(memoryRequest,
> > >>> executorCores)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val requests = new ArrayBuffer[ContainerRequest]()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error] 40 errors found
> > >>>
> > >>> [error] (yarn-stable/compile:compile) Compilation failed
> > >>>
> > >>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sandy.ryza@cloudera.com
> >
> > >>> wrote:
> > >>>
> > >>>> Hi Ron,
> > >>>>
> > >>>> I just checked and this bug is fixed in recent releases of Spark.
> > >>>>
> > >>>> -Sandy
> > >>>>
> > >>>>
> > >>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <
> chester@alpinenow.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Ron,
> > >>>>> Which distribution and Version of Hadoop are you using ?
> > >>>>>
> > >>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> > >>>>> 2.3.0-cdh5.0.0),
> > >>>>>
> > >>>>> MRJobConfig does have the field :
> > >>>>>
> > >>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>>>
> > >>>>> Chester
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <
> zlgonzalez@yahoo.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
> > >>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
> > >>>>>>
> > >>>>>> val field =
> > >>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> > >>>>>> MRJobConfig doesn't have this field so the created launch env is
> > >>>>>> incomplete. Workaround is to set yarn.application.classpath with
> > the value
> > >>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
> > >>>>>>
> > >>>>>> This results in having the spark job hang if the submission config
> > is
> > >>>>>> different from the default config. For example, if my resource
> > manager port
> > >>>>>> is 8050 instead of 8030, then the spark app is not able to
> register
> > itself
> > >>>>>> and stays in ACCEPTED state.
> > >>>>>>
> > >>>>>> I can easily fix this by changing this to YarnConfiguration
> instead
> > of
> > >>>>>> MRJobConfig but was wondering what the steps are for submitting a
> > fix.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Ron
> > >>>>>>
> > >>>>>> Sent from my iPhone
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
Re: Possible bug in ClientBase.scala?
Posted by Sandy Ryza <sa...@cloudera.com>.
To add, we've made some effort to yarn-alpha to work with the 2.0.x line,
but this was a time when YARN went through wild API changes. The only line
that the yarn-alpha profile is guaranteed to work against is the 0.23 line.
On Thu, Jul 17, 2014 at 12:40 AM, Sean Owen <so...@cloudera.com> wrote:
> Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
> "projects", shows it as a module. You should only build yarn-stable
> *or* yarn-alpha at any given time.
>
> I don't remember the modules changing in a while. 'yarn-alpha' is for
> YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
> for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
> is code common to both, so should compile with yarn-alpha.
>
> What's the compile error, and are you setting yarn.version? the
> default is to use hadoop.version, but that defaults to 1.0.4 and there
> is no such YARN.
>
> Unless I missed it, I only see compile errors in yarn-stable, and you
> are trying to compile vs YARN alpha versions no?
>
> On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ch...@alpinenow.com>
> wrote:
> > Looking further, the yarn and yarn-stable are both for the stable version
> > of Yarn, that explains the compilation errors when using 2.0.5-alpha
> > version of hadoop.
> >
> > the module yarn-alpha ( although is still on SparkBuild.scala), is no
> > longer there in sbt console.
> >
> >
> >> projects
> >
> > [info] In file:/Users/chester/projects/spark/
> >
> > [info] assembly
> >
> > [info] bagel
> >
> > [info] catalyst
> >
> > [info] core
> >
> > [info] examples
> >
> > [info] graphx
> >
> > [info] hive
> >
> > [info] mllib
> >
> > [info] oldDeps
> >
> > [info] repl
> >
> > [info] spark
> >
> > [info] sql
> >
> > [info] streaming
> >
> > [info] streaming-flume
> >
> > [info] streaming-kafka
> >
> > [info] streaming-mqtt
> >
> > [info] streaming-twitter
> >
> > [info] streaming-zeromq
> >
> > [info] tools
> >
> > [info] yarn
> >
> > [info] * yarn-stable
> >
> >
> > On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
> >
> >> Hmm
> >> looks like a Build script issue:
> >>
> >> I run the command with :
> >>
> >> sbt/sbt clean *yarn/*test:compile
> >>
> >> but errors came from
> >>
> >> [error] 40 errors found
> >>
> >> [error] (*yarn-stable*/compile:compile) Compilation failed
> >>
> >>
> >> Chester
> >>
> >>
> >> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
> >> wrote:
> >>
> >>> Hi, Sandy
> >>>
> >>> We do have some issue with this. The difference is in Yarn-Alpha
> and
> >>> Yarn Stable ( I noticed that in the latest build, the module name has
> >>> changed,
> >>> yarn-alpha --> yarn
> >>> yarn --> yarn-stable
> >>> )
> >>>
> >>> For example: MRJobConfig.class
> >>> the field:
> >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
> >>>
> >>>
> >>> In Yarn-Alpha : the field returns java.lang.String[]
> >>>
> >>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>
> >>> while in Yarn-Stable, it returns a String
> >>>
> >>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>
> >>> So in ClientBaseSuite.scala
> >>>
> >>> The following code:
> >>>
> >>> val knownDefMRAppCP: Seq[String] =
> >>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
> >>>
> >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> >>> Seq[String]())(a =>
> >>> *a.split(",")*)
> >>>
> >>>
> >>> works for yarn-stable, but doesn't work for yarn-alpha.
> >>>
> >>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
> >>> believe this can be refactored to yarn-alpha module and make different
> >>> tests according different API signatures.
> >>>
> >>> I just update the master branch and build doesn't even compile for
> >>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test
> passed.
> >>>
> >>>
> >>> Does the Spark Jenkins job run against yarn-alpha ?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Here is output from yarn-alpha compilation:
> >>>
> >>> I got the 40 compilation errors.
> >>>
> >>> sbt/sbt clean yarn/test:compile
> >>>
> >>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
> as
> >>> default JAVA_HOME.
> >>>
> >>> Note, this will be overridden by -java-home if it is set.
> >>>
> >>> [info] Loading project definition from
> >>> /Users/chester/projects/spark/project/project
> >>>
> >>> [info] Loading project definition from
> >>>
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
> >>>
> >>> [warn] Multiple resolvers having different access mechanism configured
> >>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove
> duplicate
> >>> project resolvers (`resolvers`) or rename publishing resolver
> (`publishTo`).
> >>>
> >>> [info] Loading project definition from
> >>> /Users/chester/projects/spark/project
> >>>
> >>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> >>> -Dhadoop.version=2.0.5-alpha
> >>>
> >>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
> >>>
> >>> [info] Set current project to spark-parent (in build
> >>> file:/Users/chester/projects/spark/)
> >>>
> >>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}core...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [info] Resolving commons-net#commons-net;3.1 ...
> >>>
> >>> [info] Compiling 358 Scala sources and 34 Java sources to
> >>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> >>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> >>> corresponding Javadoc for more information.
> >>>
> >>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> >>> attemptId)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewHadoopJob(conf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> >>> constructor TaskID in class TaskID is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
> >>> attemptID))
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> >>> method makeQualified in class Path is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] outputPath.makeQualified(fs)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] if (!fs.getFileStatus(path).isDir) {
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] val logDirs = if (logStatus != null)
> >>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] if (file.isDir) 0L else file.getLen
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> >>> method getDefaultReplication in class FileSystem is deprecated: see
> >>> corresponding Javadoc for more information.
> >>>
> >>> [warn] fs.create(tempOutputPath, false, bufferSize,
> >>> fs.getDefaultReplication, blockSize)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> >>> constructor TaskID in class TaskID is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
> >>> splitId), attemptId)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new Job(conf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn] 14 warnings found
> >>>
> >>> [warn] Note:
> >>>
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> >>> uses unchecked or unsafe operations.
> >>>
> >>> [warn] Note: Recompile with -Xlint:unchecked for details.
> >>>
> >>> [info] Compiling 15 Scala sources to
> >>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> >>> not found: value YarnClient
> >>>
> >>> [error] val yarnClient = YarnClient.createYarnClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> >>>
> >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> >>> value RM_AM_MAX_ATTEMPTS is not a member of object
> >>> org.apache.hadoop.yarn.conf.YarnConfiguration
> >>>
> >>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> >>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> >>> not found: type AMRMClient
> >>>
> >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> >>> not found: value AMRMClient
> >>>
> >>> [error] amClient = AMRMClient.createAMRMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> >>> not found: value WebAppUtils
> >>>
> >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> >>> value CONTAINER_ID is not a member of object
> >>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> >>>
> >>> [error] val containerIdString = System.getenv(
> >>> ApplicationConstants.Environment.CONTAINER_ID.name())
> >>>
> >>> [error]
> >>> ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> >>> value setTokens is not a member of
> >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> >>>
> >>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> >>>
> >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> >>> not found: type AMRMClient
> >>>
> >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> >>> not found: value AMRMClient
> >>>
> >>> [error] amClient = AMRMClient.createAMRMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> >>> not found: value WebAppUtils
> >>>
> >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> >>> object ProtoUtils is not a member of package
> >>> org.apache.hadoop.yarn.api.records.impl.pb
> >>>
> >>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.NMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> >>> not found: type NMClient
> >>>
> >>> [error] var nmClient: NMClient = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> >>> not found: value NMClient
> >>>
> >>> [error] nmClient = NMClient.createNMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> >>> value setTokens is not a member of
> >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> >>>
> >>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> >>> object ApplicationMasterProtocol is not a member of package
> >>> org.apache.hadoop.yarn.api
> >>>
> >>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> >>> not found: type AMRMClient
> >>>
> >>> [error] val amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] hostContainers: ArrayBuffer[ContainerRequest]
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val requestedContainers = new
> >>> ArrayBuffer[ContainerRequest](rackToCounts.size)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val containerRequests: List[ContainerRequest] =
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val hostContainerRequests = new
> >>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val rackContainerRequests: List[ContainerRequest] =
> >>> createRackResourceRequests(
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val containerRequestBuffer = new
> >>> ArrayBuffer[ContainerRequest](
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> >>> value newInstance is not a member of object
> >>> org.apache.hadoop.yarn.api.records.Resource
> >>>
> >>> [error] val resource = Resource.newInstance(memoryRequest,
> >>> executorCores)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val requests = new ArrayBuffer[ContainerRequest]()
> >>>
> >>> [error] ^
> >>>
> >>> [error] 40 errors found
> >>>
> >>> [error] (yarn-stable/compile:compile) Compilation failed
> >>>
> >>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
> >>> wrote:
> >>>
> >>>> Hi Ron,
> >>>>
> >>>> I just checked and this bug is fixed in recent releases of Spark.
> >>>>
> >>>> -Sandy
> >>>>
> >>>>
> >>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
> >>>> wrote:
> >>>>
> >>>>> Ron,
> >>>>> Which distribution and Version of Hadoop are you using ?
> >>>>>
> >>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> >>>>> 2.3.0-cdh5.0.0),
> >>>>>
> >>>>> MRJobConfig does have the field :
> >>>>>
> >>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>>>
> >>>>> Chester
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
> >>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
> >>>>>>
> >>>>>> val field =
> >>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> >>>>>> MRJobConfig doesn't have this field so the created launch env is
> >>>>>> incomplete. Workaround is to set yarn.application.classpath with
> the value
> >>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
> >>>>>>
> >>>>>> This results in having the spark job hang if the submission config
> is
> >>>>>> different from the default config. For example, if my resource
> manager port
> >>>>>> is 8050 instead of 8030, then the spark app is not able to register
> itself
> >>>>>> and stays in ACCEPTED state.
> >>>>>>
> >>>>>> I can easily fix this by changing this to YarnConfiguration instead
> of
> >>>>>> MRJobConfig but was wondering what the steps are for submitting a
> fix.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ron
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: Possible bug in ClientBase.scala?
Posted by Sean Owen <so...@cloudera.com>.
Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
"projects", shows it as a module. You should only build yarn-stable
*or* yarn-alpha at any given time.
I don't remember the modules changing in a while. 'yarn-alpha' is for
YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
is code common to both, so should compile with yarn-alpha.
What's the compile error, and are you setting yarn.version? the
default is to use hadoop.version, but that defaults to 1.0.4 and there
is no such YARN.
Unless I missed it, I only see compile errors in yarn-stable, and you
are trying to compile vs YARN alpha versions no?
On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ch...@alpinenow.com> wrote:
> Looking further, the yarn and yarn-stable are both for the stable version
> of Yarn, that explains the compilation errors when using 2.0.5-alpha
> version of hadoop.
>
> the module yarn-alpha ( although is still on SparkBuild.scala), is no
> longer there in sbt console.
>
>
>> projects
>
> [info] In file:/Users/chester/projects/spark/
>
> [info] assembly
>
> [info] bagel
>
> [info] catalyst
>
> [info] core
>
> [info] examples
>
> [info] graphx
>
> [info] hive
>
> [info] mllib
>
> [info] oldDeps
>
> [info] repl
>
> [info] spark
>
> [info] sql
>
> [info] streaming
>
> [info] streaming-flume
>
> [info] streaming-kafka
>
> [info] streaming-mqtt
>
> [info] streaming-twitter
>
> [info] streaming-zeromq
>
> [info] tools
>
> [info] yarn
>
> [info] * yarn-stable
>
>
> On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com> wrote:
>
>> Hmm
>> looks like a Build script issue:
>>
>> I run the command with :
>>
>> sbt/sbt clean *yarn/*test:compile
>>
>> but errors came from
>>
>> [error] 40 errors found
>>
>> [error] (*yarn-stable*/compile:compile) Compilation failed
>>
>>
>> Chester
>>
>>
>> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
>> wrote:
>>
>>> Hi, Sandy
>>>
>>> We do have some issue with this. The difference is in Yarn-Alpha and
>>> Yarn Stable ( I noticed that in the latest build, the module name has
>>> changed,
>>> yarn-alpha --> yarn
>>> yarn --> yarn-stable
>>> )
>>>
>>> For example: MRJobConfig.class
>>> the field:
>>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>>>
>>>
>>> In Yarn-Alpha : the field returns java.lang.String[]
>>>
>>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> while in Yarn-Stable, it returns a String
>>>
>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> So in ClientBaseSuite.scala
>>>
>>> The following code:
>>>
>>> val knownDefMRAppCP: Seq[String] =
>>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>>>
>>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
>>> Seq[String]())(a =>
>>> *a.split(",")*)
>>>
>>>
>>> works for yarn-stable, but doesn't work for yarn-alpha.
>>>
>>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
>>> believe this can be refactored to yarn-alpha module and make different
>>> tests according different API signatures.
>>>
>>> I just update the master branch and build doesn't even compile for
>>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>>>
>>>
>>> Does the Spark Jenkins job run against yarn-alpha ?
>>>
>>>
>>>
>>>
>>>
>>> Here is output from yarn-alpha compilation:
>>>
>>> I got the 40 compilation errors.
>>>
>>> sbt/sbt clean yarn/test:compile
>>>
>>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
>>> default JAVA_HOME.
>>>
>>> Note, this will be overridden by -java-home if it is set.
>>>
>>> [info] Loading project definition from
>>> /Users/chester/projects/spark/project/project
>>>
>>> [info] Loading project definition from
>>> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>>>
>>> [warn] Multiple resolvers having different access mechanism configured
>>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
>>> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>>>
>>> [info] Loading project definition from
>>> /Users/chester/projects/spark/project
>>>
>>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
>>> -Dhadoop.version=2.0.5-alpha
>>>
>>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>>>
>>> [info] Set current project to spark-parent (in build
>>> file:/Users/chester/projects/spark/)
>>>
>>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}core...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [info] Resolving commons-net#commons-net;3.1 ...
>>>
>>> [info] Compiling 358 Scala sources and 34 Java sources to
>>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
>>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
>>> corresponding Javadoc for more information.
>>>
>>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
>>> attemptId)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewHadoopJob(conf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
>>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
>>> attemptID))
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
>>> method makeQualified in class Path is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] outputPath.makeQualified(fs)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] if (!fs.getFileStatus(path).isDir) {
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] val logDirs = if (logStatus != null)
>>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] if (file.isDir) 0L else file.getLen
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
>>> method getDefaultReplication in class FileSystem is deprecated: see
>>> corresponding Javadoc for more information.
>>>
>>> [warn] fs.create(tempOutputPath, false, bufferSize,
>>> fs.getDefaultReplication, blockSize)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
>>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
>>> splitId), attemptId)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new Job(conf)
>>>
>>> [warn] ^
>>>
>>> [warn] 14 warnings found
>>>
>>> [warn] Note:
>>> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
>>> uses unchecked or unsafe operations.
>>>
>>> [warn] Note: Recompile with -Xlint:unchecked for details.
>>>
>>> [info] Compiling 15 Scala sources to
>>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>>> not found: value YarnClient
>>>
>>> [error] val yarnClient = YarnClient.createYarnClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>>
>>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
>>> value RM_AM_MAX_ATTEMPTS is not a member of object
>>> org.apache.hadoop.yarn.conf.YarnConfiguration
>>>
>>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
>>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
>>> not found: type AMRMClient
>>>
>>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
>>> not found: value AMRMClient
>>>
>>> [error] amClient = AMRMClient.createAMRMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
>>> not found: value WebAppUtils
>>>
>>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
>>> value CONTAINER_ID is not a member of object
>>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>>>
>>> [error] val containerIdString = System.getenv(
>>> ApplicationConstants.Environment.CONTAINER_ID.name())
>>>
>>> [error]
>>> ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
>>> value setTokens is not a member of
>>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>>
>>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
>>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>>
>>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
>>> not found: type AMRMClient
>>>
>>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
>>> not found: value AMRMClient
>>>
>>> [error] amClient = AMRMClient.createAMRMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
>>> not found: value WebAppUtils
>>>
>>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
>>> object ProtoUtils is not a member of package
>>> org.apache.hadoop.yarn.api.records.impl.pb
>>>
>>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.NMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
>>> not found: type NMClient
>>>
>>> [error] var nmClient: NMClient = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
>>> not found: value NMClient
>>>
>>> [error] nmClient = NMClient.createNMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
>>> value setTokens is not a member of
>>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>>
>>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
>>> object ApplicationMasterProtocol is not a member of package
>>> org.apache.hadoop.yarn.api
>>>
>>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
>>> not found: type AMRMClient
>>>
>>> [error] val amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
>>> not found: type ContainerRequest
>>>
>>> [error] hostContainers: ArrayBuffer[ContainerRequest]
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
>>> not found: type ContainerRequest
>>>
>>> [error] val requestedContainers = new
>>> ArrayBuffer[ContainerRequest](rackToCounts.size)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
>>> not found: type ContainerRequest
>>>
>>> [error] val containerRequests: List[ContainerRequest] =
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
>>> not found: type ContainerRequest
>>>
>>> [error] val hostContainerRequests = new
>>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
>>> not found: type ContainerRequest
>>>
>>> [error] val rackContainerRequests: List[ContainerRequest] =
>>> createRackResourceRequests(
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
>>> not found: type ContainerRequest
>>>
>>> [error] val containerRequestBuffer = new
>>> ArrayBuffer[ContainerRequest](
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
>>> value newInstance is not a member of object
>>> org.apache.hadoop.yarn.api.records.Resource
>>>
>>> [error] val resource = Resource.newInstance(memoryRequest,
>>> executorCores)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
>>> not found: type ContainerRequest
>>>
>>> [error] val requests = new ArrayBuffer[ContainerRequest]()
>>>
>>> [error] ^
>>>
>>> [error] 40 errors found
>>>
>>> [error] (yarn-stable/compile:compile) Compilation failed
>>>
>>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
>>> wrote:
>>>
>>>> Hi Ron,
>>>>
>>>> I just checked and this bug is fixed in recent releases of Spark.
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
>>>> wrote:
>>>>
>>>>> Ron,
>>>>> Which distribution and Version of Hadoop are you using ?
>>>>>
>>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>>>> 2.3.0-cdh5.0.0),
>>>>>
>>>>> MRJobConfig does have the field :
>>>>>
>>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>>>
>>>>> Chester
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
>>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
>>>>>>
>>>>>> val field =
>>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>>>> MRJobConfig doesn't have this field so the created launch env is
>>>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>>>
>>>>>> This results in having the spark job hang if the submission config is
>>>>>> different from the default config. For example, if my resource manager port
>>>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>>>> and stays in ACCEPTED state.
>>>>>>
>>>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>>>
>>>>>> Thanks,
>>>>>> Ron
>>>>>>
>>>>>> Sent from my iPhone
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
Looking further, the yarn and yarn-stable are both for the stable version
of Yarn, that explains the compilation errors when using 2.0.5-alpha
version of hadoop.
the module yarn-alpha ( although is still on SparkBuild.scala), is no
longer there in sbt console.
> projects
[info] In file:/Users/chester/projects/spark/
[info] assembly
[info] bagel
[info] catalyst
[info] core
[info] examples
[info] graphx
[info] hive
[info] mllib
[info] oldDeps
[info] repl
[info] spark
[info] sql
[info] streaming
[info] streaming-flume
[info] streaming-kafka
[info] streaming-mqtt
[info] streaming-twitter
[info] streaming-zeromq
[info] tools
[info] yarn
[info] * yarn-stable
On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com> wrote:
> Hmm
> looks like a Build script issue:
>
> I run the command with :
>
> sbt/sbt clean *yarn/*test:compile
>
> but errors came from
>
> [error] 40 errors found
>
> [error] (*yarn-stable*/compile:compile) Compilation failed
>
>
> Chester
>
>
> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
>
>> Hi, Sandy
>>
>> We do have some issue with this. The difference is in Yarn-Alpha and
>> Yarn Stable ( I noticed that in the latest build, the module name has
>> changed,
>> yarn-alpha --> yarn
>> yarn --> yarn-stable
>> )
>>
>> For example: MRJobConfig.class
>> the field:
>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>>
>>
>> In Yarn-Alpha : the field returns java.lang.String[]
>>
>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>
>> while in Yarn-Stable, it returns a String
>>
>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>
>> So in ClientBaseSuite.scala
>>
>> The following code:
>>
>> val knownDefMRAppCP: Seq[String] =
>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>>
>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
>> Seq[String]())(a =>
>> *a.split(",")*)
>>
>>
>> works for yarn-stable, but doesn't work for yarn-alpha.
>>
>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
>> believe this can be refactored to yarn-alpha module and make different
>> tests according different API signatures.
>>
>> I just update the master branch and build doesn't even compile for
>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>>
>>
>> Does the Spark Jenkins job run against yarn-alpha ?
>>
>>
>>
>>
>>
>> Here is output from yarn-alpha compilation:
>>
>> I got the 40 compilation errors.
>>
>> sbt/sbt clean yarn/test:compile
>>
>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
>> default JAVA_HOME.
>>
>> Note, this will be overridden by -java-home if it is set.
>>
>> [info] Loading project definition from
>> /Users/chester/projects/spark/project/project
>>
>> [info] Loading project definition from
>> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>>
>> [warn] Multiple resolvers having different access mechanism configured
>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
>> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>>
>> [info] Loading project definition from
>> /Users/chester/projects/spark/project
>>
>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
>> -Dhadoop.version=2.0.5-alpha
>>
>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>>
>> [info] Set current project to spark-parent (in build
>> file:/Users/chester/projects/spark/)
>>
>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>>
>> [info] Updating {file:/Users/chester/projects/spark/}core...
>>
>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>
>> [info] Done updating.
>>
>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>>
>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>>
>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>
>> [info] Done updating.
>>
>> [info] Resolving commons-net#commons-net;3.1 ...
>>
>> [info] Compiling 358 Scala sources and 34 Java sources to
>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>>
>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>
>> [info] Done updating.
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
>> corresponding Javadoc for more information.
>>
>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
>> attemptId)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewHadoopJob(conf)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
>> attemptID))
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
>> method makeQualified in class Path is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] outputPath.makeQualified(fs)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] if (!fs.getFileStatus(path).isDir) {
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] val logDirs = if (logStatus != null)
>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] if (file.isDir) 0L else file.getLen
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
>> method getDefaultReplication in class FileSystem is deprecated: see
>> corresponding Javadoc for more information.
>>
>> [warn] fs.create(tempOutputPath, false, bufferSize,
>> fs.getDefaultReplication, blockSize)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
>> splitId), attemptId)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new Job(conf)
>>
>> [warn] ^
>>
>> [warn] 14 warnings found
>>
>> [warn] Note:
>> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
>> uses unchecked or unsafe operations.
>>
>> [warn] Note: Recompile with -Xlint:unchecked for details.
>>
>> [info] Compiling 15 Scala sources to
>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>> not found: value YarnClient
>>
>> [error] val yarnClient = YarnClient.createYarnClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import
>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>
>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
>> value RM_AM_MAX_ATTEMPTS is not a member of object
>> org.apache.hadoop.yarn.conf.YarnConfiguration
>>
>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
>> not found: type AMRMClient
>>
>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
>> not found: value AMRMClient
>>
>> [error] amClient = AMRMClient.createAMRMClient()
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
>> not found: value WebAppUtils
>>
>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>> not found: type AMRMClient
>>
>> [error] amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>> not found: type AMRMClient
>>
>> [error] amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>> not found: type AMRMClient
>>
>> [error] amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
>> value CONTAINER_ID is not a member of object
>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>>
>> [error] val containerIdString = System.getenv(
>> ApplicationConstants.Environment.CONTAINER_ID.name())
>>
>> [error]
>> ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
>> value setTokens is not a member of
>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>
>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import
>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>
>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
>> not found: type AMRMClient
>>
>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
>> not found: value AMRMClient
>>
>> [error] amClient = AMRMClient.createAMRMClient()
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
>> not found: value WebAppUtils
>>
>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
>> object ProtoUtils is not a member of package
>> org.apache.hadoop.yarn.api.records.impl.pb
>>
>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.NMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
>> not found: type NMClient
>>
>> [error] var nmClient: NMClient = _
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
>> not found: value NMClient
>>
>> [error] nmClient = NMClient.createNMClient()
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
>> value setTokens is not a member of
>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>
>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
>> object ApplicationMasterProtocol is not a member of package
>> org.apache.hadoop.yarn.api
>>
>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import
>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
>> not found: type AMRMClient
>>
>> [error] val amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
>> not found: type ContainerRequest
>>
>> [error] ): ArrayBuffer[ContainerRequest] = {
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
>> not found: type ContainerRequest
>>
>> [error] hostContainers: ArrayBuffer[ContainerRequest]
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
>> not found: type ContainerRequest
>>
>> [error] val requestedContainers = new
>> ArrayBuffer[ContainerRequest](rackToCounts.size)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
>> not found: type ContainerRequest
>>
>> [error] val containerRequests: List[ContainerRequest] =
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
>> not found: type ContainerRequest
>>
>> [error] ): ArrayBuffer[ContainerRequest] = {
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
>> not found: type ContainerRequest
>>
>> [error] val hostContainerRequests = new
>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
>> not found: type ContainerRequest
>>
>> [error] val rackContainerRequests: List[ContainerRequest] =
>> createRackResourceRequests(
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
>> not found: type ContainerRequest
>>
>> [error] val containerRequestBuffer = new
>> ArrayBuffer[ContainerRequest](
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
>> not found: type ContainerRequest
>>
>> [error] ): ArrayBuffer[ContainerRequest] = {
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
>> value newInstance is not a member of object
>> org.apache.hadoop.yarn.api.records.Resource
>>
>> [error] val resource = Resource.newInstance(memoryRequest,
>> executorCores)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
>> not found: type ContainerRequest
>>
>> [error] val requests = new ArrayBuffer[ContainerRequest]()
>>
>> [error] ^
>>
>> [error] 40 errors found
>>
>> [error] (yarn-stable/compile:compile) Compilation failed
>>
>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
>> wrote:
>>
>>> Hi Ron,
>>>
>>> I just checked and this bug is fixed in recent releases of Spark.
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
>>> wrote:
>>>
>>>> Ron,
>>>> Which distribution and Version of Hadoop are you using ?
>>>>
>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>>> 2.3.0-cdh5.0.0),
>>>>
>>>> MRJobConfig does have the field :
>>>>
>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>>
>>>> Chester
>>>>
>>>>
>>>>
>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
>>>>>
>>>>> val field =
>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>>> MRJobConfig doesn't have this field so the created launch env is
>>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>>
>>>>> This results in having the spark job hang if the submission config is
>>>>> different from the default config. For example, if my resource manager port
>>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>>> and stays in ACCEPTED state.
>>>>>
>>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>>
>>>>> Thanks,
>>>>> Ron
>>>>>
>>>>> Sent from my iPhone
>>>>
>>>>
>>>>
>>>
>>
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
Hmm
looks like a Build script issue:
I run the command with :
sbt/sbt clean *yarn/*test:compile
but errors came from
[error] 40 errors found
[error] (*yarn-stable*/compile:compile) Compilation failed
Chester
On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com> wrote:
> Hi, Sandy
>
> We do have some issue with this. The difference is in Yarn-Alpha and
> Yarn Stable ( I noticed that in the latest build, the module name has
> changed,
> yarn-alpha --> yarn
> yarn --> yarn-stable
> )
>
> For example: MRJobConfig.class
> the field:
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>
>
> In Yarn-Alpha : the field returns java.lang.String[]
>
> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> while in Yarn-Stable, it returns a String
>
> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> So in ClientBaseSuite.scala
>
> The following code:
>
> val knownDefMRAppCP: Seq[String] =
> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> Seq[String]())(a =>
> *a.split(",")*)
>
>
> works for yarn-stable, but doesn't work for yarn-alpha.
>
> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
> believe this can be refactored to yarn-alpha module and make different
> tests according different API signatures.
>
> I just update the master branch and build doesn't even compile for
> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>
>
> Does the Spark Jenkins job run against yarn-alpha ?
>
>
>
>
>
> Here is output from yarn-alpha compilation:
>
> I got the 40 compilation errors.
>
> sbt/sbt clean yarn/test:compile
>
> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
> default JAVA_HOME.
>
> Note, this will be overridden by -java-home if it is set.
>
> [info] Loading project definition from
> /Users/chester/projects/spark/project/project
>
> [info] Loading project definition from
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>
> [warn] Multiple resolvers having different access mechanism configured
> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>
> [info] Loading project definition from
> /Users/chester/projects/spark/project
>
> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> -Dhadoop.version=2.0.5-alpha
>
> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>
> [info] Set current project to spark-parent (in build
> file:/Users/chester/projects/spark/)
>
> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>
> [info] Updating {file:/Users/chester/projects/spark/}core...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>
> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [info] Resolving commons-net#commons-net;3.1 ...
>
> [info] Compiling 358 Scala sources and 34 Java sources to
> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> corresponding Javadoc for more information.
>
> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> attemptId)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewHadoopJob(conf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
> attemptID))
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> method makeQualified in class Path is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] outputPath.makeQualified(fs)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] if (!fs.getFileStatus(path).isDir) {
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] val logDirs = if (logStatus != null)
> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] if (file.isDir) 0L else file.getLen
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> method getDefaultReplication in class FileSystem is deprecated: see
> corresponding Javadoc for more information.
>
> [warn] fs.create(tempOutputPath, false, bufferSize,
> fs.getDefaultReplication, blockSize)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true, splitId),
> attemptId)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new Job(conf)
>
> [warn] ^
>
> [warn] 14 warnings found
>
> [warn] Note:
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> uses unchecked or unsafe operations.
>
> [warn] Note: Recompile with -Xlint:unchecked for details.
>
> [info] Compiling 15 Scala sources to
> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> not found: value YarnClient
>
> [error] val yarnClient = YarnClient.createYarnClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> object util is not a member of package org.apache.hadoop.yarn.webapp
>
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> value RM_AM_MAX_ATTEMPTS is not a member of object
> org.apache.hadoop.yarn.conf.YarnConfiguration
>
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> not found: type AMRMClient
>
> [error] private var amClient: AMRMClient[ContainerRequest] = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> not found: value AMRMClient
>
> [error] amClient = AMRMClient.createAMRMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> not found: value WebAppUtils
>
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> value CONTAINER_ID is not a member of object
> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>
> [error] val containerIdString = System.getenv(
> ApplicationConstants.Environment.CONTAINER_ID.name())
>
> [error]
> ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>
> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> object util is not a member of package org.apache.hadoop.yarn.webapp
>
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> not found: type AMRMClient
>
> [error] private var amClient: AMRMClient[ContainerRequest] = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> not found: value AMRMClient
>
> [error] amClient = AMRMClient.createAMRMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> not found: value WebAppUtils
>
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> object ProtoUtils is not a member of package
> org.apache.hadoop.yarn.api.records.impl.pb
>
> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.NMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> not found: type NMClient
>
> [error] var nmClient: NMClient = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> not found: value NMClient
>
> [error] nmClient = NMClient.createNMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>
> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> object ApplicationMasterProtocol is not a member of package
> org.apache.hadoop.yarn.api
>
> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> not found: type AMRMClient
>
> [error] val amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> not found: type ContainerRequest
>
> [error] hostContainers: ArrayBuffer[ContainerRequest]
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> not found: type ContainerRequest
>
> [error] val requestedContainers = new
> ArrayBuffer[ContainerRequest](rackToCounts.size)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> not found: type ContainerRequest
>
> [error] val containerRequests: List[ContainerRequest] =
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> not found: type ContainerRequest
>
> [error] val hostContainerRequests = new
> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> not found: type ContainerRequest
>
> [error] val rackContainerRequests: List[ContainerRequest] =
> createRackResourceRequests(
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> not found: type ContainerRequest
>
> [error] val containerRequestBuffer = new
> ArrayBuffer[ContainerRequest](
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> value newInstance is not a member of object
> org.apache.hadoop.yarn.api.records.Resource
>
> [error] val resource = Resource.newInstance(memoryRequest,
> executorCores)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> not found: type ContainerRequest
>
> [error] val requests = new ArrayBuffer[ContainerRequest]()
>
> [error] ^
>
> [error] 40 errors found
>
> [error] (yarn-stable/compile:compile) Compilation failed
>
> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
>> Hi Ron,
>>
>> I just checked and this bug is fixed in recent releases of Spark.
>>
>> -Sandy
>>
>>
>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
>> wrote:
>>
>>> Ron,
>>> Which distribution and Version of Hadoop are you using ?
>>>
>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>> 2.3.0-cdh5.0.0),
>>>
>>> MRJobConfig does have the field :
>>>
>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> Chester
>>>
>>>
>>>
>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I was doing programmatic submission of Spark yarn jobs and I saw code
>>>> in ClientBase.getDefaultYarnApplicationClasspath():
>>>>
>>>> val field =
>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>> MRJobConfig doesn't have this field so the created launch env is
>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>
>>>> This results in having the spark job hang if the submission config is
>>>> different from the default config. For example, if my resource manager port
>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>> and stays in ACCEPTED state.
>>>>
>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>
>>>> Thanks,
>>>> Ron
>>>>
>>>> Sent from my iPhone
>>>
>>>
>>>
>>
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
Hi, Sandy
We do have some issue with this. The difference is in Yarn-Alpha and
Yarn Stable ( I noticed that in the latest build, the module name has
changed,
yarn-alpha --> yarn
yarn --> yarn-stable
)
For example: MRJobConfig.class
the field:
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
In Yarn-Alpha : the field returns java.lang.String[]
java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
while in Yarn-Stable, it returns a String
java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
So in ClientBaseSuite.scala
The following code:
val knownDefMRAppCP: Seq[String] =
getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
Seq[String]())(a => *a.split(",")*)
works for yarn-stable, but doesn't work for yarn-alpha.
This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
believe this can be refactored to yarn-alpha module and make different
tests according different API signatures.
I just update the master branch and build doesn't even compile for
Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
Does the Spark Jenkins job run against yarn-alpha ?
Here is output from yarn-alpha compilation:
I got the 40 compilation errors.
sbt/sbt clean yarn/test:compile
Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
[info] Loading project definition from
/Users/chester/projects/spark/project/project
[info] Loading project definition from
/Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
[warn] Multiple resolvers having different access mechanism configured with
same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Loading project definition from /Users/chester/projects/spark/project
NOTE: SPARK_HADOOP_VERSION is deprecated, please use
-Dhadoop.version=2.0.5-alpha
NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
[info] Set current project to spark-parent (in build
file:/Users/chester/projects/spark/)
[success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
[info] Updating {file:/Users/chester/projects/spark/}core...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/Users/chester/projects/spark/}yarn...
[info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Resolving commons-net#commons-net;3.1 ...
[info] Compiling 358 Scala sources and 34 Java sources to
/Users/chester/projects/spark/core/target/scala-2.10/classes...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
constructor TaskAttemptID in class TaskAttemptID is deprecated: see
corresponding Javadoc for more information.
[warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId, attemptId)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewHadoopJob(hadoopConfiguration)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewHadoopJob(conf)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
for more information.
[warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
attemptID))
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
method makeQualified in class Path is deprecated: see corresponding Javadoc
for more information.
[warn] outputPath.makeQualified(fs)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] if (!fs.getFileStatus(path).isDir) {
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] val logDirs = if (logStatus != null)
logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] if (file.isDir) 0L else file.getLen
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
method getDefaultReplication in class FileSystem is deprecated: see
corresponding Javadoc for more information.
[warn] fs.create(tempOutputPath, false, bufferSize,
fs.getDefaultReplication, blockSize)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
for more information.
[warn] val taId = new TaskAttemptID(new TaskID(jobID, true, splitId),
attemptId)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewAPIHadoopJob(hadoopConf)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewAPIHadoopJob(hadoopConf)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new Job(conf)
[warn] ^
[warn] 14 warnings found
[warn] Note:
/Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
uses unchecked or unsafe operations.
[warn] Note: Recompile with -Xlint:unchecked for details.
[info] Compiling 15 Scala sources to
/Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.YarnClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
not found: value YarnClient
[error] val yarnClient = YarnClient.createYarnClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
object util is not a member of package org.apache.hadoop.yarn.webapp
[error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
value RM_AM_MAX_ATTEMPTS is not a member of object
org.apache.hadoop.yarn.conf.YarnConfiguration
[error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
not found: type AMRMClient
[error] private var amClient: AMRMClient[ContainerRequest] = _
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
not found: value AMRMClient
[error] amClient = AMRMClient.createAMRMClient()
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
not found: value WebAppUtils
[error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
not found: type AMRMClient
[error] amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
not found: type AMRMClient
[error] amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
not found: type AMRMClient
[error] amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
value CONTAINER_ID is not a member of object
org.apache.hadoop.yarn.api.ApplicationConstants.Environment
[error] val containerIdString = System.getenv(
ApplicationConstants.Environment.CONTAINER_ID.name())
[error]
^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
value setTokens is not a member of
org.apache.hadoop.yarn.api.records.ContainerLaunchContext
[error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
object util is not a member of package org.apache.hadoop.yarn.webapp
[error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
not found: type AMRMClient
[error] private var amClient: AMRMClient[ContainerRequest] = _
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
not found: value AMRMClient
[error] amClient = AMRMClient.createAMRMClient()
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
not found: value WebAppUtils
[error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
object ProtoUtils is not a member of package
org.apache.hadoop.yarn.api.records.impl.pb
[error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.NMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
not found: type NMClient
[error] var nmClient: NMClient = _
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
not found: value NMClient
[error] nmClient = NMClient.createNMClient()
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
value setTokens is not a member of
org.apache.hadoop.yarn.api.records.ContainerLaunchContext
[error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
object ApplicationMasterProtocol is not a member of package
org.apache.hadoop.yarn.api
[error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
not found: type AMRMClient
[error] val amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
not found: type ContainerRequest
[error] ): ArrayBuffer[ContainerRequest] = {
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
not found: type ContainerRequest
[error] hostContainers: ArrayBuffer[ContainerRequest]
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
not found: type ContainerRequest
[error] val requestedContainers = new
ArrayBuffer[ContainerRequest](rackToCounts.size)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
not found: type ContainerRequest
[error] val containerRequests: List[ContainerRequest] =
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
not found: type ContainerRequest
[error] ): ArrayBuffer[ContainerRequest] = {
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
not found: type ContainerRequest
[error] val hostContainerRequests = new
ArrayBuffer[ContainerRequest](preferredHostToCount.size)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
not found: type ContainerRequest
[error] val rackContainerRequests: List[ContainerRequest] =
createRackResourceRequests(
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
not found: type ContainerRequest
[error] val containerRequestBuffer = new
ArrayBuffer[ContainerRequest](
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
not found: type ContainerRequest
[error] ): ArrayBuffer[ContainerRequest] = {
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
value newInstance is not a member of object
org.apache.hadoop.yarn.api.records.Resource
[error] val resource = Resource.newInstance(memoryRequest,
executorCores)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
not found: type ContainerRequest
[error] val requests = new ArrayBuffer[ContainerRequest]()
[error] ^
[error] 40 errors found
[error] (yarn-stable/compile:compile) Compilation failed
[error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Ron,
>
> I just checked and this bug is fixed in recent releases of Spark.
>
> -Sandy
>
>
> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
>
>> Ron,
>> Which distribution and Version of Hadoop are you using ?
>>
>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>> 2.3.0-cdh5.0.0),
>>
>> MRJobConfig does have the field :
>>
>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>
>> Chester
>>
>>
>>
>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>> wrote:
>>
>>> Hi,
>>> I was doing programmatic submission of Spark yarn jobs and I saw code
>>> in ClientBase.getDefaultYarnApplicationClasspath():
>>>
>>> val field =
>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>> MRJobConfig doesn't have this field so the created launch env is
>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>
>>> This results in having the spark job hang if the submission config is
>>> different from the default config. For example, if my resource manager port
>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>> and stays in ACCEPTED state.
>>>
>>> I can easily fix this by changing this to YarnConfiguration instead of
>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>
>>> Thanks,
>>> Ron
>>>
>>> Sent from my iPhone
>>
>>
>>
>
Re: Possible bug in ClientBase.scala?
Posted by Sandy Ryza <sa...@cloudera.com>.
Hi Ron,
I just checked and this bug is fixed in recent releases of Spark.
-Sandy
On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com> wrote:
> Ron,
> Which distribution and Version of Hadoop are you using ?
>
> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> 2.3.0-cdh5.0.0),
>
> MRJobConfig does have the field :
>
> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> Chester
>
>
>
> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
> wrote:
>
>> Hi,
>> I was doing programmatic submission of Spark yarn jobs and I saw code
>> in ClientBase.getDefaultYarnApplicationClasspath():
>>
>> val field =
>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>> MRJobConfig doesn't have this field so the created launch env is
>> incomplete. Workaround is to set yarn.application.classpath with the value
>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>
>> This results in having the spark job hang if the submission config is
>> different from the default config. For example, if my resource manager port
>> is 8050 instead of 8030, then the spark app is not able to register itself
>> and stays in ACCEPTED state.
>>
>> I can easily fix this by changing this to YarnConfiguration instead of
>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>
>> Thanks,
>> Ron
>>
>> Sent from my iPhone
>
>
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
Ron,
Which distribution and Version of Hadoop are you using ?
I just looked at CDH5 ( hadoop-mapreduce-client-core-
2.3.0-cdh5.0.0),
MRJobConfig does have the field :
java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
Chester
On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com> wrote:
> Hi,
> I was doing programmatic submission of Spark yarn jobs and I saw code in
> ClientBase.getDefaultYarnApplicationClasspath():
>
> val field =
> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> MRJobConfig doesn't have this field so the created launch env is
> incomplete. Workaround is to set yarn.application.classpath with the value
> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>
> This results in having the spark job hang if the submission config is
> different from the default config. For example, if my resource manager port
> is 8050 instead of 8030, then the spark app is not able to register itself
> and stays in ACCEPTED state.
>
> I can easily fix this by changing this to YarnConfiguration instead of
> MRJobConfig but was wondering what the steps are for submitting a fix.
>
> Thanks,
> Ron
>
> Sent from my iPhone