You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@spark.apache.org by Chester Chen <ch...@alpinenow.com> on 2014/07/17 02:18:26 UTC
Re: Possible bug in ClientBase.scala?
Hi, Sandy
We do have some issue with this. The difference is in Yarn-Alpha and
Yarn Stable ( I noticed that in the latest build, the module name has
changed,
yarn-alpha --> yarn
yarn --> yarn-stable
)
For example: MRJobConfig.class
the field:
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
In Yarn-Alpha : the field returns java.lang.String[]
java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
while in Yarn-Stable, it returns a String
java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
So in ClientBaseSuite.scala
The following code:
val knownDefMRAppCP: Seq[String] =
getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
Seq[String]())(a => *a.split(",")*)
works for yarn-stable, but doesn't work for yarn-alpha.
This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
believe this can be refactored to yarn-alpha module and make different
tests according different API signatures.
I just update the master branch and build doesn't even compile for
Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
Does the Spark Jenkins job run against yarn-alpha ?
Here is output from yarn-alpha compilation:
I got the 40 compilation errors.
sbt/sbt clean yarn/test:compile
Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
default JAVA_HOME.
Note, this will be overridden by -java-home if it is set.
[info] Loading project definition from
/Users/chester/projects/spark/project/project
[info] Loading project definition from
/Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
[warn] Multiple resolvers having different access mechanism configured with
same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
[info] Loading project definition from /Users/chester/projects/spark/project
NOTE: SPARK_HADOOP_VERSION is deprecated, please use
-Dhadoop.version=2.0.5-alpha
NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
[info] Set current project to spark-parent (in build
file:/Users/chester/projects/spark/)
[success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
[info] Updating {file:/Users/chester/projects/spark/}core...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Updating {file:/Users/chester/projects/spark/}yarn...
[info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[info] Resolving commons-net#commons-net;3.1 ...
[info] Compiling 358 Scala sources and 34 Java sources to
/Users/chester/projects/spark/core/target/scala-2.10/classes...
[info] Resolving org.fusesource.jansi#jansi;1.4 ...
[info] Done updating.
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
constructor TaskAttemptID in class TaskAttemptID is deprecated: see
corresponding Javadoc for more information.
[warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId, attemptId)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewHadoopJob(hadoopConfiguration)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewHadoopJob(conf)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
for more information.
[warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
attemptID))
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
method makeQualified in class Path is deprecated: see corresponding Javadoc
for more information.
[warn] outputPath.makeQualified(fs)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] if (!fs.getFileStatus(path).isDir) {
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] val logDirs = if (logStatus != null)
logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] if (file.isDir) 0L else file.getLen
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
method getDefaultReplication in class FileSystem is deprecated: see
corresponding Javadoc for more information.
[warn] fs.create(tempOutputPath, false, bufferSize,
fs.getDefaultReplication, blockSize)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
for more information.
[warn] val taId = new TaskAttemptID(new TaskID(jobID, true, splitId),
attemptId)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewAPIHadoopJob(hadoopConf)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new NewAPIHadoopJob(hadoopConf)
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
method isDir in class FileStatus is deprecated: see corresponding Javadoc
for more information.
[warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
[warn] ^
[warn]
/Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
constructor Job in class Job is deprecated: see corresponding Javadoc for
more information.
[warn] val job = new Job(conf)
[warn] ^
[warn] 14 warnings found
[warn] Note:
/Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
uses unchecked or unsafe operations.
[warn] Note: Recompile with -Xlint:unchecked for details.
[info] Compiling 15 Scala sources to
/Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.YarnClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
not found: value YarnClient
[error] val yarnClient = YarnClient.createYarnClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
object util is not a member of package org.apache.hadoop.yarn.webapp
[error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
value RM_AM_MAX_ATTEMPTS is not a member of object
org.apache.hadoop.yarn.conf.YarnConfiguration
[error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
not found: type AMRMClient
[error] private var amClient: AMRMClient[ContainerRequest] = _
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
not found: value AMRMClient
[error] amClient = AMRMClient.createAMRMClient()
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
not found: value WebAppUtils
[error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
not found: type AMRMClient
[error] amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
not found: type AMRMClient
[error] amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
not found: type AMRMClient
[error] amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
value CONTAINER_ID is not a member of object
org.apache.hadoop.yarn.api.ApplicationConstants.Environment
[error] val containerIdString = System.getenv(
ApplicationConstants.Environment.CONTAINER_ID.name())
[error]
^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
value setTokens is not a member of
org.apache.hadoop.yarn.api.records.ContainerLaunchContext
[error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
object util is not a member of package org.apache.hadoop.yarn.webapp
[error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
not found: type AMRMClient
[error] private var amClient: AMRMClient[ContainerRequest] = _
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
not found: value AMRMClient
[error] amClient = AMRMClient.createAMRMClient()
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
not found: value WebAppUtils
[error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
object ProtoUtils is not a member of package
org.apache.hadoop.yarn.api.records.impl.pb
[error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.NMClient
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
not found: type NMClient
[error] var nmClient: NMClient = _
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
not found: value NMClient
[error] nmClient = NMClient.createNMClient()
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
value setTokens is not a member of
org.apache.hadoop.yarn.api.records.ContainerLaunchContext
[error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
object ApplicationMasterProtocol is not a member of package
org.apache.hadoop.yarn.api
[error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
object api is not a member of package org.apache.hadoop.yarn.client
[error] import org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
not found: type AMRMClient
[error] val amClient: AMRMClient[ContainerRequest],
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
not found: type ContainerRequest
[error] ): ArrayBuffer[ContainerRequest] = {
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
not found: type ContainerRequest
[error] hostContainers: ArrayBuffer[ContainerRequest]
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
not found: type ContainerRequest
[error] val requestedContainers = new
ArrayBuffer[ContainerRequest](rackToCounts.size)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
not found: type ContainerRequest
[error] val containerRequests: List[ContainerRequest] =
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
not found: type ContainerRequest
[error] ): ArrayBuffer[ContainerRequest] = {
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
not found: type ContainerRequest
[error] val hostContainerRequests = new
ArrayBuffer[ContainerRequest](preferredHostToCount.size)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
not found: type ContainerRequest
[error] val rackContainerRequests: List[ContainerRequest] =
createRackResourceRequests(
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
not found: type ContainerRequest
[error] val containerRequestBuffer = new
ArrayBuffer[ContainerRequest](
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
not found: type ContainerRequest
[error] ): ArrayBuffer[ContainerRequest] = {
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
value newInstance is not a member of object
org.apache.hadoop.yarn.api.records.Resource
[error] val resource = Resource.newInstance(memoryRequest,
executorCores)
[error] ^
[error]
/Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
not found: type ContainerRequest
[error] val requests = new ArrayBuffer[ContainerRequest]()
[error] ^
[error] 40 errors found
[error] (yarn-stable/compile:compile) Compilation failed
[error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com> wrote:
> Hi Ron,
>
> I just checked and this bug is fixed in recent releases of Spark.
>
> -Sandy
>
>
> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
>
>> Ron,
>> Which distribution and Version of Hadoop are you using ?
>>
>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>> 2.3.0-cdh5.0.0),
>>
>> MRJobConfig does have the field :
>>
>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>
>> Chester
>>
>>
>>
>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>> wrote:
>>
>>> Hi,
>>> I was doing programmatic submission of Spark yarn jobs and I saw code
>>> in ClientBase.getDefaultYarnApplicationClasspath():
>>>
>>> val field =
>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>> MRJobConfig doesn't have this field so the created launch env is
>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>
>>> This results in having the spark job hang if the submission config is
>>> different from the default config. For example, if my resource manager port
>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>> and stays in ACCEPTED state.
>>>
>>> I can easily fix this by changing this to YarnConfiguration instead of
>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>
>>> Thanks,
>>> Ron
>>>
>>> Sent from my iPhone
>>
>>
>>
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
OK I will create PR.
thanks
On Thu, Jul 17, 2014 at 7:58 AM, Sean Owen <so...@cloudera.com> wrote:
> Looks like a real problem. I see it too. I think the same workaround
> found in ClientBase.scala needs to be used here. There, the fact that
> this field can be a String or String[] is handled explicitly. In fact
> I think you can just call to ClientBase for this? PR it, I say.
>
> On Thu, Jul 17, 2014 at 3:24 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
> > val knownDefMRAppCP: Seq[String] =
> > getFieldValue[String, Seq[String]](classOf[MRJobConfig],
> >
> > "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> > Seq[String]())(a =>
> a.split(","))
> >
> > will fail for yarn-alpha.
> >
> > sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
> >
>
Re: Possible bug in ClientBase.scala?
Posted by Sean Owen <so...@cloudera.com>.
Looks like a real problem. I see it too. I think the same workaround
found in ClientBase.scala needs to be used here. There, the fact that
this field can be a String or String[] is handled explicitly. In fact
I think you can just call to ClientBase for this? PR it, I say.
On Thu, Jul 17, 2014 at 3:24 PM, Chester Chen <ch...@alpinenow.com> wrote:
> val knownDefMRAppCP: Seq[String] =
> getFieldValue[String, Seq[String]](classOf[MRJobConfig],
>
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> Seq[String]())(a => a.split(","))
>
> will fail for yarn-alpha.
>
> sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
@Sean and @Sandy
Thanks for the reply. I used to be able to see yarn-alpha and yarn
directories which corresponding to the modules.
I guess due to the recent SparkBuild.scala changes, I did not see
yarn-alpha (by default) and I thought yarn-alpha is renamed to "yarn" and
"yarn-stable" is the old yarn. So I compiled "yarn" against the
hadoop.version = 2.0.5-alpha. My mistake.
I tried
export SPARK_HADOOP_VERSION=2.0.5-alpha
sbt/sbt -Pyarn-alpha yarn-alpha/test
the compilation errors are all gone.
sbt/sbt -Pyarn-alpha projects
does show the yarn-alpha project, I did not realize this is dynamically
enabled based on yarn flag. Thanks Sean for pointing that out.
To Sandy's point, I am not trying to use alpha version of Yarn. I am
experimenting some changes in Yarn Client and refactoring code and just
want to make sure I am passing tests for both yarn-alpha and yarn-stable.
The yarn-alpha tests actually failing due to the yarn API changes in
MRJobConfig class.
as I mentioned in earlier email
The field
DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH
returns String in yarn-stable, but returns String array in yarn-alpha API.
So the method in ClientBaseSuite.scala
val knownDefMRAppCP: Seq[String] =
getFieldValue[String, Seq[String]](classOf[MRJobConfig],
"DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
Seq[String]())(a => a.split(","))
will fail for yarn-alpha.
sbt/sbt -Pyarn-alpha -Dhadoop.version=2.0.5-alpha yarn-alpha/test
...
4/07/17 07:07:16 INFO ClientBase: Using Spark's default log4j profile:
org/apache/spark/log4j-defaults.properties
[info] - default Yarn application classpath *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply$mcV$sp(ClientBaseSuite.scala:47)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply(ClientBaseSuite.scala:47)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$1.apply(ClientBaseSuite.scala:47)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - default MR application classpath *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply$mcV$sp(ClientBaseSuite.scala:51)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply(ClientBaseSuite.scala:51)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$2.apply(ClientBaseSuite.scala:51)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines a classpath
for YARN *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply$mcV$sp(ClientBaseSuite.scala:55)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply(ClientBaseSuite.scala:55)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$3.apply(ClientBaseSuite.scala:55)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines a classpath
for MR *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply$mcV$sp(ClientBaseSuite.scala:64)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply(ClientBaseSuite.scala:64)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$4.apply(ClientBaseSuite.scala:64)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - resultant classpath for an application that defines both
classpaths, YARN and MR *** FAILED ***
[info] java.lang.ClassCastException: [Ljava.lang.String; cannot be cast
to java.lang.String
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$$anonfun$12.apply(ClientBaseSuite.scala:152)
[info] at scala.Option.map(Option.scala:145)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.getFieldValue(ClientBaseSuite.scala:180)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$Fixtures$.<init>(ClientBaseSuite.scala:152)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures$lzycompute(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite.Fixtures(ClientBaseSuite.scala:141)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply$mcV$sp(ClientBaseSuite.scala:73)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply(ClientBaseSuite.scala:73)
[info] at
org.apache.spark.deploy.yarn.ClientBaseSuite$$anonfun$5.apply(ClientBaseSuite.scala:73)
[info] at
org.scalatest.Transformer$$anonfun$apply$1.apply(Transformer.scala:22)
[info] ...
[info] - Local jar URIs
On Thu, Jul 17, 2014 at 12:44 AM, Sandy Ryza <sa...@cloudera.com>
wrote:
> To add, we've made some effort to yarn-alpha to work with the 2.0.x line,
> but this was a time when YARN went through wild API changes. The only line
> that the yarn-alpha profile is guaranteed to work against is the 0.23 line.
>
>
> On Thu, Jul 17, 2014 at 12:40 AM, Sean Owen <so...@cloudera.com> wrote:
>
> > Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
> > "projects", shows it as a module. You should only build yarn-stable
> > *or* yarn-alpha at any given time.
> >
> > I don't remember the modules changing in a while. 'yarn-alpha' is for
> > YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
> > for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
> > is code common to both, so should compile with yarn-alpha.
> >
> > What's the compile error, and are you setting yarn.version? the
> > default is to use hadoop.version, but that defaults to 1.0.4 and there
> > is no such YARN.
> >
> > Unless I missed it, I only see compile errors in yarn-stable, and you
> > are trying to compile vs YARN alpha versions no?
> >
> > On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ch...@alpinenow.com>
> > wrote:
> > > Looking further, the yarn and yarn-stable are both for the stable
> version
> > > of Yarn, that explains the compilation errors when using 2.0.5-alpha
> > > version of hadoop.
> > >
> > > the module yarn-alpha ( although is still on SparkBuild.scala), is no
> > > longer there in sbt console.
> > >
> > >
> > >> projects
> > >
> > > [info] In file:/Users/chester/projects/spark/
> > >
> > > [info] assembly
> > >
> > > [info] bagel
> > >
> > > [info] catalyst
> > >
> > > [info] core
> > >
> > > [info] examples
> > >
> > > [info] graphx
> > >
> > > [info] hive
> > >
> > > [info] mllib
> > >
> > > [info] oldDeps
> > >
> > > [info] repl
> > >
> > > [info] spark
> > >
> > > [info] sql
> > >
> > > [info] streaming
> > >
> > > [info] streaming-flume
> > >
> > > [info] streaming-kafka
> > >
> > > [info] streaming-mqtt
> > >
> > > [info] streaming-twitter
> > >
> > > [info] streaming-zeromq
> > >
> > > [info] tools
> > >
> > > [info] yarn
> > >
> > > [info] * yarn-stable
> > >
> > >
> > > On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com>
> > wrote:
> > >
> > >> Hmm
> > >> looks like a Build script issue:
> > >>
> > >> I run the command with :
> > >>
> > >> sbt/sbt clean *yarn/*test:compile
> > >>
> > >> but errors came from
> > >>
> > >> [error] 40 errors found
> > >>
> > >> [error] (*yarn-stable*/compile:compile) Compilation failed
> > >>
> > >>
> > >> Chester
> > >>
> > >>
> > >> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
> > >> wrote:
> > >>
> > >>> Hi, Sandy
> > >>>
> > >>> We do have some issue with this. The difference is in Yarn-Alpha
> > and
> > >>> Yarn Stable ( I noticed that in the latest build, the module name has
> > >>> changed,
> > >>> yarn-alpha --> yarn
> > >>> yarn --> yarn-stable
> > >>> )
> > >>>
> > >>> For example: MRJobConfig.class
> > >>> the field:
> > >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
> > >>>
> > >>>
> > >>> In Yarn-Alpha : the field returns java.lang.String[]
> > >>>
> > >>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>
> > >>> while in Yarn-Stable, it returns a String
> > >>>
> > >>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>
> > >>> So in ClientBaseSuite.scala
> > >>>
> > >>> The following code:
> > >>>
> > >>> val knownDefMRAppCP: Seq[String] =
> > >>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
> > >>>
> > >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> > >>> Seq[String]())(a =>
> > >>> *a.split(",")*)
> > >>>
> > >>>
> > >>> works for yarn-stable, but doesn't work for yarn-alpha.
> > >>>
> > >>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago.
> I
> > >>> believe this can be refactored to yarn-alpha module and make
> different
> > >>> tests according different API signatures.
> > >>>
> > >>> I just update the master branch and build doesn't even compile for
> > >>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test
> > passed.
> > >>>
> > >>>
> > >>> Does the Spark Jenkins job run against yarn-alpha ?
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> Here is output from yarn-alpha compilation:
> > >>>
> > >>> I got the 40 compilation errors.
> > >>>
> > >>> sbt/sbt clean yarn/test:compile
> > >>>
> > >>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
> > as
> > >>> default JAVA_HOME.
> > >>>
> > >>> Note, this will be overridden by -java-home if it is set.
> > >>>
> > >>> [info] Loading project definition from
> > >>> /Users/chester/projects/spark/project/project
> > >>>
> > >>> [info] Loading project definition from
> > >>>
> >
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
> > >>>
> > >>> [warn] Multiple resolvers having different access mechanism
> configured
> > >>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove
> > duplicate
> > >>> project resolvers (`resolvers`) or rename publishing resolver
> > (`publishTo`).
> > >>>
> > >>> [info] Loading project definition from
> > >>> /Users/chester/projects/spark/project
> > >>>
> > >>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> > >>> -Dhadoop.version=2.0.5-alpha
> > >>>
> > >>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
> > >>>
> > >>> [info] Set current project to spark-parent (in build
> > >>> file:/Users/chester/projects/spark/)
> > >>>
> > >>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}core...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
> > >>>
> > >>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [info] Resolving commons-net#commons-net;3.1 ...
> > >>>
> > >>> [info] Compiling 358 Scala sources and 34 Java sources to
> > >>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
> > >>>
> > >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> > >>>
> > >>> [info] Done updating.
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> > >>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> > >>> corresponding Javadoc for more information.
> > >>>
> > >>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> > >>> attemptId)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewHadoopJob(conf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> > >>> constructor TaskID in class TaskID is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] new TaskAttemptID(new TaskID(jID.value, true,
> splitID),
> > >>> attemptID))
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> > >>> method makeQualified in class Path is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] outputPath.makeQualified(fs)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] if (!fs.getFileStatus(path).isDir) {
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] val logDirs = if (logStatus != null)
> > >>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
> > >>>
> > >>> [warn]
> ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] if (file.isDir) 0L else file.getLen
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> > >>> method getDefaultReplication in class FileSystem is deprecated: see
> > >>> corresponding Javadoc for more information.
> > >>>
> > >>> [warn] fs.create(tempOutputPath, false, bufferSize,
> > >>> fs.getDefaultReplication, blockSize)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> > >>> constructor TaskID in class TaskID is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
> > >>> splitId), attemptId)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> > >>> method isDir in class FileStatus is deprecated: see corresponding
> > Javadoc
> > >>> for more information.
> > >>>
> > >>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn]
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> > >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> > for
> > >>> more information.
> > >>>
> > >>> [warn] val job = new Job(conf)
> > >>>
> > >>> [warn] ^
> > >>>
> > >>> [warn] 14 warnings found
> > >>>
> > >>> [warn] Note:
> > >>>
> >
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> > >>> uses unchecked or unsafe operations.
> > >>>
> > >>> [warn] Note: Recompile with -Xlint:unchecked for details.
> > >>>
> > >>> [info] Compiling 15 Scala sources to
> > >>>
> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> > >>> not found: value YarnClient
> > >>>
> > >>> [error] val yarnClient = YarnClient.createYarnClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> > >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> > >>> value RM_AM_MAX_ATTEMPTS is not a member of object
> > >>> org.apache.hadoop.yarn.conf.YarnConfiguration
> > >>>
> > >>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> > >>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> > >>> not found: value AMRMClient
> > >>>
> > >>> [error] amClient = AMRMClient.createAMRMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> > >>> not found: value WebAppUtils
> > >>>
> > >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> > >>> value CONTAINER_ID is not a member of object
> > >>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> > >>>
> > >>> [error] val containerIdString = System.getenv(
> > >>> ApplicationConstants.Environment.CONTAINER_ID.name())
> > >>>
> > >>> [error]
> > >>> ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> > >>> value setTokens is not a member of
> > >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> > >>>
> > >>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> > >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> > >>> not found: value AMRMClient
> > >>>
> > >>> [error] amClient = AMRMClient.createAMRMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> > >>> not found: value WebAppUtils
> > >>>
> > >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> > >>> object ProtoUtils is not a member of package
> > >>> org.apache.hadoop.yarn.api.records.impl.pb
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.client.api.NMClient
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> > >>> not found: type NMClient
> > >>>
> > >>> [error] var nmClient: NMClient = _
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> > >>> not found: value NMClient
> > >>>
> > >>> [error] nmClient = NMClient.createNMClient()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> > >>> value setTokens is not a member of
> > >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> > >>>
> > >>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> > >>> object ApplicationMasterProtocol is not a member of package
> > >>> org.apache.hadoop.yarn.api
> > >>>
> > >>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> > >>> object api is not a member of package org.apache.hadoop.yarn.client
> > >>>
> > >>> [error] import
> > >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> > >>> not found: type AMRMClient
> > >>>
> > >>> [error] val amClient: AMRMClient[ContainerRequest],
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] hostContainers: ArrayBuffer[ContainerRequest]
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val requestedContainers = new
> > >>> ArrayBuffer[ContainerRequest](rackToCounts.size)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val containerRequests: List[ContainerRequest] =
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val hostContainerRequests = new
> > >>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val rackContainerRequests: List[ContainerRequest] =
> > >>> createRackResourceRequests(
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val containerRequestBuffer = new
> > >>> ArrayBuffer[ContainerRequest](
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] ): ArrayBuffer[ContainerRequest] = {
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> > >>> value newInstance is not a member of object
> > >>> org.apache.hadoop.yarn.api.records.Resource
> > >>>
> > >>> [error] val resource = Resource.newInstance(memoryRequest,
> > >>> executorCores)
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error]
> > >>>
> >
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> > >>> not found: type ContainerRequest
> > >>>
> > >>> [error] val requests = new ArrayBuffer[ContainerRequest]()
> > >>>
> > >>> [error] ^
> > >>>
> > >>> [error] 40 errors found
> > >>>
> > >>> [error] (yarn-stable/compile:compile) Compilation failed
> > >>>
> > >>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>>
> > >>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sandy.ryza@cloudera.com
> >
> > >>> wrote:
> > >>>
> > >>>> Hi Ron,
> > >>>>
> > >>>> I just checked and this bug is fixed in recent releases of Spark.
> > >>>>
> > >>>> -Sandy
> > >>>>
> > >>>>
> > >>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <
> chester@alpinenow.com>
> > >>>> wrote:
> > >>>>
> > >>>>> Ron,
> > >>>>> Which distribution and Version of Hadoop are you using ?
> > >>>>>
> > >>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> > >>>>> 2.3.0-cdh5.0.0),
> > >>>>>
> > >>>>> MRJobConfig does have the field :
> > >>>>>
> > >>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> > >>>>>
> > >>>>> Chester
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <
> zlgonzalez@yahoo.com>
> > >>>>> wrote:
> > >>>>>
> > >>>>>> Hi,
> > >>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
> > >>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
> > >>>>>>
> > >>>>>> val field =
> > >>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> > >>>>>> MRJobConfig doesn't have this field so the created launch env is
> > >>>>>> incomplete. Workaround is to set yarn.application.classpath with
> > the value
> > >>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
> > >>>>>>
> > >>>>>> This results in having the spark job hang if the submission config
> > is
> > >>>>>> different from the default config. For example, if my resource
> > manager port
> > >>>>>> is 8050 instead of 8030, then the spark app is not able to
> register
> > itself
> > >>>>>> and stays in ACCEPTED state.
> > >>>>>>
> > >>>>>> I can easily fix this by changing this to YarnConfiguration
> instead
> > of
> > >>>>>> MRJobConfig but was wondering what the steps are for submitting a
> > fix.
> > >>>>>>
> > >>>>>> Thanks,
> > >>>>>> Ron
> > >>>>>>
> > >>>>>> Sent from my iPhone
> > >>>>>
> > >>>>>
> > >>>>>
> > >>>>
> > >>>
> > >>
> >
>
Re: Possible bug in ClientBase.scala?
Posted by Sandy Ryza <sa...@cloudera.com>.
To add, we've made some effort to yarn-alpha to work with the 2.0.x line,
but this was a time when YARN went through wild API changes. The only line
that the yarn-alpha profile is guaranteed to work against is the 0.23 line.
On Thu, Jul 17, 2014 at 12:40 AM, Sean Owen <so...@cloudera.com> wrote:
> Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
> "projects", shows it as a module. You should only build yarn-stable
> *or* yarn-alpha at any given time.
>
> I don't remember the modules changing in a while. 'yarn-alpha' is for
> YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
> for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
> is code common to both, so should compile with yarn-alpha.
>
> What's the compile error, and are you setting yarn.version? the
> default is to use hadoop.version, but that defaults to 1.0.4 and there
> is no such YARN.
>
> Unless I missed it, I only see compile errors in yarn-stable, and you
> are trying to compile vs YARN alpha versions no?
>
> On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ch...@alpinenow.com>
> wrote:
> > Looking further, the yarn and yarn-stable are both for the stable version
> > of Yarn, that explains the compilation errors when using 2.0.5-alpha
> > version of hadoop.
> >
> > the module yarn-alpha ( although is still on SparkBuild.scala), is no
> > longer there in sbt console.
> >
> >
> >> projects
> >
> > [info] In file:/Users/chester/projects/spark/
> >
> > [info] assembly
> >
> > [info] bagel
> >
> > [info] catalyst
> >
> > [info] core
> >
> > [info] examples
> >
> > [info] graphx
> >
> > [info] hive
> >
> > [info] mllib
> >
> > [info] oldDeps
> >
> > [info] repl
> >
> > [info] spark
> >
> > [info] sql
> >
> > [info] streaming
> >
> > [info] streaming-flume
> >
> > [info] streaming-kafka
> >
> > [info] streaming-mqtt
> >
> > [info] streaming-twitter
> >
> > [info] streaming-zeromq
> >
> > [info] tools
> >
> > [info] yarn
> >
> > [info] * yarn-stable
> >
> >
> > On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
> >
> >> Hmm
> >> looks like a Build script issue:
> >>
> >> I run the command with :
> >>
> >> sbt/sbt clean *yarn/*test:compile
> >>
> >> but errors came from
> >>
> >> [error] 40 errors found
> >>
> >> [error] (*yarn-stable*/compile:compile) Compilation failed
> >>
> >>
> >> Chester
> >>
> >>
> >> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
> >> wrote:
> >>
> >>> Hi, Sandy
> >>>
> >>> We do have some issue with this. The difference is in Yarn-Alpha
> and
> >>> Yarn Stable ( I noticed that in the latest build, the module name has
> >>> changed,
> >>> yarn-alpha --> yarn
> >>> yarn --> yarn-stable
> >>> )
> >>>
> >>> For example: MRJobConfig.class
> >>> the field:
> >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
> >>>
> >>>
> >>> In Yarn-Alpha : the field returns java.lang.String[]
> >>>
> >>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>
> >>> while in Yarn-Stable, it returns a String
> >>>
> >>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>
> >>> So in ClientBaseSuite.scala
> >>>
> >>> The following code:
> >>>
> >>> val knownDefMRAppCP: Seq[String] =
> >>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
> >>>
> >>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> >>> Seq[String]())(a =>
> >>> *a.split(",")*)
> >>>
> >>>
> >>> works for yarn-stable, but doesn't work for yarn-alpha.
> >>>
> >>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
> >>> believe this can be refactored to yarn-alpha module and make different
> >>> tests according different API signatures.
> >>>
> >>> I just update the master branch and build doesn't even compile for
> >>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test
> passed.
> >>>
> >>>
> >>> Does the Spark Jenkins job run against yarn-alpha ?
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> Here is output from yarn-alpha compilation:
> >>>
> >>> I got the 40 compilation errors.
> >>>
> >>> sbt/sbt clean yarn/test:compile
> >>>
> >>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home
> as
> >>> default JAVA_HOME.
> >>>
> >>> Note, this will be overridden by -java-home if it is set.
> >>>
> >>> [info] Loading project definition from
> >>> /Users/chester/projects/spark/project/project
> >>>
> >>> [info] Loading project definition from
> >>>
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
> >>>
> >>> [warn] Multiple resolvers having different access mechanism configured
> >>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove
> duplicate
> >>> project resolvers (`resolvers`) or rename publishing resolver
> (`publishTo`).
> >>>
> >>> [info] Loading project definition from
> >>> /Users/chester/projects/spark/project
> >>>
> >>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> >>> -Dhadoop.version=2.0.5-alpha
> >>>
> >>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
> >>>
> >>> [info] Set current project to spark-parent (in build
> >>> file:/Users/chester/projects/spark/)
> >>>
> >>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}core...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
> >>>
> >>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [info] Resolving commons-net#commons-net;3.1 ...
> >>>
> >>> [info] Compiling 358 Scala sources and 34 Java sources to
> >>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
> >>>
> >>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
> >>>
> >>> [info] Done updating.
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> >>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> >>> corresponding Javadoc for more information.
> >>>
> >>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> >>> attemptId)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewHadoopJob(conf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> >>> constructor TaskID in class TaskID is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
> >>> attemptID))
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> >>> method makeQualified in class Path is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] outputPath.makeQualified(fs)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] if (!fs.getFileStatus(path).isDir) {
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] val logDirs = if (logStatus != null)
> >>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] if (file.isDir) 0L else file.getLen
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> >>> method getDefaultReplication in class FileSystem is deprecated: see
> >>> corresponding Javadoc for more information.
> >>>
> >>> [warn] fs.create(tempOutputPath, false, bufferSize,
> >>> fs.getDefaultReplication, blockSize)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> >>> constructor TaskID in class TaskID is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
> >>> splitId), attemptId)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> >>> method isDir in class FileStatus is deprecated: see corresponding
> Javadoc
> >>> for more information.
> >>>
> >>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
> >>>
> >>> [warn] ^
> >>>
> >>> [warn]
> >>>
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> >>> constructor Job in class Job is deprecated: see corresponding Javadoc
> for
> >>> more information.
> >>>
> >>> [warn] val job = new Job(conf)
> >>>
> >>> [warn] ^
> >>>
> >>> [warn] 14 warnings found
> >>>
> >>> [warn] Note:
> >>>
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> >>> uses unchecked or unsafe operations.
> >>>
> >>> [warn] Note: Recompile with -Xlint:unchecked for details.
> >>>
> >>> [info] Compiling 15 Scala sources to
> >>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> >>> not found: value YarnClient
> >>>
> >>> [error] val yarnClient = YarnClient.createYarnClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> >>>
> >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> >>> value RM_AM_MAX_ATTEMPTS is not a member of object
> >>> org.apache.hadoop.yarn.conf.YarnConfiguration
> >>>
> >>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> >>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> >>> not found: type AMRMClient
> >>>
> >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> >>> not found: value AMRMClient
> >>>
> >>> [error] amClient = AMRMClient.createAMRMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> >>> not found: value WebAppUtils
> >>>
> >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> >>> not found: type AMRMClient
> >>>
> >>> [error] amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> >>> value CONTAINER_ID is not a member of object
> >>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
> >>>
> >>> [error] val containerIdString = System.getenv(
> >>> ApplicationConstants.Environment.CONTAINER_ID.name())
> >>>
> >>> [error]
> >>> ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> >>> value setTokens is not a member of
> >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> >>>
> >>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> >>> object util is not a member of package org.apache.hadoop.yarn.webapp
> >>>
> >>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> >>> not found: type AMRMClient
> >>>
> >>> [error] private var amClient: AMRMClient[ContainerRequest] = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> >>> not found: value AMRMClient
> >>>
> >>> [error] amClient = AMRMClient.createAMRMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> >>> not found: value WebAppUtils
> >>>
> >>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> >>> object ProtoUtils is not a member of package
> >>> org.apache.hadoop.yarn.api.records.impl.pb
> >>>
> >>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import org.apache.hadoop.yarn.client.api.NMClient
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> >>> not found: type NMClient
> >>>
> >>> [error] var nmClient: NMClient = _
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> >>> not found: value NMClient
> >>>
> >>> [error] nmClient = NMClient.createNMClient()
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> >>> value setTokens is not a member of
> >>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
> >>>
> >>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> >>> object ApplicationMasterProtocol is not a member of package
> >>> org.apache.hadoop.yarn.api
> >>>
> >>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> >>> object api is not a member of package org.apache.hadoop.yarn.client
> >>>
> >>> [error] import
> >>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> >>> not found: type AMRMClient
> >>>
> >>> [error] val amClient: AMRMClient[ContainerRequest],
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] hostContainers: ArrayBuffer[ContainerRequest]
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val requestedContainers = new
> >>> ArrayBuffer[ContainerRequest](rackToCounts.size)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val containerRequests: List[ContainerRequest] =
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val hostContainerRequests = new
> >>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val rackContainerRequests: List[ContainerRequest] =
> >>> createRackResourceRequests(
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val containerRequestBuffer = new
> >>> ArrayBuffer[ContainerRequest](
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] ): ArrayBuffer[ContainerRequest] = {
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> >>> value newInstance is not a member of object
> >>> org.apache.hadoop.yarn.api.records.Resource
> >>>
> >>> [error] val resource = Resource.newInstance(memoryRequest,
> >>> executorCores)
> >>>
> >>> [error] ^
> >>>
> >>> [error]
> >>>
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> >>> not found: type ContainerRequest
> >>>
> >>> [error] val requests = new ArrayBuffer[ContainerRequest]()
> >>>
> >>> [error] ^
> >>>
> >>> [error] 40 errors found
> >>>
> >>> [error] (yarn-stable/compile:compile) Compilation failed
> >>>
> >>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
> >>> wrote:
> >>>
> >>>> Hi Ron,
> >>>>
> >>>> I just checked and this bug is fixed in recent releases of Spark.
> >>>>
> >>>> -Sandy
> >>>>
> >>>>
> >>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
> >>>> wrote:
> >>>>
> >>>>> Ron,
> >>>>> Which distribution and Version of Hadoop are you using ?
> >>>>>
> >>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
> >>>>> 2.3.0-cdh5.0.0),
> >>>>>
> >>>>> MRJobConfig does have the field :
> >>>>>
> >>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
> >>>>>
> >>>>> Chester
> >>>>>
> >>>>>
> >>>>>
> >>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi,
> >>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
> >>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
> >>>>>>
> >>>>>> val field =
> >>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
> >>>>>> MRJobConfig doesn't have this field so the created launch env is
> >>>>>> incomplete. Workaround is to set yarn.application.classpath with
> the value
> >>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
> >>>>>>
> >>>>>> This results in having the spark job hang if the submission config
> is
> >>>>>> different from the default config. For example, if my resource
> manager port
> >>>>>> is 8050 instead of 8030, then the spark app is not able to register
> itself
> >>>>>> and stays in ACCEPTED state.
> >>>>>>
> >>>>>> I can easily fix this by changing this to YarnConfiguration instead
> of
> >>>>>> MRJobConfig but was wondering what the steps are for submitting a
> fix.
> >>>>>>
> >>>>>> Thanks,
> >>>>>> Ron
> >>>>>>
> >>>>>> Sent from my iPhone
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>>
> >>
>
Re: Possible bug in ClientBase.scala?
Posted by Sean Owen <so...@cloudera.com>.
Are you setting -Pyarn-alpha? ./sbt/sbt -Pyarn-alpha, followed by
"projects", shows it as a module. You should only build yarn-stable
*or* yarn-alpha at any given time.
I don't remember the modules changing in a while. 'yarn-alpha' is for
YARN before it stabilized, circa early Hadoop 2.0.x. 'yarn-stable' is
for beta and stable YARN, circa late Hadoop 2.0.x and onwards. 'yarn'
is code common to both, so should compile with yarn-alpha.
What's the compile error, and are you setting yarn.version? the
default is to use hadoop.version, but that defaults to 1.0.4 and there
is no such YARN.
Unless I missed it, I only see compile errors in yarn-stable, and you
are trying to compile vs YARN alpha versions no?
On Thu, Jul 17, 2014 at 5:39 AM, Chester Chen <ch...@alpinenow.com> wrote:
> Looking further, the yarn and yarn-stable are both for the stable version
> of Yarn, that explains the compilation errors when using 2.0.5-alpha
> version of hadoop.
>
> the module yarn-alpha ( although is still on SparkBuild.scala), is no
> longer there in sbt console.
>
>
>> projects
>
> [info] In file:/Users/chester/projects/spark/
>
> [info] assembly
>
> [info] bagel
>
> [info] catalyst
>
> [info] core
>
> [info] examples
>
> [info] graphx
>
> [info] hive
>
> [info] mllib
>
> [info] oldDeps
>
> [info] repl
>
> [info] spark
>
> [info] sql
>
> [info] streaming
>
> [info] streaming-flume
>
> [info] streaming-kafka
>
> [info] streaming-mqtt
>
> [info] streaming-twitter
>
> [info] streaming-zeromq
>
> [info] tools
>
> [info] yarn
>
> [info] * yarn-stable
>
>
> On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com> wrote:
>
>> Hmm
>> looks like a Build script issue:
>>
>> I run the command with :
>>
>> sbt/sbt clean *yarn/*test:compile
>>
>> but errors came from
>>
>> [error] 40 errors found
>>
>> [error] (*yarn-stable*/compile:compile) Compilation failed
>>
>>
>> Chester
>>
>>
>> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
>> wrote:
>>
>>> Hi, Sandy
>>>
>>> We do have some issue with this. The difference is in Yarn-Alpha and
>>> Yarn Stable ( I noticed that in the latest build, the module name has
>>> changed,
>>> yarn-alpha --> yarn
>>> yarn --> yarn-stable
>>> )
>>>
>>> For example: MRJobConfig.class
>>> the field:
>>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>>>
>>>
>>> In Yarn-Alpha : the field returns java.lang.String[]
>>>
>>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> while in Yarn-Stable, it returns a String
>>>
>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> So in ClientBaseSuite.scala
>>>
>>> The following code:
>>>
>>> val knownDefMRAppCP: Seq[String] =
>>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>>>
>>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
>>> Seq[String]())(a =>
>>> *a.split(",")*)
>>>
>>>
>>> works for yarn-stable, but doesn't work for yarn-alpha.
>>>
>>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
>>> believe this can be refactored to yarn-alpha module and make different
>>> tests according different API signatures.
>>>
>>> I just update the master branch and build doesn't even compile for
>>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>>>
>>>
>>> Does the Spark Jenkins job run against yarn-alpha ?
>>>
>>>
>>>
>>>
>>>
>>> Here is output from yarn-alpha compilation:
>>>
>>> I got the 40 compilation errors.
>>>
>>> sbt/sbt clean yarn/test:compile
>>>
>>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
>>> default JAVA_HOME.
>>>
>>> Note, this will be overridden by -java-home if it is set.
>>>
>>> [info] Loading project definition from
>>> /Users/chester/projects/spark/project/project
>>>
>>> [info] Loading project definition from
>>> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>>>
>>> [warn] Multiple resolvers having different access mechanism configured
>>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
>>> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>>>
>>> [info] Loading project definition from
>>> /Users/chester/projects/spark/project
>>>
>>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
>>> -Dhadoop.version=2.0.5-alpha
>>>
>>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>>>
>>> [info] Set current project to spark-parent (in build
>>> file:/Users/chester/projects/spark/)
>>>
>>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}core...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>>>
>>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [info] Resolving commons-net#commons-net;3.1 ...
>>>
>>> [info] Compiling 358 Scala sources and 34 Java sources to
>>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>>>
>>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>>
>>> [info] Done updating.
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
>>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
>>> corresponding Javadoc for more information.
>>>
>>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
>>> attemptId)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewHadoopJob(conf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
>>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
>>> attemptID))
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
>>> method makeQualified in class Path is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] outputPath.makeQualified(fs)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] if (!fs.getFileStatus(path).isDir) {
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] val logDirs = if (logStatus != null)
>>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] if (file.isDir) 0L else file.getLen
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
>>> method getDefaultReplication in class FileSystem is deprecated: see
>>> corresponding Javadoc for more information.
>>>
>>> [warn] fs.create(tempOutputPath, false, bufferSize,
>>> fs.getDefaultReplication, blockSize)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
>>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
>>> splitId), attemptId)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
>>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>>> for more information.
>>>
>>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>>>
>>> [warn] ^
>>>
>>> [warn]
>>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
>>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>>> more information.
>>>
>>> [warn] val job = new Job(conf)
>>>
>>> [warn] ^
>>>
>>> [warn] 14 warnings found
>>>
>>> [warn] Note:
>>> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
>>> uses unchecked or unsafe operations.
>>>
>>> [warn] Note: Recompile with -Xlint:unchecked for details.
>>>
>>> [info] Compiling 15 Scala sources to
>>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>>> not found: value YarnClient
>>>
>>> [error] val yarnClient = YarnClient.createYarnClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>>
>>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
>>> value RM_AM_MAX_ATTEMPTS is not a member of object
>>> org.apache.hadoop.yarn.conf.YarnConfiguration
>>>
>>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
>>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
>>> not found: type AMRMClient
>>>
>>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
>>> not found: value AMRMClient
>>>
>>> [error] amClient = AMRMClient.createAMRMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
>>> not found: value WebAppUtils
>>>
>>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>>> not found: type AMRMClient
>>>
>>> [error] amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
>>> value CONTAINER_ID is not a member of object
>>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>>>
>>> [error] val containerIdString = System.getenv(
>>> ApplicationConstants.Environment.CONTAINER_ID.name())
>>>
>>> [error]
>>> ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
>>> value setTokens is not a member of
>>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>>
>>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
>>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>>
>>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
>>> not found: type AMRMClient
>>>
>>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
>>> not found: value AMRMClient
>>>
>>> [error] amClient = AMRMClient.createAMRMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
>>> not found: value WebAppUtils
>>>
>>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
>>> object ProtoUtils is not a member of package
>>> org.apache.hadoop.yarn.api.records.impl.pb
>>>
>>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import org.apache.hadoop.yarn.client.api.NMClient
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
>>> not found: type NMClient
>>>
>>> [error] var nmClient: NMClient = _
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
>>> not found: value NMClient
>>>
>>> [error] nmClient = NMClient.createNMClient()
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
>>> value setTokens is not a member of
>>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>>
>>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
>>> object ApplicationMasterProtocol is not a member of package
>>> org.apache.hadoop.yarn.api
>>>
>>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
>>> object api is not a member of package org.apache.hadoop.yarn.client
>>>
>>> [error] import
>>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
>>> not found: type AMRMClient
>>>
>>> [error] val amClient: AMRMClient[ContainerRequest],
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
>>> not found: type ContainerRequest
>>>
>>> [error] hostContainers: ArrayBuffer[ContainerRequest]
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
>>> not found: type ContainerRequest
>>>
>>> [error] val requestedContainers = new
>>> ArrayBuffer[ContainerRequest](rackToCounts.size)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
>>> not found: type ContainerRequest
>>>
>>> [error] val containerRequests: List[ContainerRequest] =
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
>>> not found: type ContainerRequest
>>>
>>> [error] val hostContainerRequests = new
>>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
>>> not found: type ContainerRequest
>>>
>>> [error] val rackContainerRequests: List[ContainerRequest] =
>>> createRackResourceRequests(
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
>>> not found: type ContainerRequest
>>>
>>> [error] val containerRequestBuffer = new
>>> ArrayBuffer[ContainerRequest](
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
>>> not found: type ContainerRequest
>>>
>>> [error] ): ArrayBuffer[ContainerRequest] = {
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
>>> value newInstance is not a member of object
>>> org.apache.hadoop.yarn.api.records.Resource
>>>
>>> [error] val resource = Resource.newInstance(memoryRequest,
>>> executorCores)
>>>
>>> [error] ^
>>>
>>> [error]
>>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
>>> not found: type ContainerRequest
>>>
>>> [error] val requests = new ArrayBuffer[ContainerRequest]()
>>>
>>> [error] ^
>>>
>>> [error] 40 errors found
>>>
>>> [error] (yarn-stable/compile:compile) Compilation failed
>>>
>>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
>>> wrote:
>>>
>>>> Hi Ron,
>>>>
>>>> I just checked and this bug is fixed in recent releases of Spark.
>>>>
>>>> -Sandy
>>>>
>>>>
>>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
>>>> wrote:
>>>>
>>>>> Ron,
>>>>> Which distribution and Version of Hadoop are you using ?
>>>>>
>>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>>>> 2.3.0-cdh5.0.0),
>>>>>
>>>>> MRJobConfig does have the field :
>>>>>
>>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>>>
>>>>> Chester
>>>>>
>>>>>
>>>>>
>>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>>>>> wrote:
>>>>>
>>>>>> Hi,
>>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
>>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
>>>>>>
>>>>>> val field =
>>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>>>> MRJobConfig doesn't have this field so the created launch env is
>>>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>>>
>>>>>> This results in having the spark job hang if the submission config is
>>>>>> different from the default config. For example, if my resource manager port
>>>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>>>> and stays in ACCEPTED state.
>>>>>>
>>>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>>>
>>>>>> Thanks,
>>>>>> Ron
>>>>>>
>>>>>> Sent from my iPhone
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
Looking further, the yarn and yarn-stable are both for the stable version
of Yarn, that explains the compilation errors when using 2.0.5-alpha
version of hadoop.
the module yarn-alpha ( although is still on SparkBuild.scala), is no
longer there in sbt console.
> projects
[info] In file:/Users/chester/projects/spark/
[info] assembly
[info] bagel
[info] catalyst
[info] core
[info] examples
[info] graphx
[info] hive
[info] mllib
[info] oldDeps
[info] repl
[info] spark
[info] sql
[info] streaming
[info] streaming-flume
[info] streaming-kafka
[info] streaming-mqtt
[info] streaming-twitter
[info] streaming-zeromq
[info] tools
[info] yarn
[info] * yarn-stable
On Wed, Jul 16, 2014 at 5:41 PM, Chester Chen <ch...@alpinenow.com> wrote:
> Hmm
> looks like a Build script issue:
>
> I run the command with :
>
> sbt/sbt clean *yarn/*test:compile
>
> but errors came from
>
> [error] 40 errors found
>
> [error] (*yarn-stable*/compile:compile) Compilation failed
>
>
> Chester
>
>
> On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com>
> wrote:
>
>> Hi, Sandy
>>
>> We do have some issue with this. The difference is in Yarn-Alpha and
>> Yarn Stable ( I noticed that in the latest build, the module name has
>> changed,
>> yarn-alpha --> yarn
>> yarn --> yarn-stable
>> )
>>
>> For example: MRJobConfig.class
>> the field:
>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>>
>>
>> In Yarn-Alpha : the field returns java.lang.String[]
>>
>> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>
>> while in Yarn-Stable, it returns a String
>>
>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>
>> So in ClientBaseSuite.scala
>>
>> The following code:
>>
>> val knownDefMRAppCP: Seq[String] =
>> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>>
>> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
>> Seq[String]())(a =>
>> *a.split(",")*)
>>
>>
>> works for yarn-stable, but doesn't work for yarn-alpha.
>>
>> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
>> believe this can be refactored to yarn-alpha module and make different
>> tests according different API signatures.
>>
>> I just update the master branch and build doesn't even compile for
>> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>>
>>
>> Does the Spark Jenkins job run against yarn-alpha ?
>>
>>
>>
>>
>>
>> Here is output from yarn-alpha compilation:
>>
>> I got the 40 compilation errors.
>>
>> sbt/sbt clean yarn/test:compile
>>
>> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
>> default JAVA_HOME.
>>
>> Note, this will be overridden by -java-home if it is set.
>>
>> [info] Loading project definition from
>> /Users/chester/projects/spark/project/project
>>
>> [info] Loading project definition from
>> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>>
>> [warn] Multiple resolvers having different access mechanism configured
>> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
>> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>>
>> [info] Loading project definition from
>> /Users/chester/projects/spark/project
>>
>> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
>> -Dhadoop.version=2.0.5-alpha
>>
>> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>>
>> [info] Set current project to spark-parent (in build
>> file:/Users/chester/projects/spark/)
>>
>> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>>
>> [info] Updating {file:/Users/chester/projects/spark/}core...
>>
>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>
>> [info] Done updating.
>>
>> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>>
>> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>>
>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>
>> [info] Done updating.
>>
>> [info] Resolving commons-net#commons-net;3.1 ...
>>
>> [info] Compiling 358 Scala sources and 34 Java sources to
>> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>>
>> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>>
>> [info] Done updating.
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
>> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
>> corresponding Javadoc for more information.
>>
>> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
>> attemptId)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewHadoopJob(conf)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
>> attemptID))
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
>> method makeQualified in class Path is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] outputPath.makeQualified(fs)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] if (!fs.getFileStatus(path).isDir) {
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] val logDirs = if (logStatus != null)
>> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] if (file.isDir) 0L else file.getLen
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
>> method getDefaultReplication in class FileSystem is deprecated: see
>> corresponding Javadoc for more information.
>>
>> [warn] fs.create(tempOutputPath, false, bufferSize,
>> fs.getDefaultReplication, blockSize)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
>> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true,
>> splitId), attemptId)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
>> method isDir in class FileStatus is deprecated: see corresponding Javadoc
>> for more information.
>>
>> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>>
>> [warn] ^
>>
>> [warn]
>> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
>> constructor Job in class Job is deprecated: see corresponding Javadoc for
>> more information.
>>
>> [warn] val job = new Job(conf)
>>
>> [warn] ^
>>
>> [warn] 14 warnings found
>>
>> [warn] Note:
>> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
>> uses unchecked or unsafe operations.
>>
>> [warn] Note: Recompile with -Xlint:unchecked for details.
>>
>> [info] Compiling 15 Scala sources to
>> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
>> not found: value YarnClient
>>
>> [error] val yarnClient = YarnClient.createYarnClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import
>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>
>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
>> value RM_AM_MAX_ATTEMPTS is not a member of object
>> org.apache.hadoop.yarn.conf.YarnConfiguration
>>
>> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
>> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
>> not found: type AMRMClient
>>
>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
>> not found: value AMRMClient
>>
>> [error] amClient = AMRMClient.createAMRMClient()
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
>> not found: value WebAppUtils
>>
>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
>> not found: type AMRMClient
>>
>> [error] amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
>> not found: type AMRMClient
>>
>> [error] amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
>> not found: type AMRMClient
>>
>> [error] amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
>> value CONTAINER_ID is not a member of object
>> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>>
>> [error] val containerIdString = System.getenv(
>> ApplicationConstants.Environment.CONTAINER_ID.name())
>>
>> [error]
>> ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
>> value setTokens is not a member of
>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>
>> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import
>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
>> object util is not a member of package org.apache.hadoop.yarn.webapp
>>
>> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
>> not found: type AMRMClient
>>
>> [error] private var amClient: AMRMClient[ContainerRequest] = _
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
>> not found: value AMRMClient
>>
>> [error] amClient = AMRMClient.createAMRMClient()
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
>> not found: value WebAppUtils
>>
>> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
>> object ProtoUtils is not a member of package
>> org.apache.hadoop.yarn.api.records.impl.pb
>>
>> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import org.apache.hadoop.yarn.client.api.NMClient
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
>> not found: type NMClient
>>
>> [error] var nmClient: NMClient = _
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
>> not found: value NMClient
>>
>> [error] nmClient = NMClient.createNMClient()
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
>> value setTokens is not a member of
>> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>>
>> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
>> object ApplicationMasterProtocol is not a member of package
>> org.apache.hadoop.yarn.api
>>
>> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
>> object api is not a member of package org.apache.hadoop.yarn.client
>>
>> [error] import
>> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
>> not found: type AMRMClient
>>
>> [error] val amClient: AMRMClient[ContainerRequest],
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
>> not found: type ContainerRequest
>>
>> [error] ): ArrayBuffer[ContainerRequest] = {
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
>> not found: type ContainerRequest
>>
>> [error] hostContainers: ArrayBuffer[ContainerRequest]
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
>> not found: type ContainerRequest
>>
>> [error] val requestedContainers = new
>> ArrayBuffer[ContainerRequest](rackToCounts.size)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
>> not found: type ContainerRequest
>>
>> [error] val containerRequests: List[ContainerRequest] =
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
>> not found: type ContainerRequest
>>
>> [error] ): ArrayBuffer[ContainerRequest] = {
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
>> not found: type ContainerRequest
>>
>> [error] val hostContainerRequests = new
>> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
>> not found: type ContainerRequest
>>
>> [error] val rackContainerRequests: List[ContainerRequest] =
>> createRackResourceRequests(
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
>> not found: type ContainerRequest
>>
>> [error] val containerRequestBuffer = new
>> ArrayBuffer[ContainerRequest](
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
>> not found: type ContainerRequest
>>
>> [error] ): ArrayBuffer[ContainerRequest] = {
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
>> value newInstance is not a member of object
>> org.apache.hadoop.yarn.api.records.Resource
>>
>> [error] val resource = Resource.newInstance(memoryRequest,
>> executorCores)
>>
>> [error] ^
>>
>> [error]
>> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
>> not found: type ContainerRequest
>>
>> [error] val requests = new ArrayBuffer[ContainerRequest]()
>>
>> [error] ^
>>
>> [error] 40 errors found
>>
>> [error] (yarn-stable/compile:compile) Compilation failed
>>
>> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
>> wrote:
>>
>>> Hi Ron,
>>>
>>> I just checked and this bug is fixed in recent releases of Spark.
>>>
>>> -Sandy
>>>
>>>
>>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
>>> wrote:
>>>
>>>> Ron,
>>>> Which distribution and Version of Hadoop are you using ?
>>>>
>>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>>> 2.3.0-cdh5.0.0),
>>>>
>>>> MRJobConfig does have the field :
>>>>
>>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>>
>>>> Chester
>>>>
>>>>
>>>>
>>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>>>> wrote:
>>>>
>>>>> Hi,
>>>>> I was doing programmatic submission of Spark yarn jobs and I saw
>>>>> code in ClientBase.getDefaultYarnApplicationClasspath():
>>>>>
>>>>> val field =
>>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>>> MRJobConfig doesn't have this field so the created launch env is
>>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>>
>>>>> This results in having the spark job hang if the submission config is
>>>>> different from the default config. For example, if my resource manager port
>>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>>> and stays in ACCEPTED state.
>>>>>
>>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>>
>>>>> Thanks,
>>>>> Ron
>>>>>
>>>>> Sent from my iPhone
>>>>
>>>>
>>>>
>>>
>>
>
Re: Possible bug in ClientBase.scala?
Posted by Chester Chen <ch...@alpinenow.com>.
Hmm
looks like a Build script issue:
I run the command with :
sbt/sbt clean *yarn/*test:compile
but errors came from
[error] 40 errors found
[error] (*yarn-stable*/compile:compile) Compilation failed
Chester
On Wed, Jul 16, 2014 at 5:18 PM, Chester Chen <ch...@alpinenow.com> wrote:
> Hi, Sandy
>
> We do have some issue with this. The difference is in Yarn-Alpha and
> Yarn Stable ( I noticed that in the latest build, the module name has
> changed,
> yarn-alpha --> yarn
> yarn --> yarn-stable
> )
>
> For example: MRJobConfig.class
> the field:
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH"
>
>
> In Yarn-Alpha : the field returns java.lang.String[]
>
> java.lang.String[] DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> while in Yarn-Stable, it returns a String
>
> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>
> So in ClientBaseSuite.scala
>
> The following code:
>
> val knownDefMRAppCP: Seq[String] =
> getFieldValue[*String*, Seq[String]](classOf[MRJobConfig],
>
> "DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH",
> Seq[String]())(a =>
> *a.split(",")*)
>
>
> works for yarn-stable, but doesn't work for yarn-alpha.
>
> This is the only failure for the SNAPSHOT I downloaded 2 weeks ago. I
> believe this can be refactored to yarn-alpha module and make different
> tests according different API signatures.
>
> I just update the master branch and build doesn't even compile for
> Yarn-Alpha (yarn) model. Yarn-Stable compile with no error and test passed.
>
>
> Does the Spark Jenkins job run against yarn-alpha ?
>
>
>
>
>
> Here is output from yarn-alpha compilation:
>
> I got the 40 compilation errors.
>
> sbt/sbt clean yarn/test:compile
>
> Using /Library/Java/JavaVirtualMachines/jdk1.7.0_51.jdk/Contents/Home as
> default JAVA_HOME.
>
> Note, this will be overridden by -java-home if it is set.
>
> [info] Loading project definition from
> /Users/chester/projects/spark/project/project
>
> [info] Loading project definition from
> /Users/chester/.sbt/0.13/staging/ec3aa8f39111944cc5f2/sbt-pom-reader/project
>
> [warn] Multiple resolvers having different access mechanism configured
> with same name 'sbt-plugin-releases'. To avoid conflict, Remove duplicate
> project resolvers (`resolvers`) or rename publishing resolver (`publishTo`).
>
> [info] Loading project definition from
> /Users/chester/projects/spark/project
>
> NOTE: SPARK_HADOOP_VERSION is deprecated, please use
> -Dhadoop.version=2.0.5-alpha
>
> NOTE: SPARK_YARN is deprecated, please use -Pyarn flag.
>
> [info] Set current project to spark-parent (in build
> file:/Users/chester/projects/spark/)
>
> [success] Total time: 0 s, completed Jul 16, 2014 5:13:06 PM
>
> [info] Updating {file:/Users/chester/projects/spark/}core...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [info] Updating {file:/Users/chester/projects/spark/}yarn...
>
> [info] Updating {file:/Users/chester/projects/spark/}yarn-stable...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [info] Resolving commons-net#commons-net;3.1 ...
>
> [info] Compiling 358 Scala sources and 34 Java sources to
> /Users/chester/projects/spark/core/target/scala-2.10/classes...
>
> [info] Resolving org.fusesource.jansi#jansi;1.4 ...
>
> [info] Done updating.
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/hadoop/mapred/SparkHadoopMapRedUtil.scala:43:
> constructor TaskAttemptID in class TaskAttemptID is deprecated: see
> corresponding Javadoc for more information.
>
> [warn] new TaskAttemptID(jtIdentifier, jobId, isMap, taskId,
> attemptId)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:501:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewHadoopJob(hadoopConfiguration)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkContext.scala:634:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewHadoopJob(conf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:167:
> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] new TaskAttemptID(new TaskID(jID.value, true, splitID),
> attemptID))
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/SparkHadoopWriter.scala:188:
> method makeQualified in class Path is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] outputPath.makeQualified(fs)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:84:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] if (!fs.getFileStatus(path).isDir) {
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/deploy/history/FsHistoryProvider.scala:118:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] val logDirs = if (logStatus != null)
> logStatus.filter(_.isDir).toSeq else Seq[FileStatus]()
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/input/WholeTextFileInputFormat.scala:56:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] if (file.isDir) 0L else file.getLen
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/CheckpointRDD.scala:110:
> method getDefaultReplication in class FileSystem is deprecated: see
> corresponding Javadoc for more information.
>
> [warn] fs.create(tempOutputPath, false, bufferSize,
> fs.getDefaultReplication, blockSize)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/HadoopRDD.scala:267:
> constructor TaskID in class TaskID is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] val taId = new TaskAttemptID(new TaskID(jobID, true, splitId),
> attemptId)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:767:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/rdd/PairRDDFunctions.scala:830:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new NewAPIHadoopJob(hadoopConf)
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/EventLoggingListener.scala:185:
> method isDir in class FileStatus is deprecated: see corresponding Javadoc
> for more information.
>
> [warn] fileStatuses.filter(!_.isDir).map(_.getPath).toSeq
>
> [warn] ^
>
> [warn]
> /Users/chester/projects/spark/core/src/main/scala/org/apache/spark/scheduler/InputFormatInfo.scala:106:
> constructor Job in class Job is deprecated: see corresponding Javadoc for
> more information.
>
> [warn] val job = new Job(conf)
>
> [warn] ^
>
> [warn] 14 warnings found
>
> [warn] Note:
> /Users/chester/projects/spark/core/src/main/java/org/apache/spark/api/java/JavaSparkContextVarargsWorkaround.java
> uses unchecked or unsafe operations.
>
> [warn] Note: Recompile with -Xlint:unchecked for details.
>
> [info] Compiling 15 Scala sources to
> /Users/chester/projects/spark/yarn/stable/target/scala-2.10/classes...
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:26:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.YarnClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:40:
> not found: value YarnClient
>
> [error] val yarnClient = YarnClient.createYarnClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:32:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:36:
> object util is not a member of package org.apache.hadoop.yarn.webapp
>
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:63:
> value RM_AM_MAX_ATTEMPTS is not a member of object
> org.apache.hadoop.yarn.conf.YarnConfiguration
>
> [error] YarnConfiguration.RM_AM_MAX_ATTEMPTS,
> YarnConfiguration.DEFAULT_RM_AM_MAX_ATTEMPTS)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:65:
> not found: type AMRMClient
>
> [error] private var amClient: AMRMClient[ContainerRequest] = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:91:
> not found: value AMRMClient
>
> [error] amClient = AMRMClient.createAMRMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:136:
> not found: value WebAppUtils
>
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:40:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:618:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:596:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:577:
> not found: type AMRMClient
>
> [error] amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ApplicationMaster.scala:452:
> value CONTAINER_ID is not a member of object
> org.apache.hadoop.yarn.api.ApplicationConstants.Environment
>
> [error] val containerIdString = System.getenv(
> ApplicationConstants.Environment.CONTAINER_ID.name())
>
> [error]
> ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/Client.scala:128:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>
> [error] amContainer.setTokens(ByteBuffer.wrap(dob.getData()))
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:36:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.AMRMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:37:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:39:
> object util is not a member of package org.apache.hadoop.yarn.webapp
>
> [error] import org.apache.hadoop.yarn.webapp.util.WebAppUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:62:
> not found: type AMRMClient
>
> [error] private var amClient: AMRMClient[ContainerRequest] = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:99:
> not found: value AMRMClient
>
> [error] amClient = AMRMClient.createAMRMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorLauncher.scala:158:
> not found: value WebAppUtils
>
> [error] val proxy = WebAppUtils.getProxyHostAndPort(conf)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:31:
> object ProtoUtils is not a member of package
> org.apache.hadoop.yarn.api.records.impl.pb
>
> [error] import org.apache.hadoop.yarn.api.records.impl.pb.ProtoUtils
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:33:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import org.apache.hadoop.yarn.client.api.NMClient
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:53:
> not found: type NMClient
>
> [error] var nmClient: NMClient = _
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:59:
> not found: value NMClient
>
> [error] nmClient = NMClient.createNMClient()
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/ExecutorRunnable.scala:79:
> value setTokens is not a member of
> org.apache.hadoop.yarn.api.records.ContainerLaunchContext
>
> [error] ctx.setTokens(ByteBuffer.wrap(dob.getData()))
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:35:
> object ApplicationMasterProtocol is not a member of package
> org.apache.hadoop.yarn.api
>
> [error] import org.apache.hadoop.yarn.api.ApplicationMasterProtocol
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:41:
> object api is not a member of package org.apache.hadoop.yarn.client
>
> [error] import
> org.apache.hadoop.yarn.client.api.AMRMClient.ContainerRequest
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:65:
> not found: type AMRMClient
>
> [error] val amClient: AMRMClient[ContainerRequest],
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:389:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:388:
> not found: type ContainerRequest
>
> [error] hostContainers: ArrayBuffer[ContainerRequest]
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:405:
> not found: type ContainerRequest
>
> [error] val requestedContainers = new
> ArrayBuffer[ContainerRequest](rackToCounts.size)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:434:
> not found: type ContainerRequest
>
> [error] val containerRequests: List[ContainerRequest] =
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:508:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:446:
> not found: type ContainerRequest
>
> [error] val hostContainerRequests = new
> ArrayBuffer[ContainerRequest](preferredHostToCount.size)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:458:
> not found: type ContainerRequest
>
> [error] val rackContainerRequests: List[ContainerRequest] =
> createRackResourceRequests(
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:467:
> not found: type ContainerRequest
>
> [error] val containerRequestBuffer = new
> ArrayBuffer[ContainerRequest](
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:542:
> not found: type ContainerRequest
>
> [error] ): ArrayBuffer[ContainerRequest] = {
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:545:
> value newInstance is not a member of object
> org.apache.hadoop.yarn.api.records.Resource
>
> [error] val resource = Resource.newInstance(memoryRequest,
> executorCores)
>
> [error] ^
>
> [error]
> /Users/chester/projects/spark/yarn/stable/src/main/scala/org/apache/spark/deploy/yarn/YarnAllocationHandler.scala:550:
> not found: type ContainerRequest
>
> [error] val requests = new ArrayBuffer[ContainerRequest]()
>
> [error] ^
>
> [error] 40 errors found
>
> [error] (yarn-stable/compile:compile) Compilation failed
>
> [error] Total time: 98 s, completed Jul 16, 2014 5:14:44 PM
>
>
>
>
>
>
>
>
>
>
>
>
>
> On Wed, Jul 16, 2014 at 4:19 PM, Sandy Ryza <sa...@cloudera.com>
> wrote:
>
>> Hi Ron,
>>
>> I just checked and this bug is fixed in recent releases of Spark.
>>
>> -Sandy
>>
>>
>> On Sun, Jul 13, 2014 at 8:15 PM, Chester Chen <ch...@alpinenow.com>
>> wrote:
>>
>>> Ron,
>>> Which distribution and Version of Hadoop are you using ?
>>>
>>> I just looked at CDH5 ( hadoop-mapreduce-client-core-
>>> 2.3.0-cdh5.0.0),
>>>
>>> MRJobConfig does have the field :
>>>
>>> java.lang.String DEFAULT_MAPREDUCE_APPLICATION_CLASSPATH;
>>>
>>> Chester
>>>
>>>
>>>
>>> On Sun, Jul 13, 2014 at 6:49 PM, Ron Gonzalez <zl...@yahoo.com>
>>> wrote:
>>>
>>>> Hi,
>>>> I was doing programmatic submission of Spark yarn jobs and I saw code
>>>> in ClientBase.getDefaultYarnApplicationClasspath():
>>>>
>>>> val field =
>>>> classOf[MRJobConfig].getField("DEFAULT_YARN_APPLICATION_CLASSPATH)
>>>> MRJobConfig doesn't have this field so the created launch env is
>>>> incomplete. Workaround is to set yarn.application.classpath with the value
>>>> from YarnConfiguration.DEFAULT_YARN_APPLICATION_CLASSPATH.
>>>>
>>>> This results in having the spark job hang if the submission config is
>>>> different from the default config. For example, if my resource manager port
>>>> is 8050 instead of 8030, then the spark app is not able to register itself
>>>> and stays in ACCEPTED state.
>>>>
>>>> I can easily fix this by changing this to YarnConfiguration instead of
>>>> MRJobConfig but was wondering what the steps are for submitting a fix.
>>>>
>>>> Thanks,
>>>> Ron
>>>>
>>>> Sent from my iPhone
>>>
>>>
>>>
>>
>