You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@spark.apache.org by Anfernee Xu <an...@gmail.com> on 2015/10/15 01:20:12 UTC
Application not found in Spark historyserver in yarn-client mode
Hi,
Here's the problem I'm facing, I have a standalone java application which
is periodically submit Spark jobs to my yarn cluster, btw I'm not using
'spark-submit' or 'org.apache.spark.launcher' to submit my jobs. These jobs
are successful and I can see them on Yarn RM webUI, but when I want to
follow the link to the app history, I always got 404(application is not
found) from Spark historyserver.
My code looks likes as below
SparkConf conf = new
SparkConf().setAppName("testSpak").setMaster("yarn-client")
.setJars(new String[]{IOUtil.getJar(MySparkApp.class)});
conf.set("spark.yarn.historyServer.address", "10.247.44.155:18080");
conf.set("spark.history.fs.logDirectory",
"
hdfs://myHdfsNameNode:55310/scratch/tie/spark/applicationHistory");
JavaSparkContext sc = new JavaSparkContext(conf);
try {
... my application code
}finally{
sc.stop();
}
--
--Anfernee
Re: Application not found in Spark historyserver in yarn-client mode
Posted by Ted Yu <yu...@gmail.com>.
Which Spark release are you using ?
Thanks
On Wed, Oct 14, 2015 at 4:20 PM, Anfernee Xu <an...@gmail.com> wrote:
> Hi,
>
> Here's the problem I'm facing, I have a standalone java application which
> is periodically submit Spark jobs to my yarn cluster, btw I'm not using
> 'spark-submit' or 'org.apache.spark.launcher' to submit my jobs. These jobs
> are successful and I can see them on Yarn RM webUI, but when I want to
> follow the link to the app history, I always got 404(application is not
> found) from Spark historyserver.
>
> My code looks likes as below
>
>
> SparkConf conf = new
> SparkConf().setAppName("testSpak").setMaster("yarn-client")
> .setJars(new String[]{IOUtil.getJar(MySparkApp.class)});
>
> conf.set("spark.yarn.historyServer.address", "10.247.44.155:18080");
> conf.set("spark.history.fs.logDirectory",
> "
> hdfs://myHdfsNameNode:55310/scratch/tie/spark/applicationHistory");
>
> JavaSparkContext sc = new JavaSparkContext(conf);
>
> try {
>
> ... my application code
>
> }finally{
> sc.stop();
>
> }
>
> --
> --Anfernee
>