[jira] [Created] (AMBARI-17537) Spark history server stopped after deploy

["Weiqing Yang (JIRA)" <ji...@apache.org>]

Weiqing Yang created AMBARI-17537:
-------------------------------------

             Summary: Spark history server stopped after deploy
                 Key: AMBARI-17537
                 URL: https://issues.apache.org/jira/browse/AMBARI-17537
             Project: Ambari
          Issue Type: Bug
          Components: ambari-server
    Affects Versions: 2.4.0
         Environment: ambari-server version: ii  ambari-agent                               2.4.0.0-694                        amd64        Ambari Agent
ii  ambari-metrics-assembly                    2.4.0.0-694                        amd64        Ambari Metrics Assembly
ii  ambari-server                              2.4.0.0-694                        amd64        Ambari Server
ambari-server --hash: 678877dcd8342d6a6c8edfdacbf215d02c89f580
HDP Stack: 2.3
HDP Version: 2.3.2.0-2950
Ambari DB: :Oracle
Oozie/Hive DB: Oracle/Oracle
Security:yes
Security Type:MIT/MIT
Blueprints: false
Umask: 
JDK: OracleJDK8
HA: yes
Repo URL: http://dev.hortonworks.com.s3.amazonaws.com/ambari/debian7/2.x/updates/2.4.0.0/ambariqe.list
OS: Debian 7
Ambari User: root
Agents User: root
MOTD enabled: true
Customized service users: true
Is tmp exec: false
            Reporter: Weiqing Yang
            Assignee: Weiqing Yang
             Fix For: trunk, 2.4.0


After deploy Spark history server is stopped.
16/06/27 19:55:45 INFO HistoryServer: History provider class: org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
Exception in thread "main" java.lang.ClassNotFoundException: org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider
        at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
        at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
        at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
        at java.lang.Class.forName0(Native Method)
        at java.lang.Class.forName(Class.java:264)
        at org.apache.spark.deploy.history.HistoryServer$.main(HistoryServer.scala:224)
 at org.apache.spark.deploy.history.HistoryServer.main(HistoryServer.scala).

Why this jira happened?
In the jira, the spark version is 1.5.1. which doesnot has the class org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider. Instead, it has "org.apache.spark.deploy.yarn.history.YarnHistoryProvider" and "org.apache.spark.deploy.history.yarn.YarnHistoryProvider".

How to fix? 
Actually, in spark internal logic, "org.apache.spark.deploy.yarn.history.YarnHistoryProvider" is a wrapper of "org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider" and "org.apache.spark.deploy.history.yarn.YarnHistoryProvider".
The property "spark.history.provider" appears twice in the stack definition, having two values: one is "org.apache.spark.deploy.yarn.history.YarnHistoryProvider", and the other one is "org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider". In this patch, remove the one whose value is "org.apache.spark.deploy.history.yarn.server.YarnHistoryProvider".

The jira is originally reported by Denis Tarasyuk 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)