You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Ismaël Mejía (JIRA)" <ji...@apache.org> on 2016/07/19 15:23:20 UTC

[jira] [Created] (BEAM-470) Spark Runner does not send the job execution information into the Spark History Server

Ismaël Mejía created BEAM-470:
---------------------------------

             Summary: Spark Runner does not send the job execution information into the Spark History Server
                 Key: BEAM-470
                 URL: https://issues.apache.org/jira/browse/BEAM-470
             Project: Beam
          Issue Type: Improvement
          Components: runner-spark
    Affects Versions: 0.2.0-incubating
            Reporter: Ismaël Mejía
            Assignee: Amit Sela
            Priority: Minor


If you run a Beam pipeline using the spark runner from spark (via spark-submit), the execution is registered in the spark-history-server if it is active and configured.
if you do this directly from a main method with --runner=SparkRunner (the beam way) the Beam runner does not report the execution to the history server, it seems the issue is the runner does not take into account an existing spark configuration file SPARK_HOME/conf/spark-defaults.conf (or there is not a way to tell the runner to take such conf into account).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)