You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "Kris Geusebroek (JIRA)" <ji...@apache.org> on 2017/05/17 13:44:04 UTC

[jira] [Created] (AIRFLOW-1219) Spark submit hook fix realtime logging.

Kris Geusebroek created AIRFLOW-1219:
----------------------------------------

             Summary: Spark submit hook fix realtime logging.
                 Key: AIRFLOW-1219
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1219
             Project: Apache Airflow
          Issue Type: Bug
            Reporter: Kris Geusebroek


Redirecting the logging to airflow doesn't work correctly.

Spark's logging goes to stderr and output of the spark program goes to stdout. In the loop of the two sources stdout is used first.
It seems to be blocking until output is written at the end of the spark-submit execution.

Fix this by connecting stderr to stdout in the subprocess invocation and only pipe stdout to the airflow logging



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)