You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@airflow.apache.org by "Fokko Driesprong (JIRA)" <ji...@apache.org> on 2017/09/04 18:37:00 UTC

[jira] [Created] (AIRFLOW-1562) Spark-sql deadlock in logging

Fokko Driesprong created AIRFLOW-1562:
-----------------------------------------

             Summary: Spark-sql deadlock in logging
                 Key: AIRFLOW-1562
                 URL: https://issues.apache.org/jira/browse/AIRFLOW-1562
             Project: Apache Airflow
          Issue Type: Bug
          Components: hooks
    Affects Versions: Airflow 1.8
            Reporter: Fokko Driesprong


Related to Issue 1255

Logging in SparkSubmitOperator does not work as intended (continuous logging as received in the subprocess). This is because, spark-submit internally redirects all logs to stdout (including stderr), which causes the current two iterator logging to get stuck with empty stderr pipe. The logs are written only when the subprocess finishes. This leads to yarn_application_id not being available until the end of application.




--
This message was sent by Atlassian JIRA
(v6.4.14#64029)