You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/11/28 07:56:53 UTC

[GitHub] [airflow] Athlestan opened a new issue #12680: SparkSubmitHook - allow log parsing

Athlestan opened a new issue #12680:
URL: https://github.com/apache/airflow/issues/12680


   **Description**
   
   Create a method that will allow developer to parse spark log as needed in SparkSubmitOperator to extract usefull information. Method should be designed to be overwritten.
   
   **Use case / motivation**
   
   I want to extract what business dates my spark application processed/inserted to hive table and pass it to next task in dag. My business logic is in my Spark application written in Scala that I run using Airflow SparkSubmitOperator. 
   
   My idea was to parse spark log in `SparkSubmitOperator` using `SparkSubmitHook` in method `_process_spark_submit_log` looking for specific information that I will log in my spark application. Then I would be able to pass that information using xcom to another task.
   
   
   **Related Issues**
   
   No.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #12680: SparkSubmitHook - allow log parsing

Posted by GitBox <gi...@apache.org>.
boring-cyborg[bot] commented on issue #12680:
URL: https://github.com/apache/airflow/issues/12680#issuecomment-735060480


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #12680: SparkSubmitHook - allow log parsing

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12680:
URL: https://github.com/apache/airflow/issues/12680#issuecomment-928205522


   This feels like a very custom use case for your needs.
   You can do that with a custom operator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #12680: SparkSubmitHook - allow log parsing

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12680:
URL: https://github.com/apache/airflow/issues/12680


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal closed issue #12680: SparkSubmitHook - allow log parsing

Posted by GitBox <gi...@apache.org>.
eladkal closed issue #12680:
URL: https://github.com/apache/airflow/issues/12680


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] eladkal commented on issue #12680: SparkSubmitHook - allow log parsing

Posted by GitBox <gi...@apache.org>.
eladkal commented on issue #12680:
URL: https://github.com/apache/airflow/issues/12680#issuecomment-928205522


   This feels like a very custom use case for your needs.
   You can do that with a custom operator.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org