[GitHub] [airflow] jdonnelly-apixio opened a new issue #16620: additional spark on k8s operator sensor for sparkapp log tail -f

[GitBox <gi...@apache.org>]

jdonnelly-apixio opened a new issue #16620:
URL: https://github.com/apache/airflow/issues/16620


   **Description**
   I think it would be useful to have an additional sensor (or.. I'm not certain if it should be a sensor or something else) that would `tail -f` the spark app log. That log output in the airflow ui would make it a more user friendly way to see what the app is up to or why it failed.
   
   The current poke sensor is here:
   https://github.com/apache/airflow/blob/main/airflow/providers/cncf/kubernetes/sensors/spark_kubernetes.py
   
   
   **Use case / motivation**
   
   I was hoping to make it easy for developers to investigate spark app issues in the airflow ui.
   
   **Are you willing to submit a PR?**
   
   I plan on hacking something together that would do this at some point, but I will try to read up on docs and see if I can do it the correct way. If the correct way seams feasible, I will give it a shot.
   
   **Related Issues**
   
   n/a


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] boring-cyborg[bot] commented on issue #16620: additional spark on k8s operator sensor for sparkapp log tail -f

[GitBox <gi...@apache.org>]

boring-cyborg[bot] commented on issue #16620:
URL: https://github.com/apache/airflow/issues/16620#issuecomment-867209451


   Thanks for opening your first issue here! Be sure to follow the issue template!
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org