You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2020/08/11 20:26:51 UTC

[GitHub] [airflow] jaketf opened a new issue #10293: Add Dataproc Job Sensor

jaketf opened a new issue #10293:
URL: https://github.com/apache/airflow/issues/10293


   **Description**
   
   The Dataproc hook now has separate methods for [submitting](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataproc.py#L778) and [waiting for job completion](https://github.com/apache/airflow/blob/master/airflow/providers/google/cloud/hooks/dataproc.py#L705).
   We should improve this further by adding a method to simply poke the status of the job.
   Finally implement a `DataprocJobSensor` so we can use reschedule mode for jobs known to run for a long time. 
   
   **Use case / motivation**
   
   For Long running dataproc jobs that are known to run for a long time (e.g. > 1 hr) we should not block a slot for the entire duration of the job.
   
   **Related Issues**
   
   N/A
   
   **Other notes **
   @varundhussa is currently working on this for a customer with this use case.
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] mik-laj commented on issue #10293: Add Dataproc Job Sensor

Posted by GitBox <gi...@apache.org>.
mik-laj commented on issue #10293:
URL: https://github.com/apache/airflow/issues/10293#issuecomment-678667092


   > @varundhussa is currently working on this for a customer with this use case.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] varundhussa commented on issue #10293: Add Dataproc Job Sensor

Posted by GitBox <gi...@apache.org>.
varundhussa commented on issue #10293:
URL: https://github.com/apache/airflow/issues/10293#issuecomment-678921754


   Thanks @rootcss and @mik-laj 
   I have completed the sensor and am completing the test cases. I will raise a PR soon.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] kaxil closed issue #10293: Add Dataproc Job Sensor

Posted by GitBox <gi...@apache.org>.
kaxil closed issue #10293:
URL: https://github.com/apache/airflow/issues/10293


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [airflow] rootcss commented on issue #10293: Add Dataproc Job Sensor

Posted by GitBox <gi...@apache.org>.
rootcss commented on issue #10293:
URL: https://github.com/apache/airflow/issues/10293#issuecomment-678666802


   I'd like to work on this. Can you please assign this to me @mik-laj 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org