You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "alizademhdi (via GitHub)" <gi...@apache.org> on 2024/03/29 22:17:18 UTC

[PR] Fix inital value for _batch_id attribute in livy operator [airflow]

alizademhdi opened a new pull request, #38628:
URL: https://github.com/apache/airflow/pull/38628

   ### Related issue
   #37898
   
   ### Explain
   
   when the livy operator is rewaked up after trigger is time out, the _batch_id is not initialized, which cause it to raise this exeption:
   `AttributeError: 'LivyOperator' object has no attribute '_batch_id'`
    
   this problem arises because an initial value is not defined for _batch_id. by initializing this attribute to None, this problem will be solved.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2036432683

   Trigger should (even if it times out) produce a trigger event with batch_id information in it. So Airflow should not kill trigger, it should timeout itself before Airflow does it and signal timeout operation.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "mateuslatrova (via GitHub)" <gi...@apache.org>.
mateuslatrova commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2048507160

   Thanks for the idea, @potiuk! That makes much more sense. I will try to implement it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "alizademhdi (via GitHub)" <gi...@apache.org>.
alizademhdi commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2028073596

   Fixed static tests :D 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2131753717

   This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2028687443

   Can you add unit test to avoid regression?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "mateuslatrova (via GitHub)" <gi...@apache.org>.
mateuslatrova commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2035812370

   Hi! I was also working on issue #37898 , and one of the requirements the author mentioned was: 
   
   > "The Livy operator should fail due to a timeout, and simultaneously, it should kill the Spark job."
   
   The way you solved it makes the task fail, but it does not kill the Spark Job. 
   
   When the task is rewaked up after the trigger times out its `_batch_id` is `None`. So when the task's `kill` method is called, it will not call the `self.hook.delete_batch` method inside the `kill` method.
   
   I am trying to figure out a way to get that `_batch_id` when the task is rewaked up so that we could kill the Spark job, but it doesn't seem trivial... Any ideias?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "alizademhdi (via GitHub)" <gi...@apache.org>.
alizademhdi commented on PR #38628:
URL: https://github.com/apache/airflow/pull/38628#issuecomment-2030790977

   Sure. I will try.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix inital value for _batch_id attribute in livy operator [airflow]

Posted by "alizademhdi (via GitHub)" <gi...@apache.org>.
alizademhdi closed pull request #38628: Fix inital value for _batch_id attribute in livy operator
URL: https://github.com/apache/airflow/pull/38628


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org