You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "fdemiane (via GitHub)" <gi...@apache.org> on 2023/10/04 12:27:59 UTC
[PR] Fix cloud run operation timeout error [airflow]
fdemiane opened a new pull request, #34755:
URL: https://github.com/apache/airflow/pull/34755
The CloudRunExectureJob operator is failing randomly in deferrable mode with a grpc error code 13: internal error. After investigation, it turns out that getting the execute operation status was timing out. The fix was to add a 2 minutes timeout to the get_operation function.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix cloud run operation timeout error [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #34755:
URL: https://github.com/apache/airflow/pull/34755#discussion_r1348312105
##########
airflow/providers/google/cloud/hooks/cloud_run.py:
##########
@@ -175,4 +175,6 @@ def get_conn(self):
return self._client
async def get_operation(self, operation_name: str) -> operations_pb2.Operation:
- return await self.get_conn().get_operation(operations_pb2.GetOperationRequest(name=operation_name))
+ return await self.get_conn().get_operation(
+ operations_pb2.GetOperationRequest(name=operation_name), timeout=120
Review Comment:
Should we make this `120` configurable?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix cloud run operation timeout error [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #34755:
URL: https://github.com/apache/airflow/pull/34755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix cloud run operation timeout error [airflow]
Posted by "fdemiane (via GitHub)" <gi...@apache.org>.
fdemiane commented on PR #34755:
URL: https://github.com/apache/airflow/pull/34755#issuecomment-1767977529
Hello @potiuk,
I would appreciate if you can take a look on this PR. Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Fix cloud run operation timeout error [airflow]
Posted by "fdemiane (via GitHub)" <gi...@apache.org>.
fdemiane commented on code in PR #34755:
URL: https://github.com/apache/airflow/pull/34755#discussion_r1356455859
##########
airflow/providers/google/cloud/hooks/cloud_run.py:
##########
@@ -175,4 +175,6 @@ def get_conn(self):
return self._client
async def get_operation(self, operation_name: str) -> operations_pb2.Operation:
- return await self.get_conn().get_operation(operations_pb2.GetOperationRequest(name=operation_name))
+ return await self.get_conn().get_operation(
+ operations_pb2.GetOperationRequest(name=operation_name), timeout=120
Review Comment:
@Lee-W Thank you for the comment, I don't think it makes sense to make it configurable. By default the operation is never expected to timeout. The timeout I added was just to make sure that the client has enough time to fetch the operation status from the CloudRun service/database in case of (let's say) load.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org