You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "vandonr-amz (via GitHub)" <gi...@apache.org> on 2023/06/15 18:47:37 UTC
[GitHub] [airflow] vandonr-amz opened a new pull request, #31942: Use a waiter in athena hook
vandonr-amz opened a new pull request, #31942:
URL: https://github.com/apache/airflow/pull/31942
also took the opportunity to regroup sleep time and max attempts in the same place, to get a more consistent interface.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas merged pull request #31942: Use a waiter in `AthenaHook`
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas merged PR #31942:
URL: https://github.com/apache/airflow/pull/31942
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] o-nikolas commented on a diff in pull request #31942: Use a waiter in athena hook
Posted by "o-nikolas (via GitHub)" <gi...@apache.org>.
o-nikolas commented on code in PR #31942:
URL: https://github.com/apache/airflow/pull/31942#discussion_r1232600792
##########
airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -229,51 +242,33 @@ def get_query_results_paginator(
return paginator.paginate(**result_params)
def poll_query_status(
- self,
- query_execution_id: str,
- max_polling_attempts: int | None = None,
+ self, query_execution_id: str, max_polling_attempts: int | None = None, sleep_time: int | None = None
) -> str | None:
"""Poll the state of a submitted query until it reaches final state.
:param query_execution_id: ID of submitted athena query
- :param max_polling_attempts: Number of times to poll for query state
- before function exits
+ :param max_polling_attempts: Number of times to poll for query state before function exits
+ :param sleep_time: Time (in seconds) to wait between two consecutive query status checks.
:return: One of the final states
"""
- try_number = 1
- final_query_state = None # Query state when query reaches final state or max_polling_attempts reached
- while True:
- query_state = self.check_query_status(query_execution_id)
- if query_state is None:
- self.log.info(
- "Query execution id: %s, trial %s: Invalid query state. Retrying again",
- query_execution_id,
- try_number,
- )
- elif query_state in self.TERMINAL_STATES:
- self.log.info(
- "Query execution id: %s, trial %s: Query execution completed. Final state is %s",
- query_execution_id,
- try_number,
- query_state,
- )
- final_query_state = query_state
- break
- else:
- self.log.info(
- "Query execution id: %s, trial %s: Query is still in non-terminal state - %s",
Review Comment:
We lose this progress tracking logging, which was a huge deal to people for the sensors we were working to deferrablize, perhaps use the new waiter helper @syedahsn wrote to get logging while waiting? It's still in PR [here](https://github.com/apache/airflow/pull/30463/files#diff-7da50792b111859f0571f56ec8d7b43b3dd44d0ac78626444b446d61de6a2720)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vincbeck commented on a diff in pull request #31942: Use a waiter in athena hook
Posted by "vincbeck (via GitHub)" <gi...@apache.org>.
vincbeck commented on code in PR #31942:
URL: https://github.com/apache/airflow/pull/31942#discussion_r1232381091
##########
airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -38,8 +41,7 @@ class AthenaHook(AwsBaseHook):
Provide thick wrapper around
:external+boto3:py:class:`boto3.client("athena") <Athena.Client>`.
- :param sleep_time: Time (in seconds) to wait between two consecutive calls
- to check query status on Athena.
+ :param sleep_time: obsolete, please use the parameter of poll_query_status instead
Review Comment:
```suggestion
:param sleep_time: obsolete, please use the parameter of `poll_query_status` method instead
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] vandonr-amz commented on a diff in pull request #31942: Use a waiter in `AthenaHook`
Posted by "vandonr-amz (via GitHub)" <gi...@apache.org>.
vandonr-amz commented on code in PR #31942:
URL: https://github.com/apache/airflow/pull/31942#discussion_r1234494084
##########
airflow/providers/amazon/aws/hooks/athena.py:
##########
@@ -229,51 +242,33 @@ def get_query_results_paginator(
return paginator.paginate(**result_params)
def poll_query_status(
- self,
- query_execution_id: str,
- max_polling_attempts: int | None = None,
+ self, query_execution_id: str, max_polling_attempts: int | None = None, sleep_time: int | None = None
) -> str | None:
"""Poll the state of a submitted query until it reaches final state.
:param query_execution_id: ID of submitted athena query
- :param max_polling_attempts: Number of times to poll for query state
- before function exits
+ :param max_polling_attempts: Number of times to poll for query state before function exits
+ :param sleep_time: Time (in seconds) to wait between two consecutive query status checks.
:return: One of the final states
"""
- try_number = 1
- final_query_state = None # Query state when query reaches final state or max_polling_attempts reached
- while True:
- query_state = self.check_query_status(query_execution_id)
- if query_state is None:
- self.log.info(
- "Query execution id: %s, trial %s: Invalid query state. Retrying again",
- query_execution_id,
- try_number,
- )
- elif query_state in self.TERMINAL_STATES:
- self.log.info(
- "Query execution id: %s, trial %s: Query execution completed. Final state is %s",
- query_execution_id,
- try_number,
- query_state,
- )
- final_query_state = query_state
- break
- else:
- self.log.info(
- "Query execution id: %s, trial %s: Query is still in non-terminal state - %s",
Review Comment:
ok, I used @syedahsn 's code, but as I merged his branch into mine, it's going to show up in the diff until his PR gets merged
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org