[GitHub] [airflow] phanikumv commented on a diff in pull request #28874: Add `HivePartitionAsyncSensor`

[GitBox <gi...@apache.org>]

phanikumv commented on code in PR #28874:
URL: https://github.com/apache/airflow/pull/28874#discussion_r1067780016


##########
airflow/providers/apache/hive/provider.yaml:
##########
@@ -56,6 +56,7 @@ dependencies:
   # the sasl library anyway (and there sasl library version is not relevant)
   - sasl>=0.3.1; python_version>="3.9"
   - thrift>=0.9.2
+  - impyla

Review Comment:
   We have written our own asyncio method as shown below, because  impyla give us a handle immediately after submitting the query
    
   ```
   async def partition_exists(self, table: str, schema: str, partition: str, polling_interval: float) -> str:
           """
           Checks for the existence of a partition in the given hive table.
   
           :param table: table in hive where the partition exists.
           :param schema: database where the hive table exists
           :param partition: partition to check for in given hive database and hive table.
           :param polling_interval: polling interval in seconds to sleep between checks
           """
           client = self.get_hive_client()
           cursor = client.cursor()
           query = f"show partitions {schema}.{table} partition({partition})"
           cursor.execute_async(query)
           while cursor.is_executing():
               await asyncio.sleep(polling_interval)
           results = cursor.fetchall()
           if len(results) == 0:
               return "failure"
           return "success"
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org