You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "henry3260 (via GitHub)" <gi...@apache.org> on 2026/07/08 13:10:41 UTC

[PR] Fix Hive metastore partition sensor initialization [airflow]

henry3260 opened a new pull request, #69612:
URL: https://github.com/apache/airflow/pull/69612

   ## Why
   
   `MetastorePartitionSensor` documents `mysql_conn_id` as the connection argument and is intended to generate the Hive metastore partition query internally. After inheriting from `SqlSensor`, the documented constructor path still required callers to pass the parent `conn_id` and `sql` arguments manually, so normal usage did not match the public API.
   
   The sensor also interpolated templated table, schema, and partition values directly into the SQL string. Passing those values through SQL parameters keeps the query construction aligned with `SqlSensor` behavior.
   
   ## What
   
   - Updated `providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py` so `MetastorePartitionSensor` passes `mysql_conn_id` to `SqlSensor` as `conn_id` and owns the metastore partition SQL.
   - Changed `poke()` to split `schema.table` input at runtime and pass `table`, `schema`, and `partition_name` through `SqlSensor.parameters`.
   - Expanded `SqlSensor` template fields while preserving `partition_name`, `table`, and `schema` templating.
   - Updated `providers/apache/hive/tests/unit/apache/hive/sensors/test_metastore_partition.py` to cover the documented constructor, schema/table splitting, and parameterized SQL call.
   - Updated the existing Hive metastore sensor test setup to use `mysql_conn_id`.
   
   ---
   
   ##### Was generative AI tooling used to co-author this PR?
   
   <!--
   If generative AI tooling has been used in the process of authoring this PR, please
   change below checkbox to `[X]` followed by the name of the tool, uncomment the "Generated-by".
   -->
   
   - [ ] Yes (please specify the tool below)
   
   <!--
   Generated-by: [Tool Name] following [the guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#gen-ai-assisted-contributions)
   -->
   
   ---
   
   * Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information. Note: commit author/co-author name and email in commits become permanently public when merged.
   * For fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
   * When adding dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
   * For significant user-facing changes create newsfragment: `{pr_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments). You can add this file in a follow-up commit after the PR is created so you know the PR number.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Hive metastore partition sensor initialization [airflow]

Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 commented on code in PR #69612:
URL: https://github.com/apache/airflow/pull/69612#discussion_r3579134098


##########
providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py:
##########
@@ -58,28 +70,16 @@ def __init__(
         self.partition_name = partition_name
         self.table = table
         self.schema = schema
-        self.first_poke = True
-        self.conn_id = mysql_conn_id
-        # TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor.
-        # The problem is the way apply_defaults works isn't compatible with inheritance.
-        # The inheritance model needs to be reworked in order to support overriding args/
-        # kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the
-        # constructor below and apply_defaults will no longer throw an exception.
-        super().__init__(**kwargs)
+        super().__init__(conn_id=mysql_conn_id, sql=_METASTORE_PARTITION_SQL, **kwargs)

Review Comment:
   > If someone already use `MetastorePartitionSensor(con_id=...)`, it would cause an issue.
   > 
   > something like
   > 
   > works better
   
   Thanks for catching this!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Hive metastore partition sensor initialization [airflow]

Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 merged PR #69612:
URL: https://github.com/apache/airflow/pull/69612


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Hive metastore partition sensor initialization [airflow]

Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #69612:
URL: https://github.com/apache/airflow/pull/69612#discussion_r3578009445


##########
providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py:
##########
@@ -58,28 +70,16 @@ def __init__(
         self.partition_name = partition_name
         self.table = table
         self.schema = schema
-        self.first_poke = True
-        self.conn_id = mysql_conn_id
-        # TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor.
-        # The problem is the way apply_defaults works isn't compatible with inheritance.
-        # The inheritance model needs to be reworked in order to support overriding args/
-        # kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the
-        # constructor below and apply_defaults will no longer throw an exception.
-        super().__init__(**kwargs)
+        super().__init__(conn_id=mysql_conn_id, sql=_METASTORE_PARTITION_SQL, **kwargs)
 
     def poke(self, context: Context) -> Any:
-        if self.first_poke:
-            self.first_poke = False
-            if "." in self.table:
-                self.schema, self.table = self.table.split(".")
-            self.sql = f"""
-            SELECT 'X'
-            FROM PARTITIONS A0
-            LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID
-            LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID
-            WHERE
-                B0.TBL_NAME = '{self.table}' AND
-                C0.NAME = '{self.schema}' AND
-                A0.PART_NAME = '{self.partition_name}';
-            """
+        schema = self.schema
+        table = self.table
+        if "." in table:
+            schema, table = table.split(".", maxsplit=1)

Review Comment:
   ```suggestion
           if "." in self.table:
               schema, table = self.table.split(".", maxsplit=1)
           else:
           	schema, table = self.schema, self.table
   ```



##########
providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py:
##########
@@ -58,28 +70,16 @@ def __init__(
         self.partition_name = partition_name
         self.table = table
         self.schema = schema
-        self.first_poke = True
-        self.conn_id = mysql_conn_id
-        # TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor.
-        # The problem is the way apply_defaults works isn't compatible with inheritance.
-        # The inheritance model needs to be reworked in order to support overriding args/
-        # kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the
-        # constructor below and apply_defaults will no longer throw an exception.
-        super().__init__(**kwargs)
+        super().__init__(conn_id=mysql_conn_id, sql=_METASTORE_PARTITION_SQL, **kwargs)

Review Comment:
   If someone already use `MetastorePartitionSensor(con_id=...)`, it would cause an issue.
   
   something like
   ```suggestion
           _kwargs = {"conn_id": mysql_conn_id, sql: _METASTORE_PARTITION_SQL}
           if kwargs:
               _kwargs |= kwargs
           super().__init__(**_kwargs)
   ```
   works better



##########
providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py:
##########
@@ -58,28 +70,16 @@ def __init__(
         self.partition_name = partition_name
         self.table = table
         self.schema = schema
-        self.first_poke = True
-        self.conn_id = mysql_conn_id
-        # TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor.
-        # The problem is the way apply_defaults works isn't compatible with inheritance.
-        # The inheritance model needs to be reworked in order to support overriding args/
-        # kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the
-        # constructor below and apply_defaults will no longer throw an exception.
-        super().__init__(**kwargs)
+        super().__init__(conn_id=mysql_conn_id, sql=_METASTORE_PARTITION_SQL, **kwargs)
 
     def poke(self, context: Context) -> Any:
-        if self.first_poke:
-            self.first_poke = False
-            if "." in self.table:
-                self.schema, self.table = self.table.split(".")
-            self.sql = f"""
-            SELECT 'X'
-            FROM PARTITIONS A0
-            LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID
-            LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID
-            WHERE
-                B0.TBL_NAME = '{self.table}' AND
-                C0.NAME = '{self.schema}' AND
-                A0.PART_NAME = '{self.partition_name}';
-            """
+        schema = self.schema
+        table = self.table
+        if "." in table:
+            schema, table = table.split(".", maxsplit=1)

Review Comment:
   also, do we really need `maxsplit=1` here? wouldn't it cause more trouble if someone passes `a.b.c`



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] Fix Hive metastore partition sensor initialization [airflow]

Posted by "henry3260 (via GitHub)" <gi...@apache.org>.
henry3260 commented on code in PR #69612:
URL: https://github.com/apache/airflow/pull/69612#discussion_r3579587346


##########
providers/apache/hive/src/airflow/providers/apache/hive/sensors/metastore_partition.py:
##########
@@ -58,28 +70,16 @@ def __init__(
         self.partition_name = partition_name
         self.table = table
         self.schema = schema
-        self.first_poke = True
-        self.conn_id = mysql_conn_id
-        # TODO(aoen): We shouldn't be using SqlSensor here but MetastorePartitionSensor.
-        # The problem is the way apply_defaults works isn't compatible with inheritance.
-        # The inheritance model needs to be reworked in order to support overriding args/
-        # kwargs with arguments here, then 'conn_id' and 'sql' can be passed into the
-        # constructor below and apply_defaults will no longer throw an exception.
-        super().__init__(**kwargs)
+        super().__init__(conn_id=mysql_conn_id, sql=_METASTORE_PARTITION_SQL, **kwargs)
 
     def poke(self, context: Context) -> Any:
-        if self.first_poke:
-            self.first_poke = False
-            if "." in self.table:
-                self.schema, self.table = self.table.split(".")
-            self.sql = f"""
-            SELECT 'X'
-            FROM PARTITIONS A0
-            LEFT OUTER JOIN TBLS B0 ON A0.TBL_ID = B0.TBL_ID
-            LEFT OUTER JOIN DBS C0 ON B0.DB_ID = C0.DB_ID
-            WHERE
-                B0.TBL_NAME = '{self.table}' AND
-                C0.NAME = '{self.schema}' AND
-                A0.PART_NAME = '{self.partition_name}';
-            """
+        schema = self.schema
+        table = self.table
+        if "." in table:
+            schema, table = table.split(".", maxsplit=1)

Review Comment:
   > also, do we really need `maxsplit=1` here? wouldn't it cause more trouble if someone passes `a.b.c`
   
   Applied! using another way to avoid that



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org