You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "pawelgrochowicz (via GitHub)" <gi...@apache.org> on 2025/09/03 15:44:41 UTC
[PR] feature: Add OpenLineage support for BigQueryToMySqlOperator [airflow]
pawelgrochowicz opened a new pull request, #55219:
URL: https://github.com/apache/airflow/pull/55219
This PR adds OpenLineage support for BigQueryToMySqlOperator.
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [airflow-core/newsfragments](https://github.com/apache/airflow/tree/main/airflow-core/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feature: Add OpenLineage support for BigQueryToMySqlOperator [airflow]
Posted by "pawelgrochowicz (via GitHub)" <gi...@apache.org>.
pawelgrochowicz commented on code in PR #55219:
URL: https://github.com/apache/airflow/pull/55219#discussion_r2323452531
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py:
##########
@@ -76,5 +86,73 @@ def __init__(
)
self.mysql_conn_id = mysql_conn_id
- def get_sql_hook(self) -> MySqlHook:
+ @cached_property
+ def mysql_hook(self) -> MySqlHook:
return MySqlHook(schema=self.database, mysql_conn_id=self.mysql_conn_id)
+
+ def get_sql_hook(self) -> MySqlHook:
+ return self.mysql_hook
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_source_project_dataset_table(self, project_id: str = PROVIDE_PROJECT_ID) -> str:
+ return project_id
+
+ def execute(self, context):
+ # Set source_project_dataset_table here, after hooks are initialized and project_id is available
+ project_id = self.bigquery_hook.project_id
Review Comment:
I'm adding the OpenLinage support to BigQueryToMsSQLOperator, BigQueryToMySQLOperator and next to BigQueryToPostgresOperator. Initially, I'm adding basic support to each of them, as they are not unified in terms of input arguments. Then, I would like to refactor the code and, in the BigQueryToSQLOperator (the base for all three), create a common version of the code. As for now, as @GoogleBaseHook.fallback_to_default_project_id is mostly used in hooks, I just reused what is theren
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py:
##########
@@ -76,5 +86,73 @@ def __init__(
)
self.mysql_conn_id = mysql_conn_id
- def get_sql_hook(self) -> MySqlHook:
+ @cached_property
+ def mysql_hook(self) -> MySqlHook:
return MySqlHook(schema=self.database, mysql_conn_id=self.mysql_conn_id)
+
+ def get_sql_hook(self) -> MySqlHook:
+ return self.mysql_hook
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_source_project_dataset_table(self, project_id: str = PROVIDE_PROJECT_ID) -> str:
+ return project_id
+
+ def execute(self, context):
+ # Set source_project_dataset_table here, after hooks are initialized and project_id is available
+ project_id = self.bigquery_hook.project_id
Review Comment:
I'm adding the OpenLinage support to BigQueryToMsSQLOperator, BigQueryToMySQLOperator and next to BigQueryToPostgresOperator. Initially, I'm adding basic support to each of them, as they are not unified in terms of input arguments. Then, I would like to refactor the code and, in the BigQueryToSQLOperator (the base for all three), create a common version of the code. As for now, as @GoogleBaseHook.fallback_to_default_project_id is mostly used in hooks, I just reused what is there
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feature: Add OpenLineage support for BigQueryToMySqlOperator [airflow]
Posted by "pawelgrochowicz (via GitHub)" <gi...@apache.org>.
pawelgrochowicz commented on PR #55219:
URL: https://github.com/apache/airflow/pull/55219#issuecomment-3252434617
FYI @mobuchowski
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feature: Add OpenLineage support for BigQueryToMySqlOperator [airflow]
Posted by "mobuchowski (via GitHub)" <gi...@apache.org>.
mobuchowski merged PR #55219:
URL: https://github.com/apache/airflow/pull/55219
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feature: Add OpenLineage support for BigQueryToMySqlOperator [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on code in PR #55219:
URL: https://github.com/apache/airflow/pull/55219#discussion_r2321529448
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py:
##########
@@ -76,5 +86,73 @@ def __init__(
)
self.mysql_conn_id = mysql_conn_id
- def get_sql_hook(self) -> MySqlHook:
+ @cached_property
+ def mysql_hook(self) -> MySqlHook:
return MySqlHook(schema=self.database, mysql_conn_id=self.mysql_conn_id)
+
+ def get_sql_hook(self) -> MySqlHook:
+ return self.mysql_hook
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_source_project_dataset_table(self, project_id: str = PROVIDE_PROJECT_ID) -> str:
+ return project_id
+
+ def execute(self, context):
+ # Set source_project_dataset_table here, after hooks are initialized and project_id is available
+ project_id = self.bigquery_hook.project_id
Review Comment:
is there a reason you want to get project_id from hook specifically and not from self of the operator?
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py:
##########
@@ -76,5 +86,73 @@ def __init__(
)
self.mysql_conn_id = mysql_conn_id
- def get_sql_hook(self) -> MySqlHook:
+ @cached_property
+ def mysql_hook(self) -> MySqlHook:
return MySqlHook(schema=self.database, mysql_conn_id=self.mysql_conn_id)
+
+ def get_sql_hook(self) -> MySqlHook:
+ return self.mysql_hook
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_source_project_dataset_table(self, project_id: str = PROVIDE_PROJECT_ID) -> str:
+ return project_id
Review Comment:
I don't quite get this function's purpose :)
Does it not just return project_id? in general you can use this decorator @GoogleBaseHook.fallback_to_default_project_id to get project_id
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] feature: Add OpenLineage support for BigQueryToMySqlOperator [airflow]
Posted by "pawelgrochowicz (via GitHub)" <gi...@apache.org>.
pawelgrochowicz commented on code in PR #55219:
URL: https://github.com/apache/airflow/pull/55219#discussion_r2323417324
##########
providers/google/src/airflow/providers/google/cloud/transfers/bigquery_to_mysql.py:
##########
@@ -76,5 +86,73 @@ def __init__(
)
self.mysql_conn_id = mysql_conn_id
- def get_sql_hook(self) -> MySqlHook:
+ @cached_property
+ def mysql_hook(self) -> MySqlHook:
return MySqlHook(schema=self.database, mysql_conn_id=self.mysql_conn_id)
+
+ def get_sql_hook(self) -> MySqlHook:
+ return self.mysql_hook
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def get_source_project_dataset_table(self, project_id: str = PROVIDE_PROJECT_ID) -> str:
+ return project_id
Review Comment:
I removed it. It was not used. Originally, I wanted to use the function, but ended up getting project_id from BigQueryHook
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org