You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by GitBox <gi...@apache.org> on 2022/06/13 15:30:29 UTC
[GitHub] [airflow] lwyszomi opened a new pull request, #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
lwyszomi opened a new pull request, #24416:
URL: https://github.com/apache/airflow/pull/24416
Refactor of the BigQueryToGCPOperator, now we using `insert_job` method not `run_extract`. Also adjustment for the links in the BigQueryInsertJobOperator
FYI @potiuk
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code change, Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvements+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragement file, named `{pr_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a diff in pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
turbaszek commented on code in PR #24416:
URL: https://github.com/apache/airflow/pull/24416#discussion_r896036850
##########
airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -2167,16 +2156,34 @@ def execute(self, context: Any):
f"Or, if you want to reattach in this scenario add {job.state} to `reattach_states`"
)
- if "query" in job.to_api_repr()["configuration"]:
- if "destinationTable" in job.to_api_repr()["configuration"]["query"]:
- table = job.to_api_repr()["configuration"]["query"]["destinationTable"]
- BigQueryTableLink.persist(
- context=context,
- task_instance=self,
- dataset_id=table["datasetId"],
- project_id=table["projectId"],
- table_id=table["tableId"],
- )
+ job_types = {
+ LoadJob._JOB_TYPE: ["sourceTable", "destinationTable"],
+ CopyJob._JOB_TYPE: ["sourceTable", "destinationTable"],
+ ExtractJob._JOB_TYPE: ["sourceTable"],
+ QueryJob._JOB_TYPE: ["destinationTable"],
+ }
+
+ for job_type, tables_prop in job_types.items():
+ if job_type in job.to_api_repr()["configuration"]:
+ for table_prop in tables_prop:
+ if table_prop in job.to_api_repr()["configuration"][job_type]:
+ table = job.to_api_repr()["configuration"][job_type][table_prop]
+ if self.project_id:
+ if isinstance(table, str):
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ project_id=self.project_id,
+ table_id=table,
+ )
+ else:
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table["datasetId"],
+ project_id=self.project_id,
+ table_id=table["tableId"],
+ )
Review Comment:
Consider using opossite assertion and `continue` to keep the code clean. We don't have pylint but too much indentation is not good 😉
Also `if not self.project_id:` can be moved outside loop as it's crucial, so if missing then no need to do any work, right?
Apart from that we can introduce `job_configuration`
```suggestion
job_configuration = job.to_api_repr()["configuration"]
if not job_type in job_configuration:
continue
for table_prop in tables_prop:
if not table_prop in job_configuration[job_type]:
continue
table = job_configuration[job_type][table_prop]
if isinstance(table, str):
BigQueryTableLink.persist(
context=context,
task_instance=self,
project_id=self.project_id,
table_id=table,
)
else:
BigQueryTableLink.persist(
context=context,
task_instance=self,
dataset_id=table["datasetId"],
project_id=self.project_id,
table_id=table["tableId"],
)
```
Additionally I would do:
```
persist_kwargs = {"context": context, task_instance: self, project_id: self.project_id, table_id: table}
if not isinstance(table, str):
persist_kwargs["table_id"] = table["tableId"]
persist_kwargs["dataset_id"] = table["datasetId"]
BigQueryTableLink.persist(**kwargs)
```
instead of calling `BigQueryTableLink.persist` twice.
WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24416:
URL: https://github.com/apache/airflow/pull/24416#issuecomment-1155721133
Ah yeah. there are also few comments from @josh-fell - so @lwyszomi -> if you can address them tomorrow morning, I am happy to wait.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24416:
URL: https://github.com/apache/airflow/pull/24416#issuecomment-1155719670
Yeah. I think that last comment from @turbaszek and we can merge (and release 8.0.1)
BTW. @turbaszek - we want to relase 8.0.1 as this one fixes a serious problem in Google Provider 8.0.0 - so if you are not too strong with your comment, I'd merge and release it even now.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lwyszomi commented on pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
lwyszomi commented on PR #24416:
URL: https://github.com/apache/airflow/pull/24416#issuecomment-1156028613
@potiuk I pushed fixes for @josh-fell and @turbaszek comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] turbaszek commented on a diff in pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
turbaszek commented on code in PR #24416:
URL: https://github.com/apache/airflow/pull/24416#discussion_r897322128
##########
airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -2167,16 +2156,30 @@ def execute(self, context: Any):
f"Or, if you want to reattach in this scenario add {job.state} to `reattach_states`"
)
- if "query" in job.to_api_repr()["configuration"]:
- if "destinationTable" in job.to_api_repr()["configuration"]["query"]:
- table = job.to_api_repr()["configuration"]["query"]["destinationTable"]
- BigQueryTableLink.persist(
- context=context,
- task_instance=self,
- dataset_id=table["datasetId"],
- project_id=table["projectId"],
- table_id=table["tableId"],
- )
+ job_types = {
+ LoadJob._JOB_TYPE: ["sourceTable", "destinationTable"],
+ CopyJob._JOB_TYPE: ["sourceTable", "destinationTable"],
+ ExtractJob._JOB_TYPE: ["sourceTable"],
+ QueryJob._JOB_TYPE: ["destinationTable"],
+ }
+
+ if self.project_id:
+ for job_type, tables_prop in job_types.items():
+ if job_type in job.to_api_repr()["configuration"]:
+ for table_prop in tables_prop:
+ if table_prop in job.to_api_repr()["configuration"][job_type]:
+ table = job.to_api_repr()["configuration"][job_type][table_prop]
Review Comment:
Do we need to write `job.to_api_repr()["configuration"]` 3 times? Could you please introduce a variable for this value?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] josh-fell commented on a diff in pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
josh-fell commented on code in PR #24416:
URL: https://github.com/apache/airflow/pull/24416#discussion_r896939077
##########
airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:
##########
@@ -115,17 +178,49 @@ def execute(self, context: 'Context'):
location=self.location,
impersonation_chain=self.impersonation_chain,
)
- job: BigQueryJob = hook.run_extract(
- source_project_dataset_table=self.source_project_dataset_table,
- destination_cloud_storage_uris=self.destination_cloud_storage_uris,
- compression=self.compression,
- export_format=self.export_format,
- field_delimiter=self.field_delimiter,
- print_header=self.print_header,
- labels=self.labels,
- return_full_job=True,
+ self.hook = hook
+
+ configuration = self._prepare_configuration()
+ job_id = hook.generate_job_id(
+ job_id=self.job_id,
+ dag_id=self.dag_id,
+ task_id=self.task_id,
+ logical_date=context["logical_date"],
+ configuration=configuration,
+ force_rerun=self.force_rerun,
)
- conf = job["configuration"]["extract"]["sourceTable"]
+
+ try:
+ self.log.info(f"Executing: {configuration}")
Review Comment:
We should use %-formatting in logging so the formatting is deferred until the latest possible moment.
##########
airflow/providers/google/cloud/transfers/bigquery_to_gcs.py:
##########
@@ -78,6 +93,7 @@ def __init__(
*,
source_project_dataset_table: str,
destination_cloud_storage_uris: List[str],
+ project_id: Optional[str] = None,
Review Comment:
Would you mind adding this to the operator's docstring too?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
potiuk merged PR #24416:
URL: https://github.com/apache/airflow/pull/24416
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24416:
URL: https://github.com/apache/airflow/pull/24416#issuecomment-1155720526
Soory @lwyszomi - it TOTALLY got under the radar - not sure how, I was waiting for that PR!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
potiuk commented on PR #24416:
URL: https://github.com/apache/airflow/pull/24416#issuecomment-1155713561
Ah. How could I missed it !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] lwyszomi commented on a diff in pull request #24416: Use insert_job in the BigQueryToGCPOpertor and adjust links
Posted by GitBox <gi...@apache.org>.
lwyszomi commented on code in PR #24416:
URL: https://github.com/apache/airflow/pull/24416#discussion_r896432278
##########
airflow/providers/google/cloud/operators/bigquery.py:
##########
@@ -2167,16 +2156,34 @@ def execute(self, context: Any):
f"Or, if you want to reattach in this scenario add {job.state} to `reattach_states`"
)
- if "query" in job.to_api_repr()["configuration"]:
- if "destinationTable" in job.to_api_repr()["configuration"]["query"]:
- table = job.to_api_repr()["configuration"]["query"]["destinationTable"]
- BigQueryTableLink.persist(
- context=context,
- task_instance=self,
- dataset_id=table["datasetId"],
- project_id=table["projectId"],
- table_id=table["tableId"],
- )
+ job_types = {
+ LoadJob._JOB_TYPE: ["sourceTable", "destinationTable"],
+ CopyJob._JOB_TYPE: ["sourceTable", "destinationTable"],
+ ExtractJob._JOB_TYPE: ["sourceTable"],
+ QueryJob._JOB_TYPE: ["destinationTable"],
+ }
+
+ for job_type, tables_prop in job_types.items():
+ if job_type in job.to_api_repr()["configuration"]:
+ for table_prop in tables_prop:
+ if table_prop in job.to_api_repr()["configuration"][job_type]:
+ table = job.to_api_repr()["configuration"][job_type][table_prop]
+ if self.project_id:
+ if isinstance(table, str):
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ project_id=self.project_id,
+ table_id=table,
+ )
+ else:
+ BigQueryTableLink.persist(
+ context=context,
+ task_instance=self,
+ dataset_id=table["datasetId"],
+ project_id=self.project_id,
+ table_id=table["tableId"],
+ )
Review Comment:
Yeap, makes sense. thanks. I already push the update
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org