You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "tnk-ysk (via GitHub)" <gi...@apache.org> on 2023/03/06 13:29:41 UTC
[GitHub] [airflow] tnk-ysk opened a new pull request, #29937: Fixed location on cloud build operators
tnk-ysk opened a new pull request, #29937:
URL: https://github.com/apache/airflow/pull/29937
Fixed an issue where some cloud build operators did not work for private pools or a non-global default pools.
In this case CloudBuildClient need to set the api_endpoint.
Related https://github.com/apache/airflow/pull/29689
Targets
- CloudBuildCancelBuildOperator
- CloudBuildGetBuildOperator
- CloudBuildListBuildsOperator
- CloudBuildRetryBuildOperator
- (CloudbuildCreateBuildOperator https://github.com/apache/airflow/pull/29689)
## Reference
- https://github.com/googleapis/python-cloudbuild/blob/9cf4f4269b55213c1f84206923d3cdc0308ecf57/samples/snippets/quickstart.py#L35-L43
- https://cloud.google.com/build/docs/private-pools/private-pools-overview
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] boring-cyborg[bot] commented on pull request #29937: Fix location on cloud build operators
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1463715978
Awesome work, congrats on your first merged pull request!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tnk-ysk commented on pull request #29937: Fix location on cloud build operators
Posted by "tnk-ysk (via GitHub)" <gi...@apache.org>.
tnk-ysk commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1459145119
@potiuk
> tests needs fixing though
I fixed it.
https://github.com/apache/airflow/pull/29937/commits/23cd526d1eb6e802ac6baf970ec893e8c8e435f8
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29937: Fix location on cloud build operators
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1463716148
Cool. Thank @VladaZakharova !
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] tnk-ysk commented on pull request #29937: Fix location on cloud build operators
Posted by "tnk-ysk (via GitHub)" <gi...@apache.org>.
tnk-ysk commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1460208035
@VladaZakharova
I checked and fixed some issues.
1. Fixed some bugs.
2. CloudBuildTriggerOperator had a similar issue, fixed.
3. Fixed CloudBuild links.
Test dag
```
from __future__ import print_function
import datetime
import os
from airflow import models
from airflow.providers.google.cloud.operators import cloud_build
from airflow.providers.google.cloud.utils import credentials_provider
PROJECT_ID = "xxxxxxx"
REPO_NAME = "sample"
gcp_conn = credentials_provider.build_gcp_conn(
"/files/key.json",
["https://www.googleapis.com/auth/cloud-platform"],
PROJECT_ID,
)
os.environ["AIRFLOW_CONN_GOOGLE_CLOUD_DEFAULT"] = gcp_conn
default_dag_args = {
'start_date': datetime.datetime(2022, 9, 28),
}
with models.DAG(
'CloudBuildTest',
schedule_interval=None,
default_args=default_dag_args,
catchup=False
) as dag:
build = {
"steps": [{
"name": "alpine",
"script": "sleep 10 && echo hello",
}],
}
trigger = {
"name": "trigger",
"trigger_template": {
"repo_name": REPO_NAME,
"branch_name": "main",
},
"filename": "cloudbuild.yaml",
}
# global
create_build_global = cloud_build.CloudBuildCreateBuildOperator(
task_id="create_build_global",
project_id=PROJECT_ID,
build=build,
)
create_build_global_deferrable = cloud_build.CloudBuildCreateBuildOperator(
task_id="create_build_global_deferrable",
project_id=PROJECT_ID,
build=build,
deferrable=True,
)
list_build_global = cloud_build.CloudBuildListBuildsOperator(
task_id="list_build_global",
project_id=PROJECT_ID,
page_size=1,
)
retry_build_global = cloud_build.CloudBuildRetryBuildOperator(
task_id="retry_build_global",
project_id=PROJECT_ID,
id_='{{ ti.xcom_pull(task_ids="create_build_global", key="id") }}',
)
get_build_global = cloud_build.CloudBuildGetBuildOperator(
task_id="get_build_global",
project_id=PROJECT_ID,
id_='{{ ti.xcom_pull(task_ids="retry_build_global", key="id") }}',
)
list_build_triggers_global = cloud_build.CloudBuildListBuildTriggersOperator(
task_id="list_build_triggers_global",
project_id=PROJECT_ID,
)
create_build_global_for_cancel = cloud_build.CloudBuildCreateBuildOperator(
task_id="create_build_global_for_cancel",
project_id=PROJECT_ID,
build=build,
wait=False,
)
cancel_build_global = cloud_build.CloudBuildCancelBuildOperator(
task_id="cancel_build_global",
project_id=PROJECT_ID,
id_='{{ ti.xcom_pull(task_ids="create_build_global_for_cancel", key="id") }}',
)
create_build_trigger_global = cloud_build.CloudBuildCreateBuildTriggerOperator(
task_id="create_build_trigger_global",
project_id=PROJECT_ID,
trigger=trigger,
)
get_build_trigger_global = cloud_build.CloudBuildGetBuildTriggerOperator(
task_id="get_build_trigger_global",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
)
update_build_trigger_global = cloud_build.CloudBuildUpdateBuildTriggerOperator(
task_id="update_build_trigger_global",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
trigger=trigger,
)
run_build_trigger_global = cloud_build.CloudBuildRunBuildTriggerOperator(
task_id="run_build_trigger_global",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
source={
"branch_name": "main"
}
)
delete_build_trigger_global = cloud_build.CloudBuildDeleteBuildTriggerOperator(
task_id="delete_build_trigger_global",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_global", key="id") }}',
)
# us-centrarl
create_build_us_central1 = cloud_build.CloudBuildCreateBuildOperator(
task_id="create_build_us_central1",
project_id=PROJECT_ID,
build=build,
location="us-central1",
)
create_build_us_central1_deferrable = cloud_build.CloudBuildCreateBuildOperator(
task_id="create_build_us_central1_deferrable",
project_id=PROJECT_ID,
build=build,
deferrable=True,
location="us-central1",
)
list_build_us_central1 = cloud_build.CloudBuildListBuildsOperator(
task_id="list_build_us_central1",
project_id=PROJECT_ID,
page_size=1,
location="us-central1",
)
retry_build_us_central1 = cloud_build.CloudBuildRetryBuildOperator(
task_id="retry_build_us_central1",
project_id=PROJECT_ID,
id_='{{ ti.xcom_pull(task_ids="create_build_us_central1", key="id") }}',
location="us-central1",
)
get_build_us_central1 = cloud_build.CloudBuildGetBuildOperator(
task_id="get_build_us_central1",
project_id=PROJECT_ID,
id_='{{ ti.xcom_pull(task_ids="retry_build_us_central1", key="id") }}',
location="us-central1",
)
list_build_triggers_us_central1 = cloud_build.CloudBuildListBuildTriggersOperator(
task_id="list_build_triggers_us_central1",
project_id=PROJECT_ID,
location="us-central1",
)
create_build_us_central1_for_cancel = cloud_build.CloudBuildCreateBuildOperator(
task_id="create_build_us_central1_for_cancel",
project_id=PROJECT_ID,
build=build,
wait=False,
location="us-central1",
)
cancel_build_us_central1 = cloud_build.CloudBuildCancelBuildOperator(
task_id="cancel_build_us_central1",
project_id=PROJECT_ID,
id_='{{ ti.xcom_pull(task_ids="create_build_us_central1_for_cancel", key="id") }}',
location="us-central1",
)
create_build_trigger_us_central1 = cloud_build.CloudBuildCreateBuildTriggerOperator(
task_id="create_build_trigger_us_central1",
project_id=PROJECT_ID,
trigger=trigger,
location="us-central1",
)
get_build_trigger_us_central1 = cloud_build.CloudBuildGetBuildTriggerOperator(
task_id="get_build_trigger_us_central1",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
location="us-central1",
)
update_build_trigger_us_central1 = cloud_build.CloudBuildUpdateBuildTriggerOperator(
task_id="update_build_trigger_us_central1",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
trigger=trigger,
location="us-central1",
)
run_build_trigger_us_central1 = cloud_build.CloudBuildRunBuildTriggerOperator(
task_id="run_build_trigger_us_central1",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
source={
"branch_name": "main"
},
location="us-central1",
)
delete_build_trigger_us_central1 = cloud_build.CloudBuildDeleteBuildTriggerOperator(
task_id="delete_build_trigger_us_central1",
project_id=PROJECT_ID,
trigger_id='{{ ti.xcom_pull(task_ids="create_build_trigger_us_central1", key="id") }}',
location="us-central1",
)
[
# global
create_build_global >> retry_build_global >> get_build_global,
create_build_global_deferrable,
list_build_global >> list_build_triggers_global,
create_build_global_for_cancel >> cancel_build_global,
create_build_trigger_global >> get_build_trigger_global >> update_build_trigger_global
>> run_build_trigger_global >> delete_build_trigger_global,
# us-central1
create_build_us_central1 >> retry_build_us_central1 >> get_build_us_central1,
create_build_us_central1_deferrable,
list_build_us_central1 >> list_build_triggers_us_central1,
create_build_us_central1_for_cancel >> cancel_build_us_central1,
create_build_trigger_us_central1 >> get_build_trigger_us_central1 >> update_build_trigger_us_central1
>> run_build_trigger_us_central1 >> delete_build_trigger_us_central1,
]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on pull request #29937: Fix location on cloud build operators
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1459731408
Hi!
Did you check if, for example, BigQuery service is working okay with this changes without providing location to the operator?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29937: Fix location on cloud build operators
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1458941673
tests needs fixing though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29937: Fix location on cloud build operators
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1458903187
LGTM. @VladaZakharova - WDYT?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk commented on pull request #29937: Fix location on cloud build operators
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1463716457
And @tnk-ysk of course :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] potiuk merged pull request #29937: Fix location on cloud build operators
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #29937:
URL: https://github.com/apache/airflow/pull/29937
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [airflow] VladaZakharova commented on pull request #29937: Fix location on cloud build operators
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #29937:
URL: https://github.com/apache/airflow/pull/29937#issuecomment-1460212204
Cool! Thanks you for the fix :)
LGTM
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org