You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "e-galan (via GitHub)" <gi...@apache.org> on 2024/03/07 09:11:32 UTC
[PR] Deferrable mode for RunPipelineJobOperator [airflow]
e-galan opened a new pull request, #37969:
URL: https://github.com/apache/airflow/pull/37969
<!--
Licensed to the Apache Software Foundation (ASF) under one
or more contributor license agreements. See the NOTICE file
distributed with this work for additional information
regarding copyright ownership. The ASF licenses this file
to you under the Apache License, Version 2.0 (the
"License"); you may not use this file except in compliance
with the License. You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing,
software distributed under the License is distributed on an
"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
KIND, either express or implied. See the License for the
specific language governing permissions and limitations
under the License.
-->
<!--
Thank you for contributing! Please make sure that your code changes
are covered with tests. And in case of new features or big changes
remember to adjust the documentation.
Feel free to ping committers for the review!
In case of an existing issue, reference it using one of the following:
closes: #ISSUE
related: #ISSUE
How to write a good git commit message:
http://chris.beams.io/posts/git-commit/
-->
### List of changes:
- Add the deferrable mode to `RunPipelineJobOperator`;
- Move persisting the pipeline job link to the beginning of the operator's execution;
- Add new unit tests, refactor some of the existing unit tests.
<!-- Please keep an empty line above the dashes. -->
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/contributing-docs/05_pull_requests.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "e-galan (via GitHub)" <gi...@apache.org>.
e-galan commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1528297369
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
Review Comment:
@Lee-W Could you please elaborate? I am not sure what you meant here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "e-galan (via GitHub)" <gi...@apache.org>.
e-galan commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1528297369
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
Review Comment:
@Lee-W Could you please elaborate? I am not sure about what you meant here.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1526769734
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
+ service_account: str | None = None,
+ network: str | None = None,
+ create_request_timeout: float | None = None,
+ experiment: str | experiment_resources.Experiment | None = None,
+ # END: run param
+ ) -> PipelineJob:
+ """
+ Create and start a PipelineJob run.
+ For more info about the client method please see:
+ https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.PipelineJob#google_cloud_aiplatform_PipelineJob_submit
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
Review Comment:
Not sure whether we should keep `Required`, `Optional` here as we've already specify them in type annotation
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -407,3 +504,123 @@ def delete_pipeline_job(
metadata=metadata,
)
return result
+
+ @staticmethod
+ def extract_pipeline_job_id(obj: dict) -> str:
+ """Return unique id of a pipeline job from its name."""
+ return obj["name"].rpartition("/")[-1]
+
+
+class PipelineJobAsyncHook(GoogleBaseAsyncHook):
+ """Asynchronous hook for Google Cloud Vertex AI Pipeline Job APIs."""
+
+ sync_hook_class = PipelineJobHook
+ PIPELINE_COMPLETE_STATES = (
+ PipelineState.PIPELINE_STATE_CANCELLED,
+ PipelineState.PIPELINE_STATE_FAILED,
+ PipelineState.PIPELINE_STATE_PAUSED,
+ PipelineState.PIPELINE_STATE_SUCCEEDED,
+ )
+
+ def __init__(
+ self,
+ gcp_conn_id: str = "google_cloud_default",
+ impersonation_chain: str | Sequence[str] | None = None,
+ **kwargs,
+ ) -> None:
+ super().__init__(
+ gcp_conn_id=gcp_conn_id,
+ impersonation_chain=impersonation_chain,
+ **kwargs,
+ )
+
+ async def get_credentials(self) -> Credentials:
+ return (await self.get_sync_hook()).get_credentials()
+
+ async def get_project_id(self) -> str:
+ sync_hook = await self.get_sync_hook()
+ return sync_hook.project_id
+
+ async def get_location(self) -> str:
+ sync_hook = await self.get_sync_hook()
+ return sync_hook.location
+
+ async def get_pipeline_service_client(
+ self,
+ region: str | None = None,
+ ) -> PipelineServiceAsyncClient:
+ if region and region != "global":
+ client_options = ClientOptions(api_endpoint=f"{region}-aiplatform.googleapis.com:443")
+ else:
+ client_options = ClientOptions()
+ return PipelineServiceAsyncClient(
+ credentials=await self.get_credentials(),
+ client_info=CLIENT_INFO,
+ client_options=client_options,
+ )
+
+ async def get_pipeline_job(
+ self,
+ project_id: str,
+ location: str,
+ job_id: str,
+ retry: AsyncRetry | _MethodDefault = DEFAULT,
+ timeout: float | _MethodDefault | None = DEFAULT,
+ metadata: Sequence[tuple[str, str]] = (),
+ ) -> types.PipelineJob:
+ """
+ Get a PipelineJob proto message from PipelineServiceAsyncClient.
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
+ :param location: Required. The ID of the Google Cloud region that the service belongs to.
+ :param job_id: Required. The ID of the PipelineJob resource.
+ :param retry: Designation of what errors, if any, should be retried.
+ :param timeout: The timeout for this request.
+ :param metadata: Strings which should be sent along with the request as metadata.
+ """
+ client = await self.get_pipeline_service_client(region=location)
+ pipeline_job_name = client.pipeline_job_path(
+ project=project_id,
+ location=location,
+ pipeline_job=job_id,
+ )
+ response: types.PipelineJob = await client.get_pipeline_job(
+ request={"name": pipeline_job_name},
+ retry=retry,
+ timeout=timeout,
+ metadata=metadata,
+ )
+ return response
Review Comment:
```suggestion
pipeline_job: types.PipelineJob = await client.get_pipeline_job(
request={"name": pipeline_job_name},
retry=retry,
timeout=timeout,
metadata=metadata,
)
return pipeline_job
```
##########
airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py:
##########
@@ -173,20 +181,46 @@ def execute(self, context: Context):
create_request_timeout=self.create_request_timeout,
experiment=self.experiment,
)
-
- pipeline_job = result.to_dict()
- pipeline_job_id = self.hook.extract_pipeline_job_id(pipeline_job)
+ pipeline_job_id = pipeline_job_obj.job_id
self.log.info("Pipeline job was created. Job id: %s", pipeline_job_id)
-
self.xcom_push(context, key="pipeline_job_id", value=pipeline_job_id)
VertexAIPipelineJobLink.persist(context=context, task_instance=self, pipeline_id=pipeline_job_id)
+
+ if self.deferrable:
+ pipeline_job_obj.wait_for_resource_creation()
+ self.defer(
+ trigger=RunPipelineJobTrigger(
+ conn_id=self.gcp_conn_id,
+ project_id=self.project_id,
+ location=pipeline_job_obj.location,
+ job_id=pipeline_job_id,
+ poll_interval=self.poll_interval,
+ impersonation_chain=self.impersonation_chain,
+ ),
+ method_name="execute_complete",
+ )
+
+ pipeline_job_obj.wait()
+ pipeline_job = pipeline_job_obj.to_dict()
return pipeline_job
+ def execute_complete(self, context: Context, event: dict[str, Any]) -> None:
Review Comment:
It seems the type is wrongly annotated
##########
tests/providers/google/cloud/triggers/test_vertex_ai.py:
##########
@@ -46,6 +55,44 @@
BASE_STRING = "airflow.providers.google.common.hooks.base_google.{}"
+# PYTEST FIXTURES
Review Comment:
```suggestion
```
Not sure whether we really need this comment as they're obviously pytest fixture an no other module seem to follow this convention
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
Review Comment:
not sure whether it makes more sense to make them keword only args
```suggestion
*,
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #37969:
URL: https://github.com/apache/airflow/pull/37969#issuecomment-1991313576
Hi @potiuk @eladkal @Lee-W !
Can you please check the changes in this PR? Thank you so much!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "Lee-W (via GitHub)" <gi...@apache.org>.
Lee-W commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1535091325
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -408,3 +505,123 @@ def delete_pipeline_job(
metadata=metadata,
)
return result
+
+ @staticmethod
+ def extract_pipeline_job_id(obj: dict) -> str:
+ """Return unique id of a pipeline job from its name."""
+ return obj["name"].rpartition("/")[-1]
+
+
+class PipelineJobAsyncHook(GoogleBaseAsyncHook):
+ """Asynchronous hook for Google Cloud Vertex AI Pipeline Job APIs."""
+
+ sync_hook_class = PipelineJobHook
+ PIPELINE_COMPLETE_STATES = (
+ PipelineState.PIPELINE_STATE_CANCELLED,
+ PipelineState.PIPELINE_STATE_FAILED,
+ PipelineState.PIPELINE_STATE_PAUSED,
+ PipelineState.PIPELINE_STATE_SUCCEEDED,
+ )
+
+ def __init__(
+ self,
+ gcp_conn_id: str = "google_cloud_default",
+ impersonation_chain: str | Sequence[str] | None = None,
+ **kwargs,
+ ) -> None:
+ super().__init__(
+ gcp_conn_id=gcp_conn_id,
+ impersonation_chain=impersonation_chain,
+ **kwargs,
+ )
+
+ async def get_credentials(self) -> Credentials:
+ return (await self.get_sync_hook()).get_credentials()
+
Review Comment:
```suggestion
async def get_credentials(self) -> Credentials:
sync_hook = await self.get_sync_hook()
return sync_hook.get_credentials()
```
nitpick: unify how these get` methods are implemented
##########
airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py:
##########
@@ -96,6 +101,10 @@ class RunPipelineJobOperator(GoogleCloudBaseOperator):
If set as a sequence, the identities from the list must grant
Service Account Token Creator IAM role to the directly preceding identity, with first
account from the list granting this role to the originating account (templated).
+ :param deferrable: If True, run the task in the deferrable mode.
Review Comment:
```suggestion
:param deferrable: If True, run the task in the deferrable mode.
```
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
Review Comment:
I'm good with not adding it, either. What I thought was to avoid users from passing wrong arg due to wrong order
##########
tests/providers/google/cloud/operators/test_vertex_ai.py:
##########
@@ -2041,9 +2042,70 @@ def test_execute(self, mock_hook, to_dict_mock):
experiment=None,
)
+ @mock.patch(VERTEX_AI_PATH.format("pipeline_job.PipelineJobHook"))
+ def test_execute_enters_deferred_state(self, mock_hook):
+ task = RunPipelineJobOperator(
+ task_id=TASK_ID,
+ gcp_conn_id=GCP_CONN_ID,
+ impersonation_chain=IMPERSONATION_CHAIN,
+ region=GCP_LOCATION,
+ project_id=GCP_PROJECT,
+ display_name=DISPLAY_NAME,
+ template_path=TEST_TEMPLATE_PATH,
+ job_id=TEST_PIPELINE_JOB_ID,
+ deferrable=True,
+ )
+ mock_hook.return_value.exists.return_value = False
+ with pytest.raises(TaskDeferred) as exc:
+ task.execute(context={"ti": mock.MagicMock()})
+ assert isinstance(exc.value.trigger, RunPipelineJobTrigger), "Trigger is not a RunPipelineJobTrigger"
+
+ @mock.patch(
+ "airflow.providers.google.cloud.operators.vertex_ai.pipeline_job.RunPipelineJobOperator.xcom_push"
+ )
Review Comment:
```suggestion
@mock.patch(VERTEX_AI_PATH.format("pipeline_job.RunPipelineJobOperator.xcom_push"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #37969:
URL: https://github.com/apache/airflow/pull/37969#issuecomment-2020050078
Awesome work, congrats on your first merged pull request! You are invited to check our [Issue Tracker](https://github.com/apache/airflow/issues) for additional contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "e-galan (via GitHub)" <gi...@apache.org>.
e-galan commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1528226597
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
+ service_account: str | None = None,
+ network: str | None = None,
+ create_request_timeout: float | None = None,
+ experiment: str | experiment_resources.Experiment | None = None,
+ # END: run param
+ ) -> PipelineJob:
+ """
+ Create and start a PipelineJob run.
+ For more info about the client method please see:
+ https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.PipelineJob#google_cloud_aiplatform_PipelineJob_submit
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
Review Comment:
I agree, it does not seem necessary, but the same parameters and docstrings are used in the other class methods, so I decided to just stick with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "e-galan (via GitHub)" <gi...@apache.org>.
e-galan commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1528298787
##########
airflow/providers/google/cloud/operators/vertex_ai/pipeline_job.py:
##########
@@ -173,20 +181,46 @@ def execute(self, context: Context):
create_request_timeout=self.create_request_timeout,
experiment=self.experiment,
)
-
- pipeline_job = result.to_dict()
- pipeline_job_id = self.hook.extract_pipeline_job_id(pipeline_job)
+ pipeline_job_id = pipeline_job_obj.job_id
self.log.info("Pipeline job was created. Job id: %s", pipeline_job_id)
-
self.xcom_push(context, key="pipeline_job_id", value=pipeline_job_id)
VertexAIPipelineJobLink.persist(context=context, task_instance=self, pipeline_id=pipeline_job_id)
+
+ if self.deferrable:
+ pipeline_job_obj.wait_for_resource_creation()
+ self.defer(
+ trigger=RunPipelineJobTrigger(
+ conn_id=self.gcp_conn_id,
+ project_id=self.project_id,
+ location=pipeline_job_obj.location,
+ job_id=pipeline_job_id,
+ poll_interval=self.poll_interval,
+ impersonation_chain=self.impersonation_chain,
+ ),
+ method_name="execute_complete",
+ )
+
+ pipeline_job_obj.wait()
+ pipeline_job = pipeline_job_obj.to_dict()
return pipeline_job
+ def execute_complete(self, context: Context, event: dict[str, Any]) -> None:
Review Comment:
@Lee-W Fixed, thanks for pointing that out.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "e-galan (via GitHub)" <gi...@apache.org>.
e-galan commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1528297369
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
Review Comment:
@Lee-W Could you please elaborate? I am not sure about what you meant here.
Update:
I think I got it. I like this Python feature, but it is not really used in the rest of the Vertex AI code. Therefore I doubt that adding it to a single hook method would bring much utility.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "e-galan (via GitHub)" <gi...@apache.org>.
e-galan commented on code in PR #37969:
URL: https://github.com/apache/airflow/pull/37969#discussion_r1528226597
##########
airflow/providers/google/cloud/hooks/vertex_ai/pipeline_job.py:
##########
@@ -243,15 +245,110 @@ def run_pipeline_job(
location=region,
failure_policy=failure_policy,
)
+ self._pipeline_job.submit(
+ service_account=service_account,
+ network=network,
+ create_request_timeout=create_request_timeout,
+ experiment=experiment,
+ )
+ self._pipeline_job.wait()
+
+ return self._pipeline_job
+
+ @GoogleBaseHook.fallback_to_default_project_id
+ def submit_pipeline_job(
+ self,
+ project_id: str,
+ region: str,
+ display_name: str,
+ template_path: str,
+ job_id: str | None = None,
+ pipeline_root: str | None = None,
+ parameter_values: dict[str, Any] | None = None,
+ input_artifacts: dict[str, str] | None = None,
+ enable_caching: bool | None = None,
+ encryption_spec_key_name: str | None = None,
+ labels: dict[str, str] | None = None,
+ failure_policy: str | None = None,
+ # START: run param
+ service_account: str | None = None,
+ network: str | None = None,
+ create_request_timeout: float | None = None,
+ experiment: str | experiment_resources.Experiment | None = None,
+ # END: run param
+ ) -> PipelineJob:
+ """
+ Create and start a PipelineJob run.
+ For more info about the client method please see:
+ https://cloud.google.com/python/docs/reference/aiplatform/latest/google.cloud.aiplatform.PipelineJob#google_cloud_aiplatform_PipelineJob_submit
+
+ :param project_id: Required. The ID of the Google Cloud project that the service belongs to.
Review Comment:
I agree, it does not seem necessary, but the same parameters and docstrings are used in the other class methods, so I just decided to stick with it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #37969:
URL: https://github.com/apache/airflow/pull/37969#issuecomment-2011849286
Hi @potiuk !
Can you please check changes here? :) Thanks you <3
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "VladaZakharova (via GitHub)" <gi...@apache.org>.
VladaZakharova commented on PR #37969:
URL: https://github.com/apache/airflow/pull/37969#issuecomment-2020045226
Hi @potiuk @eladkal!
Can you please merge the changes in this PR? Thank you!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Deferrable mode for RunPipelineJobOperator [airflow]
Posted by "potiuk (via GitHub)" <gi...@apache.org>.
potiuk merged PR #37969:
URL: https://github.com/apache/airflow/pull/37969
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org