You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "KamranImaaz (via GitHub)" <gi...@apache.org> on 2026/01/17 15:32:47 UTC
[PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
KamranImaaz opened a new pull request, #60712:
URL: https://github.com/apache/airflow/pull/60712
Fix #60687
Have Checked the Code and Modified accordingly also checked when the instance is not available to delete it should not log an error(Idempotent).
Tested Working Fine from My Side
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3764088551
> > @shahar1 not only delete operator but execute method in every class is having try and except block. will remove all in One PR? or Individual PR is required for every class in `Bigtable.py`?
>
> For this PR - you could also insert the other class (there is only one). For other files (bigquery, etc.) - please a separate PR for each.
Okay Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3767363857
I have reverted the changes for `delete table operator`. Kindly verify it
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on code in PR #60712:
URL: https://github.com/apache/airflow/pull/60712#discussion_r2701248282
##########
providers/google/tests/unit/google/cloud/operators/test_bigtable.py:
##########
@@ -584,66 +584,21 @@ def test_empty_attribute(self, mock_hook, missing_attribute, project_id, instanc
mock_hook.assert_not_called()
@mock.patch("airflow.providers.google.cloud.operators.bigtable.BigtableHook")
- def test_deleting_instance_that_doesnt_exists(self, mock_hook):
- op = BigtableDeleteInstanceOperator(
- project_id=PROJECT_ID,
- instance_id=INSTANCE_ID,
- task_id="id",
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.side_effect = mock.Mock(
- side_effect=google.api_core.exceptions.NotFound("Instance not found.")
- )
- op.execute(None)
- mock_hook.assert_called_once_with(
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.assert_called_once_with(
- project_id=PROJECT_ID, instance_id=INSTANCE_ID
- )
-
- @mock.patch("airflow.providers.google.cloud.operators.bigtable.BigtableHook")
- def test_deleting_instance_that_doesnt_exists_empty_project_id(self, mock_hook):
- op = BigtableDeleteInstanceOperator(
- instance_id=INSTANCE_ID,
- task_id="id",
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.side_effect = mock.Mock(
- side_effect=google.api_core.exceptions.NotFound("Instance not found.")
- )
- op.execute(None)
- mock_hook.assert_called_once_with(
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.assert_called_once_with(
- project_id=None, instance_id=INSTANCE_ID
- )
+ def test_delete_instance_when_instance_missing_does_not_fail(self, mock_hook):
+ # Simulate hook handling missing instance gracefully
+ mock_hook.return_value.delete_instance.return_value = None
- @mock.patch("airflow.providers.google.cloud.operators.bigtable.BigtableHook")
- def test_different_error_reraised(self, mock_hook):
op = BigtableDeleteInstanceOperator(
project_id=PROJECT_ID,
instance_id=INSTANCE_ID,
task_id="id",
gcp_conn_id=GCP_CONN_ID,
impersonation_chain=IMPERSONATION_CHAIN,
)
- mock_hook.return_value.delete_instance.side_effect = mock.Mock(
- side_effect=google.api_core.exceptions.GoogleAPICallError("error")
- )
- with pytest.raises(google.api_core.exceptions.GoogleAPICallError):
- op.execute(None)
+ # Should NOT raise
Review Comment:
Okay Sure
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3779484671
Hello @KamranImaaz thank you for your PR!
You mentioned in the [issue](https://github.com/apache/airflow/issues/60687) that we do not need this `try ... except` block in the Operator side because we handle it on the Hook side. I have checked the Hook code for `delete_instance` and `get_instance` methods and I do not see any `try...except` block in these methods which handles the `google.api_core.exceptions.NotFound` exception. Here is the code for [delete_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L84C1-L105C1) and [get_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L69C1-L85C1). As I see `get_instance` method has a `if-clause` statement which is checking `instance` for existing and returns `None` in case if `instance` does not exist and `delete_instance` relies on it and prints warning message in the logs. @KamranImaaz could you please clarify if I missed something?
In my opinion it is not the same as handling errors by the `try ... except` block. This line of the code:
`instance = self._get_client(project_id=project_id).instance(instance_id)` still can produce `google.api_core.exceptions.NotFound` exception and I do not see any proves that removing `try ... execept` block does not break `BigtableDeleteInstanceOperator` operator. Because this `try ... execpt` block is needed for finishing the task successfully in case when `instance` is not found. @KamranImaaz could you please share with us the screenshots that the task which runs this operator has the same behavior as before and will not fail in case when the instance is not found?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3765784530
btw, there **might** be a case for the other operator (DeleteInstance) to remove handling of `NotFound` resulting from the `delete_instance` (because the latter handles it internally in the hook) - but currently it seems that someone else opened a PR for that: https://github.com/apache/airflow/pull/60727.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #60712:
URL: https://github.com/apache/airflow/pull/60712#discussion_r2702728664
##########
providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py:
##########
@@ -251,9 +251,11 @@ def delete_table(self, instance_id: str, table_id: str, project_id: str) -> None
"""
instance = self.get_instance(instance_id=instance_id, project_id=project_id)
if instance is None:
- raise RuntimeError(f"Instance {instance_id} did not exist; unable to delete table {table_id}")
- table = instance.table(table_id=table_id)
- table.delete()
+ self.log.warning("Instance '%s' does not exist in project '%s'. Exiting", instance_id, project_id)
Review Comment:
Here it actually makes sense to raise a `RuntimeError`, because the instance needs to exists so we could actually delete the table. After we ensure that it exists, then we could actually use `try..except` to supress the table if it doesn't exist - which apparently we already do in the operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 merged PR #60712:
URL: https://github.com/apache/airflow/pull/60712
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3764087406
> @shahar1 not only delete operator but execute method in every class is having try and except block. will remove all in One PR? or Individual PR is required for every class in `Bigtable.py`?
For this PR - you could also insert the other class (there is only one).
For other files (bigquery, etc.) - please a separate PR for each.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3764025274
Hi @shahar1 Can you review it once. If it's good from your side.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3776585730
> > Hi @shahar1 All CI checks have passed. Can you please review it once
>
> I have reviewed it. Please note that there's no need to tag me in each comment - once I'm tagged as reviewer, I get notified for every change and comment in the PR. I'll let Google's team a couple of days to review it before merging - in the meanwhile, you could work on the other operators or start working on new PRs labeled as [good first issues](https://github.com/apache/airflow/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22).
I Generally have the habit of tagging. But I'll follow your advice. I Apologize for the inconvinience caused.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3774716630
> Hi @shahar1 All CI checks have passed. Can you please review it once
I have reviewed it. Please note that there's no need to tag me in each comment - once I'm tagged as reviewer, I get notified for every change and comment in the PR.
I'll let Google's team a couple of days to review it before merging - in the meanwhile, you could start working on new PRs labeled as [good first issues](https://github.com/apache/airflow/issues?q=is%3Aissue%20state%3Aopen%20label%3A%22good%20first%20issue%22).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3764063185
> Overall LGTM, small comment.
>
> Also, could you please apply the same fix to the `try..except` block of `BigtableDeleteTableOperator`? (same principle)
>
> @VladaZakharova I'll be happy for your review here as well - allegedly both `Delete` operators should never reach the `NotFound` case as it is already handled in the hook (it's a 7 years old code, so maybe things have been a bit different in the past).
Yes Once it is approved I will raise another PR and modify the `Bigtable.py` file.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3764071287
@shahar1 not only delete operator but execute method in every class is having try and except block. will remove all in One PR? or Individual PR is required for every class in `Bigtable.py`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #60712:
URL: https://github.com/apache/airflow/pull/60712#discussion_r2702728664
##########
providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py:
##########
@@ -251,9 +251,11 @@ def delete_table(self, instance_id: str, table_id: str, project_id: str) -> None
"""
instance = self.get_instance(instance_id=instance_id, project_id=project_id)
if instance is None:
- raise RuntimeError(f"Instance {instance_id} did not exist; unable to delete table {table_id}")
- table = instance.table(table_id=table_id)
- table.delete()
+ self.log.warning("Instance '%s' does not exist in project '%s'. Exiting", instance_id, project_id)
Review Comment:
Here it actually makes sense to raise a `RuntimeError`, because the instance needs to exists so we could actually delete the table. After we ensure that it exists, then we could actually use `try..except` to supress the table if it doesn't exist - which apparently we do only in the operator.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3765633053
@shahar1 Apologies for Late Updates. Also in `Bigtable.py` there are still many classes such as `BigtableCreateInstanceOperator`, `BigtableUpdateInstanceOperator`, `BigtableCreateTableOperator`, `BigtableUpdateClusterOperator` which are having redundant try..except blocks will remove it in the next PR
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3767283665
> Hey @KamranImaaz , I went over the PR more thoroughly and I think that I've accidentally misled you - I had initially thought that we use `try..except` in both the operators and hooks (which is indeed not necessary), but as it seems - the hook actually calls the API straight forward, and then the operators use `try..except` to reach idempotency in action, which is already the desired behavior. When you delete a table and the instance does not exist, it rightously raises a `RuntimeError` error - because the instance is assumed to exist so you would be able to delete the table (whether it exists or not). If you run the Delete table operator, and the instance does not exist - it means that this assumption is wrong (regardless of the idempotency of the table), and the user needs to be alerted about it as a failure of the operator. I sincerely apologize for missing it - but unfortunately it seems that we won't get too much from this one. I'm aware that it's important for you to get y
our 3rd PR merged soon - so please try to look for existing issues labeled as [good first issue](https://github.com/apache/airflow/issues?q=is%3Aissue%20state%3Aopen%20label%3Aprovider%3Agoogle%20label%3A%22good%20first%20issue%22), and let me know.
Yes @shahar1 That doubt I was having initially. That's why I did not pushed it in initial commit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3780825877
> Hello @KamranImaaz thank you for your PR! You mentioned in the [issue](https://github.com/apache/airflow/issues/60687) that we do not need this `try ... except` block in the Operator side because we handle it on the Hook side. I have checked the Hook code for `delete_instance` and `get_instance` methods and I do not see any `try...except` block in these methods which handles the `google.api_core.exceptions.NotFound` exception. Here is the code for [delete_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L84C1-L105C1) and [get_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L69C1-L85C1). As I see `get_instance` method has a `if-clause` statement which is checking `instance` for existing and returns `None` in case if `instance` does not exist and `delete_instance` relies on it and prints warning message in the logs. @KamranImaaz cou
ld you please clarify if I missed something?
>
> In my opinion it is not the same as handling errors by the `try ... except` block. This line of the code: `instance = self._get_client(project_id=project_id).instance(instance_id)` still can produce `google.api_core.exceptions.NotFound` exception and I do not see any proves that removing `try ... execept` block does not break `BigtableDeleteInstanceOperator` operator. Because this `try ... execpt` block is needed for finishing the task successfully in case when `instance` is not found. @KamranImaaz could you please share with us the screenshots that the task which runs this operator has the same behavior as before and will not fail in case when the instance is not found?
Correct me if Iam wrong. Removing `try..except` block in the operator is safe because the hook already checks `instance.exists()` and returns `None` when the instance does not exist. `client.instance(instance_id)` does not make a API call. `instance.exists()` is the one who calls the API and it returns `False`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3783088333
> Hello @KamranImaaz thank you for your PR! You mentioned in the [issue](https://github.com/apache/airflow/issues/60687) that we do not need this `try ... except` block in the Operator side because we handle it on the Hook side. I have checked the Hook code for `delete_instance` and `get_instance` methods and I do not see any `try...except` block in these methods which handles the `google.api_core.exceptions.NotFound` exception. Here is the code for [delete_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L84C1-L105C1) and [get_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L69C1-L85C1). As I see `get_instance` method has a `if-clause` statement which is checking `instance` for existing and returns `None` in case if `instance` does not exist and `delete_instance` relies on it and prints warning message in the logs. @KamranImaaz cou
ld you please clarify if I missed something?
>
> In my opinion it is not the same as handling errors by the `try ... except` block. This line of the code: `instance = self._get_client(project_id=project_id).instance(instance_id)` still can produce `google.api_core.exceptions.NotFound` exception and I do not see any proves that removing `try ... execept` block does not break `BigtableDeleteInstanceOperator` operator. Because this `try ... execpt` block is needed for finishing the task successfully in case when `instance` is not found. @KamranImaaz could you please share with us the screenshots that the task which runs this operator has the same behavior as before and will not fail in case when the instance is not found?
> > Hello @KamranImaaz thank you for your PR! You mentioned in the [issue](https://github.com/apache/airflow/issues/60687) that we do not need this `try ... except` block in the Operator side because we handle it on the Hook side. I have checked the Hook code for `delete_instance` and `get_instance` methods and I do not see any `try...except` block in these methods which handles the `google.api_core.exceptions.NotFound` exception. Here is the code for [delete_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L84C1-L105C1) and [get_instance](https://github.com/apache/airflow/blob/main/providers/google/src/airflow/providers/google/cloud/hooks/bigtable.py#L69C1-L85C1). As I see `get_instance` method has a `if-clause` statement which is checking `instance` for existing and returns `None` in case if `instance` does not exist and `delete_instance` relies on it and prints warning message in the logs. @KamranImaaz c
ould you please clarify if I missed something?
> > In my opinion it is not the same as handling errors by the `try ... except` block. This line of the code: `instance = self._get_client(project_id=project_id).instance(instance_id)` still can produce `google.api_core.exceptions.NotFound` exception and I do not see any proves that removing `try ... execept` block does not break `BigtableDeleteInstanceOperator` operator. Because this `try ... execpt` block is needed for finishing the task successfully in case when `instance` is not found. @KamranImaaz could you please share with us the screenshots that the task which runs this operator has the same behavior as before and will not fail in case when the instance is not found?
>
> In the issue I mean that the exceptions raised at the Operator Level is already taken care in Hook Level(Checks the existence of instance or not i.e `if...else` blocks).
>
> Correct me if Iam wrong. Removing `try..except` block in the operator is safe because the hook already checks `instance.exists()` and returns `False` when the instance does not exist. `client.instance(instance_id)` does not make a API call. `instance.exists()` is the one who calls the API and it returns `False`
+1 for that:
```python
instance = self._get_client(project_id=project_id).instance(instance_id)
if not instance.exists():
return None
return instance
```
How can the first line produce a `NotFound` error if the next in line checks for existence?
It just constructs the `Instance` object without calling the API (I checked also the nested calls, only assigning attributes).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in airflow/providers/google/cloud/operators/bigtable.py [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on code in PR #60712:
URL: https://github.com/apache/airflow/pull/60712#discussion_r2701235434
##########
providers/google/tests/unit/google/cloud/operators/test_bigtable.py:
##########
@@ -584,66 +584,21 @@ def test_empty_attribute(self, mock_hook, missing_attribute, project_id, instanc
mock_hook.assert_not_called()
@mock.patch("airflow.providers.google.cloud.operators.bigtable.BigtableHook")
- def test_deleting_instance_that_doesnt_exists(self, mock_hook):
- op = BigtableDeleteInstanceOperator(
- project_id=PROJECT_ID,
- instance_id=INSTANCE_ID,
- task_id="id",
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.side_effect = mock.Mock(
- side_effect=google.api_core.exceptions.NotFound("Instance not found.")
- )
- op.execute(None)
- mock_hook.assert_called_once_with(
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.assert_called_once_with(
- project_id=PROJECT_ID, instance_id=INSTANCE_ID
- )
-
- @mock.patch("airflow.providers.google.cloud.operators.bigtable.BigtableHook")
- def test_deleting_instance_that_doesnt_exists_empty_project_id(self, mock_hook):
- op = BigtableDeleteInstanceOperator(
- instance_id=INSTANCE_ID,
- task_id="id",
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.side_effect = mock.Mock(
- side_effect=google.api_core.exceptions.NotFound("Instance not found.")
- )
- op.execute(None)
- mock_hook.assert_called_once_with(
- gcp_conn_id=GCP_CONN_ID,
- impersonation_chain=IMPERSONATION_CHAIN,
- )
- mock_hook.return_value.delete_instance.assert_called_once_with(
- project_id=None, instance_id=INSTANCE_ID
- )
+ def test_delete_instance_when_instance_missing_does_not_fail(self, mock_hook):
+ # Simulate hook handling missing instance gracefully
+ mock_hook.return_value.delete_instance.return_value = None
- @mock.patch("airflow.providers.google.cloud.operators.bigtable.BigtableHook")
- def test_different_error_reraised(self, mock_hook):
op = BigtableDeleteInstanceOperator(
project_id=PROJECT_ID,
instance_id=INSTANCE_ID,
task_id="id",
gcp_conn_id=GCP_CONN_ID,
impersonation_chain=IMPERSONATION_CHAIN,
)
- mock_hook.return_value.delete_instance.side_effect = mock.Mock(
- side_effect=google.api_core.exceptions.GoogleAPICallError("error")
- )
- with pytest.raises(google.api_core.exceptions.GoogleAPICallError):
- op.execute(None)
+ # Should NOT raise
Review Comment:
Please delete this comment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "shahar1 (via GitHub)" <gi...@apache.org>.
shahar1 commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3783225261
@KamranImaaz great job! Looking forward for more PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "MaksYermak (via GitHub)" <gi...@apache.org>.
MaksYermak commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3783182352
@KamranImaaz @shahar1 thank you for the explanation!
My fault now I see that `instance()` is just a construct and, also, I see that we handle `NotFound` exception on the Client side inside `exists` method [here](https://github.com/googleapis/python-bigtable/blob/main/google/cloud/bigtable/instance.py#L361) .
This PR looks good to me, thank you one more time for your clarification
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3783280118
Thanks @shahar1 I will raise different PR for other operators and link it to the same issue. And it was nice working with you. I learned a lot from you. Once again Apologize for any inconvenience caused to you.
Thank you have a great day!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3783283047
Thanks @MaksYermak for your review.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Remove redundant try/except blocks in `BigtableDeleteInstanceOperator` [airflow]
Posted by "KamranImaaz (via GitHub)" <gi...@apache.org>.
KamranImaaz commented on PR #60712:
URL: https://github.com/apache/airflow/pull/60712#issuecomment-3773016546
Hi @shahar1 All CI checks have passed. Can you please review it once
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org