You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@airflow.apache.org by "dmedora (via GitHub)" <gi...@apache.org> on 2023/10/16 20:15:54 UTC
[PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
dmedora opened a new pull request, #34982:
URL: https://github.com/apache/airflow/pull/34982
Closes: #34980
Updated _prepare_sync_plan function to get blobs using the Objects Get API (.get_blob(..)) so that their crc32c value is included in the returned Blob object. As discussed in #34980, the current Objects List method does not return this for CMEK-protected objets, leading to inaccurate hash comparisons.
---
**^ Add meaningful description above**
Read the **[Pull Request Guidelines](https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#pull-request-guidelines)** for more information.
In case of fundamental code changes, an Airflow Improvement Proposal ([AIP](https://cwiki.apache.org/confluence/display/AIRFLOW/Airflow+Improvement+Proposals)) is needed.
In case of a new dependency, check compliance with the [ASF 3rd Party License Policy](https://www.apache.org/legal/resolved.html#category-x).
In case of backwards incompatible changes please leave a note in a newsfragment file, named `{pr_number}.significant.rst` or `{issue_number}.significant.rst`, in [newsfragments](https://github.com/apache/airflow/tree/main/newsfragments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on code in PR #34982:
URL: https://github.com/apache/airflow/pull/34982#discussion_r1362568877
##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -1326,6 +1326,11 @@ def _prepare_sync_plan(
for current_name in names_to_check:
source_blob = source_names_index[current_name]
destination_blob = destination_names_index[current_name]
+ # If either object is CMEK-protected, use the Cloud Storage Objects Get API to retrieve them so that the crc32c is included
+ if source_blob.kms_key_name != None:
+ source_blob = source_bucket.get_blob(source_blob.name)
+ if destination_blob.kms_key_name != None:
Review Comment:
```suggestion
if destination_blob.kms_key_name:
```
##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -1326,6 +1326,11 @@ def _prepare_sync_plan(
for current_name in names_to_check:
source_blob = source_names_index[current_name]
destination_blob = destination_names_index[current_name]
+ # If either object is CMEK-protected, use the Cloud Storage Objects Get API to retrieve them so that the crc32c is included
+ if source_blob.kms_key_name != None:
Review Comment:
```suggestion
if source_blob.kms_key_name:
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "dmedora (via GitHub)" <gi...@apache.org>.
dmedora commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765496038
> Shouldn't this need a test against whatever that blob should be?
Not sure I understand, could you please explain further or with an example? Thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed pull request #34982: Update GCS hook to get crc32c hash for CMEK-protected objects
URL: https://github.com/apache/airflow/pull/34982
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "boring-cyborg[bot] (via GitHub)" <gi...@apache.org>.
boring-cyborg[bot] commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765202922
Congratulations on your first Pull Request and welcome to the Apache Airflow community! If you have any issues or are unsure about any anything please check our Contribution Guide (https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst)
Here are some useful points:
- Pay attention to the quality of your code (ruff, mypy and type annotations). Our [pre-commits]( https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#prerequisites-for-pre-commit-hooks) will help you with that.
- In case of a new feature add useful documentation (in docstrings or in `docs/` directory). Adding a new operator? Check this short [guide](https://github.com/apache/airflow/blob/main/docs/apache-airflow/howto/custom-operator.rst) Consider adding an example DAG that shows how users should use it.
- Consider using [Breeze environment](https://github.com/apache/airflow/blob/main/BREEZE.rst) for testing locally, it's a heavy docker but it ships with a working Airflow and a lot of integrations.
- Be patient and persistent. It might take some time to get a review or get the final approval from Committers.
- Please follow [ASF Code of Conduct](https://www.apache.org/foundation/policies/conduct) for all communication including (but not limited to) comments on Pull Requests, Mailing list and Slack.
- Be sure to read the [Airflow Coding style]( https://github.com/apache/airflow/blob/main/CONTRIBUTING.rst#coding-style-and-best-practices).
- Always keep your Pull Requests rebased, otherwise your build might fail due to changes not related to your commits.
Apache Airflow is a community-driven project and together we are making it better 🚀.
In case of doubts contact the developers at:
Mailing List: dev@airflow.apache.org
Slack: https://s.apache.org/airflow-slack
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #34982:
URL: https://github.com/apache/airflow/pull/34982#discussion_r1430556197
##########
tests/providers/google/cloud/hooks/test_gcs.py:
##########
@@ -1253,6 +1253,45 @@ def test_should_overwrite_files(self, mock_get_conn, mock_delete, mock_rewrite,
any_order=True,
)
mock_copy.assert_not_called()
+
Review Comment:
```suggestion
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "dmedora (via GitHub)" <gi...@apache.org>.
dmedora commented on code in PR #34982:
URL: https://github.com/apache/airflow/pull/34982#discussion_r1362613278
##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -1326,6 +1326,11 @@ def _prepare_sync_plan(
for current_name in names_to_check:
source_blob = source_names_index[current_name]
destination_blob = destination_names_index[current_name]
+ # If either object is CMEK-protected, use the Cloud Storage Objects Get API to retrieve them so that the crc32c is included
+ if source_blob.kms_key_name != None:
Review Comment:
Updated in latest commit, thanks.
##########
airflow/providers/google/cloud/hooks/gcs.py:
##########
@@ -1326,6 +1326,11 @@ def _prepare_sync_plan(
for current_name in names_to_check:
source_blob = source_names_index[current_name]
destination_blob = destination_names_index[current_name]
+ # If either object is CMEK-protected, use the Cloud Storage Objects Get API to retrieve them so that the crc32c is included
+ if source_blob.kms_key_name != None:
+ source_blob = source_bucket.get_blob(source_blob.name)
+ if destination_blob.kms_key_name != None:
Review Comment:
Updated in latest commit, thanks.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on code in PR #34982:
URL: https://github.com/apache/airflow/pull/34982#discussion_r1430574117
##########
tests/providers/google/cloud/hooks/test_gcs.py:
##########
@@ -1285,6 +1285,44 @@ def test_should_overwrite_files(self, mock_get_conn, mock_delete, mock_rewrite,
any_order=True,
)
mock_copy.assert_not_called()
+ @mock.patch(GCS_STRING.format("GCSHook.copy"))
Review Comment:
```suggestion
mock_copy.assert_not_called()
@mock.patch(GCS_STRING.format("GCSHook.copy"))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "heaven00 (via GitHub)" <gi...@apache.org>.
heaven00 commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765512475
sure,
so the function that has been changed is here is a private function, so we can't test the logic that has been added but from what I can understand there should be some behavioral changes to the `sync` function https://github.com/apache/airflow/pull/34982/files#diff-82854006b5553665046db26d43a9dfa90bec78d4ba93e2d2ca7ff5bf632fa624R1179
in terms of the data it is parsing from the blob and in my opinion there should be respective tests added to https://github.com/RNHTTR/airflow/blob/78d1031b34d70784d22fc1360c6f3deeb2c66a2c/airflow/providers/google/cloud/hooks/gcs.py so that the behavior is captured.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1778304223
Are you worry about that you can't test internal static method `_prepare_sync_plan`?
I don't see any problem here because it is internal part of Google Provider so you could also test it as part of Google Provider
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1967947102
This pull request has been automatically marked as stale because it has not had recent activity. It will be closed in 5 days if no further activity occurs. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "heaven00 (via GitHub)" <gi...@apache.org>.
heaven00 commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1765470742
Shouldn't this need a test against whatever that blob should be?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "Taragolis (via GitHub)" <gi...@apache.org>.
Taragolis commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1778200695
Need to fix [static checks](https://github.com/apache/airflow/blob/main/STATIC_CODE_CHECKS.rst#id5) and failed test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "dmedora (via GitHub)" <gi...@apache.org>.
dmedora commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1778213394
Thanks, still figuring out how to write the test correctly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Update GCS hook to get crc32c hash for CMEK-protected objects [airflow]
Posted by "eladkal (via GitHub)" <gi...@apache.org>.
eladkal commented on PR #34982:
URL: https://github.com/apache/airflow/pull/34982#issuecomment-1890686018
@dmedora can you please fix the test?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@airflow.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org