You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "tarun-google (via GitHub)" <gi...@apache.org> on 2026/06/17 21:47:11 UTC
[PR] Validate Project of staging bucket [beam]
tarun-google opened a new pull request, #39008:
URL: https://github.com/apache/beam/pull/39008
Parity feature, similar to Java. Validate that bucket is in the same project as the pipeline when Beam defaults the default naming conventions.
Introducing google-cloud-resource-manager for mapping project_name to project_number. Java used the same dependency for this https://github.com/apache/beam/blob/849fcf89c81e7b8ac2ccdf8e3596d35993ff33ab/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L527
Existing Java:
<img width="964" height="374" alt="Screenshot 2026-06-17 at 2 42 06 PM" src="https://github.com/user-attachments/assets/a4023bc4-e732-4a8c-bdd2-4d724e93a43e" />
Python:
<img width="964" height="374" alt="Screenshot 2026-06-17 at 2 42 36 PM" src="https://github.com/user-attachments/assets/cf0fe9af-6a84-4c9d-b9b2-e9917a8f7010" />
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn merged PR #39008:
URL: https://github.com/apache/beam/pull/39008
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3431705955
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -77,6 +77,24 @@ def default_gcs_bucket_name(project, region):
region, md5(project.encode('utf8')).hexdigest())
+def _get_project_number(project_id, credentials=None):
+ """Resolves a project ID to its project number using Cloud Resource Manager API."""
+ from google.cloud import resourcemanager_v3
+ client = resourcemanager_v3.ProjectsClient(credentials=credentials)
+ project_info = client.get_project(name=f"projects/{project_id}")
+ # project_info.name is of the form "projects/PROJECT_NUMBER"
+ return int(project_info.name.split('/')[-1])
+
+
+def _validate_bucket_project(bucket, project_id, credentials=None):
+ """Verifies that the GCS bucket is owned by the executing project."""
+ bucket_project_number = bucket.project_number
+ project_number = _get_project_number(project_id, credentials=credentials)
+ if bucket_project_number != project_number:
+ raise ValueError(
+ f'Bucket gs://{bucket.name} is not owned by project {project_id}.')
Review Comment:
If, in java we are expecting Cloud Resource Manager API is enabled, it's valid expectation python pipelines also enabled it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3437116229
##########
sdks/python/setup.py:
##########
@@ -531,6 +531,7 @@ def get_portability_package_data():
'google-cloud-datastore>=2.0.0,<3',
'google-cloud-pubsub>=2.1.0,<3',
'google-cloud-storage>=2.18.2,<4',
+ 'google-cloud-resource-manager>=1.12.0,<2',
Review Comment:
sgtm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3431672865
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -77,6 +77,24 @@ def default_gcs_bucket_name(project, region):
region, md5(project.encode('utf8')).hexdigest())
+def _get_project_number(project_id, credentials=None):
+ """Resolves a project ID to its project number using Cloud Resource Manager API."""
+ from google.cloud import resourcemanager_v3
+ client = resourcemanager_v3.ProjectsClient(credentials=credentials)
+ project_info = client.get_project(name=f"projects/{project_id}")
+ # project_info.name is of the form "projects/PROJECT_NUMBER"
+ return int(project_info.name.split('/')[-1])
+
+
+def _validate_bucket_project(bucket, project_id, credentials=None):
+ """Verifies that the GCS bucket is owned by the executing project."""
+ bucket_project_number = bucket.project_number
+ project_number = _get_project_number(project_id, credentials=credentials)
+ if bucket_project_number != project_number:
+ raise ValueError(
+ f'Bucket gs://{bucket.name} is not owned by project {project_id}.')
Review Comment:
I do not think we should make integrity checks defensive.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #39008:
URL: https://github.com/apache/beam/pull/39008#issuecomment-4736419678
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #39008:
URL: https://github.com/apache/beam/pull/39008#issuecomment-4736414443
R: @Abacn @damccorm
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3431676713
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -90,16 +108,18 @@ def get_or_create_default_gcs_bucket(options):
return None
bucket_name = default_gcs_bucket_name(project, region)
- bucket = GcsIO(pipeline_options=options).get_bucket(bucket_name)
+ gcs = GcsIO(pipeline_options=options)
+ bucket = gcs.get_bucket(bucket_name)
if bucket:
+ # Validate project ownership of the pre-existing bucket
+ _validate_bucket_project(bucket, project, credentials=gcs.client._credentials)
Review Comment:
good one, i will fix this
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3431744129
##########
sdks/python/setup.py:
##########
@@ -531,6 +531,7 @@ def get_portability_package_data():
'google-cloud-datastore>=2.0.0,<3',
'google-cloud-pubsub>=2.1.0,<3',
'google-cloud-storage>=2.18.2,<4',
+ 'google-cloud-resource-manager>=1.12.0,<2',
Review Comment:
looks like google-cloud-resource-manager is already introduced as a transitive dependency. It is brought from google-cloud-aiplatform(VertexAI). It is good to add explicitly as being used may be?
https://github.com/apache/beam/blob/master/sdks/python/container/py313/base_image_requirements.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on PR #39008:
URL: https://github.com/apache/beam/pull/39008#issuecomment-4735833168
## Summary of Changes
Hello, I'm Gemini Code Assist[^1]! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!
This pull request introduces a parity feature with the Java SDK by enforcing project ownership validation for default GCS staging buckets. By utilizing the Cloud Resource Manager API, the pipeline now verifies that the staging bucket belongs to the executing project, preventing potential cross-project access issues.
### Highlights
* **Bucket Ownership Validation**: Added a validation check to ensure that the GCS staging bucket is owned by the same project as the pipeline, matching existing Java SDK behavior.
* **Resource Manager Integration**: Introduced the google-cloud-resource-manager dependency to resolve project IDs to project numbers for accurate ownership verification.
* **Testing**: Added unit tests to verify both successful ownership matching and error handling when a project mismatch is detected.
<details>
<summary><b>New Features</b></summary>
🧠 You can now enable **Memory** (public preview) to help **Gemini Code Assist** learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. **Click [here](https://codeassist.google/code-review/login) to enable Memory in your admin console.**
</details>
<details>
<summary><b>Using Gemini Code Assist</b></summary>
<br>
The full guide for Gemini Code Assist can be found on our [documentation page](https://developers.google.com/gemini-code-assist/docs/review-github-code), here are some quick tips.
<b>Invoking Gemini</b>
You can request assistance from Gemini at any point by creating a comment using either `/gemini <command>` or `@gemini-code-assist <command>`. Below is a summary of the supported commands on the current page.
Feature | Command | Description
--- | --- | ---
Code Review | `/gemini review` | Performs a code review for the current pull request in its current state.
Pull Request Summary | `/gemini summary` | Provides a summary of the current pull request in its current state.
Comment | @gemini-code-assist | Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help | `/gemini help` | Displays a list of available commands.
<b>Customization</b>
To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a `.gemini/` folder in the base of the repository. Detailed instructions can be found [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github).
<b>Limitations & Feedback</b>
Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with :thumbsup: and :thumbsdown: on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up [here](https://google.qualtrics.com/jfe/form/SV_2cyuGuTWsEw84yG).
</details>
[^1]: Review the [Privacy Notices](https://policies.google.com/privacy), [Generative AI Prohibited Use Policy](https://policies.google.com/terms/generative-ai/use-policy), [Terms of Service](https://policies.google.com/terms), and learn how to configure Gemini Code Assist in GitHub [here](https://developers.google.com/gemini-code-assist/docs/customize-gemini-behavior-github). Gemini can make mistakes, so double check it and [use code with caution](https://support.google.com/legal/answer/13505487).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3437121969
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -77,6 +77,24 @@ def default_gcs_bucket_name(project, region):
region, md5(project.encode('utf8')).hexdigest())
+def _get_project_number(project_id, credentials=None):
+ """Resolves a project ID to its project number using Cloud Resource Manager API."""
+ from google.cloud import resourcemanager_v3
+ client = resourcemanager_v3.ProjectsClient(credentials=credentials)
+ project_info = client.get_project(name=f"projects/{project_id}")
+ # project_info.name is of the form "projects/PROJECT_NUMBER"
+ return int(project_info.name.split('/')[-1])
+
+
+def _validate_bucket_project(bucket, project_id, credentials=None):
+ """Verifies that the GCS bucket is owned by the executing project."""
+ bucket_project_number = bucket.project_number
+ project_number = _get_project_number(project_id, credentials=credentials)
+ if bucket_project_number != project_number:
+ raise ValueError(
+ f'Bucket gs://{bucket.name} is not owned by project {project_id}.')
Review Comment:
It may be a valid concern. Python SDK is not only used in Beam but also internal FlumePython, and we should probably handle mock as well, to avoid future back-and-forth if breaking tests and production use cases
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3431744129
##########
sdks/python/setup.py:
##########
@@ -531,6 +531,7 @@ def get_portability_package_data():
'google-cloud-datastore>=2.0.0,<3',
'google-cloud-pubsub>=2.1.0,<3',
'google-cloud-storage>=2.18.2,<4',
+ 'google-cloud-resource-manager>=1.12.0,<2',
Review Comment:
looks like google-cloud-resource-manager is already introduced as a transitive dependency. Should we just rely on that and remove this?
https://github.com/apache/beam/blob/master/sdks/python/container/py313/base_image_requirements.txt
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #39008:
URL: https://github.com/apache/beam/pull/39008#discussion_r3431669058
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -77,6 +77,24 @@ def default_gcs_bucket_name(project, region):
region, md5(project.encode('utf8')).hexdigest())
+def _get_project_number(project_id, credentials=None):
+ """Resolves a project ID to its project number using Cloud Resource Manager API."""
+ from google.cloud import resourcemanager_v3
+ client = resourcemanager_v3.ProjectsClient(credentials=credentials)
+ project_info = client.get_project(name=f"projects/{project_id}")
+ # project_info.name is of the form "projects/PROJECT_NUMBER"
+ return int(project_info.name.split('/')[-1])
+
+
+def _validate_bucket_project(bucket, project_id, credentials=None):
+ """Verifies that the GCS bucket is owned by the executing project."""
+ bucket_project_number = bucket.project_number
+ project_number = _get_project_number(project_id, credentials=credentials)
+ if bucket_project_number != project_number:
+ raise ValueError(
+ f'Bucket gs://{bucket.name} is not owned by project {project_id}.')
Review Comment:
To prevent pipeline startup failures, we should make this validation more defensive:
1. Handle cases where `bucket.project_number` is `None` (e.g., when using local GCS emulators or mock buckets in tests).
2. Catch potential exceptions from `_get_project_number` (such as when the Cloud Resource Manager API is not enabled, or the executing service account lacks permissions to call it) and log a warning instead of crashing the pipeline.
```python
def _validate_bucket_project(bucket, project_id, credentials=None):
"""Verifies that the GCS bucket is owned by the executing project."""
bucket_project_number = bucket.project_number
if bucket_project_number is None:
_LOGGER.warning(
'Bucket gs://%s does not have a project number. Skipping ownership validation.',
bucket.name)
return
try:
project_number = _get_project_number(project_id, credentials=credentials)
except Exception as e:
_LOGGER.warning(
'Failed to resolve project number for project %s: %s. '
'Skipping bucket ownership validation.',
project_id,
e)
return
if bucket_project_number != project_number:
raise ValueError(
f'Bucket gs://{bucket.name} is not owned by project {project_id}.')
```
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -77,6 +77,24 @@ def default_gcs_bucket_name(project, region):
region, md5(project.encode('utf8')).hexdigest())
+def _get_project_number(project_id, credentials=None):
+ """Resolves a project ID to its project number using Cloud Resource Manager API."""
+ from google.cloud import resourcemanager_v3
+ client = resourcemanager_v3.ProjectsClient(credentials=credentials)
+ project_info = client.get_project(name=f"projects/{project_id}")
+ # project_info.name is of the form "projects/PROJECT_NUMBER"
+ return int(project_info.name.split('/')[-1])
Review Comment:
If `project_id` is already a project number (either as an integer or a numeric string), we can return it directly and avoid making a redundant, slow network call to the Cloud Resource Manager API.
```suggestion
def _get_project_number(project_id, credentials=None):
"""Resolves a project ID to its project number using Cloud Resource Manager API."""
if isinstance(project_id, int):
return project_id
if isinstance(project_id, str) and project_id.isdigit():
return int(project_id)
from google.cloud import resourcemanager_v3
client = resourcemanager_v3.ProjectsClient(credentials=credentials)
project_info = client.get_project(name=f"projects/{project_id}")
# project_info.name is of the form "projects/PROJECT_NUMBER"
return int(project_info.name.split('/')[-1])
```
##########
sdks/python/apache_beam/io/gcp/gcsio.py:
##########
@@ -90,16 +108,18 @@ def get_or_create_default_gcs_bucket(options):
return None
bucket_name = default_gcs_bucket_name(project, region)
- bucket = GcsIO(pipeline_options=options).get_bucket(bucket_name)
+ gcs = GcsIO(pipeline_options=options)
+ bucket = gcs.get_bucket(bucket_name)
if bucket:
+ # Validate project ownership of the pre-existing bucket
+ _validate_bucket_project(bucket, project, credentials=gcs.client._credentials)
Review Comment:
Accessing the private attribute `_credentials` directly from `gcs.client` is risky because private attributes are not part of the public API and can change or be removed in future versions of `google-cloud-storage`. Using `getattr` with a fallback is safer.
```python
# Validate project ownership of the pre-existing bucket
_validate_bucket_project(
bucket,
project,
credentials=getattr(gcs.client, '_credentials', None))
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Validate Project of staging bucket [beam]
Posted by "tarun-google (via GitHub)" <gi...@apache.org>.
tarun-google commented on PR #39008:
URL: https://github.com/apache/beam/pull/39008#issuecomment-4744708292
Filing Tests are not relevant. This is done from my end. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org