You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2020/06/10 23:59:23 UTC
[GitHub] [beam] pabloem opened a new pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
pabloem opened a new pull request #11982:
URL: https://github.com/apache/beam/pull/11982
Following the logic from Java (https://github.com/apache/beam/blob/b56740f0e8cd80c2873412847d0b336837429fb9/sdks/java/extensions/google-cloud-platform-core/src/main/java/org/apache/beam/sdk/extensions/gcp/options/GcpOptions.java#L332).
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Samza | Spark
--- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Java11/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11982:
URL: https://github.com/apache/beam/pull/11982#discussion_r441274707
##########
File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
##########
@@ -93,7 +93,7 @@ def test_missing_required_options(self):
self.assertEqual(
self.check_errors_for_arguments(
- errors, ['project', 'staging_location', 'temp_location', 'region']),
+ errors, ['project', 'temp_location', 'staging_location', 'region']),
Review comment:
Fixed this mixup.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642897706
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-645071314
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643510455
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642331818
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643510550
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643509531
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11982:
URL: https://github.com/apache/beam/pull/11982#discussion_r439674896
##########
File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
##########
@@ -93,7 +93,7 @@ def test_missing_required_options(self):
self.assertEqual(
self.check_errors_for_arguments(
- errors, ['project', 'staging_location', 'temp_location', 'region']),
+ errors, ['project', 'temp_location', 'staging_location', 'region']),
Review comment:
Rest of the PR LG
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-644269595
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-645000120
Run Python 3.7 PostCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11982:
URL: https://github.com/apache/beam/pull/11982#discussion_r439078406
##########
File path: sdks/python/apache_beam/io/gcp/gcsio.py
##########
@@ -110,6 +110,27 @@ def parse_gcs_path(gcs_path, object_optional=False):
return match.group(1), match.group(2)
+def default_gcs_bucket_name(project, region):
+ from hashlib import md5
+ return 'dataflow-staging-%s-%s' % (
+ region, md5(project.encode('utf8')).hexdigest())
+
+
+def get_or_create_default_gcs_bucket(project, region, kms_key=None):
Review comment:
I realize that there's a check for this in `_create_default_gcs_bucket`, but this method should fail if kms_key is set. Otherwise, it's saying that it's okay to have a default bucket with a KMS key.
(The default bucket should not use a KMS key, and if a KMS key is specified Beam should not use a bucket with some other key (KMS or Google managed).)
Refs:
https://github.com/apache/beam/pull/8135#discussion_r274695249
https://github.com/apache/beam/pull/8830/files
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643510430
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642929886
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11982:
URL: https://github.com/apache/beam/pull/11982#discussion_r439141406
##########
File path: sdks/python/apache_beam/io/gcp/gcsio.py
##########
@@ -110,6 +110,27 @@ def parse_gcs_path(gcs_path, object_optional=False):
return match.group(1), match.group(2)
+def default_gcs_bucket_name(project, region):
+ from hashlib import md5
+ return 'dataflow-staging-%s-%s' % (
+ region, md5(project.encode('utf8')).hexdigest())
+
+
+def get_or_create_default_gcs_bucket(project, region, kms_key=None):
Review comment:
I still need to review the rest of the code. Can look tomorrow, overloaded today
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem merged pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem merged pull request #11982:
URL: https://github.com/apache/beam/pull/11982
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642327786
I've tested this on my machine
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642929745
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643511451
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] udim commented on a change in pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
udim commented on a change in pull request #11982:
URL: https://github.com/apache/beam/pull/11982#discussion_r439674768
##########
File path: sdks/python/apache_beam/options/pipeline_options_validator_test.py
##########
@@ -93,7 +93,7 @@ def test_missing_required_options(self):
self.assertEqual(
self.check_errors_for_arguments(
- errors, ['project', 'staging_location', 'temp_location', 'region']),
+ errors, ['project', 'temp_location', 'staging_location', 'region']),
Review comment:
I don't understand this change - the order shouldn't matter, right?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642839805
retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642331889
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642884059
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642331957
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on a change in pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on a change in pull request #11982:
URL: https://github.com/apache/beam/pull/11982#discussion_r439094715
##########
File path: sdks/python/apache_beam/io/gcp/gcsio.py
##########
@@ -110,6 +110,27 @@ def parse_gcs_path(gcs_path, object_optional=False):
return match.group(1), match.group(2)
+def default_gcs_bucket_name(project, region):
+ from hashlib import md5
+ return 'dataflow-staging-%s-%s' % (
+ region, md5(project.encode('utf8')).hexdigest())
+
+
+def get_or_create_default_gcs_bucket(project, region, kms_key=None):
Review comment:
okay I've made this not work in that case. WDYT?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643512286
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643442446
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-643507056
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] pabloem commented on pull request #11982: [BEAM-6892] Supporting bucket auto-creation for Dataflow.
Posted by GitBox <gi...@apache.org>.
pabloem commented on pull request #11982:
URL: https://github.com/apache/beam/pull/11982#issuecomment-642327079
r: @udim
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org