You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by GitBox <gi...@apache.org> on 2020/03/04 02:30:49 UTC
[GitHub] [beam] ihji opened a new pull request #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
ihji opened a new pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Go_VR_Spark/lastCompletedBuild/)
Java | [](https://builds.apache.org/job/beam_PostCommit_Java/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Java11/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink_Streaming/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Spark_Batch/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_SparkStructuredStreaming/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python36/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python37/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow_V2/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PostCommit_Python35_VR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Spark/lastCompletedBuild/)
XLang | --- | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Flink/lastCompletedBuild/) | --- | --- | [](https://builds.apache.org/job/beam_PostCommit_XVR_Spark/lastCompletedBuild/)
Pre-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
--- |Java | Python | Go | Website
--- | --- | --- | --- | ---
Non-portable | [](https://builds.apache.org/job/beam_PreCommit_Java_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Python_Cron/lastCompletedBuild/)<br>[](https://builds.apache.org/job/beam_PreCommit_PythonLint_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Go_Cron/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PreCommit_Website_Cron/lastCompletedBuild/)
Portable | --- | [](https://builds.apache.org/job/beam_PreCommit_Portable_Python_Cron/lastCompletedBuild/) | --- | ---
See [.test-infra/jenkins/README](https://github.com/apache/beam/blob/master/.test-infra/jenkins/README.md) for trigger phrase, status and link of all Jenkins jobs.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-602822540
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530581
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
Review comment:
Please add a unit test
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-601864064
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-604655018
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-602855911
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605473827
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r393879915
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
google_cloud_options = options.view_as(GoogleCloudOptions)
if google_cloud_options.staging_location is None:
raise RuntimeError('The --staging_location option must be specified.')
if google_cloud_options.temp_location is None:
raise RuntimeError('The --temp_location option must be specified.')
+ resources = []
+ for _, env in pipeline.components.environments.items():
+ for dep in env.dependencies:
+ if dep.type_urn != common_urns.artifact_types.FILE.urn:
+ raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
+ if dep.role_urn != common_urns.artifact_roles.STAGING_TO.urn:
+ raise RuntimeError('unsupported role type %s' % dep.role_urn)
+ type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
+ dep.type_payload)
+ role_payload = \
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on issue #11039: [BEAM-9383] Staging Dataflow
artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-600925930
@chamikaramj PTAL and rerun the tests.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530711
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
google_cloud_options = options.view_as(GoogleCloudOptions)
if google_cloud_options.staging_location is None:
raise RuntimeError('The --staging_location option must be specified.')
if google_cloud_options.temp_location is None:
raise RuntimeError('The --temp_location option must be specified.')
+ resources = []
+ for _, env in pipeline.components.environments.items():
+ for dep in env.dependencies:
+ if dep.type_urn != common_urns.artifact_types.FILE.urn:
+ raise RuntimeError('unsupported artifact type %s' % dep.type_urn)
+ if dep.role_urn != common_urns.artifact_roles.STAGING_TO.urn:
+ raise RuntimeError('unsupported role type %s' % dep.role_urn)
+ type_payload = beam_runner_api_pb2.ArtifactFilePayload.FromString(
+ dep.type_payload)
+ role_payload = \
Review comment:
Nit: Pls use ( instead of \ for formatting.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605031622
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-603263934
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on issue #11039: [BEAM-9383] Staging Dataflow
artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-594932376
This PR depends on #10621. needs to be rebased before merging.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605032008
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-606018431
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-604072538
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605300008
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605472960
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r393879970
##########
File path: sdks/python/apache_beam/runners/dataflow/internal/apiclient.py
##########
@@ -563,19 +562,31 @@ def _gcs_file_copy(self, from_path, to_path):
with open(from_path, 'rb') as f:
self.stage_file(to_folder, to_name, f, total_size=total_size)
- def _stage_resources(self, options):
+ def _stage_resources(self, pipeline, options):
Review comment:
done
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605031496
Run Java PreCommi
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r394721868
##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger debuggerClient, String uniquifie
}
}
+ private List<DataflowPackage> stageArtifacts(RunnerApi.Pipeline pipeline) {
Review comment:
It's private method so we can't directly test it. Existing test (such as https://github.com/apache/beam/blob/master/runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/DataflowRunnerTest.java#L758) should cover this method too.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530472
##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger debuggerClient, String uniquifie
}
}
+ private List<DataflowPackage> stageArtifacts(RunnerApi.Pipeline pipeline) {
Review comment:
Please add a unit test.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605473440
Run Portable_Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605475407
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r392530455
##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger debuggerClient, String uniquifie
}
}
+ private List<DataflowPackage> stageArtifacts(RunnerApi.Pipeline pipeline) {
+ ImmutableList.Builder<String> filesToStageBuilder = ImmutableList.builder();
+ for (Map.Entry<String, RunnerApi.Environment> entry :
+ pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info : entry.getValue().getDependenciesList()) {
+ if (!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn())) {
Review comment:
Check the ROLE as well ?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605031824
Run Python2_PVR_Flink PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605536418
Run Java PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605031722
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605298244
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on issue #11039: [BEAM-9383] Staging Dataflow
artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-598435272
R: @robertwb, @chamikaramj
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-606018509
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] ihji commented on a change in pull request #11039:
[BEAM-9383] Staging Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
ihji commented on a change in pull request #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#discussion_r394722832
##########
File path: runners/google-cloud-dataflow-java/src/main/java/org/apache/beam/runners/dataflow/DataflowRunner.java
##########
@@ -752,6 +759,27 @@ private Debuggee registerDebuggee(CloudDebugger debuggerClient, String uniquifie
}
}
+ private List<DataflowPackage> stageArtifacts(RunnerApi.Pipeline pipeline) {
+ ImmutableList.Builder<String> filesToStageBuilder = ImmutableList.builder();
+ for (Map.Entry<String, RunnerApi.Environment> entry :
+ pipeline.getComponents().getEnvironmentsMap().entrySet()) {
+ for (RunnerApi.ArtifactInformation info : entry.getValue().getDependenciesList()) {
+ if (!BeamUrns.getUrn(RunnerApi.StandardArtifacts.Types.FILE).equals(info.getTypeUrn())) {
Review comment:
Currently, DataflowRunner auto-generates staging names based on source file names. So ROLE (STAGING_TO) is ignored anyway. We could change this behavior later in BEAM-9455.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-605298889
Run Python PreCommit
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-603263845
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
[GitHub] [beam] chamikaramj commented on issue #11039: [BEAM-9383] Staging
Dataflow artifacts from environment
Posted by GitBox <gi...@apache.org>.
chamikaramj commented on issue #11039: [BEAM-9383] Staging Dataflow artifacts from environment
URL: https://github.com/apache/beam/pull/11039#issuecomment-598458492
Retest this please
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services