You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ro...@apache.org on 2020/03/31 22:50:03 UTC
[beam] branch master updated (2f12213 -> dbc21d9)
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 2f12213 Merge pull request #11254: [BEAM-7961] Refactors X-Lang test pipelines
add b6f2db5 [BEAM-9577] Rename the Artifact{Staging,Retrieval}Service.
add 4c25de6 [BEAM-9577] Define the new Artifact{Staging,Retrieval}Service.
add f2a58b4 [BEAM-9577] Regenerate protos.
add b2ad0f9 [BEAM-9577] Implement the new Artifact{Staging,Retrieval}Services in Python.
new dbc21d9 Merge pull request #11203 [BEAM-9577] Define and implement dependency-aware artifact staging service.
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../src/main/proto/beam_provision_api.proto | 2 +-
.../src/main/proto/beam_artifact_api.proto | 92 +-
.../core/construction/ArtifactServiceStager.java | 16 +-
.../InMemoryArtifactStagerService.java | 8 +-
.../runners/flink/FlinkJobServerDriverTest.java | 2 +-
...=> AbstractLegacyArtifactRetrievalService.java} | 22 +-
...a => AbstractLegacyArtifactStagingService.java} | 13 +-
...mFileSystemLegacyArtifactRetrievalService.java} | 17 +-
...eamFileSystemLegacyArtifactStagingService.java} | 22 +-
...ClassLoaderLegacyArtifactRetrievalService.java} | 11 +-
...avaFilesystemLegacyArtifactStagingService.java} | 11 +-
...ce.java => LegacyArtifactRetrievalService.java} | 2 +-
.../control/DefaultJobBundleFactory.java | 18 +-
.../environment/DockerEnvironmentFactory.java | 10 +-
.../environment/EmbeddedEnvironmentFactory.java | 4 +-
.../environment/EnvironmentFactory.java | 4 +-
.../environment/ExternalEnvironmentFactory.java | 10 +-
.../environment/ProcessEnvironmentFactory.java | 10 +-
.../StaticRemoteEnvironmentFactory.java | 4 +-
.../fnexecution/jobsubmission/JobServerDriver.java | 19 +-
.../jobsubmission/PortablePipelineJarCreator.java | 29 +-
.../BeamFileSystemArtifactServicesTest.java | 72 +-
.../artifact/ClassLoaderArtifactServiceTest.java | 84 +-
.../control/DefaultJobBundleFactoryTest.java | 6 +-
.../environment/DockerEnvironmentFactoryTest.java | 4 +-
.../environment/ProcessEnvironmentFactoryTest.java | 4 +-
.../beam/runners/samza/SamzaJobServerDriver.java | 4 +-
sdks/go/cmd/beamctl/cmd/artifact.go | 4 +-
sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go | 2 +-
sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 4 +-
sdks/go/pkg/beam/artifact/materialize.go | 12 +-
sdks/go/pkg/beam/artifact/materialize_test.go | 8 +-
sdks/go/pkg/beam/artifact/server_test.go | 8 +-
sdks/go/pkg/beam/artifact/stage.go | 10 +-
sdks/go/pkg/beam/artifact/stage_test.go | 8 +-
.../model/fnexecution_v1/beam_provision_api.pb.go | 2 +-
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 929 +++++++++++++++++----
.../pkg/beam/runners/universal/runnerlib/stage.go | 2 +-
sdks/python/.pylintrc | 1 +
.../runners/portability/abstract_job_service.py | 2 +-
.../runners/portability/artifact_service.py | 263 +++++-
.../runners/portability/artifact_service_test.py | 170 +++-
.../runners/portability/fn_api_runner/fn_runner.py | 4 -
.../portability/fn_api_runner/worker_handlers.py | 8 +-
.../runners/portability/local_job_service.py | 2 +-
.../runners/portability/local_job_service_test.py | 2 +-
.../runners/portability/portable_stager.py | 14 +-
.../runners/portability/portable_stager_test.py | 11 +-
.../portability/spark_uber_jar_job_server_test.py | 2 +-
49 files changed, 1552 insertions(+), 416 deletions(-)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{AbstractArtifactRetrievalService.java => AbstractLegacyArtifactRetrievalService.java} (90%)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{AbstractArtifactStagingService.java => AbstractLegacyArtifactStagingService.java} (95%)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{BeamFileSystemArtifactRetrievalService.java => BeamFileSystemLegacyArtifactRetrievalService.java} (84%)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{BeamFileSystemArtifactStagingService.java => BeamFileSystemLegacyArtifactStagingService.java} (90%)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{ClassLoaderArtifactRetrievalService.java => ClassLoaderLegacyArtifactRetrievalService.java} (80%)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{JavaFilesystemArtifactStagingService.java => JavaFilesystemLegacyArtifactStagingService.java} (87%)
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/{ArtifactRetrievalService.java => LegacyArtifactRetrievalService.java} (93%)
[beam] 01/01: Merge pull request #11203 [BEAM-9577] Define and
implement dependency-aware artifact staging service.
Posted by ro...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
robertwb pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit dbc21d97de83de08568ddd45ba5ec21d3f83d118
Merge: 2f12213 b2ad0f9
Author: Robert Bradshaw <ro...@google.com>
AuthorDate: Tue Mar 31 15:49:48 2020 -0700
Merge pull request #11203 [BEAM-9577] Define and implement dependency-aware artifact staging service.
.../src/main/proto/beam_provision_api.proto | 2 +-
.../src/main/proto/beam_artifact_api.proto | 92 +-
.../core/construction/ArtifactServiceStager.java | 16 +-
.../InMemoryArtifactStagerService.java | 8 +-
.../runners/flink/FlinkJobServerDriverTest.java | 2 +-
...=> AbstractLegacyArtifactRetrievalService.java} | 22 +-
...a => AbstractLegacyArtifactStagingService.java} | 13 +-
...mFileSystemLegacyArtifactRetrievalService.java} | 17 +-
...eamFileSystemLegacyArtifactStagingService.java} | 22 +-
...ClassLoaderLegacyArtifactRetrievalService.java} | 11 +-
...avaFilesystemLegacyArtifactStagingService.java} | 11 +-
...ce.java => LegacyArtifactRetrievalService.java} | 2 +-
.../control/DefaultJobBundleFactory.java | 18 +-
.../environment/DockerEnvironmentFactory.java | 10 +-
.../environment/EmbeddedEnvironmentFactory.java | 4 +-
.../environment/EnvironmentFactory.java | 4 +-
.../environment/ExternalEnvironmentFactory.java | 10 +-
.../environment/ProcessEnvironmentFactory.java | 10 +-
.../StaticRemoteEnvironmentFactory.java | 4 +-
.../fnexecution/jobsubmission/JobServerDriver.java | 19 +-
.../jobsubmission/PortablePipelineJarCreator.java | 29 +-
.../BeamFileSystemArtifactServicesTest.java | 72 +-
.../artifact/ClassLoaderArtifactServiceTest.java | 84 +-
.../control/DefaultJobBundleFactoryTest.java | 6 +-
.../environment/DockerEnvironmentFactoryTest.java | 4 +-
.../environment/ProcessEnvironmentFactoryTest.java | 4 +-
.../beam/runners/samza/SamzaJobServerDriver.java | 4 +-
sdks/go/cmd/beamctl/cmd/artifact.go | 4 +-
sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go | 2 +-
sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 4 +-
sdks/go/pkg/beam/artifact/materialize.go | 12 +-
sdks/go/pkg/beam/artifact/materialize_test.go | 8 +-
sdks/go/pkg/beam/artifact/server_test.go | 8 +-
sdks/go/pkg/beam/artifact/stage.go | 10 +-
sdks/go/pkg/beam/artifact/stage_test.go | 8 +-
.../model/fnexecution_v1/beam_provision_api.pb.go | 2 +-
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 929 +++++++++++++++++----
.../pkg/beam/runners/universal/runnerlib/stage.go | 2 +-
sdks/python/.pylintrc | 1 +
.../runners/portability/abstract_job_service.py | 2 +-
.../runners/portability/artifact_service.py | 263 +++++-
.../runners/portability/artifact_service_test.py | 170 +++-
.../runners/portability/fn_api_runner/fn_runner.py | 4 -
.../portability/fn_api_runner/worker_handlers.py | 8 +-
.../runners/portability/local_job_service.py | 2 +-
.../runners/portability/local_job_service_test.py | 2 +-
.../runners/portability/portable_stager.py | 14 +-
.../runners/portability/portable_stager_test.py | 11 +-
.../portability/spark_uber_jar_job_server_test.py | 2 +-
49 files changed, 1552 insertions(+), 416 deletions(-)