You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by re...@apache.org on 2018/04/11 15:36:39 UTC
[beam] branch master updated (262932d -> c78b96a)
This is an automated email from the ASF dual-hosted git repository.
reuvenlax pushed a change to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 262932d [BEAM-4035] Fix spark validatesrunner
add bfa8136 Update python version.
add 0a4774d Update container version.
add cfbacaa Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO.
add 5601f0a This closes #4002: Cherrypick of #3995 to 2.2.0: Add custom tempLocation support to BigQueryIO
add e7a8620 [BEAM-1017] Add RedisIO
add ba9d1f9 Sets user agent in BigTableIO.Read.getBigTableService().
add 09431da This closes #3998
add 1b10b91 Use 2.2.0 container tag for Dataflow
add 582ddef This closes #4006: Use 2.2.0 container tag for Dataflow
add c93ee2c Pin runner harness container image for Dataflow distributions of BEAM Python SDK
add 2be277d Make linter happy.
add e5e10b6 [BEAM-2600] Add minimal python SDK harness container
add 423e470 Use 2.2.0 version instead of 2.3.0 for container Maven project on the release branch. This reverts eeb4d73.
add 7ff94e6 Use tox instead of detox. Looks like we reverted to tox on the master branch.
add 961fa46 Pin runner harness also for official BEAM releases.
add c2cb1dd This closes #3970
add 2c5f85a [maven-release-plugin] prepare release v-RC1-DupdateWorkingCopyVersions=false
add b6d15dc [maven-release-plugin] prepare for next development iteration
add b6a3528 Revert "[maven-release-plugin] prepare for next development iteration"
add e47b477 Revert "[maven-release-plugin] prepare release v-RC1-DupdateWorkingCopyVersions=false"
add 9b709e7 [maven-release-plugin] prepare release v2.2.0-RC1
add 576d22a [maven-release-plugin] rollback changes from release preparation of v2.2.0-RC1
add e474a22 Add an element batching transform.
add c2e0306 This closes #4054
add a375b2e Align names with those produced by the dataflow runner harness.
add 9b063eb Fix from any -> bytes transition.
add 6db1db7 This closes #4055
add 48ae7d1 Add an option for dataflow job labels.
add 6f07e3f Unit test for label pipeline option
add c6b6668 This closes #3993
add 3718680 Update container version for python DataflowRunner for the 2.2.0 release
add acd4e1d This closes #4059
add d84da51 [maven-release-plugin] prepare release v2.2.0-RC2
add 2d254b1 [maven-release-plugin] rollback changes from release preparation of v2.2.0-RC2
add b75fe35 [maven-release-plugin] prepare release v2.2.0-RC3
add 36ae989 [maven-release-plugin] rollback changes from release preparation of v2.2.0-RC3
add 22dcd73 Set save_main_session to True for csv module
add bddb869 Set OnTimeBehavior in windowing strategy to FIRE_ALWAYS.
add 13dcec9 Update urns.py
add d28ee9b Cherrypicks for 2 2 0 rc 4
add 381e86c CP #4112, #4122: Properly handle side input exception when all reader threads complete
add 8da5c1b This closes #4114
add 55a1124 Cherrypick WriteFiles fix.
add 6c93105 [maven-release-plugin] prepare release v2.2.0-RC4
add fd85372 [maven-release-plugin] rollback changes from release preparation of v2.2.0-RC4
add d12b903 initial commit for BEAM-2871. WIP Todo: Add in the unit tests to work with build env Remove the FinanceExample protoc created file and let build do it Wait for resolution of BEAM-2868 to remove GCS dependency
add 1d897e6 [maven-release-plugin] prepare for next development iteration
add 1d81b36 Bump Python version for next release.
add a96971e Only fail nightly release at end, for more detailed signal
add 7d4beed Only fail Java postcommit at end
add a4bc3fe Move the Job Server into a Submodule
add 54e7fbd Add Artifact Staging Endpoint to PrepareJobResponse
add 4cd4978 Add model/ and submodules to Maven structure and gen_protos.py
add 4fa28a1 Move sdk/common/runner-api protos into model/pipeline and model/job-management
add e61c0a3 Move sdks/common/fn-api protos to model/fn-execution
add 8db5a59 Remove sdks/common
add e3b65bc Re-namespace model/pipeline to model.pipeline
add b89d577 Re-namespace model/job-management to model.job_management
add cbd2410 Re-namespace model/fn-execution to model.fn_execution
add 45798ca Improve error message when standard_coders.yaml not found
add 93fb657 Dataflow runner: shade protobuf and beam model
add b4a9391 PipelineOptionsFactory should prevent non PipelineOptions interfaces from being constructed
add 1ac8e1a [BEAM-3016] Fix blocking issue within run() when channel terminates while blocking within DirectStreamObserver.
add 056e1d4 [BEAM-2576] Reflect portability proto changes in Go
add 466a474 Add licenses to generated code
add f6606e8 Improve the documentation for CombineFn
add cd75629 Remove coders from DoFn's Remove extra space in hello.cc file Remove generated proto class Move construct for SubProcessConfiguration.java to the PipelineOptions
add dc97ab1 initial commit for BEAM-2871. WIP Todo: Add in the unit tests to work with build env Remove the FinanceExample protoc created file and let build do it Wait for resolution of BEAM-2868 to remove GCS dependency
add 9978d1c Merge branch 'beam-2871' of https://github.com/rezarokni/beam into beam-2871
add 96cc47b Merge branch 'master' of https://github.com/apache/beam into beam-2871
add 313ec71 Merge branch 'beam-2871' of https://source.developers.google.com/p/bigdatapivot/r/timeseries into beam-2871
add 6bc90ea Fix Bug if more than one external lib DoFn is used in same pipeline the file would never be downloaded
add 36a573f Removed more complex examples for later PR Added semaphore logic to allow for different bin's to have different conc
add 90ce578 Added Assert for test pipeline. Added comments and new file names for main pipeline.
add 6b495d7 Added ability to read Binary output from files for use with Protobuf
add bec2b18 Moved files from java8 to java
add 8f5b4bd Merge branch 'master' into beam-2871
add 39fbf03 Correct checkstyle issues
add 176fe7d Fix findBugs issues
add 026748b Merge branch 'master' of https://github.com/apache/beam into beam-2871
new c78b96a Merge pull request #3961: [BEAM-2871] Add advanced examples of running external libraries on workers
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.../org/apache/beam/examples/subprocess/Echo.cc | 30 +-
.../apache/beam/examples/subprocess/EchoAgain.cc | 30 +-
.../examples/subprocess/ExampleEchoPipeline.java | 135 +++++++++
.../subprocess/SubProcessPipelineOptions.java | 97 +++++++
.../configuration/SubProcessConfiguration.java | 83 ++++++
.../kernel/SubProcessCommandLineArgs.java | 76 +++++
.../subprocess/kernel/SubProcessIOFiles.java | 131 +++++++++
.../subprocess/kernel/SubProcessKernel.java | 311 +++++++++++++++++++++
.../subprocess/utils/CallingSubProcessUtils.java | 110 ++++++++
.../examples/subprocess/utils/ExecutableFile.java | 77 +++++
.../beam/examples/subprocess/utils/FileUtils.java | 164 +++++++++++
.../subprocess/ExampleEchoPipelineTest.java | 203 ++++++++++++++
12 files changed, 1419 insertions(+), 28 deletions(-)
copy sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/CloseableFnDataReceiver.java => examples/java/src/main/java/org/apache/beam/examples/subprocess/Echo.cc (66%)
copy sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/CloseableFnDataReceiver.java => examples/java/src/main/java/org/apache/beam/examples/subprocess/EchoAgain.cc (62%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/ExampleEchoPipeline.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/SubProcessPipelineOptions.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/configuration/SubProcessConfiguration.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessCommandLineArgs.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessIOFiles.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/kernel/SubProcessKernel.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/CallingSubProcessUtils.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/ExecutableFile.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/subprocess/utils/FileUtils.java
create mode 100644 examples/java/src/test/java/org/apache/beam/examples/subprocess/ExampleEchoPipelineTest.java
--
To stop receiving notification emails like this one, please contact
reuvenlax@apache.org.
[beam] 01/01: Merge pull request #3961: [BEAM-2871] Add advanced
examples of running external libraries on workers
Posted by re...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
reuvenlax pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/beam.git
commit c78b96a67c0313f44d78262ee46f86aa4866a3aa
Merge: 262932d 026748b
Author: reuvenlax <re...@google.com>
AuthorDate: Wed Apr 11 16:36:27 2018 +0100
Merge pull request #3961: [BEAM-2871] Add advanced examples of running external libraries on workers
.../org/apache/beam/examples/subprocess/Echo.cc | 36 +++
.../apache/beam/examples/subprocess/EchoAgain.cc | 36 +++
.../examples/subprocess/ExampleEchoPipeline.java | 135 +++++++++
.../subprocess/SubProcessPipelineOptions.java | 97 +++++++
.../configuration/SubProcessConfiguration.java | 83 ++++++
.../kernel/SubProcessCommandLineArgs.java | 76 +++++
.../subprocess/kernel/SubProcessIOFiles.java | 131 +++++++++
.../subprocess/kernel/SubProcessKernel.java | 311 +++++++++++++++++++++
.../subprocess/utils/CallingSubProcessUtils.java | 110 ++++++++
.../examples/subprocess/utils/ExecutableFile.java | 77 +++++
.../beam/examples/subprocess/utils/FileUtils.java | 164 +++++++++++
.../subprocess/ExampleEchoPipelineTest.java | 203 ++++++++++++++
12 files changed, 1459 insertions(+)
--
To stop receiving notification emails like this one, please contact
reuvenlax@apache.org.