You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mw...@apache.org on 2020/06/02 10:46:12 UTC
[beam] branch BEAM-9646-cloud-vision updated (16e087a -> c40306a)
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a change to branch BEAM-9646-cloud-vision
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 16e087a Add default parallelism value
discard 0367c61 Change param name from numKeys to desiredRequestParallelism
discard cd7d3de Update README.md
discard 747ed7c Added changes to CHANGES.md
discard ef78421 Add customizable number of keys to the transform
discard 372e385 Set the same key for each element so that grouping is done correctly by existing transforms
discard 23cf44d Fixes after code review
discard 1b1694b Add dependency to build.gradle
discard 16c9bdd Add @Experimental annotation, make parent class package-private
discard 6f67f71 Fix spotbugs errors
discard 6ca6372 [BEAM-9646] Add Google Cloud vision integration transform
add 972c331 Avoid re-encoding value state cells each commit (#11823)
add 0e48498 [BEAM-10067] Minify website assets with --minify flag
add d09ef10 Merge pull request #11801 from epicfaace/minify
add e667344 Add a bare bones powered by page.
add b43b2f1 Merge pull request #11826 from aaltay/rela
add 04c2a2d Reject fractional values outside of (0.0, 1.0)
add e42d217 Merge pull request #11580 from kmjung/bigquery_split
add 4676c43 [BEAM-10106] Script the deployment of artifacts to pypi
add 10732c7 Add pypi script to release guide.
add eae2113 Improve log messages in release scripts.
add b6100fd Install twine in virtual env.
add 409607a Merge pull request #11828 from ibzib/BEAM-10106
add 47ea112 [BEAM-9679] Flatten Kata for Go (#11806)
add 018e889 [BEAM-9220] Go Dataflow jobs to use runner v2 (#11207)
add a5e642a [BEAM-9935] Respect allowed split points and fraction in Go.
add e57c144 Merge pull request #11791: [BEAM-9935] Respect allowed split points and fraction in Go.
add 775ad8e [BEAM-8280] Blog post: Python typing changes (#11070)
add d172594 Enables proto holders for testing runner API. Without this x-lang can be broken for some execution paths where this property get enabled automatically.
add 74962ce Merge pull request #11844 [BEAM-8019] Enables proto holders when 'test_runner_api' is True.
add d68f7f2 Update multi-language roadmap status.
add e7abbdb Update website/www/site/content/en/roadmap/connectors-multi-sdk.md
add 1e7d194 Merge pull request #11853 Update multi-language roadmap status.
add d71ee35 [BEAM-9916] Fixed some IO Urls on website
add 0addd1f Merge pull request #11848: [BEAM-9916] Fixed some IO Urls on website
add 7978c0f [BEAM-10078] uniquify Dataflow specific jars when staging (#11814)
add fc941a0 [BEAM-10107] Remove outdated instructions for website updates in rele… (#11852)
add 0e74474 remove as_cloud_object override
add b09e38e Merge pull request #11839 [BEAM-10122] Python RowCoder throws NotImplementedError in DataflowRunner
add b843bfc [BEAM-9964] Move workerCacheMb to a user-visible place
add 8420eee Merge pull request #11849 from [BEAM-9964] Move workerCacheMb to a user-visible place
add 193e54c [BEAM-9679] Add a CoGroupByKey lesson to the Core Transforms section (#11803)
add 1bcb551 [BEAM-10110] Propagate ids for custom coders. (#11832)
add 9e5dd68 [BEAM-10125] adding cross-language KafkaIO integration test
add 858fe4e Merge pull request #11847: [BEAM-10125] adding cross-language KafkaIO integration test
add 12878fc [BEAM-10121] Python RowCoder doesn't support nested structs (#11841)
add d87fc52 [BEAM-7746] Fix type errors and enable checks for apache_beam.dataframe.* (#11632)
add 21848c6 Moves apply_WriteToBigQuery to override
add b365680 Skip test if py=3.8 see BEAM-9754
add dba5f2b Merge pull request #11594 from [BEAM-9692] Replace apply_WriteToBigQuery with PTransformOverride
add a0297da Update programming-guide.md
add 1bd1508 Merge pull request #11871: [website] Fix programming guide typo
add 6dc08a4 [BEAM-8608] Chain DoFns in Flink batch runner when possible.
add 3689296 [BEAM-8608] Code review. Fix checkstyle.
add 1a8c563 Merge pull request #11858: [BEAM-8608] Chain DoFns in Flink batch runner when possible.
add 1b14085 [BEAM-9421] Better documentation of output results from AnnotateText transform (#11776)
add e9837fe [BEAM-9421] Website section that describes getting predictions using AI Platform Prediciton (#11075)
add a1031c8 [BEAM-10162, BEAM-10163] Change PubSubIO and BigQueryIO job names; update README.md
add 194f981 [BEAM-10051] Scope instruction data. (#11864)
add e0217d1 [BEAM-9679] Add Combine/Simple Function to Core Transforms (#11866)
add 280e243 Updates CHANGES.md to include new I/Os for Beam 2.22.0. (#11879)
add 6598f10 Update programming-guide.md
add 2163ff8 Merge pull request #11872: Fix Programming guide typo
add 05df703 [BEAM-10154] Remove stray version number.
add 3ebc397 Merge pull request #11865 from ibzib/BEAM-10154
add 89fc35b [BEAM-10158] Use a shared unbounded thread pool within Beam Python. (#11867)
add 01c11e7 [BEAM-9825] | Implement Intersect,Union,Except transforms (#11610)
add b6ca2ab [BEAM-9723] Add DLP integration transforms (#11566)
add be5d91a [BEAM-9646] Add Google Cloud vision integration transform
add 0100fdc Fix spotbugs errors
add e0ed48d Add @Experimental annotation, make parent class package-private
add b8f048e Add dependency to build.gradle
add 6aa6815 Fixes after code review
add b108cb3 Set the same key for each element so that grouping is done correctly by existing transforms
add 962aefa Add customizable number of keys to the transform
add 60ee68e Added changes to CHANGES.md
add 5860c85 Update README.md
add 7145d47 Change param name from numKeys to desiredRequestParallelism
add 6038bfc Add default parallelism value
add c40306a Fix code comments
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (16e087a)
\
N -- N -- N refs/heads/BEAM-9646-cloud-vision (c40306a)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.test-infra/jenkins/README.md | 3 +
.../job_PerformanceTests_BigQueryIO_Python.groovy | 8 +-
.../job_PerformanceTests_PubsubIO_Python.groovy | 4 +-
CHANGES.md | 36 +-
.../CoGroupByKey/cmd/main.go} | 37 +-
.../ParDo => CoGroupByKey/CoGroupByKey}/go.mod | 13 +-
.../CoGroupByKey/CoGroupByKey/go.sum | 314 ++++++++++
.../CoGroupByKey/CoGroupByKey/pkg/task/task.go | 52 ++
.../CoGroupByKey/task-info.yaml} | 19 +-
.../CoGroupByKey/task-remote-info.yaml | 2 +
.../CoGroupByKey/CoGroupByKey/task.md | 104 ++++
.../CoGroupByKey/CoGroupByKey/test/task_test.go | 55 ++
.../Core Transforms/CoGroupByKey/lesson-info.yaml | 0
.../CoGroupByKey/lesson-remote-info.yaml | 3 +
.../go.mod => Combine/Simple Function/cmd/main.go} | 36 +-
.../Combine/Simple Function}/go.mod | 16 +-
.../Core Transforms/Combine/Simple Function/go.sum | 310 ++++++++++
.../Combine/Simple Function/pkg/task/task.go} | 17 +-
.../Simple Function/task-info.yaml} | 19 +-
.../Combine/Simple Function/task-remote-info.yaml | 2 +
.../Combine/Simple Function/task.md | 43 ++
.../Simple Function/test/task_test.go} | 37 +-
.../lesson-info.yaml} | 3 +-
.../Combine/lesson-remote-info.yaml | 3 +
.../go.mod => Flatten/Flatten/cmd/main.go} | 38 +-
.../ParDo OneToMany => Flatten/Flatten}/go.mod | 3 +-
.../ParDo OneToMany => Flatten/Flatten}/go.sum | 6 -
.../Flatten/Flatten/pkg/task/task.go} | 13 +-
.../Core Transforms/Flatten/Flatten/task-info.yaml | 31 +
.../Flatten/Flatten/task-remote-info.yaml | 2 +
.../go/Core Transforms/Flatten/Flatten/task.md | 38 ++
.../go.mod => Flatten/Flatten/test/task_test.go} | 40 +-
.../lesson-info.yaml} | 3 +-
.../Flatten/lesson-remote-info.yaml | 3 +
.../go/Core Transforms/Map/ParDo OneToMany/go.mod | 3 +-
.../go/Core Transforms/Map/ParDo OneToMany/go.sum | 5 +-
.../Map/ParDo OneToMany/task-remote-info.yaml | 2 +-
.../go/Core Transforms/Map/ParDo Struct/go.mod | 3 +-
.../go/Core Transforms/Map/ParDo Struct/go.sum | 5 +-
.../Map/ParDo Struct/task-remote-info.yaml | 2 +-
learning/katas/go/Core Transforms/Map/ParDo/go.mod | 3 +-
learning/katas/go/Core Transforms/Map/ParDo/go.sum | 5 +-
.../Map/ParDo/task-remote-info.yaml | 2 +-
.../katas/go/Core Transforms/section-info.yaml | 3 +
.../Introduction/Hello Beam/Hello Beam Test/go.mod | 2 +-
.../Hello Beam Test/task-remote-info.yaml | 2 +-
.../go/Introduction/Hello Beam/Hello Beam/go.mod | 2 +-
.../Hello Beam/Hello Beam/task-remote-info.yaml | 2 +-
learning/katas/go/course-remote-info.yaml | 2 +-
release/src/main/scripts/cut_release_branch.sh | 9 +-
release/src/main/scripts/deploy_pypi.sh | 61 ++
.../beam/runners/flink/FlinkCapabilities.java | 34 ++
.../beam/runners/flink/FlinkCapabilities.java | 34 ++
.../flink/FlinkBatchTransformTranslators.java | 21 +-
.../translation/functions/FlinkDoFnFunction.java | 156 +++--
.../functions/FlinkStatefulDoFnFunction.java | 7 +-
.../options/DataflowPipelineDebugOptions.java | 12 +
.../options/DataflowWorkerHarnessOptions.java | 12 -
.../beam/runners/dataflow/util/PackageUtil.java | 19 +-
.../dataflow/worker/WindmillStateInternals.java | 20 +-
sdks/go/pkg/beam/core/graph/coder/coder.go | 17 +-
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 105 +++-
.../pkg/beam/core/runtime/exec/datasource_test.go | 135 +++-
sdks/go/pkg/beam/core/runtime/exec/plan.go | 6 +-
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 3 +
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 16 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 128 +++-
.../pkg/beam/core/runtime/harness/datamgr_test.go | 73 ++-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 6 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 12 +
.../src/main/resources/beam/suppressions.xml | 1 +
.../java/org/apache/beam/sdk/transforms/Sets.java | 680 +++++++++++++++++++++
.../org/apache/beam/sdk/transforms/SetsTest.java | 324 ++++++++++
sdks/java/extensions/ml/build.gradle | 14 +-
.../beam/sdk/extensions/ml/AnnotateImages.java | 6 +-
.../beam/sdk/extensions/ml/BatchRequestForDLP.java | 113 ++++
.../apache/beam/sdk/extensions/ml/CloudVision.java | 30 +-
.../beam/sdk/extensions/ml/DLPDeidentifyText.java | 282 +++++++++
.../beam/sdk/extensions/ml/DLPInspectText.java | 240 ++++++++
.../beam/sdk/extensions/ml/DLPReidentifyText.java | 286 +++++++++
.../beam/sdk/extensions/ml/MapStringToDlpRow.java | 59 ++
.../sdk/extensions/ml/BatchRequestForDlpTest.java | 63 ++
.../sdk/extensions/ml/DLPDeidentifyTextTest.java | 101 +++
.../beam/sdk/extensions/ml/DLPInspectTextTest.java | 101 +++
.../sdk/extensions/ml/DLPReidentifyTextTest.java | 101 +++
.../sdk/extensions/ml/DLPTextOperationsIT.java | 159 +++++
.../sdk/extensions/ml/MapStringToDlpRowTest.java | 69 +++
.../sql/impl/rel/BeamSetOperatorRelBase.java | 43 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 5 +
sdks/python/apache_beam/coders/row_coder.py | 5 +-
sdks/python/apache_beam/coders/row_coder_test.py | 22 +-
sdks/python/apache_beam/dataframe/convert.py | 41 +-
sdks/python/apache_beam/dataframe/doctests.py | 5 +-
sdks/python/apache_beam/dataframe/frame_base.py | 3 +-
sdks/python/apache_beam/dataframe/transforms.py | 26 +-
.../apache_beam/examples/snippets/snippets.py | 91 +++
.../io/external/xlang_kafkaio_it_test.py | 141 +++++
sdks/python/apache_beam/pipeline.py | 3 +-
.../runners/dataflow/dataflow_runner.py | 40 +-
.../runners/dataflow/dataflow_runner_test.py | 67 ++
.../runners/dataflow/ptransform_overrides.py | 136 +++++
.../runners/portability/artifact_service_test.py | 6 +-
.../runners/portability/expansion_service_test.py | 4 +-
.../runners/portability/fn_api_runner/fn_runner.py | 4 +-
.../portability/fn_api_runner/worker_handlers.py | 14 +-
.../runners/portability/local_job_service.py | 7 +-
.../runners/portability/portable_stager_test.py | 4 +-
.../apache_beam/runners/worker/data_plane_test.py | 4 +-
.../apache_beam/runners/worker/log_handler_test.py | 4 +-
.../apache_beam/runners/worker/sdk_worker.py | 9 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 4 +-
.../apache_beam/runners/worker/worker_pool_main.py | 5 +-
.../runners/worker/worker_status_test.py | 4 +-
.../apache_beam/utils/thread_pool_executor.py | 13 +
.../apache_beam/utils/thread_pool_executor_test.py | 15 +
sdks/python/mypy.ini | 3 -
website/www/package.json | 2 +-
website/www/site/content/en/blog/python-typing.md | 136 +++++
.../www/site/content/en/community/powered-by.md | 25 +
.../site/content/en/contribute/release-guide.md | 46 +-
.../content/en/documentation/dsls/sql/overview.md | 2 +-
.../en/documentation/patterns/ai-platform.md | 216 +++++++
.../content/en/documentation/patterns/overview.md | 4 +
.../content/en/documentation/programming-guide.md | 12 +-
.../content/en/roadmap/connectors-multi-sdk.md | 19 +-
website/www/site/data/authors.yml | 3 +
website/www/site/data/io_matrix.yaml | 10 +-
.../partials/section-menu/en/community.html | 1 +
.../partials/section-menu/en/documentation.html | 27 +-
129 files changed, 5625 insertions(+), 477 deletions(-)
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany/go.mod => CoGroupByKey/CoGroupByKey/cmd/main.go} (54%)
copy learning/katas/go/Core Transforms/{Map/ParDo => CoGroupByKey/CoGroupByKey}/go.mod (77%)
create mode 100644 learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/go.sum
create mode 100644 learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/pkg/task/task.go
copy learning/katas/go/Core Transforms/{section-info.yaml => CoGroupByKey/CoGroupByKey/task-info.yaml} (73%)
create mode 100644 learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
create mode 100644 learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/task.md
create mode 100644 learning/katas/go/Core Transforms/CoGroupByKey/CoGroupByKey/test/task_test.go
copy learning/katas/{python => go}/Core Transforms/CoGroupByKey/lesson-info.yaml (100%)
create mode 100644 learning/katas/go/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany/go.mod => Combine/Simple Function/cmd/main.go} (58%)
copy learning/katas/go/{Introduction/Hello Beam/Hello Beam => Core Transforms/Combine/Simple Function}/go.mod (72%)
create mode 100644 learning/katas/go/Core Transforms/Combine/Simple Function/go.sum
copy learning/katas/go/{Introduction/Hello Beam/Hello Beam Test/go.mod => Core Transforms/Combine/Simple Function/pkg/task/task.go} (78%)
copy learning/katas/go/Core Transforms/{section-info.yaml => Combine/Simple Function/task-info.yaml} (74%)
create mode 100644 learning/katas/go/Core Transforms/Combine/Simple Function/task-remote-info.yaml
create mode 100644 learning/katas/go/Core Transforms/Combine/Simple Function/task.md
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany/go.mod => Combine/Simple Function/test/task_test.go} (56%)
copy learning/katas/go/Core Transforms/{section-info.yaml => Combine/lesson-info.yaml} (97%)
create mode 100644 learning/katas/go/Core Transforms/Combine/lesson-remote-info.yaml
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany/go.mod => Flatten/Flatten/cmd/main.go} (56%)
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany => Flatten/Flatten}/go.mod (95%)
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany => Flatten/Flatten}/go.sum (95%)
copy learning/katas/go/{Introduction/Hello Beam/Hello Beam Test/go.mod => Core Transforms/Flatten/Flatten/pkg/task/task.go} (78%)
create mode 100644 learning/katas/go/Core Transforms/Flatten/Flatten/task-info.yaml
create mode 100644 learning/katas/go/Core Transforms/Flatten/Flatten/task-remote-info.yaml
create mode 100644 learning/katas/go/Core Transforms/Flatten/Flatten/task.md
copy learning/katas/go/Core Transforms/{Map/ParDo OneToMany/go.mod => Flatten/Flatten/test/task_test.go} (51%)
copy learning/katas/go/Core Transforms/{section-info.yaml => Flatten/lesson-info.yaml} (97%)
create mode 100644 learning/katas/go/Core Transforms/Flatten/lesson-remote-info.yaml
create mode 100755 release/src/main/scripts/deploy_pypi.sh
create mode 100644 runners/flink/1.10/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java
create mode 100644 runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/FlinkCapabilities.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/Sets.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/SetsTest.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/BatchRequestForDLP.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPDeidentifyText.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPInspectText.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/DLPReidentifyText.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/MapStringToDlpRow.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/BatchRequestForDlpTest.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/DLPDeidentifyTextTest.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/DLPInspectTextTest.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/DLPReidentifyTextTest.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/DLPTextOperationsIT.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/MapStringToDlpRowTest.java
create mode 100644 sdks/python/apache_beam/io/external/xlang_kafkaio_it_test.py
create mode 100644 website/www/site/content/en/blog/python-typing.md
create mode 100644 website/www/site/content/en/community/powered-by.md
create mode 100644 website/www/site/content/en/documentation/patterns/ai-platform.md