You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/04/08 05:01:17 UTC
[beam] branch nightly-refs/heads/master updated (e596abf7660 -> e9afde092c1)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from e596abf7660 Merge pull request #17262: [BEAM-14244] Use the supplied output timestamp for processing time timers rather than the input watermark
add 9578d82e5f9 [BEAM-13015] Lookup the container for the step once when registering instead of on each execution. (#17281)
add af006afe9c0 [BEAM-14175] Log read loop abort at debug rather than error (#17183)
add dce3fabf1fa [BEAM-11745] Fix author list rendering (#17308)
add 8c1fdaaaf76 [BEAM-14144] Record JFR profiles when GC thrashing is detected (#17151)
add 416fc9ba752 [BEAM-14104] Support shard aware aggregation in Kinesis writer.
add ad4561ea5d8 Merge pull request #17113 from mosche/BEAM-14104-ShardAwareAggregation
add a88b2a109e1 Factors enable_prime flag in when checking use_unified_worker conditions (#17271)
add 9e4c288627c [BEAM-11104] Add ProcessContinuation type to Go SDK (#17265)
add e44d8a72d4a BEAM-13939: Restructure Protos to fix namespace conflicts (#16961)
add 81d3de2007f [BEAM-14270] Mark {Snowflake/BigQuery}Services as @Internal (#17309)
add 26f6d86b556 [BEAM-13901] Add unit tests for graphx/cogbk.go
add f6cfa2739f4 [BEAM-13992] [Website] update Contribute/Code Contribution Guide page
add 468994f37ed [BEAM-13992] [Website] change text, transfer tag a
add 3951a9941c1 [BEAM-13992] [Website] change code tags
add 9642881a6d7 [BEAM-13992] [Website] change text
add 3137b6ef960 [BEAM-13992] [Website] change text and links, add empty lines
add 7a4df7de59e Merge branch 'master' into BEAM-13992-update-contribution-guide-page
add cf9ae549e03 [BEAM-13992] [Website] change links, change text
add f82be4394c9 [BEAM-13992] [Website] change links, add text, add dots
add da76ae13d10 [BEAM-13992] [Website] change links, change text
add dc824115352 [BEAM-13992] [Website] change text, delete whitespace
add 33cc5cda880 Merge branch 'master' into BEAM-13992-update-contribution-guide-page
add a63c3973647 [BEAM-13992] [Website] update text
add acc7dc6bbb3 Merge branch 'master' into BEAM-13992-update-contribution-guide-page
add 24c20ff8709 [BEAM-13991] [Website] change links, add contribute file
add 882b9afc580 [BEAM-13991] [Website] add content, add styles
add 661d34a21d8 Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page
add 6d1dab25a3d [BEAM-13991] [Website] add images, add styles, delete spaces
add 1f2f1c17f93 [BEAM-13991] [Website] change url and aliases, delete bullet points
add 30af0b9aff6 [BEAM-13991] [Website] add empty line
add b77d513845c Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page
add a6143af4cd5 [BEAM-13991] [Website] change styles, change quotes
add e8a155f07a4 [BEAM-13991] [Website] change link color
add 8ec18a2403a Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page
add 7788ba26832 Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page
add 3f0540ce075 [BEAM-13991] [Website] change text
add 63ee894027c Merge remote-tracking branch 'origin/BEAM-13991-update-contribute-to-beam-page' into BEAM-13991-update-contribute-to-beam-page
add 1978c9040b7 [BEAM-13991] [Website] added changes from PR 13992, changed get-starting contributing
add d27119c63c0 Merge branch 'master' into BEAM-13991-update-contribute-to-beam-page
add 0035f37e4e0 Merge pull request #17136 from akvelon/BEAM-13991-update-contribute-to-beam-page
add f9e317cebec Sets 'sdk_harness_container_images' property for all Dataflow jobs - Python SDK
add 238d7b99b9e Merge pull request #16974: [BEAM-14009] Sets 'sdk_harness_container_images' property for all Dataflow jobs - Python SDK
add 5343f1a606d Sets 'sdk_harness_container_images' property for all Dataflow jobs - Java SDK
add 25d14a7f12a Merge pull request #16981: [BEAM-14009] Sets 'sdk_harness_container_images' property for all Dataflow jobs - Java SDK
add 443924a76d0 [BEAM-14259, BEAM-14266] Remove unused function, replace use of ptypes (#17301)
add 397b1809957 [BEAM-14274] Fix staticcheck warnings in pipelinex (#17311)
add f42d4375e18 [BEAM-13857] Switched Go IT script to using Go flags for expansion services (#17161)
add b4b0025acd6 [BEAM-10529] add java and generic components of nullable xlang tests
add 798fd532832 [BEAM-10529] fix test case
add 87435496681 [BEAM-10529] add coders and typehints to support nullable xlang coders
add 1ce66059a97 [BEAM-10529] update external builder to support nullable coder
add 04868d6a76a Merge remote-tracking branch 'origin/master' into feature/BEAM-10529-nullable-Xlang-Coder
add a6a38d0ccc9 [BEAM-10529] clean up coders.py
add e3e02235a5a [BEAM-10529] add coder translation test
add 30f06d69dfd [BEAM-10529] add additional check to typecoder to not accidentally misidentify coders as nullable
add 24dd8767553 [BEAM-10529] add test to retrieve nullable coder from typehint
add 3e93b56c886 [BEAM-10529] improve comments, clean up python
add e642e7c437a [BEAM-10529] run spotless
add f55074c7aed Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlang-Coder' into feature/BEAM-10529-nullable-Xlang-Coder
add 4dd1c548139 [BEAM-10529] add go nullable coder
add 43c84e62b5b [BEAM-10529] cleanup extra println
add 48b6f34fe70 Merge remote-tracking branch 'origin/feature/BEAM-10529-nullable-Xlang-Coder' into feature/BEAM-10529-nullable-Xlang-Coder
add 5be0e94b5e5 Merge remote-tracking branch 'origin/master' into feature/BEAM-10529-enable-nullable-for-kafka
add e9afde092c1 Merge pull request #17268: [BEAM-10529] enable nullable for kafkaIO xlang
No new revisions were added by this update.
Summary of changes:
.gitignore | 3 +-
build.gradle.kts | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 7 +-
.../beam/model/fn_execution/v1}/beam_fn_api.proto | 6 +-
.../fn_execution/v1}/beam_provision_api.proto | 4 +-
.../interactive/v1}/beam_interactive_api.proto | 2 +-
.../job_management/v1}/beam_artifact_api.proto | 2 +-
.../job_management/v1}/beam_expansion_api.proto | 2 +-
.../model/job_management/v1}/beam_job_api.proto | 6 +-
.../beam/model/pipeline/v1}/beam_runner_api.proto | 2 +-
.../apache/beam/model/pipeline/v1}/endpoints.proto | 0
.../model/pipeline/v1}/external_transforms.proto | 4 +-
.../apache/beam/model/pipeline/v1}/metrics.proto | 2 +-
.../apache/beam/model/pipeline/v1}/schema.proto | 0
.../model/pipeline/v1}/standard_window_fns.proto | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 39 +-
.../options/DataflowPipelineDebugOptions.java | 19 +
.../beam/runners/dataflow/DataflowRunnerTest.java | 19 +-
.../dataflow/worker/WorkItemStatusClient.java | 14 +-
.../dataflow/worker/status/JfrzServlet.java | 69 +
.../dataflow/worker/status/WorkerStatusPages.java | 6 +-
.../runners/dataflow/worker/util/JfrInterop.java | 84 +
.../dataflow/worker/util/MemoryMonitor.java | 168 +-
.../worker/util/common/worker/ReadOperation.java | 10 +-
.../dataflow/worker/util/MemoryMonitorTest.java | 84 +-
sdks/go/pkg/beam/core/funcx/fn.go | 32 +-
sdks/go/pkg/beam/core/funcx/fn_test.go | 7 +
sdks/go/pkg/beam/core/runtime/exec/translate.go | 57 +-
sdks/go/pkg/beam/core/runtime/graphx/cogbk_test.go | 125 +
sdks/go/pkg/beam/core/runtime/graphx/v1/gen.go | 2 +-
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 664 ++--
sdks/go/pkg/beam/core/runtime/harness/gen.go | 2 +-
.../go/pkg/beam/core/runtime/harness/session.proto | 2 +-
.../core/runtime/harness/session/session.pb.go | 245 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 5 +-
sdks/go/pkg/beam/core/runtime/pipelinex/util.go | 2 +-
.../pkg/beam/core/runtime/pipelinex/util_test.go | 15 -
sdks/go/pkg/beam/core/sdf/continuation.go | 62 +
.../graphx/v1/gen.go => sdf/continuation_test.go} | 25 +-
sdks/go/pkg/beam/model/PROTOBUF.md | 10 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 2540 +++++++------
.../model/fnexecution_v1/beam_fn_api_grpc.pb.go | 17 +-
.../model/fnexecution_v1/beam_provision_api.pb.go | 258 +-
.../fnexecution_v1/beam_provision_api_grpc.pb.go | 7 +-
sdks/go/pkg/beam/model/gen.go | 4 +-
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 665 ++--
.../jobmanagement_v1/beam_artifact_api_grpc.pb.go | 13 +-
.../jobmanagement_v1/beam_expansion_api.pb.go | 167 +-
.../jobmanagement_v1/beam_expansion_api_grpc.pb.go | 7 +-
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 841 ++---
.../model/jobmanagement_v1/beam_job_api_grpc.pb.go | 7 +-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 3935 +++++++++++---------
.../model/pipeline_v1/beam_runner_api_grpc.pb.go | 7 +-
sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go | 116 +-
.../model/pipeline_v1/external_transforms.pb.go | 467 ++-
sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 1049 +++---
sdks/go/pkg/beam/model/pipeline_v1/schema.pb.go | 632 ++--
.../model/pipeline_v1/standard_window_fns.pb.go | 224 +-
sdks/go/pkg/beam/transforms/stats/util_gen.go | 1 +
sdks/go/pkg/beam/transforms/stats/util_gen.tmpl | 2 +-
sdks/go/scripts/genproto.sh | 159 +
sdks/go/test/build.gradle | 57 +-
sdks/go/test/integration/expansions.go | 7 +-
sdks/go/test/integration/flags.go | 16 -
.../integration/io/xlang/debezium/debezium_test.go | 23 +-
.../go/test/integration/io/xlang/jdbc/jdbc_test.go | 29 +-
.../test/integration/io/xlang/kafka/kafka_test.go | 26 +-
sdks/go/test/integration/xlang/xlang_test.go | 38 +-
.../harness/data/PTransformFunctionRegistry.java | 2 +-
.../apache/beam/sdk/io/aws2/common/ClientPool.java | 123 -
.../apache/beam/sdk/io/aws2/common/ObjectPool.java | 151 +
.../sdk/io/aws2/common/RetryConfiguration.java | 2 +-
.../apache/beam/sdk/io/aws2/kinesis/KinesisIO.java | 270 +-
.../sdk/io/aws2/kinesis/KinesisPartitioner.java | 28 +-
.../sdk/io/aws2/kinesis/RecordsAggregator.java | 4 -
.../{ClientPoolTest.java => ObjectPoolTest.java} | 99 +-
.../sdk/io/aws2/kinesis/KinesisIOWriteTest.java | 243 +-
.../sdk/io/aws2/kinesis/PutRecordsHelpers.java | 8 +
.../sdk/io/aws2/kinesis/testing/KinesisIOIT.java | 5 +-
.../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 2 +
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 17 +-
.../io/snowflake/services/SnowflakeServices.java | 2 +
sdks/python/.yapfignore | 2 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 4 +-
.../python/apache_beam/portability/api/__init__.py | 21 -
sdks/python/apache_beam/portability/common_urns.py | 43 +-
.../runners/dataflow/internal/apiclient.py | 60 +-
.../runners/dataflow/internal/apiclient_test.py | 140 +-
.../runners/interactive/caching/streaming_cache.py | 20 +-
.../interactive/caching/streaming_cache_test.py | 246 +-
.../interactive/options/capture_control_test.py | 16 +-
.../interactive/options/capture_limiters.py | 11 +-
.../interactive/options/capture_limiters_test.py | 6 +-
.../runners/interactive/recording_manager.py | 4 +-
.../runners/interactive/recording_manager_test.py | 5 +-
.../interactive/testing/test_cache_manager.py | 31 +-
.../apache_beam/runners/interactive/utils.py | 8 +-
.../apache_beam/runners/interactive/utils_test.py | 8 +-
.../runners/portability/expansion_service_test.py | 4 +-
.../runners/portability/fn_api_runner/execution.py | 2 +-
.../portability/fn_api_runner/worker_handlers.py | 2 +-
sdks/python/apache_beam/testing/test_stream.py | 34 +-
.../apache_beam/testing/test_stream_service.py | 4 +-
.../testing/test_stream_service_test.py | 14 +-
.../python/apache_beam/testing/test_stream_test.py | 161 +-
sdks/python/apache_beam/transforms/external.py | 10 +-
.../python/apache_beam/transforms/external_test.py | 34 +-
.../transforms/fully_qualified_named_transform.py | 7 +-
sdks/python/gen_protos.py | 506 ++-
sdks/python/mypy.ini | 2 +-
sdks/python/scripts/run_pylint.sh | 6 +-
sdks/python/scripts/run_pylint_2to3.sh | 6 +-
sdks/python/setup.py | 41 +-
website/www/site/assets/scss/_case_study.scss | 2 +-
website/www/site/assets/scss/_contribute.sass | 41 +
website/www/site/assets/scss/_traits.scss | 2 +-
website/www/site/assets/scss/main.scss | 1 +
website/www/site/content/en/contribute/_index.md | 578 ++-
.../en/contribute/get-started-contributing.md | 390 ++
website/www/site/layouts/blog/single.html | 6 +-
.../partials/section-menu/en/contribute.html | 2 +-
.../talat_uyarer.png => talat_uyarer_photo.png} | Bin
122 files changed, 9761 insertions(+), 6800 deletions(-)
rename model/fn-execution/src/main/proto/{ => org/apache/beam/model/fn_execution/v1}/beam_fn_api.proto (99%)
rename model/fn-execution/src/main/proto/{ => org/apache/beam/model/fn_execution/v1}/beam_provision_api.proto (96%)
rename model/interactive/src/main/proto/{ => org/apache/beam/model/interactive/v1}/beam_interactive_api.proto (96%)
rename model/job-management/src/main/proto/{ => org/apache/beam/model/job_management/v1}/beam_artifact_api.proto (99%)
rename model/job-management/src/main/proto/{ => org/apache/beam/model/job_management/v1}/beam_expansion_api.proto (97%)
rename model/job-management/src/main/proto/{ => org/apache/beam/model/job_management/v1}/beam_job_api.proto (98%)
rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/beam_runner_api.proto (99%)
rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/endpoints.proto (100%)
rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/external_transforms.proto (97%)
rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/metrics.proto (99%)
rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/schema.proto (100%)
rename model/pipeline/src/main/proto/{ => org/apache/beam/model/pipeline/v1}/standard_window_fns.proto (98%)
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/JfrzServlet.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/JfrInterop.java
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/cogbk_test.go
create mode 100644 sdks/go/pkg/beam/core/sdf/continuation.go
copy sdks/go/pkg/beam/core/{runtime/graphx/v1/gen.go => sdf/continuation_test.go} (62%)
create mode 100755 sdks/go/scripts/genproto.sh
delete mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ClientPool.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/common/ObjectPool.java
rename sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/common/{ClientPoolTest.java => ObjectPoolTest.java} (63%)
delete mode 100644 sdks/python/apache_beam/portability/api/__init__.py
create mode 100644 website/www/site/assets/scss/_contribute.sass
create mode 100644 website/www/site/content/en/contribute/get-started-contributing.md
copy website/www/site/static/images/{case-study/paloalto/talat_uyarer.png => talat_uyarer_photo.png} (100%)