You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/12/10 04:28:44 UTC
[beam] branch nightly-refs/heads/master updated (ad92087babb -> 359a6415848)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git
from ad92087babb Update dataflow container version for Pandas upgrade (#24532)
add ff60ac3147a [Website] center the main content #24521
add e52039c5ada Merge pull request #24427: [Website] center the main content of about page
add 895eb19ad78 [Website] update copy icon positioning #24426
add cceac3b55d9 Merge pull request #24399: [Website] Update copy icon positioning
add b4d6b6652bf [Spark Dataset runner] Broadcast pipeline options
add 4fd73625660 Merge pull request #24523: [Spark Dataset runner] Broadcast pipeline options
add 89089cecbb4 Update google-cloud-bigquery-storage requirement from <2.14,>=2.6.3 to >=2.6.3,<2.17 in /sdks/python (#24539)
add 2aa55911283 Bump cloud.google.com/go/pubsub from 1.27.1 to 1.28.0 in /sdks (#24534)
add f27b6697a56 Bump golang.org/x/net from 0.2.0 to 0.3.0 in /sdks (#24544)
add 3f0b5fa95e8 Precommit python version update (#24526)
add ac29569ae1a [CdapIO] Add CdapIO and SparkReceiverIO documentation in website (#23338)
add 953f895e377 Adding support for Pubsub Lite Writes in SchemaTransforms (#24359)
add 8ec0568d12d Disallow using the JRH with Python streaming pipelines (#24513)
add 40aca90f8d0 fix null pointer exception caused by clearing member variable
add ed29ceb90fa Merge pull request #24546: Fix null pointer exception caused by clearing member variable
add 19514938176 Add RunInference example for TensorFlow Hub pre-trained model (#24529)
add 457bf397ca9 update(PULL Request template) remove Choose reviewer (#24540)
add edaa553bb17 Revert "Bump actions/setup-java from 3.6.0 to 3.7.0 (#24484)" (#24551)
add 03512910dbb Interface{}->any for more subfolders (#24553)
add 49c044b90ef Support multiple gradle tasks in one precommit job
add c48a4e7b5c7 Split up some IOs from Java PreCommit
add 1ba7821a09c Merge pull request #24554: Split some IO precommits out of the Java precommit
add da100f967a6 Migrate testing subpackages from interface{} to any (#24570)
add 6c24637b451 fix go lints (#24566)
add 11ed0e1857f Samza runner support for non unique stateId across multiple ParDos (#24276)
add e0c494d9b57 Suppress keyfor warnings
add a22e204fd32 Suppress checker warnings that are confusing and difficult
add efa8a08549a Add @Pure annotations to MongoDbIO autovalue fields
add 186c37fe121 Suppress checker in FnApiDoFnRunner due to crash
add 3ae4efe3b64 Suppress checker framework in Dataflow FetchAndFilterStreamingSideInputOperation due to crash
add f32c50d35eb Fix some nullness errors in Spark runner
add eb39a1b41fe Upgrade checker framework to 3.27.0
add de16b05f361 Merge pull request #24410: Upgrade checker framework to 3.27.0
add fb06ef3abc8 Bump to Hadoop 3.3.4 for performance tests (#24550)
add 4c0253d8009 pubsub: fix typo in grpc client factory
add 46055564fcc Merge pull request #24398 from Illedran/patch-1
add e233ef4d2a6 regenerate python dependencies (#24582)
add 80980b8be48 Return empty splits if unable to split, not errors (#24508)
add a40e79b16f7 [Tour Of Beam] Playground Router GRPC API host (#24542)
add d1a29704529 Bump golang.org/x/net from 0.3.0 to 0.4.0 in /sdks (#24587)
add dd7a581188e Replaced finalize with DoFn Teardown in Neo4jIO (#24571)
add da3af55f1e3 Simplify bug report templates
add 73e99b664ec Merge pull request #24585: Simplify issue report templates
add 495e1e0a309 Fix bugs in issue template yml
add 0ddf1f7ea8a Merge pull request #24595: Fix bugs in issue template yml
add 7f5ea613012 Fix issue templates (#24597)
add c0de849c1ce [#24024] Stop wrapping light weight functions with Contextful as they add a lot of overhead for functions that are meant to do almost no work. (#24025)
add 27c355c534b Sample window size as well (#24388)
add 00bf2d42eac Implement Kafka Write Schema Transform (#24495)
add 25365bde72f Exclude IOs already split from Java Precommit job
add f7c9f85f365 Move expansion services into appropriate precommits
add cd06ee187c5 Split more IOs out of Java precommit
add b4c33ca2dcb Fix trigger paths for separated IOs
add e8814846f5b Merge pull request #24584: Split java precommit more
add 90d280b43e6 Eliminate null errors from JdbcIO
add 451378851ab Merge pull request #22862: Eliminate null errors from JdbcIO
add b8a335be8ab docs(fix): Filter.whereFieldName(s?) -> Filter.create().whereFieldName(s?) (#24576)
add d523a536a08 ElasticsearchIO: Lower log level in flushBatch to avoid noisy log
add ba38e41c702 Merge pull request #24604 from andthezhang/master
add a9cba697e05 Turn rawtype checking back on for core Java SDK
add 58b4d46655d Merge pull request #24574: Turn rawtype checking back on for core Java SDK
add 884673b4fb6 interface{} -> any for registration (#24600)
add f57c7b94a7a Restrict tox to be in 3.x version (#24601)
add 816f3515c4a [Playground] support for Kafka-enabled examples (#24459)
add 4baee1aa9f3 Fix some small notebook typos (#24616)
add c8c59e91e5d Fixing a breakage of multi-lang auto Runner v2 enabling
add 8f7e9a18f6f Merge pull request #24579: Fixing a breakage of multi-lang auto Runner v2 enabling
add 1f0a53ba040 initialize and increment metrics properly (#24592)
add 70fd788e2b0 Add schema conversion support from Kafka Connect Record schemas to Beam schemas (#24605)
add 0f79b419f8f interface{} -> any for starcgen (#24618)
add aa1d7126499 interface{} -> any for remaining references (#24625)
add 691337e9910 Updating issue-tagger Workflow (#171) (#23143)
add 9689995f21f [GitHub Actions] - Updates in Build Playground Backend to runs-on Self-hosted runners (#23134)
add 9cdea7e39b4 [GitHub Actions] - Updates in Build Playground Frontend to runs-on Self-hosted runners (#23135)
add 43ca241e4d6 [GitHub Actions] - Updates in Go Tests to runs-on Self-hosted runners (#23140)
add 71d09d9ec5b [GitHub Actions] - Updates in Java Tests to runs-on Self-hosted runners (#23160)
add fca93a5a480 Updated label_prs workflow (#173) (#23145)
add 6db3030481e [CdapIO] CdapIO and SparkReceiverIO updates (#24436)
add 61e2ec1811e Revert "[GitHub Actions] - Updates in Java Tests to runs-on Self-hosted runners (#23160)" (#24628)
add 359a6415848 Disallow sliding windows with combiner fanout to prevent data loss (#24606)
No new revisions were added by this update.
Summary of changes:
.github/ISSUE_TEMPLATE/bug.yml | 137 +-
.github/ISSUE_TEMPLATE/failing_test.yml | 136 +-
.github/ISSUE_TEMPLATE/feature.yml | 134 +-
.github/ISSUE_TEMPLATE/task.yml | 135 +-
.github/PULL_REQUEST_TEMPLATE.md | 1 -
.github/issue-rules.yml | 345 +---
.github/workflows/build_playground_backend.yml | 25 +-
.github/workflows/build_playground_frontend.yml | 29 +-
.github/workflows/go_tests.yml | 12 +-
.github/workflows/issue-tagger.yml | 2 +-
.github/workflows/java_tests.yml | 2 +-
.github/workflows/label_prs.yml | 6 +-
.github/workflows/playground_deploy_backend.yml | 2 +-
.github/workflows/playground_deploy_examples.yml | 2 +-
.github/workflows/playground_examples_ci.yml | 2 +-
.../workflows/playground_examples_ci_reusable.yml | 20 +-
.github/workflows/python_tests.yml | 2 +-
.github/workflows/run_rc_validation.yml | 4 +-
.../workflows/tour_of_beam_backend_integration.yml | 2 +-
.pre-commit-config.yaml | 2 -
.test-infra/jenkins/PrecommitJobBuilder.groovy | 9 +-
.../job_PerformanceTests_FileBasedIO_IT.groovy | 4 +-
.../job_PerformanceTests_PubsubIO_Python.groovy | 6 +-
...stCommit_Python_ValidatesRunner_Dataflow.groovy | 1 -
.test-infra/jenkins/job_PreCommit_Java.groovy | 37 +-
.../job_PreCommit_Java_AWS2_IO_Direct.groovy | 7 +-
.../job_PreCommit_Java_AWS_IO_Direct.groovy | 8 +-
...vy => job_PreCommit_Java_Amqp_IO_Direct.groovy} | 9 +-
...y => job_PreCommit_Java_Azure_IO_Direct.groovy} | 9 +-
... job_PreCommit_Java_Cassandra_IO_Direct.groovy} | 10 +-
...vy => job_PreCommit_Java_Cdap_IO_Direct.groovy} | 12 +-
...job_PreCommit_Java_Clickhouse_IO_Direct.groovy} | 9 +-
.../job_PreCommit_Java_Debezium_IO_Direct.groovy | 9 +-
..._PreCommit_Java_ElasticSearch_IO_Direct.groovy} | 16 +-
.../job_PreCommit_Java_GCP_IO_Direct.groovy | 15 +-
...> job_PreCommit_Java_HCatalog_IO_Direct.groovy} | 11 +-
... => job_PreCommit_Java_Hadoop_IO_Direct.groovy} | 15 +-
...y => job_PreCommit_Java_Hbase_IO_Direct.groovy} | 11 +-
...> job_PreCommit_Java_InfluxDb_IO_Direct.groovy} | 10 +-
.../job_PreCommit_Java_JDBC_IO_Direct.groovy | 8 +-
...ovy => job_PreCommit_Java_Jms_IO_Direct.groovy} | 9 +-
.../job_PreCommit_Java_Kafka_IO_Direct.groovy | 12 +-
.../job_PreCommit_Java_Kinesis_IO_Direct.groovy | 8 +-
...vy => job_PreCommit_Java_Kudu_IO_Direct.groovy} | 10 +-
...=> job_PreCommit_Java_MongoDb_IO_Direct.groovy} | 11 +-
...vy => job_PreCommit_Java_Mqtt_IO_Direct.groovy} | 10 +-
.../job_PreCommit_Java_Neo4j_IO_Direct.groovy | 8 +-
...=> job_PreCommit_Java_Parquet_IO_Direct.groovy} | 10 +-
... => job_PreCommit_Java_Pulsar_IO_Direct.groovy} | 9 +-
...> job_PreCommit_Java_RabbitMq_IO_Direct.groovy} | 10 +-
...y => job_PreCommit_Java_Redis_IO_Direct.groovy} | 10 +-
...ob_PreCommit_Java_Singlestore_IO_Direct.groovy} | 11 +-
... job_PreCommit_Java_Snowflake_IO_Direct.groovy} | 13 +-
...vy => job_PreCommit_Java_Solr_IO_Direct.groovy} | 10 +-
... => job_PreCommit_Java_Splunk_IO_Direct.groovy} | 9 +-
... => job_PreCommit_Java_Thrift_IO_Direct.groovy} | 9 +-
...vy => job_PreCommit_Java_Tika_IO_Direct.groovy} | 9 +-
.../LargeITCluster/hdfs-multi-datanode-cluster.yml | 4 +-
CHANGES.md | 34 +
build.gradle.kts | 46 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 4 +-
.../apache/beam/examples/KafkaWordCountAvro.java | 120 ++
.../apache/beam/examples/KafkaWordCountJson.java | 120 ++
.../beam-ml/dataframe_api_preprocessing.ipynb | 2 +-
.../beam-ml/run_inference_tensorflow_hub.ipynb | 543 ++++++
learning/tour-of-beam/backend/README.md | 18 +-
playground/api/v1/api.proto | 13 +
playground/backend/CONTRIBUTE.md | 30 +-
playground/backend/cmd/server/controller.go | 30 +-
playground/backend/cmd/server/controller_test.go | 85 +-
playground/backend/containers/java/Dockerfile | 26 +-
playground/backend/containers/router/Dockerfile | 19 +-
playground/backend/datasets/CountWords.avro | Bin 0 -> 888 bytes
playground/backend/datasets/CountWords.json | 22 +
playground/backend/go.mod | 53 +-
playground/backend/go.sum | 350 +++-
playground/backend/internal/api/v1/api.pb.go | 1221 +++++++------
playground/backend/internal/api/v1/api_grpc.pb.go | 17 -
.../internal/code_processing/code_processing.go | 25 +-
.../code_processing/code_processing_test.go | 10 +-
.../internal/constants/datastore_constants.go | 1 +
.../snippet.go => constants/emulator_constants.go} | 10 +-
.../backend/internal/db/datastore/datastore_db.go | 74 +-
.../backend/internal/db/dto/precompiled_object.go | 27 +-
.../internal/db/dto/precompiled_object_test.go | 219 +++
playground/backend/internal/db/dto/snippet.go | 10 +
playground/backend/internal/db/entity/snippet.go | 35 +-
playground/backend/internal/db/mapper/mapper.go | 1 +
.../db/mapper/precompiled_object_mapper.go | 63 +-
.../db/mapper/precompiled_object_mapper_test.go | 82 +-
playground/backend/internal/emulators/emulator.go | 106 ++
playground/backend/internal/emulators/kafka.go | 181 ++
.../backend/internal/executors/executor_test.go | 10 +-
playground/backend/internal/fs_tool/fs.go | 6 +-
playground/backend/internal/fs_tool/fs_test.go | 8 +-
.../internal/preparers/go_preparers_test.go | 7 +-
.../backend/internal/preparers/java_preparers.go | 45 +-
.../internal/preparers/java_preparers_test.go | 23 +-
playground/backend/internal/preparers/preparer.go | 14 +-
.../backend/internal/preparers/preparer_test.go | 53 +-
.../internal/preparers/python_preparers_test.go | 10 +-
.../internal/preparers/scio_preparers_test.go | 7 +-
.../internal/setup_tools/builder/setup_builder.go | 13 +-
.../setup_tools/builder/setup_builder_test.go | 26 +-
.../setup_tools/life_cycle/life_cycle_setuper.go | 27 +-
.../life_cycle/life_cycle_setuper_test.go | 12 +-
.../backend/internal/tests/test_data/test_data.go | 29 +-
.../backend/internal/utils/datastore_utils.go | 5 +
playground/backend/internal/utils/file_utils.go | 3 +-
.../backend/internal/utils/file_utils_test.go | 5 +
.../lib/src/api/v1/api.pb.dart | 119 +-
.../lib/src/api/v1/api.pbenum.dart | 35 +-
.../lib/src/api/v1/api.pbgrpc.dart | 20 +-
.../lib/src/api/v1/api.pbjson.dart | 63 +-
playground/infrastructure/api/v1/api_pb2.py | 1889 +++-----------------
playground/infrastructure/cd_helper.py | 7 +-
playground/infrastructure/checker.py | 4 +-
playground/infrastructure/ci_helper.py | 5 +-
playground/infrastructure/config.py | 37 +-
playground/infrastructure/datastore_client.py | 267 +--
playground/infrastructure/grpc_client.py | 8 +-
.../templates/deployment-router-grpc.yml | 48 +
.../templates/service-router-grpc.yml | 32 +-
playground/infrastructure/helper.py | 95 +-
playground/infrastructure/proxy/allow_list.py | 6 +-
.../{proxy/allow_list.py => repository.py} | 36 +-
playground/infrastructure/requirements.txt | 1 +
playground/infrastructure/test_datastore_client.py | 43 +-
playground/infrastructure/test_grpc_client.py | 2 +-
playground/infrastructure/test_helper.py | 381 +++-
playground/infrastructure/test_repository.py | 53 +
.../google-cloud-dataflow-java/worker/build.gradle | 1 +
.../worker/legacy-worker/build.gradle | 1 +
.../runners/fnexecution/wire/CommonCoderTest.java | 6 +-
runners/samza/build.gradle | 4 -
.../org/apache/beam/runners/samza/SamzaRunner.java | 8 +-
.../apache/beam/runners/samza/runtime/DoFnOp.java | 6 +-
.../runners/samza/runtime/SamzaDoFnRunners.java | 10 +-
.../samza/runtime/SamzaStoreStateInternals.java | 40 +-
.../runners/samza/translation/ConfigBuilder.java | 5 +-
.../runners/samza/translation/ConfigContext.java | 13 +-
.../translation/ParDoBoundMultiTranslator.java | 32 +-
.../translation/PortableTranslationContext.java | 2 +-
.../samza/translation/SamzaPipelineTranslator.java | 4 +-
.../runners/samza/translation/StateIdParser.java | 69 +
.../samza/translation/TranslationContext.java | 8 +
.../samza/util/SamzaPipelineTranslatorUtils.java | 4 +-
.../beam/runners/samza/util/StoreIdGenerator.java | 48 +
.../runtime/SamzaStoreStateInternalsTest.java | 102 ++
.../samza/translation/ConfigGeneratorTest.java | 146 +-
.../samza/translation/TranslationContextTest.java | 6 +-
.../io/BoundedDatasetFactory.java | 10 +-
.../translation/PipelineTranslator.java | 39 +-
.../translation/SparkSessionFactory.java | 5 +
.../translation/TransformTranslator.java | 9 +-
.../translation/batch/Aggregators.java | 18 +-
.../batch/DoFnMapPartitionsFactory.java | 7 +-
.../batch/GroupByKeyTranslatorBatch.java | 2 +-
.../translation/batch/ParDoTranslatorBatch.java | 2 +-
.../batch/ReadSourceTranslatorBatch.java | 5 +-
.../batch/functions/CachedSideInputReader.java | 4 +-
.../GroupAlsoByWindowViaOutputBufferFn.java | 7 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 11 +-
sdks/go.mod | 8 +-
sdks/go.sum | 16 +-
sdks/go/cmd/specialize/main.go | 8 +-
sdks/go/cmd/symtab/main.go | 2 +-
.../nativepubsubio/subscriptiontracker.go | 6 +-
sdks/go/examples/snippets/01_03intro.go | 8 +-
sdks/go/examples/snippets/06schemas.go | 8 +-
sdks/go/examples/snippets/06schemas_test.go | 2 +-
sdks/go/examples/snippets/12splittabledofns.go | 2 +-
sdks/go/pkg/beam/artifact/materialize_test.go | 4 +-
sdks/go/pkg/beam/beam.shims.go | 86 +-
sdks/go/pkg/beam/coder.go | 8 +-
sdks/go/pkg/beam/coder_test.go | 6 +-
sdks/go/pkg/beam/combine.go | 8 +-
sdks/go/pkg/beam/core/graph/xlang.go | 2 +-
.../pkg/beam/core/runtime/coderx/coderx.shims.go | 88 +-
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 6 +-
.../pkg/beam/core/runtime/exec/datasource_test.go | 32 +-
.../go/pkg/beam/core/runtime/exec/dynsplit_test.go | 20 +-
sdks/go/pkg/beam/core/runtime/exec/plan.go | 13 +-
sdks/go/pkg/beam/core/runtime/graphx/v1/v1.pb.go | 2 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 13 +-
.../core/runtime/harness/session/session.pb.go | 3 +-
sdks/go/pkg/beam/core/typex/special.go | 12 +-
sdks/go/pkg/beam/create.go | 12 +-
sdks/go/pkg/beam/create_test.go | 24 +-
sdks/go/pkg/beam/encoding.go | 32 +-
sdks/go/pkg/beam/example_schema_test.go | 12 +-
sdks/go/pkg/beam/forward.go | 6 +-
sdks/go/pkg/beam/internal/errors/errors.go | 8 +-
sdks/go/pkg/beam/io/avroio/avroio.go | 4 +-
sdks/go/pkg/beam/io/avroio/avroio_test.go | 6 +-
sdks/go/pkg/beam/io/bigqueryio/bigquery.go | 6 +-
sdks/go/pkg/beam/io/bigtableio/bigtable_test.go | 2 +-
sdks/go/pkg/beam/io/databaseio/database.go | 6 +-
sdks/go/pkg/beam/io/databaseio/loader.go | 4 +-
sdks/go/pkg/beam/io/databaseio/mapper.go | 32 +-
sdks/go/pkg/beam/io/databaseio/util.go | 6 +-
sdks/go/pkg/beam/io/databaseio/writer.go | 10 +-
sdks/go/pkg/beam/io/datastoreio/datastore_test.go | 4 +-
sdks/go/pkg/beam/io/fhirio/execute_bundles.go | 4 +-
sdks/go/pkg/beam/io/fhirio/search.go | 4 +-
sdks/go/pkg/beam/io/filesystem/s3/helper_test.go | 2 +-
sdks/go/pkg/beam/io/parquetio/parquetio.go | 2 +-
sdks/go/pkg/beam/io/parquetio/parquetio_test.go | 2 +-
.../beam/io/rtrackers/offsetrange/offsetrange.go | 8 +-
.../io/rtrackers/offsetrange/offsetrange_test.go | 12 +-
sdks/go/pkg/beam/io/textio/textio.go | 2 +-
sdks/go/pkg/beam/io/xlang/jdbcio/jdbc.go | 2 +-
sdks/go/pkg/beam/io/xlang/schemaio/schemaio.go | 6 +-
sdks/go/pkg/beam/log/log.go | 36 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 17 +-
.../model/fnexecution_v1/beam_fn_api_grpc.pb.go | 2 +-
.../model/fnexecution_v1/beam_provision_api.pb.go | 2 +-
.../fnexecution_v1/beam_provision_api_grpc.pb.go | 2 +-
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 5 +-
.../jobmanagement_v1/beam_artifact_api_grpc.pb.go | 2 +-
.../jobmanagement_v1/beam_expansion_api.pb.go | 315 +++-
.../jobmanagement_v1/beam_expansion_api_grpc.pb.go | 42 +-
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 3 +-
.../model/jobmanagement_v1/beam_job_api_grpc.pb.go | 2 +-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 274 +--
.../model/pipeline_v1/beam_runner_api_grpc.pb.go | 2 +-
sdks/go/pkg/beam/model/pipeline_v1/endpoints.pb.go | 2 +-
.../model/pipeline_v1/external_transforms.pb.go | 217 ++-
sdks/go/pkg/beam/model/pipeline_v1/metrics.pb.go | 5 +-
sdks/go/pkg/beam/model/pipeline_v1/schema.pb.go | 280 +--
.../model/pipeline_v1/standard_window_fns.pb.go | 2 +-
sdks/go/pkg/beam/option.go | 4 +-
sdks/go/pkg/beam/options/jobopts/stringSlice.go | 2 +-
sdks/go/pkg/beam/pardo.go | 22 +-
sdks/go/pkg/beam/partition.go | 14 +-
sdks/go/pkg/beam/partition_test.go | 8 +-
sdks/go/pkg/beam/provision/provision.go | 4 +-
sdks/go/pkg/beam/register/emitter.go | 8 +-
sdks/go/pkg/beam/register/iter.go | 4 +-
sdks/go/pkg/beam/register/register.go | 1338 +++++++-------
sdks/go/pkg/beam/register/register.tmpl | 120 +-
sdks/go/pkg/beam/register/register_test.go | 70 +-
.../beam/runners/dataflow/dataflowlib/messages.go | 20 +-
.../beam/runners/dataflow/dataflowlib/metrics.go | 6 +-
.../runners/dataflow/dataflowlib/metrics_test.go | 2 +-
sdks/go/pkg/beam/runners/direct/buffer.go | 2 +-
.../runners/vet/testpipeline/testpipeline.shims.go | 38 +-
sdks/go/pkg/beam/runners/vet/vet.go | 6 +-
sdks/go/pkg/beam/schema.go | 8 +-
sdks/go/pkg/beam/testing/passert/equals.go | 4 +-
sdks/go/pkg/beam/testing/passert/floats.go | 2 +-
sdks/go/pkg/beam/testing/passert/passert.go | 4 +-
sdks/go/pkg/beam/testing/passert/passert.shims.go | 130 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 8 +-
sdks/go/pkg/beam/testing/ptest/ptest_test.go | 6 +-
sdks/go/pkg/beam/testing/teststream/teststream.go | 6 +-
.../pkg/beam/testing/teststream/teststream_test.go | 20 +-
.../go/pkg/beam/transforms/filter/distinct_test.go | 28 +-
sdks/go/pkg/beam/transforms/filter/filter.go | 4 +-
sdks/go/pkg/beam/transforms/filter/filter.shims.go | 36 +-
sdks/go/pkg/beam/transforms/filter/filter_test.go | 4 +-
sdks/go/pkg/beam/transforms/stats/max_switch.go | 2 +-
sdks/go/pkg/beam/transforms/stats/max_switch.tmpl | 2 +-
sdks/go/pkg/beam/transforms/stats/mean.go | 2 +-
sdks/go/pkg/beam/transforms/stats/min_switch.go | 2 +-
sdks/go/pkg/beam/transforms/stats/min_switch.tmpl | 2 +-
sdks/go/pkg/beam/transforms/stats/quantiles.go | 22 +-
.../go/pkg/beam/transforms/stats/quantiles_test.go | 6 +-
sdks/go/pkg/beam/transforms/stats/stats.shims.go | 146 +-
sdks/go/pkg/beam/transforms/stats/sum_switch.go | 2 +-
sdks/go/pkg/beam/transforms/stats/sum_switch.tmpl | 2 +-
sdks/go/pkg/beam/transforms/stats/util.go | 4 +-
sdks/go/pkg/beam/transforms/top/top.go | 16 +-
sdks/go/pkg/beam/transforms/top/top.shims.go | 40 +-
.../pkg/beam/transforms/xlang/python/external.go | 8 +-
sdks/go/pkg/beam/util.go | 2 +-
sdks/go/pkg/beam/util/shimx/generate.go | 20 +-
sdks/go/pkg/beam/util/shimx/generate_test.go | 2 +-
sdks/go/pkg/beam/util/starcgenx/starcgenx.go | 2 +-
sdks/go/pkg/beam/util/starcgenx/starcgenx_test.go | 6 +-
sdks/go/pkg/beam/x/debug/debug.shims.go | 66 +-
sdks/go/pkg/beam/xlang.go | 2 +-
sdks/go/test/integration/flags.go | 2 +-
.../integration/io/xlang/bigquery/bigquery_test.go | 9 +-
sdks/go/test/integration/primitives/windowinto.go | 6 +-
.../go/test/regression/coders/fromyaml/fromyaml.go | 22 +-
sdks/go/test/regression/pardo.go | 4 +-
sdks/java/core/build.gradle | 5 +-
.../org/apache/beam/sdk/coders/DefaultCoder.java | 4 +-
.../org/apache/beam/sdk/coders/SortedMapCoder.java | 2 +
.../beam/sdk/runners/PTransformOverride.java | 4 +-
.../sdk/schemas/FieldValueTypeInformation.java | 22 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 4 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 8 +-
.../apache/beam/sdk/schemas/transforms/Filter.java | 4 +-
.../apache/beam/sdk/schemas/transforms/Group.java | 2 +-
.../sdk/schemas/transforms/SchemaAggregateFn.java | 4 +
.../beam/sdk/schemas/utils/ReflectUtils.java | 18 +-
.../beam/sdk/transforms/FlatMapElements.java | 257 ++-
.../apache/beam/sdk/transforms/MapElements.java | 250 ++-
.../beam/sdk/transforms/join/CoGbkResult.java | 9 +-
.../transforms/resourcehints/ResourceHints.java | 2 +-
.../resourcehints/ResourceHintsOptions.java | 5 +-
.../java/org/apache/beam/sdk/util/MoreFutures.java | 10 +-
.../apache/beam/sdk/values/PCollectionList.java | 2 +-
.../apache/beam/sdk/values/PCollectionTuple.java | 2 +-
.../main/java/org/apache/beam/sdk/values/Row.java | 12 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 1 +
.../apache/beam/sdk/values/TypeDescriptors.java | 2 +-
.../sql/impl/rel/BeamAggregationRel.java | 4 +-
sdks/java/harness/build.gradle | 1 +
.../beam/fn/harness/PrecombineGroupingTable.java | 82 +-
.../fn/harness/PrecombineGroupingTableTest.java | 68 +-
sdks/java/io/cdap/build.gradle | 5 +-
.../java/org/apache/beam/sdk/io/cdap/CdapIO.java | 89 +-
.../org/apache/beam/sdk/io/cdap/MappingUtils.java | 102 +-
.../java/org/apache/beam/sdk/io/cdap/Plugin.java | 127 +-
.../beam/sdk/io/cdap/context/BatchContextImpl.java | 51 +-
.../io/cdap/context/FailureCollectorWrapper.java | 18 +
.../org/apache/beam/sdk/io/cdap/CdapIOTest.java | 36 +-
.../org/apache/beam/sdk/io/cdap/PluginTest.java | 12 +-
.../cdap/context/FailureCollectorWrapperTest.java | 24 +
sdks/java/io/debezium/build.gradle | 1 +
.../apache/beam/io/debezium/KafkaConnectUtils.java | 77 +
.../beam/io/debezium/KafkaConnectSchemaTest.java | 58 +
.../beam/io/debezium/SourceRecordJsonTest.java | 65 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 2 +-
.../beam/sdk/io/gcp/bigquery/AppendClientInfo.java | 12 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 39 +-
.../bigquery/StorageApiWritesShardedRecords.java | 62 +-
.../beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java | 2 +-
.../PubsubLiteWriteSchemaTransformProvider.java | 167 ++
.../beam/sdk/io/gcp/pubsublite/ReadWriteIT.java | 63 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 392 ++--
.../beam/sdk/io/jdbc/JdbcSchemaIOProvider.java | 51 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 83 +-
.../org/apache/beam/sdk/io/jdbc/LogicalTypes.java | 7 +-
.../org/apache/beam/sdk/io/jdbc/SchemaUtil.java | 38 +-
.../io/kafka/KafkaReadSchemaTransformProvider.java | 38 +-
.../kafka/KafkaWriteSchemaTransformProvider.java | 166 ++
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 144 ++
.../KafkaReadSchemaTransformProviderTest.java | 2 +-
.../beam/sdk/io/mongodb/AggregationQuery.java | 5 +
.../java/org/apache/beam/sdk/io/neo4j/Neo4jIO.java | 4 +-
.../ReadFromSparkReceiverWithOffsetDoFn.java | 37 +-
.../beam/sdk/io/sparkreceiver/SparkReceiverIO.java | 28 +-
.../sdk/io/sparkreceiver/SparkReceiverIOTest.java | 27 +
.../apache_beam/io/gcp/pubsub_io_perf_test.py | 1 -
.../python/apache_beam/options/pipeline_options.py | 7 -
.../runners/dataflow/dataflow_runner.py | 295 ++-
.../runners/dataflow/dataflow_runner_test.py | 208 ++-
.../runners/dataflow/internal/apiclient.py | 84 +-
.../runners/dataflow/internal/apiclient_test.py | 125 +-
.../runners/dataflow/ptransform_overrides.py | 60 +-
.../apache_beam/runners/portability/stager.py | 8 -
.../apache_beam/testing/load_tests/build.gradle | 8 -
.../apache_beam/transforms/combiners_test.py | 21 +
sdks/python/apache_beam/transforms/core.py | 6 +
.../container/py310/base_image_requirements.txt | 77 +-
.../container/py37/base_image_requirements.txt | 79 +-
.../container/py38/base_image_requirements.txt | 84 +-
.../container/py39/base_image_requirements.txt | 83 +-
sdks/python/scripts/run_integration_test.sh | 12 -
sdks/python/setup.py | 4 +-
sdks/python/test-suites/dataflow/common.gradle | 46 +-
website/www/site/assets/scss/_table-wrapper.scss | 10 +
.../content/en/documentation/io/built-in/cdap.md | 396 ++++
.../en/documentation/io/built-in/sparkreceiver.md | 103 ++
.../site/content/en/documentation/io/connectors.md | 26 +
.../content/en/documentation/programming-guide.md | 2 +-
website/www/site/layouts/about/baseof.html | 2 +-
.../partials/section-menu/en/documentation.html | 2 +
372 files changed, 10423 insertions(+), 7335 deletions(-)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Amqp_IO_Direct.groovy} (87%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Azure_IO_Direct.groovy} (87%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Cassandra_IO_Direct.groovy} (84%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Cdap_IO_Direct.groovy} (80%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Clickhouse_IO_Direct.groovy} (86%)
copy .test-infra/jenkins/{job_PreCommit_Java_Neo4j_IO_Direct.groovy => job_PreCommit_Java_ElasticSearch_IO_Direct.groovy} (65%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_HCatalog_IO_Direct.groovy} (82%)
copy .test-infra/jenkins/{job_PreCommit_Java_JDBC_IO_Direct.groovy => job_PreCommit_Java_Hadoop_IO_Direct.groovy} (72%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Hbase_IO_Direct.groovy} (83%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_InfluxDb_IO_Direct.groovy} (84%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Jms_IO_Direct.groovy} (87%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Kudu_IO_Direct.groovy} (85%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_MongoDb_IO_Direct.groovy} (82%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Mqtt_IO_Direct.groovy} (85%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Parquet_IO_Direct.groovy} (84%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Pulsar_IO_Direct.groovy} (87%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_RabbitMq_IO_Direct.groovy} (84%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Redis_IO_Direct.groovy} (85%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Singlestore_IO_Direct.groovy} (81%)
copy .test-infra/jenkins/{job_PreCommit_Java_Neo4j_IO_Direct.groovy => job_PreCommit_Java_Snowflake_IO_Direct.groovy} (76%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Solr_IO_Direct.groovy} (85%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Splunk_IO_Direct.groovy} (87%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Thrift_IO_Direct.groovy} (87%)
copy .test-infra/jenkins/{job_PreCommit_Java_Debezium_IO_Direct.groovy => job_PreCommit_Java_Tika_IO_Direct.groovy} (87%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountAvro.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/KafkaWordCountJson.java
create mode 100644 examples/notebooks/beam-ml/run_inference_tensorflow_hub.ipynb
create mode 100755 playground/backend/datasets/CountWords.avro
create mode 100644 playground/backend/datasets/CountWords.json
copy playground/backend/internal/{db/dto/snippet.go => constants/emulator_constants.go} (86%)
create mode 100644 playground/backend/internal/db/dto/precompiled_object_test.go
create mode 100644 playground/backend/internal/emulators/emulator.go
create mode 100644 playground/backend/internal/emulators/kafka.go
create mode 100644 playground/infrastructure/helm-playground/templates/deployment-router-grpc.yml
copy .github/workflows/issue-tagger.yml => playground/infrastructure/helm-playground/templates/service-router-grpc.yml (68%)
copy playground/infrastructure/{proxy/allow_list.py => repository.py} (54%)
create mode 100644 playground/infrastructure/test_repository.py
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/StateIdParser.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/StoreIdGenerator.java
create mode 100644 sdks/java/io/debezium/src/main/java/org/apache/beam/io/debezium/KafkaConnectUtils.java
create mode 100644 sdks/java/io/debezium/src/test/java/org/apache/beam/io/debezium/KafkaConnectSchemaTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteWriteSchemaTransformProvider.java
create mode 100644 sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/KafkaWriteSchemaTransformProvider.java
create mode 100644 website/www/site/content/en/documentation/io/built-in/cdap.md
create mode 100644 website/www/site/content/en/documentation/io/built-in/sparkreceiver.md