You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by bh...@apache.org on 2021/02/17 00:07:39 UTC
[beam] branch website-revamp updated (67508d0 -> 10328bc)
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a change to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 67508d0 Remove website/www/site/content/en/documentation/runners/basics.md
add 2e43854 [BEAM-11652] bump aircompressor to 0.1.8
add 3d6cb85 Merge pull request #13855: [BEAM-11652] bump aircompressor to 0.1.8
add 4ea3602 [BEAM-10379] Remove BIT_XOR from ZetaSQL supported functions list.
add 41bd965 Merge pull request #13866 from ibzib/xor-rollback
add 8c2db83 [BEAM-11731] Restrict to numpy <1.20.0 (#13870)
add 2930de1 [BEAM-11732] Revert flink-clients from runtime to compile configuration.
add 423659c Merge pull request #13868 from ibzib/BEAM-11732
add 83d680b [BEAM-11357] Copy Annotations when cloning PTransforms (#13865)
add b0d8d60 Setup InfluxDbIO_IT jenkins job cron
add 41ebec8 Merge pull request #13849 from y1chi/master
add 63acfb0 [BEAM-11693] Update formatting. Fix email template (#13815)
add 5b105d4 Merge pull request #13723 from [BEAM-11736] Propagate pipeline options to direct runner
add 8078706 [BEAM-11688] Support partial proto encoding
add 5356056 Merge pull request #13812: [BEAM-11688] Support partial proto encoding
add 9fe8c34 Fix example syntax in SQL walkthough
add 79e8f55 Merge pull request #13776: Fix example syntax in SQL walk though
add 636ac14 [BEAM-11733] Skip flaky healthcare IO integration tests.
add fb8fcd5 Merge pull request #13869 from [BEAM-11733] Skip flaky healthcare IO integration tests.
add ad0f575 [BEAM-11715] Add ValidatesRunner test for translations.pack_combiners (#13851)
add 7c07ceb [BEAM-11720] Don't assume a particular pip location, use default pip path. (#13850)
add 0078bb3 [BEAM-11589] Migrated settings.gradle file to Kotlin script. (#13837)
add 34ecb8b Merge pull request #13824 from [BEAM-11700] Fix PortableRunner skipping optimizations
add 2237c69 [BEAM-11581] Minor fix to skip ExecutionStateSampler.reset() in ProcessBundleHandler
add 0e91f1f Merge pull request #13876 from [BEAM-11581] Minor fix to skip ExecutionStateSampler.reset() in ProcessBundleHandler
add 453b115 Add tag to docker push
add e24892b Merge pull request #13880 from [BEAM-11739] Add tag to docker image when pushing to container registry
add afb6455 Document recommended pipeline run pattern for Python.
add eb733a4 Merge pull request #13873 from robertwb/pipeline-run
add 489ab66 [BEAM-11531] Allow pandas <1.3.0 (#13681)
add 816f297 Add transform translator for GroupIntoBatches in Java.
add 9b92f41 Fix checkStyle error; add a condition to disable autosharding for JRH
add b4a9a99 Fail jobs that would otherwise fall back to the default implementation of GIB
add 4a10c01 Fix failed test; update checkArgument logs.
add b56b61a Merge pull request #13805 from [BEAM-10475] Add transform translator for GroupIntoBatches in Java.
add 7962018 [BEAM-11705] Fixing ignore_insert_id implementation
add 9e18310 Merge pull request #13827 from [BEAM-11705] Fixing ignore_insert_id implementation
add 9cbef09 Bump Dataflow worker container version.
add 4be9e20 Merge pull request #13890 from Bump Dataflow worker container version.
add 291ced1 Create Spark Metrics in directory using Spark History Server format
add f044ec1 java spotless apply
add c578c84 add --spark-history-dir option in spark job server
add cb1b46f add driver distribution logs
add 864603a remove start import check
add f0c7412 spotless apply
add ad359f5 render all metrics & add eventLogEnabled
add bb7c05e add exception when eventLogEnabled is true
add cac34a7 handle eventLogEnabled
add e669894 apply spotless
add 8be02d6 remove star import
add 5392f66 run java spotless
add 1da04b4 fix renderName is static from now on
add 9bec263 spark.executor.id loop and switch to boolean
add 31f86b8 spotless apply
add c98a89e remove uncessary code
add 6529c7a scope.Option.apply
add 03c2457 remove options from spark job server configuration & fix the spark listener executor added logic
add d3b7838 remove options from spark job server configuration & fix the spark listener executor added logic
add ba68363 remove options from spark job server configuration & fix the spark listener executor added logic
add 6acf592 remove options from spark job server configuration & fix the spark listener executor added logic
add a513fbf remove options from spark job server configuration & fix the spark listener executor added logic
add f383e17 remove options from spark job server configuration & fix the spark listener executor added logic
add e105b9f minor cleanup
add 068a33e minor cleanup
add b35c282 minor improvments
add 4b64b5f remove whitespace changes
add 6329b57 remove whitespace changes
add 654ad2b Merge pull request #13743 from tszerszen/sparkmetrics
add 3fdb136 [BEAM-11476] Resolve flaky tests (#13881)
add 8da6a6c [BEAM-11519] Adding PYPI_INSTALL_REQ role and SDK container support
add 7cb5691 move idCounter variable and add comments
add c2b7fba simplify the branches, change _req to _requirement
add 715bcd0 change urn for pip install requirements file
add cb52848 Merge pull request #13606 from ihji/BEAM-11519
add 842711d [BEAM-11657] Avoid repeated reflection calls for Kafka deserialization when not using headers
add 2a8daff Merge pull request #13782: [BEAM-11657] Reduce reflection calls when reading from Kafka
add 0cfa80f Remove an unused reference to staleTimerSet and reword the commentary. (#13268)
add ef5aeef [BEAM-10925] Enable user-defined Java scalar functions in ZetaSQL.
add feee800 address review comments
add 9bbd5bd Merge pull request #13891 from ibzib/scalarfn-test
add ebd2a3a Fix the check on maxBufferingDuration
add 5c31997 Merge pull request #13894 from [BEAM-11746] Fix test flakiness: GroupIntoBatches.testInGlobalWindow
add 0fbb21f [BEAM-11325] Support KafkaIO dynamic read
add 47d3326 Merge pull request #13750 from [BEAM-11325] Kafka Dynamic Read
add 5ea504d [BEAM-11715] Partial revert of "Combiner packing in Dataflow" (#13763) (#13884)
add 3261fe3 [BEAM-10961] Enable strict dependency checking for sdks/java/io/snowflake (#13800)
add 623f06d [BEAM-11723] Avoid eliminate_common_key_with_none without combiner packing.
add f9707b1 Merge pull request #13904: [BEAM-11723] Avoid eliminate_common_key_with_none without combiner packing.
add 94c2739 Renamed build.gradle to build.gradle.kts
add 45a6868 Migrated build.gradle file to Kotlin script.
add d483d3a Updated autolabeler.yml
add 52d0bd9 Merge pull request #13888 from akvelon/gradle_to_kotlin
add 25b4090 [BEAM-11045] Advance chrome version for screen diff integration test 1. Advanced chromedriver-binary version to current stable chrome browser version 88. 2. Updated the golden screenshots to new ones. Screenshot size in this version of chrome browser is now back to the viewport size.
add 2151954 Merge pull request #13905 from KevinGG/BEAM-11045
add b74fcf7 [BEAM-11731][BEAM-10582] Allow pyarrow<4,numpy<1.21.0, improve pyarrow verification (#13892)
add a8b0573 [BEAM-11595] bump jackson to 2.12.1
add befcc3d Merge pull request #13900: [BEAM-11595] bump jackson to 2.12.1
add d9011f6 [BEAM-11729] Remove calls to Class.getSimpleName() for WindowTracing debug logs. These are evaluated eagerly and showed up on CPU profiles.
add 123901f Merge #13864: [BEAM-11729] Remove calls to Class.getSimpleName() for WindowTracing
add 5565e18 [BEAM-11773] Add bhulette as an owner for pandas, pyarrow, numpy requirements (#13919)
add b787dd1 [BEAM-9547] Produce better errors for some groupby() and set_index() configurations (#13853)
add 86f266b [BEAM-11762] Upgrades Tensorflow, numpy, and grpcio dependencies (#13911)
add 5d1a034 [BEAM-9918] Support Tagged external PTransforms (#13872)
add 804bc28 Using LoadingCache instead of Map to cache BundleProcessor
add 022a2ab Merge pull request #13893 from [BEAM-11752] Using LoadingCache instead of Map to cache BundleProcessor
add 1d921ee Remove translations.eliminate_common_key_with_none from DataflowRunner
add 091ecdf Merge pull request #13921: [BEAM-11723] Remove translations.eliminate_common_key_with_none from DataflowRunner when pre_optimize=all
add 6637224 [BEAM-11503] bump rockdbjni to 6.15.2
add 6935e10 Merge pull request #13916: [BEAM-11503] bump rockdbjni to 6.15.2
add f0bd066 [BEAM-11265] make sure sample text file gets copied into archetype (#13918)
add 7b7ab4b [BEAM-11723] Use packed name for PairWithVoid consolidation.
add 6e0ca2f Only do pair-with-none packing when combiner packing is desired.
add 2c559ab Make eliminate_common_key_with_none private.
add d0c5e80 Merge branch 'master' into pack-combine-naming
add a35752f Merge branch 'master' into pack-combine-naming
add 4b53ffc Merge pull request #13856 [BEAM-11723] More conservative PairWithVoid consolidation.
add c6db9ca [BEAM-11324] Impose consistent ordering on partitionings used in PartitioningSession (#13854)
add e6eba9e [BEAM-11659] Extract common payload serialization behavior for Row transforms. (#13825)
add ee4ccef Explicitly set dataflow runner v2 test shuffle mode to appliance
add 5adf500 Merge pull request #13928 from [BEAM-11779] Explicitly set dataflow runner v2 test shuffle mode to appliance
add b3fd2bd [BEAM-11377] Add retries to mvn command.
add a272999 Merge pull request #13923 from tysonjh/snapshot-fix
add ef685e6 [BEAM-11168] bump powermock to 2.0.9
add 6edc333 Merge pull request #13936: [BEAM-11168] bump powermock to 2.0.9
add 827c938 [BEAM-10961] make mongo-java-driver dependency a named variable
add 705649d Merge pull request #13910 from sonam-vend/sdks-io-mongodb-driver-version
add 595c252 [BEAM-11781]: Update kafka-clients version in samza-runner (#13935)
add a9a0cee Renamed build.gradle to build.gradle.kts
add ec4e268 Migrated build file to kotlin
add 771edd0 Reverted extra change
add b19e751 Removed kotlin-dsl plugin from buildSrc
add d7eac76 [BEAM-11589] Migrated buildSrc build.gradle file to Kotlin script (#13917)
add e28fdba [BEAM-11045] Updated screen diff goldens
add 1293acb Merge pull request #13943 from [BEAM-11045] Updated screen diff goldens
add ca5e018 Update programming guide: New python expansion service binary
add c349a5b Merge pull request #13945: Update programming guide: New python expansion service target
add 3e69e31 Maintain topological order for root transforms
add bc1569c Merge pull request #13942: [BEAM-11784] Maintain topological order of root transforms
add fc8f4e6 [BEAM-10925] Test Java UDF on columns
add 16f757c Merge pull request #13947 from amaliujia/rw-more-testing
add 514a27e [BEAM-11789] Upgrade gradle-dependency-analyze plugin.
add 44652b8 Merge pull request #13949 from ibzib/BEAM-11789-kt
add 4cb02a5 [BEAM-11743] countif aggregate function implementation for zetasql dialect (#13895)
add 207e03a [BEAM-11708] Skip combinefn_lifecycle_test.test_combining_value_state (#13950)
add e2bdd6d [BEAM-11377] Disable mvn http keepalive.
add 6e03cfd Merge pull request #13952 from [BEAM-11377] Disable mvn http keepalive.
add 88190dc Removes VR test
add 7502992 Merge pull request #13957: [BEAM-11787] Removes new VR test that fails for Dataflow
add 3f76b5e [BEAM-11787] Do not call `translations.optimize_pipeline` in Dataflow Streaming (#13951)
add e0ba4dc update versions
add 5136c4e update autovalue version
add a5750ab Merge pull request #13765: [BEAM-11648] In preparation for new BigQuery sink, update GCP BOM
add 625ee1f [BEAM-2914] Add portable merging window support to Python. (#12995)
add e26fe06 [BEAM-11791] Continuously running microbenchmark for FnApiRunner
add ee4cf67 Merge pull request #13906 from pabloem/fnrunner-ubench
add 97a4a19 [BEAM-11647] Fix go:goBuild gradle rules for build collision (#13958)
add 5880bf1 Fixes a checkstyle error in UdfTestProvider
add 7e18e71 Merge pull request #13959: [BEAM-11794] Fixes a checkstyle error in UdfTestProvider
add 8f219a2 [BEAM-11611] Add transformation for computing approximate quantiles. (#13889)
add 04eff7d Integrate BigQuery sink file loads with GroupIntoBatches
add dc29b8c Merge pull request #13859 from [BEAM-11772] Integrate BigQuery sink file loads with GroupIntoBatches
add e95c5dd [BEAM-10925] Don't publish udf-test-provider to Maven.
add b2c79ad Merge pull request #13967 from ibzib/dont-publish-test
add adf4555 [BEAM-11780] Use vendored cloudbuild python client. (#13933)
add dcf3e78 [BEAM-11804] Remove vendors/sdk-java-extensions-protobuf (#13968)
add 9aa9c86 [BEAM-7372][BEAM-9372] cleanup python 2.x and 3.5 codepaths (#13913)
add 4dd7789 Simplify LateDataDropping runner.
add 3640ae2 Merge pull request #13523: Simplify LateDataDropping runner.
add a73153b Changing query to sql to fix doc
add 9b79d9c Merge pull request #13860 from bitnahian/hotfix/spanner-doc-fix
add 436527c [BEAM-11807] SDK Worker multithreading causes boto3 the KeyError
add 4fb1f05 Merge pull request #13977 from [BEAM-11807] Fix issue with sdk worker multi-threading and boto3 client
add 06e0616 [BEAM-11520] Stage extra PyPI dependencies with generated requirements
add 0397c51 raise exception for non-file type artifacts
add 31dbf8e Merge pull request #13607 from ihji/BEAM-11520
add bb1cf0e [BEAM-11476] Resolve flakiness (#13984)
add b6eaf4f Revert "update versions"
add 7929d6f Revert "update autovalue version"
add e092703 [BEAM-11805] Revert version updates from #13765 (#13972)
add bfc858a [BEAM-11806] Explicit Partition Support for KafkaIO.WriteRecords (#13975)
add 9631007 [BEAM-11124] bump joda-time to 2.10.10
add 3930d12 Merge pull request #13981: [BEAM-11124] bump joda-time to 2.10.10
add f53bfee Move payload providers out into their own module. (#13963)
add 0566418 Removed conscrypt from a shaded dataflow-worker jar (#13846)
add 09e65cb [BEAM-11747] Reject the mixed Java UDF and ZetaSQL builtin operator cases (#13912)
add c9b60bc Minor fixes for python nexmark benchmarks
add d9d1daf Merge pull request #13969 from Minor fixes for python nexmark benchmarks
add 5996801 [BEAM-11589] Migrated release build.gradle file to Kotlin script (#13948)
add 7b4962e [BEAM-11811] [Dataflow Java] Don't allow numWorkers > maxNumWorkers.
add cbe2b9c Don't create unassigned variable.
add fac5f97 Merge pull request #13989 from ibzib/BEAM-11811
add b4413d6 Bump dataflow container version (#13988)
add 797985a Merge remote-tracking branch 'origin/master' into website-revamp
add 10328bc Fix committer trait links
No new revisions were added by this update.
Summary of changes:
.github/autolabeler.yml | 2 +-
.gitignore | 1 +
.test-infra/jenkins/PrecommitJobBuilder.groovy | 2 +-
.../job_LoadTests_FnApiRunner_Python.groovy | 64 +
.../job_PerformanceTests_InfluxDBIO_IT.groovy | 1 +
.../Python_FnApiRunner_ubenchmarks.json | 174 ++
build.gradle | 396 ----
build.gradle.kts | 402 ++++
buildSrc/build.gradle | 98 -
buildSrc/build.gradle.kts | 99 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 7 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 6 +-
ownership/PYTHON_DEPENDENCY_OWNERS.yaml | 9 +
release/{build.gradle => build.gradle.kts} | 29 +-
.../src/main/groovy/MobileGamingCommands.groovy | 3 +
.../main/groovy/quickstart-java-dataflow.groovy | 3 +
.../src/main/groovy/quickstart-java-direct.groovy | 3 +
.../main/groovy/quickstart-java-flinklocal.groovy | 3 +
.../main/groovy/quickstart-java-twister2.groovy | 3 +
runners/core-construction-java/build.gradle | 1 +
.../construction/GroupIntoBatchesTranslation.java | 100 +
.../core/construction/PTransformTranslation.java | 5 +
.../core/construction/PipelineTranslation.java | 5 +-
.../GroupIntoBatchesTranslationTest.java | 106 ++
.../runners/core/LateDataDroppingDoFnRunner.java | 74 +-
.../apache/beam/runners/core/ReduceFnRunner.java | 6 +-
runners/flink/flink_runner.gradle | 6 +-
.../streaming/ExecutableStageDoFnOperator.java | 2 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 6 +-
.../dataflow/DataflowPipelineTranslator.java | 7 +
.../beam/runners/dataflow/DataflowRunner.java | 31 +-
.../runners/dataflow/GroupIntoBatchesOverride.java | 5 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 171 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 11 +-
.../google-cloud-dataflow-java/worker/build.gradle | 6 +-
.../worker/legacy-worker/build.gradle | 13 +-
runners/java-fn-execution/build.gradle | 1 -
.../control/ProcessBundleDescriptors.java | 2 +-
.../fnexecution/state/StateRequestHandlers.java | 2 +-
.../runners/fnexecution/wire/ByteStringCoder.java | 124 ++
.../fnexecution/wire/ByteStringCoderTest.java | 135 ++
runners/samza/build.gradle | 4 +-
.../beam/runners/spark/SparkPipelineOptions.java | 12 +
.../beam/runners/spark/SparkPipelineRunner.java | 58 +-
.../runners/spark/metrics/SparkBeamMetric.java | 24 +-
.../runners/spark/metrics/SparkBeamMetricTest.java | 2 +-
sdks/go/build.gradle | 5 +-
sdks/go/container/boot.go | 5 +-
sdks/go/pkg/beam/artifact/materialize.go | 115 +-
sdks/go/pkg/beam/artifact/materialize_test.go | 9 +-
sdks/go/pkg/beam/core/graph/edge.go | 26 +
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 29 +-
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 20 +-
sdks/go/pkg/beam/core/runtime/pipelinex/clone.go | 1 +
sdks/go/pkg/beam/core/runtime/xlangx/resolve.go | 12 +-
sdks/go/pkg/beam/external.go | 65 +-
sdks/go/pkg/beam/external_test.go | 92 +
sdks/go/pkg/beam/transforms/stats/quantiles.go | 711 ++++++++
.../go/pkg/beam/transforms/stats/quantiles_test.go | 275 +++
sdks/go/pkg/beam/util.go | 8 +
sdks/go/test/run_integration_tests.sh | 2 +-
.../src/main/resources/beam/suppressions.xml | 3 +-
sdks/java/container/boot.go | 9 +-
.../container/license_scripts/dep_urls_java.yaml | 4 +
sdks/java/core/build.gradle | 2 +-
.../org/apache/beam/sdk/schemas/io/GenericDlq.java | 17 +-
.../beam/sdk/schemas/io/GenericDlqProvider.java | 6 +-
.../io/{GenericDlqProvider.java => Providers.java} | 36 +-
.../AvroPayloadSerializerProvider.java} | 26 +-
.../io/payloads/JsonPayloadSerializerProvider.java | 52 +
.../PayloadSerializer.java} | 35 +-
.../PayloadSerializerProvider.java} | 23 +-
.../PayloadSerializers.java} | 31 +-
.../sdk/schemas/io/payloads/package-info.java} | 30 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 5 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 68 +-
.../org/apache/beam/sdk/util/RowJsonUtils.java | 2 +-
.../io/AvroPayloadSerializerProviderTest.java | 64 +
.../io/JsonPayloadSerializerProviderTest.java | 60 +
sdks/java/extensions/protobuf/build.gradle | 1 +
.../protobuf/ProtoPayloadSerializerProvider.java | 84 +
.../ProtoPayloadSerializerProviderTest.java | 110 ++
.../src/test/proto/payload_messages.proto} | 2 +-
sdks/java/extensions/sql/build.gradle | 7 +-
.../impl/transform/BeamBuiltinAggregations.java | 2 +
.../extensions/sql/impl/transform/agg/CountIf.java | 73 +
.../meta/provider/kafka/BeamKafkaAvroTable.java | 89 -
.../meta/provider/kafka/BeamKafkaJsonTable.java | 107 --
.../meta/provider/kafka/BeamKafkaProtoTable.java | 119 --
.../meta/provider/kafka/BeamKafkaThriftTable.java | 138 --
.../meta/provider/kafka/KafkaTableProvider.java | 86 +-
.../kafka/PayloadSerializerKafkaTable.java | 69 +
.../extensions/sql/BeamSqlDslAggregationTest.java | 28 +
.../provider/kafka/BeamKafkaTableAvroTest.java | 12 +-
.../provider/kafka/BeamKafkaTableJsonTest.java | 12 +-
.../provider/kafka/BeamKafkaTableProtoTest.java | 41 +-
.../provider/kafka/BeamKafkaTableThriftTest.java | 37 +-
.../meta/provider/kafka/KafkaTableProviderIT.java | 7 +-
.../provider/kafka/KafkaTableProviderTest.java | 17 +-
.../extensions/sql/udf-test-provider/build.gradle | 7 +-
.../extensions/sql/provider/UdfTestProvider.java | 40 +-
sdks/java/extensions/sql/zetasql/build.gradle | 7 +
.../sql/zetasql/BeamJavaUdfCalcRule.java | 2 +-
.../sql/zetasql/BeamZetaSqlCalcRule.java | 2 +-
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java | 54 +-
.../zetasql/SupportedZetaSqlBuiltinFunctions.java | 5 +-
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 61 +-
.../sql/zetasql/ZetaSQLQueryPlanner.java | 48 +-
.../zetasql/translation/ExpressionConverter.java | 28 +-
.../translation/SqlOperatorMappingTable.java | 1 +
.../sql/zetasql/translation/SqlOperators.java | 7 +
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 18 +
.../extensions/sql/zetasql/ZetaSqlJavaUdfTest.java | 265 +++
.../sdk/extensions/sql/zetasql/ZetaSqlUdfTest.java | 26 +
.../fn/harness/control/ProcessBundleHandler.java | 59 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 130 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 17 +-
.../gcp/bigquery/WriteGroupedRecordsToFiles.java | 21 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessageToRow.java | 98 +-
.../sdk/io/gcp/pubsub/PubsubSchemaIOProvider.java | 49 +-
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 60 +-
.../sdk/io/gcp/testing/FakeDatasetService.java | 10 +
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 69 +
.../beam/sdk/io/gcp/healthcare/FhirIOLROIT.java | 2 +
.../beam/sdk/io/gcp/healthcare/FhirIOWriteIT.java | 2 +
.../sdk/io/gcp/pubsub/PubsubMessageToRowTest.java | 40 +-
.../org/apache/beam/sdk/io/kafka/ConsumerSpEL.java | 97 +-
.../beam/sdk/io/kafka/KafkaCommitOffset.java | 2 -
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 164 +-
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 20 +-
.../beam/sdk/io/kafka/KafkaUnboundedSource.java | 4 +
.../org/apache/beam/sdk/io/kafka/KafkaWriter.java | 7 +-
.../beam/sdk/io/kafka/ReadFromKafkaDoFn.java | 22 +-
.../beam/sdk/io/kafka/TopicPartitionCoder.java | 56 +
.../sdk/io/kafka/WatchKafkaTopicPartitionDoFn.java | 156 ++
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 117 +-
.../sdk/io/kafka/TopicPartitionCoderTest.java} | 35 +-
.../io/kafka/WatchKafkaTopicPartitionDoFnTest.java | 422 +++++
sdks/java/io/mongodb/build.gradle | 6 +-
sdks/java/io/snowflake/build.gradle | 5 +-
.../io/snowflake/expansion-service/build.gradle | 4 +-
sdks/java/io/thrift/build.gradle | 1 +
.../io/thrift/ThriftPayloadSerializerProvider.java | 107 ++
.../ThriftPayloadSerializerProviderTest.java | 154 ++
.../sdk/io/thrift/payloads}/ItThriftMessage.java | 56 +-
.../io/thrift/payloads}/SimpleThriftMessage.java | 51 +-
.../sdk/io/thrift/payloads}/TestThriftMessage.java | 90 +-
.../sdk/io/thrift/payloads}/thrift_test.thrift | 0
.../thrift/src/test/thrift/payload.thrift} | 6 +-
.../maven-archetypes/examples/generate-sources.sh | 15 +-
sdks/java/testing/nexmark/build.gradle | 1 +
.../org/apache/beam/sdk/nexmark/NexmarkUtils.java | 3 +-
sdks/python/apache_beam/coders/coder_impl.py | 8 +-
sdks/python/apache_beam/coders/coders.py | 16 +-
.../apache_beam/coders/coders_test_common.py | 57 +-
sdks/python/apache_beam/coders/row_coder_test.py | 6 +-
sdks/python/apache_beam/coders/slow_stream.py | 22 +-
sdks/python/apache_beam/dataframe/doctests_test.py | 2 -
sdks/python/apache_beam/dataframe/expressions.py | 32 +-
sdks/python/apache_beam/dataframe/frame_base.py | 14 +-
sdks/python/apache_beam/dataframe/frames.py | 11 +-
sdks/python/apache_beam/dataframe/frames_test.py | 100 +-
sdks/python/apache_beam/dataframe/io_test.py | 21 +-
.../apache_beam/dataframe/pandas_doctests_test.py | 17 +-
sdks/python/apache_beam/dataframe/schemas.py | 21 +-
sdks/python/apache_beam/dataframe/schemas_test.py | 26 +-
.../apache_beam/examples/snippets/snippets_test.py | 38 +-
.../transforms/aggregation/combineglobally_test.py | 21 -
.../transforms/aggregation/groupby_test.py | 3 -
.../snippets/transforms/aggregation/top_test.py | 7 -
.../snippets/transforms/elementwise/pardo_test.py | 15 +-
sdks/python/apache_beam/internal/pickler.py | 10 +-
.../apache_beam/io/aws/clients/s3/boto3_client.py | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 5 +-
sdks/python/apache_beam/io/gcp/spanner.py | 2 +-
sdks/python/apache_beam/io/parquetio_test.py | 2 +
.../runners/dataflow/dataflow_runner.py | 104 +-
.../runners/dataflow/dataflow_runner_test.py | 1 +
.../internal/clients/cloudbuild/__init__.py} | 29 +-
.../clients/cloudbuild/cloudbuild_v1_client.py | 686 +++++++
.../clients/cloudbuild/cloudbuild_v1_messages.py | 1911 ++++++++++++++++++++
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/dataflow/ptransform_overrides.py | 27 -
.../apache_beam/runners/direct/direct_runner.py | 11 +-
.../interactive/interactive_environment_test.py | 114 +-
.../Darwin/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 746483 -> 748019 bytes
.../Linux/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 703677 -> 728121 bytes
.../integration/tests/init_square_cube_test.py | 2 +-
.../runners/portability/flink_runner_test.py | 8 +
.../runners/portability/fn_api_runner/execution.py | 241 ++-
.../runners/portability/fn_api_runner/fn_runner.py | 1 -
.../portability/fn_api_runner/fn_runner_test.py | 37 +
.../portability/fn_api_runner/translations.py | 101 +-
.../portability/fn_api_runner/translations_test.py | 67 +-
.../portability/fn_api_runner/worker_handlers.py | 5 +-
.../runners/portability/local_job_service.py | 10 +-
.../runners/portability/portable_runner.py | 60 +-
.../runners/portability/portable_runner_test.py | 13 +
.../runners/portability/sdk_container_builder.py | 61 +-
.../runners/portability/spark_runner_test.py | 3 +
.../apache_beam/runners/portability/stager.py | 162 +-
.../apache_beam/runners/portability/stager_test.py | 19 +
.../apache_beam/runners/worker/bundle_processor.py | 43 +
.../apache_beam/runners/worker/sdk_worker_main.py | 2 +
.../testing/benchmarks/nexmark/nexmark_launcher.py | 10 +-
.../testing/benchmarks/nexmark/queries/query0.py | 2 +-
.../testing/benchmarks/nexmark/queries/query1.py | 2 +-
.../testing/benchmarks/nexmark/queries/query10.py | 4 +-
.../testing/benchmarks/nexmark/queries/query11.py | 2 +-
.../testing/benchmarks/nexmark/queries/query12.py | 2 +-
.../testing/benchmarks/nexmark/queries/query2.py | 2 +-
.../testing/benchmarks/nexmark/queries/query3.py | 2 +-
.../testing/benchmarks/nexmark/queries/query4.py | 2 +-
.../testing/benchmarks/nexmark/queries/query5.py | 2 +-
.../testing/benchmarks/nexmark/queries/query6.py | 2 +-
.../testing/benchmarks/nexmark/queries/query7.py | 2 +-
.../testing/benchmarks/nexmark/queries/query8.py | 2 +-
.../testing/benchmarks/nexmark/queries/query9.py | 2 +-
.../apache_beam/testing/load_tests/load_test.py | 3 +-
.../testing/load_tests/load_test_metrics_utils.py | 24 +-
.../testing/load_tests/microbenchmarks_test.py | 85 +
.../apache_beam/testing/synthetic_pipeline_test.py | 9 +-
.../tools/fn_api_runner_microbenchmark.py | 7 +-
.../transforms/combinefn_lifecycle_test.py | 3 +
sdks/python/apache_beam/transforms/environments.py | 14 +-
sdks/python/container/base_image_requirements.txt | 6 +-
sdks/python/container/boot.go | 22 +-
.../container/license_scripts/dep_urls_py.yaml | 2 +
sdks/python/container/piputil.go | 3 +-
sdks/python/pytest.ini | 2 +
sdks/python/setup.py | 12 +-
sdks/python/test-suites/tox/common.gradle | 3 +-
sdks/python/test-suites/tox/py38/build.gradle | 2 +
sdks/python/tox.ini | 15 +-
settings.gradle | 226 ---
settings.gradle.kts | 226 +++
vendor/sdks-java-extensions-protobuf/build.gradle | 62 -
.../www/site/content/en/blog/timely-processing.md | 9 +-
.../site/content/en/contribute/release-guide.md | 546 +++---
.../content/en/documentation/dsls/sql/shell.md | 2 +-
.../en/documentation/dsls/sql/walkthrough.md | 6 +-
.../content/en/documentation/programming-guide.md | 6 +-
.../en/documentation/sdks/java/testing/nexmark.md | 2 +-
.../content/en/get-started/wordcount-example.md | 4 +-
website/www/site/data/en/committer_traits.yaml | 6 +-
246 files changed, 10344 insertions(+), 3026 deletions(-)
create mode 100644 .test-infra/jenkins/job_LoadTests_FnApiRunner_Python.groovy
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/Python_FnApiRunner_ubenchmarks.json
delete mode 100644 build.gradle
create mode 100644 build.gradle.kts
delete mode 100644 buildSrc/build.gradle
create mode 100644 buildSrc/build.gradle.kts
rename release/{build.gradle => build.gradle.kts} (53%)
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/GroupIntoBatchesTranslation.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/GroupIntoBatchesTranslationTest.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/wire/ByteStringCoder.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/wire/ByteStringCoderTest.java
create mode 100644 sdks/go/pkg/beam/external_test.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/quantiles.go
create mode 100644 sdks/go/pkg/beam/transforms/stats/quantiles_test.go
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{GenericDlqProvider.java => Providers.java} (52%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{GenericDlqProvider.java => payloads/AvroPayloadSerializerProvider.java} (61%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/payloads/JsonPayloadSerializerProvider.java
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{GenericDlqProvider.java => payloads/PayloadSerializer.java} (57%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{GenericDlqProvider.java => payloads/PayloadSerializerProvider.java} (65%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/io/{GenericDlqProvider.java => payloads/PayloadSerializers.java} (52%)
copy sdks/java/{extensions/sql/src/test/proto/kafka/kafka_messages.proto => core/src/main/java/org/apache/beam/sdk/schemas/io/payloads/package-info.java} (63%)
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/AvroPayloadSerializerProviderTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/io/JsonPayloadSerializerProviderTest.java
create mode 100644 sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoPayloadSerializerProvider.java
create mode 100644 sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoPayloadSerializerProviderTest.java
rename sdks/java/extensions/{sql/src/test/proto/kafka/kafka_messages.proto => protobuf/src/test/proto/payload_messages.proto} (93%)
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/transform/agg/CountIf.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaAvroTable.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaJsonTable.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaProtoTable.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaThriftTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/PayloadSerializerKafkaTable.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlJavaUdfTest.java
create mode 100644 sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/TopicPartitionCoder.java
create mode 100644 sdks/java/io/kafka/src/main/java/org/apache/beam/sdk/io/kafka/WatchKafkaTopicPartitionDoFn.java
copy sdks/java/{extensions/sql/udf-test-provider/src/main/java/org/apache/beam/sdk/extensions/sql/provider/UdfTestProvider.java => io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/TopicPartitionCoderTest.java} (52%)
create mode 100644 sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/WatchKafkaTopicPartitionDoFnTest.java
create mode 100644 sdks/java/io/thrift/src/main/java/org/apache/beam/sdk/io/thrift/ThriftPayloadSerializerProvider.java
create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/ThriftPayloadSerializerProviderTest.java
rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift => io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/payloads}/ItThriftMessage.java (90%)
rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift => io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/payloads}/SimpleThriftMessage.java (90%)
rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift => io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/payloads}/TestThriftMessage.java (90%)
rename sdks/java/io/thrift/src/test/resources/{thrift => org/apache/beam/sdk/io/thrift/payloads}/thrift_test.thrift (100%)
rename sdks/java/{extensions/sql/src/test/thrift/kafka/messages.thrift => io/thrift/src/test/thrift/payload.thrift} (86%)
copy sdks/python/apache_beam/runners/{interactive/testing/integration/tests/init_square_cube_test.py => dataflow/internal/clients/cloudbuild/__init__.py} (56%)
create mode 100644 sdks/python/apache_beam/runners/dataflow/internal/clients/cloudbuild/cloudbuild_v1_client.py
create mode 100644 sdks/python/apache_beam/runners/dataflow/internal/clients/cloudbuild/cloudbuild_v1_messages.py
create mode 100644 sdks/python/apache_beam/testing/load_tests/microbenchmarks_test.py
delete mode 100644 settings.gradle
create mode 100644 settings.gradle.kts
delete mode 100644 vendor/sdks-java-extensions-protobuf/build.gradle