You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/08/22 02:46:26 UTC
[beam] tag nightly-master updated (cfa448d -> bdb71bc)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/beam.git.
*** WARNING: tag nightly-master was modified! ***
from cfa448d (commit)
to bdb71bc (commit)
from cfa448d [BEAM-9546] DataframeTransform can now consume a schema-aware PCollection (#11980)
add 081ad8c [BEAM-10049] Add licenses for go dependencies in python, java, and go SDK containers
add 209bc21 Merge pull request #12650 from alanmyrvold/licensesGo
add 0040865 Propagate BigQuery streaming insert throttled time to Dataflow worker in Java SDK (#12403)
add 2489cf1 Color Jenkins logs support
add 6b472e1 Merge pull request #11112: Color Jenkins logs support
add cdc6778 Added a whitespace lint precommit job.
add e81c6f4 Removed trailing whitespaces from all markdown and gradle build files.
add 344599e Updated jenkins precommit job README, added comments of what needs to be linted.
add 4e573fd Removed trailing whitespaces from HEAD changes
add dfb5cd5 Merge pull request #12444 from [BEAM-10771] Added a whitespace lint as part of python lint precommit
add 21de8ba [BEAM-10764] Make is_in_ipython robust (#12641)
add e6db419 [BEAM-10766] Fix flake where ByteString coercion could modify cached type hints (#12654)
add 7c6dfd2 [BEAM-10751] Revert "Extending archiveJunit post-commit task with stability history"
add f82ae1c Merge pull request #12664: [BEAM-10751] Revert "Extending archiveJunit post-commit task with sta…
add d7972c9 Use setCoder, not setSchema (#12662)
add 3d01eed [BEAM-10771] Added Whitespacelint job to pull request template
add 3d0f7dc Merge pull request #12665 from [BEAM-10771] Added Whitespacelint job to pull request template
add 60c1e8d [BEAM-10703] Prepare Dataflow Java runner for shardable states (#12578)
add c9bed1e [BEAM-10549] Improve runtime type checking performance for the Python SDK (#12352)
add 5e783c6 [BEAM-10788] Also disable immutability checking for transforms within Read. (#12660)
add 7608c2d [BEAM-5715, BEAM-8862, BEAM-8702] Removes all references to grpc_all and replaces them with more specific dependencies. Also fixes a few other dependency errors. (#12508)
add 665a94b [BEAM-10460] Remove SparkPortableExecutionTest altogether.
add f8b77e0 Merge pull request #12668 from ibzib/BEAM-10460-2
add d3fdda0 [BEAM-10761] Prefer TestPubsub#assertSubscriptionEventuallyExists when waiting to inject data (#12634)
add b57aab0 Fix bug in JoinScanWithRefConverter
add 0ade7bd Merge pull request #12648 from robinyqiu/with-join
add ba89b85 [BEAM-10654] Implemented ExternalSchemaIOTransformRegistrar for jdbc (#12498)
add 66055db Merge pull request #12580 from [BEAM-2855] nexmark python suite implement implement query 3, 4, 5, 6, 7, 8, 11
add bdb71bc [BEAM-9918] Adding tests and documentation to xlang components (#12667)
No new revisions were added by this update.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 8 +-
.test-infra/dockerized-jenkins/README.md | 8 +-
.test-infra/jenkins/CommonJobProperties.groovy | 9 +-
.test-infra/jenkins/README.md | 1 +
.../job_PerformanceTests_BigQueryIO_Java.groovy | 2 +-
.../jenkins/job_PerformanceTests_Python.groovy | 2 +-
.../jenkins/job_PerformanceTests_SQLIO_Java.groovy | 2 +-
...mmit_CrossLanguageValidatesRunner_Direct.groovy | 2 +-
...ommit_CrossLanguageValidatesRunner_Flink.groovy | 2 +-
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Java.groovy | 2 +-
...PostCommit_Java_Dataflow_Examples_Java11.groovy | 2 +-
...Java_PortableValidatesRunner_Flink_Batch.groovy | 2 +-
..._PortableValidatesRunner_Flink_Streaming.groovy | 2 +-
...Java_PortableValidatesRunner_Spark_Batch.groovy | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow.groovy | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Java11.groovy | 2 +-
...b_PostCommit_Java_ValidatesRunner_Direct.groovy | 2 +-
...ommit_Java_ValidatesRunner_Direct_Java11.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 2 +-
...Commit_Java_ValidatesRunner_Flink_Java11.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Samza.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Spark.groovy | 2 +-
...ValidatesRunner_SparkStructuredStreaming.groovy | 2 +-
...PostCommit_Java_ValidatesRunner_Twister2.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Python2.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Python35.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Python36.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Python37.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Python38.groovy | 2 +-
...ommit_Python_ValidatesContainer_Dataflow.groovy | 2 +-
...stCommit_Python_ValidatesRunner_Dataflow.groovy | 2 +-
...ommit_Python_ValidatesRunner_Dataflow_V2.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_SQL.groovy | 2 +-
.test-infra/jenkins/job_PreCommit_Java.groovy | 3 +-
.../job_PreCommit_Java_Examples_Dataflow.groovy | 3 +-
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 2 +-
.../job_PreCommit_Java_PortabilityApi.groovy | 3 +-
...job_PreCommit_Java_PortabilityApi_Java11.groovy | 2 +-
.test-infra/jenkins/job_PreCommit_Python.groovy | 3 +-
.test-infra/jenkins/job_PreCommit_SQL.groovy | 3 +-
.../jenkins/job_PreCommit_SQL_Java11.groovy | 2 +-
...thon.groovy => job_PreCommit_Whitespace.groovy} | 18 +-
.test-infra/jenkins/job_sonarqube_report.groovy | 2 +-
.test-infra/kubernetes/kafka-cluster/README.md | 4 +-
.test-infra/metrics/README.md | 2 +-
CHANGES.md | 22 +-
CI.md | 10 +-
README.md | 2 +-
build.gradle | 6 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 6 +-
learning/katas/README.md | 6 +-
learning/katas/go/README.md | 4 +-
.../go/common_transforms/filter/filter/task.md | 2 +-
.../go/common_transforms/filter/pardo/task.md | 4 +-
.../additional_outputs/additional_outputs/task.md | 10 +-
.../cogroupbykey/cogroupbykey/task.md | 30 +-
.../core_transforms/combine/combine_perkey/task.md | 6 +-
.../go/core_transforms/combine/combinefn/task.md | 22 +-
.../combine/simple_function/task.md | 12 +-
.../go/core_transforms/composite/composite/task.md | 10 +-
.../go/core_transforms/flatten/flatten/task.md | 8 +-
.../core_transforms/groupbykey/groupbykey/task.md | 8 +-
.../katas/go/core_transforms/map/pardo/task.md | 6 +-
.../go/core_transforms/map/pardo_struct/task.md | 2 +-
.../go/core_transforms/partition/partition/task.md | 6 +-
.../core_transforms/side_input/side_input/task.md | 30 +-
.../go/core_transforms/windowing/windowing/task.md | 14 +-
.../java/Common Transforms/Filter/Filter/task.md | 2 +-
.../java/Common Transforms/Filter/ParDo/task.md | 2 +-
.../Common Transforms/WithKeys/WithKeys/task.md | 2 +-
.../Core Transforms/Branching/Branching/task.md | 4 +-
.../CoGroupByKey/CoGroupByKey/task.md | 8 +-
.../Combine/BinaryCombineFn Lambda/task.md | 4 +-
.../Combine/BinaryCombineFn/task.md | 14 +-
.../Core Transforms/Combine/Combine PerKey/task.md | 10 +-
.../java/Core Transforms/Combine/CombineFn/task.md | 18 +-
.../Combine/Simple Function/task.md | 12 +-
.../Composite Transform/task.md | 20 +-
.../DoFn Additional Parameters/task.md | 16 +-
.../java/Core Transforms/Flatten/Flatten/task.md | 6 +-
.../Core Transforms/GroupByKey/GroupByKey/task.md | 12 +-
.../Core Transforms/Map/FlatMapElements/task.md | 4 +-
.../java/Core Transforms/Map/MapElements/task.md | 2 +-
.../Core Transforms/Map/ParDo OneToMany/task.md | 2 +-
.../katas/java/Core Transforms/Map/ParDo/task.md | 6 +-
.../Core Transforms/Partition/Partition/task.md | 12 +-
.../Core Transforms/Side Input/Side Input/task.md | 14 +-
.../Side Output/Side Output/task.md | 6 +-
.../java/IO/Built-in IOs/Built-in IOs/task.md | 4 +-
learning/katas/java/IO/TextIO/TextIO Read/task.md | 10 +-
.../Introduction/Hello Beam/Hello Beam/task.md | 16 +-
learning/katas/java/README.md | 4 +-
.../Triggers/Early Triggers/Early Triggers/task.md | 8 +-
.../Event Time Triggers/task.md | 20 +-
.../Window Accumulation Mode/task.md | 10 +-
.../java/Windowing/Adding Timestamp/ParDo/task.md | 4 +-
.../Adding Timestamp/WithTimestamps/task.md | 4 +-
.../Fixed Time Window/Fixed Time Window/task.md | 16 +-
learning/katas/java/build.gradle | 6 +-
.../kotlin/Common Transforms/Filter/Filter/task.md | 2 +-
.../kotlin/Common Transforms/Filter/ParDo/task.md | 2 +-
.../Common Transforms/WithKeys/WithKeys/task.md | 2 +-
.../Core Transforms/Branching/Branching/task.md | 4 +-
.../CoGroupByKey/CoGroupByKey/task.md | 8 +-
.../Combine/BinaryCombineFn Lambda/task.md | 4 +-
.../Combine/BinaryCombineFn/task.md | 14 +-
.../Core Transforms/Combine/Combine PerKey/task.md | 10 +-
.../Core Transforms/Combine/CombineFn/task.md | 18 +-
.../Combine/Simple Function/task.md | 12 +-
.../Composite Transform/task.md | 20 +-
.../DoFn Additional Parameters/task.md | 16 +-
.../kotlin/Core Transforms/Flatten/Flatten/task.md | 6 +-
.../Core Transforms/GroupByKey/GroupByKey/task.md | 12 +-
.../Core Transforms/Map/FlatMapElements/task.md | 4 +-
.../kotlin/Core Transforms/Map/MapElements/task.md | 2 +-
.../Core Transforms/Map/ParDo OneToMany/task.md | 2 +-
.../katas/kotlin/Core Transforms/Map/ParDo/task.md | 6 +-
.../Core Transforms/Partition/Partition/task.md | 12 +-
.../Side Input/Side Input/src/task.md | 14 +-
.../Core Transforms/Side Input/Side Input/task.md | 14 +-
.../Side Output/Side Output/src/task.md | 6 +-
.../Side Output/Side Output/task.md | 6 +-
.../kotlin/IO/Built-in IOs/Built-in IOs/task.md | 4 +-
.../katas/kotlin/IO/TextIO/TextIO Read/task.md | 10 +-
.../Introduction/Hello Beam/Hello Beam/task.md | 16 +-
learning/katas/kotlin/README.md | 4 +-
.../Triggers/Early Triggers/Early Triggers/task.md | 8 +-
.../Event Time Triggers/task.md | 20 +-
.../Window Accumulation Mode/task.md | 10 +-
.../Windowing/Adding Timestamp/ParDo/task.md | 4 +-
.../Adding Timestamp/WithTimestamps/task.md | 4 +-
.../Fixed Time Window/Fixed Time Window/task.md | 16 +-
learning/katas/kotlin/build.gradle | 6 +-
.../python/Common Transforms/Filter/Filter/task.md | 2 +-
.../python/Common Transforms/Filter/ParDo/task.md | 2 +-
.../Common Transforms/WithKeys/WithKeys/task.md | 2 +-
.../Core Transforms/Branching/Branching/task.md | 4 +-
.../CoGroupByKey/CoGroupByKey/task.md | 8 +-
.../Core Transforms/Combine/Combine PerKey/task.md | 6 +-
.../Core Transforms/Combine/CombineFn/task.md | 14 +-
.../Combine/Simple Function/task.md | 10 +-
.../Composite Transform/task.md | 18 +-
.../python/Core Transforms/Flatten/Flatten/task.md | 6 +-
.../Core Transforms/GroupByKey/GroupByKey/task.md | 12 +-
.../python/Core Transforms/Map/FlatMap/task.md | 4 +-
.../katas/python/Core Transforms/Map/ParDo/task.md | 2 +-
.../Core Transforms/Partition/Partition/task.md | 12 +-
.../Core Transforms/Side Input/Side Input/task.md | 14 +-
.../Side Output/Side Output/task.md | 6 +-
.../python/IO/Built-in IOs/Built-in IOs/task.md | 4 +-
.../katas/python/IO/TextIO/ReadFromText/task.md | 10 +-
.../Introduction/Hello Beam/Hello Beam/task.md | 14 +-
learning/katas/python/README.md | 4 +-
.../Windowing/Adding Timestamp/ParDo/task.md | 4 +-
.../Fixed Time Window/Fixed Time Window/task.md | 16 +-
.../container => release/go-licenses}/Dockerfile | 21 +-
.../go-licenses/common.gradle | 36 +-
release/go-licenses/get-licenses.sh | 26 +
.../py38 => release/go-licenses/go}/build.gradle | 20 +-
.../py38 => release/go-licenses/java}/build.gradle | 20 +-
.../py38 => release/go-licenses/py}/build.gradle | 20 +-
.../apache/beam/runners/direct/DirectRunner.java | 6 +-
.../direct/ImmutabilityEnforcementFactory.java | 2 +-
.../examples/build.gradle | 2 +-
.../dataflow/worker/BatchModeExecutionContext.java | 28 +-
.../worker/MetricTrackingWindmillServerStub.java | 21 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 48 +-
.../dataflow/worker/StreamingDataflowWorker.java | 118 +++--
.../worker/StreamingModeExecutionContext.java | 22 +-
.../dataflow/worker/WindmillStateCache.java | 57 +-
.../worker/BatchModeExecutionContextTest.java | 8 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 46 +-
.../worker/StreamingDataflowWorkerTest.java | 213 ++++++--
.../dataflow/worker/WindmillStateCacheTest.java | 136 ++++-
.../worker/WindmillStateInternalsTest.java | 9 +-
.../worker/windmill/src/main/proto/windmill.proto | 1 +
runners/jet/build.gradle | 2 +-
.../runners/spark/SparkPortableExecutionTest.java | 195 -------
sdks/go/BUILD.md | 4 +-
sdks/go/README.md | 12 +-
sdks/go/container/Dockerfile | 12 +
sdks/go/container/build.gradle | 13 +-
sdks/go/examples/xlang/cogroup_by/cogroup_by.go | 107 ++++
sdks/go/examples/xlang/combine/combine.go | 101 ++++
.../xlang/combine_globally/combine_globally.go | 79 +++
sdks/go/examples/xlang/flatten/flatten.go | 81 +++
sdks/go/examples/xlang/group_by/group_by.go | 105 ++++
sdks/go/examples/xlang/multi_input_output/multi.go | 74 +++
sdks/go/examples/xlang/partition/partition.go | 83 +++
sdks/go/examples/xlang/wordcount/wordcount.go | 50 +-
sdks/go/pkg/beam/core/graph/edge.go | 33 +-
sdks/go/pkg/beam/core/graph/xlang.go | 43 +-
sdks/go/pkg/beam/core/graph/xlang_test.go | 86 +++
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 5 +-
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 10 +
sdks/go/pkg/beam/core/runtime/graphx/xlang_test.go | 192 +++++++
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 1 +
sdks/go/pkg/beam/core/runtime/xlangx/namespace.go | 43 +-
.../pkg/beam/core/runtime/xlangx/namespace_test.go | 579 +++++++++++++++++++++
sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 60 ++-
sdks/go/pkg/beam/runners/universal/universal.go | 3 +
sdks/go/pkg/beam/xlang.go | 186 +++++--
sdks/java/container/Dockerfile | 13 +
sdks/java/container/Dockerfile-java11 | 12 +
sdks/java/container/build.gradle | 9 +
sdks/java/core/build.gradle | 2 +-
.../org/apache/beam/sdk/schemas/io/SchemaIO.java | 2 +-
.../beam/sdk/schemas/transforms/Convert.java | 7 +-
sdks/java/extensions/euphoria/README.md | 2 +-
.../gcp/util/RetryHttpRequestInitializer.java | 6 +-
.../schemaio-expansion-service}/build.gradle | 39 +-
.../ExternalSchemaIOTransformRegistrar.java | 150 ++++++
.../schemaio/expansion/package-info.java | 24 +-
.../ExternalSchemaIOTransformRegistrarTest.java | 180 +++++++
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 4 +-
.../provider/SchemaIOTableProviderWrapper.java | 2 +-
.../sdk/extensions/sql/PubsubToBigqueryIT.java | 20 +-
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 45 +-
sdks/java/extensions/sql/zetasql/build.gradle | 2 +-
.../zetasql/translation/ExpressionConverter.java | 97 ----
.../sql/zetasql/translation/JoinScanConverter.java | 4 +-
.../translation/JoinScanWithRefConverter.java | 100 ----
.../translation/QueryStatementConverter.java | 1 -
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 34 ++
sdks/java/io/azure/README.md | 4 +-
sdks/java/io/google-cloud-platform/build.gradle | 5 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 15 +
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 6 +-
.../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 24 +-
.../beam/sdk/io/jdbc/JdbcSchemaIOProvider.java | 188 +++++++
.../beam/sdk/io/jdbc/JdbcTransformsRegistrar.java | 165 ------
sdks/java/io/xml/build.gradle | 4 +-
sdks/java/maven-archetypes/examples/build.gradle | 2 +-
.../apache_beam/examples/snippets/snippets.py | 3 +-
.../io/external/xlang_jdbcio_it_test.py | 6 +-
sdks/python/apache_beam/io/jdbc.py | 113 ++--
.../python/apache_beam/options/pipeline_options.py | 7 +
sdks/python/apache_beam/pipeline.py | 16 +
sdks/python/apache_beam/runners/common.py | 3 +
.../apache_beam/runners/worker/bundle_processor.py | 3 +-
.../apache_beam/runners/worker/opcounters.pxd | 2 +
.../apache_beam/runners/worker/opcounters.py | 26 +-
.../apache_beam/runners/worker/operations.py | 66 ++-
.../benchmarks/nexmark/models/auction_bid.py | 3 +
.../benchmarks/nexmark/models/auction_price.py | 56 --
.../benchmarks/nexmark/models/nexmark_model.py | 9 +
.../testing/benchmarks/nexmark/nexmark_launcher.py | 37 +-
.../testing/benchmarks/nexmark/nexmark_util.py | 3 -
.../nexmark/queries/nexmark_query_util.py | 39 +-
.../testing/benchmarks/nexmark/queries/query11.py | 56 ++
.../testing/benchmarks/nexmark/queries/query2.py | 8 +-
.../testing/benchmarks/nexmark/queries/query3.py | 158 ++++++
.../testing/benchmarks/nexmark/queries/query4.py | 81 +++
.../testing/benchmarks/nexmark/queries/query5.py | 97 ++++
.../testing/benchmarks/nexmark/queries/query6.py | 92 ++++
.../testing/benchmarks/nexmark/queries/query7.py | 59 +++
.../testing/benchmarks/nexmark/queries/query8.py | 75 +++
.../tools/runtime_type_check_microbenchmark.py | 131 +++++
sdks/python/apache_beam/transforms/core.py | 5 +
sdks/python/apache_beam/transforms/external.py | 12 +-
.../python/apache_beam/transforms/external_test.py | 7 +
sdks/python/apache_beam/transforms/ptransform.py | 8 +
.../apache_beam/transforms/ptransform_test.py | 44 +-
sdks/python/apache_beam/typehints/typecheck.py | 78 ++-
.../apache_beam/typehints/typecheck_test_py3.py | 208 +++++++-
.../apache_beam/typehints/typed_pipeline_test.py | 8 +-
.../typehints/typed_pipeline_test_py3.py | 8 +-
sdks/python/apache_beam/utils/interactive_utils.py | 19 +-
.../apache_beam/utils/interactive_utils_test.py | 68 +++
sdks/python/container/Dockerfile | 12 +
sdks/python/container/common.gradle | 9 +
.../container/license_scripts/pull_licenses_py.py | 1 -
sdks/python/scripts/run_tox.sh | 4 +
sdks/python/scripts/run_whitespacelint.sh | 59 +++
sdks/python/test-suites/portable/common.gradle | 4 +-
sdks/python/test-suites/tox/py38/build.gradle | 22 +
sdks/python/tox.ini | 9 +-
settings.gradle | 4 +
website/CONTRIBUTE.md | 6 +-
website/www/site/archetypes/blog.md | 4 +-
.../www/site/content/en/blog/added-apex-runner.md | 4 +-
website/www/site/content/en/blog/beam-2.10.0.md | 4 +-
website/www/site/content/en/blog/beam-2.11.0.md | 2 +-
website/www/site/content/en/blog/beam-2.13.0.md | 28 +-
website/www/site/content/en/blog/beam-2.14.0.md | 36 +-
website/www/site/content/en/blog/beam-2.18.0.md | 4 +-
website/www/site/content/en/blog/beam-2.19.0.md | 12 +-
website/www/site/content/en/blog/beam-2.20.0.md | 4 +-
website/www/site/content/en/blog/beam-2.21.0.md | 18 +-
website/www/site/content/en/blog/beam-2.23.0.md | 2 +-
website/www/site/content/en/blog/beam-2.5.0.md | 4 +-
website/www/site/content/en/blog/beam-2.7.0.md | 2 +-
.../content/en/blog/beam-first-stable-release.md | 6 +-
.../www/site/content/en/blog/beam-has-a-logo.md | 2 +-
.../www/site/content/en/blog/beam-kata-release.md | 32 +-
.../content/en/blog/beam-katas-kotlin-release.md | 20 +-
website/www/site/content/en/blog/beam-kotlin.md | 8 +-
.../site/content/en/blog/beam-summit-aftermath.md | 2 +-
.../content/en/blog/beam-summit-digital-2020.md | 2 +-
.../content/en/blog/beam-summit-europe-2019.md | 8 +-
.../www/site/content/en/blog/beam-summit-europe.md | 10 +-
.../www/site/content/en/blog/beam-summit-site.md | 2 +-
website/www/site/content/en/blog/first-release.md | 4 +-
website/www/site/content/en/blog/gsoc-19.md | 4 +-
website/www/site/content/en/blog/looping-timers.md | 2 +-
website/www/site/content/en/blog/python-typing.md | 32 +-
website/www/site/content/en/blog/season-of-docs.md | 4 +-
website/www/site/content/en/blog/six-months.md | 4 +-
.../www/site/content/en/blog/splittable-do-fn.md | 14 +-
.../site/content/en/blog/stateful-processing.md | 36 +-
.../www/site/content/en/blog/timely-processing.md | 10 +-
website/www/site/content/en/community/in-person.md | 2 +-
website/www/site/content/en/community/mascot.md | 4 +-
.../site/content/en/community/twitter-handle.md | 4 +-
.../content/en/contribute/become-a-committer.md | 2 +-
.../site/content/en/contribute/committer-guide.md | 4 +-
.../www/site/content/en/contribute/dependencies.md | 16 +-
.../site/content/en/contribute/design-documents.md | 2 +-
.../en/contribute/postcommits-policies-details.md | 6 +-
.../site/content/en/contribute/release-guide.md | 134 ++---
.../www/site/content/en/contribute/runner-guide.md | 2 +-
.../dsls/sql/calcite/lexical-structure.md | 2 +-
.../en/documentation/dsls/sql/calcite/overview.md | 4 +-
.../dsls/sql/extensions/user-defined-functions.md | 12 +-
.../sql/extensions/windowing-and-triggering.md | 24 +-
.../content/en/documentation/dsls/sql/overview.md | 6 +-
.../content/en/documentation/dsls/sql/shell.md | 6 +-
.../en/documentation/dsls/sql/walkthrough.md | 26 +-
.../en/documentation/dsls/sql/zetasql/overview.md | 8 +-
.../content/en/documentation/io/built-in/hadoop.md | 10 +-
.../en/documentation/io/built-in/hcatalog.md | 12 +-
.../en/documentation/io/built-in/parquet.md | 2 +-
.../en/documentation/io/built-in/snowflake.md | 28 +-
.../en/documentation/io/developing-io-java.md | 34 +-
.../en/documentation/io/developing-io-overview.md | 22 +-
.../en/documentation/io/developing-io-python.md | 6 +-
.../site/content/en/documentation/io/testing.md | 48 +-
.../en/documentation/patterns/ai-platform.md | 12 +-
.../en/documentation/patterns/bigqueryio.md | 6 +-
.../en/documentation/patterns/custom-windows.md | 2 +-
.../content/en/documentation/patterns/overview.md | 2 +-
.../content/en/documentation/patterns/schema.md | 8 +-
.../en/documentation/patterns/side-inputs.md | 2 +-
.../content/en/documentation/programming-guide.md | 564 ++++++++++----------
.../documentation/resources/learning-resources.md | 8 +-
.../documentation/resources/videos-and-podcasts.md | 8 +-
.../content/en/documentation/runners/direct.md | 2 +-
.../site/content/en/documentation/runners/jet.md | 34 +-
.../site/content/en/documentation/runners/nemo.md | 2 +-
.../site/content/en/documentation/runners/samza.md | 4 +-
.../site/content/en/documentation/runners/spark.md | 2 +-
.../en/documentation/runtime/environments.md | 12 +-
.../en/documentation/sdks/java-dependencies.md | 6 +-
.../en/documentation/sdks/java-thirdparty.md | 16 +-
.../content/en/documentation/sdks/java/euphoria.md | 22 +-
.../en/documentation/sdks/java/testing/nexmark.md | 6 +-
.../en/documentation/sdks/python-dependencies.md | 12 +-
.../en/documentation/sdks/python-type-safety.md | 14 +-
.../java/aggregation/approximatequantiles.md | 4 +-
.../java/aggregation/approximateunique.md | 4 +-
.../transforms/java/aggregation/cogroupbykey.md | 4 +-
.../transforms/java/aggregation/combine.md | 6 +-
.../java/aggregation/combinewithcontext.md | 2 +-
.../transforms/java/aggregation/count.md | 4 +-
.../transforms/java/aggregation/distinct.md | 4 +-
.../transforms/java/aggregation/groupbykey.md | 4 +-
.../java/aggregation/groupintobatches.md | 4 +-
.../transforms/java/aggregation/latest.md | 2 +-
.../transforms/java/aggregation/max.md | 4 +-
.../transforms/java/aggregation/mean.md | 4 +-
.../transforms/java/aggregation/min.md | 4 +-
.../transforms/java/aggregation/sample.md | 4 +-
.../transforms/java/aggregation/sum.md | 4 +-
.../transforms/java/aggregation/top.md | 4 +-
.../transforms/java/elementwise/filter.md | 4 +-
.../transforms/java/elementwise/flatmapelements.md | 6 +-
.../transforms/java/elementwise/keys.md | 4 +-
.../transforms/java/elementwise/kvswap.md | 4 +-
.../transforms/java/elementwise/mapelements.md | 8 +-
.../transforms/java/elementwise/pardo.md | 4 +-
.../transforms/java/elementwise/partition.md | 8 +-
.../transforms/java/elementwise/regex.md | 4 +-
.../transforms/java/elementwise/reify.md | 2 +-
.../transforms/java/elementwise/tostring.md | 4 +-
.../transforms/java/elementwise/values.md | 2 +-
.../transforms/java/elementwise/withkeys.md | 4 +-
.../transforms/java/elementwise/withtimestamps.md | 4 +-
.../documentation/transforms/java/other/create.md | 2 +-
.../documentation/transforms/java/other/flatten.md | 4 +-
.../documentation/transforms/java/other/passert.md | 6 +-
.../en/documentation/transforms/java/other/view.md | 4 +-
.../documentation/transforms/java/other/window.md | 4 +-
.../en/documentation/transforms/java/overview.md | 14 +-
.../python/aggregation/approximatequantiles.md | 4 +-
.../python/aggregation/approximateunique.md | 4 +-
.../python/aggregation/combinewithcontext.md | 4 +-
.../transforms/python/elementwise/flatmap.md | 4 +-
.../transforms/python/elementwise/reify.md | 6 +-
.../transforms/python/elementwise/withkeys.md | 4 +-
.../transforms/python/other/create.md | 4 +-
.../transforms/python/other/flatten.md | 6 +-
.../transforms/python/other/passert.md | 4 +-
.../transforms/python/other/reshuffle.md | 4 +-
.../documentation/transforms/python/other/view.md | 4 +-
.../transforms/python/other/windowinto.md | 4 +-
.../en/documentation/transforms/python/overview.md | 4 +-
.../site/content/en/get-started/quickstart-java.md | 28 +-
.../content/en/get-started/wordcount-example.md | 8 +-
website/www/site/content/en/roadmap/_index.md | 10 +-
.../content/en/roadmap/connectors-python-sdk.md | 8 +-
website/www/site/content/en/roadmap/euphoria.md | 2 +-
website/www/site/content/en/roadmap/go-sdk.md | 6 +-
website/www/site/content/en/roadmap/python-sdk.md | 6 +-
website/www/site/content/en/security/_index.md | 12 +-
415 files changed, 6145 insertions(+), 2681 deletions(-)
copy .test-infra/jenkins/{job_PreCommit_Python.groovy => job_PreCommit_Whitespace.groovy} (73%)
copy {sdks/go/container => release/go-licenses}/Dockerfile (71%)
copy sdks/python/test-suites/tox/py38/build.gradle => release/go-licenses/common.gradle (54%)
create mode 100755 release/go-licenses/get-licenses.sh
copy {sdks/python/test-suites/tox/py38 => release/go-licenses/go}/build.gradle (69%)
copy {sdks/python/test-suites/tox/py38 => release/go-licenses/java}/build.gradle (69%)
copy {sdks/python/test-suites/tox/py38 => release/go-licenses/py}/build.gradle (69%)
delete mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/SparkPortableExecutionTest.java
create mode 100644 sdks/go/examples/xlang/cogroup_by/cogroup_by.go
create mode 100644 sdks/go/examples/xlang/combine/combine.go
create mode 100644 sdks/go/examples/xlang/combine_globally/combine_globally.go
create mode 100644 sdks/go/examples/xlang/flatten/flatten.go
create mode 100644 sdks/go/examples/xlang/group_by/group_by.go
create mode 100644 sdks/go/examples/xlang/multi_input_output/multi.go
create mode 100644 sdks/go/examples/xlang/partition/partition.go
create mode 100644 sdks/go/pkg/beam/core/graph/xlang_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/xlang_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/namespace_test.go
copy sdks/{python/test-suites/tox/py38 => java/extensions/schemaio-expansion-service}/build.gradle (51%)
create mode 100644 sdks/java/extensions/schemaio-expansion-service/src/main/java/org/apache/beam/sdk/extensions/schemaio/expansion/ExternalSchemaIOTransformRegistrar.java
copy .test-infra/jenkins/job_PreCommit_Python.groovy => sdks/java/extensions/schemaio-expansion-service/src/main/java/org/apache/beam/sdk/extensions/schemaio/expansion/package-info.java (62%)
create mode 100644 sdks/java/extensions/schemaio-expansion-service/src/test/java/org/apache/beam/sdk/extensions/schemaio/expansion/ExternalSchemaIOTransformRegistrarTest.java
delete mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
create mode 100644 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcSchemaIOProvider.java
delete mode 100644 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcTransformsRegistrar.java
delete mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/models/auction_price.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query11.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query3.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query4.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query5.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query6.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query7.py
create mode 100644 sdks/python/apache_beam/testing/benchmarks/nexmark/queries/query8.py
create mode 100644 sdks/python/apache_beam/tools/runtime_type_check_microbenchmark.py
create mode 100644 sdks/python/apache_beam/utils/interactive_utils_test.py
create mode 100755 sdks/python/scripts/run_whitespacelint.sh