You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by bh...@apache.org on 2021/01/13 01:56:14 UTC
[beam] branch website-revamp updated (8b22fb4 -> 7d0d220)
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a change to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 8b22fb4 Merge remote-tracking branch 'origin/master' into website-revamp
add ce69ec6 [BEAM-11312] Minor improvement in retrieving build log (#13501)
add 196fd9d [BEAM-11207] Return valid error on job failures (#13484)
add 0498a17 [BEAM-11413] Fix default job server for Spark tests.
add 7b39702 Merge pull request #13498 from ibzib/BEAM-11413
add 80eff6c [BEAM-11417] Use Cache with time eviction policy for commitCallbacks
add a9a1c8e Merge pull request #13507 from [BEAM-11417] Use Cache with time eviction policy for commitCallbacks
add bbaaeb0 Decode data channel elements after, rather than before, placing them in the queue. (#13387)
add 5caeb5d Merge pull request #13395 from [BEAM-11426] Add FHIR Search to io/gcp/healthcare/FhirIO
add 05c8471 Add GroupIntoBatches to runner API; add Dataflow override in Python SDK
add 2c96aeb Merge pull request #13405 from [BEAM-10703, BEAM-10475] Add GroupIntoBatches to runner API; add Dataflow override in Python SDK
add 62f385c [BEAM-11419] Add Flink 1.12 build target.
add a35513d Merge pull request #13504: [BEAM-11419] Add Flink 1.12 build target
add 15f9270 [BEAM-11075] Fix go sdk gbk load test (#13494)
add e1cff1a [BEAM-11400] Change StreamingDataflowWorker to ignore commits that don't match.
add dce9158 Merge pull request #13486: [BEAM-11400] Change StreamingDataflowWorker to ignore commits that do…
add b39602a [BEAM-11384] Change StreamingDataflowWorker work id for logs to sharding key.
add 5f50f17 Merge pull request #13459: [BEAM-11384] Change StreamingDataflowWorker from encoding key
add adf5515 Better logging for Python Docker environment (#13508)
add a2815b5 [BEAM-11415] Setup and bugfixes for Go ValidatesRunner framework.
add 310a710 Merge pull request #13502: [BEAM-11415] Setup and bugfixes for Go ValidatesRunner framework.
add b942279 [BEAM-10114] Bump Pub/Sub lite version and clean up settings classes.
add c1d9794 [BEAM-10114] Fix lint errors
add bc0bac7 Merge pull request #13509 from [BEAM-10114] Bump Pub/Sub lite version and clean up settings classes.
add e8a9c7b [BEAM-11415] In-progress Go ValidatesRunner framework.
add d009fe9 Merge pull request #13503: [BEAM-11415] In-progress Go ValidatesRunner framework.
add cd2c312 [BEAM-11435] Reuse already set timers.
add 744637d Merge pull request #13522: [BEAM-11435] Reuse already set timers.
add 190261a Log exception caught during UnboundedSource#split
add 76723d8 Merge pull request #13520: Log exception caught during UnboundedSource#split
add 01a63c7 [BEAM-10556][BEAM-10402] Cleanup DoFnOperatorTest warnings.
add 33b3ac4 Merge pull request #13521: [BEAM-10556][BEAM-10402] Cleanup DoFnOperatorTest warnings
add 2356367 [BEAM-11425] Metrics extraction via Monitoring API (#13505)
add aa1498c [BEAM-11374] Add key regex filter to Bigtable for BeamSQL
add d4e9171 Merge pull request #13495 from piotr-szuberski/bigtable-filters
add 5e1c544 Add SDF capability_matrix.
add 1481977 Merge pull request #13368 from [BEAM-10480] Add SDF capability_matrix.
add 379d796 Add a small announcement for Splittable DoFn.
add c4af7f9 Merge pull request #13456 from [BEAM-10480] Add a small announcement for Splittable DoFn.
add 44fc9fe [BEAM-10402] Supress nullness and keyfor DoFnOperatorTest warnings.
add 0424485 Merge pull request #13525: [BEAM-10402] Supress nullness and keyfor DoFnOperatorTest warnings.
add 0353554 [BEAM-11337] Make the ThriftCoder class visible from outside its package
add 9317a6b Merge pull request #13415: [BEAM-11337] Make the ThriftCoder class visible from outside its package
add fd56cf1 [BEAM-7794] Enable DynamoDBIO tests (Use localstack version 0.11.4)
add b850e76 Merge pull request #13510: [BEAM-7794] Enable DynamoDBIO tests (Use localstack version 0.11.4)
add bf3c7eb Serialize AwsCredentialsProvider for AWS SDK v2 IO connectors
add 32598ff Merge pull request #13233: [BEAM-11016] Serialize AwsCredentialsProvider for AWS SDK v2 IO connectors
add 77683be [BEAM-11439] Fix Snowflake streaming docs (#13528)
add a679bab [BEAM-9602] Add timer family support for python SDK
add 8cf88c3 Merge pull request #13421 from [BEAM-9602] Add timer family support for python SDK
add 566ee5e [BEAM-10706] Fix duplicate key error in DynamoDBIO.Write
add a1676e9 Merge pull request #12583: [BEAM-10706] Fix duplicate key error in DynamoDBIO.Write
add 20bab56 Add myself (lostluck) to the authors.yml list (#13531)
add 1c453dc [BEAM-11208] Fix for QUOTA_EXCEEDED failures in BQ Storage streams sp… (#13378)
add 2733e95 [BEAM-11434]Expose batch/admin clients in Spanner Accessor
add 70ac878 Merge pull request #13519: [BEAM-11434]Expose batch/admin clients in Spanner Accessor
add f36cd5b Update Beam website to release 2.26.0. (#13481)
add fdbf47d Add a blogpost for ApacheBeam 2.26.0. (#13482)
add dc8d82e Fix typo in publish_docker_images.sh (#13532)
add b9cc076 [BEAM-11338] Beam schema for thrift data (#13428)
add ee911d2 Fix missing braket in the 2.25 signature. (#13534)
add 302de87 Workaround for incremental in read_json. (#13489)
add dc7ecb9 [BEAM-9196] Update testcontainers to 1.15.1
add fc42781 Merge pull request #13530: [BEAM-9196] Update testcontainers to 1.15.1
add 82bbdf7 [BEAM-10124] Add missing announcement from CHANGES.md (#13537)
add 38d3c5d [BEAM-11075] Enable Go load tests for Dataflow (#13529)
add 755eab2 [BEAM-11338] Addendum: handle private fields in generated thrift code (#13539)
add 7e85cc7 Add output type hints for GroupIntoBatches. (#13493)
add 7bdc297 [BEAM-11444] Python IO MongoDB: mask password in display_data uri and spec. (#13541)
add 18b0ee8 Change the output type of FHIR Search from String to JsonArray (#13516)
add 8649972 [BEAM-11429] User-friendly names for packed combiners (#13535)
add be85be2 [Beam-11438] start-build-env.sh improve for macOS (#13545)
add 34ae21b [BEAM-11065] Apache Beam pipeline example to ingest from Apache Kafka to Google Pub/Sub (#13112)
add 52a2270 [BEAM-11196] Fix `None` parent when fusing >2 stages (#13549)
add f23ebad [BEAM-10641] Add Combiner Packing to Dataflow (#13455)
add 75e167e [BEAM-11460] Implement reading unknown schema files for ParquetIO
add 0686879 Update ParquertIO capability in CHANGES.md
add de6d268 improve backward compatibility by creating separate `Parse<T>` and `ParseFiles<T>` implementation for supporting files with unknown schema.
add 431831b Fix review comments by @iemejia
add 6faeda3 Make GenericRecordPassthroughFn singleton
add afe11c6 fix spotless Apply
add eb16936 [BEAM-11230] Fix a bug with decoding repeated fields (#13544)
add 447d9ae Enable more tests on Java + Python FnRunner
add 531cded Merge pull request #13547 from [BEAM-10450][BEAM-10449][BEAM-10453][BEAM-10995] Enable more Java tests on Python FnRunner
add ba11ed7 Do not add unnecessary experiment use_multiple_sdk_containers. (#13475)
add 7e09870 Skip dynamic timer test in portable spark test
add 5878cc5 Merge pull request #13553 from Skip dynamic timer test in portable spark test
add 3d62aab Bump up python container versions
add 3710357 Merge pull request #13552 from Bump up python container versions
add 9f97585 [BEAM-11360] Updates Dataflow Python multi-language pipelines to use portable job submission by default (#13536)
add 8dd9b38 [BEAM-11401][BEAM-11366] Change ReaderCache invalidations to be asynchronous and make the timeout configurable.
add 0548643 Merge pull request #13485: [BEAM-11401][BEAM-11366] Change ReaderCache invalidations to be async…
add 5184eac Revert "Do not add unnecessary experiment use_multiple_sdk_containers." (#13555)
add c60b70e Adding limitations of SchemaUpdateOptions to BQIO javadoc.
add 5d9f4de Merge pull request #13551 from Adding limitations of SchemaUpdateOptions to BQIO javadoc.
add 15ffb07 [BEAM-11472] Support SDF on non-splitting runners such as legacy Dataflow.
add f137246 Merge pull request #13556 [BEAM-11472] Support SDF on non-splitting runners such as legacy Dataflow.
add 31dcd53 [BEAM-10114] Convert PubsubLiteIO read to use SplittableDoFn.
add ec3efe2 Fix test.
add d3336fe fix: Testing issues
add 8892c1c [BEAM-10114] Convert PubsubLiteIO to use an SDF for reads.
add f5c6108 [BEAM-10114] Add byte and time limiting
add 6d8c0a9 [BEAM-10114] Add byte and time limiting
add 762b7ba [BEAM-10114] Add byte and time limiting
add 65b2c42 [BEAM-10114] Add byte and time limiting
add 418b3ac [BEAM-10114] Add byte and time limiting
add 843fc4a [BEAM-10114] Add byte and time limiting
add fa3fc2d fix: Linter issues
add ea33f9e fix: linters
add 88a3021 fix: linters
add b731da8 fix: Remove unnecessary shuffle.
add a432e8f fix: Address comments
add f87c984 Merge pull request #13470 from [BEAM-10114] Convert PubsubLiteIO read to use SplittableDoFn.
add dee8d49 [BEAM-11443] Implement byte-based GBK for Flink Streaming Runner.
add 56d4ac9 Merge pull request #13538: [BEAM-11443] Implement byte-based GBK for Flink Streaming Runner.
add ded0325 [BEAM-11075] Go SDK SideInput load tests (#13436)
add f1c1514 [BEAM-11267] Remove unecessary reshuffle for stateful ParDo after keyed operation.
add 76c6129 Merge pull request #13353: [BEAM-11267] Remove unnecessary reshuffle for stateful ParDo after key…
add 12b6015 Update DoFn docstring (#13559)
add cf20794 Fix concat source fraction_consumed() in degenerate case.
add 8f79065 Merge pull request #13563 from robertwb/concat-isdone
add b2f4784 [BEAM-11361] Dynamic splitting of dataframe csv reads. (#13488)
add 97ad770 fix PerPartitionSubscriptionSdf methods with incorrect visibility
add 92209d1 Merge pull request #13566 from fix PerPartitionSubscriptionSdf methods with incorrect visibility
add 1ccdee8 [BEAM-11432] put latest tag when publishing SDK head containers to gcr (#13517)
add 8de3288 [BEAM-10925] Add interface for SQL Java scalar UDF. (#13305)
add 554d254 Track transform processing thread in Java SDK harness and set log entry transform id field.
add a6ba301 Add javadoc
add 23b2007 Add tracking to other ProcessElement methods
add 945e794 Use LoadingCache instead of ConcurrentHashMap to limit the size for thread tracker
add 857d41f Address comment
add c8d9d1d Fix checkstyle
add 6869dfa Merge pull request #13533 from [BEAM-11474] Track transform processing thread in Java SDK harness and set log entry field
add 3de4f88 [BEAM-10925] Add SQL AggregateFn interface.
add a7457c0 Merge pull request #13306 from ibzib/aggregate-udf
add 3cacd4a [BEAM-11407] Add IT test to Bigtable for BeamSQL
add 1a5505b Merge pull request #13512 from piotr-szuberski/bigtable-it
add 8fa3e87 Thread safe addition of new metrics
add c7efdc1 Thread safe addition of new metrics
add 5f253b6 Merge pull request #13526 from [BEAM-11459] Thread safe addition of new metrics
add b8c001c Moving to 2.28.0-SNAPSHOT on master branch.
add 9c69229 [Beam-11002] Fixes BufferOverflowException in XMLReader (#13513)
add db84863 Adding combiner packing optimization for Portable Runner
add 2202581 fix formatting
add fc84d02 Merge pull request #13573 from pabloem/cl151
add 4f6f923 [BEAM-10658] Update website to recommend the right version
add 0757359 Merge pull request #13576: [BEAM-10658] Update website to recommend the right version
add f8d326e Rewrite container docs for custom containers (#13420)
add 77ff32c Cleanup directory with SVN repo for PMC members
add 1542971 Merge pull request #13580 from Cleanup directory with SVN repo for PMC members
add 3663e03 [BEAM-11458] Upgrade SamzRunner to Samza 1.5 (#13550)
add 9732fa3 Add DataFrame Preview announcment blog post (#13561)
add 019db83 Dataframe docs (#13560)
add 7d88858 [BEAM-11300] Improve Nexmark performance:
add 80248d0 Merge pull request #13318: [BEAM-11300] Improve Nexmark performance:
add 8de6faa [BEAM-11481] emit output watermark on watermark hold change
add bea6f88 Merge pull request #13571: [BEAM-11481] emit output watermark on watermark hold change
add 287f77a [BEAM-11329]:HDFS not deduplicating identical configuration paths (#13514)
add 4b1c4aa [BEAM-10925] SQL AggregateFn: Separate mutable accumulator from immutables.
add 199fea8 Merge pull request #13579 from ibzib/accumulators
add c4b1359 [BEAM-10925] Create ZetaSQL-specific subclass of ScalarFunctionImpl that knows which function group it belongs to.
add 542e8ce Make createUd(a)fOperator methods package-private.
add 1466db9 Merge pull request #13309 from ibzib/fngroup
add 243e5df [BEAM-11391] Improve Nexmark Kafka support to track watermarks using commit time, allow specifying the number of partitions instead of requiring launcher to have access to kafka cluster, and removing unneeded deserialization of keys.
add bfdb3e5 Merge pull request #13472: [BEAM-11391] Improve Nexmark Kafka support to track
add 164a7b6 [BEAM-9426] Add JVM properties to JavaJarJobServer. (#13581)
add e9fa621 Release bundle processor when any exceptions during processing.
add 060b083 Merge pull request #13568 from [BEAM-3245] Release bundle processor when any exceptions during processing.
add b1fcfe9 [BEAM-10925] Add UdfProvider interface.
add e4ee5f5 Merge pull request #13587 from ibzib/udf-provider
add c679de6 [BEAM-9637] Add all runners to Python --runner help text.
add 0350f2e Merge pull request #13584 from ibzib/BEAM-9637
add 926c649 Fix typo in join explanation
add 44f198d Merge pull request #13577: Fix typo in join explanation
add 8a9a259 Fix regex in Kotlin MinimalWordCount example. (#13570)
add 6171a36 [BEAM-11456] Fix flaky table provider tests (Kafka and Pubsub) (#13542)
add c2e07fb [BEAM-10925] Convert ScalarFn to Method. (#13593)
add a929141 [BEAM-11415][BEAM-11416] Move Go ValidatesRunner tests and add filtering (#13518)
add 256dafd [BEAM-11515] Add check for missing authors, add missing authors (#13599)
add 98ef617 Add 2.28.0 section to CHANGES.md (#13610)
add c17270b [BEAM-11159] Use official GCP client in TestPubSubSignal (#13598)
add fbeb28e [BEAM-11515] Minor release guide updates (#13609)
add 30a1baf [BEAM-11522] Revert "Optimizes extract_output for 1 element accumulator case." (#13612)
add 0e8725c Bump Gradle to 6.7.1
add 5c755ea Merge pull request #13414: [BEAM-11523] Bump Gradle to 6.7.1
add feb2b84 Remove usages of WriteStringsToPubSub in examples
add b0e3d36 Merge pull request #13615 from [BEAM-11524] Remove usages of WriteStringsToPubSub in examples
add 38eeec9 Merge pull request 2.27.0 from apache/master
add e4c3c94 Move ParquetIO Changes to 2.28.0
add 0d344ba Make passthroughfunction singleton by making the class with private constructor.
add 8149158 Merge pull request #13616: [BEAM-11460] Support reading Parquet files with unknown schema
add 98ee1f1 Cache UnboundedReader per CheckpointMark in SDF Wrapper DoFn.
add b6243e7 Merge pull request #13592 from [BEAM-11403] Cache UnboundedReader per UnboundedSourceRestriction in SDF Wrapper DoFn.
add 3256be5 Fix for the case when truststoreLocation and keystoreLocation are not specified
add 06f8853 Fix nullable coders casting
add a751c11 Merge pull request #13608 from Fix for Kafka To PubSub Example
add d61a1fa [BEAM-11517] Enable test_file_loads on Dataflow
add e9a7b44 Merge pull request #13601 from [BEAM-11517] Fix test_file_loads on Dataflow
add 593b576 [BEAM-11531] Restrict pandas version to <1.2.0 (#13623)
add 8b050a3 [BEAM-10925] Change UserFunctionDefinitions into an AutoValue and Builder class.
add 356e2ae Merge pull request #13625 from ibzib/ufd-builder
add 9f234d6 [BEAM-9980] Combine job_PostCommit_Python*.groovy files into one (#13461)
add b8f273e [BEAM-9980] use constant for job_Inventory.groovy and do not use hardcoded python versions (#13460)
add 8df07b6 [BEAM-11524] Change WriteStringsToPubsub to use WriteToPubsub. (#13614)
add 2ceef2d [BEAM-9615] Initial Custom Schema Coder Support (#13611)
add 894231f [BEAM-10925] Default to empty maps in UserFunctionDefinitions.Builder.
add 7ef8bd1 Merge pull request #13628 from ibzib/ufd-builder
add d84699e Oxford comma
add ba305fa Merge pull request #13631 from elharo/patch-3
add 4827bea [BEAM-11537] Fix TestPubSubSignal.waitForStart (#13632)
add 9de4ea4 [BEAM-11482] Thrift support for KafkaTableProvider (#13572)
add 4364cc8 [BEAM-10986] Fix for update shadow jar plugin. (#13586)
add 0c84a51 BEAM-11536. Test "beam:window_fn:serialized_java:v1" in WindowStrategyTranslation
add 5e17b69 Merge pull request #13630 from amaliujia/BEAM-11536
add fe0dfa1 [BEAM-11530] Consider all kinds of annotatios on schema creation (#13622)
add cec16b6 [BEAM-11532] Fix edge case in merge where left_on and right_on contain equivalent column names (#13634)
add 9bd4a42 [BEAM-9187] Address flake in loadBalancesBundles (#13448)
add 48673e8 [BEAM-11533] Add logic to convert Beam schema to DataCatalog schema to SchemaUtils (#13588)
add a7301b7 [BEAM-11527] Add builder parameter to allow user defined Hadoop ReadSupport flags in Hadoop Configuration.
add ff5a094 Consistency improvements and other fixes
add cf74313 Merge pull request #13619: [BEAM-11527] Allow user defined Hadoop ReadSupport flags for ParquetReader
add 1a617be Add Beam schema support for ParquetIO reads.
add 2f205f1 Merge pull request #13639: [BEAM-11526] Add Beam schema support for ParquetIO reads
add 4165016 [BEAM-11457] Add option to skip key-value clone (#13543)
add 4370926 [BEAM-10234] Create ApproximateDistinct using HLL Impl
add 47d6ab5 Merge pull request #12973: [BEAM-10234] Create ApproximateDistinct using HLL Impl
add c7d80a8 Revert "[BEAM-10234] Create ApproximateDistinct using HLL Impl"
add da1113b Merge pull request #13653 from apache/revert-12973-hll [BEAM-10324] Reverts apache/beam#12973 Create ApproximateDistinct using HLL Impl
add cd2d208 Replace . with ;
add e28cb28 Merge pull request #13659: Replace . with ; in schema programming guide snippet
add 34a8fcd [BEAM-11526] Cleanup Code for Schema support (#13646)
add 47ca61a [BEAM-10324] Create ApproximateDistinct using HLL Impl (#13655)
add 3f8a4ff [BEAM-8829] only drop event_timestamp when it exists (#13638)
add f120eb7 [BEAM-9541] Add flink_versions to gradle.properties.
add 7eacb40 [BEAM-9541] All Gradle tasks use latest Flink version.
add 08706df Disable Flink classloader leak check when using local execution mode.
add e12a0d8 [BEAM-11570] Comment with link to context.
add 9b85af0 Merge pull request #13633 from ibzib/latest-flink
add 9f9dcd6 [BEAM-9980] do not hardcode Python version for dataflow validate runner tests and make it switchable (#13330)
add 9276a8c Copying Github action into our own repository to comply with Apache security requirements
add b2ad4dd Merge pull request #13670 from [BEAM-11569] Clone ad-m/github-push-action into Beam repository to comply with Apache security requirements
add 9ebdc8a replace Assert.asserThat with MacherAssert.assertThat across entire codebase
add 3915c3f [BEAM-11572] Add lint rule to avoid Assert.assertThat usage
add 9df5db9 Fix Assertion in Nexmark (non-test) class.
add 60834ae Merge pull request #13673 from anantdamle/beam_11572_fix_assertThat
add c72fac9 Fix sdk.properties caching, remove timestamp.
add d3422f3 Merge pull request #13652 from runningcode/no/remove-timestamp
add 686b1ff [BEAM-11569] Building wheels with airflow-clone of github-push action
add 0183cdb2 Merge pull request #13677 from [BEAM-11569] Building wheels with airflow-clone of github-push action
add 6a21b06 Fix for Java container release
add 26d27c8 Merge pull request #13604 from pabloem/fixup-jv811
add 3b12ce0 [BEAM-11569] Fixing cancel workflows
add 26c2466 Merge pull request #13679 from [BEAM-11569] Fixing cancel workflows
add d8966d6 [BEAM-11415] Add Dataflow Runner to new Go ValidatesRunner framework. (#13678)
add 065ed7c [BEAM-11415] Adds documentation to integration package. (#13683)
add d42c67c [BEAM-11580] Add disk_size_gb flag for Go SDK. (#13693)
add 854ed5b [BEAM-10983] Add getting started from Spark page
add c484e68 Add "From Apache Spark" entry to side menu
add 17aed03 Add links to transform catalog
add eaff2e9 Merge branch 'master' of github.com:apache/beam into coming-from-spark
add 26a2a04 Add more links
add 2f39799 Add RDD/DataFrame clarification
add a751df8 Merge branch 'master' of github.com:apache/beam into coming-from-spark
add 6c338ce Add link to from-spark page
add a3406b3 Added extra explanations
add 67af84a Merge branch 'master' of github.com:apache/beam into coming-from-spark
add 9557470 Fixed typos + removed italics
add 2ad2854 Merge pull request #12963 from davidcavazos/coming-from-spark
add 96a659a Clarify some assertions (#13515)
add 36bf02c [BEAM-11504] Clean up direct runner parallelism section.
add d72b5d9 Merge pull request #13589 from ibzib/BEAM-11504
add 2239b3a [BEAM-11419] Update Flink supported versions in the website
add c5badae Merge pull request #13620 from iemejia/website-flink-versions
add 41c3811 Revert "[BEAM-11474] Track transform processing thread in Java SDK harness and set log entry field"
add 6af9d75 Merge pull request #13696 from y1chi/revert-13533-test_logging
add 93c2bd8 [BEAM-1833] Give transforms more control over the names of their inputs and outputs.
add 7e8fa4e mypy
add 74ec609 Merge pull request #13672 [BEAM-1833] Give transforms more control over the names of their inputs and outputs.
add 587a3ff [BEAM-11582] Merge website/.gitignore into .gitignore (#13697)
add 2e95a20 Correct the license in website package.json file
add df74d74 Merge pull request #13698 from apache/aaltay-patch-1
add 078b24d [BEAM-11543] Increase Spanner streaming read timeout to 2 hours
add f0db6e7 Merge pull request #13640: [BEAM-11543] Increase Spanner streaming read timeout to 2 hours
add 995c776 Remove unused endOfStreamMonitor from nexmark
add b0b8ffd Also rename metric
add 38464d1 Merge pull request #13435 from rHermes/remove-unused-monitor-in-nexmark
add d2f76f9 [BEAM-11327] Replace Charset.defaultCharset() with StandardCharsets.UTF_8 (#13410)
add 3fa9846 [BEAM-11579] Specify flink_job_server_jar instead of using default.
add 4e83ac5 Merge pull request #13682 from ibzib/BEAM-11579
add c9f868a [BEAM-11508] Set sdkHarnessContainerImages in workerpool configuration for Dataflow runner v2
add ac71556 add utility function to check runner v2, set single-core flag for python
add a3fef8b Merge pull request #13605 from ihji/BEAM-11508
add b378764 [BEAM-11327] Update JulHandler with Charset.defaultCharset() (#13701)
add 6933486 Adding website update for release 2.27.0
add ec36893 Merge pull request #13602 from pabloem/2_27_0_website_update
add ff40901 Fixing script for Java8/Java11 docker container push
add fbaf292 Merge pull request #13703 from Fixing script for Java8/Java11 docker container push
add dd12a85 Add 2.27.0 blog post, and update CHANGES.md
add 2888c0b Merge branch 'master' into blog-2.27.0
add 922d7d7 Merge pull request #13603 from pabloem/blog-2.27.0
add f8fe5a6 Update the blog date for 2.27
add 868a888 Merge pull request #13708 from apache/aaltay-patch-1
add 9e7535c [BEAM-11571] Support Conversion to GenericRecords in Convert.to transform
add bfee142 [BEAM-11571] Don't transform if input and output types are equal on Convert transform
add 45ff8df Merge pull request #13671: [BEAM-11571] Support Conversion to GenericRecords in Convert.to transform
add 4c51121 [BEAM-11584] Upgrade junit to version 4.13.1
add 0994dfe Merge pull request #13700: [BEAM-11584] Upgrade junit to version 4.13.1
add 20cf66a [BEAM-8493] Add test to standard double coder (#13713)
add 5f3a86c [BEAM-11588] Fix the dependency (#13712)
add bf814cc Update Dataflow worker container image version.
add 15d282b Merge pull request #13699: Update Dataflow worker harness container image version.
add 1396a70 don't use artifact to output java 3rd party licenses
add ca207f6 remove java 3pl directory from copy
add 7f93e42 Merge pull request #13711 from emilymye/fixjavaartifact
add 51183ac Add version and sdk_version as inputs to ProcessResources task.
add 2c63124 Also add docker image to sdk.properties
add a1d97b8 Fix incorrect input property
add 0a7edfc Update sdks/java/core/src/main/resources/org/apache/beam/sdk/sdk.properties
add 75d3864 Merge pull request #13691 from runningcode/no/sdk-properties
add f46954c [BEAM-11588] Move forward jupyter-client version.
add 04e3343 Merge pull request #13721 from KevinGG/BEAM-11588
add 8e1a8b8 [BEAM-10402] Trigger#subTriggers does not return null
add 7aa7e13 Merge pull request #13719: [BEAM-10402] Trigger#subTriggers might return null
add dd71c9a Update doc/examples: BigQuerySource to ReadFromBigQuery (#13239)
add 4c8f0b0 [BEAM-11474] Set log entry transform id with best effort
add 327ec71 Address comment
add 6c9da02 Merge pull request #13702 from [BEAM-11474] Set log entry transform id with best effort in Java SDK harness
add 1aca0c7 Note about SQL in Python.
add 53615fe Merge pull request #13738 Update docs wrt SQL in Python.
new 7d0d220 Merge remote-tracking branch 'origin/master' into website-revamp
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/workflows/build_wheels.yml | 2 +-
.github/workflows/cancel.yml | 2 +-
.gitignore | 10 +
.test-infra/jenkins/README.md | 9 +-
.test-infra/jenkins/job_Inventory.groovy | 15 +-
.../jenkins/job_LoadTests_Combine_Flink_Go.groovy | 2 +-
...k_Go.groovy => job_LoadTests_Combine_Go.groovy} | 85 +-
.../jenkins/job_LoadTests_GBK_Flink_Go.groovy | 19 +-
.test-infra/jenkins/job_LoadTests_GBK_Go.groovy | 211 +++++
.test-infra/jenkins/job_LoadTests_ParDo_Go.groovy | 145 ++++
.../job_LoadTests_SideInput_Flink_Go.groovy | 103 +++
.../jenkins/job_LoadTests_SideInput_Go.groovy | 94 +++
.test-infra/jenkins/job_LoadTests_coGBK_Go.groovy | 169 ++++
...ython36.groovy => job_PostCommit_Python.groovy} | 39 +-
.test-infra/jenkins/job_PostCommit_Python37.groovy | 44 -
.test-infra/jenkins/job_PostCommit_Python38.groovy | 44 -
...stCommit_Python_ValidatesRunner_Dataflow.groovy | 8 +-
...ommit_Python_ValidatesRunner_Dataflow_V2.groovy | 9 +-
.../jenkins/job_Publish_SDK_Image_Snapshots.groovy | 8 +-
.../perftests_metrics/SideInput_Load_Tests.json | 2 +-
CHANGES.md | 51 +-
build.gradle | 9 -
buildSrc/build.gradle | 4 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 31 +-
examples/java/build.gradle | 11 +-
.../org/apache/beam/examples/complete/README.md | 5 +
.../kafkatopubsub/KafkaPubsubConstants.java | 15 +-
.../complete/kafkatopubsub/KafkaToPubsub.java | 235 ++++++
.../beam/examples/complete/kafkatopubsub/README.md | 200 +++++
.../complete/kafkatopubsub/avro/AvroDataClass.java | 63 ++
.../avro/AvroDataClassKafkaAvroDeserializer.java | 27 +-
.../complete/kafkatopubsub/avro/package-info.java | 10 +-
.../kafka/consumer/SslConsumerFactoryFn.java | 135 ++++
.../kafkatopubsub/kafka/consumer/Utils.java | 165 ++++
.../kafkatopubsub/kafka/consumer/package-info.java | 10 +-
.../options/KafkaToPubsubOptions.java | 90 +++
.../kafkatopubsub/options/package-info.java | 10 +-
.../complete/kafkatopubsub/package-info.java | 10 +-
.../kafkatopubsub/transforms/FormatTransform.java | 128 +++
.../kafkatopubsub/transforms/package-info.java | 10 +-
.../examples/complete/game/LeaderBoardTest.java | 2 +-
.../complete/kafkatopubsub/KafkaToPubsubTest.java | 90 +++
.../cookbook/CombinePerKeyExamplesTest.java | 13 +-
examples/kotlin/build.gradle | 2 +-
.../beam/examples/kotlin/MinimalWordCount.kt | 2 +-
gradle.properties | 7 +-
gradle/wrapper/gradle-wrapper.properties | 2 +-
.../beam/learning/katas/util/ContainsKvs.java | 2 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 18 +
.../src/main/scripts/build_release_candidate.sh | 6 +-
.../src/main/scripts/preparation_before_release.sh | 2 +
release/src/main/scripts/publish_docker_images.sh | 4 +-
.../core/construction/CoderTranslationTest.java | 2 +-
.../runners/core/construction/CommonCoderTest.java | 2 +-
.../CreatePCollectionViewTranslationTest.java | 2 +-
.../DeduplicatedFlattenFactoryTest.java | 2 +-
.../construction/DefaultArtifactResolverTest.java | 2 +-
.../EmptyFlattenAsCreateFactoryTest.java | 2 +-
.../core/construction/EnvironmentsTest.java | 2 +-
.../ExecutableStageTranslationTest.java | 2 +-
.../construction/ForwardingPTransformTest.java | 2 +-
.../construction/GroupByKeyTranslationTest.java | 2 +-
.../runners/core/construction/ModelCodersTest.java | 2 +-
.../core/construction/MorePipelineTest.java | 2 +-
.../construction/PCollectionTranslationTest.java | 2 +-
.../core/construction/PTransformMatchersTest.java | 2 +-
.../construction/PTransformReplacementsTest.java | 2 +-
.../construction/PTransformTranslationTest.java | 2 +-
.../core/construction/ParDoTranslationTest.java | 2 +-
.../PipelineOptionsTranslationTest.java | 2 +-
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/construction/ReadTranslationTest.java | 2 +-
.../construction/RehydratedComponentsTest.java | 2 +-
.../core/construction/ReplacementOutputsTest.java | 2 +-
.../construction/ReshuffleTranslationTest.java | 2 +-
.../core/construction/SchemaTranslationTest.java | 2 +-
.../core/construction/SdkComponentsTest.java | 2 +-
.../SingleInputOutputOverrideFactoryTest.java | 2 +-
.../construction/TestStreamTranslationTest.java | 2 +-
.../core/construction/TransformInputsTest.java | 2 +-
.../core/construction/TriggerTranslationTest.java | 2 +-
.../core/construction/UnconsumedReadsTest.java | 2 +-
.../core/construction/ValidateRunnerXlangTest.java | 2 +-
.../WindowingStrategyTranslationTest.java | 161 +++-
.../construction/WriteFilesTranslationTest.java | 2 +-
.../construction/graph/ExecutableStageTest.java | 2 +-
.../core/construction/graph/FusedPipelineTest.java | 2 +-
.../graph/GreedyPipelineFuserTest.java | 2 +-
.../construction/graph/GreedyStageFuserTest.java | 2 +-
.../graph/ImmutableExecutableStageTest.java | 2 +-
.../core/construction/graph/NetworksTest.java | 2 +-
.../construction/graph/OutputDeduplicatorTest.java | 2 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
.../construction/graph/QueryablePipelineTest.java | 2 +-
.../core/metrics/ExecutionStateTracker.java | 20 +-
.../runners/core/InMemoryStateInternalsTest.java | 2 +-
.../runners/core/InMemoryTimerInternalsTest.java | 2 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 2 +-
.../beam/runners/core/LateDataUtilsTest.java | 2 +-
...BoundedSplittableProcessElementInvokerTest.java | 2 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 2 +-
.../beam/runners/core/SideInputHandlerTest.java | 2 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 2 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 2 +-
.../runners/core/SplittableParDoProcessFnTest.java | 2 +-
.../beam/runners/core/StateNamespacesTest.java | 2 +-
.../beam/runners/core/TimerInternalsTest.java | 2 +-
.../beam/runners/core/WindowMatchersTest.java | 2 +-
.../beam/runners/core/metrics/CounterCellTest.java | 2 +-
.../beam/runners/core/metrics/DirtyStateTest.java | 2 +-
.../runners/core/metrics/DistributionCellTest.java | 2 +-
.../core/metrics/ExecutionStateSamplerTest.java | 2 +-
.../core/metrics/ExecutionStateTrackerTest.java | 2 +-
.../beam/runners/core/metrics/GaugeCellTest.java | 2 +-
.../core/metrics/MetricsContainerImplTest.java | 2 +-
.../core/metrics/MetricsContainerStepMapTest.java | 2 +-
.../beam/runners/core/metrics/MetricsMapTest.java | 2 +-
.../runners/core/metrics/MetricsPusherTest.java | 2 +-
.../core/metrics/MonitoringInfoConstantsTest.java | 2 +-
.../core/metrics/SimpleStateRegistryTest.java | 2 +-
.../triggers/AfterWatermarkStateMachineTest.java | 2 +-
.../core/triggers/FinishedTriggersBitSetTest.java | 2 +-
.../core/triggers/FinishedTriggersSetTest.java | 2 +-
.../core/triggers/NeverStateMachineTest.java | 2 +-
.../core/triggers/TriggerStateMachinesTest.java | 2 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 2 +-
.../runners/direct/CloningBundleFactoryTest.java | 2 +-
.../beam/runners/direct/CommittedResultTest.java | 2 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 2 +-
.../runners/direct/DirectGraphVisitorTest.java | 2 +-
.../DirectGroupByKeyOverrideFactoryTest.java | 2 +-
.../beam/runners/direct/DirectMetricsTest.java | 2 +-
.../runners/direct/DirectRunnerApiSurfaceTest.java | 2 +-
.../beam/runners/direct/DirectRunnerTest.java | 2 +-
.../runners/direct/DirectTimerInternalsTest.java | 2 +-
.../direct/DirectTransformExecutorTest.java | 2 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 2 +-
.../runners/direct/DoFnLifecycleManagerTest.java | 2 +-
.../beam/runners/direct/EvaluationContextTest.java | 2 +-
.../direct/FlattenEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyOnlyEvaluatorFactoryTest.java | 2 +-
.../ImmutabilityCheckingBundleFactoryTest.java | 2 +-
.../direct/ImmutableListBundleFactoryTest.java | 2 +-
.../direct/ImpulseEvaluatorFactoryTest.java | 2 +-
.../direct/KeyedPValueTrackingVisitorTest.java | 2 +-
.../beam/runners/direct/MultiStepCombineTest.java | 2 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 2 +-
.../runners/direct/SideInputContainerTest.java | 2 +-
.../direct/StatefulParDoEvaluatorFactoryTest.java | 2 +-
.../runners/direct/StepTransformResultTest.java | 2 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 2 +-
.../direct/UnboundedReadDeduplicatorTest.java | 2 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 2 +-
.../runners/direct/ViewEvaluatorFactoryTest.java | 2 +-
.../direct/WatermarkCallbackExecutorTest.java | 2 +-
.../beam/runners/direct/WatermarkManagerTest.java | 2 +-
.../runners/direct/WindowEvaluatorFactoryTest.java | 2 +-
.../direct/WriteWithShardingFactoryTest.java | 2 +-
.../runners/flink/streaming/StreamSources.java | 13 +
{sdks/go => runners/flink/1.12}/build.gradle | 38 +-
.../job-server-container/build.gradle | 0
.../flink/1.12/job-server}/build.gradle | 34 +-
.../streaming/AbstractStreamOperatorCompat.java | 53 ++
.../runners/flink/streaming/StreamSources.java | 14 +
.../runners/flink/streaming/StreamSources.java | 13 +
.../runners/flink/streaming/StreamSources.java | 13 +
runners/flink/flink_runner.gradle | 19 +-
.../runners/flink/FlinkExecutionEnvironments.java | 12 +
.../flink/FlinkPortableClientEntryPoint.java | 4 +-
.../FlinkStreamingPortablePipelineTranslator.java | 27 +-
.../flink/FlinkStreamingTransformTranslators.java | 215 +++--
.../flink/FlinkStreamingTranslationContext.java | 24 +-
.../wrappers/streaming/DoFnOperator.java | 77 +-
.../streaming/KeyedPushedBackElementsHandler.java | 5 +-
.../streaming/SingletonKeyedWorkItemCoder.java | 13 +-
.../wrappers/streaming/WorkItemKeySelector.java | 13 +-
.../streaming/state/FlinkStateInternals.java | 4 +-
.../FlinkStreamingPipelineTranslatorTest.java | 109 +++
.../FlinkStreamingTransformTranslatorsTest.java | 7 +-
.../runners/flink/FlinkTransformOverridesTest.java | 2 +-
.../wrappers/streaming/DedupingOperatorTest.java | 2 +-
.../wrappers/streaming/DoFnOperatorTest.java | 315 ++++++--
.../wrappers/streaming/WindowDoFnOperatorTest.java | 5 +-
runners/google-cloud-dataflow-java/build.gradle | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 53 +-
.../options/DataflowPipelineDebugOptions.java | 10 +
.../dataflow/BatchStatefulParDoOverridesTest.java | 2 +-
.../runners/dataflow/BatchViewOverridesTest.java | 2 +-
.../dataflow/DataflowPTransformMatchersTest.java | 2 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 2 +-
.../runners/dataflow/DataflowRunnerInfoTest.java | 2 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 56 ++
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../runners/dataflow/TestDataflowRunnerTest.java | 2 +-
.../options/DataflowPipelineDebugOptionsTest.java | 2 +-
.../options/DataflowProfilingOptionsTest.java | 2 +-
.../runners/dataflow/util/PackageUtilTest.java | 2 +-
.../beam/runners/dataflow/util/StructsTest.java | 25 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 38 +-
.../dataflow/worker/StreamingDataflowWorker.java | 61 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 14 +-
.../logging/DataflowWorkerLoggingInitializer.java | 14 +-
.../JulHandlerPrintStreamAdapterFactory.java | 15 +-
.../dataflow/harness/test/TestStreamsTest.java | 2 +-
.../dataflow/worker/AvroByteReaderFactoryTest.java | 5 +-
.../dataflow/worker/AvroByteSinkFactoryTest.java | 3 +-
.../dataflow/worker/BatchDataflowWorkerTest.java | 2 +-
.../worker/BatchModeExecutionContextTest.java | 2 +-
.../dataflow/worker/ConcatReaderFactoryTest.java | 2 +-
.../runners/dataflow/worker/ConcatReaderTest.java | 2 +-
.../ContextActivationObserverRegistryTest.java | 2 +-
...CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java | 2 +-
.../DataflowElementExecutionTrackerTest.java | 2 +-
.../worker/DataflowExecutionStateTrackerTest.java | 2 +-
.../worker/DataflowSideInputReadCounterTest.java | 2 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 2 +-
.../dataflow/worker/DefaultParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/DeltaCounterCellTest.java | 2 +-
.../dataflow/worker/DoFnInstanceManagersTest.java | 2 +-
.../dataflow/worker/FakeWindmillServer.java | 14 +-
.../runners/dataflow/worker/FilepatternsTest.java | 2 +-
.../GroupAlsoByWindowParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/InMemoryReaderFactoryTest.java | 3 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 2 +-
.../worker/IntrinsicMapTaskExecutorTest.java | 2 +-
.../runners/dataflow/worker/IsmFormatTest.java | 2 +-
.../dataflow/worker/IsmSideInputReaderTest.java | 2 +-
.../dataflow/worker/LogRecordMatcherTest.java | 2 +-
.../beam/runners/dataflow/worker/LogSaverTest.java | 2 +-
...tContextActivationObserverRegistrationTest.java | 2 +-
.../worker/PairWithConstantKeyDoFnFactoryTest.java | 2 +-
.../worker/PartialGroupByKeyParDoFnsTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 6 +-
.../runners/dataflow/worker/ReaderFactoryTest.java | 6 +-
...ReifyTimestampAndWindowsParDoFnFactoryTest.java | 2 +-
...essCoderCloudObjectTranslatorRegistrarTest.java | 2 +-
.../dataflow/worker/ShuffleReaderFactoryTest.java | 3 +-
.../dataflow/worker/ShuffleSinkFactoryTest.java | 3 +-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 2 +-
.../runners/dataflow/worker/SinkRegistryTest.java | 5 +-
.../worker/SourceOperationExecutorFactoryTest.java | 2 +-
.../runners/dataflow/worker/StateFetcherTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 10 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
.../worker/StreamingModeExecutionContextTest.java | 5 +-
...eamingPCollectionViewWriterDoFnFactoryTest.java | 2 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 2 +-
.../worker/StreamingStepMetricsContainerTest.java | 2 +-
.../ToIsmRecordForMultimapDoFnFactoryTest.java | 2 +-
.../dataflow/worker/UserParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/ValuesDoFnFactoryTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillStateInternalsTest.java | 2 +-
.../dataflow/worker/WindmillStateReaderTest.java | 2 +-
.../worker/WindmillTimerInternalsTest.java | 2 +-
.../dataflow/worker/WorkItemStatusClientTest.java | 2 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 2 +-
.../dataflow/worker/counters/CounterSetTest.java | 2 +-
.../fn/control/BeamFnMapTaskExecutorTest.java | 2 +-
.../RegisterAndProcessBundleOperationTest.java | 2 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 2 +-
.../fn/data/RemoteGrpcPortReadOperationTest.java | 2 +-
.../fn/data/RemoteGrpcPortWriteOperationTest.java | 2 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 2 +-
.../fn/stream/ServerStreamObserverFactoryTest.java | 2 +-
.../graph/CloneAmbiguousFlattensFunctionTest.java | 2 +-
.../CreateRegisterFnOperationFunctionTest.java | 2 +-
.../graph/DeduceNodeLocationsFunctionTest.java | 2 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 2 +-
.../worker/graph/MapTaskToNetworkFunctionTest.java | 2 +-
.../dataflow/worker/graph/NetworksTest.java | 2 +-
.../RemoveFlattenInstructionsFunctionTest.java | 2 +-
.../ReplacePgbkWithPrecombineFunctionTest.java | 2 +-
.../DataflowWorkerLoggingInitializerTest.java | 2 +-
.../JulHandlerPrintStreamAdapterFactoryTest.java | 16 +-
.../worker/profiler/ScopedProfilerTest.java | 2 +-
.../dataflow/worker/status/ThreadzServletTest.java | 2 +-
.../worker/status/WorkerStatusPagesTest.java | 2 +-
.../worker/util/BatchGroupAlsoByWindowFnsTest.java | 2 +-
.../BatchGroupAlsoByWindowReshuffleDoFnTest.java | 2 +-
.../worker/util/GroupAlsoByWindowProperties.java | 2 +-
.../dataflow/worker/util/MemoryMonitorTest.java | 2 +-
.../worker/util/ScalableBloomFilterTest.java | 2 +-
.../dataflow/worker/util/TimerOrElementTest.java | 2 +-
.../worker/BatchingShuffleEntryReaderTest.java | 2 +-
.../worker/CachingShuffleBatchReaderTest.java | 2 +-
.../util/common/worker/FlattenOperationTest.java | 5 +-
.../util/common/worker/GroupingTablesTest.java | 2 +-
.../util/common/worker/MapTaskExecutorTest.java | 2 +-
.../util/common/worker/OutputReceiverTest.java | 5 +-
.../util/common/worker/ParDoOperationTest.java | 2 +-
.../util/common/worker/ReadOperationTest.java | 2 +-
.../util/common/worker/ShuffleEntryTest.java | 2 +-
.../util/common/worker/WriteOperationTest.java | 2 +-
.../runners/fnexecution/data/GrpcDataService.java | 6 +-
.../runners/fnexecution/ServerFactoryTest.java | 2 +-
.../control/DefaultJobBundleFactoryTest.java | 62 +-
.../control/FnApiControlClientPoolServiceTest.java | 2 +-
.../control/FnApiControlClientTest.java | 2 +-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 2 +-
.../fnexecution/data/GrpcDataServiceTest.java | 2 +-
.../environment/ProcessEnvironmentFactoryTest.java | 2 +-
.../environment/ProcessManagerTest.java | 8 +-
.../environment/RemoteEnvironmentTest.java | 2 +-
.../logging/GrpcLoggingServiceTest.java | 2 +-
.../StaticGrpcProvisionServiceTest.java | 2 +-
.../fnexecution/state/GrpcStateServiceTest.java | 2 +-
.../translation/PipelineTranslatorUtilsTest.java | 2 +-
.../beam/runners/local/StructuralKeyTest.java | 18 +-
runners/portability/java/build.gradle | 24 -
runners/samza/build.gradle | 17 +-
.../beam/runners/samza/SamzaJobInvocation.java | 114 +++
.../beam/runners/samza/SamzaJobServerDriver.java | 60 +-
.../beam/runners/samza/SamzaPipelineOptions.java | 22 +-
.../samza/SamzaPipelineOptionsValidator.java | 38 +-
.../samza/SamzaPortablePipelineOptions.java | 13 +
.../runners/samza/SamzaRunnerOverrideConfigs.java | 15 +
.../samza/adapter/UnboundedSourceSystem.java | 39 +-
.../samza/container/BeamContainerRunner.java | 6 +-
.../samza/container/BeamJobCoordinatorRunner.java | 78 ++
...inerCfgFactory.java => ContainerCfgLoader.java} | 18 +-
.../ContainerCfgLoaderFactory.java} | 16 +-
.../beam/runners/samza/runtime/BundleManager.java | 349 ++++++++
.../apache/beam/runners/samza/runtime/DoFnOp.java | 277 ++++---
.../runners/samza/runtime/FutureCollector.java | 60 ++
.../beam/runners/samza/runtime/GroupByKeyOp.java | 11 +-
.../beam/runners/samza/runtime/KeyedTimerData.java | 1 +
.../beam/runners/samza/runtime/OpAdapter.java | 48 +-
.../beam/runners/samza/runtime/OpEmitter.java | 5 +
.../samza/runtime/OutputManagerFactory.java | 5 +
.../samza/runtime/SamzaStoreStateInternals.java | 230 ++++--
.../samza/runtime/SamzaTimerInternalsFactory.java | 217 ++++-
.../runners/samza/translation/ConfigBuilder.java | 77 +-
.../runners/samza/translation/ConfigContext.java | 8 +
.../translation/FlattenPCollectionsTranslator.java | 2 +-
.../samza/translation/GroupByKeyTranslator.java | 24 +-
.../translation/ParDoBoundMultiTranslator.java | 56 +-
.../samza/translation/SamzaPipelineTranslator.java | 21 +-
.../translation/SamzaTestStreamSystemFactory.java | 179 +++++
.../translation/SamzaTestStreamTranslator.java | 100 +++
.../translation/SplittableParDoTranslators.java | 6 +-
.../samza/translation/TranslationContext.java | 59 +-
.../samza/translation/WindowAssignTranslator.java | 4 +-
.../beam/runners/samza/util/FutureUtils.java | 50 ++
.../samza/SamzaPipelineOptionsValidatorTest.java | 60 ++
.../samza/adapter/UnboundedSourceSystemTest.java | 28 +
.../runners/samza/runtime/BundleManagerTest.java | 474 +++++++++++
.../samza/runtime/FutureCollectorImplTest.java | 92 +++
.../runners/samza/runtime/KeyedTimerDataTest.java | 14 +-
.../runtime/SamzaStoreStateInternalsTest.java | 38 +-
.../runtime/SamzaTimerInternalsFactoryTest.java | 416 +++++++++-
.../samza/translation/ConfigGeneratorTest.java | 86 +-
.../samza/translation/TranslationContextTest.java | 94 +++
.../beam/runners/samza/util/FutureUtilsTest.java | 107 +++
.../runners/spark/GlobalWatermarkHolderTest.java | 2 +-
.../beam/runners/spark/SparkPipelineStateTest.java | 2 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 2 +-
.../spark/io/ReaderToIteratorAdapterTest.java | 2 +-
.../runners/spark/metrics/SparkBeamMetricTest.java | 2 +-
.../spark/metrics/SparkMetricsPusherTest.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 2 +-
.../metrics/BeamMetricTest.java | 2 +-
.../translation/batch/GroupByKeyTest.java | 2 +-
.../translation/streaming/CreateStreamTest.java | 2 +-
.../ResumeFromCheckpointStreamingTest.java | 2 +-
.../streaming/SparkCoGroupByKeyStreamingTest.java | 2 +-
.../streaming/StreamingSourceMetricsTest.java | 2 +-
.../streaming/TrackStreamingSourcesTest.java | 2 +-
sdks/go/build.gradle | 2 +-
sdks/go/container/build.gradle | 2 +
sdks/go/pkg/beam/core/graph/coder/double_test.go | 53 ++
sdks/go/pkg/beam/core/graph/coder/map_test.go | 11 +-
sdks/go/pkg/beam/core/graph/coder/row.go | 337 ++------
sdks/go/pkg/beam/core/graph/coder/row_decoder.go | 308 +++++++
sdks/go/pkg/beam/core/graph/coder/row_encoder.go | 271 +++++++
sdks/go/pkg/beam/core/graph/coder/row_test.go | 488 +++++++++---
.../pkg/beam/core/graph/coder/testutil/testutil.go | 154 ++++
.../core/graph/coder/testutil/testutil_test.go | 201 +++++
sdks/go/pkg/beam/core/metrics/metrics.go | 39 +
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 9 +-
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 38 +-
sdks/go/pkg/beam/pipeline.go | 1 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 5 +-
.../beam/runners/dataflow/dataflowlib/execute.go | 57 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 8 +
.../beam/runners/dataflow/dataflowlib/metrics.go | 124 +++
.../runners/dataflow/dataflowlib/metrics_test.go | 133 ++++
.../beam/runners/universal/runnerlib/compile.go | 13 +-
.../beam/runners/universal/runnerlib/execute.go | 21 +-
sdks/go/pkg/beam/runners/universal/universal.go | 1 +
sdks/go/pkg/beam/testing/ptest/ptest.go | 21 +
sdks/go/test/build.gradle | 94 ++-
sdks/go/test/integration/{ => driver}/driver.go | 0
sdks/go/test/integration/integration.go | 147 ++++
sdks/go/test/integration/primitives/cogbk_test.go | 16 +-
.../go/test/integration/primitives/flatten_test.go | 11 +-
sdks/go/test/integration/primitives/pardo_test.go | 16 +-
.../test/integration/synthetic/synthetic_test.go | 75 ++
.../test/integration/wordcount/wordcount_test.go | 6 +
sdks/go/test/integration/xlang/xlang_test.go | 260 ++++++
sdks/go/test/load/build.gradle | 1 +
sdks/go/test/load/sideinput/sideinput.go | 100 +++
sdks/go/test/run_integration_tests.sh | 2 +-
...ation_tests.sh => run_validatesrunner_tests.sh} | 220 +++--
.../src/main/resources/beam/checkstyle.xml | 17 +
.../src/main/resources/beam/suppressions.xml | 4 +
sdks/java/container/build.gradle | 4 +-
sdks/java/container/common.gradle | 12 +-
sdks/java/core/build.gradle | 5 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 38 +-
.../apache/beam/sdk/io/AvroSchemaIOProvider.java | 4 +-
.../beam/sdk/io/GenericRecordWriteConverter.java | 67 --
.../src/main/java/org/apache/beam/sdk/io/Read.java | 97 ++-
.../sdk/schemas/FieldValueTypeInformation.java | 53 +-
.../org/apache/beam/sdk/schemas/RowMessages.java | 132 +++
.../beam/sdk/schemas/transforms/Convert.java | 5 +
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 4 +-
.../beam/sdk/schemas/utils/ConvertHelpers.java | 14 +-
.../apache/beam/sdk/testing/CoderProperties.java | 2 +-
.../apache/beam/sdk/testing/CombineFnTester.java | 2 +-
.../java/org/apache/beam/sdk/testing/PAssert.java | 2 +-
.../apache/beam/sdk/testing/SourceTestUtils.java | 2 +-
.../org/apache/beam/sdk/testing/TestPipeline.java | 45 ++
.../apache/beam/sdk/testing/WindowFnTestUtils.java | 2 +-
.../beam/sdk/transforms/ApproximateUnique.java | 20 +-
.../beam/sdk/transforms/windowing/Trigger.java | 3 +-
.../resources/org/apache/beam/sdk/sdk.properties | 5 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 2 +-
.../org/apache/beam/sdk/coders/AvroCoderTest.java | 2 +-
.../beam/sdk/coders/BigDecimalCoderTest.java | 2 +-
.../beam/sdk/coders/BigEndianIntegerCoderTest.java | 2 +-
.../beam/sdk/coders/BigEndianLongCoderTest.java | 2 +-
.../beam/sdk/coders/BigIntegerCoderTest.java | 2 +-
.../apache/beam/sdk/coders/BitSetCoderTest.java | 2 +-
.../apache/beam/sdk/coders/ByteArrayCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/ByteCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/coders/CoderTest.java | 2 +-
.../beam/sdk/coders/CollectionCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DefaultCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DelegateCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DoubleCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DurationCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/FloatCoderTest.java | 2 +-
.../apache/beam/sdk/coders/InstantCoderTest.java | 2 +-
.../apache/beam/sdk/coders/IterableCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/KvCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/ListCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/MapCoderTest.java | 2 +-
.../apache/beam/sdk/coders/NullableCoderTest.java | 2 +-
.../sdk/coders/PCollectionCustomCoderTest.java | 2 +-
.../beam/sdk/coders/SerializableCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/SetCoderTest.java | 2 +-
.../beam/sdk/coders/StringDelegateCoderTest.java | 2 +-
.../beam/sdk/coders/StringUtf8CoderTest.java | 2 +-
.../beam/sdk/coders/StructuredCoderTest.java | 11 +-
.../beam/sdk/coders/TextualIntegerCoderTest.java | 2 +-
.../apache/beam/sdk/coders/VarIntCoderTest.java | 2 +-
.../apache/beam/sdk/coders/VarLongCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/VoidCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 2 +-
.../org/apache/beam/sdk/io/AvroSourceTest.java | 2 +-
.../sdk/io/BoundedReadFromUnboundedSourceTest.java | 2 +-
.../apache/beam/sdk/io/CompressedSourceTest.java | 2 +-
.../org/apache/beam/sdk/io/CountingSourceTest.java | 2 +-
.../org/apache/beam/sdk/io/FileBasedSinkTest.java | 2 +-
.../apache/beam/sdk/io/FileBasedSourceTest.java | 2 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 2 +-
.../beam/sdk/io/LocalFileSystemRegistrarTest.java | 2 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 2 +-
.../test/java/org/apache/beam/sdk/io/ReadTest.java | 130 +++
.../org/apache/beam/sdk/io/TextIOReadTest.java | 2 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 2 +-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 2 +-
.../io/range/ByteKeyRangeEstimateFractionTest.java | 2 +-
.../io/range/ByteKeyRangeInterpolateKeyTest.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRangeTest.java | 2 +-
.../org/apache/beam/sdk/io/range/ByteKeyTest.java | 2 +-
.../org/apache/beam/sdk/metrics/MetricsTest.java | 2 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 2 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 2 +-
.../beam/sdk/runners/PipelineRunnerTest.java | 2 +-
.../beam/sdk/runners/TransformHierarchyTest.java | 2 +-
.../apache/beam/sdk/runners/TransformTreeTest.java | 2 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 4 +-
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 38 +-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 6 +-
.../apache/beam/sdk/schemas/SchemaCoderTest.java | 2 +-
.../sdk/schemas/transforms/CastValidatorTest.java | 2 +-
.../beam/sdk/schemas/transforms/CoGroupTest.java | 2 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 15 +
.../beam/sdk/schemas/transforms/GroupTest.java | 2 +-
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 26 +-
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 6 +-
.../beam/sdk/schemas/utils/SchemaZipFoldTest.java | 2 +-
.../beam/sdk/schemas/utils/TestJavaBeans.java | 17 +
.../apache/beam/sdk/state/StateContextsTest.java | 2 +-
.../beam/sdk/testing/CoderPropertiesTest.java | 2 +-
.../beam/sdk/testing/CombineFnTesterTest.java | 2 +-
.../org/apache/beam/sdk/testing/PAssertTest.java | 2 +-
.../beam/sdk/testing/PaneExtractorsTest.java | 2 +-
.../beam/sdk/testing/SerializableMatchersTest.java | 2 +-
.../apache/beam/sdk/testing/StaticWindowsTest.java | 2 +-
.../apache/beam/sdk/testing/TestPipelineTest.java | 2 +-
.../beam/sdk/testing/WindowSupplierTest.java | 2 +-
.../apache/beam/sdk/transforms/CombineFnsTest.java | 2 +-
.../apache/beam/sdk/transforms/CombineTest.java | 2 +-
.../org/apache/beam/sdk/transforms/CreateTest.java | 2 +-
.../apache/beam/sdk/transforms/DistinctTest.java | 2 +-
.../org/apache/beam/sdk/transforms/DoFnTest.java | 2 +-
.../apache/beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 2 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 2 +-
.../apache/beam/sdk/transforms/PTransformTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ReifyTest.java | 2 +-
.../beam/sdk/transforms/ReifyTimestampsTest.java | 2 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 2 +-
.../beam/sdk/transforms/SimpleFunctionTest.java | 2 +-
.../beam/sdk/transforms/WithFailuresTest.java | 2 +-
.../display/DisplayDataEvaluatorTest.java | 2 +-
.../display/DisplayDataMatchersTest.java | 2 +-
.../sdk/transforms/display/DisplayDataTest.java | 2 +-
.../beam/sdk/transforms/join/CoGbkResultTest.java | 2 +-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 2 +-
.../beam/sdk/transforms/join/UnionCoderTest.java | 2 +-
.../sdk/transforms/reflect/DoFnInvokersTest.java | 2 +-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 2 +-
.../transforms/reflect/OnTimerInvokersTest.java | 2 +-
.../sdk/transforms/windowing/FixedWindowsTest.java | 2 +-
.../transforms/windowing/IntervalWindowTest.java | 2 +-
.../sdk/transforms/windowing/RepeatedlyTest.java | 2 +-
.../sdk/transforms/windowing/SessionsTest.java | 2 +-
.../transforms/windowing/SlidingWindowsTest.java | 2 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 2 +-
.../org/apache/beam/sdk/util/ApiSurfaceTest.java | 2 +-
.../BufferedElementCountingOutputStreamTest.java | 2 +-
.../apache/beam/sdk/util/FluentBackoffTest.java | 2 +-
.../org/apache/beam/sdk/util/HistogramTest.java | 2 +-
.../org/apache/beam/sdk/util/MoreFuturesTest.java | 2 +-
.../org/apache/beam/sdk/util/ReleaseInfoTest.java | 2 +-
.../beam/sdk/util/UserCodeExceptionTest.java | 2 +-
.../java/org/apache/beam/sdk/util/VarIntTest.java | 2 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 2 +-
.../org/apache/beam/sdk/util/ZipFilesTest.java | 2 +-
.../beam/sdk/util/common/ReflectHelpersTest.java | 2 +-
.../java/org/apache/beam/sdk/values/KVTest.java | 2 +-
.../beam/sdk/values/PCollectionListTest.java | 2 +-
.../beam/sdk/values/PCollectionTupleTest.java | 2 +-
.../org/apache/beam/sdk/values/TupleTagTest.java | 2 +-
.../beam/sdk/values/TypeDescriptorsTest.java | 2 +-
.../apache/beam/sdk/values/TypedPValueTest.java | 2 +-
.../beam/sdk/expansion/service/ExternalTest.java | 2 +-
.../sdk/extensions/gcp/util/GceMetadataUtil.java | 4 +-
.../gcp/storage/GcsFileSystemRegistrarTest.java | 2 +-
.../extensions/gcp/storage/GcsFileSystemTest.java | 2 +-
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 2 +-
...LatencyRecordingHttpRequestInitializerTest.java | 2 +-
.../gcp/util/RetryHttpRequestInitializerTest.java | 2 +-
.../sdk/extensions/gcp/util/gcsfs/GcsPathTest.java | 2 +-
.../extensions/protobuf/ProtoMessageSchema.java | 68 +-
.../extensions/protobuf/ByteStringCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtobufUtilTest.java | 2 +-
.../ExternalSchemaIOTransformRegistrarTest.java | 4 +-
.../extensions/sketching/TDigestQuantilesTest.java | 2 +-
.../extensions/sorter/ExternalSorterBenchmark.java | 6 +-
.../beam/sdk/extensions/sorter/SortValuesTest.java | 2 +-
sdks/java/extensions/sql/build.gradle | 15 +-
.../datacatalog/DataCatalogTableProvider.java | 61 +-
.../sql/meta/provider/datacatalog/SchemaUtils.java | 93 +++
.../meta/provider/datacatalog/SchemaUtilsTest.java | 157 ++++
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 2 +-
.../sdk/extensions/sql/jdbc/BeamSqlLineTest.java | 2 +-
.../sdk/extensions/sql/impl/ScalarFnReflector.java | 73 ++
.../extensions/sql/impl/ScalarFunctionImpl.java | 47 +-
.../sql/meta/provider/bigtable/BigtableFilter.java | 136 ++++
.../sql/meta/provider/bigtable/BigtableTable.java | 47 +-
.../meta/provider/kafka/BeamKafkaThriftTable.java | 138 ++++
.../meta/provider/kafka/KafkaTableProvider.java | 56 +-
.../sql/BeamSqlDslSqlStdOperatorsTest.java | 2 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java | 2 +-
.../extensions/sql/impl/ScalarFnReflectorTest.java | 141 ++++
.../provider/bigtable/BigtableClientWrapper.java | 115 +++
.../meta/provider/bigtable/BigtableFilterTest.java | 114 +++
.../BigtableTableCreationFailuresTest.java | 2 +-
.../provider/bigtable/BigtableTableFlatTest.java | 132 +--
.../meta/provider/bigtable/BigtableTableIT.java | 200 +++++
.../meta/provider/bigtable/BigtableTableTest.java | 94 ---
.../provider/bigtable/BigtableTableTestUtils.java | 237 ++++++
.../bigtable/BigtableTableWithRowsTest.java | 120 ++-
.../provider/kafka/BeamKafkaTableProtoTest.java | 2 +-
...rotoTest.java => BeamKafkaTableThriftTest.java} | 53 +-
.../meta/provider/kafka/KafkaTableProviderIT.java | 214 ++++-
.../provider/kafka/KafkaTableProviderProtoIT.java | 53 --
.../provider/kafka/KafkaTableProviderTest.java | 55 +-
.../provider/kafka/thrift/ItThriftMessage.java | 612 ++++++++++++++
.../provider/kafka/thrift/SimpleThriftMessage.java | 509 ++++++++++++
.../provider/kafka/thrift/TestThriftMessage.java | 881 +++++++++++++++++++++
.../sql/meta/provider/pubsub/PubsubAvroIT.java | 102 ---
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 76 --
.../provider/pubsub/PubsubTableProviderIT.java | 273 +++++--
.../meta/provider/text/TextTableProviderTest.java | 2 +-
.../sql/meta/store/InMemoryMetaStoreTest.java | 2 +-
.../sql/src/test/thrift/kafka/messages.thrift | 36 +-
sdks/{go => java/extensions/sql/udf}/build.gradle | 27 +-
.../beam/sdk/extensions/sql/udf/AggregateFn.java | 71 ++
.../beam/sdk/extensions/sql/udf/ScalarFn.java | 53 ++
.../beam/sdk/extensions/sql/udf/UdfProvider.java} | 23 +-
.../beam/sdk/extensions/sql/udf/package-info.java | 16 +-
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java | 6 +
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 5 +-
.../sql/zetasql/translation/ConversionContext.java | 2 +-
.../zetasql/translation/ExpressionConverter.java | 11 +-
.../sql/zetasql/translation/SqlOperators.java | 76 +-
.../translation/UserFunctionDefinitions.java | 30 +-
.../translation/ZetaSqlScalarFunctionImpl.java | 86 ++
.../zetasketch/ApproximateCountDistinct.java | 288 +++++++
.../zetasketch/ApproximateCountDistinctTest.java | 342 ++++++++
.../sdk/fn/data/BeamFnDataInboundObserver.java | 35 +-
.../beam/sdk/fn/data/DecodingFnDataReceiver.java | 46 ++
.../org/apache/beam/sdk/fn/IdGeneratorsTest.java | 2 +-
.../sdk/fn/channel/SocketAddressFactoryTest.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 2 +-
...DataSizeBasedBufferingOutboundObserverTest.java | 2 +-
.../CompletableFutureInboundDataClientTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 2 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 2 +-
.../beam/sdk/fn/stream/AdvancingPhaserTest.java | 2 +-
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 2 +-
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 2 +-
.../apache/beam/sdk/fn/test/TestStreamsTest.java | 2 +-
.../fn/harness/control/ProcessBundleHandler.java | 87 +-
.../beam/fn/harness/data/BeamFnDataClient.java | 14 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 10 +-
.../fn/harness/data/QueueingBeamFnDataClient.java | 11 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 15 +
.../beam/fn/harness/AssignWindowsRunnerTest.java | 2 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 2 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 2 +-
.../apache/beam/fn/harness/CombineRunnersTest.java | 2 +-
.../apache/beam/fn/harness/FlattenRunnerTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 2 +-
.../apache/beam/fn/harness/MapFnRunnersTest.java | 2 +-
.../fn/harness/PrecombineGroupingTableTest.java | 2 +-
.../beam/fn/harness/WindowMergingFnRunnerTest.java | 2 +-
.../harness/control/BeamFnControlClientTest.java | 2 +-
.../harness/control/BundleSplitListenerTest.java | 2 +-
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 2 +-
.../data/BeamFnDataInboundObserverTest.java | 13 +-
.../data/PCollectionConsumerRegistryTest.java | 2 +-
.../harness/data/QueueingBeamFnDataClientTest.java | 2 +-
.../harness/logging/BeamFnLoggingClientTest.java | 2 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 2 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 2 +-
.../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 40 +-
.../beam/sdk/io/aws/coders/AwsCodersTest.java | 2 +-
.../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 68 +-
.../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java | 2 +-
.../beam/sdk/io/aws/options/AwsModuleTest.java | 2 +-
.../beam/sdk/io/aws/s3/S3FileSystemTest.java | 2 +-
.../sdk/io/aws/sns/PublishResultCodersTest.java | 2 +-
.../aws2/dynamodb/BasicDynamoDbClientProvider.java | 32 +-
.../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 40 +-
.../sdk/io/aws2/options/AwsSerializableUtils.java | 50 ++
.../io/aws2/sns/BasicSnsAsyncClientProvider.java | 32 +-
.../sdk/io/aws2/sns/BasicSnsClientProvider.java | 34 +-
.../sdk/io/aws2/sqs/BasicSqsClientProvider.java | 34 +-
.../dynamodb/BasicDynamoDbClientProviderTest.java | 51 ++
.../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 85 +-
.../beam/sdk/io/aws2/options/AwsModuleTest.java | 2 +-
.../io/aws2/options/AwsSerializableUtilsTest.java | 70 ++
.../aws2/sns/BasicSnsAsyncClientProviderTest.java | 51 ++
.../io/aws2/sns/BasicSnsClientProviderTest.java | 51 ++
.../io/aws2/sqs/BasicSqsClientProviderTest.java | 51 ++
.../blobstore/AzureBlobStoreFileSystemTest.java | 2 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 2 +-
.../io/contextualtextio/ContextualTextIOTest.java | 2 +-
.../elasticsearch/ElasticsearchIOTestCommon.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 7 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 6 +
.../apache/beam/sdk/io/gcp/bigtable/RowUtils.java | 10 +
.../apache/beam/sdk/io/gcp/healthcare/FhirIO.java | 232 ++++++
.../sdk/io/gcp/healthcare/HealthcareApiClient.java | 17 +
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 174 ++++
.../sdk/io/gcp/healthcare/JsonArrayCoder.java} | 34 +-
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 28 +-
.../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 17 +-
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 150 ++--
.../sdk/io/gcp/pubsublite/CloudPubsubChecks.java | 21 +-
...fsetFinalizer.java => InitialOffsetReader.java} | 12 +-
.../io/gcp/pubsublite/InitialOffsetReaderImpl.java | 54 ++
.../gcp/pubsublite/LimitingTopicBacklogReader.java | 81 ++
...ffsetFinalizer.java => OffsetByteProgress.java} | 17 +-
.../io/gcp/pubsublite/OffsetByteRangeTracker.java | 172 ++++
.../io/gcp/pubsublite/OffsetCheckpointMark.java | 78 --
.../pubsublite/PerSubscriptionPartitionSdf.java | 110 +++
.../beam/sdk/io/gcp/pubsublite/PublisherCache.java | 10 +-
.../sdk/io/gcp/pubsublite/PublisherOptions.java | 47 +-
.../sdk/io/gcp/pubsublite/PublisherOrError.java | 6 +-
.../beam/sdk/io/gcp/pubsublite/Publishers.java | 56 ++
.../beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java | 14 +-
.../beam/sdk/io/gcp/pubsublite/PubsubLiteSink.java | 18 +-
.../gcp/pubsublite/PubsubLiteUnboundedReader.java | 333 --------
.../gcp/pubsublite/PubsubLiteUnboundedSource.java | 140 ----
...zer.java => SerializableSubscriberFactory.java} | 17 +-
.../io/gcp/pubsublite/SerializableSupplier.java | 5 +-
.../sdk/io/gcp/pubsublite/SubscribeTransform.java | 124 +++
.../sdk/io/gcp/pubsublite/SubscriberOptions.java | 157 ++--
...etFinalizer.java => SubscriptionPartition.java} | 19 +-
...dCoder.java => SubscriptionPartitionCoder.java} | 36 +-
...er.java => SubscriptionPartitionProcessor.java} | 16 +-
... => SubscriptionPartitionProcessorFactory.java} | 15 +-
.../SubscriptionPartitionProcessorImpl.java | 152 ++++
.../sdk/io/gcp/pubsublite/TopicBacklogReader.java | 34 +-
.../io/gcp/pubsublite/TopicBacklogReaderImpl.java | 80 +-
.../gcp/pubsublite/TopicBacklogReaderSettings.java | 50 +-
.../gcp/pubsublite/TranslatingPullSubscriber.java | 48 --
.../beam/sdk/io/gcp/pubsublite/UuidCoder.java | 7 +-
.../gcp/pubsublite/UuidDeduplicationOptions.java | 15 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 22 +-
.../io/gcp/testing/BigtableEmulatorWrapper.java | 75 --
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 154 ----
.../beam/sdk/io/gcp/testing/BigtableUtils.java | 52 +-
.../cloud/spanner/FakeBatchTransactionId.java | 1 +
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 3 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 2 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 13 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 2 +-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 2 +-
.../sdk/io/gcp/bigquery/TableRowJsonCoderTest.java | 2 +-
.../bigtable/BeamRowToBigtableMutationTest.java | 14 +-
.../sdk/io/gcp/bigtable/BigtableConfigTest.java | 2 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 2 +-
.../gcp/bigtable/BigtableRowToBeamRowFlatTest.java | 14 +-
.../io/gcp/bigtable/BigtableRowToBeamRowTest.java | 17 +-
.../{TestUtils.java => BigtableTestUtils.java} | 50 +-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 2 +-
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 13 +-
.../io/gcp/datastore/AdaptiveThrottlerTest.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 2 +-
.../gcp/datastore/EntityToRowRowToEntityTest.java | 6 +-
.../beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java | 156 ++++
.../beam/sdk/io/gcp/healthcare/FhirIOTest.java | 21 +
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 2 +-
.../pubsub/PubsubMessagePayloadOnlyCoderTest.java | 2 +-
...MessageWithAttributesAndMessageIdCoderTest.java | 2 +-
.../PubsubMessageWithAttributesCoderTest.java | 2 +-
.../PubsubMessageWithMessageIdCoderTest.java | 2 +-
.../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 2 +-
.../gcp/pubsublite/OffsetByteRangeTrackerTest.java | 158 ++++
.../gcp/pubsublite/OffsetCheckpointMarkTest.java | 81 --
.../PerSubscriptionPartitionSdfTest.java | 189 +++++
.../sdk/io/gcp/pubsublite/PubsubLiteSinkTest.java | 47 +-
.../pubsublite/PubsubLiteUnboundedReaderTest.java | 330 --------
.../SubscriptionPartitionProcessorImplTest.java | 227 ++++++
.../gcp/pubsublite/TopicBacklogReaderImplTest.java | 140 +---
.../pubsublite/UuidDeduplicationTransformTest.java | 13 +-
.../io/gcp/spanner/MutationSizeEstimatorTest.java | 2 +-
.../sdk/io/gcp/spanner/ReadSpannerSchemaTest.java | 2 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 2 +-
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 2 +-
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 2 +-
.../sdk/io/hadoop/SerializableConfiguration.java | 11 +
.../io/hadoop/SerializableConfigurationTest.java | 9 +
.../beam/sdk/io/hadoop/WritableCoderTest.java | 2 +-
.../beam/sdk/io/hdfs/HadoopFileSystemOptions.java | 9 +-
.../sdk/io/hdfs/HadoopFileSystemModuleTest.java | 2 +-
.../hdfs/HadoopFileSystemOptionsRegistrarTest.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemOptionsTest.java | 22 +-
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 2 +-
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 85 +-
.../hadoop/format/HadoopFormatIOCassandraTest.java | 6 +-
.../io/hadoop/format/HadoopFormatIOReadTest.java | 162 +++-
.../org/apache/beam/sdk/io/hbase/HBaseIOTest.java | 2 +-
.../beam/sdk/io/hcatalog/HCatalogIOTest.java | 2 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 2 +-
.../CustomTimestampPolicyWithLimitedDelayTest.java | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 8 +-
.../apache/beam/sdk/io/kinesis/KinesisIOIT.java | 2 +-
.../java/org/apache/beam/sdk/io/kudu/KuduIOIT.java | 4 +-
.../sdk/io/parquet/GenericRecordReadConverter.java | 64 --
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 472 +++++++++--
.../sdk/io/parquet/ParquetSchemaIOProvider.java | 11 +-
.../sdk/io/parquet/GenericRecordToRowTest.java | 79 --
.../apache/beam/sdk/io/parquet/ParquetIOTest.java | 132 ++-
.../apache/beam/sdk/io/snowflake/KeyPairUtils.java | 4 +-
.../io/snowflake/crosslanguage/ReadBuilder.java | 4 +-
.../services/SnowflakeBatchServiceImpl.java | 4 +-
.../test/FakeSnowflakeStreamingServiceImpl.java | 4 +-
.../beam/sdk/io/snowflake/test/TestUtils.java | 6 +-
.../beam/sdk/io/solr/JavaBinCodecCoderTest.java | 2 +-
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 4 +-
sdks/java/io/thrift/build.gradle | 10 +-
.../org/apache/beam/sdk/io/thrift/ThriftCoder.java | 9 +-
.../apache/beam/sdk/io/thrift/ThriftSchema.java | 408 ++++++++++
.../apache/beam/sdk/io/thrift/TestThriftEnum.java} | 45 +-
.../beam/sdk/io/thrift/TestThriftInnerStruct.java | 526 ++++++++++++
.../beam/sdk/io/thrift/TestThriftStruct.java | 760 ++++++++++++++++--
.../apache/beam/sdk/io/thrift/TestThriftUnion.java | 401 ++++++++++
.../apache/beam/sdk/io/thrift/ThriftIOTest.java | 38 +-
.../beam/sdk/io/thrift/ThriftSchemaTest.java | 251 ++++++
.../src/test/resources/thrift/thrift_test.thrift | 35 +-
.../apache/beam/sdk/io/tika/ParseResultTest.java | 2 +-
.../org/apache/beam/sdk/io/tika/TikaIOTest.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlSource.java | 22 +-
.../org/apache/beam/sdk/io/xml/JAXBCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOTest.java | 11 +-
.../org/apache/beam/sdk/io/xml/XmlSourceTest.java | 44 +-
sdks/java/maven-archetypes/examples/build.gradle | 2 +-
sdks/java/testing/jpms-tests/build.gradle | 2 +-
.../java/org/apache/beam/sdk/nexmark/Monitor.java | 13 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 50 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 14 +
.../beam/sdk/nexmark/queries/NexmarkQuery.java | 3 -
.../sdk/nexmark/queries/NexmarkQueryModel.java | 4 +-
.../apache/beam/sdk/nexmark/queries/Query10.java | 8 +-
.../apache/beam/sdk/nexmark/queries/Query3.java | 191 +++--
.../apache/beam/sdk/nexmark/queries/Query5.java | 133 +++-
.../beam/sdk/nexmark/NexmarkConfigurationTest.java | 2 +-
.../beam/sdk/nexmark/PerfsToBigQueryTest.java | 2 +-
.../nexmark/queries/BoundedSideInputJoinTest.java | 2 +-
.../nexmark/queries/SessionSideInputJoinTest.java | 2 +-
.../queries/sql/SqlBoundedSideInputJoinTest.java | 2 +-
sdks/python/apache_beam/coders/coders.py | 71 +-
.../apache_beam/coders/coders_test_common.py | 1 -
sdks/python/apache_beam/coders/typecoders.py | 3 -
sdks/python/apache_beam/dataframe/frames.py | 26 +-
sdks/python/apache_beam/dataframe/frames_test.py | 78 +-
sdks/python/apache_beam/dataframe/io.py | 263 ++++--
sdks/python/apache_beam/dataframe/io_test.py | 88 +-
.../examples/cookbook/bigquery_side_input.py | 7 +-
.../apache_beam/examples/cookbook/filters.py | 2 +-
.../apache_beam/examples/snippets/snippets.py | 16 +-
.../apache_beam/examples/snippets/snippets_test.py | 10 +-
sdks/python/apache_beam/examples/sql_taxi.py | 3 +-
sdks/python/apache_beam/io/concat_source.py | 9 +-
sdks/python/apache_beam/io/concat_source_test.py | 8 +
.../flink/flink_streaming_impulse_source_test.py | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 2 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 21 +-
.../apache_beam/io/gcp/bigquery_read_internal.py | 40 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 14 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 8 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 3 +-
sdks/python/apache_beam/io/mongodbio.py | 29 +-
sdks/python/apache_beam/io/mongodbio_test.py | 41 +
sdks/python/apache_beam/io/restriction_trackers.py | 2 +-
sdks/python/apache_beam/metrics/execution.pxd | 1 +
sdks/python/apache_beam/metrics/execution.py | 9 +-
.../python/apache_beam/options/pipeline_options.py | 29 +-
sdks/python/apache_beam/pipeline.py | 163 ++--
sdks/python/apache_beam/portability/common_urns.py | 1 +
sdks/python/apache_beam/runners/common.pxd | 1 +
sdks/python/apache_beam/runners/common.py | 36 +-
.../runners/dataflow/dataflow_runner.py | 184 ++---
.../runners/dataflow/dataflow_runner_test.py | 62 ++
.../apache_beam/runners/dataflow/internal/names.py | 9 +-
.../runners/dataflow/ptransform_overrides.py | 45 ++
.../apache_beam/runners/direct/direct_runner.py | 4 +-
.../apache_beam/runners/direct/direct_userstate.py | 35 +-
.../runners/direct/transform_evaluator.py | 8 +-
sdks/python/apache_beam/runners/direct/util.py | 18 +-
.../runners/direct/watermark_manager.py | 10 +-
.../python/apache_beam/runners/pipeline_context.py | 2 -
.../runners/portability/flink_runner_test.py | 3 +
.../portability/fn_api_runner/fn_runner_test.py | 24 +
.../portability/fn_api_runner/translations.py | 81 +-
.../portability/fn_api_runner/translations_test.py | 18 +-
.../portability/fn_api_runner/worker_handlers.py | 7 +-
.../apache_beam/runners/portability/job_server.py | 10 +-
.../runners/portability/job_server_test.py | 84 ++
.../runners/portability/portable_runner.py | 85 +-
.../runners/portability/sdk_container_builder.py | 11 +-
.../runners/portability/spark_runner_test.py | 5 +-
sdks/python/apache_beam/runners/runner.py | 24 +-
.../apache_beam/runners/worker/bundle_processor.py | 41 +-
.../apache_beam/runners/worker/operations.py | 6 +-
sdks/python/apache_beam/transforms/core.py | 145 +---
sdks/python/apache_beam/transforms/environments.py | 26 +-
sdks/python/apache_beam/transforms/external.py | 24 +-
sdks/python/apache_beam/transforms/ptransform.py | 46 +-
sdks/python/apache_beam/transforms/trigger.py | 39 +-
sdks/python/apache_beam/transforms/trigger_test.py | 10 +-
sdks/python/apache_beam/transforms/userstate.py | 55 +-
.../apache_beam/transforms/userstate_test.py | 216 ++++-
sdks/python/apache_beam/transforms/util.py | 123 ++-
sdks/python/apache_beam/transforms/util_test.py | 65 ++
.../typehints/native_type_compatibility.py | 7 -
.../apache_beam/typehints/sharded_key_type.py | 31 +-
.../apache_beam/typehints/sharded_key_type_test.py | 35 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/container/common.gradle | 2 +
sdks/python/setup.py | 6 +-
sdks/python/test-suites/dataflow/build.gradle | 24 +
sdks/python/test-suites/gradle.properties | 5 +
sdks/python/test-suites/portable/common.gradle | 34 +-
settings.gradle | 5 +
start-build-env.sh | 16 +-
website/.gitignore | 23 -
website/www/package.json | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.13.0.md | 2 +-
website/www/site/content/en/blog/beam-2.21.0.md | 2 +-
website/www/site/content/en/blog/beam-2.22.0.md | 2 +-
website/www/site/content/en/blog/beam-2.23.0.md | 4 +-
website/www/site/content/en/blog/beam-2.24.0.md | 4 +-
website/www/site/content/en/blog/beam-2.25.0.md | 4 +-
website/www/site/content/en/blog/beam-2.26.0.md | 65 ++
website/www/site/content/en/blog/beam-2.27.0.md | 69 ++
.../en/blog/dataframe-api-preview-available.md | 178 +++++
.../site/content/en/blog/pattern-match-beam-sql.md | 2 +-
.../en/blog/splittable-do-fn-is-available.md | 91 +++
website/www/site/content/en/contribute/_index.md | 2 +-
.../site/content/en/contribute/release-guide.md | 7 +-
.../en/documentation/dsls/dataframes/overview.md | 131 +++
.../dsls/sql/extensions/create-external-table.md | 15 +-
.../content/en/documentation/dsls/sql/overview.md | 2 +-
.../documentation/io/built-in/google-bigquery.md | 15 +-
.../en/documentation/io/built-in/snowflake.md | 6 +-
.../content/en/documentation/programming-guide.md | 4 +-
.../content/en/documentation/runners/direct.md | 112 +--
.../site/content/en/documentation/runners/flink.md | 42 +-
.../site/content/en/documentation/runners/samza.md | 5 +
.../en/documentation/runtime/environments.md | 305 ++++---
.../site/content/en/get-started/beam-overview.md | 5 +-
.../www/site/content/en/get-started/downloads.md | 28 +-
.../www/site/content/en/get-started/from-spark.md | 268 +++++++
website/www/site/data/authors.yml | 38 +-
website/www/site/data/capability_matrix.yaml | 714 +++++++++++++++--
.../www/site/data/capability_matrix_snapshot.yaml | 195 +++++
website/www/site/layouts/blog/single.html | 1 +
.../partials/section-menu/en/get-started.html | 7 +-
.../layouts/partials/section-menu/en/sdks.html | 17 +-
website/www/site/static/.htaccess | 2 +-
941 files changed, 26256 insertions(+), 6803 deletions(-)
copy .test-infra/jenkins/{job_LoadTests_Combine_Flink_Go.groovy => job_LoadTests_Combine_Go.groovy} (56%)
create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Go.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Go.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_SideInput_Flink_Go.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_SideInput_Go.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_coGBK_Go.groovy
rename .test-infra/jenkins/{job_PostCommit_Python36.groovy => job_PostCommit_Python.groovy} (51%)
delete mode 100644 .test-infra/jenkins/job_PostCommit_Python37.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Python38.groovy
copy runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OutputManagerFactory.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaPubsubConstants.java (64%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsub.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/README.md
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/avro/AvroDataClass.java
rename sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderJsonIT.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/avro/AvroDataClassKafkaAvroDeserializer.java (53%)
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/avro/package-info.java (74%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/SslConsumerFactoryFn.java
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/Utils.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/kafka/consumer/package-info.java (74%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options/KafkaToPubsubOptions.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableStatusFunction.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/options/package-info.java (72%)
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/package-info.java (74%)
create mode 100644 examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/transforms/FormatTransform.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SerializableSupplier.java => examples/java/src/main/java/org/apache/beam/examples/complete/kafkatopubsub/transforms/package-info.java (74%)
create mode 100644 examples/java/src/test/java/org/apache/beam/examples/complete/kafkatopubsub/KafkaToPubsubTest.java
copy {sdks/go => runners/flink/1.12}/build.gradle (50%)
copy runners/flink/{1.9 => 1.12}/job-server-container/build.gradle (100%)
copy {sdks/go => runners/flink/1.12/job-server}/build.gradle (53%)
create mode 100644 runners/flink/1.12/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/AbstractStreamOperatorCompat.java
copy runners/flink/{1.10 => 1.12}/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java (80%)
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaJobInvocation.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/container/BeamJobCoordinatorRunner.java
rename runners/samza/src/main/java/org/apache/beam/runners/samza/container/{ContainerCfgFactory.java => ContainerCfgLoader.java} (79%)
copy runners/samza/src/main/java/org/apache/beam/runners/samza/{runtime/OutputManagerFactory.java => container/ContainerCfgLoaderFactory.java} (65%)
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/BundleManager.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/FutureCollector.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTestStreamSystemFactory.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/translation/SamzaTestStreamTranslator.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/FutureUtils.java
create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/SamzaPipelineOptionsValidatorTest.java
create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/BundleManagerTest.java
create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/runtime/FutureCollectorImplTest.java
create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/translation/TranslationContextTest.java
create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/FutureUtilsTest.java
create mode 100644 sdks/go/pkg/beam/core/graph/coder/double_test.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/row_decoder.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/row_encoder.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/testutil/testutil.go
create mode 100644 sdks/go/pkg/beam/core/graph/coder/testutil/testutil_test.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/metrics.go
create mode 100644 sdks/go/pkg/beam/runners/dataflow/dataflowlib/metrics_test.go
rename sdks/go/test/integration/{ => driver}/driver.go (100%)
create mode 100644 sdks/go/test/integration/integration.go
create mode 100644 sdks/go/test/integration/synthetic/synthetic_test.go
create mode 100644 sdks/go/test/integration/xlang/xlang_test.go
create mode 100644 sdks/go/test/load/sideinput/sideinput.go
copy sdks/go/test/{run_integration_tests.sh => run_validatesrunner_tests.sh} (57%)
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/GenericRecordWriteConverter.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/RowMessages.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/SchemaUtilsTest.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFnReflector.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilter.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/BeamKafkaThriftTable.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/ScalarFnReflectorTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableClientWrapper.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableFilterTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableIT.java
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableTestUtils.java
copy sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/{BeamKafkaTableProtoTest.java => BeamKafkaTableThriftTest.java} (63%)
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderProtoIT.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift/ItThriftMessage.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift/SimpleThriftMessage.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/thrift/TestThriftMessage.java
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubAvroIT.java
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/pubsub/PubsubJsonIT.java
copy runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineOptions.java => sdks/java/extensions/sql/src/test/thrift/kafka/messages.thrift (54%)
copy sdks/{go => java/extensions/sql/udf}/build.gradle (56%)
create mode 100644 sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/AggregateFn.java
create mode 100644 sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/ScalarFn.java
copy sdks/java/{io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetFinalizer.java => extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/UdfProvider.java} (57%)
copy runners/samza/src/main/java/org/apache/beam/runners/samza/runtime/OutputManagerFactory.java => sdks/java/extensions/sql/udf/src/main/java/org/apache/beam/sdk/extensions/sql/udf/package-info.java (68%)
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ZetaSqlScalarFunctionImpl.java
create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/ApproximateCountDistinct.java
create mode 100644 sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/ApproximateCountDistinctTest.java
create mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/data/DecodingFnDataReceiver.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsSerializableUtils.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProviderTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsSerializableUtilsTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsAsyncClientProviderTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProviderTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sqs/BasicSqsClientProviderTest.java
rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderCSVIT.java => io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/JsonArrayCoder.java} (52%)
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => InitialOffsetReader.java} (77%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/InitialOffsetReaderImpl.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/LimitingTopicBacklogReader.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => OffsetByteProgress.java} (68%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetByteRangeTracker.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetCheckpointMark.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PerSubscriptionPartitionSdf.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/Publishers.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedReader.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedSource.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => SerializableSubscriberFactory.java} (64%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SubscribeTransform.java
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => SubscriptionPartition.java} (64%)
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{UuidCoder.java => SubscriptionPartitionCoder.java} (50%)
rename sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{OffsetFinalizer.java => SubscriptionPartitionProcessor.java} (69%)
rename sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/{SerializableStatusFunction.java => SubscriptionPartitionProcessorFactory.java} (60%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/SubscriptionPartitionProcessorImpl.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsublite/TranslatingPullSubscriber.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableEmulatorWrapper.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableTestUtils.java
copy runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/translation/PipelineTranslatorUtilsTest.java => sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableUtils.java (54%)
rename sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/{TestUtils.java => BigtableTestUtils.java} (71%)
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetByteRangeTrackerTest.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/OffsetCheckpointMarkTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/PerSubscriptionPartitionSdfTest.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/PubsubLiteUnboundedReaderTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsublite/SubscriptionPartitionProcessorImplTest.java
delete mode 100644 sdks/java/io/parquet/src/main/java/org/apache/beam/sdk/io/parquet/GenericRecordReadConverter.java
delete mode 100644 sdks/java/io/parquet/src/test/java/org/apache/beam/sdk/io/parquet/GenericRecordToRowTest.java
create mode 100644 sdks/java/io/thrift/src/main/java/org/apache/beam/sdk/io/thrift/ThriftSchema.java
rename sdks/java/{extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/kafka/KafkaTableProviderAvroIT.java => io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/TestThriftEnum.java} (50%)
create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/TestThriftInnerStruct.java
create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/TestThriftUnion.java
create mode 100644 sdks/java/io/thrift/src/test/java/org/apache/beam/sdk/io/thrift/ThriftSchemaTest.java
create mode 100644 sdks/python/apache_beam/runners/portability/job_server_test.py
delete mode 100644 website/.gitignore
create mode 100644 website/www/site/content/en/blog/beam-2.26.0.md
create mode 100644 website/www/site/content/en/blog/beam-2.27.0.md
create mode 100644 website/www/site/content/en/blog/dataframe-api-preview-available.md
create mode 100644 website/www/site/content/en/blog/splittable-do-fn-is-available.md
create mode 100644 website/www/site/content/en/documentation/dsls/dataframes/overview.md
create mode 100644 website/www/site/content/en/get-started/from-spark.md
[beam] 01/01: Merge remote-tracking branch 'origin/master' into
website-revamp
Posted by bh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 7d0d220278b65376ab84832421be613e901b1177
Merge: 8b22fb4 53615fe
Author: Brian Hulette <bh...@google.com>
AuthorDate: Tue Jan 12 17:53:24 2021 -0800
Merge remote-tracking branch 'origin/master' into website-revamp
.github/workflows/build_wheels.yml | 2 +-
.github/workflows/cancel.yml | 2 +-
.gitignore | 10 +
.test-infra/jenkins/README.md | 9 +-
.test-infra/jenkins/job_Inventory.groovy | 15 +-
.../jenkins/job_LoadTests_Combine_Flink_Go.groovy | 2 +-
...k_Go.groovy => job_LoadTests_Combine_Go.groovy} | 85 +-
.../jenkins/job_LoadTests_GBK_Flink_Go.groovy | 19 +-
.test-infra/jenkins/job_LoadTests_GBK_Go.groovy | 211 +++++
.test-infra/jenkins/job_LoadTests_ParDo_Go.groovy | 145 ++++
.../job_LoadTests_SideInput_Flink_Go.groovy | 103 +++
.../jenkins/job_LoadTests_SideInput_Go.groovy | 94 +++
.test-infra/jenkins/job_LoadTests_coGBK_Go.groovy | 169 ++++
...ython36.groovy => job_PostCommit_Python.groovy} | 39 +-
.test-infra/jenkins/job_PostCommit_Python37.groovy | 44 -
.test-infra/jenkins/job_PostCommit_Python38.groovy | 44 -
...stCommit_Python_ValidatesRunner_Dataflow.groovy | 8 +-
...ommit_Python_ValidatesRunner_Dataflow_V2.groovy | 9 +-
.../jenkins/job_Publish_SDK_Image_Snapshots.groovy | 8 +-
.../perftests_metrics/SideInput_Load_Tests.json | 2 +-
CHANGES.md | 51 +-
build.gradle | 9 -
buildSrc/build.gradle | 4 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 31 +-
examples/java/build.gradle | 11 +-
.../org/apache/beam/examples/complete/README.md | 5 +
.../kafkatopubsub/KafkaPubsubConstants.java | 15 +-
.../complete/kafkatopubsub/KafkaToPubsub.java | 235 ++++++
.../beam/examples/complete/kafkatopubsub/README.md | 200 +++++
.../complete/kafkatopubsub/avro/AvroDataClass.java | 63 ++
.../avro/AvroDataClassKafkaAvroDeserializer.java | 27 +-
.../complete/kafkatopubsub/avro/package-info.java | 10 +-
.../kafka/consumer/SslConsumerFactoryFn.java | 135 ++++
.../kafkatopubsub/kafka/consumer/Utils.java | 165 ++++
.../kafkatopubsub/kafka/consumer/package-info.java | 10 +-
.../options/KafkaToPubsubOptions.java | 90 +++
.../kafkatopubsub/options/package-info.java | 10 +-
.../complete/kafkatopubsub/package-info.java | 10 +-
.../kafkatopubsub/transforms/FormatTransform.java | 128 +++
.../kafkatopubsub/transforms/package-info.java | 10 +-
.../examples/complete/game/LeaderBoardTest.java | 2 +-
.../complete/kafkatopubsub/KafkaToPubsubTest.java | 90 +++
.../cookbook/CombinePerKeyExamplesTest.java | 13 +-
examples/kotlin/build.gradle | 2 +-
.../beam/examples/kotlin/MinimalWordCount.kt | 2 +-
gradle.properties | 7 +-
gradle/wrapper/gradle-wrapper.properties | 2 +-
.../beam/learning/katas/util/ContainsKvs.java | 2 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 18 +
.../src/main/scripts/build_release_candidate.sh | 6 +-
.../src/main/scripts/preparation_before_release.sh | 2 +
release/src/main/scripts/publish_docker_images.sh | 4 +-
.../core/construction/CoderTranslationTest.java | 2 +-
.../runners/core/construction/CommonCoderTest.java | 2 +-
.../CreatePCollectionViewTranslationTest.java | 2 +-
.../DeduplicatedFlattenFactoryTest.java | 2 +-
.../construction/DefaultArtifactResolverTest.java | 2 +-
.../EmptyFlattenAsCreateFactoryTest.java | 2 +-
.../core/construction/EnvironmentsTest.java | 2 +-
.../ExecutableStageTranslationTest.java | 2 +-
.../construction/ForwardingPTransformTest.java | 2 +-
.../construction/GroupByKeyTranslationTest.java | 2 +-
.../runners/core/construction/ModelCodersTest.java | 2 +-
.../core/construction/MorePipelineTest.java | 2 +-
.../construction/PCollectionTranslationTest.java | 2 +-
.../core/construction/PTransformMatchersTest.java | 2 +-
.../construction/PTransformReplacementsTest.java | 2 +-
.../construction/PTransformTranslationTest.java | 2 +-
.../core/construction/ParDoTranslationTest.java | 2 +-
.../PipelineOptionsTranslationTest.java | 2 +-
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/construction/ReadTranslationTest.java | 2 +-
.../construction/RehydratedComponentsTest.java | 2 +-
.../core/construction/ReplacementOutputsTest.java | 2 +-
.../construction/ReshuffleTranslationTest.java | 2 +-
.../core/construction/SchemaTranslationTest.java | 2 +-
.../core/construction/SdkComponentsTest.java | 2 +-
.../SingleInputOutputOverrideFactoryTest.java | 2 +-
.../construction/TestStreamTranslationTest.java | 2 +-
.../core/construction/TransformInputsTest.java | 2 +-
.../core/construction/TriggerTranslationTest.java | 2 +-
.../core/construction/UnconsumedReadsTest.java | 2 +-
.../core/construction/ValidateRunnerXlangTest.java | 2 +-
.../WindowingStrategyTranslationTest.java | 161 +++-
.../construction/WriteFilesTranslationTest.java | 2 +-
.../construction/graph/ExecutableStageTest.java | 2 +-
.../core/construction/graph/FusedPipelineTest.java | 2 +-
.../graph/GreedyPipelineFuserTest.java | 2 +-
.../construction/graph/GreedyStageFuserTest.java | 2 +-
.../graph/ImmutableExecutableStageTest.java | 2 +-
.../core/construction/graph/NetworksTest.java | 2 +-
.../construction/graph/OutputDeduplicatorTest.java | 2 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
.../construction/graph/QueryablePipelineTest.java | 2 +-
.../core/metrics/ExecutionStateTracker.java | 20 +-
.../runners/core/InMemoryStateInternalsTest.java | 2 +-
.../runners/core/InMemoryTimerInternalsTest.java | 2 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 2 +-
.../beam/runners/core/LateDataUtilsTest.java | 2 +-
...BoundedSplittableProcessElementInvokerTest.java | 2 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 2 +-
.../beam/runners/core/SideInputHandlerTest.java | 2 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 2 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 2 +-
.../runners/core/SplittableParDoProcessFnTest.java | 2 +-
.../beam/runners/core/StateNamespacesTest.java | 2 +-
.../beam/runners/core/TimerInternalsTest.java | 2 +-
.../beam/runners/core/WindowMatchersTest.java | 2 +-
.../beam/runners/core/metrics/CounterCellTest.java | 2 +-
.../beam/runners/core/metrics/DirtyStateTest.java | 2 +-
.../runners/core/metrics/DistributionCellTest.java | 2 +-
.../core/metrics/ExecutionStateSamplerTest.java | 2 +-
.../core/metrics/ExecutionStateTrackerTest.java | 2 +-
.../beam/runners/core/metrics/GaugeCellTest.java | 2 +-
.../core/metrics/MetricsContainerImplTest.java | 2 +-
.../core/metrics/MetricsContainerStepMapTest.java | 2 +-
.../beam/runners/core/metrics/MetricsMapTest.java | 2 +-
.../runners/core/metrics/MetricsPusherTest.java | 2 +-
.../core/metrics/MonitoringInfoConstantsTest.java | 2 +-
.../core/metrics/SimpleStateRegistryTest.java | 2 +-
.../triggers/AfterWatermarkStateMachineTest.java | 2 +-
.../core/triggers/FinishedTriggersBitSetTest.java | 2 +-
.../core/triggers/FinishedTriggersSetTest.java | 2 +-
.../core/triggers/NeverStateMachineTest.java | 2 +-
.../core/triggers/TriggerStateMachinesTest.java | 2 +-
.../direct/BoundedReadEvaluatorFactoryTest.java | 2 +-
.../runners/direct/CloningBundleFactoryTest.java | 2 +-
.../beam/runners/direct/CommittedResultTest.java | 2 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 2 +-
.../runners/direct/DirectGraphVisitorTest.java | 2 +-
.../DirectGroupByKeyOverrideFactoryTest.java | 2 +-
.../beam/runners/direct/DirectMetricsTest.java | 2 +-
.../runners/direct/DirectRunnerApiSurfaceTest.java | 2 +-
.../beam/runners/direct/DirectRunnerTest.java | 2 +-
.../runners/direct/DirectTimerInternalsTest.java | 2 +-
.../direct/DirectTransformExecutorTest.java | 2 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 2 +-
.../runners/direct/DoFnLifecycleManagerTest.java | 2 +-
.../beam/runners/direct/EvaluationContextTest.java | 2 +-
.../direct/FlattenEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyOnlyEvaluatorFactoryTest.java | 2 +-
.../ImmutabilityCheckingBundleFactoryTest.java | 2 +-
.../direct/ImmutableListBundleFactoryTest.java | 2 +-
.../direct/ImpulseEvaluatorFactoryTest.java | 2 +-
.../direct/KeyedPValueTrackingVisitorTest.java | 2 +-
.../beam/runners/direct/MultiStepCombineTest.java | 2 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 2 +-
.../runners/direct/SideInputContainerTest.java | 2 +-
.../direct/StatefulParDoEvaluatorFactoryTest.java | 2 +-
.../runners/direct/StepTransformResultTest.java | 2 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 2 +-
.../direct/UnboundedReadDeduplicatorTest.java | 2 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 2 +-
.../runners/direct/ViewEvaluatorFactoryTest.java | 2 +-
.../direct/WatermarkCallbackExecutorTest.java | 2 +-
.../beam/runners/direct/WatermarkManagerTest.java | 2 +-
.../runners/direct/WindowEvaluatorFactoryTest.java | 2 +-
.../direct/WriteWithShardingFactoryTest.java | 2 +-
.../runners/flink/streaming/StreamSources.java | 13 +
{sdks/go => runners/flink/1.12}/build.gradle | 38 +-
.../flink/1.12/job-server-container/build.gradle | 16 +-
.../flink/1.12/job-server}/build.gradle | 34 +-
.../streaming/AbstractStreamOperatorCompat.java | 53 ++
.../runners/flink/streaming/StreamSources.java | 14 +
.../runners/flink/streaming/StreamSources.java | 13 +
.../runners/flink/streaming/StreamSources.java | 13 +
runners/flink/flink_runner.gradle | 19 +-
.../runners/flink/FlinkExecutionEnvironments.java | 12 +
.../flink/FlinkPortableClientEntryPoint.java | 4 +-
.../FlinkStreamingPortablePipelineTranslator.java | 27 +-
.../flink/FlinkStreamingTransformTranslators.java | 215 +++--
.../flink/FlinkStreamingTranslationContext.java | 24 +-
.../wrappers/streaming/DoFnOperator.java | 77 +-
.../streaming/KeyedPushedBackElementsHandler.java | 5 +-
.../streaming/SingletonKeyedWorkItemCoder.java | 13 +-
.../wrappers/streaming/WorkItemKeySelector.java | 13 +-
.../streaming/state/FlinkStateInternals.java | 4 +-
.../FlinkStreamingPipelineTranslatorTest.java | 109 +++
.../FlinkStreamingTransformTranslatorsTest.java | 7 +-
.../runners/flink/FlinkTransformOverridesTest.java | 2 +-
.../wrappers/streaming/DedupingOperatorTest.java | 2 +-
.../wrappers/streaming/DoFnOperatorTest.java | 315 ++++++--
.../wrappers/streaming/WindowDoFnOperatorTest.java | 5 +-
runners/google-cloud-dataflow-java/build.gradle | 2 +-
.../beam/runners/dataflow/DataflowRunner.java | 53 +-
.../options/DataflowPipelineDebugOptions.java | 10 +
.../dataflow/BatchStatefulParDoOverridesTest.java | 2 +-
.../runners/dataflow/BatchViewOverridesTest.java | 2 +-
.../dataflow/DataflowPTransformMatchersTest.java | 2 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 2 +-
.../runners/dataflow/DataflowRunnerInfoTest.java | 2 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 56 ++
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../runners/dataflow/TestDataflowRunnerTest.java | 2 +-
.../options/DataflowPipelineDebugOptionsTest.java | 2 +-
.../options/DataflowProfilingOptionsTest.java | 2 +-
.../runners/dataflow/util/PackageUtilTest.java | 2 +-
.../beam/runners/dataflow/util/StructsTest.java | 25 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 38 +-
.../dataflow/worker/StreamingDataflowWorker.java | 61 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 14 +-
.../logging/DataflowWorkerLoggingInitializer.java | 14 +-
.../JulHandlerPrintStreamAdapterFactory.java | 15 +-
.../dataflow/harness/test/TestStreamsTest.java | 2 +-
.../dataflow/worker/AvroByteReaderFactoryTest.java | 5 +-
.../dataflow/worker/AvroByteSinkFactoryTest.java | 3 +-
.../dataflow/worker/BatchDataflowWorkerTest.java | 2 +-
.../worker/BatchModeExecutionContextTest.java | 2 +-
.../dataflow/worker/ConcatReaderFactoryTest.java | 2 +-
.../runners/dataflow/worker/ConcatReaderTest.java | 2 +-
.../ContextActivationObserverRegistryTest.java | 2 +-
...CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java | 2 +-
.../DataflowElementExecutionTrackerTest.java | 2 +-
.../worker/DataflowExecutionStateTrackerTest.java | 2 +-
.../worker/DataflowSideInputReadCounterTest.java | 2 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 2 +-
.../dataflow/worker/DefaultParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/DeltaCounterCellTest.java | 2 +-
.../dataflow/worker/DoFnInstanceManagersTest.java | 2 +-
.../dataflow/worker/FakeWindmillServer.java | 14 +-
.../runners/dataflow/worker/FilepatternsTest.java | 2 +-
.../GroupAlsoByWindowParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/InMemoryReaderFactoryTest.java | 3 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 2 +-
.../worker/IntrinsicMapTaskExecutorTest.java | 2 +-
.../runners/dataflow/worker/IsmFormatTest.java | 2 +-
.../dataflow/worker/IsmSideInputReaderTest.java | 2 +-
.../dataflow/worker/LogRecordMatcherTest.java | 2 +-
.../beam/runners/dataflow/worker/LogSaverTest.java | 2 +-
...tContextActivationObserverRegistrationTest.java | 2 +-
.../worker/PairWithConstantKeyDoFnFactoryTest.java | 2 +-
.../worker/PartialGroupByKeyParDoFnsTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 6 +-
.../runners/dataflow/worker/ReaderFactoryTest.java | 6 +-
...ReifyTimestampAndWindowsParDoFnFactoryTest.java | 2 +-
...essCoderCloudObjectTranslatorRegistrarTest.java | 2 +-
.../dataflow/worker/ShuffleReaderFactoryTest.java | 3 +-
.../dataflow/worker/ShuffleSinkFactoryTest.java | 3 +-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 2 +-
.../runners/dataflow/worker/SinkRegistryTest.java | 5 +-
.../worker/SourceOperationExecutorFactoryTest.java | 2 +-
.../runners/dataflow/worker/StateFetcherTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 10 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
.../worker/StreamingModeExecutionContextTest.java | 5 +-
...eamingPCollectionViewWriterDoFnFactoryTest.java | 2 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 2 +-
.../worker/StreamingStepMetricsContainerTest.java | 2 +-
.../ToIsmRecordForMultimapDoFnFactoryTest.java | 2 +-
.../dataflow/worker/UserParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/ValuesDoFnFactoryTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillStateInternalsTest.java | 2 +-
.../dataflow/worker/WindmillStateReaderTest.java | 2 +-
.../worker/WindmillTimerInternalsTest.java | 2 +-
.../dataflow/worker/WorkItemStatusClientTest.java | 2 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 2 +-
.../dataflow/worker/counters/CounterSetTest.java | 2 +-
.../fn/control/BeamFnMapTaskExecutorTest.java | 2 +-
.../RegisterAndProcessBundleOperationTest.java | 2 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 2 +-
.../fn/data/RemoteGrpcPortReadOperationTest.java | 2 +-
.../fn/data/RemoteGrpcPortWriteOperationTest.java | 2 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 2 +-
.../fn/stream/ServerStreamObserverFactoryTest.java | 2 +-
.../graph/CloneAmbiguousFlattensFunctionTest.java | 2 +-
.../CreateRegisterFnOperationFunctionTest.java | 2 +-
.../graph/DeduceNodeLocationsFunctionTest.java | 2 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 2 +-
.../worker/graph/MapTaskToNetworkFunctionTest.java | 2 +-
.../dataflow/worker/graph/NetworksTest.java | 2 +-
.../RemoveFlattenInstructionsFunctionTest.java | 2 +-
.../ReplacePgbkWithPrecombineFunctionTest.java | 2 +-
.../DataflowWorkerLoggingInitializerTest.java | 2 +-
.../JulHandlerPrintStreamAdapterFactoryTest.java | 16 +-
.../worker/profiler/ScopedProfilerTest.java | 2 +-
.../dataflow/worker/status/ThreadzServletTest.java | 2 +-
.../worker/status/WorkerStatusPagesTest.java | 2 +-
.../worker/util/BatchGroupAlsoByWindowFnsTest.java | 2 +-
.../BatchGroupAlsoByWindowReshuffleDoFnTest.java | 2 +-
.../worker/util/GroupAlsoByWindowProperties.java | 2 +-
.../dataflow/worker/util/MemoryMonitorTest.java | 2 +-
.../worker/util/ScalableBloomFilterTest.java | 2 +-
.../dataflow/worker/util/TimerOrElementTest.java | 2 +-
.../worker/BatchingShuffleEntryReaderTest.java | 2 +-
.../worker/CachingShuffleBatchReaderTest.java | 2 +-
.../util/common/worker/FlattenOperationTest.java | 5 +-
.../util/common/worker/GroupingTablesTest.java | 2 +-
.../util/common/worker/MapTaskExecutorTest.java | 2 +-
.../util/common/worker/OutputReceiverTest.java | 5 +-
.../util/common/worker/ParDoOperationTest.java | 2 +-
.../util/common/worker/ReadOperationTest.java | 2 +-
.../util/common/worker/ShuffleEntryTest.java | 2 +-
.../util/common/worker/WriteOperationTest.java | 2 +-
.../runners/fnexecution/data/GrpcDataService.java | 6 +-
.../runners/fnexecution/ServerFactoryTest.java | 2 +-
.../control/DefaultJobBundleFactoryTest.java | 62 +-
.../control/FnApiControlClientPoolServiceTest.java | 2 +-
.../control/FnApiControlClientTest.java | 2 +-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 2 +-
.../fnexecution/data/GrpcDataServiceTest.java | 2 +-
.../environment/ProcessEnvironmentFactoryTest.java | 2 +-
.../environment/ProcessManagerTest.java | 8 +-
.../environment/RemoteEnvironmentTest.java | 2 +-
.../logging/GrpcLoggingServiceTest.java | 2 +-
.../StaticGrpcProvisionServiceTest.java | 2 +-
.../fnexecution/state/GrpcStateServiceTest.java | 2 +-
.../translation/PipelineTranslatorUtilsTest.java | 2 +-
.../beam/runners/local/StructuralKeyTest.java | 18 +-
runners/portability/java/build.gradle | 24 -
runners/samza/build.gradle | 17 +-
.../beam/runners/samza/SamzaJobInvocation.java | 114 +++
.../beam/runners/samza/SamzaJobServerDriver.java | 60 +-
.../beam/runners/samza/SamzaPipelineOptions.java | 22 +-
.../samza/SamzaPipelineOptionsValidator.java | 38 +-
.../samza/SamzaPortablePipelineOptions.java | 13 +
.../runners/samza/SamzaRunnerOverrideConfigs.java | 15 +
.../samza/adapter/UnboundedSourceSystem.java | 39 +-
.../samza/container/BeamContainerRunner.java | 6 +-
.../samza/container/BeamJobCoordinatorRunner.java | 78 ++
...inerCfgFactory.java => ContainerCfgLoader.java} | 18 +-
.../ContainerCfgLoaderFactory.java} | 16 +-
.../beam/runners/samza/runtime/BundleManager.java | 349 ++++++++
.../apache/beam/runners/samza/runtime/DoFnOp.java | 277 ++++---
.../runners/samza/runtime/FutureCollector.java | 60 ++
.../beam/runners/samza/runtime/GroupByKeyOp.java | 11 +-
.../beam/runners/samza/runtime/KeyedTimerData.java | 1 +
.../beam/runners/samza/runtime/OpAdapter.java | 48 +-
.../beam/runners/samza/runtime/OpEmitter.java | 5 +
.../samza/runtime/OutputManagerFactory.java | 5 +
.../samza/runtime/SamzaStoreStateInternals.java | 230 ++++--
.../samza/runtime/SamzaTimerInternalsFactory.java | 217 ++++-
.../runners/samza/translation/ConfigBuilder.java | 77 +-
.../runners/samza/translation/ConfigContext.java | 8 +
.../translation/FlattenPCollectionsTranslator.java | 2 +-
.../samza/translation/GroupByKeyTranslator.java | 24 +-
.../translation/ParDoBoundMultiTranslator.java | 56 +-
.../samza/translation/SamzaPipelineTranslator.java | 21 +-
.../translation/SamzaTestStreamSystemFactory.java | 179 +++++
.../translation/SamzaTestStreamTranslator.java | 100 +++
.../translation/SplittableParDoTranslators.java | 6 +-
.../samza/translation/TranslationContext.java | 59 +-
.../samza/translation/WindowAssignTranslator.java | 4 +-
.../beam/runners/samza/util/FutureUtils.java | 50 ++
.../samza/SamzaPipelineOptionsValidatorTest.java | 60 ++
.../samza/adapter/UnboundedSourceSystemTest.java | 28 +
.../runners/samza/runtime/BundleManagerTest.java | 474 +++++++++++
.../samza/runtime/FutureCollectorImplTest.java | 92 +++
.../runners/samza/runtime/KeyedTimerDataTest.java | 14 +-
.../runtime/SamzaStoreStateInternalsTest.java | 38 +-
.../runtime/SamzaTimerInternalsFactoryTest.java | 416 +++++++++-
.../samza/translation/ConfigGeneratorTest.java | 86 +-
.../samza/translation/TranslationContextTest.java | 94 +++
.../beam/runners/samza/util/FutureUtilsTest.java | 107 +++
.../runners/spark/GlobalWatermarkHolderTest.java | 2 +-
.../beam/runners/spark/SparkPipelineStateTest.java | 2 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 2 +-
.../spark/io/ReaderToIteratorAdapterTest.java | 2 +-
.../runners/spark/metrics/SparkBeamMetricTest.java | 2 +-
.../spark/metrics/SparkMetricsPusherTest.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 2 +-
.../metrics/BeamMetricTest.java | 2 +-
.../translation/batch/GroupByKeyTest.java | 2 +-
.../translation/streaming/CreateStreamTest.java | 2 +-
.../ResumeFromCheckpointStreamingTest.java | 2 +-
.../streaming/SparkCoGroupByKeyStreamingTest.java | 2 +-
.../streaming/StreamingSourceMetricsTest.java | 2 +-
.../streaming/TrackStreamingSourcesTest.java | 2 +-
sdks/go/build.gradle | 2 +-
sdks/go/container/build.gradle | 2 +
sdks/go/pkg/beam/core/graph/coder/double_test.go | 53 ++
sdks/go/pkg/beam/core/graph/coder/map_test.go | 11 +-
sdks/go/pkg/beam/core/graph/coder/row.go | 337 ++------
sdks/go/pkg/beam/core/graph/coder/row_decoder.go | 308 +++++++
sdks/go/pkg/beam/core/graph/coder/row_encoder.go | 271 +++++++
sdks/go/pkg/beam/core/graph/coder/row_test.go | 488 +++++++++---
.../pkg/beam/core/graph/coder/testutil/testutil.go | 154 ++++
.../core/graph/coder/testutil/testutil_test.go | 201 +++++
sdks/go/pkg/beam/core/metrics/metrics.go | 39 +
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 9 +-
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 38 +-
sdks/go/pkg/beam/pipeline.go | 1 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 5 +-
.../beam/runners/dataflow/dataflowlib/execute.go | 57 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 8 +
.../beam/runners/dataflow/dataflowlib/metrics.go | 124 +++
.../runners/dataflow/dataflowlib/metrics_test.go | 133 ++++
.../beam/runners/universal/runnerlib/compile.go | 13 +-
.../beam/runners/universal/runnerlib/execute.go | 21 +-
sdks/go/pkg/beam/runners/universal/universal.go | 1 +
sdks/go/pkg/beam/testing/ptest/ptest.go | 21 +
sdks/go/test/build.gradle | 94 ++-
sdks/go/test/integration/{ => driver}/driver.go | 0
sdks/go/test/integration/integration.go | 147 ++++
sdks/go/test/integration/primitives/cogbk_test.go | 16 +-
.../go/test/integration/primitives/flatten_test.go | 11 +-
sdks/go/test/integration/primitives/pardo_test.go | 16 +-
.../test/integration/synthetic/synthetic_test.go | 75 ++
.../test/integration/wordcount/wordcount_test.go | 6 +
sdks/go/test/integration/xlang/xlang_test.go | 260 ++++++
sdks/go/test/load/build.gradle | 1 +
sdks/go/test/load/sideinput/sideinput.go | 100 +++
sdks/go/test/run_integration_tests.sh | 2 +-
...ation_tests.sh => run_validatesrunner_tests.sh} | 220 +++--
.../src/main/resources/beam/checkstyle.xml | 17 +
.../src/main/resources/beam/suppressions.xml | 4 +
sdks/java/container/build.gradle | 4 +-
sdks/java/container/common.gradle | 12 +-
sdks/java/core/build.gradle | 5 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 38 +-
.../apache/beam/sdk/io/AvroSchemaIOProvider.java | 4 +-
.../beam/sdk/io/GenericRecordWriteConverter.java | 67 --
.../src/main/java/org/apache/beam/sdk/io/Read.java | 97 ++-
.../sdk/schemas/FieldValueTypeInformation.java | 53 +-
.../org/apache/beam/sdk/schemas/RowMessages.java | 132 +++
.../beam/sdk/schemas/transforms/Convert.java | 5 +
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 4 +-
.../beam/sdk/schemas/utils/ConvertHelpers.java | 14 +-
.../apache/beam/sdk/testing/CoderProperties.java | 2 +-
.../apache/beam/sdk/testing/CombineFnTester.java | 2 +-
.../java/org/apache/beam/sdk/testing/PAssert.java | 2 +-
.../apache/beam/sdk/testing/SourceTestUtils.java | 2 +-
.../org/apache/beam/sdk/testing/TestPipeline.java | 45 ++
.../apache/beam/sdk/testing/WindowFnTestUtils.java | 2 +-
.../beam/sdk/transforms/ApproximateUnique.java | 20 +-
.../beam/sdk/transforms/windowing/Trigger.java | 3 +-
.../resources/org/apache/beam/sdk/sdk.properties | 5 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 2 +-
.../org/apache/beam/sdk/coders/AvroCoderTest.java | 2 +-
.../beam/sdk/coders/BigDecimalCoderTest.java | 2 +-
.../beam/sdk/coders/BigEndianIntegerCoderTest.java | 2 +-
.../beam/sdk/coders/BigEndianLongCoderTest.java | 2 +-
.../beam/sdk/coders/BigIntegerCoderTest.java | 2 +-
.../apache/beam/sdk/coders/BitSetCoderTest.java | 2 +-
.../apache/beam/sdk/coders/ByteArrayCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/ByteCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/coders/CoderTest.java | 2 +-
.../beam/sdk/coders/CollectionCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DefaultCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DelegateCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DoubleCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DurationCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/FloatCoderTest.java | 2 +-
.../apache/beam/sdk/coders/InstantCoderTest.java | 2 +-
.../apache/beam/sdk/coders/IterableCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/KvCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/ListCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/MapCoderTest.java | 2 +-
.../apache/beam/sdk/coders/NullableCoderTest.java | 2 +-
.../sdk/coders/PCollectionCustomCoderTest.java | 2 +-
.../beam/sdk/coders/SerializableCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/SetCoderTest.java | 2 +-
.../beam/sdk/coders/StringDelegateCoderTest.java | 2 +-
.../beam/sdk/coders/StringUtf8CoderTest.java | 2 +-
.../beam/sdk/coders/StructuredCoderTest.java | 11 +-
.../beam/sdk/coders/TextualIntegerCoderTest.java | 2 +-
.../apache/beam/sdk/coders/VarIntCoderTest.java | 2 +-
.../apache/beam/sdk/coders/VarLongCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/VoidCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 2 +-
.../org/apache/beam/sdk/io/AvroSourceTest.java | 2 +-
.../sdk/io/BoundedReadFromUnboundedSourceTest.java | 2 +-
.../apache/beam/sdk/io/CompressedSourceTest.java | 2 +-
.../org/apache/beam/sdk/io/CountingSourceTest.java | 2 +-
.../org/apache/beam/sdk/io/FileBasedSinkTest.java | 2 +-
.../apache/beam/sdk/io/FileBasedSourceTest.java | 2 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 2 +-
.../beam/sdk/io/LocalFileSystemRegistrarTest.java | 2 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 2 +-
.../test/java/org/apache/beam/sdk/io/ReadTest.java | 130 +++
.../org/apache/beam/sdk/io/TextIOReadTest.java | 2 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 2 +-
.../org/apache/beam/sdk/io/WriteFilesTest.java | 2 +-
.../io/range/ByteKeyRangeEstimateFractionTest.java | 2 +-
.../io/range/ByteKeyRangeInterpolateKeyTest.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRangeTest.java | 2 +-
.../org/apache/beam/sdk/io/range/ByteKeyTest.java | 2 +-
.../org/apache/beam/sdk/metrics/MetricsTest.java | 2 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 2 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 2 +-
.../beam/sdk/runners/PipelineRunnerTest.java | 2 +-
.../beam/sdk/runners/TransformHierarchyTest.java | 2 +-
.../apache/beam/sdk/runners/TransformTreeTest.java | 2 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 4 +-
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 38 +-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 6 +-
.../apache/beam/sdk/schemas/SchemaCoderTest.java | 2 +-
.../sdk/schemas/transforms/CastValidatorTest.java | 2 +-
.../beam/sdk/schemas/transforms/CoGroupTest.java | 2 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 15 +
.../beam/sdk/schemas/transforms/GroupTest.java | 2 +-
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 26 +-
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 6 +-
.../beam/sdk/schemas/utils/SchemaZipFoldTest.java | 2 +-
.../beam/sdk/schemas/utils/TestJavaBeans.java | 17 +
.../apache/beam/sdk/state/StateContextsTest.java | 2 +-
.../beam/sdk/testing/CoderPropertiesTest.java | 2 +-
.../beam/sdk/testing/CombineFnTesterTest.java | 2 +-
.../org/apache/beam/sdk/testing/PAssertTest.java | 2 +-
.../beam/sdk/testing/PaneExtractorsTest.java | 2 +-
.../beam/sdk/testing/SerializableMatchersTest.java | 2 +-
.../apache/beam/sdk/testing/StaticWindowsTest.java | 2 +-
.../apache/beam/sdk/testing/TestPipelineTest.java | 2 +-
.../beam/sdk/testing/WindowSupplierTest.java | 2 +-
.../apache/beam/sdk/transforms/CombineFnsTest.java | 2 +-
.../apache/beam/sdk/transforms/CombineTest.java | 2 +-
.../org/apache/beam/sdk/transforms/CreateTest.java | 2 +-
.../apache/beam/sdk/transforms/DistinctTest.java | 2 +-
.../org/apache/beam/sdk/transforms/DoFnTest.java | 2 +-
.../apache/beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 2 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 2 +-
.../apache/beam/sdk/transforms/PTransformTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ReifyTest.java | 2 +-
.../beam/sdk/transforms/ReifyTimestampsTest.java | 2 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 2 +-
.../beam/sdk/transforms/SimpleFunctionTest.java | 2 +-
.../beam/sdk/transforms/WithFailuresTest.java | 2 +-
.../display/DisplayDataEvaluatorTest.java | 2 +-
.../display/DisplayDataMatchersTest.java | 2 +-
.../sdk/transforms/display/DisplayDataTest.java | 2 +-
.../beam/sdk/transforms/join/CoGbkResultTest.java | 2 +-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 2 +-
.../beam/sdk/transforms/join/UnionCoderTest.java | 2 +-
.../sdk/transforms/reflect/DoFnInvokersTest.java | 2 +-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 2 +-
.../transforms/reflect/OnTimerInvokersTest.java | 2 +-
.../sdk/transforms/windowing/FixedWindowsTest.java | 2 +-
.../transforms/windowing/IntervalWindowTest.java | 2 +-
.../sdk/transforms/windowing/RepeatedlyTest.java | 2 +-
.../sdk/transforms/windowing/SessionsTest.java | 2 +-
.../transforms/windowing/SlidingWindowsTest.java | 2 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 2 +-
.../org/apache/beam/sdk/util/ApiSurfaceTest.java | 2 +-
.../BufferedElementCountingOutputStreamTest.java | 2 +-
.../apache/beam/sdk/util/FluentBackoffTest.java | 2 +-
.../org/apache/beam/sdk/util/HistogramTest.java | 2 +-
.../org/apache/beam/sdk/util/MoreFuturesTest.java | 2 +-
.../org/apache/beam/sdk/util/ReleaseInfoTest.java | 2 +-
.../beam/sdk/util/UserCodeExceptionTest.java | 2 +-
.../java/org/apache/beam/sdk/util/VarIntTest.java | 2 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 2 +-
.../org/apache/beam/sdk/util/ZipFilesTest.java | 2 +-
.../beam/sdk/util/common/ReflectHelpersTest.java | 2 +-
.../java/org/apache/beam/sdk/values/KVTest.java | 2 +-
.../beam/sdk/values/PCollectionListTest.java | 2 +-
.../beam/sdk/values/PCollectionTupleTest.java | 2 +-
.../org/apache/beam/sdk/values/TupleTagTest.java | 2 +-
.../beam/sdk/values/TypeDescriptorsTest.java | 2 +-
.../apache/beam/sdk/values/TypedPValueTest.java | 2 +-
.../beam/sdk/expansion/service/ExternalTest.java | 2 +-
.../sdk/extensions/gcp/util/GceMetadataUtil.java | 4 +-
.../gcp/storage/GcsFileSystemRegistrarTest.java | 2 +-
.../extensions/gcp/storage/GcsFileSystemTest.java | 2 +-
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 2 +-
...LatencyRecordingHttpRequestInitializerTest.java | 2 +-
.../gcp/util/RetryHttpRequestInitializerTest.java | 2 +-
.../sdk/extensions/gcp/util/gcsfs/GcsPathTest.java | 2 +-
.../extensions/protobuf/ProtoMessageSchema.java | 68 +-
.../extensions/protobuf/ByteStringCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtobufUtilTest.java | 2 +-
.../ExternalSchemaIOTransformRegistrarTest.java | 4 +-
.../extensions/sketching/TDigestQuantilesTest.java | 2 +-
.../extensions/sorter/ExternalSorterBenchmark.java | 6 +-
.../beam/sdk/extensions/sorter/SortValuesTest.java | 2 +-
sdks/java/extensions/sql/build.gradle | 15 +-
.../datacatalog/DataCatalogTableProvider.java | 61 +-
.../sql/meta/provider/datacatalog/SchemaUtils.java | 93 +++
.../meta/provider/datacatalog/SchemaUtilsTest.java | 157 ++++
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 2 +-
.../sdk/extensions/sql/jdbc/BeamSqlLineTest.java | 2 +-
.../sdk/extensions/sql/impl/ScalarFnReflector.java | 73 ++
.../extensions/sql/impl/ScalarFunctionImpl.java | 47 +-
.../sql/meta/provider/bigtable/BigtableFilter.java | 136 ++++
.../sql/meta/provider/bigtable/BigtableTable.java | 47 +-
.../meta/provider/kafka/BeamKafkaThriftTable.java | 138 ++++
.../meta/provider/kafka/KafkaTableProvider.java | 56 +-
.../sql/BeamSqlDslSqlStdOperatorsTest.java | 2 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java | 2 +-
.../extensions/sql/impl/ScalarFnReflectorTest.java | 141 ++++
.../provider/bigtable/BigtableClientWrapper.java | 115 +++
.../meta/provider/bigtable/BigtableFilterTest.java | 114 +++
.../BigtableTableCreationFailuresTest.java | 2 +-
.../provider/bigtable/BigtableTableFlatTest.java | 132 +--
.../meta/provider/bigtable/BigtableTableIT.java | 200 +++++
.../meta/provider/bigtable/BigtableTableTest.java | 94 ---
.../provider/bigtable/BigtableTableTestUtils.java | 237 ++++++
.../bigtable/BigtableTableWithRowsTest.java | 120 ++-
.../provider/kafka/BeamKafkaTableProtoTest.java | 2 +-
...rotoTest.java => BeamKafkaTableThriftTest.java} | 53 +-
.../meta/provider/kafka/KafkaTableProviderIT.java | 214 ++++-
.../provider/kafka/KafkaTableProviderProtoIT.java | 53 --
.../provider/kafka/KafkaTableProviderTest.java | 55 +-
.../provider/kafka/thrift/ItThriftMessage.java | 612 ++++++++++++++
.../provider/kafka/thrift/SimpleThriftMessage.java | 509 ++++++++++++
.../provider/kafka/thrift/TestThriftMessage.java | 881 +++++++++++++++++++++
.../sql/meta/provider/pubsub/PubsubAvroIT.java | 102 ---
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 76 --
.../provider/pubsub/PubsubTableProviderIT.java | 273 +++++--
.../meta/provider/text/TextTableProviderTest.java | 2 +-
.../sql/meta/store/InMemoryMetaStoreTest.java | 2 +-
.../sql/src/test/thrift/kafka/messages.thrift | 36 +-
sdks/{go => java/extensions/sql/udf}/build.gradle | 27 +-
.../beam/sdk/extensions/sql/udf/AggregateFn.java | 71 ++
.../beam/sdk/extensions/sql/udf/ScalarFn.java | 53 ++
.../beam/sdk/extensions/sql/udf/UdfProvider.java} | 23 +-
.../beam/sdk/extensions/sql/udf/package-info.java | 16 +-
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java | 6 +
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 5 +-
.../sql/zetasql/translation/ConversionContext.java | 2 +-
.../zetasql/translation/ExpressionConverter.java | 11 +-
.../sql/zetasql/translation/SqlOperators.java | 76 +-
.../translation/UserFunctionDefinitions.java | 30 +-
.../translation/ZetaSqlScalarFunctionImpl.java | 86 ++
.../zetasketch/ApproximateCountDistinct.java | 288 +++++++
.../zetasketch/ApproximateCountDistinctTest.java | 342 ++++++++
.../sdk/fn/data/BeamFnDataInboundObserver.java | 35 +-
.../beam/sdk/fn/data/DecodingFnDataReceiver.java | 46 ++
.../org/apache/beam/sdk/fn/IdGeneratorsTest.java | 2 +-
.../sdk/fn/channel/SocketAddressFactoryTest.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 2 +-
...DataSizeBasedBufferingOutboundObserverTest.java | 2 +-
.../CompletableFutureInboundDataClientTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 2 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 2 +-
.../beam/sdk/fn/stream/AdvancingPhaserTest.java | 2 +-
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 2 +-
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 2 +-
.../apache/beam/sdk/fn/test/TestStreamsTest.java | 2 +-
.../fn/harness/control/ProcessBundleHandler.java | 87 +-
.../beam/fn/harness/data/BeamFnDataClient.java | 14 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 10 +-
.../fn/harness/data/QueueingBeamFnDataClient.java | 11 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 15 +
.../beam/fn/harness/AssignWindowsRunnerTest.java | 2 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 2 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 2 +-
.../apache/beam/fn/harness/CombineRunnersTest.java | 2 +-
.../apache/beam/fn/harness/FlattenRunnerTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 2 +-
.../apache/beam/fn/harness/MapFnRunnersTest.java | 2 +-
.../fn/harness/PrecombineGroupingTableTest.java | 2 +-
.../beam/fn/harness/WindowMergingFnRunnerTest.java | 2 +-
.../harness/control/BeamFnControlClientTest.java | 2 +-
.../harness/control/BundleSplitListenerTest.java | 2 +-
.../harness/control/ProcessBundleHandlerTest.java | 2 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 2 +-
.../data/BeamFnDataInboundObserverTest.java | 13 +-
.../data/PCollectionConsumerRegistryTest.java | 2 +-
.../harness/data/QueueingBeamFnDataClientTest.java | 2 +-
.../harness/logging/BeamFnLoggingClientTest.java | 2 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 2 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 2 +-
.../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 40 +-
.../beam/sdk/io/aws/coders/AwsCodersTest.java | 2 +-
.../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 68 +-
.../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java | 2 +-
.../beam/sdk/io/aws/options/AwsModuleTest.java | 2 +-
.../beam/sdk/io/aws/s3/S3FileSystemTest.java | 2 +-
.../sdk/io/aws/sns/PublishResultCodersTest.java | 2 +-
.../aws2/dynamodb/BasicDynamoDbClientProvider.java | 32 +-
.../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 40 +-
.../sdk/io/aws2/options/AwsSerializableUtils.java | 50 ++
.../io/aws2/sns/BasicSnsAsyncClientProvider.java | 32 +-
.../sdk/io/aws2/sns/BasicSnsClientProvider.java | 34 +-
.../sdk/io/aws2/sqs/BasicSqsClientProvider.java | 34 +-
.../dynamodb/BasicDynamoDbClientProviderTest.java | 51 ++
.../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 85 +-
.../beam/sdk/io/aws2/options/AwsModuleTest.java | 2 +-
.../io/aws2/options/AwsSerializableUtilsTest.java | 70 ++
.../aws2/sns/BasicSnsAsyncClientProviderTest.java | 51 ++
.../io/aws2/sns/BasicSnsClientProviderTest.java | 51 ++
.../io/aws2/sqs/BasicSqsClientProviderTest.java | 51 ++
.../blobstore/AzureBlobStoreFileSystemTest.java | 2 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 2 +-
.../io/contextualtextio/ContextualTextIOTest.java | 2 +-
.../elasticsearch/ElasticsearchIOTestCommon.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 7 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 6 +
.../apache/beam/sdk/io/gcp/bigtable/RowUtils.java | 10 +
.../apache/beam/sdk/io/gcp/healthcare/FhirIO.java | 232 ++++++
.../sdk/io/gcp/healthcare/HealthcareApiClient.java | 17 +
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 174 ++++
.../sdk/io/gcp/healthcare/JsonArrayCoder.java} | 34 +-
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 28 +-
.../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 17 +-
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 150 ++--
.../sdk/io/gcp/pubsublite/CloudPubsubChecks.java | 21 +-
...fsetFinalizer.java => InitialOffsetReader.java} | 12 +-
.../io/gcp/pubsublite/InitialOffsetReaderImpl.java | 54 ++
.../gcp/pubsublite/LimitingTopicBacklogReader.java | 81 ++
...ffsetFinalizer.java => OffsetByteProgress.java} | 17 +-
.../io/gcp/pubsublite/OffsetByteRangeTracker.java | 172 ++++
.../io/gcp/pubsublite/OffsetCheckpointMark.java | 78 --
.../pubsublite/PerSubscriptionPartitionSdf.java | 110 +++
.../beam/sdk/io/gcp/pubsublite/PublisherCache.java | 10 +-
.../sdk/io/gcp/pubsublite/PublisherOptions.java | 47 +-
.../sdk/io/gcp/pubsublite/PublisherOrError.java | 6 +-
.../beam/sdk/io/gcp/pubsublite/Publishers.java | 56 ++
.../beam/sdk/io/gcp/pubsublite/PubsubLiteIO.java | 14 +-
.../beam/sdk/io/gcp/pubsublite/PubsubLiteSink.java | 18 +-
.../gcp/pubsublite/PubsubLiteUnboundedReader.java | 333 --------
.../gcp/pubsublite/PubsubLiteUnboundedSource.java | 140 ----
...zer.java => SerializableSubscriberFactory.java} | 17 +-
.../io/gcp/pubsublite/SerializableSupplier.java | 5 +-
.../sdk/io/gcp/pubsublite/SubscribeTransform.java | 124 +++
.../sdk/io/gcp/pubsublite/SubscriberOptions.java | 157 ++--
...etFinalizer.java => SubscriptionPartition.java} | 19 +-
...dCoder.java => SubscriptionPartitionCoder.java} | 36 +-
...er.java => SubscriptionPartitionProcessor.java} | 16 +-
... => SubscriptionPartitionProcessorFactory.java} | 15 +-
.../SubscriptionPartitionProcessorImpl.java | 152 ++++
.../sdk/io/gcp/pubsublite/TopicBacklogReader.java | 34 +-
.../io/gcp/pubsublite/TopicBacklogReaderImpl.java | 80 +-
.../gcp/pubsublite/TopicBacklogReaderSettings.java | 50 +-
.../gcp/pubsublite/TranslatingPullSubscriber.java | 48 --
.../beam/sdk/io/gcp/pubsublite/UuidCoder.java | 7 +-
.../gcp/pubsublite/UuidDeduplicationOptions.java | 15 +-
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 22 +-
.../io/gcp/testing/BigtableEmulatorWrapper.java | 75 --
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 154 ----
.../beam/sdk/io/gcp/testing/BigtableUtils.java | 52 +-
.../cloud/spanner/FakeBatchTransactionId.java | 1 +
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 3 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 2 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 13 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 2 +-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 2 +-
.../sdk/io/gcp/bigquery/TableRowJsonCoderTest.java | 2 +-
.../bigtable/BeamRowToBigtableMutationTest.java | 14 +-
.../sdk/io/gcp/bigtable/BigtableConfigTest.java | 2 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 2 +-
.../gcp/bigtable/BigtableRowToBeamRowFlatTest.java | 14 +-
.../io/gcp/bigtable/BigtableRowToBeamRowTest.java | 17 +-
.../{TestUtils.java => BigtableTestUtils.java} | 50 +-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 2 +-
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 13 +-
.../io/gcp/datastore/AdaptiveThrottlerTest.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 2 +-
.../gcp/datastore/EntityToRowRowToEntityTest.java | 6 +-
.../beam/sdk/io/gcp/healthcare/FhirIOSearchIT.java | 156 ++++
.../beam/sdk/io/gcp/healthcare/FhirIOTest.java | 21 +
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 2 +-
.../pubsub/PubsubMessagePayloadOnlyCoderTest.java | 2 +-
...MessageWithAttributesAndMessageIdCoderTest.java | 2 +-
.../PubsubMessageWithAttributesCoderTest.java | 2 +-
.../PubsubMessageWithMessageIdCoderTest.java | 2 +-
.../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 2 +-
.../gcp/pubsublite/OffsetByteRangeTrackerTest.java | 158 ++++
.../gcp/pubsublite/OffsetCheckpointMarkTest.java | 81 --
.../PerSubscriptionPartitionSdfTest.java | 189 +++++
.../sdk/io/gcp/pubsublite/PubsubLiteSinkTest.java | 47 +-
.../pubsublite/PubsubLiteUnboundedReaderTest.java | 330 --------
.../SubscriptionPartitionProcessorImplTest.java | 227 ++++++
.../gcp/pubsublite/TopicBacklogReaderImplTest.java | 140 +---
.../pubsublite/UuidDeduplicationTransformTest.java | 13 +-
.../io/gcp/spanner/MutationSizeEstimatorTest.java | 2 +-
.../sdk/io/gcp/spanner/ReadSpannerSchemaTest.java | 2 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 2 +-
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 2 +-
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 2 +-
.../sdk/io/hadoop/SerializableConfiguration.java | 11 +
.../io/hadoop/SerializableConfigurationTest.java | 9 +
.../beam/sdk/io/hadoop/WritableCoderTest.java | 2 +-
.../beam/sdk/io/hdfs/HadoopFileSystemOptions.java | 9 +-
.../sdk/io/hdfs/HadoopFileSystemModuleTest.java | 2 +-
.../hdfs/HadoopFileSystemOptionsRegistrarTest.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemOptionsTest.java | 22 +-
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 2 +-
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 85 +-
.../hadoop/format/HadoopFormatIOCassandraTest.java | 6 +-
.../io/hadoop/format/HadoopFormatIOReadTest.java | 162 +++-
.../org/apache/beam/sdk/io/hbase/HBaseIOTest.java | 2 +-
.../beam/sdk/io/hcatalog/HCatalogIOTest.java | 2 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 2 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 2 +-
.../CustomTimestampPolicyWithLimitedDelayTest.java | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 8 +-
.../apache/beam/sdk/io/kinesis/KinesisIOIT.java | 2 +-
.../java/org/apache/beam/sdk/io/kudu/KuduIOIT.java | 4 +-
.../sdk/io/parquet/GenericRecordReadConverter.java | 64 --
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 472 +++++++++--
.../sdk/io/parquet/ParquetSchemaIOProvider.java | 11 +-
.../sdk/io/parquet/GenericRecordToRowTest.java | 79 --
.../apache/beam/sdk/io/parquet/ParquetIOTest.java | 132 ++-
.../apache/beam/sdk/io/snowflake/KeyPairUtils.java | 4 +-
.../io/snowflake/crosslanguage/ReadBuilder.java | 4 +-
.../services/SnowflakeBatchServiceImpl.java | 4 +-
.../test/FakeSnowflakeStreamingServiceImpl.java | 4 +-
.../beam/sdk/io/snowflake/test/TestUtils.java | 6 +-
.../beam/sdk/io/solr/JavaBinCodecCoderTest.java | 2 +-
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 4 +-
sdks/java/io/thrift/build.gradle | 10 +-
.../org/apache/beam/sdk/io/thrift/ThriftCoder.java | 9 +-
.../apache/beam/sdk/io/thrift/ThriftSchema.java | 408 ++++++++++
.../apache/beam/sdk/io/thrift/TestThriftEnum.java} | 45 +-
.../beam/sdk/io/thrift/TestThriftInnerStruct.java | 526 ++++++++++++
.../beam/sdk/io/thrift/TestThriftStruct.java | 760 ++++++++++++++++--
.../apache/beam/sdk/io/thrift/TestThriftUnion.java | 401 ++++++++++
.../apache/beam/sdk/io/thrift/ThriftIOTest.java | 38 +-
.../beam/sdk/io/thrift/ThriftSchemaTest.java | 251 ++++++
.../src/test/resources/thrift/thrift_test.thrift | 35 +-
.../apache/beam/sdk/io/tika/ParseResultTest.java | 2 +-
.../org/apache/beam/sdk/io/tika/TikaIOTest.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlSource.java | 22 +-
.../org/apache/beam/sdk/io/xml/JAXBCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOTest.java | 11 +-
.../org/apache/beam/sdk/io/xml/XmlSourceTest.java | 44 +-
sdks/java/maven-archetypes/examples/build.gradle | 2 +-
sdks/java/testing/jpms-tests/build.gradle | 2 +-
.../java/org/apache/beam/sdk/nexmark/Monitor.java | 13 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 50 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 14 +
.../beam/sdk/nexmark/queries/NexmarkQuery.java | 3 -
.../sdk/nexmark/queries/NexmarkQueryModel.java | 4 +-
.../apache/beam/sdk/nexmark/queries/Query10.java | 8 +-
.../apache/beam/sdk/nexmark/queries/Query3.java | 191 +++--
.../apache/beam/sdk/nexmark/queries/Query5.java | 133 +++-
.../beam/sdk/nexmark/NexmarkConfigurationTest.java | 2 +-
.../beam/sdk/nexmark/PerfsToBigQueryTest.java | 2 +-
.../nexmark/queries/BoundedSideInputJoinTest.java | 2 +-
.../nexmark/queries/SessionSideInputJoinTest.java | 2 +-
.../queries/sql/SqlBoundedSideInputJoinTest.java | 2 +-
sdks/python/apache_beam/coders/coders.py | 71 +-
.../apache_beam/coders/coders_test_common.py | 1 -
sdks/python/apache_beam/coders/typecoders.py | 3 -
sdks/python/apache_beam/dataframe/frames.py | 26 +-
sdks/python/apache_beam/dataframe/frames_test.py | 78 +-
sdks/python/apache_beam/dataframe/io.py | 263 ++++--
sdks/python/apache_beam/dataframe/io_test.py | 88 +-
.../examples/cookbook/bigquery_side_input.py | 7 +-
.../apache_beam/examples/cookbook/filters.py | 2 +-
.../apache_beam/examples/snippets/snippets.py | 16 +-
.../apache_beam/examples/snippets/snippets_test.py | 10 +-
sdks/python/apache_beam/examples/sql_taxi.py | 3 +-
sdks/python/apache_beam/io/concat_source.py | 9 +-
sdks/python/apache_beam/io/concat_source_test.py | 8 +
.../flink/flink_streaming_impulse_source_test.py | 3 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 2 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 21 +-
.../apache_beam/io/gcp/bigquery_read_internal.py | 40 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 14 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 8 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 3 +-
sdks/python/apache_beam/io/mongodbio.py | 29 +-
sdks/python/apache_beam/io/mongodbio_test.py | 41 +
sdks/python/apache_beam/io/restriction_trackers.py | 2 +-
sdks/python/apache_beam/metrics/execution.pxd | 1 +
sdks/python/apache_beam/metrics/execution.py | 9 +-
.../python/apache_beam/options/pipeline_options.py | 29 +-
sdks/python/apache_beam/pipeline.py | 163 ++--
sdks/python/apache_beam/portability/common_urns.py | 1 +
sdks/python/apache_beam/runners/common.pxd | 1 +
sdks/python/apache_beam/runners/common.py | 36 +-
.../runners/dataflow/dataflow_runner.py | 184 ++---
.../runners/dataflow/dataflow_runner_test.py | 62 ++
.../apache_beam/runners/dataflow/internal/names.py | 9 +-
.../runners/dataflow/ptransform_overrides.py | 45 ++
.../apache_beam/runners/direct/direct_runner.py | 4 +-
.../apache_beam/runners/direct/direct_userstate.py | 35 +-
.../runners/direct/transform_evaluator.py | 8 +-
sdks/python/apache_beam/runners/direct/util.py | 18 +-
.../runners/direct/watermark_manager.py | 10 +-
.../python/apache_beam/runners/pipeline_context.py | 2 -
.../runners/portability/flink_runner_test.py | 3 +
.../portability/fn_api_runner/fn_runner_test.py | 24 +
.../portability/fn_api_runner/translations.py | 81 +-
.../portability/fn_api_runner/translations_test.py | 18 +-
.../portability/fn_api_runner/worker_handlers.py | 7 +-
.../apache_beam/runners/portability/job_server.py | 10 +-
.../runners/portability/job_server_test.py | 84 ++
.../runners/portability/portable_runner.py | 85 +-
.../runners/portability/sdk_container_builder.py | 11 +-
.../runners/portability/spark_runner_test.py | 5 +-
sdks/python/apache_beam/runners/runner.py | 24 +-
.../apache_beam/runners/worker/bundle_processor.py | 41 +-
.../apache_beam/runners/worker/operations.py | 6 +-
sdks/python/apache_beam/transforms/core.py | 145 +---
sdks/python/apache_beam/transforms/environments.py | 26 +-
sdks/python/apache_beam/transforms/external.py | 24 +-
sdks/python/apache_beam/transforms/ptransform.py | 46 +-
sdks/python/apache_beam/transforms/trigger.py | 39 +-
sdks/python/apache_beam/transforms/trigger_test.py | 10 +-
sdks/python/apache_beam/transforms/userstate.py | 55 +-
.../apache_beam/transforms/userstate_test.py | 216 ++++-
sdks/python/apache_beam/transforms/util.py | 123 ++-
sdks/python/apache_beam/transforms/util_test.py | 65 ++
.../typehints/native_type_compatibility.py | 7 -
.../apache_beam/typehints/sharded_key_type.py | 31 +-
.../apache_beam/typehints/sharded_key_type_test.py | 35 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/container/common.gradle | 2 +
sdks/python/setup.py | 6 +-
sdks/python/test-suites/dataflow/build.gradle | 24 +
sdks/python/test-suites/gradle.properties | 5 +
sdks/python/test-suites/portable/common.gradle | 34 +-
settings.gradle | 5 +
start-build-env.sh | 16 +-
website/.gitignore | 23 -
website/www/package.json | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.13.0.md | 2 +-
website/www/site/content/en/blog/beam-2.21.0.md | 2 +-
website/www/site/content/en/blog/beam-2.22.0.md | 2 +-
website/www/site/content/en/blog/beam-2.23.0.md | 4 +-
website/www/site/content/en/blog/beam-2.24.0.md | 4 +-
website/www/site/content/en/blog/beam-2.25.0.md | 4 +-
website/www/site/content/en/blog/beam-2.26.0.md | 65 ++
website/www/site/content/en/blog/beam-2.27.0.md | 69 ++
.../en/blog/dataframe-api-preview-available.md | 178 +++++
.../site/content/en/blog/pattern-match-beam-sql.md | 2 +-
.../en/blog/splittable-do-fn-is-available.md | 91 +++
website/www/site/content/en/contribute/_index.md | 2 +-
.../site/content/en/contribute/release-guide.md | 7 +-
.../en/documentation/dsls/dataframes/overview.md | 131 +++
.../dsls/sql/extensions/create-external-table.md | 15 +-
.../content/en/documentation/dsls/sql/overview.md | 2 +-
.../documentation/io/built-in/google-bigquery.md | 15 +-
.../en/documentation/io/built-in/snowflake.md | 6 +-
.../content/en/documentation/programming-guide.md | 4 +-
.../content/en/documentation/runners/direct.md | 112 +--
.../site/content/en/documentation/runners/flink.md | 42 +-
.../site/content/en/documentation/runners/samza.md | 5 +
.../en/documentation/runtime/environments.md | 305 ++++---
.../site/content/en/get-started/beam-overview.md | 5 +-
.../www/site/content/en/get-started/downloads.md | 28 +-
.../www/site/content/en/get-started/from-spark.md | 268 +++++++
website/www/site/data/authors.yml | 38 +-
website/www/site/data/capability_matrix.yaml | 714 +++++++++++++++--
.../www/site/data/capability_matrix_snapshot.yaml | 195 +++++
website/www/site/layouts/blog/single.html | 1 +
.../partials/section-menu/en/get-started.html | 7 +-
.../layouts/partials/section-menu/en/sdks.html | 17 +-
website/www/site/static/.htaccess | 2 +-
941 files changed, 26264 insertions(+), 6811 deletions(-)