You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ie...@apache.org on 2019/08/01 15:33:06 UTC
[beam] branch spark-runner_structured-streaming updated (2c5d37a ->
174595b)
This is an automated email from the ASF dual-hosted git repository.
iemejia pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 2c5d37a Fix hardcoded value
discard b63892e Use "sparkMaster" in local mode to obtain number of shuffle partitions + spotless apply
discard 228b924 Improve Pardo translation performance: avoid calling a filter transform when there is only one output tag
discard 446e221 Add a TODO on perf improvement of Pardo translation
discard c088517 After testing performance and correctness, launch pipeline with dataset.foreach(). Make both test mode and production mode use foreach for uniformity. Move dataset print as a utility method
discard 3d20031 Remove no more needed AggregatorCombinerPerKey (there is only AggregatorCombiner)
discard 3e6a9e2 fixup! Add PipelineResults to Spark structured streaming.
discard 9a1e4dc Print number of leaf datasets
discard b62a593 Add spark execution plans extended debug messages.
discard 935e703 Update log4j configuration
discard 797b40d Add PipelineResults to Spark structured streaming.
omit b9c8925 Make spotless happy
omit 50b3a97 Added metrics sinks and tests
omit df41a8b Persist all output Dataset if there are multiple outputs in pipeline Enabled Use*Metrics tests
omit a37817d Add a test to check that CombineGlobally preserves windowing
omit 170568b Fix accumulators initialization in Combine that prevented CombineGlobally to work.
omit 0fc97ee Fix javadoc
omit a3d1061 Add setEnableSparkMetricSinks() method
omit f88db1b Add missing dependencies to run Spark Structured Streaming Runner on Nexmark
omit 12eb126 Add metrics support in DoFn
omit 703e773 Ignore for now not working test testCombineGlobally
omit 5bb44d6 Add a test that combine per key preserves windowing
omit bea2c60 Clean groupByKeyTest
omit 777be24 add comment in combine globally test
omit 78270bf Fixed immutable list bug
omit 721b1d0 Fix javadoc of AggregatorCombiner
omit ea959c6 Clean not more needed WindowingHelpers
omit e84c0bb Clean not more needed RowHelpers
omit 2851c72 Clean no more needed KVHelpers
omit f05e17c Now that there is only Combine.PerKey translation, make only one Aggregator
omit 18a2e2a Remove CombineGlobally translation because it is less performant than the beam sdk one (key + combinePerKey.withHotkeyFanout)
omit 6450bda Remove the mapPartition that adds a key per partition because otherwise spark will reduce values per key instead of globally
omit 11981e5 Fix bug in the window merging logic
omit ddce346 Fix wrong encoder in combineGlobally GBK
omit 3ae7e09 Fix case when a window does not merge into any other window
omit e77e951 Apply a groupByKey avoids for some reason that the spark structured streaming fmwk casts data to Row which makes it impossible to deserialize without the coder shipped into the data. For performance reasons (avoid memory consumption and having to deserialize), we do not ship coder + data. Also add a mapparitions before GBK to avoid shuffling
omit b1cf536 Revert extractKey while combinePerKey is not done (so that it compiles)
omit 28067b2 Fix encoder in combine call
omit a4af031 Implement merge accumulators part of CombineGlobally translation with windowing
omit 3e49bf0 Output data after combine
omit 233e532 Implement reduce part of CombineGlobally translation with windowing
omit a9e7bf4 Fix comment about schemas
omit 05cb7b8 Update KVHelpers.extractKey() to deal with WindowedValue and update GBK and CPK
omit 1a85496 Add TODO in Combine translations
omit 92bf293 Add a test that GBK preserves windowing
omit 41de6da Build fixup: enable UsesBoundedSplittableParDo category of tests Since the default overwrite is already in place and it is breaking only in one subcase
omit ee91d24 Improve visibility of debug messages
omit 6c9a3a8 re-enable reduceFnRunner timers for output
omit 6bc9b17 Re-code GroupByKeyTranslatorBatch to conserve windowing instead of unwindowing/windowing(GlobalWindow): simplify code, use ReduceFnRunner to merge the windows
omit ede9118 Add comment about checkpoint mark
omit ab26efc Update windowAssignTest
omit 9d29d76 Put back batch/simpleSourceTest.testBoundedSource
omit 9c3b009 Consider null object case on RowHelpers, fixes empty side inputs tests.
omit ebc6ca9 Build fixup: hadoop-format is not mandataory to run ValidatesRunner tests
omit cb1158b Pass transform based doFnSchemaInformation in ParDo translation
omit ff50759 Build fixup: enable UsesFailureMessage and UsesSchema categories on validatesRunner tests
omit 1f4f0bc Fixes ParDo not calling setup and not tearing down if exception on startBundle
omit 1d52f35 Build fixup: Limit the number of partitions to make tests go 300% faster
omit 82e28d1 Build fixup: enable batch Validates Runner tests for Structured Streaming Runner
omit 72f18c7 Apply Spotless
omit a75e88a Update javadoc
omit 82ed0fc implement source.stop
omit e04d455 Ignore spark offsets (cf javadoc)
omit 4b568da Use PAssert in Spark Structured Streaming transform tests
omit 9d6caa1 Rename SparkPipelineResult to SparkStructuredStreamingPipelineResult This is done to avoid an eventual collision with the one in SparkRunner. However this cannot happen at this moment because it is package private, so it is also done for consistency.
omit 4f0e0e6 Add SparkStructuredStreamingPipelineOptions and SparkCommonPipelineOptions - SparkStructuredStreamingPipelineOptions was added to have the new runner rely only on its specific options.
omit 45781a8 Fix logging levels in Spark Structured Streaming translation
omit 1d48ae5 Fix spotless issues after rebase
omit 8602eaa Pass doFnSchemaInformation to ParDo batch translation
omit f36c3b8 Fix non-vendored imports from Spark Streaming Runner classes
omit 07b01b1 Merge Spark Structured Streaming runner into main Spark module. Remove spark-structured-streaming module.Rename Runner to SparkStructuredStreamingRunner. Remove specific PipelineOotions and Registrar for Structured Streaming Runner
omit 51630e2 Fix access level issues, typos and modernize code to Java 8 style
omit df02413 Disable never ending test SimpleSourceTest.testUnboundedSource
omit 85635ff Apply spotless and fix spotbugs warnings
omit 1bc4d75 Deal with checkpoint and offset based read
omit 559e80f Continue impl of offsets for streaming source
omit b9af1b5 Clean streaming source
omit 4ef8635 Clean unneeded 0 arg constructor in batch source
omit e0ecd02 Specify checkpointLocation at the pipeline start
omit a14698f Add source streaming test
omit c46ce22 Add transformators registry in PipelineTranslatorStreaming
omit 10fac36 Add a TODO on spark output modes
omit 9803c07 Implement first streaming source
omit cb261f2 Add streaming source initialisation
omit 8bd0f66 And unchecked warning suppression
omit 9ff127d Added TODO comment for ReshuffleTranslatorBatch
omit 519c9b2 Added using CachedSideInputReader
omit d629b1c Don't use Reshuffle translation
omit d705c22 Fix CheckStyle violations
omit 2128544 Added SideInput support
omit 8e029bd Fix build wordcount:disable deps versions forcing, remove fail on warnings
omit f7f3b54 Fix javadoc
omit 07bceca Implement WindowAssignTest
omit dafe5c4 Implement WindowAssignTranslatorBatch
omit e712187 Cleaning
omit 976783c [TO UPGRADE WITH THE 2 SPARK RUNNERS BEFORE MERGE] Change de wordcount build to test on new spark runner
omit 2a8d92e Fix encoder bug in combinePerkey
omit 0bd6370 Add explanation about receiving a Row as input in the combiner
omit 5d86507 Use more generic Row instead of GenericRowWithSchema
omit c453b90 Fix combine. For unknown reason GenericRowWithSchema is used as input of combine so extract its content to be able to proceed
omit ac7f271 Update test with Long
omit 3ede119 Fix various type checking issues in Combine.Globally
omit 571b3e9 Get back to classes in translators resolution because URNs cannot translate Combine.Globally
omit b390509 Cleaning
omit 0c48a94 Add CombineGlobally translation to avoid translating Combine.perKey as a composite transform based on Combine.PerKey (which uses low perf GBK)
omit 7629b22 Introduce RowHelpers
omit 11f6322 Add combinePerKey and CombineGlobally tests
omit 9b3c604 Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless
omit 6da1c02 Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey
omit 9407138 Improve type checking of Tuple2 encoder
omit d1bafd1 First version of combinePerKey
omit c7256da Extract binary schema creation in a helper class
omit 05a1a11 Fix getSideInputs
omit 28d0be3 Generalize the use of SerializablePipelineOptions in place of (not serializable) PipelineOptions
omit a33477f Rename SparkDoFnFilterFunction to DoFnFilterFunction for consistency
omit 3bbb0d1 Add a test for the most simple possible Combine
omit d7104ee Added "testTwoPardoInRow"
omit 2b32d52 Fix for test elements container in GroupByKeyTest
omit 8ede26e Rename SparkOutputManager for consistency
omit 106e54e Fix kryo issue in GBK translator with a workaround
omit 72bf4d3 Simplify logic of ParDo translator
omit c8cefc8 Don't use deprecated sideInput.getWindowingStrategyInternal()
omit 1f6c3b7 Rename SparkSideInputReader class and rename pruneOutput() to pruneOutputFilteredByTag()
omit 8407488 Fixed Javadoc error
omit 6efe901 Apply spotless
omit fe00df0 Fix Encoders: create an Encoder for every manipulated type to avoid Spark fallback to genericRowWithSchema and cast to allow having Encoders with generic types such as WindowedValue<T> and get the type checking back
omit 24a0f7d Fail in case of having SideInouts or State/Timers
omit d3c6e42 Add ComplexSourceTest
omit e4948df Remove no more needed putDatasetRaw
omit 4f55612 Port latest changes of ReadSourceTranslatorBatch to ReadSourceTranslatorStreaming
omit 55a6419 Fix type checking with Encoder of WindowedValue<T>
omit 6eb2ff1 Add comments and TODO to GroupByKeyTranslatorBatch
omit df89095 Add GroupByKeyTest
omit 8e999c9 Clean
omit cd58952 Address minor review notes
omit 32bf4da Add ParDoTest
omit afc53ee Clean
omit bab98f1 Fix split bug
omit 6b87e80 Remove bundleSize parameter and always use spark default parallelism
omit 53ddaa0 Cleaning
omit 152d858 Fix testMode output to comply with new binary schema
omit 355fa9e Fix errorprone
omit 80aa7ea Comment schema choices
omit d9d424f Serialize windowedValue to byte[] in source to be able to specify a binary dataset schema and deserialize windowedValue from Row to get a dataset<WindowedValue>
omit 000381c First attempt for ParDo primitive implementation
omit 5067ca1 Add flatten test
omit 21c426e Enable gradle build scan
omit 7c4ff5c Enable test mode
omit 33754a8 Put all transform translators Serializable
omit b39024a Simplify beam reader creation as it created once the source as already been partitioned
omit b6a6912 Fix SourceTest
omit 155a65b Move SourceTest to same package as tested class
omit 4d18207 Add serialization test
omit 8ba03d6f Add SerializationDebugger
omit ec13b5b Fix serialization issues
omit 0cfde2b Clean unneeded fields in DatasetReader
omit 3b868db improve readability of options passing to the source
omit 38b1514 Fix pipeline triggering: use a spark action instead of writing the dataset
omit 6173a33 Deactivate deps resolution forcing that prevent using correct spark transitive dep
omit 4f6d5b9 Refactor SourceTest to a UTest instaed of a main
omit ab70e9e Checkstyle and Findbugs
omit b9acd02 Clean
omit 2bc0d23 Add empty 0-arg constructor for mock source
omit 0bdbf93 Add a dummy schema for reader
omit 73d8c71 Apply spotless and fix checkstyle
omit c38d1d2 Use new PipelineOptionsSerializationUtils
omit af3dfad Add missing 0-arg public constructor
omit 84f1963 Wire real SourceTransform and not mock and update the test
omit 0c64b46 Refactor DatasetSource fields
omit a96039e Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
omit 8bfa1a5 Move Source and translator mocks to a mock package.
omit 5559890 Add ReadSourceTranslatorStreaming
omit 2dca3c2 Clean
omit 98d6e1e Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
omit 772efc4 Split batch and streaming sources and translators
omit 0b9875f Run pipeline in batch mode or in streaming mode
omit a0243bf Move DatasetSourceMock to proper batch mode
omit 80b91bd clean deps
omit b5cc52a Use raw WindowedValue so that spark Encoders could work (temporary)
omit f99d1d1 fix mock, wire mock in translators and create a main test.
omit 8a0343a Add source mocks
omit bc6b4e6 Experiment over using spark Catalog to pass in Beam Source through spark Table
omit 6cfdf31 Improve type enforcement in ReadSourceTranslator
omit aec130e Improve exception flow
omit 9b3a72e start source instanciation
omit 4b55ed6 Apply spotless
omit 039976a update TODO
omit dfcfa6a Implement read transform
omit 6a9021d Use Iterators.transform() to return Iterable
omit 3491edf Add primitive GroupByKeyTranslatorBatch implementation
omit b045f0d Add Flatten transformation translator
omit 6ff5192 Create Datasets manipulation methods
omit 7d7dcc6 Create PCollections manipulation methods
omit e3ac83a Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
omit 52da1ea Added SparkRunnerRegistrar
omit d23f4fd Add precise TODO for multiple TransformTranslator per transform URN
omit 60bc61b Post-pone batch qualifier in all classes names for readability
omit 419211a Add TODOs
omit 3504543 Make codestyle and firebug happy
omit af2e92b apply spotless
omit e4a647e Move common translation context components to superclass
omit cc80092 Move SparkTransformOverrides to correct package
omit 1cd8dab Improve javadocs
omit b8e9f7a Make transform translation clearer: renaming, comments
omit 0621611 Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
omit 2fa095a Initialise BatchTranslationContext
omit 77286bd Organise methods in PipelineTranslator
omit 72e7fb9 Renames: better differenciate pipeline translator for transform translator
omit 73d37f2 Wire node translators with pipeline translator
omit 4ecaa74 Add nodes translators structure
omit e837be2 Add global pipeline translation structure
omit 721fa2d Start pipeline translation
omit 4e8f89e Add SparkPipelineOptions
omit 56ad12e Fix missing dep
omit 474c0dc Add an empty spark-structured-streaming runner project targeting spark 2.4.0
add 4c328fe [BEAM-7005] Add streaming GBK load test
add 39661a5 Merge pull request #8355: [BEAM-7005] Add streaming GBK load test
add 6182a40 [BEAM-7228] Add a Jenkins job running ValidatesRunner tests for Direct runner
add 260677a Merge pull request #8528: [BEAM-7228] Add a Jenkins job running ValidatesRunner tests for Direct runner
add eb21fb8 [BEAM-7012] Support TestStream in portable Flink Runner
add e55ab73 Merge pull request #8411: [BEAM-7012] Support TestStream in portable Flink Runner
add e21cd54 Fix py3 "Map modified during iteration"
add a3c99f7 Add and disable tests for optimized portable pipeline
add c14b6e8 Add Python optimized test for Flink
add a095321 Merge pull request #8488 from angoenka/py3_fix_iteration
add 2d671a6 Moving to 2.14.0-SNAPSHOT on master branch.
add 477a2a0 [BEAM-7143] adding withConsumerConfigUpdates
add 1ac8e7a Merge pull request #8398: [BEAM-7143] adding withConsumerConfigUpdates
add 759ea22 Add Snippets for 3 new Beam Patterns 2 X FilePatterns 1 X SideInput Pattern
add 311edfd Merge pull request #8538 from rezarokni/Beam-Patterns
add dae6e60 [BEAM-7257] adding withProducerConfigUpdates
add 8154ec5 Merge pull request #8541: [BEAM-7257] adding withProducerConfigUpdates
add 6643853 Update reviewer list for Py3 contributions.
add ada26ba Merge pull request #8529 from tvalentyn/patch-49
add 7d8447e [BEAM-7263] Deprecate set/getClientConfiguration in JdbcIO
add 765fe3b Merge pull request #8547 from iemejia/BEAM-7263-jdbc-deprecations
add ac48aab [BEAM-7265] Update Spark runner to use spark version 2.4.3
add 6679b00 Merge pull request #8548: [BEAM-7265] Update Spark runner to use spark version 2.4.3
add 40da292 [BEAM-7145] Make FlinkRunner compatible with Flink 1.8
add 01e9878 Merge pull request #8540: [BEAM-7145] Make FlinkRunner compatible with Flink 1.8
add c4dbe12 [BEAM-7260] UTF8 coder is breaking dataflow tests
add da4bb49 Merge pull request #8545 from ihji/BEAM-7260
add 8d59768 [BEAM-7267] Add python3*-dev to verify script
add f310234 Merge pull request #8551 from angoenka/fix_verify_scipt
add 2650c52 Downgrade missing coder error logs to info logs.
add 117a2d6 Merge pull request #8533 from aaltay/errtinf
add cc8bc6c Revert "Merge pull request #8228: [BEAM-7008] adding UTF8 String coder to Java SDK ModelCoders"
add 466038d Revert "Merge pull request #8545 from ihji/BEAM-7260"
add 0a1ec0c Merge pull request #8544 from angoenka/revert_utf8
add c02af60 [BEAM-4858] Increase tolerance on linear regression tests (#8556)
add c9142ad2 [BEAM-6627] Generify ByteMonitor to accept any type, not just KV<byte[], byte[]>
add d2f0631 [BEAM-6627] Add CountMonitor to IOIT metric utils
add 59e68a2 [BEAM-6627] Add byte and element counting to TextIOIT
add b8d7fe8 [BEAM-6627] Add byte and element counting to AvroIOIT
add 74c0a47 [BEAM-6627] Add byte and element counting to ParquetIOIT
add 535c7cf [BEAM-6627] Add byte and element count metric to XmlIOIT
add b022a1a [BEAM-6627] Add byte and element count metric in TFRecordIOIT
add a57c679 [BEAM-6627] Add byte and element counter metric to JdbcIOIT
add bdefbef [BEAM-6627] Add byte and item count metrics to MongoDBIOIT
add 30aa39f [BEAM-6627] Add byte and item count metrics to JdbcIOIT
add a16a5b7 Merge pull request #8400: [BEAM-6627] Added byte and item counting metrics to integration tests
add 0acb798 [BEAM-6627] Add item and byte counters to HadoopFormatIOIT
add 9cc27c3 Merge pull request #8561: [BEAM-6627] Add item and byte counters to HadoopFormatIOIT
add 6a36ad1 Updating python containers to beam-master-20190509
add b780403 Merge pull request #8546 from angoenka/update_container_20190509
add 7bc1f48 [BEAM-7269] remove StateSpec from hashCode of SimpleStateTag
add b6f22ef [BEAM-7269] remove StateSpec from hashCode of SimpleStateTag
add 6c1d448 Small doc clean up
add b4f1e37 [SQL] SpotlessApply for minor docs PR #8567
add 1715194 Merge pull request #8567 from akedin/minor-docs-cleanup
add 1e7ea0d [BEAM-4046] decouple gradle project names and maven artifact ids
add f198de0 [BEAM-4046] add project mappings and enable parsing of legacy commands
add abece47 Merge pull request #8194: [BEAM-4046] decouple gradle project names and maven artifact ids
add 460fbd3 [BEAM-7281] Update Python deps page to current release
add dca4fe8 Merge pull request #8568: [BEAM-7281] Update Python deps page to current release
add d618dc1 [BEAM-7253] test_with_jar_packages_invalid_file_name test fails on windows
add 2527817 Merge pull request #8537: [BEAM-7253] test_with_jar_packages_invalid_file_name test fails on Windows
add 9cb1c3c [BEAM-7181][BEAM-7243] add streaming it tests python 3.6
add 8d9abd0 Merge pull request #8550 from Juta/it-tests
add 4629e82 [BEAM-562] Add DoFn.setup and DoFn.teardown to Python SDK (#7994)
add 52be416 [BEAM-7282] portable Spark: don't fuse already optimized graph
add 0cb53c1 Merge pull request #8569 from ibzib/dont-fuse
add b0b2ac2 Beam community update blogpost (#8559)
add c03bbec [BEAM-7239] Do not close DataSource on Teardown on JdbcIO
add 5c7ee60 Merge pull request #8565 from iemejia/beam-master
add 0a5c700 [BEAM-7126] Remove LengthPrefixCoder from Flink's key Coder
add 2b0e572 Merge pull request #8472: [BEAM-7126] Remove redundant key encoding for state backend
add 126d8a0 Use a singleton PIPELINE_OPTION_PARAMETER in DoFnSignature.Parameter
add eb302dd Merge #8341: Use a singleton PIPELINE_OPTION_PARAMETER in DoFnSignature.Parameter
add 2f6d730 [BEAM-7271] adding UTF8 String coder to Java SDK ModelCoder
add d9430fc Merge pull request #8557: [BEAM-7271] adding UTF8 String coder to Java SDK ModelCoder
add 7e3559b Fix trigger paths filters to execute precommit tests only if needed
add a648016 Merge pull request #8483 from iemejia/fix-precommits-triggers
add 0fdfaca [BEAM-7028] Change ParDo load test according to the proposal, add a jenkins job for the load test
add 3b7f085 Merge pull request #8539 from mwalenia/BEAM-7028-pardo-load-test
add 891e52f Add first version of Hazelcast Jet based Java Runner
add 5674996 Merge pull request #8410: [BEAM-7305] Add first version of Hazelcast Jet based Java Runner
add bfc2471 [website] Update Flink Runner page for version 1.8
add ed6f68a Merge pull request #8573: [website] Update Flink Runner page for version 1.8
add fd627fc Update references Dataflow to Beam in Flink Runner
add 41dfd84 Add Coder components for BufferedElement Coder
add 9b27720 Remove UnboundedSocketSource from Flink Runner
add a975cc7 Merge pull request #8574: [cleanup] Remove UnboundedSocketSource from Flink Runner
add b988307 [BEAM-6493] : Add cookbook and snippets examples in Kotlin (#8439)
add 238659b Merge pull request #8358: Beam Kata for Java and Python SDKs
add efb1fa1 Fix project dependencies in pom.xml outputs
add 690e898 Merge pull request #8577: Fix project dependencies in pom.xml outputs
add a32c8db [BEAM-4046] revert test infra to be compatible with pending PRs
add 26583d6 fix syntax errors in groovy files
add 3806d52 Merge pull request #8581 from adude3141/BEAM-4046-revert-test-infra
add 9778f42 [BEAM-7251] Increase timeout for test BQ queries.
add 36082de Add missing javadoc to BigqueryClient.java
add 31785cc Merge pull request #8536 from udim/bq-test-client
add c630217 Merge pull request #8425: [BEAM-7174] Add schema modification transforms
add 32624ad [BEAM-5865] Create optional auto-balancing sharding function for Flink
add cc5c897 [BEAM-5865] Add unit tests and fix bug + changes from review
add 9ce0d8e [BEAM-5865] Improve FlinkPipelineOptions javadoc
add 434f0ec Merge pull request #8499: [BEAM-5865] Create optional auto-balancing sharding function for Flink
add 98a4d2b [BEAM-7121] Add deterministic proto coder (#8377)
add 8b8fe76 [BEAM-7220] add JavaReadViaImpulse substitution to Java expansion service
add a806c13 Merge pull request #8491 [BEAM-7220] Use JavaReadViaImpulse in Java expansion service
add 0ad7f12 [BEAM-7311] merge internal commits to beam open source trunk to prepare for the security patch (#8582)
add 4d54a33 [BEAM-7103] Adding AvroCoderTranslator for cross-language data transfer
add c52935f Merge pull request #8342: [BEAM-7103] Adding AvroCoderTranslator for cross-language data transfer
add 568c3db upgrade avatica to 1.14.0
add e2c9571 upgrade to avatica 1.15.0
add 7b1abc9 Merge pull request #8566 from vectorijk/upgrade-avatica
add 8e41ab8 [BEAM-6138] Update java SDK to report user distribution tuple metrics over the FN API
add 0294039 Merge pull request #8280 from ajamato/java_user_distribution
add 163f781 Categorize tests from UsesTestStream to UsesTestStreamWithProcessingTime
add f0b03b4 Merge pull request #8586: Categorize tests from UsesTestStream to UsesTestStreamWithProcessingTime
add 7c92fe0 [BEAM-7154] Updating Go SDK errors (Part 3)
add 4f9361b Merge pull request #8560 from youngoli/beam7154-2
add b253305 Create PAssert util function for late panes.
add 1945762 [BEAM-7331] Create PAssert util function for late panes.
add 931a0de Simple fix: make use of Reify in Latest to replace duplicate code
add 02ac871 Merge #8589: Simple fix: make use of Reify in Latest to replace duplicate code
add 2fbb1ee [BEAM-7287] Change JAVA_HOME to match new Jenkins agents (#8584)
add 9fe0a03 Modified KinesisIO to use WatermarkPolicyFactory for watermark computation. Added tests
add 288c95f Merge pull request #8513: [BEAM-7240] Kinesis IO Watermark Computation Improvements
add f3dc7dd [website] Add design doc on portable representation of schemas
add 07d8d5c Merge pull request #8591: [website] Add design doc on Portable Beam Schemas
add feaec61 [BEAM-7230] Add javadoc on provided poolable data sources on JdbcIO
add b570842 Merge pull request #8595: [BEAM-7230] Add javadoc on provided poolable data sources on JdbcIO
add 94a97fb [BEAM-7283] Update aggregated javadoc versions during linking to point to javadoc.io
add 8cc7c53 [BEAM-7283] Build javadocs using online links for all dependencies instead of a curated list.
add ece60d8 [BEAM-7283] Update aggregated javadoc versions during linking to point to javadoc.io
add 62f431f Add Postcommit test suite for Python 3.7
add adb6d0c Merge pull request #8564 from fredo838/add-suites
add 3cfcb41 Add retry strategy when TableRow or TableCell is null from BQ response.
add 4691d85 Remove unused attemptNum
add 38166bc Merge pull request #8583 from boyuanzz/patch_test
add d49a79c [BEAM-6959] Go SDK: support integration tests on Flink
add 1a09f79 run job server on an unused port
add 11675bf Merge pull request #8531 from ibzib/go-flink-subprocess
add 951cee0 [BEAM-7339] Make input and checksum configurable for Python WordCountIT
add 5e29d31 Merge pull request #8596 [BEAM-7339] Make input and checksum configurable for Python WordCountIT
add d77474d [BEAM-6908] Refactor Python performance test groovy file for easy configuration (#8518)
add 8a3480f [BEAM-7302] ensure artifactId set on pom also with unconfigured project
add 548bbd4 Merge pull request #8594: [BEAM-7302] ensure artifactId set on pom also with unconfigured project
add 3240ce0 Fix a typo in setting.gradle
add 2ec133f Merge pull request #8601 from tvalentyn/patch-50
add 42ada03 [BEAM-7347] Pass dpb_service_zone flag to benchmarks to fix breakage
add 15deadf Merge pull request #8602 from markflyhigh/fix-perf
add ddbdcd1 [BEAM-7339] Enable 1Gb input for Python wordcount benchmark
add 21fece9 Merge pull request #8608 from markflyhigh/py-large-input
add a1cb61a Updates python SDK GCP client API in sync with the current Google internal definition. (#8554)
add 38bef4a [BEAM-7275] ParDoLifeCycleTest: collect lifecycle info for DoFn insta… (#8563)
add 5687218 [BEAM-7349] Get back to "provided" scope for "kafka_clients"
add bd6bada Merge pull request #8600: [BEAM-7349] Get back to "provided" scope for "kafka_clients"
add a06b7a6 [BEAM-7359] Fix static analysis issues for HadoopFormatIO (#8616)
add 099fad8 [BEAM-7360] Fix static analysis issues for HCatalogIO (#8617)
add b936299 Fix the ContainsKv assertion which at times didn't work with some JDKs
add 4896d47 Merge pull request #8614: Fix the ContainsKv assertion that didn't work with some JDKs
add 1782292 [BEAM-7154] Fixing small error bug.
add bb046df Merge pull request #8610 [BEAM-7154] Fixing small error bug.
add dabf9c4 [BEAM-7367] Add Kafka clients jar as a Docker dependency
add 6ab90e9 Merge pull request #8625: [BEAM-7367] Add Kafka clients jar as a Docker dependency
add 411de45 [BEAM-6988] Implement a Python 3 version of getcallargs_forhints (#8590)
add b7c23d0 Update README for beam_PerformanceTests_WordCountIT Jenkins job
add 8bf45e9 Merge pull request #8609 from markflyhigh/update-doc
add f05c612 Add various Jet Runner fixes & improvements
add 7d62bb0 Add Jet-Runner to maven archetypes to facilitate easier running of examples
add 5ccaa72 Merge pull request #8592: [BEAM-7305] Improve and extend Hazelcast Jet based Java Runner
add 77eb66a Reorg Beam SQL docs and added Calcite section
add 7a65876 Merge pull request #8455: [BEAM-6916] Reorg Beam SQL docs and add Calcite section
add eddc83a [BEAM-6880] Remove deprecated Reference Runner code. (#8380)
add 66eb4bf [BEAM-7376] Update tox version used by gradle
add 4f76268 Merge pull request #8631 from udim/datastore-version-pip-check
add 2c9344d [BEAM-7373] Migrating vet runner
add 8da20c2 Merge pull request #8632 from youngoli/beam7373
add cabbce6 [BEAM-7286] RedisIO support for INCRBY/DECRBY operations
add 74899d6 Merge pull request: [BEAM-7286] RedisIO support for INCRBY/DECRBY operations
add bec0f3e [BEAM-7378] Fix Python PVR tests by adding a Kafka client dependency
add 76a7247 Change scope of runtime dependency to runtime
add 71759cb Merge pull request #8633: [BEAM-7378] Fix Python PVR tests by adding a Kafka client dependency
add 385f842 [BEAM-7378] Add trigger to Python Flink PVR tests for Kafka module
add 3e976e6 Merge pull request #8634: [BEAM-7378] Add trigger to Python Flink PVR tests for Kafka module
add ad27400 [BEAM-7135] Spark: close job bundle factory
add 34d8e50 Merge pull request #8605 [BEAM-7135] Spark: close job bundle factory
add 8b73a96 [BEAM-7230] Make PoolableDataSourceProvider a static singleton And simplifies all the inner method construction that was error-prone.
add a37ba1a Merge pull request #8635 from iemejia/BEAM-7230-jdbc-fix-pool-instantiation
add fc36c9e Update stability critical jobs status
add 79125ae Merge pull request #8630 from Ardagan/MonthDash
add 27771ef Add a passing test to postcommit suites.
add 088d16c Merge pull request #8627 from tvalentyn/patch-51
add 2638883 [BEAM-6872] Add hook for user-defined JVM initialization in workers (#8104)
add 8946e3e [BEAM-6908] Enable WordCountIT benchmark on Py36+37
add 033deea Merge pull request #8628 from markflyhigh/py36-ws-benchmark
add a6ae08a [BEAM-7371] fix dependency check
add e71af5d Merge pull request #8615 [BEAM-7371] fix dependency check
add ac4ff1d Update .test-infra/jenkins/README
add a71f305 Merge pull request #8643 from markflyhigh/update-readme
add 766a765 [BEAM-6695] Latest PTransform for Python SDK (#8206)
add 5e1c530 [BEAM-7364] Avoid possible signed integer overflow in hash.
add 6b328be Merge pull request #8593: [BEAM-7364] Avoid possible signed integer overflow in hash.
add 938ee3e [BEAM-7354] Starcgen fix when no identifiers specified. (#8611)
add e754666 [BEAM-6888] Enable SpotBugs in JdbcIO
add 67e0eaf Merge pull request #8647 from iemejia/BEAM-6888-jdbc-enable-spotbugs
add be8d10f [BEAM-7366] Include the project's SourceSets in Spotless target
add c8f3e88 Apply Spotless after fixing source directory detection
add 809c335 Move StateContextsTest to let it execute as part of the tests
add 38c62a8 Merge pull request #8624: [BEAM-7366] Include the project's SourceSets in Spotless target
add 74d8727 [BEAM-5822] Vendor bytebuddy (#8357)
add 7245eac [BEAM-7365] Reduce the volume of test data in fastavro branch of test_dynamic_work_rebalancing_exhaustive to match the volume of avro branch.
add 26a1635 Support sync_interval in TestAvro._write_data() to simplify test scenarios.
add de08c89 Merge pull request #8646: [BEAM-7365] Reduces the volume of test data in fastavro branch to match the volume of avro branch
add a7c9c01 [BEAM-6880] Removing python integration test for Java ULR.
add 2b2d9e0 Merge pull request #8658 [BEAM-6880] Removing python integration test for Java ULR.
add a9039ba Make Jet Runner usable and documented
add 9a7b60f Merge pull request #8637: [BEAM-7305] Make Jet Runner usable and documented
add ec70892 Make sfl4j runtime only for Samza Job Server too
add a9800b6 Merge pull request #8642: Make sfl4j runtime only for Samza Job Server
add b21ce5b Merge pull request #8613: [BEAM-7356] Support generic field selections in Filter.
add 6d75846 [BEAM-7393] Misc. improvements to Go error formatting.
add 4e20419 Merge pull request #8655 from youngoli/go-errors
add 8b379b4 [BEAM-7348] Support environment expiration
add 8e0e83f Merge pull request #8599: [BEAM-7348] Support environment expiration
add 8d68bac [BEAM-7023] WithKeys PTransform for Python SDK
add f0ed4a9 Merge pull request #8663 from ttanay/beam-7023
add 8d83576 [BEAM-7203] Signal Dataflow workers to use fastavro library on Python 3. (#8652)
add bb92d46 [BEAM-7409] Fix minor issues with \ and = in release guide.
add d810ab6 Merge pull request #8672 from lukecwik/release_guide
add 4cf2830 [BEAM-6027] Fix slow downloads when reading from GCS (#8553)
add a87ca3f [BEAM-4046] Allow running gradlew from repo subdirs
add 3eaab8b Merge pull request #8656: [BEAM-4046] Allow running gradlew from repo subdirs
add 0c50494 [BEAM-6380] Allow rewinding PipeStream
add f2e9f64 Merge pull request #8661: [BEAM-6380] Allow rewinding PipeStream
add 7840446 Reduce verbosity of setup.py sdist and futurize
add b404755 Merge pull request #8657: Reduce verbosity of setup.py sdist and futurize
add aef01da [BEAM-7086] Refactoring repetitive error code.
add 5556467 Merge pull request #8674 from youngoli/beam7086
add ea32ab9 [BEAM-6284] Improve error message on waitUntilFinish. (#8629)
add 97edda3 [BEAM-7131] cache executable stage output to prevent re-computation
add 7775871 Spark portable runner: cache all re-used RDDs by default
add 65f3c27 refactor to use pushDataset everywhere
add 05b9dbe Merge pull request #8558 [BEAM-7131] Spark: cache output to prevent re-computation
add 4243fa2 Refine runners/reference/java dependencies and fix minor static analysis issues
add f38f606 Merge pull request #8665: Refine runners/reference/java dependencies and fix minor static analysis issues
add 51ea73f [BEAM-7408] Make code example consistent with diagram
add 3691bf7 Merge pull request #8673: [BEAM-7408] Make code example consistent with diagram
add 5e161bb Merge pull request #8670: [BEAM-7408] Make custom options example consistent across languages
add 29c604d Fix command on flink runner docs
add aa8709e Merge pull request #8678: [BEAM-7417] Fix incorrect command on flink runner docs
add 7b851e6 Replace assertEquals(boolean, xyz) with assertBoolean(xyz) (#8037)
add 72dd802 [BEAM-4046] Allow running gradlew from repo subdirs (windows)
add fab12c7 Merge pull request #8676: [BEAM-4046] Allow running gradlew from repo subdirs (windows)
add 7a5252e [BEAM-6693] ApproximateUnique transforms for Python SDK (#8535)
add 606b18b [BEAM-7406] Include slf4j-jdk14 in Dataflow worker uber jar
add 73c6c26 [BEAM-7422] Create new java worker in flink python test
add dc26d2e Merge pull request #8682 from angoenka/fix_expansion_task
add 0f9f757 [BEAM-6429] side-input type coercion for multimaps
add edfd894 Merge pull request #8654: [BEAM-6429] side-input type coercion for multimaps
add 23714a3 Attempt to fix lots of URNs.
add 34c1511 Fix spotless errors
add c79f879 Update Python Dataflow precommit to use worker jar built locally
add 064be0e [BEAM-7387] Attempt to fix lots of URNs to follow common pattern.
add a91f4f8 splitting support over FnAPI for Go SDK
add 1809cf4 splitting support over FnAPI for Go SDK. updated
add 7512579 Update datasource.go
add 943e765 Merge pull request #8683: [BEAM-5791] splitting support over FnAPI for Go SDK
add 5d93491 [BEAM-3934] Use try with resources to close the reader on JavaReadViaImpulse
add 89c61b3 Merge pull request #8648: [BEAM-3934] Use try with resources to close the reader on JavaReadViaImpulse
add 519e045 Fix small typo in vet.go
add b1ab819 Merge pull request #8685 from lostluck/patch-4
add 838c030 [BEAM-7383] Adding strict flag to runners to validate with vet runner
add e0c7eb6 Merge pull request #8644 from youngoli/beam7383
add 6054c37 Refactor default bundle size into a constant in TextIO
add 9428cc8 Merge pull request #8692: Refactor default bundle size into a constant in TextIO
add 9ed7ca2 [cleanup] Remove dead code from Flink Runner
add 1da00e2 Merge pull request #8694: [cleanup] Remove dead code from Flink Runner
add 42cfb31 [BEAM-7421] Add missing transform payload translator for Reshuffle
add 8bf679c Merge pull request #8693: [BEAM-7421] Add missing transform payload translator for Reshuffle
add 75580db [BEAM-7426] Add underscore support on Field lexer
add 8304e94 Merge pull request #8689: [BEAM-7426] Add underscore support on Field lexer
add 882eca8 [BEAM-7141] Add window & timestamp in timer callback method argument (#8408)
add a7833bc add error log when fail to begin staging artifact
add dd0bf1a Merge pull request #8700 [BEAM-7441] Log when failing to begin staging artifact.
add 1ffe760 [BEAM-7436] Use coder for Flink's encoded key instead of falling back to Kryo
add 266ab37 Merge pull request #8696: [BEAM-7436] Use coder for Flink's encoded key instead of falling back to Kryo
add 2df702a [BEAM-7114] Move dot pipeline graph renderer to runners-core-construction-java
add 95297dd Merge pull request #8649: [BEAM-7114] Move dot pipeline graph renderer to runners-core-construc…
add 5f772bd [SQL] Add support for `FLOAT64` to DataCatalogTableProvider
add 5f83637 Merge pull request #8701 from akedin/dc-float64
add 23115ce [cleanup] Remove unncecessary Flink timestamp from UnboundedSourceWrapper
add 9fc25de Remove unused Accumulator in SparkGroupAlsoByWindowViaOutputBufferFn
add fa96678 [BEAM-7448] Remove unneeded external windowedValue in collections for window merging as the windowedValue is already with the elements. Clean related functions and avoid calling so mappers.
add cd8c6ec Merge pull request #8711: [BEAM-7448] Remove redundant windowing information and unused accumulators on Spark runner
add c552ce1 Remove usages of deprecated self.assertEquals in Python codebase
add 5623d8b Merge pull request #8612: [BEAM-7101] Remove usages of deprecated self.assertEquals in Python codebase
add a9d7887 Close grpc channel after pulling messages from pubsub
add d5d7a7b Merge pull request #8708 from ostrokach/close_grcp_channel_pubsub
add 1c26009 [BEAM-5354] Add side input nodes to their consumer's parent-scope.
add 9647808 Merge pull request #8713 from lostluck/sideforms
add 11a93ff [BEAM-3204] Clean-up Coder to only contain FunctionSpec removing SdkFunctionSpec.
add 1082584 [BEAM-3204] Clean-up Go code related to removing SdkFunctionSpec layer from Coder
add 35d3d15 [BEAM-3204] Update Python SDK Coder to only use FunctionSpec instead of SdkFunctionSpec
add a0388f0 fixup! Fix missed Python coder change
add b6ceda1 [BEAM-3204] Clean-up Coder to only contain FunctionSpec removing its usage of SdkFunctionSpec
add 07190ee [BEAM-6769][BEAM-7327][BEAM-7382] add it test for writing and reading with bigqu… (#8621)
add 9f4ce1c Update thread count default comment in sdk_worker_main.py
add 9dbf29c Merge pull request #8707 from apache/charlesccychen-patch-1
add 593ed6c [BEAM-7221] don't test timers on portable Spark runner yet
add 3455c9f Merge pull request #8723: [BEAM-7221] don't test timers on portable Spark runner yet
add de19c75 Merge pull request #7776: Make sure that BigQuery temp tables match the final output table.
add 767b1a8 [BEAM-7021] added ToString transform on PythonSDK
add 7943d5f Merge pull request #8698 from mszb/BEAM-7021
add 466a17d Fix build break caused by #7776
add b715c45 Merge pull request #8726 from apilloud/break
add b74cb17 [SQL] Add array constructor test
add e8666a4 Merge pull request #8727 from akedin/array-constructor
add 5edc1e9 Add flavor text in 1st kata descriptions as suggested by JetBrains Educational Content Manager
add d59cc7c Update Stepik for JetBrains review
add 8607140 Change Kata to Katas
add 48c5cb2 Merge pull request #8623 from henryken/kata-flavor-text
add d43b12d [BEAM-7341] Elements produced by bounded GBK are always ON_TIME and ONLY_FIRING
add 51c59b2 Merge pull request #8618: [BEAM-7341] Elements produced by bounded GBK are always ON_TIME and ONLY_FIRING
add fe30366 [BEAM-7332] Add blog post announcing Beam Katas
add 5f654cf Merge pull request #8724 from henryken/BEAM-7332-kata-blog-post
add 3f8ee2b [BEAM-6623] Complete DataflowValidatesRunner test coverage for Python 3. (#8721)
add 77cf84c [Beam 7447] Add Snippets for Options pattern. (#8710)
add c4c3fd8 [BEAM-7465] Upgrade Jackson to version 2.9.9
add ecfb432 Merge pull request #8729: [BEAM-7465] Upgrade Jackson to version 2.9.9
add 14bd7a6 Make metrics flush periodically
add f4987e2 Rename pipeline option for parallelism
add d6884d6 Make improvements in documentation
add 52443d7 Add support for running "NeedsRunner" category of tests
add 143efd8 Make cooperative ParDo execution an option, when possible
add 41c865e Remove previous edge optimization, it's not playing well with file-write sinks
add 570ad98 Make ParDoLifecycleTest less restrictive
add 25227c3 Make IMPULSE work in Jet Runner
add ab6b65e Merge pull request #8699: [BEAM-7305] Fix issues in and extend documentation for Hazelcast Jet Runner
add b57c996 [BEAM-7442][BEAM-5650] Read sequentially from bounded sources in UnboundedSourceWrapper
add 207f34e Merge pull request #8715: [BEAM-7442][BEAM-5650] Read sequentially from bounded sources in UnboundedSourceWrapper
add fb51440 [BEAM-7357] KinesisIO: fix too many checks that writing stream exists.
add 14879ed Merge pull request #8730: [BEAM-7357] KinesisIO: fix too many checks that writing stream exists.
add d2dc5c3 Add Precommit it tests for Python 3.
add c47d8d1 Merge pull request #8728: [BEAM-7459] Add Precommit IT test suites for Python 3
add c661d18 Transform catalog for Python
add 2d3bcdc Merge pull request #8534: [BEAM-7464] Transform catalog for Python
add b62b571 Downgrade verbose BeamFnMapTaskExecutor log messages
add 2471e3e Downgrade verbose BeamFnMapTaskExecutor log messages
add 784d2c2 [BEAM-7397] Avoid String.format in state namespace key construction
add c0de457 [BEAM-7397] Avoid String.format in state namespace key construction
add 15d0341 [BEAM-5878] add tests for kwonly args in python 3 (#8505)
add 8820f28 [BEAM-5709] BeamFnControlServiceTest - use latch instead of sleep
add f2236e5 Merge pull request #8671 from youngoli/beam5709
add 107849c [BEAM-7412] shut down executor service in fn harness
add 0b58c88 Merge pull request #8722 from ibzib/gcs-options
add 265fb6b Add enable_health_checker flag for Dataflow FnAPI worker
add 088c011 Merge pull request #8681 from robinyqiu/resilience
add ae60a72 [BEAM-6959] add beam_PostCommit_Go_VR_Flink job (#8606)
add e21d28f Fix float coder test on Windows.
add 465784a Merge pull request #8695 from robertwb/windows-float
add 2959fcc BEAM-7477 add is not empty helper test method (#8742)
add f50e7d0 Beam Katas - Update Beam to v2.12.0, javadoc, pydoc to use 'current' (#8735)
add 8f80722 [BEAM-5995] Added hot key params and method generating hot keys
add 22d958a [BEAM-5995] Add Python load tests that uses hot key
add 63e0170 Merge pull request #8664 from kkucharc/add-hot-key
add a189e31 Announcing Beam Summit Site (#8075)
add 8bbdc8c Prototype for wrapping python BoundedSoruce with SDF
add 37326d3 [BEAM-7443] Create a BoundedSource -> SDF wrapper in Python SDK
add fc7dedf [BEAM-7479] dynamically allocate expansion port in portable runner tests
add a0c90c7 Merge pull request #8746: [BEAM-7479] dynamically allocate expansion port in portable runner tests
add 81813c0 Revert "[BEAM-5878] add tests for kwonly args in python 3 (#8505)"
add 291f1e9 Merge pull request #8750: [BEAM-7481] Revert "[BEAM-5878] add tests for kwonly args in python 3"
add 5082ef8 [BEAM-7362] Added a performance test for BigQueryIO write
add 5d617ae Merge pull request #8626: [BEAM-7362] BigQuery IO write performance test
add 2839706 [BEAM-2943] Check existence of to-be-staged files in PipelineResources
add 0cb4275 Merge pull request #8744: [BEAM-2943] Check existence of to-be-staged files in PipelineResources
add 69f6951 Add support for full unboxing in Convert
add 79d5671 Merge pull request #8651: Add support for full unboxing in Convert
add 949c972 [BEAM-7487] Avoid throwing an exception in FlinkRunnerResult#cancel
add ace6ce8 Merge pull request #8755: [BEAM-7487] Avoid throwing an exception in FlinkRunnerResult#cancel
add 28bb357 Fixing a typo
add e3926d8 Merge pull request #8749 from apache/aaltay-patch-1
add 6d17908 [BEAM-3606] Remove guava-testlib from sdks/java/core main dependency set
add 19a45bf [BEAM-3606] Remove guava-testlib from sdks/java/core main dependency set
add c3ea964 [BEAM-7175] Add Spark Portable ValidatesRunner Batch postcommit test
add 3710eea Merge pull request #8432: [BEAM-7175] Add Spark Portable ValidatesRunner Batch postcommit test
add a69e76a [BEAM-3606] Remove unused guava_testlib from dependency sets.
add 0b96270 [BEAM-3606] Remove unused guava_testlib from dependency sets.
add 1de0517 [BEAM-6620, BEAM-3606] Stop shading by default.
add 44b042e fixup! Fix sdks/java/container Kafka dependency
add 163dbe0 fixup! Fixup JDBC gradle build to use shaded configuration for shaded project.
add 2c95ac1 Fix Spark ValidatesRunner since the shadowTest classpath before didn't include testCompile dependencies as they were expected to be shaded into the test jar which hadoop-format wasn't doing.
add 8232336 fixup! Fix publishing to use the correct configuration property, also ensure that the jar classifier when not shading is unspecified.
add 2ede2b3 [BEAM-7388] Reify PTransform for Python SDK
add 9fdf6f7 Merge pull request #8717 from ttanay/reify-python
add 64c56d7 [BEAM-7357] Remove stream existance check
add 85f51cc Merge pull request #8758: [BEAM-7357] Remove stream existence check
add 5b7f365 [BEAM-7351] increment ack deadline for flaky pubsub test
add 4973c57 Merge pull request #8752 from Juta/flaky-it2
add a012a3a Update Beam site to use 2.13.0 as the latest release
add b2b08d2 Mark old release archived
add 366cdea Merge pull request #8645 from angoenka/website_change_2_13
add 28818e3 [BEAM-2857] Implementing WriteToFiles transform for fileio (Python) (#8394)
add 5f26c5d Update Flink artifact name in archetypes
add a3a608d Resolve Flink artifact name dynamically
add 7c8530d Merge pull request #8753: Update Flink artifact name in archetypes
add e6104a2 Adding a Sequence Generator for SQL, and posting about SQL data sources (#8542)
add e42028b [BEAM-7437] BQ integration test for streaming inserts in streaming
add 2b74933 Merge pull request #8748 from ttanay/it-streaming-inserts
add 8097972 Update python containers to beam-master-20190605
add 7c2eb17 Update python containers to beam-master-20190605
add ae3a1f2 [BEAM-6356] Add the option to use TFRecord to store cache results using PCollection's PCoder (#8687)
add e8ecea0 Use artifactId as archivesBaseName for vendor modules
add 907d269 Merge pull request #8777: [BEAM-7493] Use artifactId as archivesBaseName for vendor modules
add 6315d6f Performs some simple clean-up for the tornadoes demo.
add 3928a55 Merge pull request #8736: Perform some simple clean-up for the tornadoes demo
add af9c5dc Add explicit archivesBaseName to applyJavaNature and applyPortabilityNature
add 9f46698 Merge pull request #8767: [BEAM-7493] Use specified archivesBaseName for project dependencies
add 2305904 TfIdfTest should not be marked ValidatesRunner
add d783082 : [BEAM-7490] TfIdfTest should not be marked ValidatesRunner from kennknowles/tfidf
add 6ffd60e [BEAM-7499] Skip test since it doesn't work on DirectRunner.
add 67e3c34 Merge pull request #8779 from lukecwik/beam7499
add e363d8c Merge branch 'master' into beam3606
add b26e622 [BEAM-6620, BEAM-3606] Stop shading by default.
add 0f9d71a Fix archivesBaseName method
add fc8dc15 Merge pull request #8782: Fix archivesBaseName method
add 97eb7d5 Fix python io developer guide document errors
add 5319373 Merge pull request #8781: Fix python io developer guide document errors
add c8631cb Importing hamcrest dependency at matcher runtime
add 37dc031 Merge pull request #8778 from pabloem/hamcrest-in-matcher
add 706fd9c Fix code block formatting in docstring. (#8756)
add db82464 [BEAM-6952] fixes concatenated compressed files bug with python sdk (#8187)
add 28c35be Increase font and wording for click to expand link.
add cc5b4da Merge pull request #8773: [BEAM-7456] Increase font and wording for click to expand link
add d633888 [BEAM-7492] [BEAM-7491] add Spark runner to Go SDK
add 4ffdf1c add badges for Go-on-Spark test
add c63218d Merge pull request #8768: [BEAM-7492] [BEAM-7491] add Spark runner to Go SDK
add 6c37774 Lookup split requests on transform id rather than constant string.
add 94967b3 Merge pull request #8771 Lookup split requests on transform id rather than constant string.
add f3d2187 [BEAM-5664] Improve cancellation of portable pipelines
add 9430d31 Merge pull request #8754: [BEAM-5664] Improve cancellation of portable pipelines
add db13cfc [BEAM-5995] Fix: cron job naming
add 0ca5a1b Merge pull request #8788: [BEAM-5995] Fix: cron Jenkins job naming
add 00c0861 Using the default partitioner based on parallelism
add 2e0e6a8 Merge pull request #8785: [BEAM-7413] Fixes bug in partitioning of GroupNonMergingWindowsFunctions
add a9fa019 [BEAM-6199] Remove outdated Combine URNs
add 5ee4cf4 [BEAM-6199] Remove outdated Combine URNs
add 58be4e8 [BEAM-7507] Use TextFormat to escape workIds to a string rather than utf8
add 99204ad [BEAM-7507] Use TextFormat to escape workIds to a string rather than utf8
add 0ee670e Test
add b2329fc Adding synthetic pipeline changes.
add 37b53df Changes to restriction provider.
add 7117e2e Fixing some indents..
add 5832fee Fixing wrapping indent
add 6a1371d Fixing more indents
add 26ae7a1 Some more
add abc413a Some more
add afa0ffe And almost done
add 5d3dc85 Fixing some more things.
add 8f8b0f2 Fixing OffsetRange, some more whitespace issues, and lint issues.
add 9a039a1 Review changes.
add 2154add FIx for linter
add 16f13c4 Some small fixes and splitting time between elements.
add 479bf50 [BEAM-7342] Extend SyntheticPipeline map steps to be splittable.
add 39d695f [BEAM-7449] Add pages documenting common Beam pipeline patterns (with code samples)
add 1d98301 Add work-in-progress drain and checkpoint to runner comparison
add cd7f4e2 Merge pull request #8576: [BEAM-2888] Add the not-yet-fully-designed drain and checkpoint to runner comparison
add 532eb97 Beam 2.13.0 Blog Post
add d516914 Merge pull request #8667 from angoenka/blog_2_13
add b24e82c Merge pull request #8620: [BEAM-6673] Add schema support to BigQuery reads
add c375e89 Skip flaky test
add 79d478a Merge pull request #8792 from pabloem/ff
add 43b3e68 Fix CountingSource example stop_position bug
add 38166b0 Merge pull request #8794: Fix CountingSource example stop_position bug
add 13fcf78 [BEAM-6821] FileBasedSink improper paths (#8054)
add cc9eec9 [BEAM-7512] Replace deprecated self.assertEquals with self.assertEqual
add 6eef1ba [BEAM-7512] Fix up some other deprecated usages of self.assertEquals
add fc24deb [BEAM-7512] Replace deprecated self.assertEquals with self.assertEqual
add 89b08e1 [BEAM-7470] Update proto and all SDKs to make the logical data stream over the data plane identified solely by instruction id and transform id.
add c85f0f5 [BEAM-7470] Update proto and all SDKs to make the logical data stream over the data plane identified solely by instruction id and transform id.
add ce63125 BEAM-7475: Update programming guide with stateful and timer example (#8740)
add 25a6874 [BEAM-6620] Remove references to DEFAULT_SHADOW_CLOSURE since it does nothing now.
add 58634bc [BEAM-6620] Remove references to DEFAULT_SHADOW_CLOSURE since it does nothing now.
add b7c2512 Fixes the version number for concat compressed files fix
add 0eec015 Merge pull request #8784: [BEAM-6952]Fixes the version number for concat compressed files fix
add f42d92e Adding Python samples to the Stateful Processing post. (#8363)
add d6301f59 fix minor typo
add b6cb4f4 fix minor typo in BatchLoads.java
add d445287 Update Python dependencies page for 2.13.0
add f16b689 Merge pull request #8810: [BEAM-7524] Update Python dependencies page for 2.13.0
add 3e0b352 [BEAM-7511] Fixes the bug in KafkaTable Initialization.
add 550c1cc [BEAM-7511] Fixes the bug in KafkaTable Initialization, returned the deleted methods back.
add 93d9334 [BEAM-7511] Fixes the bug in KafkaTable Initialization, returned the deleted methods back.
add b7b0868 Merge pull request #8797 from riazela/master
add a8d0094 [BEAM-6693] replace mmh3 with default hash function (#8799)
add 6cb1fd5 [website] Correct date in release blog post
add bd81d3f Merge pull request #8806 from mxm/release-blog-post
add bae5db1 [BEAM-6865] refactor InMemoryBagUserStateFactory to standalone class
add 2451601 Merge pull request #8802: [BEAM-6865] refactor InMemoryBagUserStateFactory to standalone class
add 2e17f12 [BEAM-7526] Fix toBeamValue logic in BigQueryUtils
add 01f14a0 Merge pull request #8814 from charithe/beam-7526
add a8bdcc5 Remove the unnecessary enable_health_checker flag
add 47ba869 Merge pull request #8783 from robinyqiu/resilience
add 76a1d51 [BEAM-5995] add Jenkins job with GBK Python load tests (#8151)
add 36df7b4 Use when/then style for Mockito stubs
add e7aadf5 Use when/then style for Mockito stubs
add d779459 [BEAM-7044] portable Spark: support stateful dofns
add 4f22c54 Merge pull request #8812: [BEAM-7044] portable Spark: support stateful dofns
add 37b76b6 Add aliases for machine_type and disk_type flags to match existing documentation (#8603)
add 1aee9a8 Merge #8686: [BEAM-7399] Blog post on looping timers
add 1f53d9e Publish SDF progress information on root node.
add b71cf58 Merge pull request #8816 Publish SDF progress information on root node.
add 6af9567 Make the Go SDK more robust with respect to requested split points.
add 0d61308 Merge pull request #8786 More robust Go SDK split point handling.
add 1159dd6 [website] Add artifact staging document to design documents
add 2b2b5d5 Merge pull request #8837: [website] Add artifact staging document to design documents
add 2798d62 [BEAM-7510] Fixing fileio tests checking JSON serialization (flaky) (#8795)
add c603563 [BEAM-7529] Add Sums.ofFloats() and Sums.ofDoubles()
add d754094 Merge pull request #8820: [BEAM-7529] Add Sums.ofFloats() and Sums.ofDoubles()
add 7421a17 Add side input and side output java katas
add bb39cf0 Add side input and side output python katas
add 2bddf9c Modify the TestPipeline declaration to be 'final transient'
add a5ed7c3 Add Create.of transform names in CoGroupByKey kata
add c34f8a7 Improve the task description of ParDo OneToMany to describe OutputReceiver
add ab40ce5 Update offset for some tasks
add e2959b3 Merge branch 'master' into add-katas
add 2ac75ce Update the task descriptions: formatting, conventions, and styles
add 4e53ea4 Update Beam version to v2.13.0
add f19cf0e Add BinaryCombineFn Lambda java kata
add 29f0553 Add packages for all Java katas
add ac6faff Add composite transform Java kata
add 64bd536 Add composite transform Python kata
add 8adf0e3 Add 'public' modifier to early Task classes
add a7e64d4 Rename Tests classes to TaskTest
add 7a410ea Add branching Java kata
add 6f14fe0 Add branching Python kata
add 3541d17 Add TextIO Read Java kata
add 23ed747 Add the Kata descriptions for Branching and TextIO Read
add d43a97b Add TextIO ReadFromText Python kata
add 58738a8 Add Kata description from Branching Python kata
add 1a524aa Add Built-in IOs Java kata
add 8502d8e Add Built-in IOs Python kata
add 7ebe261 Add missing copyright for Python kata "Built-in IOs" test file
add ef8665e Add rat exclusion for Katas IO txt files
add e60e173 Removed unused entry in Java kata study_project.xml
add 3e3961b Merge pull request #8804 from henryken/add-katas
add af7ad3f [BEAM-7407] Adds portable wordcount test for Python 3 with Flink runner. (#8745)
add 999bc5a Merge pull request #8552: [BEAM-7268] make sorter extension Hadoop-free
add c9fb261 [BEAM-7221] portable Spark: timers
add 66b090d Merge pull request #8841: [BEAM-7221] Spark portable runner: support timers
add 278c1e8 Add new proposal to design docs
add 6f7415c Merge pull request #8846: [website] Add Portable batch & streaming load tests in all sdks proposal to design docs
add e9c6207 [BEAM-7368] Add method to prepare SDK harness image for jenkins jobs
add 55e1624 [BEAM-7368] Add method to prepare Flink Job server's image for jenkins jobs
add fd5c7a8 [BEAM-7368] Add cluster Flink creation and teardown steps
add e5db98c [BEAM-7368] Add GBK load test for Python + Flink
add c83018c [BEAM-7368] Specify a cron job for the load test
add 0a6390c [BEAM-7368] Apply code review suggestions
add ba797dd [BEAM-7368] Use post build script in Jenkins
add 4c77324 [BEAM-7368] Allow defining number of slots per worker. Set to 1 by default.
add b35efe8 [BEAM-7368] Added parameters that allow using different flink versions
add 0e90336 [BEAM-7368] increase worker count to be equal to Flink's parallelism
add db4ae61 [BEAM-7368] Bugfix: create proper amount of VMs for Flink Dataproc cluster
add 9e31c94 [BEAM-7368] Remove deps to external spark config in Flink init action
add fde7793 Merge pull request #8636: [BEAM-7368] Flink + Python + gbk load test
add 54a730e Documenting and refactoring fn api runner code (#8659)
add 1362956 [BEAM-7541] provide runner & filesystem deps for IOIT in runtime
add 0d9212e Merge PR 8835
add e933ddc Adding helper functions, and documentation to fn_api_runner classes. (#8793)
add 4e05a7f Read YAML as binary stream in standard_coders_test.
add 8c8ca66 Merge pull request: [BEAM-5270] Read YAML as binary stream in standard_coders_test
add 681b8db [BEAM-5315] Fix Py3 incompatibility in tfrecordio_test.
add d0de455 Merge pull request #8852 from tvalentyn/patch-53
add c7a592e Remove Python batch support for Pub/Sub from docs
add f49d1fd Merge pull request #8840: [BEAM-7136] Remove Python batch support for Pub/Sub from docs
add 06c89de [BEAM-7445] Add search field to Beam website
add 5200161 Merge pull request #8838: [BEAM-7445] Add search field to Beam website
add 0e265f4 Fixed spaces for author in looping blog. Added twitter handle for rarokni
add 460aba5 Merge pull request #8844: [BEAM-7399] Fix Looping Timer spacing
add f9d476d [BEAM-7033] Updates for reading with query using BigQuery Storage API
add 9d0eaf1 Merge pull request #8829: [BEAM-7033] Updates for reading with query using BigQuery Storage API
add e4008d1 [BEAM-6769] allow read None bytes
add 82bf1ae Merge pull request #8817: [BEAM-7530] allow read None bytes
add 614dad7 [BEAM-7553] add portability to Spark webpage
add 84e54af Merge pull request #8854: [BEAM-7553] Add portable runner to Spark webpage
add 3fdf691 [BEAM-7043] Add DynamoDBIO
add 33d7cd0 Merge pull request #8390: [BEAM-7043] Add DynamoDBIO
add 121df06 [BEAM-7533] Fix CoderRegistry for Float
add 6d093f5 Merge pull request #8825: [BEAM-7533] Fix CoderRegistry for Float
add f4cf92d [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization
add 211c788 Merge pull request #8697: [BEAM-7126] Fix StateRequestHandler type variables to allow for proper key parameterization
add c3a0fda [BEAM-7402] Added a performance test for BigQuery IO read
add 59ea050 Merge pull request #8675: [BEAM-7402] BigQuery IO read performance tests
add 64abb60 [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction
add f2067fe Merge pull request #8857: [BEAM-7551] Checkpoint Flink's ImpulseSourceFunction
add d38a4da Merge pull request #8858: [BEAM-7542] Fix faulty cast in BigQueryUtils
add f5bc2fe Adding a Link to The Doc for Cost Estimation
add 2be7457 Adding a Link to The Doc for Cost Estimation
add 5b3807b [BEAM-7531] Remove and disallow KMS key setting
add e46d0f8 [BEAM-7531] Remove and disallow KMS key setting
add 55e3f16 Remove unused **kwargs arguments for various transforms.
add 11aacd2 Remove unused **kwargs arguments for various transforms.
add 9b99768 Reduce gradle build verbosity
add 6b3b970 Merge pull request #8831: Reduce gradle build verbosity
add b25be12 [BEAM-7432] Set input and output files for sdist
add a71bfda Merge pull request #8765: [BEAM-7432] Set input and output files for sdist
add 9bbe6f5 [BEAM-7450] Support unbounded reads with HCatalogIO
add 19804ac Merge pull request #8718: [BEAM-7450] Support unbounded reads with HCatalogIO
add 2d49e58 Add link to type hints design doc
add 4d2eba2 Merge pull request #8865: [website] Add link to the type hints design doc for python 3
add 1642655 [BEAM-7144] Fix for rescaling problem on Flink >= 1.6
add 89c5abf Merge pull request #8850: [BEAM-7144] Fix for rescaling problem on Flink >= 1.6
add e432b96 [BEAM-6777] Let HealthzServlet respond actual health information of SDK harnesses (#8827)
add 18100ca Use code.jquery.com for jQuery CDN
add ff89b2e use https
add 36f4f85 [website] Use code.jquery.com for jQuery CDN
add bc2c286 [BEAM-7513] Implements row estimation for BigQuery.
add 2de7d07 Merge pull request #8822 from riazela/bigquery-rowcount
add 8f300f9 Fixes filesystem_test for Windows.
add 7e633f3 Merge pull request #8869: [BEAM-7569] Fixes filesystem_test for Windows
add 96934f2 Merge pull request #8836: Add Beam Katas to the website's "Learning Resources" page
add e295aa8 Add Python 3.6, 3.7 to the list of supported version classifiers.
add 3c61ab8 Merge pull request #8867 from tvalentyn/classifiers
add 7b7c215 Revert "[BEAM-7513] Adding Row Count for Bigquery Table"
add 35f99ba Merge pull request #8882 from apache/revert-8822-bigquery-rowcount
add 77456c1 [BEAM-7467] Add dependency classifier to published pom (#8868)
add fe6891a Fixing file naming for windows (#8870)
add 944646e [BEAM-7461] disalbe flaky tests due to flaky org.apache.beam.sdk.io.gcp.pubsub.TestPubsub.checkIfAnySubscriptionExists]
add ce09b84 [BEAM-7461] Disalbe flaky tests
add 83bc006 Use StateTags.ID_EQUIVALENCE when using comparing StateTags
add a6c9c92 Use StateTags.ID_EQUIVALENCE when using comparing StateTags in WindmillStateCache
add e7e13f9 [BEAM-5148] Add MongoDB IO to python sdk
add acb1f56 Merge pull request #8826: [BEAM-5148] Implement MongoDB IO for Python SDK
add e1f0783 support Datetime value conversion.
add f50f4d5 Merge pull request #8376: [BEAM-7130]Support Datetime value conversion in convertAvroFieldStrict
add 08a84e4 [BEAM-6674] Add schema support to JdbcIO read (#8725)
add 408f744 [BEAM-7561] HdfsFileSystems.match can match a directory.
add edd0e0c [BEAM-7561] Validate correct directory matching in LocalFileSystemsTest
add 5e5e7a5 Merge pull request #8849: [BEAM-7561] Enable HdfsFileSystems.match to match a directory
add b0bf009 [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction<V, V, V>
add 175a8d2 [BEAM-7543] deprecate old #combineBy(Sums.ofLongs(), TypeDescriptors.longs())
add eb5a8c2 Merge pull request #8848: [BEAM-7543] ReduceByKey.combineBy must accept BinaryFunction<V, V, V>
add 5c2b362 [BEAM-7579] Use bucket with default key in ITs
add 4d33e4b Address review comments
add 25955c6 [BEAM-7579] Use bucket with default key in ITs
add 15d7be3 [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8 in hot paths
add 29868a7 [BEAM-7507] Avoid more String.format and ByteString.toStringUtf8
add 814713c [BEAM-7467] Do not run gearpump pipeline remotely by default
add 4f7d802 [BEAM-7467] Do not run gearpump pipeline remotely by default
add db30622 [BEAM-6877] Fix trivial_inference for Python 3.x
add 29807c8 Address review comments
add 4dc1c0b Allow more time for IT pipeline to run
add 221e0a5 Merge pull request #8856: [BEAM-6877] Fix trivial_inference for Python 3.x
add 441b6b1 [BEAM-6955] Use base version component of Beam Python SDK version when choosing Dataflow container image to use. (#8885)
add c32acce Add common code infrastructure for element-wise snippets
add 3124531 Merge pull request #8811 from davidcavazos/element-wise-snippets
add 6d21877 Moving to 2.15.0-SNAPSHOT on master branch.
add e868b80 [BEAM-7513] Adding RowCount to BigQueryTable.
add b2fd4e3 Merge pull request #8892 from riazela/bigquery-rowcount
add 3cddc42 [BEAM-7603] Adding support for ValueProvider in Custom GCS Temp Location (#8833)
add ea2675d Add link to gRPC.io on website portability page
add 0fd2cd3 Merge pull request #8909 [BEAM-7602] Add link to gRPC.io on website portability page
add 6915114 [BEAM7597] Fix Typo on website
add 94260de Merge pull request #8911: [BEAM-7597] Fix Typo on website
add 2bf6fde [BEAM-5806] Update PubsubIO to be able to change the PubsubClientFactory
add e9debbf [BEAM-5806] Update PubsubIO to be able to change the PubsubClientFactory
add fa37fc5 [BEAM-4046, BEAM-7307] revert PR 8581, migrate jenkins tests to use directory based names instead of artifact names (#8881)
add c775eda Make PubsubIO#withCoderAndParseFn public (#8879)
add da79821 Add Python snippet for Keys, Values, KvSwap, and ToString transform
add 778bf33 Merge pull request #8906 from davidcavazos/element-wise-simple
add 1a39ecf Add Python snippet for Filter transform
add 4c1f28a Merge pull request #8900 from davidcavazos/element-wise-filter
add 03aa755 add compatibility check badges to README
add df35d5c address github comment
add 0c04a02 Merge pull request #8791 from ylil93/master
add 2b41800 Fixed typo and removed whitespace
add 553c560 Merge pull request #8910: [BEAM-7601] Fix typo on community page
add f1a6e06 Add a timeout to urlopen calls
add 30c0dd8 Merge pull request #8925 from apache/aaltay-patch-1
add ad88aa8 [Beam-6696] GroupIntoBatches transform for Python SDK (#8914)
add 19b636d Add Python snippet for FlatMap transform
add 31342b0 Merge pull request #8901 from davidcavazos/element-wise-flat-map
add b406dbc Add README files on how to setup the project for both Java and Python Katas
add 1d9d216 Merge pull request #8862: Beam Katas README to setup project
add a2aa4b9 Add to docstring of advance_watermark_to_infinity
add b4093ad Merge pull request #8922 from ttanay/adv_watermark_docstring
add ce4574f Revert "Merge pull request #8561: [BEAM-6627] Add item and byte counters to HadoopFormatIOIT"
add 3d2c969 Revert "Merge pull request #8400: [BEAM-6627] Added byte and item counting metrics to integration tests"
add e0cbd2a Merge pull request #8861: [BEAM-7431] revert expensive byte monitor operations from Core Operations and IO tests. Revert element counters too.
add 690ab87 [BEAM-7307] Fix load tests failures due to mistakes in PR 8881
add a229027 [BEAM-7307] Fix load tests failures due to mistake in previous PR
add e3f3b57 Fix erros in py sdk io utils and add unit tests
add abede33 Merge pull request #8887: Fix errors in py sdk io utils and add unit tests
add fa96609 [BEAM-7572] ApproximateUnique.Globally and ApproximateUnique.PerKey classes are inaccessible
add 0f862ba [BEAM-7572] ApproximateUnique.Globally and ApproximateUnique.PerKey classes are inaccessible
add 2d28efe [BEAM-7598] Do not build Python tar file in run_integration_test.sh
add 5b91cb4 Merge pull request #8916 from markflyhigh/fix-run-it-script
add 08f600b Add Python snippet for ParDo transform
add 754a2ec Merge pull request #8903 from davidcavazos/element-wise-pardo
add fb8efe3 BEAM-7141: add key value timer callback (#8739)
add 4416b56 [BEAM-7606] Fix JDBC time conversion tests
add 7a08617 Merge pull request #8924: [BEAM-7606] Fix JDBC time conversion tests
add e65c176 Update portable schema representation and java SchemaTranslation (#8853)
add 8745d6e [BEAM-7475] update wordcount example (#8803)
add aa8b3d0 Implement splitAtFraction for the BigQuery Storage API
add fd9ad83 Merge pull request #8832: [BEAM-7495] Add dynamic worker rebalancing to BigQuery Storage
add 826b796 [BEAM-7013] Add HLL doc link to Beam website
add 6143271 Merge pull request #8947 from robinyqiu/patch-2
add 740d804 Drop experimental from python streaming in the doc
add 350da8e Merge pull request #8932 from apache/aaltay-patch-1
add 137953b [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Update the version of guava to 26.0-jre for all our vendored artifacts containing guava
add af93621 Merge pull request #8942: [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Update the version of guava to 26.0-jre for vendored gRPC and guava
add e858f49 [BEAM-4046, BEAM-7527] Fix benchmark with correct Gradle project
add af0ca9b Merge pull request #8919: [BEAM-4046, BEAM-7527] Fix benchmark with correct Gradle project
add c44f352 Match Python 3 warning message in __init__.py with the one in setup.py.
add 063091e Merge pull request #8956 from tvalentyn/classifiers
add e6cc73b Spotless config update to include java files only under src directory
add c39ba4a Merge pull request #8954 from akedin/spotless-include-src
add 1992cde [BEAM-7548] fix flaky tests for ApproximateUnique (#8948)
add 713fd6e [BEAM-7326] add documentation bigquery data types
add 0c0234b Merge pull request #8873: [BEAM-7326] add documentation bigquery data types
add 17bf89d [BEAM-7586] Add Integration test for python mongodb io
add 00c402d Merge pull request #8895: [BEAM-7586] Add Integration test for python mongodb io
add 3c43cc1 [BEAM-7547] Avoid WindmillStateCache cache hits for stale work.
add 9cea449 [BEAM-7547] Avoid WindmillStateCache cache hits for stale work.
add 0d5e711 Add Python snippet for Map transform
add d1058a6 Merge pull request #8902: [BEAM-7389] Add Python snippet for Map transform
add 6786afd [BEAM-7548] Fix flaky tests for ApproximateUnique (#8960)
add 43bb514 [BEAM-7589] Use only one KinesisProducer instance per JVM
add b81025c [BEAM-7589] Make KinesisIOIT compatible with all other ITs
add 69f5b7ec Merge pull request #8955: [BEAM-7589] Use only one KinesisProducer instance per JVM
add b6e45f4 [BEAM-7424] Retry HTTP 429 errors from GCS
add 30fa5d8 Merge pull request #8933: [BEAM-7424] Retry HTTP 429 errors from GCS
add a4fffee [BEAM-6692] portable Spark: reshuffle translation
add f22070b Merge pull request #8966: [BEAM-6692] portable Spark: reshuffle translation
add 54475b9 [BEAM-7640] Create amazon-web-services2 module and AwsOptions
add 0d74293 Merge pull request #8963: [BEAM-7640] Create amazon-web-services2 module and AwsOptions
add 16ceca5 [BEAM-7545] Adding RowCount to TextTable.
add 3d576f7 Merge pull request #8951 from riazela/TextTableRowCount
add b31f930 BEAM-3645 add ParallelBundleProcessor
add 1f38467 BEAM-3645 reflect comments
add b1e7644 BEAM-3645 add changes from review comments
add f391aa4 BEAM-3645 add thread lock when generating process_bundle_id
add 45d8a28 [BEAM-3645] add ParallelBundleManager
add ef9f61b Tiny typo fix
add 05900a4 Merge pull request #8974 from youngoli/patch-7
add d87c46d [BEAM-7536] Fixed BQ dataset name in collecting Load Tests metrics
add 6aceab4 Merge pull request #8834: [BEAM-7536] Fix BigQuery dataset name in collecting Load Tests metrics
add 80c7d25 Update python containers to beam-master-20190605
add fbd1f4c Update python containers to beam-master-20190628
add 88f0cef [BEAM-7640] Rename the package name for amazon-web-services2 from aws to aws2
add 3643f43 Merge pull request #8983: [BEAM-7640] Rename the package name for amazon-web-services2 from aws to aws2
add 9f6fbf2 [BEAM-7504] Added top_count parameter
add 884d1f2 [BEAM-7504] Create Combine Python Load Test Jenkins job
add 7f83599 Merge pull request #8813: [BEAM-7504] Created Combine Python Load Test Jenkins job
add 6a9d246 [BEAM-5315] improve test coverage bigquery special chars
add b79f24c Merge pull request #8878: [BEAM-5315] improve test coverage bigquery special chars
add 3e4cbf2 [BEAM-4420] Allow connecting to zookeeper using external ip
add b8a7edd [BEAM-4420] Add KafkaIO integration test pipeline
add 32e2e3e Merge pull request #8986: [BEAM-4420] Add KafkaIO Integration Tests
add 79878c3 [BEAM-7683] - fix withQueryFn when split is more than 0
add cfecfb0 Merge pull request #9004: [BEAM-7683] MongoDBIO.withQueryFn ignores custom filter
add 0859e88 [BEAM-7682] Fix Combine.GroupedValues javadoc code snippet
add b2f0171 Merge pull request #8995: [BEAM-7682] Fix Combine.GroupedValues javadoc code snippet
add 707be06 Add Jet Runner to the Get Started page
add 73735cc Merge pull request #8937: [BEAM-7305] Add Jet Runner to Get Started documentation page
add 92eab27 [BEAM-7550] Reimplement Python ParDo load test according to the proposal
add 2d5e493 Merge pull request #8847: [BEAM-7550] Missing pipeline parameters in ParDo Load Test
add 279f2d4 [BEAM-5605] Update Beam Java SDK backlog to track latest changes in Beam Python SDK.
add 82cfdb8 [BEAM-5605] Update Beam Java SDK backlog to track latest changes in Beam Python SDK.
add 3772902 Fix RestrictionTracker docstring
add 69bb363 Merge pull request #8999 from boyuanzz/doc_fix
add ef4a51d Consider Elasticsearch as one word in camelCase.
add c2343c8 Merge pull request #9008 from RyanSkraba/BEAM-7698-rename-elasticsearch
add 142b581 [BEAM-6480] Adds AvroIO sink for generic records. (#9005)
add 0624ba6 unskip ReifyTest.test_window
add 42ef04a Merge pull request #8866 from ttanay/unskip-reify-test
add 67ed7ac [BEAM-6740] Add PTransformTranslator for Combine.Globally
add 289867e [BEAM-6740] Add extractAcummulatorCoder for Combine.Globally and fix tests
add dbe59e4 [BEAM-7640] Change tests to use PayloadTranslator instead of unused toProto
add 18e0ac8 [BEAM-6740] Refactor to remove duplicated code in CombineTranslation
add 16e4fbc [BEAM-6740] Add PTransformTranslator for Combine.Globally
add 3fc4478 [BEAM-7437] Add streaming flag to BQ streaming inserts IT test
add db92536 Change default timeout to 5 mins
add db913ef Merge pull request #8934: [BEAM-7437] Add streaming flag to BQ streaming inserts IT test
add a00f753 [BEAM-7708] don't expect SQL shell bundled dependencies to be shadowed
add 600099a [BEAM-7708] don't expect SQL shell bundled dependencies to be shadowed
add 4e1db10 [SQL][Doc] fix broken gradle command.
add a2f8458 Merge pull request #9021 from amaliujia/patch-1
add b2ef9ca Added new example on how to create a custom unbounded streaming source
add 66bd44f Added new example on how to create a custom unbounded streaming source
add cbac53b Update code comments to improve readability in docs (#9024)
add fab3ae0 [BEAM-7709] Re-use node for explicit flattens
add 14d8c3d [BEAM-7709] Re-use node for explicit flattens
add a25c843 Reformat CamelCase function naming style to underscore style for synthetic_pipeline.
add 3417c3a fix lint
add 5c0905c Merge pull request #9016 from boyuanzz/fix_format
add f8cd4a8 [BEAM-7653] Add PTransformTranslator for Combine.GroupedValues
add f34c286 [BEAM-7653] Add PTransformTranslator for Combine.GroupedValues
add eae055a Update pyarrow version requirement in setup.py
add 0b83651 Merge pull request #9027 from paulgc/master
add f17d1db BEAM-3645 add thread lock
add 2a00f2e [BEAM-3645] add thread lock for ParallelBundleManager
add 265127f [BEAM-7535] Created Jenkins job for BQ performance tests
add fe36909 [BEAM-7535] Delete existing data if the table already exists
add b4efbb2 Merge pull request #8874 from kamilwu/bq-perf-jenkins
add 485a781 Update Python Dataflow runner to patch side input coders on the unified worker, too
add 7d7f897 Update Python Dataflow runner to patch side input coders on the unified worker, too
add 51c8873 [BEAM-7690] Port WordCountTest off DoFnTester
add 4ba9989 [BEAM-7690] Port WordCountTest off DoFnTester
add e2025ff Use Beam's abstraction of pickler instead of dill in coder tests.
add 3376417 Merge pull request #8975 Use Beam's abstraction of pickler instead of dill in coder tests.
add 7d11bca [BEAM-7668] Add ability to query a pipeline definition from a gRPC JobService
add 05dba6a Merge PR #8977
add 5952ea0 [BEAM-7710] [website] remove outdated reference to KeyedCombineFn
add c23cddb Merge pull request #9017: [BEAM-7710] [website] remove outdated reference to KeyedCombineFn
add ad6229c [BEAM-7389] Add Python snippet for Partition transform (#8904)
add 3c1acb0 Add Python snippet for WithTimestamps transform
add 4169307 Merge pull request #8907 from davidcavazos/element-wise-with-timestamps
add b72d848 [BEAM-7689] make a temporary directory unique for FileBaseSink
add 5b68aae Merge pull request #9013: [BEAM-7689] make a temporary directory unique for FileBaseSink
add daa5ca1 [BEAM-5994] Fix condition which allows to publish metrics.
add 8e0f2d8 Merge pull request #9035: [BEAM-5994] Fix condition which allows to publish metrics
add 0740132 Update Beam Katas (Java) course on Stepik
add f3babfe Update Beam Katas (Python) course on Stepik
add 9ade438 Update the ParDo OneToMany task description to follow latest convention
add e8f8a60 Add Windowing Adding Timestamp using ParDo Java Kata
add 713f5e4 Add Windowing Adding Timestamp using WithTimestamps Java Kata
add 626037f Add Fixed Time Window Java Kata
add 2e9667c Add kata description for the Adding Timestamp katas
add 7e1f1e8 Rename 'util' package to 'org.apache.beam.learning.katas.util'
add e2f6a5d Add Triggers "Event Time Triggers" Java kata
add 8e6f5cc Add "DoFn Additional Parameters" Java kata
add 521ddee Move Built-in IOs Task into a package
add 6d9e2a2 Add WithKeys Java kata
add fe5c3c6 Change logging library to use log4j2 and add log4j2.xml in the util module
add 2cd28fc Add "Early Triggers" Java kata
add 5eee523 Update WithKeys task description javadoc link to use 'current'
add 0afe06b Add "Window Accumulation Mode" Java kata
add 8d25f7b Add package for "Early Triggers" and "Window Accumulation Mode"
add 5d8066f Update study_project.xml
add a18f0eb Fix course syllabus on Stepik
add 4789098 Fix the wrong file path for TextIO Read kata
add 7e1fd00 Update the allowed lateness and discarding accumulation mode kata descriptions to include the links to javadoc
add 6a5ddcf Update course to Stepik
add e2996a3 Reupload Python Katas lessons
add 1c9799d Merge pull request #8889 from henryken/add-windowing-katas
add f8732f5 [BEAM-7503] Added iteration parameter to CoGBK test in Python
add 39d4f89 [BEAM-7503] Created CoGBK Python Load Test Jenkins job
add df04c07 [BEAM-3959] Added a flake F821 test
add a199e6b Merge pull request #8969 from kamilwu/cogbk-jenkins
add 854489a Remove unused NEWLINE in TextSink.TextWriter
add f2a2f2a Merge pull request #8989: Remove unused NEWLINE in TextSink.TextWriter
add 25df6c9 [BEAM-7621] Null pointer exception when accessing null row fields in BeamSql
add 10fd794 Merge pull request #8930: [BEAM-7621] Null pointer exception when accessing null row fields in BeamSql
add 0b61d3e Ensure that publishing vendored artifacts checks the contents of the jar before publishing.
add 1912b6f Merge pull request #9036 from lukecwik/vendor
add b1fe777 [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Fix shading of vendored guava to exclude classes from transitive dependencies which aren't needed at runtime.a
add 0fce2b8 Merge pull request #9038 from lukecwik/vendor2
add 2215b77 [BEAM-7729] Fixes the bug by checking the value first before parsing it.
add 60f70bf Merge pull request #9045 from riazela/BigQueryNullableBugFix
add 6f8324c [BEAM-7731] Adding helper function to handle if null else pattern for expression generation
add d6044fa Merge pull request #9047 [BEAM-7731] Adding helper function to handle if null else pattern for expression generatioin.
add f7cbf88 [BEAM-7437] Raise RuntimeError for PY2 in BigqueryFullResultStreamingMatcher (#9044)
add f34d835 Some trivial typos
add 3f53bec Merge pull request #9068 Some trivial typos
add a353733 [BEAM-7737] Fix microbenchmark scripts compiled check (#9066)
add 60ed617 [BEAM-7694]Fix error spelling in annotation of SparkTransformOverrides.
add dce8f0d update annotation for PTransformMatcher.
add a89cc2a Merge pull request #9006: [BEAM-7694] Fix error spelling in annotation of SparkTransformOverrides
add 1e47cc0 [BEAM-7736] Free worker when work is skipped
add 93be6ae [BEAM-7736] Free worker when work is skipped
add 3f3e93d Done notifications for BigtableIO.Write
add 5fb21e3 Merge pull request #7805: [BEAM-3061] Done notifications for BigtableIO.Write
add 1d624b6 Add estimate_size() to source_test.LineSource
add 92b283f fix lint
add fa1c239 Add _get_file_size and UT
add 6fd2490 Merge pull request #8981 from boyuanzz/source_test
add c773822 [BEAM-7546] Increasing environment cache to avoid chances of recreating the environment
add d8eb89e Merge pull request #9046 from angoenka/fix_portable_wordcount
add de4dd5b [BEAM-7656] Add sdk-worker-parallelism arg to flink job server shadow jar
add b60aef1 Merge pull request #8967 from ibzib/parallelism
add 2364886 Merge pull request #4 from apache/master
add c27bef6 [BEAM-7666] Memory monitor change
add beb62c1 [BEAM-7666] Adds the counter
add 8001ad2 Merge pull request #8971 from dustin12/ThrottlePiping
add 8d33dae Edit of Looping Timer Blog to fix issue with Timer State
add 708cea5 Merge pull request #9010: Edit of Looping Timer Blog to fix issue with Timer State
add 41478d0 [BEAM-6675] Generate JDBC statement and preparedStatementSetter automatically when schema is available (#8962)
add e02cb19 [BEAM-7715] Mark user facing APIs related to External transforms as @Experimental
add 3363721 Merge pull request #9031: [BEAM-7715] Mark user facing APIs related to External transforms as @Experimental
add 88c3f6b [BEAM-7578] add py37 hdfs integration test (#8970)
add a035703 [BEAM-7674] Combine batch and streaming BQ Streaming Insert ITs
add 3bbec05 Merge pull request #9074 from ttanay/unify-bqsi-test
add e821038 [BEAM-7665] Support TypeDefinition options in beam.Combine()
add 126bb85 Merge branch 'master' of https://github.com/apache/beam into combine-typedef
add aa3fe32 Revise the comments to only mention the TypeDefinition options.
add b9f8b61 Merge pull request #8976 from htyleo/combine-typedef
add ea82a89 Label the ParDos with their name
add da45987 Merge pull request #9053 from happygiraffe/patch-1
add b5f7a5d [BEAM-7632] Update Python quickstart guide for Flink and Spark
add 4a7ba2d Merge pull request #8949 from ibzib/quickstart
add f745702 Add Python snippet for Regex transform
add db59a3d Merge pull request #8905 from davidcavazos/element-wise-regex
add d7bfb06 Fix stream position bug in BigQuery Storage stream source.
add a0ec59b Merge pull request #9059: [BEAM-7743] Fix stream position bug in BigQuery Storage stream source.
add 885bd50 [BEAM-7557] - Migrate DynamoDBIO to AWS SDK for Java 2
add ccfd31e Merge pull request #9086: [BEAM-7557] Migrate DynamoDBIO to AWS SDK for Java 2
add 066d11f [BEAM-4420] Extract method for publishing already collected metrics
add 6e6675c [BEAM-4420] Collect & write metrics to BigQuery and console
add adb281f Merge pull request #9073: [BEAM-4420] KafkaIOIT metrics collection
add cead81f [BEAM-7530] Add it test to read None values from BigQuery (#8875)
add 608a861 Adding logging and adding one more retry
add 29daca2 Merge pull request #9084 from pabloem/bqlggn
add 43d953d [BEAM-7499] Fixup for tricky Reify testing issue (#9077)
add 3b11a4c [BEAM-7630] add ITs for writing and reading bytes from pubsub
add 2b5e30d Merge pull request #8985 from Juta/bytes
add e851f6b BEAM-3645 improve test cases
add 38f8f3a Merge pull request #9051 from Hannah-Jiang/python-ParallelBundleProcessor
add eeee1de [BEAM-7641] Collect xunit statistics for Py ITs (#8952)
add c2f0d28 [BEAM-4948, BEAM-6267, BEAM-5559, BEAM-7289] Update the version of guava to 26.0-jre for all our vendored artifacts containing guava (#8899)
add 76bc70d [BEAM-7484] Metrics collection in BigQuery perf tests (#8766)
add a223b09 [BEAM-7545] Reordering Beam Joins and check if the produced join is correct.
add 8656d4c Merge pull request #9040 from riazela/JoinReordering
add 26358b8 [BEAM-7284] enabled to pickle MappingProxyType in order to pickle DataClass and added unittests for them
add 719da2c Merge pull request #9050 [BEAM-7284] Enabled pickling of python3 dataclasses
add 7934826 Fix documentation on iterable coder spec.
add 9e8a564 Add the ability to set the service account email for dataflow jobs
add c503f54 Matching on filename, not directory for fileio
add 90d8570 Lint fixup
add f41e29f Lint fixup
add 265c780 Merge pull request #9097 from pabloem/fx-fio-win
add e901a1e Add retractions doc to design doc page.
add d626584 Merge pull request #9102: [website] Add retractions doc to the design doc page
add 8c8a902 Add last_attempted_record_start to python OffsetRangeTracker (#9058)
add 8da5e09 [BEAM-7784] Fixup for Guava upgrade
add 35c30fe Merge pull request #9105: [BEAM-7784] Fixup for Guava upgrade
add 648277a Refactor portable JobService to allow better sharing of code.
add b05554b Simplify known runner parsing code.
add e8a2de8 [BEAM-7722] Add a Python FlinkRunner that fetches and uses released artifacts.
add c5f4334 Pull out generic java job server helpers from flink.
add 9589835 Merge pull request #9043 [BEAM-7722] Python FlinkRunner fetching released artifacts.
add e40cde7 Split Python 3 Postcommits into several jobs
add 2bbf6eb Merge pull request #9093: [BEAM-7714] [BEAM-7257] Split Python 3 postcommits into several Jenkins jobs.
add 500c570 Skip DoFn params test in Python 2 on Windows
add 11215fc Merge pull request #9081 from davidcavazos/pardo_test_fix
add 471be56 Add transform_name_mapping pipeline option for python sdk
add 3115bab Check streaming option when validate transform_name_mapping
add 7fe54a0 Merge pull request #9072 from y1chi/transform_name_mapping
add 31c8675 Revert incorrect doc on flink runner version.
add 23a879f Merge pull request #9110: Revert incorrect doc on flink runner version
add 304882c [BEAM-5191] Support for BigQuery clustering
add 94a3c85 Merge pull request #8945: [BEAM-5191] Support for BigQuery clustering
add 00383f9 Replace old badges for Python PostCommit test with split ones
add 4c86b91 Merge pull request #9114 Update badges for Python PostCommit in README.md
add 9ab0dae [BEAM-7060] Support translation of native type variables.
add 87ba0c0 [BEAM-7060] Translate the typing.Iterable type hint.
add d2b5751 Minor fixes discovered in migration.
add 21147f1 [BEAM-7060] Automated replace of typehints with typing.
add f843044 Post-rewrite lint fixes.
add f67f1dd Manual fixes for over-agressive replace.
add 0533f8f More conservative typing module translation.
add 3dd2888 Merge pull request #9109 [BEAM-7060] Migrate to native typing types where possible.
add 6b33c0e Relax pydot requirements.
add c188582 Merge pull request #9113 Relax pydot requirements.
add 10c51f5 Remove 4 empty spaces from PR template that mess up Python postcommit build status icons.
add f6367cc Merge pull request #9112 Fix PR template.
add f7f0661 [BEAM-7679] Add randomness to ITs' BQ dataset name
add 7ca8b76 Merge pull request #8991 [BEAM-7679] Add randomness to ITs' BQ dataset name
add 3a0d9c4 [BEAM-7783] Adding BeamTableStatistics.
add ff7a803 Merge pull request #9104 from riazela/BeamTableStatistics
add b6f43f1 Default to PiplelineState.UNKNOWN when job state returned from v1beta3 cannot be recogninzed.
add 7b9b7bf fixup: Address review feedback.
add ab80cc5 Merge pull request #9094 from tvalentyn/default_to_unknown
add 0dee4e3 Update Python 3 entry in Python SDK roadmap.
add 6518abf Merge pull request #9123 from tvalentyn/patch-57
add 8afee95 Revert "[BEAM-7060] Migrate to native typing types where possible."
add 8bd3b50 Merge pull request #9122: [BEAM-7798] Revert "[BEAM-7060] Migrate to native typing types where possible."
add c604768 [BEAM-7502] Create ParDo Python Load Test Jenkins job
add 69e3b3e [BEAM-7502] Renamed file with Python GBK Load Test job definition
add 5d473b5 [BEAM-7502] Reduced number of iterations to 1 in Java ParDo job
add 1640133 Merge pull request #9042: [BEAM-7502] Create ParDo Python Load Test Jenkins job
add 8e9f980 Temporary workaround for [BEAM-7473] (#9023)
add 75875de Make SDFBoundedSource wrapper work with dynamic splitting (#8944)
add 3a4d636 [BEAM-7775] Remove job_PerformanceTests_Spark
add ed8a3f5 [BEAM-7775] Remove job_PerformanceTests_Dataflow
add e9009cf Merge pull request #9101 from lgajowy/BEAM-7775-remove-obsolete--performance-test-jobs
add 5597086 Merge pull request #8943: Schema conversion cleanup
add 7567bcf [BEAM-7801] Fix wordcount_xlang postcommit.
add de8c987 Merge pull request #9126: [BEAM-7801] Fix wordcount_xlang postcommit.
add dc171ae [BEAM-7781] Update from Go Version 1.10 to 1.12
add 38cf8ad [BEAM-7781] Use correct Go gradle tasks
add 5ef5402 [BEAM-7781] Update to latest GoGradle
add 3015444 [BEAM-7696]prepare files to stage also in local master of spark runner.
add 2ae62ff Merge pull request #9019: [BEAM-7696] Prepare files to stage also in local master of spark runner
add 84ee4c6 [BEAM-7794] DynamoDBIOTest is blocking forever
add 4777f09 Merge pull request #9121: [BEAM-7794] DynamoDBIOTest is blocking forever
add 84ef8aa [BEAM-7785] synchronize watermark update with bundle processing
add 35e1cc3 Merge pull request #9108: [BEAM-7785] synchronize watermark update with bundle processing
add 6867465 [BEAM-6783] byte[] breaks in BeamSQL codegen
add f3a2f96 [SQL] Annotate BuiltinStringFunctions with Strict
add 0e98899 Merge pull request #9119: [BEAM-6783] byte[] breaks in BeamSQL codegen
add c2b600b Add setFailOnNoMatchingTests(true)
add 0f06f2e Merge pull request #9120: Add setFailOnNoMatchingTests(true)
add 5588d27 [BEAM-7726] Add tests for DataSource protocol
add 26f921c [BEAM-7726] Refactor to prepare for state backing.
add 9c1589c [BEAM-7726] Rename some SideInput code to State
add 7688bcf [BEAM-7726] Handle State Backed iterables
add 78e95cc [BEAM-7680] Removed flaky assertion and separated unit tests in synthetic source test (#9049)
add c76c807 [BEAM-7650] Add docker system prune to inventory job
add d510c2b [BEAM-7650] Add docker system prune to inventory job
add 27bb5bc Revert "Revert "[BEAM-7060] Migrate to native typing types where possible.""
add f357c48 [BEAM-7798] Unit test for BundleBasedRunner type hints.
add d080028 [BEAM-7798] Typehints fix for bundle based runner.
add 4935d0c Merge pull request #9125 [BEAM-7798] [BEAM-7060] Native types with revert and fix.
add 2f239e4 [BEAM-7747] Close the file handle owned by fastavro.write.Writer in _FastAvroSink.close().
add 09fe498 Merge pull request #9111 [BEAM-7747] Propertly close file handle in FastAvroSink.
add 082f5bc [BEAM-7794] Ignore blocking DynamoDBIOtest class
add 4672208 Merge pull request #9139: [BEAM-7794] Ignore blocking DynamoDBIOTest classes
add 7cc8613 [BEAM-7661] Create Python coGBK load test job on Flink
add edac4cb [BEAM-7661] Generate the same number of hot keys from both sources
add 5796931 Merge pull request #9127: [BEAM-7661] Create Python CoGBK load test job on Flink
add 667bf85 [BEAM-7797] Move upload_graph handling logic before template generation. (#9128)
add 6967d23 Update the Dataflow API version to 0607
add c8343eb Merge pull request #9149 from rohdesamuel/update-versions
add 7f94621 Configure console output collection for Python 3.6, 3.7 postcommit suites
add 8050915 Merge pull request #9143: [BEAM-7714] Configure console output collection for Python 3.6, 3.7 postcommit suites.
add ea024fc Weaken overly-strict microbenchmark check_complied test. (#9100)
add 9f933a6 Revert "[BEAM-7785] synchronize watermark update with bundle processing" (#9151)
add 8ea6f41 [BEAM-7814] Wait for BQ query results in tests
add 5c6f428 Merge pull request #9150 from udim/bq-test-client-wait
add 090ad02 Add progress reporting to the BigQuery source
add 4eb7610 Merge pull request #9079: [BEAM-7495] Add progress reporting to the BigQuery source
add 28281a6 [BEAM-4181] Add readFiles transform to TfRecordIO.
add c791104 Merge pull request #9088 from RyanSkraba/BEAM-4181-tfrecord-readfiles
add 0ae24ea Add hot key detection protos to Windmill
add f03b6ba Merge pull request #9155 from rohdesamuel/update-versions
add 7a63558 [BEAM-7823] Implementing a Custom NonCumulativeCost Handler
add 1c639d1 Merge pull request #9159 from riazela/CostMetadataHandler
add 0008687 Update nexmark.md
add a999e85 Merge pull request #9163: [website] Remove duplicate arguments of Nexmark Dataflow command
add 82e419c [Java] remove unused variable in DirectMetricsTest.java
add 389f649 [BEAM-7755] adding support for nested rows and arrays in BigQuery to Beam row conversion (#9089)
add 9f88443 Upgrade google-cloud-core and google-cloud-bigquery dependencies
add 8117b58 Merge pull request #9162 from aaltay/gcc
add 0650b9b [BEAM-7641] Add xunit results to run_validatescontainer.sh
add 9a3adc7 Merge pull request #9171: [BEAM-7641] Add xunit results to run_validatescontainer.sh
add 7095832 [BEAM-7608] Honor DATASTORE_EMULATOR_HOST env var
add 4eab256 Merge pull request #8921: [BEAM-7608] Honor DATASTORE_EMULATOR_HOST env var
add 6cf3b11 [BEAM-6611] BigQuery file loads in Streaming for Python SDK (#8871)
add 4c95d57 [BEAM-7662] Create Python Combine load test job on Flink
add bf19854 Merge pull request #9115: [BEAM-7662] Create Python Combine load test job on Flink
add 1cabb1a Set job name in Jet Runner
add 85bb004 Merge pull request #9177: [BEAM-7305] Set job name in Jet Runner
add 45ef1f2 [BEAM-7830]: make spark default duration consistent with PipelineResult
add d1e3e90 fixup! [BEAM-7830]: make spark default duration consistent with PipelineResult
add 4e40b53 Merge pull request #9170: [BEAM-7830] Make spark default duration consistent with PipelineResult
add 3d087be [BEAM-7366] Restore Spotless check for Flink Runner
add 8f6247e Merge pull request #9176: [BEAM-7366] Restore Spotless check for Flink Runner
add abe5a7e [BEAM-7783] BeamIOSourceRel using custom BeamStatistics
add e9e8731 Merge pull request #9161 from riazela/BeamTableStatistics
add 206bf5f [BEAM-7807] Forbid TestNG imports on Beam classes
add 707f10b [BEAM-7807] Update Beam tests to not import TestNG related classes
add 0fcb9c6 Merge pull request #9141 from iemejia/BEAM-7807-ban-testng
add f9e0969 [BEAM-7814] Make BigqueryMatcher wait for results. (#9154)
add e813673 [BEAM-7623] Add support to select MAP with Row as values in Beam SQL
add c80ca6b Merge pull request #9181 from bmv126/map_with_row_as_value
add f75c2fb Provision datasources and dashboards from config.
add 83bb3dd [BEAM-7700] Java transform catalog (#9129)
add 3ee9da9 [BEAM-7574] Use Map-based window combining
add 6c1d115 [BEAM-7574] use Merging and NonMerging accumulating strategies
add 945a606 [BEAM-7574] unify Combine.perKey and Combine.globally
add 7be5dff [BEAM-7574] Explode windows to key when needed
add 1691935 [BEAM-7574] Optimize GC performance, finalize implementation
add fb83467 Merge pull request #8946: [BEAM-7574] fix Combine performance for SparkRunner
add bfdb169 Add arbitrary value matchers to MetricsResultMatchers.
add d088904 [BEAM-6883] Improve long-running spark streaming test.
add c787b5e Merge pull request #9178: [BEAM-6883] Improve long-running spark streaming test.
add c8164a4 Nitpicky grammar fix in programming guide
add ccaf3fe Merge pull request #9180 Nitpicky grammar fix in programming guide
add 0e60dfa Merge pull request #9183: [BEAM-7675] Unify Gradle test configuration across Py2 and Py3 (#9183)
add c31a2b1 [BEAM-7837] Add adapter class for API incompatible changes in Spark 3
add a60d550 Merge pull request #9191: [BEAM-7837] Add adapter class for API incompatible changes in Spark 3
add c4c75bd [BEAM-6202] Change retry policy for dataflow's get_job (#9182)
add 80a99ed Sets a default environment for Dataflow runner
add 0f2c9dd Merge pull request #9165: [BEAM-7824] Sets a default environment for Dataflow runner
add 30989de [BEAM-6877] trivial_inference: make remaining tests pass (#8893)
add e378f13 [BEAM-7820] Add hot key detection to Batch Dataflow Worker (#9186)
add 68acce8 [BEAM-2264] Reuse GCP credentials in GCS calls.
add 786ea64 Merge pull request #9060: [BEAM-2264] Reuse GCP credentials in GCS calls.
add 5f36ea1 Change retry policy for dataflow's list_messages
add 2a18da4 Merge pull request #9196 from aaltay/m404
add 5fb6609 [BEAM-7728] [SQL] Support ParquetTable (#9054)
add 5a340c7 Fixed type on singleton side inputs sample
add 60fab9b Merge pull request #9195 from davidcavazos/filter-sample-fix
add e8fad15 [BEAM-3645] Add multiplexing for python FnApiRunner (#8979)
add 43a6430 [BEAM-7840] Provide MapTuple and FlatMapTuple for Python 3 users. (#9168)
add 841610f [BEAM-4948] Switches dataflow runner to use a container image with updated Guava dependency.
add 380d5c9 Merge pull request #9205 [BEAM-4948] Switches Dataflow to container with updated Guava dependency.
add 6cc59ac [BEAM-6783] byte[] breaks in BeamSQL codegen
add afeb935 Merge pull request #9169: [BEAM-6783] byte[] breaks in BeamSQL codegen
add 172b563 [BEAM-7808] Add method to convert avro field to beam field
add 866b8e7 Merge pull request #9142 from bmv126/Avro_field_to_beam_field
add f8b3d3e Add "--workerHarnessContainerImage=" to dataflow java11 legacy worker test target
add 4684cb0 Merge pull request #9203 from boyuanzz/fix
add 7c6241f Moving to 2.16.0-SNAPSHOT on master branch.
add e641e27 Update design-documents.md
add e4e4590 Merge pull request #9207 from riazela/patch-2
add 7472f81 [BEAM-7389] Add code examples for Filter page (#9184)
add 06021ac Add git scm extension to wipe workspace before cloning project
add 2793a72 Merge pull request #9200: [BEAM-7788] Add git scm extension to wipe workspace before cloning project
add 199618b Use dateutil for more flexible parsing of RFC-3339 / ISO-8601 formatted strings
add 4805f49 Pin python-dateutil to current major version
add 2dc8609 Merge pull request #9147: Use dateutil for more flexible parsing of RFC-3339 formatted strings
add 0285f29 [BEAM-7751] Add a CoGroupByKey load testing Jenkins job
add b9f8590 Merge pull request #9087: [BEAM-7751] Add a CoGroupByKey load testing Jenkins job
new b703f96 Add an empty spark-structured-streaming runner project targeting spark 2.4.0
new 7f33be3 Fix missing dep
new 44d3fab Add SparkPipelineOptions
new 23c42d3 Start pipeline translation
new 4247c23 Add global pipeline translation structure
new 3bb80f7 Add nodes translators structure
new 684b735 Wire node translators with pipeline translator
new c52ea32 Renames: better differenciate pipeline translator for transform translator
new 070a6d7 Organise methods in PipelineTranslator
new 2f0309b Initialise BatchTranslationContext
new c5f43ab Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
new 21cc0b5 Make transform translation clearer: renaming, comments
new 30b0117 Improve javadocs
new 9e035a0 Move SparkTransformOverrides to correct package
new ee7d8ec Move common translation context components to superclass
new 22c99d2 apply spotless
new c223c7e Make codestyle and firebug happy
new 6252c89 Add TODOs
new 4a22ca4 Post-pone batch qualifier in all classes names for readability
new 63e4b97 Add precise TODO for multiple TransformTranslator per transform URN
new f80db00 Added SparkRunnerRegistrar
new 663aeec Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
new 2f6ace4 Create PCollections manipulation methods
new 3be3536 Create Datasets manipulation methods
new e4afbb4 Add Flatten transformation translator
new 58d9f0d Add primitive GroupByKeyTranslatorBatch implementation
new b88d12c Use Iterators.transform() to return Iterable
new 37357d5 Implement read transform
new dd1bb28 update TODO
new 0ee3697 Apply spotless
new fd72ddd start source instanciation
new 6916dcc Improve exception flow
new fa8b8cc Improve type enforcement in ReadSourceTranslator
new ecd15ff Experiment over using spark Catalog to pass in Beam Source through spark Table
new 3fff9c7 Add source mocks
new eaf06e8 fix mock, wire mock in translators and create a main test.
new 32a0f31 Use raw WindowedValue so that spark Encoders could work (temporary)
new 99d06c5 clean deps
new c3d886f Move DatasetSourceMock to proper batch mode
new 46b6b75 Run pipeline in batch mode or in streaming mode
new af423ad Split batch and streaming sources and translators
new 40d9059 Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
new deee71f Clean
new ddbc0fc Add ReadSourceTranslatorStreaming
new 3668323 Move Source and translator mocks to a mock package.
new 73eb96e Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
new 8288575 Refactor DatasetSource fields
new 86f50a4 Wire real SourceTransform and not mock and update the test
new e944d40 Add missing 0-arg public constructor
new 5829530 Use new PipelineOptionsSerializationUtils
new 59b56bc Apply spotless and fix checkstyle
new 2424be8 Add a dummy schema for reader
new 6338e78 Add empty 0-arg constructor for mock source
new f53becb Clean
new 38e918a Checkstyle and Findbugs
new ab9acc2 Refactor SourceTest to a UTest instaed of a main
new f9bdc19 Fix pipeline triggering: use a spark action instead of writing the dataset
new 591aaf2 improve readability of options passing to the source
new 0e9d7fa Clean unneeded fields in DatasetReader
new d99dcc3 Fix serialization issues
new e28e3fd Add SerializationDebugger
new f369ddb Add serialization test
new 70b145a Move SourceTest to same package as tested class
new df8d0ef Fix SourceTest
new ab8b22e Simplify beam reader creation as it created once the source as already been partitioned
new ca02796 Put all transform translators Serializable
new 6a683a3 Enable test mode
new 1ef2566 Enable gradle build scan
new 188e10d Add flatten test
new 739ad71 First attempt for ParDo primitive implementation
new 6e00851 Serialize windowedValue to byte[] in source to be able to specify a binary dataset schema and deserialize windowedValue from Row to get a dataset<WindowedValue>
new a0df1f9 Comment schema choices
new ca5a2f8 Fix errorprone
new f530a62 Fix testMode output to comply with new binary schema
new d8b1531 Cleaning
new 4f0b82d Remove bundleSize parameter and always use spark default parallelism
new c3e448c Fix split bug
new a98b158 Clean
new 6005962 Add ParDoTest
new b77a225 Address minor review notes
new baf6885 Clean
new c7116de Add GroupByKeyTest
new 409f3cb Add comments and TODO to GroupByKeyTranslatorBatch
new 54a88b6 Fix type checking with Encoder of WindowedValue<T>
new a2ed2db Port latest changes of ReadSourceTranslatorBatch to ReadSourceTranslatorStreaming
new 58e9108 Remove no more needed putDatasetRaw
new 951efec Add ComplexSourceTest
new 8f8fff6 Fail in case of having SideInouts or State/Timers
new c805783 Fix Encoders: create an Encoder for every manipulated type to avoid Spark fallback to genericRowWithSchema and cast to allow having Encoders with generic types such as WindowedValue<T> and get the type checking back
new 12a86ec Apply spotless
new fc9eed2 Fixed Javadoc error
new dfac8e3 Rename SparkSideInputReader class and rename pruneOutput() to pruneOutputFilteredByTag()
new 10b07ad Don't use deprecated sideInput.getWindowingStrategyInternal()
new 2bbd8c3 Simplify logic of ParDo translator
new 424d39c Fix kryo issue in GBK translator with a workaround
new 898b299 Rename SparkOutputManager for consistency
new 17b520b Fix for test elements container in GroupByKeyTest
new 873bc95 Added "testTwoPardoInRow"
new 181e4c5 Add a test for the most simple possible Combine
new c17ce6e Rename SparkDoFnFilterFunction to DoFnFilterFunction for consistency
new 3d2a05e Generalize the use of SerializablePipelineOptions in place of (not serializable) PipelineOptions
new 4a91a3c Fix getSideInputs
new 4f1676d Extract binary schema creation in a helper class
new 8429fb6 First version of combinePerKey
new 788aefb Improve type checking of Tuple2 encoder
new f145076 Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey
new ad8b109 Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless
new e0de5d4 Add combinePerKey and CombineGlobally tests
new 6d1301c Introduce RowHelpers
new e870813 Add CombineGlobally translation to avoid translating Combine.perKey as a composite transform based on Combine.PerKey (which uses low perf GBK)
new 4a1fd5b Cleaning
new f4a193c Get back to classes in translators resolution because URNs cannot translate Combine.Globally
new e11a72d Fix various type checking issues in Combine.Globally
new 143f09f Update test with Long
new ab06b8d Fix combine. For unknown reason GenericRowWithSchema is used as input of combine so extract its content to be able to proceed
new b8cf669 Use more generic Row instead of GenericRowWithSchema
new 4f63965 Add explanation about receiving a Row as input in the combiner
new bb210c4 Fix encoder bug in combinePerkey
new 1f772db Cleaning
new f5a3a6a Implement WindowAssignTranslatorBatch
new 7e39e2a Implement WindowAssignTest
new 077a563 Fix javadoc
new 5dd6884 Added SideInput support
new dd3b0ef Fix CheckStyle violations
new 4d540fe Don't use Reshuffle translation
new d7b8ea2 Added using CachedSideInputReader
new 2551488 Added TODO comment for ReshuffleTranslatorBatch
new 62cbf48 And unchecked warning suppression
new a476cb2 Add streaming source initialisation
new 66c6090 Implement first streaming source
new 3c9e3ad Add a TODO on spark output modes
new bac7b46 Add transformators registry in PipelineTranslatorStreaming
new f446d14 Add source streaming test
new fde0e46 Specify checkpointLocation at the pipeline start
new e25571b Clean unneeded 0 arg constructor in batch source
new 041d8e3 Clean streaming source
new 5499990 Continue impl of offsets for streaming source
new dddbccf Deal with checkpoint and offset based read
new 4e87333 Apply spotless and fix spotbugs warnings
new 2b37240 Disable never ending test SimpleSourceTest.testUnboundedSource
new 470085a Fix access level issues, typos and modernize code to Java 8 style
new dac1ce9 Merge Spark Structured Streaming runner into main Spark module. Remove spark-structured-streaming module.Rename Runner to SparkStructuredStreamingRunner. Remove specific PipelineOotions and Registrar for Structured Streaming Runner
new 696d17d Fix non-vendored imports from Spark Streaming Runner classes
new 6e0341b Pass doFnSchemaInformation to ParDo batch translation
new 0687f3c Fix spotless issues after rebase
new 6b83da4 Fix logging levels in Spark Structured Streaming translation
new be96e01 Add SparkStructuredStreamingPipelineOptions and SparkCommonPipelineOptions - SparkStructuredStreamingPipelineOptions was added to have the new runner rely only on its specific options.
new 16b3eb0 Rename SparkPipelineResult to SparkStructuredStreamingPipelineResult This is done to avoid an eventual collision with the one in SparkRunner. However this cannot happen at this moment because it is package private, so it is also done for consistency.
new 0d2dc3e Use PAssert in Spark Structured Streaming transform tests
new 035d261 Ignore spark offsets (cf javadoc)
new ee67ef3 implement source.stop
new aef79ae Update javadoc
new a69cc67 Apply Spotless
new e2248ae Enable batch Validates Runner tests for Structured Streaming Runner
new 7aba40d Limit the number of partitions to make tests go 300% faster
new 01dc170 Fixes ParDo not calling setup and not tearing down if exception on startBundle
new 246e8c3 Pass transform based doFnSchemaInformation in ParDo translation
new 74316dc Consider null object case on RowHelpers, fixes empty side inputs tests.
new f460906 Put back batch/simpleSourceTest.testBoundedSource
new 81d687c Update windowAssignTest
new b9c48ac Add comment about checkpoint mark
new 53fb97b Re-code GroupByKeyTranslatorBatch to conserve windowing instead of unwindowing/windowing(GlobalWindow): simplify code, use ReduceFnRunner to merge the windows
new 7d8ab5f re-enable reduceFnRunner timers for output
new edecedf Improve visibility of debug messages
new 02f690f Add a test that GBK preserves windowing
new 06f88a6 Add TODO in Combine translations
new e7c9bed Update KVHelpers.extractKey() to deal with WindowedValue and update GBK and CPK
new 859b4ce Fix comment about schemas
new fcd021b Implement reduce part of CombineGlobally translation with windowing
new 2726c5a Output data after combine
new 80ecd5b Implement merge accumulators part of CombineGlobally translation with windowing
new c4c4d8d Fix encoder in combine call
new d47f721 Revert extractKey while combinePerKey is not done (so that it compiles)
new c48421d Apply a groupByKey avoids for some reason that the spark structured streaming fmwk casts data to Row which makes it impossible to deserialize without the coder shipped into the data. For performance reasons (avoid memory consumption and having to deserialize), we do not ship coder + data. Also add a mapparitions before GBK to avoid shuffling
new 9f83191 Fix case when a window does not merge into any other window
new ef54b3d Fix wrong encoder in combineGlobally GBK
new 537dfcc Fix bug in the window merging logic
new 007c937 Remove the mapPartition that adds a key per partition because otherwise spark will reduce values per key instead of globally
new f060076 Remove CombineGlobally translation because it is less performant than the beam sdk one (key + combinePerKey.withHotkeyFanout)
new ef47a2b Now that there is only Combine.PerKey translation, make only one Aggregator
new c4b0610 Clean no more needed KVHelpers
new a80e8c4 Clean not more needed RowHelpers
new 5840fa1 Clean not more needed WindowingHelpers
new 3c4f55c Fix javadoc of AggregatorCombiner
new 1b17242 Fixed immutable list bug
new 9a5d813 add comment in combine globally test
new b8592f6 Clean groupByKeyTest
new 8760699 Add a test that combine per key preserves windowing
new 928d5ef Ignore for now not working test testCombineGlobally
new 3f72747 Add metrics support in DoFn
new b27090b Add missing dependencies to run Spark Structured Streaming Runner on Nexmark
new 0252c70 Add setEnableSparkMetricSinks() method
new efd3070 Fix javadoc
new 4f27def Fix accumulators initialization in Combine that prevented CombineGlobally to work.
new e4d2e85 Add a test to check that CombineGlobally preserves windowing
new dc5bcae Persist all output Dataset if there are multiple outputs in pipeline Enabled Use*Metrics tests
new 735e312 Added metrics sinks and tests
new 9e81526 Make spotless happy
new 7386c77 Add PipelineResults to Spark structured streaming.
new 55539f5 Update log4j configuration
new c98460a Add spark execution plans extended debug messages.
new dfe4f93 Print number of leaf datasets
new 9d02370 fixup! Add PipelineResults to Spark structured streaming.
new aed7062 Remove no more needed AggregatorCombinerPerKey (there is only AggregatorCombiner)
new 9cc8f17 After testing performance and correctness, launch pipeline with dataset.foreach(). Make both test mode and production mode use foreach for uniformity. Move dataset print as a utility method
new 4b36264 Add a TODO on perf improvement of Pardo translation
new 66123bd Improve Pardo translation performance: avoid calling a filter transform when there is only one output tag
new 066fe45 Use "sparkMaster" in local mode to obtain number of shuffle partitions + spotless apply
new 174595b Fix hardcoded value
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (2c5d37a)
\
N -- N -- N refs/heads/spark-runner_structured-streaming (174595b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
The 209 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 6 +-
.gitignore | 6 +-
.test-infra/dataproc/create_flink_cluster.sh | 14 +-
.test-infra/dataproc/init-actions/flink.sh | 24 +-
.test-infra/jenkins/CommonJobProperties.groovy | 12 +-
.test-infra/jenkins/CommonTestProperties.groovy | 10 +-
.test-infra/jenkins/Infrastructure.groovy | 120 +
.test-infra/jenkins/LoadTestsBuilder.groovy | 37 +-
.test-infra/jenkins/NexmarkBuilder.groovy | 2 +-
.test-infra/jenkins/README.md | 50 +-
.../dependency_check_report_generator_test.py | 12 +-
.../jenkins/dependency_check/generate_report.sh | 4 +-
.../dependency_check/version_comparer_test.py | 2 +-
.test-infra/jenkins/jira_utils/jira_manager.py | 2 +-
.../jenkins/jira_utils/jira_manager_test.py | 14 +-
.test-infra/jenkins/job_Inventory.groovy | 3 +-
.../jenkins/job_LoadTests_CoGBK_Java.groovy | 209 ++
.../job_LoadTests_Combine_Flink_Python.groovy | 186 ++
.../jenkins/job_LoadTests_Combine_Java.groovy | 27 +-
.../jenkins/job_LoadTests_Combine_Python.groovy | 161 +
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 87 +
.test-infra/jenkins/job_LoadTests_GBK_Java.groovy | 259 ++
.../jenkins/job_LoadTests_GBK_Python.groovy | 151 +
.../job_LoadTests_GBK_Python_reiterate.groovy | 97 +
.test-infra/jenkins/job_LoadTests_Java.groovy | 231 --
.../jenkins/job_LoadTests_Java_Smoke.groovy | 21 +-
.../jenkins/job_LoadTests_ParDo_Java.groovy | 181 ++
.../jenkins/job_LoadTests_ParDo_Python.groovy | 137 +
.test-infra/jenkins/job_LoadTests_Python.groovy | 80 -
.../jenkins/job_LoadTests_Python_Smoke.groovy | 70 +
.../job_LoadTests_coGBK_Flink_Python.groovy | 187 ++
.../jenkins/job_LoadTests_coGBK_Python.groovy | 167 ++
.../job_PerformanceTests_BigQueryIO_Python.groovy | 101 +
.../jenkins/job_PerformanceTests_Dataflow.groovy | 43 -
.../job_PerformanceTests_FileBasedIO_IT.groovy | 2 +-
...job_PerformanceTests_FileBasedIO_IT_HDFS.groovy | 2 +-
.../job_PerformanceTests_HadoopFormat.groovy | 2 +-
.../jenkins/job_PerformanceTests_JDBC.groovy | 2 +-
.../job_PerformanceTests_MongoDBIO_IT.groovy | 2 +-
.../jenkins/job_PerformanceTests_Python.groovy | 197 +-
.../jenkins/job_PerformanceTests_Spark.groovy | 46 -
.../jenkins/job_Performancetests_Python35.groovy | 63 -
.../job_PostCommit_Go_ValidatesRunner_Flink.groovy | 38 +
.../job_PostCommit_Go_ValidatesRunner_Spark.groovy | 38 +
.../job_PostCommit_Java11_Dataflow_Examples.groovy | 2 +-
...mit_Java11_Dataflow_Portability_Examples.groovy | 2 +-
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 2 +-
.../job_PostCommit_Java_Nexmark_Dataflow.groovy | 16 +-
.../job_PostCommit_Java_Nexmark_Direct.groovy | 16 +-
.../job_PostCommit_Java_Nexmark_Flink.groovy | 16 +-
.../job_PostCommit_Java_Nexmark_Spark.groovy | 8 +-
...Java_PortableValidatesRunner_Flink_Batch.groovy | 2 +-
..._PortableValidatesRunner_Flink_Streaming.groovy | 2 +-
...Java_PortableValidatesRunner_Spark_Batch.groovy | 43 +
...job_PostCommit_Java_ValidatesRunner_Apex.groovy | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow.groovy | 2 +-
...unner_DataflowPortabilityExecutableStage.groovy | 2 +-
...mit_Java_ValidatesRunner_Dataflow_Java11.groovy | 2 +-
...b_PostCommit_Java_ValidatesRunner_Direct.groovy | 43 +
...ommit_Java_ValidatesRunner_Direct_Java11.groovy | 10 +-
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 2 +-
...PostCommit_Java_ValidatesRunner_Gearpump.groovy | 2 +-
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Samza.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Spark.groovy | 2 +-
.test-infra/jenkins/job_PostCommit_Python2.groovy | 44 +
.test-infra/jenkins/job_PostCommit_Python35.groovy | 44 +
.test-infra/jenkins/job_PostCommit_Python36.groovy | 44 +
.test-infra/jenkins/job_PostCommit_Python37.groovy | 44 +
.../jenkins/job_PostCommit_Python3_Verify.groovy | 38 -
.../job_PostCommit_Python_MongoDBIO_IT.groovy | 38 +
...ommit_Python_ValidatesContainer_Dataflow.groovy | 4 +
...stCommit_Python_ValidatesRunner_Dataflow.groovy | 14 +-
..._PostCommit_Python_ValidatesRunner_Spark.groovy | 38 +
.../jenkins/job_PostCommit_Python_Verify.groovy | 40 -
.../jenkins/job_PostCommit_Website_Publish.groovy | 4 +-
.../jenkins/job_PostCommit_Website_Test.groovy | 2 +-
.test-infra/jenkins/job_PreCommit_Go.groovy | 1 -
.../jenkins/job_PreCommit_Portable_Python.groovy | 7 +-
.test-infra/jenkins/job_PreCommit_Python.groovy | 3 +-
...b_PreCommit_Python_ValidatesRunner_Flink.groovy | 27 +-
.../jenkins/job_PreCommit_Website_Stage_GCS.groovy | 2 +-
.../job_Release_Python_NightlySnapshot.groovy | 4 +-
.../kafka-cluster/03-zookeeper/30service.yml | 1 +
.test-infra/metrics/beamgrafana-deploy.yaml | 20 +-
.../metrics/build_and_publish_containers.sh | 57 +
.../dashboards/Post-Commits_status_dashboard.json | 175 --
.test-infra/metrics/dashboards/code_velocity.json | 1167 --------
.../metrics/dashboards/post-commit_tests.json | 836 ------
.../metrics/dashboards/pre-commit_tests.json | 241 --
.../metrics/dashboards/source_data_freshness.json | 258 --
.../dashboards/stability_critical_jobs_status.json | 332 ---
.test-infra/metrics/docker-compose.yml | 12 +-
.test-infra/metrics/grafana/Dockerfile | 25 +
.../dashboards/Post-Commits_status_dashboard.json | 175 ++
.../metrics/grafana/dashboards/code_velocity.json | 1128 +++++++
.../grafana/dashboards/post-commit_tests.json | 797 +++++
.../grafana/dashboards/pre-commit_tests.json | 435 +++
.../grafana/dashboards/source_data_freshness.json | 229 ++
.../dashboards/stability_critical_jobs_status.json | 410 +++
.../grafana/provisioning/dashboards/all.yaml | 32 +
.../provisioning/datasources/beampostgresql.yaml | 35 +
.test-infra/metrics/postgres/init.sql | 18 +
README.md | 8 +-
build.gradle | 112 +-
buildSrc/build.gradle | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 553 ++--
.../org/apache/beam/gradle/GrpcVendoring.groovy | 48 +-
.../org/apache/beam/gradle/VendorJavaPlugin.groovy | 11 +-
deprecation-warning.txt | 18 +
examples/java/README.md | 5 +-
examples/java/build.gradle | 60 +-
.../apache/beam/examples/common/ExampleUtils.java | 8 +-
.../examples/common/WriteOneFilePerWindow.java | 2 +-
.../beam/examples/complete/AutoComplete.java | 4 +-
.../org/apache/beam/examples/complete/TfIdf.java | 2 +-
.../examples/complete/TopWikipediaSessions.java | 2 +-
.../beam/examples/complete/TrafficRoutes.java | 2 +-
.../beam/examples/complete/game/LeaderBoard.java | 2 +-
.../examples/complete/game/StatefulTeamScore.java | 4 +-
.../examples/complete/game/injector/Injector.java | 2 +-
.../complete/game/injector/InjectorUtils.java | 2 +-
.../game/injector/RetryHttpInitializerWrapper.java | 2 +-
.../examples/complete/game/utils/WriteToText.java | 2 +-
.../beam/examples/cookbook/BigQueryTornadoes.java | 46 +-
.../apache/beam/examples/snippets/Snippets.java | 206 +-
.../kernel/SubProcessCommandLineArgs.java | 2 +-
.../subprocess/utils/CallingSubProcessUtils.java | 2 +-
.../beam/examples/DebuggingWordCountTest.java | 2 +-
.../apache/beam/examples/MinimalWordCountTest.java | 2 +-
.../apache/beam/examples/WindowedWordCountIT.java | 6 +-
.../org/apache/beam/examples/WordCountTest.java | 19 +-
.../apache/beam/examples/complete/TfIdfTest.java | 3 -
.../examples/complete/TrafficMaxLaneFlowIT.java | 42 +-
.../beam/examples/complete/TrafficRoutesIT.java | 38 +-
.../examples/complete/game/LeaderBoardTest.java | 2 +-
.../beam/examples/complete/game/UserScoreTest.java | 2 +-
.../examples/cookbook/BigQueryTornadoesTest.java | 2 +-
.../examples/cookbook/MaxPerKeyExamplesTest.java | 2 +-
.../beam/examples/cookbook/TriggerExampleTest.java | 4 +-
.../subprocess/ExampleEchoPipelineTest.java | 2 +-
examples/kotlin/OWNERS | 5 -
examples/kotlin/build.gradle | 60 +-
.../beam/examples/kotlin/common/ExampleUtils.kt | 8 +-
.../kotlin/common/WriteOneFilePerWindow.kt | 2 +-
.../examples/kotlin/cookbook/BigQueryTornadoes.kt | 195 ++
.../kotlin/cookbook/CombinePerKeyExamples.kt | 187 ++
.../examples/kotlin/cookbook/DistinctExample.kt | 94 +
.../examples/kotlin/cookbook/FilterExamples.kt | 229 ++
.../beam/examples/kotlin/cookbook/JoinExamples.kt | 174 ++
.../examples/kotlin/cookbook/MaxPerKeyExamples.kt | 155 +
.../beam/examples/kotlin}/cookbook/README.md | 0
.../examples/kotlin/cookbook/TriggerExample.kt | 524 ++++
.../beam/examples/kotlin/snippets/Snippets.kt | 382 +++
.../examples/DebuggingWordCountTestKotlin.java | 2 +-
.../beam/examples/MinimalWordCountTestKotlin.java | 2 +-
.../beam/examples/WindowedWordCountITKotlin.kt | 6 +-
.../get-started/try-apache-beam-java.ipynb | 12 +-
gradle.properties | 2 +-
gradlew | 184 +-
gradlew.bat | 93 +-
gradlew => gradlew_orig | 0
gradlew.bat => gradlew_orig.bat | 0
learning/katas/README.md | 26 +
learning/katas/java/.idea/study_project.xml | 3151 ++++++++++++++++++++
.../commontransforms/aggregation/count/Task.java | 48 +
.../Common Transforms/Aggregation/Count/task.html | 29 +
.../aggregation/count/TaskTest.java | 46 +
.../commontransforms/aggregation/max/Task.java | 48 +
.../Common Transforms/Aggregation/Max/task.html | 29 +
.../commontransforms/aggregation/max/TaskTest.java | 46 +
.../commontransforms/aggregation/mean/Task.java | 48 +
.../Common Transforms/Aggregation/Mean/task.html | 29 +
.../aggregation/mean/TaskTest.java | 46 +
.../commontransforms/aggregation/min/Task.java | 48 +
.../Common Transforms/Aggregation/Min/task.html | 29 +
.../commontransforms/aggregation/min/TaskTest.java | 46 +
.../commontransforms/aggregation/sum/Task.java | 48 +
.../Common Transforms/Aggregation/Sum/task.html | 29 +
.../commontransforms/aggregation/sum/TaskTest.java | 46 +
.../katas/commontransforms/filter/filter/Task.java | 49 +
.../java/Common Transforms/Filter/Filter/task.html | 34 +
.../commontransforms/filter/filter/TaskTest.java | 46 +
.../katas/commontransforms/filter/pardo/Task.java | 59 +
.../java/Common Transforms/Filter/ParDo/task.html | 33 +
.../commontransforms/filter/pardo/TaskTest.java | 46 +
.../katas/commontransforms/withkeys/Task.java | 55 +
.../Common Transforms/WithKeys/WithKeys/task.html | 34 +
.../katas/commontransforms/withkeys/TaskTest.java | 57 +
.../katas/coretransforms/branching/Task.java | 57 +
.../Core Transforms/Branching/Branching/task.html | 35 +
.../katas/coretransforms/branching/TaskTest.java | 50 +
.../katas/coretransforms/cogroupbykey/Task.java | 100 +
.../coretransforms/cogroupbykey/WordsAlphabet.java | 42 +
.../CoGroupByKey/CoGroupByKey/task.html | 45 +
.../coretransforms/cogroupbykey/TaskTest.java | 57 +
.../combine/binarycombinefnlambda/Task.java | 54 +
.../Combine/BinaryCombineFn Lambda/task.html | 43 +
.../combine/binarycombinefnlambda/TaskTest.java | 50 +
.../combine/binarycombinefn/Task.java | 64 +
.../Combine/BinaryCombineFn/task.html | 50 +
.../combine/binarycombinefn/TaskTest.java | 50 +
.../coretransforms/combine/combineperkey/Task.java | 68 +
.../Combine/Combine PerKey/task.html | 48 +
.../combine/combineperkey/TaskTest.java | 53 +
.../coretransforms/combine/combinefn/Task.java | 108 +
.../Core Transforms/Combine/CombineFn/task.html | 52 +
.../coretransforms/combine/combinefn/TaskTest.java | 46 +
.../katas/coretransforms/combine/simple/Task.java | 64 +
.../Combine/Simple Function/task.html | 49 +
.../coretransforms/combine/simple/TaskTest.java | 46 +
.../katas/coretransforms/composite/Task.java | 69 +
.../Composite Transform/task.html | 51 +
.../katas/coretransforms/composite/TaskTest.java | 49 +
.../coretransforms/dofnadditionalparams/Task.java | 36 +
.../DoFn Additional Parameters/task.html | 52 +
.../dofnadditionalparams/TaskTest.java | 22 +
.../katas/coretransforms/flatten/Task.java | 60 +
.../java/Core Transforms/Flatten/Flatten/task.html | 41 +
.../katas/coretransforms/flatten/TaskTest.java | 50 +
.../katas/coretransforms/groupbykey/Task.java | 60 +
.../GroupByKey/GroupByKey/task.html | 45 +
.../katas/coretransforms/groupbykey/TaskTest.java | 56 +
.../coretransforms/map/flatmapelements/Task.java | 54 +
.../Core Transforms/Map/FlatMapElements/task.html | 44 +
.../map/flatmapelements/TaskTest.java | 46 +
.../katas/coretransforms/map/mapelements/Task.java | 53 +
.../java/Core Transforms/Map/MapElements/task.html | 42 +
.../coretransforms/map/mapelements/TaskTest.java | 46 +
.../coretransforms/map/pardoonetomany/Task.java | 61 +
.../Core Transforms/Map/ParDo OneToMany/task.html | 38 +
.../map/pardoonetomany/TaskTest.java | 46 +
.../katas/coretransforms/map/pardo/Task.java | 57 +
.../katas/java/Core Transforms/Map/ParDo/task.html | 42 +
.../katas/coretransforms/map/pardo/TaskTest.java | 46 +
.../katas/coretransforms/partition/Task.java | 62 +
.../Core Transforms/Partition/Partition/task.html | 48 +
.../katas/coretransforms/partition/TaskTest.java | 52 +
.../katas/coretransforms/sideinput/Person.java | 79 +
.../katas/coretransforms/sideinput/Task.java | 94 +
.../Side Input/Side Input/task.html | 54 +
.../katas/coretransforms/sideinput/TaskTest.java | 75 +
.../katas/coretransforms/sideoutput/Task.java | 70 +
.../Side Output/Side Output/task.html | 44 +
.../katas/coretransforms/sideoutput/TaskTest.java | 54 +
.../learning/katas/examples/wordcount/Task.java | 76 +
.../java/Examples/Word Count/Word Count/task.html | 36 +
.../katas/examples/wordcount/TaskTest.java | 56 +
.../beam/learning/katas/io/builtinios/Task.java | 36 +
.../java/IO/Built-in IOs/Built-in IOs/task.html | 29 +
.../learning/katas/io/builtinios/TaskTest.java | 22 +
.../katas/java/IO/TextIO/TextIO Read/countries.txt | 10 +
.../beam/learning/katas/io/textio/read/Task.java | 53 +
.../katas/java/IO/TextIO/TextIO Read/task.html | 47 +
.../learning/katas/io/textio/read/TaskTest.java | 57 +
.../beam/learning/katas/intro/hello/Task.java | 45 +
.../Introduction/Hello Beam/Hello Beam/task.html | 53 +
.../beam/learning/katas/intro/hello/TaskTest.java | 43 +
learning/katas/java/README.md | 31 +
.../triggers/earlytriggers/GenerateEvent.java | 42 +
.../katas/triggers/earlytriggers/Task.java | 64 +
.../Early Triggers/Early Triggers/task.html | 59 +
.../katas/triggers/earlytriggers/TaskTest.java | 67 +
.../triggers/eventtimetriggers/GenerateEvent.java | 42 +
.../katas/triggers/eventtimetriggers/Task.java | 60 +
.../Event Time Triggers/task.html | 78 +
.../katas/triggers/eventtimetriggers/TaskTest.java | 70 +
.../triggers/windowaccummode/GenerateEvent.java | 42 +
.../katas/triggers/windowaccummode/Task.java | 64 +
.../Window Accumulation Mode/task.html | 63 +
.../katas/triggers/windowaccummode/TaskTest.java | 67 +
.../windowing/addingtimestamp/pardo/Event.java | 92 +
.../windowing/addingtimestamp/pardo/Task.java | 66 +
.../Windowing/Adding Timestamp/ParDo/task.html | 48 +
.../windowing/addingtimestamp/pardo/TaskTest.java | 85 +
.../addingtimestamp/withtimestamps/Event.java | 92 +
.../addingtimestamp/withtimestamps/Task.java | 58 +
.../Adding Timestamp/WithTimestamps/task.html | 42 +
.../addingtimestamp/withtimestamps/TaskTest.java | 85 +
.../learning/katas/windowing/fixedwindow/Task.java | 70 +
.../Fixed Time Window/Fixed Time Window/task.html | 61 +
.../katas/windowing/fixedwindow/TaskTest.java | 86 +
.../katas/windowing/fixedwindow/WindowedEvent.java | 64 +
learning/katas/java/build.gradle | 118 +
.../katas/java/gradle/wrapper/gradle-wrapper.jar | Bin 0 -> 54413 bytes
.../java/gradle/wrapper/gradle-wrapper.properties | 23 +
learning/katas/java/gradlew | 190 ++
gradlew.bat => learning/katas/java/gradlew.bat | 0
learning/katas/java/settings.gradle | 43 +
learning/katas/java/util/resources/log4j2.xml | 31 +
.../org/apache/beam/learning/katas/util/Log.java | 81 +
.../beam/learning/katas/util/ContainsKvs.java | 60 +
.../apache/beam/learning/katas/util/KvMatcher.java | 56 +
learning/katas/python/.idea/study_project.xml | 2317 ++++++++++++++
.../Common Transforms/Aggregation/Count/task.html | 29 +
.../Common Transforms/Aggregation/Count/task.py | 27 +
.../Common Transforms/Aggregation/Count/tests.py | 33 +
.../Aggregation/Largest/task.html | 29 +
.../Common Transforms/Aggregation/Largest/task.py | 27 +
.../Common Transforms/Aggregation/Largest/tests.py | 33 +
.../Common Transforms/Aggregation/Mean/task.html | 29 +
.../Common Transforms/Aggregation/Mean/task.py | 27 +
.../Common Transforms/Aggregation/Mean/tests.py | 33 +
.../Aggregation/Smallest/task.html | 29 +
.../Common Transforms/Aggregation/Smallest/task.py | 27 +
.../Aggregation/Smallest/tests.py | 33 +
.../Common Transforms/Aggregation/Sum/task.html | 30 +
.../Common Transforms/Aggregation/Sum/task.py | 27 +
.../Common Transforms/Aggregation/Sum/tests.py | 33 +
.../Common Transforms/Filter/Filter/task.html | 34 +
.../python/Common Transforms/Filter/Filter/task.py | 27 +
.../Common Transforms/Filter/Filter/tests.py | 44 +
.../Common Transforms/Filter/ParDo/task.html | 31 +
.../python/Common Transforms/Filter/ParDo/task.py | 35 +
.../python/Common Transforms/Filter/ParDo/tests.py | 33 +
.../Core Transforms/Branching/Branching/task.html | 35 +
.../Core Transforms/Branching/Branching/task.py | 31 +
.../Core Transforms/Branching/Branching/tests.py | 43 +
.../CoGroupByKey/CoGroupByKey/task.html | 42 +
.../CoGroupByKey/CoGroupByKey/task.py | 57 +
.../CoGroupByKey/CoGroupByKey/tests.py | 37 +
.../Combine/Combine PerKey/task.html | 48 +
.../Core Transforms/Combine/Combine PerKey/task.py | 32 +
.../Combine/Combine PerKey/tests.py | 49 +
.../Core Transforms/Combine/CombineFn/task.html | 52 +
.../Core Transforms/Combine/CombineFn/task.py | 46 +
.../Core Transforms/Combine/CombineFn/tests.py | 44 +
.../Combine/Simple Function/task.html | 47 +
.../Combine/Simple Function/task.py | 37 +
.../Combine/Simple Function/tests.py | 44 +
.../Composite Transform/task.html | 49 +
.../Composite Transform/task.py | 37 +
.../Composite Transform/tests.py | 44 +
.../Core Transforms/Flatten/Flatten/task.html | 40 +
.../python/Core Transforms/Flatten/Flatten/task.py | 33 +
.../Core Transforms/Flatten/Flatten/tests.py | 44 +
.../GroupByKey/GroupByKey/task.html | 44 +
.../Core Transforms/GroupByKey/GroupByKey/task.py | 28 +
.../Core Transforms/GroupByKey/GroupByKey/tests.py | 35 +
.../python/Core Transforms/Map/FlatMap/task.html | 43 +
.../python/Core Transforms/Map/FlatMap/task.py | 27 +
.../python/Core Transforms/Map/FlatMap/tests.py | 44 +
.../katas/python/Core Transforms/Map/Map/task.html | 39 +
.../katas/python/Core Transforms/Map/Map/task.py | 27 +
.../katas/python/Core Transforms/Map/Map/tests.py | 44 +
.../Core Transforms/Map/ParDo OneToMany/task.html | 43 +
.../Core Transforms/Map/ParDo OneToMany/task.py | 34 +
.../Core Transforms/Map/ParDo OneToMany/tests.py | 55 +
.../python/Core Transforms/Map/ParDo/task.html | 45 +
.../katas/python/Core Transforms/Map/ParDo/task.py | 34 +
.../python/Core Transforms/Map/ParDo/tests.py | 55 +
.../Core Transforms/Partition/Partition/task.html | 47 +
.../Core Transforms/Partition/Partition/task.py | 38 +
.../Core Transforms/Partition/Partition/tests.py | 53 +
.../Side Input/Side Input/task.html | 53 +
.../Core Transforms/Side Input/Side Input/task.py | 69 +
.../Core Transforms/Side Input/Side Input/tests.py | 70 +
.../Side Output/Side Output/task.html | 44 +
.../Side Output/Side Output/task.py | 45 +
.../Side Output/Side Output/tests.py | 65 +
.../Examples/Word Count/Word Count/task.html | 36 +
.../python/Examples/Word Count/Word Count/task.py | 36 +
.../python/Examples/Word Count/Word Count/tests.py | 39 +
.../python/IO/Built-in IOs/Built-in IOs/task.html | 29 +
.../python/IO/Built-in IOs/Built-in IOs/task.py | 22 +
.../python/IO/Built-in IOs/Built-in IOs/tests.py | 16 +
.../python/IO/TextIO/ReadFromText/countries.txt | 10 +
.../katas/python/IO/TextIO/ReadFromText/task.html | 45 +
.../katas/python/IO/TextIO/ReadFromText/task.py | 29 +
.../katas/python/IO/TextIO/ReadFromText/tests.py | 55 +
.../Introduction/Hello Beam/Hello Beam/task.html | 53 +
.../Introduction/Hello Beam/Hello Beam/task.py | 26 +
.../Introduction/Hello Beam/Hello Beam/tests.py | 41 +
learning/katas/python/README.md | 32 +
learning/katas/python/log_elements.py | 37 +
learning/katas/python/requirements.txt | 18 +
learning/katas/python/test_helper.py | 240 ++
model/fn-execution/build.gradle | 4 +-
.../fn-execution/src/main/proto/beam_fn_api.proto | 69 +-
model/job-management/build.gradle | 4 +-
.../src/main/proto/beam_job_api.proto | 16 +
.../pipeline/src/main/proto/beam_runner_api.proto | 88 +-
model/pipeline/src/main/proto/schema.proto | 85 +
ownership/PYTHON_DEPENDENCY_OWNERS.yaml | 3 +
project-mappings | 112 +
release/build.gradle | 14 +-
.../python-release/python_release_automation.sh | 16 +-
.../src/main/scripts/build_release_candidate.sh | 2 +-
release/src/main/scripts/run_rc_validation.sh | 12 +-
release/src/main/scripts/verify_release_build.sh | 4 +
runners/apex/build.gradle | 42 +-
.../org/apache/beam/runners/apex/ApexRunner.java | 6 +-
.../beam/runners/apex/ApexRunnerRegistrar.java | 2 +-
.../apache/beam/runners/apex/ApexYarnLauncher.java | 8 +-
.../translation/FlattenPCollectionTranslator.java | 4 +-
.../runners/apex/translation/ParDoTranslator.java | 4 +-
.../apex/translation/TranslationContext.java | 4 +-
.../operators/ApexGroupByKeyOperator.java | 4 +-
.../translation/operators/ApexParDoOperator.java | 8 +-
.../operators/ApexProcessFnOperator.java | 4 +-
.../operators/ApexReadUnboundedInputOperator.java | 2 +-
.../translation/operators/ApexTimerInternals.java | 8 +-
.../apex/translation/utils/ApexStateInternals.java | 4 +-
.../apex/translation/utils/ApexStreamTuple.java | 2 +-
.../translation/utils/CoderAdapterStreamCodec.java | 2 +-
.../apache/beam/runners/apex/ApexRunnerTest.java | 2 +-
.../beam/runners/apex/examples/WordCountTest.java | 2 +-
.../translation/ApexGroupByKeyOperatorTest.java | 2 +-
.../FlattenPCollectionTranslatorTest.java | 4 +-
.../apex/translation/GroupByKeyTranslatorTest.java | 4 +-
.../apex/translation/ParDoTranslatorTest.java | 4 +-
.../translation/ReadUnboundTranslatorTest.java | 10 +-
.../apex/translation/SideInputTranslationTest.java | 2 +-
.../operators/ApexTimerInternalsTest.java | 2 +-
runners/core-construction-java/build.gradle | 35 +-
.../core/construction/ArtifactServiceStager.java | 14 +-
.../core/construction/AvroCoderRegistrar.java | 40 +
.../core/construction/AvroCoderTranslator.java | 44 +
.../beam/runners/core/construction/BeamUrns.java | 2 +-
.../core/construction/CoderTranslation.java | 43 +-
.../core/construction/CoderTranslators.java | 2 +-
.../core/construction/CombineTranslation.java | 235 +-
.../CreatePCollectionViewTranslation.java | 4 +-
.../construction/DeduplicatedFlattenFactory.java | 2 +-
.../DefaultExpansionServiceClientFactory.java | 2 +-
.../core/construction/DisplayDataTranslation.java | 4 +-
.../construction/EmptyFlattenAsCreateFactory.java | 2 +-
.../runners/core/construction/Environments.java | 10 +-
.../construction/ExecutableStageTranslation.java | 10 +-
.../beam/runners/core/construction/External.java | 10 +-
.../core/construction/ExternalTranslation.java | 8 +-
.../core/construction/JavaReadViaImpulse.java | 17 +-
.../core/construction/ModelCoderRegistrar.java | 12 +-
.../runners/core/construction/ModelCoders.java | 16 +-
.../construction/PCollectionViewTranslation.java | 4 +-
.../core/construction/PTransformMatchers.java | 2 +-
.../core/construction/PTransformReplacements.java | 4 +-
.../core/construction/PTransformTranslation.java | 12 +-
.../core/construction/ParDoTranslation.java | 21 +-
.../construction/PipelineOptionsTranslation.java | 10 +-
.../core/construction/PipelineResources.java | 23 +-
.../core/construction/PipelineTranslation.java | 6 +-
.../runners/core/construction/PrimitiveCreate.java | 2 +-
.../runners/core/construction/ReadTranslation.java | 12 +-
.../core/construction/RehydratedComponents.java | 8 +-
.../core/construction/ReplacementOutputs.java | 6 +-
.../core/construction/ReshuffleTranslation.java | 58 +
.../core/construction/SchemaTranslation.java | 196 +-
.../runners/core/construction/SdkComponents.java | 10 +-
.../runners/core/construction/SplittableParDo.java | 6 +-
.../construction/SplittableParDoNaiveBounded.java | 4 +-
.../core/construction/TestStreamTranslation.java | 6 +-
.../runners/core/construction/TransformInputs.java | 4 +-
.../core/construction/TriggerTranslation.java | 4 +-
.../UnboundedReadFromBoundedSource.java | 10 +-
.../core/construction/WindowIntoTranslation.java | 6 +-
.../construction/WindowingStrategyTranslation.java | 8 +-
.../core/construction/WriteFilesTranslation.java | 15 +-
.../construction/expansion/ExpansionServer.java | 6 +-
.../construction/expansion/ExpansionService.java | 26 +-
.../core/construction/graph/FusedPipeline.java | 2 +-
.../graph/GreedyPCollectionFusers.java | 8 +-
.../construction/graph/GreedyPipelineFuser.java | 14 +-
.../core/construction/graph/GreedyStageFuser.java | 4 +-
.../graph/ImmutableExecutableStage.java | 2 +-
.../runners/core/construction/graph/Networks.java | 31 +-
.../construction/graph/OutputDeduplicator.java | 6 +-
.../core/construction/graph/PipelineValidator.java | 12 +-
.../core/construction/graph/ProtoOverrides.java | 2 +-
.../core/construction/graph/QueryablePipeline.java | 18 +-
.../construction/graph/SideInputReference.java | 2 +-
.../construction/renderer/PipelineDotRenderer.java | 126 +
.../renderer/PortablePipelineDotRenderer.java | 113 +
.../core/construction/renderer/package-info.java | 20 +
.../construction/ArtifactServiceStagerTest.java | 12 +-
.../core/construction/CoderTranslationTest.java | 11 +-
.../core/construction/CombineTranslationTest.java | 28 +-
.../runners/core/construction/CommonCoderTest.java | 14 +-
.../CreatePCollectionViewTranslationTest.java | 2 +-
.../runners/core/construction/ExternalTest.java | 12 +-
.../construction/ForwardingPTransformTest.java | 2 +-
.../InMemoryArtifactStagerService.java | 8 +-
.../core/construction/JavaReadViaImpulseTest.java | 2 +-
.../runners/core/construction/ModelCodersTest.java | 13 +-
.../construction/PCollectionTranslationTest.java | 2 +-
.../core/construction/PTransformMatchersTest.java | 4 +-
.../construction/PTransformReplacementsTest.java | 2 +-
.../construction/PTransformTranslationTest.java | 2 +-
.../core/construction/ParDoTranslationTest.java | 2 +-
.../PipelineOptionsTranslationTest.java | 8 +-
.../core/construction/PipelineResourcesTest.java | 15 +-
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/construction/ReadTranslationTest.java | 2 +-
.../core/construction/ReplacementOutputsTest.java | 4 +-
.../construction/ReshuffleTranslationTest.java | 41 +
.../core/construction/SchemaTranslationTest.java | 89 +
.../construction/TestStreamTranslationTest.java | 2 +-
.../core/construction/TriggerTranslationTest.java | 2 +-
.../UnboundedReadFromBoundedSourceTest.java | 6 +-
.../construction/WindowIntoTranslationTest.java | 6 +-
.../WindowingStrategyTranslationTest.java | 2 +-
.../construction/WriteFilesTranslationTest.java | 2 +-
.../expansion/ExpansionServiceTest.java | 10 +-
.../expansion/TestExpansionService.java | 6 +-
.../construction/graph/ExecutableStageMatcher.java | 2 +-
.../construction/graph/ExecutableStageTest.java | 2 +-
.../core/construction/graph/FusedPipelineTest.java | 4 +-
.../graph/GreedyPipelineFuserTest.java | 2 +-
.../construction/graph/GreedyStageFuserTest.java | 4 +-
.../core/construction/graph/NetworksTest.java | 20 +-
.../construction/graph/OutputDeduplicatorTest.java | 4 +-
.../construction/graph/ProtoOverridesTest.java | 9 +-
.../construction/graph/QueryablePipelineTest.java | 4 +-
.../renderer/PipelineDotRendererTest.java | 97 +
.../renderer/PortablePipelineDotRendererTest.java | 68 +
runners/core-java/build.gradle | 30 +-
.../apache/beam/runners/core/ActiveWindowSet.java | 2 +-
.../beam/runners/core/GlobalCombineFnRunners.java | 2 +-
.../runners/core/GroupByKeyViaGroupByKeyOnly.java | 2 +-
.../core/InMemoryMultimapSideInputView.java | 6 +-
.../beam/runners/core/InMemoryStateInternals.java | 6 +-
.../beam/runners/core/InMemoryTimerInternals.java | 12 +-
.../beam/runners/core/KeyedWorkItemCoder.java | 2 +-
.../apache/beam/runners/core/KeyedWorkItems.java | 4 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 4 +-
.../apache/beam/runners/core/LateDataUtils.java | 4 +-
.../beam/runners/core/MergingActiveWindowSet.java | 10 +-
.../runners/core/NonMergingActiveWindowSet.java | 4 +-
.../beam/runners/core/NullSideInputReader.java | 2 +-
...TimeBoundedSplittableProcessElementInvoker.java | 8 +-
.../apache/beam/runners/core/PaneInfoTracker.java | 4 +-
.../beam/runners/core/PeekingReiterator.java | 4 +-
.../apache/beam/runners/core/ProcessFnRunner.java | 4 +-
.../beam/runners/core/ReduceFnContextFactory.java | 6 +-
.../apache/beam/runners/core/ReduceFnRunner.java | 10 +-
.../apache/beam/runners/core/SideInputHandler.java | 2 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 10 +-
.../core/SimplePushbackSideInputDoFnRunner.java | 4 +-
.../core/SplittableParDoViaKeyedWorkItems.java | 4 +-
.../core/SplittableProcessElementInvoker.java | 2 +-
.../apache/beam/runners/core/StateNamespaces.java | 20 +-
.../org/apache/beam/runners/core/StateTable.java | 6 +-
.../org/apache/beam/runners/core/StateTags.java | 9 +-
.../apache/beam/runners/core/SystemReduceFn.java | 2 +-
.../apache/beam/runners/core/TimerInternals.java | 2 +-
.../apache/beam/runners/core/WatermarkHold.java | 4 +-
.../runners/core/metrics/DefaultMetricResults.java | 2 +-
.../core/metrics/ExecutionStateSampler.java | 6 +-
.../core/metrics/ExecutionStateTracker.java | 6 +-
.../beam/runners/core/metrics/MetricUpdates.java | 2 +-
.../runners/core/metrics/MetricsContainerImpl.java | 54 +-
.../core/metrics/MetricsContainerStepMap.java | 2 +-
.../beam/runners/core/metrics/MetricsMap.java | 4 +-
.../beam/runners/core/metrics/MetricsPusher.java | 4 +-
.../runners/core/metrics/MetricsTranslation.java | 6 +-
.../core/metrics/MonitoringInfoConstants.java | 2 +
.../core/metrics/MonitoringInfoMetricName.java | 6 +-
.../runners/core/metrics/SimpleExecutionState.java | 2 +-
.../core/metrics/SimpleMonitoringInfoBuilder.java | 30 +-
.../core/metrics/SpecMonitoringInfoValidator.java | 2 +-
.../core/triggers/AfterAllStateMachine.java | 6 +-
.../AfterDelayFromFirstElementStateMachine.java | 2 +-
.../core/triggers/AfterEachStateMachine.java | 6 +-
.../core/triggers/AfterFirstStateMachine.java | 6 +-
...fterSynchronizedProcessingTimeStateMachine.java | 2 +-
.../core/triggers/AfterWatermarkStateMachine.java | 4 +-
.../triggers/ExecutableTriggerStateMachine.java | 4 +-
.../runners/core/triggers/FinishedTriggersSet.java | 2 +-
.../core/triggers/OrFinallyStateMachine.java | 2 +-
.../runners/core/triggers/TriggerStateMachine.java | 2 +-
.../TriggerStateMachineContextFactory.java | 6 +-
.../core/triggers/TriggerStateMachineRunner.java | 6 +-
.../core/InMemoryMultimapSideInputViewTest.java | 2 +-
.../beam/runners/core/KeyedWorkItemCoderTest.java | 2 +-
.../core/LateDataDroppingDoFnRunnerTest.java | 4 +-
.../runners/core/MergingActiveWindowSetTest.java | 4 +-
...BoundedSplittableProcessElementInvokerTest.java | 2 +-
.../beam/runners/core/ReduceFnRunnerTest.java | 2 +-
.../apache/beam/runners/core/ReduceFnTester.java | 10 +-
.../beam/runners/core/SideInputHandlerTest.java | 2 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 4 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 2 +-
.../runners/core/SplittableParDoProcessFnTest.java | 2 +-
.../beam/runners/core/StateInternalsTest.java | 2 +-
.../org/apache/beam/runners/core/StateTagTest.java | 20 +-
.../beam/runners/core/StatefulDoFnRunnerTest.java | 4 +-
.../apache/beam/runners/core/WindowMatchers.java | 2 +-
.../beam/runners/core/WindowMatchersTest.java | 2 +-
.../core/metrics/MetricsContainerImplTest.java | 56 +
.../core/metrics/MetricsTranslationTest.java | 4 +-
.../metrics/SimpleMonitoringInfoBuilderTest.java | 31 +-
.../metrics/SpecMonitoringInfoValidatorTest.java | 15 +-
.../core/triggers/StubTriggerStateMachine.java | 2 +-
.../core/triggers/TriggerStateMachineTester.java | 8 +-
.../core/triggers/TriggerStateMachinesTest.java | 2 +-
runners/direct-java/build.gradle | 104 +-
.../direct/BoundedReadEvaluatorFactory.java | 12 +-
.../beam/runners/direct/CloningBundleFactory.java | 2 +-
.../beam/runners/direct/CommittedResult.java | 2 +-
.../direct/CopyOnAccessInMemoryStateInternals.java | 6 +-
.../apache/beam/runners/direct/DirectGraph.java | 2 +-
.../beam/runners/direct/DirectGraphVisitor.java | 8 +-
.../beam/runners/direct/DirectGroupByKey.java | 6 +-
.../direct/DirectGroupByKeyOverrideFactory.java | 2 +-
.../apache/beam/runners/direct/DirectMetrics.java | 2 +-
.../beam/runners/direct/DirectRegistrar.java | 2 +-
.../apache/beam/runners/direct/DirectRunner.java | 14 +-
.../runners/direct/DirectTransformExecutor.java | 4 +-
.../beam/runners/direct/DoFnLifecycleManager.java | 10 +-
.../beam/runners/direct/EvaluationContext.java | 12 +-
.../direct/ExecutorServiceParallelExecutor.java | 16 +-
.../runners/direct/FlattenEvaluatorFactory.java | 2 +-
.../direct/GroupAlsoByWindowEvaluatorFactory.java | 4 +-
.../direct/GroupByKeyOnlyEvaluatorFactory.java | 4 +-
.../direct/ImmutabilityCheckingBundleFactory.java | 6 +-
.../runners/direct/ImmutableListBundleFactory.java | 8 +-
.../runners/direct/ImpulseEvaluatorFactory.java | 4 +-
.../runners/direct/KeyedPValueTrackingVisitor.java | 4 +-
.../beam/runners/direct/MultiStepCombine.java | 12 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 4 +-
.../beam/runners/direct/ParDoEvaluatorFactory.java | 6 +-
.../runners/direct/ParDoMultiOverrideFactory.java | 7 +-
.../beam/runners/direct/QuiescenceDriver.java | 4 +-
.../beam/runners/direct/RootProviderRegistry.java | 4 +-
.../beam/runners/direct/SideInputContainer.java | 18 +-
.../SplittableProcessElementsEvaluatorFactory.java | 8 +-
.../direct/StatefulParDoEvaluatorFactory.java | 10 +-
.../org/apache/beam/runners/direct/StepAndKey.java | 2 +-
.../beam/runners/direct/StepTransformResult.java | 2 +-
.../runners/direct/TestStreamEvaluatorFactory.java | 8 +-
.../runners/direct/TransformEvaluatorRegistry.java | 6 +-
.../runners/direct/TransformExecutorServices.java | 2 +-
.../runners/direct/UnboundedReadDeduplicator.java | 6 +-
.../direct/UnboundedReadEvaluatorFactory.java | 6 +-
.../beam/runners/direct/ViewEvaluatorFactory.java | 2 +-
.../beam/runners/direct/ViewOverrideFactory.java | 3 +-
.../runners/direct/WatermarkCallbackExecutor.java | 11 +-
.../beam/runners/direct/WatermarkManager.java | 32 +-
.../runners/direct/WindowEvaluatorFactory.java | 2 +-
.../runners/direct/WriteWithShardingFactory.java | 6 +-
.../runners/direct/portable/BundleFactory.java | 47 -
.../BundleFactoryOutputReceiverFactory.java | 68 -
.../runners/direct/portable/BundleProcessor.java | 33 -
.../runners/direct/portable/CommittedBundle.java | 86 -
.../runners/direct/portable/CommittedResult.java | 63 -
.../direct/portable/CompletionCallback.java | 42 -
.../CopyOnAccessInMemoryStateInternals.java | 463 ---
.../runners/direct/portable/DirectGroupByKey.java | 133 -
.../runners/direct/portable/DirectMetrics.java | 307 --
.../direct/portable/DirectStateAndTimers.java | 70 -
.../direct/portable/DirectTimerInternals.java | 105 -
.../direct/portable/DirectTransformExecutor.java | 151 -
.../direct/portable/DisplayDataValidator.java | 67 -
.../direct/portable/EmptyInputProvider.java | 38 -
.../runners/direct/portable/EvaluationContext.java | 333 ---
...valuationContextStepStateAndTimersProvider.java | 39 -
.../direct/portable/ExecutorServiceFactory.java | 30 -
.../portable/ExecutorServiceParallelExecutor.java | 392 ---
.../direct/portable/FlattenEvaluatorFactory.java | 79 -
.../GroupAlsoByWindowEvaluatorFactory.java | 282 --
.../portable/GroupByKeyOnlyEvaluatorFactory.java | 157 -
.../portable/ImmutableListBundleFactory.java | 171 --
.../direct/portable/ImpulseEvaluatorFactory.java | 110 -
.../direct/portable/PCollectionViewWindow.java | 66 -
.../direct/portable/PCollectionViewWriter.java | 34 -
.../portable/PassthroughTransformEvaluator.java | 47 -
.../runners/direct/portable/PipelineExecutor.java | 61 -
.../runners/direct/portable/PortableGraph.java | 73 -
.../runners/direct/portable/QuiescenceDriver.java | 328 --
.../runners/direct/portable/ReferenceRunner.java | 559 ----
.../portable/RemoteStageEvaluatorFactory.java | 95 -
.../runners/direct/portable/RootInputProvider.java | 44 -
.../direct/portable/RootProviderRegistry.java | 68 -
.../beam/runners/direct/portable/SourceShard.java | 33 -
.../SplittableRemoteStageEvaluatorFactory.java | 174 --
.../beam/runners/direct/portable/StepAndKey.java | 67 -
.../direct/portable/StepStateAndTimers.java | 34 -
.../direct/portable/StepTransformResult.java | 135 -
.../direct/portable/TransformEvaluator.java | 45 -
.../direct/portable/TransformEvaluatorFactory.java | 58 -
.../portable/TransformEvaluatorRegistry.java | 126 -
.../runners/direct/portable/TransformExecutor.java | 21 -
.../direct/portable/TransformExecutorFactory.java | 29 -
.../direct/portable/TransformExecutorService.java | 38 -
.../direct/portable/TransformExecutorServices.java | 180 --
.../runners/direct/portable/TransformResult.java | 101 -
.../runners/direct/portable/UncommittedBundle.java | 56 -
.../direct/portable/WatermarkCallbackExecutor.java | 171 --
.../direct/portable/WindowEvaluatorFactory.java | 123 -
.../artifact/LocalArtifactStagingLocation.java | 116 -
.../LocalFileSystemArtifactRetrievalService.java | 119 -
.../LocalFileSystemArtifactStagerService.java | 282 --
.../UnsupportedArtifactRetrievalService.java | 42 -
.../direct/portable/artifact/package-info.java | 20 -
.../runners/direct/portable/job/PreparingJob.java | 59 -
.../portable/job/ReferenceRunnerJobServer.java | 164 -
.../portable/job/ReferenceRunnerJobService.java | 308 --
.../runners/direct/portable/job/package-info.java | 23 -
.../beam/runners/direct/portable/package-info.java | 25 -
.../direct/BoundedReadEvaluatorFactoryTest.java | 4 +-
.../runners/direct/CloningBundleFactoryTest.java | 4 +-
.../beam/runners/direct/CommittedResultTest.java | 4 +-
.../CopyOnAccessInMemoryStateInternalsTest.java | 2 +-
.../runners/direct/DirectGraphVisitorTest.java | 2 +-
.../beam/runners/direct/DirectMetricsTest.java | 3 +-
.../beam/runners/direct/DirectRegistrarTest.java | 4 +-
.../runners/direct/DirectRunnerApiSurfaceTest.java | 4 +-
.../beam/runners/direct/DirectRunnerTest.java | 4 +-
.../direct/DirectTransformExecutorTest.java | 6 +-
.../runners/direct/DoFnLifecycleManagerTest.java | 2 +-
.../runners/direct/DoFnLifecycleManagersTest.java | 2 +-
.../beam/runners/direct/EvaluationContextTest.java | 4 +-
.../ExecutorServiceParallelExecutorTest.java | 4 +-
.../direct/FlattenEvaluatorFactoryTest.java | 2 +-
.../direct/GroupByKeyEvaluatorFactoryTest.java | 6 +-
.../direct/GroupByKeyOnlyEvaluatorFactoryTest.java | 6 +-
.../direct/ImmutableListBundleFactoryTest.java | 2 +-
.../direct/ImpulseEvaluatorFactoryTest.java | 2 +-
.../org/apache/beam/runners/direct/MockClock.java | 2 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 6 +-
.../runners/direct/SideInputContainerTest.java | 6 +-
.../direct/StatefulParDoEvaluatorFactoryTest.java | 4 +-
.../direct/TestStreamEvaluatorFactoryTest.java | 2 +-
.../direct/TransformExecutorServicesTest.java | 2 +-
.../direct/UnboundedReadDeduplicatorTest.java | 8 +-
.../direct/UnboundedReadEvaluatorFactoryTest.java | 14 +-
.../runners/direct/ViewEvaluatorFactoryTest.java | 2 +-
.../beam/runners/direct/WatermarkManagerTest.java | 6 +-
.../runners/direct/WindowEvaluatorFactoryTest.java | 6 +-
.../direct/WriteWithShardingFactoryTest.java | 2 +-
.../BundleFactoryOutputReceiverFactoryTest.java | 262 --
.../direct/portable/CommittedResultTest.java | 129 -
.../CopyOnAccessInMemoryStateInternalsTest.java | 591 ----
.../runners/direct/portable/DirectMetricsTest.java | 232 --
.../direct/portable/DirectTimerInternalsTest.java | 132 -
.../portable/DirectTransformExecutorTest.java | 330 --
.../direct/portable/EvaluationContextTest.java | 315 --
.../direct/portable/ExecutableGraphBuilder.java | 81 -
.../portable/FlattenEvaluatorFactoryTest.java | 122 -
.../GroupByKeyOnlyEvaluatorFactoryTest.java | 247 --
.../portable/ImmutableListBundleFactoryTest.java | 234 --
.../portable/ImpulseEvaluatorFactoryTest.java | 130 -
.../beam/runners/direct/portable/MockClock.java | 65 -
.../runners/direct/portable/PortableGraphTest.java | 176 --
.../direct/portable/ReferenceRunnerTest.java | 213 --
.../portable/RemoteStageEvaluatorFactoryTest.java | 238 --
.../direct/portable/StepTransformResultTest.java | 95 -
.../portable/TransformExecutorServicesTest.java | 143 -
.../portable/WatermarkCallbackExecutorTest.java | 136 -
.../portable/WindowEvaluatorFactoryTest.java | 314 --
.../artifact/LocalArtifactStagingLocationTest.java | 122 -
...ocalFileSystemArtifactRetrievalServiceTest.java | 240 --
.../LocalFileSystemArtifactStagerServiceTest.java | 313 --
.../UnsupportedArtifactRetrievalServiceTest.java | 150 -
.../job/ReferenceRunnerJobServiceTest.java | 150 -
runners/extensions-java/metrics/build.gradle | 12 +-
runners/flink/1.5/build.gradle | 34 +
.../{1.7 => 1.5}/job-server-container/build.gradle | 0
runners/flink/1.5/job-server/build.gradle | 31 +
.../translation/types/CoderTypeSerializer.java | 208 ++
.../translation/types/EncodedValueSerializer.java | 0
.../FlinkBroadcastStateInternalsTest.java | 0
.../flink/streaming/FlinkStateInternalsTest.java | 0
.../translation/types/CoderTypeSerializerTest.java | 0
runners/flink/1.6/build.gradle | 9 +-
runners/flink/1.6/job-server/build.gradle | 1 +
runners/flink/1.7/build.gradle | 9 +-
runners/flink/1.7/job-server/build.gradle | 1 +
runners/flink/1.8/build.gradle | 34 +
.../{1.7 => 1.8}/job-server-container/build.gradle | 0
runners/flink/1.8/job-server/build.gradle | 31 +
.../translation/types/CoderTypeSerializer.java | 165 +
.../translation/types/EncodedValueSerializer.java | 121 +
.../FlinkBroadcastStateInternalsTest.java | 96 +
.../flink/streaming/FlinkStateInternalsTest.java | 151 +
.../translation/types/CoderTypeSerializerTest.java | 73 +
runners/flink/build.gradle | 31 -
runners/flink/flink_runner.gradle | 107 +-
runners/flink/job-server-container/build.gradle | 24 -
.../flink_job_server_container.gradle | 2 +-
runners/flink/job-server/build.gradle | 30 -
runners/flink/job-server/flink_job_server.gradle | 38 +-
.../runners/flink/CreateStreamingFlinkView.java | 2 +-
.../FlinkBatchPortablePipelineTranslator.java | 18 +-
.../flink/FlinkBatchTransformTranslators.java | 8 +-
.../flink/FlinkBatchTranslationContext.java | 2 +-
.../runners/flink/FlinkExecutionEnvironments.java | 14 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 4 +-
.../flink/FlinkPipelineExecutionEnvironment.java | 6 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 12 +
.../org/apache/beam/runners/flink/FlinkRunner.java | 4 +-
.../beam/runners/flink/FlinkRunnerRegistrar.java | 2 +-
.../beam/runners/flink/FlinkRunnerResult.java | 6 +-
.../flink/FlinkStreamingPipelineTranslator.java | 195 +-
.../FlinkStreamingPortablePipelineTranslator.java | 66 +-
.../flink/FlinkStreamingTransformTranslators.java | 19 +-
.../flink/FlinkStreamingTranslationContext.java | 9 +-
.../runners/flink/FlinkTransformOverrides.java | 7 +-
.../flink/metrics/FlinkMetricContainer.java | 2 +-
.../functions/AbstractFlinkCombineRunner.java | 2 +-
.../translation/functions/FlinkAssignContext.java | 2 +-
.../FlinkDefaultExecutableStageContext.java | 4 +-
.../translation/functions/FlinkDoFnFunction.java | 2 +-
.../functions/FlinkExecutableStageFunction.java | 215 +-
.../functions/FlinkSideInputReader.java | 4 +-
.../functions/FlinkStatefulDoFnFunction.java | 2 +-
.../FlinkStreamingSideInputHandlerFactory.java | 6 +-
.../functions/HashingFlinkCombineRunner.java | 2 +-
.../functions/ImpulseSourceFunction.java | 33 +-
...CountingFlinkExecutableStageContextFactory.java | 4 +-
.../functions/SideInputInitializer.java | 2 +-
.../functions/SortingFlinkCombineRunner.java | 4 +-
.../translation/types/CoderTypeInformation.java | 7 +-
.../translation/types/CoderTypeSerializer.java | 208 --
.../types/InspectableByteArrayOutputStream.java | 32 -
.../translation/wrappers/DataInputViewWrapper.java | 10 +-
.../wrappers/DataOutputViewWrapper.java | 8 +-
.../translation/wrappers/ImpulseInputFormat.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 30 +-
.../streaming/ExecutableStageDoFnOperator.java | 69 +-
.../wrappers/streaming/FlinkKeyUtils.java | 55 +-
.../streaming/KeyedPushedBackElementsHandler.java | 38 +-
.../streaming/KvToByteBufferKeySelector.java | 6 +-
.../NonKeyedPushedBackElementsHandler.java | 2 +-
.../streaming/SingletonKeyedWorkItemCoder.java | 2 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 5 +-
.../wrappers/streaming/WindowDoFnOperator.java | 3 +-
.../wrappers/streaming/io/TestStreamSource.java | 11 +-
.../streaming/io/UnboundedSocketSource.java | 261 --
.../streaming/io/UnboundedSourceWrapper.java | 51 +-
.../streaming/stableinput/BufferedElements.java | 4 +-
.../stableinput/KeyedBufferingElementsHandler.java | 45 +-
.../NonKeyedBufferingElementsHandler.java | 2 +-
.../streaming/state/FlinkStateInternals.java | 126 +-
.../state/KeyGroupCheckpointedOperator.java | 34 -
.../streaming/state/KeyGroupRestoringOperator.java | 31 -
.../runners/flink/FlinkJobServerDriverTest.java | 2 +-
.../FlinkPipelineExecutionEnvironmentTest.java | 49 +-
.../beam/runners/flink/FlinkRunnerResultTest.java | 50 +
.../beam/runners/flink/FlinkSavepointTest.java | 111 +-
.../FlinkStreamingPipelineTranslatorTest.java | 123 +
.../beam/runners/flink/FlinkTestPipeline.java | 4 +-
.../beam/runners/flink/PortableExecutionTest.java | 6 +-
.../runners/flink/PortableStateExecutionTest.java | 4 +-
.../runners/flink/PortableTimersExecutionTest.java | 4 +-
.../beam/runners/flink/ReadSourcePortableTest.java | 6 +-
.../runners/flink/ReadSourceStreamingTest.java | 2 +-
.../apache/beam/runners/flink/ReadSourceTest.java | 2 +-
.../flink/metrics/FlinkMetricContainerTest.java | 2 +-
.../flink/streaming/GroupByNullKeyTest.java | 2 +-
.../flink/streaming/TopWikipediaSessionsTest.java | 2 +-
.../FlinkDefaultExecutableStageContextTest.java | 2 +-
.../FlinkExecutableStageFunctionTest.java | 4 +-
.../functions/ImpulseSourceFunctionTest.java | 83 +-
...tingFlinkExecutableStageContextFactoryTest.java | 2 +-
.../wrappers/streaming/DoFnOperatorTest.java | 10 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 63 +-
.../wrappers/streaming/StreamRecordStripper.java | 4 +-
.../wrappers/streaming/WindowDoFnOperatorTest.java | 2 +-
.../streaming/io/UnboundedSourceWrapperTest.java | 82 +-
.../stableinput/BufferedElementsTest.java | 2 +-
runners/gearpump/build.gradle | 40 +-
.../runners/gearpump/GearpumpPipelineOptions.java | 2 +-
.../runners/gearpump/GearpumpRunnerRegistrar.java | 2 +-
.../translators/CreateStreamingGearpumpView.java | 2 +-
.../translators/FlattenPCollectionsTranslator.java | 2 +-
.../translators/GearpumpPipelineTranslator.java | 2 +-
.../gearpump/translators/GroupByKeyTranslator.java | 4 +-
.../gearpump/translators/TranslationContext.java | 4 +-
.../translators/WindowAssignTranslator.java | 2 +-
.../translators/functions/DoFnFunction.java | 2 +-
.../translators/utils/TranslatorUtils.java | 2 +-
.../gearpump/GearpumpRunnerRegistrarTest.java | 2 +-
.../beam/runners/gearpump/PipelineOptionsTest.java | 2 +-
.../translators/GroupByKeyTranslatorTest.java | 4 +-
.../translators/WindowAssignTranslatorTest.java | 2 +-
.../translators/io/GearpumpSourceTest.java | 2 +-
.../gearpump/translators/io/ValueSoureTest.java | 4 +-
.../translators/utils/TranslatorUtilsTest.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 166 +-
.../examples-streaming/build.gradle | 24 +-
.../examples/build.gradle | 38 +-
.../dataflow/BatchStatefulParDoOverrides.java | 2 +-
.../beam/runners/dataflow/BatchViewOverrides.java | 32 +-
.../beam/runners/dataflow/DataflowClient.java | 2 +-
.../beam/runners/dataflow/DataflowMetrics.java | 6 +-
.../dataflow/DataflowPTransformMatchers.java | 2 +-
.../beam/runners/dataflow/DataflowPipelineJob.java | 239 +-
.../dataflow/DataflowPipelineRegistrar.java | 2 +-
.../dataflow/DataflowPipelineTranslator.java | 16 +-
.../beam/runners/dataflow/DataflowRunner.java | 46 +-
.../beam/runners/dataflow/DataflowRunnerInfo.java | 4 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 6 +-
.../beam/runners/dataflow/TestDataflowRunner.java | 8 +-
.../runners/dataflow/internal/CustomSources.java | 4 +-
.../beam/runners/dataflow/internal/IsmFormat.java | 12 +-
.../options/DataflowWorkerLoggingOptions.java | 2 +-
.../beam/runners/dataflow/util/CloudKnownType.java | 2 +-
.../beam/runners/dataflow/util/CloudObject.java | 2 +-
.../dataflow/util/CloudObjectTranslators.java | 4 +-
.../beam/runners/dataflow/util/CloudObjects.java | 8 +-
.../runners/dataflow/util/DataflowTemplateJob.java | 2 +-
.../runners/dataflow/util/DataflowTransport.java | 2 +-
...DefaultCoderCloudObjectTranslatorRegistrar.java | 12 +-
.../beam/runners/dataflow/util/GcsStager.java | 6 +-
.../beam/runners/dataflow/util/MonitoringUtil.java | 6 +-
.../runners/dataflow/util/OutputReference.java | 2 +-
.../beam/runners/dataflow/util/PackageUtil.java | 18 +-
.../runners/dataflow/util/RandomAccessData.java | 10 +-
.../util/SchemaCoderCloudObjectTranslator.java | 6 +-
.../SerializableCoderCloudObjectTranslator.java | 2 +-
.../beam/runners/dataflow/util/TimeUtil.java | 2 +-
.../dataflow/BatchStatefulParDoOverridesTest.java | 2 +-
.../runners/dataflow/BatchViewOverridesTest.java | 4 +-
.../beam/runners/dataflow/DataflowMetricsTest.java | 6 +-
.../runners/dataflow/DataflowPipelineJobTest.java | 46 +-
.../dataflow/DataflowPipelineRegistrarTest.java | 4 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 8 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 36 +-
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../runners/dataflow/TestDataflowRunnerTest.java | 6 +-
.../options/DataflowPipelineOptionsTest.java | 2 +-
.../options/DataflowWorkerLoggingOptionsTest.java | 2 +-
.../runners/dataflow/util/CloudObjectsTest.java | 4 +-
.../runners/dataflow/util/PackageUtilTest.java | 12 +-
.../dataflow/util/RandomAccessDataTest.java | 2 +-
.../beam/runners/dataflow/util/StructsTest.java | 4 +-
.../google-cloud-dataflow-java/worker/build.gradle | 38 +-
.../worker/legacy-worker/build.gradle | 92 +-
.../worker/ApplianceShuffleEntryReader.java | 2 +-
.../worker/AssignWindowsParDoFnFactory.java | 4 +-
.../runners/dataflow/worker/AvroByteReader.java | 2 +-
.../dataflow/worker/AvroByteReaderFactory.java | 2 +-
.../beam/runners/dataflow/worker/AvroByteSink.java | 2 +-
.../dataflow/worker/AvroByteSinkFactory.java | 2 +-
.../dataflow/worker/BatchDataflowWorker.java | 21 +-
.../dataflow/worker/BatchModeExecutionContext.java | 10 +-
.../worker/BeamFnMapTaskExecutorFactory.java | 30 +-
.../runners/dataflow/worker/ByteStringCoder.java | 4 +-
.../worker/ChunkingShuffleBatchReader.java | 2 +-
.../dataflow/worker/CombineValuesFnFactory.java | 2 +-
.../beam/runners/dataflow/worker/ConcatReader.java | 6 +-
.../worker/ContextActivationObserverRegistry.java | 6 +-
.../dataflow/worker/CounterShortIdCache.java | 6 +-
.../CreateIsmShardKeyAndSortKeyDoFnFactory.java | 4 +-
.../runners/dataflow/worker/DataflowApiUtils.java | 4 +-
.../worker/DataflowBatchWorkerHarness.java | 9 +-
.../worker/DataflowElementExecutionTracker.java | 10 +-
.../dataflow/worker/DataflowExecutionContext.java | 6 +-
.../dataflow/worker/DataflowExecutionStateKey.java | 4 +-
.../worker/DataflowExecutionStateRegistry.java | 4 +-
.../worker/DataflowMapTaskExecutorFactory.java | 2 +-
.../dataflow/worker/DataflowOperationContext.java | 6 +-
.../dataflow/worker/DataflowOutputCounter.java | 2 +-
.../dataflow/worker/DataflowProcessFnRunner.java | 4 +-
.../dataflow/worker/DataflowRunnerHarness.java | 4 +-
.../worker/DataflowSideInputReadCounter.java | 2 +-
.../dataflow/worker/DataflowSystemMetrics.java | 5 +-
.../worker/DataflowWorkProgressUpdater.java | 50 +-
.../dataflow/worker/DataflowWorkUnitClient.java | 8 +-
.../worker/DataflowWorkerHarnessHelper.java | 2 +-
.../dataflow/worker/DefaultParDoFnFactory.java | 2 +-
.../runners/dataflow/worker/ExperimentContext.java | 4 +-
...FetchAndFilterStreamingSideInputsOperation.java | 2 +-
.../beam/runners/dataflow/worker/Filepatterns.java | 2 +-
.../dataflow/worker/FnApiWindowMappingFn.java | 24 +-
.../worker/GroupAlsoByWindowParDoFnFactory.java | 6 +-
.../dataflow/worker/GroupAlsoByWindowsParDoFn.java | 2 +-
.../dataflow/worker/GroupingShuffleReader.java | 4 +-
.../worker/GroupingShuffleReaderFactory.java | 4 +-
...ingShuffleReaderWithFaultyBytesReadCounter.java | 2 +-
.../runners/dataflow/worker/InMemoryReader.java | 8 +-
.../dataflow/worker/InMemoryReaderFactory.java | 2 +-
.../worker/IntrinsicMapTaskExecutorFactory.java | 10 +-
.../beam/runners/dataflow/worker/IsmReader.java | 2 +-
.../runners/dataflow/worker/IsmReaderFactory.java | 6 +-
.../runners/dataflow/worker/IsmReaderImpl.java | 30 +-
.../dataflow/worker/IsmSideInputReader.java | 40 +-
.../beam/runners/dataflow/worker/IsmSink.java | 12 +-
.../runners/dataflow/worker/IsmSinkFactory.java | 6 +-
.../worker/LazilyInitializedSideInputReader.java | 4 +-
.../worker/MetricTrackingWindmillServerStub.java | 4 +-
.../dataflow/worker/MetricsContainerRegistry.java | 2 +-
...nmentContextActivationObserverRegistration.java | 2 +-
.../beam/runners/dataflow/worker/OrderedCode.java | 4 +-
.../worker/PairWithConstantKeyDoFnFactory.java | 2 +-
.../dataflow/worker/PartialGroupByKeyParDoFns.java | 6 +-
.../dataflow/worker/PartitioningShuffleReader.java | 2 +-
.../worker/PartitioningShuffleReaderFactory.java | 4 +-
.../beam/runners/dataflow/worker/PubsubReader.java | 4 +-
.../beam/runners/dataflow/worker/PubsubSink.java | 4 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 12 +-
.../runners/dataflow/worker/ReaderRegistry.java | 8 +-
.../ReifyTimestampAndWindowsParDoFnFactory.java | 2 +-
...HarnessCoderCloudObjectTranslatorRegistrar.java | 2 +-
.../dataflow/worker/SdkHarnessRegistries.java | 19 +-
.../dataflow/worker/SdkHarnessRegistry.java | 3 +
.../beam/runners/dataflow/worker/ShuffleSink.java | 4 +-
.../dataflow/worker/ShuffleSinkFactory.java | 4 +-
.../runners/dataflow/worker/SimpleParDoFn.java | 8 +-
.../beam/runners/dataflow/worker/SinkRegistry.java | 6 +-
.../dataflow/worker/SizeReportingSinkWrapper.java | 2 +-
.../worker/SourceOperationExecutorFactory.java | 2 +-
.../beam/runners/dataflow/worker/StateFetcher.java | 14 +-
.../dataflow/worker/StreamingDataflowWorker.java | 94 +-
.../worker/StreamingGroupAlsoByWindowsDoFns.java | 2 +-
.../StreamingKeyedWorkItemSideInputDoFnRunner.java | 6 +-
.../worker/StreamingModeExecutionContext.java | 21 +-
.../StreamingPCollectionViewWriterDoFnFactory.java | 4 +-
.../StreamingPCollectionViewWriterParDoFn.java | 4 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 10 +-
.../worker/StreamingStepMetricsContainer.java | 20 +-
.../worker/ToIsmRecordForMultimapDoFnFactory.java | 4 +-
.../dataflow/worker/UngroupedShuffleReader.java | 2 +-
.../worker/UngroupedShuffleReaderFactory.java | 2 +-
.../dataflow/worker/UngroupedWindmillReader.java | 4 +-
.../dataflow/worker/UserParDoFnFactory.java | 6 +-
.../runners/dataflow/worker/ValuesDoFnFactory.java | 2 +-
.../beam/runners/dataflow/worker/Weighers.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItem.java | 8 +-
.../dataflow/worker/WindmillNamespacePrefix.java | 2 +-
.../worker/WindmillReaderIteratorBase.java | 2 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 4 +-
.../dataflow/worker/WindmillStateCache.java | 85 +-
.../dataflow/worker/WindmillStateInternals.java | 14 +-
.../dataflow/worker/WindmillStateReader.java | 27 +-
.../runners/dataflow/worker/WindmillTimeUtils.java | 2 +-
.../dataflow/worker/WindmillTimerInternals.java | 14 +-
.../dataflow/worker/WindowingWindmillReader.java | 4 +-
.../dataflow/worker/WorkItemStatusClient.java | 10 +-
.../runners/dataflow/worker/WorkUnitClient.java | 2 +-
.../dataflow/worker/WorkerCustomSources.java | 12 +-
.../worker/WorkerUncaughtExceptionHandler.java | 2 +-
.../runners/dataflow/worker/apiary/Apiary.java | 2 +-
.../FixMultiOutputInfosOnParDoInstructions.java | 2 +-
.../runners/dataflow/worker/counters/Counter.java | 2 +-
.../dataflow/worker/counters/CounterFactory.java | 10 +-
.../dataflow/worker/counters/CounterName.java | 6 +-
.../dataflow/worker/counters/CounterSet.java | 4 +-
.../counters/DataflowCounterUpdateExtractor.java | 2 +-
.../dataflow/worker/fn/BeamFnControlService.java | 2 +-
.../worker/fn/control/BeamFnMapTaskExecutor.java | 12 +-
.../control/DataflowSideInputHandlerFactory.java | 4 +-
...piMonitoringInfoToCounterUpdateTransformer.java | 2 +-
...ecMonitoringInfoToCounterUpdateTransformer.java | 2 +-
.../control/RegisterAndProcessBundleOperation.java | 16 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 6 +-
.../fn/data/RemoteGrpcPortReadOperation.java | 15 +-
.../fn/data/RemoteGrpcPortWriteOperation.java | 25 +-
.../worker/fn/logging/BeamFnLoggingService.java | 6 +-
.../fn/stream/ServerStreamObserverFactory.java | 6 +-
.../graph/CloneAmbiguousFlattensFunction.java | 6 +-
.../graph/CreateExecutableStageNodeFunction.java | 28 +-
.../graph/CreateRegisterFnOperationFunction.java | 27 +-
.../graph/DeduceFlattenLocationsFunction.java | 10 +-
.../worker/graph/DeduceNodeLocationsFunction.java | 4 +-
.../beam/runners/dataflow/worker/graph/Edges.java | 2 +-
...nsertFetchAndFilterStreamingSideInputNodes.java | 12 +-
.../worker/graph/LengthPrefixUnknownCoders.java | 11 +-
.../worker/graph/MapTaskToNetworkFunction.java | 6 +-
.../runners/dataflow/worker/graph/Networks.java | 16 +-
.../beam/runners/dataflow/worker/graph/Nodes.java | 19 +-
.../worker/graph/RegisterNodeFunction.java | 34 +-
.../graph/RemoveFlattenInstructionsFunction.java | 8 +-
.../graph/ReplacePgbkWithPrecombineFunction.java | 2 +-
.../logging/DataflowWorkerLoggingHandler.java | 6 +-
.../logging/DataflowWorkerLoggingInitializer.java | 6 +-
.../JulHandlerPrintStreamAdapterFactory.java | 2 +-
.../dataflow/worker/profiler/ScopedProfiler.java | 2 +-
.../dataflow/worker/status/BaseStatusServlet.java | 2 +-
.../dataflow/worker/status/DebugCapture.java | 2 +-
.../dataflow/worker/status/HealthzServlet.java | 17 +-
.../dataflow/worker/status/HeapzServlet.java | 2 +-
.../dataflow/worker/status/ThreadzServlet.java | 2 +-
.../dataflow/worker/status/WorkerStatusPages.java | 12 +-
.../util/BatchGroupAlsoByWindowAndCombineFn.java | 6 +-
.../util/BatchGroupAlsoByWindowViaIteratorsFn.java | 10 +-
.../worker/util/BatchGroupAlsoByWindowsDoFns.java | 2 +-
.../dataflow/worker/util/MemoryMonitor.java | 14 +-
.../dataflow/worker/util/ScalableBloomFilter.java | 14 +-
.../dataflow/worker/util/TimerOrElement.java | 4 +-
.../dataflow/worker/util/ValueInEmptyWindows.java | 2 +-
.../worker/util/common/ForwardingReiterator.java | 2 +-
.../common/worker/BatchingShuffleEntryReader.java | 4 +-
.../common/worker/ByteArrayShufflePosition.java | 6 +-
.../common/worker/CachingShuffleBatchReader.java | 12 +-
.../util/common/worker/FlattenOperation.java | 2 +-
.../worker/GroupingShuffleEntryIterator.java | 4 +-
.../common/worker/GroupingShuffleRangeTracker.java | 4 +-
.../worker/util/common/worker/GroupingTables.java | 2 +-
.../worker/util/common/worker/MapTaskExecutor.java | 4 +-
.../worker/util/common/worker/ParDoOperation.java | 2 +-
.../common/worker/ProgressTrackingReiterator.java | 2 +-
.../worker/util/common/worker/ReadOperation.java | 6 +-
.../util/common/worker/ShuffleReadCounter.java | 2 +-
.../worker/util/common/worker/WorkExecutor.java | 2 +-
.../util/common/worker/WorkProgressUpdater.java | 6 +-
.../worker/util/common/worker/WriteOperation.java | 2 +-
.../worker/windmill/DirectStreamObserver.java | 4 +-
.../windmill/ForwardingClientResponseObserver.java | 6 +-
.../worker/windmill/GrpcWindmillServer.java | 46 +-
.../worker/windmill/StreamObserverFactory.java | 4 +-
.../worker/windmill/WindmillServerBase.java | 2 +-
.../worker/windmill/WindmillServerStub.java | 2 +-
.../dataflow/harness/test/TestExecutors.java | 2 +-
.../runners/dataflow/harness/test/TestStreams.java | 4 +-
.../dataflow/worker/BatchDataflowWorkerTest.java | 4 +-
.../worker/CombineValuesFnFactoryTest.java | 6 +-
...CreateIsmShardKeyAndSortKeyDoFnFactoryTest.java | 4 +-
.../DataflowElementExecutionTrackerTest.java | 2 +-
.../worker/DataflowExecutionStateTrackerTest.java | 2 +-
.../runners/dataflow/worker/DataflowMatchers.java | 2 +-
.../worker/DataflowOperationContextTest.java | 2 +-
.../worker/DataflowWorkProgressUpdaterTest.java | 61 +
.../worker/DataflowWorkUnitClientTest.java | 6 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 2 +-
.../dataflow/worker/DefaultParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/ExperimentContextTest.java | 2 +-
.../dataflow/worker/FakeWindmillServer.java | 4 +-
.../dataflow/worker/FnApiWindowMappingFnTest.java | 2 +-
.../GroupAlsoByWindowParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/GroupingShuffleReaderTest.java | 2 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 10 +-
.../worker/IntrinsicMapTaskExecutorTest.java | 2 +-
.../runners/dataflow/worker/IsmFormatTest.java | 2 +-
.../dataflow/worker/IsmReaderFactoryTest.java | 8 +-
.../runners/dataflow/worker/IsmReaderTest.java | 20 +-
.../dataflow/worker/IsmSideInputReaderTest.java | 32 +-
.../beam/runners/dataflow/worker/IsmSinkTest.java | 2 +-
.../LazilyInitializedSideInputReaderTest.java | 2 +-
.../dataflow/worker/LogRecordMatcherTest.java | 2 +-
.../beam/runners/dataflow/worker/LogSaverTest.java | 2 +-
.../runners/dataflow/worker/OrderedCodeTest.java | 4 +-
.../worker/PairWithConstantKeyDoFnFactoryTest.java | 2 +-
.../worker/PartialGroupByKeyParDoFnsTest.java | 6 +-
.../worker/PartitioningShuffleReaderTest.java | 2 +-
.../runners/dataflow/worker/PubsubReaderTest.java | 2 +-
.../runners/dataflow/worker/PubsubSinkTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 4 +-
...ReifyTimestampAndWindowsParDoFnFactoryTest.java | 4 +-
...essCoderCloudObjectTranslatorRegistrarTest.java | 4 +-
.../runners/dataflow/worker/ShuffleSinkTest.java | 2 +-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 6 +-
.../runners/dataflow/worker/StateFetcherTest.java | 8 +-
.../worker/StreamingDataflowWorkerTest.java | 29 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
...eamingKeyedWorkItemSideInputDoFnRunnerTest.java | 4 +-
.../worker/StreamingModeExecutionContextTest.java | 4 +-
...eamingPCollectionViewWriterDoFnFactoryTest.java | 2 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 4 +-
.../worker/StreamingSideInputFetcherTest.java | 6 +-
.../runners/dataflow/worker/TestShuffleReader.java | 2 +-
.../ToIsmRecordForMultimapDoFnFactoryTest.java | 4 +-
.../worker/UngroupedShuffleReaderTest.java | 2 +-
.../dataflow/worker/UserParDoFnFactoryTest.java | 2 +-
.../dataflow/worker/ValuesDoFnFactoryTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillReaderIteratorBaseTest.java | 2 +-
.../dataflow/worker/WindmillStateCacheTest.java | 141 +-
.../worker/WindmillStateInternalsTest.java | 49 +-
.../dataflow/worker/WindmillStateReaderTest.java | 4 +-
.../worker/WindmillTimerInternalsTest.java | 2 +-
.../dataflow/worker/WorkItemStatusClientTest.java | 2 +-
.../WorkerCustomSourcesSplitOnlySourceTest.java | 2 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 12 +-
.../worker/WorkerPipelineOptionsFactoryTest.java | 2 +-
...FixMultiOutputInfosOnParDoInstructionsTest.java | 2 +-
.../worker/counters/CounterFactoryTest.java | 2 +-
.../dataflow/worker/counters/CounterTest.java | 34 +-
.../worker/fn/BeamFnControlServiceTest.java | 43 +-
.../fn/control/BeamFnMapTaskExecutorTest.java | 8 +-
.../DataflowSideInputHandlerFactoryTest.java | 4 +-
.../RegisterAndProcessBundleOperationTest.java | 8 +-
.../worker/fn/control/TimerReceiverTest.java | 34 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 50 +-
.../fn/data/RemoteGrpcPortReadOperationTest.java | 15 +-
.../fn/data/RemoteGrpcPortWriteOperationTest.java | 14 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 14 +-
.../fn/stream/ServerStreamObserverFactoryTest.java | 4 +-
.../graph/CloneAmbiguousFlattensFunctionTest.java | 13 +-
.../CreateRegisterFnOperationFunctionTest.java | 44 +-
.../graph/DeduceFlattenLocationsFunctionTest.java | 16 +-
.../graph/DeduceNodeLocationsFunctionTest.java | 18 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 18 +-
.../graph/LengthPrefixUnknownCodersTest.java | 21 +-
.../worker/graph/MapTaskToNetworkFunctionTest.java | 6 +-
.../dataflow/worker/graph/NetworksTest.java | 10 +-
.../runners/dataflow/worker/graph/NodesTest.java | 4 +-
.../RemoveFlattenInstructionsFunctionTest.java | 17 +-
.../ReplacePgbkWithPrecombineFunctionTest.java | 8 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 4 +-
.../DataflowWorkerLoggingInitializerTest.java | 2 +-
.../JulHandlerPrintStreamAdapterFactoryTest.java | 2 +-
.../worker/profiler/ScopedProfilerTest.java | 2 +-
.../dataflow/worker/status/ThreadzServletTest.java | 2 +-
.../worker/status/WorkerStatusPagesTest.java | 20 +-
.../worker/util/CounterHamcrestMatchers.java | 4 +-
.../util/DataflowCounterUpdateExtractorTest.java | 2 +-
.../worker/util/GroupAlsoByWindowProperties.java | 10 +-
.../dataflow/worker/util/ListOutputManager.java | 4 +-
.../util/common/worker/ExecutorTestUtils.java | 2 +-
.../worker/GroupingShuffleEntryIteratorTest.java | 4 +-
.../util/common/worker/MapTaskExecutorTest.java | 2 +-
.../util/common/worker/ReadOperationTest.java | 2 +-
.../util/common/worker/TestOutputReceiver.java | 2 +-
.../worker/windmill/GrpcWindmillServerTest.java | 14 +-
.../worker/windmill/build.gradle | 5 +-
.../worker/windmill/src/main/proto/windmill.proto | 11 +
runners/java-fn-execution/build.gradle | 32 +-
.../apache/beam/runners/fnexecution/FnService.java | 6 +-
.../GrpcContextHeaderAccessorProvider.java | 16 +-
.../beam/runners/fnexecution/GrpcFnServer.java | 4 +-
.../fnexecution/InProcessServerFactory.java | 8 +-
.../beam/runners/fnexecution/ServerFactory.java | 24 +-
.../BeamFileSystemArtifactRetrievalService.java | 28 +-
.../BeamFileSystemArtifactStagingService.java | 18 +-
.../control/DefaultJobBundleFactory.java | 249 +-
.../fnexecution/control/FnApiControlClient.java | 6 +-
.../control/FnApiControlClientPoolService.java | 6 +-
.../fnexecution/control/MapControlClientPool.java | 2 +-
.../control/ProcessBundleDescriptors.java | 85 +-
.../fnexecution/control/SdkHarnessClient.java | 37 +-
.../SingleEnvironmentInstanceJobBundleFactory.java | 15 +-
.../fnexecution/control/TimerReceiverFactory.java | 90 +
.../runners/fnexecution/data/GrpcDataService.java | 12 +-
.../fnexecution/data/RemoteInputDestination.java | 6 +-
.../fnexecution/environment/DockerCommand.java | 4 +-
.../environment/DockerEnvironmentFactory.java | 8 +-
.../environment/EmbeddedEnvironmentFactory.java | 2 +-
.../environment/ExternalEnvironmentFactory.java | 2 +-
.../environment/ProcessEnvironmentFactory.java | 4 +-
.../fnexecution/environment/ProcessManager.java | 6 +-
.../jobsubmission/InMemoryJobService.java | 33 +-
.../fnexecution/jobsubmission/JobInvocation.java | 56 +-
.../fnexecution/jobsubmission/JobInvoker.java | 8 +-
.../fnexecution/jobsubmission/JobPreparation.java | 2 +-
.../fnexecution/jobsubmission/JobServerDriver.java | 2 +-
.../fnexecution/logging/GrpcLoggingService.java | 4 +-
.../runners/fnexecution/provisioning/JobInfo.java | 2 +-
.../provisioning/StaticGrpcProvisionService.java | 2 +-
.../splittabledofn/SDFFeederViaStateAndTimers.java | 10 +-
.../fnexecution/state/GrpcStateService.java | 6 +-
.../state/InMemoryBagUserStateFactory.java | 118 +
.../fnexecution/state/StateRequestHandlers.java | 16 +-
.../translation/BatchSideInputHandlerFactory.java | 8 +-
.../translation/PipelineTranslatorUtils.java | 68 +-
.../wire/LengthPrefixUnknownCoders.java | 11 +-
.../beam/runners/fnexecution/wire/WireCoders.java | 2 +-
.../runners/fnexecution/EmbeddedSdkHarness.java | 2 +-
.../GrpcContextHeaderAccessorProviderTest.java | 22 +-
.../runners/fnexecution/ServerFactoryTest.java | 16 +-
.../BeamFileSystemArtifactServicesTest.java | 18 +-
.../control/DefaultJobBundleFactoryTest.java | 131 +-
.../control/FnApiControlClientPoolServiceTest.java | 4 +-
.../control/FnApiControlClientTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 206 +-
.../fnexecution/control/SdkHarnessClientTest.java | 67 +-
...gleEnvironmentInstanceJobBundleFactoryTest.java | 2 +-
.../fnexecution/data/GrpcDataServiceTest.java | 24 +-
.../data/RemoteInputDestinationTest.java | 14 +-
.../fnexecution/environment/DockerCommandTest.java | 4 +-
.../jobsubmission/InMemoryJobServiceTest.java | 74 +-
.../jobsubmission/JobInvocationTest.java | 192 ++
.../logging/GrpcLoggingServiceTest.java | 6 +-
.../StaticGrpcProvisionServiceTest.java | 10 +-
.../fnexecution/state/GrpcStateServiceTest.java | 4 +-
.../translation/PipelineTranslatorUtilsTest.java | 4 +-
.../wire/LengthPrefixUnknownCodersTest.java | 2 +-
runners/jet-experimental/build.gradle | 106 +
.../org/apache/beam/runners/jet/DAGBuilder.java | 226 ++
.../runners/jet/FailedRunningPipelineResults.java | 91 +
.../apache/beam/runners/jet/JetGraphVisitor.java | 100 +
.../beam/runners/jet/JetPipelineOptions.java | 68 +
.../apache/beam/runners/jet/JetPipelineResult.java | 132 +
.../org/apache/beam/runners/jet/JetRunner.java | 207 ++
.../beam/runners/jet/JetRunnerRegistrar.java | 54 +
.../beam/runners/jet/JetTransformTranslator.java | 33 +
.../beam/runners/jet/JetTransformTranslators.java | 431 +++
.../beam/runners/jet/JetTranslationContext.java | 52 +
.../java/org/apache/beam/runners/jet/Utils.java | 292 ++
.../beam/runners/jet/metrics/AbstractMetric.java | 35 +
.../beam/runners/jet/metrics/CounterImpl.java | 56 +
.../beam/runners/jet/metrics/DistributionImpl.java | 51 +
.../apache/beam/runners/jet/metrics/GaugeImpl.java | 42 +
.../beam/runners/jet/metrics/JetMetricResults.java | 212 ++
.../runners/jet/metrics/JetMetricsContainer.java | 138 +
.../beam/runners/jet/metrics/package-info.java | 20 +
.../org/apache/beam/runners/jet/package-info.java | 20 +
.../runners/jet/processors/AbstractParDoP.java | 521 ++++
.../beam/runners/jet/processors/AssignWindowP.java | 130 +
.../runners/jet/processors/BoundedSourceP.java | 215 ++
.../beam/runners/jet/processors/FlattenP.java | 87 +
.../beam/runners/jet/processors/ImpulseP.java | 107 +
.../apache/beam/runners/jet/processors/ParDoP.java | 191 ++
.../runners/jet/processors/StatefulParDoP.java | 302 ++
.../runners/jet/processors/UnboundedSourceP.java | 281 ++
.../apache/beam/runners/jet/processors/ViewP.java | 130 +
.../beam/runners/jet/processors/WindowGroupP.java | 339 +++
.../beam/runners/jet/processors/package-info.java | 20 +
.../beam/runners/jet/JetTestRunnerRegistrar.java | 54 +
.../org/apache/beam/runners/jet/TestJetRunner.java | 147 +
.../org/apache/beam/runners/jet/TestStreamP.java | 101 +
runners/local-java/build.gradle | 15 +-
runners/reference/java/build.gradle | 24 +-
.../beam/runners/reference/CloseableResource.java | 4 +-
.../runners/reference/ExternalWorkerService.java | 4 +-
.../reference/JobServicePipelineResult.java | 4 +-
.../beam/runners/reference/PortableRunner.java | 12 +-
.../runners/reference/PortableRunnerRegistrar.java | 4 +-
.../runners/reference/testing/TestJobService.java | 2 +-
.../testing/TestPortablePipelineOptions.java | 2 +-
.../runners/reference/CloseableResourceTest.java | 6 +-
.../beam/runners/reference/PortableRunnerTest.java | 6 +-
runners/reference/job-server/build.gradle | 54 -
runners/samza/build.gradle | 64 +-
runners/samza/job-server/build.gradle | 8 +-
.../beam/runners/samza/SamzaExecutionContext.java | 2 +-
.../beam/runners/samza/SamzaJobServerDriver.java | 80 +-
.../samza/SamzaPipelineLifeCycleListener.java | 2 +-
.../samza/SamzaPipelineOptionsValidator.java | 2 +-
.../beam/runners/samza/SamzaPipelineResult.java | 16 +-
.../beam/runners/samza/SamzaPipelineRunner.java | 4 +-
.../samza/SamzaPortablePipelineOptions.java | 36 +
.../org/apache/beam/runners/samza/SamzaRunner.java | 15 +-
.../beam/runners/samza/SamzaRunnerRegistrar.java | 2 +-
.../runners/samza/adapter/BoundedSourceSystem.java | 2 +-
.../samza/adapter/UnboundedSourceSystem.java | 4 +-
.../apache/beam/runners/samza/runtime/DoFnOp.java | 2 +-
.../runtime/DoFnRunnerWithKeyedInternals.java | 2 +-
.../beam/runners/samza/runtime/KeyedInternals.java | 2 +-
.../runners/samza/runtime/SamzaAssignContext.java | 2 +-
.../runners/samza/runtime/SamzaDoFnRunners.java | 2 +-
.../samza/runtime/SamzaStoreStateInternals.java | 6 +-
.../runners/samza/translation/ConfigBuilder.java | 6 +-
.../runners/samza/translation/ConfigContext.java | 2 +-
.../translation/FlattenPCollectionsTranslator.java | 4 +-
.../samza/translation/GroupByKeyTranslator.java | 2 +-
.../translation/ParDoBoundMultiTranslator.java | 2 +-
.../translation/PortableTranslationContext.java | 2 +-
.../runners/samza/translation/ReadTranslator.java | 2 +-
.../samza/translation/SamzaPipelineTranslator.java | 4 +-
.../SamzaPortablePipelineTranslator.java | 2 +-
.../SamzaPublishViewTransformOverride.java | 2 +-
.../samza/translation/SamzaTransformOverrides.java | 2 +-
.../samza/translation/TranslationContext.java | 2 +-
.../samza/translation/WindowAssignTranslator.java | 2 +-
.../runners/samza/util/PipelineDotRenderer.java | 128 -
.../samza/util/PortablePipelineDotRenderer.java | 124 -
.../samza/util/SamzaPipelineTranslatorUtils.java | 4 +-
.../samza/adapter/BoundedSourceSystemTest.java | 2 +-
.../runtime/SamzaStoreStateInternalsTest.java | 6 +-
.../samza/translation/ConfigGeneratorTest.java | 4 +-
runners/spark/build.gradle | 66 +-
runners/spark/job-server/build.gradle | 18 +-
.../apache/beam/runners/spark/SparkJobInvoker.java | 4 +-
.../runners/spark/SparkNativePipelineVisitor.java | 4 +-
.../beam/runners/spark/SparkPipelineOptions.java | 34 +-
.../beam/runners/spark/SparkPipelineResult.java | 8 +-
.../beam/runners/spark/SparkPipelineRunner.java | 14 +-
.../org/apache/beam/runners/spark/SparkRunner.java | 6 +-
.../beam/runners/spark/SparkRunnerRegistrar.java | 2 +-
.../runners/spark/SparkTransformOverrides.java | 4 +-
.../apache/beam/runners/spark/TestSparkRunner.java | 4 +-
.../spark/aggregators/AggregatorsAccumulator.java | 4 +-
.../spark/aggregators/NamedAggregators.java | 4 +-
.../beam/runners/spark/coders/CoderHelpers.java | 43 +-
.../spark/coders/SparkRunnerKryoRegistrator.java | 6 +-
.../apache/beam/runners/spark/io/CreateStream.java | 4 +-
.../beam/runners/spark/io/MicrobatchSource.java | 12 +-
.../beam/runners/spark/io/SourceDStream.java | 2 +-
.../apache/beam/runners/spark/io/SourceRDD.java | 6 +-
.../runners/spark/io/SparkUnboundedSource.java | 2 +-
.../runners/spark/metrics/MetricsAccumulator.java | 4 +-
.../runners/spark/metrics/SparkBeamMetric.java | 4 +-
.../runners/spark/metrics/WithMetricsSupport.java | 16 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 21 +-
.../spark/stateful/SparkStateInternals.java | 4 +-
.../spark/stateful/SparkTimerInternals.java | 6 +-
.../runners/spark/stateful/StateSpecFunctions.java | 8 +-
.../aggregators/AggregatorsAccumulator.java | 2 +-
.../aggregators/NamedAggregators.java | 4 +-
.../metrics/MetricsAccumulator.java | 2 +-
.../metrics/SparkBeamMetric.java | 4 +-
.../metrics/WithMetricsSupport.java | 16 +-
.../translation/SparkTransformOverrides.java | 2 +-
.../translation/TranslationContext.java | 2 +-
.../translation/batch/AggregatorCombiner.java | 4 +-
.../translation/batch/DatasetSourceBatch.java | 2 +-
.../translation/batch/DoFnFunction.java | 8 +-
.../translation/batch/FlattenTranslatorBatch.java | 2 +-
.../batch/GroupByKeyTranslatorBatch.java | 2 +-
.../translation/batch/ParDoTranslatorBatch.java | 2 +-
.../translation/batch/ProcessContext.java | 2 +-
.../batch/functions/SparkSideInputReader.java | 4 +-
.../translation/helpers/WindowingHelpers.java | 2 +-
.../streaming/DatasetSourceStreaming.java | 2 +-
.../translation/utils/CachedSideInputReader.java | 2 +-
.../translation/utils/SideInputStorage.java | 4 +-
.../runners/spark/translation/BoundedDataset.java | 2 +-
.../spark/translation/EvaluationContext.java | 4 +-
.../spark/translation/GroupCombineFunctions.java | 151 +-
.../GroupNonMergingWindowsFunctions.java | 226 +-
.../spark/translation/MultiDoFnFunction.java | 8 +-
.../spark/translation/SparkAbstractCombineFn.java | 124 -
.../spark/translation/SparkAssignWindowFn.java | 2 +-
.../SparkBatchPortablePipelineTranslator.java | 220 +-
.../runners/spark/translation/SparkCombineFn.java | 833 ++++++
.../translation/SparkExecutableStageFunction.java | 203 +-
.../spark/translation/SparkGlobalCombineFn.java | 261 --
.../SparkGroupAlsoByWindowViaOutputBufferFn.java | 13 +-
.../spark/translation/SparkKeyedCombineFn.java | 278 --
.../spark/translation/SparkProcessContext.java | 4 +-
.../spark/translation/SparkTranslationContext.java | 24 +-
.../spark/translation/TransformTranslator.java | 84 +-
.../spark/translation/TranslationUtils.java | 57 +-
.../translation/ValueAndCoderLazySerializable.java | 4 +-
.../SparkRunnerStreamingContextFactory.java | 2 +-
.../streaming/StreamingTransformTranslator.java | 17 +-
.../streaming/WatermarkSyncedDStream.java | 6 +-
.../apache/beam/runners/spark/util/ByteArray.java | 2 +-
.../runners/spark/util/CachedSideInputReader.java | 2 +-
.../runners/spark/util/GlobalWatermarkHolder.java | 12 +-
.../beam/runners/spark/util/SideInputStorage.java | 4 +-
.../beam/runners/spark/util/SparkCompat.java | 65 +
.../runners/spark/util/SparkSideInputReader.java | 4 +-
.../runners/spark/ProvidedSparkContextTest.java | 4 +-
.../runners/spark/SparkPortableExecutionTest.java | 132 +-
.../runners/spark/SparkRunnerDebuggerTest.java | 4 +-
.../runners/spark/SparkRunnerRegistrarTest.java | 4 +-
.../spark/TestSparkPipelineOptionsRegistrar.java | 2 +-
.../aggregators/metrics/sink/InMemoryMetrics.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 4 +-
.../beam/runners/spark/io/AvroPipelineTest.java | 4 +-
.../beam/runners/spark/io/NumShardsTest.java | 6 +-
.../spark/io/ReaderToIteratorAdapterTest.java | 2 +-
.../aggregators/metrics/sink/InMemoryMetrics.java | 2 +-
.../metrics/sink/SparkMetricsSinkTest.java | 4 +-
.../translation/batch/GroupByKeyTest.java | 2 +-
.../GroupNonMergingWindowsFunctionsTest.java | 124 +-
.../spark/translation/SparkCombineFnTest.java | 298 ++
.../SparkExecutableStageFunctionTest.java | 5 +-
.../ResumeFromCheckpointStreamingTest.java | 10 +-
.../streaming/SparkCoGroupByKeyStreamingTest.java | 2 +-
.../streaming/StreamingSourceMetricsTest.java | 29 +-
sdks/go/README.md | 2 +-
sdks/go/cmd/starcgen/starcgen.go | 6 +-
sdks/go/container/build.gradle | 6 +-
sdks/go/examples/build.gradle | 6 +-
sdks/go/pkg/beam/combine.go | 28 +-
sdks/go/pkg/beam/combine_test.go | 57 +
sdks/go/pkg/beam/core/funcx/fn.go | 10 +-
sdks/go/pkg/beam/core/funcx/signature.go | 15 +-
sdks/go/pkg/beam/core/graph/bind.go | 64 +-
sdks/go/pkg/beam/core/graph/coder/coder.go | 13 +-
sdks/go/pkg/beam/core/graph/coder/registry.go | 7 +-
sdks/go/pkg/beam/core/graph/coder/varint.go | 2 +-
sdks/go/pkg/beam/core/graph/edge.go | 46 +-
sdks/go/pkg/beam/core/graph/fn.go | 16 +-
sdks/go/pkg/beam/core/graph/graph.go | 7 +-
sdks/go/pkg/beam/core/runtime/coderx/float.go | 5 +-
sdks/go/pkg/beam/core/runtime/coderx/varint.go | 9 +-
sdks/go/pkg/beam/core/runtime/exec/coder.go | 3 +-
sdks/go/pkg/beam/core/runtime/exec/cogbk.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/combine.go | 23 +-
sdks/go/pkg/beam/core/runtime/exec/combine_test.go | 5 +-
sdks/go/pkg/beam/core/runtime/exec/data.go | 28 +-
sdks/go/pkg/beam/core/runtime/exec/datasink.go | 3 +-
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 326 +-
.../pkg/beam/core/runtime/exec/datasource_test.go | 248 ++
sdks/go/pkg/beam/core/runtime/exec/fn.go | 15 +-
.../pkg/beam/core/runtime/exec/fullvalue_test.go | 10 +
sdks/go/pkg/beam/core/runtime/exec/input.go | 3 +-
sdks/go/pkg/beam/core/runtime/exec/pardo.go | 13 +-
sdks/go/pkg/beam/core/runtime/exec/plan.go | 32 +-
sdks/go/pkg/beam/core/runtime/exec/sideinput.go | 21 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 59 +-
sdks/go/pkg/beam/core/runtime/exec/unit_test.go | 56 +-
sdks/go/pkg/beam/core/runtime/exec/util.go | 5 +-
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 83 +-
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 27 +-
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 149 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 21 +-
.../pkg/beam/core/runtime/graphx/translate_test.go | 130 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 36 +-
.../pkg/beam/core/runtime/harness/datamgr_test.go | 8 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 40 +-
sdks/go/pkg/beam/core/runtime/harness/logging.go | 3 +-
sdks/go/pkg/beam/core/runtime/harness/session.go | 13 +-
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 84 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 3 +-
sdks/go/pkg/beam/core/runtime/symbols.go | 4 +-
sdks/go/pkg/beam/core/typex/fulltype.go | 12 +-
sdks/go/pkg/beam/core/util/dot/dot.go | 20 +-
sdks/go/pkg/beam/core/util/hooks/hooks.go | 12 +-
sdks/go/pkg/beam/core/util/ioutilx/read.go | 3 +-
sdks/go/pkg/beam/core/util/protox/any.go | 9 +-
sdks/go/pkg/beam/core/util/protox/base64.go | 4 +-
sdks/go/pkg/beam/core/util/reflectx/call.go | 4 +-
sdks/go/pkg/beam/core/util/reflectx/json.go | 5 +-
sdks/go/pkg/beam/core/util/symtab/symtab.go | 15 +-
sdks/go/pkg/beam/internal/errors/errors.go | 17 +-
sdks/go/pkg/beam/internal/errors/errors_test.go | 4 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 801 ++---
.../jobmanagement_v1/beam_expansion_api.pb.go | 45 +-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 1295 ++++----
sdks/go/pkg/beam/options/jobopts/options.go | 4 +
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 34 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 16 +-
sdks/go/pkg/beam/runners/direct/buffer.go | 2 +-
sdks/go/pkg/beam/runners/direct/direct.go | 10 +
sdks/go/pkg/beam/runners/session/session.go | 4 +-
sdks/go/pkg/beam/runners/spark/spark.go | 34 +
sdks/go/pkg/beam/runners/universal/universal.go | 13 +
.../pkg/beam/runners/vet/testpipeline/functions.go | 52 +
.../beam/runners/vet/testpipeline/testpipeline.go | 84 +
.../runners/vet/testpipeline/testpipeline.shims.go | 190 ++
sdks/go/pkg/beam/runners/vet/vet.go | 598 ++++
sdks/go/pkg/beam/runners/vet/vet_test.go | 65 +
sdks/go/pkg/beam/transforms/stats/stats.shims.go | 143 +-
sdks/go/pkg/beam/x/beamx/run.go | 1 +
sdks/go/test/build.gradle | 38 +-
sdks/go/test/integration/primitives/pardo.go | 3 +-
sdks/go/test/run_integration_tests.sh | 133 +-
sdks/java/bom/build.gradle | 10 +-
.../src/main/resources/beam/checkstyle.xml | 8 +
.../src/main/resources/beam/spotbugs-filter.xml | 38 +-
.../src/main/resources/beam/suppressions.xml | 7 +-
sdks/java/container/build.gradle | 12 +-
sdks/java/core/build.gradle | 9 +-
.../parser/generated/FieldSpecifierNotation.g4 | 2 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 43 +-
.../java/org/apache/beam/sdk/PipelineRunner.java | 2 +-
.../java/org/apache/beam/sdk/coders/AvroCoder.java | 4 +-
.../apache/beam/sdk/coders/BigDecimalCoder.java | 2 +-
.../apache/beam/sdk/coders/BigIntegerCoder.java | 2 +-
.../org/apache/beam/sdk/coders/ByteArrayCoder.java | 2 +-
.../java/org/apache/beam/sdk/coders/Coder.java | 12 +-
.../org/apache/beam/sdk/coders/CoderProviders.java | 4 +-
.../org/apache/beam/sdk/coders/CoderRegistry.java | 23 +-
.../org/apache/beam/sdk/coders/DefaultCoder.java | 2 +-
.../org/apache/beam/sdk/coders/DelegateCoder.java | 4 +-
.../apache/beam/sdk/coders/IterableLikeCoder.java | 2 +-
.../apache/beam/sdk/coders/LengthPrefixCoder.java | 6 +-
.../java/org/apache/beam/sdk/coders/MapCoder.java | 2 +-
.../org/apache/beam/sdk/coders/NullableCoder.java | 4 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 4 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 6 +-
.../apache/beam/sdk/coders/SerializableCoder.java | 2 +-
.../apache/beam/sdk/coders/ShardedKeyCoder.java | 2 +-
.../org/apache/beam/sdk/coders/SnappyCoder.java | 2 +-
.../apache/beam/sdk/coders/StringUtf8Coder.java | 4 +-
.../beam/sdk/coders/StructuralByteArray.java | 2 +-
.../sdk/expansion/ExternalTransformRegistrar.java | 2 +
.../apache/beam/sdk/harness/JvmInitializer.java | 56 +
.../org/apache/beam/sdk/harness/package-info.java | 24 +
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 37 +-
.../java/org/apache/beam/sdk/io/AvroSource.java | 10 +-
.../java/org/apache/beam/sdk/io/AvroUtils.java | 6 +-
.../sdk/io/BoundedReadFromUnboundedSource.java | 2 +-
.../org/apache/beam/sdk/io/CompressedSource.java | 4 +-
.../java/org/apache/beam/sdk/io/Compression.java | 6 +-
.../beam/sdk/io/ConstantAvroDestination.java | 11 +-
.../org/apache/beam/sdk/io/CountingSource.java | 6 +-
.../apache/beam/sdk/io/DefaultFilenamePolicy.java | 8 +-
.../beam/sdk/io/DynamicAvroDestinations.java | 2 +-
.../beam/sdk/io/DynamicFileDestinations.java | 2 +-
.../java/org/apache/beam/sdk/io/FileBasedSink.java | 46 +-
.../org/apache/beam/sdk/io/FileBasedSource.java | 10 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 94 +-
.../java/org/apache/beam/sdk/io/FileSystems.java | 35 +-
.../org/apache/beam/sdk/io/GenerateSequence.java | 10 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 14 +-
.../beam/sdk/io/LocalFileSystemRegistrar.java | 2 +-
.../org/apache/beam/sdk/io/LocalResourceId.java | 6 +-
.../org/apache/beam/sdk/io/OffsetBasedSource.java | 2 +-
.../beam/sdk/io/SerializableAvroCodecFactory.java | 4 +-
.../org/apache/beam/sdk/io/ShardingFunction.java | 31 +
.../main/java/org/apache/beam/sdk/io/Source.java | 2 +-
.../java/org/apache/beam/sdk/io/TFRecordIO.java | 53 +-
.../main/java/org/apache/beam/sdk/io/TextIO.java | 18 +-
.../apache/beam/sdk/io/TextRowCountEstimator.java | 219 ++
.../main/java/org/apache/beam/sdk/io/TextSink.java | 1 -
.../java/org/apache/beam/sdk/io/TextSource.java | 4 +-
.../java/org/apache/beam/sdk/io/WriteFiles.java | 104 +-
.../org/apache/beam/sdk/io/WriteFilesResult.java | 2 +-
.../apache/beam/sdk/io/fs/ResourceIdTester.java | 68 +-
.../java/org/apache/beam/sdk/io/range/ByteKey.java | 2 +-
.../org/apache/beam/sdk/io/range/ByteKeyRange.java | 12 +-
.../beam/sdk/io/range/ByteKeyRangeTracker.java | 4 +-
.../org/apache/beam/sdk/io/range/OffsetRange.java | 2 +-
.../beam/sdk/io/range/OffsetRangeTracker.java | 4 +-
.../apache/beam/sdk/metrics/MetricFiltering.java | 2 +-
.../org/apache/beam/sdk/metrics/MetricName.java | 4 +-
.../apache/beam/sdk/metrics/MetricNameFilter.java | 2 +-
.../beam/sdk/metrics/MetricQueryResults.java | 2 +-
.../org/apache/beam/sdk/metrics/MetricsFilter.java | 2 +-
.../org/apache/beam/sdk/metrics/SourceMetrics.java | 2 +-
.../options/DefaultPipelineOptionsRegistrar.java | 2 +-
.../beam/sdk/options/ExperimentalOptions.java | 2 +-
.../options/ManualDockerEnvironmentOptions.java | 2 +-
.../apache/beam/sdk/options/PipelineOptions.java | 2 +-
.../beam/sdk/options/PipelineOptionsFactory.java | 90 +-
.../beam/sdk/options/PipelineOptionsReflector.java | 6 +-
.../beam/sdk/options/PipelineOptionsValidator.java | 12 +-
.../beam/sdk/options/PortablePipelineOptions.java | 6 +
.../beam/sdk/options/ProxyInvocationHandler.java | 22 +-
.../apache/beam/sdk/options/SdkHarnessOptions.java | 2 +-
.../org/apache/beam/sdk/options/ValueProvider.java | 5 +-
.../apache/beam/sdk/options/ValueProviders.java | 2 +-
.../apache/beam/sdk/runners/PTransformMatcher.java | 2 +-
.../beam/sdk/runners/TransformHierarchy.java | 14 +-
.../apache/beam/sdk/schemas/AutoValueSchema.java | 2 +-
.../beam/sdk/schemas/FieldAccessDescriptor.java | 74 +-
.../beam/sdk/schemas/FieldTypeDescriptors.java | 6 +-
.../sdk/schemas/FieldValueTypeInformation.java | 2 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 8 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 2 +-
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 2 +-
.../org/apache/beam/sdk/schemas/LogicalTypes.java | 2 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 18 +-
.../apache/beam/sdk/schemas/SchemaRegistry.java | 8 +-
.../schemas/SchemaUserTypeConstructorCreator.java | 2 +-
.../sdk/schemas/annotations/DefaultSchema.java | 6 +-
.../parser/FieldAccessDescriptorParser.java | 4 +-
.../beam/sdk/schemas/transforms/AddFields.java | 434 +++
.../apache/beam/sdk/schemas/transforms/Cast.java | 6 +-
.../beam/sdk/schemas/transforms/CoGroup.java | 6 +-
.../beam/sdk/schemas/transforms/Convert.java | 43 +-
.../beam/sdk/schemas/transforms/DropFields.java | 149 +
.../apache/beam/sdk/schemas/transforms/Filter.java | 162 +-
.../beam/sdk/schemas/transforms/RenameFields.java | 187 ++
.../sdk/schemas/transforms/SchemaAggregateFn.java | 2 +-
.../apache/beam/sdk/schemas/transforms/Unnest.java | 2 +-
.../beam/sdk/schemas/utils/AutoValueUtils.java | 4 +-
.../beam/sdk/schemas/utils/AvroByteBuddyUtils.java | 2 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 40 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 6 +-
.../beam/sdk/schemas/utils/ConvertHelpers.java | 6 +-
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 2 +-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 2 +-
.../beam/sdk/schemas/utils/ReflectUtils.java | 6 +-
.../beam/sdk/schemas/utils/SchemaZipFold.java | 2 +-
.../beam/sdk/schemas/utils/SelectHelpers.java | 8 +-
.../sdk/schemas/utils/StaticSchemaInference.java | 4 +-
.../java/org/apache/beam/sdk/state/StateSpecs.java | 2 +-
.../org/apache/beam/sdk/testing/Annotations.java | 4 +-
.../apache/beam/sdk/testing/CoderProperties.java | 12 +-
.../beam/sdk/testing/FileChecksumMatcher.java | 10 +-
.../beam/sdk/testing/MatcherDeserializer.java | 2 +-
.../apache/beam/sdk/testing/MatcherSerializer.java | 2 +-
.../java/org/apache/beam/sdk/testing/PAssert.java | 46 +-
.../apache/beam/sdk/testing/PaneExtractors.java | 18 +
.../beam/sdk/testing/SerializableMatchers.java | 2 +-
.../apache/beam/sdk/testing/SourceTestUtils.java | 6 +-
.../org/apache/beam/sdk/testing/StaticWindows.java | 6 +-
.../apache/beam/sdk/testing/SuccessOrFailure.java | 4 +-
.../org/apache/beam/sdk/testing/TestPipeline.java | 14 +-
.../org/apache/beam/sdk/testing/TestStream.java | 8 +-
.../java/org/apache/beam/sdk/testing/UsesKms.java | 26 +
.../apache/beam/sdk/testing/WindowFnTestUtils.java | 6 +-
.../apache/beam/sdk/testing/WindowSupplier.java | 6 +-
.../beam/sdk/transforms/ApproximateQuantiles.java | 8 +-
.../beam/sdk/transforms/ApproximateUnique.java | 17 +-
.../org/apache/beam/sdk/transforms/Combine.java | 19 +-
.../apache/beam/sdk/transforms/CombineFnBase.java | 2 +-
.../org/apache/beam/sdk/transforms/CombineFns.java | 14 +-
.../org/apache/beam/sdk/transforms/Contextful.java | 2 +-
.../org/apache/beam/sdk/transforms/Create.java | 12 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 71 +
.../beam/sdk/transforms/DoFnOutputReceivers.java | 4 +-
.../beam/sdk/transforms/DoFnSchemaInformation.java | 2 +-
.../org/apache/beam/sdk/transforms/DoFnTester.java | 8 +-
.../sdk/transforms/ExternalTransformBuilder.java | 2 +
.../beam/sdk/transforms/FlatMapElements.java | 4 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 6 +-
.../org/apache/beam/sdk/transforms/Latest.java | 43 +-
.../apache/beam/sdk/transforms/MapElements.java | 6 +-
.../java/org/apache/beam/sdk/transforms/Mean.java | 2 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 4 +-
.../apache/beam/sdk/transforms/Requirements.java | 2 +-
.../org/apache/beam/sdk/transforms/Reshuffle.java | 2 +-
.../org/apache/beam/sdk/transforms/Sample.java | 2 +-
.../org/apache/beam/sdk/transforms/ToString.java | 2 +-
.../java/org/apache/beam/sdk/transforms/Top.java | 4 +-
.../java/org/apache/beam/sdk/transforms/Wait.java | 4 +-
.../java/org/apache/beam/sdk/transforms/Watch.java | 26 +-
.../apache/beam/sdk/transforms/WithFailures.java | 2 +-
.../org/apache/beam/sdk/transforms/WithKeys.java | 2 +-
.../apache/beam/sdk/transforms/WithTimestamps.java | 2 +-
.../beam/sdk/transforms/display/DisplayData.java | 14 +-
.../beam/sdk/transforms/join/CoGbkResult.java | 8 +-
.../sdk/transforms/join/KeyedPCollectionTuple.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 4 +-
.../reflect/ByteBuddyOnTimerInvokerFactory.java | 12 +-
.../beam/sdk/transforms/reflect/DoFnSignature.java | 6 +-
.../sdk/transforms/reflect/DoFnSignatures.java | 12 +-
.../reflect/StableInvokerNamingStrategy.java | 2 +-
.../sdk/transforms/splittabledofn/Backlog.java | 90 -
.../sdk/transforms/splittabledofn/Backlogs.java | 58 -
.../splittabledofn/ByteKeyRangeTracker.java | 27 +-
.../splittabledofn/OffsetRangeTracker.java | 17 +-
.../beam/sdk/transforms/splittabledofn/Sizes.java | 57 +
.../beam/sdk/transforms/windowing/AfterAll.java | 4 +-
.../beam/sdk/transforms/windowing/AfterEach.java | 4 +-
.../beam/sdk/transforms/windowing/AfterFirst.java | 4 +-
.../transforms/windowing/AfterProcessingTime.java | 2 +-
.../windowing/AfterSynchronizedProcessingTime.java | 2 +-
.../sdk/transforms/windowing/AfterWatermark.java | 4 +-
.../sdk/transforms/windowing/OrFinallyTrigger.java | 2 +-
.../beam/sdk/transforms/windowing/PaneInfo.java | 8 +-
.../beam/sdk/transforms/windowing/Sessions.java | 6 +
.../transforms/windowing/TimestampCombiner.java | 6 +-
.../beam/sdk/transforms/windowing/Trigger.java | 2 +-
.../beam/sdk/transforms/windowing/Window.java | 4 +-
.../java/org/apache/beam/sdk/util/ApiSurface.java | 32 +-
.../org/apache/beam/sdk/util/AppliedCombineFn.java | 2 +-
.../apache/beam/sdk/util/BucketingFunction.java | 2 +-
.../util/BufferedElementCountingOutputStream.java | 2 +-
.../java/org/apache/beam/sdk/util/ClassPath.java | 36 +-
.../java/org/apache/beam/sdk/util/CoderUtils.java | 4 +-
.../apache/beam/sdk/util/ExplicitShardedFile.java | 8 +-
.../sdk/util/FilePatternMatchingShardedFile.java | 10 +-
.../org/apache/beam/sdk/util/FluentBackoff.java | 4 +-
.../org/apache/beam/sdk/util/InstanceBuilder.java | 6 +-
.../org/apache/beam/sdk/util/MovingFunction.java | 2 +-
.../java/org/apache/beam/sdk/util/NameUtils.java | 6 +-
.../apache/beam/sdk/util/NumberedShardedFile.java | 14 +-
.../java/org/apache/beam/sdk/util/ReleaseInfo.java | 2 +-
.../apache/beam/sdk/util/RowJsonDeserializer.java | 2 +-
.../apache/beam/sdk/util/RowJsonValidation.java | 2 +-
.../apache/beam/sdk/util/SerializableUtils.java | 2 +-
.../java/org/apache/beam/sdk/util/StringUtils.java | 2 +-
.../apache/beam/sdk/util/UnownedInputStream.java | 2 +-
.../apache/beam/sdk/util/UnownedOutputStream.java | 2 +-
.../org/apache/beam/sdk/util/WindowedValue.java | 63 +-
.../java/org/apache/beam/sdk/util/ZipFiles.java | 16 +-
.../beam/sdk/util/common/ReflectHelpers.java | 55 +-
.../main/java/org/apache/beam/sdk/values/KV.java | 2 +-
.../org/apache/beam/sdk/values/PCollection.java | 4 +-
.../apache/beam/sdk/values/PCollectionList.java | 4 +-
.../apache/beam/sdk/values/PCollectionTuple.java | 2 +-
.../apache/beam/sdk/values/PCollectionViews.java | 10 +-
.../org/apache/beam/sdk/values/PValueBase.java | 2 +-
.../main/java/org/apache/beam/sdk/values/Row.java | 10 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 4 +-
.../org/apache/beam/sdk/values/TaggedPValue.java | 2 +-
.../apache/beam/sdk/values/TimestampedValue.java | 2 +-
.../java/org/apache/beam/sdk/values/TupleTag.java | 4 +-
.../org/apache/beam/sdk/values/TupleTagList.java | 4 +-
.../org/apache/beam/sdk/values/TypeDescriptor.java | 10 +-
.../org/apache/beam/sdk/values/TypeParameter.java | 2 +-
.../beam/sdk/values/ValueInSingleWindow.java | 2 +-
.../apache/beam/sdk/values/ValueWithRecordId.java | 2 +-
.../apache/beam/sdk/values/WindowingStrategy.java | 4 +-
.../java/org/apache/beam/sdk/PipelineTest.java | 4 +-
.../apache/beam/sdk/coders/AvroCoderTestPojo.java | 2 +-
.../beam/sdk/coders/BigDecimalCoderTest.java | 2 +-
.../beam/sdk/coders/BigIntegerCoderTest.java | 2 +-
.../apache/beam/sdk/coders/CoderRegistryTest.java | 6 +-
.../apache/beam/sdk/coders/DefaultCoderTest.java | 2 +-
.../apache/beam/sdk/coders/DelegateCoderTest.java | 4 +-
.../apache/beam/sdk/coders/DurationCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/FloatCoderTest.java | 92 +
.../apache/beam/sdk/coders/InstantCoderTest.java | 4 +-
.../beam/sdk/coders/LengthPrefixCoderTest.java | 2 +-
.../org/apache/beam/sdk/coders/MapCoderTest.java | 2 +-
.../apache/beam/sdk/coders/NullableCoderTest.java | 4 +-
.../beam/sdk/coders/PrintBase64Encodings.java | 4 +-
.../org/apache/beam/sdk/coders/RowCoderTest.java | 2 +-
.../beam/sdk/coders/StructuralByteArrayTest.java | 2 +-
.../beam/sdk/coders/StructuredCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroIOTest.java | 181 +-
.../org/apache/beam/sdk/io/AvroSourceTest.java | 2 +-
.../apache/beam/sdk/io/CompressedSourceTest.java | 10 +-
.../org/apache/beam/sdk/io/FileBasedSinkTest.java | 4 +-
.../java/org/apache/beam/sdk/io/FileIOTest.java | 2 +-
.../org/apache/beam/sdk/io/FileSystemsTest.java | 8 +-
.../beam/sdk/io/LocalFileSystemRegistrarTest.java | 2 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 31 +-
.../org/apache/beam/sdk/io/TFRecordIOTest.java | 80 +-
.../org/apache/beam/sdk/io/TextIOReadTest.java | 10 +-
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 20 +-
.../beam/sdk/io/TextRowCountEstimatorTest.java | 116 +
.../org/apache/beam/sdk/io/WriteFilesTest.java | 90 +-
.../io/range/ByteKeyRangeEstimateFractionTest.java | 2 +-
.../io/range/ByteKeyRangeInterpolateKeyTest.java | 2 +-
.../apache/beam/sdk/io/range/ByteKeyRangeTest.java | 2 +-
.../beam/sdk/metrics/MetricResultsMatchers.java | 158 +-
.../sdk/options/PipelineOptionsFactoryTest.java | 16 +-
.../sdk/options/PipelineOptionsReflectorTest.java | 2 +-
.../beam/sdk/options/PipelineOptionsTest.java | 2 +-
.../sdk/options/ProxyInvocationHandlerTest.java | 8 +-
.../beam/sdk/options/SdkHarnessOptionsTest.java | 2 +-
.../apache/beam/sdk/options/ValueProviderTest.java | 2 +-
.../beam/sdk/options/ValueProvidersTest.java | 2 +-
.../beam/sdk/runners/TransformHierarchyTest.java | 2 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 5 +-
.../apache/beam/sdk/schemas/AvroSchemaTest.java | 4 +-
.../sdk/schemas/FieldAccessDescriptorTest.java | 4 +-
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 19 +-
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 17 +-
.../beam/sdk/schemas/SchemaRegistryTest.java | 2 +-
.../beam/sdk/schemas/transforms/AddFieldsTest.java | 356 +++
.../beam/sdk/schemas/transforms/CastTest.java | 2 +-
.../sdk/schemas/transforms/CastValidatorTest.java | 4 +-
.../beam/sdk/schemas/transforms/CoGroupTest.java | 4 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 30 +-
.../sdk/schemas/transforms/DropFieldsTest.java | 161 +
.../beam/sdk/schemas/transforms/FilterTest.java | 106 +-
.../beam/sdk/schemas/transforms/GroupTest.java | 4 +-
.../beam/sdk/schemas/transforms/JoinTest.java | 2 +-
.../beam/sdk/schemas/transforms/JoinTestUtils.java | 2 +-
.../sdk/schemas/transforms/RenameFieldsTest.java | 261 ++
.../beam/sdk/schemas/transforms/SelectTest.java | 4 +-
.../beam/sdk/schemas/utils/AvroGenerators.java | 6 +-
.../beam/sdk/schemas/utils/AvroUtilsTest.java | 40 +-
.../beam/sdk/schemas/utils/JavaBeanUtilsTest.java | 8 +-
.../beam/sdk/schemas/utils/POJOUtilsTest.java | 8 +-
.../beam/sdk/schemas/utils/SchemaZipFoldTest.java | 2 +-
.../beam/sdk/schemas/utils/SelectHelpersTest.java | 25 +-
.../apache/beam/sdk/state/StateContextsTest.java | 0
.../beam/sdk/testing/CoderPropertiesTest.java | 2 +-
.../beam/sdk/testing/CombineFnTesterTest.java | 2 +-
.../beam/sdk/testing/FileChecksumMatcherTest.java | 2 +-
.../beam/sdk/testing/GatherAllPanesTest.java | 2 +-
.../sdk/testing/InterceptingUrlClassLoader.java | 6 +-
.../org/apache/beam/sdk/testing/PAssertTest.java | 7 +-
.../beam/sdk/testing/PCollectionViewTesting.java | 2 +-
.../beam/sdk/testing/PaneExtractorsTest.java | 35 +-
.../beam/sdk/testing/SerializableMatchersTest.java | 2 +-
.../beam/sdk/testing/SourceTestUtilsTest.java | 2 +-
.../apache/beam/sdk/testing/StaticWindowsTest.java | 2 +-
.../beam/sdk/testing/WindowSupplierTest.java | 2 +-
.../sdk/transforms/ApproximateQuantilesTest.java | 2 +-
.../beam/sdk/transforms/ApproximateUniqueTest.java | 4 +-
.../apache/beam/sdk/transforms/CombineFnsTest.java | 4 +-
.../apache/beam/sdk/transforms/CombineTest.java | 12 +-
.../org/apache/beam/sdk/transforms/CreateTest.java | 4 +-
.../apache/beam/sdk/transforms/DistinctTest.java | 11 +-
.../apache/beam/sdk/transforms/DoFnTesterTest.java | 2 +-
.../beam/sdk/transforms/FlatMapElementsTest.java | 4 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 2 +-
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 2 +-
.../beam/sdk/transforms/GroupIntoBatchesTest.java | 2 +-
.../apache/beam/sdk/transforms/LatestFnTest.java | 2 +-
.../beam/sdk/transforms/MapElementsTest.java | 2 +-
.../org/apache/beam/sdk/transforms/MaxTest.java | 2 +-
.../org/apache/beam/sdk/transforms/MeanTest.java | 2 +-
.../org/apache/beam/sdk/transforms/MinTest.java | 2 +-
.../beam/sdk/transforms/ParDoLifecycleTest.java | 177 +-
.../beam/sdk/transforms/ParDoSchemaTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 16 +-
.../apache/beam/sdk/transforms/ReshuffleTest.java | 2 +-
.../org/apache/beam/sdk/transforms/SampleTest.java | 6 +-
.../beam/sdk/transforms/SplittableDoFnTest.java | 4 +-
.../org/apache/beam/sdk/transforms/SumTest.java | 2 +-
.../org/apache/beam/sdk/transforms/ViewTest.java | 4 +-
.../org/apache/beam/sdk/transforms/WaitTest.java | 18 +-
.../org/apache/beam/sdk/transforms/WatchTest.java | 18 +-
.../beam/sdk/transforms/WithFailuresTest.java | 2 +-
.../transforms/display/DisplayDataEvaluator.java | 2 +-
.../transforms/display/DisplayDataMatchers.java | 2 +-
.../sdk/transforms/display/DisplayDataTest.java | 8 +-
.../sdk/transforms/join/CoGbkResultCoderTest.java | 2 +-
.../beam/sdk/transforms/join/CoGroupByKeyTest.java | 4 +-
.../beam/sdk/transforms/join/UnionCoderTest.java | 2 +-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 2 +-
.../splittabledofn/ByteKeyRangeTrackerTest.java | 17 +-
.../splittabledofn/OffsetRangeTrackerTest.java | 13 +-
.../sdk/transforms/windowing/GlobalWindowTest.java | 4 +-
.../transforms/windowing/IntervalWindowTest.java | 2 +-
.../sdk/transforms/windowing/SessionsTest.java | 2 +-
.../beam/sdk/transforms/windowing/StubTrigger.java | 2 +-
.../beam/sdk/transforms/windowing/WindowTest.java | 2 +-
.../sdk/transforms/windowing/WindowingTest.java | 2 +-
.../org/apache/beam/sdk/util/ApiSurfaceTest.java | 8 +-
.../BufferedElementCountingOutputStreamTest.java | 4 +-
.../apache/beam/sdk/util/CombineFnUtilTest.java | 2 +-
.../sdk/util/ExposedByteArrayInputStreamTest.java | 2 +-
.../sdk/util/ExposedByteArrayOutputStreamTest.java | 2 +-
.../util/FilePatternMatchingShardedFileTest.java | 2 +-
.../beam/sdk/util/MutationDetectorsTest.java | 6 +-
.../beam/sdk/util/NumberedShardedFileTest.java | 2 +-
.../beam/sdk/util/SerializableUtilsTest.java | 4 +-
.../apache/beam/sdk/util/WindowedValueTest.java | 24 +-
.../org/apache/beam/sdk/util/ZipFilesTest.java | 10 +-
.../beam/sdk/util/common/ReflectHelpersTest.java | 41 +
.../java/org/apache/beam/sdk/values/KVTest.java | 2 +-
.../beam/sdk/values/PCollectionListTest.java | 2 +-
.../beam/sdk/values/PCollectionTupleTest.java | 2 +-
.../java/org/apache/beam/sdk/values/RowTest.java | 11 +-
.../org/apache/beam/sdk/values/TupleTagTest.java | 4 +-
.../apache/beam/sdk/values/TypeDescriptorTest.java | 2 +-
....sdk.util.common.ReflectHelpersTest$FakeService | 19 +
sdks/java/extensions/euphoria/README.md | 2 +-
sdks/java/extensions/euphoria/build.gradle | 10 +-
.../euphoria/core/client/operator/CountByKey.java | 2 +-
.../euphoria/core/client/operator/Distinct.java | 2 +-
.../euphoria/core/client/operator/FlatMap.java | 2 +-
.../euphoria/core/client/operator/ReduceByKey.java | 307 +-
.../core/client/operator/ReduceWindow.java | 294 +-
.../euphoria/core/client/operator/Union.java | 2 +-
.../core/client/operator/WindowBuilder.java | 2 +-
.../core/client/util/PCollectionLists.java | 2 +-
.../extensions/euphoria/core/client/util/Sums.java | 58 +-
.../core/translate/AbstractJoinTranslator.java | 2 +-
.../translate/BroadcastHashJoinTranslator.java | 6 +-
.../translate/CompositeOperatorTranslator.java | 2 +-
.../euphoria/core/translate/OperatorTransform.java | 2 +-
.../core/translate/ReduceByKeyTranslator.java | 85 +-
.../provider/GenericTranslatorProvider.java | 2 +-
.../core/client/operator/ReduceByKeyTest.java | 99 +-
.../euphoria/core/client/util/SumsTest.java | 52 +
.../core/docs/DocumentationExamplesTest.java | 2 +-
.../euphoria/core/testkit/ReduceByKeyTest.java | 49 +-
.../core/testkit/accumulators/NanosecondTimer.java | 2 +-
.../google-cloud-platform-core/build.gradle | 43 +-
.../sdk/extensions/gcp/options/GcpOptions.java | 21 +-
.../gcp/options/GcpPipelineOptionsRegistrar.java | 2 +-
.../sdk/extensions/gcp/options/GcsOptions.java | 4 +-
.../sdk/extensions/gcp/storage/GcsFileSystem.java | 18 +-
.../gcp/storage/GcsFileSystemRegistrar.java | 4 +-
.../extensions/gcp/storage/GcsPathValidator.java | 2 +-
.../sdk/extensions/gcp/storage/GcsResourceId.java | 6 +-
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 12 +-
.../beam/sdk/extensions/gcp/util/Transport.java | 2 +-
.../sdk/extensions/gcp/util/gcsfs/GcsPath.java | 4 +-
.../sdk/extensions/gcp/GcpCoreApiSurfaceTest.java | 2 +-
.../sdk/extensions/gcp/options/GcpOptionsTest.java | 18 +-
.../gcp/storage/GcsFileSystemRegistrarTest.java | 2 +-
.../extensions/gcp/storage/GcsFileSystemTest.java | 4 +-
.../extensions/gcp/util/CustomHttpErrorsTest.java | 8 +-
.../beam/sdk/extensions/gcp/util/GcsUtilIT.java | 12 +-
.../beam/sdk/extensions/gcp/util/GcsUtilTest.java | 4 +-
sdks/java/extensions/jackson/build.gradle | 12 +-
.../beam/sdk/extensions/jackson/AsJsons.java | 2 +-
.../beam/sdk/extensions/jackson/ParseJsons.java | 2 +-
.../extensions/jackson/JacksonTransformsTest.java | 2 +-
sdks/java/extensions/join-library/build.gradle | 6 +-
.../beam/sdk/extensions/joinlibrary/Join.java | 2 +-
sdks/java/extensions/kryo/build.gradle | 8 +-
.../sdk/extensions/kryo/KryoCoderProvider.java | 2 +-
sdks/java/extensions/protobuf/build.gradle | 8 +-
.../sdk/extensions/protobuf/ByteStringCoder.java | 2 +-
.../beam/sdk/extensions/protobuf/ProtoCoder.java | 6 +-
.../protobuf/ProtobufCoderProviderRegistrar.java | 2 +-
.../beam/sdk/extensions/protobuf/ProtobufUtil.java | 2 +-
.../extensions/protobuf/ByteStringCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtoCoderTest.java | 2 +-
.../sdk/extensions/protobuf/ProtobufUtilTest.java | 4 +-
sdks/java/extensions/sketching/build.gradle | 24 +-
.../extensions/sketching/ApproximateDistinct.java | 2 +-
.../extensions/sketching/SketchFrequencies.java | 2 +-
sdks/java/extensions/sorter/build.gradle | 6 +-
.../extensions/sorter/BufferedExternalSorter.java | 24 +-
.../beam/sdk/extensions/sorter/ExternalSorter.java | 198 +-
.../extensions/sorter/HadoopExternalSorter.java | 199 ++
.../beam/sdk/extensions/sorter/InMemorySorter.java | 6 +-
.../extensions/sorter/NativeExternalSorter.java | 77 +
.../sdk/extensions/sorter/NativeFileSorter.java | 283 ++
.../extensions/sorter/ExternalSorterBenchmark.java | 62 +
.../sdk/extensions/sorter/ExternalSorterTest.java | 56 +-
.../sdk/extensions/sorter/SorterTestUtils.java | 2 +-
sdks/java/extensions/sql/build.gradle | 36 +-
sdks/java/extensions/sql/datacatalog/build.gradle | 16 +-
.../sql/example/BeamSqlDataCatalogExample.java | 4 +-
.../datacatalog/DataCatalogTableProvider.java | 2 +-
.../sql/meta/provider/datacatalog/SchemaUtils.java | 5 +-
.../sql/meta/provider/datacatalog/TableUtils.java | 2 +-
sdks/java/extensions/sql/hcatalog/build.gradle | 8 +-
.../meta/provider/hcatalog/DatabaseProvider.java | 2 +-
.../provider/hcatalog/BeamSqlHiveSchemaTest.java | 2 +-
sdks/java/extensions/sql/jdbc/build.gradle | 10 +-
.../sdk/extensions/sql/jdbc/BeamSqlLineIT.java | 7 +-
sdks/java/extensions/sql/shell/build.gradle | 10 +-
.../apache/beam/sdk/extensions/sql/BeamSqlCli.java | 7 +-
.../beam/sdk/extensions/sql/BeamSqlTable.java | 7 +
.../beam/sdk/extensions/sql/SqlTransform.java | 6 +-
.../sdk/extensions/sql/example/BeamSqlExample.java | 2 +-
.../extensions/sql/example/BeamSqlPojoExample.java | 2 +-
.../sdk/extensions/sql/impl/BeamCalciteSchema.java | 5 +-
.../sql/impl/BeamCalciteSchemaFactory.java | 14 +-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 46 +-
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 45 +-
.../sql/impl/BeamSqlPipelineOptionsRegistrar.java | 2 +-
.../extensions/sql/impl/BeamTableStatistics.java | 93 +
.../extensions/sql/impl/CalciteQueryPlanner.java | 74 +-
.../sdk/extensions/sql/impl/JdbcConnection.java | 11 +-
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 6 +-
.../extensions/sql/impl/ScalarFunctionImpl.java | 67 +-
.../sql/impl/UdfImplReflectiveFunctionBase.java | 2 +-
.../sql/impl/parser/SqlColumnDeclaration.java | 2 +-
.../sql/impl/parser/SqlCreateExternalTable.java | 2 +-
.../extensions/sql/impl/parser/SqlDropObject.java | 2 +-
.../extensions/sql/impl/planner/BeamRuleSets.java | 9 +-
.../sql/impl/rel/BeamAggregationRel.java | 4 +-
.../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 235 +-
.../sql/impl/rel/BeamEnumerableConverter.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java | 2 +-
.../extensions/sql/impl/rel/BeamIOSourceRel.java | 42 +-
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 37 +-
.../sql/impl/rel/BeamSetOperatorRelBase.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamSortRel.java | 23 +-
.../extensions/sql/impl/rel/BeamUncollectRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 2 +-
.../sdk/extensions/sql/impl/rel/BeamValuesRel.java | 4 +-
.../sql/impl/rule/BeamJoinAssociateRule.java | 52 +
.../sql/impl/rule/BeamJoinPushThroughJoinRule.java | 70 +
.../sql/impl/rule/JoinRelOptRuleCall.java | 105 +
.../impl/transform/BeamBuiltinAggregations.java | 2 +-
.../impl/transform/BeamSetOperatorsTransforms.java | 2 +-
.../sql/impl/transform/agg/CovarianceFn.java | 2 +-
.../sql/impl/udf/BuiltinStringFunctions.java | 57 +-
.../sql/impl/utils/BigDecimalConverter.java | 2 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 6 +-
.../sql/meta/provider/ReadOnlyTableProvider.java | 2 +-
.../sql/meta/provider/bigquery/BigQueryTable.java | 41 +-
.../provider/bigquery/BigQueryTableProvider.java | 2 +-
.../sql/meta/provider/kafka/BeamKafkaTable.java | 8 +-
.../parquet/GenericRecordReadConverter.java | 64 +
.../sql/meta/provider/parquet/ParquetTable.java | 60 +
.../provider/parquet/ParquetTableProvider.java | 52 +
.../sql/meta/provider/parquet/package-info.java | 20 +
.../provider/seqgen/GenerateSequenceTable.java | 69 +
.../seqgen/GenerateSequenceTableProvider.java | 51 +
.../sql/meta/provider/seqgen/package-info.java | 20 +
.../sql/meta/provider/test/TestBoundedTable.java | 7 +
.../sql/meta/provider/test/TestTableProvider.java | 10 +-
.../sql/meta/provider/test/TestTableUtils.java | 2 +-
.../sql/meta/provider/test/TestUnboundedTable.java | 14 +
.../sql/meta/provider/text/TextTable.java | 35 +
.../sql/meta/provider/text/TextTableProvider.java | 8 +-
.../sql/meta/store/InMemoryMetaStore.java | 2 +-
.../sdk/extensions/sql/BeamComplexTypeTest.java | 328 +-
.../sdk/extensions/sql/BeamSqlDslFilterTest.java | 22 +
.../sdk/extensions/sql/BeamSqlDslProjectTest.java | 14 +
.../sql/BeamSqlDslSqlStdOperatorsTest.java | 8 +-
.../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 2 +-
.../beam/sdk/extensions/sql/BeamSqlMapTest.java | 2 +-
.../extensions/sql/BeamSqlMultipleSchemasTest.java | 2 +-
.../sdk/extensions/sql/PubsubToBigqueryIT.java | 4 +-
.../apache/beam/sdk/extensions/sql/TestUtils.java | 2 +-
.../sdk/extensions/sql/impl/BeamSqlEnvTest.java | 7 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java | 33 +-
.../extensions/sql/impl/parser/BeamDDLTest.java | 7 +-
.../sdk/extensions/sql/impl/rel/BaseRelTest.java | 6 +-
.../sql/impl/rel/BeamIOSourceRelTest.java | 128 +
.../rel/BeamJoinRelUnboundedVsBoundedTest.java | 2 +-
.../sql/impl/rule/JoinReorderingTest.java | 463 +++
...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 13 +-
.../provider/bigquery/BigQueryReadWriteIT.java | 2 +-
.../meta/provider/bigquery/BigQueryRowCountIT.java | 160 +
.../meta/provider/bigquery/BigQueryTestTable.java | 45 +
.../bigquery/BigQueryTestTableProvider.java | 71 +
.../provider/kafka/KafkaTableProviderTest.java | 2 +-
.../provider/parquet/GenericRecordToRowTest.java | 79 +
.../provider/parquet/ParquetTableReadTest.java | 90 +
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 11 +-
.../provider/pubsub/PubsubMessageToRowTest.java | 6 +-
.../meta/provider/text/TextTableProviderTest.java | 2 +-
.../beam/sdk/extensions/sql/utils/RowAsserts.java | 2 +-
.../sql/src/test/resources/users.parquet | Bin 0 -> 615 bytes
sdks/java/fn-execution/build.gradle | 14 +-
.../org/apache/beam/sdk/fn/JvmInitializers.java | 49 +
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 18 +-
.../beam/sdk/fn/channel/SocketAddressFactory.java | 6 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 14 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 20 +-
.../sdk/fn/data/BeamFnDataInboundObserver.java | 4 +-
.../apache/beam/sdk/fn/data/LogicalEndpoint.java | 10 +-
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 8 +-
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 8 +-
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 41 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 8 +-
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 8 +-
.../beam/sdk/fn/stream/DirectStreamObserver.java | 4 +-
.../stream/ForwardingClientResponseObserver.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactory.java | 4 +-
.../sdk/fn/stream/SynchronizedStreamObserver.java | 2 +-
.../fn/test/InProcessManagedChannelFactory.java | 4 +-
.../org/apache/beam/sdk/fn/test/TestExecutors.java | 2 +-
.../org/apache/beam/sdk/fn/test/TestStreams.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindow.java | 4 +-
.../apache/beam/sdk/fn/JvmInitializersTest.java | 78 +
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 6 +-
.../sdk/fn/channel/SocketAddressFactoryTest.java | 2 +-
.../BeamFnDataBufferingOutboundObserverTest.java | 14 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 16 +-
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 2 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 52 +-
.../sdk/fn/stream/BufferingStreamObserverTest.java | 4 +-
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 10 +-
.../sdk/fn/stream/DirectStreamObserverTest.java | 4 +-
.../ForwardingClientResponseObserverTest.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 2 +-
sdks/java/harness/build.gradle | 21 +-
.../beam/fn/harness/AssignWindowsRunner.java | 8 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 23 +-
.../beam/fn/harness/BeamFnDataWriteRunner.java | 24 +-
.../beam/fn/harness/BoundedSourceRunner.java | 6 +-
.../org/apache/beam/fn/harness/CombineRunners.java | 6 +-
.../fn/harness/DoFnPTransformRunnerFactory.java | 14 +-
.../org/apache/beam/fn/harness/FlattenRunner.java | 4 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 10 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 65 +-
.../org/apache/beam/fn/harness/MapFnRunners.java | 4 +-
.../beam/fn/harness/PrecombineGroupingTable.java | 6 +-
.../harness/SplittableProcessElementsRunner.java | 12 +-
.../beam/fn/harness/WindowMappingFnRunner.java | 2 +-
.../beam/fn/harness/WindowMergingFnRunner.java | 4 +-
.../harness/control/AddHarnessIdInterceptor.java | 10 +-
.../fn/harness/control/BeamFnControlClient.java | 8 +-
.../fn/harness/control/ProcessBundleHandler.java | 24 +-
.../beam/fn/harness/control/RegisterHandler.java | 2 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 10 +-
.../harness/data/MultiplexingFnDataReceiver.java | 2 +-
.../harness/data/PCollectionConsumerRegistry.java | 4 +-
.../fn/harness/data/QueueingBeamFnDataClient.java | 8 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 18 +-
.../apache/beam/fn/harness/state/BagUserState.java | 6 +-
.../harness/state/BeamFnStateGrpcClientCache.java | 4 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 10 +-
.../state/LazyCachingIteratorToIterable.java | 2 +-
.../beam/fn/harness/state/MultimapSideInput.java | 2 +-
.../fn/harness/state/StateFetchingIterators.java | 4 +-
.../stream/HarnessStreamObserverFactories.java | 2 +-
.../beam/fn/harness/AssignWindowsRunnerTest.java | 4 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 27 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 27 +-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 12 +-
.../apache/beam/fn/harness/CombineRunnersTest.java | 2 +-
.../apache/beam/fn/harness/FlattenRunnerTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 8 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 69 +-
.../apache/beam/fn/harness/MapFnRunnersTest.java | 4 +-
.../beam/fn/harness/WindowMergingFnRunnerTest.java | 6 +-
.../harness/control/BeamFnControlClientTest.java | 12 +-
.../harness/control/ProcessBundleHandlerTest.java | 10 +-
.../fn/harness/control/RegisterHandlerTest.java | 14 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 38 +-
.../data/BeamFnDataInboundObserverTest.java | 7 +-
.../data/MultiplexingFnDataReceiverTest.java | 2 +-
.../harness/data/QueueingBeamFnDataClientTest.java | 40 +-
.../harness/logging/BeamFnLoggingClientTest.java | 18 +-
.../beam/fn/harness/state/BagUserStateTest.java | 6 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 18 +-
.../fn/harness/state/FakeBeamFnStateClient.java | 2 +-
.../state/LazyCachingIteratorToIterableTest.java | 4 +-
.../fn/harness/state/MultimapSideInputTest.java | 6 +-
.../harness/state/StateFetchingIteratorsTest.java | 4 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 4 +-
sdks/java/io/amazon-web-services/build.gradle | 43 +-
.../sdk/io/aws/dynamodb/AttributeValueCoder.java | 166 ++
.../AttributeValueCoderProviderRegistrar.java | 37 +
.../sdk/io/aws/dynamodb/AwsClientsProvider.java | 34 +
.../sdk/io/aws/dynamodb/BasicDynamoDBProvider.java | 75 +
.../beam/sdk/io/aws/dynamodb/DynamoDBIO.java | 536 ++++
.../beam/sdk/io/aws/dynamodb/package-info.java | 19 +
.../apache/beam/sdk/io/aws/options/AwsModule.java | 2 +-
.../aws/options/AwsPipelineOptionsRegistrar.java | 2 +-
.../io/aws/s3/DefaultS3ClientBuilderFactory.java | 2 +-
.../apache/beam/sdk/io/aws/s3/S3FileSystem.java | 30 +-
.../beam/sdk/io/aws/s3/S3FileSystemRegistrar.java | 4 +-
.../io/aws/s3/S3ReadableSeekableByteChannel.java | 4 +-
.../apache/beam/sdk/io/aws/s3/S3ResourceId.java | 10 +-
.../beam/sdk/io/aws/s3/S3WritableByteChannel.java | 6 +-
.../beam/sdk/io/aws/sns/BasicSnsProvider.java | 2 +-
.../beam/sdk/io/aws/sns/PublishResultCoder.java | 2 +-
.../sdk/io/aws/sns/SnsCoderProviderRegistrar.java | 2 +-
.../java/org/apache/beam/sdk/io/aws/sns/SnsIO.java | 6 +-
.../beam/sdk/io/aws/sqs/SqsCheckpointMark.java | 4 +-
.../java/org/apache/beam/sdk/io/aws/sqs/SqsIO.java | 2 +-
.../beam/sdk/io/aws/sqs/SqsUnboundedSource.java | 4 +-
.../io/aws/dynamodb/AttributeValueCoderTest.java | 211 ++
.../io/aws/dynamodb/AwsClientsProviderMock.java | 46 +
.../beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java | 215 ++
.../sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java | 168 ++
.../beam/sdk/io/aws/s3/MatchResultMatcher.java | 6 +-
.../beam/sdk/io/aws/s3/S3FileSystemTest.java | 4 +-
.../org/apache/beam/sdk/io/aws/sns/SnsIOTest.java | 2 +-
sdks/java/io/amazon-web-services2/build.gradle | 52 +
.../sdk/io/aws2/dynamodb/AttributeValueCoder.java | 158 +
.../aws2/dynamodb/BasicDynamoDbClientProvider.java | 57 +
.../beam/sdk/io/aws2/dynamodb/DynamoDBIO.java | 532 ++++
.../io/aws2/dynamodb/DynamoDbClientProvider.java | 31 +
.../beam/sdk/io/aws2/dynamodb/package-info.java | 19 +
.../apache/beam/sdk/io/aws2/options/AwsModule.java | 197 ++
.../beam/sdk/io/aws2/options/AwsOptions.java | 101 +
.../aws2/options/AwsPipelineOptionsRegistrar.java | 33 +
.../beam/sdk/io/aws2/options/package-info.java | 22 +
.../io/aws2/dynamodb/AttributeValueCoderTest.java | 208 ++
.../beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java | 313 ++
.../sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java | 216 ++
.../aws2/dynamodb/DynamoDbClientProviderMock.java | 39 +
.../beam/sdk/io/aws2/options/AwsModuleTest.java | 119 +
sdks/java/io/amqp/build.gradle | 10 +-
.../java/org/apache/beam/sdk/io/amqp/AmqpIO.java | 4 +-
.../apache/beam/sdk/io/amqp/AmqpMessageCoder.java | 2 +-
.../io/amqp/AmqpMessageCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/amqp/AmqpMessageCoderTest.java | 2 +-
sdks/java/io/cassandra/build.gradle | 32 +-
.../apache/beam/sdk/io/cassandra/CassandraIO.java | 10 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 6 +-
sdks/java/io/clickhouse/build.gradle | 8 +-
.../beam/sdk/io/clickhouse/ClickHouseIO.java | 6 +-
.../beam/sdk/io/clickhouse/ClickHouseWriter.java | 4 +-
.../beam/sdk/io/clickhouse/BaseClickHouseTest.java | 2 +-
.../beam/sdk/io/clickhouse/TableSchemaTest.java | 2 +-
sdks/java/io/common/build.gradle | 4 +-
.../beam/sdk/io/common/DatabaseTestHelper.java | 18 +
.../org/apache/beam/sdk/io/common/HashingFn.java | 6 +-
.../org/apache/beam/sdk/io/common/IOITHelper.java | 6 +-
.../org/apache/beam/sdk/io/common/TestRow.java | 2 +-
.../elasticsearch-tests-2/build.gradle | 16 +-
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 6 +-
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 2 +-
.../elasticsearch-tests-5/build.gradle | 14 +-
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 8 +-
.../elasticsearch-tests-6/build.gradle | 14 +-
.../sdk/io/elasticsearch/ElasticsearchIOIT.java | 8 +-
.../elasticsearch-tests-common/build.gradle | 12 +-
.../io/elasticsearch/ElasticSearchIOTestUtils.java | 262 --
.../io/elasticsearch/ElasticsearchIOITCommon.java | 2 +-
.../elasticsearch/ElasticsearchIOTestCommon.java | 58 +-
.../io/elasticsearch/ElasticsearchIOTestUtils.java | 262 ++
sdks/java/io/elasticsearch/build.gradle | 20 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 10 +-
sdks/java/io/file-based-io-tests/build.gradle | 18 +-
.../beam/sdk/io/common/FileBasedIOITHelper.java | 4 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOIT.java | 2 +-
sdks/java/io/google-cloud-platform/build.gradle | 113 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 27 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 16 +-
.../bigquery/BigQueryCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/gcp/bigquery/BigQueryHelpers.java | 27 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 197 +-
.../sdk/io/gcp/bigquery/BigQueryQuerySource.java | 105 +-
.../io/gcp/bigquery/BigQueryQuerySourceDef.java | 162 +
.../bigquery/BigQuerySchemaRetrievalException.java | 25 +
.../beam/sdk/io/gcp/bigquery/BigQueryServices.java | 19 +-
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 38 +-
.../sdk/io/gcp/bigquery/BigQuerySourceBase.java | 12 +-
.../sdk/io/gcp/bigquery/BigQuerySourceDef.java | 52 +
.../gcp/bigquery/BigQueryStorageQuerySource.java | 2 +-
.../io/gcp/bigquery/BigQueryStorageSourceBase.java | 26 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 195 +-
.../gcp/bigquery/BigQueryStorageTableSource.java | 6 +-
.../sdk/io/gcp/bigquery/BigQueryTableSource.java | 63 +-
.../io/gcp/bigquery/BigQueryTableSourceDef.java | 104 +
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 188 +-
.../beam/sdk/io/gcp/bigquery/CreateTables.java | 23 +-
.../sdk/io/gcp/bigquery/DynamicDestinations.java | 13 +-
.../gcp/bigquery/DynamicDestinationsHelpers.java | 142 +-
.../sdk/io/gcp/bigquery/InsertRetryPolicy.java | 2 +-
.../io/gcp/bigquery/PassThroughThenCleanup.java | 2 +-
.../beam/sdk/io/gcp/bigquery/PrepareWrite.java | 2 +-
.../beam/sdk/io/gcp/bigquery/StreamingWriteFn.java | 4 +-
.../beam/sdk/io/gcp/bigquery/TableDestination.java | 68 +-
.../io/gcp/bigquery/TableDestinationCoderV2.java | 7 +-
.../io/gcp/bigquery/TableDestinationCoderV3.java | 64 +
.../sdk/io/gcp/bigquery/TableRowInfoCoder.java | 2 +-
.../beam/sdk/io/gcp/bigquery/TableRowWriter.java | 4 +-
.../beam/sdk/io/gcp/bigquery/TagWithUniqueIds.java | 2 +-
.../beam/sdk/io/gcp/bigquery/TestBigQuery.java | 2 +-
.../sdk/io/gcp/bigquery/WriteBundlesToFiles.java | 6 +-
.../beam/sdk/io/gcp/bigquery/WritePartition.java | 4 +-
.../beam/sdk/io/gcp/bigquery/WriteRename.java | 6 +-
.../beam/sdk/io/gcp/bigquery/WriteResult.java | 4 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 27 +-
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 6 +-
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 272 +-
.../sdk/io/gcp/bigtable/BigtableServiceImpl.java | 15 +-
.../sdk/io/gcp/bigtable/BigtableWriteResult.java | 37 +
.../io/gcp/bigtable/BigtableWriteResultCoder.java | 55 +
.../bigtable/VendoredListenableFutureAdapter.java | 2 +-
.../gcp/common/GcpIoPipelineOptionsRegistrar.java | 2 +-
.../sdk/io/gcp/datastore/AdaptiveThrottler.java | 2 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1.java | 16 +-
.../beam/sdk/io/gcp/pubsub/PubsubClient.java | 10 +-
.../gcp/pubsub/PubsubCoderProviderRegistrar.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubGrpcClient.java | 8 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 65 +-
.../beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 8 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessage.java | 2 +-
.../gcp/pubsub/PubsubMessagePayloadOnlyCoder.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubTestClient.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSink.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 12 +-
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 8 +-
.../beam/sdk/io/gcp/spanner/BatchSpannerRead.java | 2 +-
.../sdk/io/gcp/spanner/MutationCellCounter.java | 2 +-
.../beam/sdk/io/gcp/spanner/MutationGroup.java | 4 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 2 +-
.../beam/sdk/io/gcp/spanner/NaiveSpannerRead.java | 2 +-
.../beam/sdk/io/gcp/spanner/OrderedCode.java | 10 +-
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 4 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 12 +-
.../beam/sdk/io/gcp/spanner/SpannerSchema.java | 12 +-
.../sdk/io/gcp/spanner/SpannerWriteResult.java | 2 +-
.../beam/sdk/io/gcp/testing/BigqueryClient.java | 15 +-
.../beam/sdk/io/gcp/testing/BigqueryMatcher.java | 10 +-
.../cloud/spanner/FakeBatchTransactionId.java | 2 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 2 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 6 +-
.../sdk/io/gcp/bigquery/BigQueryHelpersTest.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryIOReadTest.java | 154 +-
.../io/gcp/bigquery/BigQueryIOStorageQueryIT.java | 2 +-
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 29 +-
.../io/gcp/bigquery/BigQueryIOStorageReadIT.java | 2 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 568 +++-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 132 +-
.../beam/sdk/io/gcp/bigquery/BigQueryKmsKeyIT.java | 5 +-
.../io/gcp/bigquery/BigQueryServicesImplTest.java | 6 +-
.../BigQueryTimePartitioningClusteringIT.java | 228 ++
.../sdk/io/gcp/bigquery/BigQueryToTableIT.java | 4 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 229 +-
.../sdk/io/gcp/bigquery/FakeBigQueryServices.java | 24 +-
.../sdk/io/gcp/bigquery/FakeDatasetService.java | 8 +-
.../beam/sdk/io/gcp/bigquery/FakeJobService.java | 26 +-
.../sdk/io/gcp/bigquery/InsertRetryPolicyTest.java | 2 +-
.../sdk/io/gcp/bigtable/BigtableConfigTest.java | 4 +-
.../beam/sdk/io/gcp/bigtable/BigtableIOTest.java | 166 +-
.../io/gcp/bigtable/BigtableServiceImplTest.java | 2 +-
.../beam/sdk/io/gcp/bigtable/BigtableWriteIT.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubClientTest.java | 2 +-
.../sdk/io/gcp/pubsub/PubsubGrpcClientTest.java | 6 +-
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 40 +-
.../sdk/io/gcp/pubsub/PubsubJsonClientTest.java | 4 +-
.../beam/sdk/io/gcp/pubsub/PubsubReadIT.java | 2 +-
.../sdk/io/gcp/pubsub/PubsubTestClientTest.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSinkTest.java | 6 +-
.../io/gcp/pubsub/PubsubUnboundedSourceTest.java | 2 +-
.../sdk/io/gcp/spanner/MutationKeyEncoderTest.java | 2 +-
.../beam/sdk/io/gcp/spanner/OrderedCodeTest.java | 10 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 6 +-
.../beam/sdk/io/gcp/spanner/SpannerWriteIT.java | 6 +-
.../beam/sdk/io/gcp/storage/GcsKmsKeyIT.java | 31 +-
.../sdk/io/gcp/testing/BigqueryMatcherTest.java | 2 +-
sdks/java/io/hadoop-common/build.gradle | 2 +-
sdks/java/io/hadoop-file-system/build.gradle | 17 +-
.../apache/beam/sdk/io/hdfs/HadoopFileSystem.java | 26 +-
.../beam/sdk/io/hdfs/HadoopFileSystemOptions.java | 8 +-
.../io/hdfs/HadoopFileSystemOptionsRegistrar.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemRegistrar.java | 4 +-
.../apache/beam/sdk/io/hdfs/HadoopResourceId.java | 4 +-
.../hdfs/HadoopFileSystemOptionsRegistrarTest.java | 2 +-
.../sdk/io/hdfs/HadoopFileSystemOptionsTest.java | 6 +-
.../sdk/io/hdfs/HadoopFileSystemRegistrarTest.java | 6 +-
.../beam/sdk/io/hdfs/HadoopFileSystemTest.java | 27 +-
sdks/java/io/hadoop-format/build.gradle | 32 +-
.../sdk/io/hadoop/format/HDFSSynchronization.java | 4 +-
.../beam/sdk/io/hadoop/format/HadoopFormatIO.java | 132 +-
.../beam/sdk/io/hadoop/format/HadoopFormats.java | 6 +-
.../format/ConfigurableEmployeeInputFormat.java | 2 +-
.../apache/beam/sdk/io/hadoop/format/Employee.java | 10 +-
.../sdk/io/hadoop/format/EmployeeInputFormat.java | 2 +-
.../io/hadoop/format/HDFSSynchronizationTest.java | 7 +-
.../hadoop/format/HadoopFormatIOCassandraTest.java | 2 -
.../hadoop/format/HadoopFormatIOElasticTest.java | 8 +-
.../sdk/io/hadoop/format/HadoopFormatIOIT.java | 2 +-
.../io/hadoop/format/HadoopFormatIOReadTest.java | 18 +-
.../format/HadoopFormatIOSequenceFileTest.java | 18 +-
.../format/ReuseObjectsEmployeeInputFormat.java | 4 +-
.../sdk/io/hadoop/format/TestEmployeeDataSet.java | 4 +-
.../sdk/io/hadoop/format/TestRowDBWritable.java | 5 +-
sdks/java/io/hbase/build.gradle | 16 +-
.../sdk/io/hbase/HBaseCoderProviderRegistrar.java | 2 +-
.../java/org/apache/beam/sdk/io/hbase/HBaseIO.java | 4 +-
sdks/java/io/hcatalog/build.gradle | 14 +-
.../org/apache/beam/sdk/io/hcatalog/HCatToRow.java | 2 +-
.../beam/sdk/io/hcatalog/HCatalogBeamSchema.java | 6 +-
.../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 101 +-
.../apache/beam/sdk/io/hcatalog/HCatalogUtils.java | 87 +
.../beam/sdk/io/hcatalog/PartitionPollerFn.java | 56 +
.../beam/sdk/io/hcatalog/PartitionReaderFn.java | 111 +
.../apache/beam/sdk/io/hcatalog/SchemaUtils.java | 2 +-
.../sdk/io/hcatalog/HCatalogBeamSchemaTest.java | 13 +-
.../apache/beam/sdk/io/hcatalog/HCatalogIOIT.java | 2 +-
.../beam/sdk/io/hcatalog/HCatalogIOTest.java | 95 +-
sdks/java/io/jdbc/build.gradle | 22 +-
.../sdk/io/jdbc/BeamSchemaInferenceException.java | 25 +
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 442 ++-
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 235 ++
.../org/apache/beam/sdk/io/jdbc/LogicalTypes.java | 241 ++
.../org/apache/beam/sdk/io/jdbc/SchemaUtil.java | 411 +++
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 429 +++
.../org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java | 44 +
.../org/apache/beam/sdk/io/jdbc/RowWithSchema.java | 44 +
.../apache/beam/sdk/io/jdbc/SchemaUtilTest.java | 345 +++
sdks/java/io/jms/build.gradle | 12 +-
.../java/org/apache/beam/sdk/io/jms/JmsIO.java | 4 +-
.../java/org/apache/beam/sdk/io/jms/JmsIOTest.java | 2 +-
sdks/java/io/kafka/build.gradle | 28 +-
.../org/apache/beam/sdk/io/kafka/ConsumerSpEL.java | 6 +-
.../CustomTimestampPolicyWithLimitedDelay.java | 2 +-
.../beam/sdk/io/kafka/KafkaCheckpointMark.java | 2 +-
.../beam/sdk/io/kafka/KafkaExactlyOnceSink.java | 26 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 89 +-
.../org/apache/beam/sdk/io/kafka/KafkaRecord.java | 2 +-
.../beam/sdk/io/kafka/KafkaUnboundedReader.java | 10 +-
.../beam/sdk/io/kafka/KafkaUnboundedSource.java | 6 +-
.../org/apache/beam/sdk/io/kafka/ProducerSpEL.java | 2 +-
.../beam/sdk/io/kafka/TimestampPolicyFactory.java | 2 +-
.../CustomTimestampPolicyWithLimitedDelayTest.java | 2 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 14 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 209 ++
.../org/apache/beam/sdk/io/kafka/KafkaIOTest.java | 19 +-
sdks/java/io/kinesis/build.gradle | 36 +-
.../beam/sdk/io/kinesis/BasicKinesisProvider.java | 2 +-
.../sdk/io/kinesis/DynamicCheckpointGenerator.java | 2 +-
.../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 253 +-
.../apache/beam/sdk/io/kinesis/KinesisReader.java | 23 +-
.../sdk/io/kinesis/KinesisReaderCheckpoint.java | 6 +-
.../apache/beam/sdk/io/kinesis/KinesisSource.java | 11 +-
.../beam/sdk/io/kinesis/KinesisWatermark.java | 116 -
.../apache/beam/sdk/io/kinesis/RecordFilter.java | 2 +-
.../beam/sdk/io/kinesis/ShardCheckpoint.java | 4 +-
.../beam/sdk/io/kinesis/ShardReadersPool.java | 33 +-
.../beam/sdk/io/kinesis/ShardRecordsIterator.java | 26 +-
.../sdk/io/kinesis/SimplifiedKinesisClient.java | 4 +-
.../apache/beam/sdk/io/kinesis/StartingPoint.java | 2 +-
.../sdk/io/kinesis/StartingPointShardsFinder.java | 2 +-
.../sdk/io/kinesis/StaticCheckpointGenerator.java | 2 +-
.../beam/sdk/io/kinesis/WatermarkParameters.java | 98 +
.../beam/sdk/io/kinesis/WatermarkPolicy.java | 29 +
.../sdk/io/kinesis/WatermarkPolicyFactory.java | 152 +
.../beam/sdk/io/kinesis/AmazonKinesisMock.java | 4 +-
.../io/kinesis/DynamicCheckpointGeneratorTest.java | 30 +-
.../apache/beam/sdk/io/kinesis/KinesisIOIT.java | 124 +-
.../beam/sdk/io/kinesis/KinesisMockReadTest.java | 5 +-
.../beam/sdk/io/kinesis/KinesisMockWriteTest.java | 38 +-
.../beam/sdk/io/kinesis/KinesisProducerMock.java | 39 +-
.../io/kinesis/KinesisReaderCheckpointTest.java | 2 +-
.../beam/sdk/io/kinesis/KinesisReaderTest.java | 41 +-
.../beam/sdk/io/kinesis/KinesisServiceMock.java | 2 +-
.../beam/sdk/io/kinesis/KinesisTestOptions.java | 12 +
.../beam/sdk/io/kinesis/KinesisWatermarkTest.java | 138 -
.../beam/sdk/io/kinesis/RecordFilterTest.java | 14 +-
.../beam/sdk/io/kinesis/ShardCheckpointTest.java | 5 +-
.../beam/sdk/io/kinesis/ShardReadersPoolTest.java | 65 +-
.../sdk/io/kinesis/ShardRecordsIteratorTest.java | 16 +-
.../io/kinesis/SimplifiedKinesisClientTest.java | 50 +-
.../io/kinesis/StartingPointShardsFinderTest.java | 40 +-
.../beam/sdk/io/kinesis/WatermarkPolicyTest.java | 152 +
sdks/java/io/kudu/build.gradle | 14 +-
.../java/org/apache/beam/sdk/io/kudu/KuduIO.java | 8 +-
.../apache/beam/sdk/io/kudu/KuduServiceImpl.java | 4 +-
.../org/apache/beam/sdk/io/kudu/KuduTestUtils.java | 2 +-
sdks/java/io/mongodb/build.gradle | 16 +-
.../org/apache/beam/sdk/io/mongodb/FindQuery.java | 2 +-
.../beam/sdk/io/mongodb/MongoDbGridFSIO.java | 4 +-
.../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 13 +-
.../apache/beam/sdk/io/mongodb/MongoDBIOIT.java | 2 +-
.../apache/beam/sdk/io/mongodb/MongoDbIOTest.java | 2 +-
sdks/java/io/mqtt/build.gradle | 16 +-
.../java/org/apache/beam/sdk/io/mqtt/MqttIO.java | 4 +-
sdks/java/io/parquet/build.gradle | 22 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 2 +-
sdks/java/io/rabbitmq/build.gradle | 13 +-
.../apache/beam/sdk/io/rabbitmq/RabbitMqIO.java | 2 +-
sdks/java/io/redis/build.gradle | 10 +-
.../sdk/io/redis/RedisConnectionConfiguration.java | 2 +-
.../java/org/apache/beam/sdk/io/redis/RedisIO.java | 32 +-
.../org/apache/beam/sdk/io/redis/RedisIOTest.java | 31 +
sdks/java/io/solr/build.gradle | 14 +-
.../beam/sdk/io/solr/AuthorizedSolrClient.java | 2 +-
.../java/org/apache/beam/sdk/io/solr/SolrIO.java | 10 +-
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 2 +-
sdks/java/io/synthetic/build.gradle | 16 +-
.../sdk/io/synthetic/SyntheticBoundedSource.java | 4 +-
.../beam/sdk/io/synthetic/SyntheticOptions.java | 6 +-
.../sdk/io/synthetic/SyntheticSourceOptions.java | 4 +-
.../beam/sdk/io/synthetic/SyntheticStep.java | 10 +-
.../sdk/io/synthetic/delay/SyntheticDelay.java | 6 +-
.../beam/sdk/io/synthetic/SyntheticStepTest.java | 2 +-
sdks/java/io/tika/build.gradle | 10 +-
.../org/apache/beam/sdk/io/tika/ParseResult.java | 8 +-
.../java/org/apache/beam/sdk/io/tika/TikaIO.java | 4 +-
sdks/java/io/xml/build.gradle | 14 +-
.../java/org/apache/beam/sdk/io/xml/JAXBCoder.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIO.java | 4 +-
.../org/apache/beam/sdk/io/xml/JAXBCoderTest.java | 2 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOTest.java | 2 +-
.../org/apache/beam/sdk/io/xml/XmlSourceTest.java | 2 +-
sdks/java/javadoc/apiclient-docs/package-list | 34 -
sdks/java/javadoc/avro-docs/package-list | 30 -
sdks/java/javadoc/bq-docs/package-list | 2 -
sdks/java/javadoc/build.gradle | 31 +-
sdks/java/javadoc/datastore-docs/package-list | 2 -
sdks/java/javadoc/guava-docs/package-list | 15 -
sdks/java/javadoc/hamcrest-docs/package-list | 10 -
.../javadoc/jackson-annotations-docs/package-list | 1 -
.../javadoc/jackson-databind-docs/package-list | 20 -
sdks/java/javadoc/joda-docs/package-list | 7 -
sdks/java/javadoc/junit-docs/package-list | 7 -
sdks/java/javadoc/oauth-docs/package-list | 11 -
sdks/java/maven-archetypes/examples/build.gradle | 9 +-
.../src/main/resources/archetype-resources/pom.xml | 19 +-
sdks/java/maven-archetypes/starter/build.gradle | 4 +-
sdks/java/testing/load-tests/build.gradle | 39 +-
.../beam/sdk/loadtests/CoGroupByKeyLoadTest.java | 2 +-
.../apache/beam/sdk/loadtests/CombineLoadTest.java | 4 +-
.../beam/sdk/loadtests/GroupByKeyLoadTest.java | 2 +-
.../org/apache/beam/sdk/loadtests/LoadTest.java | 8 +-
.../apache/beam/sdk/loadtests/LoadTestResult.java | 2 +-
.../apache/beam/sdk/loadtests/ParDoLoadTest.java | 62 +-
.../beam/sdk/loadtests/SyntheticDataPublisher.java | 2 +-
sdks/java/testing/nexmark/build.gradle | 68 +-
.../java/org/apache/beam/sdk/nexmark/Main.java | 4 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 12 +-
.../org/apache/beam/sdk/nexmark/NexmarkUtils.java | 10 +-
.../org/apache/beam/sdk/nexmark/model/Auction.java | 2 +-
.../apache/beam/sdk/nexmark/model/AuctionBid.java | 2 +-
.../beam/sdk/nexmark/model/BidsPerSession.java | 2 +-
.../beam/sdk/nexmark/model/CategoryPrice.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Done.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Event.java | 2 +-
.../beam/sdk/nexmark/model/IdNameReserve.java | 2 +-
.../org/apache/beam/sdk/nexmark/model/Person.java | 2 +-
.../apache/beam/sdk/nexmark/model/SellerPrice.java | 2 +-
.../sdk/nexmark/queries/BoundedSideInputJoin.java | 2 +-
.../beam/sdk/nexmark/queries/Query3Model.java | 4 +-
.../apache/beam/sdk/nexmark/queries/Query6.java | 2 +-
.../beam/sdk/nexmark/queries/Query8Model.java | 4 +-
.../sdk/nexmark/queries/SessionSideInputJoin.java | 2 +-
.../nexmark/queries/SessionSideInputJoinModel.java | 6 +-
.../beam/sdk/nexmark/queries/WinningBids.java | 2 +-
.../queries/sql/SqlBoundedSideInputJoin.java | 2 +-
.../beam/sdk/nexmark/queries/sql/SqlQuery5.java | 2 +-
.../sdk/nexmark/sources/generator/Generator.java | 2 +-
.../sources/generator/GeneratorCheckpoint.java | 2 +-
.../apache/beam/sdk/nexmark/NexmarkUtilsTest.java | 10 +-
.../beam/sdk/nexmark/model/sql/RowSizeTest.java | 2 +-
.../nexmark/queries/BoundedSideInputJoinTest.java | 2 +-
.../nexmark/queries/SessionSideInputJoinTest.java | 2 +-
.../queries/sql/SqlBoundedSideInputJoinTest.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery2Test.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery3Test.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery5Test.java | 2 +-
.../sdk/nexmark/queries/sql/SqlQuery7Test.java | 2 +-
sdks/java/testing/test-utils/build.gradle | 23 +-
.../apache/beam/sdk/testutils/NamedTestResult.java | 2 +-
.../beam/sdk/testutils/metrics/IOITMetrics.java | 19 +-
.../beam/sdk/testutils/metrics/MetricsReader.java | 6 +-
.../publishing/BigQueryResultsPublisherTest.java | 2 +-
sdks/python/apache_beam/__init__.py | 4 +-
sdks/python/apache_beam/coders/avro_coder.py | 99 +
sdks/python/apache_beam/coders/avro_coder_test.py | 71 +
sdks/python/apache_beam/coders/coder_impl.py | 7 +
sdks/python/apache_beam/coders/coders.py | 50 +-
sdks/python/apache_beam/coders/coders_test.py | 38 +-
.../apache_beam/coders/coders_test_common.py | 6 +-
sdks/python/apache_beam/coders/observable_test.py | 6 +-
.../apache_beam/coders/standard_coders_test.py | 27 +-
sdks/python/apache_beam/coders/stream_test.py | 46 +-
.../apache_beam/examples/complete/estimate_pi.py | 6 +-
.../examples/complete/game/game_stats_it_test.py | 6 -
.../examples/complete/game/leader_board_it_test.py | 6 -
.../examples/cookbook/group_with_coder.py | 4 +-
.../apache_beam/examples/snippets/snippets.py | 96 +-
.../apache_beam/examples/snippets/snippets_test.py | 9 +-
.../snippets/transforms}/__init__.py | 0
.../snippets/transforms/element_wise}/__init__.py | 0
.../snippets/transforms/element_wise/filter.py | 182 ++
.../transforms/element_wise/filter_test.py | 80 +
.../snippets/transforms/element_wise/flat_map.py | 221 ++
.../transforms/element_wise/flat_map_test.py | 87 +
.../snippets/transforms/element_wise/keys.py | 42 +
.../snippets/transforms/element_wise/keys_test.py | 55 +
.../snippets/transforms/element_wise/kvswap.py | 42 +
.../transforms/element_wise/kvswap_test.py | 55 +
.../snippets/transforms/element_wise/map.py | 203 ++
.../snippets/transforms/element_wise/map_test.py | 85 +
.../snippets/transforms/element_wise/pardo.py | 83 +
.../snippets/transforms/element_wise/pardo_test.py | 82 +
.../snippets/transforms/element_wise/partition.py | 136 +
.../transforms/element_wise/partition_test.py | 84 +
.../snippets/transforms/element_wise/regex.py | 161 +
.../snippets/transforms/element_wise/regex_test.py | 115 +
.../snippets/transforms/element_wise/to_string.py | 42 +
.../transforms/element_wise/to_string_test.py | 55 +
.../snippets/transforms/element_wise/values.py | 42 +
.../transforms/element_wise/values_test.py | 55 +
.../transforms/element_wise/with_timestamps.py | 106 +
.../element_wise/with_timestamps_test.py | 95 +
sdks/python/apache_beam/examples/snippets/util.py | 56 +
.../apache_beam/examples/snippets/util_test.py | 67 +
.../examples/streaming_wordcount_it_test.py | 14 +-
.../apache_beam/examples/wordcount_it_test.py | 33 +-
.../python/apache_beam/examples/wordcount_xlang.py | 3 +-
sdks/python/apache_beam/internal/gcp/auth.py | 149 +-
.../apache_beam/internal/gcp/json_value_test.py | 36 +-
.../apache_beam/internal/http_client_test.py | 24 +-
sdks/python/apache_beam/internal/module_test.py | 10 +
sdks/python/apache_beam/internal/pickler.py | 7 +
sdks/python/apache_beam/internal/pickler_test.py | 43 +-
sdks/python/apache_beam/internal/util_test.py | 20 +-
sdks/python/apache_beam/io/__init__.py | 2 +-
sdks/python/apache_beam/io/avroio.py | 1 +
sdks/python/apache_beam/io/avroio_test.py | 56 +-
sdks/python/apache_beam/io/concat_source_test.py | 78 +-
.../apache_beam/io/external/generate_sequence.py | 2 +
sdks/python/apache_beam/io/external/kafka.py | 4 +
sdks/python/apache_beam/io/filebasedsink.py | 3 +-
sdks/python/apache_beam/io/filebasedsource_test.py | 34 +-
sdks/python/apache_beam/io/fileio.py | 558 +++-
sdks/python/apache_beam/io/fileio_test.py | 370 ++-
sdks/python/apache_beam/io/filesystem.py | 34 +-
sdks/python/apache_beam/io/filesystem_test.py | 73 +
sdks/python/apache_beam/io/filesystemio.py | 50 +-
sdks/python/apache_beam/io/filesystemio_test.py | 51 +-
.../io/gcp/big_query_query_to_table_it_test.py | 6 -
sdks/python/apache_beam/io/gcp/bigquery.py | 47 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 162 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 102 +-
.../apache_beam/io/gcp/bigquery_read_it_test.py | 213 ++
.../apache_beam/io/gcp/bigquery_read_perf_test.py | 147 +
sdks/python/apache_beam/io/gcp/bigquery_test.py | 75 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 12 +
.../apache_beam/io/gcp/bigquery_tools_test.py | 6 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 274 ++
.../apache_beam/io/gcp/bigquery_write_perf_test.py | 115 +
.../apache_beam/io/gcp/datastore/v1new/helper.py | 6 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 38 +-
.../apache_beam/io/gcp/gcsio_integration_test.py | 2 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 2 +-
.../internal/clients/storage/storage_v1_client.py | 2 +-
.../apache_beam/io/gcp/pubsub_integration_test.py | 34 +-
sdks/python/apache_beam/io/gcp/pubsub_test.py | 35 +-
.../apache_beam/io/gcp/tests/bigquery_matcher.py | 102 +-
.../io/gcp/tests/bigquery_matcher_test.py | 25 +-
.../apache_beam/io/gcp/tests/pubsub_matcher.py | 2 +-
.../io/gcp/tests/pubsub_matcher_test.py | 6 +-
sdks/python/apache_beam/io/gcp/tests/utils.py | 5 +-
.../python/apache_beam/io/hadoopfilesystem_test.py | 4 +-
.../io/hdfs_integration_test/Dockerfile | 3 +-
.../hdfs_integration_test/hdfs_integration_test.sh | 10 +-
sdks/python/apache_beam/io/iobase.py | 269 +-
sdks/python/apache_beam/io/iobase_test.py | 158 +
sdks/python/apache_beam/io/localfilesystem_test.py | 4 +-
sdks/python/apache_beam/io/mongodbio.py | 370 +++
sdks/python/apache_beam/io/mongodbio_it_test.py | 92 +
sdks/python/apache_beam/io/mongodbio_test.py | 215 ++
sdks/python/apache_beam/io/parquetio_test.py | 6 +-
sdks/python/apache_beam/io/range_trackers.py | 37 +-
sdks/python/apache_beam/io/range_trackers_test.py | 27 +-
sdks/python/apache_beam/io/restriction_trackers.py | 5 +-
.../apache_beam/io/source_test_utils_test.py | 12 +-
sdks/python/apache_beam/io/sources_test.py | 35 +-
sdks/python/apache_beam/io/tfrecordio_test.py | 4 +-
sdks/python/apache_beam/io/utils.py | 11 +-
sdks/python/apache_beam/io/utils_test.py | 66 +
.../python/apache_beam/options/pipeline_options.py | 21 +-
.../apache_beam/options/pipeline_options_test.py | 25 +
.../options/pipeline_options_validator.py | 15 +
.../options/pipeline_options_validator_test.py | 33 +
sdks/python/apache_beam/pipeline.py | 16 +-
sdks/python/apache_beam/pipeline_test.py | 81 +-
sdks/python/apache_beam/pvalue.py | 16 +-
sdks/python/apache_beam/runners/common.pxd | 6 +
sdks/python/apache_beam/runners/common.py | 87 +-
sdks/python/apache_beam/runners/common_test.py | 59 +
.../dataflow_exercise_metrics_pipeline_test.py | 6 -
.../runners/dataflow/dataflow_runner.py | 56 +-
.../runners/dataflow/dataflow_runner_test.py | 25 +
.../runners/dataflow/internal/apiclient.py | 66 +-
.../runners/dataflow/internal/apiclient_test.py | 82 +-
.../clients/dataflow/dataflow_v1b3_client.py | 218 ++
.../clients/dataflow/dataflow_v1b3_messages.py | 182 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/dataflow/ptransform_overrides.py | 17 +
.../apache_beam/runners/direct/direct_runner.py | 13 +-
.../runners/direct/direct_runner_test.py | 8 +
.../runners/direct/helper_transforms.py | 7 +-
.../runners/direct/transform_evaluator.py | 45 +-
.../runners/interactive/cache_manager.py | 88 +-
.../runners/interactive/cache_manager_test.py | 36 +-
.../runners/interactive/interactive_runner.py | 9 +-
.../python/apache_beam/runners/pipeline_context.py | 7 +-
.../runners/portability/flink_runner.py | 69 +
.../runners/portability/flink_runner_test.py | 14 +-
.../runners/portability/fn_api_runner.py | 1048 ++++---
.../runners/portability/fn_api_runner_test.py | 190 +-
.../portability/fn_api_runner_transforms.py | 73 +-
.../portability/java_reference_runner_test.py | 166 --
.../apache_beam/runners/portability/job_server.py | 290 +-
.../runners/portability/local_job_service.py | 21 +-
.../runners/portability/portable_runner.py | 68 +-
.../runners/portability/portable_runner_test.py | 10 +
.../runners/portability/spark_runner_test.py | 20 +-
.../apache_beam/runners/portability/stager.py | 2 +-
.../apache_beam/runners/portability/stager_test.py | 2 +-
sdks/python/apache_beam/runners/runner.py | 64 +-
.../apache_beam/runners/worker/bundle_processor.py | 120 +-
.../apache_beam/runners/worker/data_plane.py | 80 +-
.../apache_beam/runners/worker/data_plane_test.py | 53 +-
.../apache_beam/runners/worker/operations.pxd | 1 +
.../apache_beam/runners/worker/operations.py | 14 +
.../apache_beam/runners/worker/sdk_worker.py | 17 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 7 +-
.../runners/worker/sdk_worker_main_test.py | 2 +-
.../apache_beam/testing/load_tests/build.gradle | 5 +-
.../testing/load_tests/co_group_by_key_test.py | 33 +-
.../apache_beam/testing/load_tests/combine_test.py | 20 +-
.../testing/load_tests/group_by_key_test.py | 4 +-
.../apache_beam/testing/load_tests/load_test.py | 34 +-
.../testing/load_tests/load_test_metrics_utils.py | 39 +-
.../apache_beam/testing/load_tests/pardo_test.py | 70 +-
.../testing/load_tests/sideinput_test.py | 4 +-
.../apache_beam/testing/synthetic_pipeline.py | 260 +-
.../apache_beam/testing/synthetic_pipeline_test.py | 127 +-
sdks/python/apache_beam/testing/test_stream.py | 2 +-
sdks/python/apache_beam/testing/util.py | 32 +
sdks/python/apache_beam/testing/util_test.py | 11 +
.../apache_beam/tools/coders_microbenchmark.py | 32 +-
.../apache_beam/tools/microbenchmarks_test.py | 20 +
sdks/python/apache_beam/tools/utils.py | 3 +-
sdks/python/apache_beam/transforms/__init__.py | 1 +
sdks/python/apache_beam/transforms/combiners.py | 96 +-
.../apache_beam/transforms/combiners_test.py | 91 +-
sdks/python/apache_beam/transforms/core.py | 232 +-
.../dataflow_distribution_counter_test.py | 24 +-
.../apache_beam/transforms/dofn_lifecycle_test.py | 89 +
sdks/python/apache_beam/transforms/external.py | 7 +
sdks/python/apache_beam/transforms/ptransform.py | 11 +-
.../apache_beam/transforms/ptransform_test.py | 142 +-
sdks/python/apache_beam/transforms/stats.py | 236 ++
sdks/python/apache_beam/transforms/stats_test.py | 349 +++
sdks/python/apache_beam/transforms/trigger.py | 14 +-
sdks/python/apache_beam/transforms/trigger_test.py | 4 +-
.../apache_beam/transforms/userstate_test.py | 49 +-
sdks/python/apache_beam/transforms/util.py | 230 +-
sdks/python/apache_beam/transforms/util_test.py | 213 +-
sdks/python/apache_beam/transforms/window.py | 16 +-
sdks/python/apache_beam/typehints/decorators.py | 62 +-
.../typehints/native_type_compatibility.py | 62 +-
.../typehints/native_type_compatibility_test.py | 16 +-
sdks/python/apache_beam/typehints/opcodes.py | 113 +-
.../apache_beam/typehints/trivial_inference.py | 177 +-
.../typehints/trivial_inference_test.py | 107 +-
.../apache_beam/typehints/typed_pipeline_test.py | 22 +-
sdks/python/apache_beam/typehints/typehints.py | 26 +-
.../python/apache_beam/typehints/typehints_test.py | 87 +-
sdks/python/apache_beam/utils/retry.py | 9 +
sdks/python/apache_beam/utils/timestamp.py | 30 +-
sdks/python/apache_beam/utils/timestamp_test.py | 9 +
sdks/python/apache_beam/utils/windowed_value.py | 16 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/build.gradle | 412 +--
sdks/python/container/build.gradle | 8 +-
sdks/python/container/py3/build.gradle | 4 +-
sdks/python/container/run_validatescontainer.sh | 7 +-
sdks/python/gen_protos.py | 3 +-
sdks/python/scripts/run_dependency_check.sh | 3 +-
sdks/python/scripts/run_integration_test.sh | 33 +-
sdks/python/scripts/run_pylint.sh | 3 +-
sdks/python/scripts/run_snapshot_publish.sh | 2 +-
sdks/python/setup.py | 19 +-
sdks/python/test-suites/dataflow/build.gradle | 47 -
sdks/python/test-suites/dataflow/py2/build.gradle | 116 +
sdks/python/test-suites/dataflow/py35/build.gradle | 35 +-
sdks/python/test-suites/dataflow/py36/build.gradle | 36 +-
sdks/python/test-suites/dataflow/py37/build.gradle | 126 +
sdks/python/test-suites/direct/py2/build.gradle | 114 +
sdks/python/test-suites/direct/py35/build.gradle | 5 +-
sdks/python/test-suites/direct/py36/build.gradle | 6 +-
sdks/python/test-suites/direct/py37/build.gradle | 63 +
sdks/python/test-suites/portable/py2/build.gradle | 235 ++
sdks/python/test-suites/portable/py35/build.gradle | 32 +
sdks/python/test-suites/tox/py2/build.gradle | 59 +
settings.gradle | 349 +--
vendor/bytebuddy-1_9_3/build.gradle | 34 +
vendor/grpc-1_13_1/build.gradle | 36 -
vendor/grpc-1_21_0/build.gradle | 36 +
vendor/guava-20_0/build.gradle | 35 -
vendor/guava-26_0-jre/build.gradle | 42 +
vendor/sdks-java-extensions-protobuf/build.gradle | 26 +-
website/README.md | 4 +-
website/_config.yml | 2 +-
website/_config_test.yml | 2 +-
website/build.gradle | 2 +-
website/src/.htaccess | 2 +-
website/src/_data/authors.yml | 12 +
website/src/_data/capability-matrix.yml | 217 ++
website/src/_includes/capability-matrix.md | 9 +-
website/src/_includes/head.html | 2 +-
website/src/_includes/header.html | 16 +
.../src/_includes/section-menu/documentation.html | 133 +-
website/src/_includes/section-menu/runners.html | 1 +
website/src/_includes/section-menu/sdks.html | 29 +-
.../src/_posts/2017-02-13-stateful-processing.md | 110 +-
website/src/_posts/2018-01-09-beam-a-look-back.md | 2 +-
website/src/_posts/2019-03-18-beam-summit-site.md | 38 +
.../2019-05-01-adding-data-sources-to-sql.md | 202 ++
.../_posts/2019-05-11-beam-summit-europe-2019.md | 88 +
website/src/_posts/2019-05-22-beam-2.13.0.md | 77 +
website/src/_posts/2019-05-30-beam-kata-release.md | 59 +
website/src/_posts/2019-06-11-looping-timers.md | 349 +++
website/src/_sass/_global.sass | 2 +-
website/src/_sass/capability-matrix.scss | 2 +-
website/src/community/in-person.md | 6 +-
website/src/contribute/design-documents.md | 7 +
website/src/contribute/ptransform-style-guide.md | 4 +-
website/src/contribute/release-guide.md | 24 +-
website/src/contribute/runner-guide.md | 4 +-
.../documentation/dsls/sql/aggregate-functions.md | 34 -
.../dsls/sql/calcite/aggregate-functions.md | 37 +
.../documentation/dsls/sql/calcite/data-types.md | 48 +
.../dsls/sql/calcite/lexical-structure.md | 1050 +++++++
.../src/documentation/dsls/sql/calcite/overview.md | 88 +
.../documentation/dsls/sql/calcite/query-syntax.md | 716 +++++
.../dsls/sql/calcite/scalar-functions.md | 137 +
.../dsls/sql/create-external-table.md | 364 ---
website/src/documentation/dsls/sql/data-types.md | 46 -
.../dsls/sql/extensions/create-external-table.md | 365 +++
.../src/documentation/dsls/sql/extensions/joins.md | 73 +
.../src/documentation/dsls/sql/extensions/set.md | 56 +
.../dsls/sql/extensions/user-defined-functions.md | 128 +
.../sql/extensions/windowing-and-triggering.md | 67 +
website/src/documentation/dsls/sql/joins.md | 76 -
.../documentation/dsls/sql/lexical-structure.md | 1048 -------
website/src/documentation/dsls/sql/overview.md | 37 +-
.../src/documentation/dsls/sql/scalar-functions.md | 134 -
website/src/documentation/dsls/sql/select.md | 715 -----
website/src/documentation/dsls/sql/set.md | 55 -
website/src/documentation/dsls/sql/shell.md | 10 +-
.../dsls/sql/user-defined-functions.md | 124 -
.../dsls/sql/windowing-and-triggering.md | 66 -
website/src/documentation/index.md | 3 +-
.../documentation/io/built-in-google-bigquery.md | 51 +-
website/src/documentation/io/built-in-hadoop.md | 16 +-
website/src/documentation/io/built-in.md | 1 -
.../documentation/patterns/custom-io-patterns.md | 42 +
.../patterns/file-processing-patterns.md | 107 +
website/src/documentation/patterns/overview.md | 45 +
.../patterns/pipeline-option-patterns.md | 47 +
.../documentation/patterns/side-input-patterns.md | 48 +
website/src/documentation/programming-guide.md | 122 +-
.../documentation/resources/learning-resources.md | 27 +
website/src/documentation/runners/flink.md | 31 +-
website/src/documentation/runners/jet.md | 197 ++
website/src/documentation/runners/spark.md | 132 +-
website/src/documentation/sdks/euphoria.md | 4 +-
.../src/documentation/sdks/java-dependencies.md | 17 +-
website/src/documentation/sdks/nexmark.md | 45 +-
.../src/documentation/sdks/python-dependencies.md | 93 +-
website/src/documentation/sdks/python-streaming.md | 2 +-
.../java/aggregation/approximatequantiles.md | 43 +
.../java/aggregation/approximateunique.md | 40 +
.../transforms/java/aggregation/cogroupbykey.md | 73 +
.../transforms/java/aggregation/combine.md | 82 +
.../java/aggregation/combinewithcontext.md | 37 +
.../transforms/java/aggregation/count.md | 50 +
.../transforms/java/aggregation/distinct.md | 43 +
.../transforms/java/aggregation/groupbykey.md | 50 +
.../java/aggregation/groupintobatches.md | 42 +
.../transforms/java/aggregation/latest.md | 52 +
.../transforms/java/aggregation/max.md | 56 +
.../transforms/java/aggregation/mean.md | 58 +
.../transforms/java/aggregation/min.md | 42 +
.../transforms/java/aggregation/sample.md | 40 +
.../transforms/java/aggregation/sum.md | 51 +
.../transforms/java/aggregation/top.md | 39 +
.../transforms/java/element-wise/filter.md | 62 +
.../java/element-wise/flatmapelements.md | 40 +
.../transforms/java/element-wise/keys.md | 43 +
.../transforms/java/element-wise/kvswap.md | 43 +
.../transforms/java/element-wise/mapelements.md | 63 +
.../transforms/java/element-wise/pardo.md | 152 +
.../transforms/java/element-wise/partition.md | 62 +
.../transforms/java/element-wise/regex.md | 36 +
.../transforms/java/element-wise/reify.md | 39 +
.../transforms/java/element-wise/tostring.md | 37 +
.../transforms/java/element-wise/values.md | 44 +
.../transforms/java/element-wise/withkeys.md | 55 +
.../transforms/java/element-wise/withtimestamps.md | 36 +
website/src/documentation/transforms/java/index.md | 81 +
.../documentation/transforms/java/other/create.md | 40 +
.../documentation/transforms/java/other/flatten.md | 67 +
.../documentation/transforms/java/other/passert.md | 61 +
.../documentation/transforms/java/other/view.md | 37 +
.../documentation/transforms/java/other/window.md | 40 +
.../python/aggregation/approximatequantiles.md | 26 +
.../python/aggregation/approximateunique.md | 25 +
.../transforms/python/aggregation/cogroupbykey.md | 45 +
.../python/aggregation/combineglobally.md | 43 +
.../python/aggregation/combinewithcontext.md | 26 +
.../transforms/python/aggregation/count.md | 36 +
.../transforms/python/aggregation/distinct.md | 37 +
.../transforms/python/aggregation/groupbykey.md | 41 +
.../python/aggregation/groupintobatches.md | 26 +
.../transforms/python/aggregation/latest.md | 26 +
.../transforms/python/aggregation/max.md | 26 +
.../transforms/python/aggregation/mean.md | 39 +
.../transforms/python/aggregation/min.md | 26 +
.../transforms/python/aggregation/sample.md | 38 +
.../transforms/python/aggregation/sum.md | 26 +
.../transforms/python/aggregation/top.md | 38 +
.../transforms/python/element-wise/filter.md | 195 ++
.../transforms/python/element-wise/flatmap.md | 42 +
.../transforms/python/element-wise/keys.md | 39 +
.../transforms/python/element-wise/kvswap.md | 39 +
.../transforms/python/element-wise/map.md | 42 +
.../transforms/python/element-wise/pardo.md | 44 +
.../transforms/python/element-wise/partition.md | 50 +
.../transforms/python/element-wise/regex.md | 28 +
.../transforms/python/element-wise/reify.md | 28 +
.../transforms/python/element-wise/tostring.md | 27 +
.../transforms/python/element-wise/values.md | 37 +
.../transforms/python/element-wise/withkeys.md | 27 +
.../python/element-wise/withtimestamps.md | 28 +
.../src/documentation/transforms/python/index.md | 86 +
.../transforms/python/other/create.md | 38 +
.../transforms/python/other/flatten.md | 43 +
.../transforms/python/other/passert.md | 25 +
.../transforms/python/other/reshuffle.md | 41 +
.../documentation/transforms/python/other/view.md | 25 +
.../transforms/python/other/windowinto.md | 41 +
website/src/get-started/beam-overview.md | 1 +
website/src/get-started/downloads.md | 25 +-
website/src/get-started/quickstart-java.md | 45 +-
website/src/get-started/quickstart-py.md | 9 +-
website/src/get-started/wordcount-example.md | 38 +-
.../blog/beam-kata/beam-kata-intellij-edu-1.png | Bin 0 -> 52004 bytes
.../blog/beam-kata/beam-kata-intellij-edu-2.png | Bin 0 -> 148295 bytes
.../blog/beam-kata/beam-kata-pycharm-edu-1.png | Bin 0 -> 44655 bytes
.../blog/beam-kata/beam-kata-pycharm-edu-2.png | Bin 0 -> 164407 bytes
website/src/images/logos/runners/jet.png | Bin 0 -> 9128 bytes
website/src/js/section-nav.js | 6 +-
website/src/roadmap/portability.md | 33 +-
website/src/roadmap/python-sdk.md | 10 +-
2935 files changed, 86951 insertions(+), 36364 deletions(-)
create mode 100644 .test-infra/jenkins/Infrastructure.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_CoGBK_Java.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_Combine_Flink_Python.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_Combine_Python.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Flink_Python.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Java.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Python.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Python_reiterate.groovy
delete mode 100644 .test-infra/jenkins/job_LoadTests_Java.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Java.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Python.groovy
delete mode 100644 .test-infra/jenkins/job_LoadTests_Python.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_Python_Smoke.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_coGBK_Flink_Python.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_coGBK_Python.groovy
create mode 100644 .test-infra/jenkins/job_PerformanceTests_BigQueryIO_Python.groovy
delete mode 100644 .test-infra/jenkins/job_PerformanceTests_Dataflow.groovy
delete mode 100644 .test-infra/jenkins/job_PerformanceTests_Spark.groovy
delete mode 100644 .test-infra/jenkins/job_Performancetests_Python35.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Flink.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Go_ValidatesRunner_Spark.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Java_PortableValidatesRunner_Spark_Batch.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_Direct.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python2.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python35.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python36.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python37.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Python3_Verify.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python_MongoDBIO_IT.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python_ValidatesRunner_Spark.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Python_Verify.groovy
create mode 100755 .test-infra/metrics/build_and_publish_containers.sh
delete mode 100644 .test-infra/metrics/dashboards/Post-Commits_status_dashboard.json
delete mode 100644 .test-infra/metrics/dashboards/code_velocity.json
delete mode 100644 .test-infra/metrics/dashboards/post-commit_tests.json
delete mode 100644 .test-infra/metrics/dashboards/pre-commit_tests.json
delete mode 100644 .test-infra/metrics/dashboards/source_data_freshness.json
delete mode 100644 .test-infra/metrics/dashboards/stability_critical_jobs_status.json
create mode 100644 .test-infra/metrics/grafana/Dockerfile
create mode 100644 .test-infra/metrics/grafana/dashboards/Post-Commits_status_dashboard.json
create mode 100644 .test-infra/metrics/grafana/dashboards/code_velocity.json
create mode 100644 .test-infra/metrics/grafana/dashboards/post-commit_tests.json
create mode 100644 .test-infra/metrics/grafana/dashboards/pre-commit_tests.json
create mode 100644 .test-infra/metrics/grafana/dashboards/source_data_freshness.json
create mode 100644 .test-infra/metrics/grafana/dashboards/stability_critical_jobs_status.json
create mode 100644 .test-infra/metrics/grafana/provisioning/dashboards/all.yaml
create mode 100644 .test-infra/metrics/grafana/provisioning/datasources/beampostgresql.yaml
create mode 100644 .test-infra/metrics/postgres/init.sql
create mode 100644 deprecation-warning.txt
delete mode 100644 examples/kotlin/OWNERS
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/BigQueryTornadoes.kt
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/CombinePerKeyExamples.kt
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/DistinctExample.kt
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/FilterExamples.kt
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/JoinExamples.kt
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/MaxPerKeyExamples.kt
copy examples/{java/src/main/java/org/apache/beam/examples => kotlin/src/main/java/org/apache/beam/examples/kotlin}/cookbook/README.md (100%)
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/cookbook/TriggerExample.kt
create mode 100644 examples/kotlin/src/main/java/org/apache/beam/examples/kotlin/snippets/Snippets.kt
copy gradlew => gradlew_orig (100%)
copy gradlew.bat => gradlew_orig.bat (100%)
create mode 100644 learning/katas/README.md
create mode 100644 learning/katas/java/.idea/study_project.xml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Count/src/org/apache/beam/learning/katas/commontransforms/aggregation/count/Task.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Count/task.html
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Count/test/org/apache/beam/learning/katas/commontransforms/aggregation/count/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Max/src/org/apache/beam/learning/katas/commontransforms/aggregation/max/Task.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Max/task.html
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Max/test/org/apache/beam/learning/katas/commontransforms/aggregation/max/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Mean/src/org/apache/beam/learning/katas/commontransforms/aggregation/mean/Task.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Mean/task.html
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Mean/test/org/apache/beam/learning/katas/commontransforms/aggregation/mean/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Min/src/org/apache/beam/learning/katas/commontransforms/aggregation/min/Task.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Min/task.html
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Min/test/org/apache/beam/learning/katas/commontransforms/aggregation/min/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Sum/src/org/apache/beam/learning/katas/commontransforms/aggregation/sum/Task.java
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Sum/task.html
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Sum/test/org/apache/beam/learning/katas/commontransforms/aggregation/sum/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/Filter/Filter/src/org/apache/beam/learning/katas/commontransforms/filter/filter/Task.java
create mode 100644 learning/katas/java/Common Transforms/Filter/Filter/task.html
create mode 100644 learning/katas/java/Common Transforms/Filter/Filter/test/org/apache/beam/learning/katas/commontransforms/filter/filter/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/Filter/ParDo/src/org/apache/beam/learning/katas/commontransforms/filter/pardo/Task.java
create mode 100644 learning/katas/java/Common Transforms/Filter/ParDo/task.html
create mode 100644 learning/katas/java/Common Transforms/Filter/ParDo/test/org/apache/beam/learning/katas/commontransforms/filter/pardo/TaskTest.java
create mode 100644 learning/katas/java/Common Transforms/WithKeys/WithKeys/src/org/apache/beam/learning/katas/commontransforms/withkeys/Task.java
create mode 100644 learning/katas/java/Common Transforms/WithKeys/WithKeys/task.html
create mode 100644 learning/katas/java/Common Transforms/WithKeys/WithKeys/test/org/apache/beam/learning/katas/commontransforms/withkeys/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Branching/Branching/src/org/apache/beam/learning/katas/coretransforms/branching/Task.java
create mode 100644 learning/katas/java/Core Transforms/Branching/Branching/task.html
create mode 100644 learning/katas/java/Core Transforms/Branching/Branching/test/org/apache/beam/learning/katas/coretransforms/branching/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/Task.java
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/src/org/apache/beam/learning/katas/coretransforms/cogroupbykey/WordsAlphabet.java
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/test/org/apache/beam/learning/katas/coretransforms/cogroupbykey/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/Task.java
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task.html
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefnlambda/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/Task.java
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task.html
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/binarycombinefn/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Combine/Combine PerKey/src/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/Task.java
create mode 100644 learning/katas/java/Core Transforms/Combine/Combine PerKey/task.html
create mode 100644 learning/katas/java/Core Transforms/Combine/Combine PerKey/test/org/apache/beam/learning/katas/coretransforms/combine/combineperkey/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Combine/CombineFn/src/org/apache/beam/learning/katas/coretransforms/combine/combinefn/Task.java
create mode 100644 learning/katas/java/Core Transforms/Combine/CombineFn/task.html
create mode 100644 learning/katas/java/Core Transforms/Combine/CombineFn/test/org/apache/beam/learning/katas/coretransforms/combine/combinefn/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Combine/Simple Function/src/org/apache/beam/learning/katas/coretransforms/combine/simple/Task.java
create mode 100644 learning/katas/java/Core Transforms/Combine/Simple Function/task.html
create mode 100644 learning/katas/java/Core Transforms/Combine/Simple Function/test/org/apache/beam/learning/katas/coretransforms/combine/simple/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/Composite Transform/src/org/apache/beam/learning/katas/coretransforms/composite/Task.java
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task.html
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/Composite Transform/test/org/apache/beam/learning/katas/coretransforms/composite/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/src/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/Task.java
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task.html
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/test/org/apache/beam/learning/katas/coretransforms/dofnadditionalparams/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Flatten/Flatten/src/org/apache/beam/learning/katas/coretransforms/flatten/Task.java
create mode 100644 learning/katas/java/Core Transforms/Flatten/Flatten/task.html
create mode 100644 learning/katas/java/Core Transforms/Flatten/Flatten/test/org/apache/beam/learning/katas/coretransforms/flatten/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/GroupByKey/src/org/apache/beam/learning/katas/coretransforms/groupbykey/Task.java
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task.html
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/GroupByKey/test/org/apache/beam/learning/katas/coretransforms/groupbykey/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Map/FlatMapElements/src/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/Task.java
create mode 100644 learning/katas/java/Core Transforms/Map/FlatMapElements/task.html
create mode 100644 learning/katas/java/Core Transforms/Map/FlatMapElements/test/org/apache/beam/learning/katas/coretransforms/map/flatmapelements/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Map/MapElements/src/org/apache/beam/learning/katas/coretransforms/map/mapelements/Task.java
create mode 100644 learning/katas/java/Core Transforms/Map/MapElements/task.html
create mode 100644 learning/katas/java/Core Transforms/Map/MapElements/test/org/apache/beam/learning/katas/coretransforms/map/mapelements/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo OneToMany/src/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/Task.java
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo OneToMany/task.html
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo OneToMany/test/org/apache/beam/learning/katas/coretransforms/map/pardoonetomany/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo/src/org/apache/beam/learning/katas/coretransforms/map/pardo/Task.java
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo/task.html
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo/test/org/apache/beam/learning/katas/coretransforms/map/pardo/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Partition/Partition/src/org/apache/beam/learning/katas/coretransforms/partition/Task.java
create mode 100644 learning/katas/java/Core Transforms/Partition/Partition/task.html
create mode 100644 learning/katas/java/Core Transforms/Partition/Partition/test/org/apache/beam/learning/katas/coretransforms/partition/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Person.java
create mode 100644 learning/katas/java/Core Transforms/Side Input/Side Input/src/org/apache/beam/learning/katas/coretransforms/sideinput/Task.java
create mode 100644 learning/katas/java/Core Transforms/Side Input/Side Input/task.html
create mode 100644 learning/katas/java/Core Transforms/Side Input/Side Input/test/org/apache/beam/learning/katas/coretransforms/sideinput/TaskTest.java
create mode 100644 learning/katas/java/Core Transforms/Side Output/Side Output/src/org/apache/beam/learning/katas/coretransforms/sideoutput/Task.java
create mode 100644 learning/katas/java/Core Transforms/Side Output/Side Output/task.html
create mode 100644 learning/katas/java/Core Transforms/Side Output/Side Output/test/org/apache/beam/learning/katas/coretransforms/sideoutput/TaskTest.java
create mode 100644 learning/katas/java/Examples/Word Count/Word Count/src/org/apache/beam/learning/katas/examples/wordcount/Task.java
create mode 100644 learning/katas/java/Examples/Word Count/Word Count/task.html
create mode 100644 learning/katas/java/Examples/Word Count/Word Count/test/org/apache/beam/learning/katas/examples/wordcount/TaskTest.java
create mode 100644 learning/katas/java/IO/Built-in IOs/Built-in IOs/src/org/apache/beam/learning/katas/io/builtinios/Task.java
create mode 100644 learning/katas/java/IO/Built-in IOs/Built-in IOs/task.html
create mode 100644 learning/katas/java/IO/Built-in IOs/Built-in IOs/test/org/apache/beam/learning/katas/io/builtinios/TaskTest.java
create mode 100644 learning/katas/java/IO/TextIO/TextIO Read/countries.txt
create mode 100644 learning/katas/java/IO/TextIO/TextIO Read/src/org/apache/beam/learning/katas/io/textio/read/Task.java
create mode 100644 learning/katas/java/IO/TextIO/TextIO Read/task.html
create mode 100644 learning/katas/java/IO/TextIO/TextIO Read/test/org/apache/beam/learning/katas/io/textio/read/TaskTest.java
create mode 100644 learning/katas/java/Introduction/Hello Beam/Hello Beam/src/org/apache/beam/learning/katas/intro/hello/Task.java
create mode 100644 learning/katas/java/Introduction/Hello Beam/Hello Beam/task.html
create mode 100644 learning/katas/java/Introduction/Hello Beam/Hello Beam/test/org/apache/beam/learning/katas/intro/hello/TaskTest.java
create mode 100644 learning/katas/java/README.md
create mode 100644 learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/GenerateEvent.java
create mode 100644 learning/katas/java/Triggers/Early Triggers/Early Triggers/src/org/apache/beam/learning/katas/triggers/earlytriggers/Task.java
create mode 100644 learning/katas/java/Triggers/Early Triggers/Early Triggers/task.html
create mode 100644 learning/katas/java/Triggers/Early Triggers/Early Triggers/test/org/apache/beam/learning/katas/triggers/earlytriggers/TaskTest.java
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/GenerateEvent.java
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/src/org/apache/beam/learning/katas/triggers/eventtimetriggers/Task.java
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task.html
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/test/org/apache/beam/learning/katas/triggers/eventtimetriggers/TaskTest.java
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/GenerateEvent.java
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/src/org/apache/beam/learning/katas/triggers/windowaccummode/Task.java
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task.html
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/test/org/apache/beam/learning/katas/triggers/windowaccummode/TaskTest.java
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Event.java
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/ParDo/src/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/Task.java
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/ParDo/task.html
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/ParDo/test/org/apache/beam/learning/katas/windowing/addingtimestamp/pardo/TaskTest.java
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Event.java
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/src/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/Task.java
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task.html
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/test/org/apache/beam/learning/katas/windowing/addingtimestamp/withtimestamps/TaskTest.java
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/src/org/apache/beam/learning/katas/windowing/fixedwindow/Task.java
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task.html
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/TaskTest.java
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/test/org/apache/beam/learning/katas/windowing/fixedwindow/WindowedEvent.java
create mode 100644 learning/katas/java/build.gradle
create mode 100644 learning/katas/java/gradle/wrapper/gradle-wrapper.jar
create mode 100644 learning/katas/java/gradle/wrapper/gradle-wrapper.properties
create mode 100755 learning/katas/java/gradlew
copy gradlew.bat => learning/katas/java/gradlew.bat (100%)
create mode 100644 learning/katas/java/settings.gradle
create mode 100644 learning/katas/java/util/resources/log4j2.xml
create mode 100644 learning/katas/java/util/src/org/apache/beam/learning/katas/util/Log.java
create mode 100644 learning/katas/java/util/test/org/apache/beam/learning/katas/util/ContainsKvs.java
create mode 100644 learning/katas/java/util/test/org/apache/beam/learning/katas/util/KvMatcher.java
create mode 100644 learning/katas/python/.idea/study_project.xml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Count/task.html
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Count/task.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Count/tests.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Largest/task.html
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Largest/task.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Largest/tests.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Mean/task.html
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Mean/task.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Mean/tests.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Smallest/task.html
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Smallest/task.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Smallest/tests.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Sum/task.html
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Sum/task.py
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Sum/tests.py
create mode 100644 learning/katas/python/Common Transforms/Filter/Filter/task.html
create mode 100644 learning/katas/python/Common Transforms/Filter/Filter/task.py
create mode 100644 learning/katas/python/Common Transforms/Filter/Filter/tests.py
create mode 100644 learning/katas/python/Common Transforms/Filter/ParDo/task.html
create mode 100644 learning/katas/python/Common Transforms/Filter/ParDo/task.py
create mode 100644 learning/katas/python/Common Transforms/Filter/ParDo/tests.py
create mode 100644 learning/katas/python/Core Transforms/Branching/Branching/task.html
create mode 100644 learning/katas/python/Core Transforms/Branching/Branching/task.py
create mode 100644 learning/katas/python/Core Transforms/Branching/Branching/tests.py
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.html
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task.py
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/tests.py
create mode 100644 learning/katas/python/Core Transforms/Combine/Combine PerKey/task.html
create mode 100644 learning/katas/python/Core Transforms/Combine/Combine PerKey/task.py
create mode 100644 learning/katas/python/Core Transforms/Combine/Combine PerKey/tests.py
create mode 100644 learning/katas/python/Core Transforms/Combine/CombineFn/task.html
create mode 100644 learning/katas/python/Core Transforms/Combine/CombineFn/task.py
create mode 100644 learning/katas/python/Core Transforms/Combine/CombineFn/tests.py
create mode 100644 learning/katas/python/Core Transforms/Combine/Simple Function/task.html
create mode 100644 learning/katas/python/Core Transforms/Combine/Simple Function/task.py
create mode 100644 learning/katas/python/Core Transforms/Combine/Simple Function/tests.py
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.html
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task.py
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/Composite Transform/tests.py
create mode 100644 learning/katas/python/Core Transforms/Flatten/Flatten/task.html
create mode 100644 learning/katas/python/Core Transforms/Flatten/Flatten/task.py
create mode 100644 learning/katas/python/Core Transforms/Flatten/Flatten/tests.py
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.html
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task.py
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/GroupByKey/tests.py
create mode 100644 learning/katas/python/Core Transforms/Map/FlatMap/task.html
create mode 100644 learning/katas/python/Core Transforms/Map/FlatMap/task.py
create mode 100644 learning/katas/python/Core Transforms/Map/FlatMap/tests.py
create mode 100644 learning/katas/python/Core Transforms/Map/Map/task.html
create mode 100644 learning/katas/python/Core Transforms/Map/Map/task.py
create mode 100644 learning/katas/python/Core Transforms/Map/Map/tests.py
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.html
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo OneToMany/task.py
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo OneToMany/tests.py
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo/task.html
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo/task.py
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo/tests.py
create mode 100644 learning/katas/python/Core Transforms/Partition/Partition/task.html
create mode 100644 learning/katas/python/Core Transforms/Partition/Partition/task.py
create mode 100644 learning/katas/python/Core Transforms/Partition/Partition/tests.py
create mode 100644 learning/katas/python/Core Transforms/Side Input/Side Input/task.html
create mode 100644 learning/katas/python/Core Transforms/Side Input/Side Input/task.py
create mode 100644 learning/katas/python/Core Transforms/Side Input/Side Input/tests.py
create mode 100644 learning/katas/python/Core Transforms/Side Output/Side Output/task.html
create mode 100644 learning/katas/python/Core Transforms/Side Output/Side Output/task.py
create mode 100644 learning/katas/python/Core Transforms/Side Output/Side Output/tests.py
create mode 100644 learning/katas/python/Examples/Word Count/Word Count/task.html
create mode 100644 learning/katas/python/Examples/Word Count/Word Count/task.py
create mode 100644 learning/katas/python/Examples/Word Count/Word Count/tests.py
create mode 100644 learning/katas/python/IO/Built-in IOs/Built-in IOs/task.html
create mode 100644 learning/katas/python/IO/Built-in IOs/Built-in IOs/task.py
create mode 100644 learning/katas/python/IO/Built-in IOs/Built-in IOs/tests.py
create mode 100644 learning/katas/python/IO/TextIO/ReadFromText/countries.txt
create mode 100644 learning/katas/python/IO/TextIO/ReadFromText/task.html
create mode 100644 learning/katas/python/IO/TextIO/ReadFromText/task.py
create mode 100644 learning/katas/python/IO/TextIO/ReadFromText/tests.py
create mode 100644 learning/katas/python/Introduction/Hello Beam/Hello Beam/task.html
create mode 100644 learning/katas/python/Introduction/Hello Beam/Hello Beam/task.py
create mode 100644 learning/katas/python/Introduction/Hello Beam/Hello Beam/tests.py
create mode 100644 learning/katas/python/README.md
create mode 100644 learning/katas/python/log_elements.py
create mode 100644 learning/katas/python/requirements.txt
create mode 100644 learning/katas/python/test_helper.py
create mode 100644 model/pipeline/src/main/proto/schema.proto
create mode 100644 project-mappings
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderRegistrar.java
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/AvroCoderTranslator.java
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ReshuffleTranslation.java
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRenderer.java
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRenderer.java
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/renderer/package-info.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/ReshuffleTranslationTest.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/SchemaTranslationTest.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PipelineDotRendererTest.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/renderer/PortablePipelineDotRendererTest.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/BundleProcessor.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedBundle.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CommittedResult.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CompletionCallback.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternals.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectGroupByKey.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectMetrics.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectStateAndTimers.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTimerInternals.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DirectTransformExecutor.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/DisplayDataValidator.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EmptyInputProvider.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContext.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/EvaluationContextStepStateAndTimersProvider.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ExecutorServiceParallelExecutor.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupAlsoByWindowEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWindow.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PCollectionViewWriter.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PassthroughTransformEvaluator.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PipelineExecutor.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/PortableGraph.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/QuiescenceDriver.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/ReferenceRunner.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootInputProvider.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/RootProviderRegistry.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SourceShard.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/SplittableRemoteStageEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepAndKey.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepStateAndTimers.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/StepTransformResult.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluator.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformEvaluatorRegistry.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutor.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorService.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformExecutorServices.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/TransformResult.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/UncommittedBundle.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutor.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactory.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocation.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalService.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerService.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalService.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/artifact/package-info.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/PreparingJob.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServer.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobService.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/job/package-info.java
delete mode 100644 runners/direct-java/src/main/java/org/apache/beam/runners/direct/portable/package-info.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/BundleFactoryOutputReceiverFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CommittedResultTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/CopyOnAccessInMemoryStateInternalsTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectMetricsTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTimerInternalsTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/DirectTransformExecutorTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/EvaluationContextTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ExecutableGraphBuilder.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/FlattenEvaluatorFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/GroupByKeyOnlyEvaluatorFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImmutableListBundleFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ImpulseEvaluatorFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/MockClock.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/PortableGraphTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/ReferenceRunnerTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/RemoteStageEvaluatorFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/StepTransformResultTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/TransformExecutorServicesTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WatermarkCallbackExecutorTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/WindowEvaluatorFactoryTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalArtifactStagingLocationTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactRetrievalServiceTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/LocalFileSystemArtifactStagerServiceTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/artifact/UnsupportedArtifactRetrievalServiceTest.java
delete mode 100644 runners/direct-java/src/test/java/org/apache/beam/runners/direct/portable/job/ReferenceRunnerJobServiceTest.java
create mode 100644 runners/flink/1.5/build.gradle
copy runners/flink/{1.7 => 1.5}/job-server-container/build.gradle (100%)
create mode 100644 runners/flink/1.5/job-server/build.gradle
create mode 100644 runners/flink/1.5/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
rename runners/flink/{ => 1.5}/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java (100%)
rename runners/flink/{ => 1.5}/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java (100%)
rename runners/flink/{ => 1.5}/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java (100%)
rename runners/flink/{ => 1.5}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java (100%)
create mode 100644 runners/flink/1.8/build.gradle
copy runners/flink/{1.7 => 1.8}/job-server-container/build.gradle (100%)
create mode 100644 runners/flink/1.8/job-server/build.gradle
create mode 100644 runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
create mode 100644 runners/flink/1.8/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java
create mode 100644 runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java
create mode 100644 runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
create mode 100644 runners/flink/1.8/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java
delete mode 100644 runners/flink/build.gradle
delete mode 100644 runners/flink/job-server-container/build.gradle
delete mode 100644 runners/flink/job-server/build.gradle
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/types/InspectableByteArrayOutputStream.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/UnboundedSocketSource.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupCheckpointedOperator.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/state/KeyGroupRestoringOperator.java
create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkRunnerResultTest.java
create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/FlinkStreamingPipelineTranslatorTest.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/TimerReceiverFactory.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/state/InMemoryBagUserStateFactory.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/JobInvocationTest.java
create mode 100644 runners/jet-experimental/build.gradle
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunner.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/package-info.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java
create mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/package-info.java
create mode 100644 runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java
create mode 100644 runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java
create mode 100644 runners/jet-experimental/src/test/java/org/apache/beam/runners/jet/TestStreamP.java
delete mode 100644 runners/reference/job-server/build.gradle
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineOptions.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/PipelineDotRenderer.java
delete mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/PortablePipelineDotRenderer.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkAbstractCombineFn.java
create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkCombineFn.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkGlobalCombineFn.java
delete mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkKeyedCombineFn.java
create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/util/SparkCompat.java
create mode 100644 runners/spark/src/test/java/org/apache/beam/runners/spark/translation/SparkCombineFnTest.java
create mode 100644 sdks/go/pkg/beam/combine_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/datasource_test.go
create mode 100644 sdks/go/pkg/beam/runners/spark/spark.go
create mode 100644 sdks/go/pkg/beam/runners/vet/testpipeline/functions.go
create mode 100644 sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.go
create mode 100644 sdks/go/pkg/beam/runners/vet/testpipeline/testpipeline.shims.go
create mode 100644 sdks/go/pkg/beam/runners/vet/vet.go
create mode 100644 sdks/go/pkg/beam/runners/vet/vet_test.go
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/harness/JvmInitializer.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/harness/package-info.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/ShardingFunction.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/TextRowCountEstimator.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/AddFields.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/DropFields.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/transforms/RenameFields.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesKms.java
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlog.java
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Backlogs.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Sizes.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/coders/FloatCoderTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextRowCountEstimatorTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/AddFieldsTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/DropFieldsTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/transforms/RenameFieldsTest.java
rename sdks/java/core/src/test/{avro => java}/org/apache/beam/sdk/state/StateContextsTest.java (100%)
create mode 100644 sdks/java/core/src/test/resources/META-INF/services/org.apache.beam.sdk.util.common.ReflectHelpersTest$FakeService
create mode 100644 sdks/java/extensions/euphoria/src/test/java/org/apache/beam/sdk/extensions/euphoria/core/client/util/SumsTest.java
create mode 100644 sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/HadoopExternalSorter.java
create mode 100644 sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeExternalSorter.java
create mode 100644 sdks/java/extensions/sorter/src/main/java/org/apache/beam/sdk/extensions/sorter/NativeFileSorter.java
create mode 100644 sdks/java/extensions/sorter/src/test/java/org/apache/beam/sdk/extensions/sorter/ExternalSorterBenchmark.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/BeamTableStatistics.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinAssociateRule.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinPushThroughJoinRule.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinRelOptRuleCall.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordReadConverter.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableProvider.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/package-info.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/GenerateSequenceTableProvider.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/seqgen/package-info.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamIOSourceRelTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/JoinReorderingTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryRowCountIT.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTable.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryTestTableProvider.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/GenericRecordToRowTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/parquet/ParquetTableReadTest.java
create mode 100644 sdks/java/extensions/sql/src/test/resources/users.parquet
create mode 100644 sdks/java/fn-execution/src/main/java/org/apache/beam/sdk/fn/JvmInitializers.java
create mode 100644 sdks/java/fn-execution/src/test/java/org/apache/beam/sdk/fn/JvmInitializersTest.java
create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoder.java
create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderProviderRegistrar.java
create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProvider.java
create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/BasicDynamoDBProvider.java
create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIO.java
create mode 100644 sdks/java/io/amazon-web-services/src/main/java/org/apache/beam/sdk/io/aws/dynamodb/package-info.java
create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AttributeValueCoderTest.java
create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/AwsClientsProviderMock.java
create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTest.java
create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/dynamodb/DynamoDBIOTestHelper.java
create mode 100644 sdks/java/io/amazon-web-services2/build.gradle
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoder.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/BasicDynamoDbClientProvider.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIO.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProvider.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/dynamodb/package-info.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsModule.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsOptions.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/AwsPipelineOptionsRegistrar.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/options/package-info.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/AttributeValueCoderTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDBIOTestHelper.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/dynamodb/DynamoDbClientProviderMock.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/options/AwsModuleTest.java
delete mode 100644 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticSearchIOTestUtils.java
create mode 100644 sdks/java/io/elasticsearch-tests/elasticsearch-tests-common/src/test/java/org/apache/beam/sdk/io/elasticsearch/ElasticsearchIOTestUtils.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryQuerySourceDef.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySchemaRetrievalException.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQuerySourceDef.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTableSourceDef.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/TableDestinationCoderV3.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResult.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableWriteResultCoder.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryTimePartitioningClusteringIT.java
create mode 100644 sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/HCatalogUtils.java
create mode 100644 sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionPollerFn.java
create mode 100644 sdks/java/io/hcatalog/src/main/java/org/apache/beam/sdk/io/hcatalog/PartitionReaderFn.java
create mode 100644 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/BeamSchemaInferenceException.java
create mode 100644 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java
create mode 100644 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/LogicalTypes.java
create mode 100644 sdks/java/io/jdbc/src/main/java/org/apache/beam/sdk/io/jdbc/SchemaUtil.java
create mode 100644 sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java
create mode 100644 sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/RowWithSchema.java
create mode 100644 sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/SchemaUtilTest.java
create mode 100644 sdks/java/io/kafka/src/test/java/org/apache/beam/sdk/io/kafka/KafkaIOIT.java
delete mode 100644 sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/KinesisWatermark.java
create mode 100644 sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkParameters.java
create mode 100644 sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicy.java
create mode 100644 sdks/java/io/kinesis/src/main/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicyFactory.java
delete mode 100644 sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/KinesisWatermarkTest.java
create mode 100644 sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/WatermarkPolicyTest.java
delete mode 100644 sdks/java/javadoc/apiclient-docs/package-list
delete mode 100644 sdks/java/javadoc/avro-docs/package-list
delete mode 100644 sdks/java/javadoc/bq-docs/package-list
delete mode 100644 sdks/java/javadoc/datastore-docs/package-list
delete mode 100644 sdks/java/javadoc/guava-docs/package-list
delete mode 100644 sdks/java/javadoc/hamcrest-docs/package-list
delete mode 100644 sdks/java/javadoc/jackson-annotations-docs/package-list
delete mode 100644 sdks/java/javadoc/jackson-databind-docs/package-list
delete mode 100644 sdks/java/javadoc/joda-docs/package-list
delete mode 100644 sdks/java/javadoc/junit-docs/package-list
delete mode 100644 sdks/java/javadoc/oauth-docs/package-list
create mode 100644 sdks/python/apache_beam/coders/avro_coder.py
create mode 100644 sdks/python/apache_beam/coders/avro_coder_test.py
copy sdks/python/apache_beam/{io/gcp/tests => examples/snippets/transforms}/__init__.py (100%)
copy sdks/python/apache_beam/{io/gcp/tests => examples/snippets/transforms/element_wise}/__init__.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/map.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/values.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/util.py
create mode 100644 sdks/python/apache_beam/examples/snippets/util_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_read_it_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_read_perf_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_write_it_test.py
create mode 100644 sdks/python/apache_beam/io/gcp/bigquery_write_perf_test.py
create mode 100644 sdks/python/apache_beam/io/iobase_test.py
create mode 100644 sdks/python/apache_beam/io/mongodbio.py
create mode 100644 sdks/python/apache_beam/io/mongodbio_it_test.py
create mode 100644 sdks/python/apache_beam/io/mongodbio_test.py
create mode 100644 sdks/python/apache_beam/io/utils_test.py
create mode 100644 sdks/python/apache_beam/runners/portability/flink_runner.py
delete mode 100644 sdks/python/apache_beam/runners/portability/java_reference_runner_test.py
create mode 100644 sdks/python/apache_beam/transforms/dofn_lifecycle_test.py
create mode 100644 sdks/python/apache_beam/transforms/stats.py
create mode 100644 sdks/python/apache_beam/transforms/stats_test.py
delete mode 100644 sdks/python/test-suites/dataflow/build.gradle
create mode 100644 sdks/python/test-suites/dataflow/py2/build.gradle
create mode 100644 sdks/python/test-suites/dataflow/py37/build.gradle
create mode 100644 sdks/python/test-suites/direct/py2/build.gradle
create mode 100644 sdks/python/test-suites/direct/py37/build.gradle
create mode 100644 sdks/python/test-suites/portable/py2/build.gradle
create mode 100644 sdks/python/test-suites/portable/py35/build.gradle
create mode 100644 sdks/python/test-suites/tox/py2/build.gradle
create mode 100644 vendor/bytebuddy-1_9_3/build.gradle
delete mode 100644 vendor/grpc-1_13_1/build.gradle
create mode 100644 vendor/grpc-1_21_0/build.gradle
delete mode 100644 vendor/guava-20_0/build.gradle
create mode 100644 vendor/guava-26_0-jre/build.gradle
create mode 100644 website/src/_posts/2019-03-18-beam-summit-site.md
create mode 100644 website/src/_posts/2019-05-01-adding-data-sources-to-sql.md
create mode 100644 website/src/_posts/2019-05-11-beam-summit-europe-2019.md
create mode 100644 website/src/_posts/2019-05-22-beam-2.13.0.md
create mode 100644 website/src/_posts/2019-05-30-beam-kata-release.md
create mode 100644 website/src/_posts/2019-06-11-looping-timers.md
delete mode 100644 website/src/documentation/dsls/sql/aggregate-functions.md
create mode 100644 website/src/documentation/dsls/sql/calcite/aggregate-functions.md
create mode 100644 website/src/documentation/dsls/sql/calcite/data-types.md
create mode 100644 website/src/documentation/dsls/sql/calcite/lexical-structure.md
create mode 100644 website/src/documentation/dsls/sql/calcite/overview.md
create mode 100644 website/src/documentation/dsls/sql/calcite/query-syntax.md
create mode 100644 website/src/documentation/dsls/sql/calcite/scalar-functions.md
delete mode 100644 website/src/documentation/dsls/sql/create-external-table.md
delete mode 100644 website/src/documentation/dsls/sql/data-types.md
create mode 100644 website/src/documentation/dsls/sql/extensions/create-external-table.md
create mode 100644 website/src/documentation/dsls/sql/extensions/joins.md
create mode 100644 website/src/documentation/dsls/sql/extensions/set.md
create mode 100644 website/src/documentation/dsls/sql/extensions/user-defined-functions.md
create mode 100644 website/src/documentation/dsls/sql/extensions/windowing-and-triggering.md
delete mode 100644 website/src/documentation/dsls/sql/joins.md
delete mode 100644 website/src/documentation/dsls/sql/lexical-structure.md
delete mode 100644 website/src/documentation/dsls/sql/scalar-functions.md
delete mode 100644 website/src/documentation/dsls/sql/select.md
delete mode 100644 website/src/documentation/dsls/sql/set.md
delete mode 100644 website/src/documentation/dsls/sql/user-defined-functions.md
delete mode 100644 website/src/documentation/dsls/sql/windowing-and-triggering.md
create mode 100644 website/src/documentation/patterns/custom-io-patterns.md
create mode 100644 website/src/documentation/patterns/file-processing-patterns.md
create mode 100644 website/src/documentation/patterns/overview.md
create mode 100644 website/src/documentation/patterns/pipeline-option-patterns.md
create mode 100644 website/src/documentation/patterns/side-input-patterns.md
create mode 100644 website/src/documentation/runners/jet.md
create mode 100644 website/src/documentation/transforms/java/aggregation/approximatequantiles.md
create mode 100644 website/src/documentation/transforms/java/aggregation/approximateunique.md
create mode 100644 website/src/documentation/transforms/java/aggregation/cogroupbykey.md
create mode 100644 website/src/documentation/transforms/java/aggregation/combine.md
create mode 100644 website/src/documentation/transforms/java/aggregation/combinewithcontext.md
create mode 100644 website/src/documentation/transforms/java/aggregation/count.md
create mode 100644 website/src/documentation/transforms/java/aggregation/distinct.md
create mode 100644 website/src/documentation/transforms/java/aggregation/groupbykey.md
create mode 100644 website/src/documentation/transforms/java/aggregation/groupintobatches.md
create mode 100644 website/src/documentation/transforms/java/aggregation/latest.md
create mode 100644 website/src/documentation/transforms/java/aggregation/max.md
create mode 100644 website/src/documentation/transforms/java/aggregation/mean.md
create mode 100644 website/src/documentation/transforms/java/aggregation/min.md
create mode 100644 website/src/documentation/transforms/java/aggregation/sample.md
create mode 100644 website/src/documentation/transforms/java/aggregation/sum.md
create mode 100644 website/src/documentation/transforms/java/aggregation/top.md
create mode 100644 website/src/documentation/transforms/java/element-wise/filter.md
create mode 100644 website/src/documentation/transforms/java/element-wise/flatmapelements.md
create mode 100644 website/src/documentation/transforms/java/element-wise/keys.md
create mode 100644 website/src/documentation/transforms/java/element-wise/kvswap.md
create mode 100644 website/src/documentation/transforms/java/element-wise/mapelements.md
create mode 100644 website/src/documentation/transforms/java/element-wise/pardo.md
create mode 100644 website/src/documentation/transforms/java/element-wise/partition.md
create mode 100644 website/src/documentation/transforms/java/element-wise/regex.md
create mode 100644 website/src/documentation/transforms/java/element-wise/reify.md
create mode 100644 website/src/documentation/transforms/java/element-wise/tostring.md
create mode 100644 website/src/documentation/transforms/java/element-wise/values.md
create mode 100644 website/src/documentation/transforms/java/element-wise/withkeys.md
create mode 100644 website/src/documentation/transforms/java/element-wise/withtimestamps.md
create mode 100644 website/src/documentation/transforms/java/index.md
create mode 100644 website/src/documentation/transforms/java/other/create.md
create mode 100644 website/src/documentation/transforms/java/other/flatten.md
create mode 100644 website/src/documentation/transforms/java/other/passert.md
create mode 100644 website/src/documentation/transforms/java/other/view.md
create mode 100644 website/src/documentation/transforms/java/other/window.md
create mode 100644 website/src/documentation/transforms/python/aggregation/approximatequantiles.md
create mode 100644 website/src/documentation/transforms/python/aggregation/approximateunique.md
create mode 100644 website/src/documentation/transforms/python/aggregation/cogroupbykey.md
create mode 100644 website/src/documentation/transforms/python/aggregation/combineglobally.md
create mode 100644 website/src/documentation/transforms/python/aggregation/combinewithcontext.md
create mode 100644 website/src/documentation/transforms/python/aggregation/count.md
create mode 100644 website/src/documentation/transforms/python/aggregation/distinct.md
create mode 100644 website/src/documentation/transforms/python/aggregation/groupbykey.md
create mode 100644 website/src/documentation/transforms/python/aggregation/groupintobatches.md
create mode 100644 website/src/documentation/transforms/python/aggregation/latest.md
create mode 100644 website/src/documentation/transforms/python/aggregation/max.md
create mode 100644 website/src/documentation/transforms/python/aggregation/mean.md
create mode 100644 website/src/documentation/transforms/python/aggregation/min.md
create mode 100644 website/src/documentation/transforms/python/aggregation/sample.md
create mode 100644 website/src/documentation/transforms/python/aggregation/sum.md
create mode 100644 website/src/documentation/transforms/python/aggregation/top.md
create mode 100644 website/src/documentation/transforms/python/element-wise/filter.md
create mode 100644 website/src/documentation/transforms/python/element-wise/flatmap.md
create mode 100644 website/src/documentation/transforms/python/element-wise/keys.md
create mode 100644 website/src/documentation/transforms/python/element-wise/kvswap.md
create mode 100644 website/src/documentation/transforms/python/element-wise/map.md
create mode 100644 website/src/documentation/transforms/python/element-wise/pardo.md
create mode 100644 website/src/documentation/transforms/python/element-wise/partition.md
create mode 100644 website/src/documentation/transforms/python/element-wise/regex.md
create mode 100644 website/src/documentation/transforms/python/element-wise/reify.md
create mode 100644 website/src/documentation/transforms/python/element-wise/tostring.md
create mode 100644 website/src/documentation/transforms/python/element-wise/values.md
create mode 100644 website/src/documentation/transforms/python/element-wise/withkeys.md
create mode 100644 website/src/documentation/transforms/python/element-wise/withtimestamps.md
create mode 100644 website/src/documentation/transforms/python/index.md
create mode 100644 website/src/documentation/transforms/python/other/create.md
create mode 100644 website/src/documentation/transforms/python/other/flatten.md
create mode 100644 website/src/documentation/transforms/python/other/passert.md
create mode 100644 website/src/documentation/transforms/python/other/reshuffle.md
create mode 100644 website/src/documentation/transforms/python/other/view.md
create mode 100644 website/src/documentation/transforms/python/other/windowinto.md
create mode 100644 website/src/images/blog/beam-kata/beam-kata-intellij-edu-1.png
create mode 100644 website/src/images/blog/beam-kata/beam-kata-intellij-edu-2.png
create mode 100644 website/src/images/blog/beam-kata/beam-kata-pycharm-edu-1.png
create mode 100644 website/src/images/blog/beam-kata/beam-kata-pycharm-edu-2.png
create mode 100644 website/src/images/logos/runners/jet.png