You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ec...@apache.org on 2019/10/28 14:01:24 UTC
[beam] branch spark-runner_structured-streaming updated (37ae68a ->
9c4a6ac)
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 37ae68a [BEAM-8470] Apply spotless
discard 3417103 [BEAM-8470] Add a jenkins job for validates runner tests in the new spark runner
discard a0e6ca4 [BEAM-8470] Remove Encoders based on kryo now that we call Beam coders in the runner
discard 9e7f118 [BEAM-8470] Fix: Remove generic hack of using object. Use actual Coder encodedType in Encoders
discard 2e1ed4b [BEAM-8470] Remove unneeded cast
discard fbb8355 [BEAM-8470] Use beam encoders also in the output of the source translation
discard f180b4d [BEAM-8470] Apply spotless, fix typo and javadoc
discard de92516 [BEAM-8470] Apply new Encoders to Pardo. Replace Tuple2Coder with MultiOutputCoder to deal with multiple output to use in Spark Encoder for DoFnRunner
discard fbda17a [BEAM-8470] Apply new Encoders to GroupByKey
discard f1f2163 [BEAM-8470] Create a Tuple2Coder to encode scala tuple2
discard 54fd760 [BEAM-8470] Apply new Encoders to AggregatorCombiner
discard a7629eb [BEAM-8470] Apply new Encoders to Window assign translation
discard 163b157 [BEAM-8470] Ignore long time failing test: SparkMetricsSinkTest
discard cb82036 [BEAM-8470] Improve performance of source: the mapper already calls windowedValueCoder.decode, no need to call it also in the Spark encoder
discard 7e5b6df [BEAM-8470] Apply new Encoders to Read source
discard 5ebc1e9 [BEAM-8470] Apply new Encoders to CombinePerKey
discard e8463ce [BEAM-8470] Catch Exception instead of IOException because some coders to not throw Exceptions at all (e.g.VoidCoder)
discard 2daed76 [BEAM-8470] Put Encoders expressions serializable
discard 4055d11 [BEAM-8470] Wrap exceptions in UserCoderExceptions
discard ba33722 [BEAM-8470] Apply spotless and checkstyle and add javadocs
discard 6b31168 [BEAM-8470] Add an assert of equality in the encoders test
discard dc5b243 [BEAM-8470] Fix generated code: uniform exceptions catching, fix parenthesis and variable declarations
discard 83421c2 [BEAM-8470] Fix equal and hashcode
discard 72ba1ea [BEAM-8470] Remove example code
discard cfff6f7 [BEAM-8470] Remove lazy init of beam coder because there is no generic way on instanciating a beam coder
discard e2a134c [BEAM-8470] Fix beam coder lazy init using reflexion: use .clas + try catch + cast
discard 44274ca [BEAM-8470] Fix getting the output value in code generation
discard 482bcf6 [BEAM-8470] Fix ExpressionEncoder generated code: typos, try catch, fqcn
discard f1a555e [BEAM-8470] Fix warning in coder construction by reflexion
discard 8e341a1 [BEAM-8470] Lazy init coder because coder instance cannot be interpolated by catalyst
discard cca4034 [BEAM-8470] Fix code generation in Beam coder wrapper
discard 3a87d37 [BEAM-8470] Add a simple spark native test to test Beam coders wrapping into Spark Encoders
discard a0706d7 [BEAM-8470] Conform to spark ExpressionEncoders: pass classTags, implement scala Product, pass children from within the ExpressionEncoder, fix visibilities
discard 02a0f01 [BEAM-8470] Fix scala Product in Encoders to avoid StackEverflow
discard 69aebb1 [BEAM-8470] type erasure: spark encoders require a Class<T>, pass Object and cast to Class<T>
discard 750037f [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: deserialization part
discard 5ce206b [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: serialization part
discard c5b8799 [BEAM-8470] Use "sparkMaster" in local mode to obtain number of shuffle partitions + spotless apply
discard a8e0ad9 [BEAM-8470] Improve Pardo translation performance: avoid calling a filter transform when there is only one output tag
discard 134ee35 [BEAM-8470] After testing performance and correctness, launch pipeline with dataset.foreach(). Make both test mode and production mode use foreach for uniformity. Move dataset print as a utility method
discard f2388dc [BEAM-8470] Remove no more needed AggregatorCombinerPerKey (there is only AggregatorCombiner)
discard 65c8457 [BEAM-8470] fixup! Add PipelineResults to Spark structured streaming.
discard 7f71572 [BEAM-8470] Print number of leaf datasets
discard 8c91c11 [BEAM-8470] Add spark execution plans extended debug messages.
discard 7d7503a [BEAM-8470] Update log4j configuration
discard 3391f8d [BEAM-8470] Add PipelineResults to Spark structured streaming.
discard 745ab6e [BEAM-8470] Make spotless happy
discard a004a56 [BEAM-8470] Added metrics sinks and tests
discard b59d8c5 [BEAM-8470] Persist all output Dataset if there are multiple outputs in pipeline Enabled Use*Metrics tests
discard 37dcb9a [BEAM-8470] Add a test to check that CombineGlobally preserves windowing
discard a6d0e58 [BEAM-8470] Fix accumulators initialization in Combine that prevented CombineGlobally to work.
discard a7f04ed [BEAM-8470] Fix javadoc
discard dc1abf5 [BEAM-8470] Add setEnableSparkMetricSinks() method
discard 35fef0f [BEAM-8470] Add missing dependencies to run Spark Structured Streaming Runner on Nexmark
discard 8215482 [BEAM-8470] Add metrics support in DoFn
discard b12e878 [BEAM-8470] Ignore for now not working test testCombineGlobally
discard e205425 [BEAM-8470] Add a test that combine per key preserves windowing
discard 85fe2d0 [BEAM-8470] Clean groupByKeyTest
discard 98bc7a4 [BEAM-8470] add comment in combine globally test
discard c178172 [BEAM-8470] Fixed immutable list bug
discard b19b792 [BEAM-8470] Fix javadoc of AggregatorCombiner
discard d6683ad [BEAM-8470] Clean not more needed WindowingHelpers
discard 96b147e [BEAM-8470] Clean not more needed RowHelpers
discard 5a7bbf5 [BEAM-8470] Clean no more needed KVHelpers
discard e7d3283 [BEAM-8470] Now that there is only Combine.PerKey translation, make only one Aggregator
discard c0e52c2 [BEAM-8470] Remove CombineGlobally translation because it is less performant than the beam sdk one (key + combinePerKey.withHotkeyFanout)
discard 5427856 [BEAM-8470] Remove the mapPartition that adds a key per partition because otherwise spark will reduce values per key instead of globally
discard d093cad [BEAM-8470] Fix bug in the window merging logic
discard 6f47514 [BEAM-8470] Fix wrong encoder in combineGlobally GBK
discard 26f778e [BEAM-8470] Fix case when a window does not merge into any other window
discard a460052 [BEAM-8470] Apply a groupByKey avoids for some reason that the spark structured streaming fmwk casts data to Row which makes it impossible to deserialize without the coder shipped into the data. For performance reasons (avoid memory consumption and having to deserialize), we do not ship coder + data. Also add a mapparitions before GBK to avoid shuffling
discard 7237dca [BEAM-8470] Revert extractKey while combinePerKey is not done (so that it compiles)
discard 0bdc83c [BEAM-8470] Fix encoder in combine call
discard a8ca39f [BEAM-8470] Implement merge accumulators part of CombineGlobally translation with windowing
discard 7222d49 [BEAM-8470] Output data after combine
discard 4c2585b [BEAM-8470] Implement reduce part of CombineGlobally translation with windowing
discard 09bc7a1 [BEAM-8470] Fix comment about schemas
discard dc4dda7 [BEAM-8470] Update KVHelpers.extractKey() to deal with WindowedValue and update GBK and CPK
discard 22abb05 [BEAM-8470] Add TODO in Combine translations
discard c5970f2 [BEAM-8470] Add a test that GBK preserves windowing
discard 26fcb28 [BEAM-8470] Improve visibility of debug messages
discard d52ef97 [BEAM-8470] re-enable reduceFnRunner timers for output
discard 211cb98 [BEAM-8470] Re-code GroupByKeyTranslatorBatch to conserve windowing instead of unwindowing/windowing(GlobalWindow): simplify code, use ReduceFnRunner to merge the windows
discard 0705e50 [BEAM-8470] Add comment about checkpoint mark
discard 2c49efc [BEAM-8470] Update windowAssignTest
discard 355e95d [BEAM-8470] Put back batch/simpleSourceTest.testBoundedSource
discard f780659 [BEAM-8470] Consider null object case on RowHelpers, fixes empty side inputs tests.
discard 386080c [BEAM-8470] Pass transform based doFnSchemaInformation in ParDo translation
discard 48eaf9d [BEAM-8470] Fixes ParDo not calling setup and not tearing down if exception on startBundle
discard 7891116 [BEAM-8470] Limit the number of partitions to make tests go 300% faster
discard 9b7986a [BEAM-8470] Enable batch Validates Runner tests for Structured Streaming Runner
discard 0c082b89 [BEAM-8470] Apply Spotless
discard 0031ef9 [BEAM-8470] Update javadoc
discard 2427002 [BEAM-8470] implement source.stop
discard cbe80d4 [BEAM-8470] Ignore spark offsets (cf javadoc)
discard c9b3e51 [BEAM-8470] Use PAssert in Spark Structured Streaming transform tests
discard 5e4d878 [BEAM-8470] Rename SparkPipelineResult to SparkStructuredStreamingPipelineResult This is done to avoid an eventual collision with the one in SparkRunner. However this cannot happen at this moment because it is package private, so it is also done for consistency.
discard f2dd748 [BEAM-8470] Add SparkStructuredStreamingPipelineOptions and SparkCommonPipelineOptions - SparkStructuredStreamingPipelineOptions was added to have the new runner rely only on its specific options.
discard fecc40b [BEAM-8470] Fix logging levels in Spark Structured Streaming translation
discard 102f6fc [BEAM-8470] Fix spotless issues after rebase
discard 53acbda [BEAM-8470] Pass doFnSchemaInformation to ParDo batch translation
discard 1c82b5e [BEAM-8470] Fix non-vendored imports from Spark Streaming Runner classes
discard f106a04 [BEAM-8470] Merge Spark Structured Streaming runner into main Spark module. Remove spark-structured-streaming module.Rename Runner to SparkStructuredStreamingRunner. Remove specific PipelineOotions and Registrar for Structured Streaming Runner
discard 673731f [BEAM-8470] Fix access level issues, typos and modernize code to Java 8 style
discard 52e8689 [BEAM-8470] Disable never ending test SimpleSourceTest.testUnboundedSource
discard 2070278 [BEAM-8470] Apply spotless and fix spotbugs warnings
discard 45cd090 [BEAM-8470] Deal with checkpoint and offset based read
discard e260811 [BEAM-8470] Continue impl of offsets for streaming source
discard 850f56a [BEAM-8470] Clean streaming source
discard 79f22a8 [BEAM-8470] Clean unneeded 0 arg constructor in batch source
discard 9f2caa8 [BEAM-8470] Specify checkpointLocation at the pipeline start
discard 4ab7b03 [BEAM-8470] Add source streaming test
discard 2470f3e [BEAM-8470] Add transformators registry in PipelineTranslatorStreaming
discard 0863bf5 [BEAM-8470] Add a TODO on spark output modes
discard f20a878 [BEAM-8470] Implement first streaming source
discard f5cbbda [BEAM-8470] Add streaming source initialisation
discard ff2ed77 [BEAM-8470] And unchecked warning suppression
discard 4c10a48 [BEAM-8470] Added TODO comment for ReshuffleTranslatorBatch
discard c4b8e86 [BEAM-8470] Added using CachedSideInputReader
discard 0f1c9ff [BEAM-8470] Don't use Reshuffle translation
discard d51e934 [BEAM-8470] Fix CheckStyle violations
discard 95564cd [BEAM-8470] Added SideInput support
discard cb0e79a [BEAM-8470] Fix javadoc
discard 75dc161 [BEAM-8470] Implement WindowAssignTest
discard 0c53291 [BEAM-8470] Implement WindowAssignTranslatorBatch
discard a781ec1 [BEAM-8470] Cleaning
discard 82131f5 [BEAM-8470] Fix encoder bug in combinePerkey
discard c3b4686 [BEAM-8470] Add explanation about receiving a Row as input in the combiner
discard 4bb19ce [BEAM-8470] Use more generic Row instead of GenericRowWithSchema
discard f94f5ca [BEAM-8470] Fix combine. For unknown reason GenericRowWithSchema is used as input of combine so extract its content to be able to proceed
discard 3499784 [BEAM-8470] Update test with Long
discard cd19577 [BEAM-8470] Fix various type checking issues in Combine.Globally
discard 2aff552 [BEAM-8470] Get back to classes in translators resolution because URNs cannot translate Combine.Globally
discard 071aab5 [BEAM-8470] Cleaning
discard 4f670a5 [BEAM-8470] Add CombineGlobally translation to avoid translating Combine.perKey as a composite transform based on Combine.PerKey (which uses low perf GBK)
discard f416e20 [BEAM-8470] Introduce RowHelpers
discard ca25210 [BEAM-8470] Add combinePerKey and CombineGlobally tests
discard 35f7128 [BEAM-8470] Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless
discard f277be2 [BEAM-8470] Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey
discard e282801 [BEAM-8470] Improve type checking of Tuple2 encoder
discard cb666d3 [BEAM-8470] First version of combinePerKey
discard 126ee79 [BEAM-8470] Extract binary schema creation in a helper class
discard d24bc86 [BEAM-8470] Fix getSideInputs
discard 65dcd0b [BEAM-8470] Generalize the use of SerializablePipelineOptions in place of (not serializable) PipelineOptions
discard 222ce45 [BEAM-8470] Rename SparkDoFnFilterFunction to DoFnFilterFunction for consistency
discard a9eed5b [BEAM-8470] Add a test for the most simple possible Combine
discard b136699 [BEAM-8470] Added "testTwoPardoInRow"
discard 9275c82 [BEAM-8470] Fix for test elements container in GroupByKeyTest
discard b7ec102 [BEAM-8470] Rename SparkOutputManager for consistency
discard 359a6b0 [BEAM-8470] Fix kryo issue in GBK translator with a workaround
discard 6657d7d [BEAM-8470] Simplify logic of ParDo translator
discard 313bc66 [BEAM-8470] Don't use deprecated sideInput.getWindowingStrategyInternal()
discard 8286afc [BEAM-8470] Rename SparkSideInputReader class and rename pruneOutput() to pruneOutputFilteredByTag()
discard a048341 [BEAM-8470] Fixed Javadoc error
discard 2bea926 [BEAM-8470] Apply spotless
discard 0cd30d1 [BEAM-8470] Fix Encoders: create an Encoder for every manipulated type to avoid Spark fallback to genericRowWithSchema and cast to allow having Encoders with generic types such as WindowedValue<T> and get the type checking back
discard 728dc73 [BEAM-8470] Fail in case of having SideInouts or State/Timers
discard f7a008c [BEAM-8470] Add ComplexSourceTest
discard c8f2078 [BEAM-8470] Remove no more needed putDatasetRaw
discard 61bf40c [BEAM-8470] Port latest changes of ReadSourceTranslatorBatch to ReadSourceTranslatorStreaming
discard 997043d [BEAM-8470] Fix type checking with Encoder of WindowedValue<T>
discard 990e220 [BEAM-8470] Add comments and TODO to GroupByKeyTranslatorBatch
discard bdeb934 [BEAM-8470] Add GroupByKeyTest
discard bd385ff [BEAM-8470] Clean
discard 7fd2de1 [BEAM-8470] Address minor review notes
discard f152deb [BEAM-8470] Add ParDoTest
discard fc2239d [BEAM-8470] Clean
discard e63d794 [BEAM-8470] Fix split bug
discard 7e5399e [BEAM-8470] Remove bundleSize parameter and always use spark default parallelism
discard 1960559 [BEAM-8470] Cleaning
discard 067e756 [BEAM-8470] Fix testMode output to comply with new binary schema
discard 730eed3 [BEAM-8470] Fix errorprone
discard 176342f [BEAM-8470] Comment schema choices
discard 30cbbf4 [BEAM-8470] Serialize windowedValue to byte[] in source to be able to specify a binary dataset schema and deserialize windowedValue from Row to get a dataset<WindowedValue>
discard 5dc598a [BEAM-8470] First attempt for ParDo primitive implementation
discard 068f63d [BEAM-8470] Add flatten test
discard 3dc7d95 [BEAM-8470] Enable gradle build scan
discard 3004db7 [BEAM-8470] Enable test mode
discard 833df0e [BEAM-8470] Put all transform translators Serializable
discard bb7db53 [BEAM-8470] Simplify beam reader creation as it created once the source as already been partitioned
discard baea877 [BEAM-8470] Fix SourceTest
discard 63aa493 [BEAM-8470] Move SourceTest to same package as tested class
discard f5a0ba7 [BEAM-8470] Add serialization test
discard 1dc0352 [BEAM-8470] Add SerializationDebugger
discard b3796a2 [BEAM-8470] Fix serialization issues
discard 1102280 [BEAM-8470] Clean unneeded fields in DatasetReader
discard 244459f [BEAM-8470] improve readability of options passing to the source
discard 285aab4 [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset
discard 344f69e [BEAM-8470] Refactor SourceTest to a UTest instaed of a main
discard 8b232df [BEAM-8470] Checkstyle and Findbugs
discard 9537add [BEAM-8470] Clean
discard e9b0488 [BEAM-8470] Add empty 0-arg constructor for mock source
discard aeee309 [BEAM-8470] Add a dummy schema for reader
discard adfd237 [BEAM-8470] Apply spotless and fix checkstyle
discard dc615f0 [BEAM-8470] Use new PipelineOptionsSerializationUtils
discard 6cac8b2 [BEAM-8470] Add missing 0-arg public constructor
discard db036ca [BEAM-8470] Wire real SourceTransform and not mock and update the test
discard c569f1b [BEAM-8470] Refactor DatasetSource fields
discard 3368a9e [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
discard 68f7230 [BEAM-8470] Move Source and translator mocks to a mock package.
discard 4a59b09 [BEAM-8470] Add ReadSourceTranslatorStreaming
discard ae75196 [BEAM-8470] Clean
discard ee0ea0e [BEAM-8470] Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
discard 8051fac [BEAM-8470] Split batch and streaming sources and translators
discard 5385a8f [BEAM-8470] Run pipeline in batch mode or in streaming mode
discard 26b2a91 [BEAM-8470] Move DatasetSourceMock to proper batch mode
discard 4480578 [BEAM-8470] clean deps
discard 756eec3 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary)
discard be8344e [BEAM-8470] fix mock, wire mock in translators and create a main test.
discard 2a960dd [BEAM-8470] Add source mocks
discard c863dca [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table
discard 1970760 [BEAM-8470] Improve type enforcement in ReadSourceTranslator
discard 6bb32a5 [BEAM-8470] Improve exception flow
discard cb9bb99 [BEAM-8470] start source instanciation
discard eb7c77e [BEAM-8470] Apply spotless
discard 7901d73 [BEAM-8470] update TODO
discard cce30c9 [BEAM-8470] Implement read transform
discard 6965842 [BEAM-8470] Use Iterators.transform() to return Iterable
discard 129e95f [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation
discard ad26cec [BEAM-8470] Add Flatten transformation translator
discard e1b7644 [BEAM-8470] Create Datasets manipulation methods
discard a36bae0 [BEAM-8470] Create PCollections manipulation methods
discard d757185 [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
discard 7c8cd47 [BEAM-8470] Added SparkRunnerRegistrar
discard 79b4541 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN
discard 3f29bff [BEAM-8470] Post-pone batch qualifier in all classes names for readability
discard fbc7fbc [BEAM-8470] Add TODOs
discard 12ed3f5 [BEAM-8470] Make codestyle and firebug happy
discard 4e94975 [BEAM-8470] apply spotless
discard fd888deb [BEAM-8470] Move common translation context components to superclass
discard 5eeca80 [BEAM-8470] Move SparkTransformOverrides to correct package
discard 666c011 [BEAM-8470] Improve javadocs
discard fef01b3 [BEAM-8470] Make transform translation clearer: renaming, comments
discard b8d4a96 [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
discard 1c1bbab [BEAM-8470] Initialise BatchTranslationContext
discard 1ffbf4e [BEAM-8470] Organise methods in PipelineTranslator
discard 2222e24 [BEAM-8470] Renames: better differenciate pipeline translator for transform translator
discard c4c38b3 [BEAM-8470] Wire node translators with pipeline translator
discard c3828ea [BEAM-8470] Add nodes translators structure
discard 5b7efb1 [BEAM-8470] Add global pipeline translation structure
discard 8b1f07e [BEAM-8470] Start pipeline translation
discard 44aa39d [BEAM-8470] Add SparkPipelineOptions
discard 0d314d2 [BEAM-8470] Fix missing dep
discard 14a7a24 [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0
add 9146b9c [BEAM-4775] Ensure each JobInvocation test is run in isolation
add f392558 [BEAM-4775] Refactor SamzaRunner - separate pipeline result creation
add c81dd52 [BEAM-4775] Introduce PortablePipelineResult interface
add fc6cf80 [BEAM-4775] Use PortablePipelineResult in all portable runners
add e3ac1c1 [BEAM-4775] Allow reading portableMetrics from JobInvocation using grpc calls
add ec647ee [BEAM-4775] Stash handle to pipeline result to query metrics later
add 79fcd3b [BEAM-4775] Read portable metrics from grpc response in Python SDK
add 5705e1f [BEAM-4775] Return portable metrics from Flink runner
add 2cfa48b Merge pull request #9020: [BEAM-4775] Support returning metrics from job service
add 83bd4f1 [BEAM-7772] Remove Pkb from file-based IOIT jobs
add 716b9aa [BEAM-7772] Refactor: improve naming in the file
add f458576 [BEAM-7772] Refactor: autoformat for consistency (intelij rules applied)
add 1668956 Merge pull request #9287: [BEAM-7776] Remove Pkb from file-based IOIT jobs
add 6943270 upgrade calcite to 1.20.0
add 4501f60 api changes in calcite 1.20.0
add 26d5642 Merge pull request #9311: [SQL] Upgrade to Calcite 1.20.0
add 8e989a3 [BEAM-5148] Optionally load bson (#9308)
add babf86b [BEAM-5822] Remove non vendored byte-buddy dependency from sdks/java/core
add 6c65527 [BEAM-5822] Remove non vendored byte-buddy dependency from sdks/java/core
add 2bebbfe [BEAM-7940] Quick fix sdks:python:depSnapshot
add 0704bb8 Merge pull request #9317: [BEAM-7940] Quick fix sdks:python:depSnapshot
add d2d8bac Add JobService.GetJobs to the job API
add e0a65ec Fix python code style
add e5e9cab [BEAM-7927] Add JobService.GetJobs to the job API
add f7eb6fe [SQL] Disable KafkaCSVTableIT
add de18f4b Merge pull request #9322 from akedin/disable-kafkatableit
add 13b45e6 [BEAM-3608] Remove remaining unused non-vendored guava dependencies
add 12837e8 [BEAM-3608] Remove remaining unused non-vendored guava dependencies
add 8b388a7 Fix: Make sure experiments list if modifiable
add 7eca117 Fix: Make sure experiments list if modifiable
add f79b55c [BEAM-7389] Add outputs for Filter page examples
add f3f3171 Merge pull request #9259 from davidcavazos/filter-page
add 813ceb4 [BEAM-7389] Add code examples for Values page
add 1e8f8b0 Merge pull request #9264 from davidcavazos/values-page
add 27b526f [BEAM-7389] Add code examples for FlatMap page
add 0c56384 Merge pull request #9260 from davidcavazos/flatmap-page
add 339f2a7 Add Nexmark talk (ApacheCon 2017) and Universal metrics talk (ApacheCon 2018) slides and audio
add a56a779 Make it explicit that the link starts to play the audio podcast
add 6012f87 Address review comments: nit
add 948e84e Merge pull request #9107 from echauchot/website_etienne_talks
add 579a7df Added Paris meetup into the list of meetups
add 4b6800a Merge pull request #9326: [website] Added Paris meetup into the list of meetups
add 57237ad [BEAM-7721]Add write test to BigQueryIOIT
add 56f008c [BEAM-7721]Code reformat
add d79c06d Merge pull request #9295: [BEAM-7721] Add write test to BigQueryIOIT
add 614916a [BEAM-7854] Resolve parent folder recursively in LocalFileSystem match + test
add e3cd3b9 Use regex to detect non glob prefix of path
add 770519a Change tests
add 48955be Refactor + put test back
add 21ddfd1 Change logic + fix tests
add 66c3dea Spotless
add badef9c Spotless
add 22d70f9 [BEAM-7854] Resolve parent folder recursively in LocalFileSystem matc…
add c0bebd7 [BEAM-7916] Add ElasticsearchIO query parameter to take a ValueProvider (#9285)
add 4ff1347 Merge pull request #9314: [BEAM-7916] Add ElasticsearchIO query parameter to take a ValueProvider
add 89498f2 [BEAM-7959] Simplify use of Gradle property and task in Python
add 22eff79 [BEAM-7959] Simplify use of Gradle property and task in Python
add ead80b4 [BEAM-7954] Validate beam modules to disable guava imports from vendored gRPC
add b645856 [BEAM-7954] Validate beam modules to disable guava imports from vendored gRPC
add e62bb34 [BEAM-7946] changed name of sdist task in LoadTests gradle.
add c950d30 Merge pull request #9312: [BEAM-7946] Changed name of sdist task in LoadTests gradle
add 98e9760 [BEAM-7816] Support Avro dates in Schemas
add 5d41100 [BEAM-7817] Support Avro enums in Schemas
add e4b268e Merge pull request #9152: [BEAM-7816] [BEAM-7817] Support Avro dates and enums in Schemas
add 1fad0a7 Add fine-grained progress reporting
add 946596b Merge pull request #9156: [BEAM-7495] Add fine-grained progress reporting
add 97b99b1 [BEAM-7866] Fix python ReadFromMongoDB potential data loss issue (#9233)
add 04119cc [BEAM-7667] report GCS throttling time to Dataflow autoscaler
add 8eddf8e Merge pull request #8973: [BEAM-7667] report GCS throttling time to Dataflow autoscaler
add 075b094 [BEAM-7856] Suppress error on table bigquery table already exists
add 5de3583 Merge pull request #9204 from angoenka/bq_create_race
add 2f1b56c [BEAM-7420]: Allow including the flink runner without flink on the classpath.
add ae83448 Merge pull request #8894: [BEAM-7420]: Allow including the Flink runner without Flink dependencies on the classpath
add d1cf9bb Refactor release guide.
add b7bca84 Merge pull request #9319 from akedin/refactor-release-guide
add ed95df5 Fix command format in Release Guide
add ab15bee Merge pull request #9349: Fix command format in Release Guide
add 0aa7d15 [BEAM-7389] Add code examples for Map page
add ab37b0f Merge pull request #9265 from davidcavazos/map-page
add d4b6f32 [BEAM-7389] Add code examples for WithTimestamps page
add 899037d Merge pull request #9267 from davidcavazos/withtimestamps-page
add ba5256d [BEAM-7389] Add code examples for KvSwap page
add b39d7e7 Merge pull request #9258 from davidcavazos/kvswap-page
add b4d3474 [BEAM-7389] Add code examples for Keys page
add d93f188 Merge pull request #9263 from davidcavazos/keys-page
add dbac355 Update build plugins
add 64507fe Merge pull request #9325 from iemejia/update-plugins
add 579355c Update stager.py
add 22cee91 Merge pull request #9350 from the1plummie/patch-1
add 2dc4667 [SQL] Add custom table name resolution
add a44f7d8 Merge pull request #9343 from akedin/custom-table-name-resolution
add 989c928 [SQL] Support complex identifiers in DataCatalog
add fd67fd3 Merge pull request #9353 from akedin/datacatalog-custom-name-resolution
add 4059cf5 Downgrade log message level
add 2785f14 Merge pull request #9354 from apache/aaltay-patch-2-1
add 5d9bb45 [BEAM-7987] Drop empty Windmill workitem in WindowingWindmillReader (#9336)
add d70aea3 [BEAM-7936] Update portable WordCount Gradle task on portability page
add b134e67 Merge pull request #9348: [BEAM-7936] Update portable WordCount Gradle task on portability page
add f97b04a release vendor calcite
add cde2c71 Merge pull request #9333 from vectorijk/release-vendor-calcite
add 530f1d7 [BEAM-7965] add retracting mode to model proto.
add 544604c Merge pull request #9329 from amaliujia/add_retraction_in_java_sdk
add 6c60e80 [BEAM-6694] Added Approximate Quantile Transfrom on Python SDK (#9153)
add 4083127 Move Design Documents index to cwiki
add 442fbcd Merge pull request #9360: [website] Move Design Documents index to cwiki
add 0015cae [BEAM-7802] Make SQL example slightly simpler
add 0c1e63d [BEAM-7802] Inline AvroUtils methods to have only one public AvroUtils class in core SDK
add 3cf7e4b [BEAM-7802] Fix minor issues (access modifiers + static) in AvroUtilsTest
add be6e077 [BEAM-7802] Make Schema.toString method multi OS friendly
add 6978655 [BEAM-7802] Add AvroUtils.schemaCoder method to infer a Beam schema from Avro
add c825d71 Merge pull request #9130: [BEAM-7802] Expose a method to make an Avro-based PCollection into an Schema-based one
add 4335f36 [BEAM-7989] Remove side inputs from CacheVisitor calculation.
add ee8a333 Merge pull request #9357: [BEAM-7989] Remove side inputs from CacheVisitor calculation
add 6fd3e71 [BEAM-7389] Add code examples for Partition page
add 96abacb Merge pull request #9261 from davidcavazos/partition-page
add c662f28 [BEAM-7882] Invoke Spark API incompatible methods by reflection
add 3ce0674 Merge pull request #9370: [BEAM-7882] Invoke Spark API incompatible methods by reflection
add face1f6 [BEAM-5164]: Add documentation for ParquetIO.
add 2b83b8f fixup! [BEAM-5164]: Add documentation for ParquetIO.
add 9909541 fixup! [BEAM-5164]: Add documentation for ParquetIO.
add 78ce9f0 fixup! [BEAM-5164]: Add documentation for ParquetIO.
add 0467ac4 Merge pull request #9339: [BEAM-5164]: Add documentation for ParquetIO.
add 6b153c5 [BEAM-7772] Remove PKB from JdbcIOIT Jenkins job
add f5bd005 [BEAM-7772] Remove PKB from HadoopFormatIOIT Jenkins job
add da9c5f9 [BEAM-7772] Remove PKB in hdfs file based IOITs
add b2cd084 [BEAM-7772] Refactor: rename consistently with job_PerformanceTests_FileBasedIO_IT.groovy
add f2b1c0f [BEAM-7772] Remove obsolete pkb config files
add 7fa63b8 [BEAM-7772] Refactor: reformat code, fix variable naming
add e9cf1f4 [BEAM-7772] Move all file based ioits to one file
add c8eff45 Merge pull request #9315: [BEAM-7772] remove pkb from JDBCIOIT, HadoopFormatIOIT and hdfs tests
add c5d4533 [BEAM-5519] Remove redundant groupByKey in Spark Streaming.
add 9c89493 Merge pull request #6511: [BEAM-5519] Remove call to groupByKey in Spark Streaming
add d44db9a [BEAM-5980] Refactor: keep all Flink Python Combine load test scenarios in one place
add 830e99b [BEAM-5980] Remove redundant Combine Performance tests
add d1f99ce Merge pull request #9286: [BEAM-5980] Remove redundant combine tests
add 873f776 [BEAM-7994] Fixing unsafe pointer usage for Go 1.13
add 797c27f [BEAM-7994] Fixing unsafe pointer usage for Go 1.13
add c170081 Fix the exception type of InMemoryJobService when job id not found
add 86c2ebd Update to also catch StatusRuntimeException
add eddaf92 [BEAM-7720] Fix the exception type of InMemoryJobService when job id not found
add c7ace41 [BEAM-7670] portable py: prevent race opening worker subprocess
add 7906149 Merge pull request #8992 from ibzib/subprocess
add 57aa9ef [BEAM-7676] uniquely identify SDK workers by the factory that created them
add 45e07fb Merge pull request #8990 from ibzib/worker-id
add 5ec2497 [BEAM-7986] Upgrade grpcio
add 6fbae6c Merge pull request #9356: [BEAM-7986] Upgrade grpcio
add 5b6a8b6 [BEAM-1974] Add Metrics user-oriented documentation (#9328)
add cff2669 [BEAM-8004] Fully restart flink cluster to prevent UnknownHostException
add 43c9e75 Merge pull request #9381: [BEAM-8004] Fix flaky Flink load tests
add 8bbfa10 [BEAM-7264] Remove set/getClientConfiguration in JdbcIO
add df36e3a Merge pull request #9373: [BEAM-7264] Remove set/getClientConfiguration in JdbcIO
add aaf1558 Add a datatype_inference module for inferring coders and schema from data (#9281)
add c0f456f Remove BEAM-6675 from 2.14.0 release notes (#9385)
add ff7964c [BEAM-8006] Add retracting to windowing strategy translation.
add 5b43602 [BEAM-7924] fix options attribute error
add bb278f4 Merge pull request #9365 from ibzib/py-opts
add 9d975a6 [BEAM-1009] Upgrade to Mockito 3.0.0 and Powermock 2.0.2
add 44fa33e Merge pull request #9338: [BEAM-1009] Upgrade to Mockito 3
add b16996c [BEAM-8010] Remove translation of bounded reads from portable Flink Runner
add 4575a80 Merge pull request #9383: [BEAM-8010] Remove translation of bounded reads from portable Flink Runner
add 197ddf7 [BEAM-7804] Update python sdk transform programming guide.
add 5994182 Merge pull request #9379 from y1chi/BEAM-7804
add 68734a2 [BEAM-7832] Add ZetaSQL as a dialect in BeamSQL (#9210)
add 7f66c22 [BEAM-8022] Add new section in Dockerized Jenkins README
add 098c11f Merge pull request #9392 from kamilwu/seed-job-from-local-git
add 9266887 [BEAM-7980] Containerized Python worker pool for external environment
add 3561100 Merge pull request #9371: [BEAM-7980] External environment with containerized worker pool
add b50d9f1 [BEAM-8020] Remove translation of bounded reads from portable Spark runner
add 727f85f Merge pull request #9391: [BEAM-8020] Remove translation of bounded reads from portable Spark runner
add 9f87c75 [BEAM-8011] Remove workaround to run unbounded sources in portable pipelines
add 2c0fef2 Merge pull request #9384: [BEAM-8011] Remove workaround to run unbounded sources in portable pipelines
add f039451 [BEAM-8018] Detect unexported fields earlier
add d6c87aa !fixup typos
add 8dd6db3 [BEAM-7969] Report FnAPI counters as deltas in streaming jobs. (#9330)
add 4f6b6be [BEAM-7742] Partition files in BQFL to cater to quotas & limits (#9242)
add abad054 [BEAM-7856] Re Raise exception for code other than 409
add 6486fcf Merge pull request #9396 from angoenka/fix_bq_exception
add 5ec2b3f [BEAM-8037] add beam_fn_api experiment when using FlinkRunner
add 59a60b8 Merge pull request #9402 from ibzib/flink-override
add cd3b2cb [BEAM-6114] Calcite Rules to Select Type of Join in BeamSQL
add 1efa67a Merge pull request #9395 from rahul8383/beamsql-join-rules
add 9d45346 [BEAM-8038] Fix worker pool exit hook
add cacb931 [BEAM-8036] fix failed postcommit (#9409)
add 5cffc64 update beam website for 2.15.0
add ba7e16c Merge pull request #9341 Update beam website for 2.15.0
add 57ac356 add blog post for 2.15.0 release
add 90939f5 Add highlights and known issues.
add 93f02f9 Merge pull request #9346 Add blog post for 2.15.0 release
add d943bd6 fix 2.15.0 release blog
add 0020398 Merge pull request #9412 fix 2.15.0 release blog
add db9b497 fix in the beam website download file
add b830a6c Merge pull request #9413 fix in the beam website download file
add f3be08a SDK 2.15.0 Python dependencies
add dcb389d Merge pull request #9394 [BEAM-8007] Document SDK 2.15.0 Python dependencies
add 4648e13 [BEAM-8082] Do not publish any metrics in Load tests on Flink
add dc63296 Merge pull request #9416: [BEAM-8082] Do not publish any metrics in Load tests on Flink
add ad92513 Add PubSub BigQuery IT in Python for Batch Loads in Streaming (#9309)
add 4670398 [BEAM-7642] Fix python sdk AfterProcessingTime unit discrepancy
add de1cfd1 Merge pull request #9397 from y1chi/BEAM-7642
add 1041db2 [BEAM-7988] py: include runner name when runner is invalid
add 862b04c Merge pull request #9355 from ibzib/bad-runner
add c1b45a7 [BEAM-6783] Unignore ZetaSQLDialectSpecTestZetaSQL#testSelectBytes
add d009f72 Merge pull request #9415 from kanterov/zetasql-bytes-literals
add 41efa38 Updated 2.15.0 blog post
add 279c55f Merge pull request #9420 Updated 2.15.0 blog post
add aa8b29b [BEAM-7909] upgrade python lib versions to match to dataflow worker (#9388)
add aeada18 Rename assertArrayCountEqual to assertUnhashableCountEqual and fix typo
add f085cb5 Merge pull request #9352 Fix and rename assertUnhashableCountEqual.
add 64262a6 Merge pull request #9275: [BEAM-6858] Support side inputs injected into a DoFn
add fe07ce9 [Beam ZetaSQL] remove @Ignore from fixed tests.
add e14f8dc Merge pull request #9423: [SQL][ZetaSQL] remove @Ignore from fixed tests.
add 91c129d [BEAM-8080] [SQL] Fix relocation of com.google.types
add 47c51ad [BEAM-8080] [SQL] Do not relocate com.google.type.*
add 664e250 Merge pull request #9414 from kanterov/beam-8080
add f2e32af Skip PubSubBigQueryIT.test_file_loads on Dataflow
add 842942e Merge pull request #9427 from ttanay/skip-test-file-loads
add 692de7c [BEAM-8079] Move release build verification to Jenkins
add 823de61 Merge pull request #9411: [BEAM-8079] Move release Gradle build to a Jenkins job (Part - 1)
add 41dd4bf [BEAM-7389] Update to use util.Regex transform
add da6c1a8 Merge pull request #9435 from davidcavazos/regex-sample
add 36e0b7c [BEAM-8015] Get logs from Docker containers
add 514cf21 Merge pull request #9389: [BEAM-8015] Get logs from Docker containers
add fb9c868 [BEAM-8079] Move release Gradle build to a Jenkins job (Part - 2)
add 3d4ebaf Merge pull request #9434: [BEAM-8079] Move release Gradle build to a Jenkins job (Part - 2)
add 31562e7 [BEAM-7872] Implement Flink class which takes responsibility of creating Flink cluster
add cec26bf [BEAM-7872] Rewrite create_flink_cluster.sh so that it supports other operation
add a92f1e8 [BEAM-7872] Create a class for publishing docker images
add 157df49 Merge pull request #9213: [BEAM-7872] Simpler Flink cluster set up in load tests
add 36ddc79 [BEAM-7528] Save metrics according to distribution name.
add 605ba11 [BEAM-7528] Refactor of load test utils and added documentation
add 777fd37 Merge pull request #8941 from kkucharc/BEAM-7528-save-correctly-named--metrics
add 368e28c Removed unused arg from ToString.Element
add e06ef8e Merge pull request #9437 from aaltay/ts
add 058f5bc Integrating ZetaSketch's HLL++ algorithm with Beam (#9144)
add f6c1d6e Revert #9319 - Changes to release guide (#9436)
add 468ca09 [BEAM-6127] upgrade junit dependency
add cb6a86d [BEAM-6127] upgrade junit dependency
add 45b595b Update beam_fn_api.proto
add 2e9f052 Update beam_fn_api.proto
add 5bb21c8 Document StateKey enums for state backed iterable use case.
add 203a98f Use consistent names for auto-generated GCP client artifacts.
add 6303d41 Alphabetize imports of GCP library entries
add 26ecb3d Depend on proto- instead of grpc- package for google_cloud_bigtable
add 74e9fd5 Merge branch 'master' into beam_java_dependencies
add a1bfde3 Merge branch 'master' into beam_java_dependencies
add 8a3399b Housekeeping for GCP dependencies
add a8d8235 Add mongodbio to beam website io links
add f0618c5 Add python mongodbio to beam website IO links
add 744fe67 [BEAM-8008] show error message from expansion service in Java External transform
add a6f219a Merge pull request #9377: [BEAM-8008] show error message from expansion service in Java External transform
add 5b5658e [BEAM-8023] Add value provider interfaces for BigQueryIO.Read using Method.DIRECT_READ (#9405)
add 4836b14 [BEAM-7864] Simplify/generalize Spark reshuffle translation
add a5e7e67 Merge pull request #9410 from ibzib/spark-reshuffle
add 5a0ed84 [BEAM-7973] py: shut down Flink job server automatically
add 98e553b Merge pull request #9340 from ibzib/stop-flink-job-server
add d94e348 [BEAM-7999] Fix BigQuery timestamp handling for Schema Aware PCollection
add 5a62f94 Merge pull request #9369 from alexvanboxel/feature/BEAM-7999-bq-timestamp
add 8e3afa4 Remove unwanted byte-buddy forced version.
add 7a2b847 Merge pull request #9442: Fix broken master
add b3a2fe2 [BEAM-8107] Update commons-compress to version 1.19
add c7f0b49 Merge pull request #9439 from iemejia/BEAM-8107-commons-compress-update
add c9da964 Revert "Merge pull request #8943: Schema conversion cleanup"
add dbcb14c Revert "Update portable schema representation and java SchemaTranslation (#8853)"
add 7f09832 Merge pull request #9443 from TheNeuralBit/revert-portable-schema
add 18071d6 Add SchemaCoder known issue to 2.15.0 release blog
add afeba4c Merge pull request #9444 from TheNeuralBit/patch-1
add 65b79fb [BEAM-7829] Add schema names when converting with AvroUtils.toAvroSchema
add 0871831 [BEAM-7829] Avoid name collisions in generated avro
add 9e152b7 Merge pull request #9247: [BEAM-7829] Add schema names when converting with AvroUtils.toAvroSchema
add 3d906d3 [BEAM-7978] Return BACKLOG_UNKNOWN in case of unknown watermark
add 5af8a6a Merge pull request #9432: [BEAM-7978] Return BACKLOG_UNKNOWN in case of unknown watermark
add b90cb51 Bump container version
add 62a3234 Merge pull request #9455 from TheNeuralBit/container-version-update
add 526e6db [BEAM-5428] Modify cache token Proto design to only include tokens in ProcessBundleRequest
add 9bbd89a Merge pull request #9440: [BEAM-5428] Modify cache token Proto design to only include tokens in ProcessBundleRequest
add f1d02e9 [BEAM-3489] add PubSub messageId to PubsubMessage
add 71f6ddf [BEAM-3489] add Coder test
add d0b9cac [BEAM-3489] add messageId coders to PubsubUnboundedSource.getOutputCoder()
add b8313b0 [BEAM-3489] update docs
add a9e14ff [BEAM-3489] add PubSub messageId in PubsubMessage
add 26c91a9 Remove the dot to make job logs link clickable.
add d4f8068 Merge pull request #9438: Remove the dot to make job logs link clickable
add f22231c [BEAM-7972] Always use Global window in reshuffle and then apply window again.
add d8c1146 Merge pull request #9334 from angoenka/fix_reshuffle
add 0c61e3c [BEAM-7389] Update to use util.ToString transform
add 445d6ba Merge pull request #9433 from davidcavazos/tostring-sample
add 7ab55ac [BEAM-5428] Update generated proto bindings for Go.
add 22b4899 Merge pull request #9459: [BEAM-5428] Update generated portability proto bindings for Go.
add 0df9543 Support side-input injection in FnApiDoFnRunner.
add 43b6163 Merge pull request #9461: [BEAM-8102] Support side-input injection in FnApiDoFnRunner.
add 6a8ed72 [BEAM-8116] Update website instructions in the build_rc script.
add 8db7849 Merge pull request #9457 [BEAM-8116] Update website instructions in the build_rc script.
add a79e5f6 [BEAM-8129] Update Spark runner to use spark version 2.4.4
add 59632ee Merge pull request #9465: [BEAM-8129] Update Spark runner to use spark version 2.4.4
add 367f85d Release upper bound for dill
add dad64bf Merge pull request #9419 Relax upper bound for dill
add 032201c [BEAM-8114] Fix failing Chicago Taxi Example job on Jenkins (#9456)
add 9eda188 [BEAM-7980] Exactly once artifact retrieval for Python SDK worker pool
add a141c4e Merge pull request #9398: [BEAM-7980] Exactly once artifact retrieval for Python SDK worker pool
add f0e6c8b [BEAM-7966] Write portable Flink application jar
add 4460f03 Merge pull request #9331 from ibzib/flink-create-jar
add 0c2d8be [BEAM-5878] support DoFns with Keyword-only arguments (#9237)
add 01b3bfb [BEAM-7993] Run Portable PreCommit tests sequentially
add d53cd53 Merge pull request #9460: [BEAM-7993] Run Portable PreCommit tests sequentially
add 56ff5db [BEAM-8097] update the release doc
add 8f01a44 Merge pull request #9467 from yifanzou/BEAM-8097/update_release_guide
add 6175f44 [BEAM-8117] add notes when generating the gpg key.
add c381678 Merge pull request #9463: [BEAM-8117] add notes when generating the gpg key
add 6075b1e Makes subnetwork configurable
add bf2d943 [BEAM-8014] Using OffsetRange as restriction for OffsetRestrictionTracker (#9376)
add e184e5a [BEAM-7721] Add Jenkins job configuration for BigQueryIO performance tests (#9382)
add 96d8e68 [BEAM-6114] Handle Unsupported Lookup Joins
add 84d5c87 Merge pull request #9453 from rahul8383/handle-unsupported-lookupjoins
add 562f928 [BEAM-8025] Temporarily disable CassandraIOTest
add a1b1cfe Merge pull request #9470: [BEAM-8025] Temporarily disable CassandraIOTest
add 0b5beac [BEAM-7819] Python - parse PubSub message_id into attributes property (#9232)
add 89f3645 Add blog for GSoC '19
add d251702 Merge pull request #9472 from ttanay/gsoc-blog
add 47feeaf [BEAM-7909] Python3 docker containers #9351
add ce10f5f Max publish batch byte size set to a value which takes base64 encoding size increase into account
add 4a1cfdd PR Comments
add fe8a9de Merge pull request #8184: [BEAM-8145] Max publish batch byte size set to a value which takes base64 encodin…
add b4e02a1 Allow arguments to be passed to python tests from tox cli.
add 1f3415e Merge pull request #9474 [BEAM-8148] Allow arguments passed from tox cli
add 964b964 [BEAM-7909] add Portable Precommit test for py36 and py37 (#9475)
add 23b0b73 Revert "[BEAM-7819] Python - parse PubSub message_id into attributes property (#9232)"
add 4c1dd37 Merge pull request #9479 from apache/revert-9232-BEAM-7819-parse-pubsub-message_id
add 7e1eddf [BEAM-7909] fix python container image name (#9478)
add 4a0eaf3 [hotfix] Encode keys as NESTED for flink keyselector
add ce4cd3b Merge pull request #9464: [hotfix] Encode keys as NESTED for flink keyselector
add 79d4bf2 Track PCollection boundedness in python sdk
add 2e39266 Add tests
add 9983115 Merge pull request #9426: [BEAM-8088] Track PCollection boundedness in Python SDK
add 7bf6c61 [hotfix] Remove duplicate words for the doc of flink runner
add 8869fce Merge pull request #9481: Remove duplicate word from the Flink runner doc
add be30557 Revert "[hotfix] Encode keys as NESTED for flink keyselector"
add 54a1296 Merge pull request #9483: Revert "[hotfix] Encode keys as NESTED for flink keyselector"
add c9d0ff2 Add static Pubsub.readMessagesWithCoderAndParseFn method
add 38dd8ce Merge pull request #9473: [BEAM-8155] Add static PubsubIO.readMessagesWithCoderAndParseFn method
add e73f581 Fix slice inference to not always assume List.
add f747159 Merge pull request #9486 Fix slice inference to not always assume List.
add 393396b [BEAM-7060] Type hints from Python 3 annotations (#9283)
add 4dbfc6f [BEAM-7859] set SDK worker parallelism to 1 in word count test
add fba3524 Merge pull request #9421 from ibzib/sdk-worker-test
add d28791e Introduce state when job state reported by a runner cannot be interpreted by the Java SDK.
add d87d8e4 Introduce state when job state reported by a runner cannot be interpreted by the Python SDK.
add 55b4815 Merge pull request #9132 [BEAM-7839] Default to UNRECOGNIZED state when a state cannot be accurately interpreted by the SDK.
add 6f88601 [BEAM-7389] Add code examples for Regex page (#9262)
add 468cac6 [BEAM-7389] Add DoFn methods sample (#9257)
add 846ada6 The coder returned for typehints.List should be IterableCoder
add f88989b Clarify the story surrounding the list return value guarantee
add eb5c794 lint
add d21bbaf Merge pull request #9344 [BEAM-7984] Return IterableCoder for typehints.List.
add 8b7a3e3 Fix minor typos (#9498)
add 3c3d59c [BEAM-7616] apitools use urllib with the global timeout. Set it to 60 seconds # to prevent network related stuckness issues. (#9401)
add 4f980f9 [GoSDK] Improve singleton side input runtime error
add 0dcbb3f5 Improve go worker harness error messages.
add 062dec2 Add non-terminal option for go harness worker hook
add 4a6dc28 [BEAM-7969] Fix doublecount on Cross SDK boundary GRPC PCollections in streaming jobs. (#9494)
add 58a12b6 Merge pull request #9454: [BEAM-8111] Add ValidatesRunner test to AvroSchemaTest
add 36aaabc [BEAM-8161] Update joda time to 2.10.3 to get TZDB 2019b
add 014bbee [BEAM-8161] Update joda time to 2.10.3 to get TZDB 2019b
add d0faa50 [BEAM-8167] Replace Postcommit_Python_Verify in Grafana dashboards.
add 7f64067 Fix typo in Cloud Pub/Sub topic path
add 4b94239 Merge pull request #9500 from anguillanneuf/patch-1
add 9043b3e Merge pull request #9372: [Beam-6858] Validate that side-input parameters match the type of the PCollectionView
add eefe965 Remove obsolete field from proto.
add 9390339 Remove obsolete field from proto.
add 1231873 [BEAM-7970] Adding correct version of Go protobuf compiler.
add 66dbff9 [BEAM-8144] Update hub setup in validates_rc_candidate.sh
add d07946e7 Merge pull request #9469: [BEAM-8144] Update hub setup in run_rc_validation.sh
add 483b9fb Replace tabs with spaces
add 3945827 Fix docker workers on os-x.
add 9fdd279 Merge pull request #9508 Fix docker workers on os-x.
add 0438908 [BEAM-8170] Don't import PyArrow on Python 2 + Windows
add 47ad06e Merge pull request #9522: [BEAM-8170] Don't import PyArrow on Python 2 + Windows
add 4eabda8 Revert "[BEAM-5878] support DoFns with Keyword-only arguments (#9237)"
add a757475 Merge pull request #9517 from apache/revert-9237-support_keyword-only-arguments
add 6007d8b Update beamgrafana deploy to match latest deployment
add 4a618a2 [Go SDK] Off By One error on splitting
add 1ebdf3f Set diskType on disks (#9515)
add 0db535d Revert "[Go SDK] Off By One error on splitting"
add 7c240da [go-sdk] Never split on the 0th element.
add 335de9d [BEAM-6865] make executable stage context management code reuseable
add df7f774 [BEAM-7600] borrow SDK harness management code into Spark runner
add 2954ee3 refactor Flink and Spark contexts into separate classes
add 6c27e3d s/state/jobFactory
add 932bd80 Merge pull request #9095 from ibzib/spark-multiworker
add b6f0487 Don't require pyarrow on Win32 installations.
add d4856b3 Merge pull request #9526 [BEAM-8181] Don't require pyarrow on Win32 installations.
add 6ab904d Allow cross-language for Direct runner.
add 8284e5b Merge pull request #9507 Allow cross-language for Direct runner.
add f3c3624 [BEAM-8187] Do not publish gradlew_orig in release
add 25760d2 Merge pull request #9533: [BEAM-8187] Do not publish gradlew_orig in release
add ccac658 Temporarily replace super() calls with calls to __init__ of parent class in Beam examples.
add 740ddc6 Merge pull request #9513 [BEAM-6158] Temporarily remove super() calls in Beam examples.
add 822c6b8 BEAM-7995: python PGBKCVOperation using wrong timestamp (#9364)
add 80a44fb Warn users about Beam becoming Python3-only in the future to encourage migration.
add 341fc1d Merge pull request #9537 [BEAM-1251] Warn Python 2 users about Beam becoming Python 3-only in the future to encourage migration.
add 8ea4755 Update verify release branch in release-guide.md
add f1d48fa Merge pull request #9540 Update verify release branch in release-guide.md
add 2b3599c [BEAM-8119] Properly cleanup Pubsub in release script
add 361a7b7 Merge pull request #9534: [BEAM-8119] Properly cleanup Pubsub in release script
add ab2a21e Enables asynchronous execution
add a0c10f5 Renames flag to reduce ambiguity, as per review
add 473377e [BEAM-8105] change default docker image name
add 53f069d [BEAM-8165] read version to set tag for default docker image
add 8790822 Merge pull request #9487 from Hannah-Jiang/add_containers_to_release
add ea301de [BEAM-8032] Fix exception in JDBC readRows when statement preparator is null
add 7accdc6 Set default statement preparator for readRows
add f0b7f8b Merge pull request #9425 from charithe/beam-8032
add 67afe5e [BEAM-8188] Allow PyArrow imports to fail on systems other than Win + Py27
add fcf1698 Merge pull request #9546 [BEAM-8188] Allow PyArrow imports to fail on systems other than Win+Py27
add d0fd5b3 Downgrade warning log to debug log.
add 6a63c06 Make JRH logs less noisy
add 40ccf08 [BEAM-7389] Add code examples for ParDo page (#9503)
add cd7b025 [BEAM-7389] Add code examples for ToString page (#9289)
add 7ddb127 [BEAM-6185] add image artifacts to release process
add da2fec3 Merge pull request #9506: [BEAM-8105] Docker images release scripts
add 804a295 [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions
add 6df5f74 fixup! [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions
add 86daba6 fixup! fixup! [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions
add cb10019 [BEAM-8169] DataCatalogTableProvider to use GCP credentials from PipelineOptions
add 7e91a7f [BEAM-7947] Improves the interfaces of classes such as FnDataService, BundleProcessor, ActiveBundle, etc to change the parameter type from WindowedValue<T> to T
add be2cfec fix code style
add 6b3f7d6 [BEAM-7947] Improves the interfaces of classes such as FnDataService,…
add 03bc7e7 Use a 0-length byte array to represent empty sketch in HllCount
add 17490af Use a 0-length byte array to represent empty sketch in HllCount (#9519)
add 81c44b4 [BEAM-7967] Execute portable Flink application jar
add c4674b6 Merge pull request #9408 from ibzib/flink-execute-jar
add b9be6bb Exclude AvroSchemaTest for Apex and Flink runner for now because they are failing
add 3fc26cd Exclude ParDoTest.testSideInputAnnotationWithMultipleSideInputs
add 8f91984 Merge pull request #9541: [BEAM-8204][BEAM-8205] Exclude AvroSchemaTest for Apex and Flink runner for now
add 41946dd Moving to 2.17.0-SNAPSHOT on master branch.
add 87e41df [BEAM-8208] [website] fix Python container build invocation
add daea77d Merge pull request #9545 from ibzib/py-container
add ea45ee4 [BEAM-8165] add _sdk to container name
add 73d42a4 Merge pull request #9549: [BEAM-8165] add _sdk to container name
add b07c38d Use common metric matchers for fnapi and legacy python dataflow metric integration tests (#9539)
add 051164e Make Jet Runner related documentation improvements
add c8767d3 Accommodate changes proposed during review
add 15eb089 Merge pull request #9471: [BEAM-7305] Improve Jet Runner related documentation
add da9efa8 [BEAM-8196] Add wait_until_finish_duration
add 3ed0e18 Merge pull request #9547: [BEAM-8196] Add wait_until_finish_duration
add fbe65ee [BEAM-8156] Add convert_to_typing_type
add 7a18873 Add typing.Generator -> typehints.Iterator conversion
add 632b54b Merge pull request #9509: [BEAM-8156] Add convert_to_typing_type
add a0e02ab [BEAM-5539] Upgrade google-cloud-pubsub and bigtable packages.
add bce94cd Merge pull request #9491: [BEAM-5539] Upgrade google-cloud-pubsub and bigtable packages.
add b82b96b [BEAM-8217] Encode str to bytes py3 in sdk harness status server
add abf00f4 Merge pull request #9551 from angoenka/fix_py3_status_serever
add 49e77e8 [BEAM-7911] Fix flakiness in avroio_test.py
add ec7a3d2 Merge pull request #9554: [BEAM-7911] Fix flakiness in avroio_test.py
add 2f4be02 [Beam 8031] Add snippets for pattern on custom windows (#9538)
add 2bd2a95 Improve documentation on BigQueryIO
add 0d2fbf2 Merge pull request #9555 from robinyqiu/hll
add 551faa1 Create a more user friendly external transform API
add 356f470 KafkaIO: use StringUtf8Coder in Java to match python
add ef6a5dc Encode and decode using nested context
add 932a696 Add better error message for configuration type mismatch
add 0c31f7c Allow tests to specify a python 3 minor version, to isolate syntax changes in 3.6
add e470c42 Split dataclass-based external transform tests to run in python 3.7 only
add 0ae5582 Add dataclasses as a native library for isort
add 535bf37 Merge pull request #9098 [7746] Create a more user friendly external transform API
add 59c81e4 [BEAM-7060] Prefer inspect module over funcsigs
add 43ae56a Merge pull request #9562: [BEAM-7060] Prefer inspect module over funcsigs
add 3fbba03 [BEAM-8225] Fix portable runner test Bugfix (#9557)
add 3528aa8 Fix bug in run_pylint.sh
add 6737a09 Merge pull request #9542: Fix exit status bug in run_pylint.sh
add 85ac361 Direct - Fix ParDo with side input after CoGroupByKey
add a039f43 Fix state sampler dropping the stage and step name from reported current state
add ba5c9de Merge pull request #9568: [BEAM-8231] Fix state sampler dropping the stage and step name from reported curr…
add f9751fd Exclude python 3-specific test files from running in post-commit
add b04895b Merge pull request #9570: [BEAM-8229] Exclude python 3-specific test files from running in post-commit
add 371c122 [BEAM-8226] Alter getInputElementsConsumed computation for monitoringInfos in RegisterAndProcessBundleOperation.java (#9571)
add 6b5084b [BEAM-7389] Generate notebook for filter (#9561)
add 8446745 Add test for versioned encoding
add b9de43c Merge pull request #9573 from pabloem/BEAM-1080
add 46a8dac Update index.md
add 1fcd209 Merge pull request #9574 from jesusrv1103/patch-1
add 0914293 BEAM-8138: Code snippet in FileIO Java docs fixed
add fb41d6f Merge branch 'master' of https://github.com/apache/beam into BEAM-8138
add a06be47 Merge pull request #9575 from Luis-MX/BEAM-8138
add 00e3bf5 BEAM-7596: Fixed 'concepts' section to make it parallel
add 43cea184 Merge pull request #9580 from saulchavez93/master
add 72f15b5 [BEAM-8178] Switch to a new gradle task that builds python docker image
add d3ffdce Merge pull request #9521: [BEAM-8178] Fix broken Load Tests running on Flink
add f39da9d Update execution-model.md
add d93e4e0 [BEAM-7595]Update execution-model.md
add dd8fe38 [BEAM-8034] Upgrade Flink Runner to 1.8.2
add 76b28ab Merge pull request #9535: [BEAM-8034] Upgrade Flink Runner to 1.8.2
add 0122d4d BEAM-8165 pass root as a param and keep default root
add 85d6d38 Merge pull request #9588 from Hannah-Jiang/container
add af00fc4 [BEAM-8224] Fix bug in _fn_takes_side_inputs (#9563)
add 451dd9c [BEAM-8093] Commit change to ensure py36-gcp runs tests
add 2e59c27 [BEAM-8093] Commit change to ensure py36-gcp runs tests
add 1e00170 [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override
add 06b91a7 [BEAM-8240] Ensure that the pipeline proto contains the worker_harness_container_image override
add 433f0e7 [BEAM-8093] Fail tox environments the don't set command
add 28525b6 [BEAM-8093] Fail tox environments that don't set command
add 113d9c1 Fix typo for procesing --> processing in programming-guide.md
add 511caa3 Fix typo for procesing --> processing in programming-guide.md
add 28a6177 [BEAM-8105] update release guide with docker images
add 0d0a3de [BEAM-8105] fixup: Mark's comment
add 74cffa3 [BEAM-8105] fix
add cb8770d [BEAM-8105] add a command to update version at gradle.properties to manual scripts
add ff4d432 BEAM-8105 fixup
add 6d56040 Merge pull request #9510: [BEAM-8105] update release guide with docker images
add 593cb6f [BEAM-8241] SQL code gen is more restrictive than Calcite
add 703536d Added comments
add fb31cb7 Merge pull request #9586: [BEAM-8241] SQL code gen is more restrictive than Calcite
add ddf8a53 [BEAM-8267] remove spurious error on Py worker shutdown
add f0aa877 Merge pull request #9600 from ibzib/no-process
add 510427e Fix traceback.format_exc(e) calls
add 4aad615 Merge pull request #9599 from Pehat/patch-1
add 2d6d8c7 gen_protos: don't run futurize twice when grpcio-tools is not installed
add 8ae6dfc gen_protos: use disutils log so that output is printed during normal use
add f470357 gen_protos: log the reason that protos are being regenerated
add c3e46dd gen_protos: cleanup pb2 files when there are more than expected
add d3b8338 gen_protos: stop checking protos twice for sdist
add 2a0b45f Merge pull request #9572 [BEAM-8238] Improve python gen_protos script
add 9f495fe [BEAM-8252] sync python SDK GCP client API with the current Google internal definition
add c445fdf Merge pull request #9589 from ibzib/dataflow-internal-py
add 6686cc4 [BEAM-8185] Properly backup sys config files in release script
add a3f06d1 Merge pull request #9532: [BEAM-8185] Properly backup sys config files in release script
add cbf8a90 Compatibility matrix: Remove unnecessary dependency on docker task of Java SDK Harness
add 9bc0fce Merge pull request #9603: Compatibility matrix: Remove dependency on Java SDK Harness docker task
add 1a8a88d Minor fix for python mongodbio display data
add ab1e16c Merge pull request #9601: Minor fix for python mongodbio display data
add c383c04 [BEAM-8242] Support -buildmode=pie with unregistered Go functions (#9585)
add 95717fa [BEAM-8228] Changed suffix of Dataflow counter metrics names filtered after adding distribution metric
add f9a95cd Merge pull request #9564 from kkucharc/BEAM-8228-change-counter-metrics-suffix-filter
add 62fd60f [BEAM-8165] use default tag
add eba9138 Merge branch 'master' into container
add 909bc5a Merge pull request #9610 [BEAM-8165] use default tag
add 06a206a [BEAM-7760] Added Interactive Beam module
add 6e9675b Merge pull request #9278 from KevinGG/master
add 6b663e0 Update capability-matrix.yml (#9234)
add a81edd0 [BEAM-8279] Disable IOTypeHints.from_callable
add eb76a71 Merge pull request #9611: [BEAM-8279] Disable IOTypeHints.from_callable
add 3d6afca Update index.md (#9576)
add fbc84b6 [BEAM-7945] Allow runner to configure semi_persist_dir which is used in the SDK harness
add 864e2e0 Merge pull request #9452: [BEAM-7945] Allow configuration of semi_persist_dir
add b26eb94 [BEAM-7629] Go SDK additional Validation for DoFns (1st impl)
add 4e88fa0 [BEAM-7629] Additional checks for context param and error return.
add 2adc2d1 [BEAM-7389] Generate notebook for FlatMap (#9584)
add 02c290f [BEAM-8286] replace dead intersphinx link for google-cloud-python
add 5c5a04f Merge pull request #9620: [BEAM-8286] replace dead intersphinx link for google-cloud-python
add 2e06a74 [BEAM-8285] Upgrade to ZetaSQL 2019.09.1
add 2393b3f Merge pull request #9621 from apilloud/zetasql-2019.09.1
add 279a8e5 [BEAM-8233] [BEAM-8214] [BEAM-8232] Document environment_type flag
add 6a81cdc Merge pull request #9605 from ibzib/sdk-harness-doc
add 14e5632 Several changes to support cross-language transforms on DataflowRunner
add 7c2254f Merge pull request #9390: [BEAM-8019] Several changes to support cross-language transforms on DataflowRunner
add b092462 Add explanation of triaging release blockers
add ec3e770 Merge pull request #9608 Add explanation of triaging release blockers
add 94a8b26 [BEAM-8203] Add support for AvroTable in SQL
add b8b7bce Merge pull request #9597 from bmv126/avro-table-provider-support-in-sql
add 5a80d3d fix typo
add cb71f03 Merge pull request #9624: [BEAM-8290] Fixes typo(ptions -> options)
add 592b2fa [BEAM-8003] Update IOIT documentation. Remove all mentions of Perfkit.
add dd955f4 [BEAM-8003] Update descriptions of file-based ioits
add 23605d8 [BEAM-8003] Remove "performanceTest" gradle task (obsolete)
add bb6f9ed Merge pull request #9450: [BEAM-8003] Remove Perfkit leftovers
add 49c3978 Document creating and implementing custom windows
add b19c5b5 Merge pull request #9406 from soyrice/custom-window-pattern
add 42566ff Refactor job server subprocess logic to be used elsewhere.
add 5845fe9 Add java jar based expansion service.
add 6a54170 More documentation.
add d963aeb Merge pull request #9525 Add java jar based expansion service.
add d69290f Avoid comparison with param.default in type signature analysis.
add 16f1379 Merge pull request #9627 [BEAM-7060] Avoid comparison with param.default in type signature analysis.
add 7aa42a7 Merge pull request #9604: Restore portable schema changes, with simple LogicalType support
add 6bfb947 Update Flink runner page to refer to classic instead of legacy runner
add cb30e25 Merge pull request #9635 from tweise/classicFlinkRunner
add 905491d replace unnecessary identity calls in sql tests
add a15ccc8 Merge pull request #9631: Replace unnecessary identity calls in sql tests
add 8070931 [BEAM-8299] Upgrade Jackson to version 2.9.10
add f376861 Merge pull request #9636 from iemejia/BEAM-8299-jackson-2.9.10
add 4299e38 Sets workerHarnessContaienrImage as the containerImage of the DockerPayload of the default environment for DataflowRunner
add a7cf016 Merge pull request #9629: [BEAM-8240] Sets workerHarnessContainerImage in the default Environment of DataflowRunner
add 1b2ad50 Remove several supported dataflow python streaming items from unsupported list
add 24c7a62 Merge pull request #9643 from y1chi/streaming-doc
add 113461a [BEAM-8160] Add FnApi execution mode instruction (#9628)
add ce27e30 [BEAM-7962] Drop support for Flink 1.5 and 1.6
add 661237c [BEAM-7962] Update version compatibility section on Flink Runner page
add 0dc2ad2 Merge pull request #9632: [BEAM-7962] Drop support for Flink 1.5 and 1.6
add c4f4109 [BEAM-8157] Introduce encode_nested method on Python SDK Coder
add 61b8fa0 [BEAM-8157] Ensure key encoding for state requests is consistent across SDKs
add fce6380 Merge pull request #9484: [BEAM-8157] Ensure key encoding for state requests is consistent across SDKs
add cb3772c [BEAM-7660] Create Python ParDo load test job on Flink
add 8bb6982 [BEAM-7660] Parameter names changes
add 6e3c352 Merge pull request #9449: [BEAM-7660] Create Python ParDo load test job on Flink
add b063a9e [BEAM-8256] Set fixed number of workers for file-based IOITs
add d61c227 Merge pull request #9596: [BEAM-8256] Set fixed number of workers for Java IOITs
add f4a1358 [BEAM-8131] Provide Kubernetes setup for Prometheus (#9482)
add 32f03f9 More guarding against missing grpc.
add 6eb8605 Merge pull request #9645 from robertwb/no-grpc
add 899f45c [BEAM-5820] Switch Beam SQL to vendored Calcite
add 0d9972f not include shadow direct runner dependency in nexmark
add 031b378 [BEAM-5820] Use vendored calcite
add 77bfad4 Remove the 'experimental' word from Jet Runner modules/directories
add ac45af9 [BEAM-7305] Remove the 'experimental' word from Jet Runner modules/directories
add 15e4bd1 [BEAM-8302] Fix PostCommit_XVR_Flink
add 40f2562 Merge pull request #9644 [BEAM-8302] Fix PostCommit_XVR_Flink
add 340c320 [BEAM-8293] prescriptive log message for artifact retrieval failure
add f07bd7a Merge pull request #9646 from ibzib/retrieve-fail
add b32e332 [BEAM-8021] Stop publishing unneeded artifacts
add 5ea2878 [BEAM-8021] Add Automatic-Module-Name headers to beam's artifacts.
add 2acbfbd Merge branch 'lgajowy-BEAM-8021'
add 7bee065 [BEAM-8301] Fix incomparable defaults.
add 80bfcc6 Merge pull request #9641 [BEAM-8301] Fix incomparable defaults.
add 656cc76 [BEAM-8311] Fix py mongodbio display data
add 4ea54ca Merge pull request #9656 from y1chi/BEAM-8311
add a40d643 [BEAM-3372] Remove duplicated zone in Dataflow worker options (#9578)
add 9324e26 [BEAM-8100] Exception handling for AsJsons and ParseJsons transforms
add 7f3cded Merge pull request #9499: [BEAM-8100] Exception handling for AsJsons and ParseJsons
add 72f4366 [BEAM-5428] Implement Runner support for cache tokens
add 948c6fa Merge pull request #9374: [BEAM-5428] Implement Runner support for cache tokens
add a97ff41 [Go SDK] Fix Off by One error splitting DataSource
add 04dc3c3 Update sdks/go/pkg/beam/core/runtime/exec/datasource.go
add 43bb1ba Add artifacts-dir param to flink runner
add ba9a7ff Merge pull request #9648 from angoenka/flink_options
add 17805bc [BEAM-8313] Rename certain proto fields to be consistent across fn/runner APIs
add 97c4381 [BEAM-7389] Use includes for buttons
add 7120ceb Merge pull request #9661 from davidcavazos/include-buttons
add 7644806 Fixing issue in AvroCoder documentation.
add a2fdb5c Merge pull request #9659 from apache/pabloem-patch-1
add 3231ac5 [BEAM-8315] Clean-up unused fields for splitting SDFs
add cb414aa Merge pull request #9666 from lukecwik/beam8315
add 74ea749 [BEAM-8313] Follow up on PR comments for #9663 (#9668)
add b63bac0 [BEAM-1296] Providing a small dataset for "Apache Beam Mobile Gaming … (#9633)
add 82f6656 Add test demonstrating BEAM-8317
add e78943c Merge pull request #9672: [BEAM-8317] Add (skipped) test for aggregating after a filter
add 42cae0a [BEAM-8157] Increase parallelism for Python PVR tests
add 469618c Merge pull request #9649: [BEAM-8157] Increase parallelism for Python PVR tests
add 1572ce0 [BEAM-7919] Add MongoDB IO integration test for py3.5 (#9639)
add b2ed0da Add ReadFromParquetBatched and ReadAllFromParquetBatched
add 19f8812 Merge pull request #9361: [BEAM-7990] Add ability to read parquet files into PCollection<pyarrow.Table>
add 0256f38 [BEAM-8314] Add aggregation logic to beam_fn_api metric counter updat… (#9679)
add c5bbb51 Adds a pipeline option to Python SDK for controlling the number of threads per worker.
add 7627c82 Merge pull request #9675: [BEAM-8318] Adds a pipeline option to Python SDK for controlling the number of threads per worker.
add 6060df3 [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode
add 0bda32d Addressed review comments
add f6f7c88 Added a test for BigQuery SQL read in EXPORT mode
add ee45cb0 Merge pull request #9625: [BEAM-8275] Beam SQL should support BigQuery in DIRECT_READ mode
add 731d5bf Add clarification about authorized views
add 3125a83 Merge pull request #9617: BigQueryIO: add clarification about authorized views (doc only pr)
add 487c9dfc [BEAM-6923] limit gcs buffer size to 1MB for artifact upload (#9647)
add 3cbcf7d Helper to get the value of a KV type
add 5cf2029 [BEAM-8321] fix Flink portable jar test
add 2534330 Merge pull request #9680 from ibzib/pipeline-jar-test
add 82742c1 Restrict dill's upper bound.
add bed4201 Merge pull request #9695 [BEAM-8324] Restrict the upper bound for dill due to incompatibility between versions 0.3.0 and 0.3.1.1.
add 8e35464 [BEAM-8021] Swap build-tools to be compile only so it isn't a "required" dependency of Flink.
add de9b5b6 Merge pull request #9698: [BEAM-8021] Swap build-tools to be compile only so it isn't a "required" dependency of Flink.
add 0128ceb [BEAM-5428] Implement cross-bundle user state caching in the Python SDK
add 55588e9 [BEAM-5428] Add pipeline option to enable caching / Disable caching by default
add c1007b6 Merge pull request #9418: [BEAM-5428] Implement cross-bundle user state caching in the Python SDK
add 92399c6 [BEAM-8300] - Add readObject to initialize producer if it is null
add 710d400 Merge pull request #9640: [BEAM-8300] KinesisIO.write throws NPE because producer is null
add 698b111 [BEAM-6896] Loosen PyYAML dependency.
add 0152c42 Merge pull request #9699 from robertwb/update-yaml
add 869c8c7 Always use UTC when creating instances of baselocal
add 40d5167 Merge pull request #9678: [BEAM-8291] Always use UTC in generated code to convert to a LocalDate
add a53fdf4 Fix Python RC staging URL in release script
add 5ced442 Merge pull request #9707: Fix Python RC staging URL in release script
add 71ca273 Move Flink VR test code into a common file to be shared across multiple py versions.
add 902a089 Include common portable runner tasks in python 3.x suites.
add 2366e43 Evaluate the name of Python container task based on interpreter version.
add 70f2146 Add Python 3.5 ValidatesRunner Flink Jenkins suite.
add c475507 Match test suite name in the UI with the trigger command.
add 3233506 Update postcommit status links in README.md.
add 4f49063 Rename existing Python_PVR_Flink precommit job to Python2_PVR_Flink.
add 001e165 Merge pull request #9691 [BEAM-7223] Add Python 3.5 Flink ValidatesRunner postcommit suite.
add 94c6d8d Pull out reusable parts of BeamFileSystemArtifact*.java into abstract superclass.
add 6ada564 [BEAM-8312] ArtifactRetrievalService serving artifacts from jar resources.
add cb4f82b Merge pull request #9681 [BEAM-8312] ArtifactRetrievalService serving artifacts from jar.
add a1e09e5 Fix broken :runners:java-fn-execution:spotbugsMain task
add 652ed87 Merge pull request #9710: Fix broken :runners:java-fn-execution:spotbugsMain task
add 85090ab [BEAM-8303] Ensure FileSystems registration code runs in non UDF Flink operators
add 6551d09 Merge pull request #9688: [BEAM-8303] Ensure FileSystems registration code runs in non UDF Flink operators
add 1c1fc74 [BEAM-8286] move datastore intersphinx link again
add c1fa9b1 Merge pull request #9714: [BEAM-8286] move datastore intersphinx link again
add 2b0716f UT for BeamSql DDL with field having map with row as value
add b4cd1d6 Merge pull request #9701: [BEAM-7623] UT for BeamSql DDL with map field having row as value
add 01ccafe Fix download_files in the python RC validation script
add 979225d Merge pull request #9715 Fix download_files in the python RC validation script
add f207721 Fix traceback.format_exc(e) call
add 8589c85 Merge pull request #9689 from Pehat/patch-2
add 8c3d9bf Split out lint job from monolithic python preCommit tests on jenkins
add a203945 Merge pull request #9706 [BEAM-8213] Split out lint job from monolithic python preCommit tests on jenkins
add e20b9e1 [Go-SDK] Check correct cache for wrapped structs.
add 9bb7dd5 Handle newlines in dataflow side input labels. (#9712)
add 10243dc Add job server request timeout (default to 60 seconds)
add dae22d0 Merge pull request #9673 from ecanzonieri/BEAM-7933_add_jobserver_timeout
add 6579be6 [BEAM-8334] Expose Language Options for testing
add b01cc60 Merge pull request #9704: [BEAM-8334] Expose Language Options for testing
add 0afcc04 Fix wrong temp_location in run_rc_validation.sh
add 4f7f89f Merge pull request #9716: [BEAM-8339] Fix wrong temp_location in run_rc_validation.sh
add c1ca55f Quietly skip cleanup_pubsub in automation script
add 7e65dc8 Merge pull request #9719: [BEAM-8339] Quietly skip cleanup_pubsub in automation script
add c24df32 [BEAM-8346] Increase timeout of beam_PostRelease_Python_Candidate to 6hours
add 088a502 Merge pull request #9723: [BEAM-8346] Increase timeout of Python Release Validation job
add e14f3f8 [BEAM-8111] Enable CloudObjectsTest$DefaultCoders (#9446)
add 6cc1c1f Merge pull request #9612: [BEAM-6855] Side inputs are not supported when using the state API
add be5df76 [SQL] Add support for GCS entries in DataCatalog
add c5c100a Port GCS SQL table to latest Data Catalog client
add b1fe782 Merge #9727: Add support for GCS entries in DataCatalog
add 478fcb0 Add Prerequisites(Requirements) to run gradlew
add 6fb6873 Update portability.md
add daa36e7 Merge pull request #9497 from ChethanUK/patch-1
add 929c7ee Update capability-matrix.yml
add 63344d58 Merge pull request #9670 from Jofre/patch-2
add 2d42592 Update fixed and sliding window examples to be consistent with the figures
add 37e8926 Merge pull request #9468 from ageron/patch-1
add 69bfdac [BEAM-7389] Update include buttons to support multiple languages
add 8e75935 Merge pull request #9669 from davidcavazos/include-buttons
add 0216d53 [BEAM-5707] Modify Flink streaming impulse function to include a counter
add 6a3fdfd Merge pull request #9728: [BEAM-5707] Modify Flink streaming impulse function to include a counter
add bfdf6b8 [BEAM-8092] changed guava's Optional to java.util
add ff8653d Merge pull request #9431 from je-ik/BEAM-8092: [BEAM-8092] change guava's Optional to java.util in DirectRunner
add 5c76769 use LOG.info instead of System.out.println
add c327f50 Merge pull request #9337 from sgrj/sj/BEAM-7977/change-std-out-to-logger
add 13cea03 Add TimestampConversion to AvroCoder ReflectData, and accompanying test
add 01fa52a Add similar test to PubsubIO
add 3c718c3 Merge #9702: [BEAM-8329] Add TimestampConversion to AvroCoder ReflectData
add c0b3fda [BEAM-7389] Created elementwise for consistency with docs
add 50c5281 Merge pull request #9664 from davidcavazos/normalize-filenames
add f857f0f [BEAM-7760] Added pipeline_instrument module (#9619)
add f748dd2 [BEAM-6228]: Website build from source release fails due to git dependency
add df3a188 Merge pull request #9734: [BEAM-6228]: Website build from source release fails due to git dependency
add 7abf88c Merge pull request #9424: Make RowCoder a sub-class of SchemaCoder<Row>
add 9578559 Add support to pass params along to external worker pool.
add 916e170 Add unittest for external environment.
add 64e5a83 Merge pull request #9730: [BEAM-8351] Support passing in KV pairs to sdk worker via external environment config
add a2214c5 [BEAM-3845] Remove deprecated Class.newInstance() method usage
add 5a7bd66 Merge pull request #9613: [BEAM-3845] Remove deprecated Class.newInstance() method usage
add 153f806 [BEAM-8343] Moved BeamTable classes to a more appropriate location
add 77f047b Merge pull request #9718: [BEAM-8343] Moved BeamTable classes to a more appropriate location
add a927a3a [BEAM-7657] fix misleading sdk_worker_parallelism description (#9709)
add 798fa4b Add RowJsonSerializer and tests
add ba0e404 Merge pull request #9732: [BEAM-8354] Add `RowJsonSerializer`
add 53d6e09 Move ZetaSQL to Calcite translator to separate module
add 659d84b Merge pull request #9737: [BEAM-8359] Move ZetaSQL to Calcite translator to separate module
add 1cc27c3 Custom container docs
add 1df2e6c Merge pull request #9607 from soyrice/sdk_2-16_docs-updates
add 48ce3a9 Fix closing region tags
add be5753d Merge pull request #9738 from davidcavazos/region-tag-fix
add eff9eb1 [BEAM-6995] Modify BeamBasicAggregationRule to only be applied when windowing is not used
add 5d83b15 Merge pull request #9703: [BEAM-6995] Beam basic aggregation rule only when not windowed
add 45abd1d Update beam website for 2.16.0
add 4733374 Merge pull request #9667: Update beam website for 2.16.0
add 6a3c008 Update downloads.md to correct release date
add 820bb29 [BEAM-7389] Update docs with matching code files (#9692)
add 7327e45 Merge pull request #9671: Beam 2.16.0 release Blog post
add 85c0e83 Add ToJson transform (#9744)
add bf90551 [BEAM-7635] Migrate SnsIO to AWS SDK for Java 2
add ad5d383 Merge pull request #9393: [BEAM-7635] Migrate SnsIO to AWS SDK for Java 2
add 7587aa4 Merge pull request: [BEAM-8309] Document SDK 2.16.0 Python dependencies (#9653)
add 3f705bb [BEAM-6829] Use transform/pcollection name for metric namespace if none provided (#8585)
add 3d700af Add a standard boolean coder
add 85af506 Reject values outside of 0, 1
add b736bad Add BooleanCoder to standard coders in python registry
add 67b4b20 Merge pull request #9735: [BEAM-8355] Add a standard boolean coder
add 153bd848 [BEAM-8333] Adding lull logging for SDK harness (#9746)
add d57e05e [BEAM-8312] Flink jars retrieve artifacts from classloader
add 376c41b Merge pull request #9717 [BEAM-8312] Flink jars retrieve artifacts from classloader
add f6a7e73 [BEAM-8164][website] Correct document for building the python SDK harness container
add 8267c22 Merge pull request #9536 from sunjincheng121/BEAM-8164-PR
add ee5a671 [BEAM-3713] Convert ITs to not use save_main_session
add 603d68a Merge pull request #9514: [BEAM-3713] Convert ITs to not use save_main_session
add cd47522 Remove DoubleCoder from dataflow known coders (#9755)
add a37b5fe Changing visibility of PubsubTestClient
add e6aef76 Merge pull request #9740: Changing Visibility of PubsubTestClient
add 14e68fb [BEAM-8369] update link to new container docs
add 67ef076 Merge pull request #9753: [BEAM-8369] update link to new container docs
add 56a4ea4 [BEAM-8370] update outdated Flink gradle command
add 6b1fc06 Merge pull request #9751: [BEAM-8370] update outdated Flink gradle command
add ea0977d [BEAM-8343] Added nessesary methods to BeamSqlTable to enable support for predicate/project push-down
add 0cb56a2 Merge pull request #9731: [BEAM-8343] Added nessesary methods to BeamSqlTable to enable support for predicate/project push-down
add 025e514 [BEAM-8378] Downgrade build-scan plugin to 2.3 (#9762)
add 07068e2 Use assertIn for inclusion check.
add 9757da1 Merge pull request #9747 Use assertIn for inclusion check in a recently added test.
add 71a4690 [BEAM-8287] Python 3 docs updates for 2.16.0 (#9700)
add f3ce866 Change the default status port of BatchDataflowWorker to 8081 (#9763)
add ee954b5 Document ZetaSQL dialect and add generated ZetaSQL dialect reference
add b288d5a Merge pull request #9306: [BEAM-7939] ZetaSQL dialect documentation
add e026b6d Updates Dataflow container images used by unreleased SDKs.
add 9b372d6 Merge pull request #9766 from apache/tvalentyn-patch-2
add c5b4e64 Tweak ZetaSQL docs wording
add 713e44d Merge pull request #9767: Tweak ZetaSQL docs wording
add 937520d [BEAM-8340]: Support stable id in stateful transforms (#9733)
add 04c3731 BEAM-7739: Rename ValueStateSpec to ReadModifyWrite
add 2e548c5 Merge pull request #9687 from rakeshcusat/BEAM-7739-modify-idl-definition
add 4f4c7c6 [BEAM-8183] restructure Flink portable jars to support multiple pipelines
add 2ff462d Merge pull request #9752: [BEAM-8183] restructure Flink portable jars to support multiple pipelines
add 7f66522 fix for message being not serializable due to LongString in headers
add 05db193 [BEAM-7414] applying Beam project style guides (or trying so)
add 3c845cb [BEAM-7414] Fixing test failures : seems like generated test messages have no header values, which made LongString-elimination code failing
add 36d289b Seems like i changed that line by error. Rolling that back.
add 51fabee That code was useless Indeed, if there is an IOException when trying to ack the message, there most probably be one when trying to nack the message.
add 5252a20 According to jbonofre, the transformation of deliverable into serializable objects lies in RabbitMqMessage And i agree, it's way better.
add d8d8761 fixed one last formatting point
add 5a851b7 Merge pull request #8677 from Riduidel/fix/rabbitmq-message-not-serializable
add b68d70a [BEAM-8391] Update type id write methods in `AwsModule` from deprecated (in jackson 2.9) to supported ones.
add 8543801 Merge pull request #9783: [BEAM-8391] Update type id write methods in
add 453db82 Add project push-down capability to TestTableProvider
add 1c0ab42 Merge pull request #9743: [BEAM-8365] Project push-down for test table provider
add 383f7cd [BEAM-8368] Add a <0.14.0 bound for pyarrow
add 246cc8d Merge pull request #9768 from kamilwu/pyarrow-downgrade
add 3ed2684 [BEAM-7389] Remove old element-wise snippets directory
add 0a00a8f Merge pull request #9786 from davidcavazos/remove-old-element-wise-snippets
add 71b9292 [BEAM-8387] Remove sdk-worker-parallelism option from JobServerDriver
add 6ab60dc Merge pull request #9785: [BEAM-8387] Remove sdk-worker-parallelism option from JobServerDriver
add 8a661a4 [BEAM-7073]: Add unit test for Avro logical type datum.
add b8bef24 Merge pull request #9773 from RyanSkraba/BEAM-7073-avro-sql-unit-test
add 2a367c3 [BEAM-4046] Remove old project name mappings. (#8915)
add 69e840b [BEAM-8375]Correct the command for Start a Flink job server in `environments.md`
add 14b5ef8 Merge pull request #9759: [BEAM-8375] Correct Gradle command on environments page
add ad5a2c3 [BEAM-8176] Fix BigQuery Java test jobs
add 8b475bc Merge pull request #9520: [BEAM-8176] Fix BigQuery Java test jobs
add e8b101f [BEAM-8401] Upgrade to ZetaSQL 2019.10.1
add d2f974c Merge pull request #9802: [BEAM-8401] Upgrade to ZetaSQL 2019.10.1
add 4f3bdb3 Implement ZipFile backed artifact service.
add b3efb01 Make compatible with Java artifact services.
add 5618350 [BEAM-8372] Job server submitting UberJars directly to Flink Runner.
add b658172 Account for post BEAM-8183 file structure.
add fa74467 Merge pull request #9775 from robertwb/easy-flink
add 1458891 Fixing a logging bug (#9795)
add 0ff53f1 Fixing a logging bug in auth (#9796)
add 2aa2e76 Replaced depricated calls
add ab4d995 Merge pull request #9798: [BEAM-8404] Replaced deprecated calls
add 21bd455 Showing the Java environment used by Dataflow in the UI (#8680)
add 3556b67 Clean up a few places where we pass arbitrary kwargs to PTransform constructor.
add e4ca3e1 Merge pull request #9674 from tvalentyn/ptransform_cleanup
add c96ce16 [BEAM-8394]: Add withDataSourceConfiguration() method in JdbcIO.ReadRows
add fe4db4e Merge pull request #9799: [BEAM-8394]: Add withDataSourceConfiguration() method in JdbcIO.ReadRows
add 2e91b7a [BEAM-8403] Guard request id generation to prevent concurrent worker access
add 1ddaa3d Merge pull request #9800: [BEAM-8403] Guard request id generation to prevent concurrent worker access
add 6b18af9 [BEAM-8390] usable useCorrelationId with RabbitMq
add 58ad984 Merge pull request #9782: [BEAM-8390] Allow specifying useCorrelationId for RabbitMqIO.Read
add cc533c1 [Go SDK] Fix subtle split error. Add more tests. (#9750)
add ae76f28 [BEAM-8385] Add a KafkaIOIT hash for 100M records (#9770)
add 3330069 [BEAM-8350] Upgrade to Pylint 2.4 (#9725)
add 12d0774 [BEAM-8367] Using insertId for BQ streaming inserts (#9797)
add 5776e43 [BEAM-8372] Follow-up on Flink UberJar submission.
add 432e17f Merge pull request #9803: [BEAM-8372] Follow-up to Flink UberJar submission.
add e06cece Updated price model and added a rule to perform project push-down for TestTableProvider
add f1aee7c Merge pull request #9764: [BEAM-8365] Project push-down for TestTableProvider
add bb8d54a [BEAM-8355] Add standard bool coder to Go SDK
add d9c3ccf add missing license to exec/coder_test.go
add aa8acb0 Merge pull request #9809 [BEAM-8355] Add standard bool coder to Go SDK
add 5e94175 Small improvement. (#9818)
add 55b8b0a [BEAM-8424] Lower the timeout value for Dataflow ValidatesRunner tests
add 82b3e39 [BEAM-8424] Set timeout value of java 11 dataflow validates runner tests equal to java 8 equivalents
add d9add56 Merge pull request #9819 from lgajowy/BEAM-8424
add d6bcb03 [BEAM-8417] Expose ExternalWorkerHandler hostname
add 274f73b Merge pull request #9814: [BEAM-8417] Expose ExternalWorkerHandler hostname
add cdb38bf [BEAM-8416] Add locking to zipfile reading as well as zipfile writing.
add 358104c Merge pull request #9825 [BEAM-8416] Add locking to zipfile reading as well as zipfile writing.
add 4da0afe [BEAM-8281] Refactor the IOIT settings
add 7b67a92 Merge pull request #9638: [BEAM-8281] Resize IOITs datasets
add 3ff698e [BEAM-8412] xlang test: set sdk worker parallelism to 1
add 16b4afa Merge pull request #9817 from ibzib/xlang-test
add 030231a Test DataCatalogBigQueryIT on ZetaSQL Planner
add ca9f1a2 Merge pull request #9828: Test ZetaSQLPlanner in DataCatalogBigQueryIT
add 9231341 Adds a section on how to make reviewer's job easier in Contributor Guide (#9826)
add 31cff02 [BEAM-8433] DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialects.
add a070c16 Merge pull request #9831: [BEAM-8433] DataCatalogBigQueryIT runs for both Calcite and ZetaSQL dialect
add 7262b62 Fixing BQ streaming inserts test (#9824)
add ae8f383 [BEAM-8415] Improving error message when applying PTransform with a name that already exists in the pipeline
add c2df856 Merge pull request #9812 from davidyan74/err
add 19cf849 Fix javadoc of Pardo
add a834ab3 Merge pull request #9801 from echauchot/fix_pardo_javadoc
add 416f62b [BEAM-8410] JdbcIO should support setConnectionInitSqls in its DataSource
add e065ce8 Merge pull request #9808: [BEAM-8410] JdbcIO should support setConnectionInitSqls in its DataSource
add 0c2174b [BEAM-8430] Change py default sdk_worker_parallelism to 1
add 6df727b Merge pull request #9829: [BEAM-8430] Change py default sdk_worker_parallelism to 1
add 19e9d10 [BEAM-5690] Fix Zero value issue with countByKey transform in Spark Runner
add a24e256 [BEAM-5690] Addressing review comments
add d577262 Merge pull request #9567 from bmv126/count_by_key_fix_sr
add 49ef57b [BEAM-8396] Default to LOOPBACK mode for local flink runner.
add 3b491ba Merge pull request #9833 [BEAM-8396] Default to LOOPBACK mode for local flink runner.
add 49def1e [BEAM-8439] Delay bundle start until pushed back data is emitted
add 7f8e4e4 Merge pull request #9841 from je-ik/BEAM-8439: [BEAM-8439] avoid creation of empty bundles
add 828fae2 Move test dependency to testCompile scope
add 934d5d1 Make CalciteQueryPlanner public, since users can choose query planner
add 210f611 Inline DataCatalogBigQueryIT to its only subclass
add 6ed6497 Merge pull request #9838: [BEAM-8433] Minor refactors to Data Catalog BigQuery integration test config & test class
add 5a8b276 Reduce the verbosity of local_job_service_main
add 0301cda Merge pull request #9813 from udim/portability-less-logging
add f2af848 [BEAM-8416] Hold the lock for the duration, not just the creation, of the iterator.
add c4ffa78 Merge pull request #9845 [BEAM-8416] Hold the lock for the duration, not just the creation, of the iterator.
add 8915d6e Fix some not allowed nulls in ZetaSQL translator
add e2f0e5c Use UnsupportedOperationException for not implemented methods in ZetaSQL planner
add 0d4bf60 Merge pull request #9837: [BEAM-8445] Fix some not allowed nulls in ZetaSQL translator
add 44b2c65 Add project push-down for BigQuery
add e4aab40 Merge pull request #9823: [BEAM-8428] [SQL] Add project push-down for BigQuery
add a799dd0 Use JUnit Parameterized runner for exercising different SQL planners
add face155 Merge pull request #9835: [BEAM-8433] Use JUnit parameterized runner for dialect-sensitive integration tests
add 2575d9d [BEAM-8405] Support embedded Datastore entities (#9805)
add f454f97 [BEAM-7730] Introduce Flink 1.9 Runner.
add 4a64965 [BEAM-7730] Reuse sources overrides from previous flink versions.
add d34bbbd [BEAM-7730] Fix spotless plugin for flink runner.
add 686bfd2 Merge pull request #9296: [BEAM-7730] Introduce Flink 1.9 Runner
add a423bb5 Updating the flink runner docs
add 0f35360 Merge pull request #9761: Update the Flink runner docs to include Docker images
add 307bce9 [BEAM-7730] Migrate docs and integration tests to Flink 1.9
add 99484db Merge pull request #9850: [BEAM-7730] Migrate docs and integration tests to Flink 1.9
add e0adc9a Make beam_fn_api opt-out rather than opt-in for runners.
add 03f681e Workaround for BEAM-8436, BEAM-8444, and BEAM-8448.
add 4ba731f Merge pull request #9834: Make beam_fn_api opt-out rather than opt-in for runners
add 6ccc9e7 [BEAM-7730] Upgrade Flink 1.9.x to 1.9.1.
add 1b01d91 Merge pull request #9851: [BEAM-7730] Upgrade Flink from 1.9.0 to 1.9.1.
add e43c3cd [BEAM-8449] Document known issue with macOS installation (#9847)
add fccd4e2 Add ValidatesRunner test for Impulse transform
add 250ab6f Use base64 string for representing impulse payload in legacy DF runner codepath.
add b3e55a3 Merge pull request #9822 [BEAM-8418] Use base64 string for representing impulse payload in DF runner legacy codepath.
add 4fa1501 [BEAM-7389] Show code snippet outputs as stdout (#9790)
add a7c8766 [BEAM-7520] Add ValidatesRunner test for timer ordering
add c29d06b [BEAM-7520] Synchronize watermarks with bundle processing and add timer pushback
add 0049037 [BEAM-7520] add UsesStrictTimerOrdering category, ignore test for samza and portable flink
add 2ef2736 Merge pull request #9190 from je-ik/BEAM-7520: [BEAM-7520] Fix timer firing order in DirectRunner
add 52c45a7 Fixing BQ write transform
add eb40876 Merge pull request #9856 from pabloem/fixbqwr
add 1a8391d Merge pull request #9854 from [BEAM-8457] Label Dataflow jobs from Notebook
add a116aed [Go SDK] Fix Logging call depths.
add c211d36 [Go SDK] Track failed instructions in harness.
add 8e574a1 [Go SDK] Fix log on failed data channel read.
add 06fa1cc [BEAM-7981] Fix double iterable stripping (#9708)
add 65e6d7b [BEAM-7926] Update dependencies in Java Katas
add 9f27a1d Merge pull request #9771 from leonardoam/BEAM-8356
add ba15657 Add DoFn lifecycle sequence diagram to website
add 5add560 Add Source lifecycle sequence diagram to website
add fe09249 Fix menus and width for lifecycle images
add 6623569 Add introductory content to the lifecycle sequence diagrams
add 20b6913 Merge pull request #9807 from echauchot/update_website_with_sequence_diagrams
add 64263a5 [BEAM-8393] Fix Java BigQueryIO clustering support for multiple partitions
add 19799cd Merge pull request #9784: [BEAM-8393] Fix Java BigQueryIO clustering support for multiple partitions
add 8142228 Moving to 2.18.0-SNAPSHOT on master branch.
add 7f93150 Revert "Moving to 2.18.0-SNAPSHOT on master branch."
add f21f417 Moving to 2.18.0-SNAPSHOT on master branch.
add 25054b8 [BEAM-5967] Add handling of DynamicMessage in ProtoCoder
add ba4a121 [BEAM-5967] Add Nullable annotations and corrected JavaDoc after review
add cb77473 Backported ProtoDomainTest for schema aware PR
add 64829e1 Split the ProtoCoder into ProtoCoder and DynamicProtoCoder.
add f67e3d8 Merge pull request #8496 from [BEAM-5967] Add handling of DynamicMessage in ProtoCoder
add a9cace3 [BEAM-8341]: basic bundling support for portable runner (#9777)
add 177c134 [BEAM-7389] Add code snippet for CoGroupByKey (#9791)
add 43ce5ca [Go SDK] Fix post commits: runtime error in logger
add 4725b54 [BEAM-7520] ignore UsesStrictTimerOrdering for both batch and streaming flink
add 21f9fe1 Merge pull request #9869 from je-ik/BEAM-7520-ignore-timer-ordering-for-flink-pvr-batch:[BEAM-7520] ignore UsesStrictTimerOrdering for Flink
add 7392db0 [BEAM-7730] Add ability to override resources for a specific flink runner version.
add f40cbad Merge pull request #9857: [BEAM-7730] Add ability to override resources for a specific flink runner verison
add 05ee7f1 [BEAM-8462] Bump up Flink version inside gradle task name
add e456b7a [BEAM-8462] Put Hadoop jar into the FLINK_INSTALL_DIR/lib directory
add c4287a3 [BEAM-8462] Add --region=global parameter to gcloud dataproc calls
add 90460b4 Merge pull request #9867: [BEAM-8462] Upgrade Flink version in load tests to 1.9
add 1d59072 Revert "Merge pull request #9854 from [BEAM-8457] Label Dataflow jobs from Notebook"
add 9a9b5f4 Merge pull request #9879 from apache/revert-9854-BEAM-8457
add b0d4dbf [Go SDK] Propagate data channel failures.
add e9c7f99 [Go SDK] Simplify datamgr tests.
add 95bd4b2 [BEAM-7738] Add external transform support to PubsubIO
add 61814dd Adapt io.external.pubsub reader and writer to new API
add 6dff94d Adjust the way that the new API is used to handle the encoded data
add 63cabe7 Must use Boolean arg to make use of the BooleanCoder in Java
add b1e3fd1 Fix docs
add 7b3ce81 Mark kafka and pubsub external transforms as experimental
add 73642eb Merge pull request #9268: [BEAM-7738] Add external transform support to PubsubIO
add a365c67 Combine serializer and deserializer
add d335596 Pass along exception
add 1239781 expose of method
add 6f9e11a fixup!
add 0cf8ae8 Merge incorrectly split strings
add 9f8fdd1 Merge pull request #9757: [BEAM-8484] RowJsonDeserializer/ToJson cleanup
add 90f5dff [BEAM-8456] Add pipeline option to have Data Catalog truncate sub-millisecond precision.
add b2329dc Inline Data Catalog TableUtils to its only use
add 01c355b Make BigQueryUtils, PubsubUtils, GcsUtils implementations of TableFactory; respect truncateTimestamps option
add 23f9e5c Merge pull request #9849: [BEAM-8456] Add pipeline option to have Data Catalog truncate sub-millisecond precision
add 9c0a628 In Go, base counter names on PTransform unique_name
add 70190ab [BEAM-8480] Explicitly set restriction coder for bounded reader wrapper SDF.
add 51efa2e Merge pull request #9877 [BEAM-8480] Explicitly set restriction coder for bounded reader wrapper SDF.
add 4b5f7e5 Merge #9858 for [BEAM-8446] Adding a test checking the wait for BQ jobs (#9858)
add c3c5999 Improve user support story - and add community events link (#9875)
add 59e11f7 Job timeout after 120 minutes instead of 100
add 8972e5e Merge pull request #9896 from [BEAM-8481] Increasing Py3.7 postcommit job timeout after 120 minutes instead of 100
add 501df3a [BEAM-8493] Add standard double coder to Go SDK.
add 6418fb6 Move sequence diagrams to PlantUML SVG.
add 98eb81d Merge pull request #9870 from RyanSkraba/rskraba/svg-sequence-diags
add 14c33ec [BEAM-8372] Support both flink_master and flink_master_url parameter in FlinkJobServerDriver
add f7c069a [BEAM-8372] Match the Java SDK's default of 'flink_master'
add e6eb97c Merge pull request #9844: [BEAM-8372] Support flink_master parameter in job server
add 263abfe [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0
add fc882e8 [BEAM-8470] Fix missing dep
add 3c24550 [BEAM-8470] Add SparkPipelineOptions
add 6f6e650 [BEAM-8470] Start pipeline translation
add 3c16dc5 [BEAM-8470] Add global pipeline translation structure
add be3ff45 [BEAM-8470] Add nodes translators structure
add 003b239 [BEAM-8470] Wire node translators with pipeline translator
add 27d1e23 [BEAM-8470] Renames: better differenciate pipeline translator for transform translator
add 0471b35 [BEAM-8470] Organise methods in PipelineTranslator
add 9b367bd [BEAM-8470] Initialise BatchTranslationContext
add 5b1c2af [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
add 7bb6a3e [BEAM-8470] Make transform translation clearer: renaming, comments
add 357e77a [BEAM-8470] Improve javadocs
add c7250cd [BEAM-8470] Move SparkTransformOverrides to correct package
add 2f5aa47 [BEAM-8470] Move common translation context components to superclass
add 3bf9f4d [BEAM-8470] apply spotless
add 580ccca [BEAM-8470] Make codestyle and firebug happy
add 3cc6a3e [BEAM-8470] Add TODOs
add 82001d4 [BEAM-8470] Post-pone batch qualifier in all classes names for readability
add fe5afe5 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN
add d8bbeeb [BEAM-8470] Added SparkRunnerRegistrar
add c6dc3ca [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
add 345b963 [BEAM-8470] Create PCollections manipulation methods
add 0e0a754 [BEAM-8470] Create Datasets manipulation methods
add efbcb27 [BEAM-8470] Add Flatten transformation translator
add 91388b1 [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation
add 2996fc0 [BEAM-8470] Use Iterators.transform() to return Iterable
add 4b8bae4 [BEAM-8470] Implement read transform
add 7d5421a [BEAM-8470] update TODO
add b1e5eab [BEAM-8470] Apply spotless
add dd62b16 [BEAM-8470] start source instanciation
add b91cd39 [BEAM-8470] Improve exception flow
add a4f4c27 [BEAM-8470] Improve type enforcement in ReadSourceTranslator
add 4d920fd [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table
add 99f1d37 [BEAM-8470] Add source mocks
add a0a072e [BEAM-8470] fix mock, wire mock in translators and create a main test.
add e14ea16 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary)
add ec0a390 [BEAM-8470] clean deps
add 3ba8cf2 [BEAM-8470] Move DatasetSourceMock to proper batch mode
add f2c3e98 [BEAM-8470] Run pipeline in batch mode or in streaming mode
add 15ae8ba [BEAM-8470] Split batch and streaming sources and translators
add 788f960 [BEAM-8470] Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
add 71a5d91 [BEAM-8470] Clean
add 2742c60 [BEAM-8470] Add ReadSourceTranslatorStreaming
add 412e5b8 [BEAM-8470] Move Source and translator mocks to a mock package.
add 0d4e7e5 [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
add 1ded190 [BEAM-8470] Refactor DatasetSource fields
add c662ee6 [BEAM-8470] Wire real SourceTransform and not mock and update the test
add 1ec8213 [BEAM-8470] Add missing 0-arg public constructor
add 3120789 [BEAM-8470] Use new PipelineOptionsSerializationUtils
add fed2cfd [BEAM-8470] Apply spotless and fix checkstyle
add 749668a [BEAM-8470] Add a dummy schema for reader
add e9fe234 [BEAM-8470] Add empty 0-arg constructor for mock source
add 0f2cfb6 [BEAM-8470] Clean
add 383c0a6 [BEAM-8470] Checkstyle and Findbugs
add abc4600 [BEAM-8470] Refactor SourceTest to a UTest instaed of a main
add 475ed56 [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset
add 085b516 [BEAM-8470] improve readability of options passing to the source
add 8416e7b [BEAM-8470] Clean unneeded fields in DatasetReader
add e260def [BEAM-8470] Fix serialization issues
add 1f35595 [BEAM-8470] Add SerializationDebugger
add bde9f8b [BEAM-8470] Add serialization test
add fc21488 [BEAM-8470] Move SourceTest to same package as tested class
add 8948cf3 [BEAM-8470] Fix SourceTest
add 231285c [BEAM-8470] Simplify beam reader creation as it created once the source as already been partitioned
add 5920917 [BEAM-8470] Put all transform translators Serializable
add 8202fa1 [BEAM-8470] Enable test mode
add 0a15995 [BEAM-8470] Enable gradle build scan
add 4f808c1 [BEAM-8470] Add flatten test
add 5219d9b [BEAM-8470] First attempt for ParDo primitive implementation
add f30982a [BEAM-8470] Serialize windowedValue to byte[] in source to be able to specify a binary dataset schema and deserialize windowedValue from Row to get a dataset<WindowedValue>
add eab170c [BEAM-8470] Comment schema choices
add 0c9ff45 [BEAM-8470] Fix errorprone
add 99b4c4a [BEAM-8470] Fix testMode output to comply with new binary schema
add ab0ff96 [BEAM-8470] Cleaning
add 0913635 [BEAM-8470] Remove bundleSize parameter and always use spark default parallelism
add 54d015a [BEAM-8470] Fix split bug
add e15e6cd [BEAM-8470] Clean
add ec584e8 [BEAM-8470] Add ParDoTest
add 105ec04 [BEAM-8470] Address minor review notes
add 8642beb [BEAM-8470] Clean
add eb3e6c3 [BEAM-8470] Add GroupByKeyTest
add 6d5d382 [BEAM-8470] Add comments and TODO to GroupByKeyTranslatorBatch
add bed3340 [BEAM-8470] Fix type checking with Encoder of WindowedValue<T>
add b2b742b [BEAM-8470] Port latest changes of ReadSourceTranslatorBatch to ReadSourceTranslatorStreaming
add 268ac53 [BEAM-8470] Remove no more needed putDatasetRaw
add 2a1ea36 [BEAM-8470] Add ComplexSourceTest
add c5e4407 [BEAM-8470] Fail in case of having SideInouts or State/Timers
add f600fd1 [BEAM-8470] Fix Encoders: create an Encoder for every manipulated type to avoid Spark fallback to genericRowWithSchema and cast to allow having Encoders with generic types such as WindowedValue<T> and get the type checking back
add ea34002 [BEAM-8470] Apply spotless
add 83547f4 [BEAM-8470] Fixed Javadoc error
add 75d32df [BEAM-8470] Rename SparkSideInputReader class and rename pruneOutput() to pruneOutputFilteredByTag()
add 0b800d9 [BEAM-8470] Don't use deprecated sideInput.getWindowingStrategyInternal()
add 6dd56cc [BEAM-8470] Simplify logic of ParDo translator
add e144c24 [BEAM-8470] Fix kryo issue in GBK translator with a workaround
add 338f836 [BEAM-8470] Rename SparkOutputManager for consistency
add 88b8edd [BEAM-8470] Fix for test elements container in GroupByKeyTest
add e61507d [BEAM-8470] Added "testTwoPardoInRow"
add b5057cc [BEAM-8470] Add a test for the most simple possible Combine
add 06343d4 [BEAM-8470] Rename SparkDoFnFilterFunction to DoFnFilterFunction for consistency
add 6c7a9ed [BEAM-8470] Generalize the use of SerializablePipelineOptions in place of (not serializable) PipelineOptions
add bc3ca37 [BEAM-8470] Fix getSideInputs
add 4bd38c14 [BEAM-8470] Extract binary schema creation in a helper class
add 52634ef [BEAM-8470] First version of combinePerKey
add 2474926 [BEAM-8470] Improve type checking of Tuple2 encoder
add 746fee2 [BEAM-8470] Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey
add fc8843d [BEAM-8470] Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless
add b1749ca [BEAM-8470] Add combinePerKey and CombineGlobally tests
add edc7a81 [BEAM-8470] Introduce RowHelpers
add 911cdb4 [BEAM-8470] Add CombineGlobally translation to avoid translating Combine.perKey as a composite transform based on Combine.PerKey (which uses low perf GBK)
add c2eb95b [BEAM-8470] Cleaning
add 237ee28 [BEAM-8470] Get back to classes in translators resolution because URNs cannot translate Combine.Globally
add 6f40095 [BEAM-8470] Fix various type checking issues in Combine.Globally
add bd985de [BEAM-8470] Update test with Long
add 2a04f4a [BEAM-8470] Fix combine. For unknown reason GenericRowWithSchema is used as input of combine so extract its content to be able to proceed
add 642833d [BEAM-8470] Use more generic Row instead of GenericRowWithSchema
add 5f5f097 [BEAM-8470] Add explanation about receiving a Row as input in the combiner
add 6dbe5af [BEAM-8470] Fix encoder bug in combinePerkey
add 0d8595a [BEAM-8470] Cleaning
add 1eb712c [BEAM-8470] Implement WindowAssignTranslatorBatch
add 55cd936 [BEAM-8470] Implement WindowAssignTest
add 3986c35 [BEAM-8470] Fix javadoc
add 4ea5960 [BEAM-8470] Added SideInput support
add 231f1cd [BEAM-8470] Fix CheckStyle violations
add bd54aa4 [BEAM-8470] Don't use Reshuffle translation
add 7a02467 [BEAM-8470] Added using CachedSideInputReader
add 1dd1edd [BEAM-8470] Added TODO comment for ReshuffleTranslatorBatch
add 6011073 [BEAM-8470] And unchecked warning suppression
add 04ed002 [BEAM-8470] Add streaming source initialisation
add 683e6f8 [BEAM-8470] Implement first streaming source
add 4eff413 [BEAM-8470] Add a TODO on spark output modes
add eafbd73 [BEAM-8470] Add transformators registry in PipelineTranslatorStreaming
add ebba00f [BEAM-8470] Add source streaming test
add 801c823 [BEAM-8470] Specify checkpointLocation at the pipeline start
add 64e8adc [BEAM-8470] Clean unneeded 0 arg constructor in batch source
add e3c3960 [BEAM-8470] Clean streaming source
add 94d4fae [BEAM-8470] Continue impl of offsets for streaming source
add f4cd419 [BEAM-8470] Deal with checkpoint and offset based read
add 63be481 [BEAM-8470] Apply spotless and fix spotbugs warnings
add a8112e0 [BEAM-8470] Disable never ending test SimpleSourceTest.testUnboundedSource
add b02809e [BEAM-8470] Fix access level issues, typos and modernize code to Java 8 style
add a0b2f1b [BEAM-8470] Merge Spark Structured Streaming runner into main Spark module. Remove spark-structured-streaming module.Rename Runner to SparkStructuredStreamingRunner. Remove specific PipelineOotions and Registrar for Structured Streaming Runner
add 31ae06e [BEAM-8470] Fix non-vendored imports from Spark Streaming Runner classes
add 1e2b5af [BEAM-8470] Pass doFnSchemaInformation to ParDo batch translation
add 8a99531 [BEAM-8470] Fix spotless issues after rebase
add 1b4a21c [BEAM-8470] Fix logging levels in Spark Structured Streaming translation
add ae130cb [BEAM-8470] Add SparkStructuredStreamingPipelineOptions and SparkCommonPipelineOptions - SparkStructuredStreamingPipelineOptions was added to have the new runner rely only on its specific options.
add 9448630 [BEAM-8470] Rename SparkPipelineResult to SparkStructuredStreamingPipelineResult This is done to avoid an eventual collision with the one in SparkRunner. However this cannot happen at this moment because it is package private, so it is also done for consistency.
add 81bae1a [BEAM-8470] Use PAssert in Spark Structured Streaming transform tests
add 71dfb9b [BEAM-8470] Ignore spark offsets (cf javadoc)
add d8c27b8 [BEAM-8470] implement source.stop
add 85fe1f1 [BEAM-8470] Update javadoc
add d3e1d91 [BEAM-8470] Apply Spotless
add 91b00f8 [BEAM-8470] Enable batch Validates Runner tests for Structured Streaming Runner
add 745aee3 [BEAM-8470] Limit the number of partitions to make tests go 300% faster
add e03157d [BEAM-8470] Fixes ParDo not calling setup and not tearing down if exception on startBundle
add 0910904 [BEAM-8470] Pass transform based doFnSchemaInformation in ParDo translation
add b348923 [BEAM-8470] Consider null object case on RowHelpers, fixes empty side inputs tests.
add 4ee0993 [BEAM-8470] Put back batch/simpleSourceTest.testBoundedSource
add e9eb45c [BEAM-8470] Update windowAssignTest
add 657ab5e [BEAM-8470] Add comment about checkpoint mark
add 654ae07 [BEAM-8470] Re-code GroupByKeyTranslatorBatch to conserve windowing instead of unwindowing/windowing(GlobalWindow): simplify code, use ReduceFnRunner to merge the windows
add 5dc128d [BEAM-8470] re-enable reduceFnRunner timers for output
add bffa608 [BEAM-8470] Improve visibility of debug messages
add 97a1bc8 [BEAM-8470] Add a test that GBK preserves windowing
add cf58316 [BEAM-8470] Add TODO in Combine translations
add cd494bd [BEAM-8470] Update KVHelpers.extractKey() to deal with WindowedValue and update GBK and CPK
add 04bc6a3 [BEAM-8470] Fix comment about schemas
add 37ca462 [BEAM-8470] Implement reduce part of CombineGlobally translation with windowing
add e1dc45c [BEAM-8470] Output data after combine
add dec6a1d [BEAM-8470] Implement merge accumulators part of CombineGlobally translation with windowing
add bca3996 [BEAM-8470] Fix encoder in combine call
add e92dca1 [BEAM-8470] Revert extractKey while combinePerKey is not done (so that it compiles)
add 94d0057 [BEAM-8470] Apply a groupByKey avoids for some reason that the spark structured streaming fmwk casts data to Row which makes it impossible to deserialize without the coder shipped into the data. For performance reasons (avoid memory consumption and having to deserialize), we do not ship coder + data. Also add a mapparitions before GBK to avoid shuffling
add 9f7136b [BEAM-8470] Fix case when a window does not merge into any other window
add 2bb7a3a [BEAM-8470] Fix wrong encoder in combineGlobally GBK
add fb3c17c [BEAM-8470] Fix bug in the window merging logic
add 2d89260 [BEAM-8470] Remove the mapPartition that adds a key per partition because otherwise spark will reduce values per key instead of globally
add 30ad19c [BEAM-8470] Remove CombineGlobally translation because it is less performant than the beam sdk one (key + combinePerKey.withHotkeyFanout)
add c0b7159 [BEAM-8470] Now that there is only Combine.PerKey translation, make only one Aggregator
add 3df2363 [BEAM-8470] Clean no more needed KVHelpers
add 62e003c [BEAM-8470] Clean not more needed RowHelpers
add d6f5126 [BEAM-8470] Clean not more needed WindowingHelpers
add a63290d [BEAM-8470] Fix javadoc of AggregatorCombiner
add 588ca31 [BEAM-8470] Fixed immutable list bug
add aa3288e [BEAM-8470] add comment in combine globally test
add 737999f [BEAM-8470] Clean groupByKeyTest
add 0266777 [BEAM-8470] Add a test that combine per key preserves windowing
add c76654f [BEAM-8470] Ignore for now not working test testCombineGlobally
add b1d60d4 [BEAM-8470] Add metrics support in DoFn
add ef70702 [BEAM-8470] Add missing dependencies to run Spark Structured Streaming Runner on Nexmark
add e5dabeb [BEAM-8470] Add setEnableSparkMetricSinks() method
add 2fbcba8 [BEAM-8470] Fix javadoc
add 66b61d6 [BEAM-8470] Fix accumulators initialization in Combine that prevented CombineGlobally to work.
add 78fea89 [BEAM-8470] Add a test to check that CombineGlobally preserves windowing
add 8303a47 [BEAM-8470] Persist all output Dataset if there are multiple outputs in pipeline Enabled Use*Metrics tests
add 820d422 [BEAM-8470] Added metrics sinks and tests
add e039015 [BEAM-8470] Make spotless happy
add c6b638e [BEAM-8470] Add PipelineResults to Spark structured streaming.
add 62e6b01 [BEAM-8470] Update log4j configuration
add 4bbca76 [BEAM-8470] Add spark execution plans extended debug messages.
add 4e06a92 [BEAM-8470] Print number of leaf datasets
add 829b76e [BEAM-8470] fixup! Add PipelineResults to Spark structured streaming.
add c4b2af1 [BEAM-8470] Remove no more needed AggregatorCombinerPerKey (there is only AggregatorCombiner)
add ff72b1b [BEAM-8470] After testing performance and correctness, launch pipeline with dataset.foreach(). Make both test mode and production mode use foreach for uniformity. Move dataset print as a utility method
add 8a866cd [BEAM-8470] Improve Pardo translation performance: avoid calling a filter transform when there is only one output tag
add cb36530 [BEAM-8470] Use "sparkMaster" in local mode to obtain number of shuffle partitions + spotless apply
add 666a060 [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: serialization part
add 2a53396 [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: deserialization part
add eef809c [BEAM-8470] type erasure: spark encoders require a Class<T>, pass Object and cast to Class<T>
add 637d8dd [BEAM-8470] Fix scala Product in Encoders to avoid StackEverflow
add 299bb3d [BEAM-8470] Conform to spark ExpressionEncoders: pass classTags, implement scala Product, pass children from within the ExpressionEncoder, fix visibilities
add cf4cc45 [BEAM-8470] Add a simple spark native test to test Beam coders wrapping into Spark Encoders
add d36d977 [BEAM-8470] Fix code generation in Beam coder wrapper
add 4f95b9d [BEAM-8470] Lazy init coder because coder instance cannot be interpolated by catalyst
add a4ddeef [BEAM-8470] Fix warning in coder construction by reflexion
add f4ebf7d [BEAM-8470] Fix ExpressionEncoder generated code: typos, try catch, fqcn
add 5941bca [BEAM-8470] Fix getting the output value in code generation
add c237789 [BEAM-8470] Fix beam coder lazy init using reflexion: use .clas + try catch + cast
add 31bc268 [BEAM-8470] Remove lazy init of beam coder because there is no generic way on instanciating a beam coder
add 261c181 [BEAM-8470] Remove example code
add 4f4478a [BEAM-8470] Fix equal and hashcode
add 6b53a09 [BEAM-8470] Fix generated code: uniform exceptions catching, fix parenthesis and variable declarations
add 0ad6034 [BEAM-8470] Add an assert of equality in the encoders test
add 3120646 [BEAM-8470] Apply spotless and checkstyle and add javadocs
add 96ff856 [BEAM-8470] Wrap exceptions in UserCoderExceptions
add 8de3a5d [BEAM-8470] Put Encoders expressions serializable
add 887f4d3 [BEAM-8470] Catch Exception instead of IOException because some coders to not throw Exceptions at all (e.g.VoidCoder)
add 69dfd52 [BEAM-8470] Apply new Encoders to CombinePerKey
add d398ea0 [BEAM-8470] Apply new Encoders to Read source
add 1429d09 [BEAM-8470] Improve performance of source: the mapper already calls windowedValueCoder.decode, no need to call it also in the Spark encoder
add dd52c2f [BEAM-8470] Ignore long time failing test: SparkMetricsSinkTest
add f594e96 [BEAM-8470] Apply new Encoders to Window assign translation
add 766ad7f [BEAM-8470] Apply new Encoders to AggregatorCombiner
add f4f34fc [BEAM-8470] Create a Tuple2Coder to encode scala tuple2
add 5f0f28b [BEAM-8470] Apply new Encoders to GroupByKey
add 717aafc [BEAM-8470] Apply new Encoders to Pardo. Replace Tuple2Coder with MultiOutputCoder to deal with multiple output to use in Spark Encoder for DoFnRunner
add 1835cdb [BEAM-8470] Apply spotless, fix typo and javadoc
add 216b9ca [BEAM-8470] Use beam encoders also in the output of the source translation
add c794671 [BEAM-8470] Remove unneeded cast
add 47526cf [BEAM-8470] Fix: Remove generic hack of using object. Use actual Coder encodedType in Encoders
add 573d177 [BEAM-8470] Remove Encoders based on kryo now that we call Beam coders in the runner
add f73ba27 [BEAM-8470] Add a jenkins job for validates runner tests in the new spark runner
add 9c4a6ac [BEAM-8470] Apply spotless
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (37ae68a)
\
N -- N -- N refs/heads/spark-runner_structured-streaming (9c4a6ac)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.gitattributes | 3 +-
.github/PULL_REQUEST_TEMPLATE.md | 6 +-
.test-infra/dataproc/create_flink_cluster.sh | 148 -
.test-infra/dataproc/flink_cluster.sh | 161 +
.test-infra/dataproc/init-actions/flink.sh | 9 +-
.test-infra/dockerized-jenkins/README.md | 23 +
.test-infra/jenkins/CommonTestProperties.groovy | 2 +-
.test-infra/jenkins/Docker.groovy | 76 +
.test-infra/jenkins/Flink.groovy | 107 +
.test-infra/jenkins/Infrastructure.groovy | 121 -
.test-infra/jenkins/LoadTestsBuilder.groovy | 4 +-
.test-infra/jenkins/README.md | 4 +-
.../jenkins/job_LoadTests_CoGBK_Java.groovy | 40 +-
.../job_LoadTests_Combine_Flink_Python.groovy | 134 +-
.../jenkins/job_LoadTests_Combine_Java.groovy | 130 +-
.../jenkins/job_LoadTests_Combine_Python.groovy | 76 +-
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 154 +-
.test-infra/jenkins/job_LoadTests_GBK_Java.groovy | 67 +-
.../jenkins/job_LoadTests_GBK_Python.groovy | 50 +-
.../job_LoadTests_GBK_Python_reiterate.groovy | 20 +-
.../jenkins/job_LoadTests_Java_Smoke.groovy | 32 +-
.../job_LoadTests_ParDo_Flink_Python.groovy | 150 +
.../jenkins/job_LoadTests_ParDo_Java.groovy | 40 +-
.../jenkins/job_LoadTests_ParDo_Python.groovy | 34 +-
.../jenkins/job_LoadTests_Python_Smoke.groovy | 18 +-
.../job_LoadTests_coGBK_Flink_Python.groovy | 81 +-
.../jenkins/job_LoadTests_coGBK_Python.groovy | 38 +-
.../job_PerformanceTests_BigQueryIO_Java.groovy | 113 +
.../job_PerformanceTests_BigQueryIO_Python.groovy | 20 +-
.../job_PerformanceTests_FileBasedIO_IT.groovy | 418 ++-
...job_PerformanceTests_FileBasedIO_IT_HDFS.groovy | 187 -
.../job_PerformanceTests_HadoopFormat.groovy | 81 +-
.../jenkins/job_PerformanceTests_JDBC.groovy | 80 +-
.../job_PerformanceTests_MongoDBIO_IT.groovy | 20 +-
.../jenkins/job_PerformanceTests_Python.groovy | 12 +-
...ommit_CrossLanguageValidatesRunner_Flink.groovy | 2 +-
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 2 +-
.../job_PostCommit_Java_Nexmark_Flink.groovy | 8 +-
...Java_PortableValidatesRunner_Flink_Batch.groovy | 2 +-
..._PortableValidatesRunner_Flink_Streaming.groovy | 2 +-
...PostCommit_Java_ValidatesRunner_Dataflow.groovy | 3 +-
...mit_Java_ValidatesRunner_Dataflow_Java11.groovy | 2 +-
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 2 +-
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 3 +-
.../job_PostCommit_PortableJar_Flink.groovy | 38 +
...ostCommit_Python35_ValidatesRunner_Flink.groovy | 38 +
.test-infra/jenkins/job_PostCommit_Python37.groovy | 2 +-
.../job_PostCommit_Python_MongoDBIO_IT.groovy | 1 +
...ommit_Python_ValidatesContainer_Dataflow.groovy | 4 +-
.../jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy | 34 +
.../jenkins/job_PreCommit_Portable_Python.groovy | 32 +-
.../jenkins/job_PreCommit_PythonLint.groovy | 30 +
...b_PreCommit_Python_ValidatesRunner_Flink.groovy | 7 +-
.../jenkins/job_ReleaseCandidate_Python.groovy | 2 +-
.../jenkins/job_Release_Gradle_Build.groovy | 45 +
.../hadoop/LargeITCluster/pkb-config.yml | 44 -
.../hadoop/SmallITCluster/pkb-config.yml | 40 -
.../kafka-cluster/05-kafka/10broker-config.yml | 2 +-
.../kubernetes/kafka-cluster/05-kafka/50kafka.yml | 7 +-
.../kafka-cluster/05-kafka/configmap-config.yaml | 2 +-
.../mongodb/load-balancer/pkb-config.yml | 32 -
.../kubernetes/postgres/pkb-config-local.yml | 34 -
.test-infra/kubernetes/postgres/pkb-config.yml | 32 -
.test-infra/metrics/README.md | 31 +-
.test-infra/metrics/apply_configmaps.sh | 26 +
.test-infra/metrics/beamgrafana-deploy.yaml | 8 +-
.test-infra/metrics/beamprometheus-deploy.yaml | 125 +
.test-infra/metrics/build.gradle | 3 +-
.test-infra/metrics/docker-compose.yml | 28 +-
.../dashboards/stability_critical_jobs_status.json | 13 +-
.../alertmanager/config/alertmanager.yml | 41 +
.../prometheus/prometheus/config/prometheus.yml | 40 +
.../metrics/prometheus/prometheus/config/rules.yml | 35 +
README.md | 6 +-
build.gradle | 32 +-
buildSrc/build.gradle | 10 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 222 +-
deprecation-warning.txt | 18 -
examples/java/build.gradle | 18 +-
.../beam/examples/complete/game/UserScore.java | 7 +-
.../beam/examples/cookbook/BigQueryTornadoes.java | 9 +-
.../apache/beam/examples/snippets/Snippets.java | 118 +
examples/kotlin/build.gradle | 18 +-
.../transforms/python/elementwise/filter-py.ipynb | 512 +++
.../transforms/python/elementwise/flatmap-py.ipynb | 672 ++++
.../transforms/python/elementwise/keys-py.ipynb | 195 +
.../transforms/python/elementwise/kvswap-py.ipynb | 196 +
.../transforms/python/elementwise/map-py.ipynb | 616 ++++
.../transforms/python/elementwise/pardo-py.ipynb | 414 +++
.../python/elementwise/partition-py.ipynb | 404 +++
.../transforms/python/elementwise/regex-py.ipynb | 719 ++++
.../python/elementwise/tostring-py.ipynb | 314 ++
.../transforms/python/elementwise/values-py.ipynb | 195 +
.../python/elementwise/withtimestamps-py.ipynb | 369 ++
gradle.properties | 3 +-
gradlew | 184 +-
gradlew.bat | 93 +-
gradlew_orig | 190 -
gradlew_orig.bat | 102 -
learning/katas/java/build.gradle | 20 +-
model/fn-execution/build.gradle | 5 +-
.../fn-execution/src/main/proto/beam_fn_api.proto | 112 +-
.../beam/model/fnexecution/v1/standard_coders.yaml | 28 +
model/job-management/build.gradle | 10 +-
.../src/main/proto/beam_artifact_api.proto | 4 -
.../src/main/proto/beam_job_api.proto | 19 +
model/pipeline/build.gradle | 5 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 10 +-
model/pipeline/src/main/proto/schema.proto | 2 +-
project-mappings | 112 -
release/build.gradle | 2 +-
release/src/main/groovy/TestScripts.groovy | 2 +-
.../python_release_automation_utils.sh | 14 +-
.../run_release_candidate_python_quickstart.sh | 12 +-
.../src/main/scripts/build_release_candidate.sh | 55 +-
release/src/main/scripts/cut_release_branch.sh | 2 +
.../src/main/scripts/preparation_before_release.sh | 1 +
release/src/main/scripts/publish_docker_images.sh | 77 +
release/src/main/scripts/run_rc_validation.sh | 194 +-
release/src/main/scripts/script.config | 49 +-
release/src/main/scripts/set_version.sh | 2 +
release/src/main/scripts/verify_release_build.sh | 287 +-
runners/apex/build.gradle | 8 +-
.../runners/apex/translation/ParDoTranslator.java | 33 +-
.../translation/operators/ApexParDoOperator.java | 26 +-
.../apex/translation/ParDoTranslatorTest.java | 1 +
runners/core-construction-java/build.gradle | 2 +-
.../runners/core/construction/Environments.java | 3 +-
.../beam/runners/core/construction/External.java | 6 +
.../core/construction/ModelCoderRegistrar.java | 3 +
.../runners/core/construction/ModelCoders.java | 2 +
.../core/construction/ParDoTranslation.java | 48 +-
.../runners/core/construction/ReadTranslation.java | 6 +-
.../core/construction/SchemaTranslation.java | 48 +-
.../runners/core/construction/SplittableParDo.java | 7 +-
.../construction/SplittableParDoNaiveBounded.java | 5 +
.../construction/WindowingStrategyTranslation.java | 4 +
.../construction/expansion/ExpansionService.java | 10 +-
.../construction/graph/GreedyPipelineFuser.java | 9 +-
.../core/construction/CoderTranslationTest.java | 2 +
.../runners/core/construction/CommonCoderTest.java | 8 +
.../CreatePCollectionViewTranslationTest.java | 8 +-
.../core/construction/EnvironmentsTest.java | 3 +-
.../PCollectionViewTranslationTest.java | 6 +
.../core/construction/PTransformMatchersTest.java | 5 +-
.../core/construction/ParDoTranslationTest.java | 4 +-
.../core/construction/ReadTranslationTest.java | 2 +-
.../core/construction/SchemaTranslationTest.java | 8 +-
.../WindowingStrategyTranslationTest.java | 7 +
runners/core-java/build.gradle | 2 +-
.../org/apache/beam/runners/core/DoFnRunners.java | 12 +-
...TimeBoundedSplittableProcessElementInvoker.java | 5 +
.../apache/beam/runners/core/SimpleDoFnRunner.java | 28 +-
.../runners/core/metrics/DefaultMetricResults.java | 2 +-
.../core/metrics/MonitoringInfoMetricName.java | 7 +
.../beam/runners/core/SimpleDoFnRunnerTest.java | 27 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 220 ++
.../beam/runners/core/StatefulDoFnRunnerTest.java | 3 +-
.../core/metrics/MonitoringInfoMetricNameTest.java | 20 +-
runners/direct-java/build.gradle | 24 +-
.../beam/runners/direct/CommittedResult.java | 2 +-
.../direct/CopyOnAccessInMemoryStateInternals.java | 8 +-
.../beam/runners/direct/DirectGraphVisitor.java | 3 +-
.../beam/runners/direct/DirectTimerInternals.java | 12 +
.../beam/runners/direct/EvaluationContext.java | 15 +-
.../direct/ExecutorServiceParallelExecutor.java | 4 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 13 +-
.../beam/runners/direct/ParDoEvaluatorFactory.java | 9 +-
.../runners/direct/ParDoMultiOverrideFactory.java | 24 +-
.../beam/runners/direct/QuiescenceDriver.java | 12 +-
.../beam/runners/direct/SideInputContainer.java | 4 +-
.../SplittableProcessElementsEvaluatorFactory.java | 10 +-
.../direct/StatefulParDoEvaluatorFactory.java | 29 +-
.../runners/direct/WatermarkCallbackExecutor.java | 23 +-
.../beam/runners/direct/WatermarkManager.java | 199 +-
.../beam/runners/direct/CommittedResultTest.java | 10 +-
.../beam/runners/direct/DirectRunnerTest.java | 230 +-
.../direct/DirectTransformExecutorTest.java | 4 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 8 +-
.../beam/runners/direct/ParDoEvaluatorTest.java | 2 +
.../direct/StatefulParDoEvaluatorFactoryTest.java | 20 +-
.../beam/runners/direct/WatermarkManagerTest.java | 39 +-
runners/extensions-java/metrics/build.gradle | 2 +-
runners/flink/1.5/build.gradle | 34 -
.../flink/1.5/job-server-container/build.gradle | 26 -
runners/flink/1.5/job-server/build.gradle | 31 -
.../flink/streaming/FlinkStateInternalsTest.java | 144 -
runners/flink/1.6/build.gradle | 34 -
runners/flink/1.6/job-server/build.gradle | 31 -
runners/flink/1.7/build.gradle | 10 +-
.../translation/types/CoderTypeSerializer.java | 0
.../translation/types/EncodedValueSerializer.java | 0
.../streaming/io/BeamStoppableFunction.java | 29 +
.../FlinkBroadcastStateInternalsTest.java | 0
.../flink/streaming/FlinkStateInternalsTest.java | 145 +
.../runners/flink/streaming/StreamSources.java | 37 +
.../translation/types/CoderTypeSerializerTest.java | 0
runners/flink/1.8/build.gradle | 12 +-
.../flink/streaming/FlinkStateInternalsTest.java | 5 +-
runners/flink/1.9/build.gradle | 33 +
.../{1.6 => 1.9}/job-server-container/build.gradle | 0
runners/flink/1.9/job-server/build.gradle | 31 +
.../streaming/io/BeamStoppableFunction.java | 31 +
.../runners/flink/streaming/StreamSources.java | 49 +
runners/flink/flink_runner.gradle | 74 +-
runners/flink/job-server/flink_job_server.gradle | 21 +
runners/flink/job-server/test_pipeline_jar.sh | 124 +
.../FlinkBatchPortablePipelineTranslator.java | 61 +-
.../flink/FlinkBatchTransformTranslators.java | 10 +-
.../runners/flink/FlinkExecutionEnvironments.java | 20 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 33 +-
.../beam/runners/flink/FlinkJobServerDriver.java | 21 +-
.../beam/runners/flink/FlinkPipelineOptions.java | 33 +-
.../beam/runners/flink/FlinkPipelineRunner.java | 131 +-
.../runners/flink/FlinkPortableRunnerResult.java | 53 +
.../runners/flink/FlinkStateBackendFactory.java | 25 +
.../FlinkStreamingPortablePipelineTranslator.java | 234 +-
.../flink/FlinkStreamingTransformTranslators.java | 107 +-
.../flink/metrics/DoFnRunnerWithMetricsUpdate.java | 5 +-
.../FlinkDefaultExecutableStageContext.java | 122 -
.../translation/functions/FlinkDoFnFunction.java | 20 +-
.../functions/FlinkExecutableStageContext.java | 45 -
.../FlinkExecutableStageContextFactory.java | 70 +
.../functions/FlinkExecutableStageFunction.java | 28 +-
.../FlinkExecutableStagePruningFunction.java | 19 +-
.../FlinkMergingNonShuffleReduceFunction.java | 9 +
.../functions/FlinkMultiOutputPruningFunction.java | 18 +-
.../functions/FlinkPartialReduceFunction.java | 9 +
.../translation/functions/FlinkReduceFunction.java | 9 +
.../functions/FlinkStatefulDoFnFunction.java | 20 +-
...CountingFlinkExecutableStageContextFactory.java | 248 --
.../wrappers/streaming/DoFnOperator.java | 28 +-
.../streaming/ExecutableStageDoFnOperator.java | 58 +-
.../wrappers/streaming/FlinkKeyUtils.java | 11 +-
.../wrappers/streaming/SplittableDoFnOperator.java | 3 +-
.../wrappers/streaming/WindowDoFnOperator.java | 3 +-
.../wrappers/streaming/io/DedupingOperator.java | 15 +
.../streaming/io/StreamingImpulseSource.java | 19 +-
.../streaming/io/UnboundedSourceWrapper.java | 9 +-
.../streaming/state/FlinkStateInternals.java | 12 +-
.../flink/FlinkExecutionEnvironmentsTest.java | 2 +-
.../runners/flink/FlinkJobServerDriverTest.java | 7 +-
.../FlinkPipelineExecutionEnvironmentTest.java | 2 +-
.../runners/flink/FlinkPipelineOptionsTest.java | 12 +-
.../beam/runners/flink/FlinkSavepointTest.java | 6 +-
.../FlinkStreamingTransformTranslatorsTest.java | 37 +-
.../beam/runners/flink/PortableExecutionTest.java | 4 +-
.../runners/flink/PortableStateExecutionTest.java | 4 +-
.../runners/flink/PortableTimersExecutionTest.java | 4 +-
.../beam/runners/flink/ReadSourcePortableTest.java | 4 +-
.../flink/metrics/FlinkMetricContainerTest.java | 36 +-
.../flink/streaming/BoundedSourceRestoreTest.java | 6 +-
.../FlinkDefaultExecutableStageContextTest.java | 76 -
.../FlinkExecutableStageFunctionTest.java | 21 +-
.../functions/ImpulseSourceFunctionTest.java | 4 +-
...tingFlinkExecutableStageContextFactoryTest.java | 115 -
.../wrappers/streaming/DedupingOperatorTest.java | 3 +-
.../wrappers/streaming/DoFnOperatorTest.java | 79 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 160 +-
.../wrappers/streaming/FlinkKeyUtilsTest.java | 26 +-
.../streaming/io/UnboundedSourceWrapperTest.java | 13 +-
runners/gearpump/build.gradle | 2 +-
.../translators/ParDoMultiOutputTranslator.java | 8 +-
.../translators/functions/DoFnFunction.java | 6 +-
.../translators/utils/DoFnRunnerFactory.java | 8 +-
.../FlattenPCollectionsTranslatorTest.java | 6 +-
.../translators/ReadBoundedTranslatorTest.java | 4 +-
.../translators/ReadUnboundedTranslatorTest.java | 4 +-
runners/google-cloud-dataflow-java/build.gradle | 30 +-
.../dataflow/DataflowPipelineTranslator.java | 73 +-
.../beam/runners/dataflow/DataflowRunner.java | 24 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 15 +-
.../options/DataflowPipelineWorkerPoolOptions.java | 17 +-
.../beam/runners/dataflow/util/MonitoringUtil.java | 2 +-
.../util/SchemaCoderCloudObjectTranslator.java | 2 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 34 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 2 +-
.../dataflow/PrimitiveParDoSingleFactoryTest.java | 2 +-
.../runners/dataflow/util/CloudObjectsTest.java | 78 +-
.../runners/dataflow/util/MonitoringUtilTest.java | 6 +-
.../dataflow/worker/BatchDataflowWorker.java | 2 +-
.../dataflow/worker/BatchModeExecutionContext.java | 22 +-
.../dataflow/worker/CombineValuesFnFactory.java | 13 +-
.../dataflow/worker/DataflowOperationContext.java | 16 +-
.../worker/DataflowPortabilityPCollectionView.java | 6 +
.../runners/dataflow/worker/DoFnRunnerFactory.java | 3 +-
.../dataflow/worker/FnApiWindowMappingFn.java | 2 +-
.../worker/MetricsToCounterUpdateConverter.java | 1 +
.../beam/runners/dataflow/worker/PubsubReader.java | 4 +-
.../dataflow/worker/SimpleDoFnRunnerFactory.java | 6 +-
.../runners/dataflow/worker/SimpleParDoFn.java | 7 +-
.../worker/SplittableProcessFnFactory.java | 9 +-
.../dataflow/worker/StreamingDataflowWorker.java | 70 +-
.../dataflow/worker/UserParDoFnFactory.java | 2 +
.../dataflow/worker/WindowingWindmillReader.java | 62 +-
.../worker/counters/CounterUpdateAggregator.java | 38 +
.../worker/counters/CounterUpdateAggregators.java | 75 +
.../DistributionCounterUpdateAggregator.java | 65 +
.../counters/MeanCounterUpdateAggregator.java | 55 +
.../counters/SumCounterUpdateAggregator.java | 47 +
.../worker/fn/control/BeamFnMapTaskExecutor.java | 50 +-
...ntMonitoringInfoToCounterUpdateTransformer.java | 3 +-
...ecMonitoringInfoToCounterUpdateTransformer.java | 3 +-
...ntMonitoringInfoToCounterUpdateTransformer.java | 6 +-
.../control/RegisterAndProcessBundleOperation.java | 113 +-
...onMonitoringInfoToCounterUpdateTransformer.java | 3 +-
...erMonitoringInfoToCounterUpdateTransformer.java | 3 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 12 +-
.../logging/DataflowWorkerLoggingHandler.java | 6 +-
.../dataflow/worker/BatchDataflowWorkerTest.java | 4 +-
.../worker/BatchModeExecutionContextTest.java | 11 +-
.../dataflow/worker/DefaultParDoFnFactoryTest.java | 4 +-
.../dataflow/worker/DoFnInstanceManagersTest.java | 16 +-
.../dataflow/worker/FnApiWindowMappingFnTest.java | 18 +-
.../IntrinsicMapTaskExecutorFactoryTest.java | 4 +-
.../dataflow/worker/IsmSideInputReaderTest.java | 3 +-
.../runners/dataflow/worker/SimpleParDoFnTest.java | 24 +-
.../worker/StreamingDataflowWorkerTest.java | 9 +-
.../worker/StreamingModeExecutionContextTest.java | 5 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 3 +-
.../worker/StreamingStepMetricsContainerTest.java | 2 +-
.../dataflow/worker/UserParDoFnFactoryTest.java | 3 +-
.../worker/WindmillStateInternalsTest.java | 4 +-
.../dataflow/worker/WorkItemStatusClientTest.java | 9 +-
.../counters/CounterUpdateAggregatorsTest.java | 96 +
.../DistributionCounterUpdateAggregatorTest.java | 72 +
.../counters/MeanCounterUpdateAggregatorTest.java | 66 +
.../counters/SumCounterUpdateAggregatorTest.java | 62 +
.../RegisterAndProcessBundleOperationTest.java | 13 +-
.../worker/fn/control/TimerReceiverTest.java | 2 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 20 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 18 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 2 +-
.../util/common/worker/ReadOperationTest.java | 2 +-
.../common/worker/WorkProgressUpdaterTest.java | 12 +-
.../worker/windmill/build.gradle | 1 +
runners/java-fn-execution/build.gradle | 2 +-
.../artifact/AbstractArtifactRetrievalService.java | 199 +
.../artifact/AbstractArtifactStagingService.java | 227 ++
.../BeamFileSystemArtifactRetrievalService.java | 169 +-
.../BeamFileSystemArtifactStagingService.java | 233 +-
.../ClassLoaderArtifactRetrievalService.java | 58 +
.../JavaFilesystemArtifactStagingService.java | 93 +
.../control/DefaultExecutableStageContext.java | 99 +
.../control/DefaultJobBundleFactory.java | 42 +-
.../control/ExecutableStageContext.java | 38 +
.../control/ProcessBundleDescriptors.java | 18 +-
...renceCountingExecutableStageContextFactory.java | 243 ++
.../runners/fnexecution/control/RemoteBundle.java | 3 +-
.../fnexecution/control/SdkHarnessClient.java | 30 +-
.../SingleEnvironmentInstanceJobBundleFactory.java | 8 +-
.../runners/fnexecution/data/FnDataService.java | 8 +-
.../runners/fnexecution/data/GrpcDataService.java | 12 +-
.../fnexecution/environment/DockerCommand.java | 86 +-
.../environment/DockerContainerEnvironment.java | 20 +-
.../environment/DockerEnvironmentFactory.java | 83 +-
.../environment/ExternalEnvironmentFactory.java | 40 +-
.../environment/ProcessEnvironmentFactory.java | 43 +-
.../jobsubmission/InMemoryJobService.java | 59 +
.../fnexecution/jobsubmission/JobInvocation.java | 43 +-
.../fnexecution/jobsubmission/JobServerDriver.java | 13 -
.../jobsubmission/PortablePipelineJarCreator.java | 292 ++
.../jobsubmission/PortablePipelineJarUtils.java | 145 +
.../jobsubmission/PortablePipelineResult.java | 36 +
.../jobsubmission/PortablePipelineRunner.java | 3 +-
.../fnexecution/state/GrpcStateService.java | 5 +-
.../state/InMemoryBagUserStateFactory.java | 11 +
.../fnexecution/state/StateRequestHandler.java | 7 +
.../fnexecution/state/StateRequestHandlers.java | 79 +-
.../runners/fnexecution/ServerFactoryTest.java | 4 +-
.../artifact/ClassLoaderArtifactServiceTest.java | 406 +++
.../control/DefaultExecutableStageContextTest.java | 72 +
.../control/DefaultJobBundleFactoryTest.java | 2 +
...eCountingExecutableStageContextFactoryTest.java | 114 +
.../fnexecution/control/RemoteExecutionTest.java | 36 +-
.../fnexecution/control/SdkHarnessClientTest.java | 58 +-
.../fnexecution/data/GrpcDataServiceTest.java | 18 +-
.../environment/DockerEnvironmentFactoryTest.java | 191 +-
.../environment/ProcessEnvironmentFactoryTest.java | 5 +-
.../jobsubmission/InMemoryJobServiceTest.java | 25 +
.../jobsubmission/JobInvocationTest.java | 62 +-
.../PortablePipelineJarCreatorTest.java | 172 +
.../logging/GrpcLoggingServiceTest.java | 18 +-
.../fnexecution/state/GrpcStateServiceTest.java | 4 +-
runners/jet-experimental/build.gradle | 106 -
.../java/org/apache/beam/runners/jet/Utils.java | 292 --
.../runners/jet/processors/AbstractParDoP.java | 521 ---
.../apache/beam/runners/jet/processors/ParDoP.java | 191 -
.../runners/jet/processors/StatefulParDoP.java | 302 --
runners/jet/build.gradle | 106 +
.../org/apache/beam/runners/jet/DAGBuilder.java | 0
.../runners/jet/FailedRunningPipelineResults.java | 0
.../apache/beam/runners/jet/JetGraphVisitor.java | 0
.../beam/runners/jet/JetPipelineOptions.java | 0
.../apache/beam/runners/jet/JetPipelineResult.java | 0
.../org/apache/beam/runners/jet/JetRunner.java | 0
.../beam/runners/jet/JetRunnerRegistrar.java | 0
.../beam/runners/jet/JetTransformTranslator.java | 0
.../beam/runners/jet/JetTransformTranslators.java | 0
.../beam/runners/jet/JetTranslationContext.java | 0
.../java/org/apache/beam/runners/jet/Utils.java | 292 ++
.../beam/runners/jet/metrics/AbstractMetric.java | 0
.../beam/runners/jet/metrics/CounterImpl.java | 0
.../beam/runners/jet/metrics/DistributionImpl.java | 0
.../apache/beam/runners/jet/metrics/GaugeImpl.java | 0
.../beam/runners/jet/metrics/JetMetricResults.java | 0
.../runners/jet/metrics/JetMetricsContainer.java | 0
.../beam/runners/jet/metrics/package-info.java | 0
.../org/apache/beam/runners/jet/package-info.java | 0
.../runners/jet/processors/AbstractParDoP.java | 524 +++
.../beam/runners/jet/processors/AssignWindowP.java | 0
.../runners/jet/processors/BoundedSourceP.java | 0
.../beam/runners/jet/processors/FlattenP.java | 0
.../beam/runners/jet/processors/ImpulseP.java | 0
.../apache/beam/runners/jet/processors/ParDoP.java | 193 +
.../runners/jet/processors/StatefulParDoP.java | 304 ++
.../runners/jet/processors/UnboundedSourceP.java | 0
.../apache/beam/runners/jet/processors/ViewP.java | 0
.../beam/runners/jet/processors/WindowGroupP.java | 0
.../beam/runners/jet/processors/package-info.java | 0
.../beam/runners/jet/JetTestRunnerRegistrar.java | 0
.../org/apache/beam/runners/jet/TestJetRunner.java | 0
.../org/apache/beam/runners/jet/TestStreamP.java | 0
runners/local-java/build.gradle | 1 +
runners/reference/java/build.gradle | 2 +-
.../runners/reference/ExternalWorkerService.java | 11 +-
runners/samza/build.gradle | 3 +-
runners/samza/job-server/build.gradle | 2 +-
.../beam/runners/samza/SamzaPipelineOptions.java | 18 +-
.../samza/SamzaPipelineOptionsValidator.java | 41 +-
.../beam/runners/samza/SamzaPipelineRunner.java | 4 +-
.../runners/samza/SamzaPortablePipelineResult.java | 48 +
.../org/apache/beam/runners/samza/SamzaRunner.java | 23 +-
.../runners/samza/adapter/BoundedSourceSystem.java | 3 +-
.../apache/beam/runners/samza/runtime/DoFnOp.java | 168 +-
.../beam/runners/samza/runtime/GroupByKeyOp.java | 28 +-
.../org/apache/beam/runners/samza/runtime/Op.java | 2 +-
.../beam/runners/samza/runtime/OpAdapter.java | 2 +-
.../runners/samza/runtime/SamzaDoFnRunners.java | 93 +-
.../samza/runtime/SamzaTimerInternalsFactory.java | 316 +-
.../samza/translation/GroupByKeyTranslator.java | 24 +-
.../translation/ParDoBoundMultiTranslator.java | 33 +-
.../translation/PortableTranslationContext.java | 38 +-
.../samza/translation/SamzaPipelineTranslator.java | 6 +-
.../SamzaPortablePipelineTranslator.java | 7 +-
.../translation/SamzaPublishViewTranslator.java | 2 +-
.../samza/translation/TranslationContext.java | 25 +-
.../beam/runners/samza/util/HashIdGenerator.java | 66 +
.../runtime/SamzaTimerInternalsFactoryTest.java | 57 +-
.../runners/samza/util/TestHashIdGenerator.java | 85 +
runners/spark/build.gradle | 4 +-
runners/spark/job-server/build.gradle | 1 +
.../beam/runners/spark/SparkJobServerDriver.java | 8 +-
.../beam/runners/spark/SparkPipelineResult.java | 19 +
.../beam/runners/spark/SparkPipelineRunner.java | 6 +-
.../org/apache/beam/runners/spark/SparkRunner.java | 7 +-
.../SparkGroupAlsoByWindowViaWindowSet.java | 66 +-
.../spark/stateful/SparkTimerInternals.java | 2 +-
.../spark/translation/GroupCombineFunctions.java | 25 +-
.../spark/translation/MultiDoFnFunction.java | 9 +-
.../ReifyTimestampsAndWindowsFunction.java | 22 +-
.../SparkBatchPortablePipelineTranslator.java | 64 +-
.../SparkExecutableStageContextFactory.java | 69 +
.../translation/SparkExecutableStageFunction.java | 30 +-
.../spark/translation/TransformTranslator.java | 15 +-
.../spark/translation/TranslationUtils.java | 5 +-
.../streaming/StreamingTransformTranslator.java | 32 +-
.../beam/runners/spark/util/SparkCompat.java | 68 +-
.../apache/beam/runners/spark/util/TimerUtils.java | 44 +
.../org/apache/beam/runners/spark/CacheTest.java | 32 +-
.../SparkExecutableStageFunctionTest.java | 20 +-
.../translation/streaming/CreateStreamTest.java | 47 +
sdks/CONTAINERS.md | 191 -
sdks/go/container/build.gradle | 5 +-
sdks/go/gogradle.lock | 9 +-
sdks/go/pkg/beam/coder.go | 11 +-
sdks/go/pkg/beam/core/funcx/fn.go | 56 +-
sdks/go/pkg/beam/core/funcx/fn_test.go | 161 +-
sdks/go/pkg/beam/core/graph/coder/coder.go | 12 +
sdks/go/pkg/beam/core/graph/coder/double.go | 41 +
sdks/go/pkg/beam/core/graph/fn.go | 184 +-
sdks/go/pkg/beam/core/graph/fn_test.go | 295 +-
sdks/go/pkg/beam/core/runtime/exec/coder.go | 63 +
sdks/go/pkg/beam/core/runtime/exec/coder_test.go | 84 +
sdks/go/pkg/beam/core/runtime/exec/datasource.go | 59 +-
.../pkg/beam/core/runtime/exec/datasource_test.go | 246 +-
sdks/go/pkg/beam/core/runtime/exec/fn.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/plan.go | 6 +-
sdks/go/pkg/beam/core/runtime/exec/translate.go | 11 +-
sdks/go/pkg/beam/core/runtime/exec/unit_test.go | 58 +
sdks/go/pkg/beam/core/runtime/graphx/coder.go | 14 +
sdks/go/pkg/beam/core/runtime/graphx/coder_test.go | 8 +
sdks/go/pkg/beam/core/runtime/graphx/dataflow.go | 16 +-
sdks/go/pkg/beam/core/runtime/graphx/serialize.go | 7 +-
.../pkg/beam/core/runtime/graphx/serialize_test.go | 69 +
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 86 +-
.../pkg/beam/core/runtime/harness/datamgr_test.go | 106 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 98 +-
sdks/go/pkg/beam/core/runtime/harness/init/init.go | 43 +-
sdks/go/pkg/beam/core/runtime/harness/logging.go | 10 +-
.../pkg/beam/core/runtime/harness/logging_test.go | 51 +
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 35 +-
sdks/go/pkg/beam/core/typex/fulltype.go | 8 +
sdks/go/pkg/beam/core/util/dot/dot.go | 6 +-
sdks/go/pkg/beam/core/util/reflectx/functions.go | 4 +-
.../pkg/beam/core/util/reflectx/functions_test.go | 43 +
sdks/go/pkg/beam/core/util/reflectx/structs.go | 2 +-
sdks/go/pkg/beam/core/util/symtab/symtab.go | 47 +-
sdks/go/pkg/beam/core/util/symtab/symtab_test.go | 113 +
sdks/go/pkg/beam/create_test.go | 4 +
sdks/go/pkg/beam/doc_test.go | 4 +-
sdks/go/pkg/beam/log/log.go | 36 +-
sdks/go/pkg/beam/model/PROTOBUF.md | 7 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 1093 ++++--
.../model/jobmanagement_v1/beam_artifact_api.pb.go | 125 +-
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 507 ++-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 1440 +++-----
sdks/go/pkg/beam/options/jobopts/options.go | 2 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 5 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 3 +
sdks/go/pkg/beam/runners/direct/buffer.go | 15 +-
sdks/go/test/build.gradle | 4 +-
sdks/go/test/run_integration_tests.sh | 2 +-
sdks/java/build-tools/build.gradle | 2 +-
.../src/main/resources/beam/checkstyle.xml | 24 +
.../src/main/resources/beam/suppressions.xml | 5 +
sdks/java/container/build.gradle | 5 +-
sdks/java/core/build.gradle | 24 +-
.../java/org/apache/beam/sdk/PipelineResult.java | 7 +-
.../java/org/apache/beam/sdk/coders/AvroCoder.java | 8 +-
.../org/apache/beam/sdk/coders/BooleanCoder.java | 8 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 230 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 3 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 62 +-
.../main/java/org/apache/beam/sdk/io/AvroSink.java | 2 +-
.../java/org/apache/beam/sdk/io/AvroUtils.java | 40 -
.../java/org/apache/beam/sdk/io/FileSystem.java | 2 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 19 +-
.../beam/sdk/metrics/MetricQueryResults.java | 2 +-
.../beam/sdk/metrics/MetricsEnvironment.java | 2 +-
.../beam/sdk/options/PortablePipelineOptions.java | 19 +
.../beam/sdk/options/RemoteEnvironmentOptions.java | 44 +
.../java/org/apache/beam/sdk/schemas/Schema.java | 5 +-
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 168 +-
.../sdk/schemas/transforms/SchemaAggregateFn.java | 5 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 133 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 182 +-
.../apache/beam/sdk/testing/CoderProperties.java | 11 +
.../org/apache/beam/sdk/testing/TestStream.java | 4 +
.../testing/UsesSideInputsWithDifferentCoders.java | 21 +
.../beam/sdk/testing/UsesStrictTimerOrdering.java | 24 +
.../beam/sdk/transforms/ApproximateUnique.java | 14 +
.../org/apache/beam/sdk/transforms/Combine.java | 4 +
.../java/org/apache/beam/sdk/transforms/DoFn.java | 8 +
.../org/apache/beam/sdk/transforms/DoFnTester.java | 5 +
.../org/apache/beam/sdk/transforms/JsonToRow.java | 6 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 124 +-
.../org/apache/beam/sdk/transforms/ToJson.java | 76 +
.../java/org/apache/beam/sdk/transforms/View.java | 28 +-
.../org/apache/beam/sdk/transforms/ViewFn.java | 4 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 12 +
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 11 +
.../beam/sdk/transforms/reflect/DoFnSignature.java | 48 +
.../sdk/transforms/reflect/DoFnSignatures.java | 37 +-
.../java/org/apache/beam/sdk/util/DoFnInfo.java | 24 +-
.../sdk/util/DoFnWithExecutionInformation.java | 12 +-
.../org/apache/beam/sdk/util/JsonToRowUtils.java | 52 -
.../java/org/apache/beam/sdk/util/RowJson.java | 359 ++
.../apache/beam/sdk/util/RowJsonDeserializer.java | 271 --
.../org/apache/beam/sdk/util/RowJsonUtils.java | 74 +
.../apache/beam/sdk/util/RowJsonValidation.java | 2 +-
.../beam/sdk/util/RowJsonValueExtractors.java | 2 +-
.../org/apache/beam/sdk/values/PCollection.java | 2 +-
.../apache/beam/sdk/values/PCollectionView.java | 1 -
.../apache/beam/sdk/values/PCollectionViews.java | 99 +-
.../apache/beam/sdk/values/WindowingStrategy.java | 12 +-
.../avro/org/apache/beam/sdk/schemas/test.avsc | 5 +-
.../org/apache/beam/sdk/coders/AvroCoderTest.java | 42 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 31 +
.../sdk/options/RemoteEnvironmentOptionsTest.java | 39 +
.../apache/beam/sdk/schemas/AvroSchemaTest.java | 76 +-
.../beam/sdk/schemas/utils/AvroUtilsTest.java | 164 +-
.../beam/sdk/transforms/FlatMapElementsTest.java | 2 +-
.../beam/sdk/transforms/MapElementsTest.java | 5 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 645 +++-
.../org/apache/beam/sdk/transforms/ToJsonTest.java | 95 +
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 9 +-
.../beam/sdk/util/RowJsonDeserializerTest.java | 496 ---
.../java/org/apache/beam/sdk/util/RowJsonTest.java | 520 +++
sdks/java/extensions/euphoria/build.gradle | 3 +-
.../google-cloud-platform-core/build.gradle | 2 +-
.../sdk/extensions/gcp/options/GcpOptions.java | 1 +
.../gcp/util/RetryHttpRequestInitializer.java | 15 +-
sdks/java/extensions/jackson/build.gradle | 1 +
.../beam/sdk/extensions/jackson/AsJsons.java | 193 +-
.../beam/sdk/extensions/jackson/ParseJsons.java | 182 +-
.../extensions/jackson/JacksonTransformsTest.java | 268 +-
sdks/java/extensions/join-library/build.gradle | 2 +-
sdks/java/extensions/kryo/build.gradle | 1 +
sdks/java/extensions/protobuf/build.gradle | 2 +-
.../sdk/extensions/protobuf/DynamicProtoCoder.java | 205 ++
.../beam/sdk/extensions/protobuf/ProtoCoder.java | 63 +-
.../beam/sdk/extensions/protobuf/ProtoDomain.java | 248 ++
.../extensions/protobuf/DynamicProtoCoderTest.java | 92 +
.../extensions/protobuf/IsDynamicMessageEqual.java | 69 +
.../sdk/extensions/protobuf/ProtoCoderTest.java | 7 +
.../sdk/extensions/protobuf/ProtoDomainTest.java | 55 +
sdks/java/extensions/sketching/build.gradle | 2 +-
.../extensions/sketching/ApproximateDistinct.java | 5 +
sdks/java/extensions/sorter/build.gradle | 2 +-
sdks/java/extensions/sql/build.gradle | 106 +-
sdks/java/extensions/sql/datacatalog/build.gradle | 42 +-
.../sql/example/BeamSqlDataCatalogExample.java | 7 +-
.../provider/datacatalog/BigQueryTableFactory.java | 74 +
.../meta/provider/datacatalog/BigQueryUtils.java | 58 -
.../provider/datacatalog/ChainedTableFactory.java | 50 +
.../datacatalog/DataCatalogClientAdapter.java | 94 -
.../datacatalog/DataCatalogPipelineOptions.java | 8 +
.../datacatalog/DataCatalogTableProvider.java | 116 +-
.../meta/provider/datacatalog/GcsTableFactory.java | 62 +
.../provider/datacatalog/PubsubTableFactory.java | 63 +
.../sql/meta/provider/datacatalog/PubsubUtils.java | 53 -
.../sql/meta/provider/datacatalog/SchemaUtils.java | 4 +-
.../meta/provider/datacatalog/TableFactory.java | 37 +
.../sql/meta/provider/datacatalog/TableUtils.java | 59 -
.../meta/provider/datacatalog/ZetaSqlIdUtils.java | 79 +
.../datacatalog/DataCatalogBigQueryIT.java | 150 +
.../provider/datacatalog/DataCatalogGCSIT.java | 82 +
.../provider/datacatalog/ZetaSqlIdUtilsTest.java | 64 +
sdks/java/extensions/sql/hcatalog/build.gradle | 4 +-
.../meta/provider/hcatalog/DatabaseProvider.java | 2 +-
.../sql/meta/provider/hcatalog/HCatalogTable.java | 4 +-
.../provider/hcatalog/HCatalogTableProvider.java | 2 +-
sdks/java/extensions/sql/jdbc/build.gradle | 3 +-
sdks/java/extensions/sql/shell/build.gradle | 4 +-
.../extensions/sql/src/main/codegen/config.fmpp | 14 +-
.../beam/sdk/extensions/sql/BeamSqlTable.java | 47 -
.../beam/sdk/extensions/sql/SqlTransform.java | 5 +-
.../extensions/sql/TableNameExtractionUtils.java | 98 +
.../sdk/extensions/sql/example/BeamSqlExample.java | 22 +-
.../extensions/sql/example/BeamSqlPojoExample.java | 17 +-
.../sdk/extensions/sql/impl/BeamCalciteSchema.java | 17 +-
.../sql/impl/BeamCalciteSchemaFactory.java | 20 +-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 42 +-
.../beam/sdk/extensions/sql/impl/BeamSqlEnv.java | 16 +-
.../sql/impl/BeamSqlPipelineOptionsRegistrar.java | 2 +-
.../extensions/sql/impl/BeamTableStatistics.java | 14 +-
.../sql/impl/CalciteConnectionWrapper.java | 16 +-
.../extensions/sql/impl/CalciteFactoryWrapper.java | 24 +-
.../extensions/sql/impl/CalciteQueryPlanner.java | 79 +-
.../sdk/extensions/sql/impl/JdbcConnection.java | 8 +-
.../beam/sdk/extensions/sql/impl/JdbcDriver.java | 30 +-
.../beam/sdk/extensions/sql/impl/JdbcFactory.java | 36 +-
.../beam/sdk/extensions/sql/impl/QueryPlanner.java | 2 +-
.../extensions/sql/impl/ScalarFunctionImpl.java | 61 +-
.../beam/sdk/extensions/sql/impl/TableName.java | 100 +
.../extensions/sql/impl/TableResolutionUtils.java | 214 ++
.../beam/sdk/extensions/sql/impl/UdafImpl.java | 12 +-
.../beam/sdk/extensions/sql/impl/UdfImpl.java | 12 +-
.../sql/impl/UdfImplReflectiveFunctionBase.java | 19 +-
.../sql/impl/parser/SqlCheckConstraint.java | 18 +-
.../sql/impl/parser/SqlColumnDeclaration.java | 20 +-
.../sql/impl/parser/SqlCreateExternalTable.java | 34 +-
.../extensions/sql/impl/parser/SqlDdlNodes.java | 20 +-
.../extensions/sql/impl/parser/SqlDropObject.java | 24 +-
.../extensions/sql/impl/parser/SqlDropTable.java | 10 +-
.../sql/impl/parser/SqlSetOptionBeam.java | 20 +-
.../extensions/sql/impl/planner/BeamCostModel.java | 11 +-
.../sql/impl/planner/BeamJavaTypeFactory.java | 12 +-
.../sql/impl/planner/BeamRelDataTypeSystem.java | 4 +-
.../extensions/sql/impl/planner/BeamRuleSets.java | 66 +-
.../sql/impl/planner/NodeStatsMetadata.java | 12 +-
.../sql/impl/planner/RelMdNodeStats.java | 12 +-
.../sql/impl/rel/BeamAggregationRel.java | 34 +-
.../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 70 +-
.../extensions/sql/impl/rel/BeamCoGBKJoinRel.java | 213 ++
.../sql/impl/rel/BeamEnumerableConverter.java | 40 +-
.../sdk/extensions/sql/impl/rel/BeamIOSinkRel.java | 24 +-
.../extensions/sql/impl/rel/BeamIOSourceRel.java | 57 +-
.../extensions/sql/impl/rel/BeamIntersectRel.java | 14 +-
.../sdk/extensions/sql/impl/rel/BeamJoinRel.java | 476 +--
.../sql/impl/rel/BeamLogicalConvention.java | 12 +-
.../sdk/extensions/sql/impl/rel/BeamMinusRel.java | 14 +-
.../sdk/extensions/sql/impl/rel/BeamRelNode.java | 12 +-
.../sql/impl/rel/BeamSetOperatorRelBase.java | 2 +-
.../sql/impl/rel/BeamSideInputJoinRel.java | 181 +
.../sql/impl/rel/BeamSideInputLookupJoinRel.java | 149 +
.../sdk/extensions/sql/impl/rel/BeamSortRel.java | 36 +-
.../extensions/sql/impl/rel/BeamSqlRelUtils.java | 6 +-
.../extensions/sql/impl/rel/BeamUncollectRel.java | 14 +-
.../sdk/extensions/sql/impl/rel/BeamUnionRel.java | 14 +-
.../sdk/extensions/sql/impl/rel/BeamUnnestRel.java | 24 +-
.../sdk/extensions/sql/impl/rel/BeamValuesRel.java | 20 +-
.../sdk/extensions/sql/impl/rel/package-info.java | 5 +-
.../sql/impl/rule/BeamAggregationRule.java | 33 +-
.../sql/impl/rule/BeamBasicAggregationRule.java | 87 +-
.../sdk/extensions/sql/impl/rule/BeamCalcRule.java | 14 +-
.../sql/impl/rule/BeamCoGBKJoinRule.java | 82 +
.../sql/impl/rule/BeamEnumerableConverterRule.java | 8 +-
.../sql/impl/rule/BeamIOPushDownRule.java | 248 ++
.../extensions/sql/impl/rule/BeamIOSinkRule.java | 6 +-
.../sql/impl/rule/BeamIntersectRule.java | 10 +-
.../sql/impl/rule/BeamJoinAssociateRule.java | 15 +-
.../sql/impl/rule/BeamJoinPushThroughJoinRule.java | 19 +-
.../sdk/extensions/sql/impl/rule/BeamJoinRule.java | 51 -
.../extensions/sql/impl/rule/BeamMinusRule.java | 10 +-
.../sql/impl/rule/BeamSideInputJoinRule.java | 82 +
.../sql/impl/rule/BeamSideInputLookupJoinRule.java | 69 +
.../sdk/extensions/sql/impl/rule/BeamSortRule.java | 10 +-
.../sql/impl/rule/BeamUncollectRule.java | 8 +-
.../extensions/sql/impl/rule/BeamUnionRule.java | 13 +-
.../extensions/sql/impl/rule/BeamUnnestRule.java | 26 +-
.../extensions/sql/impl/rule/BeamValuesRule.java | 10 +-
.../sql/impl/rule/JoinRelOptRuleCall.java | 14 +-
.../sdk/extensions/sql/impl/rule/package-info.java | 4 +-
.../extensions/sql/impl/schema/BaseBeamTable.java | 36 -
.../sql/impl/schema/BeamPCollectionTable.java | 3 +-
.../extensions/sql/impl/schema/BeamTableUtils.java | 2 +-
.../impl/transform/BeamBuiltinAggregations.java | 2 +-
.../sql/impl/transform/BeamJoinTransforms.java | 10 +-
.../impl/transform/BeamSetOperatorsTransforms.java | 2 +-
.../transform/agg/AggregationCombineFnAdapter.java | 4 +-
.../sql/impl/transform/agg/CovarianceFn.java | 4 +-
.../sql/impl/transform/agg/VarianceFn.java | 2 +-
.../sql/impl/udf/BuiltinStringFunctions.java | 2 +-
.../sql/impl/utils/BigDecimalConverter.java | 2 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 16 +-
.../sql/impl/utils/SerializableRexFieldAccess.java | 4 +-
.../sql/impl/utils/SerializableRexInputRef.java | 2 +-
.../sql/impl/utils/SerializableRexNode.java | 6 +-
.../sdk/extensions/sql/meta/BaseBeamTable.java | 51 +
.../beam/sdk/extensions/sql/meta/BeamSqlTable.java | 58 +
.../extensions/sql/meta/BeamSqlTableFilter.java | 34 +
.../extensions/sql/meta/CustomTableResolver.java | 45 +
.../extensions/sql/meta/DefaultTableFilter.java | 45 +
.../extensions/sql/meta/SchemaBaseBeamTable.java | 35 +
.../sql/meta/provider/FullNameTableProvider.java | 166 +
.../sql/meta/provider/ReadOnlyTableProvider.java | 4 +-
.../sql/meta/provider/TableProvider.java | 2 +-
.../sql/meta/provider/avro/AvroTable.java | 82 +
.../sql/meta/provider/avro/AvroTableProvider.java | 52 +
.../provider/avro/GenericRecordWriteConverter.java | 67 +
.../sql/meta/provider/avro/package-info.java | 20 +
.../sql/meta/provider/bigquery/BigQueryTable.java | 99 +-
.../provider/bigquery/BigQueryTableProvider.java | 4 +-
.../sql/meta/provider/kafka/BeamKafkaTable.java | 6 +-
.../meta/provider/kafka/KafkaTableProvider.java | 2 +-
.../sql/meta/provider/parquet/ParquetTable.java | 7 +-
.../provider/parquet/ParquetTableProvider.java | 2 +-
.../meta/provider/pubsub/PubsubIOJsonTable.java | 5 +-
.../provider/pubsub/PubsubJsonTableProvider.java | 2 +-
.../meta/provider/pubsub/PubsubMessageToRow.java | 10 +-
.../provider/seqgen/GenerateSequenceTable.java | 4 +-
.../seqgen/GenerateSequenceTableProvider.java | 2 +-
.../sql/meta/provider/test/TestTable.java | 4 +-
.../sql/meta/provider/test/TestTableProvider.java | 73 +-
.../sql/meta/provider/test/TestTableUtils.java | 16 +-
.../sql/meta/provider/test/TestUnboundedTable.java | 7 +-
.../sql/meta/provider/text/TextTable.java | 9 +-
.../sql/meta/provider/text/TextTableProvider.java | 10 +-
.../sql/meta/store/InMemoryMetaStore.java | 4 +-
...ckaged.org.codehaus.commons.compiler.properties | 18 -
...1_20_0.org.codehaus.commons.compiler.properties | 18 +
.../sdk/extensions/sql/BeamComplexTypeTest.java | 4 +-
.../beam/sdk/extensions/sql/BeamSqlCastTest.java | 11 +-
.../sql/BeamSqlDslAggregationCovarianceTest.java | 7 +-
.../sql/BeamSqlDslAggregationNullableTest.java | 4 +-
.../extensions/sql/BeamSqlDslAggregationTest.java | 45 +-
.../sql/BeamSqlDslAggregationVarianceTest.java | 7 +-
.../sdk/extensions/sql/BeamSqlDslArrayTest.java | 41 +-
.../beam/sdk/extensions/sql/BeamSqlDslBase.java | 51 +-
.../sdk/extensions/sql/BeamSqlDslJoinTest.java | 4 +-
.../extensions/sql/BeamSqlDslNestedRowsTest.java | 16 +-
.../sql/BeamSqlDslSqlStdOperatorsTest.java | 18 +-
.../sdk/extensions/sql/BeamSqlDslUdfUdafTest.java | 15 +-
.../sdk/extensions/sql/BeamSqlExplainTest.java | 8 +-
.../beam/sdk/extensions/sql/BeamSqlMapTest.java | 8 +-
.../extensions/sql/BeamSqlMultipleSchemasTest.java | 2 +-
.../CalciteCannotParseSimpleIdentifiersTest.java | 77 +
.../sql/CalciteParsesSimpleIdentifiersTest.java | 133 +
.../sdk/extensions/sql/PubsubToBigqueryIT.java | 4 +-
.../apache/beam/sdk/extensions/sql/TestUtils.java | 15 +-
.../sdk/extensions/sql/impl/JdbcDriverTest.java | 8 +-
.../sql/impl/parser/BeamDDLNestedTypesTest.java | 2 +-
.../extensions/sql/impl/parser/BeamDDLTest.java | 15 +
.../extensions/sql/impl/planner/NodeStatsTest.java | 10 +-
.../sdk/extensions/sql/impl/rel/BaseRelTest.java | 2 +-
.../sql/impl/rel/BeamAggregationRelTest.java | 2 +-
.../extensions/sql/impl/rel/BeamCalcRelTest.java | 2 +-
.../rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java | 378 ++
.../BeamCoGBKJoinRelUnboundedVsUnboundedTest.java | 254 ++
.../sql/impl/rel/BeamEnumerableConverterTest.java | 28 +-
.../sql/impl/rel/BeamIOSourceRelTest.java | 4 +-
.../sql/impl/rel/BeamIntersectRelTest.java | 2 +-
.../impl/rel/BeamJoinRelBoundedVsBoundedTest.java | 378 --
.../rel/BeamJoinRelUnboundedVsBoundedTest.java | 365 --
.../rel/BeamJoinRelUnboundedVsUnboundedTest.java | 254 --
.../extensions/sql/impl/rel/BeamMinusRelTest.java | 2 +-
.../sql/impl/rel/BeamSideInputJoinRelTest.java | 280 ++
.../impl/rel/BeamSideInputLookupJoinRelTest.java | 295 ++
.../extensions/sql/impl/rel/BeamSortRelTest.java | 2 +-
.../sql/impl/rel/BeamUncollectRelTest.java | 2 +-
.../extensions/sql/impl/rel/BeamUnionRelTest.java | 2 +-
.../extensions/sql/impl/rel/BeamValuesRelTest.java | 2 +-
.../sql/impl/rule/IOPushDownRuleTest.java | 177 +
.../sql/impl/rule/JoinReorderingTest.java | 77 +-
.../sql/impl/schema/BeamSqlRowCoderTest.java | 13 +-
.../sql/impl/utils/CalciteUtilsTest.java | 10 +-
...BeamSqlBuiltinFunctionsIntegrationTestBase.java | 14 +-
.../BeamSqlDateFunctionsIntegrationTest.java | 2 +-
.../sql/meta/CustomTableResolverTest.java | 481 +++
.../meta/provider/avro/AvroTableProviderTest.java | 77 +
.../provider/bigquery/BigQueryReadWriteIT.java | 227 +-
.../meta/provider/bigquery/BigQueryRowCountIT.java | 4 +-
.../bigquery/BigQueryTableProviderTest.java | 81 +-
.../bigquery/BigQueryTestTableProvider.java | 4 +-
.../meta/provider/kafka/BeamKafkaCSVTableTest.java | 10 +-
.../sql/meta/provider/kafka/KafkaCSVTableIT.java | 4 +-
.../provider/kafka/KafkaTableProviderTest.java | 4 +-
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 15 +-
.../pubsub/PubsubJsonTableProviderTest.java | 2 +-
.../provider/pubsub/PubsubMessageToRowTest.java | 6 +-
.../meta/provider/test/TestTableProviderTest.java | 128 +
.../test/TestTableProviderWithProjectPushDown.java | 206 ++
.../meta/provider/text/TextTableProviderTest.java | 2 +-
.../sql/meta/store/InMemoryMetaStoreTest.java | 2 +-
.../beam/sdk/extensions/sql/utils/RowAsserts.java | 2 +-
sdks/java/extensions/sql/zetasql/build.gradle | 51 +
.../extensions/sql/zetasql/BeamBuiltinMethods.java | 71 +
.../extensions/sql/zetasql/BeamCodegenUtils.java | 37 +
.../sdk/extensions/sql/zetasql/DateFunctions.java | 38 +
.../sdk/extensions/sql/zetasql/DateTimeUtils.java | 279 ++
.../sdk/extensions/sql/zetasql/QueryTrait.java | 76 +
.../sdk/extensions/sql/zetasql/SqlAnalyzer.java | 284 ++
.../zetasql/SqlCaseWithValueOperatorRewriter.java | 77 +
.../sql/zetasql/SqlCoalesceOperatorRewriter.java | 67 +
.../SqlExtractTimestampOperatorRewriter.java | 47 +
.../sql/zetasql/SqlIfNullOperatorRewriter.java | 51 +
.../sql/zetasql/SqlNullIfOperatorRewriter.java | 53 +
.../sql/zetasql/SqlOperatorRewriter.java | 35 +
.../sdk/extensions/sql/zetasql/SqlOperators.java | 197 +
.../sql/zetasql/SqlStdOperatorMappingTable.java | 359 ++
.../extensions/sql/zetasql/StringFunctions.java | 182 +
.../extensions/sql/zetasql/TableResolution.java | 91 +
.../sql/zetasql/TableResolutionContext.java | 73 +
.../sdk/extensions/sql/zetasql/TableResolver.java | 36 +
.../extensions/sql/zetasql/TableResolverImpl.java | 61 +
.../beam/sdk/extensions/sql/zetasql/TestInput.java | 255 ++
.../extensions/sql/zetasql/TimestampFunctions.java | 48 +
.../beam/sdk/extensions/sql/zetasql/TypeUtils.java | 166 +
.../sql/zetasql/ZetaSQLCastFunctionImpl.java | 116 +
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 203 ++
.../sql/zetasql/ZetaSQLQueryPlanner.java | 151 +
.../sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java | 84 +
.../sdk/extensions/sql/zetasql/package-info.java | 20 +
.../translation/AggregateScanConverter.java | 231 ++
.../translation/ArrayScanToJoinConverter.java | 101 +
.../translation/ArrayScanToUncollectConverter.java | 62 +
.../sql/zetasql/translation/ConversionContext.java | 65 +
.../zetasql/translation/ExpressionConverter.java | 1031 ++++++
.../zetasql/translation/FilterScanConverter.java | 52 +
.../sql/zetasql/translation/JoinScanConverter.java | 101 +
.../translation/JoinScanWithRefConverter.java | 95 +
.../LimitOffsetScanToLimitConverter.java | 63 +
.../LimitOffsetScanToOrderByLimitConverter.java | 101 +
.../OrderByScanUnsupportedConverter.java | 39 +
.../zetasql/translation/ProjectScanConverter.java | 49 +
.../translation/QueryStatementConverter.java | 114 +
.../sql/zetasql/translation/RelConverter.java | 74 +
.../translation/SetOperationScanConverter.java | 114 +
.../translation/SingleRowScanConverter.java | 41 +
.../zetasql/translation/TableScanConverter.java | 131 +
.../zetasql/translation/WithRefScanConverter.java | 53 +
.../sql/zetasql/translation/WithScanConverter.java | 55 +
.../sql/zetasql/translation/package-info.java | 20 +
.../sql/zetasql/JoinCompoundIdentifiersTest.java | 342 ++
.../sql/zetasql/ZetaSQLDialectSpecTest.java | 3787 ++++++++++++++++++++
sdks/java/extensions/zetasketch/build.gradle | 77 +
.../beam/sdk/extensions/zetasketch/HllCount.java | 411 +++
.../sdk/extensions/zetasketch/HllCountInitFn.java | 166 +
.../zetasketch/HllCountMergePartialFn.java | 115 +
.../zetasketch/HyperLogLogPlusPlusCoder.java | 64 +
.../sdk/extensions/zetasketch/package-info.java | 23 +
.../BigQueryHllSketchCompatibilityIT.java | 197 +
.../sdk/extensions/zetasketch/HllCountTest.java | 487 +++
sdks/java/fn-execution/build.gradle | 2 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 24 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 11 +-
.../sdk/fn/data/BeamFnDataInboundObserver.java | 17 +-
.../apache/beam/sdk/fn/data/LogicalEndpoint.java | 2 +-
.../BeamFnDataBufferingOutboundObserverTest.java | 8 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 8 +-
sdks/java/harness/build.gradle | 2 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 13 +
.../harness/SplittableProcessElementsRunner.java | 4 +-
.../fn/harness/control/ProcessBundleHandler.java | 6 +-
.../beam/fn/harness/data/BeamFnDataClient.java | 9 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 13 +-
.../fn/harness/data/QueueingBeamFnDataClient.java | 41 +-
.../apache/beam/fn/harness/state/BagUserState.java | 4 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 10 +-
.../beam/fn/harness/state/MultimapSideInput.java | 4 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 63 +-
.../harness/control/ProcessBundleHandlerTest.java | 28 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 20 +-
.../data/BeamFnDataInboundObserverTest.java | 8 +-
.../data/ElementCountFnDataReceiverTest.java | 2 +-
.../data/PCollectionConsumerRegistryTest.java | 4 +-
.../data/PTransformFunctionRegistryTest.java | 4 +-
.../harness/data/QueueingBeamFnDataClientTest.java | 20 +-
.../beam/fn/harness/state/BagUserStateTest.java | 2 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 14 +-
.../fn/harness/state/MultimapSideInputTest.java | 2 +-
sdks/java/io/amazon-web-services/build.gradle | 2 +-
.../apache/beam/sdk/io/aws/options/AwsModule.java | 9 +-
.../beam/sdk/io/aws/s3/S3FileSystemTest.java | 35 +-
.../sdk/io/aws/s3/S3WritableByteChannelTest.java | 9 +-
sdks/java/io/amazon-web-services2/build.gradle | 3 +-
.../apache/beam/sdk/io/aws2/options/AwsModule.java | 8 +-
.../sdk/io/aws2/sns/BasicSnsClientProvider.java | 55 +
.../beam/sdk/io/aws2/sns/PublishResponseCoder.java | 48 +
.../beam/sdk/io/aws2/sns/SnsClientProvider.java | 31 +
.../sdk/io/aws2/sns/SnsCoderProviderRegistrar.java | 38 +
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 371 ++
.../apache/beam/sdk/io/aws2/sns/package-info.java | 19 +
.../beam/sdk/io/aws2/sns/SnsClientMockErrors.java | 56 +
.../beam/sdk/io/aws2/sns/SnsClientMockSuccess.java | 67 +
.../org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java | 100 +
.../org.mockito.plugins.MockMaker | 1 +
sdks/java/io/amqp/build.gradle | 2 +-
sdks/java/io/bigquery-io-perf-tests/build.gradle | 3 +-
.../beam/sdk/bigqueryioperftests/BigQueryIOIT.java | 221 ++
.../BigQueryIOReadPerformanceIT.java | 203 --
sdks/java/io/cassandra/build.gradle | 2 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 2 +
sdks/java/io/clickhouse/build.gradle | 1 +
sdks/java/io/common/build.gradle | 2 +-
.../elasticsearch-tests-2/build.gradle | 1 +
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 10 +-
.../elasticsearch-tests-5/build.gradle | 1 +
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 13 +-
.../elasticsearch-tests-6/build.gradle | 1 +
.../sdk/io/elasticsearch/ElasticsearchIOTest.java | 13 +-
.../elasticsearch-tests-common/build.gradle | 1 +
.../elasticsearch/ElasticsearchIOTestCommon.java | 25 +-
sdks/java/io/elasticsearch/build.gradle | 2 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 21 +-
sdks/java/io/file-based-io-tests/build.gradle | 3 +-
.../java/org/apache/beam/sdk/io/avro/AvroIOIT.java | 17 +-
.../beam/sdk/io/common/FileBasedIOITHelper.java | 15 -
.../io/common/FileBasedIOTestPipelineOptions.java | 12 +
.../apache/beam/sdk/io/parquet/ParquetIOIT.java | 20 +-
.../java/org/apache/beam/sdk/io/text/TextIOIT.java | 16 +-
.../apache/beam/sdk/io/tfrecord/TFRecordIOIT.java | 20 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOIT.java | 27 +-
sdks/java/io/google-cloud-platform/build.gradle | 60 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 165 +-
.../gcp/bigquery/BigQueryStorageQuerySource.java | 2 +-
.../io/gcp/bigquery/BigQueryStorageSourceBase.java | 23 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 71 +-
.../gcp/bigquery/BigQueryStorageTableSource.java | 18 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 68 +-
.../gcp/bigquery/DynamicDestinationsHelpers.java | 3 +-
.../beam/sdk/io/gcp/bigquery/TestBigQuery.java | 6 +-
.../gcp/pubsub/PubsubCoderProviderRegistrar.java | 7 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 242 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessage.java | 18 +-
...bsubMessageWithAttributesAndMessageIdCoder.java | 64 +
.../pubsub/PubsubMessageWithMessageIdCoder.java | 54 +
.../beam/sdk/io/gcp/pubsub/PubsubTestClient.java | 6 +-
.../sdk/io/gcp/pubsub/PubsubUnboundedSource.java | 78 +-
.../beam/sdk/io/gcp/pubsub/TestPubsubSignal.java | 2 +-
.../beam/sdk/io/gcp/testing/BigqueryClient.java | 21 +-
.../beam/sdk/io/gcp/testing/BigqueryMatcher.java | 22 +-
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 44 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 325 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 140 +-
.../beam/sdk/io/gcp/datastore/DatastoreV1Test.java | 3 +-
.../sdk/io/gcp/pubsub/PubsubIOExternalTest.java | 243 ++
.../beam/sdk/io/gcp/pubsub/PubsubIOTest.java | 55 +-
.../pubsub/PubsubMessagePayloadOnlyCoderTest.java | 56 +
...MessageWithAttributesAndMessageIdCoderTest.java | 62 +
.../PubsubMessageWithAttributesCoderTest.java | 60 +
.../PubsubMessageWithMessageIdCoderTest.java | 57 +
.../beam/sdk/io/gcp/pubsub/PubsubReadIT.java | 47 +
.../sdk/io/gcp/spanner/ReadSpannerSchemaTest.java | 4 +-
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 17 +-
sdks/java/io/hadoop-common/build.gradle | 2 +-
sdks/java/io/hadoop-file-system/build.gradle | 2 +-
sdks/java/io/hadoop-format/build.gradle | 2 +-
.../io/hadoop/format/HadoopFormatIOReadTest.java | 8 +-
sdks/java/io/hbase/build.gradle | 2 +-
sdks/java/io/hcatalog/build.gradle | 3 +-
.../apache/beam/sdk/io/hcatalog/HCatalogIO.java | 6 +-
sdks/java/io/jdbc/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 68 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 52 +-
sdks/java/io/jms/build.gradle | 2 +-
sdks/java/io/kafka/build.gradle | 3 +-
.../java/org/apache/beam/sdk/io/kafka/KafkaIO.java | 68 +-
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 49 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 21 +-
sdks/java/io/kinesis/build.gradle | 4 +-
.../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 19 +-
.../apache/beam/sdk/io/kinesis/KinesisReader.java | 13 +-
.../beam/sdk/io/kinesis/KinesisProducerMock.java | 2 +-
.../beam/sdk/io/kinesis/KinesisReaderTest.java | 21 +-
.../beam/sdk/io/kinesis/ShardReadersPoolTest.java | 4 +-
.../sdk/io/kinesis/ShardRecordsIteratorTest.java | 4 +-
sdks/java/io/kudu/build.gradle | 2 +-
sdks/java/io/mongodb/build.gradle | 2 +-
sdks/java/io/mqtt/build.gradle | 2 +-
sdks/java/io/parquet/build.gradle | 2 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 11 +-
sdks/java/io/rabbitmq/build.gradle | 2 +-
.../apache/beam/sdk/io/rabbitmq/RabbitMqIO.java | 18 +-
.../beam/sdk/io/rabbitmq/RabbitMqMessage.java | 66 +
sdks/java/io/redis/build.gradle | 2 +-
sdks/java/io/solr/build.gradle | 2 +-
sdks/java/io/synthetic/build.gradle | 3 +-
sdks/java/io/tika/build.gradle | 2 +-
sdks/java/io/xml/build.gradle | 2 +-
sdks/java/javadoc/build.gradle | 2 +-
sdks/java/maven-archetypes/examples/build.gradle | 5 +-
.../src/main/resources/archetype-resources/pom.xml | 13 +-
sdks/java/maven-archetypes/starter/build.gradle | 2 +-
sdks/java/testing/expansion-service/build.gradle | 2 +-
sdks/java/testing/load-tests/build.gradle | 1 +
sdks/java/testing/nexmark/build.gradle | 6 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 1 +
.../apache/beam/sdk/nexmark/model/sql/RowSize.java | 83 -
.../beam/sdk/nexmark/model/sql/RowSizeTest.java | 114 -
sdks/java/testing/test-utils/build.gradle | 1 +
sdks/python/.pylintrc | 18 +-
sdks/python/apache_beam/__init__.py | 8 +-
sdks/python/apache_beam/coders/coder_impl.pxd | 4 +
sdks/python/apache_beam/coders/coder_impl.py | 32 +
sdks/python/apache_beam/coders/coders.py | 41 +-
sdks/python/apache_beam/coders/coders_test.py | 4 +-
.../apache_beam/coders/coders_test_common.py | 12 +-
sdks/python/apache_beam/coders/fast_coders_test.py | 2 +-
sdks/python/apache_beam/coders/slow_coders_test.py | 4 +-
.../apache_beam/coders/standard_coders_test.py | 1 +
sdks/python/apache_beam/coders/typecoders.py | 4 +-
sdks/python/apache_beam/coders/typecoders_test.py | 22 +
sdks/python/apache_beam/examples/avro_bitcoin.py | 4 +-
.../apache_beam/examples/complete/autocomplete.py | 4 +-
.../apache_beam/examples/complete/distribopt.py | 5 +-
.../examples/complete/distribopt_test.py | 7 +-
.../examples/complete/game/game_stats.py | 16 +-
.../examples/complete/game/game_stats_it_test.py | 3 +-
.../examples/complete/game/hourly_team_score.py | 20 +-
.../complete/game/hourly_team_score_it_test.py | 3 +-
.../examples/complete/game/leader_board.py | 24 +-
.../examples/complete/game/leader_board_it_test.py | 3 +-
.../examples/complete/game/user_score.py | 12 +-
.../examples/complete/game/user_score_it_test.py | 3 +-
.../examples/complete/juliaset/setup.py | 3 +-
sdks/python/apache_beam/examples/complete/tfidf.py | 4 +-
.../apache_beam/examples/complete/tfidf_test.py | 7 +-
.../examples/complete/top_wikipedia_sessions.py | 4 +-
.../examples/cookbook/bigtableio_it_test.py | 4 +-
.../examples/cookbook/datastore_wordcount.py | 4 +-
.../examples/cookbook/group_with_coder.py | 4 +-
.../examples/cookbook/group_with_coder_test.py | 16 +-
.../apache_beam/examples/cookbook/mergecontacts.py | 4 +-
.../examples/cookbook/mergecontacts_test.py | 14 +-
.../examples/cookbook/multiple_output_pardo.py | 4 +-
.../cookbook/multiple_output_pardo_test.py | 7 +-
.../apache_beam/examples/snippets/snippets.py | 7 +-
.../{element_wise => aggregation}/__init__.py | 0
.../transforms/aggregation/cogroupbykey.py | 49 +
.../transforms/aggregation/cogroupbykey_test.py | 59 +
.../transforms/element_wise/filter_test.py | 80 -
.../snippets/transforms/element_wise/flat_map.py | 248 --
.../transforms/element_wise/flat_map_test.py | 92 -
.../snippets/transforms/element_wise/keys_test.py | 55 -
.../transforms/element_wise/kvswap_test.py | 55 -
.../snippets/transforms/element_wise/map_test.py | 90 -
.../snippets/transforms/element_wise/pardo.py | 83 -
.../snippets/transforms/element_wise/pardo_test.py | 82 -
.../snippets/transforms/element_wise/partition.py | 136 -
.../transforms/element_wise/partition_test.py | 84 -
.../snippets/transforms/element_wise/regex.py | 161 -
.../snippets/transforms/element_wise/regex_test.py | 115 -
.../snippets/transforms/element_wise/to_string.py | 42 -
.../transforms/element_wise/to_string_test.py | 55 -
.../transforms/element_wise/values_test.py | 55 -
.../transforms/element_wise/with_timestamps.py | 128 -
.../element_wise/with_timestamps_test.py | 103 -
.../snippets/transforms/elementwise}/__init__.py | 0
.../{element_wise => elementwise}/filter.py | 0
.../snippets/transforms/elementwise/filter_test.py | 75 +
.../snippets/transforms/elementwise/flatmap.py | 248 ++
.../transforms/elementwise/flatmap_test.py | 86 +
.../{element_wise => elementwise}/keys.py | 0
.../snippets/transforms/elementwise/keys_test.py | 52 +
.../{element_wise => elementwise}/kvswap.py | 0
.../snippets/transforms/elementwise/kvswap_test.py | 52 +
.../{element_wise => elementwise}/map.py | 0
.../snippets/transforms/elementwise/map_test.py | 84 +
.../snippets/transforms/elementwise/pardo.py | 125 +
.../snippets/transforms/elementwise/pardo_test.py | 123 +
.../snippets/transforms/elementwise/partition.py | 121 +
.../transforms/elementwise/partition_test.py | 102 +
.../snippets/transforms/elementwise/regex.py | 236 ++
.../snippets/transforms/elementwise/regex_test.py | 155 +
.../snippets/transforms/elementwise/tostring.py | 86 +
.../transforms/elementwise/tostring_test.py | 80 +
.../{element_wise => elementwise}/values.py | 0
.../snippets/transforms/elementwise/values_test.py | 52 +
.../transforms/elementwise/withtimestamps.py | 128 +
.../transforms/elementwise/withtimestamps_test.py | 95 +
sdks/python/apache_beam/examples/snippets/util.py | 40 +-
.../apache_beam/examples/snippets/util_test.py | 65 +-
.../apache_beam/examples/streaming_wordcount.py | 6 +-
.../examples/streaming_wordcount_it_test.py | 3 +-
sdks/python/apache_beam/examples/wordcount.py | 7 +-
.../apache_beam/examples/wordcount_debugging.py | 8 +-
.../examples/wordcount_debugging_test.py | 9 +-
.../apache_beam/examples/wordcount_it_test.py | 3 +-
.../apache_beam/examples/wordcount_minimal.py | 4 +-
.../apache_beam/examples/wordcount_minimal_test.py | 7 +-
sdks/python/apache_beam/examples/wordcount_test.py | 7 +-
sdks/python/apache_beam/internal/gcp/auth.py | 9 +
sdks/python/apache_beam/internal/http_client.py | 4 +-
sdks/python/apache_beam/internal/util.py | 11 +-
sdks/python/apache_beam/io/avroio_test.py | 12 +-
.../io/{gcp/tests => external/gcp}/__init__.py | 0
sdks/python/apache_beam/io/external/gcp/pubsub.py | 168 +
.../apache_beam/io/external/generate_sequence.py | 62 +-
sdks/python/apache_beam/io/external/kafka.py | 174 +-
sdks/python/apache_beam/io/filebasedsink.py | 4 +-
sdks/python/apache_beam/io/filebasedsink_test.py | 8 +-
sdks/python/apache_beam/io/filebasedsource_test.py | 20 +-
sdks/python/apache_beam/io/filesystem.py | 11 +-
sdks/python/apache_beam/io/filesystems_test.py | 18 +-
.../io/flink/flink_streaming_impulse_source.py | 2 +-
.../io/gcp/big_query_query_to_table_it_test.py | 6 +
sdks/python/apache_beam/io/gcp/bigquery.py | 109 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 376 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 253 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 179 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 56 +-
.../apache_beam/io/gcp/bigquery_tools_test.py | 54 +-
.../apache_beam/io/gcp/datastore/v1/datastoreio.py | 4 +-
.../io/gcp/datastore/v1/query_splitter_test.py | 12 +-
.../datastore/v1new/datastore_write_it_pipeline.py | 2 -
.../io/gcp/datastore/v1new/query_splitter_test.py | 4 +-
.../apache_beam/io/gcp/datastore/v1new/types.py | 6 +
.../io/gcp/datastore/v1new/types_test.py | 58 +-
.../io/gcp/datastore_write_it_pipeline.py | 2 -
.../apache_beam/io/gcp/gcsfilesystem_test.py | 22 +-
sdks/python/apache_beam/io/gcp/gcsio_overrides.py | 51 +
sdks/python/apache_beam/io/gcp/gcsio_test.py | 14 +-
.../internal/clients/storage/storage_v1_client.py | 2 +
.../apache_beam/io/gcp/pubsub_it_pipeline.py | 4 -
sdks/python/apache_beam/io/gcp/pubsub_test.py | 28 +-
.../io/gcp/tests/bigquery_matcher_test.py | 1 -
.../io/gcp/tests/pubsub_matcher_test.py | 8 +-
sdks/python/apache_beam/io/gcp/tests/utils_test.py | 8 +-
.../python/apache_beam/io/hadoopfilesystem_test.py | 30 +-
sdks/python/apache_beam/io/iobase.py | 6 +-
sdks/python/apache_beam/io/localfilesystem_test.py | 30 +-
sdks/python/apache_beam/io/mongodbio.py | 234 +-
sdks/python/apache_beam/io/mongodbio_it_test.py | 18 +-
sdks/python/apache_beam/io/mongodbio_test.py | 305 +-
sdks/python/apache_beam/io/parquetio.py | 126 +-
sdks/python/apache_beam/io/parquetio_it_test.py | 10 +-
sdks/python/apache_beam/io/parquetio_test.py | 147 +-
sdks/python/apache_beam/io/restriction_trackers.py | 27 +-
.../apache_beam/io/restriction_trackers_test.py | 60 +-
sdks/python/apache_beam/io/source_test_utils.py | 6 +-
sdks/python/apache_beam/io/tfrecordio.py | 21 +-
sdks/python/apache_beam/io/tfrecordio_test.py | 4 +-
sdks/python/apache_beam/io/utils_test.py | 2 +-
sdks/python/apache_beam/io/vcfio.py | 10 +-
.../python/apache_beam/options/pipeline_options.py | 41 +-
.../apache_beam/options/value_provider_test.py | 2 +-
sdks/python/apache_beam/pipeline.py | 11 +-
sdks/python/apache_beam/pipeline_test.py | 87 +-
sdks/python/apache_beam/portability/python_urns.py | 4 +-
sdks/python/apache_beam/pvalue.py | 19 +-
sdks/python/apache_beam/pvalue_test.py | 7 +-
sdks/python/apache_beam/runners/common.pxd | 2 +
sdks/python/apache_beam/runners/common.py | 97 +-
.../dataflow/dataflow_exercise_metrics_pipeline.py | 85 +-
.../dataflow_exercise_metrics_pipeline_test.py | 8 +-
...dataflow_exercise_streaming_metrics_pipeline.py | 100 +
...low_exercise_streaming_metrics_pipeline_test.py | 177 +
.../runners/dataflow/dataflow_metrics.py | 12 +-
.../runners/dataflow/dataflow_runner.py | 61 +-
.../runners/dataflow/dataflow_runner_test.py | 32 +-
.../runners/dataflow/internal/apiclient.py | 25 +-
.../runners/dataflow/internal/apiclient_test.py | 27 +-
.../clients/dataflow/dataflow_v1b3_client.py | 230 --
.../clients/dataflow/dataflow_v1b3_messages.py | 373 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/dataflow/native_io/iobase.py | 2 -
.../runners/dataflow/native_io/streaming_create.py | 2 +-
.../runners/dataflow/test_dataflow_runner.py | 2 +-
.../apache_beam/runners/direct/direct_runner.py | 11 +-
.../runners/direct/sdf_direct_runner.py | 2 +-
.../runners/direct/sdf_direct_runner_test.py | 10 +-
.../apache_beam/runners/interactive/README.md | 4 +-
.../runners/interactive/interactive_beam.py | 86 +
.../runners/interactive/interactive_beam_test.py | 71 +
.../runners/interactive/interactive_environment.py | 107 +
.../interactive/interactive_environment_test.py | 93 +
.../runners/interactive/interactive_runner.py | 4 +
.../runners/interactive/pipeline_instrument.py | 433 +++
.../interactive/pipeline_instrument_test.py | 283 ++
.../python/apache_beam/runners/pipeline_context.py | 4 +-
.../runners/portability/abstract_job_service.py | 139 +
.../runners/portability/artifact_service.py | 207 +-
.../runners/portability/artifact_service_test.py | 38 +-
.../runners/portability/flink_runner.py | 31 +-
.../runners/portability/flink_runner_test.py | 26 +-
.../portability/flink_uber_jar_job_server.py | 245 ++
.../portability/flink_uber_jar_job_server_test.py | 138 +
.../runners/portability/fn_api_runner.py | 257 +-
.../runners/portability/fn_api_runner_test.py | 130 +-
.../apache_beam/runners/portability/job_server.py | 156 +-
.../runners/portability/local_job_service.py | 152 +-
.../runners/portability/local_job_service_main.py | 1 -
.../runners/portability/portable_runner.py | 152 +-
.../runners/portability/portable_runner_test.py | 81 +-
.../apache_beam/runners/portability/stager.py | 8 +-
sdks/python/apache_beam/runners/runner.py | 12 +-
sdks/python/apache_beam/runners/sdf_common.py | 2 +-
.../apache_beam/runners/worker/bundle_processor.py | 95 +-
.../apache_beam/runners/worker/data_plane.py | 22 +-
.../apache_beam/runners/worker/data_plane_test.py | 16 +-
.../apache_beam/runners/worker/log_handler.py | 4 +-
.../apache_beam/runners/worker/opcounters_test.py | 2 +-
.../apache_beam/runners/worker/operations.py | 5 +-
.../apache_beam/runners/worker/sdk_worker.py | 249 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 29 +-
.../runners/worker/sdk_worker_main_test.py | 5 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 3 +-
.../apache_beam/runners/worker/statecache.py | 122 +
.../apache_beam/runners/worker/statecache_test.py | 155 +
.../apache_beam/runners/worker/worker_pool_main.py | 191 +
.../testing/benchmarks/chicago_taxi/preprocess.py | 6 +-
.../benchmarks/chicago_taxi/process_tfma.py | 8 +-
.../chicago_taxi/tfdv_analyze_and_validate.py | 6 +-
.../apache_beam/testing/datatype_inference.py | 144 +
.../apache_beam/testing/datatype_inference_test.py | 207 ++
.../python/apache_beam/testing/extra_assertions.py | 6 +-
.../apache_beam/testing/extra_assertions_test.py | 23 +-
.../apache_beam/testing/load_tests/build.gradle | 12 +-
.../testing/load_tests/load_test_metrics_utils.py | 238 +-
.../apache_beam/testing/synthetic_pipeline.py | 61 +-
.../apache_beam/testing/synthetic_pipeline_test.py | 58 +-
sdks/python/apache_beam/testing/test_pipeline.py | 9 +-
sdks/python/apache_beam/testing/test_stream.py | 2 +-
sdks/python/apache_beam/testing/util.py | 17 +-
.../apache_beam/tools/sideinput_microbenchmark.py | 2 +-
sdks/python/apache_beam/transforms/core.py | 174 +-
sdks/python/apache_beam/transforms/external.py | 248 +-
.../python/apache_beam/transforms/external_test.py | 201 +-
.../apache_beam/transforms/external_test_py3.py | 100 +
.../apache_beam/transforms/external_test_py37.py | 74 +
sdks/python/apache_beam/transforms/ptransform.py | 8 +-
.../apache_beam/transforms/ptransform_test.py | 59 +-
sdks/python/apache_beam/transforms/stats.py | 402 ++-
sdks/python/apache_beam/transforms/stats_test.py | 275 ++
sdks/python/apache_beam/transforms/trigger.py | 4 +-
sdks/python/apache_beam/transforms/trigger_test.py | 6 +-
.../apache_beam/transforms/userstate_test.py | 10 +-
sdks/python/apache_beam/transforms/util.py | 35 +-
sdks/python/apache_beam/transforms/util_test.py | 5 +-
sdks/python/apache_beam/typehints/decorators.py | 300 +-
.../apache_beam/typehints/decorators_test.py | 130 +
.../apache_beam/typehints/decorators_test_py3.py | 115 +
.../typehints/native_type_compatibility.py | 86 +-
.../typehints/native_type_compatibility_test.py | 35 +-
sdks/python/apache_beam/typehints/opcodes.py | 4 +-
.../typehints/trivial_inference_test.py | 7 +
sdks/python/apache_beam/typehints/typecheck.py | 5 +-
.../apache_beam/typehints/typed_pipeline_test.py | 111 +-
.../typehints/typed_pipeline_test_py3.py | 236 ++
sdks/python/apache_beam/typehints/typehints.py | 57 +-
.../python/apache_beam/typehints/typehints_test.py | 147 +-
.../apache_beam/typehints/typehints_test_py3.py | 56 +
sdks/python/apache_beam/utils/subprocess_server.py | 227 ++
sdks/python/apache_beam/utils/timestamp_test.py | 8 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/build.gradle | 40 +-
sdks/python/container/Dockerfile | 3 +-
sdks/python/container/base_image_requirements.txt | 53 +-
sdks/python/container/boot.go | 87 +-
sdks/python/container/build.gradle | 22 +-
.../python/container/extra_requirements/Dockerfile | 27 -
sdks/python/container/py2/build.gradle | 66 +
sdks/python/container/py3/Dockerfile | 53 -
sdks/python/container/py3/build.gradle | 52 -
sdks/python/container/py35/build.gradle | 66 +
sdks/python/container/py36/build.gradle | 66 +
sdks/python/container/py37/build.gradle | 67 +
sdks/python/container/run_validatescontainer.sh | 30 +-
sdks/python/gen_protos.py | 56 +-
sdks/python/scripts/add_requirements.sh | 45 -
sdks/python/scripts/generate_pydoc.sh | 4 +-
sdks/python/scripts/run_integration_test.sh | 2 +-
sdks/python/scripts/run_mini_py2lint.sh | 63 +
sdks/python/scripts/run_mini_py3lint.sh | 51 -
sdks/python/scripts/run_pylint.sh | 19 +-
sdks/python/setup.cfg | 3 +
sdks/python/setup.py | 41 +-
sdks/python/test-suites/dataflow/py2/build.gradle | 62 +-
sdks/python/test-suites/dataflow/py35/build.gradle | 28 +-
sdks/python/test-suites/dataflow/py36/build.gradle | 26 +-
sdks/python/test-suites/dataflow/py37/build.gradle | 34 +-
sdks/python/test-suites/direct/py2/build.gradle | 24 +-
sdks/python/test-suites/direct/py35/build.gradle | 41 +-
sdks/python/test-suites/direct/py36/build.gradle | 10 +-
sdks/python/test-suites/direct/py37/build.gradle | 15 +-
sdks/python/test-suites/portable/common.gradle | 80 +
sdks/python/test-suites/portable/py2/build.gradle | 127 +-
sdks/python/test-suites/portable/py35/build.gradle | 6 +-
sdks/python/test-suites/portable/py36/build.gradle | 32 +
sdks/python/test-suites/portable/py37/build.gradle | 32 +
sdks/python/test-suites/tox/py2/build.gradle | 1 -
sdks/python/test-suites/tox/py35/build.gradle | 7 -
sdks/python/test-suites/tox/py37/build.gradle | 6 +
sdks/python/tox.ini | 54 +-
settings.gradle | 26 +-
vendor/calcite-1_20_0/build.gradle | 63 +
vendor/sdks-java-extensions-protobuf/build.gradle | 1 +
website/_config.yml | 2 +-
website/build.gradle | 16 +-
website/notebooks/docs.yaml | 95 +
website/notebooks/generate.py | 128 +
website/notebooks/imports/license.md | 19 +
website/notebooks/imports/setup.md | 30 +
website/src/.htaccess | 2 +-
website/src/_data/authors.yml | 9 +-
website/src/_data/capability-matrix.yml | 6 +-
website/src/_includes/button-pydoc.md | 23 +
website/src/_includes/button.md | 21 +
website/src/_includes/buttons-code-snippet.md | 43 +
website/src/_includes/footer.html | 3 +-
website/src/_includes/head.html | 3 +
.../src/_includes/section-menu/documentation.html | 29 +-
website/src/_includes/section-menu/sdks.html | 23 +
website/src/_posts/2019-07-31-beam-2.14.0.md | 3 +-
website/src/_posts/2019-08-22-beam-2.15.0.md | 88 +
website/src/_posts/2019-09-04-gsoc-19.md | 93 +
website/src/_posts/2019-10-07-beam-2.16.0.md | 103 +
website/src/community/contact-us.md | 4 +-
website/src/community/in-person.md | 3 +-
website/src/contribute/design-documents.md | 150 +-
website/src/contribute/index.md | 44 +-
website/src/contribute/release-guide.md | 406 ++-
.../dsls/sql/calcite/aggregate-functions.md | 10 +-
.../documentation/dsls/sql/calcite/data-types.md | 11 +-
.../dsls/sql/calcite/lexical-structure.md | 9 +-
.../src/documentation/dsls/sql/calcite/overview.md | 27 +-
.../documentation/dsls/sql/calcite/query-syntax.md | 8 +-
.../dsls/sql/calcite/scalar-functions.md | 10 +-
website/src/documentation/dsls/sql/overview.md | 30 +-
website/src/documentation/dsls/sql/shell.md | 6 +-
.../dsls/sql/zetasql/aggregate-functions.md | 210 ++
.../dsls/sql/zetasql/conditional-expressions.md | 116 +
.../dsls/sql/zetasql/conversion-rules.md | 193 +
.../documentation/dsls/sql/zetasql/data-types.md | 457 +++
.../src/documentation/dsls/sql/zetasql/lexical.md | 573 +++
.../dsls/sql/zetasql/math-functions.md | 132 +
.../documentation/dsls/sql/zetasql/operators.md | 597 +++
.../src/documentation/dsls/sql/zetasql/overview.md | 67 +
.../documentation/dsls/sql/zetasql/query-syntax.md | 1215 +++++++
.../dsls/sql/zetasql/string-functions.md | 657 ++++
.../src/documentation/dsls/sql/zetasql/syntax.md | 34 +
website/src/documentation/execution-model.md | 210 --
website/src/documentation/index.md | 4 +-
website/src/documentation/io/built-in-hcatalog.md | 4 +-
website/src/documentation/io/built-in-parquet.md | 148 +
website/src/documentation/io/built-in.md | 5 +-
.../src/documentation/io/developing-io-overview.md | 10 +
website/src/documentation/io/testing.md | 377 +-
.../documentation/patterns/custom-io-patterns.md | 42 -
website/src/documentation/patterns/custom-io.md | 42 +
.../src/documentation/patterns/custom-windows.md | 114 +
.../patterns/file-processing-patterns.md | 107 -
.../src/documentation/patterns/file-processing.md | 107 +
website/src/documentation/patterns/overview.md | 15 +-
.../patterns/pipeline-option-patterns.md | 47 -
.../src/documentation/patterns/pipeline-options.md | 47 +
.../documentation/patterns/side-input-patterns.md | 48 -
website/src/documentation/patterns/side-inputs.md | 48 +
website/src/documentation/programming-guide.md | 225 +-
.../documentation/resources/learning-resources.md | 2 +-
.../documentation/resources/videos-and-podcasts.md | 20 +
website/src/documentation/runners/direct.md | 71 +
website/src/documentation/runners/flink.md | 83 +-
website/src/documentation/runners/spark.md | 25 +-
website/src/documentation/runtime/environments.md | 183 +
website/src/documentation/runtime/model.md | 212 ++
website/src/documentation/sdks/euphoria.md | 2 +-
website/src/documentation/sdks/nexmark.md | 6 +-
.../src/documentation/sdks/python-dependencies.md | 81 +
website/src/documentation/sdks/python-streaming.md | 7 +-
website/src/documentation/sdks/python.md | 1 -
.../transforms/python/element-wise/filter.md | 195 -
.../transforms/python/element-wise/flatmap.md | 42 -
.../transforms/python/element-wise/keys.md | 39 -
.../transforms/python/element-wise/kvswap.md | 39 -
.../transforms/python/element-wise/map.md | 42 -
.../transforms/python/element-wise/pardo.md | 44 -
.../transforms/python/element-wise/partition.md | 50 -
.../transforms/python/element-wise/regex.md | 28 -
.../transforms/python/element-wise/tostring.md | 27 -
.../transforms/python/element-wise/values.md | 37 -
.../python/element-wise/withtimestamps.md | 28 -
.../transforms/python/elementwise/filter.md | 176 +
.../transforms/python/elementwise/flatmap.md | 240 ++
.../transforms/python/elementwise/keys.md | 56 +
.../transforms/python/elementwise/kvswap.md | 57 +
.../transforms/python/elementwise/map.md | 216 ++
.../transforms/python/elementwise/pardo.md | 167 +
.../transforms/python/elementwise/partition.md | 144 +
.../transforms/python/elementwise/regex.md | 299 ++
.../python/{element-wise => elementwise}/reify.md | 0
.../transforms/python/elementwise/tostring.md | 104 +
.../transforms/python/elementwise/values.md | 56 +
.../{element-wise => elementwise}/withkeys.md | 0
.../python/elementwise/withtimestamps.md | 120 +
website/src/get-started/downloads.md | 16 +-
website/src/get-started/quickstart-go.md | 2 +-
website/src/get-started/quickstart-java.md | 12 +-
website/src/get-started/quickstart-py.md | 2 +
website/src/get-started/wordcount-example.md | 55 +-
website/src/images/dofn-sequence-diagram.svg | 94 +
website/src/images/source-sequence-diagram.svg | 106 +
.../src/images/standard-vs-dynamic-sessions.png | Bin 0 -> 26026 bytes
website/src/roadmap/portability.md | 42 +-
website/src/roadmap/python-sdk.md | 2 +-
1438 files changed, 69165 insertions(+), 20675 deletions(-)
delete mode 100755 .test-infra/dataproc/create_flink_cluster.sh
create mode 100755 .test-infra/dataproc/flink_cluster.sh
create mode 100644 .test-infra/jenkins/Docker.groovy
create mode 100644 .test-infra/jenkins/Flink.groovy
delete mode 100644 .test-infra/jenkins/Infrastructure.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Flink_Python.groovy
create mode 100644 .test-infra/jenkins/job_PerformanceTests_BigQueryIO_Java.groovy
delete mode 100644 .test-infra/jenkins/job_PerformanceTests_FileBasedIO_IT_HDFS.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_PortableJar_Flink.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Python35_ValidatesRunner_Flink.groovy
create mode 100644 .test-infra/jenkins/job_PreCommit_BeamSQL_ZetaSQL.groovy
create mode 100644 .test-infra/jenkins/job_PreCommit_PythonLint.groovy
create mode 100644 .test-infra/jenkins/job_Release_Gradle_Build.groovy
delete mode 100644 .test-infra/kubernetes/hadoop/LargeITCluster/pkb-config.yml
delete mode 100644 .test-infra/kubernetes/hadoop/SmallITCluster/pkb-config.yml
delete mode 100644 .test-infra/kubernetes/mongodb/load-balancer/pkb-config.yml
delete mode 100644 .test-infra/kubernetes/postgres/pkb-config-local.yml
delete mode 100644 .test-infra/kubernetes/postgres/pkb-config.yml
create mode 100755 .test-infra/metrics/apply_configmaps.sh
create mode 100644 .test-infra/metrics/beamprometheus-deploy.yaml
create mode 100644 .test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml
create mode 100644 .test-infra/metrics/prometheus/prometheus/config/prometheus.yml
create mode 100644 .test-infra/metrics/prometheus/prometheus/config/rules.yml
delete mode 100644 deprecation-warning.txt
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/filter-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/flatmap-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/keys-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/kvswap-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/map-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/pardo-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/partition-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/regex-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/tostring-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/values-py.ipynb
create mode 100644 examples/notebooks/documentation/transforms/python/elementwise/withtimestamps-py.ipynb
delete mode 100755 gradlew_orig
delete mode 100644 gradlew_orig.bat
delete mode 100644 project-mappings
create mode 100644 release/src/main/scripts/publish_docker_images.sh
delete mode 100644 runners/flink/1.5/build.gradle
delete mode 100644 runners/flink/1.5/job-server-container/build.gradle
delete mode 100644 runners/flink/1.5/job-server/build.gradle
delete mode 100644 runners/flink/1.5/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
delete mode 100644 runners/flink/1.6/build.gradle
delete mode 100644 runners/flink/1.6/job-server/build.gradle
rename runners/flink/{1.5 => 1.7}/src/main/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializer.java (100%)
rename runners/flink/{1.5 => 1.7}/src/main/java/org/apache/beam/runners/flink/translation/types/EncodedValueSerializer.java (100%)
create mode 100644 runners/flink/1.7/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
rename runners/flink/{1.5 => 1.7}/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java (100%)
create mode 100644 runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java
create mode 100644 runners/flink/1.7/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
rename runners/flink/{1.5 => 1.7}/src/test/java/org/apache/beam/runners/flink/translation/types/CoderTypeSerializerTest.java (100%)
create mode 100644 runners/flink/1.9/build.gradle
rename runners/flink/{1.6 => 1.9}/job-server-container/build.gradle (100%)
create mode 100644 runners/flink/1.9/job-server/build.gradle
create mode 100644 runners/flink/1.9/src/main/java/org/apache/beam/runners/flink/translation/wrappers/streaming/io/BeamStoppableFunction.java
create mode 100644 runners/flink/1.9/src/test/java/org/apache/beam/runners/flink/streaming/StreamSources.java
create mode 100755 runners/flink/job-server/test_pipeline_jar.sh
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableRunnerResult.java
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkStateBackendFactory.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContext.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContext.java
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkExecutableStageContextFactory.java
delete mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactory.java
delete mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/FlinkDefaultExecutableStageContextTest.java
delete mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/functions/ReferenceCountingFlinkExecutableStageContextFactoryTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregator.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregators.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregator.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregator.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregator.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/CounterUpdateAggregatorsTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/DistributionCounterUpdateAggregatorTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/MeanCounterUpdateAggregatorTest.java
create mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/counters/SumCounterUpdateAggregatorTest.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactRetrievalService.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/AbstractArtifactStagingService.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactRetrievalService.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/JavaFilesystemArtifactStagingService.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContext.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ExecutableStageContext.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactory.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreator.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarUtils.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineResult.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ClassLoaderArtifactServiceTest.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ReferenceCountingExecutableStageContextFactoryTest.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/jobsubmission/PortablePipelineJarCreatorTest.java
delete mode 100644 runners/jet-experimental/build.gradle
delete mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/Utils.java
delete mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
delete mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
delete mode 100644 runners/jet-experimental/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
create mode 100644 runners/jet/build.gradle
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/DAGBuilder.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/FailedRunningPipelineResults.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetGraphVisitor.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetPipelineOptions.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetPipelineResult.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetRunner.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetRunnerRegistrar.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetTransformTranslator.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetTransformTranslators.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/JetTranslationContext.java (100%)
create mode 100644 runners/jet/src/main/java/org/apache/beam/runners/jet/Utils.java
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/AbstractMetric.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/CounterImpl.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/DistributionImpl.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/GaugeImpl.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricResults.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/JetMetricsContainer.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/metrics/package-info.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/package-info.java (100%)
create mode 100644 runners/jet/src/main/java/org/apache/beam/runners/jet/processors/AbstractParDoP.java
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/AssignWindowP.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/BoundedSourceP.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/FlattenP.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/ImpulseP.java (100%)
create mode 100644 runners/jet/src/main/java/org/apache/beam/runners/jet/processors/ParDoP.java
create mode 100644 runners/jet/src/main/java/org/apache/beam/runners/jet/processors/StatefulParDoP.java
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/UnboundedSourceP.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/ViewP.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/WindowGroupP.java (100%)
rename runners/{jet-experimental => jet}/src/main/java/org/apache/beam/runners/jet/processors/package-info.java (100%)
rename runners/{jet-experimental => jet}/src/test/java/org/apache/beam/runners/jet/JetTestRunnerRegistrar.java (100%)
rename runners/{jet-experimental => jet}/src/test/java/org/apache/beam/runners/jet/TestJetRunner.java (100%)
rename runners/{jet-experimental => jet}/src/test/java/org/apache/beam/runners/jet/TestStreamP.java (100%)
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/SamzaPortablePipelineResult.java
create mode 100644 runners/samza/src/main/java/org/apache/beam/runners/samza/util/HashIdGenerator.java
create mode 100644 runners/samza/src/test/java/org/apache/beam/runners/samza/util/TestHashIdGenerator.java
create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/translation/SparkExecutableStageContextFactory.java
create mode 100644 runners/spark/src/main/java/org/apache/beam/runners/spark/util/TimerUtils.java
delete mode 100644 sdks/CONTAINERS.md
create mode 100644 sdks/go/pkg/beam/core/graph/coder/double.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/coder_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/serialize_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/logging_test.go
create mode 100644 sdks/go/pkg/beam/core/util/reflectx/functions_test.go
create mode 100644 sdks/go/pkg/beam/core/util/symtab/symtab_test.go
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/io/AvroUtils.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/options/RemoteEnvironmentOptions.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesSideInputsWithDifferentCoders.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/testing/UsesStrictTimerOrdering.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/ToJson.java
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/util/JsonToRowUtils.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJson.java
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonDeserializer.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/util/RowJsonUtils.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/options/RemoteEnvironmentOptionsTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/ToJsonTest.java
delete mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonDeserializerTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/util/RowJsonTest.java
create mode 100644 sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoder.java
create mode 100644 sdks/java/extensions/protobuf/src/main/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomain.java
create mode 100644 sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/DynamicProtoCoderTest.java
create mode 100644 sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/IsDynamicMessageEqual.java
create mode 100644 sdks/java/extensions/protobuf/src/test/java/org/apache/beam/sdk/extensions/protobuf/ProtoDomainTest.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryTableFactory.java
delete mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/BigQueryUtils.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ChainedTableFactory.java
delete mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogClientAdapter.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/GcsTableFactory.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubTableFactory.java
delete mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/PubsubUtils.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableFactory.java
delete mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/TableUtils.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtils.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogBigQueryIT.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/DataCatalogGCSIT.java
create mode 100644 sdks/java/extensions/sql/datacatalog/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/datacatalog/ZetaSqlIdUtilsTest.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/BeamSqlTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/TableNameExtractionUtils.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableName.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/TableResolutionUtils.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRel.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRel.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRel.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamCoGBKJoinRule.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamIOPushDownRule.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamJoinRule.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputJoinRule.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rule/BeamSideInputLookupJoinRule.java
delete mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/schema/BaseBeamTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BaseBeamTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/BeamSqlTableFilter.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolver.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/DefaultTableFilter.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/SchemaBaseBeamTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/FullNameTableProvider.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProvider.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/GenericRecordWriteConverter.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/package-info.java
delete mode 100644 sdks/java/extensions/sql/src/main/resources/org.apache.beam.sdks.java.extensions.sql.repackaged.org.codehaus.commons.compiler.properties
create mode 100644 sdks/java/extensions/sql/src/main/resources/org.apache.beam.vendor.calcite.v1_20_0.org.codehaus.commons.compiler.properties
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteCannotParseSimpleIdentifiersTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/CalciteParsesSimpleIdentifiersTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelBoundedVsBoundedTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamCoGBKJoinRelUnboundedVsUnboundedTest.java
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelBoundedVsBoundedTest.java
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsBoundedTest.java
delete mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamJoinRelUnboundedVsUnboundedTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputJoinRelTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rel/BeamSideInputLookupJoinRelTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/impl/rule/IOPushDownRuleTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/CustomTableResolverTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/avro/AvroTableProviderTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithProjectPushDown.java
create mode 100644 sdks/java/extensions/sql/zetasql/build.gradle
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamBuiltinMethods.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/BeamCodegenUtils.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateFunctions.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/DateTimeUtils.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/QueryTrait.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlAnalyzer.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCaseWithValueOperatorRewriter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlCoalesceOperatorRewriter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlExtractTimestampOperatorRewriter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlIfNullOperatorRewriter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlNullIfOperatorRewriter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperatorRewriter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlOperators.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/SqlStdOperatorMappingTable.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/StringFunctions.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolution.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolutionContext.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolver.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TableResolverImpl.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TestInput.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TimestampFunctions.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/TypeUtils.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLCastFunctionImpl.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPlannerImpl.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLQueryPlanner.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSqlIdUtils.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/package-info.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/AggregateScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToJoinConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanToUncollectConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ConversionContext.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ExpressionConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/FilterScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/JoinScanWithRefConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToLimitConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/LimitOffsetScanToOrderByLimitConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/OrderByScanUnsupportedConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ProjectScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/QueryStatementConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/RelConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SetOperationScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/SingleRowScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/TableScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithRefScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/WithScanConverter.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/package-info.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/JoinCompoundIdentifiersTest.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLDialectSpecTest.java
create mode 100644 sdks/java/extensions/zetasketch/build.gradle
create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCount.java
create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountInitFn.java
create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HllCountMergePartialFn.java
create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/HyperLogLogPlusPlusCoder.java
create mode 100644 sdks/java/extensions/zetasketch/src/main/java/org/apache/beam/sdk/extensions/zetasketch/package-info.java
create mode 100644 sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/BigQueryHllSketchCompatibilityIT.java
create mode 100644 sdks/java/extensions/zetasketch/src/test/java/org/apache/beam/sdk/extensions/zetasketch/HllCountTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsClientProvider.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/PublishResponseCoder.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsCoderProviderRegistrar.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsIO.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/package-info.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockErrors.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsClientMockSuccess.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/resources/mockito-extensions/org.mockito.plugins.MockMaker
create mode 100644 sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOIT.java
delete mode 100644 sdks/java/io/bigquery-io-perf-tests/src/test/java/org/apache/beam/sdk/bigqueryioperftests/BigQueryIOReadPerformanceIT.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoder.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoder.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubIOExternalTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessagePayloadOnlyCoderTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesAndMessageIdCoderTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithAttributesCoderTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/pubsub/PubsubMessageWithMessageIdCoderTest.java
delete mode 100644 sdks/java/testing/nexmark/src/main/java/org/apache/beam/sdk/nexmark/model/sql/RowSize.java
delete mode 100644 sdks/java/testing/nexmark/src/test/java/org/apache/beam/sdk/nexmark/model/sql/RowSizeTest.java
rename sdks/python/apache_beam/examples/snippets/transforms/{element_wise => aggregation}/__init__.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/aggregation/cogroupbykey_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/filter_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/flat_map_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/keys_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/kvswap_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/map_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/pardo_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/partition_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/regex_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/to_string_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/values_test.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps.py
delete mode 100644 sdks/python/apache_beam/examples/snippets/transforms/element_wise/with_timestamps_test.py
copy sdks/python/apache_beam/{io/gcp/tests => examples/snippets/transforms/elementwise}/__init__.py (100%)
rename sdks/python/apache_beam/examples/snippets/transforms/{element_wise => elementwise}/filter.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/filter_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/flatmap_test.py
rename sdks/python/apache_beam/examples/snippets/transforms/{element_wise => elementwise}/keys.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/keys_test.py
rename sdks/python/apache_beam/examples/snippets/transforms/{element_wise => elementwise}/kvswap.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/kvswap_test.py
rename sdks/python/apache_beam/examples/snippets/transforms/{element_wise => elementwise}/map.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/map_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/pardo_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/partition_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/regex_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/tostring_test.py
rename sdks/python/apache_beam/examples/snippets/transforms/{element_wise => elementwise}/values.py (100%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/values_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/elementwise/withtimestamps_test.py
copy sdks/python/apache_beam/io/{gcp/tests => external/gcp}/__init__.py (100%)
create mode 100644 sdks/python/apache_beam/io/external/gcp/pubsub.py
create mode 100644 sdks/python/apache_beam/io/gcp/gcsio_overrides.py
create mode 100644 sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline.py
create mode 100644 sdks/python/apache_beam/runners/dataflow/dataflow_exercise_streaming_metrics_pipeline_test.py
create mode 100644 sdks/python/apache_beam/runners/interactive/interactive_beam.py
create mode 100644 sdks/python/apache_beam/runners/interactive/interactive_beam_test.py
create mode 100644 sdks/python/apache_beam/runners/interactive/interactive_environment.py
create mode 100644 sdks/python/apache_beam/runners/interactive/interactive_environment_test.py
create mode 100644 sdks/python/apache_beam/runners/interactive/pipeline_instrument.py
create mode 100644 sdks/python/apache_beam/runners/interactive/pipeline_instrument_test.py
create mode 100644 sdks/python/apache_beam/runners/portability/abstract_job_service.py
create mode 100644 sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server.py
create mode 100644 sdks/python/apache_beam/runners/portability/flink_uber_jar_job_server_test.py
create mode 100644 sdks/python/apache_beam/runners/worker/statecache.py
create mode 100644 sdks/python/apache_beam/runners/worker/statecache_test.py
create mode 100644 sdks/python/apache_beam/runners/worker/worker_pool_main.py
create mode 100644 sdks/python/apache_beam/testing/datatype_inference.py
create mode 100644 sdks/python/apache_beam/testing/datatype_inference_test.py
create mode 100644 sdks/python/apache_beam/transforms/external_test_py3.py
create mode 100644 sdks/python/apache_beam/transforms/external_test_py37.py
create mode 100644 sdks/python/apache_beam/typehints/decorators_test.py
create mode 100644 sdks/python/apache_beam/typehints/decorators_test_py3.py
create mode 100644 sdks/python/apache_beam/typehints/typed_pipeline_test_py3.py
create mode 100644 sdks/python/apache_beam/typehints/typehints_test_py3.py
create mode 100644 sdks/python/apache_beam/utils/subprocess_server.py
delete mode 100644 sdks/python/container/extra_requirements/Dockerfile
create mode 100644 sdks/python/container/py2/build.gradle
delete mode 100644 sdks/python/container/py3/Dockerfile
delete mode 100644 sdks/python/container/py3/build.gradle
create mode 100644 sdks/python/container/py35/build.gradle
create mode 100644 sdks/python/container/py36/build.gradle
create mode 100644 sdks/python/container/py37/build.gradle
delete mode 100755 sdks/python/scripts/add_requirements.sh
create mode 100755 sdks/python/scripts/run_mini_py2lint.sh
delete mode 100755 sdks/python/scripts/run_mini_py3lint.sh
create mode 100644 sdks/python/test-suites/portable/common.gradle
create mode 100644 sdks/python/test-suites/portable/py36/build.gradle
create mode 100644 sdks/python/test-suites/portable/py37/build.gradle
create mode 100644 vendor/calcite-1_20_0/build.gradle
create mode 100644 website/notebooks/docs.yaml
create mode 100644 website/notebooks/generate.py
create mode 100644 website/notebooks/imports/license.md
create mode 100644 website/notebooks/imports/setup.md
create mode 100644 website/src/_includes/button-pydoc.md
create mode 100644 website/src/_includes/button.md
create mode 100644 website/src/_includes/buttons-code-snippet.md
create mode 100644 website/src/_posts/2019-08-22-beam-2.15.0.md
create mode 100644 website/src/_posts/2019-09-04-gsoc-19.md
create mode 100644 website/src/_posts/2019-10-07-beam-2.16.0.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/aggregate-functions.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/conditional-expressions.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/conversion-rules.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/data-types.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/lexical.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/math-functions.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/operators.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/overview.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/query-syntax.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/string-functions.md
create mode 100644 website/src/documentation/dsls/sql/zetasql/syntax.md
delete mode 100644 website/src/documentation/execution-model.md
create mode 100644 website/src/documentation/io/built-in-parquet.md
delete mode 100644 website/src/documentation/patterns/custom-io-patterns.md
create mode 100644 website/src/documentation/patterns/custom-io.md
create mode 100644 website/src/documentation/patterns/custom-windows.md
delete mode 100644 website/src/documentation/patterns/file-processing-patterns.md
create mode 100644 website/src/documentation/patterns/file-processing.md
delete mode 100644 website/src/documentation/patterns/pipeline-option-patterns.md
create mode 100644 website/src/documentation/patterns/pipeline-options.md
delete mode 100644 website/src/documentation/patterns/side-input-patterns.md
create mode 100644 website/src/documentation/patterns/side-inputs.md
create mode 100644 website/src/documentation/runtime/environments.md
create mode 100644 website/src/documentation/runtime/model.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/filter.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/flatmap.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/keys.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/kvswap.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/map.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/pardo.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/partition.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/regex.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/tostring.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/values.md
delete mode 100644 website/src/documentation/transforms/python/element-wise/withtimestamps.md
create mode 100644 website/src/documentation/transforms/python/elementwise/filter.md
create mode 100644 website/src/documentation/transforms/python/elementwise/flatmap.md
create mode 100644 website/src/documentation/transforms/python/elementwise/keys.md
create mode 100644 website/src/documentation/transforms/python/elementwise/kvswap.md
create mode 100644 website/src/documentation/transforms/python/elementwise/map.md
create mode 100644 website/src/documentation/transforms/python/elementwise/pardo.md
create mode 100644 website/src/documentation/transforms/python/elementwise/partition.md
create mode 100644 website/src/documentation/transforms/python/elementwise/regex.md
rename website/src/documentation/transforms/python/{element-wise => elementwise}/reify.md (100%)
create mode 100644 website/src/documentation/transforms/python/elementwise/tostring.md
create mode 100644 website/src/documentation/transforms/python/elementwise/values.md
rename website/src/documentation/transforms/python/{element-wise => elementwise}/withkeys.md (100%)
create mode 100644 website/src/documentation/transforms/python/elementwise/withtimestamps.md
create mode 100644 website/src/images/dofn-sequence-diagram.svg
create mode 100644 website/src/images/source-sequence-diagram.svg
create mode 100644 website/src/images/standard-vs-dynamic-sessions.png