You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by ec...@apache.org on 2019/11/19 08:40:48 UTC
[beam] branch spark-runner_structured-streaming updated (fe9e723 ->
27dcc48)
This is an automated email from the ASF dual-hosted git repository.
echauchot pushed a change to branch spark-runner_structured-streaming
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard fe9e723 [BEAM-8470] Fix CombineTest.testCountPerElementWithSlidingWindows expected results
discard 87b4de7 [BEAM-8470] set log level to info to avoid resource consumption in production mode
discard 9d5fdb8 [BEAM-8470] Fix the output timestamps of combine: timestamps must be merged to a window before combining
discard 5a059d1 [BEAM-8470] Add a countPerElement test with sliding windows
discard d949942 [BEAM-8470] Apply spotless
discard 4a2c59d [BEAM-8470] Fix: create an empty accumulator in combine.mergeAccumulators, because this method modifies its first input accumulator. Decrease memory usage by storing only accumulator and timestamp in the combine.merge map
discard a7d67b2 [BEAM-8470] Add disclaimers about this runner being experimental
discard fba2109 [BEAM-8470] Fix wrong expected results in CombineTest.testBinaryCombineWithSlidingWindows
discard 2e048f9 [BEAM-8470] Fix tests: use correct SparkStructuredStreamingPipelineOptions, set testMode to true. Some renaming
discard c4f2575 [BEAM-8470] Add a test to test combine translation on binaryCombineFn with sliding windows
discard 9e4235d [BEAM-8470] Add a combine test with sliding windows
discard 8a3f75c [BEAM-8470] input windows can arrive exploded (for sliding windows). As a result an input has multiple windows. So we need to consider that the accumulator can have multiple windows
discard 59ba063 [BEAM-8470] Fix combiner. Do not reuse instance of accumulator
discard d224f1c [BEAM-8470] simplify coders in combinePerKey translation
discard bb62390 Fix SpotBugs
discard e40bb83 [BEAM-8470] Rebase on master: pass sideInputMapping in SimpleDoFnRunner as needed now in the API
discard 9c4a6ac [BEAM-8470] Apply spotless
discard f73ba27 [BEAM-8470] Add a jenkins job for validates runner tests in the new spark runner
discard 573d177 [BEAM-8470] Remove Encoders based on kryo now that we call Beam coders in the runner
discard 47526cf [BEAM-8470] Fix: Remove generic hack of using object. Use actual Coder encodedType in Encoders
discard c794671 [BEAM-8470] Remove unneeded cast
discard 216b9ca [BEAM-8470] Use beam encoders also in the output of the source translation
discard 1835cdb [BEAM-8470] Apply spotless, fix typo and javadoc
discard 717aafc [BEAM-8470] Apply new Encoders to Pardo. Replace Tuple2Coder with MultiOutputCoder to deal with multiple output to use in Spark Encoder for DoFnRunner
discard 5f0f28b [BEAM-8470] Apply new Encoders to GroupByKey
discard f4f34fc [BEAM-8470] Create a Tuple2Coder to encode scala tuple2
discard 766ad7f [BEAM-8470] Apply new Encoders to AggregatorCombiner
discard f594e96 [BEAM-8470] Apply new Encoders to Window assign translation
discard dd52c2f [BEAM-8470] Ignore long time failing test: SparkMetricsSinkTest
discard 1429d09 [BEAM-8470] Improve performance of source: the mapper already calls windowedValueCoder.decode, no need to call it also in the Spark encoder
discard d398ea0 [BEAM-8470] Apply new Encoders to Read source
discard 69dfd52 [BEAM-8470] Apply new Encoders to CombinePerKey
discard 887f4d3 [BEAM-8470] Catch Exception instead of IOException because some coders to not throw Exceptions at all (e.g.VoidCoder)
discard 8de3a5d [BEAM-8470] Put Encoders expressions serializable
discard 96ff856 [BEAM-8470] Wrap exceptions in UserCoderExceptions
discard 3120646 [BEAM-8470] Apply spotless and checkstyle and add javadocs
discard 0ad6034 [BEAM-8470] Add an assert of equality in the encoders test
discard 6b53a09 [BEAM-8470] Fix generated code: uniform exceptions catching, fix parenthesis and variable declarations
discard 4f4478a [BEAM-8470] Fix equal and hashcode
discard 261c181 [BEAM-8470] Remove example code
discard 31bc268 [BEAM-8470] Remove lazy init of beam coder because there is no generic way on instanciating a beam coder
discard c237789 [BEAM-8470] Fix beam coder lazy init using reflexion: use .clas + try catch + cast
discard 5941bca [BEAM-8470] Fix getting the output value in code generation
discard f4ebf7d [BEAM-8470] Fix ExpressionEncoder generated code: typos, try catch, fqcn
discard a4ddeef [BEAM-8470] Fix warning in coder construction by reflexion
discard 4f95b9d [BEAM-8470] Lazy init coder because coder instance cannot be interpolated by catalyst
discard d36d977 [BEAM-8470] Fix code generation in Beam coder wrapper
discard cf4cc45 [BEAM-8470] Add a simple spark native test to test Beam coders wrapping into Spark Encoders
discard 299bb3d [BEAM-8470] Conform to spark ExpressionEncoders: pass classTags, implement scala Product, pass children from within the ExpressionEncoder, fix visibilities
discard 637d8dd [BEAM-8470] Fix scala Product in Encoders to avoid StackEverflow
discard eef809c [BEAM-8470] type erasure: spark encoders require a Class<T>, pass Object and cast to Class<T>
discard 2a53396 [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: deserialization part
discard 666a060 [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: serialization part
discard cb36530 [BEAM-8470] Use "sparkMaster" in local mode to obtain number of shuffle partitions + spotless apply
discard 8a866cd [BEAM-8470] Improve Pardo translation performance: avoid calling a filter transform when there is only one output tag
discard ff72b1b [BEAM-8470] After testing performance and correctness, launch pipeline with dataset.foreach(). Make both test mode and production mode use foreach for uniformity. Move dataset print as a utility method
discard c4b2af1 [BEAM-8470] Remove no more needed AggregatorCombinerPerKey (there is only AggregatorCombiner)
discard 829b76e [BEAM-8470] fixup! Add PipelineResults to Spark structured streaming.
discard 4e06a92 [BEAM-8470] Print number of leaf datasets
discard 4bbca76 [BEAM-8470] Add spark execution plans extended debug messages.
discard 62e6b01 [BEAM-8470] Update log4j configuration
discard c6b638e [BEAM-8470] Add PipelineResults to Spark structured streaming.
discard e039015 [BEAM-8470] Make spotless happy
discard 820d422 [BEAM-8470] Added metrics sinks and tests
discard 8303a47 [BEAM-8470] Persist all output Dataset if there are multiple outputs in pipeline Enabled Use*Metrics tests
discard 78fea89 [BEAM-8470] Add a test to check that CombineGlobally preserves windowing
discard 66b61d6 [BEAM-8470] Fix accumulators initialization in Combine that prevented CombineGlobally to work.
discard 2fbcba8 [BEAM-8470] Fix javadoc
discard e5dabeb [BEAM-8470] Add setEnableSparkMetricSinks() method
discard ef70702 [BEAM-8470] Add missing dependencies to run Spark Structured Streaming Runner on Nexmark
discard b1d60d4 [BEAM-8470] Add metrics support in DoFn
discard c76654f [BEAM-8470] Ignore for now not working test testCombineGlobally
discard 0266777 [BEAM-8470] Add a test that combine per key preserves windowing
discard 737999f [BEAM-8470] Clean groupByKeyTest
discard aa3288e [BEAM-8470] add comment in combine globally test
discard 588ca31 [BEAM-8470] Fixed immutable list bug
discard a63290d [BEAM-8470] Fix javadoc of AggregatorCombiner
discard d6f5126 [BEAM-8470] Clean not more needed WindowingHelpers
discard 62e003c [BEAM-8470] Clean not more needed RowHelpers
discard 3df2363 [BEAM-8470] Clean no more needed KVHelpers
discard c0b7159 [BEAM-8470] Now that there is only Combine.PerKey translation, make only one Aggregator
discard 30ad19c [BEAM-8470] Remove CombineGlobally translation because it is less performant than the beam sdk one (key + combinePerKey.withHotkeyFanout)
discard 2d89260 [BEAM-8470] Remove the mapPartition that adds a key per partition because otherwise spark will reduce values per key instead of globally
discard fb3c17c [BEAM-8470] Fix bug in the window merging logic
discard 2bb7a3a [BEAM-8470] Fix wrong encoder in combineGlobally GBK
discard 9f7136b [BEAM-8470] Fix case when a window does not merge into any other window
discard 94d0057 [BEAM-8470] Apply a groupByKey avoids for some reason that the spark structured streaming fmwk casts data to Row which makes it impossible to deserialize without the coder shipped into the data. For performance reasons (avoid memory consumption and having to deserialize), we do not ship coder + data. Also add a mapparitions before GBK to avoid shuffling
discard e92dca1 [BEAM-8470] Revert extractKey while combinePerKey is not done (so that it compiles)
discard bca3996 [BEAM-8470] Fix encoder in combine call
discard dec6a1d [BEAM-8470] Implement merge accumulators part of CombineGlobally translation with windowing
discard e1dc45c [BEAM-8470] Output data after combine
discard 37ca462 [BEAM-8470] Implement reduce part of CombineGlobally translation with windowing
discard 04bc6a3 [BEAM-8470] Fix comment about schemas
discard cd494bd [BEAM-8470] Update KVHelpers.extractKey() to deal with WindowedValue and update GBK and CPK
discard cf58316 [BEAM-8470] Add TODO in Combine translations
discard 97a1bc8 [BEAM-8470] Add a test that GBK preserves windowing
discard bffa608 [BEAM-8470] Improve visibility of debug messages
discard 5dc128d [BEAM-8470] re-enable reduceFnRunner timers for output
discard 654ae07 [BEAM-8470] Re-code GroupByKeyTranslatorBatch to conserve windowing instead of unwindowing/windowing(GlobalWindow): simplify code, use ReduceFnRunner to merge the windows
discard 657ab5e [BEAM-8470] Add comment about checkpoint mark
discard e9eb45c [BEAM-8470] Update windowAssignTest
discard 4ee0993 [BEAM-8470] Put back batch/simpleSourceTest.testBoundedSource
discard b348923 [BEAM-8470] Consider null object case on RowHelpers, fixes empty side inputs tests.
discard 0910904 [BEAM-8470] Pass transform based doFnSchemaInformation in ParDo translation
discard e03157d [BEAM-8470] Fixes ParDo not calling setup and not tearing down if exception on startBundle
discard 745aee3 [BEAM-8470] Limit the number of partitions to make tests go 300% faster
discard 91b00f8 [BEAM-8470] Enable batch Validates Runner tests for Structured Streaming Runner
discard d3e1d91 [BEAM-8470] Apply Spotless
discard 85fe1f1 [BEAM-8470] Update javadoc
discard d8c27b8 [BEAM-8470] implement source.stop
discard 71dfb9b [BEAM-8470] Ignore spark offsets (cf javadoc)
discard 81bae1a [BEAM-8470] Use PAssert in Spark Structured Streaming transform tests
discard 9448630 [BEAM-8470] Rename SparkPipelineResult to SparkStructuredStreamingPipelineResult This is done to avoid an eventual collision with the one in SparkRunner. However this cannot happen at this moment because it is package private, so it is also done for consistency.
discard ae130cb [BEAM-8470] Add SparkStructuredStreamingPipelineOptions and SparkCommonPipelineOptions - SparkStructuredStreamingPipelineOptions was added to have the new runner rely only on its specific options.
discard 1b4a21c [BEAM-8470] Fix logging levels in Spark Structured Streaming translation
discard 8a99531 [BEAM-8470] Fix spotless issues after rebase
discard 1e2b5af [BEAM-8470] Pass doFnSchemaInformation to ParDo batch translation
discard 31ae06e [BEAM-8470] Fix non-vendored imports from Spark Streaming Runner classes
discard a0b2f1b [BEAM-8470] Merge Spark Structured Streaming runner into main Spark module. Remove spark-structured-streaming module.Rename Runner to SparkStructuredStreamingRunner. Remove specific PipelineOotions and Registrar for Structured Streaming Runner
discard b02809e [BEAM-8470] Fix access level issues, typos and modernize code to Java 8 style
discard a8112e0 [BEAM-8470] Disable never ending test SimpleSourceTest.testUnboundedSource
discard 63be481 [BEAM-8470] Apply spotless and fix spotbugs warnings
discard f4cd419 [BEAM-8470] Deal with checkpoint and offset based read
discard 94d4fae [BEAM-8470] Continue impl of offsets for streaming source
discard e3c3960 [BEAM-8470] Clean streaming source
discard 64e8adc [BEAM-8470] Clean unneeded 0 arg constructor in batch source
discard 801c823 [BEAM-8470] Specify checkpointLocation at the pipeline start
discard ebba00f [BEAM-8470] Add source streaming test
discard eafbd73 [BEAM-8470] Add transformators registry in PipelineTranslatorStreaming
discard 4eff413 [BEAM-8470] Add a TODO on spark output modes
discard 683e6f8 [BEAM-8470] Implement first streaming source
discard 04ed002 [BEAM-8470] Add streaming source initialisation
discard 6011073 [BEAM-8470] And unchecked warning suppression
discard 1dd1edd [BEAM-8470] Added TODO comment for ReshuffleTranslatorBatch
discard 7a02467 [BEAM-8470] Added using CachedSideInputReader
discard bd54aa4 [BEAM-8470] Don't use Reshuffle translation
discard 231f1cd [BEAM-8470] Fix CheckStyle violations
discard 4ea5960 [BEAM-8470] Added SideInput support
discard 3986c35 [BEAM-8470] Fix javadoc
discard 55cd936 [BEAM-8470] Implement WindowAssignTest
discard 1eb712c [BEAM-8470] Implement WindowAssignTranslatorBatch
discard 0d8595a [BEAM-8470] Cleaning
discard 6dbe5af [BEAM-8470] Fix encoder bug in combinePerkey
discard 5f5f097 [BEAM-8470] Add explanation about receiving a Row as input in the combiner
discard 642833d [BEAM-8470] Use more generic Row instead of GenericRowWithSchema
discard 2a04f4a [BEAM-8470] Fix combine. For unknown reason GenericRowWithSchema is used as input of combine so extract its content to be able to proceed
discard bd985de [BEAM-8470] Update test with Long
discard 6f40095 [BEAM-8470] Fix various type checking issues in Combine.Globally
discard 237ee28 [BEAM-8470] Get back to classes in translators resolution because URNs cannot translate Combine.Globally
discard c2eb95b [BEAM-8470] Cleaning
discard 911cdb4 [BEAM-8470] Add CombineGlobally translation to avoid translating Combine.perKey as a composite transform based on Combine.PerKey (which uses low perf GBK)
discard edc7a81 [BEAM-8470] Introduce RowHelpers
discard b1749ca [BEAM-8470] Add combinePerKey and CombineGlobally tests
discard fc8843d [BEAM-8470] Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless
discard 746fee2 [BEAM-8470] Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey
discard 2474926 [BEAM-8470] Improve type checking of Tuple2 encoder
discard 52634ef [BEAM-8470] First version of combinePerKey
discard 4bd38c14 [BEAM-8470] Extract binary schema creation in a helper class
discard bc3ca37 [BEAM-8470] Fix getSideInputs
discard 6c7a9ed [BEAM-8470] Generalize the use of SerializablePipelineOptions in place of (not serializable) PipelineOptions
discard 06343d4 [BEAM-8470] Rename SparkDoFnFilterFunction to DoFnFilterFunction for consistency
discard b5057cc [BEAM-8470] Add a test for the most simple possible Combine
discard e61507d [BEAM-8470] Added "testTwoPardoInRow"
discard 88b8edd [BEAM-8470] Fix for test elements container in GroupByKeyTest
discard 338f836 [BEAM-8470] Rename SparkOutputManager for consistency
discard e144c24 [BEAM-8470] Fix kryo issue in GBK translator with a workaround
discard 6dd56cc [BEAM-8470] Simplify logic of ParDo translator
discard 0b800d9 [BEAM-8470] Don't use deprecated sideInput.getWindowingStrategyInternal()
discard 75d32df [BEAM-8470] Rename SparkSideInputReader class and rename pruneOutput() to pruneOutputFilteredByTag()
discard 83547f4 [BEAM-8470] Fixed Javadoc error
discard ea34002 [BEAM-8470] Apply spotless
discard f600fd1 [BEAM-8470] Fix Encoders: create an Encoder for every manipulated type to avoid Spark fallback to genericRowWithSchema and cast to allow having Encoders with generic types such as WindowedValue<T> and get the type checking back
discard c5e4407 [BEAM-8470] Fail in case of having SideInouts or State/Timers
discard 2a1ea36 [BEAM-8470] Add ComplexSourceTest
discard 268ac53 [BEAM-8470] Remove no more needed putDatasetRaw
discard b2b742b [BEAM-8470] Port latest changes of ReadSourceTranslatorBatch to ReadSourceTranslatorStreaming
discard bed3340 [BEAM-8470] Fix type checking with Encoder of WindowedValue<T>
discard 6d5d382 [BEAM-8470] Add comments and TODO to GroupByKeyTranslatorBatch
discard eb3e6c3 [BEAM-8470] Add GroupByKeyTest
discard 8642beb [BEAM-8470] Clean
discard 105ec04 [BEAM-8470] Address minor review notes
discard ec584e8 [BEAM-8470] Add ParDoTest
discard e15e6cd [BEAM-8470] Clean
discard 54d015a [BEAM-8470] Fix split bug
discard 0913635 [BEAM-8470] Remove bundleSize parameter and always use spark default parallelism
discard ab0ff96 [BEAM-8470] Cleaning
discard 99b4c4a [BEAM-8470] Fix testMode output to comply with new binary schema
discard 0c9ff45 [BEAM-8470] Fix errorprone
discard eab170c [BEAM-8470] Comment schema choices
discard f30982a [BEAM-8470] Serialize windowedValue to byte[] in source to be able to specify a binary dataset schema and deserialize windowedValue from Row to get a dataset<WindowedValue>
discard 5219d9b [BEAM-8470] First attempt for ParDo primitive implementation
discard 4f808c1 [BEAM-8470] Add flatten test
discard 0a15995 [BEAM-8470] Enable gradle build scan
discard 8202fa1 [BEAM-8470] Enable test mode
discard 5920917 [BEAM-8470] Put all transform translators Serializable
discard 231285c [BEAM-8470] Simplify beam reader creation as it created once the source as already been partitioned
discard 8948cf3 [BEAM-8470] Fix SourceTest
discard fc21488 [BEAM-8470] Move SourceTest to same package as tested class
discard bde9f8b [BEAM-8470] Add serialization test
discard 1f35595 [BEAM-8470] Add SerializationDebugger
discard e260def [BEAM-8470] Fix serialization issues
discard 8416e7b [BEAM-8470] Clean unneeded fields in DatasetReader
discard 085b516 [BEAM-8470] improve readability of options passing to the source
discard 475ed56 [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset
discard abc4600 [BEAM-8470] Refactor SourceTest to a UTest instaed of a main
discard 383c0a6 [BEAM-8470] Checkstyle and Findbugs
discard 0f2cfb6 [BEAM-8470] Clean
discard e9fe234 [BEAM-8470] Add empty 0-arg constructor for mock source
discard 749668a [BEAM-8470] Add a dummy schema for reader
discard fed2cfd [BEAM-8470] Apply spotless and fix checkstyle
discard 3120789 [BEAM-8470] Use new PipelineOptionsSerializationUtils
discard 1ec8213 [BEAM-8470] Add missing 0-arg public constructor
discard c662ee6 [BEAM-8470] Wire real SourceTransform and not mock and update the test
discard 1ded190 [BEAM-8470] Refactor DatasetSource fields
discard 0d4e7e5 [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
discard 412e5b8 [BEAM-8470] Move Source and translator mocks to a mock package.
discard 2742c60 [BEAM-8470] Add ReadSourceTranslatorStreaming
discard 71a5d91 [BEAM-8470] Clean
discard 788f960 [BEAM-8470] Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
discard 15ae8ba [BEAM-8470] Split batch and streaming sources and translators
discard f2c3e98 [BEAM-8470] Run pipeline in batch mode or in streaming mode
discard 3ba8cf2 [BEAM-8470] Move DatasetSourceMock to proper batch mode
discard ec0a390 [BEAM-8470] clean deps
discard e14ea16 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary)
discard a0a072e [BEAM-8470] fix mock, wire mock in translators and create a main test.
discard 99f1d37 [BEAM-8470] Add source mocks
discard 4d920fd [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table
discard a4f4c27 [BEAM-8470] Improve type enforcement in ReadSourceTranslator
discard b91cd39 [BEAM-8470] Improve exception flow
discard dd62b16 [BEAM-8470] start source instanciation
discard b1e5eab [BEAM-8470] Apply spotless
discard 7d5421a [BEAM-8470] update TODO
discard 4b8bae4 [BEAM-8470] Implement read transform
discard 2996fc0 [BEAM-8470] Use Iterators.transform() to return Iterable
discard 91388b1 [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation
discard efbcb27 [BEAM-8470] Add Flatten transformation translator
discard 0e0a754 [BEAM-8470] Create Datasets manipulation methods
discard 345b963 [BEAM-8470] Create PCollections manipulation methods
discard c6dc3ca [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
discard d8bbeeb [BEAM-8470] Added SparkRunnerRegistrar
discard fe5afe5 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN
discard 82001d4 [BEAM-8470] Post-pone batch qualifier in all classes names for readability
discard 3cc6a3e [BEAM-8470] Add TODOs
discard 580ccca [BEAM-8470] Make codestyle and firebug happy
discard 3bf9f4d [BEAM-8470] apply spotless
discard 2f5aa47 [BEAM-8470] Move common translation context components to superclass
discard c7250cd [BEAM-8470] Move SparkTransformOverrides to correct package
discard 357e77a [BEAM-8470] Improve javadocs
discard 7bb6a3e [BEAM-8470] Make transform translation clearer: renaming, comments
discard 5b1c2af [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
discard 9b367bd [BEAM-8470] Initialise BatchTranslationContext
discard 0471b35 [BEAM-8470] Organise methods in PipelineTranslator
discard 27d1e23 [BEAM-8470] Renames: better differenciate pipeline translator for transform translator
discard 003b239 [BEAM-8470] Wire node translators with pipeline translator
discard be3ff45 [BEAM-8470] Add nodes translators structure
discard 3c16dc5 [BEAM-8470] Add global pipeline translation structure
discard 6f6e650 [BEAM-8470] Start pipeline translation
discard 3c24550 [BEAM-8470] Add SparkPipelineOptions
discard fc882e8 [BEAM-8470] Fix missing dep
discard 263abfe [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0
add addf288 Add note for the "Built-in IOs" tasks to inform that there is no kata to do
add ed4f82c Merge pull request #9905 from henryken/built-in-ios-task-note Add no kata note for the "Built-in IOs" tasks
add f61b380 Change from study_project.xml to yaml files for Beam Katas Python
add b58bbff Exclude Katas remote-info.yaml files from RAT check
add 6007c8c Merge pull request #9902 from henryken/katas-python-course-yaml
add 0fcdbe1 [BEAM-8432] Move javaVersion to gradle.properties (#9830)
add aeb7153 Enables Dataflow workers to use private IPs
add eb7edf3 Update the Beam Katas Java to use the new course metadata yaml format (#9901)
add d96c6b2 BEAM-3658 Port SpannerIOReadTest off DoFnTester (#9898)
add 1745a17 Increasing timeout due to still constant timeouts.
add f6adc23 Merge pull request #9908 Increasing timeout due to still constant timeouts.
add f89c1b9 [BEAM-8471] Flink native job submission for portable pipelines
add 0645404 Merge pull request #9872: [BEAM-8471] Flink native job submission for portable pipelines
add fb93d18 Move use_sdf_bounded_source tests to iobase.
add 1ceabf0 Merge pull request #9893 Move use_sdf_bounded_source tests to iobase.
add 6b6dd20 [BEAM-8471] Put back FlinkJobServerDriver.fromParams to fix post commit
add cce7a3b [BEAM-8442] Remove duplicate code for bundle register in Python SDK harness
add cdfedf0 Merge pull request #9842: [BEAM-8442] Remove duplicate code for bundle register in Python SDK harness
add 70caac8 [BEAM-8383] Fix gauge metrics in Python SDK and Flink runner
add 175a2bc [BEAM-8383] Add metrics for the Python state cache
add 7811838 Merge pull request #9769: [BEAM-8383] Add metrics for the Python state cache
add 8ee8051 [BEAM-8509] TestPortableRunner uses JobServerDriver interface
add 7047a05 Merge pull request #9914: [BEAM-8509] TestPortableRunner uses JobServerDriver interface
add caa881e [BEAM-6303] Add withSuffix(".parquet") to Parquet write javadoc.
add ff36841 Merge pull request #9916: [BEAM-6303] Add withSuffix(".parquet") to Parquet write javadoc
add 8e0feaf [BEAM-7730] Add Flink 1.9 to the list of supported Flink versions
add 79ba545 Merge pull request #9917: [BEAM-7730] Add Flink 1.9 to the list of supported Flink versions
add 749ee57 [BEAM-8269] Prefer class decorators over from_callable
add bd09a03 Merge pull request #9882 [BEAM-8269] Prefer class decorators over from_callable
add 0bb4343 [BEAM-4132] Support multi-output type inference
add 51ada98 Add assertions on pcoll types
add 860e3dc Don't try to infer recursively
add 9ffa665 Merge pull request #9810 [BEAM-4132] Default type hints for multi-output transforms
add 45a91a4 Updated PushDown rule and TestTableProvider to support predicate PushDown
add 4ba4909 Updated cost model to favor IOs with filters pushed-down, added more supported filters to TestTable
add 0c8a4c5 Added more tests, bug fixing
add e7b0284 Addressed review comments
add 254f5c7 Fixed style and readability issues
add c8bed1c Merge pull request #9863: [BEAM-8468] Predicate push down for in memory table
add aa47745 [BEAM-8517] Fix timezone conversion issue due to usage of different calls on DateTime for getting values. (#9929)
add b26e821 [BEAM-8515, BEAM-2699] Add equals/hashCode to ValueProvider classes.
add 6a900d2 [BEAM-7230] Cache DataSource instances based upon equivalent DataSourceConfiguration instances.
add b02bc8a Merge pull request #9927 from lukecwik/beam7230
add 5c3cf5f Fail when pipeline options conversion fails due to incompatible jackson-core version
add 1deab4c Merge pull request #9931: [BEAM-8518] Fail when pipeline options conversion fails due to incompatible jackson-core version
add 174b218 [BEAM-8398] Upgrade Google Cloud Java Client APIs
add ceb9a99 fix PubsubJsonClientTest NPE
add d664592 [BEAM-8398] Upgrade Dataflow Java Client API
add 7b3a3fa [BEAM-8507] Augment 'flink_master' pipeline option for Python FlinkRunner (#9918)
add 8698862 Bump Python container versions
add bb4f85a Merge pull request #9933 Update Dataflow container images to beam-master-20191029
add 1f63a9c BigQuery should return row fields in the selected order
add a23d704 Merge pull request #9864: [BEAM-8428] [SQL] BigQuery should return row fields in the selected order
add c67e7de Cleanup metrics tests.
add 3226fd2 Merge pull request #9928 Cleanup metrics tests.
add 1ccecd3 Remove --flink-master-url alias.
add 054fe9c [BEAM-4775] Extract code responsible for parsing MonitoringInfo
add 34407f6 [BEAM-4775] Expose metrics gathered by FnApiRunner in LocalJobServicer
add f1944c0 [BEAM-4775] Publish only user metrics in load tests
add f53e47a [BEAM-4775] Use result property of MetricResult in load tests
add 7a02887 Merge pull request #9843 from kamilwu/portable-runners-metrics
add cb5a467 [Go SDK] Delete spammy log in datamgr.go
add 5fa2c7f [Go SDK] Delete spammy log in datamgr.go
add dc6b7a3 Avoid polluting Stackdriver logs with noise during autoscaling events. Add message directing users to ignore them when appropriate.
add db87ebc Spotless Java applied
add f196d8e Merge pull request #9821 from nahuellofeudo/logs-cleanup
add 474dcd9 [BEAM-8474] A microbenchmark for Python FnApiRunner: (#9891)
add d85f64b Revert "Remove --flink-master-url alias."
add 18b63e4 Merge pull request #9947: Revert "Remove --flink-master-url alias."
add eb05f56 Not inject pubsub into Impulse when in fnapi streaming mode.
add 0573e10 Merge pull request #9932 from boyuanzz/windmill_create
add 04a70dd [BEAM-8306] estimate byte size by product count
add 62fc963 Merge pull request #9940: [BEAM-8306] estimate byte size by product count in ElasticsearchIO
add e98e31b Remove reference to --flink-master-url.
add d91b270 Remove occurrences of flink-master-url and add test for backwards-compatibility
add b325bd7 Merge pull request #9946: Remove reference to --flink-master-url
add 978a2ed [BEAM-8396] Restore Flink runner LOOPBACK default.
add b10a8db Merge pull request #9945: [BEAM-8396] Use LOOPBACK in FlinkRunner if no flink_master is supplied
add 8a9b034 Explicitly disable runner_api path in test_remote_runners_display_data.
add 49e3f6e Extract helper classes to module level.
add 2c282e7 Simplify test_remote_runners_display_data assertion logic.
add e5ea42e fixup: remove sorting of expected_data
add 75e64ce Merge pull request #9881 [BEAM-8397] Fix infinite recursion errors in test_remote_runner_display_data_test on Python 3.7.
add a4abe8b [BEAM-8434] Translate trigger transcripts into validates runner tests. (#9832)
add add58f8 [BEAM-8530] disable UsesStrictEventTimeOrdering for portable dataflow
add cb15994 Merge pull request #9951: [BEAM-8530] disable UsesStrictEventTimeOrdering for portable dataflow
add 791c1b6 [BEAM-8252] Python: add worker_region and worker_zone options
add 8370811 Merge pull request #9594 from ibzib/py-worker-region-zone
add 6d5ca75 [BEAM-8472] get default GCP region option (Python)
add 5030899 Merge pull request #9868 from ibzib/default-region
add af3d805 [BEAM-8451] Fix interactive beam max recursion err (#9865)
add b82c0e1 [BEAM-8514] ZetaSql should use cost based optimization (#9874)
add 9a45f96 [BEAM-7434] [BEAM-5895] and [BEAM-5894] Upgrade to rabbit amqp-client 5.x (#9900)
add 4865ec8 [BEAM-8533] Revert "[BEAM-8442] Remove duplicate code for bundle register in Python SDK harness"
add c322f76 Merge pull request #9956: [BEAM-8533] Revert "[BEAM-8442] Remove duplicate code for bundle register in Python SDK harness
add a80ae25 [BEAM-8540] Fix CSVSink example in FileIO docs
add d7b2c0e Merge pull request #9964 from jklukas/csvsink-fix
add 90abf5d [BEAM-8513] Allow reads from exchange-bound queue without declaring the exchange (#9937)
add 9a41ce0 [BEAM-8521] only build Python 2.7 for xlang test
add 980e655 Merge pull request #9939 from ibzib/xlang-version
add c870c63 [BEAM-7917] Fix datastore writes failing on retry (#9294)
add dfc230c Merge pull request #9741 from [BEAM-7926] Visualize PCollection
add bebbba8 [BEAM-7389] Add code snippets for CombinePerKey (#9921)
add 9e38cbe [BEAM-7389] Add code snippets for Distinct
add 9d2d9df Merge pull request #9924 from davidcavazos/distinct-code
add 9b30d7c [BEAM-8146,BEAM-8204,BEAM-8205] Add equals and hashCode to SchemaCoder and RowCoder (#9493)
add 56d17fb Add separate testSQLRead and testSQLWrite test in BigQueryIT
add d093625 Add PubsubJsonIT tests that pull and verify pubsub messages
add 107ac28 Add TestPubsub#{waitForNMessages, assertThatTopicEventuallyReceives}
add ed8401d Merge pull request #9880: [BEAM-8503] Improve TestBigQuery and TestPubsub
add 1cb675a Add Jira priority explanations to Beam site
add 2564cb9 Add release blocking section to Beam site
add 902dddd Merge pull request #9862: Describe Jira priorities and release blockers on Beam site
add d651434 [BEAM-8422] Notify builds@ on ABORTED job final state
add 3d865bc Merge pull request #9816: [BEAM-8422] Notify builds@ on ABORTED job final state
add 3626751 [BEAM-8491] Add ability for replacing transforms with multiple outputs (#9912)
add 1664ce2 [BEAM-8467] Enabling reading compressed files (#9861)
add a6936c8 [BEAM-3713] Add pytest for unit tests (#9756)
add 55b6d05 [BEAM-8526] Turn on metrics publishing in load tests on Flink
add e9087a0 Merge pull request #9948: [BEAM-8526] Turn on metrics publishing in load tests on Flink
add 101165a update trigger phrase for beam_PreCommit_Website_Stage_GCS in README
add 711b7aa Merge pull request #9983: [BEAM-8500] update trigger phrase for beam_PreCommit_Website_Stage_GCS in README
add 0ad6611 [BEAM-8516] only consider current version in sdist build
add 664de39 Merge pull request #9930: [BEAM-8516] only consider current version in sdist build
add 10a570f Align base image requirements with Beam's setup.py.
add ad7ae0d Merge pull request #9967 Align base image requirements with Beam's setup.py.
add 81af13c [BEAM-8544] Use ccache for compiling the Beam Python SDK.
add 660c618 Additional logging.
add c421484 Merge pull request #9966 [BEAM-8544] Use ccache for compiling the Beam Python SDK.
add 4b107d2 [BEAM-8532] Output timestamp should be the inclusive, not exclusive, end of window.
add d024a4e Merge pull request #9958 [BEAM-8532] Output timestamp inclusive, not exclusive, end of window.
add 9ed9dc7 [BEAM-8368] [BEAM-8392] Update pyarrow to the latest version 0.15.1 (#9970)
add 640ced8 Remove base tests from python precommits
add 187d71e Merge pull request #9985 from aaltay/prec
add 8f3818e [BEAM-8254] add workerRegion and workerZone options to Java SDK
add e7f1539 Merge pull request #9961 from ibzib/java-worker
add e29a0e9 [BEAM-8555] Update Python streaming limitations.
add a4141ac Merge pull request #9989 [BEAM-8555] Update Python streaming limitations.
add 1e70f1c Fix a bug related to zero-row responses
add 9233aa4 Merge pull request #9987: [BEAM-8504] Fix a bug related to zero-row responses
add 535e5fb [BEAM-8432] Enable building kotlin examples with different java versions
add d155893 Merge pull request #9984: [BEAM-8432] Enable building kotlin examples with different java versions
add c81b5f4 [BEAM-8549] Do not use keyed operator state for checkpoint buffering
add 9f56639 Merge pull request #9980 from mxm/BEAM-8549
add 7ffb218 [BEAM-7303] Rename runners.reference package to runners.portability
add 7120c2a Merge pull request #9936: [BEAM-7303] Move PortableRunner from runners.reference to runners.portability package
add d282879 [BEAM-4420] Make it possible for kafka test k8s setup to pass namespace as a parameter
add 1ece728 [BEAM-4420] Refactor multiple things:
add afa759a [BEAM-4420] Create jenkins job for Kafka IOIT
add f0b1e1e [BEAM-4420] Extract common Jenkins code
add 7f52e45 [BEAM-4420] Fix cancelling timeouting Dataflow jobs
add 315566a Merge pull request #9871: [BEAM-4420] Kafka ioit jenkins job
add 45bf690 Use better name for Flink current key supplier
add fdd3570 Merge pull request #9982: Use better name for Flink current key supplier
add 1c56a76 [BEAM-8558] Fix BigQueryIOIT flakiness (#9995)
add fc142c2 [BEAM-8435] Implement PaneInfo computation for Python.
add 16fbe89 Merge pull request #9836 [BEAM-8435] Implement PaneInfo computation for Python.
add b09b1b6 [BEAM-8539] Provide an initial definition of all job states and the state transition diagram
add ddd115a Merge pull request #9969 from apache/beam8539
add b49054d Created a MongoDbTable and a provider for it
add c24dcb5 Updated assertion format string to use [] vs ?
add 326ac36 Merge pull request #9806: [BEAM-8427] Create a table and a table provider for MongoDB
add 81ba76d Fix expected step name in Python PTransformTest test_read_metrics
add 295d9f8 generalized step name matching
add 78ce6e0 Merge branch 'master' into metrics
add c39752a Merge pull request #9941 Fix expected step name in Python PTransformTest test_read_metrics
add 7f90f2b Removing some trailing whitespace.
add 1bff5ae Merge pull request #10001 from youngoli/patch-9
add 22bd9b1 [Go SDK] Correctly return EOFs from boolDecoder
add 5926ade [BEAM-8565] Update .test-infra/jenkins/README with missing entries and correct wrong entries
add 9ad416e Merge pull request #10006: [BEAM-8565] Update .test-infra/jenkins/README to reflect the current state
add 432cfd2 [BEAM-8352] Add "withMaxCapacityPerShard()" to KinesisIO.Read
add fae96ed Merge pull request #9745: [BEAM-8352] Add "withMaxCapacityPerShard()" to KinesisIO.Read
add b3cb103 [BEAM-8450] Allow empty bundles in ParDoLifecycleTest
add 49ef06e Merge pull request #9848 from je-ik/BEAM-8450: [BEAM-8450] Allow empty bundles in ParDoLifecycleTest
add fd51123 Revert "[BEAM-8427] Create a table and a table provider for MongoDB"
add 39cbc87 Merge pull request #10018: [BEAM-8574] Revert "[BEAM-8427] Create a table and a table provider for MongoDB"
add f80b50d Add beam:coder:row:v1 to standard coders
add 05ff238 Java: Implement RowCoder fn api translation
add de196e0 Java: Add suport for beam:coder:row:v1 in CommonCoderTest
add 83322b9 Python: Add conversions between native types and protobuf schema representation
add 38459c0 Python: Add RowCoder
add bbf46c2 Python: Add support for beam:coder:row:v1 in standard_coders_test.py
add f7ce06d Add simple beam:coder:row:v1 test to standard_coders.yaml
add 01726e9 Merge pull request #9188: [BEAM-7886] Make row coder a standard coder and implement in Python
add 9bceeb8 Adjust last-updated date for the Jenkins README
add bfc65dd [BEAM-8402] Create a class hierarchy to represent environments
add 076a037 Merge pull request #9811: [BEAM-8402] Create a class hierarchy to represent Python environments
add 4caefa6 [BEAM-8568] Fixed problem that LocalFileSystem no longer supports wildcard relative paths.
add eb9dea3 Merge pull request #10024: [BEAM-8568] Fixed problem that LocalFileSystem no longer supports wil…
add 96fa3da [BEAM-8347]: Consistently advance UnboundedRabbitMqReader watermark (#9820)
add fd06b7e Fix generic comparisons on protobuf messages
add 8327160 Update DoFn.SplitRestriction signature
add df34941 Clarify that SplitRestriction outputs Restrictions
add ec4fd27 [BEAM-8573] Updates documented signature for @SplitRestriction
add 5ff7428 Filter push-down should not be reliant on project push-down [BEAM-8508]
add d9e4134 Fixed a bug with selecting the same field more than once. Changed a cost model for BeamCalcRel.
add 599bd21 Selecting fields in a different order should not drop the Calc when project push-down is not supported
add 08381fd Refactoring
add e79bcf8 spotlesApply
add 987148a IOPushDownRule should not be applied more than once to the same IORels
add a48662d Merge pull request #9943: [BEAM-8508] [SQL] Standalone filter push down
add 692da20 [BEAM-8294] run Spark portable validates runner tests in parallel
add 497e576 Merge pull request #9999 from ibzib/spark-parallel-test
add b02b856 Bump Dataflow python container versions
add 3943893 Merge pull request #10019 from TheNeuralBit/patch-2
add 3139b43 [BEAM-8570] Use SDK version in default Java container tag
add 8d2f371 Use default Java container image if none is specified
add a9ef293 flinkCompatibilityMatrix depends on Java Docker containers
add 8a1e24c Merge pull request #10017 from ibzib/java-container-version
add 90d5878 Increase overhaed budget for test_sampler_transition_overhead (#10012)
add 54e6108 Add support for exclusion in Jenkins DSL
add 72eb8c7 Add separate SQL precommit
add 5bfe1b6 Exclude SQL from Java precommit
add ccc98f0 Merge pull request #10022: Isolate sql precommit
add 66a67e6 Add portable schemas to the roadmap
add 2deb788 [BEAM-8566] Fix checkpoint buffering when another bundle is started during checkpointing
add 87488c4 Merge pull request #10007: [BEAM-8566] Fix checkpoint buffering when another bundle is started during checkpointing
add 5a23914 [BEAM-8567] Do not swallow execution errors during checkpointing
add 119f41d2 Merge pull request #10008: [BEAM-8567] Do not swallow execution errors during checkpointing
add d8e94bb [BEAM-5878] update dill min version to 0.3.1.1 and add test for functions with Keyword-only arguments (#9686)
add 329a8ce Simplify and improve performance of metrics.
add e74f00a Remove unused metrics supported global.
add 037f219 Bump Dataflow python container version
add 5dbaf66 Merge pull request #9996 Simplify and clean up metrics
add ad3fdf2 [BEAM-2879] Support writing data to BigQuery via avro
add 35da90a Merge pull request #9665: [BEAM-2879] Support writing data to BigQuery via avro
add d692d2f [BEAM-8512] Add integration tests for flink_runner.py
add 40383d8 Merge pull request #9998 from ibzib/flink-runner-test
add 8e49c5a [BEAM-8472] Get default GCP region from gcloud (Java)
add df0e7e8 Merge pull request #9974 from ibzib/java-default-region
add 2e5f89a Javadoc that the sdk.util package is internal
add 4364a21 Javadoc that the sdk.util package is internal
add da44a4e Fix typo in pardo.go documentation
add 82359a6 Merge pull request #9950 from pgudlani/patch-1
add 1e8c7a0 fix section 5.3.1 Java code example
add 18de5fa Merge pull request #9781 from shusso/prog-guide-531-fix-example-code
add 5ce20cb Bump pyyaml from 3.13 to 5.1 in /sdks/python/container
add 1d81898 Merge pull request #9963 from apache/dependabot/pip/sdks/python/container/pyyaml-5.1
add 4af507f Always use shadow configuration for direct runner dep
add f481cd4 [BEAM-8602] Always use shadow configuration for direct runner dependencies
add 894e940 [BEAM-3288] Guard against unsafe triggers at construction time
add de0b4fb Merge pull request #9960: [BEAM-3288] Guard against unsafe triggers at construction time
add 27c4190 Updates cross-language transforms roadmap
add ede14d4 Merge pull request #10054: [BEAM-8553] Updates cross-language transforms roadmap
add 81aab39 [BEAM-8585] Include path in error message in path_to_beam_jar
add 53c6212 Merge pull request #10032: [BEAM-8585] Include path in error message in path_to_beam_jar
add 6868333 [BEAM-8442] Update the client side implemetation of Fn API to make sure register is executed sucessfully before executing process_bundle
add 79d21b1 [BEAM-8442] Remove duplicate code for bundle register in Python SDK harness
add 0178652 Merge pull request #10004: [BEAM-8442] Unify bundle register in Python SDK harness
add 7937317 Merge pull request #9885 from [BEAM-8457] Label Dataflow jobs from Notebook
add 9c76dbf Add cross-language efforts to roadmap index page
add 1baad3b Option to skip display operations in beam's InteractiveRunner. This helps significantly when running large pipelines that do not need display (ie, that are not run on Jupyter/Colab).
add 0a75cb6 Merge pull request #10015 from andrefaraujo/master Option to skip display operations in beam's InteractiveRunner.
add 15b3db2 [BEAM-8472] test Java default GCP region
add c7ffbde [BEAM-8472] test Java default GCP region
add eaeeb6a Update BigQueryHllSketchCompatibilityIT to cover empty sketch cases
add 47d5ee9 Merge pull request #9778 from robinyqiu/hll
add a1d8d21 Update pipeline_graph.py
add 7124651 [BEAM-8605] Function display_graph() in example do not exist
add ad84e55 Still use fake pubsub signals when using windmill appliance with dataflow streaming.
add 60845e5 [BEAM-8524] Still use fake pubsub signals when using windmill appliance
add 043fecd Add HllCount to Java transform catalog
add 063ee99 Merge pull request #9793 from robinyqiu/tmp
add 8fb53bf Created a MongoDbTable and a provider for it
add 36d6534 Merge pull request #10031: [BEAM-8427] [SQL] Create a table and a table provider for MongoDB
add 3535958 Adds TestStream support for Dataflow runner.
add f6ce376 Marks run_TestStream as not a unittest, since it was trying to be 'run' when running other tests. Moves bad copy of internal/names.py to dataflow/internal/names.py, where it was supposed to go.
add be08efd Fixes line-to-long lint error.
add f31a0df Replaces unittest annotation with setting __test__ = False to prevent nosetest from trying to run the method.
add b951993 [BEAM-8587] TestStream for Dataflow runner
add 42da6ae BEAM-8579: Strip UTF-8 BOM from TextSource output.
add f95de04 [BEAM-8579] Strip UTF-8 BOM from TextSource output.
add d619ea1 Print pipeline proto and pipeline options in DEBUG loglevel.
add c7be064 Merge pull request #10036 [BEAM-8589] Print pipeline proto and pipeline options in DEBUG loglevel.
add c32842b [BEAM-5600, BEAM-2939] Add SplittableParDo expansion logic to runner's core.
add 45152b8 fixup! Lint
add 91fd0df fixup! Fix checkstyle & spotbugs
add e852ac8 [BEAM-5600, BEAM-2939] Add SplittableParDo expansion logic to runner's core.
add 090b510 [BEAM-8539] Make job state transitions in python-based runners consistent with java-based runners (#9965)
add 8399a12 [BEAM-8575] Add a trigger test to test Discarding accumulation mode with early data
add 0d4f3de Merge pull request #10033 [BEAM-8575] Discarding mode with early data trigger test.
add ef7ec62 [BEAM-2939, BEAM-6868, BEAM-7233] Expose residual roots and bundle finalization callbacks to portable runners.
add 212fb09 [BEAM-2939, BEAM-5600, BEAM-6868, BEAM-7233] Expose residual roots and bundle finalization callbacks to portable runners.
add a2349ea Bump jekyll from 3.2.0 to 3.6.3 in /website
add dc1fee8 Merge pull request #9973: Bump jekyll from 3.2.0 to 3.6.3 for building website
add 82132a6 [BEAM-8028] add spark_runner.py
add dc413c1 [BEAM-7224] add py wordcount tests for Spark
add c2e58c5 Merge pull request #10047: [BEAM-8028] Add spark_runner.py
add 5aaaad1 [BEAM-8594] Remove unnecessary error check in DataFlow Runner
add f038020 fixup
add 3265605 fix codestyle
add b834e53 [BEAM-8594] Remove unnecessary error check in DataFlow Runner
add fc77c31 Merge pull request #10044 6x speedup for metrics.
add 0b210c9 [BEAM-8598] Test triggering BEAM-8598 on FlinkRunner. (#10049)
add 385bba2 Implement portable representation of TestStream.
add 0ff4904 [BEAM-8597] Allow TestStream trigger tests to run on other runners.
add d02e42f Work around for BEAM-8598.
add 3f7625a Merge pull request #10043 [BEAM-8597] Allow TestStream trigger tests to run on other runners.
add 19303ab [BEAM-8157] Fail if key group for state request does not match key group range
add ee28c6e [BEAM-8157] Add LengthPrefixCoder around unknown key coder for stateful ProcessBundleDescriptor
add a8b8f64 [BEAM-8157] Add integration test for non standard key coders
add add1927 Merge pull request #9997: [BEAM-8157] Fix key encoding issues for state requests with unknown coders
add 14b7934 [BEAM-8621] Fix dependency checking job.
add 808cb35 Merge pull request #10079 [BEAM-8621] Fix dependency checking job.
add 80e8fb5 Merge pull request #10072: [BEAM-8616] Make hadoop-client a provided dependency on ParquetIO
add b94dca2 Update SDF APIs
add 352ecf3 Merge pull request #9794 from boyuanzz/watermark
add 126e5c2 [BEAM-8628] use mock GcsUtil in testDefaultGcpTempLocationDoesNotExist
add c3fd801 Merge pull request #10076 from ibzib/java-default-region
add 9783072 [BEAM-8622] Exclude UsesStrictTimerOrdering PVR tests for Spark
add da9efcd Merge pull request #10083 from mxm/BEAM-8622
add 47eac13 Remove Interactive Test Suite (#10068)
add de2cfae Beam changes to bypass recording timing for first N times seeing a new batch size. (#10063)
add c5e3253 [BEAM-8575] Windows idempotency: Applying the same window fn (or wind… (#10071)
add 54762d9 Revert "Merge pull request #10072: [BEAM-8616] Make hadoop-client a provided dependency on ParquetIO"
add 7f8b1af Merge pull request #10087: Revert "Merge pull request #10072: [BEAM-8616] Make hadoop-client a p…
add 4f48690 Update container image tags used by Dataflow runner for Beam master
add 802e7cd Merge pull request #10090 Update container image tags used by Dataflow runner for Beam master
add a093994 Created a MongoDbTable and a provider for it
add b838375 [SQL] Implemented write functionality for MongoDbTable, updated conversion logic for RowJsonSerializer
add 57d47dd spotlesApply
add f0cf1b3 ToJson should support logical types
add 221f08d Added RowJsonTest for lofical types
add a27acf3 Merge pull request #9892: [BEAM-8427] [SQL] buildIOWrite for MongoDb Table
add 7b482e4 [BEAM-7233, BEAM-6868] Create an implementation for a bundle finalizer that stores bundle finalization requests in memory. (#10074)
add ba8d4b6 [BEAM-8157] Follow-up to adjust comment and improve ProcessBundleDescriptorsTest
add c049586 Merge pull request #10085: [BEAM-8157] Adjust comment and improve ProcessBundleDescriptorsTest
add 3160584 Make key type parameter in Flink's state request handler explicit
add f96b109 Merge pull request #10026: Make key type parameter in Flink's state request handler explicit
add d0274dc [BEAM-7078] Update amazon-kinesis-client to 1.13.0
add bff9a1a Merge pull request #10086: [BEAM-7078] Update amazon-kinesis-client to 1.13.0
add ee3f951 [BEAM-3419] Explicitly model iterable side inputs as part of the Beam portability APIs. (#10092)
add 41b6716 [BEAM-8649] Eat stderr from subprocess in _get_default_gcp_region
add 15e80d2 Can't use subprocess.DEVNULL in py2
add 29cb612 Merge pull request #10088 from ibzib/process-logs
add 5d91fbf Do not override test_metrics in PortableRunner test cases.
add 55b4839 Merge pull request #10104 Do not override test_metrics in PortableRunner test cases.
add 0b78d91 [BEAM-8612] Convert []beam.T to the underlying type []T when passed to a universal type. (#10066)
add e9f766f [BEAM-8151, BEAM-7848] Add an UnboundedThreadPoolExecutor implementation.
add 1b62310 [BEAM-8151, BEAM-7848] Swap to using a thread pool which is unbounded and shrinks when threads are idle.
add e265826 Merge pull request #9477 from lukecwik/python
add 529b085 [BEAM-8598] Exclude UsesTestStreamWithMultipleStages also from Flink PVR tests
add 07077f1 Merge pull request #10103: [BEAM-8598] Exclude UsesTestStreamWithMultipleStages from Flink PVR tests
add 846369d Avoid Wimplicitly-unsigned-literal caused by -2**63.
add edf01c4 Merge pull request #10093 Avoid Wimplicitly-unsigned-literal caused by -2**63.
add 312535e [BEAM-8575] Fix window assignment idempotency tests non-deterministic elements order
add 9009dd1 Merge pull request #10111 from liumomo315/fix_window_idempotency
add e1c0970 [BEAM-8666] Remove dependency between DataflowRunner and PortableRunner introduced by PR#9811
add dd605e5 Merge pull request #10112: [BEAM-8666] Remove dependency between DataflowRunner and PortableRunner introduced by PR#9811
add 5ff4f95 [BEAM-8660] Override returned artifact staging endpoint
add 7588387 Merge pull request #10108 from ibzib/artifact-endpoint
add a1e0daa [BEAM-8661] Moving io module to have per-module logger
add 38d0f5d Fix lint
add c9e2699 Removing extra space between constants
add 8bf9468 [BEAM-8661] Moving io module to have per-module logger
add fab9e3e [BEAM-8661] Moving other modules to have per-module loggers
add d4818ec Fix lint
add ccb6af2 Removing extra line between constants
add db11e06 [BEAM-8661] Moving other modules to have per-module loggers
add 308f4ca [BEAM-8667] Bound the number of element bundles buffered off the data channel.
add e35e6b8 [BEAM-8667] Bound elements buffered off the data channel.
add 8f86340 Cleanup: move direct runner test to correct location.
add 8c3af8a Merge pull request #10114 Cleanup: move direct runner test to correct location.
add 5e2ff14 [BEAM-8657] Avoid lifting combiners for incompatible triggers.
add 3deca14 Comment on compatibility.
add 3a1a27f Merge pull request #10098 [BEAM-8657] Avoid lifting combiners for incompatible triggers.
add 96957c6 [BEAM-8663] Respect PaneInfo in bundle based direct runner.
add b653c29 Merge pull request #10106 [BEAM-8663] Respect PaneInfo in bundle based direct runner.
add 310f289 Fixes a failing check.
add 2f00613 Merge pull request #10122: Fixes a failing check.
add d3f8192 [BEAM-8657] Combiner lifting fix for bundle-based direct runner.
add 4ee06ce Merge pull request #10109 [BEAM-8657] Combiner lifting fix for bundle-based direct runner.
add 5089c39 [BEAM-8554] Use WorkItemCommitRequest protobuf fields to signal that a WorkItem needs to be broken up
add 780ef7a Merge pull request #10013: [BEAM-8554] Use WorkItemCommitRequest protobuf fields to signal that …
add cea8d03 [BEAM-8655] Strengthen trigger transcript test to use multiple keys.
add 2efca75 [BEAM-8655] Run trigger transcript tests with combiner as well as GroupByKey.
add 63d8020 [BEAM-8655] Run the subset of trigger tests that make sense in batch mode.
add 2271fa0 Merge pull request #10096 [BEAM-8655] Strengthen trigger transcript tests.
add 3f7cf68 Add option to test metrics on runners without gauge support.
add 8f54d07 Merge pull request #10052 from robertwb/no-gauge
add 4013214 [BEAM-8557] Add log for the dropped unknown response
add 85248f3 [BEAM-8557] Add log for the dropped unknown response
add de9cd0b Bump python precommit timeout
add d16f2a9 Bump python precommit timeout
add 33cc30e [BEAM-8151] Swap to create SdkWorkers on demand when processing jobs
add 5de27d5 [BEAM-8151] Swap to create SdkWorkers on demand when processing jobs
add c430c27 Filter push-down for BigQuery (kind of) working.
add 6314eb2 Added IT test for BigQuery. spotlesApply.
add 14f3d58 review comments
add a2b0df9 Merge pull request #10030: [BEAM-8583] [SQL] Big query filter push down
add 8de55cc [BEAM-8379] Cache Eviction
add 1bb581f Merge pull request #10062 from [BEAM-8379] Cache Eviction for Interactive Beam
add 9a9f045 Reduce Java Examples Dataflow Precommit timeout
add 97093f4 Merge pull request #10131 from Ardagan/LowerExamplesTimeout
add bda8f62 [BEAM-8575] Test a customized window fn work as expected
add b41fe1a fixup
add 3358b37 fixup
add c6c8fbd Merge branch 'master' into addTest
add 76b9bdd [BEAM-8575] Test a customized window fn work as expected
add 3330e1e Separate pydocs generation from py2 precommit tests.
add 6bcf166 Add settings file
add ec3515a Merge pull request #10069 Separate pydocs generation from py2 precommit tests.
add 7beb9ee Merge pull request #10117 [BEAM-8335] Add service and tagged output capabilities to TestStream
add 45dc280 [BEAM-8670] Manage environment parallelism in DefaultJobBundleFactory
add 1386b94 Merge pull request #10124: [BEAM-8670] Manage environment parallelism in DefaultJobBundleFactory
add 49d6efd [BEAM-8661] Moving runners to have per-module logger (#10097)
add bb816f6 Fix sdk_worker_parallelism pipeline option type, add test
add 1bc180c Merge pull request #10129: Make sdk_worker_parallelism an integer
add b3c12d2 Merge pull request #10003: [BEAM-6756] Create Iterable type for Schema
add 6f72c93 [BEAM-8470] Add an empty spark-structured-streaming runner project targeting spark 2.4.0
add 4ca7e55 [BEAM-8470] Fix missing dep
add 0fc0d0a4 [BEAM-8470] Add SparkPipelineOptions
add e8ca23e [BEAM-8470] Start pipeline translation
add 00964d2 [BEAM-8470] Add global pipeline translation structure
add a3b278e [BEAM-8470] Add nodes translators structure
add ef4941a [BEAM-8470] Wire node translators with pipeline translator
add 8a8dc1e [BEAM-8470] Renames: better differenciate pipeline translator for transform translator
add cdfd589 [BEAM-8470] Organise methods in PipelineTranslator
add 38eca95 [BEAM-8470] Initialise BatchTranslationContext
add 80f2d8c [BEAM-8470] Refactoring: -move batch/streaming common translation visitor and utility methods to PipelineTranslator -rename batch dedicated classes to Batch* to differentiate with their streaming counterparts -Introduce TranslationContext for common batch/streaming components
add baf210f [BEAM-8470] Make transform translation clearer: renaming, comments
add b65a9da [BEAM-8470] Improve javadocs
add 0434749 [BEAM-8470] Move SparkTransformOverrides to correct package
add 4372c7e [BEAM-8470] Move common translation context components to superclass
add 49b666b [BEAM-8470] apply spotless
add 0d6906a [BEAM-8470] Make codestyle and firebug happy
add ef97440 [BEAM-8470] Add TODOs
add 9abf8ac [BEAM-8470] Post-pone batch qualifier in all classes names for readability
add 11a6e19 [BEAM-8470] Add precise TODO for multiple TransformTranslator per transform URN
add 47ed3d1 [BEAM-8470] Added SparkRunnerRegistrar
add 022a0d0 [BEAM-8470] Add basic pipeline execution. Refactor translatePipeline() to return the translationContext on which we can run startPipeline()
add 96b3f36 [BEAM-8470] Create PCollections manipulation methods
add b0c42af [BEAM-8470] Create Datasets manipulation methods
add 2c5cb23 [BEAM-8470] Add Flatten transformation translator
add 9f1bf60 [BEAM-8470] Add primitive GroupByKeyTranslatorBatch implementation
add 98ea9fb [BEAM-8470] Use Iterators.transform() to return Iterable
add 0b55323 [BEAM-8470] Implement read transform
add 4c91a57 [BEAM-8470] update TODO
add 4adf3bb [BEAM-8470] Apply spotless
add 6b4b916 [BEAM-8470] start source instanciation
add ff60578 [BEAM-8470] Improve exception flow
add 2ee98da [BEAM-8470] Improve type enforcement in ReadSourceTranslator
add fc3abf5 [BEAM-8470] Experiment over using spark Catalog to pass in Beam Source through spark Table
add 4746d9b [BEAM-8470] Add source mocks
add e45e48d [BEAM-8470] fix mock, wire mock in translators and create a main test.
add 7d7fe77 [BEAM-8470] Use raw WindowedValue so that spark Encoders could work (temporary)
add 9d84a0f [BEAM-8470] clean deps
add b4032aa [BEAM-8470] Move DatasetSourceMock to proper batch mode
add a7ad1ab [BEAM-8470] Run pipeline in batch mode or in streaming mode
add 141e4bc [BEAM-8470] Split batch and streaming sources and translators
add 8954c50 [BEAM-8470] Use raw Encoder<WindowedValue> also in regular ReadSourceTranslatorBatch
add 00ef268 [BEAM-8470] Clean
add c426c98 [BEAM-8470] Add ReadSourceTranslatorStreaming
add 1740dc4 [BEAM-8470] Move Source and translator mocks to a mock package.
add 7819918 [BEAM-8470] Pass Beam Source and PipelineOptions to the spark DataSource as serialized strings
add b10aa53 [BEAM-8470] Refactor DatasetSource fields
add c4bb08c [BEAM-8470] Wire real SourceTransform and not mock and update the test
add 5bbea63 [BEAM-8470] Add missing 0-arg public constructor
add 0dbe26f [BEAM-8470] Use new PipelineOptionsSerializationUtils
add 43052d3 [BEAM-8470] Apply spotless and fix checkstyle
add 17ca18b [BEAM-8470] Add a dummy schema for reader
add 2e8393b [BEAM-8470] Add empty 0-arg constructor for mock source
add 43ff919 [BEAM-8470] Clean
add c8ad727 [BEAM-8470] Checkstyle and Findbugs
add 6e3575d [BEAM-8470] Refactor SourceTest to a UTest instaed of a main
add c221aaa [BEAM-8470] Fix pipeline triggering: use a spark action instead of writing the dataset
add c26c421 [BEAM-8470] improve readability of options passing to the source
add ff69ded [BEAM-8470] Clean unneeded fields in DatasetReader
add 524667e [BEAM-8470] Fix serialization issues
add 638bdae [BEAM-8470] Add SerializationDebugger
add fd354fa [BEAM-8470] Add serialization test
add 163102b [BEAM-8470] Move SourceTest to same package as tested class
add 68fc6d5 [BEAM-8470] Fix SourceTest
add e4c76fc [BEAM-8470] Simplify beam reader creation as it created once the source as already been partitioned
add 90463a0 [BEAM-8470] Put all transform translators Serializable
add 940b484 [BEAM-8470] Enable test mode
add 0b156bf [BEAM-8470] Enable gradle build scan
add 74080c1 [BEAM-8470] Add flatten test
add 740131e [BEAM-8470] First attempt for ParDo primitive implementation
add 0cedcd7 [BEAM-8470] Serialize windowedValue to byte[] in source to be able to specify a binary dataset schema and deserialize windowedValue from Row to get a dataset<WindowedValue>
add 79b075a [BEAM-8470] Comment schema choices
add 5d1b2b5 [BEAM-8470] Fix errorprone
add 0529996 [BEAM-8470] Fix testMode output to comply with new binary schema
add 696597c [BEAM-8470] Cleaning
add 91964b9 [BEAM-8470] Remove bundleSize parameter and always use spark default parallelism
add 35051e0 [BEAM-8470] Fix split bug
add 6902b4b [BEAM-8470] Clean
add e5a36f0 [BEAM-8470] Add ParDoTest
add 057e0ac [BEAM-8470] Address minor review notes
add b706a74 [BEAM-8470] Clean
add 83779db [BEAM-8470] Add GroupByKeyTest
add 0330f32 [BEAM-8470] Add comments and TODO to GroupByKeyTranslatorBatch
add 1cb53e3 [BEAM-8470] Fix type checking with Encoder of WindowedValue<T>
add e363586 [BEAM-8470] Port latest changes of ReadSourceTranslatorBatch to ReadSourceTranslatorStreaming
add 9b9cca6 [BEAM-8470] Remove no more needed putDatasetRaw
add cb1a085 [BEAM-8470] Add ComplexSourceTest
add c13e8ae [BEAM-8470] Fail in case of having SideInouts or State/Timers
add 83e4067 [BEAM-8470] Fix Encoders: create an Encoder for every manipulated type to avoid Spark fallback to genericRowWithSchema and cast to allow having Encoders with generic types such as WindowedValue<T> and get the type checking back
add d19c8b9 [BEAM-8470] Apply spotless
add 1051eba [BEAM-8470] Fixed Javadoc error
add ee67f21 [BEAM-8470] Rename SparkSideInputReader class and rename pruneOutput() to pruneOutputFilteredByTag()
add 8e0f183 [BEAM-8470] Don't use deprecated sideInput.getWindowingStrategyInternal()
add f0ff5b4 [BEAM-8470] Simplify logic of ParDo translator
add 9c4e36d [BEAM-8470] Fix kryo issue in GBK translator with a workaround
add b3a740c [BEAM-8470] Rename SparkOutputManager for consistency
add 438f1e9 [BEAM-8470] Fix for test elements container in GroupByKeyTest
add be1cd23 [BEAM-8470] Added "testTwoPardoInRow"
add ed6b12c [BEAM-8470] Add a test for the most simple possible Combine
add 35f100c [BEAM-8470] Rename SparkDoFnFilterFunction to DoFnFilterFunction for consistency
add 8111241 [BEAM-8470] Generalize the use of SerializablePipelineOptions in place of (not serializable) PipelineOptions
add 2d0c33e [BEAM-8470] Fix getSideInputs
add 8662ef8 [BEAM-8470] Extract binary schema creation in a helper class
add 9a757d0 [BEAM-8470] First version of combinePerKey
add 3da0276 [BEAM-8470] Improve type checking of Tuple2 encoder
add ca39fe1 [BEAM-8470] Introduce WindowingHelpers (and helpers package) and use it in Pardo, GBK and CombinePerKey
add f002039 [BEAM-8470] Fix combiner using KV as input, use binary encoders in place of accumulatorEncoder and outputEncoder, use helpers, spotless
add cec975f [BEAM-8470] Add combinePerKey and CombineGlobally tests
add 39a978a [BEAM-8470] Introduce RowHelpers
add c3cd76f [BEAM-8470] Add CombineGlobally translation to avoid translating Combine.perKey as a composite transform based on Combine.PerKey (which uses low perf GBK)
add 9ee65d5 [BEAM-8470] Cleaning
add 279d2ca [BEAM-8470] Get back to classes in translators resolution because URNs cannot translate Combine.Globally
add 55bf503 [BEAM-8470] Fix various type checking issues in Combine.Globally
add d5dab93 [BEAM-8470] Update test with Long
add fd1d8c5 [BEAM-8470] Fix combine. For unknown reason GenericRowWithSchema is used as input of combine so extract its content to be able to proceed
add 5c074eb [BEAM-8470] Use more generic Row instead of GenericRowWithSchema
add 341c39f [BEAM-8470] Add explanation about receiving a Row as input in the combiner
add 89d38b1 [BEAM-8470] Fix encoder bug in combinePerkey
add 6b109d3 [BEAM-8470] Cleaning
add 9494886 [BEAM-8470] Implement WindowAssignTranslatorBatch
add c83a607 [BEAM-8470] Implement WindowAssignTest
add 9bf9385 [BEAM-8470] Fix javadoc
add 8f5e9dc [BEAM-8470] Added SideInput support
add 915ca01 [BEAM-8470] Fix CheckStyle violations
add 92cab8e [BEAM-8470] Don't use Reshuffle translation
add 6bbeeff [BEAM-8470] Added using CachedSideInputReader
add f8512c7 [BEAM-8470] Added TODO comment for ReshuffleTranslatorBatch
add 19ee565 [BEAM-8470] And unchecked warning suppression
add 1bf9a4a [BEAM-8470] Add streaming source initialisation
add 6ca706a [BEAM-8470] Implement first streaming source
add b5ccec6 [BEAM-8470] Add a TODO on spark output modes
add 175940d [BEAM-8470] Add transformators registry in PipelineTranslatorStreaming
add ba59f10 [BEAM-8470] Add source streaming test
add 80dde2d [BEAM-8470] Specify checkpointLocation at the pipeline start
add e494189 [BEAM-8470] Clean unneeded 0 arg constructor in batch source
add e92adee [BEAM-8470] Clean streaming source
add 31a2be5 [BEAM-8470] Continue impl of offsets for streaming source
add 8777607 [BEAM-8470] Deal with checkpoint and offset based read
add 43088fb [BEAM-8470] Apply spotless and fix spotbugs warnings
add 75e3f7f [BEAM-8470] Disable never ending test SimpleSourceTest.testUnboundedSource
add 9e60f27 [BEAM-8470] Fix access level issues, typos and modernize code to Java 8 style
add 0492206 [BEAM-8470] Merge Spark Structured Streaming runner into main Spark module. Remove spark-structured-streaming module.Rename Runner to SparkStructuredStreamingRunner. Remove specific PipelineOotions and Registrar for Structured Streaming Runner
add 0e37bae [BEAM-8470] Fix non-vendored imports from Spark Streaming Runner classes
add 01a2fd4 [BEAM-8470] Pass doFnSchemaInformation to ParDo batch translation
add f950dbf [BEAM-8470] Fix spotless issues after rebase
add 82cf713 [BEAM-8470] Fix logging levels in Spark Structured Streaming translation
add f44c863 [BEAM-8470] Add SparkStructuredStreamingPipelineOptions and SparkCommonPipelineOptions - SparkStructuredStreamingPipelineOptions was added to have the new runner rely only on its specific options.
add d0f2ba3 [BEAM-8470] Rename SparkPipelineResult to SparkStructuredStreamingPipelineResult This is done to avoid an eventual collision with the one in SparkRunner. However this cannot happen at this moment because it is package private, so it is also done for consistency.
add 23e3c53 [BEAM-8470] Use PAssert in Spark Structured Streaming transform tests
add 3ba3e51 [BEAM-8470] Ignore spark offsets (cf javadoc)
add 05266b9 [BEAM-8470] implement source.stop
add 75209f9 [BEAM-8470] Update javadoc
add f11e382 [BEAM-8470] Apply Spotless
add 075a79b [BEAM-8470] Enable batch Validates Runner tests for Structured Streaming Runner
add de5b17b [BEAM-8470] Limit the number of partitions to make tests go 300% faster
add 71edc3e [BEAM-8470] Fixes ParDo not calling setup and not tearing down if exception on startBundle
add 5972c51 [BEAM-8470] Pass transform based doFnSchemaInformation in ParDo translation
add 3dbcaba [BEAM-8470] Consider null object case on RowHelpers, fixes empty side inputs tests.
add a687d5d [BEAM-8470] Put back batch/simpleSourceTest.testBoundedSource
add b70f605 [BEAM-8470] Update windowAssignTest
add e070a69 [BEAM-8470] Add comment about checkpoint mark
add 2c1437c [BEAM-8470] Re-code GroupByKeyTranslatorBatch to conserve windowing instead of unwindowing/windowing(GlobalWindow): simplify code, use ReduceFnRunner to merge the windows
add 01184ff [BEAM-8470] re-enable reduceFnRunner timers for output
add dfed0a5 [BEAM-8470] Improve visibility of debug messages
add c7dd248 [BEAM-8470] Add a test that GBK preserves windowing
add 65e1b79 [BEAM-8470] Add TODO in Combine translations
add 346f2f4 [BEAM-8470] Update KVHelpers.extractKey() to deal with WindowedValue and update GBK and CPK
add c9f4525 [BEAM-8470] Fix comment about schemas
add b99c9ef [BEAM-8470] Implement reduce part of CombineGlobally translation with windowing
add 1b6339e [BEAM-8470] Output data after combine
add 66f6021 [BEAM-8470] Implement merge accumulators part of CombineGlobally translation with windowing
add 6168070 [BEAM-8470] Fix encoder in combine call
add 0c00e2f [BEAM-8470] Revert extractKey while combinePerKey is not done (so that it compiles)
add f9ecf1f [BEAM-8470] Apply a groupByKey avoids for some reason that the spark structured streaming fmwk casts data to Row which makes it impossible to deserialize without the coder shipped into the data. For performance reasons (avoid memory consumption and having to deserialize), we do not ship coder + data. Also add a mapparitions before GBK to avoid shuffling
add abaf140 [BEAM-8470] Fix case when a window does not merge into any other window
add 319e01f [BEAM-8470] Fix wrong encoder in combineGlobally GBK
add a759ddc [BEAM-8470] Fix bug in the window merging logic
add d609213 [BEAM-8470] Remove the mapPartition that adds a key per partition because otherwise spark will reduce values per key instead of globally
add 2cbd35d [BEAM-8470] Remove CombineGlobally translation because it is less performant than the beam sdk one (key + combinePerKey.withHotkeyFanout)
add 24c137a [BEAM-8470] Now that there is only Combine.PerKey translation, make only one Aggregator
add b7e4a8b [BEAM-8470] Clean no more needed KVHelpers
add 5215a5a [BEAM-8470] Clean not more needed RowHelpers
add 4e06ffe [BEAM-8470] Clean not more needed WindowingHelpers
add c83168c [BEAM-8470] Fix javadoc of AggregatorCombiner
add 5898cf0 [BEAM-8470] Fixed immutable list bug
add 37b5fa8 [BEAM-8470] add comment in combine globally test
add 6eea481 [BEAM-8470] Clean groupByKeyTest
add 3e33078 [BEAM-8470] Add a test that combine per key preserves windowing
add 55d865b [BEAM-8470] Ignore for now not working test testCombineGlobally
add 65ffdd8 [BEAM-8470] Add metrics support in DoFn
add e2b57ac [BEAM-8470] Add missing dependencies to run Spark Structured Streaming Runner on Nexmark
add 323d10f [BEAM-8470] Add setEnableSparkMetricSinks() method
add 2f1d38b [BEAM-8470] Fix javadoc
add 06559d2 [BEAM-8470] Fix accumulators initialization in Combine that prevented CombineGlobally to work.
add 3cce68c [BEAM-8470] Add a test to check that CombineGlobally preserves windowing
add 68b6f98 [BEAM-8470] Persist all output Dataset if there are multiple outputs in pipeline Enabled Use*Metrics tests
add 6084764 [BEAM-8470] Added metrics sinks and tests
add 69c514f [BEAM-8470] Make spotless happy
add 4cab501 [BEAM-8470] Add PipelineResults to Spark structured streaming.
add a1ec37a [BEAM-8470] Update log4j configuration
add 2a53b32 [BEAM-8470] Add spark execution plans extended debug messages.
add f34de05 [BEAM-8470] Print number of leaf datasets
add 892b742 [BEAM-8470] fixup! Add PipelineResults to Spark structured streaming.
add 613bac9 [BEAM-8470] Remove no more needed AggregatorCombinerPerKey (there is only AggregatorCombiner)
add d024335 [BEAM-8470] After testing performance and correctness, launch pipeline with dataset.foreach(). Make both test mode and production mode use foreach for uniformity. Move dataset print as a utility method
add 7a30c5a [BEAM-8470] Improve Pardo translation performance: avoid calling a filter transform when there is only one output tag
add 0e42c18 [BEAM-8470] Use "sparkMaster" in local mode to obtain number of shuffle partitions + spotless apply
add 7f08fc8 [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: serialization part
add 2071f1c [BEAM-8470] Wrap Beam Coders into Spark Encoders using ExpressionEncoder: deserialization part
add 6f0e9fa [BEAM-8470] type erasure: spark encoders require a Class<T>, pass Object and cast to Class<T>
add 965814a [BEAM-8470] Fix scala Product in Encoders to avoid StackEverflow
add 165d8b7 [BEAM-8470] Conform to spark ExpressionEncoders: pass classTags, implement scala Product, pass children from within the ExpressionEncoder, fix visibilities
add 609aace [BEAM-8470] Add a simple spark native test to test Beam coders wrapping into Spark Encoders
add 09e882e [BEAM-8470] Fix code generation in Beam coder wrapper
add d715e37 [BEAM-8470] Lazy init coder because coder instance cannot be interpolated by catalyst
add d27962c [BEAM-8470] Fix warning in coder construction by reflexion
add fe46a63 [BEAM-8470] Fix ExpressionEncoder generated code: typos, try catch, fqcn
add 64a978d [BEAM-8470] Fix getting the output value in code generation
add cfecb40 [BEAM-8470] Fix beam coder lazy init using reflexion: use .clas + try catch + cast
add 0d13b77 [BEAM-8470] Remove lazy init of beam coder because there is no generic way on instanciating a beam coder
add 1a6e662 [BEAM-8470] Remove example code
add e45e7b5 [BEAM-8470] Fix equal and hashcode
add dd63c1b [BEAM-8470] Fix generated code: uniform exceptions catching, fix parenthesis and variable declarations
add 75a8b24 [BEAM-8470] Add an assert of equality in the encoders test
add ff50364 [BEAM-8470] Apply spotless and checkstyle and add javadocs
add 4e215e0 [BEAM-8470] Wrap exceptions in UserCoderExceptions
add 8ed3db6 [BEAM-8470] Put Encoders expressions serializable
add 465d570 [BEAM-8470] Catch Exception instead of IOException because some coders to not throw Exceptions at all (e.g.VoidCoder)
add 4579663 [BEAM-8470] Apply new Encoders to CombinePerKey
add d246520 [BEAM-8470] Apply new Encoders to Read source
add a448cfb [BEAM-8470] Improve performance of source: the mapper already calls windowedValueCoder.decode, no need to call it also in the Spark encoder
add c875dbc [BEAM-8470] Ignore long time failing test: SparkMetricsSinkTest
add be7415d [BEAM-8470] Apply new Encoders to Window assign translation
add cdd6e1a [BEAM-8470] Apply new Encoders to AggregatorCombiner
add 9ae6a57 [BEAM-8470] Create a Tuple2Coder to encode scala tuple2
add f827291 [BEAM-8470] Apply new Encoders to GroupByKey
add 76125e9 [BEAM-8470] Apply new Encoders to Pardo. Replace Tuple2Coder with MultiOutputCoder to deal with multiple output to use in Spark Encoder for DoFnRunner
add f492ccf [BEAM-8470] Apply spotless, fix typo and javadoc
add d81d2f6 [BEAM-8470] Use beam encoders also in the output of the source translation
add 9599546 [BEAM-8470] Remove unneeded cast
add 0e072d3 [BEAM-8470] Fix: Remove generic hack of using object. Use actual Coder encodedType in Encoders
add da8dfbd [BEAM-8470] Remove Encoders based on kryo now that we call Beam coders in the runner
add 470d04f [BEAM-8470] Add a jenkins job for validates runner tests in the new spark runner
add 17ace84 [BEAM-8470] Apply spotless
add 2ca30a4 [BEAM-8470] Rebase on master: pass sideInputMapping in SimpleDoFnRunner as needed now in the API
add 99e081d Fix SpotBugs
add 52aae7f [BEAM-8470] simplify coders in combinePerKey translation
add 803bb0b [BEAM-8470] Fix combiner. Do not reuse instance of accumulator
add a4c4ee7 [BEAM-8470] input windows can arrive exploded (for sliding windows). As a result an input has multiple windows. So we need to consider that the accumulator can have multiple windows
add ea47c71 [BEAM-8470] Add a combine test with sliding windows
add 89eb1f0 [BEAM-8470] Add a test to test combine translation on binaryCombineFn with sliding windows
add 4416a2e [BEAM-8470] Fix tests: use correct SparkStructuredStreamingPipelineOptions, set testMode to true. Some renaming
add b980cf9 [BEAM-8470] Fix wrong expected results in CombineTest.testBinaryCombineWithSlidingWindows
add 22d98df [BEAM-8470] Add disclaimers about this runner being experimental
add d680252 [BEAM-8470] Fix: create an empty accumulator in combine.mergeAccumulators, because this method modifies its first input accumulator. Decrease memory usage by storing only accumulator and timestamp in the combine.merge map
add 9922bcb [BEAM-8470] Apply spotless
add a0872bc [BEAM-8470] Add a countPerElement test with sliding windows
add c9ce072 [BEAM-8470] Fix the output timestamps of combine: timestamps must be merged to a window before combining
add f516554 [BEAM-8470] set log level to info to avoid resource consumption in production mode
add 27dcc48 [BEAM-8470] Fix CombineTest.testCountPerElementWithSlidingWindows expected results
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (fe9e723)
\
N -- N -- N refs/heads/spark-runner_structured-streaming (27dcc48)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.gitignore | 1 +
.test-infra/dataproc/flink_cluster.sh | 2 +-
.test-infra/jenkins/CommonJobProperties.groovy | 32 +-
.test-infra/jenkins/PrecommitJobBuilder.groovy | 13 +-
.test-infra/jenkins/README.md | 64 +-
.test-infra/jenkins/job_Dependency_Check.groovy | 2 +-
.../job_LoadTests_Combine_Flink_Python.groovy | 6 +-
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 14 +-
.../job_LoadTests_ParDo_Flink_Python.groovy | 8 +-
.../job_LoadTests_coGBK_Flink_Python.groovy | 8 +-
.../job_PerformanceTests_BigQueryIO_Java.groovy | 10 +-
.../jenkins/job_PerformanceTests_KafkaIO_IT.groovy | 68 +
.test-infra/jenkins/job_PostCommit_Python37.groovy | 2 +-
.test-infra/jenkins/job_PreCommit_Java.groovy | 3 +
.../job_PreCommit_Java_Examples_Dataflow.groovy | 3 +-
.test-infra/jenkins/job_PreCommit_Python.groovy | 16 +
...Commit_Java.groovy => job_PreCommit_SQL.groovy} | 13 +-
.../kubernetes/kafka-cluster/00-namespace.yml | 19 -
.../02-rbac-namespace-default/node-reader.yml | 12 +-
.../02-rbac-namespace-default/pod-labler.yml | 3 -
.../03-zookeeper/10zookeeper-config.yml | 1 -
.../kafka-cluster/03-zookeeper/20pzoo-service.yml | 1 -
.../kafka-cluster/03-zookeeper/30service.yml | 1 -
.../kafka-cluster/03-zookeeper/50pzoo.yml | 1 -
.../04-outside-services/outside-0.yml | 1 -
.../04-outside-services/outside-1.yml | 1 -
.../04-outside-services/outside-2.yml | 1 -
.../kafka-cluster/05-kafka/10broker-config.yml | 1 -
.../kubernetes/kafka-cluster/05-kafka/20dns.yml | 1 -
.../kafka-cluster/05-kafka/30bootstrap-service.yml | 1 -
.../kubernetes/kafka-cluster/05-kafka/50kafka.yml | 1 -
.../kafka-cluster/05-kafka/configmap-config.yaml | 5 +-
.../kafka-cluster/05-kafka/job-config.yaml | 1 -
.test-infra/kubernetes/kubernetes.sh | 2 +-
build.gradle | 22 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 42 +-
examples/kotlin/build.gradle | 4 +-
gradle.properties | 2 +
learning/katas/java/.idea/study_project.xml | 3151 --------------------
.../Aggregation/Count/task-info.yaml | 29 +
.../Aggregation/Count/task-remote-info.yaml | 2 +
.../Aggregation/Max/task-info.yaml | 29 +
.../Aggregation/Max/task-remote-info.yaml | 2 +
.../Aggregation/Mean/task-info.yaml | 29 +
.../Aggregation/Mean/task-remote-info.yaml | 2 +
.../Aggregation/Min/task-info.yaml | 29 +
.../Aggregation/Min/task-remote-info.yaml | 2 +
.../Aggregation/Sum/task-info.yaml | 29 +
.../Aggregation/Sum/task-remote-info.yaml | 2 +
.../Common Transforms/Aggregation/lesson-info.yaml | 25 +
.../Aggregation/lesson-remote-info.yaml | 3 +
.../Common Transforms/Filter/Filter/task-info.yaml | 29 +
.../Filter/Filter/task-remote-info.yaml | 2 +
.../Common Transforms/Filter/ParDo/task-info.yaml | 29 +
.../Filter/ParDo/task-remote-info.yaml | 2 +
.../java/Common Transforms/Filter/lesson-info.yaml | 22 +
.../Filter/lesson-remote-info.yaml | 3 +
.../WithKeys/WithKeys/task-info.yaml | 29 +
.../WithKeys/WithKeys/task-remote-info.yaml | 2 +
.../Common Transforms/WithKeys/lesson-info.yaml | 21 +
.../WithKeys/lesson-remote-info.yaml | 3 +
.../katas/java/Common Transforms/section-info.yaml | 23 +
.../Common Transforms/section-remote-info.yaml | 2 +
.../Branching/Branching/task-info.yaml | 32 +
.../Branching/Branching/task-remote-info.yaml | 2 +
.../Core Transforms/Branching/lesson-info.yaml | 21 +
.../Branching/lesson-remote-info.yaml | 3 +
.../CoGroupByKey/CoGroupByKey/task-info.yaml | 31 +
.../CoGroupByKey/task-remote-info.yaml | 2 +
.../Core Transforms/CoGroupByKey/lesson-info.yaml | 21 +
.../CoGroupByKey/lesson-remote-info.yaml | 3 +
.../Combine/BinaryCombineFn Lambda/task-info.yaml | 29 +
.../BinaryCombineFn Lambda/task-remote-info.yaml | 2 +
.../Combine/BinaryCombineFn/task-info.yaml | 29 +
.../Combine/BinaryCombineFn/task-remote-info.yaml | 2 +
.../Combine/Combine PerKey/task-info.yaml | 32 +
.../Combine/Combine PerKey/task-remote-info.yaml | 2 +
.../Combine/CombineFn/task-info.yaml | 29 +
.../Combine/CombineFn/task-remote-info.yaml | 2 +
.../Combine/Simple Function/task-info.yaml | 29 +
.../Combine/Simple Function/task-remote-info.yaml | 2 +
.../java/Core Transforms/Combine/lesson-info.yaml | 25 +
.../Combine/lesson-remote-info.yaml | 3 +
.../Composite Transform/task-info.yaml | 29 +
.../Composite Transform/task-remote-info.yaml | 2 +
.../Composite Transform/lesson-info.yaml | 21 +
.../Composite Transform/lesson-remote-info.yaml | 3 +
.../DoFn Additional Parameters/task-info.yaml | 25 +
.../task-remote-info.yaml | 2 +
.../DoFn Additional Parameters/lesson-info.yaml | 21 +
.../lesson-remote-info.yaml | 3 +
.../Core Transforms/Flatten/Flatten/task-info.yaml | 29 +
.../Flatten/Flatten/task-remote-info.yaml | 2 +
.../java/Core Transforms/Flatten/lesson-info.yaml | 21 +
.../Flatten/lesson-remote-info.yaml | 3 +
.../GroupByKey/GroupByKey/task-info.yaml | 29 +
.../GroupByKey/GroupByKey/task-remote-info.yaml | 2 +
.../Core Transforms/GroupByKey/lesson-info.yaml | 21 +
.../GroupByKey/lesson-remote-info.yaml | 3 +
.../Map/FlatMapElements/task-info.yaml | 29 +
.../Map/FlatMapElements/task-remote-info.yaml | 2 +
.../Core Transforms/Map/MapElements/task-info.yaml | 29 +
.../Map/MapElements/task-remote-info.yaml | 2 +
.../Map/ParDo OneToMany/task-info.yaml | 29 +
.../Map/ParDo OneToMany/task-remote-info.yaml | 2 +
.../java/Core Transforms/Map/ParDo/task-info.yaml | 29 +
.../Map/ParDo/task-remote-info.yaml | 2 +
.../java/Core Transforms/Map/lesson-info.yaml | 24 +
.../Core Transforms/Map/lesson-remote-info.yaml | 3 +
.../Partition/Partition/task-info.yaml | 29 +
.../Partition/Partition/task-remote-info.yaml | 2 +
.../Core Transforms/Partition/lesson-info.yaml | 21 +
.../Partition/lesson-remote-info.yaml | 3 +
.../Side Input/Side Input/task-info.yaml | 34 +
.../Side Input/Side Input/task-remote-info.yaml | 2 +
.../Core Transforms/Side Input/lesson-info.yaml | 21 +
.../Side Input/lesson-remote-info.yaml | 3 +
.../Side Output/Side Output/task-info.yaml | 29 +
.../Side Output/Side Output/task-remote-info.yaml | 2 +
.../Core Transforms/Side Output/lesson-info.yaml | 21 +
.../Side Output/lesson-remote-info.yaml | 3 +
.../katas/java/Core Transforms/section-info.yaml | 31 +
.../java/Core Transforms/section-remote-info.yaml | 2 +
.../Examples/Word Count/Word Count/task-info.yaml | 29 +
.../Word Count/Word Count/task-remote-info.yaml | 2 +
.../java/Examples/Word Count/lesson-info.yaml | 21 +
.../Examples/Word Count/lesson-remote-info.yaml | 3 +
learning/katas/java/Examples/section-info.yaml | 21 +
.../katas/java/Examples/section-remote-info.yaml | 2 +
.../IO/Built-in IOs/Built-in IOs/task-info.yaml | 25 +
.../Built-in IOs/task-remote-info.yaml | 2 +
.../java/IO/Built-in IOs/Built-in IOs/task.html | 4 +
.../katas/java/IO/Built-in IOs/lesson-info.yaml | 21 +
.../java/IO/Built-in IOs/lesson-remote-info.yaml | 3 +
.../java/IO/TextIO/TextIO Read/task-info.yaml | 34 +
.../IO/TextIO/TextIO Read/task-remote-info.yaml | 2 +
learning/katas/java/IO/TextIO/lesson-info.yaml | 21 +
.../katas/java/IO/TextIO/lesson-remote-info.yaml | 3 +
learning/katas/java/IO/section-info.yaml | 22 +
learning/katas/java/IO/section-remote-info.yaml | 2 +
.../Hello Beam/Hello Beam/task-info.yaml | 29 +
.../Hello Beam/Hello Beam/task-remote-info.yaml | 2 +
.../java/Introduction/Hello Beam/lesson-info.yaml | 21 +
.../Hello Beam/lesson-remote-info.yaml | 3 +
learning/katas/java/Introduction/section-info.yaml | 21 +
.../java/Introduction/section-remote-info.yaml | 2 +
.../Early Triggers/Early Triggers/task-info.yaml | 31 +
.../Early Triggers/task-remote-info.yaml | 2 +
.../java/Triggers/Early Triggers/lesson-info.yaml | 21 +
.../Early Triggers/lesson-remote-info.yaml | 3 +
.../Event Time Triggers/task-info.yaml | 31 +
.../Event Time Triggers/task-remote-info.yaml | 2 +
.../Triggers/Event Time Triggers/lesson-info.yaml | 21 +
.../Event Time Triggers/lesson-remote-info.yaml | 3 +
.../Window Accumulation Mode/task-info.yaml | 31 +
.../Window Accumulation Mode/task-remote-info.yaml | 2 +
.../Window Accumulation Mode/lesson-info.yaml | 21 +
.../lesson-remote-info.yaml | 3 +
learning/katas/java/Triggers/section-info.yaml | 23 +
.../katas/java/Triggers/section-remote-info.yaml | 2 +
.../Adding Timestamp/ParDo/task-info.yaml | 31 +
.../Adding Timestamp/ParDo/task-remote-info.yaml | 2 +
.../Adding Timestamp/WithTimestamps/task-info.yaml | 31 +
.../WithTimestamps/task-remote-info.yaml | 2 +
.../Windowing/Adding Timestamp/lesson-info.yaml | 22 +
.../Adding Timestamp/lesson-remote-info.yaml | 3 +
.../Fixed Time Window/task-info.yaml | 31 +
.../Fixed Time Window/task-remote-info.yaml | 2 +
.../Windowing/Fixed Time Window/lesson-info.yaml | 21 +
.../Fixed Time Window/lesson-remote-info.yaml | 3 +
learning/katas/java/Windowing/section-info.yaml | 22 +
.../katas/java/Windowing/section-remote-info.yaml | 2 +
learning/katas/java/course-info.yaml | 33 +
learning/katas/java/course-remote-info.yaml | 2 +
learning/katas/python/.idea/study_project.xml | 2317 --------------
.../Aggregation/Count/task-info.yaml | 29 +
.../Aggregation/Count/task-remote-info.yaml | 2 +
.../Aggregation/Largest/task-info.yaml | 29 +
.../Aggregation/Largest/task-remote-info.yaml | 2 +
.../Aggregation/Mean/task-info.yaml | 29 +
.../Aggregation/Mean/task-remote-info.yaml | 2 +
.../Aggregation/Smallest/task-info.yaml | 29 +
.../Aggregation/Smallest/task-remote-info.yaml | 2 +
.../Aggregation/Sum/task-info.yaml | 29 +
.../Aggregation/Sum/task-remote-info.yaml | 2 +
.../Common Transforms/Aggregation/lesson-info.yaml | 25 +
.../Aggregation/lesson-remote-info.yaml | 3 +
.../Common Transforms/Filter/Filter/task-info.yaml | 29 +
.../Filter/Filter/task-remote-info.yaml | 2 +
.../Common Transforms/Filter/ParDo/task-info.yaml | 29 +
.../Filter/ParDo/task-remote-info.yaml | 2 +
.../Common Transforms/Filter/lesson-info.yaml | 22 +
.../Filter/lesson-remote-info.yaml | 3 +
.../python/Common Transforms/section-info.yaml | 22 +
.../Common Transforms/section-remote-info.yaml | 2 +
.../Branching/Branching/task-info.yaml | 32 +
.../Branching/Branching/task-remote-info.yaml | 2 +
.../Core Transforms/Branching/lesson-info.yaml | 21 +
.../Branching/lesson-remote-info.yaml | 3 +
.../CoGroupByKey/CoGroupByKey/task-info.yaml | 29 +
.../CoGroupByKey/task-remote-info.yaml | 2 +
.../Core Transforms/CoGroupByKey/lesson-info.yaml | 21 +
.../CoGroupByKey/lesson-remote-info.yaml | 3 +
.../Combine/Combine PerKey/task-info.yaml | 29 +
.../Combine/Combine PerKey/task-remote-info.yaml | 2 +
.../Combine/CombineFn/task-info.yaml | 32 +
.../Combine/CombineFn/task-remote-info.yaml | 2 +
.../Combine/Simple Function/task-info.yaml | 32 +
.../Combine/Simple Function/task-remote-info.yaml | 2 +
.../Core Transforms/Combine/lesson-info.yaml | 23 +
.../Combine/lesson-remote-info.yaml | 3 +
.../Composite Transform/task-info.yaml | 32 +
.../Composite Transform/task-remote-info.yaml | 2 +
.../Composite Transform/lesson-info.yaml | 21 +
.../Composite Transform/lesson-remote-info.yaml | 3 +
.../Core Transforms/Flatten/Flatten/task-info.yaml | 29 +
.../Flatten/Flatten/task-remote-info.yaml | 2 +
.../Core Transforms/Flatten/lesson-info.yaml | 21 +
.../Flatten/lesson-remote-info.yaml | 3 +
.../GroupByKey/GroupByKey/task-info.yaml | 29 +
.../GroupByKey/GroupByKey/task-remote-info.yaml | 2 +
.../Core Transforms/GroupByKey/lesson-info.yaml | 21 +
.../GroupByKey/lesson-remote-info.yaml | 3 +
.../Core Transforms/Map/FlatMap/task-info.yaml | 29 +
.../Map/FlatMap/task-remote-info.yaml | 2 +
.../python/Core Transforms/Map/Map/task-info.yaml | 29 +
.../Core Transforms/Map/Map/task-remote-info.yaml | 2 +
.../Map/ParDo OneToMany/task-info.yaml | 32 +
.../Map/ParDo OneToMany/task-remote-info.yaml | 2 +
.../Core Transforms/Map/ParDo/task-info.yaml | 32 +
.../Map/ParDo/task-remote-info.yaml | 2 +
.../python/Core Transforms/Map/lesson-info.yaml | 24 +
.../Core Transforms/Map/lesson-remote-info.yaml | 3 +
.../Partition/Partition/task-info.yaml | 32 +
.../Partition/Partition/task-remote-info.yaml | 2 +
.../Core Transforms/Partition/lesson-info.yaml | 21 +
.../Partition/lesson-remote-info.yaml | 3 +
.../Side Input/Side Input/task-info.yaml | 32 +
.../Side Input/Side Input/task-remote-info.yaml | 2 +
.../Core Transforms/Side Input/lesson-info.yaml | 21 +
.../Side Input/lesson-remote-info.yaml | 3 +
.../Side Output/Side Output/task-info.yaml | 32 +
.../Side Output/Side Output/task-remote-info.yaml | 2 +
.../Core Transforms/Side Output/lesson-info.yaml | 21 +
.../Side Output/lesson-remote-info.yaml | 3 +
.../katas/python/Core Transforms/section-info.yaml | 30 +
.../Core Transforms/section-remote-info.yaml | 2 +
.../Examples/Word Count/Word Count/task-info.yaml | 29 +
.../Word Count/Word Count/task-remote-info.yaml | 2 +
.../python/Examples/Word Count/lesson-info.yaml | 21 +
.../Examples/Word Count/lesson-remote-info.yaml | 3 +
learning/katas/python/Examples/section-info.yaml | 21 +
.../katas/python/Examples/section-remote-info.yaml | 2 +
.../IO/Built-in IOs/Built-in IOs/task-info.yaml | 25 +
.../Built-in IOs/task-remote-info.yaml | 2 +
.../python/IO/Built-in IOs/Built-in IOs/task.html | 4 +
.../katas/python/IO/Built-in IOs/lesson-info.yaml | 21 +
.../python/IO/Built-in IOs/lesson-remote-info.yaml | 3 +
.../python/IO/TextIO/ReadFromText/task-info.yaml | 34 +
.../IO/TextIO/ReadFromText/task-remote-info.yaml | 2 +
learning/katas/python/IO/TextIO/lesson-info.yaml | 21 +
.../katas/python/IO/TextIO/lesson-remote-info.yaml | 3 +
learning/katas/python/IO/section-info.yaml | 22 +
learning/katas/python/IO/section-remote-info.yaml | 2 +
.../Hello Beam/Hello Beam/task-info.yaml | 29 +
.../Hello Beam/Hello Beam/task-remote-info.yaml | 2 +
.../Introduction/Hello Beam/lesson-info.yaml | 21 +
.../Hello Beam/lesson-remote-info.yaml | 3 +
.../katas/python/Introduction/section-info.yaml | 21 +
.../python/Introduction/section-remote-info.yaml | 2 +
learning/katas/python/course-info.yaml | 31 +
learning/katas/python/course-remote-info.yaml | 2 +
.../fn-execution/src/main/proto/beam_fn_api.proto | 43 +
.../beam/model/fnexecution/v1/standard_coders.yaml | 23 +
{runners/reference => model/interactive}/OWNERS | 2 +-
.../interactive}/build.gradle | 18 +-
.../src/main/proto/beam_interactive_api.proto | 59 +
.../src/main/proto/beam_job_api.proto | 34 +
.../pipeline/src/main/proto/beam_runner_api.proto | 87 +
model/pipeline/src/main/proto/schema.proto | 11 +-
ownership/JAVA_DEPENDENCY_OWNERS.yaml | 4 +-
runners/apex/build.gradle | 6 +-
.../core/construction/CoderTranslators.java | 33 +
.../runners/core/construction/Environments.java | 7 +-
.../core/construction/ModelCoderRegistrar.java | 3 +
.../runners/core/construction/ModelCoders.java | 5 +-
.../construction/PipelineOptionsTranslation.java | 25 +-
.../core/construction/SchemaTranslation.java | 9 +
.../graph/GreedyPCollectionFusers.java | 11 +-
.../core/construction/graph/ProtoOverrides.java | 26 +-
.../core/construction/graph/QueryablePipeline.java | 4 +
.../graph/SplittableParDoExpander.java | 273 ++
.../core/construction/CoderTranslationTest.java | 17 +-
.../runners/core/construction/CommonCoderTest.java | 117 +-
.../core/construction/PipelineTranslationTest.java | 2 +-
.../core/construction/SchemaTranslationTest.java | 5 +
.../graph/SplittableParDoExpanderTest.java | 122 +
.../beam/runners/core/metrics/GaugeData.java | 3 +-
.../runners/core/metrics/MetricsContainerImpl.java | 13 +-
runners/flink/flink_runner.gradle | 4 +-
runners/flink/job-server/flink_job_server.gradle | 9 +-
.../runners/flink/FlinkExecutionEnvironments.java | 24 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 4 +-
.../beam/runners/flink/FlinkJobServerDriver.java | 39 +-
.../beam/runners/flink/FlinkPipelineRunner.java | 13 +-
.../flink/FlinkPortableClientEntryPoint.java | 257 ++
.../flink/metrics/FlinkMetricContainer.java | 12 +-
.../FlinkExecutableStageContextFactory.java | 21 +-
.../wrappers/streaming/DoFnOperator.java | 58 +-
.../streaming/ExecutableStageDoFnOperator.java | 61 +-
.../flink/FlinkExecutionEnvironmentsTest.java | 38 +-
.../runners/flink/FlinkJobServerDriverTest.java | 22 +-
.../beam/runners/flink/FlinkSavepointTest.java | 15 +-
.../beam/runners/flink/PortableExecutionTest.java | 17 +-
.../runners/flink/PortableStateExecutionTest.java | 17 +-
.../runners/flink/PortableTimersExecutionTest.java | 17 +-
.../beam/runners/flink/ReadSourcePortableTest.java | 17 +-
.../flink/metrics/FlinkMetricContainerTest.java | 4 +-
.../wrappers/streaming/DoFnOperatorTest.java | 193 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 7 +-
runners/google-cloud-dataflow-java/build.gradle | 4 +-
.../beam/runners/dataflow/DataflowRunner.java | 43 +-
.../beam/runners/dataflow/TransformTranslator.java | 8 +
.../dataflow/options/DataflowPipelineOptions.java | 72 +-
.../util/SchemaCoderCloudObjectTranslator.java | 15 +-
.../beam/runners/dataflow/DataflowRunnerTest.java | 50 +
.../options/DataflowPipelineOptionsTest.java | 46 +
.../runners/dataflow/util/CloudObjectsTest.java | 12 +-
.../dataflow/worker/FnApiWindowMappingFn.java | 19 +-
.../dataflow/worker/StreamingDataflowWorker.java | 42 +-
.../control/RegisterAndProcessBundleOperation.java | 81 +-
.../worker/windmill/GrpcWindmillServer.java | 5 +-
.../worker/StreamingDataflowWorkerTest.java | 156 +-
.../fn/control/BeamFnMapTaskExecutorTest.java | 24 +-
.../RegisterAndProcessBundleOperationTest.java | 54 +-
.../worker/windmill/src/main/proto/windmill.proto | 9 +-
...stHandler.java => BundleCheckpointHandler.java} | 15 +-
...Handler.java => BundleFinalizationHandler.java} | 24 +-
.../control/BundleFinalizationHandlers.java | 65 +
.../control/DefaultExecutableStageContext.java | 56 +-
.../control/DefaultJobBundleFactory.java | 106 +-
.../fnexecution/control/FnApiControlClient.java | 22 +-
.../control/InstructionRequestHandler.java | 5 +-
.../control/ProcessBundleDescriptors.java | 28 +
.../fnexecution/control/SdkHarnessClient.java | 86 +-
.../fnexecution/environment/ProcessManager.java | 39 +-
.../fnexecution/jobsubmission/JobServerDriver.java | 16 +-
.../control/BundleFinalizationHandlersTest.java | 56 +
.../control/DefaultExecutableStageContextTest.java | 72 -
.../control/DefaultJobBundleFactoryTest.java | 72 +
.../control/ProcessBundleDescriptorsTest.java | 144 +
.../fnexecution/control/RemoteExecutionTest.java | 4 +-
.../fnexecution/control/SdkHarnessClientTest.java | 132 +-
.../environment/ProcessManagerTest.java | 58 +
runners/{reference => portability}/OWNERS | 0
.../{reference => portability}/java/build.gradle | 4 +-
.../runners/portability}/CloseableResource.java | 2 +-
.../portability}/ExternalWorkerService.java | 2 +-
.../portability}/JobServicePipelineResult.java | 2 +-
.../beam/runners/portability}/PortableRunner.java | 4 +-
.../portability}/PortableRunnerRegistrar.java | 2 +-
.../beam/runners/portability}/package-info.java | 2 +-
.../portability}/testing/TestJobService.java | 2 +-
.../testing/TestPortablePipelineOptions.java | 10 +-
.../portability}/testing/TestPortableRunner.java | 35 +-
.../runners/portability}/testing/package-info.java | 2 +-
.../portability}/CloseableResourceTest.java | 4 +-
.../runners/portability}/PortableRunnerTest.java | 4 +-
.../beam/runners/samza/SamzaPipelineRunner.java | 13 +-
runners/spark/job-server/build.gradle | 5 +-
.../beam/runners/spark/SparkJobServerDriver.java | 12 +-
.../beam/runners/spark/SparkPipelineRunner.java | 13 +-
.../SparkExecutableStageContextFactory.java | 24 +-
sdks/go/pkg/beam/core/runtime/exec/coder.go | 3 +
sdks/go/pkg/beam/core/runtime/exec/fullvalue.go | 59 +-
.../pkg/beam/core/runtime/exec/fullvalue_test.go | 22 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 1 -
.../pkg/beam/core/runtime/pipelinex/clone_test.go | 5 +-
.../beam/core/runtime/pipelinex/replace_test.go | 7 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 634 ++--
.../beam/model/jobmanagement_v1/beam_job_api.pb.go | 284 +-
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 949 +++---
sdks/go/pkg/beam/pardo.go | 2 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 2 +
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 7 +
sdks/go/test/run_integration_tests.sh | 2 +-
sdks/java/container/build.gradle | 4 +-
.../java/org/apache/beam/sdk/coders/RowCoder.java | 7 +-
.../apache/beam/sdk/coders/RowCoderGenerator.java | 17 +-
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 2 +
.../main/java/org/apache/beam/sdk/io/FileIO.java | 10 +-
.../org/apache/beam/sdk/io/LocalFileSystem.java | 2 +-
.../java/org/apache/beam/sdk/io/TextSource.java | 6 +
.../beam/sdk/options/PortablePipelineOptions.java | 8 +-
.../org/apache/beam/sdk/options/ValueProvider.java | 39 +
.../apache/beam/sdk/schemas/AutoValueSchema.java | 59 +-
.../apache/beam/sdk/schemas/AvroRecordSchema.java | 13 +-
.../apache/beam/sdk/schemas/CachingFactory.java | 18 +
.../beam/sdk/schemas/FieldAccessDescriptor.java | 4 +-
.../beam/sdk/schemas/FieldTypeDescriptors.java | 16 +
.../sdk/schemas/FieldValueTypeInformation.java | 20 +-
.../beam/sdk/schemas/FromRowUsingCreator.java | 72 +-
.../sdk/schemas/GetterBasedSchemaProvider.java | 73 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 80 +-
.../apache/beam/sdk/schemas/JavaFieldSchema.java | 40 +-
.../java/org/apache/beam/sdk/schemas/Schema.java | 18 +-
.../org/apache/beam/sdk/schemas/SchemaCoder.java | 50 +-
.../beam/sdk/schemas/UserTypeCreatorFactory.java | 5 +-
.../beam/sdk/schemas/transforms/AddFields.java | 13 +-
.../apache/beam/sdk/schemas/transforms/Cast.java | 6 +-
.../beam/sdk/schemas/transforms/Convert.java | 1 +
.../beam/sdk/schemas/transforms/RenameFields.java | 2 +
.../sdk/schemas/transforms/SchemaAggregateFn.java | 4 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 30 +-
.../beam/sdk/schemas/utils/ByteBuddyUtils.java | 25 +-
.../beam/sdk/schemas/utils/ConvertHelpers.java | 10 +-
.../beam/sdk/schemas/utils/SchemaZipFold.java | 1 +
.../beam/sdk/schemas/utils/SelectHelpers.java | 22 +-
.../sdk/schemas/utils/StaticSchemaInference.java | 27 +-
.../org/apache/beam/sdk/testing/TestStream.java | 4 +-
.../UsesTestStreamWithMultipleStages.java} | 13 +-
.../org/apache/beam/sdk/transforms/Create.java | 24 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 6 +-
.../org/apache/beam/sdk/transforms/GroupByKey.java | 47 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 10 +-
.../beam/sdk/transforms/windowing/AfterEach.java | 5 +
.../sdk/transforms/windowing/AfterWatermark.java | 6 +
.../sdk/transforms/windowing/DefaultTrigger.java | 6 +
.../sdk/transforms/windowing/OrFinallyTrigger.java | 5 +
.../beam/sdk/transforms/windowing/Repeatedly.java | 5 +
.../sdk/transforms/windowing/ReshuffleTrigger.java | 5 +
.../beam/sdk/transforms/windowing/Trigger.java | 14 +
.../java/org/apache/beam/sdk/util/RowJson.java | 18 +-
.../apache/beam/sdk/util/RowJsonValidation.java | 5 +
.../apache/beam/sdk/util/SerializableUtils.java | 13 +
.../org/apache/beam/sdk/util/package-info.java | 6 +-
.../org/apache/beam/sdk/values/PCollection.java | 10 +-
.../main/java/org/apache/beam/sdk/values/Row.java | 89 +-
.../org/apache/beam/sdk/values/RowWithGetters.java | 31 +
.../org/apache/beam/sdk/coders/RowCoderTest.java | 56 +-
.../apache/beam/sdk/io/LocalFileSystemTest.java | 16 +
.../org/apache/beam/sdk/io/TextIOWriteTest.java | 15 +-
.../org/apache/beam/sdk/io/TextSourceTest.java | 158 +
.../sdk/options/PortablePipelineOptionsTest.java | 44 +
.../apache/beam/sdk/options/ValueProviderTest.java | 6 +-
.../beam/sdk/schemas/AutoValueSchemaTest.java | 31 +
.../apache/beam/sdk/schemas/AvroSchemaTest.java | 16 +
.../beam/sdk/schemas/FieldTypeDescriptorsTest.java | 28 +
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 31 +
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 34 +-
.../apache/beam/sdk/schemas/SchemaCoderTest.java | 293 ++
.../org/apache/beam/sdk/schemas/SchemaTest.java | 28 +
.../beam/sdk/schemas/transforms/AddFieldsTest.java | 67 +-
.../beam/sdk/schemas/transforms/ConvertTest.java | 8 +-
.../beam/sdk/schemas/transforms/GroupTest.java | 1 +
.../beam/sdk/schemas/utils/SelectHelpersTest.java | 159 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 59 +-
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 45 +-
.../apache/beam/sdk/testing/TestStreamTest.java | 84 +-
.../org/apache/beam/sdk/transforms/CreateTest.java | 2 +
.../apache/beam/sdk/transforms/GroupByKeyTest.java | 102 +
.../beam/sdk/transforms/ParDoLifecycleTest.java | 94 +-
.../beam/sdk/transforms/ParDoSchemaTest.java | 9 +
.../beam/sdk/transforms/windowing/TriggerTest.java | 10 +
.../java/org/apache/beam/sdk/util/RowJsonTest.java | 18 +
.../java/org/apache/beam/sdk/values/RowTest.java | 111 +-
sdks/java/extensions/euphoria/build.gradle | 2 +-
.../sdk/extensions/gcp/options/GcpOptions.java | 31 +
.../sdk/extensions/gcp/options/GcpOptionsTest.java | 30 +-
sdks/java/extensions/jackson/build.gradle | 2 +-
sdks/java/extensions/join-library/build.gradle | 2 +-
sdks/java/extensions/kryo/build.gradle | 2 +-
sdks/java/extensions/sketching/build.gradle | 2 +-
sdks/java/extensions/sorter/build.gradle | 2 +-
sdks/java/extensions/sql/build.gradle | 5 +-
.../datacatalog/DataCatalogBigQueryIT.java | 52 +-
.../sdk/extensions/sql/impl/BeamCalciteTable.java | 2 -
.../sdk/extensions/sql/impl/rel/BeamCalcRel.java | 26 +-
.../sql/impl/rel/BeamEnumerableConverter.java | 5 +
.../extensions/sql/impl/rel/BeamIOSourceRel.java | 36 +-
...SourceRel.java => BeamPushDownIOSourceRel.java} | 97 +-
.../sql/impl/rule/BeamIOPushDownRule.java | 242 +-
.../extensions/sql/impl/utils/CalciteUtils.java | 2 +
.../sdk/extensions/sql/meta/BaseBeamTable.java | 4 +-
.../beam/sdk/extensions/sql/meta/BeamSqlTable.java | 2 +-
.../sdk/extensions/sql/meta/ProjectSupport.java} | 18 +-
.../provider/bigquery/BeamBigQuerySqlDialect.java | 275 ++
.../sql/meta/provider/bigquery/BigQueryFilter.java | 144 +
.../sql/meta/provider/bigquery/BigQueryTable.java | 70 +-
.../sql/meta/provider/mongodb/MongoDbTable.java | 173 ++
.../provider/mongodb/MongoDbTableProvider.java | 53 +
.../sql/meta/provider/mongodb}/package-info.java | 8 +-
.../sql/meta/provider/test/TestTableFilter.java | 118 +
.../sql/meta/provider/test/TestTableProvider.java | 171 +-
.../sql/impl/rule/IOPushDownRuleTest.java | 27 +-
.../meta/provider/bigquery/BigQueryFilterTest.java | 121 +
.../provider/bigquery/BigQueryReadWriteIT.java | 246 +-
.../meta/provider/mongodb/MongoDbReadWriteIT.java | 185 ++
.../provider/mongodb/MongoDbTableProviderTest.java | 118 +
.../meta/provider/mongodb/MongoDbTableTest.java | 136 +
.../sql/meta/provider/pubsub/PubsubJsonIT.java | 79 +-
.../meta/provider/test/TestTableProviderTest.java | 4 +-
...tTableProviderWithFilterAndProjectPushDown.java | 423 +++
.../test/TestTableProviderWithFilterPushDown.java | 308 ++
.../test/TestTableProviderWithProjectPushDown.java | 67 +-
.../extensions/sql/zetasql/ZetaSQLPlannerImpl.java | 6 -
.../sql/zetasql/ZetaSQLQueryPlanner.java | 20 +-
.../sql/zetasql/JoinCompoundIdentifiersTest.java | 3 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java | 3 +-
.../sql/zetasql/ZetaSQLPushDownTest.java | 222 ++
sdks/java/extensions/zetasketch/build.gradle | 2 +-
.../beam/sdk/extensions/zetasketch/HllCount.java | 19 +
.../BigQueryHllSketchCompatibilityIT.java | 124 +-
.../sdk/extensions/zetasketch/HllCountTest.java | 14 +
.../harness/state/BeamFnStateGrpcClientCache.java | 14 +-
sdks/java/io/amazon-web-services/build.gradle | 2 +-
sdks/java/io/amazon-web-services2/build.gradle | 2 +-
sdks/java/io/amqp/build.gradle | 2 +-
.../beam/sdk/bigqueryioperftests/BigQueryIOIT.java | 65 +-
sdks/java/io/cassandra/build.gradle | 2 +-
sdks/java/io/clickhouse/build.gradle | 2 +-
.../elasticsearch-tests-2/build.gradle | 2 +-
.../elasticsearch-tests-5/build.gradle | 2 +-
.../elasticsearch-tests-6/build.gradle | 2 +-
.../elasticsearch-tests-common/build.gradle | 2 +-
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 64 +-
.../beam/sdk/io/gcp/bigquery/AvroRowWriter.java | 62 +
.../sdk/io/gcp/bigquery/AvroWriteRequest.java} | 24 +-
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 23 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 10 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 180 +-
...{TableRowWriter.java => BigQueryRowWriter.java} | 24 +-
.../gcp/bigquery/BigQueryStorageStreamSource.java | 14 +-
.../sdk/io/gcp/bigquery/BigQueryTableSource.java | 3 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 14 +-
.../beam/sdk/io/gcp/bigquery/RowWriterFactory.java | 118 +
.../beam/sdk/io/gcp/bigquery/TableRowWriter.java | 62 +-
.../beam/sdk/io/gcp/bigquery/TestBigQuery.java | 54 +-
.../sdk/io/gcp/bigquery/WriteBundlesToFiles.java | 28 +-
.../gcp/bigquery/WriteGroupedRecordsToFiles.java | 22 +-
.../beam/sdk/io/gcp/bigquery/WritePartition.java | 19 +-
.../beam/sdk/io/gcp/bigquery/WriteTables.java | 7 +-
.../apache/beam/sdk/io/gcp/pubsub/PubsubIO.java | 10 +-
.../beam/sdk/io/gcp/pubsub/PubsubJsonClient.java | 2 +-
.../beam/sdk/io/gcp/pubsub/PubsubMessage.java | 10 +
.../apache/beam/sdk/io/gcp/pubsub/TestPubsub.java | 90 +-
.../beam/sdk/io/gcp/testing/FakeJobService.java | 28 +-
.../sdk/io/gcp/bigquery/BigQueryAvroUtilsTest.java | 4 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 3 +
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 151 +-
.../beam/sdk/io/gcp/spanner/SpannerIOReadTest.java | 173 +-
sdks/java/io/hadoop-file-system/build.gradle | 2 +-
sdks/java/io/hadoop-format/build.gradle | 2 +-
sdks/java/io/hbase/build.gradle | 2 +-
sdks/java/io/hcatalog/build.gradle | 2 +-
sdks/java/io/jdbc/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 99 +-
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 1 +
.../org/apache/beam/sdk/io/jdbc/SchemaUtil.java | 1 +
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 28 +-
sdks/java/io/jms/build.gradle | 2 +-
sdks/java/io/kafka/build.gradle | 2 +-
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 20 +-
sdks/java/io/kinesis/build.gradle | 4 +-
.../org/apache/beam/sdk/io/kinesis/KinesisIO.java | 14 +-
.../apache/beam/sdk/io/kinesis/KinesisReader.java | 16 +-
.../apache/beam/sdk/io/kinesis/KinesisSource.java | 17 +-
.../beam/sdk/io/kinesis/ShardReadersPool.java | 15 +-
.../beam/sdk/io/kinesis/KinesisReaderTest.java | 3 +-
.../beam/sdk/io/kinesis/ShardReadersPoolTest.java | 10 +-
sdks/java/io/kudu/build.gradle | 2 +-
sdks/java/io/mongodb/build.gradle | 2 +-
.../org/apache/beam/sdk/io/mongodb/MongoDbIO.java | 39 +
sdks/java/io/mqtt/build.gradle | 2 +-
sdks/java/io/parquet/build.gradle | 2 +-
.../org/apache/beam/sdk/io/parquet/ParquetIO.java | 3 +-
sdks/java/io/rabbitmq/build.gradle | 9 +-
.../apache/beam/sdk/io/rabbitmq/RabbitMqIO.java | 200 +-
.../beam/sdk/io/rabbitmq/RabbitMqMessage.java | 40 +-
.../beam/sdk/io/rabbitmq/ExchangeTestPlan.java | 102 +
.../beam/sdk/io/rabbitmq/RabbitMqIOTest.java | 301 +-
.../beam/sdk/io/rabbitmq/RabbitMqTestUtils.java | 90 +
.../sdk/io/rabbitmq/rabbitmq-io-test-config.json | 30 +
sdks/java/io/redis/build.gradle | 2 +-
sdks/java/io/solr/build.gradle | 2 +-
sdks/java/io/synthetic/build.gradle | 2 +-
sdks/java/io/tika/build.gradle | 2 +-
sdks/java/io/xml/build.gradle | 2 +-
.../src/main/resources/archetype-resources/pom.xml | 2 +-
.../apache/beam/sdk/nexmark/queries/Query10.java | 3 +-
.../queries/sql/SqlBoundedSideInputJoin.java | 2 +
.../sdk/nexmark/queries/sql/SqlQuery0Test.java | 2 +
.../sdk/nexmark/queries/sql/SqlQuery1Test.java | 2 +
sdks/python/apache_beam/coders/__init__.py | 1 +
.../apache_beam/coders/coders_test_common.py | 5 +
sdks/python/apache_beam/coders/row_coder.py | 174 ++
sdks/python/apache_beam/coders/row_coder_test.py | 168 ++
.../apache_beam/coders/standard_coders_test.py | 46 +-
.../transforms/aggregation/combineperkey.py | 270 ++
.../transforms/aggregation/combineperkey_test.py | 99 +
.../snippets/transforms/aggregation/distinct.py} | 27 +-
.../transforms/aggregation/distinct_test.py} | 54 +-
sdks/python/apache_beam/internal/gcp/auth.py | 10 +-
sdks/python/apache_beam/io/filebasedsink.py | 19 +-
sdks/python/apache_beam/io/filebasedsink_test.py | 4 +-
sdks/python/apache_beam/io/fileio.py | 37 +-
sdks/python/apache_beam/io/fileio_test.py | 64 +
sdks/python/apache_beam/io/filesystemio_test.py | 4 +-
.../io/gcp/big_query_query_to_table_it_test.py | 5 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 13 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 14 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 8 +-
.../apache_beam/io/gcp/bigquery_read_it_test.py | 9 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 9 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 33 +-
.../apache_beam/io/gcp/bigquery_write_it_test.py | 9 +-
.../apache_beam/io/gcp/datastore/v1/datastoreio.py | 21 +-
.../apache_beam/io/gcp/datastore/v1/helper.py | 5 +-
.../datastore/v1new/datastore_write_it_pipeline.py | 12 +-
.../io/gcp/datastore/v1new/datastoreio.py | 100 +-
.../io/gcp/datastore/v1new/datastoreio_test.py | 79 +
.../apache_beam/io/gcp/datastore/v1new/helper.py | 54 -
.../io/gcp/datastore/v1new/helper_test.py | 85 -
.../io/gcp/datastore/v1new/types_test.py | 7 +-
.../io/gcp/datastore_write_it_pipeline.py | 13 +-
sdks/python/apache_beam/io/gcp/gcsio.py | 21 +-
sdks/python/apache_beam/io/gcp/gcsio_overrides.py | 6 +-
.../apache_beam/io/gcp/tests/bigquery_matcher.py | 18 +-
.../io/gcp/tests/bigquery_matcher_test.py | 2 +
.../apache_beam/io/gcp/tests/pubsub_matcher.py | 4 +-
sdks/python/apache_beam/io/gcp/tests/utils.py | 5 +-
sdks/python/apache_beam/io/hadoopfilesystem.py | 4 +-
sdks/python/apache_beam/io/iobase.py | 236 +-
sdks/python/apache_beam/io/iobase_test.py | 123 +
sdks/python/apache_beam/io/mongodbio.py | 7 +-
sdks/python/apache_beam/io/mongodbio_it_test.py | 10 +-
sdks/python/apache_beam/io/parquetio_test.py | 3 +-
sdks/python/apache_beam/io/range_trackers.py | 13 +-
sdks/python/apache_beam/io/restriction_trackers.py | 128 +-
.../apache_beam/io/restriction_trackers_test.py | 12 +-
sdks/python/apache_beam/io/source_test_utils.py | 11 +-
sdks/python/apache_beam/io/textio.py | 5 +-
sdks/python/apache_beam/io/tfrecordio.py | 5 +-
sdks/python/apache_beam/io/vcfio.py | 5 +-
.../metrics/{execution.pxd => cells.pxd} | 34 +-
sdks/python/apache_beam/metrics/cells.py | 229 +-
sdks/python/apache_beam/metrics/cells_test.py | 23 -
sdks/python/apache_beam/metrics/execution.pxd | 26 +-
sdks/python/apache_beam/metrics/execution.py | 178 +-
sdks/python/apache_beam/metrics/execution_test.py | 43 +-
sdks/python/apache_beam/metrics/metric.py | 21 +-
sdks/python/apache_beam/metrics/metric_test.py | 55 +-
.../python/apache_beam/metrics/monitoring_infos.py | 26 +
.../python/apache_beam/options/pipeline_options.py | 85 +-
.../options/pipeline_options_validator.py | 17 +
.../options/pipeline_options_validator_test.py | 50 +
sdks/python/apache_beam/pipeline.py | 87 +-
sdks/python/apache_beam/pipeline_test.py | 185 +-
sdks/python/apache_beam/runners/common.pxd | 6 +-
sdks/python/apache_beam/runners/common.py | 66 +-
...dataflow_exercise_streaming_metrics_pipeline.py | 4 +-
...low_exercise_streaming_metrics_pipeline_test.py | 6 +-
.../runners/dataflow/dataflow_metrics.py | 6 +-
.../runners/dataflow/dataflow_runner.py | 99 +-
.../runners/dataflow/dataflow_runner_test.py | 58 +-
.../runners/dataflow/internal/apiclient.py | 26 +-
.../apache_beam/runners/dataflow/internal/names.py | 15 +-
.../runners/dataflow/native_io/iobase.py | 4 +-
.../runners/dataflow/test_dataflow_runner.py | 4 +-
.../apache_beam/runners/direct/bundle_factory.py | 10 +-
.../apache_beam/runners/direct/direct_runner.py | 12 +-
.../runners/direct/direct_runner_test.py | 83 +
sdks/python/apache_beam/runners/direct/executor.py | 10 +-
.../runners/direct/sdf_direct_runner_test.py | 18 +-
.../runners/direct/transform_evaluator.py | 8 +-
.../runners/interactive/display/display_manager.py | 13 +-
.../interactive/display/pcoll_visualization.py | 290 ++
.../display/pcoll_visualization_test.py | 152 +
.../runners/interactive/display/pipeline_graph.py | 4 +-
.../runners/interactive/interactive_environment.py | 116 +
.../interactive/interactive_environment_test.py | 160 +
.../runners/interactive/interactive_runner.py | 34 +-
.../runners/interactive/pipeline_analyzer.py | 11 +-
.../runners/interactive/pipeline_analyzer_test.py | 26 +
sdks/python/apache_beam/runners/job/manager.py | 4 +-
.../python/apache_beam/runners/pipeline_context.py | 21 +-
.../runners/portability/abstract_job_service.py | 18 +-
.../runners/portability/artifact_service_test.py | 6 +-
.../runners/portability/expansion_service_test.py | 10 +-
.../runners/portability/flink_runner.py | 51 +-
.../runners/portability/flink_runner_test.py | 116 +-
.../portability/flink_uber_jar_job_server.py | 14 +-
.../runners/portability/fn_api_runner.py | 138 +-
.../runners/portability/fn_api_runner_test.py | 213 +-
.../portability/fn_api_runner_transforms.py | 279 +-
.../runners/portability/local_job_service.py | 53 +-
.../runners/portability/local_job_service_main.py | 4 +-
.../runners/portability/portable_metrics.py | 71 +
.../runners/portability/portable_runner.py | 163 +-
.../runners/portability/portable_runner_test.py | 109 +-
.../runners/portability/portable_stager_test.py | 4 +-
.../runners/portability/spark_runner.py | 84 +
.../apache_beam/runners/portability/stager.py | 35 +-
.../apache_beam/runners/portability/stager_test.py | 11 +-
sdks/python/apache_beam/runners/runner.py | 14 +-
sdks/python/apache_beam/runners/utils.py | 47 +
.../apache_beam/runners/worker/bundle_processor.py | 73 +-
.../apache_beam/runners/worker/data_plane.py | 12 +-
.../apache_beam/runners/worker/data_plane_test.py | 4 +-
.../apache_beam/runners/worker/log_handler_test.py | 12 +-
.../apache_beam/runners/worker/operations.py | 13 +-
.../apache_beam/runners/worker/sdk_worker.py | 172 +-
.../apache_beam/runners/worker/sdk_worker_main.py | 52 +-
.../runners/worker/sdk_worker_main_test.py | 47 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 19 +-
.../apache_beam/runners/worker/sideinputs.py | 5 +-
.../apache_beam/runners/worker/statecache.py | 123 +-
.../apache_beam/runners/worker/statecache_test.py | 123 +-
.../runners/worker/statesampler_fast.pxd | 5 +-
.../runners/worker/statesampler_fast.pyx | 19 +-
.../runners/worker/statesampler_slow.py | 4 +
.../runners/worker/statesampler_test.py | 14 +-
.../apache_beam/runners/worker/worker_pool_main.py | 28 +-
.../testing/benchmarks/nexmark/nexmark_util.py | 4 +-
.../testing/data/trigger_transcripts.yaml | 70 +-
.../apache_beam/testing/load_tests/load_test.py | 32 +-
.../testing/load_tests/load_test_metrics_utils.py | 19 +-
.../apache_beam/testing/load_tests/pardo_test.py | 2 -
.../apache_beam/testing/pipeline_verifiers.py | 8 +-
.../apache_beam/testing/pipeline_verifiers_test.py | 3 +-
.../apache_beam/testing/synthetic_pipeline.py | 2 +-
sdks/python/apache_beam/testing/test_stream.py | 65 +-
.../python/apache_beam/testing/test_stream_test.py | 23 +-
.../tools/fn_api_runner_microbenchmark.py | 148 +
sdks/python/apache_beam/tools/utils.py | 77 +-
sdks/python/apache_beam/transforms/core.py | 95 +-
sdks/python/apache_beam/transforms/core_test.py | 54 +
sdks/python/apache_beam/transforms/environments.py | 404 +++
.../apache_beam/transforms/environments_test.py | 61 +
sdks/python/apache_beam/transforms/ptransform.py | 4 +-
.../apache_beam/transforms/ptransform_test.py | 2 +-
sdks/python/apache_beam/transforms/timeutil.py | 2 +-
.../transforms_keyword_only_args_test_py3.py | 144 +
sdks/python/apache_beam/transforms/trigger.py | 66 +-
sdks/python/apache_beam/transforms/trigger_test.py | 361 ++-
sdks/python/apache_beam/transforms/util.py | 32 +-
sdks/python/apache_beam/transforms/util_test.py | 67 +-
sdks/python/apache_beam/transforms/window.py | 8 +-
sdks/python/apache_beam/transforms/window_test.py | 99 +-
.../typehints/native_type_compatibility.py | 36 +
sdks/python/apache_beam/typehints/schemas.py | 218 ++
sdks/python/apache_beam/typehints/schemas_test.py | 270 ++
.../apache_beam/typehints/typed_pipeline_test.py | 37 +
.../typehints/typed_pipeline_test_py3.py | 22 +-
sdks/python/apache_beam/typehints/typehints.py | 6 +-
sdks/python/apache_beam/utils/profiler.py | 12 +-
sdks/python/apache_beam/utils/retry.py | 7 +-
sdks/python/apache_beam/utils/subprocess_server.py | 17 +-
.../apache_beam/utils/thread_pool_executor.py | 170 ++
.../apache_beam/utils/thread_pool_executor_test.py | 114 +
sdks/python/apache_beam/utils/timestamp.py | 7 +
sdks/python/apache_beam/utils/timestamp_test.py | 6 +
sdks/python/apache_beam/utils/windowed_value.py | 16 +-
sdks/python/build.gradle | 2 +-
.../coders/__init__.py => conftest.py} | 16 +-
sdks/python/container/Dockerfile | 11 +-
sdks/python/container/base_image_requirements.txt | 6 +-
sdks/python/gen_protos.py | 1 +
.../metrics/execution.pxd => pytest.ini} | 19 +-
sdks/python/scripts/generate_pydoc.sh | 3 +
sdks/python/scripts/run_pytest.sh | 49 +
sdks/python/setup.py | 33 +-
sdks/python/test-suites/portable/common.gradle | 20 +
sdks/python/test-suites/portable/py2/build.gradle | 9 +
sdks/python/test-suites/portable/py35/build.gradle | 9 +
sdks/python/test-suites/portable/py36/build.gradle | 9 +
sdks/python/test-suites/portable/py37/build.gradle | 9 +
sdks/python/test-suites/tox/py2/build.gradle | 18 +-
sdks/python/test-suites/tox/py35/build.gradle | 16 +-
sdks/python/test-suites/tox/py36/build.gradle | 15 +-
sdks/python/test-suites/tox/py37/build.gradle | 15 +-
.../{portable/py36 => tox/pycommon}/build.gradle | 19 +-
sdks/python/test_config.py | 1 +
sdks/python/tox.ini | 101 +-
settings.gradle | 3 +-
website/Gemfile | 2 +-
website/Gemfile.lock | 39 +-
website/src/_includes/section-menu/contribute.html | 2 +
.../src/_includes/section-menu/documentation.html | 1 +
website/src/contribute/jira-priorities.md | 76 +
website/src/contribute/release-blocking.md | 41 +
website/src/documentation/programming-guide.md | 4 +-
website/src/documentation/sdks/python-streaming.md | 14 +-
.../java/aggregation/approximateunique.md | 2 +
.../transforms/java/aggregation/hllcount.md | 77 +
website/src/documentation/transforms/java/index.md | 1 +
website/src/roadmap/connectors-multi-sdk.md | 85 +-
website/src/roadmap/index.md | 25 +-
798 files changed, 22817 insertions(+), 10663 deletions(-)
create mode 100644 .test-infra/jenkins/job_PerformanceTests_KafkaIO_IT.groovy
copy .test-infra/jenkins/{job_PreCommit_Java.groovy => job_PreCommit_SQL.groovy} (86%)
delete mode 100644 .test-infra/kubernetes/kafka-cluster/00-namespace.yml
delete mode 100644 learning/katas/java/.idea/study_project.xml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Count/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Count/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Max/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Max/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Mean/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Mean/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Min/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Min/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Sum/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/Sum/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/lesson-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Aggregation/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Filter/Filter/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Filter/Filter/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Filter/ParDo/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Filter/ParDo/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Filter/lesson-info.yaml
create mode 100644 learning/katas/java/Common Transforms/Filter/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/WithKeys/WithKeys/task-info.yaml
create mode 100644 learning/katas/java/Common Transforms/WithKeys/WithKeys/task-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/WithKeys/lesson-info.yaml
create mode 100644 learning/katas/java/Common Transforms/WithKeys/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Common Transforms/section-info.yaml
create mode 100644 learning/katas/java/Common Transforms/section-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Branching/Branching/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Branching/Branching/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Branching/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Branching/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn Lambda/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/BinaryCombineFn/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/Combine PerKey/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/CombineFn/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/CombineFn/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/Simple Function/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/Simple Function/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Combine/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Composite Transform/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/DoFn Additional Parameters/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/DoFn Additional Parameters/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Flatten/Flatten/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Flatten/Flatten/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Flatten/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Flatten/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/GroupByKey/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/FlatMapElements/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/FlatMapElements/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/MapElements/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/MapElements/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/ParDo/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Map/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Partition/Partition/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Partition/Partition/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Partition/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Partition/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Input/Side Input/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Input/Side Input/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Input/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Input/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Output/Side Output/task-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Output/Side Output/task-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Output/lesson-info.yaml
create mode 100644 learning/katas/java/Core Transforms/Side Output/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Core Transforms/section-info.yaml
create mode 100644 learning/katas/java/Core Transforms/section-remote-info.yaml
create mode 100644 learning/katas/java/Examples/Word Count/Word Count/task-info.yaml
create mode 100644 learning/katas/java/Examples/Word Count/Word Count/task-remote-info.yaml
create mode 100644 learning/katas/java/Examples/Word Count/lesson-info.yaml
create mode 100644 learning/katas/java/Examples/Word Count/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Examples/section-info.yaml
create mode 100644 learning/katas/java/Examples/section-remote-info.yaml
create mode 100644 learning/katas/java/IO/Built-in IOs/Built-in IOs/task-info.yaml
create mode 100644 learning/katas/java/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml
create mode 100644 learning/katas/java/IO/Built-in IOs/lesson-info.yaml
create mode 100644 learning/katas/java/IO/Built-in IOs/lesson-remote-info.yaml
create mode 100644 learning/katas/java/IO/TextIO/TextIO Read/task-info.yaml
create mode 100644 learning/katas/java/IO/TextIO/TextIO Read/task-remote-info.yaml
create mode 100644 learning/katas/java/IO/TextIO/lesson-info.yaml
create mode 100644 learning/katas/java/IO/TextIO/lesson-remote-info.yaml
create mode 100644 learning/katas/java/IO/section-info.yaml
create mode 100644 learning/katas/java/IO/section-remote-info.yaml
create mode 100644 learning/katas/java/Introduction/Hello Beam/Hello Beam/task-info.yaml
create mode 100644 learning/katas/java/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
create mode 100644 learning/katas/java/Introduction/Hello Beam/lesson-info.yaml
create mode 100644 learning/katas/java/Introduction/Hello Beam/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Introduction/section-info.yaml
create mode 100644 learning/katas/java/Introduction/section-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/Early Triggers/Early Triggers/task-info.yaml
create mode 100644 learning/katas/java/Triggers/Early Triggers/Early Triggers/task-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/Early Triggers/lesson-info.yaml
create mode 100644 learning/katas/java/Triggers/Early Triggers/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-info.yaml
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/Event Time Triggers/task-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/lesson-info.yaml
create mode 100644 learning/katas/java/Triggers/Event Time Triggers/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-info.yaml
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/Window Accumulation Mode/task-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/lesson-info.yaml
create mode 100644 learning/katas/java/Triggers/Window Accumulation Mode/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Triggers/section-info.yaml
create mode 100644 learning/katas/java/Triggers/section-remote-info.yaml
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/ParDo/task-info.yaml
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/ParDo/task-remote-info.yaml
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-info.yaml
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/WithTimestamps/task-remote-info.yaml
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/lesson-info.yaml
create mode 100644 learning/katas/java/Windowing/Adding Timestamp/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-info.yaml
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/Fixed Time Window/task-remote-info.yaml
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/lesson-info.yaml
create mode 100644 learning/katas/java/Windowing/Fixed Time Window/lesson-remote-info.yaml
create mode 100644 learning/katas/java/Windowing/section-info.yaml
create mode 100644 learning/katas/java/Windowing/section-remote-info.yaml
create mode 100644 learning/katas/java/course-info.yaml
create mode 100644 learning/katas/java/course-remote-info.yaml
delete mode 100644 learning/katas/python/.idea/study_project.xml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Count/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Count/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Largest/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Largest/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Mean/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Mean/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Smallest/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Smallest/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Sum/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/Sum/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/lesson-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Aggregation/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Filter/Filter/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Filter/Filter/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Filter/ParDo/task-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Filter/ParDo/task-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Filter/lesson-info.yaml
create mode 100644 learning/katas/python/Common Transforms/Filter/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Common Transforms/section-info.yaml
create mode 100644 learning/katas/python/Common Transforms/section-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Branching/Branching/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Branching/Branching/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Branching/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Branching/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/CoGroupByKey/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/CoGroupByKey/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/Combine PerKey/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/Combine PerKey/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/CombineFn/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/CombineFn/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/Simple Function/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/Simple Function/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Combine/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/Composite Transform/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Composite Transform/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Flatten/Flatten/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Flatten/Flatten/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Flatten/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Flatten/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/GroupByKey/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/GroupByKey/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/FlatMap/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/FlatMap/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/Map/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/Map/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/ParDo/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Map/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Partition/Partition/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Partition/Partition/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Partition/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Partition/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Input/Side Input/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Input/Side Input/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Input/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Input/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Output/Side Output/task-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Output/Side Output/task-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Output/lesson-info.yaml
create mode 100644 learning/katas/python/Core Transforms/Side Output/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Core Transforms/section-info.yaml
create mode 100644 learning/katas/python/Core Transforms/section-remote-info.yaml
create mode 100644 learning/katas/python/Examples/Word Count/Word Count/task-info.yaml
create mode 100644 learning/katas/python/Examples/Word Count/Word Count/task-remote-info.yaml
create mode 100644 learning/katas/python/Examples/Word Count/lesson-info.yaml
create mode 100644 learning/katas/python/Examples/Word Count/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Examples/section-info.yaml
create mode 100644 learning/katas/python/Examples/section-remote-info.yaml
create mode 100644 learning/katas/python/IO/Built-in IOs/Built-in IOs/task-info.yaml
create mode 100644 learning/katas/python/IO/Built-in IOs/Built-in IOs/task-remote-info.yaml
create mode 100644 learning/katas/python/IO/Built-in IOs/lesson-info.yaml
create mode 100644 learning/katas/python/IO/Built-in IOs/lesson-remote-info.yaml
create mode 100644 learning/katas/python/IO/TextIO/ReadFromText/task-info.yaml
create mode 100644 learning/katas/python/IO/TextIO/ReadFromText/task-remote-info.yaml
create mode 100644 learning/katas/python/IO/TextIO/lesson-info.yaml
create mode 100644 learning/katas/python/IO/TextIO/lesson-remote-info.yaml
create mode 100644 learning/katas/python/IO/section-info.yaml
create mode 100644 learning/katas/python/IO/section-remote-info.yaml
create mode 100644 learning/katas/python/Introduction/Hello Beam/Hello Beam/task-info.yaml
create mode 100644 learning/katas/python/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
create mode 100644 learning/katas/python/Introduction/Hello Beam/lesson-info.yaml
create mode 100644 learning/katas/python/Introduction/Hello Beam/lesson-remote-info.yaml
create mode 100644 learning/katas/python/Introduction/section-info.yaml
create mode 100644 learning/katas/python/Introduction/section-remote-info.yaml
create mode 100644 learning/katas/python/course-info.yaml
create mode 100644 learning/katas/python/course-remote-info.yaml
copy {runners/reference => model/interactive}/OWNERS (84%)
copy {sdks/java/extensions/join-library => model/interactive}/build.gradle (61%)
create mode 100644 model/interactive/src/main/proto/beam_interactive_api.proto
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpander.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/graph/SplittableParDoExpanderTest.java
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/FlinkPortableClientEntryPoint.java
copy runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{InstructionRequestHandler.java => BundleCheckpointHandler.java} (67%)
copy runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/{InstructionRequestHandler.java => BundleFinalizationHandler.java} (56%)
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlers.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/BundleFinalizationHandlersTest.java
delete mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/DefaultExecutableStageContextTest.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/control/ProcessBundleDescriptorsTest.java
rename runners/{reference => portability}/OWNERS (100%)
rename runners/{reference => portability}/java/build.gradle (91%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/CloseableResource.java (98%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/ExternalWorkerService.java (98%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/JobServicePipelineResult.java (99%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/PortableRunner.java (98%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/PortableRunnerRegistrar.java (96%)
copy runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/package-info.java (95%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/testing/TestJobService.java (98%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/testing/TestPortablePipelineOptions.java (88%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/testing/TestPortableRunner.java (71%)
rename runners/{reference/java/src/main/java/org/apache/beam/runners/reference => portability/java/src/main/java/org/apache/beam/runners/portability}/testing/package-info.java (94%)
rename runners/{reference/java/src/test/java/org/apache/beam/runners/reference => portability/java/src/test/java/org/apache/beam/runners/portability}/CloseableResourceTest.java (96%)
rename runners/{reference/java/src/test/java/org/apache/beam/runners/reference => portability/java/src/test/java/org/apache/beam/runners/portability}/PortableRunnerTest.java (96%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/{schemas/UserTypeCreatorFactory.java => testing/UsesTestStreamWithMultipleStages.java} (71%)
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/io/TextSourceTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/options/PortablePipelineOptionsTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/schemas/SchemaCoderTest.java
copy sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/rel/{BeamIOSourceRel.java => BeamPushDownIOSourceRel.java} (50%)
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java => extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/ProjectSupport.java} (72%)
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BeamBigQuerySqlDialect.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryFilter.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProvider.java
rename {runners/reference/java/src/main/java/org/apache/beam/runners/reference => sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb}/package-info.java (77%)
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableFilter.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigquery/BigQueryFilterTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbReadWriteIT.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableProviderTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/mongodb/MongoDbTableTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterAndProjectPushDown.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/test/TestTableProviderWithFilterPushDown.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/test/java/org/apache/beam/sdk/extensions/sql/zetasql/ZetaSQLPushDownTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroRowWriter.java
copy sdks/java/{core/src/main/java/org/apache/beam/sdk/schemas/UserTypeCreatorFactory.java => io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/AvroWriteRequest.java} (67%)
copy sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/{TableRowWriter.java => BigQueryRowWriter.java} (76%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigquery/RowWriterFactory.java
create mode 100644 sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/ExchangeTestPlan.java
create mode 100644 sdks/java/io/rabbitmq/src/test/java/org/apache/beam/sdk/io/rabbitmq/RabbitMqTestUtils.java
create mode 100644 sdks/java/io/rabbitmq/src/test/resources/org/apache/beam/sdk/io/rabbitmq/rabbitmq-io-test-config.json
create mode 100644 sdks/python/apache_beam/coders/row_coder.py
create mode 100644 sdks/python/apache_beam/coders/row_coder_test.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/aggregation/combineperkey_test.py
copy sdks/python/apache_beam/{coders/__init__.py => examples/snippets/transforms/aggregation/distinct.py} (60%)
copy sdks/python/apache_beam/{runners/portability/local_job_service_main.py => examples/snippets/transforms/aggregation/distinct_test.py} (55%)
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1new/helper_test.py
copy sdks/python/apache_beam/metrics/{execution.pxd => cells.pxd} (52%)
create mode 100644 sdks/python/apache_beam/runners/interactive/display/pcoll_visualization.py
create mode 100644 sdks/python/apache_beam/runners/interactive/display/pcoll_visualization_test.py
create mode 100644 sdks/python/apache_beam/runners/portability/portable_metrics.py
create mode 100644 sdks/python/apache_beam/runners/portability/spark_runner.py
create mode 100644 sdks/python/apache_beam/runners/utils.py
create mode 100644 sdks/python/apache_beam/tools/fn_api_runner_microbenchmark.py
create mode 100644 sdks/python/apache_beam/transforms/core_test.py
create mode 100644 sdks/python/apache_beam/transforms/environments.py
create mode 100644 sdks/python/apache_beam/transforms/environments_test.py
create mode 100644 sdks/python/apache_beam/transforms/transforms_keyword_only_args_test_py3.py
create mode 100644 sdks/python/apache_beam/typehints/schemas.py
create mode 100644 sdks/python/apache_beam/typehints/schemas_test.py
create mode 100644 sdks/python/apache_beam/utils/thread_pool_executor.py
create mode 100644 sdks/python/apache_beam/utils/thread_pool_executor_test.py
copy sdks/python/{apache_beam/coders/__init__.py => conftest.py} (67%)
copy sdks/python/{apache_beam/metrics/execution.pxd => pytest.ini} (63%)
create mode 100755 sdks/python/scripts/run_pytest.sh
copy sdks/python/test-suites/{portable/py36 => tox/pycommon}/build.gradle (67%)
create mode 100644 website/src/contribute/jira-priorities.md
create mode 100644 website/src/contribute/release-blocking.md
create mode 100644 website/src/documentation/transforms/java/aggregation/hllcount.md