You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mw...@apache.org on 2020/04/29 10:41:03 UTC
[beam] branch BEAM-9646-cloud-vision updated (6beafe0 -> 86175c8)
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a change to branch BEAM-9646-cloud-vision
in repository https://gitbox.apache.org/repos/asf/beam.git.
discard 6beafe0 Added change to CHANGES.md
discard d5f911b Fix spotbugs errors
discard 73390fd [BEAM-9646] Add Google Cloud vision integration transform
add 6e7e09f Merge pull request #11226: [BEAM-9557] Fix timer window boundary checking
add 59ea9c2 Merge pull request #11244 from [BEAM-3097] _ReadFromBigQuery supports valueprovider for table
add a271dc2 Name the pipeline_v1 proto import
add 02f328e Update materialize_test.go
add c1212a6 Merge pull request #11334 [Go SDK] Name the pipeline_v1 proto import
add 442ad98 [BEAM-9715] Ensuring annotations_test passes in all environments(Win/*nix)
add ea54794 Merge pull request #11329 from [BEAM-9715] Ensuring annotations_test passes in all environments
add d333b2c Fix flaky interactive_runner_test
add 02f6f05 Merge pull request #11337 from Fix flaky interactive_runner_test
add 0c1ac2c Merge pull request #11205 [BEAM-9578] Defer expensive artifact enumeration in Java.
add 076dccd Fix some Go SDK linter/vet warnings. (#11330)
add 1a217db [BEAM-9707] Hardcode Unified harness image for fixing dataflow VR 2 tests
add 1f48ac6 Merge pull request #11326 from angoenka/uw_image
add 4f4e9b3 [BEAM-9691] Ensuring BQSource is avoided on FnApi
add c00df44 Merge pull request #11309 from [BEAM-9691] Ensuring BQ Native Sink is avoided on FnApi pipelines
add f1fac7c Update go protos.
add 9841f38 [BEAM-9618] Pull bundle descriptors for Go.
add 197d2dd Merge pull request #11324 [BEAM-9618] Pull bundle descriptors for Go.
add ba24517 [BEAM-9577] Plumb resources through Python job service and runner.
add 8239ffa Merge pull request #11312 from [BEAM-9577] Plumb resources through Python job service and runner.
add a35afef [BEAM-9529] Remove datastore.v1, googledatastore (#11175)
add c70d30b [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
add a2766fb Merge pull request #11241: [BEAM-5422] Document DynamicDestinations.getTable uniqueness requirement
add 825363b [BEAM-9147] Add a VideoIntelligence transform to Java SDK (#11261)
add e325044 [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
add d4790fd Revert "[BEAM-9580] Downgrade Flink version to 1.9 for Nexmark and ValidatesRunner"
add b6dbf2a [BEAM-9557] Fix strings used to verify test output
add e7f6ccb Merge pull request #11346: [BEAM-9580] Allow Flink 1.10 processing timers to finish on pipeline shutdown
add c367536 [BEAM-9714] [Go SDK] Require --region flag in Dataflow runner.
add 76d7732 Merge pull request #11348 from ibzib/beam-9199-go
add 3f92f32 Update session.go
add f449dca Update stage.go
add 632535c Update server_test.go
add a4dcf29 Update materialize.go
add ca70ab8 Update materialize_test.go
add 4a378e9 Update stage_test.go
add b6e0584 Update artifact.go
add 209508c Update provision.go
add de68243 Update retrieval.go
add cf19153 Update staging.go
add 483a9c8 Update translate.go
add 11f086a Update datamgr.go
add 3c77761 Update datamgr_test.go
add 1894be7 Update logging.go
add 1378b31 Update logging_test.go
add e229c74 Update monitoring.go
add d9772ee Update session.go
add e1a2730 Update statemgr.go
add ca30a64 Update statemgr_test.go
add 1f0eb7c Update replace.go
add 46afdfd Update replace_test.go
add 68e369c Update provision.go
add c3b3c77 Update execute.go
add 532ea91 Update job.go
add a4d5648 Update translate.go
add 84ebc24 Update translate.go
add 54014d7 Update job.go
add eab9a0f Update materialize.go
add 2548664 Update translate.go
add 9da745e Update session.go
add ecb852b Update materialize_test.go
add ba6feec Merge pull request #11343 from apache/lostluck-protosuffix
add 826a68a [BEAM-9596] Ensure metrics are available in PipelineResult when the pipeline fails
add c04ac9f Merge pull request #11349: [BEAM-9596] Ensure metrics are available in PipelineResult when the pipeline fails
add 964c307 Updates documentation for WINDOWED_VALUE coder.
add 3038900 Uses iterable coder for windows and copies all of timestamp encoding specification.
add 338168d Merge pull request #11336 from acrites/documentation
add 8fb7f0f Update session.go (#11352)
add b497bf4 [BEAM-9322] [BEAM-1833] Better naming for composite transform output tags.
add cebe628 Merge pull request #11283 [BEAM-9322] [BEAM-1833] Better naming for composite transform output tags.
add 568db83 [BEAM-9618] Java SDK worker support for pulling bundle descriptors. (#11328)
add 82689f1 [BEAM-9716] Alias zone to worker_zone and warn user.
add b56740f Merge pull request #11332 from ibzib/beam-9716
add 6f6809a [Beam-9063]update documentation (#10952)
add af0acea Adds nose back under packages needed for testing.
add e7663b8 Merge pull request #11354: [BEAM-9719] Adds nose back to the base image
add cc5cf30 [BEAM-9618] Mark push registration as deprecated.
add bf4cee8 Merge pull request #11356 from robertwb/proto-notes
add 84d5402 Ensure that empty messages are not flushed to handler.
add 9280e68 Merge pull request #11351 from scwhittle/fix_log
add 71e9d3c [BEAM-9721] Add --region to Dataflow-based load tests
add bb0c0ec [BEAM-9721] LoadTestConfig: handle --region parameter and put default value if absent
add ca4e031 Merge pull request #11345: [BEAM-9721] Add --region to Dataflow-based load tests
add 237a99f [BEAM-9550] Increase JVM Metaspace size for the TaskExecutors. (#11193)
add 24108f9 [BEAM-9721]Conditionally add Dataflow region to Dataflow-based performance tests (#11344)
add 00f4f2f [BEAM-9647] fixes MQTT clientId to long
add 2b8f4d9 Merge pull request #11361: [BEAM-9647] Fix MQTT clientId too long
add ac003b6 [BEAM-8280] Document Python 3 annotations support (#11232)
add 5f432a2 [BEAM-9731] Include more detail in passert.Equals errors. (#11359)
add 79b2d87 [BEAM-9085] Fix performance regression in SyntheticSource on Python 3 (#11092)
add 7c46990 [BEAM-9726] [py] Make region optional for non-service Dataflow.
add a32613f [BEAM-9726] [java] Make region optional for non-service runner.
add c245288 Merge pull request #11353 from ibzib/beam-9726
add 80ebb70 [BEAM-4374] Fix missing deletion of metrics.
add 95a5944 Merge pull request #11363 from lukecwik/beam4374
add d36f873 [BEAM-9651] Prevent StreamPool and stream initialization livelock
add 28b081f Merge pull request #11364: [BEAM-9651] Prevent StreamPool and stream initialization livelock
add f251133 Revert "[BEAM-9651] Prevent StreamPool and stream initialization livelock"
add e8fc585 Merge pull request #11367 from scwhittle/revert-11364-contention
add cc42ab8 [BEAM-9727] Automatically set required experiment flags for dataflow runner v2. (#11355)
add 742a8ce Allow unset write threshold for state backed iterable coder.
add 3d0cde3 Merge pull request #11366 State backed iterable fixes.
add 8db19a4 [BEAM-9562, BEAM-6274] Fix-up timers to use Elements.Timer proto in data channel in Beam Java and Python
add 1de50c3 Merge pull request #11314 from boyuanzz/data
add 2498596 Update environments.py to add a method to specify container image (#11370)
add eeb0715 add missing bracket
add 0695414 Merge pull request #11247 from chie8842/patch-1
add 5f574b8 Moving to 2.22.0-SNAPSHOT on master branch.
add 247090f [BEAM-9734] Revert #11122
add 6a4ef33 Merge pull request #11376: [BEAM-9734] Revert #11122
add 1b8ac22 [BEAM-9651] Prevent StreamPool and stream initialization livelock
add 57ca62d Merge pull request #11368: [BEAM-9651] Prevent StreamPool and stream initialization livelock
add 82abec7 Add --region to changelog
add 7fb0abf Merge pull request #11379 from apache/region-changelog
add 5208c3d Upgrades gcsio to 2.1.2
add 75433f0 Merge pull request #11315: [BEAM-8889] Upgrades gcsio to 2.1.2
add c26af84 Attempt to stage resources via new API in portable runner.
add d5929ba Merge pull request #11311 [BEAM-9577] Attempt to stage resources via new API in portable runner.
add 81092b7 Fix from_container_image call
add c17e7c9 Merge pull request #11383 from [BEAM-9736] Fix from_container_image call
add d4c283e [BEAM-9562] Update Element.timer, Element.Timer to Element.timers and Element.Timers.
add 99fa8ac Merge pull request #11373 from boyuanzz/timers
add f0ba6bf [BEAM-9735] Adding Always trigger and using it in Reshuffle
add 7082c77 Merge pull request #11365 [BEAM-9735] Adding Always trigger and using it in Reshuffle
add 2e0cf8f [BEAM-9443] support direct_num_workers=0 (#11372)
add d1c04cb TOIL: Update Unified worker image
add d1e706e Merge pull request #11385 from angoenka/update_um_image
add 8458f47 google-api-client 1.30.9
add 3f99b33 Merge remote-tracking branch 'origin/master' into BEAM-8751
add cf4657b Merge pull request #11208 from suztomo/BEAM-8751
add 9b2dc82 Updates Dataflow stateful DoFn setup to support external transforms
add d566479 Merge pull request #11400: [BEAM-8019] Updates Dataflow stateful DoFn setup to support external transforms
add 537c641 [BEAM-9738] Update dataflow to setup correct docker environment options. (#11371)
add 296f5a7 [BEAM-9136]Add licenses for dependencies for Java (#11243)
add ad804f1 [BEAM-9496] Evaluation of deferred dataframes via Beam operations.
add 2197ba1 Fix and test tuple inputs and outputs.
add b557c12 Comments and clarification.
add b6e5082 Merge pull request #10760 [BEAM-9545] Dataframe transforms
add 009578e [BEAM-9562] Fix output timestamp to be inferred from scheduled time when in the event time domain.
add a7f444f Merge pull request #11402 from lukecwik/timers
add 7b454e9 [BEAM-9744] Add missing region option to py perf tests.
add 6eeed69 Merge pull request #11401 from ibzib/performance-region
add 7e07607 [BEAM-9744] Remove --region option from SQL tests.
add 40903f2 Merge pull request #11408 from ibzib/performance-region
add ec0e157 Merge pull request #11389 from Refactor the BCJ and capture controls to be more testable
add 9336272 [BEAM-i9751] upgrade zetasql to 2020.04.1 (#11410)
add 0595f79 [BEAM-9650] Add PeriodicImpulse Transform and slowly changing side input documentation (#11182)
add a57f3a0 Documentation bug fix for FlatMapElements#via() SimpleFunction in the javadoc should return List<String> instead of Integer
add 55dec55 Merge pull request #11412: Documentation minor fix for FlatMapElements#via()
add 1754d03 [BEAM-7923] Screendiff Integration Tests (#11338)
add f7647ad [BEAM-9642] Add SDF execution units. (#11327)
add f950b71 [BEAM-9136] reduce third_party_dependencies size (#11416)
add acb37bb [BEAM-9746] check for 0 length copies from state (#11413)
add 7f2ab69 [BEAM-9642] Fix infinite recursion.
add d279f4b Merge pull request #11423: [BEAM-9642] Fix infinite recursion.
add a39daa6 [BEAM-2939] Update unbounded source as SDF wrapper to resume successfully.
add 27c2c75 fixup! Fix spotbugs warning
add 1774530 Merge pull request #11409 from lukecwik/splittabledofn
add 34326a2 [BEAM-9577] Fix test to create urls from paths which are compatible with Windows.
add ef6df74 Merge pull request #11420 from lukecwik/beam9577
add 42d3d43 Removing underscore from _ReadFromBigQuery to make it external. It remains experimental.
add b85cdbe Merge pull request #11422 from [BEAM-9763] Removing underscore from _ReadFromBigQuery to make it external. It re…
add d310f98 [BEAM-9756] Nexmark: only use --region in Dataflow.
add 303b371 Merge pull request #11417 from ibzib/nexmark-region
add 744119c [BEAM-8889] add gRPC suport in GCS connector (behind an experimental-flag) (#11381)
add d7e0d5d [BEAM-8466] Make strip_iterable more strict
add 14424cc Merge pull request #11240: [BEAM-8466] Make strip_iterable more strict
add 5f5ae6f [BEAM-9119] Disable flaky test
add 26ee594 Merge pull request #11425: [BEAM-9119] Disable flaky test_large_elements
add 5c35755 Fixing type names for BQ Avro Tools
add 1db7186 Merge pull request #11404 from Fixing type names for BQ Avro Tools
add a562f1e Fix py37-lint
add 7438265 Maven compatible publish repository authentication via settings.xml
add 1658d90 Merge pull request #11399: Interpolate publish credentials as expected for Maven settings.xml
add da4a48d [BEAM-9729, BEAM-8486] Runner-side bundle registration cleanup. (#11358)
add 0a5cb43 Add new release 2.20.0 to beam website (#11285)
add f451643 add 2.20.0 blog post
add 4ebcc59 add a known issue
add 3162ed2 fixup! update 2.20.0 date
add 1d7f640 add 2.20.0 blog post (#11298)
add 0b025e3 Merge pull request #11151 from [BEAM-9468] Hl7v2 io
add 4d7ed73 Fix typo
add c1da629 Merge pull request #11434: Fix typo in WriteToKafka doc string
add a3a9c2d [BEAM-8671] Migrate Load Tests to Python 3.7
add aab7afc Merge pull request #11375: [BEAM-8671] Migrate Load Tests to Python 3.7
add b9de201 Remove outdated doc for ReadFromBigQuery transform
add 209180f Merge pull request #11424: Remove outdated doc for ReadFromBigQuery transform
add 289cb66 Add GBK load tests script for spark structured streaming runner
add 210658a Add CoGBK load tests script for spark structured streaming runner
add 1709a6b Add Combine load tests script for spark structured streaming runner
add 8a14cf0 Add Pardo load tests script for spark structured streaming runner
add 6e8e566 Merge pull request #11135 from echauchot/add_spark-ss-runner-to-GBK-load-tests
add c9d5e93 [BEAM-9765] Exclude module-info.class from vendored Calcite.
add 65c32af Merge pull request #11429 from ibzib/module-info
add 19fe4b5 [BEAM-9678] Create Go SDK introduction kata (#11340)
add 38df510 [BEAM-9769] Ensuring JSON is the default export format for BQ sink
add c21a2a1 Merge pull request #11433 from [BEAM-9769] Ensuring JSON is the default export format for BQ sink
add fc9b349 Merge pull request #11436: Document Beam Schemas (forked PR)
add c3bd485 [BEAM-9648]: DirectRunner should return null on timeout (#11275)
add 7869455 [BEAM-5379] Ignore go.sum files in RAT checks
add 0c86530 Merge pull request #11441 from apache/lostluck-gosum-rat
add 5b41b1f Include user distritribution into metric-dedicated validate runner test.
add 656869f update inline comment.
add 34031ad To allow only assert on committed metrics, bypassing the attempted ones.
add 8dbdce4 Use distribution matcher for testing.
add c7cd259 Merge pull request #11319 from HuangLED/master
add 99444c6 [BEAM-5605, BEAM-2939] Add support for FnApiDoFnRunner to handle split calls. (#11414)
add a1b79fd [BEAM-9674] Don't specify selected fields when fetching BigQuery table size (#11292)
add c38999e [BEAM-9737] Don't use docker create option -u
add 481b1d3 Merge pull request #11386: [BEAM-9737] Fix website postcommit
add 892a0a4 [BEAM-9650] Cleanup documentation on side inputs patterns (#11415)
add cb72032 [BEAM-9768] Gradle command for Python ULR.
add a69d8ec Merge pull request #11430 [BEAM-9768] Gradle command for Python ULR.
add 1b4bf6a [BEAM-8646] Fix external environment on OS X as well. (#11431)
add b6acaef [BEAM-9743] Fix TFRecordCodec to try harder to read/write (#11397)
add cae80ab Correct a typo in documentation
add c60b338 Merge pull request #11398: Correct a typo in documentation
add be57a61 Updates and clarifications on type safety. (#11449)
add 16d23bc [BEAM-9577] Small fixes to portable runner staging.
add d935b5f Merge pull request #11432 [BEAM-9577] Small fixes to portable runner staging.
add 12c66a0 [BEAM-3836] Enable dynamic splitting/checkpointing within the Java SDK harness. (#11448)
add fa4f418 [BEAM-8872] Support split at fraction for OffsetRangeTracker (#11418)
add 494482f [BEAM-9147] Add @Experimental annotation to VideoIntelligence related classes
add b8323c8 Add @Experimental to the package-info as well
add d2089bc Merge pull request #11463 from apache/BEAM-9147-add-experimental-annotation
add 37bb50d [BEAM-9692]: Make CombineValues portable (#11335)
add dee5a0a Add release date for 2.20.0
add d55cab7 Merge pull request #11465 from apache/aaltay-patch-2
add 1e7d2ff Fix user distribution metric test by using less restricting value.
add 1bae13b Apply yapf fix.
add a4e317a Replace all explicit validation.
add 5acd866 Merge pull request #11455 from HuangLED/testFix
add 30fa01f [BEAM-9773]: Update Dataflow Debug Capture to use Google API client Jackson 2
add 1da7b98 Remove all Dataflow google-api-client-jackson(1) dependencies
add b99a3dd Missed cleanup
add 2d98f80 Merge pull request #11442 from stevekoonce/master
add 95db26d [BEAM-6860] Fix iobase non-global windows bug
add a7169f9 Merge pull request #11439: [BEAM-6860] Fix iobase non-global windows bug
add b106e21 [BEAM-9788] Add RetainDockerContainers flag support to Go SDK. (#11467)
add fbf1775 [BEAM-9775] Add Go support for SDF StandardRequirements. (#11443)
add 6060d99 [BEAM-9496] Add to_dataframe and to_pcollection APIs. (#11264)
add 2a931a5 add display:block to datatable so that the columns can have various lengths based on the contents in them.
add fc8d18a Merge pull request #11462 from KevinGG/master
add 37838f2 [BEAM-9569] Fix BeamSqlPojoExample logRecords method to not require Row coder inference
add c7da1b5 Merge pull request #11460: [BEAM-9569] Fix BeamSqlPojoExample logRecords method to not require Row coder inference
add 0988049 Renames run_test -> run_scenario so that nose test runner does not think it's a test method.
add 033c1eb Merge pull request #11457 Rename test-framework-confusing run_test.
add 1b47b2e [BEAM-6960] More sophisticated job server starting and stopping.
add 5878373 Abort pipeline if docker containers die unexpectedly.
add 7181427 [BEAM-6960] Go ULR validates runner tests.
add 488232d lint
add de38f6d Merge pull request #11451 [BEAM-6960] Go ULR validates runner tests.
add 46c8d2d [BEAM-9285] Add Postcommit ValidatesRunner CI Job for Flink on Java 11
add b605c1c Merge pull request #11113: [BEAM-9285] Add Postcommit ValidatesRunner CI Job for Flink on Java 11
add 44638c5 Revert "[BEAM-8131] Provide Kubernetes setup for Prometheus (#9482)" (#11347)
add 1572d57 Merge pull request #11466 from [BEAM-9787] Clear error message on UW + BQSource
add 1905dbd [BEAM-9791] Add precommit for dataflow runner v2
add 5c27e8b Merge pull request #11470 from angoenka/uw_precommit
add 561752b [BEAM-9662][BEAM-9663] Reject Null Limit and Offset
add 8fff6d0 Merge pull request #11471: [BEAM-9662][BEAM-9663] Reject Null Limit and Offset
add 4a7f04c [BEAM-9764] multi threading & optional pulling (#11428)
add 8de324f [BEAM-9608] BundleManagers use BundleContextManager for configuration
add abb7bcb [BEAM-9639] Saving side inputs after producer execution, not before consumption.
add f7ae7f6 [BEAM-9639] Separate Stage and Bundle execution. Improve typing annotations.
add cf821e5 [BEAM-9639][BEAM-9608] Addressing review comments.
add 1fe543e Merge pull request #11270 from [BEAM-9639][BEAM-9608] Improvements for FnApiRunner
add af71782 Fix SplittableParDoNaiveBounded DoFnInvoker.
add c00dc4a Merge pull request #11475 from boyuanzz/fix
add 6e6ff61 [BEAM-9789] Fix lock error. Add test. (#11468)
add 6663da8 Revert "[BEAM-9014] CachingShuffleBatchReader use bytes to limit cache size."
add 623c5ed Merge pull request #11483 from apache/revert-10418-cachesizebytes
add fc37a04 Add a timeout to the TestStream GRPC and fix the Streaming cache timeout
add 7acbff4 Merge pull request #11440 from [BEAM-9767] Add a timeout to the TestStream GRPC and fix the Streaming cache timeout
add 880b062 [BEAM-7981] Correct ParDo(callable) test (#11446)
add d2f91a7 update release 2.20.0 blog post and added contributors name
add e13c1d6 Merge pull request #11476 from mszb/updating_release_blogpost_2_20_0
add 781ff5b Log job id similar to Java and Go
add 75151e3 Merge pull request #11458 from apache/aaltay-patch-1
add 084ac14 [BEAM-6597, BEAM-6467] Add support for reporting monitoring infos to the Java SDK harness. (#11487)
add b091db6 [BEAM-8831] Do not copy generated files for Python source copy
add 7ff0f5a Merge pull request #10230 [BEAM-8831] Exclude generated files for Python source copy
add 6cc6360 Merge pull request #11489 [BEAM-9577] Ensure required directories exist when staging.
add fe3241f [BEAM-9642] Enabling SDF exec runners and fixing small bug. (#11474)
add 66e9040 [BEAM-9755] changed MAKE_ADD stack pop order according to Python version
add 84a914b Merge pull request #11493: [BEAM-9755] changed MAKE_ADD stack pop order according to Python version
add 48db942 [BEAM-9717] update future version to 0.18.2 for Python 3.8 (#11494)
add 76387d9 [BEAM-9577] Use new artifact API to stage go artifacts. (#11490)
add 2f3edaf Force output names to be unicode-compatible on Python 2. (#11485)
add f35cf08 Make license_script.sh tolerate directory already existing (#11501)
add 8cfc8a8 [BEAM-9778] remove temp directory & add docker-pull-licenses to release scripts (#11484)
add b3596b8 [BEAM-8603] Add Python SqlTransform (#10055)
add 5b4ca7b [BEAM-9794] Reduce state cells needed for BufferingDoFnRunner
add 7869d49 Merge pull request #11478: [BEAM-9794] Reduce state cells needed for BufferingDoFnRunner
add ef267d9 [BEAM-9701] Increments fastavro version range upper bound to 0.24. (#11506)
add 93b3a0f [BEAM-2939] Expose HasProgress interface for restriction trackers and use the progress value during splitting
add 7568b06 [BEAM-2939] Add the ability for SDK harness runners to provide additional monitoring infos.
add 7ac97ea [BEAM-2939] Integrate progress reporting for splittable dofns.
add a02eb60 [BEAM-2939] Fold Sizes sub-interfaces into RestrictionTracker
add cfa6692 [BEAM-2939] Drop HasSize in favor of using HasProgress#getProgress#getWorkRemaining as the default if the DoFn doesn't override GetSize.
add 0291976 Merge pull request #11472 from lukecwik/splittabledofn
add f813a90 [BEAM-9577] New artifact staging and retrieval service for Java.
add 9c8718e [BEAM-9577] Multi-threaded artifact staging service backend.
add 01af3aa Use futures, better error handling.
add ec67a93 Merge pull request #11342 [BEAM-9577] New artifact staging and retrieval service for Java.
add 9867aab [BEAM-9812] Fixing bug causing pipelines requiring temp tables to not work
add decccbe Merge pull request #11509 from [BEAM-9812] Fixing bug causing pipelines requiring temp tables to not work
add 55b82fc [BEAM-9514] Ensure nullability passes through sum
add a905f69 Merge pull request #11488: [BEAM-9514] Ensure nullability passes through aggregate
add 71ccd7a [BEAM-7405] Workaround for bad Docker config
add 5761e33 Merge pull request #11512: [BEAM-7405] Workaround for bad Docker config on Jenkins nodes.
add 89958d8 [BEAM-8871] Support trySplit for ByteKeyRangeTracker
add 28fd597 Merge pull request #11454 from boyuanzz/restriction_tracker
add d8519dd [BEAM-9811] Nightly snapshot publish error
add e55a7b3 [BEAM-9775] Adding Go SDF example, adjusting GetProgress signature. (#11499)
add d416fd1 [BEAM-9733] Always let ImpulseSourceFunction emit a final watermark
add 643945a [BEAM-9733] Make up for timers set while processing the bundle
add 4797f31 Merge pull request #11362: [BEAM-9733] Improve watermark and timer handling
add fd74d4d [BEAM-9813] Replace outputs with respect to the correct tag.
add 5262732 Merge pull request #11515: [BEAM-9813] Replace outputs with respect to the correct tag.
add 5d350cb [BEAM-9761] Disable auto builds for tests that are currently failing.
add 022fe5e Merge pull request #11519: [BEAM-9761] Disable auto builds for tests that are currently failing.
add 6d29fb8 [BEAM-9811] Copy pom for all publishMavenJavaPublication tasks
add 1321104 add licenses and license types for zetasql packages (#11516)
add 24361d1 Added a batch example with covid tracking data for interactive notebook. (#11469)
add 4a52fb4 [BEAM-9488] Ensure we pass through PCollection ids instead of attempting to fix them up. (#11514)
add 3777b79 Fix invalid formatting specifier in verify
add 7fd4008 Merge pull request #11525: Fix invalid formatting specifier in verify
add 5fab78a [BEAM-9820] Upgrade Flink 1.9.x to 1.9.3
add b8085ad Merge pull request #11526: [BEAM-9820] Upgrade Flink 1.9.x to 1.9.3
add 9ed5a92 [BEAM-9819] Relaxes the version range for httplib2 to <0.16.0. (#11522)
add ae6ef78 [BEAM-9646] Add Google Cloud vision integration transform
add 1a33eef Fix spotbugs errors
add 86175c8 Add @Experimental annotation, make parent class package-private
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (6beafe0)
\
N -- N -- N refs/heads/BEAM-9646-cloud-vision (86175c8)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 2 +-
.test-infra/dataproc/flink_cluster.sh | 1 -
.test-infra/dataproc/init-actions/flink.sh | 6 +-
.test-infra/jenkins/CommonTestProperties.groovy | 2 +-
.test-infra/jenkins/LoadTestConfig.groovy | 5 +-
.../jenkins/NexmarkBigqueryProperties.groovy | 1 -
.../jenkins/job_LoadTests_CoGBK_Java.groovy | 4 +
...s_CoGBK_Java_spark_structured_streaming.groovy} | 64 +-
.../job_LoadTests_Combine_Flink_Python.groovy | 6 +-
.../jenkins/job_LoadTests_Combine_Java.groovy | 3 +
...Combine_Java_spark_structured_streaming.groovy} | 60 +-
.../jenkins/job_LoadTests_Combine_Python.groovy | 5 +-
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 10 +-
.test-infra/jenkins/job_LoadTests_GBK_Java.groovy | 7 +
.../jenkins/job_LoadTests_GBK_Java_Smoke.groovy | 1 +
...sts_GBK_Java_spark_structured_streaming.groovy} | 93 +-
.../jenkins/job_LoadTests_GBK_Python.groovy | 9 +-
.../job_LoadTests_GBK_Python_reiterate.groovy | 4 +-
.../job_LoadTests_ParDo_Flink_Python.groovy | 6 +-
.../jenkins/job_LoadTests_ParDo_Java.groovy | 4 +
...s_ParDo_Java_spark_structured_streaming.groovy} | 64 +-
.../jenkins/job_LoadTests_ParDo_Python.groovy | 6 +-
.../jenkins/job_LoadTests_ParDo_Python_37.groovy | 71 --
.../jenkins/job_LoadTests_Python_Smoke.groovy | 3 +-
.../job_LoadTests_coGBK_Flink_Python.groovy | 10 +-
.../jenkins/job_LoadTests_coGBK_Python.groovy | 6 +-
.../job_PerformanceTests_BigQueryIO_Python.groovy | 4 +-
.../jenkins/job_PerformanceTests_Python.groovy | 1 +
.../job_PostCommit_Java_Nexmark_Dataflow.groovy | 5 +
.../job_PostCommit_Java_Nexmark_Flink.groovy | 12 +-
...ob_PostCommit_Java_ValidatesRunner_Flink.groovy | 2 +-
...ommit_Java_ValidatesRunner_Flink_Java11.groovy} | 38 +-
.test-infra/metrics/README.md | 31 +-
.test-infra/metrics/apply_configmaps.sh | 26 -
.test-infra/metrics/beamprometheus-deploy.yaml | 125 --
.test-infra/metrics/build.gradle | 13 +
.test-infra/metrics/docker-compose.yml | 28 +-
.../alertmanager/config/alertmanager.yml | 41 -
.../prometheus/prometheus/config/prometheus.yml | 40 -
.../metrics/prometheus/prometheus/config/rules.yml | 35 -
CHANGES.md | 57 +-
build.gradle | 6 +
buildSrc/build.gradle | 1 +
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 75 +-
.../org/apache/beam/gradle/Repositories.groovy | 4 +
.../org/apache/beam/gradle/VendorJavaPlugin.groovy | 17 +-
gradle.properties | 4 +-
learning/katas/README.md | 2 +-
.../Introduction/Hello Beam/Hello Beam Test/go.mod | 47 +-
.../Introduction/Hello Beam/Hello Beam Test/go.sum | 58 +
.../Hello Beam/Hello Beam Test/pkg/task/task.go | 44 +-
.../Hello Beam/Hello Beam Test/task-info.yaml | 36 +
.../Hello Beam Test/task-remote-info.yaml | 2 +
.../Hello Beam/Hello Beam Test/task.md | 41 +
.../Hello Beam/Hello Beam Test/test/task_test.go | 49 +-
.../Introduction/Hello Beam/Hello Beam/cmd/main.go | 48 +-
.../go/Introduction/Hello Beam/Hello Beam/go.mod | 49 +-
.../go/Introduction/Hello Beam/Hello Beam/go.sum | 119 ++
.../Hello Beam/Hello Beam/pkg/task/task.go | 44 +-
.../Hello Beam/Hello Beam/task-info.yaml | 35 +
.../Hello Beam/Hello Beam/task-remote-info.yaml | 2 +
.../go/Introduction/Hello Beam/Hello Beam/task.md | 51 +
.../Hello Beam/Hello Beam/test/task_test.go | 49 +-
.../go/Introduction/Hello Beam/lesson-info.yaml | 22 +
.../Hello Beam/lesson-remote-info.yaml | 3 +
learning/katas/go/Introduction/section-info.yaml | 21 +
.../katas/go/Introduction/section-remote-info.yaml | 2 +
learning/katas/{ => go}/README.md | 17 +-
learning/katas/go/course-info.yaml | 26 +
learning/katas/go/course-remote-info.yaml | 2 +
.../fn-execution/src/main/proto/beam_fn_api.proto | 14 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 76 +-
ownership/PYTHON_DEPENDENCY_OWNERS.yaml | 6 -
.../src/main/scripts/build_release_candidate.sh | 2 +-
.../core/construction/ArtifactResolver.java | 51 +
.../core/construction/DefaultArtifactResolver.java | 93 ++
.../runners/core/construction/Environments.java | 127 ++-
.../core/construction/ParDoTranslation.java | 90 +-
.../runners/core/construction/SplittableParDo.java | 6 -
.../construction/SplittableParDoNaiveBounded.java | 7 +-
.../beam/runners/core/construction/Timer.java | 33 +-
.../construction/DefaultArtifactResolverTest.java | 131 +++
.../core/construction/EnvironmentsTest.java | 7 +
.../core/construction/ParDoTranslationTest.java | 58 +-
...TimeBoundedSplittableProcessElementInvoker.java | 31 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 27 +-
.../core/SplittableProcessElementInvoker.java | 9 +-
.../core/metrics/MonitoringInfoConstants.java | 2 +
.../runners/core/SplittableParDoProcessFnTest.java | 3 +-
.../direct/ExecutorServiceParallelExecutor.java | 36 +-
.../beam/runners/direct/DirectRunnerTest.java | 2 +-
runners/flink/1.9/build.gradle | 2 +-
runners/flink/flink_runner.gradle | 5 +-
.../runners/flink/FlinkExecutionEnvironments.java | 4 +
.../beam/runners/flink/FlinkPipelineOptions.java | 7 +
.../functions/FlinkExecutableStageFunction.java | 57 +-
.../functions/ImpulseSourceFunction.java | 12 +
.../flink/translation/utils/Workarounds.java | 25 +
.../wrappers/streaming/DoFnOperator.java | 271 +++--
.../streaming/ExecutableStageDoFnOperator.java | 324 ++++--
.../wrappers/streaming/SplittableDoFnOperator.java | 15 +-
.../wrappers/streaming/WindowDoFnOperator.java | 9 +-
.../streaming/stableinput/BufferingDoFnRunner.java | 97 +-
.../runners/flink/FlinkPipelineOptionsTest.java | 1 +
.../beam/runners/flink/FlinkSavepointTest.java | 34 +-
.../runners/flink/PortableTimersExecutionTest.java | 35 +-
.../FlinkExecutableStageFunctionTest.java | 14 +
.../functions/ImpulseSourceFunctionTest.java | 17 +-
.../wrappers/streaming/DoFnOperatorTest.java | 10 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 311 ++++-
.../wrappers/streaming/WindowDoFnOperatorTest.java | 4 +-
.../stableinput/BufferingDoFnRunnerTest.java | 171 +++
.../beam/runners/dataflow/DataflowRunner.java | 18 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 38 +-
.../dataflow/DataflowPipelineTranslatorTest.java | 18 -
.../beam/runners/dataflow/DataflowRunnerTest.java | 25 +
.../google-cloud-dataflow-java/worker/build.gradle | 2 +-
.../worker/ApplianceShuffleEntryReader.java | 6 +-
.../dataflow/worker/BatchDataflowWorker.java | 9 +-
.../worker/BeamFnMapTaskExecutorFactory.java | 11 -
.../worker/ChunkingShuffleBatchReader.java | 8 +-
.../dataflow/worker/FnApiWindowMappingFn.java | 4 +-
.../dataflow/worker/StreamingDataflowWorker.java | 9 +-
.../fn/control/ProcessRemoteBundleOperation.java | 22 +-
.../dataflow/worker/fn/control/TimerReceiver.java | 268 -----
.../worker/fn/data/BeamFnDataGrpcService.java | 2 +-
.../fn/data/RemoteGrpcPortReadOperation.java | 2 +-
.../fn/data/RemoteGrpcPortWriteOperation.java | 2 +-
.../worker/graph/RegisterNodeFunction.java | 16 +-
.../JulHandlerPrintStreamAdapterFactory.java | 44 +-
.../dataflow/worker/status/DebugCapture.java | 9 +-
.../common/worker/CachingShuffleBatchReader.java | 61 +-
.../util/common/worker/ShuffleBatchReader.java | 5 +-
.../dataflow/worker/FnApiWindowMappingFnTest.java | 4 +
.../fn/control/BeamFnMapTaskExecutorTest.java | 4 +
.../RegisterAndProcessBundleOperationTest.java | 44 +-
.../worker/fn/control/TimerReceiverTest.java | 543 ---------
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 6 +-
.../fn/data/RemoteGrpcPortReadOperationTest.java | 8 +-
.../fn/data/RemoteGrpcPortWriteOperationTest.java | 6 +-
.../graph/LengthPrefixUnknownCodersTest.java | 2 +-
.../JulHandlerPrintStreamAdapterFactoryTest.java | 21 +
.../dataflow/worker/status/DebugCaptureTest.java | 2 +-
.../worker/BatchingShuffleEntryReaderTest.java | 18 +-
.../worker/CachingShuffleBatchReaderTest.java | 4 +-
.../artifact/ArtifactRetrievalService.java | 114 ++
.../artifact/ArtifactStagingService.java | 616 ++++++++++
.../control/DefaultJobBundleFactory.java | 44 +-
.../fnexecution/control/FnApiControlClient.java | 13 +-
.../control/InstructionRequestHandler.java | 2 +
.../fnexecution/control/OutputReceiverFactory.java | 2 +-
.../control/ProcessBundleDescriptors.java | 195 ++--
.../runners/fnexecution/control/RemoteBundle.java | 16 +
.../fnexecution/control/SdkHarnessClient.java | 271 +++--
.../SingleEnvironmentInstanceJobBundleFactory.java | 21 +-
.../fnexecution/control/StageBundleFactory.java | 10 +
.../fnexecution/control/TimerReceiverFactory.java | 91 +-
.../runners/fnexecution/data/GrpcDataService.java | 2 +-
.../translation/PipelineTranslatorUtils.java | 27 +-
.../artifact/ArtifactRetrievalServiceTest.java | 132 +++
.../artifact/ArtifactStagingServiceTest.java | 173 +++
.../control/FnApiControlClientTest.java | 19 -
.../fnexecution/control/RemoteExecutionTest.java | 710 ++++++++----
.../fnexecution/control/SdkHarnessClientTest.java | 99 +-
.../control/TimerReceiverFactoryTest.java} | 24 +-
.../fnexecution/data/GrpcDataServiceTest.java | 4 +-
.../beam/runners/portability/PortableRunner.java | 2 +
.../translation/SparkExecutableStageFunction.java | 72 +-
.../SparkExecutableStageFunctionTest.java | 20 +-
sdks/go/README.md | 2 +-
sdks/go/cmd/beamctl/cmd/artifact.go | 8 +-
sdks/go/cmd/beamctl/cmd/provision.go | 6 +-
.../stringsplit/offsetrange/offsetrange.go | 119 ++
sdks/go/examples/stringsplit/stringsplit.go | 246 ++++
sdks/go/pkg/beam/artifact/gcsproxy/retrieval.go | 20 +-
sdks/go/pkg/beam/artifact/gcsproxy/staging.go | 20 +-
sdks/go/pkg/beam/artifact/materialize.go | 64 +-
sdks/go/pkg/beam/artifact/materialize_test.go | 81 +-
sdks/go/pkg/beam/artifact/server_test.go | 26 +-
sdks/go/pkg/beam/artifact/stage.go | 32 +-
sdks/go/pkg/beam/artifact/stage_test.go | 12 +-
sdks/go/pkg/beam/core/funcx/fn.go | 7 +-
sdks/go/pkg/beam/core/graph/fn.go | 21 +-
sdks/go/pkg/beam/core/graph/fn_test.go | 4 +-
sdks/go/pkg/beam/core/graph/mtime/time.go | 6 +-
sdks/go/pkg/beam/core/runtime/exec/coder.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/fn.go | 12 +-
sdks/go/pkg/beam/core/runtime/exec/pardo.go | 17 +-
sdks/go/pkg/beam/core/runtime/exec/reshuffle.go | 1 -
sdks/go/pkg/beam/core/runtime/exec/sdf.go | 315 +++++
.../beam/core/runtime/exec/sdf_invokers_test.go | 36 +-
sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 309 +++++
sdks/go/pkg/beam/core/runtime/exec/translate.go | 96 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 243 ++--
.../pkg/beam/core/runtime/graphx/translate_test.go | 54 +-
sdks/go/pkg/beam/core/runtime/harness/datamgr.go | 29 +-
.../pkg/beam/core/runtime/harness/datamgr_test.go | 14 +-
sdks/go/pkg/beam/core/runtime/harness/harness.go | 89 +-
.../pkg/beam/core/runtime/harness/harness_test.go | 165 +++
sdks/go/pkg/beam/core/runtime/harness/logging.go | 30 +-
.../pkg/beam/core/runtime/harness/logging_test.go | 6 +-
.../go/pkg/beam/core/runtime/harness/monitoring.go | 26 +-
sdks/go/pkg/beam/core/runtime/harness/session.go | 12 +-
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 58 +-
.../pkg/beam/core/runtime/harness/statemgr_test.go | 210 +++-
sdks/go/pkg/beam/core/runtime/options.go | 9 +-
sdks/go/pkg/beam/core/runtime/pipelinex/clone.go | 3 +-
sdks/go/pkg/beam/core/runtime/pipelinex/replace.go | 30 +-
.../beam/core/runtime/pipelinex/replace_test.go | 22 +-
sdks/go/pkg/beam/core/sdf/sdf.go | 7 +-
.../beam/model/fnexecution_v1/beam_fn_api.pb.go | 688 ++++++-----
.../beam/model/pipeline_v1/beam_runner_api.pb.go | 903 ++++++++-------
sdks/go/pkg/beam/options/jobopts/options.go | 4 +
sdks/go/pkg/beam/provision/provision.go | 20 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 7 +-
.../pkg/beam/runners/dataflow/dataflowlib/job.go | 4 +-
.../beam/runners/dataflow/dataflowlib/translate.go | 26 +-
sdks/go/pkg/beam/runners/session/session.go | 30 +-
.../beam/runners/universal/runnerlib/execute.go | 4 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 14 +-
.../pkg/beam/runners/universal/runnerlib/stage.go | 102 +-
sdks/go/pkg/beam/runners/universal/universal.go | 7 +-
sdks/go/pkg/beam/testing/passert/equals.go | 100 ++
sdks/go/pkg/beam/testing/passert/equals_test.go | 99 ++
sdks/go/pkg/beam/testing/passert/passert.go | 25 +-
sdks/go/pkg/beam/testing/passert/passert.shims.go | 28 +
sdks/go/test/build.gradle | 24 +
sdks/go/test/run_integration_tests.sh | 93 +-
sdks/java/bom/build.gradle | 4 +-
sdks/java/container/Dockerfile | 2 +
sdks/java/container/build.gradle | 36 +-
.../container/license_scripts/dep_urls_java.yaml | 203 ++++
.../container/license_scripts/license_script.sh | 53 +
.../license_scripts/pull_licenses_java.py | 323 ++++++
.../src/main/java/org/apache/beam/sdk/io/Read.java | 200 ++--
.../java/org/apache/beam/sdk/io/TFRecordIO.java | 55 +-
.../java/org/apache/beam/sdk/io/WriteFiles.java | 43 +-
.../org/apache/beam/sdk/io/range/ByteKeyRange.java | 2 +-
.../main/java/org/apache/beam/sdk/state/Timer.java | 22 +
.../java/org/apache/beam/sdk/transforms/DoFn.java | 27 +-
.../beam/sdk/transforms/FlatMapElements.java | 2 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 13 +-
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 12 +-
.../splittabledofn/ByteKeyRangeTracker.java | 98 +-
.../splittabledofn/OffsetRangeTracker.java | 35 +-
.../splittabledofn/RestrictionTracker.java | 76 +-
.../beam/sdk/transforms/splittabledofn/Sizes.java | 54 -
.../org/apache/beam/sdk/io/TFRecordIOTest.java | 130 +++
.../apache/beam/sdk/io/range/ByteKeyRangeTest.java | 7 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 98 +-
.../splittabledofn/ByteKeyRangeTrackerTest.java | 132 ++-
.../splittabledofn/OffsetRangeTrackerTest.java | 80 +-
.../sdk/extensions/gcp/options/GcpOptions.java | 8 +-
.../beam/sdk/extensions/gcp/util/GcsUtil.java | 18 +-
sdks/java/extensions/ml/build.gradle | 2 -
.../beam/sdk/extensions/ml/AnnotateImages.java | 4 +-
.../beam/sdk/extensions/ml/AnnotateVideo.java | 105 ++
.../apache/beam/sdk/extensions/ml/CloudVision.java | 2 +
.../beam/sdk/extensions/ml/VideoIntelligence.java | 190 +++
.../beam/sdk/extensions/ml/package-info.java | 3 +
.../beam/sdk/extensions/ml/AnnotateVideoTest.java | 73 ++
.../sdk/extensions/ml/VideoIntelligenceIT.java | 83 ++
sdks/java/extensions/sql/build.gradle | 2 -
.../extensions/sql/expansion-service/build.gradle | 40 +
.../beam/sdk/extensions/sql/SqlTransform.java | 39 +-
.../extensions/sql/example/BeamSqlPojoExample.java | 8 +-
.../sdk/extensions/sql/example/model/Customer.java | 34 +-
.../sdk/extensions/sql/example/model/Order.java | 5 +
sdks/java/extensions/sql/zetasql/build.gradle | 2 +-
.../translation/AggregateScanConverter.java | 28 +-
.../LimitOffsetScanToLimitConverter.java | 6 +
.../LimitOffsetScanToOrderByLimitConverter.java | 5 +
.../sql/zetasql/ZetaSQLDialectSpecTest.java | 41 +
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 66 +-
.../sdk/fn/data/BeamFnDataInboundObserver.java | 35 +-
...amFnDataSizeBasedBufferingOutboundObserver.java | 38 +-
.../apache/beam/sdk/fn/data/LogicalEndpoint.java | 20 +-
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 18 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 49 +-
...DataSizeBasedBufferingOutboundObserverTest.java | 148 +--
...DataTimeBasedBufferingOutboundObserverTest.java | 84 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 14 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 7 +-
.../beam/fn/harness/BeamFnDataWriteRunner.java | 5 +-
.../beam/fn/harness/BoundedSourceRunner.java | 3 +
.../org/apache/beam/fn/harness/CombineRunners.java | 3 +
.../org/apache/beam/fn/harness/FlattenRunner.java | 3 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 564 ++++++---
.../java/org/apache/beam/fn/harness/FnHarness.java | 38 +-
.../org/apache/beam/fn/harness/HandlesSplits.java | 8 +
.../org/apache/beam/fn/harness/MapFnRunners.java | 3 +
.../beam/fn/harness/PTransformRunnerFactory.java | 17 +
.../fn/harness/control/BeamFnControlClient.java | 19 +-
.../fn/harness/control/BundleSplitListener.java | 35 +-
.../fn/harness/control/ProcessBundleHandler.java | 231 ++--
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 12 +-
.../beam/fn/harness/data/BeamFnTimerClient.java | 52 +
.../fn/harness/data/BeamFnTimerGrpcClient.java | 99 ++
.../beam/fn/harness/state/FnApiStateAccessor.java | 22 +-
.../beam/fn/harness/AssignWindowsRunnerTest.java | 2 +
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 16 +-
.../beam/fn/harness/BeamFnDataWriteRunnerTest.java | 8 +-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 2 +
.../apache/beam/fn/harness/CombineRunnersTest.java | 60 +-
.../apache/beam/fn/harness/FlattenRunnerTest.java | 4 +
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 493 +++++---
.../apache/beam/fn/harness/MapFnRunnersTest.java | 6 +
.../harness/control/BundleSplitListenerTest.java | 57 +
.../harness/control/ProcessBundleHandlerTest.java | 110 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 4 +-
.../data/BeamFnDataInboundObserverTest.java | 22 +-
.../fn/harness/data/FakeBeamFnTimerClient.java | 115 ++
.../harness/data/QueueingBeamFnDataClientTest.java | 4 +-
sdks/java/io/google-cloud-platform/build.gradle | 1 +
.../sdk/io/gcp/bigquery/BigQueryAvroUtils.java | 37 +
.../io/gcp/bigquery/BigQueryStorageSourceBase.java | 7 +-
.../gcp/bigquery/BigQueryStorageTableSource.java | 12 +-
.../sdk/io/gcp/bigquery/DynamicDestinations.java | 6 +-
.../apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java | 685 +++++++++++
.../beam/sdk/io/gcp/healthcare/HL7v2Message.java | 213 ++++
.../sdk/io/gcp/healthcare/HL7v2MessageCoder.java | 65 ++
.../sdk/io/gcp/healthcare/HealthcareApiClient.java | 152 +++
.../sdk/io/gcp/healthcare/HealthcareIOError.java | 64 ++
.../io/gcp/healthcare/HealthcareIOErrorCoder.java | 59 +
.../healthcare/HealthcareIOErrorToTableRow.java | 91 ++
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 460 ++++++++
.../beam/sdk/io/gcp/healthcare}/package-info.java | 13 +-
.../apache/beam/sdk/io/gcp/GcpApiSurfaceTest.java | 1 +
.../gcp/bigquery/BigQueryIOStorageQueryTest.java | 67 +-
.../io/gcp/bigquery/BigQueryIOStorageReadTest.java | 183 ++-
.../io/gcp/healthcare/HL7V2MessagePagesTest.java | 95 ++
.../beam/sdk/io/gcp/healthcare/HL7v2IOReadIT.java | 196 ++++
.../beam/sdk/io/gcp/healthcare/HL7v2IOTest.java | 81 ++
.../sdk/io/gcp/healthcare/HL7v2IOTestUtil.java | 185 +++
.../beam/sdk/io/gcp/healthcare/HL7v2IOWriteIT.java | 93 ++
.../healthcare/HttpHealthcareApiClientTest.java | 56 +
.../java/org/apache/beam/sdk/io/mqtt/MqttIO.java | 5 +
sdks/python/apache_beam/coders/coders.py | 8 +-
sdks/python/apache_beam/dataframe/convert.py | 98 ++
sdks/python/apache_beam/dataframe/convert_test.py | 64 ++
sdks/python/apache_beam/dataframe/expressions.py | 23 +-
sdks/python/apache_beam/dataframe/frames.py | 6 +-
sdks/python/apache_beam/dataframe/transforms.py | 291 +++++
.../apache_beam/dataframe/transforms_test.py | 134 +++
.../examples/cookbook/datastore_wordcount.py | 131 +--
.../cookbook/datastore_wordcount_it_test.py | 19 +-
.../apache_beam/examples/snippets/snippets.py | 71 +-
.../apache_beam/examples/snippets/snippets_test.py | 85 +-
.../examples/snippets/snippets_test_py3.py | 21 +-
.../apache_beam/examples/wordcount_xlang_sql.py | 118 ++
sdks/python/apache_beam/io/external/kafka.py | 2 +-
.../io/gcp/big_query_query_to_table_pipeline.py | 2 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 41 +-
.../apache_beam/io/gcp/bigquery_avro_tools.py | 1 +
.../apache_beam/io/gcp/bigquery_avro_tools_test.py | 7 +-
.../apache_beam/io/gcp/bigquery_file_loads.py | 6 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 4 +-
.../io/gcp/bigquery_io_read_pipeline.py | 6 +-
.../apache_beam/io/gcp/bigquery_read_it_test.py | 4 +-
sdks/python/apache_beam/io/gcp/bigquery_test.py | 10 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 6 +-
.../apache_beam/io/gcp/datastore/v1/datastoreio.py | 516 ---------
.../io/gcp/datastore/v1/datastoreio_test.py | 313 -----
.../io/gcp/datastore/v1/fake_datastore.py | 111 --
.../apache_beam/io/gcp/datastore/v1/helper.py | 335 ------
.../apache_beam/io/gcp/datastore/v1/helper_test.py | 296 -----
.../io/gcp/datastore/v1/query_splitter.py | 283 -----
.../io/gcp/datastore/v1/query_splitter_test.py | 244 ----
.../datastore/{v1 => v1new}/adaptive_throttler.py | 2 +-
.../{v1 => v1new}/adaptive_throttler_test.py | 2 +-
.../gcp/datastore/v1new/datastore_write_it_test.py | 3 +-
.../io/gcp/datastore/v1new/datastoreio.py | 16 +-
.../io/gcp/datastore/v1new/datastoreio_test.py | 60 +-
.../io/gcp/datastore/v1new/query_splitter_test.py | 102 +-
.../apache_beam/io/gcp/datastore/v1new/types.py | 20 +-
.../io/gcp/datastore/v1new/types_test.py | 3 +-
.../io/gcp/datastore/{v1 => v1new}/util.py | 0
.../io/gcp/datastore/{v1 => v1new}/util_test.py | 2 +-
.../io/gcp/datastore_write_it_pipeline.py | 213 ----
.../apache_beam/io/gcp/datastore_write_it_test.py | 86 --
.../hdfs_integration_test/hdfs_integration_test.sh | 5 +
sdks/python/apache_beam/io/iobase.py | 2 +-
sdks/python/apache_beam/io/localfilesystem.py | 3 +
sdks/python/apache_beam/io/textio_test.py | 23 +-
sdks/python/apache_beam/metrics/metric_test.py | 21 +
.../python/apache_beam/metrics/monitoring_infos.py | 60 +-
.../python/apache_beam/options/pipeline_options.py | 3 +-
.../options/pipeline_options_validator.py | 7 +-
.../options/pipeline_options_validator_test.py | 27 +
sdks/python/apache_beam/pipeline.py | 57 +-
sdks/python/apache_beam/runners/common.py | 10 +-
.../runners/dataflow/dataflow_runner.py | 65 +-
.../runners/dataflow/dataflow_runner_test.py | 30 +-
.../runners/dataflow/internal/apiclient.py | 21 +-
.../runners/dataflow/internal/apiclient_test.py | 2 +-
.../runners/dataflow/ptransform_overrides.py | 35 +
.../apache_beam/runners/direct/direct_userstate.py | 2 +-
.../apache_beam/runners/direct/test_stream_impl.py | 20 +-
.../runners/interactive/background_caching_job.py | 58 +-
.../interactive/background_caching_job_test.py | 4 +-
.../runners/interactive/caching/streaming_cache.py | 11 +-
.../interactive/display/pcoll_visualization.py | 2 +-
.../interactive/examples/UsCovidDataExample.ipynb | 511 +++++++++
.../runners/interactive/interactive_runner_test.py | 25 +-
.../runners/interactive/options/capture_control.py | 20 +-
.../interactive/options/capture_control_test.py | 68 +-
.../interactive/options/capture_limiters.py | 70 ++
.../interactive/testing/integration}/__init__.py | 0
.../goldens/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 0 -> 754885 bytes
.../testing/integration/notebook_executor.py | 141 +++
.../interactive/testing/integration/screen_diff.py | 227 ++++
.../test_notebooks/init_square_cube.ipynb | 178 +++
.../integration/tests/init_square_cube_test.py} | 17 +
.../runners/portability/artifact_service.py | 36 +-
.../runners/portability/artifact_service_test.py | 7 +-
.../runners/portability/fn_api_runner/execution.py | 219 +++-
.../runners/portability/fn_api_runner/fn_runner.py | 512 ++++-----
.../portability/fn_api_runner/fn_runner_test.py | 25 +
.../portability/fn_api_runner/translations.py | 85 +-
.../portability/fn_api_runner/worker_handlers.py | 63 +-
.../runners/portability/local_job_service.py | 39 +-
.../runners/portability/local_job_service_main.py | 113 +-
.../runners/portability/portable_runner.py | 65 +-
.../apache_beam/runners/worker/bundle_processor.py | 272 ++---
.../apache_beam/runners/worker/data_plane.py | 201 +++-
.../apache_beam/runners/worker/operations.pxd | 4 +-
.../apache_beam/runners/worker/operations.py | 126 +-
.../apache_beam/runners/worker/sdk_worker.py | 1 -
.../apache_beam/runners/worker/worker_pool_main.py | 6 +-
.../testing/benchmarks/chicago_taxi/preprocess.py | 2 +-
.../benchmarks/chicago_taxi/process_tfma.py | 2 +-
.../chicago_taxi/tfdv_analyze_and_validate.py | 2 +-
.../testing/load_tests/co_group_by_key_test.py | 2 +
.../apache_beam/testing/load_tests/combine_test.py | 2 +
.../testing/load_tests/group_by_key_test.py | 2 +
.../apache_beam/testing/load_tests/pardo_test.py | 2 +
.../apache_beam/testing/synthetic_pipeline.py | 48 +-
sdks/python/apache_beam/transforms/core.py | 47 +-
sdks/python/apache_beam/transforms/environments.py | 11 +-
sdks/python/apache_beam/transforms/external.py | 2 +-
.../apache_beam/transforms/periodicsequence.py | 192 ++++
.../transforms/periodicsequence_test.py | 102 ++
sdks/python/apache_beam/transforms/ptransform.py | 24 +-
sdks/python/apache_beam/transforms/sql.py | 74 ++
sdks/python/apache_beam/transforms/sql_test.py | 131 +++
sdks/python/apache_beam/transforms/trigger.py | 45 +-
sdks/python/apache_beam/transforms/trigger_test.py | 34 +
sdks/python/apache_beam/transforms/userstate.py | 12 +-
sdks/python/apache_beam/transforms/util.py | 5 +-
sdks/python/apache_beam/typehints/opcodes.py | 10 +-
.../apache_beam/typehints/typed_pipeline_test.py | 5 +-
.../typehints/typed_pipeline_test_py3.py | 17 +-
sdks/python/apache_beam/utils/annotations_test.py | 6 +
sdks/python/apache_beam/utils/subprocess_server.py | 2 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/build-requirements.txt | 2 +-
sdks/python/build.gradle | 12 +
sdks/python/container/base_image_requirements.txt | 3 +-
.../container/license_scripts/dep_urls_py.yaml | 2 +
sdks/python/scripts/generate_pydoc.sh | 5 +-
sdks/python/scripts/run_integration_test.sh | 2 -
sdks/python/setup.py | 24 +-
sdks/python/test-suites/dataflow/py2/build.gradle | 87 +-
sdks/python/test-suites/dataflow/py37/build.gradle | 87 +-
sdks/python/tox.ini | 6 -
settings.gradle | 2 +
vendor/calcite-1_20_0/build.gradle | 3 +-
website/_config.yml | 2 +-
website/build.gradle | 10 +-
website/src/.htaccess | 2 +-
website/src/_data/authors.yml | 4 +
.../src/_includes/flink_java_pipeline_options.html | 5 +
.../_includes/flink_python_pipeline_options.html | 5 +
.../src/_includes/section-menu/documentation.html | 12 +
website/src/_posts/2020-04-15-beam-2.20.0.md | 77 ++
website/src/contribute/release-guide.md | 2 +-
website/src/documentation/io/built-in-hcatalog.md | 2 +-
website/src/documentation/patterns/side-inputs.md | 45 +-
.../pipelines/create-your-pipeline.md | 2 +-
website/src/documentation/programming-guide.md | 1205 +++++++++++++++++++-
website/src/documentation/runtime/environments.md | 57 +-
.../src/documentation/sdks/python-type-safety.md | 122 +-
website/src/get-started/downloads.md | 13 +-
website/src/get-started/quickstart-go.md | 1 +
website/src/get-started/wordcount-example.md | 2 +
485 files changed, 20820 insertions(+), 9350 deletions(-)
copy .test-infra/jenkins/{job_LoadTests_CoGBK_Java.groovy => job_LoadTests_CoGBK_Java_spark_structured_streaming.groovy} (78%)
copy .test-infra/jenkins/{job_LoadTests_Combine_Java.groovy => job_LoadTests_Combine_Java_spark_structured_streaming.groovy} (71%)
copy .test-infra/jenkins/{job_LoadTests_GBK_Java.groovy => job_LoadTests_GBK_Java_spark_structured_streaming.groovy} (77%)
copy .test-infra/jenkins/{job_LoadTests_ParDo_Java.groovy => job_LoadTests_ParDo_Java_spark_structured_streaming.groovy} (72%)
delete mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Python_37.groovy
copy .test-infra/jenkins/{job_PostCommit_Java_ValidatesRunner_Flink.groovy => job_PostCommit_Java_ValidatesRunner_Flink_Java11.groovy} (53%)
delete mode 100755 .test-infra/metrics/apply_configmaps.sh
delete mode 100644 .test-infra/metrics/beamprometheus-deploy.yaml
delete mode 100644 .test-infra/metrics/prometheus/alertmanager/config/alertmanager.yml
delete mode 100644 .test-infra/metrics/prometheus/prometheus/config/prometheus.yml
delete mode 100644 .test-infra/metrics/prometheus/prometheus/config/rules.yml
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam Test/go.mod (50%)
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam Test/go.sum
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam Test/pkg/task/task.go (51%)
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam Test/task-info.yaml
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam Test/task-remote-info.yaml
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam Test/task.md
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam Test/test/task_test.go (52%)
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam/cmd/main.go (52%)
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam/go.mod (50%)
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam/go.sum
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam/pkg/task/task.go (51%)
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam/task-info.yaml
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam/task-remote-info.yaml
create mode 100644 learning/katas/go/Introduction/Hello Beam/Hello Beam/task.md
copy sdks/go/cmd/beamctl/cmd/provision.go => learning/katas/go/Introduction/Hello Beam/Hello Beam/test/task_test.go (52%)
create mode 100644 learning/katas/go/Introduction/Hello Beam/lesson-info.yaml
create mode 100644 learning/katas/go/Introduction/Hello Beam/lesson-remote-info.yaml
create mode 100644 learning/katas/go/Introduction/section-info.yaml
create mode 100644 learning/katas/go/Introduction/section-remote-info.yaml
copy learning/katas/{ => go}/README.md (60%)
create mode 100644 learning/katas/go/course-info.yaml
create mode 100644 learning/katas/go/course-remote-info.yaml
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ArtifactResolver.java
create mode 100644 runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/DefaultArtifactResolver.java
create mode 100644 runners/core-construction-java/src/test/java/org/apache/beam/runners/core/construction/DefaultArtifactResolverTest.java
create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/translation/wrappers/streaming/stableinput/BufferingDoFnRunnerTest.java
delete mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/fn/control/TimerReceiver.java
delete mode 100644 runners/google-cloud-dataflow-java/worker/src/test/java/org/apache/beam/runners/dataflow/worker/fn/control/TimerReceiverTest.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactRetrievalService.java
create mode 100644 runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/artifact/ArtifactStagingService.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ArtifactRetrievalServiceTest.java
create mode 100644 runners/java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution/artifact/ArtifactStagingServiceTest.java
copy runners/java-fn-execution/src/{main/java/org/apache/beam/runners/fnexecution/control/InstructionRequestHandler.java => test/java/org/apache/beam/runners/fnexecution/control/TimerReceiverFactoryTest.java} (61%)
create mode 100644 sdks/go/examples/stringsplit/offsetrange/offsetrange.go
create mode 100644 sdks/go/examples/stringsplit/stringsplit.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sdf.go
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sdf_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/harness/harness_test.go
create mode 100644 sdks/go/pkg/beam/testing/passert/equals.go
create mode 100644 sdks/go/pkg/beam/testing/passert/equals_test.go
create mode 100644 sdks/java/container/license_scripts/dep_urls_java.yaml
create mode 100755 sdks/java/container/license_scripts/license_script.sh
create mode 100644 sdks/java/container/license_scripts/pull_licenses_java.py
delete mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/splittabledofn/Sizes.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/AnnotateVideo.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/VideoIntelligence.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/AnnotateVideoTest.java
create mode 100644 sdks/java/extensions/ml/src/test/java/org/apache/beam/sdk/extensions/ml/VideoIntelligenceIT.java
create mode 100644 sdks/java/extensions/sql/expansion-service/build.gradle
create mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnTimerClient.java
create mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/BeamFnTimerGrpcClient.java
create mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/control/BundleSplitListenerTest.java
create mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/FakeBeamFnTimerClient.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2Message.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2MessageCoder.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HealthcareApiClient.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HealthcareIOError.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HealthcareIOErrorCoder.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HealthcareIOErrorToTableRow.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/HttpHealthcareApiClient.java
copy sdks/java/{extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml => io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare}/package-info.java (66%)
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7V2MessagePagesTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOReadIT.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOTestUtil.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOWriteIT.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HttpHealthcareApiClientTest.java
create mode 100644 sdks/python/apache_beam/dataframe/convert.py
create mode 100644 sdks/python/apache_beam/dataframe/convert_test.py
create mode 100644 sdks/python/apache_beam/dataframe/transforms.py
create mode 100644 sdks/python/apache_beam/dataframe/transforms_test.py
create mode 100644 sdks/python/apache_beam/examples/wordcount_xlang_sql.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/datastoreio_test.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/fake_datastore.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/helper.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/helper_test.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore/v1/query_splitter_test.py
rename sdks/python/apache_beam/io/gcp/datastore/{v1 => v1new}/adaptive_throttler.py (98%)
rename sdks/python/apache_beam/io/gcp/datastore/{v1 => v1new}/adaptive_throttler_test.py (97%)
rename sdks/python/apache_beam/io/gcp/datastore/{v1 => v1new}/util.py (100%)
rename sdks/python/apache_beam/io/gcp/datastore/{v1 => v1new}/util_test.py (98%)
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore_write_it_pipeline.py
delete mode 100644 sdks/python/apache_beam/io/gcp/datastore_write_it_test.py
create mode 100644 sdks/python/apache_beam/runners/interactive/examples/UsCovidDataExample.ipynb
create mode 100644 sdks/python/apache_beam/runners/interactive/options/capture_limiters.py
copy sdks/python/apache_beam/{testing/load_tests/streaming => runners/interactive/testing/integration}/__init__.py (100%)
create mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/goldens/7a35f487b2a5f3a9b9852a8659eeb4bd.png
create mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/notebook_executor.py
create mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/screen_diff.py
create mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/test_notebooks/init_square_cube.ipynb
rename sdks/python/apache_beam/{io/gcp/datastore/v1/__init__.py => runners/interactive/testing/integration/tests/init_square_cube_test.py} (69%)
create mode 100644 sdks/python/apache_beam/transforms/periodicsequence.py
create mode 100644 sdks/python/apache_beam/transforms/periodicsequence_test.py
create mode 100644 sdks/python/apache_beam/transforms/sql.py
create mode 100644 sdks/python/apache_beam/transforms/sql_test.py
create mode 100644 website/src/_posts/2020-04-15-beam-2.20.0.md