You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mw...@apache.org on 2020/05/11 13:31:28 UTC
[beam] branch BEAM-9646-cloud-vision updated (a7291cb -> 53d9601)
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a change to branch BEAM-9646-cloud-vision
in repository https://gitbox.apache.org/repos/asf/beam.git.
omit a7291cb Add dependency to build.gradle
omit 86175c8 Add @Experimental annotation, make parent class package-private
omit 1a33eef Fix spotbugs errors
omit ae6ef78 [BEAM-9646] Add Google Cloud vision integration transform
add 78d2966 BEAM-9826 - Update Tika to 1.24.1
add ec8f60a Merge pull request #11531: [BEAM-9826] Update TikaIO to use Tika version 1.24.1
add 0d12aa5 [BEAM-9827] Ensure minimum watermark hold is computed across all keys
add 439728a Merge pull request #11533: [BEAM-9827] Ensure minimum watermark hold is computed across all keys
add 591de34 Merge pull request #11154: [BEAM-1819] Key should be available in @OnTimer methods
add bfc48ae Updates Dataflow Python containers to 20200423 (#11535)
add 81d7cbe [BEAM-9779] Patch HL7v2IOWriteIT Flakiness (#11450)
add 0c0e8ae Fix JavaSDK typo
add edac60d Add Python SqlTransform to CHANGES.md
add 7da0b1a Add Python SqlTransform to CHANGES.md (#11539)
add 28a7d81 [BEAM-9807] [BEAM-9808] Add missing region to Dataflow examples.
add 89c7db0 Missed a few spots.
add ea858d7 Only add --region for Dataflow runner.
add 0ddf8c4 Merge pull request #11500 from ibzib/mobile-gaming-region
add 9915525 virtualenv (#11502)
add 8ddd7ca [BEAM-9833] Add .asf.yaml file.
add fd7d20e Send issues to github@
add 9070f5e Add license header.
add 2434d49 Merge pull request #11541 from ibzib/BEAM-9833
add 1890696 Merge pull request #11546 from [BEAM-9832] Ensuring tags are not added to ProcessBundleDescriptors o…
add 482bbc0 [BEAM-9824] Do not ignore chained Reshuffles on flink batch runner.
add 164bef3 Merge pull request #11530: [BEAM-9824] Do not ignore chained Reshuffles on flink batch runner.
add 590d819 [BEAM-8790] Kubernetes deployment for InfluxDB
add daee062 [BEAM-8790] Local deployment of InfluxDB with docker-compose.yml
add 0fadb4a Merge pull request #11491: [BEAM-8790] InfluxDB deployment
add 147c34d [BEAM-9758] Use quotes in echo statements to prevent * expansion.
add 480c733 Merge pull request #11542 from ibzib/BEAM-9758
add 1c8519e [BEAM-9443] update document about direct_num_workers=0 for FnApi (#11550)
add 5d93b4e Display correct package in help message
add 2261589 Merge pull request #11461 from tobiaslieber-cognitedata/patch-1
add 7f66ea2 [BEAM-9658] Plumb through WITH OFFSET
add 7c22589 Merge pull request #11540 from apilloud/9658
add a05e18e Speedup on FnApiRunner start up time (#11543)
add 50aec39 [BEAM-6661] Properly close channel for external environment
add 278dcef [BEAM-6661] Avoid FileNotFoundException when no files have been staged
add 9a38a6d [BEAM-6661] Avoid printing empty lines in subprocess job server
add 6eb2c6e Merge pull request #11537: [BEAM-6661] Get rid of a few logging annoyances for execution and shutdown
add d15e29d All Python DirectRunners call setup/teardown on DoFns (#11547)
add 7823cd0 [BEAM-9815] Correct go integration test if clauses. (#11524)
add bfcb17e [BEAM-7923] Support docker screendiff integration tests (#11507)
add feebfcf Use logging to reduce verbosity of license checking when all goes well.
add 0276d85 Merge pull request #11561 Less verbose license checking.
add 06fe9a7 Support schemas in state API
add 74a6565 Merge pull request #10983: [BEAM-9393] Support schemas in state API
add c1319d5 Conditionally cleanup tempdir.
add 199570e Merge pull request #11497 Conditionally cleanup tempdir.
add 068d961 [BEAM-9147] Make VideoIntelligence use PTransform on user-facing API (#11464)
add 138af60 [BEAM-9802] Customization of automatically started services.
add b6f38ac Merge pull request #11495 [BEAM-9802] Customization of automatically started services.
add 2ccb4ad [9577] Attempt to stage artifacts via new API on Java.
add 81002fd Merge pull request #11511 [9577] Attempt to stage artifacts via new API on Java.
add 598402a [BEAM-9771] Fix notebook github URLs
add e173fa8 Merge pull request #11573 from davidcavazos/fix-notebook-urls
add 7f7105c Fix typo in go installation link. (#11445)
add 9ac6711 [BEAM-9846] Clean up references to unused native BQ source/sink experiment in Dataflow
add 837f4a3 Merge pull request #11562 from lukecwik/beam9846
add a010bcd [BEAM-9739]Retry SpannerIO write on Schema change
add 0f421a7 Merge pull request #11392: [BEAM-9739] Retry SpannerIO write on Schema change
add 430b36b [BEAM-9561] Add a framework for running pandas doctests with beam dataframes. (#11419)
add 5390fe2 [BEAM-9720] Add Aws Http client configurations capability. (#11341)
add 605d3c3 [BEAM-5192] move metadata serialization configuration code to WriteFn.setup to avoid doing the configuration for each element.
add 763b7cc Merge pull request #11504: [BEAM-5192] move metadata serialization configuration code to WriteFn setup
add 6196c66 skip pulling licenses by default (#11548)
add ce75c4f [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
add 144b96d Merge pull request #11479 from [BEAM-9795] Support custom avro DatumWriters when writing to BigQuery
add 20195a6 [BEAM-8078] streaming_wordcount_debugging.py is missing a test (#11568)
add de53fc3 Merge pull request #11538 from [BEAM-9831] Improve UX for HL7v2IO
add 180a810 Update container version
add fbd5b5f Merge pull request #11583 from boyuanzz/py_ver
add c7be705 Guard grpc import.
add b78c7ed Merge pull request #11587 from robertwb/no-grpc
add 44cd117 [BEAM-9692] Replace the apply_read with a PTransformOverride
add 73088bf move redundant testing pieces to function
add 6c78c79 Test simplification.
add 93181e7 Merge pull request #11452 [BEAM-9692] Move apply_Read to PTransformOverride
add c333bf8 [BEAM-9600] Get metrics in Flink uber jar job server.
add 6051439 [BEAM-9600] Add a test for Flink REST metrics.
add 3ea0670 Merge pull request #11369 from ibzib/beam-9600
add 791b3ff [BEAM-9860] Require job_endpoint when using PortableRunner.
add e0403cf5 Replace portableWordCount tests with portableWordCountFlinkRunner.
add 4335173 Merge pull request #11585 from ibzib/BEAM-9758
add 3a042b6 [BEAM-9801] Pass in fire timestamp and pane info to timer callback
add dc8f17e [BEAM-9733] Repeatedly fire in batch mode until there are no more timers
add cfa4ffa9b Merge pull request #11492 from mxm/BEAM-9801
add 848875f remove PipelineRunner.fromOptions call
add 9da835c [BEAM-2128] Remove dependency on PipelineRunner#fromOptions to stage files
add 7df2627 Modify PipelineTest to check if exception is thrown on Pipeline run rather than on creation
add 775bb7d Call FileSystems.setDefaultPipelineOptions in BatchStatefulParDoOverridesTest
add 9bb2990 [BEAM-2128] Remove PipelineRunner.fromOptions call (#11571)
add 360de38 Merge pull request #11523 from [BEAM-8414] Cleanup Python codebase to enable some of the excluded Python lint checks.
add b9f08fa Refactor timer firing in Flink & Spark portable batch.
add 1c0063a [BEAM-9801] Fire timers set within timers in Spark.
add 62d5f26 Merge pull request #11595: [BEAM-9801] Fire timers set within timers in Spark
add 8c26045 [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
add 133d798 Merge pull request #11597: [BEAM-9874] Support clearing timers in portable batch mode (Spark/Flink)
add 561b368 [BEAM-8025] Increase the number of retrials en retrial delay in case of load
add 0e93ccd [BEAM-8025] Trace caught NoHostAvailableException and InterruptedException
add 11de825 Merge pull request #11578: [BEAM-8025] Increase the number of retries in CassandraIOTest
add b00011d [BEAM-9841] Support finite wait on PortableRunner's PipelineResult
add 05bf325 Merge pull request #11556: [BEAM-9841] Support finite wait on PortableRunner's PipelineResult
add da12c3a [BEAM-9870] Always generate Dataflow-compatible FnApi protos. (#11593)
add a172313 Go fmt changes. (#11576)
add 27876f0 [BEAM-9418] Support ANY_VALUE aggregation functions
add 6453e85 Merge pull request #11333 from jhnmora000/master
add 497cd03 Make BatchGroupAlsoByWindowViaIteratorsFn extend the ElementByteSizeObservableIterable so that size estimation is lazy
add 573289a Do not attempt to estimate sizes of group-by-key values behind a key; such iterables can have enormous sizes and size estimates would cause reading the data multiple times. Furthermore, the size of the entire group-by-key collection is already known precisely at read time (and encoded into corresponding counters), which means the estimation is in fact not only expensive but redundant and unnecessary.
add 0cd43d1 remove comments
add 33091d2 Make BatchGroupAlsoByWindowViaIteratorsFn extend the ElementByteSizeObservableIterable so that size estimation is lazy
add 62e4df7 revert changes to grouping shuffle reader and iterable like coder
add fc180dd fix merge conflicts
add dd9ab00 remove obsolete class
add 4d38e39 Merge pull request #11598 from tudorm/no-expensive-size-estimation-gbk
add 95f9a15 [BEAM-9799] Automated validation for RTrackers.
add 0b96845 Merge pull request #11553: [BEAM-9799] Automated validation for RTrackers.
add 5984618 [BEAM-9880] fix cannot touch issue (#11606)
add 77309d8 [BEAM-9643] Adding Go SDF Documentation.
add bf53417 [BEAM-9643] Adjusting signature for TrySplit.
add 0c6155e Merge pull request #11517: [BEAM-9643] Adding Go SDF Documentation.
add 34c58c4 Merge pull request #11560 from [BEAM-9886] Auto-inferring project for ReadFromBigQuery
add 49d3f36 Update error prone
add 54631c0 Aling EP core and annotations versions
add 2f3cb62 Merge pull request #11527: Update error prone plugin
add 2495a04 Changed sns writes to async
add bd85e10 [BEAM-8542] V2 async changes for SNSIO
add 7c7bcd7 Merge pull request #10078: [BEAM-8542] Change write to async in AWS SNS IO & remove retry logic
add bfd661f [BEAM-9659][BEAM-9657][BEAM-9664] Reject unsupported unnest joins
add 9dc7d1e Merge pull request #11604: [BEAM-9659][BEAM-9657][BEAM-9664] Reject unsupported unnest joins
add ab04f80 [BEAM-9661] Fix ORDER BY with LIMIT
add 39495d8 Merge pull request #11602: [BEAM-9661] Fix ORDER BY with LIMIT
add 45a6fc6 [BEAM-9650] Add PeriodicSequence generator. (#11477)
add 5e15717 Pass pipeline options from caller to expansion service (#11574)
add a481397 Disable two flaky tests (BEAM-8035, BEAM-9164)
add e5dfa5b Merge pull request #11614: Disable two flaky tests (BEAM-8035, BEAM-9164)
add 96804c9 [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput
add 26f694d Merge pull request #11612: [BEAM-9888] Drop data based on input watermark in @RequiresTimeSortedInput
add dcd9f75 [BEAM-9840] Support for Parameterized Types when converting from HCatRecords to Rows in HCatalogIO (#11569)
add 24ceb95 [BEAM-9836] Exclude Spark runner from UsesKeyInParDo tests
add 65741a3 Merge pull request #11559: [BEAM-9836] Exclude Spark runner from UsesKeyInParDo tests
add 5971d77 [BEAM-9908] Fix Python build failures in release script.
add 3c8e188 Update instructions in release guide.
add bfefe18 Merge pull request #11623 from ibzib/BEAM-9908
add 85b568c [BEAM-7746] Enable mypy type checking for Beam Python code.
add 314822f Update sdks/python/mypy.ini
add 1e00a0e Merge pull request #11620 from robertwb/enable-mypy
add bd4df94 Remove a bunch of spurious warnings in tests. (#11621)
add 6505ea0 Fix thread local to be initialized on every thread.
add 76fdf60 un-duplicate _BEAM_SERVICES
add 7e0be24 Merge pull request #11627 from robertwb/import-fix-155451253
add a0d215c [BEAM-8944] Improve UnboundedThreadPoolExecutor performance
add b740404 fixup! Fix lint
add 79a66fd Merge pull request #11590: [BEAM-8944] Improve UnboundedThreadPoolExecutor performance
add f05466d [BEAM-8134] Grafana dashboards for Load Tests and IO IT Performance Tests
add 81cc7eb [BEAM-8133] Push ioit tests metrics to influxdb
add bef40cc [BEAM-8742] Add stateful and timely processing benchmarks
add d106f26 [BEAM-9900] Remove shutdownSourcesOnFinalWatermark flag
add c36f08a Merge pull request #11558: [BEAM-8742] Add stateful and timely processing benchmarks
add 6f09890 [BEAM-9913] remove license dir if exists (#11630)
add fb70c2f Merge pull request #11624 from [BEAM-9767] Make streaming_wordcount use a test timeout and increase from 5s to 30s
add df4b112 [BEAM-9731] passert.Equals: sort output strings for easier reading (#11615)
add 25b4ebc Revert "Pass pipeline options from caller to expansion service (#11574)" (#11631)
add e6fb32e [Beam-9679] Add Core Transforms section / Map lesson to the Go SDK katas (#11564)
add f97c215 Remove tests of Java on Dataflows Java Runner Harness
add f3e7f57 Merge pull request #11635: [BEAM-9868] Remove tests of Java on Dataflows Java Runner Harness
add b130aa1 [BEAM-9929] Support UNNEST(array_column) in ZetaSQL.
add 46f42d8 Merge pull request #11636 from amaliujia/rw-support_unnest_column
add e725118 Move job server to its own module.
add 61d0029 [BEAM-9845] Stage artifacts over expansion service.
add 544e568 [BEAM-9577] Resolve dependencies in Java expansion service.
add de83a82 Start artifact service in expansion service driver. 0;256;0c# Please enter the commit message for your changes. Lines starting
add 10dc1bb Merge pull request #11557: [BEAM-9845] Stage artifacts over expansion service
add 791c432 [BEAM-8132, BEAM-8133] Apply InfluxDB pipeline options in Load Tests and Performance Tests
add 734ca2c [BEAM-8132, BEAM-8133] Assume no trailing slash after a hostname
add b0a0f9c Merge pull request #11577: [BEAM-8132, BEAM-8133] Apply InfluxDB pipeline options in Load Tests and Performance Tests
add cecc4db [BEAM-2530] Compile and run tests on java 11 for Precommit portability api
add 3a117f9 Merge pull request #11619: [BEAM-2530] Compile and run tests on java 11 for PreCommit portability api
add 3661495 Remove spurious error message.
add ca3d16d Merge pull request #11438: [BEAM-9505] Remove spurious error message in SpannerIO when streaming.
add d5efe91 [BEAM-9911]Replace SpannerIO.write latency counter to distribution
add 291b5d8 Merge pull request #11628: [BEAM-9911]Replace SpannerIO.write latency counter to distribution
add 1edcb61 [BEAM-9883] Refactor SDF test restrictions. (#11605)
add dd9d73d [BEAM-4782] Remove workaround in Python multimap tests.
add d82d061 Merge pull request #11643 from ibzib/BEAM-4782
add ad0c5ad [BEAM-3288] Add suggested fix to error message (#11622)
add 5ef3d2f [BEAM-9940] Set timer family spec for TimerDeclarations in dataflow runner
add eea3a19 Merge pull request #11649 from y1chi/BEAM-9940
add 8bba19a check lastAttemptedOffset non-null
add da4a1b2 Merge pull request #11648 from jaketf/fix/OffsetRangeTrackerErrorHandling
add 9bd5aae Add a BeamJava test with Flatten with different input and output Coders
add 5312f74 Merge branch 'master' into flatten-test
add e763824 Merge pull request #11646 from CraigChambersG/flatten-test
add fd94927 [BEAM-9930] Beam Summit Digital 2020 announcement on blog
add a55dcce Correct URL in blog post
add a784ea6 Minor (markup) changes
add 127e877 Merge pull request #11640: [BEAM-9930] Beam Summit Digital 2020 announcement on blog
add e0c1b14 [BEAM-9930] Correct authors of beam summit digital blog post
add 5134cde Merge pull request #11654: [BEAM-9930] Correct authors of beam summit digital blog post
add 74ae986 [BEAM-9646] Add Google Cloud vision integration transform
add 2f4bc78 Fix spotbugs errors
add 23aab15 Add @Experimental annotation, make parent class package-private
add 19f53cf Add dependency to build.gradle
add 53d9601 Fixes after code review
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (a7291cb)
\
N -- N -- N refs/heads/BEAM-9646-cloud-vision (53d9601)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 36 +
.test-infra/dataproc/init-actions/flink.sh | 2 +
.test-infra/jenkins/CommonJobProperties.groovy | 6 +
.test-infra/jenkins/Docker.groovy | 3 +-
.../jenkins/InfluxDBCredentialsHelper.groovy | 26 +-
.test-infra/jenkins/LoadTestsBuilder.groovy | 11 +-
.test-infra/jenkins/NexmarkBuilder.groovy | 2 +-
.../jenkins/job_LoadTests_CoGBK_Java.groovy | 25 +-
...ts_CoGBK_Java_spark_structured_streaming.groovy | 21 +-
.../job_LoadTests_Combine_Flink_Python.groovy | 21 +-
.../jenkins/job_LoadTests_Combine_Java.groovy | 21 +-
..._Combine_Java_spark_structured_streaming.groovy | 17 +-
.../jenkins/job_LoadTests_Combine_Python.groovy | 13 +-
.../jenkins/job_LoadTests_GBK_Flink_Python.groovy | 21 +-
.test-infra/jenkins/job_LoadTests_GBK_Java.groovy | 31 +-
...ests_GBK_Java_spark_structured_streaming.groovy | 24 +-
.../jenkins/job_LoadTests_GBK_Python.groovy | 23 +-
.../job_LoadTests_GBK_Python_reiterate.groovy | 12 +-
.../job_LoadTests_ParDo_Flink_Python.groovy | 166 +-
.../jenkins/job_LoadTests_ParDo_Java.groovy | 23 +-
...ts_ParDo_Java_spark_structured_streaming.groovy | 17 +-
.../jenkins/job_LoadTests_ParDo_Python.groovy | 14 +-
.../job_LoadTests_coGBK_Flink_Python.groovy | 16 +-
.../jenkins/job_LoadTests_coGBK_Python.groovy | 14 +-
.../job_PerformanceTests_BigQueryIO_Java.groovy | 13 +-
.../job_PerformanceTests_BigQueryIO_Python.groovy | 8 +
.../job_PerformanceTests_FileBasedIO_IT.groovy | 12 +
.../job_PerformanceTests_HadoopFormat.groovy | 6 +
.../jenkins/job_PerformanceTests_JDBC.groovy | 7 +-
.../jenkins/job_PerformanceTests_KafkaIO_IT.groovy | 6 +
.../job_PerformanceTests_MongoDBIO_IT.groovy | 6 +
...ommit_CrossLanguageValidatesRunner_Flink.groovy | 4 +-
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 4 +-
...ava_Dataflow_Portability_Examples_Java11.groovy | 47 -
.../job_PostCommit_Java_Nexmark_Dataflow.groovy | 8 +-
.../job_PostCommit_Java_Nexmark_Direct.groovy | 8 +-
.../job_PostCommit_Java_Nexmark_Flink.groovy | 13 +-
.../job_PostCommit_Java_Nexmark_Spark.groovy | 8 +-
.../job_PostCommit_Java_PortabilityApi.groovy | 51 -
...unner_DataflowPortabilityExecutableStage.groovy | 54 -
..._ValidatesRunner_PortabilityApi_Dataflow.groovy | 51 -
...tesRunner_PortabilityApi_Dataflow_Java11.groovy | 48 -
...job_PreCommit_Java_PortabilityApi_Java11.groovy | 52 +
.test-infra/metrics/README.md | 28 +-
.../Dockerfile => beam-influxdb-autobackup.yaml} | 37 +-
...am-influxdb-backups-persistentvolumeclaim.yaml} | 17 +-
...am-influxdb-storage-persistentvolumeclaim.yaml} | 17 +-
.test-infra/metrics/beam-influxdb.yaml | 117 +
.test-infra/metrics/beamgrafana-deploy.yaml | 40 +-
.../metrics/build_and_publish_containers.sh | 6 +
.test-infra/metrics/docker-compose.yml | 35 +-
.test-infra/metrics/grafana/Dockerfile | 2 +-
.../perftests_metrics/Combine_Load_Tests.json | 459 ++++
.../perftests_metrics/GBK_Load_Tests.json | 922 +++++++
.../Java_IO_IT_Tests_Dataflow.json | 2631 ++++++++++++++++++++
.../perftests_metrics/ParDo_Load_Tests.json | 600 +++++
.../Python_IO_IT_Tests_Dataflow.json | 297 +++
.../perftests_metrics/coGBK_Load_Tests.json | 604 +++++
.../grafana/provisioning/dashboards/all.yaml | 13 +-
.../{beampostgresql.yaml => beaminfluxdb.yaml} | 16 +-
.../provisioning/datasources/beampostgresql.yaml | 8 +-
CHANGES.md | 13 +-
build.gradle | 14 +-
buildSrc/build.gradle | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 48 +-
.../beam/examples/complete/AutoComplete.java | 4 +-
.../apache/beam/examples/snippets/Snippets.java | 72 +-
.../beam/examples/snippets/SnippetsTest.java | 46 +
.../transforms/python/elementwise/filter-py.ipynb | 155 +-
.../transforms/python/elementwise/flatmap-py.ipynb | 74 +-
.../transforms/python/elementwise/keys-py.ipynb | 7 +-
.../transforms/python/elementwise/kvswap-py.ipynb | 7 +-
.../transforms/python/elementwise/map-py.ipynb | 58 +-
.../transforms/python/elementwise/pardo-py.ipynb | 35 +-
.../python/elementwise/partition-py.ipynb | 51 +-
.../transforms/python/elementwise/regex-py.ipynb | 31 +-
.../python/elementwise/tostring-py.ipynb | 13 +-
.../transforms/python/elementwise/values-py.ipynb | 7 +-
.../python/elementwise/withtimestamps-py.ipynb | 4 +-
.../Map/ParDo OneToMany/cmd/main.go | 41 +
.../go/Core Transforms/Map/ParDo OneToMany/go.mod | 27 +
.../go/Core Transforms/Map/ParDo OneToMany/go.sum | 129 +
.../Map/ParDo OneToMany/pkg/task/task.go | 33 +
.../Map/ParDo OneToMany/task-info.yaml} | 26 +-
.../Map/ParDo OneToMany/task-remote-info.yaml | 2 +
.../go/Core Transforms/Map/ParDo OneToMany/task.md | 32 +
.../Map/ParDo OneToMany/test/task_test.go | 44 +
.../Core Transforms/Map/ParDo Struct/cmd/main.go | 41 +
.../go/Core Transforms/Map/ParDo Struct/go.mod | 27 +
.../go/Core Transforms/Map/ParDo Struct/go.sum | 129 +
.../Map/ParDo Struct/pkg/task/task.go | 30 +
.../Map/ParDo Struct/task-info.yaml} | 29 +-
.../Map/ParDo Struct/task-remote-info.yaml | 2 +
.../go/Core Transforms/Map/ParDo Struct/task.md | 34 +
.../Map/ParDo Struct/test/task_test.go | 44 +
.../katas/go/Core Transforms/Map/ParDo/cmd/main.go | 43 +
learning/katas/go/Core Transforms/Map/ParDo/go.mod | 27 +
learning/katas/go/Core Transforms/Map/ParDo/go.sum | 129 +
.../go/Core Transforms/Map/ParDo/pkg/task/task.go | 26 +
.../Map/ParDo/task-info.yaml} | 26 +-
.../Map/ParDo/task-remote-info.yaml | 2 +
.../katas/go/Core Transforms/Map/ParDo/task.md | 39 +
.../go/Core Transforms/Map/ParDo/test/task_test.go | 45 +
.../Map/lesson-info.yaml} | 9 +-
.../go/Core Transforms/Map/lesson-remote-info.yaml | 3 +
.../section-info.yaml} | 7 +-
.../go/Core Transforms/section-remote-info.yaml | 2 +
.../Hello Beam Test/task-remote-info.yaml | 2 +-
learning/katas/go/course-info.yaml | 3 +-
learning/katas/go/course-remote-info.yaml | 2 +-
.../src/main/groovy/MobileGamingCommands.groovy | 20 +-
release/src/main/groovy/TestScripts.groovy | 2 +-
.../src/main/scripts/build_release_candidate.sh | 3 +-
.../translation/operators/ApexParDoOperator.java | 1 +
.../core/construction/ArtifactResolver.java | 4 +
.../core/construction/DefaultArtifactResolver.java | 19 +
.../beam/runners/core/construction/External.java | 86 +-
.../construction/SplittableParDoNaiveBounded.java | 5 +
.../org/apache/beam/runners/core/DoFnRunner.java | 3 +-
.../beam/runners/core/InMemoryTimerInternals.java | 22 +-
.../runners/core/LateDataDroppingDoFnRunner.java | 5 +-
.../apache/beam/runners/core/ProcessFnRunner.java | 3 +-
.../runners/core/PushbackSideInputDoFnRunner.java | 3 +-
.../apache/beam/runners/core/SimpleDoFnRunner.java | 24 +-
.../core/SimplePushbackSideInputDoFnRunner.java | 5 +-
.../beam/runners/core/StatefulDoFnRunner.java | 12 +-
.../core/metrics/MetricsContainerStepMap.java | 14 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 2 +
.../SimplePushbackSideInputDoFnRunnerTest.java | 12 +-
.../beam/runners/core/StatefulDoFnRunnerTest.java | 42 +
runners/direct-java/build.gradle | 7 +
...LifecycleManagerRemovingTransformEvaluator.java | 4 +-
.../apache/beam/runners/direct/ParDoEvaluator.java | 3 +-
.../direct/StatefulParDoEvaluatorFactory.java | 3 +-
...cycleManagerRemovingTransformEvaluatorTest.java | 3 +-
runners/flink/flink_runner.gradle | 1 +
runners/flink/job-server/flink_job_server.gradle | 2 -
.../flink/FlinkBatchTransformTranslators.java | 12 +-
.../runners/flink/FlinkExecutionEnvironments.java | 14 +
.../apache/beam/runners/flink/FlinkJobInvoker.java | 8 +-
.../beam/runners/flink/FlinkJobServerDriver.java | 2 +-
.../flink/FlinkPipelineExecutionEnvironment.java | 11 +
.../beam/runners/flink/FlinkPipelineOptions.java | 33 +-
.../beam/runners/flink/FlinkPipelineRunner.java | 6 +-
.../flink/FlinkPortableClientEntryPoint.java | 8 +-
.../runners/flink/FlinkPortableRunnerResult.java | 7 +-
.../org/apache/beam/runners/flink/FlinkRunner.java | 25 -
.../FlinkStreamingPortablePipelineTranslator.java | 11 +-
.../apache/beam/runners/flink/TestFlinkRunner.java | 1 -
.../flink/metrics/DoFnRunnerWithMetricsUpdate.java | 5 +-
.../functions/FlinkExecutableStageFunction.java | 32 +-
.../functions/FlinkStatefulDoFnFunction.java | 14 +-
.../functions/ImpulseSourceFunction.java | 39 +-
.../flink/translation/utils/CheckpointStats.java | 48 +
.../wrappers/streaming/DoFnOperator.java | 35 +-
.../streaming/ExecutableStageDoFnOperator.java | 5 +-
.../streaming/io/UnboundedSourceWrapper.java | 57 +-
.../streaming/stableinput/BufferedElements.java | 16 +-
.../streaming/stableinput/BufferingDoFnRunner.java | 10 +-
.../streaming/state/FlinkStateInternals.java | 93 +-
.../flink/FlinkExecutionEnvironmentsTest.java | 36 +
.../runners/flink/FlinkPipelineOptionsTest.java | 2 +-
.../flink/FlinkRequiresStableInputTest.java | 2 +-
.../beam/runners/flink/FlinkSavepointTest.java | 4 +-
.../beam/runners/flink/FlinkSubmissionTest.java | 3 -
.../beam/runners/flink/PortableExecutionTest.java | 3 +-
.../runners/flink/PortableStateExecutionTest.java | 3 +-
.../runners/flink/PortableTimersExecutionTest.java | 3 +-
.../beam/runners/flink/ReadSourcePortableTest.java | 3 +-
.../beam/runners/flink/batch/ReshuffleTest.java | 104 +
.../FlinkBroadcastStateInternalsTest.java | 0
.../flink/streaming/FlinkStateInternalsTest.java | 51 +-
.../flink/streaming/GroupByWithNullValuesTest.java | 1 -
.../functions/ImpulseSourceFunctionTest.java | 10 +-
.../wrappers/streaming/DoFnOperatorTest.java | 124 +-
.../streaming/io/UnboundedSourceWrapperTest.java | 6 +-
.../stableinput/BufferedElementsTest.java | 6 +-
runners/google-cloud-dataflow-java/build.gradle | 76 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 11 +
.../dataflow/BatchStatefulParDoOverridesTest.java | 5 +
.../dataflow/worker/DataflowProcessFnRunner.java | 3 +-
.../dataflow/worker/GroupAlsoByWindowFnRunner.java | 3 +-
.../runners/dataflow/worker/SimpleParDoFn.java | 1 +
.../dataflow/worker/StreamingDataflowWorker.java | 5 +
.../StreamingKeyedWorkItemSideInputDoFnRunner.java | 3 +-
.../worker/StreamingSideInputDoFnRunner.java | 3 +-
.../util/BatchGroupAlsoByWindowViaIteratorsFn.java | 17 +-
runners/java-fn-execution/build.gradle | 1 -
.../artifact/ArtifactRetrievalService.java | 15 +-
...BeamFileSystemLegacyArtifactStagingService.java | 4 +-
.../environment/ExternalEnvironmentFactory.java | 36 +-
.../translation/PipelineTranslatorUtils.java | 27 +-
.../build.gradle | 28 +-
.../runners}/jobsubmission/InMemoryJobService.java | 2 +-
.../beam/runners}/jobsubmission/JobInvocation.java | 2 +-
.../beam/runners}/jobsubmission/JobInvoker.java | 2 +-
.../runners}/jobsubmission/JobPreparation.java | 2 +-
.../runners}/jobsubmission/JobServerDriver.java | 2 +-
.../jobsubmission/PortablePipelineJarCreator.java | 2 +-
.../jobsubmission/PortablePipelineJarUtils.java | 2 +-
.../jobsubmission/PortablePipelineResult.java | 2 +-
.../jobsubmission/PortablePipelineRunner.java | 2 +-
.../beam/runners}/jobsubmission/package-info.java | 2 +-
.../jobsubmission/InMemoryJobServiceTest.java | 2 +-
.../runners}/jobsubmission/JobInvocationTest.java | 2 +-
.../PortablePipelineJarCreatorTest.java | 5 +-
.../runners/jet/processors/StatefulParDoP.java | 1 +
runners/portability/java/build.gradle | 1 +
.../beam/runners/portability/PortableRunner.java | 24 +-
.../testing/TestPortablePipelineOptions.java | 2 +-
.../portability/testing/TestPortableRunner.java | 2 +-
runners/samza/build.gradle | 1 +
.../beam/runners/samza/SamzaJobServerDriver.java | 6 +-
.../beam/runners/samza/SamzaPipelineRunner.java | 4 +-
.../runners/samza/SamzaPortablePipelineResult.java | 2 +-
.../org/apache/beam/runners/samza/SamzaRunner.java | 2 +-
.../samza/metrics/DoFnRunnerWithMetrics.java | 5 +-
.../apache/beam/runners/samza/runtime/DoFnOp.java | 1 +
.../runtime/DoFnRunnerWithKeyedInternals.java | 6 +-
.../runners/samza/runtime/SamzaDoFnRunners.java | 3 +-
runners/spark/build.gradle | 5 +-
runners/spark/job-server/build.gradle | 1 +
.../apache/beam/runners/spark/SparkJobInvoker.java | 8 +-
.../beam/runners/spark/SparkJobServerDriver.java | 2 +-
.../beam/runners/spark/SparkPipelineResult.java | 2 +-
.../beam/runners/spark/SparkPipelineRunner.java | 6 +-
.../translation/batch/DoFnRunnerWithMetrics.java | 5 +-
.../spark/translation/DoFnRunnerWithMetrics.java | 5 +-
.../translation/SparkExecutableStageFunction.java | 95 +-
.../spark/translation/SparkProcessContext.java | 1 +
.../runners/spark/SparkPortableExecutionTest.java | 2 +-
.../stringsplit/offsetrange/offsetrange.go | 23 +-
sdks/go/examples/stringsplit/stringsplit.go | 4 +-
sdks/go/pkg/beam/core/graph/fn_test.go | 4 +-
sdks/go/pkg/beam/core/runtime/exec/pardo.go | 13 +
.../beam/core/runtime/exec/sdf_invokers_test.go | 291 ++-
sdks/go/pkg/beam/core/runtime/exec/sdf_test.go | 84 +-
.../pkg/beam/core/runtime/graphx/translate_test.go | 12 +-
sdks/go/pkg/beam/core/sdf/sdf.go | 72 +-
sdks/go/pkg/beam/pardo.go | 81 +
.../pkg/beam/runners/universal/runnerlib/stage.go | 2 +-
sdks/go/pkg/beam/testing/passert/equals.go | 4 +-
sdks/go/test/run_integration_tests.sh | 16 +-
sdks/java/container/build.gradle | 32 +-
.../container/license_scripts/license_script.sh | 57 +-
.../license_scripts/pull_licenses_java.py | 121 +-
.../java/container/license_scripts/requirement.txt | 15 +-
.../main/java/org/apache/beam/sdk/Pipeline.java | 3 -
.../beam/sdk/options/PipelineOptionsFactory.java | 2 +-
.../java/org/apache/beam/sdk/state/StateSpecs.java | 53 +-
.../apache/beam/sdk/testing/UsesKeyInParDo.java} | 19 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 9 +
.../org/apache/beam/sdk/transforms/DoFnTester.java | 6 +
.../org/apache/beam/sdk/transforms/GroupByKey.java | 6 +-
.../java/org/apache/beam/sdk/transforms/ParDo.java | 52 +-
.../beam/sdk/transforms/PeriodicImpulse.java | 86 +
.../beam/sdk/transforms/PeriodicSequence.java | 210 ++
.../org/apache/beam/sdk/transforms/Sample.java | 38 +
.../sdk/transforms/SerializableBiConsumer.java} | 15 +-
.../reflect/ByteBuddyDoFnInvokerFactory.java | 8 +
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 16 +
.../beam/sdk/transforms/reflect/DoFnSignature.java | 21 +
.../sdk/transforms/reflect/DoFnSignatures.java | 16 +-
.../splittabledofn/OffsetRangeTracker.java | 4 +
.../java/org/apache/beam/sdk/PipelineTest.java | 15 +-
.../apache/beam/sdk/transforms/FlattenTest.java | 31 +
.../beam/sdk/transforms/ParDoSchemaTest.java | 273 ++
.../org/apache/beam/sdk/transforms/ParDoTest.java | 135 +
.../beam/sdk/transforms/PeriodicImpulseTest.java | 79 +
.../beam/sdk/transforms/PeriodicSequenceTest.java | 82 +
.../org/apache/beam/sdk/transforms/WatchTest.java | 2 +
.../splittabledofn/OffsetRangeTrackerTest.java | 8 +
sdks/java/expansion-service/build.gradle | 1 +
.../sdk/expansion/service/ExpansionServer.java | 2 +
.../sdk/expansion/service/ExpansionService.java | 23 +-
.../beam/sdk/extensions/ml/AnnotateImages.java | 22 +-
.../ml/AnnotateVideoBytesWithContextFn.java | 50 +
.../{AnnotateVideo.java => AnnotateVideoFn.java} | 12 +-
.../extensions/ml/AnnotateVideoFromBytesFn.java | 55 +
.../sdk/extensions/ml/AnnotateVideoFromURIFn.java | 54 +
.../ml/AnnotateVideoURIWithContextFn.java | 49 +
.../apache/beam/sdk/extensions/ml/CloudVision.java | 47 +-
.../beam/sdk/extensions/ml/VideoIntelligence.java | 142 +-
.../beam/sdk/extensions/ml/AnnotateVideoTest.java | 16 +-
.../sdk/extensions/ml/VideoIntelligenceIT.java | 4 +-
.../provider/bigquery/BigQueryIOPushDownIT.java | 9 +
.../impl/transform/BeamBuiltinAggregations.java | 2 +
.../meta/provider/datastore/DataStoreV1Table.java | 5 +-
.../extensions/sql/BeamSqlDslAggregationTest.java | 49 +
.../sql/zetasql/SqlStdOperatorMappingTable.java | 3 +-
.../translation/ArrayScanColumnRefToUncollect.java | 90 +
...a => ArrayScanLiteralToUncollectConverter.java} | 13 +-
.../translation/ArrayScanToJoinConverter.java | 34 +-
.../LimitOffsetScanToOrderByLimitConverter.java | 11 +-
.../translation/QueryStatementConverter.java | 3 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java | 114 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 23 +-
.../apache/beam/sdk/io/aws/options/AwsModule.java | 112 +-
.../apache/beam/sdk/io/aws/options/AwsOptions.java | 31 +-
.../options/AwsHttpClientConfigurationTest.java | 51 +
.../beam/sdk/io/aws/options/AwsModuleTest.java | 15 +
.../apache/beam/sdk/io/aws2/options/AwsModule.java | 99 +
.../beam/sdk/io/aws2/options/AwsOptions.java | 27 +
.../io/aws2/sns/BasicSnsAsyncClientProvider.java | 57 +
...ntProvider.java => SnsAsyncClientProvider.java} | 12 +-
.../beam/sdk/io/aws2/sns/SnsClientProvider.java | 2 +-
.../org/apache/beam/sdk/io/aws2/sns/SnsIO.java | 188 +-
.../apache/beam/sdk/io/aws2/sns/SnsResponse.java | 55 +
.../beam/sdk/io/aws2/sns/SnsResponseCoder.java | 95 +
.../beam/sdk/io/aws2/options/AwsModuleTest.java | 26 +
.../sdk/io/aws2/sns/MockSnsAsyncBaseClient.java | 15 +-
.../beam/sdk/io/aws2/sns/MockSnsAsyncClient.java | 46 +
.../io/aws2/sns/MockSnsAsyncExceptionClient.java} | 27 +-
.../beam/sdk/io/aws2/sns/SnsIOWriteTest.java | 165 ++
.../beam/sdk/io/aws2/sns/SnsResponseCoderTest.java | 77 +
.../beam/sdk/bigqueryioperftests/BigQueryIOIT.java | 20 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 20 +-
.../beam/sdk/io/common/IOTestPipelineOptions.java | 18 +
.../beam/sdk/io/elasticsearch/ElasticsearchIO.java | 8 +-
.../java/org/apache/beam/sdk/io/avro/AvroIOIT.java | 14 +-
.../apache/beam/sdk/io/parquet/ParquetIOIT.java | 14 +-
.../java/org/apache/beam/sdk/io/text/TextIOIT.java | 14 +-
.../apache/beam/sdk/io/tfrecord/TFRecordIOIT.java | 11 +-
.../java/org/apache/beam/sdk/io/xml/XmlIOIT.java | 14 +-
.../beam/sdk/io/gcp/bigquery/AvroRowWriter.java | 15 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 75 +-
.../beam/sdk/io/gcp/bigquery/RowWriterFactory.java | 29 +-
.../apache/beam/sdk/io/gcp/healthcare/HL7v2IO.java | 46 +-
.../beam/sdk/io/gcp/healthcare/HL7v2Message.java | 29 +-
.../sdk/io/gcp/healthcare/HL7v2MessageCoder.java | 2 +-
.../sdk/io/gcp/healthcare/HealthcareApiClient.java | 20 -
.../sdk/io/gcp/healthcare/HealthcareIOError.java | 20 +-
.../io/gcp/healthcare/HealthcareIOErrorCoder.java | 16 +-
.../healthcare/HealthcareIOErrorToTableRow.java | 14 +-
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 53 +-
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 44 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIOReadIT.java | 27 +-
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 65 +-
.../sdk/io/gcp/bigquery/BigQueryToTableIT.java | 44 -
.../beam/sdk/io/gcp/datastore/V1TestUtil.java | 2 +-
.../io/gcp/healthcare/HL7V2MessagePagesTest.java | 7 +-
.../beam/sdk/io/gcp/healthcare/HL7v2IOReadIT.java | 63 +-
.../sdk/io/gcp/healthcare/HL7v2IOReadWriteIT.java | 129 +
.../beam/sdk/io/gcp/healthcare/HL7v2IOTest.java | 5 +-
.../sdk/io/gcp/healthcare/HL7v2IOTestUtil.java | 70 +-
.../beam/sdk/io/gcp/healthcare/HL7v2IOWriteIT.java | 33 +-
.../healthcare/HttpHealthcareApiClientTest.java | 56 -
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 147 ++
.../sdk/io/hadoop/format/HadoopFormatIOIT.java | 10 +
.../apache/beam/sdk/io/hcatalog/SchemaUtils.java | 68 +-
.../beam/sdk/io/hcatalog/SchemaUtilsTest.java | 45 +
.../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 10 +
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 13 +-
.../apache/beam/sdk/io/mongodb/MongoDBIOIT.java | 10 +
sdks/java/io/tika/build.gradle | 2 +-
.../org/apache/beam/sdk/loadtests/LoadTest.java | 48 +-
.../apache/beam/sdk/loadtests/LoadTestOptions.java | 24 +
sdks/java/testing/test-utils/build.gradle | 11 +
.../beam/sdk/testutils/metrics/IOITMetrics.java | 28 +-
.../testutils/publishing/InfluxDBPublisher.java | 103 +
.../sdk/testutils/publishing/InfluxDBSettings.java | 79 +
.../testutils/jvmverification/JvmVerification.java | 88 +
sdks/python/.pylintrc | 2 -
sdks/python/apache_beam/coders/coders.py | 2 +-
.../apache_beam/coders/standard_coders_test.py | 3 +-
sdks/python/apache_beam/dataframe/doctests.py | 288 +++
sdks/python/apache_beam/dataframe/doctests_test.py | 90 +
.../examples/cookbook/bigquery_tornadoes.py | 2 +-
.../examples/streaming_wordcount_debugging.py | 26 +-
.../streaming_wordcount_debugging_it_test.py | 33 +-
.../examples/streaming_wordcount_debugging_test.py | 110 +
.../apache_beam/examples/wordcount_xlang_sql.py | 6 -
sdks/python/apache_beam/io/fileio_test.py | 8 +-
sdks/python/apache_beam/io/gcp/bigquery.py | 40 +-
.../apache_beam/io/gcp/bigquery_avro_tools.py | 2 +-
.../apache_beam/io/gcp/bigquery_file_loads_test.py | 3 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 11 +-
sdks/python/apache_beam/io/gcp/gcsio_test.py | 2 +-
sdks/python/apache_beam/io/restriction_trackers.py | 2 +-
sdks/python/apache_beam/metrics/execution_test.py | 6 +-
.../python/apache_beam/options/pipeline_options.py | 18 +
sdks/python/apache_beam/pipeline.py | 174 +-
sdks/python/apache_beam/pipeline_test.py | 54 +-
sdks/python/apache_beam/runners/common.py | 16 +-
.../runners/dataflow/dataflow_runner.py | 72 +-
.../runners/dataflow/dataflow_runner_test.py | 73 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/dataflow/ptransform_overrides.py | 38 +
.../runners/direct/transform_evaluator.py | 6 +-
.../runners/interactive/interactive_beam.py | 2 +-
.../runners/interactive/interactive_runner_test.py | 10 +-
.../goldens/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 754885 -> 0 bytes
.../Darwin/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 0 -> 762141 bytes
.../Linux/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 0 -> 717124 bytes
.../interactive/testing/integration/screen_diff.py | 35 +-
.../python/apache_beam/runners/pipeline_context.py | 2 +-
.../runners/portability/artifact_service.py | 53 +
.../runners/portability/expansion_service.py | 2 +-
.../runners/portability/flink_runner_test.py | 1 -
.../portability/flink_uber_jar_job_server.py | 18 +
.../portability/flink_uber_jar_job_server_test.py | 36 +
.../runners/portability/fn_api_runner/execution.py | 6 +
.../runners/portability/fn_api_runner/fn_runner.py | 7 +-
.../portability/fn_api_runner/fn_runner_test.py | 33 +-
.../portability/fn_api_runner/translations.py | 3 +-
.../apache_beam/runners/portability/job_server.py | 78 -
.../runners/portability/portable_runner.py | 76 +-
.../apache_beam/runners/worker/bundle_processor.py | 2 +-
.../apache_beam/runners/worker/opcounters_test.py | 2 +-
.../apache_beam/runners/worker/operations.py | 31 +-
.../apache_beam/testing/load_tests/load_test.py | 29 +-
.../testing/load_tests/load_test_metrics_utils.py | 2 +-
.../apache_beam/testing/load_tests/pardo_test.py | 105 +-
sdks/python/apache_beam/testing/test_pipeline.py | 7 +-
sdks/python/apache_beam/testing/util.py | 2 -
sdks/python/apache_beam/transforms/core.py | 13 +-
.../apache_beam/transforms/dofn_lifecycle_test.py | 16 +
sdks/python/apache_beam/transforms/environments.py | 2 +-
sdks/python/apache_beam/transforms/external.py | 96 +-
sdks/python/apache_beam/transforms/ptransform.py | 2 +-
.../apache_beam/transforms/ptransform_test.py | 7 -
sdks/python/apache_beam/transforms/sql_test.py | 22 +-
sdks/python/apache_beam/transforms/timeutil.py | 6 +-
sdks/python/apache_beam/transforms/trigger.py | 3 +-
sdks/python/apache_beam/transforms/trigger_test.py | 5 +-
sdks/python/apache_beam/transforms/userstate.py | 2 +-
sdks/python/apache_beam/transforms/util.py | 4 -
sdks/python/apache_beam/transforms/util_test.py | 4 +
.../transforms/validate_runner_xlang_test.py | 80 +-
sdks/python/apache_beam/transforms/window.py | 3 +-
sdks/python/apache_beam/utils/subprocess_server.py | 129 +-
.../apache_beam/utils/subprocess_server_test.py | 6 +
.../apache_beam/utils/thread_pool_executor.py | 102 +-
sdks/python/container/Dockerfile | 9 +-
.../container/license_scripts/pull_licenses_py.py | 14 +-
sdks/python/container/py2/build.gradle | 3 +-
sdks/python/container/py35/build.gradle | 3 +-
sdks/python/container/py36/build.gradle | 3 +-
sdks/python/container/py37/build.gradle | 3 +-
sdks/python/mypy.ini | 62 +
sdks/python/scripts/run_pylint.sh | 1 +
sdks/python/setup.py | 2 +
sdks/python/test-suites/portable/py2/build.gradle | 41 +-
sdks/python/test-suites/portable/py35/build.gradle | 9 +-
sdks/python/test-suites/portable/py36/build.gradle | 9 +-
sdks/python/test-suites/portable/py37/build.gradle | 9 +-
sdks/python/tox.ini | 3 +-
settings.gradle | 4 +-
website/notebooks/generate.py | 6 +-
website/src/_data/authors.yml | 9 +-
.../src/_includes/flink_java_pipeline_options.html | 11 +-
.../_includes/flink_python_pipeline_options.html | 11 +-
.../_posts/2020-05-08-beam-summit-digital-2020.md | 51 +
website/src/contribute/index.md | 2 +-
website/src/contribute/release-guide.md | 3 +-
website/src/documentation/patterns/side-inputs.md | 4 +-
website/src/documentation/runners/direct.md | 5 +-
457 files changed, 15268 insertions(+), 2898 deletions(-)
create mode 100644 .asf.yaml
copy sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java => .test-infra/jenkins/InfluxDBCredentialsHelper.groovy (60%)
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_Dataflow_Portability_Examples_Java11.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_PortabilityApi.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_DataflowPortabilityExecutableStage.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow.groovy
delete mode 100644 .test-infra/jenkins/job_PostCommit_Java_ValidatesRunner_PortabilityApi_Dataflow_Java11.groovy
create mode 100644 .test-infra/jenkins/job_PreCommit_Java_PortabilityApi_Java11.groovy
copy .test-infra/metrics/{grafana/Dockerfile => beam-influxdb-autobackup.yaml} (54%)
copy .test-infra/metrics/{grafana/Dockerfile => beam-influxdb-backups-persistentvolumeclaim.yaml} (84%)
copy .test-infra/metrics/{grafana/Dockerfile => beam-influxdb-storage-persistentvolumeclaim.yaml} (84%)
create mode 100644 .test-infra/metrics/beam-influxdb.yaml
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/Combine_Load_Tests.json
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/GBK_Load_Tests.json
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/Java_IO_IT_Tests_Dataflow.json
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/ParDo_Load_Tests.json
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/Python_IO_IT_Tests_Dataflow.json
create mode 100644 .test-infra/metrics/grafana/dashboards/perftests_metrics/coGBK_Load_Tests.json
copy .test-infra/metrics/grafana/provisioning/datasources/{beampostgresql.yaml => beaminfluxdb.yaml} (83%)
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/cmd/main.go
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/go.mod
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/go.sum
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/pkg/task/task.go
copy learning/katas/go/{course-info.yaml => Core Transforms/Map/ParDo OneToMany/task-info.yaml} (70%)
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/task-remote-info.yaml
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/task.md
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo OneToMany/test/task_test.go
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/cmd/main.go
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/go.mod
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/go.sum
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/pkg/task/task.go
copy learning/katas/go/{course-info.yaml => Core Transforms/Map/ParDo Struct/task-info.yaml} (66%)
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/task-remote-info.yaml
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/task.md
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo Struct/test/task_test.go
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/cmd/main.go
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/go.mod
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/go.sum
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/pkg/task/task.go
copy learning/katas/go/{course-info.yaml => Core Transforms/Map/ParDo/task-info.yaml} (70%)
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/task-remote-info.yaml
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/task.md
create mode 100644 learning/katas/go/Core Transforms/Map/ParDo/test/task_test.go
copy learning/katas/go/{course-info.yaml => Core Transforms/Map/lesson-info.yaml} (78%)
create mode 100644 learning/katas/go/Core Transforms/Map/lesson-remote-info.yaml
copy learning/katas/go/{course-info.yaml => Core Transforms/section-info.yaml} (78%)
create mode 100644 learning/katas/go/Core Transforms/section-remote-info.yaml
create mode 100644 runners/flink/src/main/java/org/apache/beam/runners/flink/translation/utils/CheckpointStats.java
create mode 100644 runners/flink/src/test/java/org/apache/beam/runners/flink/batch/ReshuffleTest.java
rename runners/flink/{1.8 => }/src/test/java/org/apache/beam/runners/flink/streaming/FlinkBroadcastStateInternalsTest.java (100%)
rename runners/flink/{1.8 => }/src/test/java/org/apache/beam/runners/flink/streaming/FlinkStateInternalsTest.java (72%)
copy runners/{java-fn-execution => java-job-service}/build.gradle (61%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/InMemoryJobService.java (99%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/JobInvocation.java (99%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/JobInvoker.java (97%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/JobPreparation.java (96%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/JobServerDriver.java (99%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/PortablePipelineJarCreator.java (99%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/PortablePipelineJarUtils.java (98%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/PortablePipelineResult.java (96%)
rename runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/PortablePipelineRunner.java (95%)
copy runners/{java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution => java-job-service/src/main/java/org/apache/beam/runners}/jobsubmission/package-info.java (93%)
rename runners/{java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution => java-job-service/src/test/java/org/apache/beam/runners}/jobsubmission/InMemoryJobServiceTest.java (99%)
rename runners/{java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution => java-job-service/src/test/java/org/apache/beam/runners}/jobsubmission/JobInvocationTest.java (99%)
rename runners/{java-fn-execution/src/test/java/org/apache/beam/runners/fnexecution => java-job-service/src/test/java/org/apache/beam/runners}/jobsubmission/PortablePipelineJarCreatorTest.java (96%)
copy .test-infra/metrics/grafana/Dockerfile => sdks/java/container/license_scripts/requirement.txt (81%)
copy sdks/java/{io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java => core/src/main/java/org/apache/beam/sdk/testing/UsesKeyInParDo.java} (64%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PeriodicImpulse.java
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/PeriodicSequence.java
copy sdks/java/{io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java => core/src/main/java/org/apache/beam/sdk/transforms/SerializableBiConsumer.java} (68%)
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/PeriodicImpulseTest.java
create mode 100644 sdks/java/core/src/test/java/org/apache/beam/sdk/transforms/PeriodicSequenceTest.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/AnnotateVideoBytesWithContextFn.java
rename sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/{AnnotateVideo.java => AnnotateVideoFn.java} (93%)
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/AnnotateVideoFromBytesFn.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/AnnotateVideoFromURIFn.java
create mode 100644 sdks/java/extensions/ml/src/main/java/org/apache/beam/sdk/extensions/ml/AnnotateVideoURIWithContextFn.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/ArrayScanColumnRefToUncollect.java
rename sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/translation/{ArrayScanToUncollectConverter.java => ArrayScanLiteralToUncollectConverter.java} (81%)
create mode 100644 sdks/java/io/amazon-web-services/src/test/java/org/apache/beam/sdk/io/aws/options/AwsHttpClientConfigurationTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/BasicSnsAsyncClientProvider.java
copy sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/{SnsClientProvider.java => SnsAsyncClientProvider.java} (71%)
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsResponse.java
create mode 100644 sdks/java/io/amazon-web-services2/src/main/java/org/apache/beam/sdk/io/aws2/sns/SnsResponseCoder.java
rename runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/jobsubmission/package-info.java => sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/MockSnsAsyncBaseClient.java (72%)
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/MockSnsAsyncClient.java
copy sdks/java/io/amazon-web-services2/src/{main/java/org/apache/beam/sdk/io/aws2/sns/SnsClientProvider.java => test/java/org/apache/beam/sdk/io/aws2/sns/MockSnsAsyncExceptionClient.java} (53%)
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsIOWriteTest.java
create mode 100644 sdks/java/io/amazon-web-services2/src/test/java/org/apache/beam/sdk/io/aws2/sns/SnsResponseCoderTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HL7v2IOReadWriteIT.java
delete mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/HttpHealthcareApiClientTest.java
create mode 100644 sdks/java/io/hcatalog/src/test/java/org/apache/beam/sdk/io/hcatalog/SchemaUtilsTest.java
create mode 100644 sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBPublisher.java
create mode 100644 sdks/java/testing/test-utils/src/main/java/org/apache/beam/sdk/testutils/publishing/InfluxDBSettings.java
create mode 100644 sdks/java/testing/test-utils/src/test/java/org/apache/beam/sdk/testutils/jvmverification/JvmVerification.java
create mode 100644 sdks/python/apache_beam/dataframe/doctests.py
create mode 100644 sdks/python/apache_beam/dataframe/doctests_test.py
create mode 100644 sdks/python/apache_beam/examples/streaming_wordcount_debugging_test.py
delete mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/goldens/7a35f487b2a5f3a9b9852a8659eeb4bd.png
create mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/goldens/Darwin/7a35f487b2a5f3a9b9852a8659eeb4bd.png
create mode 100644 sdks/python/apache_beam/runners/interactive/testing/integration/goldens/Linux/7a35f487b2a5f3a9b9852a8659eeb4bd.png
create mode 100644 website/src/_posts/2020-05-08-beam-summit-digital-2020.md