You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mi...@apache.org on 2020/01/21 17:00:00 UTC
[beam] branch Ardagan-patch-1 updated (4bc2d77 -> c3ef1cb)
This is an automated email from the ASF dual-hosted git repository.
mikhail pushed a change to branch Ardagan-patch-1
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 4bc2d77 cleanup typo
add 946060d [BEAM-9080] Support KVs in the Go SDK's Partition
add 7983403 [BEAM-8337] Hard-code Flink versions.
add 01e944c fix indentation
add 4bdf41f Update release guide
add 94b0be1 Merge pull request #10549: [BEAM-8337] Hard-code Flink versions.
add b5737cb Rephrasing lull logging to avoid alarming users (#10446)
add d64759d [BEAM-8575] Added counter tests for CombineFn (#10190)
add 4caafc7 junitxml_report: Add failure tag support
add 93c7229 Merge pull request #10472: junitxml_report: Add failure tag support
add c287f30 [BEAM-8490] Fix instance_to_type for empty containers (#9894)
add ef228d1 [BEAM-9027] Unparse DOY/DOW/WEEK Enums properly for ZetaSQL
add 909c543 Merge pull request #10539: [BEAM-9027] Unparse DOY/DOW/WEEK Enums properly for ZetaSQL
add 2cd6265 BEAM-8745 More fine-grained controls for the size of a BigQuery Load job
add a20bd7e Merge pull request #10500 from [BEAM-8745] More fine-grained controls for the size of a BigQuery Load job
add 0bfa5e7 [BEAM-8575] Test hot-key fanout with accumulation modes. (#10159)
add 603dc2a [BEAM-9027] Backport BigQuerySqlDialect fixes
add 4ff0b3a Merge pull request #10538: [BEAM-9027] Backport BigQuerySqlDialect fixes
add 33d52b2 [BEAM-9059] Use string constants in PTransformTranslation instead of getUrn() (#10521)
add 420a6e8 [BEAM-8956] Begin unifying contributor instructions into a single location
add 4abc23a Merge pull request #10366: [BEAM-8956] Begin unifying contributor instructions into a single location
add 3484558 [BEAM-7115] Fix metrics being incorrectly gathered
add c22ba7e Merge pull request #10460: [BEAM-7115] Fix metrics being incorrectly gathered
add a2ae7be Remove incorrectly tagged test annotation from test case
add 194a899 Merge pull request #10562: Remove incorrectly tagged test annotation from test case
add ce2cd50 Tighten language and remove distracting link
add 571ff82 Merge pull request #10567 Tighten language and remove distracting link.
add 4aaac9c [BEAM-7390] Add code snippet for Min
add 02efc22 Merge pull request #10177 from davidcavazos/min-code
add f0ab9dd [BEAM-7390] Add code snippet for Sum
add 0ec28de Merge pull request #10178 from davidcavazos/sum-code
add 2321127 [BEAM-7390] Add code snippet for Top (#10179)
add b599a3c Light cleanup of opcodes.py
add b7b06a7 Merge pull request #10431: Light cleanup of opcodes.py
add 5fedbe2 [BEAM-8993] [SQL] MongoDB predicate push down. (#10417)
add 04498d5 [BEAM-8740] Remove unused dependency from Spark runner (#10564)
add 1db2e39 [BEAM-6008] Propagate errors during pipeline execution in Java's PortableRunner
add d35195f Merge pull request #10566: [BEAM-6008] Propagate errors during pipeline execution in Java's PortableRunner
add a3b0f9a [BEAM-8630] Use column numbers for BeamZetaSqlCalRel
add 4135a5d Merge pull request #10537: [BEAM-8630] Use column numbers for BeamZetaSqlCalRel
add 35bf2bd [BEAM-5605] Add support for channel splitting to the gRPC read "source" and propagate "split" calls to the downstream receiver (#10501)
add f2d19fd [BEAM-5605] Add support for additional parameters to SplittableDofn methods (#10556)
add ca8b345 [BEAM-9040] Add skipQueries option to skip queries in a Nexmark suite
add 970e8ad [BEAM-9040] Add Spark Structured Streaming Runner to Nexmark PostCommit tests
add 7bea94c Merge pull request #10485: [BEAM-9040] Add Spark Structured Streaming to Nexmark PostCommit run
add 65619bd [BEAM-6587] Remove hacks due to missing common string coder.
add 3891f9f Merge pull request #10558 [BEAM-6587] Remove hacks due to missing common string coder.
add 2c00922 [BEAM-7746] Address changes in code since annotations were introduced
add 882c865 [BEAM-7746] Typing fixes that require runtime code changes
add bfc6e55 [BEAM-7746] Avoid creating attributes dynamically, so that they can be statically analyzed
add 8353a28 [BEAM-7746] Bugfix: coder id is expected to be str in python3
add 29e243e [BEAM-7746] Explicitly unpack tuple to avoid inferring unbounded tuple (Tuple[str, ...])
add 374db87 [BEAM-7746] Generate files with protobuf urns as part of gen_protos build process
add bc21b25 [BEAM-7746] Move name and coder to base StateSpec class
add 80e2c2e [BEAM-7746] Remove reference to missing attribute in statesampler_slow.StateSampler.reset()
add 9f14ce9 [BEAM-7746] Non-Optional arguments cannot default to None
add ee13d08 [BEAM-7746] Avoid reusing variables with different data types
add 83866ec [BEAM-7746] Add StateHandler abstract base class
add 6189010 [BEAM-7746] Add TODO about fixing assignment to BundleManager._skip_registration
add 3ff01eb [BEAM-7746] Fix functions that were defined twice
add f7f8792 [BEAM-7746] Fix tests that have the same name
add d03404d Merge pull request #10367 [BEAM-7746] Add python type hints (part 2)
add e44a1b6 [Go SDK Doc] Update Dead Container Link (#10585)
add 7163ede Always initalize output processor on construction.
add 9d5a3e4 Merge pull request #10570 Always initalize output processor on construction.
add 6e19ef2 Merge pull request #10582 for [INFRA-19670] Add .asf.yaml for Github testing Whitelist
add 1012408 Update data source for SQL performance tests
add 1e88385 Merge pull request #10573: [BEAM-8844] Update data source for SQL performance tests
add 281b81a Sets the correct coder when clustering is enabled for the multi-partition path
add 148cc71 Merge pull request #10584: [BEAM-9117] Sets the correct coder when clustering is enabled for the multi-partition path
add c44175c Switch to unittest.SkipTest instead of using nose.
add 6bc5f6e Merge pull request #10571 [BEAM-3713] Use unittest.SkipTest instead of nose.plugins.skip.SkipTest.
add 3d541f1 [BEAM-6008] Make sure to end stream only after sending all messages and state updates
add 4fc64ce Merge pull request #10583: [BEAM-6008] Make sure to end stream only after sending all messages and state updates
add 47ef1ea [website] Added security page
add e9311dc [website] Update the 2.17.0 release blog post to include security issues
add 2b07e0e Merge pull request #10581: [website] Add security page and update 2.17.0 release blog post to reference it
add 2e06ca4 Moving to 2.20.0-SNAPSHOT on master branch.
add 1f96eb3 [BEAM-8525] Support Const base in binary_subscr
add 8ecf55b Do not perform test on Py2.7
add 1cd8e9b Merge pull request #9944: [BEAM-8525] Support Const base in binary_subscr
add a37e5c5 [BEAM-8676] sdks/java: gax and grpc upgrades (#10554)
add 2fd785d [BEAM-9030] Migrate Beam to use beam-vendor-grpc-1_26_0 (#10578)
add 2babd75 [BEAM-9084] cleaning up docker image tag
add 99dd538 [BEAM-9084] fix Java spotless
add 9d8bfeb Merge branch 'master' into cleaning_up_docker_tags
add 003c891 Merge pull request #10557 from Hannah-Jiang/cleaning_up_docker_tags
add 17398f7 Changes watermark advance from 1001 to 1000 since Dataflow TestStream only advances watermark at 1s resolution.
add 931108c Merge pull request #10601 from acrites/window-resolution
add dc3468f [BEAM-9002] Add test_flatten_same_pcollections to fnapi runner
add 2ac548f Merge pull request #10550 from angoenka/same_pc_flatten_test
add 104033d [BEAM-9030] Align version of protoc/protoc-gen-grpc-java to vendored version.
add 72ee41f Merge pull request #10602 from lukecwik/beam9030
add cc4f603 Add version guards to requirements file for integration tests.
add 077d8d8 Merge pull request #10568 [BEAM-9061] Add version guards to requirements file for integration tests.
add e1852ca [BEAM-7951] Supports multiple inputs/outputs for wire coder settings. (#10464)
add b0f1dcc Exclude testOutputTimestamp from flink streaming tests.
add b744df1 Merge pull request #10610: [BEAM-9083] Exclude testOutputTimestamp from Flink PVR tests
add f59fa64 [BEAM-9123] HadoopResourceId returns wrong directoryName bugfix
add 0b9c65e Merge pull request #10597: [BEAM-9123] HadoopResourceId returns wrong directoryName bugfix
add 072dd4b [BEAM-9116] Limit the number of past invocations stored in JobService
add 9191690 Merge pull request #10580: [BEAM-9116] Limit the number of past invocations stored in JobServer
add f5e69be [BEAM-9124] Linkage Checker 1.1.2 to use Maven Central HTTPS URL (#10603)
add bb76eed Merge pull request #10316: [BEAM-6857] Support Dynamic Timers
add 046f76b Merge pull request #10577 from Adding Python test for ReadFromBigQuery transform
add 3edc7ac [BEAM-9127] Fix output type declaration in xlang wordcount. (#10605)
add 43bd561 Automatically convert to with pipeline syntax.
add 88fa30e Quick pass through failed auto-conversions.
add 149b9f7 Automatic conversion of more pipelines.
add 11f9ada Fix lint and tests due to autoconversion.
add efa150d A couple more conversions.
add b3a1bdd Fix lint and tests due to autoconversion.
add fad9f04 Return non-None result for Dataflow dry run.
add 5e91757 Fix lint and tests due to autoconversion.
add ff2e030 lint, reviewer comments
add 07b379c lint
add 8811f69 Merge branch 'master' into with-statement
add 4ed6084 fix merge
add ce797f0 Merge pull request #10149 [BEAM-8739] Consistently use with Pipeline(...) syntax
add d140b90 [BEAM-8625] Implement servlet for exposing sdk harness statuses in Da… (#10553)
add 4d3295c [BEAM-8939] A bash script that cancels stale dataflow jobs
add 700bd79 Merge pull request #10579: [BEAM-8939] A bash script that cancels stale dataflow jobs
add e73d633 [BEAM-9143] Make RedisIO code follow standard Beam conventions
add 56be810 [BEAM-9143] Add withOutputParallelization to RedisIO.Read/ReadAll
add 1c07929 Merge pull request #10624: [BEAM-9143] Add withOutputParallelization to RedisIO.Read/ReadAll
add c78ebc5 [BEAM-8695] Upgrade google-http-client to 1.34.0 (#10614)
add f7cb0c0 Migrate HDFS IT to use tox env.
add 86a2919 Fix for py2
add e6a8cbd Merge pull request #10629: [BEAM-9130] Migrate HDFS IT to use tox env.
add 5b6a0ea [BEAM-7246] Add Google Spanner IO Read on Python SDK (#9606)
add 02b2f20 Make apply_TestStream not a test b/c Nose thinks it is a test
add 259c21c Merge pull request #10630 from rohdesamuel/nose-fix
add 709aab4 Beam's own TimestampConversion
add 0983792 Link in Javadoc
add 89fbfbb Merge pull request #10628: [BEAM-9144] Beam's own Avro TimeConversion class in beam-sdk-java-core
add 51d194d [BEAM-7746] Fix a typing issue where SourceBase was assumed to have a coder attribute
add b5a75ae Merge pull request #10590: [BEAM-7746] Fix a typing issue where SourceBase was assumed to have a coder attribute
add ebe8bdc Merge pull request #10622: [BEAM=6857] Fix timermap test to not use TestStream
add c188c15 Report status code 0 when no stale jobs are found
add 4254673 Merge pull request #10639: [BEAM-8939] Report status code 0 when no stale jobs are found
add 61a588c [BEAM-9153] Fix release guide heading level
add 201cfc1 Merge pull request #10638: [website][BEAM-9153] Fix release guide heading level
add a93927f [BEAM-8108] Allow run_chicago.sh to take variable argument number
add 627f97d [BEAM-8108] Use _ReadFromBigQuery transform
add a0ce485 [BEAM-8108] Create Jenkins job that runs Chicago Taxi Example on Flink
add 87f705c [BEAM-8108] Rewrite Python print statement in dataproc init action
add 287b81c Merge pull request #10531 from kamilwu/chicago-flink
add 583b0c1 [BEAM-5086] Update kudu-client to version 1.11.1
add d7aefba Merge pull request #10561 from iemejia/BEAM-5086-update-kudu-client
add c3ef1cb Merge branch 'master' into Ardagan-patch-1
No new revisions were added by this update.
Summary of changes:
.asf.yaml | 177 ++++++
.test-infra/dataproc/flink_cluster.sh | 2 +-
.test-infra/jenkins/CommonTestProperties.groovy | 2 +
.test-infra/jenkins/LoadTestsBuilder.groovy | 12 +-
.test-infra/jenkins/NexmarkBuilder.groovy | 10 +
.test-infra/jenkins/README.md | 2 +
.../jenkins/job_CancelStaleDataflowJobs.groovy | 1 -
.../job_PostCommit_Java_Nexmark_Spark.groovy | 33 ++
...mit_Python_Chicago_Taxi_Example_Dataflow.groovy | 4 +-
...Commit_Python_Chicago_Taxi_Example_Flink.groovy | 72 +++
.test-infra/junitxml_report.py | 3 +
.test-infra/tools/build.gradle | 25 +-
.test-infra/tools/stale_dataflow_jobs_cleaner.go | 117 ----
.../stale_dataflow_jobs_cleaner.sh} | 15 +-
.../tools/stale_dataflow_jobs_cleaner_test.go | 74 ---
CONTRIBUTING.md | 29 -
README.md | 23 +-
build.gradle | 2 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 39 +-
.../apache/beam/gradle/GrpcVendoring_1_21_0.groovy | 150 -----
.../apache/beam/gradle/GrpcVendoring_1_26_0.groovy | 82 ++-
.../get-started/try-apache-beam-java.ipynb | 14 +-
gradle.properties | 4 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 63 +-
.../src/main/scripts/build_release_candidate.sh | 9 +-
release/src/main/scripts/publish_docker_images.sh | 52 +-
release/src/main/scripts/set_version.sh | 4 +-
.../translation/operators/ApexTimerInternals.java | 2 +-
.../operators/ApexTimerInternalsTest.java | 2 +-
runners/core-construction-java/build.gradle | 2 +-
.../core/construction/ArtifactServiceStager.java | 6 +-
.../beam/runners/core/construction/BeamUrns.java | 2 +-
.../core/construction/CoderTranslation.java | 2 +-
.../core/construction/CoderTranslators.java | 2 +-
.../core/construction/CombineTranslation.java | 2 +-
.../CreatePCollectionViewTranslation.java | 2 +-
.../DefaultExpansionServiceClientFactory.java | 2 +-
.../core/construction/DisplayDataTranslation.java | 4 +-
.../runners/core/construction/Environments.java | 4 +-
.../beam/runners/core/construction/External.java | 4 +-
.../runners/core/construction/ModelCoders.java | 2 +-
.../construction/PCollectionViewTranslation.java | 2 +-
.../core/construction/PTransformTranslation.java | 4 +
.../core/construction/ParDoTranslation.java | 35 +-
.../construction/PipelineOptionsTranslation.java | 6 +-
.../runners/core/construction/ReadTranslation.java | 4 +-
.../runners/core/construction/SplittableParDo.java | 8 +
.../construction/SplittableParDoNaiveBounded.java | 11 +
.../core/construction/TestStreamTranslation.java | 2 +-
.../core/construction/WindowIntoTranslation.java | 2 +-
.../construction/WindowingStrategyTranslation.java | 8 +-
.../core/construction/WriteFilesTranslation.java | 2 +-
.../construction/expansion/ExpansionServer.java | 4 +-
.../construction/expansion/ExpansionService.java | 6 +-
.../core/construction/graph/ExecutableStage.java | 27 +-
.../graph/GreedyPCollectionFusers.java | 2 +-
.../construction/graph/GreedyPipelineFuser.java | 2 +-
.../core/construction/graph/GreedyStageFuser.java | 4 +-
.../graph/ImmutableExecutableStage.java | 12 +-
.../construction/graph/OutputDeduplicator.java | 2 +-
.../core/construction/graph/PipelineValidator.java | 41 +-
.../core/construction/graph/QueryablePipeline.java | 2 +-
.../construction/ArtifactServiceStagerTest.java | 6 +-
.../runners/core/construction/CommonCoderTest.java | 2 +-
.../runners/core/construction/ExternalTest.java | 10 +-
.../InMemoryArtifactStagerService.java | 2 +-
.../PipelineOptionsTranslationTest.java | 6 +-
.../construction/WindowIntoTranslationTest.java | 2 +-
.../expansion/ExpansionServiceTest.java | 2 +-
.../construction/graph/ExecutableStageTest.java | 4 +-
.../graph/ImmutableExecutableStageTest.java | 4 +-
.../construction/graph/OutputDeduplicatorTest.java | 18 +-
.../construction/graph/ProtoOverridesTest.java | 2 +-
.../beam/runners/core/InMemoryTimerInternals.java | 24 +-
...TimeBoundedSplittableProcessElementInvoker.java | 13 +
.../apache/beam/runners/core/SimpleDoFnRunner.java | 129 ++++-
.../apache/beam/runners/core/TimerInternals.java | 16 +-
.../runners/core/metrics/SimpleExecutionState.java | 2 +-
.../runners/core/InMemoryTimerInternalsTest.java | 12 +-
.../beam/runners/core/SimpleDoFnRunnerTest.java | 4 +-
.../SimplePushbackSideInputDoFnRunnerTest.java | 7 +-
runners/direct-java/build.gradle | 2 +-
.../beam/runners/direct/DirectTimerInternals.java | 2 +-
.../runners/direct/ParDoMultiOverrideFactory.java | 3 +-
.../direct/StatefulParDoEvaluatorFactory.java | 3 +-
.../beam/runners/direct/WatermarkManager.java | 45 +-
runners/flink/flink_runner.gradle | 3 +-
runners/flink/job-server/flink_job_server.gradle | 1 +
.../FlinkBatchPortablePipelineTranslator.java | 2 +-
.../apache/beam/runners/flink/FlinkJobInvoker.java | 2 +-
.../beam/runners/flink/FlinkPipelineRunner.java | 2 +-
.../FlinkStreamingPortablePipelineTranslator.java | 2 +-
.../wrappers/streaming/DoFnOperator.java | 2 +-
.../streaming/ExecutableStageDoFnOperator.java | 2 +-
.../wrappers/streaming/FlinkKeyUtils.java | 2 +-
.../beam/runners/flink/FlinkJobInvokerTest.java | 2 +-
.../FlinkExecutableStageFunctionTest.java | 2 +-
.../streaming/ExecutableStageDoFnOperatorTest.java | 4 +-
.../wrappers/streaming/FlinkKeyUtilsTest.java | 2 +-
runners/google-cloud-dataflow-java/build.gradle | 3 +-
.../dataflow/DataflowPipelineTranslator.java | 4 +-
.../dataflow/PrimitiveParDoSingleFactory.java | 16 +
.../beam/runners/dataflow/util/CloudObject.java | 15 +
.../runners/dataflow/util/OutputReference.java | 18 +
.../runners/dataflow/util/CloudObjectTest.java | 26 +-
.../runners/dataflow/util/OutputReferenceTest.java | 16 +-
.../google-cloud-dataflow-java/worker/build.gradle | 2 +-
.../worker/legacy-worker/build.gradle | 2 +-
.../dataflow/worker/BatchDataflowWorker.java | 5 +-
.../runners/dataflow/worker/ByteStringCoder.java | 2 +-
.../dataflow/worker/DataflowOperationContext.java | 2 +-
.../dataflow/worker/DataflowRunnerHarness.java | 49 +-
.../worker/DataflowWorkerHarnessHelper.java | 2 +-
.../worker/GroupAlsoByWindowParDoFnFactory.java | 2 +-
.../worker/MetricTrackingWindmillServerStub.java | 2 +-
.../beam/runners/dataflow/worker/PubsubSink.java | 2 +-
.../beam/runners/dataflow/worker/ReaderCache.java | 2 +-
.../beam/runners/dataflow/worker/StateFetcher.java | 2 +-
.../dataflow/worker/StreamingDataflowWorker.java | 14 +-
.../worker/StreamingModeExecutionContext.java | 2 +-
.../dataflow/worker/StreamingSideInputFetcher.java | 4 +-
.../dataflow/worker/WindmillNamespacePrefix.java | 2 +-
.../beam/runners/dataflow/worker/WindmillSink.java | 2 +-
.../dataflow/worker/WindmillStateCache.java | 2 +-
.../dataflow/worker/WindmillStateInternals.java | 2 +-
.../dataflow/worker/WindmillStateReader.java | 2 +-
.../dataflow/worker/WindmillTimerInternals.java | 107 +++-
.../dataflow/worker/WorkerCustomSources.java | 2 +-
.../dataflow/worker/fn/BeamFnControlService.java | 2 +-
.../control/RegisterAndProcessBundleOperation.java | 4 +-
.../worker/fn/data/BeamFnDataGrpcService.java | 2 +-
.../worker/fn/logging/BeamFnLoggingService.java | 4 +-
.../fn/stream/ServerStreamObserverFactory.java | 6 +-
.../graph/CreateExecutableStageNodeFunction.java | 19 +-
...nsertFetchAndFilterStreamingSideInputNodes.java | 2 +-
.../worker/graph/RegisterNodeFunction.java | 16 +-
.../dataflow/worker/status/DebugCapture.java | 5 +-
.../worker/status/SdkWorkerStatusServlet.java | 99 ++++
.../dataflow/worker/status/WorkerStatusPages.java | 16 +-
.../worker/windmill/DirectStreamObserver.java | 4 +-
.../windmill/ForwardingClientResponseObserver.java | 6 +-
.../worker/windmill/GrpcWindmillServer.java | 30 +-
.../worker/windmill/StreamObserverFactory.java | 4 +-
.../runners/dataflow/harness/test/TestStreams.java | 4 +-
.../runners/dataflow/worker/DataflowMatchers.java | 2 +-
.../worker/DataflowOperationContextTest.java | 3 +-
.../worker/DataflowWorkerHarnessHelperTest.java | 2 +-
.../runners/dataflow/worker/PubsubReaderTest.java | 2 +-
.../runners/dataflow/worker/PubsubSinkTest.java | 2 +-
.../runners/dataflow/worker/ReaderCacheTest.java | 2 +-
.../runners/dataflow/worker/StateFetcherTest.java | 2 +-
.../worker/StreamingDataflowWorkerTest.java | 6 +-
.../worker/StreamingGroupAlsoByWindowFnsTest.java | 2 +-
...reamingGroupAlsoByWindowsReshuffleDoFnTest.java | 2 +-
.../worker/StreamingModeExecutionContextTest.java | 2 +-
.../worker/StreamingSideInputDoFnRunnerTest.java | 2 +-
.../worker/StreamingSideInputFetcherTest.java | 2 +-
.../dataflow/worker/WindmillKeyedWorkItemTest.java | 2 +-
.../worker/WindmillReaderIteratorBaseTest.java | 2 +-
.../dataflow/worker/WindmillStateCacheTest.java | 2 +-
.../worker/WindmillStateInternalsTest.java | 2 +-
.../dataflow/worker/WindmillStateReaderTest.java | 4 +-
.../dataflow/worker/WorkerCustomSourcesTest.java | 2 +-
.../worker/fn/BeamFnControlServiceTest.java | 12 +-
.../RegisterAndProcessBundleOperationTest.java | 2 +-
.../worker/fn/control/TimerReceiverTest.java | 2 +-
.../worker/fn/data/BeamFnDataGrpcServiceTest.java | 32 +-
.../fn/logging/BeamFnLoggingServiceTest.java | 12 +-
.../fn/stream/ServerStreamObserverFactoryTest.java | 4 +-
...tFetchAndFilterStreamingSideInputNodesTest.java | 2 +-
.../logging/DataflowWorkerLoggingHandlerTest.java | 2 +-
.../worker/testing/GenericJsonMatcherTest.java | 4 +-
.../worker/windmill/GrpcWindmillServerTest.java | 14 +-
runners/java-fn-execution/build.gradle | 2 +-
.../apache/beam/runners/fnexecution/FnService.java | 6 +-
.../GrpcContextHeaderAccessorProvider.java | 16 +-
.../beam/runners/fnexecution/GrpcFnServer.java | 2 +-
.../fnexecution/InProcessServerFactory.java | 8 +-
.../beam/runners/fnexecution/ServerFactory.java | 20 +-
.../artifact/AbstractArtifactRetrievalService.java | 10 +-
.../artifact/AbstractArtifactStagingService.java | 10 +-
.../BeamFileSystemArtifactStagingService.java | 4 +-
.../fnexecution/control/FnApiControlClient.java | 6 +-
.../control/FnApiControlClientPoolService.java | 2 +-
.../control/ProcessBundleDescriptors.java | 39 +-
.../runners/fnexecution/data/GrpcDataService.java | 2 +-
.../jobsubmission/InMemoryJobService.java | 110 +++-
.../fnexecution/jobsubmission/JobInvocation.java | 15 +-
.../fnexecution/jobsubmission/JobInvoker.java | 2 +-
.../fnexecution/jobsubmission/JobPreparation.java | 2 +-
.../fnexecution/jobsubmission/JobServerDriver.java | 10 +-
.../jobsubmission/PortablePipelineJarCreator.java | 6 +-
.../jobsubmission/PortablePipelineJarUtils.java | 6 +-
.../fnexecution/logging/GrpcLoggingService.java | 2 +-
.../runners/fnexecution/provisioning/JobInfo.java | 2 +-
.../provisioning/StaticGrpcProvisionService.java | 2 +-
.../splittabledofn/SDFFeederViaStateAndTimers.java | 4 +-
.../fnexecution/state/GrpcStateService.java | 4 +-
.../state/InMemoryBagUserStateFactory.java | 2 +-
.../fnexecution/state/StateRequestHandlers.java | 2 +-
.../status/BeamWorkerStatusGrpcService.java | 2 +-
.../fnexecution/status/WorkerStatusClient.java | 2 +-
.../translation/PipelineTranslatorUtils.java | 2 +-
.../beam/runners/fnexecution/wire/WireCoders.java | 13 +-
.../GrpcContextHeaderAccessorProviderTest.java | 20 +-
.../runners/fnexecution/ServerFactoryTest.java | 10 +-
.../BeamFileSystemArtifactServicesTest.java | 8 +-
.../artifact/ClassLoaderArtifactServiceTest.java | 8 +-
.../control/DefaultJobBundleFactoryTest.java | 4 +-
.../control/FnApiControlClientPoolServiceTest.java | 4 +-
.../control/FnApiControlClientTest.java | 2 +-
.../fnexecution/control/RemoteExecutionTest.java | 2 +-
.../fnexecution/data/GrpcDataServiceTest.java | 8 +-
.../jobsubmission/InMemoryJobServiceTest.java | 92 ++-
.../jobsubmission/JobInvocationTest.java | 2 +-
.../logging/GrpcLoggingServiceTest.java | 6 +-
.../StaticGrpcProvisionServiceTest.java | 10 +-
.../fnexecution/state/GrpcStateServiceTest.java | 4 +-
.../status/BeamWorkerStatusGrpcServiceTest.java | 8 +-
.../fnexecution/status/WorkerStatusClientTest.java | 2 +-
.../BatchSideInputHandlerFactoryTest.java | 4 +-
runners/portability/java/build.gradle | 2 +-
.../runners/portability/ExternalWorkerService.java | 2 +-
.../portability/JobServicePipelineResult.java | 77 ++-
.../beam/runners/portability/PortableRunner.java | 4 +-
.../portability/testing/TestJobService.java | 2 +-
.../runners/portability/PortableRunnerTest.java | 6 +-
.../beam/runners/samza/SamzaJobServerDriver.java | 5 +-
.../beam/runners/samza/runtime/KeyedInternals.java | 4 +-
.../samza/runtime/SamzaTimerInternalsFactory.java | 2 +-
.../samza/translation/WindowAssignTranslator.java | 2 +-
.../samza/util/SamzaPipelineTranslatorUtils.java | 2 +-
runners/spark/build.gradle | 1 -
.../apache/beam/runners/spark/SparkJobInvoker.java | 2 +-
.../beam/runners/spark/SparkPipelineRunner.java | 2 +-
.../spark/stateful/SparkTimerInternals.java | 2 +-
sdks/go/README.md | 3 +-
sdks/go/pkg/beam/partition.go | 77 ++-
sdks/go/pkg/beam/partition_test.go | 53 ++
sdks/java/container/build.gradle | 2 +-
sdks/java/core/build.gradle | 1 +
.../java/org/apache/beam/sdk/coders/AvroCoder.java | 38 +-
.../apache/beam/sdk/schemas/SchemaTranslation.java | 2 +-
.../apache/beam/sdk/schemas/utils/AvroUtils.java | 6 +-
.../sdk/state/{package-info.java => TimerMap.java} | 15 +-
.../java/org/apache/beam/sdk/state/TimerSpecs.java | 4 +
.../{UsesSystemMetrics.java => UsesTimerMap.java} | 7 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 91 ++-
.../org/apache/beam/sdk/transforms/DoFnTester.java | 17 +
.../java/org/apache/beam/sdk/transforms/ParDo.java | 12 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 136 ++++-
.../reflect/ByteBuddyOnTimerInvokerFactory.java | 136 +++++
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 27 +-
.../beam/sdk/transforms/reflect/DoFnSignature.java | 173 +++++-
.../sdk/transforms/reflect/DoFnSignatures.java | 415 ++++++++++++-
.../sdk/transforms/reflect/OnTimerInvokers.java | 5 +
.../java/org/apache/beam/sdk/util/ReleaseInfo.java | 10 +-
.../resources/org/apache/beam/sdk/sdk.properties | 1 +
.../apache/beam/sdk/testing/TestStreamTest.java | 5 +-
.../org/apache/beam/sdk/transforms/ParDoTest.java | 132 +++++
.../sdk/transforms/reflect/DoFnInvokersTest.java | 6 +-
.../reflect/DoFnSignaturesProcessElementTest.java | 4 +-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 111 +++-
.../sdk/transforms/reflect/DoFnSignaturesTest.java | 5 +
.../provider/bigquery/BigQueryIOPushDownIT.java | 3 +-
.../provider/bigquery/BeamBigQuerySqlDialect.java | 98 ++++
.../provider/bigquery/BeamSqlUnparseContext.java | 41 ++
.../sql/meta/provider/mongodb/MongoDbTable.java | 271 ++++++++-
.../MongoDbFilterTest.java} | 75 +--
.../meta/provider/mongodb/MongoDbReadWriteIT.java | 104 +++-
.../extensions/sql/zetasql/BeamZetaSqlCalcRel.java | 33 +-
.../sql/zetasql/ZetaSQLDialectSpecTest.java | 1 -
sdks/java/fn-execution/build.gradle | 2 +-
.../beam/sdk/fn/channel/ManagedChannelFactory.java | 18 +-
.../beam/sdk/fn/channel/SocketAddressFactory.java | 2 +-
.../data/BeamFnDataBufferingOutboundObserver.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexer.java | 4 +-
...amFnDataSizeBasedBufferingOutboundObserver.java | 4 +-
...amFnDataTimeBasedBufferingOutboundObserver.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortRead.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWrite.java | 2 +-
.../sdk/fn/stream/BufferingStreamObserver.java | 4 +-
.../org/apache/beam/sdk/fn/stream/DataStreams.java | 2 +-
.../beam/sdk/fn/stream/DirectStreamObserver.java | 4 +-
.../stream/ForwardingClientResponseObserver.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactory.java | 4 +-
.../sdk/fn/stream/SynchronizedStreamObserver.java | 2 +-
.../fn/test/InProcessManagedChannelFactory.java | 4 +-
.../org/apache/beam/sdk/fn/test/TestStreams.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindow.java | 2 +-
.../sdk/fn/channel/ManagedChannelFactoryTest.java | 6 +-
.../sdk/fn/channel/SocketAddressFactoryTest.java | 2 +-
.../sdk/fn/data/BeamFnDataGrpcMultiplexerTest.java | 2 +-
...DataSizeBasedBufferingOutboundObserverTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortReadTest.java | 2 +-
.../beam/sdk/fn/data/RemoteGrpcPortWriteTest.java | 2 +-
.../apache/beam/sdk/fn/stream/DataStreamsTest.java | 2 +-
.../ForwardingClientResponseObserverTest.java | 6 +-
.../sdk/fn/stream/OutboundObserverFactoryTest.java | 4 +-
.../sdk/fn/windowing/EncodedBoundedWindowTest.java | 2 +-
sdks/java/harness/build.gradle | 2 +-
.../beam/fn/harness/BeamFnDataReadRunner.java | 115 +++-
.../beam/fn/harness/BoundedSourceRunner.java | 2 +-
.../org/apache/beam/fn/harness/CombineRunners.java | 11 +-
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 42 +-
.../java/org/apache/beam/fn/harness/FnHarness.java | 2 +-
.../org/apache/beam/fn/harness/HandlesSplits.java | 28 +-
.../beam/fn/harness/WindowMappingFnRunner.java | 5 +-
.../beam/fn/harness/WindowMergingFnRunner.java | 5 +-
.../harness/control/AddHarnessIdInterceptor.java | 8 +-
.../fn/harness/control/BeamFnControlClient.java | 4 +-
.../fn/harness/control/ProcessBundleHandler.java | 4 +-
.../beam/fn/harness/control/RegisterHandler.java | 2 +-
.../beam/fn/harness/data/BeamFnDataGrpcClient.java | 2 +-
.../harness/data/ElementCountFnDataReceiver.java | 69 ---
.../harness/data/MultiplexingFnDataReceiver.java | 48 --
.../harness/data/PCollectionConsumerRegistry.java | 226 ++++++--
.../fn/harness/data/QueueingBeamFnDataClient.java | 3 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 12 +-
.../apache/beam/fn/harness/state/BagUserState.java | 2 +-
.../harness/state/BeamFnStateGrpcClientCache.java | 4 +-
.../beam/fn/harness/state/FnApiStateAccessor.java | 2 +-
.../beam/fn/harness/state/MultimapSideInput.java | 2 +-
.../fn/harness/state/StateFetchingIterators.java | 2 +-
.../stream/HarnessStreamObserverFactories.java | 2 +-
.../beam/fn/harness/AssignWindowsRunnerTest.java | 2 +-
.../beam/fn/harness/BeamFnDataReadRunnerTest.java | 207 ++++++-
.../beam/fn/harness/BoundedSourceRunnerTest.java | 2 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 2 +-
.../org/apache/beam/fn/harness/FnHarnessTest.java | 8 +-
.../harness/control/BeamFnControlClientTest.java | 8 +-
.../harness/control/ProcessBundleHandlerTest.java | 4 +-
.../fn/harness/data/BeamFnDataGrpcClientTest.java | 14 +-
.../data/BeamFnDataInboundObserverTest.java | 2 +-
.../data/ElementCountFnDataReceiverTest.java | 98 ----
.../data/MultiplexingFnDataReceiverTest.java | 111 ----
.../data/PCollectionConsumerRegistryTest.java | 186 +++++-
.../harness/data/QueueingBeamFnDataClientTest.java | 14 +-
.../harness/logging/BeamFnLoggingClientTest.java | 16 +-
.../beam/fn/harness/state/BagUserStateTest.java | 2 +-
.../state/BeamFnStateGrpcClientCacheTest.java | 16 +-
.../fn/harness/state/FakeBeamFnStateClient.java | 2 +-
.../fn/harness/state/MultimapSideInputTest.java | 2 +-
.../harness/state/StateFetchingIteratorsTest.java | 2 +-
.../stream/HarnessStreamObserverFactoriesTest.java | 4 +-
.../apache/beam/sdk/io/tfrecord/TFRecordIOIT.java | 103 ++--
sdks/java/io/google-cloud-platform/build.gradle | 2 +
.../beam/sdk/io/gcp/bigquery/BatchLoads.java | 15 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 20 +-
.../sdk/io/gcp/pubsub/PubsubIOExternalTest.java | 4 +-
.../apache/beam/sdk/io/hdfs/HadoopResourceId.java | 4 +
.../beam/sdk/io/hdfs/HadoopResourceIdTest.java | 15 +
.../beam/sdk/io/kafka/KafkaIOExternalTest.java | 4 +-
sdks/java/io/kudu/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/redis/RedisIO.java | 155 +++--
.../org/apache/beam/sdk/io/redis/RedisIOTest.java | 4 +-
.../beam/sdk/expansion/TestExpansionService.java | 10 +-
.../beam/sdk/nexmark/NexmarkConfiguration.java | 9 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 27 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 6 +
.../apache/beam/sdk/nexmark/NexmarkQueryName.java | 17 +
.../beam/sdk/testutils/metrics/MetricsReader.java | 18 +
sdks/python/apache_beam/coders/coder_impl.py | 3 +-
sdks/python/apache_beam/coders/row_coder.py | 2 +-
.../examples/complete/game/game_stats.py | 5 -
.../apache_beam/examples/cookbook/filters_test.py | 30 +-
.../apache_beam/examples/fastavro_it_test.py | 73 ++-
.../examples/flink/flink_streaming_impulse.py | 34 +-
.../apache_beam/examples/snippets/snippets.py | 171 +++---
.../transforms/aggregation/{mean.py => min.py} | 27 +-
.../aggregation/{max_test.py => min_test.py} | 34 +-
.../transforms/aggregation/{mean.py => sum.py} | 26 +-
.../keys_test.py => aggregation/sum_test.py} | 36 +-
.../snippets/transforms/aggregation/top.py | 153 +++++
.../snippets/transforms/aggregation/top_test.py | 111 ++++
.../apache_beam/examples/streaming_wordcount.py | 86 ++-
.../examples/streaming_wordcount_debugging.py | 150 +++--
.../python/apache_beam/examples/wordcount_xlang.py | 26 +-
sdks/python/apache_beam/internal/gcp/auth.py | 3 +-
sdks/python/apache_beam/io/concat_source_test.py | 7 +-
.../io/external/generate_sequence_test.py | 3 +-
.../io/external/xlang_parquetio_test.py | 9 +-
sdks/python/apache_beam/io/filebasedsource_test.py | 105 ++--
sdks/python/apache_beam/io/fileio_test.py | 4 +-
sdks/python/apache_beam/io/filesystems_test.py | 11 +-
.../apache_beam/io/gcp/bigquery_io_read_it_test.py | 7 +-
.../io/gcp/bigquery_io_read_pipeline.py | 29 +-
.../apache_beam/io/gcp/bigquery_read_perf_test.py | 22 +-
.../models => io/gcp/experimental}/__init__.py | 0
.../apache_beam/io/gcp/experimental/spannerio.py | 583 +++++++++++++++++++
.../io/gcp/experimental/spannerio_test.py | 338 +++++++++++
sdks/python/apache_beam/io/gcp/pubsub_test.py | 141 +++--
.../io/hdfs_integration_test/Dockerfile | 23 +-
sdks/python/apache_beam/io/iobase.py | 11 +-
sdks/python/apache_beam/io/parquetio_it_test.py | 10 +-
.../apache_beam/io/restriction_trackers_test.py | 7 -
sdks/python/apache_beam/io/sources_test.py | 7 +-
sdks/python/apache_beam/io/textio_test.py | 310 +++++-----
sdks/python/apache_beam/io/tfrecordio_test.py | 19 +-
sdks/python/apache_beam/io/vcfio.py | 3 -
sdks/python/apache_beam/io/vcfio_test.py | 25 +-
sdks/python/apache_beam/metrics/cells.py | 25 +-
sdks/python/apache_beam/pipeline.py | 40 +-
sdks/python/apache_beam/pipeline_test.py | 347 ++++++-----
sdks/python/apache_beam/portability/common_urns.py | 100 ++--
.../{internal/__init__.py => portability/utils.py} | 16 +-
sdks/python/apache_beam/pvalue.py | 11 +-
sdks/python/apache_beam/runners/common.pxd | 1 -
sdks/python/apache_beam/runners/common.py | 64 ++-
.../runners/dataflow/dataflow_runner.py | 4 +-
.../runners/dataflow/dataflow_runner_test.py | 66 +--
.../runners/dataflow/native_io/iobase.py | 6 +
.../runners/dataflow/native_io/iobase_test.py | 7 +-
.../runners/dataflow/template_runner_test.py | 24 +-
.../apache_beam/runners/direct/bundle_factory.py | 7 +-
.../apache_beam/runners/direct/direct_runner.py | 4 +
.../runners/direct/sdf_direct_runner.py | 14 +-
.../runners/direct/transform_evaluator.py | 1 +
.../runners/interactive/interactive_beam.py | 7 +-
.../python/apache_beam/runners/pipeline_context.py | 42 +-
.../runners/portability/abstract_job_service.py | 10 +-
.../runners/portability/artifact_service.py | 2 +-
.../runners/portability/fn_api_runner.py | 78 ++-
.../runners/portability/fn_api_runner_test.py | 8 +-
.../portability/fn_api_runner_transforms.py | 29 +-
.../runners/portability/portable_runner.py | 2 +-
.../apache_beam/runners/worker/bundle_processor.py | 168 ++++--
.../apache_beam/runners/worker/operations.py | 4 +
.../apache_beam/runners/worker/sdk_worker.py | 69 ++-
.../apache_beam/runners/worker/statecache.py | 15 +-
.../apache_beam/runners/worker/statesampler.py | 12 +-
.../runners/worker/statesampler_slow.py | 3 +-
.../testing/benchmarks/chicago_taxi/preprocess.py | 7 +-
.../benchmarks/chicago_taxi/process_tfma.py | 6 +-
.../testing/benchmarks/chicago_taxi/run_chicago.sh | 60 +-
.../chicago_taxi/tfdv_analyze_and_validate.py | 6 +-
sdks/python/apache_beam/testing/test_pipeline.py | 10 +-
.../apache_beam/testing/test_pipeline_test.py | 12 +-
.../python/apache_beam/testing/test_stream_test.py | 77 ++-
sdks/python/apache_beam/testing/util_test.py | 2 +-
sdks/python/apache_beam/transforms/combiners.py | 57 +-
.../apache_beam/transforms/combiners_test.py | 469 ++++++++++-----
sdks/python/apache_beam/transforms/core.py | 8 +-
sdks/python/apache_beam/transforms/display.py | 18 +-
.../apache_beam/transforms/dofn_lifecycle_test.py | 10 +-
sdks/python/apache_beam/transforms/environments.py | 78 ++-
.../python/apache_beam/transforms/external_test.py | 4 -
.../apache_beam/transforms/ptransform_test.py | 639 ++++++++++-----------
sdks/python/apache_beam/transforms/sideinputs.py | 11 +
sdks/python/apache_beam/transforms/stats_test.py | 236 ++++----
.../transforms_keyword_only_args_test_py3.py | 173 +++---
sdks/python/apache_beam/transforms/userstate.py | 33 +-
.../apache_beam/transforms/userstate_test.py | 53 +-
sdks/python/apache_beam/transforms/util_test.py | 272 +++++----
.../apache_beam/typehints/decorators_test_py3.py | 4 +-
sdks/python/apache_beam/typehints/opcodes.py | 63 +-
.../apache_beam/typehints/trivial_inference.py | 29 +-
.../typehints/trivial_inference_test.py | 35 ++
.../apache_beam/typehints/typed_pipeline_test.py | 25 +-
sdks/python/apache_beam/utils/profiler.py | 1 +
.../apache_beam/utils/thread_pool_executor.py | 2 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/build.gradle | 2 +-
sdks/python/container/py2/build.gradle | 2 +-
sdks/python/container/py35/build.gradle | 2 +-
sdks/python/container/py36/build.gradle | 2 +-
sdks/python/container/py37/build.gradle | 2 +-
sdks/python/gen_protos.py | 182 +++++-
sdks/python/scripts/run_integration_test.sh | 6 +-
sdks/python/setup.py | 7 +-
sdks/python/test-suites/dataflow/py2/build.gradle | 7 +-
sdks/python/test-suites/direct/py2/build.gradle | 3 +-
sdks/python/test-suites/direct/py37/build.gradle | 3 +-
sdks/python/test-suites/portable/py2/build.gradle | 27 +
sdks/python/tox.ini | 27 +
vendor/sdks-java-extensions-protobuf/build.gradle | 16 +-
.../src/_includes/section-menu/get-started.html | 2 +-
website/src/_posts/2020-01-06-beam-2.17.0.md | 2 +
website/src/contribute/index.md | 40 +-
website/src/contribute/release-guide.md | 6 +-
.../CVE-2020-1929.md} | 5 +-
website/src/security/index.md | 56 ++
482 files changed, 9099 insertions(+), 4354 deletions(-)
create mode 100644 .asf.yaml
create mode 100644 .test-infra/jenkins/job_PostCommit_Python_Chicago_Taxi_Example_Flink.groovy
delete mode 100644 .test-infra/tools/stale_dataflow_jobs_cleaner.go
copy .test-infra/{kubernetes/kafka-cluster/setup-cluster.sh => tools/stale_dataflow_jobs_cleaner.sh} (71%)
delete mode 100644 .test-infra/tools/stale_dataflow_jobs_cleaner_test.go
delete mode 100644 CONTRIBUTING.md
delete mode 100644 buildSrc/src/main/groovy/org/apache/beam/gradle/GrpcVendoring_1_21_0.groovy
copy sdks/java/io/kinesis/src/test/java/org/apache/beam/sdk/io/kinesis/CustomOptionalTest.java => runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/CloudObjectTest.java (55%)
copy sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigquery/BigQueryCoderProviderRegistrarTest.java => runners/google-cloud-dataflow-java/src/test/java/org/apache/beam/runners/dataflow/util/OutputReferenceTest.java (70%)
create mode 100644 runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/status/SdkWorkerStatusServlet.java
copy sdks/java/core/src/main/java/org/apache/beam/sdk/state/{package-info.java => TimerMap.java} (78%)
copy sdks/java/core/src/main/java/org/apache/beam/sdk/testing/{UsesSystemMetrics.java => UsesTimerMap.java} (82%)
copy sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/{bigquery/BigQueryFilterTest.java => mongodb/MongoDbFilterTest.java} (64%)
copy runners/java-fn-execution/src/main/java/org/apache/beam/runners/fnexecution/data/RemoteInputDestination.java => sdks/java/harness/src/main/java/org/apache/beam/fn/harness/HandlesSplits.java (62%)
delete mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/ElementCountFnDataReceiver.java
delete mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiver.java
delete mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/ElementCountFnDataReceiverTest.java
delete mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/data/MultiplexingFnDataReceiverTest.java
copy sdks/python/apache_beam/examples/snippets/transforms/aggregation/{mean.py => min.py} (75%)
copy sdks/python/apache_beam/examples/snippets/transforms/aggregation/{max_test.py => min_test.py} (67%)
copy sdks/python/apache_beam/examples/snippets/transforms/aggregation/{mean.py => sum.py} (75%)
copy sdks/python/apache_beam/examples/snippets/transforms/{elementwise/keys_test.py => aggregation/sum_test.py} (67%)
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/aggregation/top.py
create mode 100644 sdks/python/apache_beam/examples/snippets/transforms/aggregation/top_test.py
copy sdks/python/apache_beam/{testing/benchmarks/nexmark/models => io/gcp/experimental}/__init__.py (100%)
create mode 100644 sdks/python/apache_beam/io/gcp/experimental/spannerio.py
create mode 100644 sdks/python/apache_beam/io/gcp/experimental/spannerio_test.py
copy sdks/python/apache_beam/{internal/__init__.py => portability/utils.py} (70%)
copy website/src/{_includes/capability-matrix-row-blog.md => security/CVE-2020-1929.md} (81%)
create mode 100644 website/src/security/index.md