You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by bh...@apache.org on 2020/11/25 00:33:23 UTC
[beam] branch website-revamp updated (1f790fc -> 0850259)
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a change to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 1f790fc [BEAM-11182][Website revamp] Implemented Stay up to date with Beam and Changed Works with components (#13406)
add b0dd257 Squash, merge PR #13277 - [BEAM-10921] Add the UserPipelineTracker to track user pipelines for derived pipelines
add 9107a12 Increase the deadline for CreateReadSession to 2 hours
add 4e0925b Merge pull request #13295: [BEAM-11226] Increase the deadline for CreateReadSession to 2 hours
add 746cdfa Revert "Merge pull request #13321 from [BEAM-11255] Adding upper bound on urllib3 dependency"
add d09768e [BEAM-11255] Revert "Merge pull request #13321 from [BEAM-11255] Adding upper bound on urllib3 dependency" (#13328)
add fc4f1c7 add xvr dataflow badge
add beee718 Merge pull request #13336 from ihji/add_xvr_dataflow_badge
add 11c58f2 [BEAM-11211] Allow pyarrow up to 2.x, raise helpful error when trying to write wit… (#13302)
add 8e70cf6 More helpful error when we can't find a way to construct a POJO
add d9fe3f8 More helpful error when we can't construct a Java Bean
add 17c9d49 Add @SchemaCaseFormat, test with AutoValue beans and POJOs
add a6cd84c lint
add 47a6022 [BEAM-10989] Add @SchemaCaseFormat annotation (#13003)
add a8337b8 Update SDF related documentation.
add d2ff41e Merge pull request #13313 from [BEAM-10480] Update SDF related documentation.
add 31fd516 [BEAM-9980] do not hardcode Python version for cross language tests and make it switchable. (#13316)
add f46f050 use guava version specified in beam module
add 31bffe8 add guava dependency management to archetype
add a66c54d Merge pull request #13315: [BEAM-11228] Add guava to dependency management in generated pom
add 656c21b [BEAM-11192] Add use_runner_v2 to experiments.
add 43e8b99 Merge pull request #13339 from [BEAM-11192] Add use_runner_v2 to experiments.
add 7a2e401 Allow null next index for last report.
add 53a0954 Merge pull request #13343 Allow null next index for last report.
add 7aec62b Reshuffle in pd.read_* (#13346)
add 3d36122 [BEAM-11196] Ensure parent of fused stages is not one of its transforms (#13303)
add 4fd7dfd [BEAM-11188] Refactor Go xlang expansion from runners to core. (#13325)
add 01ccb08 Enable OrderedListState Dataflow tests.
add 4925b83 Merge pull request #13301: [BEAM-10650] Enable OrderedListState Dataflow tests.
add 945fa4b Change hbase-shaded-client as provided dependency
add 5ea2244 Update change.md file
add 483ab67 Merge pull request #13351: [BEAM-9278] hbase-shaded-client as provided dependency
add fb23d3f [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test in Java Postcommit
add 0ab83e9 Merge pull request #13287: [BEAM-8876] Run hadoop tests with different versions and enable HadoopFormatIT test on Postcommit
add b0ac11d [BEAM-8615 BEAM-8569 BEAM-7937] Add hadoop 3 compatibility tests
add 171c351 Merge pull request #13352: [BEAM-8615 BEAM-8569 BEAM-7937] Add hadoop 3 compatibility tests
add 143f8f9 Add trigger commands for Direct, Spark XVR Postcommits. (#13337)
add 2a22b77 [BEAM-9980] use constants of python versions in PythonTestProperties.groovy (#13349)
add 2f2ffda [BEAM-10139][BEAM-10140] Add cross-language support for Java SpannerIO with python wrapper (#12611)
add 6edc50c Use non-underscore methods for series and dataframe comparators.
add 83070b4 Only force indexing for multi-input elementwise operations.
add eba648c Merge pull request #13341 Avoid unnecessary shuffling for single-input elementwise operations.
add 252d2bc Fix Java ValidatesRunner V2 task dependency and update test exclusion.
add d66c99c Merge pull request #13259: Fix Java ValidatesRunner V2 task dependency.
add 1ae67c4 [BEAM-11265] Update quickstart-java.md (#13128)
add 78b9d05 [BEAM-11263] Java cleanUpDockerImages now force removes container images.
add 7ac82a8 Merge pull request #13344 from [BEAM-11263] Java cleanUpDockerImages now force removes container images.
add a7ea758 Further dataframe batch consolidation.
add 9ef95b5 Add another test.
add 6300c33 Update sdks/python/apache_beam/dataframe/transforms.py
add b18784f Merge pull request #13333 Further dataframe batch consolidation.
add 09a96db Upgrade gcsio to 2.1.6
add 41bdd91 Merge pull request #13311: [BEAM-8889] Upgrade GCSIO to 2.1.6
add 028becb Go redundant type cleanup. (#13348)
add baae15a [BEAM-11207] Metric Extraction via proto RPC API (#13272)
add 6c5011a [BEAM-11262] Remove numSleeps assertion in SpannerIOWriteTest (#13340)
add 9a3d3ea [BEAM-11270] Dataflow Java on runner v2 tests are failing because sdk docker container is cleaned up incorrectly
add 39b95c4 Merge pull request #13356 from [BEAM-11270] Fix container clean-up logic
add 5c8a748 Update Beam Dataflow container versions for Python
add 15cdc43 Merge pull request #13358: Update Beam Dataflow container versions for Python
add 689d89f Fix failing seed job
add 85b3023 Merge pull request #13363 from kamilwu/fix-seed-job
add 890332e [BEAM-4136] Keep strong reference to loggers to avoid potential NPE (#13345)
add a872aca Fix NPE in CountingSource
add fc41a9d Merge pull request #13361 from Fix NPE in CountingSource
add a95dbc8 Upgrade Conscrypt to 2.5.1 in GrpcVendoring_1_26_0
add ed09e96 Merge pull request #13150 from [BEAM-5748] Upgrade Conscrypt to 2.5.1 in GrpcVendoring_1_26_0
add fd87be2f add validate runner dataflow v2 java badge
add cf6f1d1 Merge pull request #13360 from ihji/add_dataflow_v2_java_badge
add 30c1eef [BEAM-8106] Separate Java8/11 container image build tasks (#13211)
add 1653a97 [BEAM-9782] Update README for instructions on updating katas (#13329)
add 62e8aeb [BEAM-8569] Add changes note about Hadoop 3 support
add 010af43 Merge pull request #13354: [BEAM-8569] Add changes note about Hadoop 3 support
add ed33e8c [BEAM-11207] Adding PipelineResult to session runner. (#13371)
add 5f133ca [BEAM-11211] parquetio_test using multiple pyarrow versions
add 1a6b9d8 Merge pull request #13369: [BEAM-11211] parquetio_test using multiple pyarrow versions
add d4c82ec Make UsesTestStream extend UsesUnboundedPCollections for exclusion in batch only runners
add 434bce5 Merge pull request #13376 from Make UsesTestStream extend UsesUnboundedPCollections for exclusion in batch-only runners
add 53a26ae [BEAM-11092] Add bigquery io request count metric, implementing HarnessMonitoringInfos and process_wide metrics
add 7c28db2 Merge pull request #13217 from [BEAM-11092] Add bigquery io request count metric, implementing HarnessMonitoringInfos and process_wide metrics
add cdb882d [BEAM-11303] Use sum as the post-agg for size (#13379)
add fc52b63 Update SDF programming guide.
add 774185b Merge pull request #13326 from [BEAM-10480] Update SDF programming guide
add a267d96 [BEAM-9615] Switch options for logical types. (#13366)
add 532b024 [BEAM-10475]Add WithShardedKey variation of GroupIntoBatches transform in Python SDK. (#13292)
add 70ddf84 [BEAM-11188] Adding unit test for mergeExpandedWithPipeline (#13370)
add 37deb55 [BEAM-11075] Pardo tests for Go SDK (#13362)
add d5eaa3f [BEAM-11304] groupby(Series).size() on a filtered dataframe is incorrect (#13380)
add c92a743 [BEAM-11308] Update deprecated Google GitHub action
add 18b82c6 Merge pull request #13384 from TobKed/beam-11308-update-deprecated-google-github-action
add b9f0538 [BEAM-10925] Move general-usage helper methods from DoFnSignatures to ReflectHelpers.
add 019ebb5 Merge pull request #13304 from ibzib/reflect-helpers
add 909a519 [BEAM-10379] Add bit_and back into Calcite SQL. (#13062)
add c4ac644 Fix Windmill timer tag output timestamp decoding
add 92ea33e Merge pull request #13373 from nehsyc/fix_timestamp_decoding
add 54225bd [BEAM-9547] Add support for Series.rename (#13385)
add fa5c512 [BEAM-113100] Exclude org.apache.beam.sdk.transforms.ViewTest.testWindowedSideInputNotPresent from flink validates runner test
add 2fed655 Merge pull request #13392 from [BEAM-11310] Exclude org.apache.beam.sdk.transforms.ViewTest.testWin
add 626551e [BEAM-11254] New documentation on multi-language pipelines (#13317)
add be9ef8c Updated shared.py comments (#13187)
add a0f1a54 [BEAM-10974] Skip GroupByKeyTest.testLargeKeys10MB.
add 1a34854 Merge pull request #13218 from ibzib/BEAM-10974
add 60e0221 Restore original str() behaviour to MetricName, for user counter(namespace+name) style MetricNames
add 6851109 Merge pull request #13400 from Restore original str() behaviour to MetricName, for user counter(namespace+name) style MetricNames
add 9054c7a set_index does not preserve partitioning (#13398)
add 07cbed5 Enable more runner v2 tests.
add 208690c Merge pull request #13372 from boyuanzz/enable_test
add b29102e [BEAM-11070] Use self-checkpoint to enfore finalization happens.
add 9b51d4b Merge pull request #13338 from [BEAM-11070] Use self-checkpoint to enforce finalization happens.
add 353ca39 fix java 11 container task
add 213385d add java 11 examples test
add 19eaf71 add java 11 image to released images
add fa7c5fc Merge pull request #13374: [BEAM-8106] Add Java 11 container to release and Java 11+V2+Dataflow testing
add 67339a9 [BEAM-11266] Python IO MongoDB: add bucket_auto aggregation option for bundling in Atlas. (#13350)
add ab2e94d Add use_unified_worker to pipeline options.
add ae1df91 Merge pull request #13396 from boyuanzz/taxi_cmd
add f5ba29f [BEAM-10593] Add Jenkins job for creating regular snapshots of the Beam SDK Harness Container Images (#13393)
add 6cc3a92 Support ZetaSQL IN operator
add 8b4b78e Merge pull request #13381 from robinyqiu/in
add 6122341 Nexmark: fix typos and clean unused options
add 4507139 Merge pull request #13403 from rHermes/nexmark-misc-improvements
add b51d64e Add an option to GroupIntoBatches to output ShardedKeys. Update Dataflow pipeline translation accordingly.
add f4d889f Merge pull request #13208 from nehsyc/fix_override
add ab1d27c [BEAM-10124] Announce ContextualTextIO in Beam 2.26.0. (#13402)
add 6a0fbe2 [BEAM-10896] Support ZetaSQL UNNEST syntax on array of struct type (#13377)
add 99b4669 add to dashboard
add 4f34b9d fix Java 11 HOME
add 245374a Merge pull request #13416: Fix Java 11 V2 CI test and add V2 tests to dashboard
add bd825f5 [BEAM-11324] Add additional verification in PartitioningSession (#13401)
add d7655c1 [BEAM-10921]: Fix BEAM-10921 and underlying issues (#13335)
new 0850259 Merge remote-tracking branch 'origin/master' into website-revamp
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.github/PULL_REQUEST_TEMPLATE.md | 4 +-
.github/workflows/build_wheels.yml | 8 +-
.github/workflows/java_tests.yml | 2 +-
.github/workflows/python_tests.yml | 2 +-
.test-infra/jenkins/CommonTestProperties.groovy | 12 +-
...Properties.groovy => JavaTestProperties.groovy} | 4 +-
.test-infra/jenkins/LoadTestsBuilder.groovy | 15 +-
.test-infra/jenkins/PythonTestProperties.groovy | 10 +
.test-infra/jenkins/README.md | 6 +-
.../jenkins/job_LoadTests_ParDo_Flink_Go.groovy | 150 +++++
...it_CrossLanguageValidatesRunner_Dataflow.groovy | 38 +-
...mmit_CrossLanguageValidatesRunner_Direct.groovy | 23 +-
...ommit_CrossLanguageValidatesRunner_Flink.groovy | 23 +-
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 23 +-
...tCommit_Java_Examples_Dataflow_V2_Java11.groovy | 50 ++
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 1 -
.../jenkins/job_Publish_Docker_Snapshots.groovy | 8 +-
.../jenkins/job_Publish_SDK_Image_Snapshots.groovy | 55 ++
CHANGES.md | 9 +-
README.md | 4 +-
build.gradle | 28 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 20 +-
.../apache/beam/gradle/GrpcVendoring_1_26_0.groovy | 2 +-
learning/katas/go/README.md | 29 +
.../src/main/scripts/build_release_candidate.sh | 6 +-
release/src/main/scripts/publish_docker_images.sh | 2 +-
.../core/construction/PTransformMatchers.java | 2 +-
runners/flink/job-server/flink_job_server.gradle | 5 +-
runners/google-cloud-dataflow-java/build.gradle | 78 ++-
.../dataflow/DataflowPipelineTranslator.java | 14 +-
.../beam/runners/dataflow/DataflowRunner.java | 29 +-
.../runners/dataflow/GroupIntoBatchesOverride.java | 132 ++++-
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../dataflow/DataflowPipelineTranslatorTest.java | 67 ++-
.../beam/runners/dataflow/DataflowRunnerTest.java | 64 ++-
.../dataflow/worker/WindmillTimerInternals.java | 13 +-
.../dataflow/worker/WorkItemStatusClient.java | 2 +-
.../worker/WindmillTimerInternalsTest.java | 4 +-
runners/portability/java/build.gradle | 9 +-
runners/spark/build.gradle | 66 ++-
sdks/go/pkg/beam/core/metrics/metrics.go | 118 ++++
.../core/runtime/graphx/schema/logicaltypes.go | 121 ++++
.../pkg/beam/core/runtime/graphx/schema/schema.go | 115 ++--
.../beam/core/runtime/graphx/schema/schema_test.go | 55 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 44 +-
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 210 +++++++
sdks/go/pkg/beam/core/runtime/graphx/xlang_test.go | 218 +++++++
.../go/pkg/beam/core/runtime/harness/monitoring.go | 187 +-----
.../beam/core/runtime/harness/monitoring_test.go | 33 +-
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 172 ++++++
.../beam/core/runtime/metricsx/metricsx_test.go | 166 ++++++
sdks/go/pkg/beam/core/runtime/metricsx/urns.go | 170 ++++++
sdks/go/pkg/beam/core/runtime/options.go | 1 +
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 69 ++-
sdks/go/pkg/beam/core/runtime/xlangx/namespace.go | 2 +-
.../pkg/beam/core/runtime/xlangx/namespace_test.go | 348 +++++------
sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 228 --------
sdks/go/pkg/beam/doc_test.go | 2 +-
sdks/go/pkg/beam/io/textio/sdf_test.go | 2 +-
sdks/go/pkg/beam/options/jobopts/options.go | 3 +
sdks/go/pkg/beam/pipeline.go | 6 +
sdks/go/pkg/beam/runner.go | 6 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 25 +-
sdks/go/pkg/beam/runners/direct/direct.go | 17 +-
sdks/go/pkg/beam/runners/dot/dot.go | 10 +-
sdks/go/pkg/beam/runners/flink/flink.go | 3 +-
sdks/go/pkg/beam/runners/session/session.go | 6 +-
sdks/go/pkg/beam/runners/spark/spark.go | 3 +-
.../beam/runners/universal/runnerlib/execute.go | 48 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 3 +
sdks/go/pkg/beam/runners/universal/universal.go | 28 +-
sdks/go/pkg/beam/runners/vet/vet.go | 8 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 3 +-
sdks/go/pkg/beam/x/beamx/run.go | 8 +
sdks/go/pkg/beam/xlang.go | 59 +-
sdks/go/test/load/build.gradle | 66 +++
sdks/go/test/load/pardo/pardo.go | 112 ++++
sdks/go/test/load/util.go | 227 ++++++++
sdks/java/container/build.gradle | 63 +-
sdks/java/container/common.gradle | 104 ++++
.../java/container/java11/build.gradle | 11 +-
.../java/container/java8/build.gradle | 11 +-
.../org/apache/beam/sdk/io/CountingSource.java | 3 +-
.../sdk/schemas/FieldValueTypeInformation.java | 32 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 27 +-
.../sdk/schemas/annotations/SchemaCaseFormat.java | 73 +++
.../sdk/schemas/annotations/SchemaFieldName.java | 11 +-
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 39 +-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 5 +-
.../apache/beam/sdk/testing/UsesTestStream.java | 2 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 4 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 70 ++-
.../sdk/transforms/reflect/DoFnSignatures.java | 77 +--
.../splittabledofn/RestrictionTracker.java | 23 +-
.../beam/sdk/util/common/ReflectHelpers.java | 60 ++
.../beam/sdk/schemas/AutoValueSchemaTest.java | 57 ++
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 73 +++
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 55 ++
.../beam/sdk/schemas/utils/SchemaTestUtils.java | 22 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 129 +++++
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 56 ++
.../beam/sdk/transforms/GroupIntoBatchesTest.java | 82 +++
sdks/java/extensions/sorter/build.gradle | 46 +-
sdks/java/extensions/sql/build.gradle | 44 +-
.../impl/transform/BeamBuiltinAggregations.java | 143 ++---
.../provider/bigquery/BeamBigQuerySqlDialect.java | 11 +
.../extensions/sql/BeamSqlDslAggregationTest.java | 263 ++++-----
.../zetasql/SupportedZetaSqlBuiltinFunctions.java | 4 +-
.../sql/zetasql/ZetaSQLQueryPlanner.java | 8 +
.../translation/ArrayScanColumnRefToUncollect.java | 4 +-
.../ArrayScanLiteralToUncollectConverter.java | 4 +-
.../translation/ArrayScanToJoinConverter.java | 5 +-
.../translation/SqlOperatorMappingTable.java | 1 +
.../zetasql/unnest/BeamZetaSqlUncollectRel.java | 120 ++++
.../zetasql/unnest/BeamZetaSqlUncollectRule.java | 54 ++
.../sql/zetasql/unnest/BeamZetaSqlUnnestRel.java | 164 ++++++
.../sql/zetasql/unnest/BeamZetaSqlUnnestRule.java | 120 ++++
.../sql/zetasql/unnest/ZetaSqlUnnest.java | 140 +++++
.../sql/zetasql/unnest/package-info.java | 8 +-
.../beam/sdk/extensions/sql/zetasql/TestInput.java | 66 ++-
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 137 ++++-
.../extensions/sql/zetasql/ZetaSqlTestBase.java | 8 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 50 +-
.../harness/logging/BeamFnLoggingClientTest.java | 50 +-
.../expansion-service/build.gradle | 37 ++
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 22 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 320 +++++++++++
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 6 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 8 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 112 +++-
.../io/gcp/spanner/SpannerTransformRegistrar.java | 374 ++++++++++++
.../beam/sdk/io/gcp/spanner/StructUtils.java | 387 +++++++++++++
.../beam/sdk/io/gcp/spanner/MutationUtilsTest.java | 285 +++++++++
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 5 +-
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 258 +++++++++
sdks/java/io/hadoop-common/build.gradle | 50 +-
sdks/java/io/hadoop-file-system/build.gradle | 57 +-
sdks/java/io/hadoop-format/build.gradle | 104 +++-
.../io/hadoop/format/HadoopFormatIOElasticIT.java | 20 +-
.../sdk/io/hadoop/format/HadoopFormatIOIT.java | 58 +-
.../hadoop/format/HadoopFormatIOTestOptions.java | 6 +
sdks/java/io/hbase/build.gradle | 2 +-
sdks/java/io/hcatalog/build.gradle | 42 +-
sdks/java/io/parquet/build.gradle | 43 +-
sdks/java/maven-archetypes/examples/build.gradle | 1 +
sdks/java/maven-archetypes/examples/sample.txt | 102 ++++
.../src/main/resources/archetype-resources/pom.xml | 8 +-
.../beam/sdk/nexmark/NexmarkConfiguration.java | 2 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 4 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 30 +-
.../sdk/nexmark/queries/SessionSideInputJoin.java | 2 +-
sdks/python/apache_beam/dataframe/expressions.py | 92 ++-
sdks/python/apache_beam/dataframe/frame_base.py | 8 +-
sdks/python/apache_beam/dataframe/frames.py | 65 ++-
sdks/python/apache_beam/dataframe/frames_test.py | 91 +--
sdks/python/apache_beam/dataframe/io.py | 1 +
.../apache_beam/dataframe/pandas_doctests_test.py | 3 +-
sdks/python/apache_beam/dataframe/partitionings.py | 26 +
sdks/python/apache_beam/dataframe/transforms.py | 62 +-
.../apache_beam/dataframe/transforms_test.py | 17 +-
.../apache_beam/examples/kafkataxi/README.md | 6 +-
sdks/python/apache_beam/internal/metrics/metric.py | 90 +++
.../apache_beam/internal/metrics/metric_test.py | 41 ++
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 36 +-
.../gcp/resource_identifiers.py} | 34 +-
sdks/python/apache_beam/io/gcp/spanner.py | 635 +++++++++++++++++++++
.../io/gcp/tests/xlang_spannerio_it_test.py | 339 +++++++++++
sdks/python/apache_beam/io/iobase.py | 12 +-
sdks/python/apache_beam/io/mongodbio.py | 166 ++++--
sdks/python/apache_beam/io/mongodbio_it_test.py | 94 ++-
sdks/python/apache_beam/io/mongodbio_test.py | 235 +++++++-
sdks/python/apache_beam/io/parquetio.py | 8 +
sdks/python/apache_beam/io/parquetio_test.py | 4 +
sdks/python/apache_beam/metrics/cells.py | 22 +-
sdks/python/apache_beam/metrics/execution.pxd | 4 +-
sdks/python/apache_beam/metrics/execution.py | 71 ++-
sdks/python/apache_beam/metrics/metric.py | 18 +-
sdks/python/apache_beam/metrics/metric_test.py | 16 +
sdks/python/apache_beam/metrics/metricbase.py | 46 +-
.../python/apache_beam/metrics/monitoring_infos.py | 21 +-
.../apache_beam/metrics/monitoring_infos_test.py | 35 ++
sdks/python/apache_beam/pipeline.py | 12 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/interactive/background_caching_job.py | 11 +-
.../interactive/background_caching_job_test.py | 42 +-
.../runners/interactive/interactive_environment.py | 49 +-
.../runners/interactive/interactive_runner.py | 4 +-
.../runners/interactive/interactive_runner_test.py | 3 -
.../runners/interactive/pipeline_fragment.py | 9 +-
.../runners/interactive/pipeline_instrument.py | 29 +-
.../runners/interactive/recording_manager.py | 20 +-
.../runners/interactive/recording_manager_test.py | 61 +-
.../runners/interactive/user_pipeline_tracker.py | 150 +++++
.../interactive/user_pipeline_tracker_test.py | 209 +++++++
.../runners/portability/flink_runner_test.py | 3 +-
.../portability/fn_api_runner/fn_runner_test.py | 81 ++-
.../portability/fn_api_runner/translations.py | 29 +-
.../portability/fn_api_runner/translations_test.py | 39 ++
.../apache_beam/runners/worker/sdk_worker.py | 38 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 60 ++
sdks/python/apache_beam/transforms/core.py | 28 +-
sdks/python/apache_beam/transforms/util.py | 49 +-
sdks/python/apache_beam/transforms/util_test.py | 17 +
sdks/python/apache_beam/utils/sharded_key.py | 2 +-
sdks/python/apache_beam/utils/shared.py | 24 +-
sdks/python/setup.py | 4 +-
sdks/python/test-suites/portable/common.gradle | 6 +-
sdks/python/test-suites/tox/common.gradle | 3 +-
sdks/python/test-suites/tox/py38/build.gradle | 7 +
sdks/python/tox.ini | 21 +
settings.gradle | 4 +
.../content/en/documentation/programming-guide.md | 365 +++++++++++-
.../en/documentation/runtime/environments.md | 18 +-
.../site/content/en/get-started/quickstart-java.md | 20 +-
.../partials/section-menu/en/documentation.html | 10 +
.../images/multi-language-pipelines-diagram.svg | 20 +
216 files changed, 10389 insertions(+), 2166 deletions(-)
copy .test-infra/jenkins/{PythonTestProperties.groovy => JavaTestProperties.groovy} (88%)
create mode 100644 .test-infra/jenkins/job_LoadTests_ParDo_Flink_Go.groovy
create mode 100644 .test-infra/jenkins/job_PostCommit_Java_Examples_Dataflow_V2_Java11.groovy
create mode 100644 .test-infra/jenkins/job_Publish_SDK_Image_Snapshots.groovy
create mode 100644 sdks/go/pkg/beam/core/runtime/graphx/schema/logicaltypes.go
create mode 100644 sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go
create mode 100644 sdks/go/pkg/beam/core/runtime/metricsx/metricsx_test.go
create mode 100644 sdks/go/pkg/beam/core/runtime/metricsx/urns.go
delete mode 100644 sdks/go/pkg/beam/core/runtime/xlangx/translate.go
create mode 100644 sdks/go/test/load/build.gradle
create mode 100644 sdks/go/test/load/pardo/pardo.go
create mode 100644 sdks/go/test/load/util.go
create mode 100644 sdks/java/container/common.gradle
copy .test-infra/jenkins/PythonTestProperties.groovy => sdks/java/container/java11/build.gradle (72%)
copy .test-infra/jenkins/PythonTestProperties.groovy => sdks/java/container/java8/build.gradle (72%)
create mode 100644 sdks/java/core/src/main/java/org/apache/beam/sdk/schemas/annotations/SchemaCaseFormat.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUncollectRel.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUncollectRule.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUnnestRel.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/BeamZetaSqlUnnestRule.java
create mode 100644 sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/ZetaSqlUnnest.java
copy .test-infra/jenkins/PythonTestProperties.groovy => sdks/java/extensions/sql/zetasql/src/main/java/org/apache/beam/sdk/extensions/sql/zetasql/unnest/package-info.java (78%)
create mode 100644 sdks/java/io/google-cloud-platform/expansion-service/build.gradle
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/SpannerTransformRegistrar.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/spanner/StructUtils.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/MutationUtilsTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/spanner/StructUtilsTest.java
create mode 100644 sdks/java/maven-archetypes/examples/sample.txt
copy sdks/python/apache_beam/{metrics/execution.pxd => io/gcp/resource_identifiers.py} (56%)
create mode 100644 sdks/python/apache_beam/io/gcp/spanner.py
create mode 100644 sdks/python/apache_beam/io/gcp/tests/xlang_spannerio_it_test.py
create mode 100644 sdks/python/apache_beam/runners/interactive/user_pipeline_tracker.py
create mode 100644 sdks/python/apache_beam/runners/interactive/user_pipeline_tracker_test.py
create mode 100644 website/www/site/static/images/multi-language-pipelines-diagram.svg
[beam] 01/01: Merge remote-tracking branch 'origin/master' into
website-revamp
Posted by bh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 0850259add2661e5d169dd939b18c270b5ffefcf
Merge: 1f790fc d7655c1
Author: Brian Hulette <bh...@google.com>
AuthorDate: Tue Nov 24 16:28:51 2020 -0800
Merge remote-tracking branch 'origin/master' into website-revamp
.github/PULL_REQUEST_TEMPLATE.md | 4 +-
.github/workflows/build_wheels.yml | 8 +-
.github/workflows/java_tests.yml | 2 +-
.github/workflows/python_tests.yml | 2 +-
.test-infra/jenkins/CommonTestProperties.groovy | 12 +-
...Properties.groovy => JavaTestProperties.groovy} | 4 +-
.test-infra/jenkins/LoadTestsBuilder.groovy | 15 +-
.test-infra/jenkins/PythonTestProperties.groovy | 10 +
.test-infra/jenkins/README.md | 6 +-
.../jenkins/job_LoadTests_ParDo_Flink_Go.groovy | 150 +++++
...it_CrossLanguageValidatesRunner_Dataflow.groovy | 38 +-
...mmit_CrossLanguageValidatesRunner_Direct.groovy | 23 +-
...ommit_CrossLanguageValidatesRunner_Flink.groovy | 23 +-
...ommit_CrossLanguageValidatesRunner_Spark.groovy | 23 +-
...tCommit_Java_Examples_Dataflow_V2_Java11.groovy | 50 ++
..._PreCommit_Java_Examples_Dataflow_Java11.groovy | 1 -
.../jenkins/job_Publish_Docker_Snapshots.groovy | 8 +-
.../jenkins/job_Publish_SDK_Image_Snapshots.groovy | 55 ++
CHANGES.md | 9 +-
README.md | 4 +-
build.gradle | 28 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 20 +-
.../apache/beam/gradle/GrpcVendoring_1_26_0.groovy | 2 +-
learning/katas/go/README.md | 29 +
.../src/main/scripts/build_release_candidate.sh | 6 +-
release/src/main/scripts/publish_docker_images.sh | 2 +-
.../core/construction/PTransformMatchers.java | 2 +-
runners/flink/job-server/flink_job_server.gradle | 5 +-
runners/google-cloud-dataflow-java/build.gradle | 78 ++-
.../dataflow/DataflowPipelineTranslator.java | 14 +-
.../beam/runners/dataflow/DataflowRunner.java | 29 +-
.../runners/dataflow/GroupIntoBatchesOverride.java | 132 ++++-
.../beam/runners/dataflow/util/PropertyNames.java | 1 +
.../dataflow/DataflowPipelineTranslatorTest.java | 67 ++-
.../beam/runners/dataflow/DataflowRunnerTest.java | 64 ++-
.../dataflow/worker/WindmillTimerInternals.java | 13 +-
.../dataflow/worker/WorkItemStatusClient.java | 2 +-
.../worker/WindmillTimerInternalsTest.java | 4 +-
runners/portability/java/build.gradle | 9 +-
runners/spark/build.gradle | 66 ++-
sdks/go/pkg/beam/core/metrics/metrics.go | 118 ++++
.../core/runtime/graphx/schema/logicaltypes.go | 121 ++++
.../pkg/beam/core/runtime/graphx/schema/schema.go | 115 ++--
.../beam/core/runtime/graphx/schema/schema_test.go | 55 +-
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 44 +-
sdks/go/pkg/beam/core/runtime/graphx/xlang.go | 210 +++++++
sdks/go/pkg/beam/core/runtime/graphx/xlang_test.go | 218 +++++++
.../go/pkg/beam/core/runtime/harness/monitoring.go | 187 +-----
.../beam/core/runtime/harness/monitoring_test.go | 33 +-
sdks/go/pkg/beam/core/runtime/metricsx/metricsx.go | 172 ++++++
.../beam/core/runtime/metricsx/metricsx_test.go | 166 ++++++
sdks/go/pkg/beam/core/runtime/metricsx/urns.go | 170 ++++++
sdks/go/pkg/beam/core/runtime/options.go | 1 +
sdks/go/pkg/beam/core/runtime/xlangx/expand.go | 69 ++-
sdks/go/pkg/beam/core/runtime/xlangx/namespace.go | 2 +-
.../pkg/beam/core/runtime/xlangx/namespace_test.go | 348 +++++------
sdks/go/pkg/beam/core/runtime/xlangx/translate.go | 228 --------
sdks/go/pkg/beam/doc_test.go | 2 +-
sdks/go/pkg/beam/io/textio/sdf_test.go | 2 +-
sdks/go/pkg/beam/options/jobopts/options.go | 3 +
sdks/go/pkg/beam/pipeline.go | 6 +
sdks/go/pkg/beam/runner.go | 6 +-
sdks/go/pkg/beam/runners/dataflow/dataflow.go | 25 +-
sdks/go/pkg/beam/runners/direct/direct.go | 17 +-
sdks/go/pkg/beam/runners/dot/dot.go | 10 +-
sdks/go/pkg/beam/runners/flink/flink.go | 3 +-
sdks/go/pkg/beam/runners/session/session.go | 6 +-
sdks/go/pkg/beam/runners/spark/spark.go | 3 +-
.../beam/runners/universal/runnerlib/execute.go | 48 +-
.../go/pkg/beam/runners/universal/runnerlib/job.go | 3 +
sdks/go/pkg/beam/runners/universal/universal.go | 28 +-
sdks/go/pkg/beam/runners/vet/vet.go | 8 +-
sdks/go/pkg/beam/testing/ptest/ptest.go | 3 +-
sdks/go/pkg/beam/x/beamx/run.go | 8 +
sdks/go/pkg/beam/xlang.go | 59 +-
sdks/go/test/load/build.gradle | 66 +++
sdks/go/test/load/pardo/pardo.go | 112 ++++
sdks/go/test/load/util.go | 227 ++++++++
sdks/java/container/build.gradle | 63 +-
sdks/java/container/common.gradle | 104 ++++
.../java/container/java11/build.gradle | 11 +-
.../java/container/java8/build.gradle | 11 +-
.../org/apache/beam/sdk/io/CountingSource.java | 3 +-
.../sdk/schemas/FieldValueTypeInformation.java | 32 +-
.../apache/beam/sdk/schemas/JavaBeanSchema.java | 27 +-
.../sdk/schemas/annotations/SchemaCaseFormat.java | 73 +++
.../sdk/schemas/annotations/SchemaFieldName.java | 11 +-
.../beam/sdk/schemas/utils/JavaBeanUtils.java | 39 +-
.../apache/beam/sdk/schemas/utils/POJOUtils.java | 5 +-
.../apache/beam/sdk/testing/UsesTestStream.java | 2 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 4 +-
.../beam/sdk/transforms/GroupIntoBatches.java | 70 ++-
.../sdk/transforms/reflect/DoFnSignatures.java | 77 +--
.../splittabledofn/RestrictionTracker.java | 23 +-
.../beam/sdk/util/common/ReflectHelpers.java | 60 ++
.../beam/sdk/schemas/AutoValueSchemaTest.java | 57 ++
.../beam/sdk/schemas/JavaBeanSchemaTest.java | 73 +++
.../beam/sdk/schemas/JavaFieldSchemaTest.java | 55 ++
.../beam/sdk/schemas/utils/SchemaTestUtils.java | 22 +
.../beam/sdk/schemas/utils/TestJavaBeans.java | 129 +++++
.../apache/beam/sdk/schemas/utils/TestPOJOs.java | 56 ++
.../beam/sdk/transforms/GroupIntoBatchesTest.java | 82 +++
sdks/java/extensions/sorter/build.gradle | 46 +-
sdks/java/extensions/sql/build.gradle | 44 +-
.../impl/transform/BeamBuiltinAggregations.java | 143 ++---
.../provider/bigquery/BeamBigQuerySqlDialect.java | 11 +
.../extensions/sql/BeamSqlDslAggregationTest.java | 263 ++++-----
.../zetasql/SupportedZetaSqlBuiltinFunctions.java | 4 +-
.../sql/zetasql/ZetaSQLQueryPlanner.java | 8 +
.../translation/ArrayScanColumnRefToUncollect.java | 4 +-
.../ArrayScanLiteralToUncollectConverter.java | 4 +-
.../translation/ArrayScanToJoinConverter.java | 5 +-
.../translation/SqlOperatorMappingTable.java | 1 +
.../zetasql/unnest/BeamZetaSqlUncollectRel.java | 120 ++++
.../zetasql/unnest/BeamZetaSqlUncollectRule.java | 54 ++
.../sql/zetasql/unnest/BeamZetaSqlUnnestRel.java | 164 ++++++
.../sql/zetasql/unnest/BeamZetaSqlUnnestRule.java | 120 ++++
.../sql/zetasql/unnest/ZetaSqlUnnest.java | 140 +++++
.../sql/zetasql/unnest/package-info.java | 8 +-
.../beam/sdk/extensions/sql/zetasql/TestInput.java | 66 ++-
.../sql/zetasql/ZetaSqlDialectSpecTest.java | 137 ++++-
.../extensions/sql/zetasql/ZetaSqlTestBase.java | 8 +-
.../fn/harness/logging/BeamFnLoggingClient.java | 50 +-
.../harness/logging/BeamFnLoggingClientTest.java | 50 +-
.../expansion-service/build.gradle | 37 ++
.../sdk/io/gcp/bigquery/BigQueryServicesImpl.java | 22 +-
.../beam/sdk/io/gcp/spanner/MutationUtils.java | 320 +++++++++++
.../beam/sdk/io/gcp/spanner/SpannerAccessor.java | 6 +
.../beam/sdk/io/gcp/spanner/SpannerConfig.java | 8 +
.../apache/beam/sdk/io/gcp/spanner/SpannerIO.java | 112 +++-
.../io/gcp/spanner/SpannerTransformRegistrar.java | 374 ++++++++++++
.../beam/sdk/io/gcp/spanner/StructUtils.java | 387 +++++++++++++
.../beam/sdk/io/gcp/spanner/MutationUtilsTest.java | 285 +++++++++
.../sdk/io/gcp/spanner/SpannerIOWriteTest.java | 5 +-
.../beam/sdk/io/gcp/spanner/StructUtilsTest.java | 258 +++++++++
sdks/java/io/hadoop-common/build.gradle | 50 +-
sdks/java/io/hadoop-file-system/build.gradle | 57 +-
sdks/java/io/hadoop-format/build.gradle | 104 +++-
.../io/hadoop/format/HadoopFormatIOElasticIT.java | 20 +-
.../sdk/io/hadoop/format/HadoopFormatIOIT.java | 58 +-
.../hadoop/format/HadoopFormatIOTestOptions.java | 6 +
sdks/java/io/hbase/build.gradle | 2 +-
sdks/java/io/hcatalog/build.gradle | 42 +-
sdks/java/io/parquet/build.gradle | 43 +-
sdks/java/maven-archetypes/examples/build.gradle | 1 +
sdks/java/maven-archetypes/examples/sample.txt | 102 ++++
.../src/main/resources/archetype-resources/pom.xml | 8 +-
.../beam/sdk/nexmark/NexmarkConfiguration.java | 2 +-
.../apache/beam/sdk/nexmark/NexmarkLauncher.java | 4 +-
.../apache/beam/sdk/nexmark/NexmarkOptions.java | 30 +-
.../sdk/nexmark/queries/SessionSideInputJoin.java | 2 +-
sdks/python/apache_beam/dataframe/expressions.py | 92 ++-
sdks/python/apache_beam/dataframe/frame_base.py | 8 +-
sdks/python/apache_beam/dataframe/frames.py | 65 ++-
sdks/python/apache_beam/dataframe/frames_test.py | 91 +--
sdks/python/apache_beam/dataframe/io.py | 1 +
.../apache_beam/dataframe/pandas_doctests_test.py | 3 +-
sdks/python/apache_beam/dataframe/partitionings.py | 26 +
sdks/python/apache_beam/dataframe/transforms.py | 62 +-
.../apache_beam/dataframe/transforms_test.py | 17 +-
.../apache_beam/examples/kafkataxi/README.md | 6 +-
sdks/python/apache_beam/internal/metrics/metric.py | 90 +++
.../apache_beam/internal/metrics/metric_test.py | 41 ++
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 36 +-
.../gcp/resource_identifiers.py} | 34 +-
sdks/python/apache_beam/io/gcp/spanner.py | 635 +++++++++++++++++++++
.../io/gcp/tests/xlang_spannerio_it_test.py | 339 +++++++++++
sdks/python/apache_beam/io/iobase.py | 12 +-
sdks/python/apache_beam/io/mongodbio.py | 166 ++++--
sdks/python/apache_beam/io/mongodbio_it_test.py | 94 ++-
sdks/python/apache_beam/io/mongodbio_test.py | 235 +++++++-
sdks/python/apache_beam/io/parquetio.py | 8 +
sdks/python/apache_beam/io/parquetio_test.py | 4 +
sdks/python/apache_beam/metrics/cells.py | 22 +-
sdks/python/apache_beam/metrics/execution.pxd | 4 +-
sdks/python/apache_beam/metrics/execution.py | 71 ++-
sdks/python/apache_beam/metrics/metric.py | 18 +-
sdks/python/apache_beam/metrics/metric_test.py | 16 +
sdks/python/apache_beam/metrics/metricbase.py | 46 +-
.../python/apache_beam/metrics/monitoring_infos.py | 21 +-
.../apache_beam/metrics/monitoring_infos_test.py | 35 ++
sdks/python/apache_beam/pipeline.py | 12 +-
.../apache_beam/runners/dataflow/internal/names.py | 4 +-
.../runners/interactive/background_caching_job.py | 11 +-
.../interactive/background_caching_job_test.py | 42 +-
.../runners/interactive/interactive_environment.py | 49 +-
.../runners/interactive/interactive_runner.py | 4 +-
.../runners/interactive/interactive_runner_test.py | 3 -
.../runners/interactive/pipeline_fragment.py | 9 +-
.../runners/interactive/pipeline_instrument.py | 29 +-
.../runners/interactive/recording_manager.py | 20 +-
.../runners/interactive/recording_manager_test.py | 61 +-
.../runners/interactive/user_pipeline_tracker.py | 150 +++++
.../interactive/user_pipeline_tracker_test.py | 209 +++++++
.../runners/portability/flink_runner_test.py | 3 +-
.../portability/fn_api_runner/fn_runner_test.py | 81 ++-
.../portability/fn_api_runner/translations.py | 29 +-
.../portability/fn_api_runner/translations_test.py | 39 ++
.../apache_beam/runners/worker/sdk_worker.py | 38 +-
.../apache_beam/runners/worker/sdk_worker_test.py | 60 ++
sdks/python/apache_beam/transforms/core.py | 28 +-
sdks/python/apache_beam/transforms/util.py | 49 +-
sdks/python/apache_beam/transforms/util_test.py | 17 +
sdks/python/apache_beam/utils/sharded_key.py | 2 +-
sdks/python/apache_beam/utils/shared.py | 24 +-
sdks/python/setup.py | 4 +-
sdks/python/test-suites/portable/common.gradle | 6 +-
sdks/python/test-suites/tox/common.gradle | 3 +-
sdks/python/test-suites/tox/py38/build.gradle | 7 +
sdks/python/tox.ini | 21 +
settings.gradle | 4 +
.../content/en/documentation/programming-guide.md | 365 +++++++++++-
.../en/documentation/runtime/environments.md | 18 +-
.../site/content/en/get-started/quickstart-java.md | 20 +-
.../partials/section-menu/en/documentation.html | 10 +
.../images/multi-language-pipelines-diagram.svg | 20 +
216 files changed, 10389 insertions(+), 2166 deletions(-)