You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by bh...@apache.org on 2020/12/08 02:10:21 UTC
[beam] branch website-revamp updated (140a913 -> 8b22fb4)
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a change to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 140a913 [Website revamp][11179 - 11180 - 11183] Implemented Quotes component, Header and Navbar (#13439)
add 8be8203 remove tabs in shared
add b21b573 Merge pull request #13422 from apache/aaltay-patch-1
add 331924f [BEAM-9980] do not use hardcoded python versions in validate container dataflow jobs (#13382)
add 0db6f9a [BEAM-9980] do not use hardcoded python versions in chicago taxi example flink job (#13412)
add e58246d Go SDK Combine load tests (#13389)
add ad0c7f6 Go SDK Group by key load tests (#13388)
add f8e8daf [BEAM-10278] Move pipeline jar tests from job server projects to python/test-suites/portable.
add b002530 Merge pull request #13417 from ibzib/BEAM-10278
add 72b8894 [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 50aaf77 [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 296f3a4 Merge branch 'master' into multi-pubsub-reader
add d0263c7 Merge remote-tracking branch 'origin/multi-pubsub-reader' into multi-pubsub-reader
add 34b92bf [BEAM-10962] Add Multiple PubSub reader to Python SDK
add e046a63 [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 660cdf5 [BEAM-10962] Add Multiple PubSub reader to Python SDK
add dfc57ee [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 0a0dc6b [BEAM-10962] Add Multiple PubSub reader to Python SDK
add c33e97f [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 2508a4f [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 2e7bc4a [BEAM-10962] Add Multiple PubSub reader to Python SDK
add a279c7d [BEAM-10962] Add Multiple PubSub reader to Python SDK
add 17af76c Merge branch 'master' into multi-pubsub-reader
add 682f2ea Merge pull request #12930 from [BEAM-10962] Add Multiple PubSub reader to Python SDK
add d29095d Advance the chromedriver-binary version with new screenshots.
add 545db73 Merge pull request #13423 from KevinGG/BEAM-11045
add e24eb88 [BEAM-11342] fix google3 import error
add fb02df3 add logging
add 8cdb807 Merge pull request #13425 from ihji/BEAM-11342
add 3734560 [BEAM-11343] Make ExpansionServiceClient accesible outside of core-construction-java
add 372a4f3 Merge pull request #13427: [BEAM-11343] Make ExpansionServiceClient accesible outside of core-construction-java
add d1ff41d [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies and add tests for the previous versions
add 4ba51cd Merge pull request #13026: [BEAM-7003 BEAM-8639 BEAM-8774] Update Kafka dependencies, enable IT test in Postcommit
add 3d23f47 Warn if temp dataset cleanup permission is denied
add bee6602 Merge pull request #13433 from Warn if temp dataset cleanup permission is denied
add a1fac1d Merge pull request #13170 from [BEAM-9650] Adding support for ReadAll from BigQuery transform
add 978b812 Merge pull request #13137 from [BEAM-11073] Dicom IO Connector for Java
add 40f517f Fixes silent fail in bigtableio.py by logging ImportError (#13332)
add fb62dd8 spotlessApply
add d85b62f Merge pull request #13445: [BEAM-11343] Run spotlessApply
add 1e63259 Update quickstart-py.md
add 4382a40 Update website/www/site/content/en/get-started/quickstart-py.md
add 47a5808 Update the other version reference
add 5e9237c [BEAM-7372] Update python version support in Python quickstart (#13442)
add 14b4889 Revert "Add use_unified_worker to pipeline options."
add eb11a3f Merge pull request #13446 from Revert "Add use_unified_worker to pipeline options."
add c1dfa1c [BEAM-11173] Add Bigtable table with read operation
add 18f7a32 Merge pull request #13319 from piotr-szuberski/bigtable-provider
add 02da3ae Update python versions in pre-requisites (#13451)
add 87f3138 Merge pull request #12779 from [BEAM-10856] Support for NestedValueProvider for Python SDK
add 4c51569 Add splittable dofn as the recommended way of building connectors.
add 3de140f Merge pull request #13227 from [BEAM-10480] Add splittable dofn as the recommended way of building connectors.
add 4e7414c [BEAM-11354] Allow DoFn itself to be used as the restriction provider.
add 6555b39 [BEAM-11354] Also allow DoFn for WatermarkEstimator.
add 218a922 [BEAM-11354] Update docs.
add 4e16a85 Test for watermark tracker.
add 3231026 Add a note to the programming guide.
add 67e8649 Skip tests unimplemented for multiple workers.
add 83ed9ae Merge pull request #13431 [BEAM-11354] Allow DoFn itself to be used as the restriction provider.
add 6c25bc0 Add start_times to MonitoringInfos and populate them in the python SDK
add eca935b Merge pull request #13429 from Add start_times to MonitoringInfos and populate them in the python SDK
add 2f9b883 [BEAM-9980] use ALL_SUPPORTED_VERSIONS for constant SUPPORTED_CONTAINER_TASKS for python (#13426)
add 8e6dae8 [BEAM-11375] Downgrade Kafka clients dependency to fix Spark postcommit
add 2fe65b9 Merge pull request #13458: [BEAM-11375] Downgrade Kafka clients dependency to fix spark postcommit
add dfc3fea [BEAM-11365] Direct Runner: Handle pipeline option string quotes under windows os
add 1e008f7 Merge pull request #13449 from Sealina/master
add bd983d2 [BEAM-11331] Change region flag to awsRegion in io.aws2
add 0d0d2d5 Merge pull request #13411: [BEAM-11331] Change region flag to awsRegion in io.aws2
add 1bdc1cd [BEAM-11376] Ignoring DicomIOReadIT (#13450)
add af6ff6f [BEAM-10911] Grafana post-commit status board: job_name cells link to post-commit job in Jenkins
add 88601d3 Merge pull request #13463 from dlu1000/testbranch
add 00baa42 [BEAM-10986] Update shadow jar plugin. (#13453)
add d98a15a Remove inappropriate assumptions about repo from linkage check script
add d70dd01 Merge pull request #13342: [BEAM-11260][BEAM-11261] Remove inappropriate assumptions about repo from linkage check script
add df6184f make checkstyle suppression filepath absolute
add e90890b Merge pull request #13466: Make checkstyle suppression filepath absolute
add 42df977 creates :sdks:java:extensions:ml:integrationTest target, run in java PostCommit
add 0a4bdb1 Merge pull request #13444: [BEAM-11363] Create `:sdks:java:extensions:ml:integrationTest` target, run in Java PostCommit
add f551c81 Fix popen() typo
add 38982ae Add missing equal signs to example in comment
add 5e132c7 Merge pull request #13465 from hacosta/nits
add bcea54f [BEAM-11361] Dynamic splitting of SDF IOs. (#13443)
add 8ab1955 [BEAM-11355] Fix topological ordering of pipeline_from_stages after sort_stages (#13432)
add e4f9054 Use EventRecorder instead of relying on class var.
add 46ac3bc Merge pull request #13478 from [BEAM-11070]Use EventRecorder instead of relying on class var.
add e0adbfa [BEAM-11392] SideInputHandlerTest: inline trivial test setup for clarity (#13473)
add 901d5e7 Add the ToStringFn to the Python SDK
add 009b5e6 Merge pull request #13262 [BEAM-11151] Adds the ToStringFn Bundle Processor to Python
add da77c35 Merge pull request #13215 [BEAM-11151] Adds the ToStringFnRunner to Java
add 413fbc0 BEAM-11224 : Apache Hop listing on the 'Powered By Beam' page
add ecced96 Merge pull request #13383 from mattcasters/master
add fa0ddc6 [BEAM-11386] Fail BQ STREAMING_INSERTS with SchemaUpdateOptions
add dd0cbec Fix condition and test
add cb9212c Merge pull request #13468 from [BEAM-11386] Fail BQ STREAMING_INSERTS with SchemaUpdateOptions
add fba7f5a Update Go SDK wordcount-example.md - missing parens (#13394)
add 73752cf Store last error status per stream to improve debugging
add 07caf33 Merge pull request #13271 from scwhittle/dataflow_debug
add edc087e [BEAM-11312] Log cloud build url and enable kaniko cache in sdk_container_builder
add 44a2ac5 [BEAM-10475] Add typehints for ShardedKeyCoder (#13474)
add 60fe232 [BEAM-10891] Standardized developer build environment using Docker (#13308)
add 30f9a60 [BEAM-11383] Add runtime normalization for sdk.properties. (#13464)
add 780ce89 [BEAM-11373] Add write to Bigtable Table
add 9862f54 Merge pull request #13452 from piotr-szuberski/bigtable-table-write
add e9380df [BEAM-11075] CoGBK tests for Go SDK (#13438)
add c3367f7 Remove Experimental annotation from SDF related APIs.
add 2462fe9 Merge pull request #13199 from [BEAM-11132] Remove Experimental annotation from SDF related APIs.
add 553553d Avoiding queries that check whether a BQ table is empty for streaming inserts (#13490)
add dd6cbc8 [BEAM-11394] Make job server Java launcher configurable via pipeline options.
add 681ea6f Rename option to disambiguate job servers and expansion servers.
add f628ced Merge pull request #13487 from ibzib/BEAM-11394
add ff1b192 [BEAM-11412] Skip SparkRunnerTest::test_sdf_with_dofn_as_watermark_estimator.
add 305dcfd Fix requirements.
add 6a91643 Merge pull request #13499 from ibzib/BEAM-11412
add 31716b0 [BEAM-7372] Remove some py 2 and 3.5 references from the website (#13500)
new 8b22fb4 Merge remote-tracking branch 'origin/master' into website-revamp
The 1 revisions listed above as "new" are entirely new to this
repository and will be described in separate emails. The revisions
listed as "add" were already present in the repository and have only
been added to this reference.
Summary of changes:
.test-infra/jenkins/PythonTestProperties.groovy | 17 +-
.../jenkins/job_LoadTests_Combine_Flink_Go.groovy | 136 ++++++++++
.../jenkins/job_LoadTests_GBK_Flink_Go.groovy | 220 ++++++++++++++++
.../jenkins/job_LoadTests_coGBK_Flink_Go.groovy | 182 +++++++++++++
.../job_PostCommit_PortableJar_Flink.groovy | 3 +-
.../job_PostCommit_PortableJar_Spark.groovy | 2 +-
...Commit_Python_Chicago_Taxi_Example_Flink.groovy | 8 +-
...ommit_Python_ValidatesContainer_Dataflow.groovy | 9 +-
.../dashboards/Post-Commits_status_dashboard.json | 2 +-
CHANGES.md | 4 +
build.gradle | 13 +-
buildSrc/build.gradle | 4 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 23 +-
dev-support/docker/Dockerfile | 117 +++++++++
dev-support/docker/bashcolors.sh | 91 +++++++
dev-support/docker/beam_env_checks.sh | 121 +++++++++
model/pipeline/src/main/proto/metrics.proto | 13 +
.../DefaultExpansionServiceClientFactory.java | 7 +-
.../core/construction/ExpansionServiceClient.java | 2 +-
.../ExpansionServiceClientFactory.java | 2 +-
.../beam/runners/core/construction/External.java | 2 +-
.../core/construction/PTransformTranslation.java | 1 +
.../beam/runners/core/SideInputHandlerTest.java | 118 +++++----
runners/direct-java/build.gradle | 22 +-
runners/flink/job-server/flink_job_server.gradle | 66 -----
.../worker/windmill/GrpcWindmillServer.java | 14 +-
.../test_flink_uber_jar.sh | 0
runners/portability/test_pipeline_jar.sh | 3 +-
runners/spark/job-server/build.gradle | 32 ---
sdks/go/test/load/build.gradle | 3 +
sdks/go/test/load/cogbk/cogbk.go | 106 ++++++++
sdks/go/test/load/combine/combine.go | 87 +++++++
sdks/go/test/load/group_by_key/group_by_key.go | 89 +++++++
sdks/java/build-tools/beam-linkage-check.sh | 80 +++---
.../apache/beam/sdk/annotations/Experimental.java | 7 -
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 7 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 5 +-
.../main/java/org/apache/beam/sdk/io/TextIO.java | 5 +-
.../java/org/apache/beam/sdk/state/Timers.java | 2 +-
.../apache/beam/sdk/transforms/Deduplicate.java | 3 -
.../java/org/apache/beam/sdk/transforms/DoFn.java | 13 -
.../beam/sdk/transforms/PeriodicSequence.java | 3 -
.../java/org/apache/beam/sdk/transforms/Watch.java | 3 -
.../splittabledofn/ByteKeyRangeTracker.java | 3 -
.../splittabledofn/GrowableOffsetRangeTracker.java | 3 -
.../splittabledofn/HasDefaultTracker.java | 4 -
.../HasDefaultWatermarkEstimator.java | 3 -
.../splittabledofn/ManualWatermarkEstimator.java | 3 -
.../splittabledofn/OffsetRangeTracker.java | 3 -
.../splittabledofn/RestrictionTracker.java | 3 -
.../sdk/transforms/splittabledofn/SplitResult.java | 3 -
.../TimestampObservingWatermarkEstimator.java | 3 -
.../splittabledofn/WatermarkEstimator.java | 3 -
.../splittabledofn/WatermarkEstimators.java | 3 -
.../transforms/splittabledofn/package-info.java | 3 -
sdks/java/extensions/ml/build.gradle | 33 ++-
sdks/java/extensions/sql/build.gradle | 1 +
.../sql/meta/provider/bigtable/BigtableTable.java | 211 +++++++++++++++
.../provider/bigtable/BigtableTableProvider.java | 85 ++++++
.../sql/meta/provider/bigtable/package-info.java | 9 +-
.../BigtableTableCreationFailuresTest.java | 145 +++++++++++
.../provider/bigtable/BigtableTableFlatTest.java | 163 ++++++++++++
.../meta/provider/bigtable/BigtableTableTest.java | 94 +++++++
.../bigtable/BigtableTableWithRowsTest.java | 158 +++++++++++
.../apache/beam/fn/harness/ToStringFnRunner.java | 66 +++++
.../beam/fn/harness/ToStringFnRunnerTest.java | 72 +++++
sdks/java/io/amazon-web-services2/build.gradle | 2 +-
.../beam/sdk/io/aws2/options/AwsOptions.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 5 +
.../io/gcp/bigtable/BeamRowToBigtableMutation.java | 123 +++++++++
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 15 ++
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 22 ++
.../sdk/io/gcp/bigtable/BigtableRowToBeamRow.java | 176 +++++++++++++
.../io/gcp/bigtable/BigtableRowToBeamRowFlat.java | 108 ++++++++
.../io/gcp/bigtable/BigtableRowToBeamRowFn.java | 46 ++++
.../beam/sdk/io/gcp/bigtable/CellValueParser.java | 106 ++++++++
.../apache/beam/sdk/io/gcp/bigtable/RowUtils.java | 13 +-
.../apache/beam/sdk/io/gcp/healthcare/DicomIO.java | 199 ++++++++++++++
.../sdk/io/gcp/healthcare/HealthcareApiClient.java | 12 +
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 79 ++++++
.../beam/sdk/io/gcp/healthcare/WebPathParser.java | 63 +++++
.../io/gcp/testing/BigtableEmulatorWrapper.java | 75 ++++++
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 154 +++++++++++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 9 +
.../bigtable/BeamRowToBigtableMutationTest.java | 89 +++++++
.../gcp/bigtable/BigtableRowToBeamRowFlatTest.java | 65 +++++
.../io/gcp/bigtable/BigtableRowToBeamRowTest.java | 93 +++++++
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 277 ++++++++++++++++++++
.../apache/beam/sdk/io/gcp/bigtable/TestUtils.java | 109 ++++++++
.../beam/sdk/io/gcp/healthcare/DicomIOReadIT.java | 93 +++++++
.../beam/sdk/io/gcp/healthcare/DicomIOTest.java | 60 +++++
.../sdk/io/gcp/healthcare/WebPathParserTest.java | 48 ++++
.../src/test/resources/DICOM/testDicomFile.dcm | Bin 0 -> 10366 bytes
.../java/org/apache/beam/sdk/io/hbase/HBaseIO.java | 1 -
sdks/java/io/kafka/build.gradle | 90 +++++--
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 69 ++++-
sdks/python/apache_beam/coders/coders.py | 16 ++
.../apache_beam/coders/coders_test_common.py | 17 ++
sdks/python/apache_beam/dataframe/io.py | 143 +++++++++-
sdks/python/apache_beam/dataframe/io_test.py | 41 +++
.../apache_beam/examples/kafkataxi/README.md | 4 +-
.../apache_beam/examples/snippets/snippets.py | 4 +
sdks/python/apache_beam/io/gcp/bigquery.py | 273 +++++++++++++------
.../apache_beam/io/gcp/bigquery_read_internal.py | 289 +++++++++++++++++++++
.../apache_beam/io/gcp/bigquery_read_it_test.py | 103 ++++++++
sdks/python/apache_beam/io/gcp/bigquery_test.py | 2 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 26 +-
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 114 ++++++++
sdks/python/apache_beam/io/gcp/pubsub_test.py | 123 +++++++++
sdks/python/apache_beam/io/iobase.py | 20 +-
sdks/python/apache_beam/io/restriction_trackers.py | 17 ++
sdks/python/apache_beam/metrics/cells.pxd | 2 +
sdks/python/apache_beam/metrics/cells.py | 15 +-
sdks/python/apache_beam/metrics/cells_test.py | 25 ++
.../python/apache_beam/options/pipeline_options.py | 6 +
sdks/python/apache_beam/options/value_provider.py | 71 ++++-
.../apache_beam/options/value_provider_test.py | 40 +++
sdks/python/apache_beam/runners/common.py | 5 +-
.../Linux/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 700548 -> 703677 bytes
.../runners/portability/flink_runner_test.py | 2 +-
.../portability/fn_api_runner/fn_runner_test.py | 115 +++++++-
.../portability/fn_api_runner/translations.py | 8 +-
.../portability/fn_api_runner/translations_test.py | 31 +++
.../portability/fn_api_runner/worker_handlers.py | 2 +-
.../apache_beam/runners/portability/job_server.py | 3 +-
.../runners/portability/sdk_container_builder.py | 30 ++-
.../runners/portability/spark_runner_test.py | 6 +-
.../apache_beam/runners/worker/bundle_processor.py | 17 ++
.../apache_beam/runners/worker/sdk_worker_test.py | 2 +
sdks/python/apache_beam/transforms/core.py | 32 ++-
.../apache_beam/typehints/sharded_key_type.py | 75 ++++++
.../apache_beam/typehints/sharded_key_type_test.py | 80 ++++++
sdks/python/apache_beam/utils/shared.py | 2 +-
sdks/python/build-requirements.txt | 3 +
sdks/python/setup.py | 2 +-
sdks/python/test-suites/portable/common.gradle | 69 ++++-
start-build-env.sh | 132 ++++++++++
.../www/site/content/en/community/powered-by.md | 2 +
website/www/site/content/en/contribute/_index.md | 16 +-
.../dsls/sql/extensions/create-external-table.md | 132 ++++++++++
.../en/documentation/io/developing-io-java.md | 3 +
.../en/documentation/io/developing-io-overview.md | 80 +++---
.../en/documentation/io/developing-io-python.md | 3 +
.../content/en/documentation/programming-guide.md | 2 +-
.../en/documentation/sdks/python-dependencies.md | 12 +-
.../site/content/en/get-started/quickstart-py.md | 4 +-
.../content/en/get-started/wordcount-example.md | 2 +-
148 files changed, 6607 insertions(+), 549 deletions(-)
create mode 100644 .test-infra/jenkins/job_LoadTests_Combine_Flink_Go.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_GBK_Flink_Go.groovy
create mode 100644 .test-infra/jenkins/job_LoadTests_coGBK_Flink_Go.groovy
create mode 100644 dev-support/docker/Dockerfile
create mode 100755 dev-support/docker/bashcolors.sh
create mode 100755 dev-support/docker/beam_env_checks.sh
rename runners/{flink/job-server => portability}/test_flink_uber_jar.sh (100%)
create mode 100644 sdks/go/test/load/cogbk/cogbk.go
create mode 100644 sdks/go/test/load/combine/combine.go
create mode 100644 sdks/go/test/load/group_by_key/group_by_key.go
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTable.java
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableProvider.java
copy runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExpansionServiceClient.java => sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/package-info.java (71%)
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableCreationFailuresTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableFlatTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableTest.java
create mode 100644 sdks/java/extensions/sql/src/test/java/org/apache/beam/sdk/extensions/sql/meta/provider/bigtable/BigtableTableWithRowsTest.java
create mode 100644 sdks/java/harness/src/main/java/org/apache/beam/fn/harness/ToStringFnRunner.java
create mode 100644 sdks/java/harness/src/test/java/org/apache/beam/fn/harness/ToStringFnRunnerTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BeamRowToBigtableMutation.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableRowToBeamRow.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableRowToBeamRowFlat.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableRowToBeamRowFn.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/CellValueParser.java
copy runners/core-construction-java/src/main/java/org/apache/beam/runners/core/construction/ExpansionServiceClient.java => sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/bigtable/RowUtils.java (70%)
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/DicomIO.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/healthcare/WebPathParser.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableEmulatorWrapper.java
create mode 100644 sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/testing/BigtableTestUtils.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BeamRowToBigtableMutationTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableRowToBeamRowFlatTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/BigtableRowToBeamRowTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/CellValueParserTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/bigtable/TestUtils.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/DicomIOReadIT.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/DicomIOTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/java/org/apache/beam/sdk/io/gcp/healthcare/WebPathParserTest.java
create mode 100644 sdks/java/io/google-cloud-platform/src/test/resources/DICOM/testDicomFile.dcm
create mode 100644 sdks/python/apache_beam/typehints/sharded_key_type.py
create mode 100644 sdks/python/apache_beam/typehints/sharded_key_type_test.py
create mode 100755 start-build-env.sh
[beam] 01/01: Merge remote-tracking branch 'origin/master' into
website-revamp
Posted by bh...@apache.org.
This is an automated email from the ASF dual-hosted git repository.
bhulette pushed a commit to branch website-revamp
in repository https://gitbox.apache.org/repos/asf/beam.git
commit 8b22fb4b9aa8607313f1e39374a226bc226a349c
Merge: 140a913 31716b0
Author: Brian Hulette <bh...@google.com>
AuthorDate: Mon Dec 7 18:09:14 2020 -0800
Merge remote-tracking branch 'origin/master' into website-revamp
.test-infra/jenkins/PythonTestProperties.groovy | 17 +-
.../jenkins/job_LoadTests_Combine_Flink_Go.groovy | 136 ++++++++++
.../jenkins/job_LoadTests_GBK_Flink_Go.groovy | 220 ++++++++++++++++
.../jenkins/job_LoadTests_coGBK_Flink_Go.groovy | 182 +++++++++++++
.../job_PostCommit_PortableJar_Flink.groovy | 3 +-
.../job_PostCommit_PortableJar_Spark.groovy | 2 +-
...Commit_Python_Chicago_Taxi_Example_Flink.groovy | 8 +-
...ommit_Python_ValidatesContainer_Dataflow.groovy | 9 +-
.../dashboards/Post-Commits_status_dashboard.json | 2 +-
CHANGES.md | 4 +
build.gradle | 13 +-
buildSrc/build.gradle | 4 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 23 +-
dev-support/docker/Dockerfile | 117 +++++++++
dev-support/docker/bashcolors.sh | 91 +++++++
dev-support/docker/beam_env_checks.sh | 121 +++++++++
model/pipeline/src/main/proto/metrics.proto | 13 +
.../DefaultExpansionServiceClientFactory.java | 7 +-
.../core/construction/ExpansionServiceClient.java | 2 +-
.../ExpansionServiceClientFactory.java | 2 +-
.../beam/runners/core/construction/External.java | 2 +-
.../core/construction/PTransformTranslation.java | 1 +
.../beam/runners/core/SideInputHandlerTest.java | 118 +++++----
runners/direct-java/build.gradle | 22 +-
runners/flink/job-server/flink_job_server.gradle | 66 -----
.../worker/windmill/GrpcWindmillServer.java | 14 +-
.../test_flink_uber_jar.sh | 0
runners/portability/test_pipeline_jar.sh | 3 +-
runners/spark/job-server/build.gradle | 32 ---
sdks/go/test/load/build.gradle | 3 +
sdks/go/test/load/cogbk/cogbk.go | 106 ++++++++
sdks/go/test/load/combine/combine.go | 87 +++++++
sdks/go/test/load/group_by_key/group_by_key.go | 89 +++++++
sdks/java/build-tools/beam-linkage-check.sh | 80 +++---
.../apache/beam/sdk/annotations/Experimental.java | 7 -
.../main/java/org/apache/beam/sdk/io/AvroIO.java | 7 +-
.../main/java/org/apache/beam/sdk/io/FileIO.java | 5 +-
.../main/java/org/apache/beam/sdk/io/TextIO.java | 5 +-
.../java/org/apache/beam/sdk/state/Timers.java | 2 +-
.../apache/beam/sdk/transforms/Deduplicate.java | 3 -
.../java/org/apache/beam/sdk/transforms/DoFn.java | 13 -
.../beam/sdk/transforms/PeriodicSequence.java | 3 -
.../java/org/apache/beam/sdk/transforms/Watch.java | 3 -
.../splittabledofn/ByteKeyRangeTracker.java | 3 -
.../splittabledofn/GrowableOffsetRangeTracker.java | 3 -
.../splittabledofn/HasDefaultTracker.java | 4 -
.../HasDefaultWatermarkEstimator.java | 3 -
.../splittabledofn/ManualWatermarkEstimator.java | 3 -
.../splittabledofn/OffsetRangeTracker.java | 3 -
.../splittabledofn/RestrictionTracker.java | 3 -
.../sdk/transforms/splittabledofn/SplitResult.java | 3 -
.../TimestampObservingWatermarkEstimator.java | 3 -
.../splittabledofn/WatermarkEstimator.java | 3 -
.../splittabledofn/WatermarkEstimators.java | 3 -
.../transforms/splittabledofn/package-info.java | 3 -
sdks/java/extensions/ml/build.gradle | 33 ++-
sdks/java/extensions/sql/build.gradle | 1 +
.../sql/meta/provider/bigtable/BigtableTable.java | 211 +++++++++++++++
.../provider/bigtable/BigtableTableProvider.java | 85 ++++++
.../sql/meta/provider/bigtable/package-info.java | 9 +-
.../BigtableTableCreationFailuresTest.java | 145 +++++++++++
.../provider/bigtable/BigtableTableFlatTest.java | 163 ++++++++++++
.../meta/provider/bigtable/BigtableTableTest.java | 94 +++++++
.../bigtable/BigtableTableWithRowsTest.java | 158 +++++++++++
.../apache/beam/fn/harness/ToStringFnRunner.java | 66 +++++
.../beam/fn/harness/ToStringFnRunnerTest.java | 72 +++++
sdks/java/io/amazon-web-services2/build.gradle | 2 +-
.../beam/sdk/io/aws2/options/AwsOptions.java | 4 +-
.../beam/sdk/io/gcp/bigquery/BigQueryIO.java | 5 +
.../io/gcp/bigtable/BeamRowToBigtableMutation.java | 123 +++++++++
.../beam/sdk/io/gcp/bigtable/BigtableConfig.java | 15 ++
.../beam/sdk/io/gcp/bigtable/BigtableIO.java | 22 ++
.../sdk/io/gcp/bigtable/BigtableRowToBeamRow.java | 176 +++++++++++++
.../io/gcp/bigtable/BigtableRowToBeamRowFlat.java | 108 ++++++++
.../io/gcp/bigtable/BigtableRowToBeamRowFn.java | 46 ++++
.../beam/sdk/io/gcp/bigtable/CellValueParser.java | 106 ++++++++
.../apache/beam/sdk/io/gcp/bigtable/RowUtils.java | 13 +-
.../apache/beam/sdk/io/gcp/healthcare/DicomIO.java | 199 ++++++++++++++
.../sdk/io/gcp/healthcare/HealthcareApiClient.java | 12 +
.../io/gcp/healthcare/HttpHealthcareApiClient.java | 79 ++++++
.../beam/sdk/io/gcp/healthcare/WebPathParser.java | 63 +++++
.../io/gcp/testing/BigtableEmulatorWrapper.java | 75 ++++++
.../beam/sdk/io/gcp/testing/BigtableTestUtils.java | 154 +++++++++++
.../sdk/io/gcp/bigquery/BigQueryIOWriteTest.java | 9 +
.../bigtable/BeamRowToBigtableMutationTest.java | 89 +++++++
.../gcp/bigtable/BigtableRowToBeamRowFlatTest.java | 65 +++++
.../io/gcp/bigtable/BigtableRowToBeamRowTest.java | 93 +++++++
.../sdk/io/gcp/bigtable/CellValueParserTest.java | 277 ++++++++++++++++++++
.../apache/beam/sdk/io/gcp/bigtable/TestUtils.java | 109 ++++++++
.../beam/sdk/io/gcp/healthcare/DicomIOReadIT.java | 93 +++++++
.../beam/sdk/io/gcp/healthcare/DicomIOTest.java | 60 +++++
.../sdk/io/gcp/healthcare/WebPathParserTest.java | 48 ++++
.../src/test/resources/DICOM/testDicomFile.dcm | Bin 0 -> 10366 bytes
.../java/org/apache/beam/sdk/io/hbase/HBaseIO.java | 1 -
sdks/java/io/kafka/build.gradle | 90 +++++--
.../org/apache/beam/sdk/io/kafka/KafkaIOIT.java | 69 ++++-
sdks/python/apache_beam/coders/coders.py | 16 ++
.../apache_beam/coders/coders_test_common.py | 17 ++
sdks/python/apache_beam/dataframe/io.py | 143 +++++++++-
sdks/python/apache_beam/dataframe/io_test.py | 41 +++
.../apache_beam/examples/kafkataxi/README.md | 4 +-
.../apache_beam/examples/snippets/snippets.py | 4 +
sdks/python/apache_beam/io/gcp/bigquery.py | 273 +++++++++++++------
.../apache_beam/io/gcp/bigquery_read_internal.py | 289 +++++++++++++++++++++
.../apache_beam/io/gcp/bigquery_read_it_test.py | 103 ++++++++
sdks/python/apache_beam/io/gcp/bigquery_test.py | 2 +-
sdks/python/apache_beam/io/gcp/bigquery_tools.py | 26 +-
sdks/python/apache_beam/io/gcp/bigtableio.py | 7 +-
sdks/python/apache_beam/io/gcp/pubsub.py | 114 ++++++++
sdks/python/apache_beam/io/gcp/pubsub_test.py | 123 +++++++++
sdks/python/apache_beam/io/iobase.py | 20 +-
sdks/python/apache_beam/io/restriction_trackers.py | 17 ++
sdks/python/apache_beam/metrics/cells.pxd | 2 +
sdks/python/apache_beam/metrics/cells.py | 15 +-
sdks/python/apache_beam/metrics/cells_test.py | 25 ++
.../python/apache_beam/options/pipeline_options.py | 6 +
sdks/python/apache_beam/options/value_provider.py | 71 ++++-
.../apache_beam/options/value_provider_test.py | 40 +++
sdks/python/apache_beam/runners/common.py | 5 +-
.../Linux/7a35f487b2a5f3a9b9852a8659eeb4bd.png | Bin 700548 -> 703677 bytes
.../runners/portability/flink_runner_test.py | 2 +-
.../portability/fn_api_runner/fn_runner_test.py | 115 +++++++-
.../portability/fn_api_runner/translations.py | 8 +-
.../portability/fn_api_runner/translations_test.py | 31 +++
.../portability/fn_api_runner/worker_handlers.py | 2 +-
.../apache_beam/runners/portability/job_server.py | 3 +-
.../runners/portability/sdk_container_builder.py | 30 ++-
.../runners/portability/spark_runner_test.py | 6 +-
.../apache_beam/runners/worker/bundle_processor.py | 17 ++
.../apache_beam/runners/worker/sdk_worker_test.py | 2 +
sdks/python/apache_beam/transforms/core.py | 32 ++-
.../apache_beam/typehints/sharded_key_type.py | 75 ++++++
.../apache_beam/typehints/sharded_key_type_test.py | 80 ++++++
sdks/python/apache_beam/utils/shared.py | 2 +-
sdks/python/build-requirements.txt | 3 +
sdks/python/setup.py | 2 +-
sdks/python/test-suites/portable/common.gradle | 69 ++++-
start-build-env.sh | 132 ++++++++++
.../www/site/content/en/community/powered-by.md | 2 +
website/www/site/content/en/contribute/_index.md | 16 +-
.../dsls/sql/extensions/create-external-table.md | 132 ++++++++++
.../en/documentation/io/developing-io-java.md | 3 +
.../en/documentation/io/developing-io-overview.md | 80 +++---
.../en/documentation/io/developing-io-python.md | 3 +
.../content/en/documentation/programming-guide.md | 2 +-
.../en/documentation/sdks/python-dependencies.md | 12 +-
.../site/content/en/get-started/quickstart-py.md | 4 +-
.../content/en/get-started/wordcount-example.md | 2 +-
148 files changed, 6607 insertions(+), 549 deletions(-)