You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by mw...@apache.org on 2020/05/28 12:45:56 UTC
[beam] branch BEAM-9723-java-dlp updated (a4411f1 -> 1f0e1ba)
This is an automated email from the ASF dual-hosted git repository.
mwalenia pushed a change to branch BEAM-9723-java-dlp
in repository https://gitbox.apache.org/repos/asf/beam.git.
from a4411f1 Add info to CHANGES.md
add f387df7 add io-matrix
add 8fe35ad add descriptions, links to guides
add ff1f9f1 fix typo
add fa805b6 format
add 37ac9ee update CHANGES.md
add 6766d0c [BEAM-10058] Provide less strict assertion to make the test more resistant against future changes in a model
add 57dc7f4 Merge pull request #11795: [BEAM-10058] Provide less strict assertion to make the test more resistant against future changes in a model
add 0f97499 [BEAM-9930] Update blog post for new Beam Summit Digital dates (#11772)
add 79dcab9 [BEAM-10073] Add pubsub performance tests results to the grafana dashboards (#11809)
add 9f8783d add def args
add 18de412 Merge pull request #11786 from TheNeuralBit/mobile-gaming-fix
add f83bc75 [BEAM-10065] Fix beam release guide template (#11797)
add f452e09 Fix typo in doc "and and" -> "and" (#11810)
add d86219f [BEAM-2939, BEAM-10057] Ensure that we can process an EmptyUnboundedSource and also prevent splitting on it. (#11781)
add 8f53b99 Fix typo Portabillity -> Portability
add 17ce646 Merge pull request #11789 from epicfaace/patch-2
add c2d6c36 [BEAM-9603] Add timer family support to FnApiDoFnRunner
add 5839915 Address comments and add tests
add d3dfa30 comments
add b1791a2 Merge pull request #11756 from y1chi/BEAM-9603
add 90a79d6 [BEAM-7370] Upgrade sphinx to 3.0.3 (#11798)
add 790ed44 Fix typo copyLicenseScrips -> copyLicenseScripts
add 6e42344 Merge pull request #11800: Fix typo copyLicenseScrips -> copyLicenseScripts
add fdd9409 Merge branch 'master' into bi
add e1531b7 Merge pull request #11802 from [BEAM-9916] Update I/O documentation links and create more complete I/O matrix
add c77bee6 Clear redundant debug statements in pr-11756
add 32c6579 Merge pull request #11822 from y1chi/clean-up
add 6022079 [BEAM-10076] Fix dataflow worker status page rendering (#11812)
add 27f0dba [BEAM-9770] BigQueryIO Patterns. Add Document update to snippit update.
add 1a5dd77 Merge pull request #11815 from rezarokni/BEAM-9770
add 630a488 [BEAM-10016, BEAM-10094] Disable single test to get portable validates runner signal back to green
add e6bbb40 Merge pull request #11825: [BEAM-10016, BEAM-10094] Disable single test to get portable validates runner signal back to green
add 63b4d3a [BEAM-7770] Make ReplicaInfo part of Read on SolrIO
add 61eebf0 [BEAM-7770] Add ReadAll transform for SolrIO
add 5d88f77 Merge pull request #11357: [BEAM-7770] Add ReadAll transform for SolrIO
add 9199598 [BEAM-10102] Fix query in pubsub read grafana dashoard
add d0d813b Merge pull request #11827: [BEAM-10102] Fix query in pubsub read grafana dashoard
add 1d84d00 [BEAM-10077] using filename + hash instead of UUID for staging name
add 8aedab2 Merge pull request #11813: [BEAM-10077] using filename + hash instead of UUID for staging name
add 5b6e84f Add a flag to tune the size of the grouping tables used by the dataflow workers (#11811)
add 292dea3 [BEAM-9936] Reduce the code duplication across python/container/py*
add 09a1677 [BEAM-9936] Create SDK harness containers with Python3.8
add a23b0e6 [BEAM-9936] Create SDK harness containers with Python 3.8 - PR #11738
add 1ec86e7 Add Pub/Sub Lite to in progress IOs
add 6906192 Merge pull request #11833 from apache/aaltay-patch-1
add ec7278b [BEAM-10108] Update Flink versions in publish_docker_images.sh.
add c27e033 Merge pull request #11829 from ibzib/BEAM-10108
add fa028e6 [BEAM-10074] | implement hashing functions
add 706a06c Merge pull request #11817 from darshanj/BEAM-10074
add 64f8a40 [BEAM-10064] Fix google3 import error for BEAM-9383
add 1671c1f Merge pull request #11793 [BEAM-10064] Fix google3 import error for BEAM-9383
add eb59a84 Merge pull request #11796 from [BEAM-10003] Use local code for building code samples on website
add 6cf105a Add to/from_runner_api_parameters to WriteToBigQuery
add 6658d62 Merge pull request #11745 from [BEAM-9692] Add to/from_runner_api_parameters to WriteToBigQuery
add 4817e84 [BEAM-9971] Do not use context classloader. (#11784)
add ad15754 Add blog post announcing the Beam 2.21.0 release.
add ff8430e Add release date.
add c5b1183 Add release date.
add e0f858f Remove HTML comment (causes formatting issues)
add a0b5fb1 Merge pull request #11729 from ibzib/2-21-blog
add 301c32b Update Beam website to release 2.21.0.
add 9939a8f Move 2.20 download addresses to archive.
add 5129587 Add release date.
add e52f58d Merge branch 'website-2-21' of https://github.com/ibzib/beam into website-2-21
add 3850133 Merge pull request #11727 from ibzib/website-2-21
add 6654a3d [BEAM-9810] Tox suites for Python 3.8
add c0c786d [BEAM-9810] Fix 'dictionary keys changed during iteration' error
add c2b9867 [BEAM-9810] Disable all failing tests until Dataflow runner supports Python 3.8
add c4f03a8 [BEAM-9810] Add a Tox (precommit) suite for Python 3.8 #11707
add 0588260 [BEAM-9363] Support TUMBLE aggregation (#11807)
add 00736f3 [BEAM-10098] Enabling javadoc export for RabbitMqIO and KuduIO
add a13ef24 Merge pull request #11837: [BEAM-10098] Enabling javadoc export for RabbitMqIO and KuduIO
add 1f0e1ba Merge branch 'master' into BEAM-9723-java-dlp
No new revisions were added by this update.
Summary of changes:
.../job_PerformanceTests_PubsubIO_Python.groovy | 5 +-
...ommit_Python_ValidatesContainer_Dataflow.groovy | 2 +
.test-infra/metrics/beamgrafana-deploy.yaml | 2 +-
.../Python_IO_IT_Tests_Dataflow.json | 256 +++++++++++++-
CHANGES.md | 5 +-
build.gradle | 4 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 7 +
.../src/main/groovy/MobileGamingCommands.groovy | 4 +-
release/src/main/scripts/publish_docker_images.sh | 2 +-
.../runners/core/construction/Environments.java | 23 +-
.../core/construction/NativeTransforms.java | 5 +-
runners/flink/job-server/flink_job_server.gradle | 4 +
.../beam/runners/dataflow/DataflowRunner.java | 12 +-
.../beam/runners/dataflow/util/PackageUtil.java | 25 +-
.../beam/runners/dataflow/util/GCSUploadMain.java | 7 +-
.../runners/dataflow/util/PackageUtilTest.java | 24 +-
.../dataflow/worker/PartialGroupByKeyParDoFns.java | 10 +-
.../dataflow/worker/StreamingDataflowWorker.java | 4 +-
.../worker/util/common/worker/GroupingTables.java | 10 +-
runners/spark/job-server/build.gradle | 4 +
.../src/main/java/org/apache/beam/sdk/io/Read.java | 12 +-
.../apache/beam/sdk/options/SdkHarnessOptions.java | 14 +
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 8 +-
.../sql/impl/rel/BeamTableFunctionScanRel.java | 37 +-
.../sql/impl/udf/BuiltinHashFunctions.java | 139 ++++++++
.../beam/sdk/extensions/sql/BeamSqlDslBase.java | 6 +
.../udf/BeamSalUhfSpecialTypeAndValueTest.java | 69 ++++
.../sql/impl/udf/BeamSqlUdfExpressionTest.java | 41 +++
.../sql/zetasql/ZetaSQLDialectSpecTest.java | 25 ++
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 131 +++++--
.../beam/fn/harness/PrecombineGroupingTable.java | 13 +-
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 264 +++++++++++----
sdks/java/io/kudu/build.gradle | 2 +-
sdks/java/io/rabbitmq/build.gradle | 2 +-
.../java/org/apache/beam/sdk/io/solr/SolrIO.java | 68 ++--
.../org/apache/beam/sdk/io/solr/SolrIOTest.java | 17 +
sdks/python/apache_beam/io/gcp/bigquery.py | 72 ++++
sdks/python/apache_beam/io/gcp/bigquery_test.py | 67 ++++
.../python/apache_beam/io/hadoopfilesystem_test.py | 15 +-
sdks/python/apache_beam/io/textio.py | 2 +-
.../ml/gcp/videointelligenceml_test_it.py | 6 +-
.../runners/dataflow/dataflow_runner_test.py | 57 ++++
.../runners/dataflow/internal/apiclient_test.py | 69 +++-
.../runners/dataflow/template_runner_test.py | 3 +
sdks/python/container/Dockerfile | 2 +-
sdks/python/container/base_image_requirements.txt | 12 +-
sdks/python/container/build.gradle | 1 +
.../container/{py37/build.gradle => common.gradle} | 16 +-
.../container/license_scripts/dep_urls_py.yaml | 4 +-
sdks/python/container/py2/build.gradle | 53 +--
sdks/python/container/py35/build.gradle | 53 +--
sdks/python/container/py36/build.gradle | 53 +--
sdks/python/container/py37/build.gradle | 53 +--
.../python/container/py38}/build.gradle | 11 +-
sdks/python/container/run_validatescontainer.sh | 20 +-
sdks/python/test-suites/tox/py37/build.gradle | 3 -
.../test-suites/tox/{py35 => py38}/build.gradle | 9 +-
sdks/python/test-suites/tox/pycommon/build.gradle | 2 +-
sdks/python/tox.ini | 29 +-
settings.gradle | 2 +
website/.gitignore | 2 +-
website/CONTRIBUTE.md | 10 +-
website/build.gradle | 10 +-
...ild_github_samples.sh => build_code_samples.sh} | 12 +-
website/www/package.json | 2 +-
website/www/site/config.toml | 2 +-
website/www/site/content/en/blog/beam-2.21.0.md | 97 ++++++
.../content/en/blog/beam-summit-digital-2020.md | 10 +-
.../site/content/en/contribute/release-guide.md | 60 ++--
.../site/content/en/documentation/io/built-in.md | 86 +----
.../documentation/io/built-in/google-bigquery.md | 50 +--
.../en/documentation/io/developing-io-python.md | 12 +-
.../en/documentation/patterns/bigqueryio.md | 46 +++
.../en/documentation/patterns/custom-windows.md | 12 +-
.../en/documentation/patterns/file-processing.md | 10 +-
.../content/en/documentation/patterns/overview.md | 3 +
.../en/documentation/patterns/pipeline-options.md | 6 +-
.../en/documentation/patterns/side-inputs.md | 6 +-
.../content/en/documentation/programming-guide.md | 94 ++---
.../en/documentation/sdks/python-type-safety.md | 22 +-
.../transforms/python/elementwise/filter.md | 24 +-
.../transforms/python/elementwise/flatmap.md | 36 +-
.../transforms/python/elementwise/keys.md | 4 +-
.../transforms/python/elementwise/kvswap.md | 4 +-
.../transforms/python/elementwise/map.md | 32 +-
.../transforms/python/elementwise/pardo.md | 12 +-
.../transforms/python/elementwise/partition.md | 12 +-
.../transforms/python/elementwise/regex.md | 36 +-
.../transforms/python/elementwise/tostring.md | 12 +-
.../transforms/python/elementwise/values.md | 4 +-
.../python/elementwise/withtimestamps.md | 16 +-
.../www/site/content/en/get-started/downloads.md | 14 +-
.../en/get-started/mobile-gaming-example.md | 44 +--
.../content/en/get-started/wordcount-example.md | 24 +-
website/www/site/content/en/roadmap/go-sdk.md | 2 +-
website/www/site/data/io_matrix.yaml | 377 +++++++++++++++++++++
.../partials/section-menu/en/documentation.html | 13 +
.../www/site/layouts/shortcodes/code_sample.html | 23 ++
.../www/site/layouts/shortcodes/github_sample.html | 15 -
website/www/site/layouts/shortcodes/io-matrix.html | 55 +++
website/www/site/static/.htaccess | 2 +-
.../blog/beamsummit/beamsummit-digital-2020.png | Bin 0 -> 72388 bytes
.../images/blog/beamsummit/beamsummit-digital.png | Bin 63238 -> 0 bytes
103 files changed, 2350 insertions(+), 840 deletions(-)
create mode 100644 sdks/java/extensions/sql/src/main/java/org/apache/beam/sdk/extensions/sql/impl/udf/BuiltinHashFunctions.java
copy sdks/python/container/{py37/build.gradle => common.gradle} (89%)
copy {.test-infra/tools => sdks/python/container/py38}/build.gradle (84%)
copy sdks/python/test-suites/tox/{py35 => py38}/build.gradle (85%)
rename website/www/{build_github_samples.sh => build_code_samples.sh} (75%)
create mode 100644 website/www/site/content/en/blog/beam-2.21.0.md
create mode 100644 website/www/site/content/en/documentation/patterns/bigqueryio.md
create mode 100644 website/www/site/data/io_matrix.yaml
create mode 100644 website/www/site/layouts/shortcodes/code_sample.html
delete mode 100644 website/www/site/layouts/shortcodes/github_sample.html
create mode 100644 website/www/site/layouts/shortcodes/io-matrix.html
create mode 100644 website/www/site/static/images/blog/beamsummit/beamsummit-digital-2020.png
delete mode 100644 website/www/site/static/images/blog/beamsummit/beamsummit-digital.png