You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2022/02/10 03:33:09 UTC
[beam] branch nightly-refs/heads/master updated (8ed4763 -> 5a9a1bc)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to branch nightly-refs/heads/master
in repository https://gitbox.apache.org/repos/asf/beam.git.
from 8ed4763 Merge pull request #16779 from kileys/changes
add f36b6e1 Update CHANGES.md with Go SDK milestones. (#16787)
add 5b9f6be [BEAM-13732] Switch x-lang BigQueryIO expansion service to GCP one.
add 06de99b Merge pull request #16784: [BEAM-13732] Switch x-lang BigQueryIO expansion service.
add 3c6fb0d [BEAM-13193] Allow BeamFnDataOutboundObserver to flush elements. (#16778)
add 256e338 [adhoc] Remove remaining usage of Powermock from aws2.
add 8bf68ad Merge pull request #16777: [adhoc] Remove remaining usage of Powermock from aws2.
add 5ba5537 [BEAM-13803] Add support for native iterable side inputs to the Go SDK (#16775)
add d7ee6b8 [BEAM-11095] Better error handling for illegal emit functions (#16776)
add df907de8 Merge pull request #16613 from Supporting JdbcIO driver in classpath for x-lang
add f3412a4 Merge pull request #15848 from [BEAM-13835] An any-type implementation for readWithPartitions for JdbcIO
add 821c432 [release-2.36.0][website] Fix github release notes script, header for release 2.36.0 (#16792)
add 6b14f19 Use shell to run python for setupVirtualenv (#16796)
add 5cc1707 Missing contribution
add c094a12 Merge pull request #16759 from davidpr91/patch1
add 0abe2e8 [BEAM-13830] Properly shut down Debezium expansion service in IT script.
add ef70473 Merge pull request #16797: [BEAM-13830] Properly shut down Debezium expansion service in IT script.
add a321802 [BEAM-12920] Assume that bare generators types define simple generators.
add 107eb3d [BEAM-12920] Assume that bare generator types define simple generators (#16791)
add 663e924 Merge pull request #16659 from [BEAM-13774][Playground] Add user to container
add 5ee8563 [BEAM-13776][Playground] (#16731)
add f9b7ab2 Add a container for Python 3.9.
add af19a1f Allow job submission with Python 3.9 on Dataflow runner
add 4c400d2 Add Python 3.9 test suites. Keep Dataflow V1 suites unchanged for now.
add 31deb25 Add py3.9 Github actions suites.
add 1ae9da0 Py39 Doc updates.
add c2d9255 [BEAM-9980] Simplify run_validates_container.sh to avoid branching.
add 8c0e265 Update Cython to a new version that has py39 wheels.
add fd0fcae [BEAM-13845] Fix comparison with potentially incomparable default values.
add e65ff00 [BEAM-12920] Assume that bare generators types define simple generators.
add 4b2e58f Mark Python 3.9 as supported version.
add 2401a11 [BEAM-13868] Remove gsutil dep from hdfs IT test.
add b07840e [BEAM-12000] Add Python 3.9 support. #16008
add 633169c [BEAM-13867] Drop NaNs returned by nlargest in flight_delays example pipeline (#16801)
add f774585 Announce Python 3.9 in CHANGES.md (#16802)
add 5a9a1bc Moving to 2.38.0-SNAPSHOT on master branch.
No new revisions were added by this update.
Summary of changes:
.github/workflows/python_tests.yml | 20 +-
.test-infra/jenkins/PythonTestProperties.groovy | 3 +-
.test-infra/jenkins/build.gradle | 1 +
.../jenkins/job_PreCommit_Portable_Python.groovy | 2 +-
.test-infra/jenkins/metrics_report/tox.ini | 2 +-
CHANGES.md | 13 +
build.gradle.kts | 17 +-
.../org/apache/beam/gradle/BeamModulePlugin.groovy | 11 +-
gradle.properties | 4 +-
playground/README.md | 15 +-
playground/backend/containers/go/Dockerfile | 16 +-
playground/backend/containers/go/build.gradle | 4 +
playground/backend/containers/go/entrypoint.sh | 8 -
.../containers/go/{entrypoint.sh => proxy.sh} | 1 -
playground/backend/containers/java/Dockerfile | 16 +-
playground/backend/containers/java/build.gradle | 4 +
playground/backend/containers/java/entrypoint.sh | 8 -
.../containers/{go/entrypoint.sh => java/proxy.sh} | 1 -
playground/backend/containers/python/Dockerfile | 15 +-
playground/backend/containers/python/build.gradle | 4 +
playground/backend/containers/python/entrypoint.sh | 9 -
.../containers/python/{entrypoint.sh => proxy.sh} | 1 -
playground/backend/containers/scio/Dockerfile | 15 +-
playground/backend/containers/scio/build.gradle | 4 +
playground/backend/containers/scio/entrypoint.sh | 8 -
.../containers/{go/entrypoint.sh => scio/proxy.sh} | 1 -
.../src/main/scripts/build_release_candidate.sh | 1 +
.../main/scripts/publish_github_release_notes.sh | 9 +-
sdks/go/pkg/beam/core/core.go | 2 +-
sdks/go/pkg/beam/core/funcx/fn.go | 5 +
sdks/go/pkg/beam/core/funcx/fn_test.go | 34 ++
sdks/go/pkg/beam/core/funcx/output.go | 44 ++-
sdks/go/pkg/beam/core/runtime/exec/data.go | 6 +-
sdks/go/pkg/beam/core/runtime/exec/sideinput.go | 61 ++-
.../pkg/beam/core/runtime/exec/sideinput_test.go | 203 ++++++++++
sdks/go/pkg/beam/core/runtime/graphx/translate.go | 43 +--
.../pkg/beam/core/runtime/graphx/translate_test.go | 6 +-
sdks/go/pkg/beam/core/runtime/harness/statemgr.go | 29 +-
sdks/go/pkg/beam/core/typex/class.go | 61 ++-
sdks/go/test/run_validatesrunner_tests.sh | 3 +
.../schemaio-expansion-service/build.gradle | 2 -
.../sdk/fn/data/BeamFnDataOutboundAggregator.java | 11 +-
.../sdk/fn/data/BeamFnDataOutboundObserver.java | 4 +-
sdks/java/io/amazon-web-services2/build.gradle | 2 -
.../sdk/io/aws2/kinesis/WatermarkPolicyTest.java | 58 ++-
sdks/java/io/common/build.gradle | 3 +
.../beam/sdk/io/common/DatabaseTestHelper.java | 56 ++-
sdks/java/io/expansion-service/build.gradle | 3 -
.../expansion-service/build.gradle | 8 +
sdks/java/io/jdbc/build.gradle | 6 +
.../java/org/apache/beam/sdk/io/jdbc/JdbcIO.java | 317 ++++++++++++----
.../java/org/apache/beam/sdk/io/jdbc/JdbcUtil.java | 172 +++++++--
.../beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java | 420 +++++++++++++++++++++
.../java/org/apache/beam/sdk/io/jdbc/JdbcIOIT.java | 9 +-
.../org/apache/beam/sdk/io/jdbc/JdbcIOTest.java | 66 ++--
.../org/apache/beam/sdk/io/jdbc/JdbcUtilTest.java | 195 ++++++++++
.../apache/beam/sdk/io/jdbc/OtherJdbcTypesIT.java | 9 +-
sdks/python/apache_beam/__init__.py | 2 +-
.../examples/dataframe/flight_delays.py | 8 +-
.../io/external/xlang_jdbcio_it_test.py | 72 +++-
sdks/python/apache_beam/io/jdbc.py | 41 +-
sdks/python/apache_beam/io/kafka.py | 2 +-
sdks/python/apache_beam/runners/common.py | 4 +-
.../runners/dataflow/internal/apiclient.py | 2 +-
.../runners/dataflow/internal/apiclient_test.py | 2 +-
sdks/python/apache_beam/transforms/external.py | 42 ++-
.../python/apache_beam/transforms/external_test.py | 25 ++
.../typehints/native_type_compatibility.py | 4 +-
.../typehints/native_type_compatibility_test.py | 4 -
sdks/python/apache_beam/utils/subprocess_server.py | 6 +-
sdks/python/apache_beam/version.py | 2 +-
sdks/python/container/build.gradle | 3 +
.../{py38 => py39}/base_image_requirements.txt | 16 +-
sdks/python/container/{py37 => py39}/build.gradle | 2 +-
sdks/python/container/run_validatescontainer.sh | 25 +-
sdks/python/setup.py | 6 +-
sdks/python/test-suites/dataflow/common.gradle | 2 +-
.../dataflow/{py37 => py39}/build.gradle | 2 +-
.../test-suites/direct/{py36 => py39}/build.gradle | 2 +-
sdks/python/test-suites/direct/xlang/build.gradle | 2 +
sdks/python/test-suites/gradle.properties | 29 +-
.../{dataflow/py37 => portable/py39}/build.gradle | 4 +-
sdks/python/test-suites/tox/py38/build.gradle | 1 +
.../test-suites/tox/{py36 => py39}/build.gradle | 6 +-
sdks/python/test-suites/tox/pycommon/build.gradle | 1 +
sdks/python/tox.ini | 22 +-
settings.gradle.kts | 5 +
website/www/site/content/en/blog/beam-2.28.0.md | 2 +-
website/www/site/content/en/blog/beam-2.35.0.md | 2 +
.../en/documentation/runtime/environments.md | 2 +
.../www/site/content/en/get-started/downloads.md | 2 +-
91 files changed, 1952 insertions(+), 451 deletions(-)
copy playground/backend/containers/go/{entrypoint.sh => proxy.sh} (96%)
mode change 100755 => 100644
copy playground/backend/containers/{go/entrypoint.sh => java/proxy.sh} (96%)
mode change 100755 => 100644
copy playground/backend/containers/python/{entrypoint.sh => proxy.sh} (96%)
mode change 100755 => 100644
copy playground/backend/containers/{go/entrypoint.sh => scio/proxy.sh} (96%)
mode change 100755 => 100644
create mode 100644 sdks/go/pkg/beam/core/runtime/exec/sideinput_test.go
create mode 100644 sdks/java/io/jdbc/src/test/java/org/apache/beam/sdk/io/jdbc/JdbcIOAutoPartitioningIT.java
copy sdks/python/container/{py38 => py39}/base_image_requirements.txt (94%)
copy sdks/python/container/{py37 => py39}/build.gradle (97%)
copy sdks/python/test-suites/dataflow/{py37 => py39}/build.gradle (97%)
copy sdks/python/test-suites/direct/{py36 => py39}/build.gradle (97%)
copy sdks/python/test-suites/{dataflow/py37 => portable/py39}/build.gradle (94%)
copy sdks/python/test-suites/tox/{py36 => py39}/build.gradle (90%)