You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by gi...@apache.org on 2020/07/19 02:47:19 UTC
[beam] tag nightly-master updated (56048f2 -> f841d60)
This is an automated email from the ASF dual-hosted git repository.
github-bot pushed a change to tag nightly-master
in repository https://gitbox.apache.org/repos/asf/beam.git.
*** WARNING: tag nightly-master was modified! ***
from 56048f2 (commit)
to f841d60 (commit)
from 56048f2 Move Java function implementations to a sub-module (#12247)
add 8250a7b Reducing how much we call BigQuery Get Table API. The BQ API is queried once per bundle. With this change, it will be queried once per worker thread. This will help with throughput for BQ streaming inserts
add e34fef8 Merge pull request #12125 from pabloem/BQ_API_quota
add 053ac16 [BEAM-10310] Adding thread-safe restriction tracker wrapper.
add 04e5cd3 [BEAM-10310] Adding GetRestriction method for RTrackers.
add 535c0f6 Merge pull request #12215: [BEAM-10310] Adding thread-safe restriction tracker wrapper.
add 2003985 [BEAM-10471] Change assertion method to improve expressiveness and increase error margin on non deterministic size test to avoid flakiness
add bedae33 Merge pull request #12276 from echauchot/BEAM-10471-cassandraIO-size-flakiness
add 188f89d [BEAM-10185] Building python wheels for Windows (on Github Actions) (#12172)
add e0a16e0 Merge pull request #12147 from [BEAM 7624] Added Null Behavior in JsonToRow
add ad7ee50 Disable flaky test: org.apache.beam.runners.samza.adapter.UnboundedSourceSystemTest.testMultipleAdvanceWatermark
add b0b8ece Disable flaky test: org.apache.beam.runners.samza.adapter.UnboundedSourceSystemTest.testMultipleAdvanceWatermark (#12291)
add 235c905 Ignore NameErrors right after WontImplementErrors.
add 5538d9e Add doctest skipping framework.
add e324b5c Add tests for all Pandas DataFrame and Series tests.
add 063a102 Merge pull request #12259 [BEAM-9561] Run Pandas tests on Beam.
add ed6bbfd [BEAM-10494] PubsubSchemaCapableIOProvider config inner class rather than row (#12266)
add cd6d904 [BEAM-10514] Restrict cache file path length
add 7e4e501 Merge pull request #12283 from KevinGG/fix_path_length
add a1c17cd [BEAM-8244] Split FlinkRunnerTest.test_external_transforms.
add 60d84dc Merge pull request #12286 from ibzib/BEAM-8244
add f509f78 [BEAM-10487] Link release blog posts on the Downloads page
add c8d2162 Merge pull request #12256 from ibzib/BEAM-10487
add 698401c Insert TruncateSizedRestriction when pipeline starts to drain.
add 0240471 Address latest comments.
add 047bed7 SpotlessApply
add 280e33d Fix java build.
add cf02dad spotlessApply
add 9b66b72 Only forward split/progress when the only consumer is splittable.
add ed39236 Merge pull request #12287 from boyuanzz/drain_java
add b7dffe3 [BEAM-8244] Don't run external transform tests with pre_optimize=all.
add f7b23ec Merge pull request #12303 from ibzib/BEAM-8244
add 373859a [BEAM-8454] Increase timeout and also enable thread stuckness detector. (#12293)
add cc27255 [BEAM-10490] Support read/write ZetaSQL DATE/TIME types from/to BigQuery (#12261)
add 8039b56 [BEAM-10526] Use GrpcCleanupRule to use consistent methodology on server shutdown. (#12301)
add 177445f [BEAM-7672] dynamically setup acceptable wheel specs according to installed python version
add 03296b3 fixup: update error message when parse failed
add 31a2657 fixup: not to abort when wheel spec setup failed
add 86456be fixup: not to use m flag at wheel name with python 3.8
add a03ebd6 fixup: simplified wheel name setting
add 0642073 [BEAM-7672] dynamically setup acceptable wheel specs according to installed python version #12115
add e688b28 [BEAM-9968] Guarantee that outstanding split/progress requests are handled before bundle completion (and before any final progress/checkpoint data sent to handlers). (#12305)
add 39bc4d3 Relax to matchers to match display data from the specific tests, not from all options.
add abf63e2 Merge pull request #12288 from aaltay/matchers
add ac6d80e [BEAM-10420] Add support for per window invocation of beam:transform:sdf_process_sized_element_and_restrictions:v1 (#12275)
add 8589741 use debian:buster for go SDK container
add 98e9922 Merge pull request #12290: Use debian:buster for go SDK container
add 2734080 Fix abi suffix tags for Python 3.8.
add af2d6b0 [BEAM-10434] Fix abi suffix tags for Python 3.8.
add 4decedd Insert TruncateSizedRestriction when pipeline starts to drain.
add f841d60 Merge pull request #12289 from boyuanzz/drain_py
No new revisions were added by this update.
Summary of changes:
.github/workflows/build_wheels.yml | 21 +-
.../pipeline/src/main/proto/beam_runner_api.proto | 10 +
.../runners/core/construction/Environments.java | 2 +
.../core/construction/PTransformTranslation.java | 5 +
.../core/construction/ParDoTranslation.java | 3 +
.../graph/SplittableParDoExpander.java | 232 +++++++++
.../core/construction/EnvironmentsTest.java | 7 +
.../core/construction/SplittableParDoTest.java | 5 +
.../graph/SplittableParDoExpanderTest.java | 56 ++
.../runners/core/SplittableParDoProcessFnTest.java | 5 +
.../worker/fn/BeamFnControlServiceTest.java | 20 +-
.../runners/fnexecution/control/RemoteBundle.java | 17 +-
.../fnexecution/control/SdkHarnessClient.java | 75 ++-
.../GrpcContextHeaderAccessorProviderTest.java | 55 +-
.../runners/fnexecution/ServerFactoryTest.java | 24 +-
.../fnexecution/control/RemoteExecutionTest.java | 6 +-
.../fnexecution/control/SdkHarnessClientTest.java | 232 +++++++++
.../runners/portability/PortableRunnerTest.java | 77 ++-
.../samza/adapter/UnboundedSourceSystemTest.java | 2 +
sdks/go/container/Dockerfile | 2 +-
sdks/go/examples/stringsplit/stringsplit.go | 15 +-
sdks/go/pkg/beam/core/graph/fn_test.go | 3 +
.../beam/core/runtime/exec/sdf_invokers_test.go | 3 +-
.../pkg/beam/core/runtime/graphx/translate_test.go | 15 +-
sdks/go/pkg/beam/core/sdf/lock.go | 82 +++
sdks/go/pkg/beam/core/sdf/sdf.go | 4 +
.../beam/io/rtrackers/offsetrange/offsetrange.go | 67 +--
sdks/go/pkg/beam/io/synthetic/source.go | 11 +-
sdks/go/pkg/beam/io/synthetic/step.go | 15 +-
.../src/main/java/org/apache/beam/sdk/io/Read.java | 15 +-
.../java/org/apache/beam/sdk/transforms/DoFn.java | 57 +++
.../org/apache/beam/sdk/transforms/JsonToRow.java | 53 +-
.../beam/sdk/transforms/PeriodicSequence.java | 5 +
.../java/org/apache/beam/sdk/transforms/Watch.java | 8 +
.../reflect/ByteBuddyDoFnInvokerFactory.java | 30 +-
.../beam/sdk/transforms/reflect/DoFnInvoker.java | 6 +
.../beam/sdk/transforms/reflect/DoFnSignature.java | 32 ++
.../sdk/transforms/reflect/DoFnSignatures.java | 82 +++
.../splittabledofn/ByteKeyRangeTracker.java | 5 +
.../splittabledofn/GrowableOffsetRangeTracker.java | 9 +
.../splittabledofn/OffsetRangeTracker.java | 5 +
.../splittabledofn/RestrictionTracker.java | 30 ++
.../apache/beam/sdk/transforms/JsonToRowTest.java | 99 ++++
.../sdk/transforms/reflect/DoFnInvokersTest.java | 240 ++++++++-
.../reflect/DoFnSignaturesSplittableDoFnTest.java | 131 +++++
.../GrowableOffsetRangeTrackerTest.java | 20 +
...amFnDataSizeBasedBufferingOutboundObserver.java | 2 +-
.../sdk/fn/splittabledofn/RestrictionTrackers.java | 5 +
...DataSizeBasedBufferingOutboundObserverTest.java | 9 +-
.../fn/splittabledofn/RestrictionTrackersTest.java | 10 +
.../apache/beam/fn/harness/FnApiDoFnRunner.java | 353 +++++++++----
.../beam/fn/harness/FnApiDoFnRunnerTest.java | 570 +++++++++++++++++++++
.../org/apache/beam/fn/harness/FnHarnessTest.java | 9 +-
.../beam/sdk/io/cassandra/CassandraIOTest.java | 9 +-
.../beam/sdk/io/gcp/bigquery/BigQueryUtils.java | 133 +++--
.../gcp/pubsub/PubsubSchemaCapableIOProvider.java | 59 ++-
.../beam/sdk/io/gcp/pubsub/RowToPubsubMessage.java | 14 +-
.../sdk/io/gcp/bigquery/BigQueryUtilsTest.java | 86 +---
sdks/python/apache_beam/dataframe/doctests.py | 39 +-
sdks/python/apache_beam/dataframe/doctests_test.py | 20 +
.../apache_beam/dataframe/pandas_doctests_test.py | 186 +++++++
sdks/python/apache_beam/io/gcp/bigquery.py | 10 +-
sdks/python/apache_beam/io/iobase.py | 33 +-
sdks/python/apache_beam/io/restriction_trackers.py | 6 +
sdks/python/apache_beam/runners/common.pxd | 5 +
sdks/python/apache_beam/runners/common.py | 378 +++++++++++---
sdks/python/apache_beam/runners/common_test.py | 438 ++++++++++++++++
.../runners/interactive/cache_manager.py | 13 +-
.../runners/interactive/interactive_environment.py | 2 +-
.../runners/interactive/pipeline_instrument.py | 5 +-
.../runners/portability/flink_runner_test.py | 12 +-
.../runners/portability/fn_api_runner/execution.py | 14 +
.../runners/portability/fn_api_runner/fn_runner.py | 8 +-
.../portability/fn_api_runner/fn_runner_test.py | 214 ++++++--
.../portability/fn_api_runner/translations.py | 66 ++-
.../runners/portability/portable_runner_test.py | 14 +-
.../apache_beam/runners/portability/stager.py | 9 +-
sdks/python/apache_beam/runners/sdf_utils.py | 6 +
.../apache_beam/runners/worker/bundle_processor.py | 60 ++-
.../runners/worker/bundle_processor_test.py | 17 +-
.../runners/worker/crossplatform_time.h | 63 +++
.../runners/worker/crossplatform_unistd.h | 37 +-
.../apache_beam/runners/worker/operations.pxd | 2 +
.../apache_beam/runners/worker/operations.py | 31 +-
.../runners/worker/statesampler_fast.pyx | 4 +-
sdks/python/apache_beam/transforms/core.py | 28 +
sdks/python/apache_beam/transforms/display_test.py | 10 +-
sdks/python/apache_beam/transforms/environments.py | 2 +
.../apache_beam/transforms/environments_test.py | 4 +
sdks/python/container/boot.go | 34 +-
sdks/python/setup.py | 18 +-
.../www/site/content/en/get-started/downloads.md | 20 +
92 files changed, 4293 insertions(+), 672 deletions(-)
create mode 100644 sdks/go/pkg/beam/core/sdf/lock.go
create mode 100644 sdks/python/apache_beam/dataframe/pandas_doctests_test.py
create mode 100644 sdks/python/apache_beam/runners/worker/crossplatform_time.h
copy runners/flink/src/main/java/org/apache/beam/runners/flink/translation/functions/FlinkIdentityFunction.java => sdks/python/apache_beam/runners/worker/crossplatform_unistd.h (57%)