You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2017/02/08 21:10:41 UTC
[jira] [Commented] (BEAM-1438) The default behavior for the Write
transform doesn't work well with the Dataflow streaming runner
[ https://issues.apache.org/jira/browse/BEAM-1438?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15858550#comment-15858550 ]
ASF GitHub Bot commented on BEAM-1438:
--------------------------------------
GitHub user reuvenlax opened a pull request:
https://github.com/apache/beam/pull/1952
BEAM-1438 Auto shard streaming sinks
If a Write requests runner-determined sharding, per-bundle sharding is the default but performs poorly in Dataflow's streaming runner. Instead, the runner statically picks a sharding based on the number of workers.
You can merge this pull request into a Git repository by running:
$ git pull https://github.com/reuvenlax/incubator-beam streaming_auto_shard_write
Alternatively you can review and apply these changes as the patch at:
https://github.com/apache/beam/pull/1952.patch
To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:
This closes #1952
----
commit a5894fd416b07b0ede094022ccf6539767507c7f
Author: Reuven Lax <re...@relax-macbookpro.roam.corp.google.com>
Date: 2016-11-14T04:29:31Z
Add PubSub attributes support to PubsubIO.
commit 459dbd2089766f4ab7b631ed1819d938da7ba5c8
Author: Reuven Lax <re...@google.com>
Date: 2017-02-08T20:53:27Z
If a Write operation requests runner-determined sharding, make the Dataflow runner default to maxNumWorkers * 2 shards.
commit 1d3a5239e3a6bd5646638e90578d8a8034fa78d1
Author: Reuven Lax <re...@google.com>
Date: 2017-02-08T20:57:33Z
Fix nullable attributes
----
> The default behavior for the Write transform doesn't work well with the Dataflow streaming runner
> -------------------------------------------------------------------------------------------------
>
> Key: BEAM-1438
> URL: https://issues.apache.org/jira/browse/BEAM-1438
> Project: Beam
> Issue Type: Bug
> Components: runner-dataflow
> Reporter: Reuven Lax
> Assignee: Davor Bonaci
>
> If a Write specifies 0 output shards, that implies the runner should pick an appropriate sharding. The default behavior is to write one shard per input bundle. This works well with the Dataflow batch runner, but not with the streaming runner which produces large numbers of small bundles.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)