You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/11/05 08:59:00 UTC
[jira] [Work logged] (BEAM-5791) Bound the amount of data on the
data plane by time.
[ https://issues.apache.org/jira/browse/BEAM-5791?focusedWorklogId=162451&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-162451 ]
ASF GitHub Bot logged work on BEAM-5791:
----------------------------------------
Author: ASF GitHub Bot
Created on: 05/Nov/18 08:58
Start Date: 05/Nov/18 08:58
Worklog Time Spent: 10m
Work Description: robertwb opened a new pull request #6940: [BEAM-5791] Implement time-based pushback in the dataflow harness data plane.
URL: https://github.com/apache/beam/pull/6940
Reverts "Merge pull request #6855: Revert #6752 #6798 #6807 #6837 to fix Dataflow test breakage"
This reverts commit a7c3078712db2d11fd0cbc86ff41b35271559093, reversing
changes made to 52721a1fabc95fe0e26fbb8b6338ae75f773ec8c.
**Please** add a meaningful description for your change here
------------------------
Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Format the pull request title like `[BEAM-XXX] Fixes bug in ApproximateQuantiles`, where you replace `BEAM-XXX` with the appropriate JIRA issue, if applicable. This will automatically link the pull request to the issue.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
It will help us expedite review of your Pull Request if you tag someone (e.g. `@username`) to look at it.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) [](https://builds.apache.org/job/beam_PostCommit_Java_PVR_Flink/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 162451)
Time Spent: 2.5h (was: 2h 20m)
> Bound the amount of data on the data plane by time.
> ---------------------------------------------------
>
> Key: BEAM-5791
> URL: https://issues.apache.org/jira/browse/BEAM-5791
> Project: Beam
> Issue Type: Improvement
> Components: runner-dataflow, sdk-java-harness, sdk-py-harness
> Reporter: Robert Bradshaw
> Assignee: Henning Rohde
> Priority: Major
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> This is especially important for Fn API reads, where each element represents a shard to read and may be very expensive, but many elements may be waiting in the Fn API buffer.
> The need for this will be mitigated with full SDF support for liquid sharding over the Fn API, but not eliminated unless the runner can "unread" elements it has already sent.
> This is especially important in for dataflow jobs that start out small but then detect that they need more workers (e.g. due to the initial inputs being an SDF).
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)