You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@beam.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/10/11 14:00:00 UTC
[jira] [Work logged] (BEAM-5713) Flink portable runner schedules
all tasks of streaming job on same task manager
[ https://issues.apache.org/jira/browse/BEAM-5713?focusedWorklogId=153507&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-153507 ]
ASF GitHub Bot logged work on BEAM-5713:
----------------------------------------
Author: ASF GitHub Bot
Created on: 11/Oct/18 13:59
Start Date: 11/Oct/18 13:59
Worklog Time Spent: 10m
Work Description: mxm opened a new pull request #6654: [BEAM-5713] Make ImpulseSourceFunction execute in parallel
URL: https://github.com/apache/beam/pull/6654
SourceFunctions only get deployed in a single task slot. Consequently, Flink
schedules subsequent tasks in task slots on the same TaskManager (if
available). This makes the source parallel which creates multiple instances of
the source. This helps to spread the tasks across all TaskManagers.
Post-Commit Tests Status (on master branch)
------------------------------------------------------------------------------------------------
Lang | SDK | Apex | Dataflow | Flink | Gearpump | Samza | Spark
--- | --- | --- | --- | --- | --- | --- | ---
Go | [](https://builds.apache.org/job/beam_PostCommit_Go_GradleBuild/lastCompletedBuild/) | --- | --- | --- | --- | --- | ---
Java | [](https://builds.apache.org/job/beam_PostCommit_Java_GradleBuild/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Apex_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Flink_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Gearpump_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Samza_Gradle/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Java_ValidatesRunner_Spark_Gradle/lastCompletedBuild/)
Python | [](https://builds.apache.org/job/beam_PostCommit_Python_Verify/lastCompletedBuild/) | --- | [](https://builds.apache.org/job/beam_PostCommit_Py_VR_Dataflow/lastCompletedBuild/) </br> [](https://builds.apache.org/job/beam_PostCommit_Py_ValCont/lastCompletedBuild/) | [](https://builds.apache.org/job/beam_PostCommit_Python_VR_Flink/lastCompletedBuild/) | --- | --- | ---
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Issue Time Tracking
-------------------
Worklog Id: (was: 153507)
Time Spent: 10m
Remaining Estimate: 0h
> Flink portable runner schedules all tasks of streaming job on same task manager
> -------------------------------------------------------------------------------
>
> Key: BEAM-5713
> URL: https://issues.apache.org/jira/browse/BEAM-5713
> Project: Beam
> Issue Type: Bug
> Components: runner-flink
> Affects Versions: 2.8.0
> Reporter: Thomas Weise
> Assignee: Maximilian Michels
> Priority: Major
> Labels: portability, portability-flink
> Attachments: With RichParallelSourceFunction and parallelism 5.png
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The cluster has 9 task managers and 144 task slots total. A simple streaming pipeline with parallelism of 8 will get all tasks scheduled on the same task manager, causing the host to be fully booked and the remaining cluster idle.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)