You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@beam.apache.org by "Thomas Weise (JIRA)" <ji...@apache.org> on 2018/09/28 05:03:00 UTC

[jira] [Created] (BEAM-5520) Flink runner per operator SDK harness option

Thomas Weise created BEAM-5520:
----------------------------------

             Summary: Flink runner per operator SDK harness option
                 Key: BEAM-5520
                 URL: https://issues.apache.org/jira/browse/BEAM-5520
             Project: Beam
          Issue Type: Improvement
          Components: runner-flink
            Reporter: Thomas Weise
            Assignee: Thomas Weise


For streaming pipelines, the runner currently uses a single SDK harness process for all subtasks of a job that get deployed on the same task manager. In common deployments with 16 or more tasks slots, many executable stage operators all use the same SDK harness process. To scale, we need an option to run separate harness processes per subtask.

   



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)