You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@beam.apache.org by "Ning (Jira)" <ji...@apache.org> on 2022/05/09 18:25:00 UTC

[jira] [Created] (BEAM-14449) Support cluster provisioning when using Flink on Dataproc

Ning created BEAM-14449:
---------------------------

             Summary: Support cluster provisioning when using Flink on Dataproc
                 Key: BEAM-14449
                 URL: https://issues.apache.org/jira/browse/BEAM-14449
             Project: Beam
          Issue Type: Improvement
          Components: runner-py-interactive
            Reporter: Ning
            Assignee: Ning


Provide the capability for the user to explicitly provision a cluster.

Current implementation provisions each cluster at the location specified by GoogleCloudOptions using 3 worker nodes. There is no explicit API to configure the number or shape of workers.

We could use the WorkerOptions to allow customers to explicitly provision a cluster and expose an explicit API (with UX in notebook extension) for customers to change the size of a cluster connected with their notebook (until we have an auto scaling solution with Dataproc for Flink).



--
This message was sent by Atlassian Jira
(v8.20.7#820007)