[jira] [Commented] (FLINK-17392) enable configuring minicluster resources in Flink SQL in IDE

["Jark Wu (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-17392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17092914#comment-17092914 ] 

Jark Wu commented on FLINK-17392:
---------------------------------

Hi [~phoenixjiangnan], sorry, I'm not fully understand. 

> in minicluster, if it is single parallelism job, then chain all operators together
Why do users want to avoid shuffling in minicluster? Performance purpose? Why do users care performance in minicluster? I think minicluster is not used for production for now. 
However, removing shuffling is in our roadmap but not in a high priority. Removing shuffling shouldn't be bound by minicluster. But this is not a small effort, we need to be careful with the keyed state.

> enable configuring minicluster in Flink SQL in IDE.
I don't think exposing minicluster to TableEnvironment is a good idea. Even {{StreamExecutionEnvironment}} doesn't provide this. If users want to modify resources. He can just set parallelism using the {{parallelism.default}} configuration.



> enable configuring minicluster resources in Flink SQL in IDE
> ------------------------------------------------------------
>
>                 Key: FLINK-17392
>                 URL: https://issues.apache.org/jira/browse/FLINK-17392
>             Project: Flink
>          Issue Type: Improvement
>          Components: Table SQL / API
>    Affects Versions: 1.11.0
>            Reporter: Bowen Li
>            Assignee: Kurt Young
>            Priority: Major
>             Fix For: 1.11.0
>
>
> It's very common case that users who want to learn and test Flink SQL will try to run a SQL job in IDE like Intellij, with Flink minicluster. Currently it's fine to do so with a simple job requiring only one task slot, which is the default resource config of minicluster.
> However, users cannot run even a little bit more complicated job since they cannot configure task slots of minicluster thru Flink SQL, e.g. single parallelism job requires shuffle. This incapability has been very frustrating to new users.
> There are two solutions to this problem:
> - in minicluster, if it is single parallelism job, then chain all operators together
> - enable configuring minicluster in Flink SQL in IDE.
> The latter feels more proper.
> Expected: users can configure minicluster resources via either SQL ("set ...=...") or TableEnvironment ("tEnv.setMiniclusterResources(..., ...)"). 
> [~jark] [~lzljs3620320]



--
This message was sent by Atlassian Jira
(v8.3.4#803005)