[jira] [Comment Edited] (FLINK-15527) can not control the number of container on yarn single job module

["chenchencc (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/FLINK-15527?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17011478#comment-17011478 ] 

chenchencc edited comment on FLINK-15527 at 1/9/20 6:41 AM:
------------------------------------------------------------

But i don't want to use so much parallelism. I accpet to use less paralilelism and use more time.

like spark has max container and min container


was (Author: chenchencc):
But i don't want to use so much parallelism. I accpet to use less paralilelism and use more time

> can not control the number of container on yarn single job module
> -----------------------------------------------------------------
>
>                 Key: FLINK-15527
>                 URL: https://issues.apache.org/jira/browse/FLINK-15527
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN
>    Affects Versions: 1.10.0
>            Reporter: chenchencc
>            Priority: Major
>             Fix For: 1.10.0
>
>         Attachments: application_1576096842353_203666.log, flink-conf.yaml, image-2020-01-09-14-30-46-973.png, yarn_application.png
>
>
> when run yarn single job run many container but paralism set 4
> *scripts:*
> ./bin/flink run -m yarn-cluster -ys 3 -p 4 -yjm 1024m -ytm 4096m -yqu bi -c com.cc.test.HiveTest2 ./cc_jars/hive-1.0-SNAPSHOT.jar 11.txt test61 6
> _notes_: in  1.9.1 has cli paramter -yn to control the number of containers and in 1.10 remove it
> *result:*
> the number of containers is 500+
>  
> *code use:*
> query the table and save it to the hdfs text
>  
> the storge of table is 200g+
>  
>  
>  
>  
> *code:*
> com.cc.test.HiveTest2
> public static void main(String[] args) throws Exception
> { EnvironmentSettings settings = EnvironmentSettings.newInstance().useBlinkPlanner().inStreamingMode().build(); StreamExecutionEnvironment settings2 = StreamExecutionEnvironment.getExecutionEnvironment();
> settings2.setParallelism(Integer.valueOf(args[2]));
> StreamTableEnvironment tableEnv = StreamTableEnvironment.create(settings2, settings); String name = "myhive"; String defaultDatabase = "test"; String hiveConfDir = "/etc/hive/conf"; String version = "1.2.1"; // or 1.2.1 2.3.4 HiveCatalog hive = new HiveCatalog(name, defaultDatabase, hiveConfDir, version); tableEnv.registerCatalog("myhive", hive); tableEnv.useCatalog("myhive"); tableEnv.listTables(); Table table = tableEnv.sqlQuery("select id from orderparent_test2 where id = 'A000021204170176'"); tableEnv.toAppendStream(table, Row.class).print(); tableEnv.toAppendStream(table, Row.class) .writeAsText("hdfs:///user/chenchao1/"+ args[0], FileSystem.WriteMode.OVERWRITE); tableEnv.execute(args[1]); }
>  



--
This message was sent by Atlassian Jira
(v8.3.4#803005)