You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@flink.apache.org by "swy (JIRA)" <ji...@apache.org> on 2018/05/26 09:26:00 UTC
[jira] [Updated] (FLINK-9442) Flink Scaling not working
[ https://issues.apache.org/jira/browse/FLINK-9442?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
swy updated FLINK-9442:
-----------------------
Description:
Hi,
We are in the middle of testing scaling ability of Flink. But we found that scaling not working, no matter increase more slot or increase number of Task Manager. We would expect a linear, if not close-to-linear scaling performance but the result even show degradation. Appreciated any comments.
Test Details,
-VMWare vsphere
-Just a simple pass through test,
- auto gen source 3mil records, each 1000k in size, parallelism=1
- source pass into next map operator, which just return the same record, and sent counter to statsD, parallelism is in cases = 2,4,6
- 3 TM, total 6 slots(2/TM) each JM/TM has 32 vCPU, 100GB memory
- Result:
- 2 slots: 26 seconds, 3mil/26=115k TPS
- 4 slots: 23 seconds, 3mil/23=130k TPS
- 6 slots: 22 seconds, 3mil/22=136k TPS
As shown the scaling is almost nothing, and capped at ~120k TPS. Any clue? Thanks.
was:
Hi,
We are in the middle of testing scaling ability of Flink. But we found that scaling not working, no matter increase more slot or increase number of Task Manager. We would expect a linear, if not close-to-linear scaling performance but the result even show degradation. Appreciated any comments.
Test Details,
-VMWare vsphere
-Just a simple pass through test,
- auto gen source 3mil records, each 1000k in size, parallelism=1
- source pass into next map operator, which just return the same record, and sent counter to statsD, parallelism is in cases = 2,4,6
- 3 TM, total 6 slots(2/TM) each JM/TM has 32 vCPU, 100GB memory
- Result:
- 2 slots: 26 seconds, 3mil/26=115k TPS
- 4 slots: 23 seconds, 3mil/23=130k TPS
- 6 slots: 22 seconds, 3mil/22=136k TPS
As shown the scaling is almost nothing. Any clue? Thanks.
> Flink Scaling not working
> -------------------------
>
> Key: FLINK-9442
> URL: https://issues.apache.org/jira/browse/FLINK-9442
> Project: Flink
> Issue Type: Bug
> Reporter: swy
> Priority: Major
>
> Hi,
>
> We are in the middle of testing scaling ability of Flink. But we found that scaling not working, no matter increase more slot or increase number of Task Manager. We would expect a linear, if not close-to-linear scaling performance but the result even show degradation. Appreciated any comments.
>
> Test Details,
>
> -VMWare vsphere
> -Just a simple pass through test,
> - auto gen source 3mil records, each 1000k in size, parallelism=1
> - source pass into next map operator, which just return the same record, and sent counter to statsD, parallelism is in cases = 2,4,6
> - 3 TM, total 6 slots(2/TM) each JM/TM has 32 vCPU, 100GB memory
> - Result:
> - 2 slots: 26 seconds, 3mil/26=115k TPS
> - 4 slots: 23 seconds, 3mil/23=130k TPS
> - 6 slots: 22 seconds, 3mil/22=136k TPS
>
> As shown the scaling is almost nothing, and capped at ~120k TPS. Any clue? Thanks.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)