[jira] [Commented] (BEAM-10774) GBK Python streaming load tests are too slow

["Danny McCormick (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/BEAM-10774?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17548788#comment-17548788 ] 

Danny McCormick commented on BEAM-10774:
----------------------------------------

This issue has been migrated to https://github.com/apache/beam/issues/20403

> GBK Python streaming load tests are too slow
> --------------------------------------------
>
>                 Key: BEAM-10774
>                 URL: https://issues.apache.org/jira/browse/BEAM-10774
>             Project: Beam
>          Issue Type: Bug
>          Components: runner-dataflow, sdk-py-core, testing
>            Reporter: Kamil Wasilewski
>            Priority: P3
>
> The following GBK streaming test cases take too long on Dataflow:
>  
> 1) 2GB of 10B records
> 2) 2GB of 100B records
> 4) fanout 4 times with 2GB 10-byte records total
> 5) fanout 8 times with 2GB 10-byte records total
>  
> Each of them needs at least 1 hour to execute, which is way too long for one Jenkins job. 
> Job's definition: [https://github.com/apache/beam/blob/master/.test-infra/jenkins/job_LoadTests_GBK_Python.groovy]
> Test pipeline: [https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/load_tests/group_by_key_test.py]
> It is probable that those cases are too extreme. The first two cases involve grouping 20M unique keys, which is a stressful operation. A solution might be to overhaul the cases so that they would be less complex.
> Both the current production Dataflow runner and the new Dataflow Runner V2 were tested.



--
This message was sent by Atlassian Jira
(v8.20.7#820007)