[jira] [Created] (BEAM-10659) ParDo Python streaming load tests timeouts on 200-iterations case

["Kasia Kucharczyk (Jira)" <ji...@apache.org>]

Kasia Kucharczyk created BEAM-10659:
---------------------------------------

             Summary: ParDo Python streaming load tests timeouts on 200-iterations case
                 Key: BEAM-10659
                 URL: https://issues.apache.org/jira/browse/BEAM-10659
             Project: Beam
          Issue Type: Bug
          Components: testing
            Reporter: Kasia Kucharczyk


Running Python Dataflow load test in streaming option timeouts on Jenkins on case 2:

{code:java}
2GB 100 byte records 200 times
{code}


 It [iterates|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/load_tests/pardo_test.py#L147] same ParDo step sequentially. 

Jenkins jobs has 2h timeout. Second case usually is [cancelled|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_05_00_47-15183151853043328210;mainTab=JOB_METRICS?project=apache-beam-testing] after 1h 47 min. The most suspicious metric here is throughput which in comparison to other jobs doesn't look steady. Sometimes there are spike after 1 hour of non action, or there are several spikes (to 30 000 elements/sec).

[Python batch case|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_06_32_29-2466435392086580014;step=s1;mainTab=JOB_METRICS?project=apache-beam-testing] scenario takes ~56 minutes, with steady throughput ~7000 elements/sec for almost whole job run.

In comparison [Java same test case|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-03_05_13_48-16554947290254286391;mainTab=JOB_GRAPH?project=apache-beam-testing] takes ~6 minutes. Here throughput goes up to ~100 000 elements/sec then after processing all elements it decreases.

 

 

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)