You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@beam.apache.org by "Kamil Wasilewski (Jira)" <ji...@apache.org> on 2020/09/15 15:20:00 UTC
[jira] [Updated] (BEAM-10659) ParDo Python streaming load tests
timeouts on 200-iterations case
[ https://issues.apache.org/jira/browse/BEAM-10659?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Kamil Wasilewski updated BEAM-10659:
------------------------------------
Status: Resolved (was: Open)
> ParDo Python streaming load tests timeouts on 200-iterations case
> -----------------------------------------------------------------
>
> Key: BEAM-10659
> URL: https://issues.apache.org/jira/browse/BEAM-10659
> Project: Beam
> Issue Type: Bug
> Components: testing
> Reporter: Kasia Kucharczyk
> Priority: P2
>
> Running Python Dataflow load test in streaming option timeouts on Jenkins on case 2:
> {code:java}
> 2GB 100 byte records 200 times
> {code}
> It [iterates|https://github.com/apache/beam/blob/master/sdks/python/apache_beam/testing/load_tests/pardo_test.py#L147] same ParDo step sequentially.
> Jenkins jobs has 2h timeout. Second case usually is [cancelled|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_05_00_47-15183151853043328210;mainTab=JOB_METRICS?project=apache-beam-testing] after 1h 47 min. The most suspicious metric here is throughput which in comparison to other jobs doesn't look steady. Sometimes there are spike after 1 hour of non action, or there are several spikes (to 30 000 elements/sec).
> [Python batch case|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-04_06_32_29-2466435392086580014;step=s1;mainTab=JOB_METRICS?project=apache-beam-testing] scenario takes ~56 minutes, with steady throughput ~7000 elements/sec for almost whole job run.
> In comparison [Java same test case|https://console.cloud.google.com/dataflow/jobs/us-central1/2020-08-03_05_13_48-16554947290254286391;mainTab=JOB_GRAPH?project=apache-beam-testing] takes ~6 minutes. Here throughput goes up to ~100 000 elements/sec then after processing all elements it decreases.
>
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)