You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "halaharr (via GitHub)" <gi...@apache.org> on 2023/10/02 14:12:04 UTC
[I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
halaharr opened a new issue, #28765:
URL: https://github.com/apache/beam/issues/28765
We are seeing Dataflow pipelines taking 2x to 3x more time to run in Apache beam SDK ver 2.50 compared to Apache beam SDK ver 2.44. As part of troubleshooting we compared the DAGS in 2.44 and 2.50 and we are seeing BQ read from table step in DAG (full table scan using DIRECT_TABLE_ACCESS) taking 3 sec to read 19 records / 13KB size in 2.44 and same exact pipeline with exactly same 19 records and 13KB size taking 1 min 5 sec in 2.50. Is this because this API has degraded in ver 2.50 since I also see throughput for this DAG step is much higher in 2.44 than 2.50. Please find the throughput graph images (elements/sec) below for both versions below
Throughput in ver 2.44 --> 0.15 sec (High)
Throughput in ver 2.50 --> 0.083 sec (Low)
<img width="598" alt="apache_beam_250" src="https://github.com/apache/beam/assets/16997826/27292c20-c2c1-4bd6-b3be-d3a74e82c638">
<img width="611" alt="apache_beam_244" src="https://github.com/apache/beam/assets/16997826/3a3481a8-3de1-4916-82e6-5b9cd4ff981f">
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
Posted by "halaharr (via GitHub)" <gi...@apache.org>.
halaharr commented on issue #28765:
URL: https://github.com/apache/beam/issues/28765#issuecomment-1744021169
> > These metrics are averaged on each minute. This small amount of data is not sufficient for comparing efficiency. Usually it needs ~10 min steady throughput and multiple runs to tell the differences.
> >
> > * Our performance tests does not show a regression - http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1&viewPanel=23&from=now-1y&to=now
>
> Thanks @Abacn appreciate your quick reply. We will test with a larger BQ table (with probably 5k to 10k records) and compare the results in both versions.
@Abacn I am assuming you are talking about the throughput metrics. when you mentioned that metrics are averaged on each minute but what about the execution time? The Dataflow pipeline runs for over 1 min 15 sec with 250 sdk version as opposed to a 30 second run for 244 sdk version.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #28765:
URL: https://github.com/apache/beam/issues/28765#issuecomment-1743206761
These metrics are averaged on each minute. This small amount of data is not sufficient for comparing efficiency. Usually it needs ~10 min steady throughput and multiple runs to tell the differences.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #28765: Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250
URL: https://github.com/apache/beam/issues/28765
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
Posted by "halaharr (via GitHub)" <gi...@apache.org>.
halaharr commented on issue #28765:
URL: https://github.com/apache/beam/issues/28765#issuecomment-1743232007
> These metrics are averaged on each minute. This small amount of data is not sufficient for comparing efficiency. Usually it needs ~10 min steady throughput and multiple runs to tell the differences.
>
> * Our performance tests does not show a regression - http://metrics.beam.apache.org/d/bnlHKP3Wz/java-io-it-tests-dataflow?orgId=1&viewPanel=23&from=now-1y&to=now
Thanks @Abacn appreciate your quick reply. We will test with a larger BQ table (with probably 5k to 10k records) and compare the results in both versions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28765:
URL: https://github.com/apache/beam/issues/28765#issuecomment-4742538447
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Dataflow pipelines in GCP are 2x to 3x times slower in apache beam java sdk ver 250 [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #28765:
URL: https://github.com/apache/beam/issues/28765#issuecomment-5015776597
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org