You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "arunpandianp (via GitHub)" <gi...@apache.org> on 2025/01/23 04:55:30 UTC
[PR] [Dataflow Streaming] Optimizing work submit loop [beam]
arunpandianp opened a new pull request, #33736:
URL: https://github.com/apache/beam/pull/33736
1. Moving recordGetWorkStreamLatencies to run outside work submission loop
2. Change OutputReceiver::outputCounters from HashMap to LinkedHashmap
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on code in PR #33736:
URL: https://github.com/apache/beam/pull/33736#discussion_r1926644689
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/OutputReceiver.java:
##########
@@ -27,7 +28,7 @@
*/
public class OutputReceiver implements Receiver {
private final List<Receiver> outputs = new ArrayList<>();
- private final HashMap<String, ElementCounter> outputCounters = new HashMap<>();
+ private final HashMap<String, ElementCounter> outputCounters = new LinkedHashMap<>();
Review Comment:
motivation for this?
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/Work.java:
##########
@@ -88,7 +87,7 @@ private Work(
this.watermarks = watermarks;
this.clock = clock;
this.startTime = clock.get();
- this.totalDurationPerState = new EnumMap<>(LatencyAttribution.State.class);
+ this.totalDurationPerState = new HashMap<>();
Review Comment:
Another idea where the map would still be stored as an array without hashing is to have an empty static EnumMap that you construct the new map referencing. Then it wouldn't have to recalculate the key universe
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/StreamingWorkScheduler.java:
##########
@@ -300,6 +301,10 @@ private void processWork(ComputationState computationState, Work work) {
}
}
+ private void processWork(ComputationState computationState, Work work) {
+ processWork(computationState, work, ImmutableList.of());
Review Comment:
instead of this delegating to the one with latencies how about that one delegates to this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #33736:
URL: https://github.com/apache/beam/pull/33736#issuecomment-2611950923
> Another idea where the map would still be stored as an array without hashing is to have an empty static EnumMap that you construct the new map referencing. Then it wouldn't have to recalculate the key universe
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #33736:
URL: https://github.com/apache/beam/pull/33736#issuecomment-2608915890
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle merged PR #33736:
URL: https://github.com/apache/beam/pull/33736
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #33736:
URL: https://github.com/apache/beam/pull/33736#issuecomment-2608916690
Run Java PreCommit
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on code in PR #33736:
URL: https://github.com/apache/beam/pull/33736#discussion_r1928318313
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/windmill/work/processing/StreamingWorkScheduler.java:
##########
@@ -300,6 +301,10 @@ private void processWork(ComputationState computationState, Work work) {
}
}
+ private void processWork(ComputationState computationState, Work work) {
+ processWork(computationState, work, ImmutableList.of());
Review Comment:
done
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on code in PR #33736:
URL: https://github.com/apache/beam/pull/33736#discussion_r1926364463
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/streaming/Work.java:
##########
@@ -88,7 +87,7 @@ private Work(
this.watermarks = watermarks;
this.clock = clock;
this.startTime = clock.get();
- this.totalDurationPerState = new EnumMap<>(LatencyAttribution.State.class);
+ this.totalDurationPerState = new HashMap<>();
Review Comment:
EnumMap is constructing a keyUniverse array in the constructor which shows up in profiles, replacing it with hashmap to speed up work submit loop.
<img width="1693" alt="Screenshot 2025-01-22 at 9 00 34 PM" src="https://github.com/user-attachments/assets/ae4bca8c-d2fa-4f07-9631-d20173623f8d" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "scwhittle (via GitHub)" <gi...@apache.org>.
scwhittle commented on PR #33736:
URL: https://github.com/apache/beam/pull/33736#issuecomment-2612062792
failure appears unrelated
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on code in PR #33736:
URL: https://github.com/apache/beam/pull/33736#discussion_r1926662161
##########
runners/google-cloud-dataflow-java/worker/src/main/java/org/apache/beam/runners/dataflow/worker/util/common/worker/OutputReceiver.java:
##########
@@ -27,7 +28,7 @@
*/
public class OutputReceiver implements Receiver {
private final List<Receiver> outputs = new ArrayList<>();
- private final HashMap<String, ElementCounter> outputCounters = new HashMap<>();
+ private final HashMap<String, ElementCounter> outputCounters = new LinkedHashMap<>();
Review Comment:
the map's values are iterated twice for every element in `proces()` below. Changed to LinkedHashmap since it is iteration friendly.
<img width="1700" alt="Screenshot 2025-01-23 at 1 35 14 AM" src="https://github.com/user-attachments/assets/c09fd669-acea-4dd7-8e41-f814d4d408e4" />
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #33736:
URL: https://github.com/apache/beam/pull/33736#issuecomment-2608863004
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control. If you'd like to restart, comment `assign set of reviewers`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Dataflow Streaming] Optimizing work submit loop [beam]
Posted by "arunpandianp (via GitHub)" <gi...@apache.org>.
arunpandianp commented on PR #33736:
URL: https://github.com/apache/beam/pull/33736#issuecomment-2608861813
R: @scwhittle
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org