You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "nbali (via GitHub)" <gi...@apache.org> on 2026/10/02 20:27:43 UTC

[PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

nbali opened a new pull request, #40391:
URL: https://github.com/apache/beam/pull/40391

   Addresses #27170 (alleviates it, does not fix it).
   
   ## What
   
   FirestoreIO batch writes now grow their ramp-up budget from one pipeline-wide start time, instead of from each DoFn instance's first write.
   
   - `FirestoreV1.BatchWriteWithSummary` and `BatchWriteWithDeadLetterQueue` create a singleton side input holding `clock.instant()`, computed once per pipeline (`Create` → `MapElements` → `View.asSingleton()`), and pass it to the write `ParDo`.
   - `FirestoreV1WriteFn.BaseBatchWriteFn.processElement` reads the side input and hands it to `RpcQos.setRampUpStart(Instant)`.
   - `RpcQosImpl.WriteRampUp` uses that instant as its ramp-up start. If it's never set, behavior is unchanged: the ramp-up starts lazily at the first write.
   
   ## Why
   
   #27170 describes how slow the write ramp-up is for small jobs. On top of the low starting budget (`500 / hintMaxNumWorkers`, 1 write/s with the default hint), each `RpcQos` instance (one per DoFn instance, created in `@Setup`) starts its own ramp-up clock at its first write. Every DoFn instance created later starts again at the base budget. This happens on autoscaled workers, when concurrency grows, and after a failed bundle discards its DoFn instances. Time the pipeline spends before its first write doesn't count either.
   
   With a shared start time:
   - DoFn instances created later join at the budget the pipeline has already grown to, instead of starting over at 1 write/s;
   - time spent before the first write (for example a long read or transform stage) counts toward the ramp-up.
   
   Trade-off, the same one DatastoreIO accepts: if the first write happens long after the pipeline starts, the budget has already grown, so those writes get less ramp-up protection.
   
   ## Where it comes from
   
   This copies the approach DatastoreIO already uses. `DatastoreV1.Mutate.expand` creates a "Generate start timestamp" side input (`Create.of("side input")` → `MapElements.via(s -> Instant.now())` → `View.asSingleton()`) and passes it to `RampupThrottlingFn`. Its `processElement` reads `c.sideInput(firstInstantSideInput)` and computes the budget relative to that instant. This PR applies the same pattern to the Firestore write path. The Firestore ramp-up lives inside `RpcQosImpl` rather than in a separate DoFn, so the instant is passed into it.
   
   ## Pipeline update compatibility
   
   The write transforms gain the side-input transforms, which changes the pipeline graph. Following the existing pattern (for example `View`), the side input is only added when `--updateCompatibilityVersion` is unset or at least `2.78.0`. Pipelines that need to update a running job from an older SDK can set it to an older version and keep the previous shape and behavior.
   
   ## Why this alleviates rather than fixes #27170
   
   The rate formula is unchanged. A small job that writes from the start still begins at `500 / hintMaxNumWorkers` writes per second and grows by 50% every 5 minutes, so with the default hint it is still slow for the first 10–15 minutes. This PR removes the extra slowdown from per-instance restarts and makes Firestore behave like Datastore. Changing the default growth is left for a follow-up.
   
   ## Testing
   
   - `RpcQosTest.rampUp_growsFromProvidedStart`: with a start 20 minutes before the first write, the first batch is sized from the grown budget (3), not the base budget (1).
   - `FirestoreV1FnBatchWriteWithSummaryTest.rampUpStartIsReadFromSideInput`: the DoFn reads the side input and passes it to `RpcQos.setRampUpStart`.
   - `FirestoreV1BatchWriteRampUpStartTest`: both write transforms add the side input to their `batchWrite` `ParDo`; with `updateCompatibilityVersion=2.77.0` they don't.
   
   ------------------------
   
   Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
   
    - [x] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
    - [x] Update `CHANGES.md` with noteworthy changes.
    - [ ] ~If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).~
   
   See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
   
   To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
   
   GitHub Actions Tests Status (on master branch)
   ------------------------------------------------------------------------------------------------
   [![Build python source distribution and wheels](https://github.com/apache/beam/actions/workflows/build_wheels.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
   [![Python tests](https://github.com/apache/beam/actions/workflows/python_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Java tests](https://github.com/apache/beam/actions/workflows/java_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
   [![Go tests](https://github.com/apache/beam/actions/workflows/go_tests.yml/badge.svg?event=schedule&&?branch=master)](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
   
   See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) for a list of workflows and how to trigger them.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-5961589498

   Assigning reviewers:
   
   R: @ahmedabu98 for label java.
   
   
   Note: If you would like to opt out of this review, comment `assign to next reviewer`.
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
   
   The PR bot will only process comments in the main thread (not review comments).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6097513678

   Reminder, please take a look at this pr: @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "nbali (via GitHub) via github" <gi...@beam.apache.org>.
nbali commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6952388420

   Note: #27170 was closed by accident when #40390 merged: a heading in the PR description ("…rather than fixes #27170") was picked up as a closing keyword. These PRs only alleviate the issue, they don't fix it, so it shouldn't have been closed. Could a maintainer reopen it? The descriptions are updated now, so this won't happen again.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on code in PR #40391:
URL: https://github.com/apache/beam/pull/40391#discussion_r4350203937


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreV1.java:
##########
@@ -230,6 +235,29 @@ public final class FirestoreV1 {
 
   private FirestoreV1() {}
 
+  /**
+   * Creates a side input holding a single pipeline-wide instant, from which the write ramp-up
+   * budget of every worker grows, rather than each worker starting its own ramp-up from its first
+   * write. Follows the approach of {@code DatastoreV1.Mutate}.
+   *
+   * <p>Returns {@code null} when {@code updateCompatibilityVersion} requests a pipeline shape from
+   * before this side input was introduced.
+   */
+  private static @Nullable PCollectionView<Instant> createRampUpStartView(
+      PCollection<?> input, JodaClock clock) {
+    if (StreamingOptions.updateCompatibilityVersionLessThan(

Review Comment:
   The way we should use this flag is that we should clone the prior `expand` method to `expand_2_77(...)`.
   
    - The `expand_2_77(...)` method will always have an unchanged topology.
    - The new `expand` delegates to `expand_2_77` in its first line(s) and after that has the new, totally separate, topology.
    - This eliminate the "action at a distance" of this return value being null causing the graph topology to be altered latter.
   
   Check other uses of this pipeline option for examples. Particularly a simple case I can think of is `Reshuffle`.
   
   They can share sub-transforms or helper functions, but the forking based on update compatibility should happen at the highest level. This is a case where redundancy is a benefit, and pushing the cyclomatic complexity to the earliest possible point is the way to do it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "github-actions[bot] (via GitHub) via github" <gi...@beam.apache.org>.
github-actions[bot] commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6630430116

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @Abacn for label java.
   Backup reviewer: chamikaramj (if the main reviewer is unresponsive, you can escalate to them by tagging their username)
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6522765192

   Reminder, please take a look at this pr: @kennknowles @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "github-actions[bot] (via GitHub) via github" <gi...@beam.apache.org>.
github-actions[bot] commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6795647563

   Reminder, please take a look at this pr: @Abacn @ahmedabu98 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6215561913

   Assigning new set of reviewers because Pr has gone too long without review. If you would like to opt out of this review, comment `assign to next reviewer`:
   
   R: @kennknowles for label java.
   Backup reviewer: damccorm (if the main reviewer is unresponsive, you can escalate to them by tagging their username)
   
   Available commands:
   - `stop reviewer notifications` - opt out of the automated review tooling
   - `remind me after tests pass` - tag the comment author after tests pass
   - `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "nbali (via GitHub) via github" <gi...@beam.apache.org>.
nbali commented on code in PR #40391:
URL: https://github.com/apache/beam/pull/40391#discussion_r4434413668


##########
sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreV1.java:
##########
@@ -230,6 +235,29 @@ public final class FirestoreV1 {
 
   private FirestoreV1() {}
 
+  /**
+   * Creates a side input holding a single pipeline-wide instant, from which the write ramp-up
+   * budget of every worker grows, rather than each worker starting its own ramp-up from its first
+   * write. Follows the approach of {@code DatastoreV1.Mutate}.
+   *
+   * <p>Returns {@code null} when {@code updateCompatibilityVersion} requests a pipeline shape from
+   * before this side input was introduced.
+   */
+  private static @Nullable PCollectionView<Instant> createRampUpStartView(
+      PCollection<?> input, JodaClock clock) {
+    if (StreamingOptions.updateCompatibilityVersionLessThan(

Review Comment:
   Thanks, done in 53a7ca9. Each batch write transform keeps the previous expansion unchanged in `expand_2_78_0`, and `expand` delegates to it in its first lines when `updateCompatibilityVersion` is 2.78.0 or earlier, as `Reshuffle` does. The new path no longer has a nullable side input.
   
   I named it `_2_78_0` rather than `_2_77` because 2.78.0 was cut without this change, so the new shape starts with 2.79.0. I also merged master, which now includes #40390.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] [Java] FirestoreIO: ramp up writes from a pipeline-wide start time [beam]

Posted by "nbali (via GitHub) via github" <gi...@beam.apache.org>.
nbali commented on PR #40391:
URL: https://github.com/apache/beam/pull/40391#issuecomment-6557870954

   The failing job seems to be a flaky one, unrelated to this PR: it's `beam_PreCommit_Java_GCP_IO_Direct_BigQuery`, which only runs the BigQuery integration tests. This PR only touches the Firestore package, and the same job failed on master on Oct 30 (https://github.com/apache/beam/actions/runs/41065748867).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: github-unsubscribe@beam.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org