You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "nbali (via GitHub)" <gi...@apache.org> on 2023/06/19 20:24:15 UTC
[GitHub] [beam] nbali opened a new issue, #27170: [Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput
nbali opened a new issue, #27170:
URL: https://github.com/apache/beam/issues/27170
### What happened?
Firestore has an 500 write/sec limitation, the java implementation splits up that quota among workers. Although it has a write rampup, the throughput is being influenced by the intial value a lot. The problem is that the default split is 1/500. So smaller jobs with just a few workers, start with 1/s and ramp up very slowly. If the job isn't sufficiently big enough, the throughtput stays low through the whole execution.
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L119-L120
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosOptions.java#L258-L275
https://github.com/apache/beam/blob/69cd6ef06b377050a1289df011c5b76bb8b5c2d9/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L594-L680
IMO the 500 as the default value should be decreased to a sensible much smaller one.
### Issue Priority
Priority: 2 (default / most bugs should be filed as P2)
### Issue Components
- [ ] Component: Python SDK
- [X] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [X] Component: IO connector
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #27170:
URL: https://github.com/apache/beam/issues/27170#issuecomment-5952745367
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput [beam]
Posted by "nbali (via GitHub)" <gi...@apache.org>.
nbali commented on issue #27170:
URL: https://github.com/apache/beam/issues/27170#issuecomment-5961038697
Two PRs that alleviate this (they don't fix it): both copy existing DatastoreIO behavior into FirestoreIO.
- #40390 adds `RpcQosOptions.Builder.withRampupThrottlingDisabled()`, mirroring `DatastoreV1.Write.withRampupThrottlingDisabled()`. With it, writes no longer wait for the 500/50/5 ramp-up budget and batches are no longer capped by it. Error-based throttling and retry backoff still apply.
- #40391 makes the ramp-up grow from one pipeline-wide start time (a singleton side input), mirroring `DatastoreV1.Mutate` / `RampupThrottlingFn`, instead of from each DoFn instance's first write. DoFn instances created later (autoscaling, retried bundles) no longer start over at 1 write/s. It's skipped when `--updateCompatibilityVersion` is below 2.78.0.
Neither changes the default: without the opt-out or `withHintMaxNumWorkers`, a small job still starts at `500 / 500 = 1` write/s and grows 50% every 5 minutes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput [beam]
Posted by "nbali (via GitHub)" <gi...@apache.org>.
nbali commented on issue #27170:
URL: https://github.com/apache/beam/issues/27170#issuecomment-5958693312
Still valid on current `master` (15a7ce17e1fb22897ac24527ad7eaa37ab75f786). Compared to 69cd6ef06b377050a1289df011c5b76bb8b5c2d9, the code linked in the original report is unchanged apart from shifted line numbers and one cast (`Math.pow(1.5, growth)` → `Math.pow(1.5, (double) growth)`) that doesn't change behavior.
Updated permalinks:
https://github.com/apache/beam/blob/15a7ce17e1fb22897ac24527ad7eaa37ab75f786/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L122-L123
https://github.com/apache/beam/blob/15a7ce17e1fb22897ac24527ad7eaa37ab75f786/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosOptions.java#L258-L275
https://github.com/apache/beam/blob/15a7ce17e1fb22897ac24527ad7eaa37ab75f786/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/RpcQosImpl.java#L597-L683
https://github.com/apache/beam/blob/15a7ce17e1fb22897ac24527ad7eaa37ab75f786/sdks/java/io/google-cloud-platform/src/main/java/org/apache/beam/sdk/io/gcp/firestore/FirestoreV1WriteFn.java#L328-L360
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn closed issue #27170: [Bug]: The default config for FirestoreV1.write().batchWrite() on smaller java jobs has an abysmal throughput
URL: https://github.com/apache/beam/issues/27170
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org