You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "dariabezkorovaina (via GitHub)" <gi...@apache.org> on 2024/02/27 10:11:15 UTC
[PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
dariabezkorovaina opened a new pull request, #30435:
URL: https://github.com/apache/beam/pull/30435
- Adding 3 prompts for container environments, resource hints, batching for external calls
- Nits and fixes to the existing prompts to match Google Writing Style Guide
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://github.com/apache/beam/blob/master/CONTRIBUTING.md#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI or the [workflows README](https://github.com/apache/beam/blob/master/.github/workflows/README.md) to see a list of phrases to trigger workflows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "dariabezkorovaina (via GitHub)" <gi...@apache.org>.
dariabezkorovaina commented on PR #30435:
URL: https://github.com/apache/beam/pull/30435#issuecomment-1985829188
@damccorm, thanks for your feedback! Implemented the edit, please take a look:)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "dariabezkorovaina (via GitHub)" <gi...@apache.org>.
dariabezkorovaina commented on PR #30435:
URL: https://github.com/apache/beam/pull/30435#issuecomment-1983439679
Thanks, @olehborysevych! @damccorm, could you please review this PR when you have a chance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on code in PR #30435:
URL: https://github.com/apache/beam/pull/30435#discussion_r1515349817
##########
learning/prompts/documentation-lookup-nolinks/47_batching_for_external_calls.md:
##########
@@ -0,0 +1,70 @@
+Prompt:
+How to reduce payload when calling external services from my Apache Beam pipeline?
+
+Response:
+To reduce payload when calling external services from your Apache Beam pipeline, you can employ batching techniques by using the `GroupIntoBatches` transform. Batching involves aggregating multiple elements into a single payload, reducing the number of requests sent to the external service and minimizing overhead.
+
+Under the hood, the `GroupIntoBatches` transform utilizes state and timers to grant users precise control over batch size and buffering duration parameters, while abstracting away the implementation details. Key parameters include:
+* `maxBufferDuration`: controls the maximum waiting time for a batch to be emitted.
+* `batchSize`: determines the maximum number of elements in each batch. Elements are buffered until the specified number is reached, then emitted as a batch.
+* `batchSizeBytes` (Java only): limits the byte size of a single batch, determined by the input coder.
+* `elementByteSize` (Java only): sets the byte size of a single batch using a user-defined function.
+* `withShardedKey()`: enhances parallelism by distributing a single key across multiple threads.
Review Comment:
withSharedKey() should be separated from other parameters since it's not a parameter but a function @dariabezkorovaina
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #30435:
URL: https://github.com/apache/beam/pull/30435#issuecomment-1983467679
Assigning reviewers. If you would like to opt out of this review, comment `assign to next reviewer`:
R: @damondouglas added as fallback since no labels match configuration
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm merged PR #30435:
URL: https://github.com/apache/beam/pull/30435
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "damccorm (via GitHub)" <gi...@apache.org>.
damccorm commented on code in PR #30435:
URL: https://github.com/apache/beam/pull/30435#discussion_r1516361339
##########
learning/prompts/documentation-lookup-nolinks/47_batching_for_external_calls.md:
##########
@@ -0,0 +1,71 @@
+Prompt:
+How to reduce payload when calling external services from my Apache Beam pipeline?
Review Comment:
I think saying "How to reduce the number of external calls when calling external services from my Apache Beam pipeline?" might be more appropriate here (and using the same wording below).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "olehborysevych (via GitHub)" <gi...@apache.org>.
olehborysevych commented on code in PR #30435:
URL: https://github.com/apache/beam/pull/30435#discussion_r1507553765
##########
learning/prompts/documentation-lookup-nolinks/47_batching_for_external_calls.md:
##########
@@ -0,0 +1,83 @@
+Prompt: How to reduce payload when calling external services from my Apache Beam pipeline?
+
+Response:
+To reduce payload when calling external services from your Apache Beam pipeline, you can employ batching techniques by using the `GroupIntoBatches` transform. Batching involves aggregating multiple elements into a single payload, reducing the number of requests sent to the external service and minimizing overhead.
+
+Under the hood, the `GroupIntoBatches` transform utilizes state and timers to grant users precise control over batch size and buffering duration parameters, while abstracting away the implementation details. Key parameters include:
+* `maxBufferDuration`: controls the maximum waiting time for a batch to be emitted.
+* `batchSize`: determines the maximum number of elements in each batch. Elements are buffered until the specified number is reached, then emitted as a batch.
+* `batchSizeBytes` (Java only): limits the byte size of a single batch, determined by the input coder.
+* `elementByteSize` (Java only): sets the byte size of a single batch using a user-defined function.
+* `withShardedKey()`: enhances parallelism by distributing a single key across multiple threads.
+
+The following sample code snippets illustrate the use of `GroupIntoBatches` to batch elements in an Apache Beam pipeline in Java and Python.
+
+Java:
+
+```java
+public class GroupIntoBatchesExample {
Review Comment:
@dariabezkorovaina I would betterreplace this with snippets from documentation or example from https://beam.apache.org/releases/javadoc/current/org/apache/beam/sdk/transforms/GroupIntoBatches.html or https://beam.apache.org/releases/pydoc/current/apache_beam.transforms.util.html
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] Duet AI prompts: containers, hints, external calls (no links) [beam]
Posted by "dariabezkorovaina (via GitHub)" <gi...@apache.org>.
dariabezkorovaina commented on PR #30435:
URL: https://github.com/apache/beam/pull/30435#issuecomment-1971328965
@olehborysevych updated the PR, please take a look :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org