You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by GitBox <gi...@apache.org> on 2022/06/23 16:06:03 UTC
[GitHub] [beam] robertwb opened a new pull request, #22020: Removes examples of unscalable sinks from documentation.
robertwb opened a new pull request, #22020:
URL: https://github.com/apache/beam/pull/22020
These sinks would only work on small data, and do not properly
handle failures. Users should instead be encouraged to use the
builtin IOs or write their own following best practices documented
in the IO guides.
Incorporated some of the relevant information into the IO page.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] [**Choose reviewer(s)**](https://beam.apache.org/contribute/#make-your-change) and mention them in a comment (`R: @username`).
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/#make-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on a diff in pull request #22020: Removes examples of unscalable sinks from documentation.
Posted by GitBox <gi...@apache.org>.
robertwb commented on code in PR #22020:
URL: https://github.com/apache/beam/pull/22020#discussion_r906168373
##########
website/www/site/content/en/documentation/io/developing-io-overview.md:
##########
@@ -180,9 +180,17 @@ To create a Beam sink, we recommend that you use a `ParDo` that writes the
received records to the data store. To develop more complex sinks (for example,
to support data de-duplication when failures are retried by a runner), use
`ParDo`, `GroupByKey`, and other available Beam transforms.
+Many data services are optimized to write batches of elements at a time,
+so it may make sense to group the elements into batches before writing.
+Persistant connectons can be initialized in a DoFn's `setUp` or `startBundle`
+method rather than upon the receipt of every element as well.
Review Comment:
I don't think any sinks to date have found a use/need for shared, but I suppose it's possible. I don't think it's common/specific enough to mention here though.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb commented on pull request #22020: Removes examples of unscalable sinks from documentation.
Posted by GitBox <gi...@apache.org>.
robertwb commented on PR #22020:
URL: https://github.com/apache/beam/pull/22020#issuecomment-1164603383
I found these while looking for incorrect usage of PDone (as is the case here).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on pull request #22020: Removes examples of unscalable sinks from documentation.
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on PR #22020:
URL: https://github.com/apache/beam/pull/22020#issuecomment-1165641196
cc: @davidcavazos
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] tvalentyn commented on a diff in pull request #22020: Removes examples of unscalable sinks from documentation.
Posted by GitBox <gi...@apache.org>.
tvalentyn commented on code in PR #22020:
URL: https://github.com/apache/beam/pull/22020#discussion_r906128814
##########
website/www/site/content/en/documentation/io/developing-io-overview.md:
##########
@@ -180,9 +180,17 @@ To create a Beam sink, we recommend that you use a `ParDo` that writes the
received records to the data store. To develop more complex sinks (for example,
to support data de-duplication when failures are retried by a runner), use
`ParDo`, `GroupByKey`, and other available Beam transforms.
+Many data services are optimized to write batches of elements at a time,
+so it may make sense to group the elements into batches before writing.
+Persistant connectons can be initialized in a DoFn's `setUp` or `startBundle`
+method rather than upon the receipt of every element as well.
Review Comment:
Would using [shared.py](https://github.com/apache/beam/blob/master/sdks/python/apache_beam/utils/shared.py) (which allows once-per-process initialization) be also applicable for sinks?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] robertwb merged pull request #22020: Removes examples of unscalable sinks from documentation.
Posted by GitBox <gi...@apache.org>.
robertwb merged PR #22020:
URL: https://github.com/apache/beam/pull/22020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org