You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "lostluck (via GitHub)" <gi...@apache.org> on 2023/03/29 17:26:32 UTC
[GitHub] [beam] lostluck opened a new pull request, #26031: [#26030] Add PTransformID to logged messages & add message batching & immeadiate flushing.
lostluck opened a new pull request, #26031:
URL: https://github.com/apache/beam/pull/26031
Re-uses the metrics package's context work to extract the PTransformID of the logged message. This allows for attribution of logged messages to specific transforms for filtering and searching purposes.
Also adds message batching to reduce round trips to the logging service, but only up to a current cap of 64 messages at a time.
To avoid delays, fatal/critical logs are immediately flushed. However, if the logger is "caught up", then a batch is sent anyway.
Refactored the log handler slightly to be easier to test, and added a unit test for batching and flushing cases.
------------------------
Thank you for your contribution! Follow this checklist to help us incorporate your contribution quickly and easily:
- [ ] Mention the appropriate issue in your description (for example: `addresses #123`), if applicable. This will automatically add a link to the pull request in the issue. If you would like the issue to automatically close on merging the pull request, comment `fixes #<ISSUE NUMBER>` instead.
- [ ] Update `CHANGES.md` with noteworthy changes.
- [ ] If this contribution is large, please file an Apache [Individual Contributor License Agreement](https://www.apache.org/licenses/icla.pdf).
See the [Contributor Guide](https://beam.apache.org/contribute) for more tips on [how to make review process smoother](https://beam.apache.org/contribute/get-started-contributing/#make-the-reviewers-job-easier).
To check the build health, please visit [https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md](https://github.com/apache/beam/blob/master/.test-infra/BUILD_STATUS.md)
GitHub Actions Tests Status (on master branch)
------------------------------------------------------------------------------------------------
[](https://github.com/apache/beam/actions?query=workflow%3A%22Build+python+source+distribution+and+wheels%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Python+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Java+Tests%22+branch%3Amaster+event%3Aschedule)
[](https://github.com/apache/beam/actions?query=workflow%3A%22Go+tests%22+branch%3Amaster+event%3Aschedule)
See [CI.md](https://github.com/apache/beam/blob/master/CI.md) for more information about GitHub Actions CI.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] github-actions[bot] commented on pull request #26031: [#26030] Add PTransformID to logged messages & add message batching & immeadiate flushing.
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #26031:
URL: https://github.com/apache/beam/pull/26031#issuecomment-1489016466
Stopping reviewer notifications for this pull request: review requested by someone other than the bot, ceding control
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] codecov[bot] commented on pull request #26031: [#26030] Add PTransformID to logged messages & add message batching & immeadiate flushing.
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #26031:
URL: https://github.com/apache/beam/pull/26031#issuecomment-1489024601
## [Codecov](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) Report
> Merging [#26031](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (451529f) into [master](https://codecov.io/gh/apache/beam/commit/5bdc4527a4702a2de3c423be172b92df0e0c13a0?el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) (5bdc452) will **increase** coverage by `0.01%`.
> The diff coverage is `n/a`.
```diff
@@ Coverage Diff @@
## master #26031 +/- ##
==========================================
+ Coverage 71.34% 71.36% +0.01%
==========================================
Files 779 779
Lines 102767 102726 -41
==========================================
- Hits 73319 73308 -11
+ Misses 27976 27947 -29
+ Partials 1472 1471 -1
```
| [Impacted Files](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation) | Coverage Δ | |
|---|---|---|
| [sdks/go/pkg/beam/core/metrics/metrics.go](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL21ldHJpY3MvbWV0cmljcy5nbw==) | `60.54% <ø> (ø)` | |
| [sdks/go/pkg/beam/core/runtime/harness/datamgr.go](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9kYXRhbWdyLmdv) | `56.91% <ø> (-0.17%)` | :arrow_down: |
| [sdks/go/pkg/beam/core/runtime/harness/harness.go](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9oYXJuZXNzLmdv) | `11.31% <ø> (+0.28%)` | :arrow_up: |
| [sdks/go/pkg/beam/core/runtime/harness/logging.go](https://codecov.io/gh/apache/beam/pull/26031?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation#diff-c2Rrcy9nby9wa2cvYmVhbS9jb3JlL3J1bnRpbWUvaGFybmVzcy9sb2dnaW5nLmdv) | `23.75% <ø> (-2.22%)` | :arrow_down: |
:mega: We’re building smart automated test selection to slash your CI/CD build times. [Learn more](https://about.codecov.io/iterative-testing/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=The+Apache+Software+Foundation)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck commented on pull request #26031: [#26030] Add PTransformID to logged messages & add message batching & immeadiate flushing.
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck commented on PR #26031:
URL: https://github.com/apache/beam/pull/26031#issuecomment-1489015148
R: @jrmccluskey
cc: @johannaojeling (Since you might be interested in this having been fixed).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [beam] lostluck merged pull request #26031: [#26030] Add PTransformID to logged messages & add message batching & immeadiate flushing.
Posted by "lostluck (via GitHub)" <gi...@apache.org>.
lostluck merged PR #26031:
URL: https://github.com/apache/beam/pull/26031
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org