You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "shunping (via GitHub)" <gi...@apache.org> on 2025/08/30 04:17:20 UTC
[PR] [Prism[ Fix two kinds of errors when using json log format. [beam]
shunping opened a new pull request, #36020:
URL: https://github.com/apache/beam/pull/36020
Two types of errors can be seen when using json log format:
```
INFO:apache_beam.utils.subprocess_server:{"time":"2025-08-29T21:58:03.309126-04:00","level":"DEBUG","source":{"function":"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism/internal.executePipeline","file":"/Users/shunping/Projects/beam-dev-python-3/sdks/go/pkg/beam/runners/prism/internal/execute.go","line":363},"msg":"pipeline done!","job":"job-001[job]","error":null,"topo":"!ERROR:json: unsupported type: func(io.Reader) []uint8"}
```
```
INFO:apache_beam.utils.subprocess_server:{"time":"2025-08-29T22:05:24.038412-04:00","level":"DEBUG","source":{"function":"github.com/apache/beam/sdks/v2/go/pkg/beam/runners/prism/internal.(*stage).Execute","file":"/Users/shunping/Projects/beam-dev-python-3/sdks/go/pkg/beam/runners/prism/internal/stage.go","line":287},"msg":"Execute: committing data","bundle":{"ID":"inst002","stage":"stage-005","watermark":"!ERROR:time.Time year outside of range [0,9999]""294247-01-10T04:00:54.775Z"},"outputsWithData":[],"outputs":[]}
```
This PR fixed both.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36020:
URL: https://github.com/apache/beam/pull/36020#discussion_r2312106177
##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -89,6 +89,15 @@ type PColInfo struct {
KeyDec func(io.Reader) []byte
}
+func (info PColInfo) LogValue() slog.Value {
+ return slog.GroupValue(
+ slog.String("GlobalID", info.GlobalID),
+ slog.String("WindowCoder", info.WindowCoder.String()),
+ slog.Any("WDec", info.WDec),
+ slog.Any("WEnc", info.WEnc),
+ )
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "gemini-code-assist[bot] (via GitHub)" <gi...@apache.org>.
gemini-code-assist[bot] commented on code in PR #36020:
URL: https://github.com/apache/beam/pull/36020#discussion_r2312071615
##########
sdks/go/pkg/beam/runners/prism/internal/execute.go:
##########
@@ -360,7 +360,11 @@ func executePipeline(ctx context.Context, wks map[string]*worker.W, j *jobservic
case rb, ok := <-bundles:
if !ok {
err := eg.Wait()
- j.Logger.Debug("pipeline done!", slog.String("job", j.String()), slog.Any("error", err), slog.Any("topo", topo))
+ var topoAttrs []slog.Attr
+ for _, s := range topo {
+ topoAttrs = append(topoAttrs, slog.Any(s.ID, s))
+ }
+ j.Logger.Debug("pipeline done!", slog.String("job", j.String()), slog.Any("error", err), slog.Any("topo", topoAttrs))
Review Comment:
Using `slog.Any("topo", topoAttrs)` will not produce the desired structured log for the topology. `topoAttrs` is a slice of `slog.Attr`, and `slog.Any` will attempt to log it as a single value, which is not what is intended and will likely result in incorrect JSON. To log a group of attributes, you should use `slog.Group`.
```suggestion
j.Logger.Debug("pipeline done!", slog.String("job", j.String()), slog.Any("error", err), slog.Group("topo", topoAttrs...))
```
##########
sdks/go/pkg/beam/runners/prism/internal/engine/elementmanager.go:
##########
@@ -89,6 +89,15 @@ type PColInfo struct {
KeyDec func(io.Reader) []byte
}
+func (info PColInfo) LogValue() slog.Value {
+ return slog.GroupValue(
+ slog.String("GlobalID", info.GlobalID),
+ slog.String("WindowCoder", info.WindowCoder.String()),
+ slog.Any("WDec", info.WDec),
+ slog.Any("WEnc", info.WEnc),
+ )
Review Comment:
The `WDec` and `WEnc` fields are function types. Attempting to log them with `slog.Any` will result in a JSON marshaling error, similar to the issue this pull request aims to fix. Function values cannot be serialized to JSON. These fields should be omitted from the log output.
```go
return slog.GroupValue(
slog.String("GlobalID", info.GlobalID),
slog.String("WindowCoder", info.WindowCoder.String()),
)
```
##########
sdks/go/pkg/beam/runners/prism/internal/stage.go:
##########
@@ -108,6 +108,19 @@ func clampTick(dur time.Duration) time.Duration {
}
}
+func (s *stage) LogValue() slog.Value {
+ var outAttrs []slog.Attr
+ for k, v := range s.OutputsToCoders {
+ outAttrs = append(outAttrs, slog.Any(k, v))
+ }
+ return slog.GroupValue(
+ slog.String("ID", s.ID),
+ slog.Any("transforms", s.transforms),
+ slog.Any("inputInfo", s.inputInfo),
+ slog.Any("outputInfo", outAttrs),
Review Comment:
Using `slog.Any("outputInfo", outAttrs)` will not produce the desired structured log for the `outputInfo` field. `outAttrs` is a slice of `slog.Attr`, and `slog.Any` will attempt to log it as a single value, which is not what is intended and will likely result in incorrect JSON. To log a group of attributes, you should use `slog.Group`.
```suggestion
slog.Group("outputInfo", outAttrs...),
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on code in PR #36020:
URL: https://github.com/apache/beam/pull/36020#discussion_r2312105954
##########
sdks/go/pkg/beam/runners/prism/internal/stage.go:
##########
@@ -108,6 +108,19 @@ func clampTick(dur time.Duration) time.Duration {
}
}
+func (s *stage) LogValue() slog.Value {
+ var outAttrs []slog.Attr
+ for k, v := range s.OutputsToCoders {
+ outAttrs = append(outAttrs, slog.Any(k, v))
+ }
+ return slog.GroupValue(
+ slog.String("ID", s.ID),
+ slog.Any("transforms", s.transforms),
+ slog.Any("inputInfo", s.inputInfo),
+ slog.Any("outputInfo", outAttrs),
Review Comment:
Done.
##########
sdks/go/pkg/beam/runners/prism/internal/execute.go:
##########
@@ -360,7 +360,11 @@ func executePipeline(ctx context.Context, wks map[string]*worker.W, j *jobservic
case rb, ok := <-bundles:
if !ok {
err := eg.Wait()
- j.Logger.Debug("pipeline done!", slog.String("job", j.String()), slog.Any("error", err), slog.Any("topo", topo))
+ var topoAttrs []slog.Attr
+ for _, s := range topo {
+ topoAttrs = append(topoAttrs, slog.Any(s.ID, s))
+ }
+ j.Logger.Debug("pipeline done!", slog.String("job", j.String()), slog.Any("error", err), slog.Any("topo", topoAttrs))
Review Comment:
Done.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping commented on PR #36020:
URL: https://github.com/apache/beam/pull/36020#issuecomment-3239493695
/gemini review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "codecov[bot] (via GitHub)" <gi...@apache.org>.
codecov[bot] commented on PR #36020:
URL: https://github.com/apache/beam/pull/36020#issuecomment-3239541549
## [Codecov](https://app.codecov.io/gh/apache/beam/pull/36020?dropdown=coverage&src=pr&el=h1&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) Report
:x: Patch coverage is `12.82051%` with `34 lines` in your changes missing coverage. Please review.
:white_check_mark: Project coverage is 56.76%. Comparing base ([`e72d9c7`](https://app.codecov.io/gh/apache/beam/commit/e72d9c7a41f071d68881631a944eae5f5797c947?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)) to head ([`b544c7e`](https://app.codecov.io/gh/apache/beam/commit/b544c7e3b7503382d865beb7bb3368eb271b27d4?dropdown=coverage&el=desc&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache)).
:warning: Report is 10 commits behind head on master.
| [Files with missing lines](https://app.codecov.io/gh/apache/beam/pull/36020?dropdown=coverage&src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Patch % | Lines |
|---|---|---|
| [...am/runners/prism/internal/engine/elementmanager.go](https://app.codecov.io/gh/apache/beam/pull/36020?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fengine%2Felementmanager.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL2VuZ2luZS9lbGVtZW50bWFuYWdlci5nbw==) | 0.00% | [17 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/36020?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [sdks/go/pkg/beam/runners/prism/internal/stage.go](https://app.codecov.io/gh/apache/beam/pull/36020?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fstage.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL3N0YWdlLmdv) | 0.00% | [11 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/36020?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
| [...go/pkg/beam/runners/prism/internal/environments.go](https://app.codecov.io/gh/apache/beam/pull/36020?src=pr&el=tree&filepath=sdks%2Fgo%2Fpkg%2Fbeam%2Frunners%2Fprism%2Finternal%2Fenvironments.go&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#diff-c2Rrcy9nby9wa2cvYmVhbS9ydW5uZXJzL3ByaXNtL2ludGVybmFsL2Vudmlyb25tZW50cy5nbw==) | 0.00% | [6 Missing :warning: ](https://app.codecov.io/gh/apache/beam/pull/36020?src=pr&el=tree&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) |
<details><summary>Additional details and impacted files</summary>
```diff
@@ Coverage Diff @@
## master #36020 +/- ##
============================================
- Coverage 56.77% 56.76% -0.01%
Complexity 3383 3383
============================================
Files 1220 1220
Lines 184968 184998 +30
Branches 3508 3508
============================================
+ Hits 105015 105016 +1
- Misses 76625 76658 +33
+ Partials 3328 3324 -4
```
| [Flag](https://app.codecov.io/gh/apache/beam/pull/36020/flags?src=pr&el=flags&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | Coverage Δ | |
|---|---|---|
| [go](https://app.codecov.io/gh/apache/beam/pull/36020/flags?src=pr&el=flag&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache) | `28.21% <12.82%> (-0.01%)` | :arrow_down: |
Flags with carried forward coverage won't be shown. [Click here](https://docs.codecov.io/docs/carryforward-flags?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache#carryforward-flags-in-the-pull-request-comment) to find out more.
</details>
[:umbrella: View full report in Codecov by Sentry](https://app.codecov.io/gh/apache/beam/pull/36020?dropdown=coverage&src=pr&el=continue&utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
:loudspeaker: Have feedback on the report? [Share it here](https://about.codecov.io/codecov-pr-comment-feedback/?utm_medium=referral&utm_source=github&utm_content=comment&utm_campaign=pr+comments&utm_term=apache).
<details><summary> :rocket: New features to boost your workflow: </summary>
- :snowflake: [Test Analytics](https://docs.codecov.com/docs/test-analytics): Detect flaky tests, report on failures, and find test suite problems.
- :package: [JS Bundle Analysis](https://docs.codecov.com/docs/javascript-bundle-analysis): Save yourself from yourself by tracking and limiting bundle sizes in JS merges.
</details>
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "shunping (via GitHub)" <gi...@apache.org>.
shunping merged PR #36020:
URL: https://github.com/apache/beam/pull/36020
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] [Prism] Fix two kinds of errors when using json log format. [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on PR #36020:
URL: https://github.com/apache/beam/pull/36020#issuecomment-3239605964
Assigning reviewers:
R: @jrmccluskey for label python.
R: @lostluck for label go.
Note: If you would like to opt out of this review, comment `assign to next reviewer`.
Available commands:
- `stop reviewer notifications` - opt out of the automated review tooling
- `remind me after tests pass` - tag the comment author after tests pass
- `waiting on author` - shift the attention set back to the author (any comment or push by the author will return the attention set to the reviewers)
The PR bot will only process comments in the main thread (not review comments).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org