You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "cozos (via GitHub)" <gi...@apache.org> on 2026/04/16 11:26:31 UTC
[I] [Feature Request]: [beam]
cozos opened a new issue, #38214:
URL: https://github.com/apache/beam/issues/38214
### What would you like to happen?
Hey team we’re hitting a logging gap with native code (pybind / C++) in the Python SDK harness. It seems like `boot.go` launches a bunch of Python child processes, and then pipes its outputs stdout/stderr to `BufferedLogger` -> `FnLogging`. But because it never goes through the Python Fn logging stack, so we lose the same worker attribution you get from normal SDK logs - particularly `portability_worker_id`.
Concretely, our Python logs come with the metadata `jsonPayload.portability_worker_id=sdk-0-0_sibling_5`. But our pybind'd C++ code logs has `jsonPayload.portability_worker_id=sdk-0-0` (from `boot.go`), so we can't correlate it.
I wonder if easily fixable in `boot.go`? In https://github.com/apache/beam/blob/ae536b6f0f83b978854f1d6852fbacb62a3e5ec0/sdks/python/container/boot.go#L321, instead of:
```go
bufLogger := tools.NewBufferedLogger(logger)
```
We would do something like:
```go
workerCtx := grpcx.WriteWorkerID(context.Background(), workerId)
bufLogger := tools.NewBufferedLoggerWithFlushInterval(workerCtx, logger, 100*time.Millisecond)
```
I have no idea if on the Dataflow Unified Harness, will actually pick up the `workerId` on the Google side, which is why I'm asking.
### Issue Priority
Priority: 2 (default / most feature requests should be filed as P2)
### Issue Components
- [x] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [x] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "AtharvUrunkar (via GitHub)" <gi...@apache.org>.
AtharvUrunkar commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-4278369056
Hi, I’ve been looking into this issue and was able to reproduce the logging inconsistency with `portability_worker_id` for child process logs.
From what I understand, the root cause seems to be that a shared logger is used in `boot.go`, which causes the Fn logging stream to be initialized once and reused across workers, leading to incorrect worker attribution.
I tried a fix by creating a separate `tools.Logger` per worker and initializing the `BufferedLogger` with a worker-specific context using `grpcx.WriteWorkerID`. This ensures each worker establishes its own logging stream with the correct metadata.
Before opening a PR, I wanted to confirm:
* Is this the expected approach for handling worker-scoped logging in the Python SDK harness?
* Are there any concerns with creating a separate logger per worker (e.g., performance or connection overhead)?
Happy to share the patch if this direction looks correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-6523070134
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "cozos (via GitHub)" <gi...@apache.org>.
cozos commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-4285623383
Thanks so much for the PR @AtharvUrunkar - I left some comments on the PR based on my findings from my experiment
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "tvalentyn (via GitHub)" <gi...@apache.org>.
tvalentyn commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-4284207041
Thanks @cozos and @AtharvUrunkar , the approach to open a separate logger per process makes sense to me. I think we should be able to handle this well from performance standpoint.
> Are there any concerns with creating a separate logger per worker
On a high level, when we use a worker ID for log streams for a particular worker, and later filter out all logs by a given ID, will we be able to filter logs from both Python logs and stdout/stderr logs coming from extensions using the same worker ID?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-5890696725
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think that’s incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "cozos (via GitHub)" <gi...@apache.org>.
cozos commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-4285399174
Hi @tvalentyn I am not sure I fully understand your question. So just to clarify
## The current state:
In order to get Python logs, we need this filter in Google Logs Explorer. This will NOT include stdout/stderr logs from extensions.
```
logName="projects/cruise-mlp-prod-13d0/logs/dataflow.googleapis.com%2Fworker"
jsonPayload.worker = "${MY_DATAFLOW_JOBNAME}-04020129-nd2t-harness-svpn"
jsonPayload.portability_worker_id" = "sdk-0-0_sibling_5"
```
In order to get the stdout/stderr logs from extensions, you need this query:
```
logName="projects/cruise-mlp-prod-13d0/logs/dataflow.googleapis.com%2Fworker"
jsonPayload.worker = "${MY_DATAFLOW_JOBNAME}-04020129-nd2t-harness-svpn"
jsonPayload.portability_worker_id" = "sdk-0-0"
```
This has two downsides:
1. All of the extension stdout/stderr logs from ALL the sibling worker are mixed together, we don't know which sibling an individual log entry came from.
2. We can't correlate the Python and C++ extensions logs from any individual sibling worker
## The proposed state:
For this feature request, I am asking that this query would return BOTH Python logs and C++ extension logs, which would be achieved by having a separate logger with a separate `workerCtx` for each sibling worker.
```
logName="projects/cruise-mlp-prod-13d0/logs/dataflow.googleapis.com%2Fworker"
jsonPayload.worker = "${MY_DATAFLOW_JOBNAME}-04020129-nd2t-harness-svpn"
jsonPayload.portability_worker_id" = "sdk-0-0_sibling_5"
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #38214: Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow
URL: https://github.com/apache/beam/issues/38214
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org