You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "AtharvUrunkar (via GitHub)" <gi...@apache.org> on 2026/04/20 06:32:09 UTC
Re: [I] Python SDK harness logs from child processes missing correct portability_worker_id in Dataflow [beam]
AtharvUrunkar commented on issue #38214:
URL: https://github.com/apache/beam/issues/38214#issuecomment-4278369056
Hi, I’ve been looking into this issue and was able to reproduce the logging inconsistency with `portability_worker_id` for child process logs.
From what I understand, the root cause seems to be that a shared logger is used in `boot.go`, which causes the Fn logging stream to be initialized once and reused across workers, leading to incorrect worker attribution.
I tried a fix by creating a separate `tools.Logger` per worker and initializing the `BufferedLogger` with a worker-specific context using `grpcx.WriteWorkerID`. This ensures each worker establishes its own logging stream with the correct metadata.
Before opening a PR, I wanted to confirm:
* Is this the expected approach for handling worker-scoped logging in the Python SDK harness?
* Are there any concerns with creating a separate logger per worker (e.g., performance or connection overhead)?
Happy to share the patch if this direction looks correct.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org