You are viewing a plain text version of this content. The canonical link for it is here.
Posted to github@beam.apache.org by "Kludex (via GitHub)" <gi...@apache.org> on 2024/11/20 12:21:04 UTC
[I] [Feature Request]: OpenTelemetry Support [beam]
Kludex opened a new issue, #33176:
URL: https://github.com/apache/beam/issues/33176
### What would you like to happen?
This is more a question than a feature request.
I was wondering... Why I don't see any OpenTelemetry related issue on Apache Beam? Is it because the runners already provide observability about their jobs?
I'm more interested in the Python side for now, but if there's no reason, would it make sense to create a `opentelemetry-instrumentation-apache-beam` package?
### Issue Priority
Priority: 3 (nice-to-have improvement)
### Issue Components
- [X] Component: Python SDK
- [ ] Component: Java SDK
- [ ] Component: Go SDK
- [ ] Component: Typescript SDK
- [ ] Component: IO connector
- [ ] Component: Beam YAML
- [ ] Component: Beam examples
- [ ] Component: Beam playground
- [ ] Component: Beam katas
- [ ] Component: Website
- [ ] Component: Infrastructure
- [ ] Component: Spark Runner
- [ ] Component: Flink Runner
- [ ] Component: Samza Runner
- [ ] Component: Twister2 Runner
- [ ] Component: Hazelcast Jet Runner
- [ ] Component: Google Cloud Dataflow Runner
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2532028912
cc @Abacn PTAL.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2532076504
BTW, I checked in Dataflow.
I made it work, but not as smooth as I wanted - I can provide code in the next days... Mainly because the context, and the exporter configured don't survive the pickling.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2517598403
> This looks interesting. I am wondering how this can work with remote Runners like Dataflow or Flink.
I'm going to run with `Dataflow` soon.
I still didn't find the what I wanted... I want to get the place where the Map function runs, for example... But I think that gets pickled or something, and then I'm having a hard time to find the right location (I think).
<img width="1478" alt="Screenshot 2024-12-04 at 15 25 25" src="https://github.com/user-attachments/assets/e8b7fa7b-04c7-400b-a912-329a61b73f1e">
This is the code I use - it's not as clean as the snippets above:
```py
def Pipeline__init_subclass__(cls: Pipeline, **kwargs):
cls.apply = logfire.instrument("{transform}", extract_args=True)(cls.apply)
cls.run = logfire.instrument("running {cls}")(cls.run)
cls.visit = logfire.instrument()(cls.visit)
def PipelineRunner__init_subclass__(cls: PipelineRunner, **kwargs):
cls.run_pipeline = logfire.instrument()(cls.run_pipeline)
def DoFnInvoker__init_subclass__(cls: DoFnInvoker, **kwargs):
cls.invoke_process = logfire.instrument("{transform}")(cls.invoke_process)
def patch_beam_pipeline():
Pipeline.__init_subclass__ = classmethod(Pipeline__init_subclass__)
Pipeline.apply = logfire.instrument("{transform}", extract_args=True)(Pipeline.apply)
Pipeline.run = logfire.instrument("running {self}")(Pipeline.run)
Pipeline.visit = logfire.instrument()(Pipeline.visit)
PipelineRunner.__init_subclass__ = classmethod(PipelineRunner__init_subclass__)
PipelineRunner.run_pipeline = logfire.instrument()(PipelineRunner.run_pipeline)
FnApiRunner.run_stages = logfire.instrument("running stages from {self}")(FnApiRunner.run_stages)
SdkWorker.process_bundle = logfire.instrument("running process_bundle from {self}")(SdkWorker.process_bundle)
patch_beam_pipeline()
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3361071500
This issue has been closed due to lack of activity. If you think that is incorrect, you can reopen the issue at any time.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2516780840
@liferoad @Abacn I was able to do what I want by manually creating a wrapper on the `PTransformer`s.
```py
class AutoSpanDoFn(beam.DoFn):
def __init__(self, transform: beam.PTransform[InputT, OutputT]):
super().__init__()
self.transform = transform
def process(self, element: Any, *args, **kwargs):
label = self.transform.label if hasattr(self.transform, "label") else "UnnamedTransform"
with logfire.span(label):
yield element
class AutoSpanTransform(beam.PTransform[InputT, OutputT]):
def __init__(self, transform: beam.PTransform[InputT, OutputT]):
super().__init__()
self.transform = transform
def expand(self, input_or_inputs: InputT) -> OutputT:
return input_or_inputs | beam.ParDo(AutoSpanDoFn(self.transform)) | self.transform
```
And then, I can use it like this:
```py
with logfire.span("main"):
with Pipeline() as pipeline:
text = [
"To be, or not to be: that is the question: ",
"Whether 'tis nobler in the mind to suffer ",
"The slings and arrows of outrageous fortune, ",
"Or to take arms against a sea of troubles, ",
]
pipeline = (
pipeline
| "Create" >> beam.Create(text)
| "Split" >> AutoSpanTransform(beam.ParDo(Split()))
| "Filter" >> AutoSpanTransform(beam.Filter(lambda x: x != "the"))
| "Print" >> AutoSpanTransform(beam.Map(logfire_print))
)
```
You can see the full code here: https://github.com/Kludex/logfire-apache-beam/blob/main/main.py
You can run it with `python main.py`.
This is what I see in Logfire (the observability platform we are developing):
Now... I need to be able to do it automatically. I tried to do some patching, like this:
```py
_original_pipeline_apply = Pipeline.apply
def patched_pipeline_apply(self, transform, *args, **kwargs):
if (
isinstance(transform, beam.PTransform)
and not isinstance(transform, AutoSpanTransform)
and not getattr(transform, "_instrumented", False)
):
transform = AutoSpanTransform(transform) # AutoSpanTrasnform has a `_instrumented = True` in this code.
return _original_pipeline_apply(self, transform, *args, **kwargs)
Pipeline.apply = patched_pipeline_apply
```
But I keep getting recursive exception - I'm still debugging it. But... I would appreciate help in two things:
1. Would you accept a PR adding some kind of API for me to not need to do this? Maybe a callback on the `Pipeline()`?
2. On my solution above, do you have any ideas on how to do it cleaner? In case I need a faster solution.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2539241871
> Is it possible to redirect regular logs to logfire?
No, only OpenTelemetry data.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2538955701
Okay... Hi there again... π
Let me try to explain where I stand now, and I'm actually going to stop on my own, to ask advice.
I've been trying to make it work in Dataflow. This is the pipeline I have in hands (for testing purposes):
```py
class Split(beam.DoFn):
def process(self, element: str):
logfire.info(f"in split {element}")
return element.split(" ")
def logfire_print(element: str):
with logfire.span("Print"):
logfire.info(element)
logfire.info("{globals}", globals=globals())
logfire.info("{locals}", locals=locals())
logfire.info("{traceback}", traceback=traceback.format_stack())
pipeline_options = PipelineOptions()
with Pipeline(options=pipeline_options) as pipeline:
text = [
"To be, or not to be: that is the question: ",
"Whether 'tis nobler in the mind to suffer ",
"The slings and arrows of outrageous fortune, ",
"Or to take arms against a sea of troubles, ",
]
pipeline = (
pipeline
| "Create" >> beam.Create(text)
| "Split" >> beam.ParDo(Split())
| "Filter" >> beam.Filter(lambda x: x != "the")
| "Print" >> beam.Map(logfire_print)
)
```
What we want is to have a span that starts when we run the pipeline, i.e. on the machine that ran it:
```bash
uv run python main.py \
--region ... \
--runner DataflowRunner \
--project ... \
--temp_location ... \
--requirements_file ./requirements.txt \
--save_main_session
```
And we want to have a span being created when each step runs e.g. when `logfire_print` is called for each element, we want to have the `logfire_print` wrapped, and then any `logfire.info()` will be contained on this automatic span that was created.
The problem is... Unfortunately, I can't monkeypatch anything on the worker side, because we actually pickle the function to send it to the worker.
<img width="883" alt="Screenshot 2024-12-12 at 14 28 07" src="https://github.com/user-attachments/assets/11a9758d-930c-4ba8-b9e3-aba9e64b16d7" />
This is what I got βοΈ
I wanted `Print` to be inside the first big span...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2773127469
Our Google team hasn't actively worked on this yet.
cc @kennknowles
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3366190810
Major challenge is extending WindowedValue and it's encoding.
#34902 was prerequisite on Java to make it easier to modify WindowedValue
#36373 is foundation work to pass any kind of metadata acros stages, starting with dataflow runner v1
Once merged we will add open telemetry SDK (we already tried that here - #34544 and #35227), drain and CDC features and start v2 and other SDK to come later
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Abacn (via GitHub)" <gi...@apache.org>.
Abacn commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3365841921
I understand this is in progress, for example, OutputBuilder (#34902) is intended for element-wise tracing that needed for OpenTelemetry support
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3380137481
> I haven't gone into the Python SDK yet. I expect that transparently adding metadata to be propagated everywhere is going to be a lot harder, because it was a massive whole-SDK type-system-assisted refactor. It will have to happen to support transparently passing the otel context everywhere - maybe the work you did can help [@Kludex](https://github.com/Kludex) ?
I was not very familiar with the tool, but what my experiment proved (at least for me) is that we need hooks, or do it internally.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "kennknowles (via GitHub)" <gi...@apache.org>.
kennknowles commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3377755610
I haven't gone into the Python SDK yet. I expect that transparently adding metadata to be propagated everywhere is going to be a lot harder, because it was a massive whole-SDK type-system-assisted refactor. It will have to happen to support transparently passing the otel context everywhere - maybe the work you did can help @Kludex ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "natanascimento (via GitHub)" <gi...@apache.org>.
natanascimento commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2772871512
any news about this topic @liferoad and @Kludex? I'm really interesting in this topic, because i'm looking how can I use open telemetry on abeam now ...
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2516887972
I got something nice running. Although not there yet...
My understanding about this package is growing a bit more... π
```py
def patch_beam_pipeline():
Pipeline.apply = logfire.instrument("{transform}", extract_args=True)(Pipeline.apply)
Pipeline.run = logfire.instrument()(Pipeline.run)
patch_beam_pipeline()
```
With the lines above, I'm able to patch `run` and `apply` - I thought the transformers would run in the `apply`, so that's what I was trying to patch, but it looks like the important bit is the `run`. Although I see how nice having both instrumented gets:
<img width="1480" alt="Screenshot 2024-12-04 at 11 26 23" src="https://github.com/user-attachments/assets/d5e8b662-d352-49ca-bf0d-df9e515b224a">
Now I need to find where the `PTransformer`s logic is executed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3361085583
That's a bit rude.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3361185199
> That's a bit rude.
We can always reopen this. :)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2488687966
This is a good idea. We do have a plan to investigate this idea. cc @Abacn
Meanwhile, you are welcome to work on this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "Kludex (via GitHub)" <gi...@apache.org>.
Kludex commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2517310515
This is becoming a great exercise to profile `apache-beam` itself π
Now I know how things run...
<img width="1482" alt="Screenshot 2024-12-04 at 14 09 02" src="https://github.com/user-attachments/assets/d0a99b9d-df38-465f-b7e7-683a11ab85f0">
I've enabled `logfire.install_auto_tracing("apache_beam", min_duration=0.00001)` to see the above.
I'm now trying to find out the best places to add spans to get execution time. I know already that `run_pipeline` and `run_stages` from `PipelineRunner` are good places.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2517551768
This looks interesting. I am wondering how this can work with remote Runners like Dataflow or Flink.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "liferoad (via GitHub)" <gi...@apache.org>.
liferoad commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2539237544
Is it possible to redirect regular logs to logfire?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "stankiewicz (via GitHub)" <gi...@apache.org>.
stankiewicz commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-2776264618
hi, we started looking into it, starting with java's per element streaming processing and several IOs that are good candidates for tracing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] commented on issue #33176:
URL: https://github.com/apache/beam/issues/33176#issuecomment-3242235117
This issue has been marked as stale due to 150 days of inactivity. It will be closed in 30 days if no further activity occurs. If you think thatβs incorrect or this issue still needs to be addressed, please simply write any comment. If closed, you can reopen the issue at any time. Thank you for your contributions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [I] [Feature Request]: OpenTelemetry Support [beam]
Posted by "github-actions[bot] (via GitHub)" <gi...@apache.org>.
github-actions[bot] closed issue #33176: [Feature Request]: OpenTelemetry Support
URL: https://github.com/apache/beam/issues/33176
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: github-unsubscribe@beam.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org