You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@iceberg.apache.org by "liurenjie1024 (via GitHub)" <gi...@apache.org> on 2024/07/26 09:23:20 UTC
[I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
liurenjie1024 opened a new issue, #482:
URL: https://github.com/apache/iceberg-rust/issues/482
As a library for production usage, it would be useful to add logging, metrics, and tracing support. We need to do some investigation about to fit into rust ecosystem as much as possible.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2760683598
I've looked into `fastrace` a bit more after our conversations about it yesterday and I must admit I have serious reservations about it.
The benchmarks as reported by `fastrace` have been pointed out by `tracing` maintainers as being somewhat misleading as they are not comparing like-for-like. Whilst `fastrace` does still come out faster, it is nowhere near the difference claimed when a like-for-like comparison is made.
Reference: https://www.reddit.com/r/rust/comments/1jh2fzg/fastrace_a_modern_approach_to_distributed_tracing/
In the blog post, the author makes claims about ecosystem fragmentation. I don't see how this is solved by introducing another tracing library. `tracing` is able to integrate with `log` already simply by enabling the `log` feature of tracing within our library. This is something that would be straightforward for consumers to do, and is also pretty common within the ecosystem already.
So, someone who already has an application that is using `tracing` for tracing and logging is able to record logs from libraries that use `tracing` or `log`. But, they would have no recourse if they wanted to capture logs from a library using `fastrace` because there is no similar means of exporting `fastrace` logs / traces to `tracing`. This seems to make fragmentation worse, not better.
The blog post claims that:
> With tokio-rs/tracing, there’s no clean way to have tracing zero-cost disabled. This makes library authors reluctant to add instrumentation to performance-sensitive code paths.
But it is trivial to do this in `tracing` too: https://docs.rs/tracing/latest/tracing/level_filters/index.html#compile-time-filters
On context propagation, the blog post also claims that:
> tokio-rs/tracing leaves this largely as an exercise for the developer.
The blogpost seems to be deliberately misleading on this. It is pretty trivial to do this in the tracing ecosystem. In a real-life service you can do this trivially in one line of code by using libraries like [tonic-tracing-opentelemetry](https://docs.rs/tonic-tracing-opentelemetry/latest/tonic_tracing_opentelemetry/index.html) for gRPC services in Tonic or [axum-tracing-opentelemetry](https://docs.rs/axum-tracing-opentelemetry/latest/axum_tracing_opentelemetry/) for HTTP services in Axum. In fact this is the exact same approach that fastrace has also used, with the fastrace-tonic and fastrace-axum crates being separate.
On the whole, the only claim in the blog that really holds up to scrutiny is that `fastrace` performs better. But even that has been exaggerated, can be mitigated, and in my real-world experience with using `tracing` with `iceberg-rust` already, is not significant enough to notice.
Conversely, `tracing` has:
* A much wider ecosystem (see https://docs.rs/tracing/latest/tracing/#related-crates)
* much greater configurability
* 10x as many GitHub stars
* 50x as many daily downloads (it gets downloaded more times in one day as fastrace has in its entire history).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "twuebi (via GitHub)" <gi...@apache.org>.
twuebi commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2255302470
Hi all,
I agree with @sdd and suggest to stick with tracing. It is the default choice for logging in async applications and has the most wide-spread adoption.
Regarding performance arguments, I doubt that this matters in any real world benchmark of this crate, any tracing incurred overhead will be dwarfed by the network latencies of whatever storage cloud service serves the data files / roundtrips to DB catalog / API calls to a Rest catalog. I'm strongly in favor with sticking with a tried and tested implementation and put some real world benchmarks in place and only if those show tracing to be a bottleneck consider choosing a less active alternative.
I also do not agree with the sentiment that it is a problem to address tracing & logging in the same crate. It rather provides very useful debugging utilities, e.g. when used in a server, some middleware can create a span containing a request id, now every (tracing-)log statement within that span will have the request id attached to it so your favorite log aggregator can filter by this request id to get all logs pertaining to it which is incredibly helpful when dealing with production issues.
Regarding metrics, I'm not sure I understand the issue, there exists https://crates.io/crates/metrics-prometheus which provides integration of prometheus with metrics.rs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2261509816
It sounds like `tracing` is the preferred option for tracing and logging. I'm happy to raise a PR to add this if we are all in agreement.
Regarding telemetry, does anyone have objections to the choice of `opentelemetry`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo closed issue #482: Investigation about tracing, logging, and metrics support.
URL: https://github.com/apache/iceberg-rust/issues/482
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2253320658
Personally I'd support `tracing` crate over `logging`. Adoption of tracing has supplanted that of logging over the past few years. It is more capable and flexible than logging. I disagree with the sentiment that it confuses things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "c-thiel (via GitHub)" <gi...@apache.org>.
c-thiel commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2257887394
Hey guys, some thoughts from my side:
- I don't think performance matters much - we are talking nanoseconds while roundtrips to S3 and catalogs are Milliseconds. Thus, the wider adoption is a clear plus for me for `tracing`.
- We are an io-heavy crate - lots of roundtrips via OpenDAL to Object Stores, Catalogs and more. IMO we would benefit alot from the `tracing` features over plain logging. It would enable us to consistently track operations across all affected systems - including forwarding of trace-ids to them. Iceberg-Rust is one component in an inherently distributed system that we call a Lakehouse. Distributed systems can best be monitored as a whole if traces are used everywhere along the way.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2746862981
There is a post for why I prefer `fastrace` + `log`: https://fast.github.io/blog/fastrace-a-modern-approach-to-distributed-tracing-in-rust/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2262103009
> It sounds like `tracing` is the preferred option for tracing and logging. I'm happy to raise a PR to add this if we are all in agreement.
I'm fine with this. Also cc @liurenjie1024 for ideas.
> Regarding telemetry, does anyone have objections to the choice of `opentelemetry`?
`opentelemetry`'s `metrics` support is currently in the `Alpha` stage. I believe most people primarily use it for `trace`. See: https://opentelemetry.io/docs/languages/rust/
I will support to use `metrics` instead if we don't want to depends on lib like `prometheus_client` directly.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2257907429
Thank you all for joining the discussion. It seems most people prefer using `tracing` for `trace` and `log`. I'm willing to accept this since it's more important for us to move forward.
So the left part is `metrics`.
What do you think about `prometheus_client` and `metrics`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2259129551
Some additional context. Renjie mentioned:
> Cons:
> It's mixing logging with tracing. If we use tracing for logging, our logging compability will not be compatible with logger system.
This is not quite right. You can use the [tracing_log](https://crates.io/crates/tracing-log) crate to redirect any other crates logs that went to `log` to `tracing`:
```rust
use tracing_log::LogTracer;
// redirects all output sent to log:* to their tracing::* equivalent
LogTracer::init().expect("Failed to set logger");
```
Regarding metrics, my preference would be either `metrics` or `opentelemetry`. Both are vendor-agnostic with pluggable exporters. My first choice would be opentelemetry as it seems to have better cross-language recognition and a bit more active / vibrant community (I'm basing this on github stars, watchers, number of contributors and commit / merge frequency)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2264300495
> It sounds like tracing is the preferred option for tracing and logging. I'm happy to raise a PR to add this if we are all in agreement.
+1.
> I will support to use metrics instead if we don't want to depends on lib like prometheus_client directly.
+1. I would also prefer to use `metrics` since `opentelemetry` is in too early stage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2760723897
Hi @sdd, I had a discussion with the fastrace developers and found that `tracing` is unable to collect the spans generated by `fastrace`.
Therefore, I agree that we should use `tracing` instead of `log` and `fastrace`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "ZENOTME (via GitHub)" <gi...@apache.org>.
ZENOTME commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2267510308
> Some additional context. Renjie mentioned:
>
> > Cons:
> > It's mixing logging with tracing. If we use tracing for logging, our logging compability will not be compatible with logger system.
>
> This is not quite right. You can use the [tracing_log](https://crates.io/crates/tracing-log) crate to redirect any other crates logs that went to `log` to `tracing`:
>
> ```rust
> use tracing_log::LogTracer;
>
> // redirects all output sent to log:* to their tracing::* equivalent
> LogTracer::init().expect("Failed to set logger");
> ```
>
> Regarding metrics, my preference would be either `metrics` or `opentelemetry`. Both are vendor-agnostic with pluggable exporters. My first choice would be opentelemetry as it seems to have better cross-language recognition and a bit more active / vibrant community (I'm basing this on github stars, watchers, number of contributors and commit / merge frequency)
I have a question: the [LogTracer](https://docs.rs/tracing-log/latest/tracing_log/struct.LogTracer.html) in tracing_log is used to convert the log record into a trace event. Does this mean that if the user uses log originally and after they introduce iceberg-rust, they have two choices:
1. maintain trace and log separately
2. use trace and redirect all original log records into trace events.
For users, is it a more ideal behavior to redirect the trace to the original log? 🤔
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2252378594
I will suggestion the following set:
- logging: [`log`](https://docs.rs/log)
- metrics: [`prometheus_client`](https://docs.rs/prometheus-client)
- tracing: [`fastrace`](https://docs.rs/fastrace)
---
## Why not `tracing`?
`Tracing` is widely used, but it often confuses the scope of logging and tracing.
From a tracing perspective, it's 100 times slower than `fastrace`.
Meanwhile, `log` offers the most compatible solution for all logging systems, making integration easier for users.
## Why not `prometheus`?
[`prometheus`](https://github.com/tikv/rust-prometheus) project maintained by tikv community is almost dead.
## Why not `metrics`?
It makes sense to introduce an abstraction based on `metrics` tailored to my own usage. However, its abstraction does not effectively support `prometheus`, which is the most widely used metrics collector.
I suggest we first integrate directly with Prometheus and consider `metrics`, while also accommodating requests for support of other metrics tools like [`tcp`](https://github.com/metrics-rs/metrics/tree/main/metrics-exporter-tcp).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2761366274
Thanks everyone for the discussion, glad to see that we reached consensus on this part!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2760738987
So our final decision will be to use `tracing` along with `metrics-rs`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "Xuanwo (via GitHub)" <gi...@apache.org>.
Xuanwo commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2253782960
> Adoption of tracing has supplanted that of logging over the past few years.
Users who rely on `tracing` can still integrate with our `log`, as tracing has native integration. However, if we adopt `tracing`, then `log` users will not be able to integrate with us. Therefore, I believe libraries like Iceberg should use `log` instead of `tracing`. Nonetheless, I'm open to using `tracing_subscriber` in our tests for a better user experience.
Here is a decision matrix:
| Does it works? | iceberg use `log` | iceberg use `tracing` |
| - | - | - |
| user use `env_logger` (`log` ecosystem) | YES | YES |
| user use `tracing_subscriber`(`tracing` ecosystem) | YES | NO |
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "liurenjie1024 (via GitHub)" <gi...@apache.org>.
liurenjie1024 commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2254555556
Hi, @sdd
> It is more capable and flexible than logging.
Could you elaborate on this? Previously for logging capability I thought `tracing` crate is better since it provides structural logging, but [log](https://github.com/rust-lang/log) also provides structural logging now.
As with using `tracing` + `logging backends`, there is [tracing-log](https://github.com/tokio-rs/tracing/tree/master/tracing-log), but it seems it's not actively maintained.
I'm trying to summarize the proposed approaches here:
## `log` + `fastrace`
Pros:
* Each crate fouces on its own functionality only, that means they have most widely supported ecosystem, and maybe easier to enable logging and tracing separately?
* [`fastrace`](https://github.com/fastracelabs/fastrace) reports that it's much faster than `tracing`.
Cons:
* [`tracing`](https://github.com/tokio-rs/tracing/tree/master) seems to have better ecosystem if you look at their main page.
* It seems that `fastrace` is less popular than `tracing`, that means it maybe more difficult to integrate with other libraries? For example if someone wants to use `iceberg-rust` in an application in [`h2`](https://github.com/hyperium/h2), which relies on `tracing` library, that means the user needs to setup two collectors/subscribers?
## `tracing` only
Pros:
* Most widely used tracing library in rust with best integrations.
Cons:
* It's mixing logging with tracing. If we use `tracing` for logging, our logging compability will not be compatible with logger system.
* It's reported to be slower than `fastrace`.
So how about we use `log` + `tracing`? We stick to the rule that using `tracing` library for tracing only, and use `log` for logging?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-3090352063
Hi All - I've been distracted with other things for a while but I've recently found time to pick this back up and have good progress so far. See the description on https://github.com/apache/iceberg-rust/pull/1502 for some details.
Some open questions / tasks:
* Feedback on the granularity of the traces I've implemented at different log levels appreciated. I've been using similar levels of tracing in production now on my iceberg fork for a few months and it has proved very useful, and I've taken the lessons I've learned from that process to feed in to this greenfield implementation.
* A documented naming scheme for trace spans would be good to have. Ideally one that aligns with cross-language iceberg implementations
* Ditto for naming tracing events
* Ditto for metrics for Prometheus etc
* Documented guidelines / policy for what / where to instrument with spans / logs / metrics and at what level (ie info/debug/trace)
* My PR contains changes necessary to the integration test setup as well as the instrumentation itself. I'd appreciate feedback / critique on the approach. It might make sense to split that part out if there is any controversy
* Where is a good location to add documentation / examples for consumers of the library for how to integrate iceberg traces/metrics with common observability stacks
One final observation. There's a trade-off to be had - at one end of the spectrum, the tracing / metrics instrumentation is as light-touch on the codebase as possible, with the implementation prioritising simplicity and clarity of the primary business logic at the expense of less cohesive / coherent traces within the observability tooling. At the other end of this spectrum, the control flow and readability of the core logic is completely disregarded in order to generate the most accurate and readable traces.
I've trodden a path closer towards the latter with this PR. It is not easy in Rust to get traces that are the most useable to consumers when you are working with streams of streams with multiple async stages. Getting the traces to look right was my goal with this PR and the cost of that is some loss of readability. My rationale is that I'd estimate that many more people will look at traces than will read the codebase. But as ever this is a discussion point.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2760713301
@ZENOTME said:
> For users, is it a more ideal behavior to redirect the trace to the original log? 🤔
Yes, and this is simple to do with `tracing` by just enabling `tracing`'s `log` feature and not using a tracing-subscriber.
See https://docs.rs/tracing/0.1.41/tracing/#crate-feature-flags
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org
Re: [I] Investigation about tracing, logging, and metrics support. [iceberg-rust]
Posted by "sdd (via GitHub)" <gi...@apache.org>.
sdd commented on issue #482:
URL: https://github.com/apache/iceberg-rust/issues/482#issuecomment-2761034967
This sounds good to me! I'm happy to pick this work up, as well as putting together examples and documentation of how to use them in practice. It will be a massive gain for all of us to finally have these pieces in place 😊
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@iceberg.apache.org
For additional commands, e-mail: issues-help@iceberg.apache.org