You are viewing a plain text version of this content. The canonical link for it is here.

Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2021/08/30 13:57:44 UTC

[GitHub] [incubator-doris] ccoffline opened a new issue #6529: [Proposal] Extend logging framework to support structured log output and custom log format

ccoffline opened a new issue #6529:
URL: https://github.com/apache/incubator-doris/issues/6529

**Is your feature request related to a problem? Please describe.**
Observability is a very important capability for a distributed system. We have encountered several problems many times while maintaining Doris, such as:
* Difficult to troubleshoot through log. Take a load transaction for example, we need to access into multiple machines and searching by multiple signatures (Label/TransactionID/QueryID), which cost a lot of time. Some logs don't even have any signatures, so we just have to guess which transaction it belongs to.
* Logs are not structured, which makes analyzing logs difficult, many is impossible.
* Our company has a centralized log collection system, which have custom log format. Even if some logs have a weak formatted output, it cannot match our custom log format. We have to custom logging for the most important processes which are query and stream load.

**Describe the solution you'd like**
We are considering a generic logging framework extension to support structured logging in Doris, and support different Doris maintainers to configure their own structured logging output format.

```java
// unstructured logging, output 'here is an info for a query, queryId=xxx'
LOG.info("here is an info for a query, queryId={}", queryId);
// structured logging, output custom log format, like 'here is an info for a query {"queryId":"xxx"}'
LOG.tag("query_id", queryId).info("here is an info for a query");
```

This allows maintainers to collect logs and transfer anyhow they want. In our case, we will collect logs into our log center and transfer them into relational records, so we can process or analyze logs in a table, maybe many tables.
This can happen without custom logging statements. Contributors can focus on adding useful information to logs and cleaning up useless ones.
Doris may need to set up specifications for tag names, like CamelCase or underline_style, or provide common tag methods and let maintainers customize their own tag names. This is open for discussion

**Describe alternatives you've considered**
We have considered import some observability framework such as `OpenTelemetry`. The current situation is that OpenTelemetry is still exploring many capabilities. For example, It doesn't support thrift in the official distribution; The cpp implementation is in pre-alpha; The logging integration is immature...
We can extend the logging capabilities to support flexible monitoring and analysis for Doris clusters' maintainers. At the same time, we can introduce `OpenTelemetry` to collect trace and metric data for telemetry, which does not conflict with log extension. Perhaps when `OpenTelemetry` is capable enough for logging, we can clean up useless logs then.

--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org