You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/05/23 15:04:09 UTC
[GitHub] [druid] FrankChen021 opened a new issue #9917: record sql executed on meta storage to log file
FrankChen021 opened a new issue #9917:
URL: https://github.com/apache/druid/issues/9917
### Description
As described in #9899, there're some performance issues for druid to get all used segments metadata info from the underlying database.
To help us easily monitor the sql performance, it's useful to log the sql and its execution time to log files so that we can analyze the sql performance.
Fortunately, DBI used by druid provides a logger to allow user to record sql and its execution time, and the code in metadata storage related to sql operation has already calls that logger to log those info.
But, DBI's default logger is `NoOpLog`, which is an empty implementation and does nothing. So here's a proposal to output the sql being executed and its execution time to log files
### Changes
Since the DBI is instantiated in meta storage implementation, and DBI is heavily used in druid-server module, my proposal here is to put the change in meta storage implementations.
In `MySQLConnector`/`PostgreSQLConnector`/`SQLServerConnector`, after the instantiation of `DBI`, change the logger of DBI from `NoOpLog` to `SLF4JLog`
Currently I think it's no need to provide a configuration item to allow user whether or not to enable this small feature, because druid does not access to the meta storage very frequently, which means this change won't introduce very large amount of logs to the log file.
But if we do need such a configuration, according to current arch and implementation, we could define a property named as 'enabledSqlProfile` in `MetadataStorageConnectorConfig`.
I will later make a PR.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] yuanlihan commented on issue #9917: record sql executed on meta storage to log file
Posted by GitBox <gi...@apache.org>.
yuanlihan commented on issue #9917:
URL: https://github.com/apache/druid/issues/9917#issuecomment-633598917
Hi @FrankChen021, it's a great idea to log the execution time of SQL.
I'm used to suffer from issues caused by bottleneck of sql execution too.
It seems that it will also be convenient if we emit [metrics](https://druid.apache.org/docs/latest/operations/metrics.html) about DBI's sql execution time apart from just logging them to logs. And what do you think about this?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] FrankChen021 commented on issue #9917: record sql executed on meta storage to log file
Posted by GitBox <gi...@apache.org>.
FrankChen021 commented on issue #9917:
URL: https://github.com/apache/druid/issues/9917#issuecomment-633621392
@yuanlihan yes, metrics is better but involves more details to think about. with the support of metrics, we can get data from kafka, which will make the analysis much more convenient.
Maybe we could add a metrics like `metaSql/time` with one dimension `sql`.
I don't know whether or not the current implementation of metrics would aggregate the values by the same dimension. if so, maybe we need to emit the raw sql instead of generalized sql output by DBI.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org