You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@doris.apache.org by GitBox <gi...@apache.org> on 2020/07/06 10:19:06 UTC
[GitHub] [incubator-doris] vagetablechicken opened a new issue #4027: DeltaWriter's FlushStatistic needs a metric
vagetablechicken opened a new issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027
More user-friendly way to debug, instead of grep logs.
DeltaWriter can flush many times, and there are too much writers opened, so we can get the max flush time in 15s. Or avg?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken commented on issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027#issuecomment-654562912
> I think the main pain points here is that the flush log isn't associated with a `transaction id` or `query id`, so it's hard to know how much time cost on `DeltaWriter.close` when we tracking a slow stream load using `transaction id` or `query id`.
Adding info in DeltaWriter is useful for tracking one slow stream load, also make sense.
But if we don't have the slow load id, e.g. users may be less sensitive to the execution time, or none of load is significantly slower than others.
Just think, for some reason the flush time or count in one BE is always longer/higher than others. But it's relative, we need the global statistics of BEs. If we have metric about it, it'll be more intuitive to us.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken commented on issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027#issuecomment-654714129
> I see.
> I am curious about detail of `some reason the flush time or count in one BE is always longer/higher than others`.
> Under what circumstances will this problem occur and what is the corresponding processing strategy?
If flush more times, it's probably because the loading data are only sent to a few BE. Maybe key cols are improper, or partition bucket num is too small(bucket num can be set in each parition, maybe a few partitions has improper bucket num, hard to find out).
If flush for too long, maybe the hard disk has some errors.
Actually, if we have tablet monitoring data shown on BE web or other place, we can easily find the problem.
e.g. it's slow read or write? which tablet is abnormal? blabla.
The tablet monitoring data could contain the flush time&count.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken commented on issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
vagetablechicken commented on issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027#issuecomment-655275835
MemTable flush count & time already have BE level stats, sorry, I didn't see it before create this issue. But no tablet level. It's better to add tablet monitoring than just adding a tablet flush metric. I'll close this issue.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
wangbo commented on issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027#issuecomment-654257097
I think the main pain points here is that the flush log isn't associated with a ```transaction id``` or ```query id```, so it's hard to know how much time cost on ```DeltaWriter.close``` when we tracking a slow stream load using ```transaction id``` or ```query id```.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
wangbo commented on issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027#issuecomment-654838945
Thanks for the answer
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] vagetablechicken closed issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
vagetablechicken closed issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org
[GitHub] [incubator-doris] wangbo commented on issue #4027: DeltaWriter's FlushStatistic needs a metric
Posted by GitBox <gi...@apache.org>.
wangbo commented on issue #4027:
URL: https://github.com/apache/incubator-doris/issues/4027#issuecomment-654591438
I see.
I am curious about detail of ```some reason the flush time or count in one BE is always longer/higher than others```.
Under what circumstances will this problem occur and what is the corresponding processing strategy?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@doris.apache.org
For additional commands, e-mail: commits-help@doris.apache.org