You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/06/14 08:30:31 UTC
[GitHub] [druid] wenraoliu opened a new issue #10031: appenderator-merge too slow(0.17.0)
wenraoliu opened a new issue #10031:
URL: https://github.com/apache/druid/issues/10031
org.apache.druid.segment.realtime.appenderator.AppenderatorImpl - Segment[***2020-06-14T05:00:00.000Z_2020-06-14T06:00:00.000Z_2020-06-14T05:00:03.291Z_171] of 602,240,691 bytes built from 16 incremental persist(s) in 1,440,695ms; pushed to deep storage in 53,893ms. Load spec is: {"type":"hdfs","path":"XX.zip"}
merge 16 Segment(500M) cost 1440s (24min), is too slow. how can make it faster.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liran-funaro commented on issue #10031: appenderator-merge too slow(0.17.0)
Posted by GitBox <gi...@apache.org>.
liran-funaro commented on issue #10031:
URL: https://github.com/apache/druid/issues/10031#issuecomment-643742931
Hi @wenraoliu,
Can you elaborate? What is the size of your data? What hardware do you run? Storage: SSD/HDD?
Can you post here also the ingestion JSON?
Also, I just published a PR that nearly doubles the ingestion speed. You can try it out and let me know if that improves anything for you: #10001.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] wenraoliu closed issue #10031: appenderator-merge too slow(0.17.0)
Posted by GitBox <gi...@apache.org>.
wenraoliu closed issue #10031:
URL: https://github.com/apache/druid/issues/10031
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liran-funaro edited a comment on issue #10031: appenderator-merge too slow(0.17.0)
Posted by GitBox <gi...@apache.org>.
liran-funaro edited a comment on issue #10031:
URL: https://github.com/apache/druid/issues/10031#issuecomment-643742931
Hi @wenraoliu,
Can you elaborate? What is the size of your data? What hardware do you use? Storage: SSD/HDD?
Can you post here also the ingestion JSON?
Also, I just published a PR that nearly doubles the ingestion speed. You can try it out and let me know if that improves anything for you: #10001.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] liran-funaro commented on issue #10031: appenderator-merge too slow(0.17.0)
Posted by GitBox <gi...@apache.org>.
liran-funaro commented on issue #10031:
URL: https://github.com/apache/druid/issues/10031#issuecomment-644163765
Thanks!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] wenraoliu edited a comment on issue #10031: appenderator-merge too slow(0.17.0)
Posted by GitBox <gi...@apache.org>.
wenraoliu edited a comment on issue #10031:
URL: https://github.com/apache/druid/issues/10031#issuecomment-644020793
I found the reason: SATA TOO slow.
the PR is a pretty work , I am studying. thanks very much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] wenraoliu commented on issue #10031: appenderator-merge too slow(0.17.0)
Posted by GitBox <gi...@apache.org>.
wenraoliu commented on issue #10031:
URL: https://github.com/apache/druid/issues/10031#issuecomment-644020793
I found the reason: SATA TOO slow.
the PR is a pretty , I am studying. thanks very much
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org