You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/01/06 07:49:30 UTC
[GitHub] [druid] smildlzj opened a new issue #9132: how auto compaction work?
smildlzj opened a new issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132
> 'druid.coordinator.merge.on' is not supported anymore. Please consider using Coordinator's automatic compaction instead. See https://druid.apache.org/docs/latest/operations/segment-optimization.html and https://druid.apache.org/docs/latest/operations/api-reference.html#compaction-configuration for more details about compaction.
but i can't find any document about auto compaction, it's mean need to use api to devlop some application to submit compaction task?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9132: how auto compaction
work?
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-576832985
Hi @smildlzj, if any compaction task is not being triggered, probably the coordinator decided to skip to compact segments in some intervals. Looking at your configuration, would you please double check the followings?
- Is the total segment size in the same interval smaller than `inputSegmentSizeBytes`?
- Is the total number of segments in the same interval smaller than `maxNumSegmentsToCompact`?
- Does the interval of the segments you want to compact overlap with the interval of `(endTimeOfTheMostRecentSegment - skipOffsetFromLatest, endTimeOfTheMostRecentSegment)`
Also, the reason should be logged in the coordinator log file as below:
```java
if (!isCompactibleSize) {
log.warn(
"total segment size[%d] for datasource[%s] and interval[%s] is larger than inputSegmentSize[%d]."
+ " Continue to the next interval.",
candidates.getTotalSize(),
candidates.segments.get(0).getDataSource(),
candidates.segments.get(0).getInterval(),
inputSegmentSize
);
}
if (!isCompactibleNum) {
log.warn(
"Number of segments[%d] for datasource[%s] and interval[%s] is larger than "
+ "maxNumSegmentsToCompact[%d]. If you see lots of shards are being skipped due to too many "
+ "segments, consider increasing 'numTargetCompactionSegments' and "
+ "'druid.indexer.runner.maxZnodeBytes'. Continue to the next interval.",
candidates.getNumSegments(),
candidates.segments.get(0).getDataSource(),
candidates.segments.get(0).getInterval(),
maxNumSegmentsToCompact
);
}
if (!needsCompaction) {
log.warn(
"Size of most of segments[%s] is larger than targetCompactionSizeBytes[%s] "
+ "for datasource[%s] and interval[%s]. Skipping compaction for this interval.",
candidates.segments.stream().map(DataSegment::getSize).collect(Collectors.toList()),
targetCompactionSizeBytes,
candidates.segments.get(0).getDataSource(),
candidates.segments.get(0).getInterval()
);
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] smildlzj commented on issue #9132: how auto compaction work?
Posted by GitBox <gi...@apache.org>.
smildlzj commented on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-573578287
look as like #9016, i want compact small segments to speed up for query
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] pawantekchandani commented on issue #9132: how auto
compaction work?
Posted by GitBox <gi...@apache.org>.
pawantekchandani commented on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-573933992
hey @smildlzj
Which version of Druid are you using?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] smildlzj edited a comment on issue #9132: how auto
compaction work?
Posted by GitBox <gi...@apache.org>.
smildlzj edited a comment on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-573958586
> hey @smildlzj
> Which version of Druid are you using?
hi @pawantekchandani, version is 0.16.1-incubating
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] smildlzj commented on issue #9132: how auto compaction work?
Posted by GitBox <gi...@apache.org>.
smildlzj commented on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-573958586
> hey @smildlzj
> Which version of Druid are you using?
hi, version is 0.16.1-incubating
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] suneet-s commented on issue #9132: how auto compaction work?
Posted by GitBox <gi...@apache.org>.
suneet-s commented on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-576807139
hi @smildlzj you can submit a compaction job just like an index task - see this tutorial https://druid.apache.org/docs/latest/tutorials/tutorial-compaction.html#compact-the-data
`bin/post-index-task --file quickstart/tutorial/compaction-keep-granularity.json --url http://localhost:8081`
Hope this helps!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] smildlzj edited a comment on issue #9132: how auto
compaction work?
Posted by GitBox <gi...@apache.org>.
smildlzj edited a comment on issue #9132: how auto compaction work?
URL: https://github.com/apache/druid/issues/9132#issuecomment-573578287
look as like #9016, i want compact small segments to speed up for query. but keep query granularity
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org