You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/06/24 19:50:00 UTC
[GitHub] [druid] pjain1 opened a new issue #10067: CostBalancerStrategy over assigns segments to historicals over their max size
pjain1 opened a new issue #10067:
URL: https://github.com/apache/druid/issues/10067
### Affected Version
All
### Description
Was hitting this issue in production cluster, so tried to reproduce it locally.
Setup - I have one empty historical with server size - 1,000,000 and one segment of size 4,821,713
Coordinator logs -
```
2020-06-24T17:50:10,751 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.LogUsedSegments - Found [1] used segments.
2020-06-24T17:50:10,752 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.ReplicationThrottler - [_default_tier]: Replicant create queue is empty.
2020-06-24T17:50:10,753 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Assigning 'primary' for segment [wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z] to server [localhost:8083] in tier [_default_tier]
2020-06-24T17:50:10,754 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - No available [_default_tier] servers or node capacity to assign segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z]! Expected Replicants[3]
2020-06-24T17:50:10,754 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Skipping replica assignment for tier [_default_tier]
2020-06-24T17:50:10,754 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.rules.LoadRule - Loading in progress, skipping drop until loading is complete
2020-06-24T17:50:10,754 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - Found 1 active servers, 0 decommissioning servers
2020-06-24T17:50:10,754 WARN [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.BalanceSegments - [_default_tier]: insufficient active servers. Cannot balance.
2020-06-24T17:50:10,754 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - [_default_tier] : Assigned 1 segments among 1 servers
2020-06-24T17:50:10,754 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Load Queues:
2020-06-24T17:50:10,754 INFO [Coordinator-Exec--0] org.apache.druid.server.coordinator.duty.EmitClusterStatsAndMetrics - Server[localhost:8083, historical, _default_tier] has 1 left to load, 0 left to drop, 4,821,713 bytes queued, 0 bytes served.
2020-06-24T17:50:10,791 ERROR [Master-PeonExec--0] org.apache.druid.server.coordinator.HttpLoadQueuePeon - Server[http://localhost:8083] Failed segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z] request[SegmentChangeRequestLoad] with cause [Exception loading segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z]].
```
Historical logs -
```
2020-06-24T17:49:40,830 WARN [SimpleDataSegmentChangeHandler-0] org.apache.druid.segment.loading.StorageLocation - Segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z:4,821,713] too large for storage[/.../cache:1,000,000]. Check your druid.segmentCache.locations maxSize param
2020-06-24T17:49:40,831 WARN [SimpleDataSegmentChangeHandler-0] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Asked to cleanup something[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z] that didn't exist. Skipping.
2020-06-24T17:49:40,831 WARN [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.coordination.BatchDataSegmentAnnouncer - No path to unannounce segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z]
2020-06-24T17:49:40,832 INFO [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.SegmentManager - Told to delete a queryable for a dataSource[wikipedia_null] that doesn't exist.
2020-06-24T17:49:40,832 WARN [SimpleDataSegmentChangeHandler-0] org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager - Asked to cleanup something[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z] that didn't exist. Skipping.
2020-06-24T17:49:40,832 WARN [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.coordination.SegmentLoadDropHandler - Unable to delete segmentInfoCacheFile[/.../cache/info_dir/wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z]
2020-06-24T17:49:40,841 ERROR [SimpleDataSegmentChangeHandler-0] org.apache.druid.server.coordination.SegmentLoadDropHandler - Failed to load segment for dataSource: {class=org.apache.druid.server.coordination.SegmentLoadDropHandler, exceptionType=class org.apache.druid.segment.loading.SegmentLoadingException, exceptionMessage=Exception loading segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z], segment=DataSegment{binaryVersion=9, id=wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z, loadSpec={type=>local, path=>/.../storage/wikipedia_null/2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z/2019-07-10T06:10:13.432Z/0/index.zip}, dimensions=[channel, cityName, comment, countryIsoCode, countryName, isAnonymous, isMinor, isNew, isRobot, isUnpatrolled, metroCode, namespace, page, regionIsoCode, regionName, user, added, deleted, delta], metrics=[], shardSpec=NumberedShardSpec{partitionNum=0, partitions=0}, lastCompactionState=null, size=4821713}}
org.apache.druid.segment.loading.SegmentLoadingException: Exception loading segment[wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:262) ~[classes/:?]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.addSegment(SegmentLoadDropHandler.java:306) ~[classes/:?]
at org.apache.druid.server.coordination.SegmentLoadDropHandler$1.lambda$addSegment$1(SegmentLoadDropHandler.java:512) ~[classes/:?]
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511) [?:1.8.0_211]
at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_211]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180) [?:1.8.0_211]
at java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149) [?:1.8.0_211]
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624) [?:1.8.0_211]
at java.lang.Thread.run(Thread.java:748) [?:1.8.0_211]
Caused by: org.apache.druid.segment.loading.SegmentLoadingException: Failed to load segment wikipedia_null_2015-09-12T00:00:00.000Z_2015-09-13T00:00:00.000Z_2019-07-10T06:10:13.432Z in all locations.
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.loadSegmentWithRetry(SegmentLoaderLocalCacheManager.java:209) ~[classes/:?]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegmentFiles(SegmentLoaderLocalCacheManager.java:163) ~[classes/:?]
at org.apache.druid.segment.loading.SegmentLoaderLocalCacheManager.getSegment(SegmentLoaderLocalCacheManager.java:130) ~[classes/:?]
at org.apache.druid.server.SegmentManager.getAdapter(SegmentManager.java:212) ~[classes/:?]
at org.apache.druid.server.SegmentManager.loadSegment(SegmentManager.java:171) ~[classes/:?]
at org.apache.druid.server.coordination.SegmentLoadDropHandler.loadSegment(SegmentLoadDropHandler.java:258) ~[classes/:?]
... 9 more
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] pjain1 closed issue #10067: CostBalancerStrategy over assigns segments to historicals over their max size
Posted by GitBox <gi...@apache.org>.
pjain1 closed issue #10067:
URL: https://github.com/apache/druid/issues/10067
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org