You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/08/07 15:56:16 UTC
[GitHub] [druid] zenfenan opened a new issue #10254: Compaction not working with segmentLock
zenfenan opened a new issue #10254:
URL: https://github.com/apache/druid/issues/10254
I updated my Kafka supervisor spec to set forceTimeChunkLock to false so that I can run compaction tasks for intervals which has realtime segments but the compaction task is stuck in `WAITING` state and overlord logs indicate it is unable to acquire a lock.
### Affected Version
0.18.1
### Description
We have a Kafka supervisor with the query granularity at hourly level and segment granularity at day level. And both of them are period granularity types with IST as the timezone. Please refer the following granularitySpec.
```
"granularitySpec": {
"type": "uniform",
"queryGranularity": {
"type": "period",
"period": "PT1H",
"timeZone": "Asia/Kolkata"
},
"segmentGranularity": {
"type": "period",
"period": "P1D",
"timeZone": "Asia/Kolkata"
}
}
```
And since we have the segment granularity as `P1D`, we can only compact segments at T-1 date since there will be realtime segments for today's interval. I was reading through Druid's [task reference](https://druid.apache.org/docs/latest/ingestion/tasks.html) page which mentioned that if we enforce the locking type to be segment lock instead of time lock, we can do compactions even for the interval which has realtime segments. So I updated my Kafka indexing spec to set `forceTimeChunkLock` to false under `context` field. However, even after updating to use segment lock, when I submitted a compaction spec, it was in `WAITING` state. Reading through the overlord.log, I found the following.
```
2020-08-07T15:34:00,529 INFO [qtp786047074-85] org.apache.druid.indexing.overlord.MetadataTaskStorage - Inserting task compact_kafka_ist_hourly_emingfdh_2020-08-07T15:34:00.527Z with status: TaskStatus{id=compact_kafka_ist_hourly_emingfdh_2020-08-07T15:34:00.527Z, status=RUNNING, duration=-1, errorMsg=null}
2020-08-07T15:34:00,531 INFO [qtp786047074-85] org.apache.druid.indexing.overlord.TaskLockbox - Adding task[compact_kafka_ist_hourly_emingfdh_2020-08-07T15:34:00.527Z] to activeTasks
2020-08-07T15:34:00,535 INFO [TaskQueue-Manager] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Detected segmentGranularity change. Using timeChunk lock
2020-08-07T15:34:00,535 INFO [TaskQueue-Manager] org.apache.druid.indexing.overlord.TaskLockbox - Cannot create a new taskLockPosse for request[TimeChunkLockRequest{lockType=EXCLUSIVE, groupId='compact_kafka_ist_hourly_emingfdh_2020-08-07T15:34:00.527Z', dataSource='kafka_ist_hourly', interval=2020-08-07T00:00:00.000+05:30/2020-08-08T00:00:00.000+05:30, preferredVersion='null', priority=25, revoked=false}] because existing locks[[TaskLockPosse{taskLock=SegmentLock{lockType=EXCLUSIVE, groupId='index_kafka_kafka_ist_hourly', dataSource='kafka_ist_hourly', interval=2020-08-06T18:30:00.000Z/2020-08-07T18:30:00.000Z, version='2020-08-06T18:29:13.134Z', partitionId=60, priority=75, revoked=false}, taskIds=[index_kafka_kafka_ist_hourly_4c1beb3f5b90384_kmcljcgk]}, TaskLockPosse{taskLock=SegmentLock{lockType=EXCLUSIVE, groupId='index_kafka_kafka_ist_hourly', dataSource='kafka_ist_hourly', interval=2020-08-06T18:30:00.000Z/2020-08-07T18:30:00.000Z, version='2020-08-06T18:29:13.134Z', partit
ionId=61, priority=75, revoked=false}, taskIds=[index_kafka_kafka_ist_hourly_505a81616efe052_ncbdikid]}, TaskLockPosse{taskLock=SegmentLock{lockType=EXCLUSIVE, groupId='index_kafka_kafka_ist_hourly', dataSource='kafka_ist_hourly', interval=2020-08-06T18:30:00.000Z/2020-08-07T18:30:00.000Z, version='2020-08-06T18:29:13.134Z', partitionId=62, priority=75, revoked=false}, taskIds=[index_kafka_kafka_ist_hourly_150349ea52a197b_gdnfljjg]}, TaskLockPosse{taskLock=SegmentLock{lockType=EXCLUSIVE, groupId='index_kafka_kafka_ist_hourly', dataSource='kafka_ist_hourly', interval=2020-08-06T18:30:00.000Z/2020-08-07T18:30:00.000Z, version='2020-08-06T18:29:13.134Z', partitionId=63, priority=75, revoked=false}, taskIds=[index_kafka_kafka_ist_hourly_7de877c9aee1a93_mefncmen]}, TaskLockPosse{taskLock=SegmentLock{lockType=EXCLUSIVE, groupId='index_kafka_kafka_ist_hourly', dataSource='kafka_ist_hourly', interval=2020-08-06T18:30:00.000Z/2020-08-07T18:30:00.000Z, version='2020-08-06T18:29:13.134Z', part
itionId=64, priority=75, revoked=false}, taskIds=[index_kafka_kafka_ist_hourly_44ab741b2682141_anfcfpna]}]] have same or higher priorities
```
So regardless of having forceTimeChunkLock set to false, it seems to enforce time chunk lock and also that it fails to acquire a lock.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10254: Compaction not working with segmentLock
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10254:
URL: https://github.com/apache/druid/issues/10254#issuecomment-670589051
> 2020-08-07T15:34:00,535 INFO [TaskQueue-Manager] org.apache.druid.indexing.common.task.AbstractBatchIndexTask - Detected segmentGranularity change. Using timeChunk lock
Hi @zenfenan, seems like the compaction task detected a change in segment granularity. Segment lock cannot be used in that case since the whole time chunk should be updated atomically. Did you set a different segment granularity for the compaction task?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zenfenan commented on issue #10254: Compaction not working with segmentLock
Posted by GitBox <gi...@apache.org>.
zenfenan commented on issue #10254:
URL: https://github.com/apache/druid/issues/10254#issuecomment-670623026
Actually, I enabled automatic compaction through the UI before but it was also failing with similar error. If you look at the log snippet, the create task lock posse will say that it cannot lock segment with time interval in IST 00:00:00.000+05:30 because another lock is active (but shows interval in 00:00:000Z i.e. UTC) So I guess this is a bug. I'll try again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #10254: Compaction not working with segmentLock
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #10254:
URL: https://github.com/apache/druid/issues/10254#issuecomment-670612596
Hmm, it should work in that case. Maybe it's a bug. Could you try again with out segment granularity in the compaction task spec and see if it works?
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zenfenan commented on issue #10254: Compaction not working with segmentLock
Posted by GitBox <gi...@apache.org>.
zenfenan commented on issue #10254:
URL: https://github.com/apache/druid/issues/10254#issuecomment-670610202
I specified the same segmentGranularity in the compaction which is in the ingestion spec. Please find it below.
```
{
"type": "compact",
"dataSource": "compact_kafka_ist_hourly",
"interval": "2020-07-28T18:30:00.000Z/2020-07-29T18:30:00.000Z",
"segmentGranularity": {
"type": "period",
"period": "P1D",
"timeZone": "Asia/Kolkata"
},
"tuningConfig": {
"type": "index_parallel",
"maxBytesInMemory": -1,
"partitionsSpec": {
"type": "dynamic",
"maxTotalRows": "750000"
}
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zenfenan edited a comment on issue #10254: Compaction not working with segmentLock
Posted by GitBox <gi...@apache.org>.
zenfenan edited a comment on issue #10254:
URL: https://github.com/apache/druid/issues/10254#issuecomment-670623026
Actually, I enabled automatic compaction through the UI before but it was also failing with similar error. So I had to update my compaction spec to include the period granularity to even work compact. If you look at the log snippet, the create task lock posse will say that it cannot lock segment with time interval in IST 00:00:00.000+05:30 because another lock is active (but shows interval in 00:00:000Z i.e. UTC) So I guess this is a bug. I'll try again.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org