You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/02/21 16:05:57 UTC
[GitHub] [druid] acdn-mpreston opened a new issue #9391: 0.17.0 Kafka
Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is
restarted
acdn-mpreston opened a new issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391
### Affected Version
0.17.0
The Druid version where the problem was encountered.
0.17.0
### Description
When provisioning a Kafka Supervisor in druid 0.17.0, a payload using the inputFormat property in the ioConfig will be accepted and the supervisor will work properly until you restart the Overlord service. When the Overlord service is restarted, it will fail to create tasks due to there no longer being any "inputFormat" data in the ioConfig.
This is due to the way the json marshalling is handled in the configuration persistence. When you issue a command to create a supervisor with an ioConfig like this:
"ioConfig": {
"topic": "some-cool-topic",
"inputFormat": {
"type": "json"
},
"replicas": 1,
"taskCount": 2,
"taskDuration": "PT1800S",
"consumerProperties": {
"bootstrap.servers": "kafka:9092"
},
"pollTimeout": 100,
"startDelay": "PT5S",
"period": "PT30S",
"useEarliestOffset": true,
"completionTimeout": "PT3600S",
"lateMessageRejectionPeriod": null,
"earlyMessageRejectionPeriod": "PT3600S",
"stream": "npav-ts-metrics",
"useEarliestSequenceNumber": true
},
It will be accepted, but anytime you try to retrieve this supervisor, you will get back an ioConfig that looks like this:
"ioConfig": {
"topic": "some-cool-topic",
"replicas": 1,
"taskCount": 2,
"taskDuration": "PT1800S",
"consumerProperties": {
"bootstrap.servers": "kafka:9092"
},
"pollTimeout": 100,
"startDelay": "PT5S",
"period": "PT30S",
"useEarliestOffset": true,
"completionTimeout": "PT3600S",
"lateMessageRejectionPeriod": null,
"earlyMessageRejectionPeriod": "PT3600S",
"lateMessageRejectionStartDateTime": null,
"stream": "npav-ts-metrics",
"useEarliestSequenceNumber": true,
"givenInputFormat": {
"type": "json",
"flattenSpec": {
"useFieldDiscovery": true,
"fields": []
},
"featureSpec": {}
}
},
Notice that the inputFormat is gone and givenInputFormat is there instead. This supervisor will work properly until you restart the overlord, at which point the supervisor ioConfig will contain the following:
"ioConfig": {
"topic": "some-cool-topic",
"replicas": 1,
"taskCount": 2,
"taskDuration": "PT1800S",
"consumerProperties": {
"bootstrap.servers": "kafka:9092"
},
"pollTimeout": 100,
"startDelay": "PT5S",
"period": "PT30S",
"useEarliestOffset": true,
"completionTimeout": "PT3600S",
"lateMessageRejectionPeriod": null,
"earlyMessageRejectionPeriod": "PT3600S",
"lateMessageRejectionStartDateTime": null,
"stream": "npav-ts-metrics",
"useEarliestSequenceNumber": true,
"givenInputFormat": null
},
Notice that there is no longer a valid inputFormat, nor is there a valid givenInputFormat. When in this state, the supervisor is unable to create new ingestion tasks.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm commented on issue #9391: 0.17.0 Kafka Supervisor
becomes "unhealthy" due to missing inputFormat when Overlord is restarted
Posted by GitBox <gi...@apache.org>.
maytasm commented on issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391#issuecomment-608116659
@acdn-mpreston
Hmm.. can you try to reproduce this with 0.17.0 and 0.18.0? I cannot reproduce this bug in either version. Also. the changeset mentioned in #9136 rename the serialize/deserialize name of givenInputFormat to inputFormat, so you should not see "givenInputFormat" in the spec when you try to retrieve this supervisor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] acdn-mpreston commented on issue #9391: 0.17.0 Kafka
Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is
restarted
Posted by GitBox <gi...@apache.org>.
acdn-mpreston commented on issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391#issuecomment-608581741
I can confirm that this is resolved in the 0.17.1 branch. Closing this bug as resolved. Thank you for looking into it!
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9391: 0.17.0 Kafka Supervisor
becomes "unhealthy" due to missing inputFormat when Overlord is restarted
Posted by GitBox <gi...@apache.org>.
maytasm3 commented on issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391#issuecomment-606408273
I cannot reproduce this issue. I believe it was fixed in this PR: https://github.com/apache/druid/pull/9136
by @clintropolis
I think we can close this issue :)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] acdn-mpreston commented on issue #9391: 0.17.0 Kafka
Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is
restarted
Posted by GitBox <gi...@apache.org>.
acdn-mpreston commented on issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391#issuecomment-607978188
I will try and reproduce, but the version of the code I generated this bug from includes the changeset mentioned in https://github.com/apache/druid/pull/9136 so I think it may still be an issue
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm edited a comment on issue #9391: 0.17.0 Kafka
Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is
restarted
Posted by GitBox <gi...@apache.org>.
maytasm edited a comment on issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391#issuecomment-608116659
@acdn-mpreston
Hmm.. can you try to reproduce this with 0.17.0 and 0.18.0 again? Also for 0.17.0 you used, is it built from the branch? I cannot reproduce this bug in either version. Also. the changeset mentioned in #9136 rename the serialize/deserialize name of givenInputFormat to inputFormat, so you should not see "givenInputFormat" in the spec when you try to retrieve this supervisor.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] maytasm3 commented on issue #9391: 0.17.0 Kafka Supervisor
becomes "unhealthy" due to missing inputFormat when Overlord is restarted
Posted by GitBox <gi...@apache.org>.
maytasm3 commented on issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391#issuecomment-605350744
I will be fixing this
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] acdn-mpreston closed issue #9391: 0.17.0 Kafka Supervisor
becomes "unhealthy" due to missing inputFormat when Overlord is restarted
Posted by GitBox <gi...@apache.org>.
acdn-mpreston closed issue #9391: 0.17.0 Kafka Supervisor becomes "unhealthy" due to missing inputFormat when Overlord is restarted
URL: https://github.com/apache/druid/issues/9391
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org