You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2019/02/02 23:25:44 UTC
[GitHub] kad3nce opened a new issue #6992: Is this supervisor spec
well-formed?
kad3nce opened a new issue #6992: Is this supervisor spec well-formed?
URL: https://github.com/apache/incubator-druid/issues/6992
I am trying to perform a realtime ingestion task from Kafka using the Kafka indexing service plugin.
I am loading the requisite plugins for my scenario (using Divolte with an Avro schema):
```druid.extensions.loadList=["druid-kafka-indexing-service", "druid-avro-extensions", "mysql-metadata-storage"]```
But submitting the supervisor spec does not seem to be accomplishing the task of creating the datasource in Druid.
```sh
$ curl -XPOST -H'Content-Type: application/json' -d @supervisor-spec.json http://localhost:8090/druid/indexer/v1/supervisor
{"id":"livestream"}
````
After submitting this supervisor spec, I can see the supervisor in the Overlord console:
But the datasource is listed as 'undefined' in the **indexing** tab of the Druid Cluster monitor:
And is not listed at all in the **datasources** tab:
I was under the impression that submitting a supervisor spec was supposed to result in that supervisor spec's datasource being automatically created by Druid. Is that not the case? Is there something wrong with my spec?
```json
{
"type": "kafka",
"dataSchema": {
"dataSource": "livestream",
"parser": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_inline",
"subjectAndIdConverter": {
"type": "avro_1124",
"topic": "livestream"
},
"schema": {
"namespace": "org.apache.druid.data",
"name": "User",
"type": "record",
"fields": [{
"name": "detectedDuplicate",
"type": "boolean"
},
{
"name": "detectedCorruption",
"type": "boolean"
},
{
"name": "firstInSession",
"type": "boolean"
},
{
"name": "timestamp",
"type": "long"
},
{
"name": "remoteHost",
"type": "string"
},
{
"name": "referer",
"type": ["null", "string"],
"default": null
},
{
"name": "location",
"type": ["null", "string"],
"default": null
},
{
"name": "viewportPixelWidth",
"type": ["null", "int"],
"default": null
},
{
"name": "viewportPixelHeight",
"type": ["null", "int"],
"default": null
},
{
"name": "screenPixelWidth",
"type": ["null", "int"],
"default": null
},
{
"name": "screenPixelHeight",
"type": ["null", "int"],
"default": null
},
{
"name": "partyId",
"type": ["null", "string"],
"default": null
},
{
"name": "sessionId",
"type": ["null", "string"],
"default": null
},
{
"name": "pageViewId",
"type": ["null", "string"],
"default": null
},
{
"name": "eventType",
"type": "string",
"default": "unknown"
},
{
"name": "userAgentString",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentName",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentFamily",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentVendor",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentType",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentVersion",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentDeviceCategory",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentOsFamily",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentOsVersion",
"type": ["null", "string"],
"default": null
},
{
"name": "userAgentOsVendor",
"type": ["null", "string"],
"default": null
},
{
"name": "technology",
"type": ["null", "string"],
"default": null
}
]
}
},
"parseSpec": {
"format": "timeAndDims",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"detectedDuplicate",
"detectedCorruption",
"firstInSession",
"remoteHost",
"referer",
"location",
"viewportPixelWidth",
"viewportPixelHeight",
"screenPixelWidth",
"screenPixelHeight",
"partyId",
"sessionId",
"pageViewId",
"eventType",
"userAgentString",
"userAgentName",
"userAgentFamily",
"userAgentVendor",
"userAgentType",
"userAgentVersion",
"userAgentDeviceCategory",
"userAgentOsFamily",
"userAgentOsVersion",
"userAgentOsVendor",
"technology"
]
}
}
},
"metricsSpec": [{
"name": "count",
"type": "count"
}],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "NONE"
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 5000000,
"logParseExceptions": true
},
"ioConfig": {
"topic": "livestream",
"consumerProperties": {
"bootstrap.servers": "10.0.1.32:9092"
},
"taskCount": 1,
"replicas": 1,
"taskDuration": "PT5S"
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org