You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by GitBox <gi...@apache.org> on 2020/01/31 18:45:42 UTC
[GitHub] [druid] zorino opened a new issue #9299: Cannot construct instance
of `org.apache.druid.data.input.AvroStreamInputRowParser`
zorino opened a new issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
URL: https://github.com/apache/druid/issues/9299
I cannot create a kafka ingestion parser with an AvroSchema...
The supervisor keep throwing :
```
{
"timestamp": "2020-01-31T18:36:25.007Z",
"exceptionClass": "java.lang.IllegalArgumentException",
"message": "Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`, problem: parseSpec\n at [Source: UNKNOWN; line: -1, column: -1]",
"streamException": false
},
```
Also logs of the Tasks are returning 404 :
Request failed with status code 404
Here is my supervisor config :
```
{
"type": "kafka",
"dataSchema": {
"dataSource": "divolte-clickstream",
"parser": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_inline",
"subjectAndIdConverter": {
"type": "avro_1124",
"topic": "divolte"
},
"schema": {
"namespace": "io.divolte.examples.record",
"type": "record",
"name": "MyEventRecord",
"fields": [
{ "name": "timestamp", "type": "long" },
{ "name": "remoteHost", "type": "string"},
{ "name": "eventType", "type": ["null", "string"], "default": null },
{ "name": "location", "type": ["null", "string"], "default": null },
{ "name": "localPath", "type": ["null", "string"], "default": null },
{ "name": "q", "type": ["null", "string"], "default": null },
{ "name": "n", "type": ["null", "int"], "default": null }
]
},
"parseSpec": {
"format": "avro",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"remoteHost",
"eventType",
"location",
"localPath",
"q",
"n"
]
}
}
},
"metricsSpec": [{
"name": "count",
"type": "count"
}],
"granularitySpec": {
"type": "uniform",
"segmentGranularity": "HOUR",
"queryGranularity": "NONE"
}
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 5000000
},
"ioConfig": {
"topic": "divolte",
"consumerProperties": {
"bootstrap.servers": "kafka:29092"
},
"workerThreads": 2,
"taskCount": 1,
"replicas": 1,
"taskDuration": "PT5M"
}
}
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zorino closed issue #9299: Cannot construct instance of
`org.apache.druid.data.input.AvroStreamInputRowParser`
Posted by GitBox <gi...@apache.org>.
zorino closed issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
URL: https://github.com/apache/druid/issues/9299
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] jihoonson commented on issue #9299: Cannot construct
instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
Posted by GitBox <gi...@apache.org>.
jihoonson commented on issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
URL: https://github.com/apache/druid/issues/9299#issuecomment-580863140
Hi, would you double check the parseSpec is at the right level? It seems like in the parser.avroBytesDecoder but should be in parser.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zorino edited a comment on issue #9299: Cannot construct
instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
Posted by GitBox <gi...@apache.org>.
zorino edited a comment on issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
URL: https://github.com/apache/druid/issues/9299#issuecomment-580974704
Yes thank you, I did but I can't make it work either here is the new config
```
{
"type": "kafka",
"dataSchema": {
"dataSource": "divolte-clickstream",
"parser": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_inline",
"subjectAndIdConverter": {
"type": "avro_1124",
"topic": "divolte"
},
"schema": {
"namespace": "io.divolte.examples.record",
"type": "record",
"name": "MyEventRecord",
"fields": [
{ "name": "timestamp", "type": "long" },
{ "name": "remoteHost", "type": "string"},
{ "name": "eventType", "type": ["null", "string"], "default": null },
{ "name": "location", "type": ["null", "string"], "default": null },
{ "name": "localPath", "type": ["null", "string"], "default": null },
{ "name": "q", "type": ["null", "string"], "default": null },
{ "name": "n", "type": ["null", "int"], "default": null }
]
}
},
"parseSpec": {
"format": "avro",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"timestamp",
"remoteHost",
"eventType",
"location",
"localPath",
"q",
"n"
],
"dimensionExclusions": []
},
"columns": [
"timestamp",
"remoteHost",
"eventType",
"location",
"localPath",
"q",
"n"
]
}
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 5000000
},
"ioConfig": {
"topic": "divolte",
"consumerProperties": {
"bootstrap.servers": "kafka:29092"
},
"taskCount": 5,
"replicas": 1,
"taskDuration": "PT1H"
}
}
```
Also the log files just disappear and return Request failed with status code 404 afterward.
I manage to successfully load the divolte raw byte data from kafka .. I just can't parse it with the Avro .. either with the schema-registry or specifying the inline_schema..
I really must do something wrong.
Thanks for your help,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] zorino commented on issue #9299: Cannot construct instance
of `org.apache.druid.data.input.AvroStreamInputRowParser`
Posted by GitBox <gi...@apache.org>.
zorino commented on issue #9299: Cannot construct instance of `org.apache.druid.data.input.AvroStreamInputRowParser`
URL: https://github.com/apache/druid/issues/9299#issuecomment-580974704
Yes thank you, I did but I can't make it work either :+1:
```
{
"type": "kafka",
"dataSchema": {
"dataSource": "divolte-clickstream",
"parser": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_inline",
"subjectAndIdConverter": {
"type": "avro_1124",
"topic": "divolte"
},
"schema": {
"namespace": "io.divolte.examples.record",
"type": "record",
"name": "MyEventRecord",
"fields": [
{ "name": "timestamp", "type": "long" },
{ "name": "remoteHost", "type": "string"},
{ "name": "eventType", "type": ["null", "string"], "default": null },
{ "name": "location", "type": ["null", "string"], "default": null },
{ "name": "localPath", "type": ["null", "string"], "default": null },
{ "name": "q", "type": ["null", "string"], "default": null },
{ "name": "n", "type": ["null", "int"], "default": null }
]
}
},
"parseSpec": {
"format": "avro",
"timestampSpec": {
"column": "timestamp",
"format": "auto"
},
"dimensionsSpec": {
"dimensions": [
"timestamp",
"remoteHost",
"eventType",
"location",
"localPath",
"q",
"n"
],
"dimensionExclusions": []
},
"columns": [
"timestamp",
"remoteHost",
"eventType",
"location",
"localPath",
"q",
"n"
]
}
}
},
"tuningConfig": {
"type": "kafka",
"maxRowsPerSegment": 5000000
},
"ioConfig": {
"topic": "divolte",
"consumerProperties": {
"bootstrap.servers": "kafka:29092"
},
"taskCount": 5,
"replicas": 1,
"taskDuration": "PT1H"
}
}
```
Also the log files just disappear and return Request failed with status code 404 afterward.
I manage to successfully load the divolte raw byte data from kafka .. I just can't parse it with the Avro .. either with the schema-registry or specifying the inline_schema..
I really must do something wrong.
Thanks for your help,
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
With regards,
Apache Git Services
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org