Fwd: Protobuf message parsing

[Neeraj Koul <ne...@go-mmt.com>]

---------- Forwarded message ---------
From: Neeraj Koul <ne...@go-mmt.com>
Date: Fri, May 3, 2019 at 3:45 PM
Subject: Protobuf message parsing
To: <dr...@googlegroups.com>


Our ingest task tries parse protobuf messages from a kafka topic. We
couldn't get it done. Followed the steps as mentioned in  "
http://druid.io/docs/latest/development/extensions-core/protobuf.html", but
that is not working out. Infact, we could read and parse those protobuf
messages in our sample code on the same druid machine. but the ingestion
task doesn't seem to get work. We are putting .desc file in the classpath
and also parseSpec formagt is json, infact we have followed whatever is
there in the url.

Our sample protobuf message has just a couple of string fields, name and
company, and we use proto3. The trace we get is like

19-05-02T12:33:54,751 ERROR [task-runner-0-priority-0]
org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner
- Encountered parse exception on row from partition[0] offset[0]
org.apache.druid.java.util.common.parsers.ParseException: Unable to parse row [
 Sam  Google]
	at org.apache.druid.java.util.common.parsers.JSONPathParser.parseToMap(JSONPathParser.java:74)
~[java-util-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.data.input.impl.StringInputRowParser.parseString(StringInputRowParser.java:155)
~[druid-api-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.data.input.impl.StringInputRowParser.buildStringKeyMap(StringInputRowParser.java:119)
~[druid-api-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.data.input.impl.StringInputRowParser.parseBatch(StringInputRowParser.java:80)
~[druid-api-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:50)
~[druid-processing-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.segment.transform.TransformingStringInputRowParser.parseBatch(TransformingStringInputRowParser.java:31)
~[druid-processing-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.runInternal(IncrementalPublishingKafkaIndexTaskRunner.java:493)
[druid-kafka-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.indexing.kafka.IncrementalPublishingKafkaIndexTaskRunner.run(IncrementalPublishingKafkaIndexTaskRunner.java:232)
[druid-kafka-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.indexing.kafka.KafkaIndexTask.run(KafkaIndexTask.java:210)
[druid-kafka-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:422)
[druid-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at org.apache.druid.indexing.overlord.SingleTaskBackgroundRunner$SingleTaskBackgroundRunnerCallable.call(SingleTaskBackgroundRunner.java:394)
[druid-indexing-service-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	at java.util.concurrent.FutureTask.run(FutureTask.java:266) [?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
[?:1.8.0_201]
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
[?:1.8.0_201]
	at java.lang.Thread.run(Thread.java:748) [?:1.8.0_201]
Caused by: com.fasterxml.jackson.core.JsonParseException: Illegal
character ((CTRL-CHAR, code 6)): only regular white space (\r, \n, \t)
is allowed between tokens
 at [Source:
 Sam  Google; line: 2, column: 2]
	at com.fasterxml.jackson.core.JsonParser._constructError(JsonParser.java:1581)
~[jackson-core-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._reportError(ParserMinimalBase.java:533)
~[jackson-core-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.core.base.ParserMinimalBase._throwInvalidSpace(ParserMinimalBase.java:484)
~[jackson-core-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser._skipWSOrEnd(ReaderBasedJsonParser.java:2056)
~[jackson-core-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.core.json.ReaderBasedJsonParser.nextToken(ReaderBasedJsonParser.java:577)
~[jackson-core-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.databind.ObjectMapper._initForReading(ObjectMapper.java:3776)
~[jackson-databind-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.databind.ObjectMapper._readMapAndClose(ObjectMapper.java:3721)
~[jackson-databind-2.6.7.jar:2.6.7]
	at com.fasterxml.jackson.databind.ObjectMapper.readValue(ObjectMapper.java:2726)
~[jackson-databind-2.6.7.jar:2.6.7]
	at org.apache.druid.java.util.common.parsers.JSONPathParser.parseToMap(JSONPathParser.java:70)
~[java-util-0.13.0-incubating-iap11.jar:0.13.0-incubating-iap11]
	... 14 more


What it seems like is that it is trying to parse proto as json.Would
be great if you could point us to the problem that we have in our
setup.


Regards,

Neeraj

-- 


::DISCLAIMER::


----------------------------------------------------------------------------------------------------------------------------------------------------





This message is intended only for the use of the addressee and may 
contain information that is privileged, confidential and exempt from 
disclosure under applicable law. If the reader of this message is not the 
intended recipient, or the employee or agent responsible for delivering the 
message to the intended recipient, you are hereby notified that any 
dissemination, distribution or copying of this communication is strictly 
prohibited. If you have received this e-mail in error, please notify us 
immediately by return e-mail and delete this e-mail and all attachments 
from your system.