You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@druid.apache.org by "rahulkg31 (via GitHub)" <gi...@apache.org> on 2023/02/06 11:08:06 UTC
[GitHub] [druid] rahulkg31 opened a new issue, #13757: Ingest base64 encoded avro messages in druid
rahulkg31 opened a new issue, #13757:
URL: https://github.com/apache/druid/issues/13757
Hi,
I want to ingest base64 encoded avro messages in druid. I am getting the following error -
Avro's unnecessary EOFException, detail: https://issues.apache.org/jira/browse/AVRO-813
Going through the code (line 88) https://github.com/apache/druid/blob/master/extensions-core/avro-extensions/src/main/java/org/apache/druid/data/input/avro/InlineSchemaAvroBytesDecoder.java , it does not seem to be decoding the messages using base64 decoder. Am I missing something? How can we configure druid to parse base64 encoded avro messages?
Spec used -
```
"inputFormat": {
"type": "avro_stream",
"avroBytesDecoder": {
"type": "schema_inline",
"schema": {
"namespace": "org.apache.druid.data",
"name": "User",
"type": "record",
"fields": [
{
"name": "id",
"type": "string"
},
{
"name": "price",
"type": "int"
}
]
}
},
"flattenSpec": {
"useFieldDiscovery": true,
"fields": [
{
"type": "path",
"name": "someRecord_subInt",
"expr": "$.someRecord.subInt"
}
]
},
"binaryAsString": false
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rahulkg31 commented on issue #13757: Ingest base64 encoded avro messages in druid
Posted by "rahulkg31 (via GitHub)" <gi...@apache.org>.
rahulkg31 commented on issue #13757:
URL: https://github.com/apache/druid/issues/13757#issuecomment-1467962012
Hi,
I could not find any existing solution for this, so I created a separate extension which is a copy of existing avro-extension. I just made the following changes in file `src/main/java/org/apache/druid/data/input/avro/InlineSchemaAvroBytesDecoder.java` and created the jar for the same and used in the setup-
```
try (ByteBufferInputStream inputStream = new ByteBufferInputStream(Collections.singletonList(bytes))) {
return reader.read(null, DecoderFactory.get().binaryDecoder(inputStream, null));
}
```
to
```
try (ByteBufferInputStream inputStream = new ByteBufferInputStream(Collections.singletonList(Base64.getDecoder().decode(bytes)))) {
return reader.read(null, DecoderFactory.get().binaryDecoder(inputStream, null));
}
```
Closing the issue!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org
[GitHub] [druid] rahulkg31 closed issue #13757: Ingest base64 encoded avro messages in druid
Posted by "rahulkg31 (via GitHub)" <gi...@apache.org>.
rahulkg31 closed issue #13757: Ingest base64 encoded avro messages in druid
URL: https://github.com/apache/druid/issues/13757
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@druid.apache.org
For additional commands, e-mail: commits-help@druid.apache.org