You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Floran Stuijt (Jira)" <ji...@apache.org> on 2019/11/29 21:22:00 UTC
[jira] [Updated] (NIFI-6918) Impossible to fill AVRO fixed type
from JSON
[ https://issues.apache.org/jira/browse/NIFI-6918?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Floran Stuijt updated NIFI-6918:
--------------------------------
Description:
Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:
{noformat}
{
"type":"record",
"name":"Message",
"namespace":"com.example",
"fields":[
{
"name":"MAC",
"type":{
"type":"fixed",
"name":"MY_FIXED_TYPE",
"size":6
}
}
]
}{noformat}
The input JSON-forms I tried are:
{noformat}
{ "MAC": [ 0, 1, 2, 3, 4, 5] }
{ "MAC": "012345" }
{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}
{"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}
{noformat}
Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead of the ConvertJsonToAvro-processor. Also without any luck.
It seems that, after some further investigation, the ConvertJsonToAvro processor *_can't_* be used to generate Avro {{FIXED}} or {{BYTES}} datum. (source: [https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nifi|https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nififor])
# The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro conversion is _not_ the same as Avro JSON encoding from the specification.
# This processor reads the incoming string into a [jackson JsonNode|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L65].
# {{FIXED}} and {{BYTES}} types need to correspond to a JsonNode where [{{isBinary()}} is true|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L324].
# As far as I can tell, parsing a JSON string with Jackson _never_ generates such a JSON node.
was:
Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:
{{{"type" : "record","name" : "Message","namespace" : "com.example","fields" : [ {"name" : "MAC","type" : {"type" : "fixed","name" : "MY_FIXED_TYPE","size" : 6}}]}}}
The input JSON-forms I tried are
{{{ "MAC": [ 0, 1, 2, 3, 4, 5] }{ "MAC": "012345" }{"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}{"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}}}
Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead of the ConvertJsonToAvro-processor. Also without any luck.
It seems that, after some further investigation, the ConvertJsonToAvro processor *_can't_* be used to generate Avro {{FIXED}} or {{BYTES}} datum. (source: [https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nifi|https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nififor])
# The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro conversion is _not_ the same as Avro JSON encoding from the specification.
# This processor reads the incoming string into a [jackson JsonNode|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L65].
# {{FIXED}} and {{BYTES}} types need to correspond to a JsonNode where [{{isBinary()}} is true|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L324].
# As far as I can tell, parsing a JSON string with Jackson _never_ generates such a JSON node.
> Impossible to fill AVRO fixed type from JSON
> --------------------------------------------
>
> Key: NIFI-6918
> URL: https://issues.apache.org/jira/browse/NIFI-6918
> Project: Apache NiFi
> Issue Type: Bug
> Affects Versions: 1.9.2
> Reporter: Floran Stuijt
> Priority: Major
>
> Currently I'm using the ConvertJsonToAvro-processor. The AVRO output schema:
>
> {noformat}
> {
> "type":"record",
> "name":"Message",
> "namespace":"com.example",
> "fields":[
> {
> "name":"MAC",
> "type":{
> "type":"fixed",
> "name":"MY_FIXED_TYPE",
> "size":6
> }
> }
> ]
> }{noformat}
>
> The input JSON-forms I tried are:
>
> {noformat}
> { "MAC": [ 0, 1, 2, 3, 4, 5] }
> { "MAC": "012345" }
> {"MAC":"\u0000\u0001\u0002\u0003\u0004\u0005"}
> {"MAC":{"MY_FIXED_TYPE": "\u0000\u0001\u0002\u0003\u0004\u0005"}}
> {noformat}
>
> Unfortunately none of them worked for me. I also tried the ConvertRecord-processor instead of the ConvertJsonToAvro-processor. Also without any luck.
>
> It seems that, after some further investigation, the ConvertJsonToAvro processor *_can't_* be used to generate Avro {{FIXED}} or {{BYTES}} datum. (source: [https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nifi|https://stackoverflow.com/questions/58391291/writing-avro-fixed-type-from-json-in-nififor])
> # The NiFi ConvertJsonToAvro uses the KiteSDK to interpret JSON into Avro data. This JSON-to-Avro conversion is _not_ the same as Avro JSON encoding from the specification.
> # This processor reads the incoming string into a [jackson JsonNode|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L65].
> # {{FIXED}} and {{BYTES}} types need to correspond to a JsonNode where [{{isBinary()}} is true|https://github.com/kite-sdk/kite/blob/cdd699a1818c8d6c573b2789ef2edf9d340c0827/kite-data/kite-data-core/src/main/java/org/kitesdk/data/spi/JsonUtil.java#L324].
> # As far as I can tell, parsing a JSON string with Jackson _never_ generates such a JSON node.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)