You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "Sandro Berger (Jira)" <ji...@apache.org> on 2024/10/02 12:41:00 UTC
[jira] [Comment Edited] (NIFI-13334) XMLReader drops name-value content tags from record arrays if one record has only one tag
[ https://issues.apache.org/jira/browse/NIFI-13334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17886360#comment-17886360 ]
Sandro Berger edited comment on NIFI-13334 at 10/2/24 12:40 PM:
----------------------------------------------------------------
Hi [~Steve Hindmarch]
I think your issue is related to the 'Infer Schema' setting in the XML Reader.
I tried both your XML examples with our Nifi 1.23.2 and as I suspected, they set different schemas (checked by using ConvertRecord with a AvroRecordSetWriter to easily get the schemas):
First example's schema has for {{Data}} an array of {{UserData_DataType}} records:
{code:json}
{
"type": "record",
"name": "nifiRecord",
"namespace": "org.apache.nifi",
"fields": [
{
"name": "Type",
"type": [
"string",
"null"
]
},
{
"name": "UserData",
"type": [
{
"type": "record",
"name": "UserDataType",
"fields": [
{
"name": "Data",
"type": [
{
"type": "array",
"items": {
"type": "record",
"name": "UserData_DataType",
"fields": [
{
"name": "Name",
"type": [
"string",
"null"
]
},
{
"name": "Value",
"type": [
"string",
"null"
]
}
]
}
},
"null"
]
}
]
},
"null"
]
}
]
}
{code}
Second has for {{Data}} a choice of a {{UserData_DataType}} record or an array of {{{}UserData_DataType{}}}. I don't understand why - but this choice seems to be the problem:
{code:json}
{
"type": "record",
"name": "nifiRecord",
"namespace": "org.apache.nifi",
"fields": [
{
"name": "Type",
"type": [
"string",
"null"
]
},
{
"name": "UserData",
"type": [
{
"type": "record",
"name": "UserDataType",
"fields": [
{
"name": "Data",
"type": [
{
"type": "record",
"name": "UserData_DataType",
"fields": [
{
"name": "Name",
"type": [
"string",
"null"
]
},
{
"name": "Value",
"type": [
"string",
"null"
]
}
]
},
{
"type": "array",
"items": "UserData_DataType"
},
"null"
]
}
]
},
"null"
]
}
]
}
{code}
When I set the first example's avro schema in the XMLReader (to ensure we always get the array of {{UserData_DataType)}} and convert your second example XML to JSON I get this:
{code:json}
[ {
"Type" : "foo",
"UserData" : {
"Data" : [ {
"Name" : "Param1",
"Value" : "String1"
} ]
}
}, {
"Type" : "bar",
"UserData" : {
"Data" : [ {
"Name" : "Param1",
"Value" : "String"
}, {
"Name" : "Param2",
"Value" : "String2"
}, {
"Name" : "Param3",
"Value" : "String3"
} ]
}
} ]
{code}
I hope this is what you were expecting and setting the schema manually could be a workaround for you.
was (Author: JIRAUSER297079):
Hi [~Steve Hindmarch]
I think your issue is related to the 'Inherit Schema' setting in the XML Reader.
I tried both your XML examples with our Nifi 1.23.2 and as I suspected, they set different schemas (checked by using ConvertRecord with a AvroRecordSetWriter to easily get the schemas):
First example's schema has for {{Data}} an array of {{UserData_DataType}} records:
{code:json}
{
"type": "record",
"name": "nifiRecord",
"namespace": "org.apache.nifi",
"fields": [
{
"name": "Type",
"type": [
"string",
"null"
]
},
{
"name": "UserData",
"type": [
{
"type": "record",
"name": "UserDataType",
"fields": [
{
"name": "Data",
"type": [
{
"type": "array",
"items": {
"type": "record",
"name": "UserData_DataType",
"fields": [
{
"name": "Name",
"type": [
"string",
"null"
]
},
{
"name": "Value",
"type": [
"string",
"null"
]
}
]
}
},
"null"
]
}
]
},
"null"
]
}
]
}
{code}
Second has for {{Data}} a choice of a {{UserData_DataType}} record or an array of {{{}UserData_DataType{}}}. I don't understand why - but this choice seems to be the problem:
{code:json}
{
"type": "record",
"name": "nifiRecord",
"namespace": "org.apache.nifi",
"fields": [
{
"name": "Type",
"type": [
"string",
"null"
]
},
{
"name": "UserData",
"type": [
{
"type": "record",
"name": "UserDataType",
"fields": [
{
"name": "Data",
"type": [
{
"type": "record",
"name": "UserData_DataType",
"fields": [
{
"name": "Name",
"type": [
"string",
"null"
]
},
{
"name": "Value",
"type": [
"string",
"null"
]
}
]
},
{
"type": "array",
"items": "UserData_DataType"
},
"null"
]
}
]
},
"null"
]
}
]
}
{code}
When I set the first example's avro schema in the XMLReader (to ensure we always get the array of {{UserData_DataType)}} and convert your second example XML to JSON I get this:
{code:json}
[ {
"Type" : "foo",
"UserData" : {
"Data" : [ {
"Name" : "Param1",
"Value" : "String1"
} ]
}
}, {
"Type" : "bar",
"UserData" : {
"Data" : [ {
"Name" : "Param1",
"Value" : "String"
}, {
"Name" : "Param2",
"Value" : "String2"
}, {
"Name" : "Param3",
"Value" : "String3"
} ]
}
} ]
{code}
I hope this is what you were expecting in the first place and setting the schema manually could be a workaround for you.
> XMLReader drops name-value content tags from record arrays if one record has only one tag
> -----------------------------------------------------------------------------------------
>
> Key: NIFI-13334
> URL: https://issues.apache.org/jira/browse/NIFI-13334
> Project: Apache NiFi
> Issue Type: Bug
> Components: Core UI
> Affects Versions: 1.24.0
> Environment: Docker
> Reporter: Stephen Jeffrey Hindmarch
> Priority: Major
>
> If you create an XMLReader service and set the following:
> * Parse XML Attributes = true
> * Expect Records as Arrays = true
> * Field Name for Content = Value
> Then use the reader in a ConvertRecord processor with a JSONRecordSetWriter
> When parsing a flow file such as
> {noformat}
> <Events>
> <Event Type="foo">
> <UserData>
> <Data Name="Param1">String1</Data>
> <Data Name="Param2">String2</Data>
> </UserData>
> </Event>
> <Event Type="bar">
> <UserData>
> <Data Name="Param1">String</Data>
> <Data Name="Param2">String2</Data>
> <Data Name="Param3">String3</Data>
> </UserData>
> </Event>
> </Events>{noformat}
> Then as expected the content tags are parsed into arrays
> {noformat}
> [ {
> "Type" : "foo",
> "UserData" : {
> "Data" : [ {
> "Name" : "Param1",
> "Value" : "String1"
> }, {
> "Name" : "Param2",
> "Value" : "String2"
> } ]
> }
> }, {
> "Type" : "bar",
> "UserData" : {
> "Data" : [ {
> "Name" : "Param1",
> "Value" : "String"
> }, {
> "Name" : "Param2",
> "Value" : "String2"
> }, {
> "Name" : "Param3",
> "Value" : "String3"
> } ]
> }
> } ]{noformat}
> But if one of the records has only one data tag, then it will not be presented in an array, and more importantly, nor will the tags for the other record. Instead, all but the last tags are dropped.
> For example
> {noformat}
> <Events>
> <Event Type="foo">
> <UserData>
> <Data Name="Param1">String1</Data>
> </UserData>
> </Event>
> <Event Type="bar">
> <UserData>
> <Data Name="Param1">String</Data>
> <Data Name="Param2">String2</Data>
> <Data Name="Param3">String3</Data>
> </UserData>
> </Event>
> </Events>{noformat}
> parses to
> {noformat}
> [ {
> "Type" : "foo",
> "UserData" : {
> "Data" : {
> "Name" : "Param1",
> "Value" : "String1"
> }
> }
> }, {
> "Type" : "bar",
> "UserData" : {
> "Data" : {
> "Name" : "Param3",
> "Value" : "String3"
> }
> }
> } ]{noformat}
> Note that the second event has lost all but the last of its data content tags.
> It does not matter which event (first or second) has 1 tag, the other event loses content.
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)