You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@nifi.apache.org by "ASF GitHub Bot (JIRA)" <ji...@apache.org> on 2018/04/02 20:37:00 UTC
[jira] [Commented] (NIFI-4035) Implement record-based Solr
processors
[ https://issues.apache.org/jira/browse/NIFI-4035?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16423094#comment-16423094 ]
ASF GitHub Bot commented on NIFI-4035:
--------------------------------------
Github user bbende commented on the issue:
https://github.com/apache/nifi/pull/2561
Nested records and arrays of nested records are not working correctly...
**Scenario 1 - Nested Record**
Schema:
```
{
"type": "record",
"name": "exams",
"fields" : [
{ "name": "first", "type": "string" },
{ "name": "last", "type": "string" },
{ "name": "grade", "type": "int" },
{
"name": "exam",
"type": {
"name" : "exam",
"type" : "record",
"fields" : [
{ "name": "subject", "type": "string" },
{ "name": "test", "type": "string" },
{ "name": "marks", "type": "int" }
]
}
}
]
}
```
Input:
```
{
"first": "Abhi",
"last": "R",
"grade": 8,
"exam": {
"subject": "Maths",
"test" : "term1",
"marks" : 90
}
}
```
Result:
```
java.util.NoSuchElementException: No value present
at java.util.Optional.get(Optional.java:135)
at org.apache.nifi.processors.solr.SolrUtils.writeRecord(SolrUtils.java:313)
at org.apache.nifi.processors.solr.SolrUtils.writeValue(SolrUtils.java:384)
at org.apache.nifi.processors.solr.SolrUtils.writeRecord(SolrUtils.java:314)
at org.apache.nifi.processors.solr.PutSolrRecord.onTrigger(PutSolrRecord.java:247)
at org.apache.nifi.processor.AbstractProcessor.onTrigger(AbstractProcessor.java:27)
```
**Scenario 2 - Array of Records**
Schema:
```
{
"type": "record",
"name": "exams",
"fields" : [
{ "name": "first", "type": "string" },
{ "name": "last", "type": "string" },
{ "name": "grade", "type": "int" },
{
"name": "exams",
"type": {
"type" : "array",
"items" : {
"name" : "exam",
"type" : "record",
"fields" : [
{ "name": "subject", "type": "string" },
{ "name": "test", "type": "string" },
{ "name": "marks", "type": "int" }
]
}
}
}
]
}
```
Input:
```
{
"first": "Abhi",
"last": "R",
"grade": 8,
"exams": [
{
"subject": "Maths",
"test" : "term1",
"marks" : 90
},
{
"subject": "Physics",
"test" : "term1",
"marks" : 95
}
]
}
```
Result:
Solr Document with multi-valued field exams where the values are the toString of a MapRecord:
```
"exams":["org.apache.nifi.serialization.record.MapRecord:MapRecord[{marks=90, test=term1, subject=Maths}]",
"org.apache.nifi.serialization.record.MapRecord:MapRecord[{marks=95, test=term1, subject=Physics}]"],
```
Should have created fields like exams_marks, exams_test, exams_subject.
Here is a full template for the two scenarios:
https://gist.githubusercontent.com/bbende/edc2e7d61db83b29533ac3fc520de30f/raw/8764d50ed5e14d876c53a0b84b3af5741d910b3b/PutSolrRecordTesting.xml
There needs to be unit tests that cover both these cases.
> Implement record-based Solr processors
> --------------------------------------
>
> Key: NIFI-4035
> URL: https://issues.apache.org/jira/browse/NIFI-4035
> Project: Apache NiFi
> Issue Type: Improvement
> Affects Versions: 1.2.0, 1.3.0
> Reporter: Bryan Bende
> Priority: Minor
>
> Now that we have record readers and writers, we should implement variants of the existing Solr processors that record-based...
> Processors to consider:
> * PutSolrRecord - uses a configured record reader to read an incoming flow file and insert records to Solr
> * GetSolrRecord - extracts records from Solr and uses a configured record writer to write them to a flow file
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)