You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Madhukar Thota <ma...@gmail.com> on 2016/04/07 13:49:32 UTC
ExecuteSQL to elasticsearch
Friends,
I am exploring ExecuteSQL processor in nifi and my goal to get sql data
ingested in Elasticsearch.
Can someone share or guide what's the flow looks like?
Thanks in Advance.
Re: ExecuteSQL to elasticsearch
Posted by Matt Burgess <ma...@gmail.com>.
Madhu,
To achieve what Arpit is saying, you can add a SplitJson processor after
your ConvertAvroToJson processor, using $[*] as the JSONPath expression.
That should split the array into individual flow files, then you would
proceed as Arpit describes.
Regards,
Matt
On Thu, Apr 7, 2016 at 2:18 PM, Arpit Gupta <ar...@apache.org> wrote:
> Each flowfile should contain a single document that gets written to
> elastic search.
>
> So if each flow file had content like
>
> {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
> "AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue":
> 9.60508652497083E-4}
>
>
> I suspect it will go through. Put Elastic search supports batching where
> multiple flow files will be batched in a single transaction but each flow
> file should only have one document.
>
> --
> Arpit
>
> On Apr 7, 2016, at 10:40 AM, Madhukar Thota <ma...@gmail.com>
> wrote:
>
> Here is the sample JSON output from ConvertAvroToJson Processor:
>
> [{"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
> "AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue":
> 9.60508652497083E-4},
> {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
> "AvgDiskSecTransfer", "InstanceName": "D:", "MetricValue":
> 2.3995189985726E-4},
> {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
> "AvgDiskSecTransfer", "InstanceName": "F:", "MetricValue":
> 2.17140579479747E-4}]
>
>
> On Thu, Apr 7, 2016 at 1:34 PM, Matt Burgess <ma...@gmail.com> wrote:
>
>> Can you provide a sample JSON output from your ConvertAvroToJson
>> processor? It could help identify the location of any mapping/parser
>> exceptions.
>>
>> Thanks,
>> Matt
>>
>> On Thu, Apr 7, 2016 at 1:31 PM, Madhukar Thota <ma...@gmail.com>
>> wrote:
>>
>>> I am able to construct the dataflow with the following processors
>>>
>>> ExecuteSQL--> ConvertAvrotoJson --> Elasticsearch.
>>>
>>> The problem i seeing is elasticsearch unable to index the data because
>>> of the Mapping parser exceptions.
>>>
>>> 13:27:37 EDT
>>> ERROR
>>> fc43fc28-215c-469a-9908-73d04d98d4c2
>>>
>>> PutElasticsearch[id=fc43fc28-215c-469a-9908-73d04d98d4c2] Failed to insert StandardFlowFileRecord[uuid=02af852b-bdf7-452f-a320-b23753c13389,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1460050039787-4636, container=default, section=540], offset=0, length=697677],offset=0,name=1386348391725491,size=697677] into Elasticsearch due to MapperParsingException[failed to parse]; nested: NotSerializableExceptionWrapper[not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes];, transferring to failure
>>>
>>>
>>>
>>>
>>> Am i doing anything wrong here or do i need extra processor to convert
>>> into right format what elasticsearch understands?
>>>
>>>
>>>
>>> On Thu, Apr 7, 2016 at 7:49 AM, Madhukar Thota <madhukar.thota@gmail.com
>>> > wrote:
>>>
>>>> Friends,
>>>>
>>>> I am exploring ExecuteSQL processor in nifi and my goal to get sql data
>>>> ingested in Elasticsearch.
>>>>
>>>> Can someone share or guide what's the flow looks like?
>>>>
>>>>
>>>> Thanks in Advance.
>>>>
>>>
>>>
>>
>
>
Re: ExecuteSQL to elasticsearch
Posted by Arpit Gupta <ar...@apache.org>.
Each flowfile should contain a single document that gets written to elastic search.
So if each flow file had content like
> {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": "AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue": 9.60508652497083E-4}
I suspect it will go through. Put Elastic search supports batching where multiple flow files will be batched in a single transaction but each flow file should only have one document.
--
Arpit
> On Apr 7, 2016, at 10:40 AM, Madhukar Thota <ma...@gmail.com> wrote:
>
> Here is the sample JSON output from ConvertAvroToJson Processor:
>
> [{"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": "AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue": 9.60508652497083E-4},
> {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": "AvgDiskSecTransfer", "InstanceName": "D:", "MetricValue": 2.3995189985726E-4},
> {"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName": "AvgDiskSecTransfer", "InstanceName": "F:", "MetricValue": 2.17140579479747E-4}]
>
>
> On Thu, Apr 7, 2016 at 1:34 PM, Matt Burgess <mattyb149@gmail.com <ma...@gmail.com>> wrote:
> Can you provide a sample JSON output from your ConvertAvroToJson processor? It could help identify the location of any mapping/parser exceptions.
>
> Thanks,
> Matt
>
> On Thu, Apr 7, 2016 at 1:31 PM, Madhukar Thota <madhukar.thota@gmail.com <ma...@gmail.com>> wrote:
> I am able to construct the dataflow with the following processors
>
> ExecuteSQL--> ConvertAvrotoJson --> Elasticsearch.
>
> The problem i seeing is elasticsearch unable to index the data because of the Mapping parser exceptions.
>
> 13:27:37 EDTERRORfc43fc28-215c-469a-9908-73d04d98d4c2
> PutElasticsearch[id=fc43fc28-215c-469a-9908-73d04d98d4c2] Failed to insert StandardFlowFileRecord[uuid=02af852b-bdf7-452f-a320-b23753c13389,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1460050039787-4636, container=default, section=540], offset=0, length=697677],offset=0,name=1386348391725491,size=697677] into Elasticsearch due to MapperParsingException[failed to parse]; nested: NotSerializableExceptionWrapper[not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes];, transferring to failure
>
>
> Am i doing anything wrong here or do i need extra processor to convert into right format what elasticsearch understands?
>
>
>
> On Thu, Apr 7, 2016 at 7:49 AM, Madhukar Thota <madhukar.thota@gmail.com <ma...@gmail.com>> wrote:
> Friends,
>
> I am exploring ExecuteSQL processor in nifi and my goal to get sql data ingested in Elasticsearch.
>
> Can someone share or guide what's the flow looks like?
>
>
> Thanks in Advance.
>
>
>
Re: ExecuteSQL to elasticsearch
Posted by Madhukar Thota <ma...@gmail.com>.
Here is the sample JSON output from ConvertAvroToJson Processor:
[{"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
"AvgDiskSecTransfer", "InstanceName": "C:", "MetricValue":
9.60508652497083E-4},
{"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
"AvgDiskSecTransfer", "InstanceName": "D:", "MetricValue":
2.3995189985726E-4},
{"DateTime": "2016-04-07 17:22:00.0", "HostName": "lab1", "CounterName":
"AvgDiskSecTransfer", "InstanceName": "F:", "MetricValue":
2.17140579479747E-4}]
On Thu, Apr 7, 2016 at 1:34 PM, Matt Burgess <ma...@gmail.com> wrote:
> Can you provide a sample JSON output from your ConvertAvroToJson
> processor? It could help identify the location of any mapping/parser
> exceptions.
>
> Thanks,
> Matt
>
> On Thu, Apr 7, 2016 at 1:31 PM, Madhukar Thota <ma...@gmail.com>
> wrote:
>
>> I am able to construct the dataflow with the following processors
>>
>> ExecuteSQL--> ConvertAvrotoJson --> Elasticsearch.
>>
>> The problem i seeing is elasticsearch unable to index the data because of
>> the Mapping parser exceptions.
>>
>> 13:27:37 EDT
>> ERROR
>> fc43fc28-215c-469a-9908-73d04d98d4c2
>>
>> PutElasticsearch[id=fc43fc28-215c-469a-9908-73d04d98d4c2] Failed to insert StandardFlowFileRecord[uuid=02af852b-bdf7-452f-a320-b23753c13389,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1460050039787-4636, container=default, section=540], offset=0, length=697677],offset=0,name=1386348391725491,size=697677] into Elasticsearch due to MapperParsingException[failed to parse]; nested: NotSerializableExceptionWrapper[not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes];, transferring to failure
>>
>>
>>
>>
>> Am i doing anything wrong here or do i need extra processor to convert
>> into right format what elasticsearch understands?
>>
>>
>>
>> On Thu, Apr 7, 2016 at 7:49 AM, Madhukar Thota <ma...@gmail.com>
>> wrote:
>>
>>> Friends,
>>>
>>> I am exploring ExecuteSQL processor in nifi and my goal to get sql data
>>> ingested in Elasticsearch.
>>>
>>> Can someone share or guide what's the flow looks like?
>>>
>>>
>>> Thanks in Advance.
>>>
>>
>>
>
Re: ExecuteSQL to elasticsearch
Posted by Matt Burgess <ma...@gmail.com>.
Can you provide a sample JSON output from your ConvertAvroToJson processor?
It could help identify the location of any mapping/parser exceptions.
Thanks,
Matt
On Thu, Apr 7, 2016 at 1:31 PM, Madhukar Thota <ma...@gmail.com>
wrote:
> I am able to construct the dataflow with the following processors
>
> ExecuteSQL--> ConvertAvrotoJson --> Elasticsearch.
>
> The problem i seeing is elasticsearch unable to index the data because of
> the Mapping parser exceptions.
>
> 13:27:37 EDT
> ERROR
> fc43fc28-215c-469a-9908-73d04d98d4c2
>
> PutElasticsearch[id=fc43fc28-215c-469a-9908-73d04d98d4c2] Failed to insert StandardFlowFileRecord[uuid=02af852b-bdf7-452f-a320-b23753c13389,claim=StandardContentClaim [resourceClaim=StandardResourceClaim[id=1460050039787-4636, container=default, section=540], offset=0, length=697677],offset=0,name=1386348391725491,size=697677] into Elasticsearch due to MapperParsingException[failed to parse]; nested: NotSerializableExceptionWrapper[not_x_content_exception: Compressor detection can only be called on some xcontent bytes or compressed xcontent bytes];, transferring to failure
>
>
>
>
> Am i doing anything wrong here or do i need extra processor to convert
> into right format what elasticsearch understands?
>
>
>
> On Thu, Apr 7, 2016 at 7:49 AM, Madhukar Thota <ma...@gmail.com>
> wrote:
>
>> Friends,
>>
>> I am exploring ExecuteSQL processor in nifi and my goal to get sql data
>> ingested in Elasticsearch.
>>
>> Can someone share or guide what's the flow looks like?
>>
>>
>> Thanks in Advance.
>>
>
>
Re: ExecuteSQL to elasticsearch
Posted by Madhukar Thota <ma...@gmail.com>.
I am able to construct the dataflow with the following processors
ExecuteSQL--> ConvertAvrotoJson --> Elasticsearch.
The problem i seeing is elasticsearch unable to index the data because of
the Mapping parser exceptions.
13:27:37 EDT
ERROR
fc43fc28-215c-469a-9908-73d04d98d4c2
PutElasticsearch[id=fc43fc28-215c-469a-9908-73d04d98d4c2] Failed to
insert StandardFlowFileRecord[uuid=02af852b-bdf7-452f-a320-b23753c13389,claim=StandardContentClaim
[resourceClaim=StandardResourceClaim[id=1460050039787-4636,
container=default, section=540], offset=0,
length=697677],offset=0,name=1386348391725491,size=697677] into
Elasticsearch due to MapperParsingException[failed to parse]; nested:
NotSerializableExceptionWrapper[not_x_content_exception: Compressor
detection can only be called on some xcontent bytes or compressed
xcontent bytes];, transferring to failure
Am i doing anything wrong here or do i need extra processor to convert
into right format what elasticsearch understands?
On Thu, Apr 7, 2016 at 7:49 AM, Madhukar Thota <ma...@gmail.com>
wrote:
> Friends,
>
> I am exploring ExecuteSQL processor in nifi and my goal to get sql data
> ingested in Elasticsearch.
>
> Can someone share or guide what's the flow looks like?
>
>
> Thanks in Advance.
>