You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@nifi.apache.org by Jean-Sebastien Vachon <js...@brizodata.com> on 2019/02/04 21:16:23 UTC
PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi all,
I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index)
Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this...
Supports Expression Language: true (will be evaluated using variable registry only)
It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation?
My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute.
What can I do about this? I don't like monkey patching.. is there any other way to get around this?
Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Posted by Luis Carmona <lc...@openpartner.cl>.
You are welcome.
Good Luck !
LC
De: "Jean-Sebastien Vachon" <js...@brizodata.com>
Para: "Luis Carmona" <lc...@openpartner.cl>, "users" <us...@nifi.apache.org>
Enviados: Lunes, 4 de Febrero 2019 18:45:34
Asunto: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi Luis,
thanks for the hint... this is indeed a good work around. So simple that I should have thought about it \uD83D\uDE09
Regards
From: Luis Carmona <lc...@openpartner.cl>
Sent: Monday, February 4, 2019 4:23 PM
To: users
Subject: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
HI Jean,
I'm not even near to be an expert on NIFI, but I did accomplish to put to work an scenario similar to the one you describe.
I was able to read some data, process it and store the Json payload in ES. I used HTTP invoke, instead of ES Processors.
Hope it helps you.
Regards,
LC
De: "Jean-Sebastien Vachon" <js...@brizodata.com>
Para: "users" <us...@nifi.apache.org>
Enviados: Lunes, 4 de Febrero 2019 18:16:23
Asunto: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi all,
I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index)
Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this...
Supports Expression Language: true (will be evaluated using variable registry only)
It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation?
My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute.
What can I do about this? I don't like monkey patching.. is there any other way to get around this?
Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
Hi Luis,
thanks for the hint... this is indeed a good work around. So simple that I should have thought about it 😉
Regards
________________________________
From: Luis Carmona <lc...@openpartner.cl>
Sent: Monday, February 4, 2019 4:23 PM
To: users
Subject: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
HI Jean,
I'm not even near to be an expert on NIFI, but I did accomplish to put to work an scenario similar to the one you describe.
I was able to read some data, process it and store the Json payload in ES. I used HTTP invoke, instead of ES Processors.
Hope it helps you.
Regards,
LC
________________________________
De: "Jean-Sebastien Vachon" <js...@brizodata.com>
Para: "users" <us...@nifi.apache.org>
Enviados: Lunes, 4 de Febrero 2019 18:16:23
Asunto: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi all,
I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index)
Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this...
Supports Expression Language: true (will be evaluated using variable registry only)
It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation?
My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute.
What can I do about this? I don't like monkey patching.. is there any other way to get around this?
Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Posted by Luis Carmona <lc...@openpartner.cl>.
HI Jean,
I'm not even near to be an expert on NIFI, but I did accomplish to put to work an scenario similar to the one you describe.
I was able to read some data, process it and store the Json payload in ES. I used HTTP invoke, instead of ES Processors.
Hope it helps you.
Regards,
LC
De: "Jean-Sebastien Vachon" <js...@brizodata.com>
Para: "users" <us...@nifi.apache.org>
Enviados: Lunes, 4 de Febrero 2019 18:16:23
Asunto: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Hi all,
I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index)
Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this...
Supports Expression Language: true (will be evaluated using variable registry only)
It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation?
My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute.
What can I do about this? I don't like monkey patching.. is there any other way to get around this?
Thanks
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Posted by Jean-Sebastien Vachon <js...@brizodata.com>.
Thanks all for the feedback.
________________________________
From: Joe Percivall <jp...@apache.org>
Sent: Monday, February 4, 2019 9:59 PM
To: users@nifi.apache.org
Subject: Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
I believe also one of the reasons this was done is because PutElasticsearchHttp takes in batches of FlowFiles and does a bulk insert. In order to support FlowFile attribute expression on the URL, we would have to either only act on one FlowFile at a time or determine another mechanism for handling that ambiguity.
PutElasticsearcHttpRecord on the other hand only takes in a single FlowFile with each onTrigger and could be more easily updated to support that use-case.
Cheers,
Joe
On Mon, Feb 4, 2019 at 5:05 PM Matt Burgess <ma...@apache.org>> wrote:
The restriction to using the variable registry only has always been
there AFAIK, but as of 1.6 we made the distinction in documentation on
how expression language would be evaluated for each property. The
choice was so that we weren't constantly recreating connections for
each flow file, in fact all concurrent tasks share the same underlying
OkHttpClient.
We could probably do something fancier where we allow flowfile
attributes to be evaluated as well, but have a modestly-sized
least-recently-used (LRU) cache of clients, keeping them open until
they are evicted (and closing them all when stopped). Please feel free
to file an improvement Jira and we can discuss further there.
Regards,
Matt
On Mon, Feb 4, 2019 at 4:16 PM Jean-Sebastien Vachon
<js...@brizodata.com>> wrote:
>
> Hi all,
>
> I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index)
> Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this...
>
> Supports Expression Language: true (will be evaluated using variable registry only)
>
>
> It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation?
>
> My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute.
>
> What can I do about this? I don't like monkey patching.. is there any other way to get around this?
>
> Thanks
--
Joe Percivall
linkedin.com/in/Percivall<http://linkedin.com/in/Percivall>
e: jpercivall@apache.com<ma...@apache.com>
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Posted by Joe Percivall <jp...@apache.org>.
I believe also one of the reasons this was done is because
PutElasticsearchHttp takes in batches of FlowFiles and does a bulk insert.
In order to support FlowFile attribute expression on the URL, we would have
to either only act on one FlowFile at a time or determine another mechanism
for handling that ambiguity.
PutElasticsearcHttpRecord on the other hand only takes in a single FlowFile
with each onTrigger and could be more easily updated to support that
use-case.
Cheers,
Joe
On Mon, Feb 4, 2019 at 5:05 PM Matt Burgess <ma...@apache.org> wrote:
> The restriction to using the variable registry only has always been
> there AFAIK, but as of 1.6 we made the distinction in documentation on
> how expression language would be evaluated for each property. The
> choice was so that we weren't constantly recreating connections for
> each flow file, in fact all concurrent tasks share the same underlying
> OkHttpClient.
>
> We could probably do something fancier where we allow flowfile
> attributes to be evaluated as well, but have a modestly-sized
> least-recently-used (LRU) cache of clients, keeping them open until
> they are evicted (and closing them all when stopped). Please feel free
> to file an improvement Jira and we can discuss further there.
>
> Regards,
> Matt
>
> On Mon, Feb 4, 2019 at 4:16 PM Jean-Sebastien Vachon
> <js...@brizodata.com> wrote:
> >
> > Hi all,
> >
> > I was just finishing modifying my flow to make it more reusable by
> having my source document containing information about where to store the
> final document (some Elasticsearch index)
> > Everything was fine until I found out that the PutElasticsearchHttp's
> documentation was saying this...
> >
> > Supports Expression Language: true (will be evaluated using variable
> registry only)
> >
> >
> > It looks like this restriction appeared around Nifi 1.6 (as per the
> documentation)... is there a reason for such a limitation?
> >
> > My current flow was extracting the information from the input JSON
> document and saving the information inside a Flow attribute.
> >
> > What can I do about this? I don't like monkey patching.. is there any
> other way to get around this?
> >
> > Thanks
>
--
*Joe Percivall*
linkedin.com/in/Percivall
e: jpercivall@apache.com
Re: PutElasticsearchHttp can not use Flowfile attribute for ES_URL
Posted by Matt Burgess <ma...@apache.org>.
The restriction to using the variable registry only has always been
there AFAIK, but as of 1.6 we made the distinction in documentation on
how expression language would be evaluated for each property. The
choice was so that we weren't constantly recreating connections for
each flow file, in fact all concurrent tasks share the same underlying
OkHttpClient.
We could probably do something fancier where we allow flowfile
attributes to be evaluated as well, but have a modestly-sized
least-recently-used (LRU) cache of clients, keeping them open until
they are evicted (and closing them all when stopped). Please feel free
to file an improvement Jira and we can discuss further there.
Regards,
Matt
On Mon, Feb 4, 2019 at 4:16 PM Jean-Sebastien Vachon
<js...@brizodata.com> wrote:
>
> Hi all,
>
> I was just finishing modifying my flow to make it more reusable by having my source document containing information about where to store the final document (some Elasticsearch index)
> Everything was fine until I found out that the PutElasticsearchHttp's documentation was saying this...
>
> Supports Expression Language: true (will be evaluated using variable registry only)
>
>
> It looks like this restriction appeared around Nifi 1.6 (as per the documentation)... is there a reason for such a limitation?
>
> My current flow was extracting the information from the input JSON document and saving the information inside a Flow attribute.
>
> What can I do about this? I don't like monkey patching.. is there any other way to get around this?
>
> Thanks