You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@nifi.apache.org by Bobby <bo...@gmail.com> on 2018/05/22 02:51:56 UTC
Put data to Elastic with static settings or index template
Hi, when inserting data to elastic using nifi's processor (putElastic), i
need to apply static setting for the index..like mentioned in
https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html
<https://www.elastic.co/guide/en/elasticsearch/reference/current/index-modules.html>
, this must be applied in index creation..
With the processor, will it be possible to use this utility? I need to do
this in order to save the space...or in other word, changing the compression
type...
As for last resort, i might need to write custom processor extended from
putElastic
Any suggestion?
Thank you
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Put data to Elastic with static settings or index template
Posted by Bobby <bo...@gmail.com>.
Yep, i used index template and it worked well...If my calculation is right,
the size decreased about 12 - 18%..huge gain!!!
Consider this issue solved..
Thank you everyone
-----
-----------------
Bobby
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Put data to Elastic with static settings or index template
Posted by Mike Thomsen <mi...@gmail.com>.
Bobby,
You need to use ElasticSearch templates for this.
https://www.elastic.co/guide/en/elasticsearch/reference/5.4/indices-templates.html
That's the official Elastic-sanctioned way of doing this.
On Tue, May 22, 2018 at 4:23 AM Koji Kawamura <ij...@gmail.com>
wrote:
> Hi Bobby,
>
> Elasticsearch creates index if it doesn't exist.
>
> I haven't tried it myself yet, but Elasticsearch's Index template
> might be useful to tweak default settings for indices those are
> created automatically.
>
> https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
>
> Thanks,
> Koji
>
> On Tue, May 22, 2018 at 3:41 PM, Bobby <bo...@gmail.com> wrote:
> > Siva,
> >
> > In my putElastic processor i only state below properties:
> >
> > <
> http://apache-nifi-developer-list.39713.n7.nabble.com/file/t921/2018-05-22_13_36_48-NiFi.png
> >
> >
> > Given the index name is using expression language, i assume it will be
> > created if it is not exist; In my example, i tend to create new index per
> > day. My team also said, he didn't create index first, the processor take
> > care of it.
> >
> > Thanks
> >
> >
> >
> > -----
> >
> > -----------------
> > Bobby
> > --
> > Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>
Re: Put data to Elastic with static settings or index template
Posted by Koji Kawamura <ij...@gmail.com>.
Hi Bobby,
Elasticsearch creates index if it doesn't exist.
I haven't tried it myself yet, but Elasticsearch's Index template
might be useful to tweak default settings for indices those are
created automatically.
https://www.elastic.co/guide/en/elasticsearch/reference/current/indices-templates.html
Thanks,
Koji
On Tue, May 22, 2018 at 3:41 PM, Bobby <bo...@gmail.com> wrote:
> Siva,
>
> In my putElastic processor i only state below properties:
>
> <http://apache-nifi-developer-list.39713.n7.nabble.com/file/t921/2018-05-22_13_36_48-NiFi.png>
>
> Given the index name is using expression language, i assume it will be
> created if it is not exist; In my example, i tend to create new index per
> day. My team also said, he didn't create index first, the processor take
> care of it.
>
> Thanks
>
>
>
> -----
>
> -----------------
> Bobby
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Put data to Elastic with static settings or index template
Posted by Bobby <bo...@gmail.com>.
Siva,
In my putElastic processor i only state below properties:
<http://apache-nifi-developer-list.39713.n7.nabble.com/file/t921/2018-05-22_13_36_48-NiFi.png>
Given the index name is using expression language, i assume it will be
created if it is not exist; In my example, i tend to create new index per
day. My team also said, he didn't create index first, the processor take
care of it.
Thanks
-----
-----------------
Bobby
--
Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
Re: Put data to Elastic with static settings or index template
Posted by Sivaprasanna <si...@gmail.com>.
Bobby,
If I'm correct, this setting is done during index creation and the
PutElasticsearch processors doesn't create index. It primarily works with
the assumption that the configured index already exists (people correct me,
if I'm wrong). If that's the case, there is no need to do anything on the
NiFi side. Rather while creating the index through ES APIs, you set the
"static" setting. Hope that helps.
-
Sivaprasanna
On Tue, May 22, 2018 at 8:21 AM, Bobby <bo...@gmail.com> wrote:
> Hi, when inserting data to elastic using nifi's processor (putElastic), i
> need to apply static setting for the index..like mentioned in
> https://www.elastic.co/guide/en/elasticsearch/reference/
> current/index-modules.html
> <https://www.elastic.co/guide/en/elasticsearch/reference/
> current/index-modules.html>
> , this must be applied in index creation..
>
> With the processor, will it be possible to use this utility? I need to do
> this in order to save the space...or in other word, changing the
> compression
> type...
>
> As for last resort, i might need to write custom processor extended from
> putElastic
>
>
> Any suggestion?
>
> Thank you
>
>
>
> --
> Sent from: http://apache-nifi-developer-list.39713.n7.nabble.com/
>